This paper proposes real-time image-based damage detection method for concrete structures using deep learning. The proposed method is composed of three steps: (1) collection of a large volume of images containing damage information from internet, (2) development of a deep learning model (i.e., convolutional neural network (CNN)) using collected images, and (3) automatic selection of damage images using the trained deep learning model. The whole procedure of the proposed method has been applied to some figures taken in a real structure. This method is expected to facilitate the regular inspection and speed up the assessment of detailed damage distribution the without losing accuracy.