This paper proposes a deep learning-based crack evaluation technique using hybrid images. The use of the hybrid images combining vision and infrared images are able to improve crack detectability while minimizing false alarms. In particular, large-scale infrastructures can be inspected by an UAV-mounted hybrid image scanning (HIS) system, and the corresponding huge amount of data is typically difficult to be analyzed by experts. To automate such making-decision process, deep convolutional neural network is used in this study. As the very first stage, a lab-scale HIS system is developed using a scanning zig and experimentally validated using a concrete specimen with various-size cracks. The test results reveal that macro- and micro-cracks are successfully and automatically detected with minimizing false-alarms.