Potholes, one of the main causes of road-surface damage, pose a physical hazard to drivers, cause vehicle damage, and increase road maintenance costs. Hence, a model that enhances the accuracy of pothole detection and improves the real-time detection speed is required. A new model based on dilated convolutional neural networks was developed using a dataset that considers various lighting conditions, road conditions, and pothole sizes and shapes. Although the existing YOLOv5 model demonstrated high speed, it exhibited some false-positive pothole detections. In contrast, the proposed dilated convolutional neural network achieved both high accuracy and an appropriate inference speed, making it suitable for real-time detection. Compared with traditional models, the proposed model demonstrated efficiency in terms of model size and inference speed, indicating its potential suitability for systems performing real-time pothole detection when installed directly in vehicles.