A video-based non-contact remote displacement measurement system is proposed and developed. The system captures images of the structure and analyzes the image to determine the displacement. Multi-thread computing and adaptive region of interest method is implemented in the system to compute the displacement in real-time. Field test on a real structure has been carried out to evaluate the performance and verify the reliability of the system.