Displacement is one of the most fundamental responses, containing useful information regarding dynamic behavior of a structure. Traditional displacement measurement devices such as LVDT have difficulties in using for full-scale civil structures mainly due to the installation inconvenience. The use of laser-based measurements is rapidly increasing; however, the high equipment cost prevents its wider adoption. This study uses a vision-based displacement measurement system as a cost-effective and practical solution for field testing. The vision-based system is used to measure vertical deflections of a railway bridge for high-speed trains to validate its applicability for the bridge deflection measurement.