Displacement is one of the most fundamental responses, containing useful information regarding dynamic behavior of a structure. Traditional displacement measurement devices such as LVDT have disadvantages in their high cost and few options on installation locations. For the sake of economic reason, vision-based displacement measurement systems using low cost cameras have been developed, yet these systems still have difficulties in finding appropriate camera positions; camera should be placed perpendicular to targets. This study presents a new vision-based displacement measurement system using the planar homography method that results in accurate displacement measurement, allowing the camera to face targets with an arbitrary angles. The proposed vision-based system was experimentally validated from a laboratory test.