In the shaking table test, contact displacement sensors are widely being applied to a structure for measuring the displacement responses, but difficulties may arise depending on the condition of the structure. Moreover, the test uneconomically needs temporary facilities for installing sensors and supplementary equipment for saving the displacement responses of a structure. The contact displacement sensors that are currently becoming popular in the shaking table tests are limited depending on their use environments; thus, the reality is that each equipment appears impractical considering its effectiveness and applicability. Therefore, in this study, to verify an algorithm for measuring the displacement responses of a structure using the vision-based system in the shaking table test, the various tests were conducted.