As various types of information technology systems are becoming more pervasive than ever, many studies have evaluated the systems from the user perspective. Some of them have used surveys to measure consumers’ cognitive responses to the target technology. However, this method may cause problems if the survey participants do not have a useful frame of reference for evaluating an unfamiliar system. To examine this issue, the current study empirically tested the effect of personal product knowledge on the predictability of a behavioral model, such as Technology Acceptance Model. A series of measurement invariance tests as well as multi-group comparison tests were conducted for rigorous examination of the data. Our analysis showed that the variance of attitude that is explained by the two believes (perceived usefulness and ease of use) was relatively small when the survey respondents had lower amount of product knowledge. Moreover, the group had weaker causal relationship between attitude and intention to use the technology, hindering the predictability of the research model. The results indicated that respondents should have a certain amount of knowledge of the target system in order to form accurate beliefs and behavioral decisions. The findings of this study provide important implications on sampling strategies for researchers with new technology.