This study investigates differences in Chinese receptive vocabulary size and perceived lexical difficulty among three groups of Korean university students: Chinese majors, liberal arts students studying Chinese, and students with no formal exposure to Chinese. A total of 156 participants completed a 47-item Vocabulary Level Test (VLT) based on HSK levels 1–5. Data were analyzed using one-way ANOVA and Item Response Theory (IRT). The results revealed significant group differences: Chinese majors demonstrated the highest lexical competence, followed by liberal arts students and then non-learners. Notably, non-learners achieved a mean accuracy rate of 38.30%, likely due to the influence of Sino-Korean cognates. IRT analysis further indicated that Chinese majors perceived the items as relatively “easy,” liberal arts students as “difficult,” and non-learners as “very difficult.” These findings underscore the critical role of linguistic and cultural familiarity—particularly knowledge of Chinese characters—in facilitating vocabulary comprehension. The study advocates for differentiated instructional strategies tailored to learners’ backgrounds and calls for future research involving students from non-Sinographic language backgrounds to enhance the generalizability of the findings.