This study examined the intended CEFR level, passage familiarity, and learner variation in a locally developed elementary EFL reading assessment. Responses from 42 Korean fifth graders to eight passages and 32 items were analyzed within a 2 × 2 design crossing intended CEFR level (A1/A2) and familiarity conditions. Rasch modeling revealed that the intended A1–A2 distinction was reflected in average item difficulty, though substantial overlap indicated that CEFR-based text design did not determine individual item difficulty. Repeated measures ANOVA showed higher accuracy for A1 and familiar passages, with no interaction between level and familiarity. Item-level binomial GLMM analysis indicated that familiarity effects differed across ability groups: the lower-ability group declined under less-familiar conditions, while the higher-ability group remained stable. Questionnaire indicators showed no statistically supported associations with reading ability. Overall, findings suggest CEFR-referenced assessment is valuable when the intended level is compared with actual item functioning and learner performance. The study advances CEFR use from linguistic calibration to response-based, learnersensitive interpretation of reading performance.