This study assessed how fruit size and cultivation region affect the distribution of carotenoids and flavonoids in sweet persimmon (Diospyros kaki Thunb.). We quantified concentrations in the whole fruit, peel, and pulp, and compared them across different size grades and cultivation regions. The fruit size did not significantly influence the levels of carotenoids or flavonoids, suggesting a limited direct relationship with phytochemical accumulation. Notably, the peel consistently contained the highest concentrations of the compounds analyzed. Regional comparisons revealed significant variability. In the whole fruit, flavonoids—except for quercetin 3-O-galactoside (Q3Gal)—were most concentrated in Gwangyang. In the peel, Q3Gal, quercetin 3-O-glucoside (Q3Glu), and zeaxanthin were most abundant in Naju–Yeongam, while β-carotene peaked in Gimhae–Changwon. To identify environmental influences, we examined climatic variables and found that a larger diurnal temperature range during the dormancy stage of fruit development correlated positively with higher flavonoid levels. Overall, these findings indicate that the functional components of sweet persimmon are primarily influenced by the growing region—potentially due to local weather conditions—rather than by fruit size. Additionally, consuming persimmons with their peel may enhance the intake of beneficial plant-derived compounds.