Although the concept of “common sense” is often taken for granted, judging whether behavior or knowledge is common sense requires a complex series of mental processes. Additionally, different perceptions of common sense can lead to social conflicts. Thus, it is important to understand how we perceive common sense and make relevant judgments. The present study investigated the dynamics of neural representations underlying judgments of what common sense is. During functional magnetic resonance imaging, participants indicated the extent to which they thought that a given sentence corresponded to common sense under the given perspective. We incorporated two different decision contexts involving different cultural perspectives to account for social variability of the judgments, an important feature of common sense judgments apart from logical true/false judgments. Our findings demonstrated that common sense versus non-common sense perceptions involve the amygdala and a brain network for episodic memory recollection, including the hippocampus, angular gyrus, posterior cingulate cortex, and ventromedial prefrontal cortex, suggesting integrated affective, mnemonic, and social functioning in common sense processing. Furthermore, functional connectivity multivariate pattern analysis revealed that interactivity among the amygdala, angular gyrus, and parahippocampal cortex reflected representational features of common sense perception and not those of non-common sense perception. Our study demonstrated that the social memory network is exclusively involved in processing common sense and not non-common sense. These results suggest that intergroup exclusion and misunderstanding can be reduced by experiencing and encoding long-term social memories about behavioral norms and knowledge that act as common sense of the outgroup.