How galaxy evolution diers in dierent environments is one of the intriguing questions in the study of structure formation. While galaxy properties are clearly distinguished in different environments in the local universe, it is still an open issue what causes this environmental dependence of various galaxy properties. To address this question, in this work, we investigate the build-up of passive galaxies over a wide redshift range, from z ~ 2 to z ~ 0.5, focusing on its dependence on galaxy environment. In the UKIDSS/Ultra Deep Survey (UDS) field, we identify high-redshift galaxy cluster candidates within this redshift range. Then, using deep optical and near-infrared data from Subaru and UKIRT available in this field, we analyze and compare the stellar population properties of galaxies in the clusters and in the field. Our results show that the environmental effect on galaxy star-formation properties is a strong function of redshift as well as stellar mass | in the sense that (1) the effect becomes signicant at small redshift, and (2) it is stronger for low-mass (M* < 1010M⊙) galaxies. We have also found that galaxy stellar mass plays a more significant role in determining their star-formation property — i.e., whether they are forming stars actively or not — than their environment throughout the redshift range.