The warm recycling technology has been increasingly used in many countries due to the environmental and financial benefits. In this study, the rheological and fatigue performance evolutions of warm-mix recycled asphalt materials during the secondary service period were evaluated in two scales, mixture and fine aggregate matrix (FAM). A laboratory simulation method was proposed to produce warm-mix recycled asphalt binders with various long-term aging levels for the mixture and FAM tests. The dynamic shear rheometer temperature and frequency sweep test and time sweep test were conducted to characterize the rheological and fatigue behavior of FAMs, respectively. The rheological and fatigue properties of asphalt mixtures were measured by the dynamic modulus test and semi-circular bending test, respectively. Effects of aging levels and recycling plans on different pavement performance were investigated. Performance correlations between the mixture and FAM were finally investigated by the statistical method. It is found that the secondary long-term aging causes the continuous increase in the stiffness and decrease in the viscoelasticity level in each material scale, indicating the improvement of the rutting resistance and the reduction of the fatigue resistance. The warm mix asphalt technology plays a positive role in the fatigue performance with a loss of the rutting resistance. Using the styrene butadiene rubber latex can improve different pavement performance within the whole time-temperature domain. Good performance correlations between the mixture and FAM are developed, indicating that the FAM may be the critical material scale for evaluating the rheological and fatigue performance of warm-mix recycled asphalt pavements.