Thermal property represents a critical metric when evaluating the performance of next generation nuclear graphite. Despite the extensive measurement data available, a detailed investigation into the influence of microstructure on graphite’s thermal conductivity remains underexplored. In this work, taking advantage of the distinct microstructures between different graphite grades, a comparative study of four graphite grades was conducted to elucidate the structure–property relationship. The microstructures of graphite were characterized by Raman spectroscopy and X-ray diffraction techniques, demonstrating specimen preparation induced damage and annealing induced restoration. Thermal properties were investigated across multiple scales using laser flash analysis and photothermal radiometry. The results indicate that despite similar densities, thermal conductivity varies significantly between different grades and correlates positively with crystallite sizes. By interpolating an infinitely large crystallite and removing the impact of macroscale porosity, an upper bound for the thermal conductivity of isotropic defect-free nuclear graphite has been established.