In the current steel structures of high-rise buildings, high heat input welding techniques are used to improve productivity in the construction industry. Under the high heat input welding, however, the microstructures of the weld metal (WM) and heat-affected zone (HAZ) coarsen, resulting in the deterioration of impact toughness. This study focuses mainly on the effects of fine TiN precipitates dispersed in steel plates and B addition in welding materials on grain refinement of the HAZ microstructure under submerged arc welding (SAW) with a high heat input of 200 kJ/cm. The study reveals that, different from that in conventional steel, the γ grain coarsening is notably retarded in the coarse grain HAZ (CGHAZ) of a newly developed steel with TiN precipitates below 70 nm in size even under the high heat input welding, and the refinement of HAZ microstructure is confirmed to have improved impact toughness. Furthermore, energy dispersive spectroscopy (EDS) and secondary-ion mass spectrometry (SIMS) analyses demonstrate that B is was identified at the interface of TiN in CGHAZ. It is likely that B atoms in the WM are diffused to CGHAZ and are segregated at the outer part of undissolved TiN, which contributes partly to a further grain refinement, and consequently, improved mechanical properties are achieved.