This study introduces a refined method for accurately determining the Invariant Point (IVP) of Very Long Baseline Interferometry (VLBI) antennas through constrained optimization, emphasizing the critical role of axis-offset sign conditions. Optical surveying techniques, commonly used to determine IVPs, inherently involve measurement limitations and observational errors, which may lead to biased estimations of the Azimuth (AZ) and Elevation (EL) axes. To mitigate these biases, we implement physical geometric constraints, including equal-radius conditions for target circles and equal inter-circle distances to ensure consistency across multiple Azimuth positions. Our method specifically incorporates a novel approach for determining the axis-offset sign, which significantly influences VLBI delay estimations. To validate the effectiveness of our method, we conducted numerical simulations using a virtual model with a predefined IVP and axis configuration. Realistic measurement noise was introduced to generate synthetic observational data. Simulation results clearly show that our constrained optimization approach substantially reduces bias and variance in IVP estimation compared to the geometric method proposed in our previous work. Specifically, the proposed method reduced the 3D RMSE of IVP estimation by approximately 40% (e.g., from 1.239 mm to 0.736 mm at 50 m observation distance) and the Interquartile Range (IQR) Error Norm by over 60% (e.g., from 0.949 mm to 0.343 mm at 50 m observation distance). The proposed method’s explicit handling of axis-offset sign conditions demonstrates originality and practical applicability, providing robust and reliable antenna reference point determination. Furthermore, we successfully applied this refined method to the Korean VLBI Network (KVN) Pyeongchang VLBI antenna, demonstrating practical effectiveness in operational geodetic VLBI environments. This advancement contributes to enhanced precision in International Terrestrial Reference Frame (ITRF) realization through improved VLBI station positioning.