This paper proposes a simple and intuitive model-free torque-tracking control for rotary electro-hydraulic actuators. The undesirable natural-velocity-feedback effect is discussed by introducing mechanical impedance into the electro-hydraulic actuation system. The proposed model-free torque control comprises inner- and outer-loop control to achieve two control objectives. Inner-loop control reduces the mechanical impedance passively and optimally. To improve the tracking accuracy, a certain form of proportional-integral-derivative control is applied to the outer loop. The robustness of the proposed closed-loop system against external disturbances is demonstrated by transforming the two-loop control structure into a disturbance observer form. The proposed method is validated on a single joint electro-hydraulic actuator.