The use of radar-based systems for vessel monitoring is not suitable in populated areas, due to the high electromagnetic emissions. In this paper, a camera based vessel recognition system for application in the context of Vessel Traffic Services (VTS) and Homeland Protection (HP) is proposed. Our approach is designed to extend the functionality of traditional VTS systems by permitting the classification of both cooperative and non-cooperative targets, using camera images only. This allows enhancing the surveillance function in populated areas, where public opinion is strongly concerned about electromagnetic emissions and therefore antennas are suspiciously observed and radars are not allowed. Experiments have been carried out on a publicly available data set of images coming from the ARGOS boat traffic monitoring system in the City of Venice (Italy). The obtained classification accuracy of 89.6% (with 11 different classes of boats) demonstrates the effectiveness of the proposed approach.