Recently there was an incident that military radars, coastal CCTVs and other surveillance equipment captured a small rubber boat smuggling a group of illegal immigrants into South Korea, but guards on duty failed to notice it until after they reached the shore and fled. After that, the detection of such vessels before it reach to the Korean shore has emerged as an important issue to be solved. In the fields of marine navigation, Automatic Identification System (AIS) is widely equipped in vessels, and the vessels incessantly transmits its position information. In this paper, we propose a method of automatically identifying abnormally behaving vessels with AIS using convolutional autoencoder (CAE). Vessel anomaly detection can be referred to as the process of detecting its trajectory that significantly deviated from the majority of the trajectories. In this method, the normal vessel trajectory is gridded as an image, and CAE are trained with images from historical normal vessel trajectories to reconstruct the input image. Features of normal trajectories are captured into weights in CAE. As a result, images of the trajectories of abnormal behaving vessels are poorly reconstructed and end up with large reconstruction errors. We show how correctly the model detects simulated abnormal trajectories shifted a few pixel from normal trajectories. Since the proposed model identifies abnormally behaving ships using actual AIS data, it is expected to contribute to the strengthening of security level when it is applied to various maritime surveillance systems.