IMPROVING THE MARITIME TRAFFIC SITUATION ASSESSMENT FOR A SINGLETARGET IN A MULTISENSOR ENVIRONMENT

Gregor Siegert, Paweł Banys and Frank Heymann

Department of Nautical SystemsGerman Aerospace Center (DLR)

Neustrelitz, GermanyEmail: gregor.siegert@dlr.de

ABSTRACTExploiting the diversity of multiple on-board sensors is apromising approach to generate a reliable picture of the traf-fic situation in the vicinity of a particular vessel. This workfocuses on multi-sensor fusion for single target tracking in aloosely-coupled architecture. An Interacting Multiple ModelMulti-Sensor Probabilistic Data Association filter is designedto capture rapidly changing vessel dynamics in the presenceof possible clutter measurements. The actual target trackingis made up of two Unscented Kalman filters each being condi-tioned on radar and AIS measurement updates. The benefitsof the proposed method will be demonstrated on behalf ofreal-world measurements obtained from the Baltic Sea.

Index Terms— AIS, IMM-MSPDA filter, UKF, radar im-age processing, sensor fusion, single target tracking

1. INTRODUCTION

The increasing challenges of the maritime traffic domain callfor advanced solutions to guarantee safety at sea. Nearly 80 %of the global trade traverses the seas and harbors worldwide(see [1]) stressing the vital economic interests in secure andefficient shipping. Key aspect to all mariners, traffic manage-ment and security authorities is a reliable and timely pictureof the traffic situation not only in their close vicinity but alsowith respect to vessels in greater distance. For better iden-tification and localization of maritime traffic participants theAutomatic Identification System (AIS) was introduced by theInternational Maritime Organization (IMO) as an ITU-R rec-ommendation [2] in 2004, yielding a mandatory standard forvessels greater than 300GRT. AIS can be understood as ad-ditional sensor that supports the use of classical surveillancetechniques for collision avoidance, e.g., radar, that are usedaboard or in shore-based Vessel Traffic Service (VTS) moni-toring stations. However, none of the available sensors, nei-ther AIS or radar, can constantly provide sufficient data ontheir own to establish a reliable and accurate traffic picture atall times. While radar may detect vessels invisible in AIS, it isin general less accurate and will always be subject to external

weather phenomena that may result in false echos or cluttermeasurements. On the contrary, AIS yields great precision ofvessel positions, but entirely relies on the cooperative natureof the system. With its open standard AIS is vulnerable to aseries of threats, such as availability disruption, ship spoof-ing or AIS hijacking, as discussed in [3]. Apart from that,unintentional misuse or imperfect equipment may introduceadditional error sources compromising the reliability of thesystem, as was also shown in a comprehensive AIS plausibil-ity analysis in [4]. To encounter these shortcomings, we pro-pose to fuse both, radar and AIS, to establish a more accurateand reliable traffic picture by exploiting the complementarynature of the two sensors. In the literature various approacheshave been published to augment maritime surveillance or col-lision avoidance systems, mostly based on radar target fusionwith additional sensors like laser in [5] or multiple radar sys-tems for exploiting aspect diversity as in [6]. The matter ofAIS and radar fusion was mainly addressed for anomaly de-tection, e.g., based on multi hypothesis tests in [7] or by ex-ploiting historical traffic route knowledge for SAR/AIS fusionin [8]. In [9] an overview was given for different AIS/radarfusion techniques incorporating online covariance estimation.

The remainder of this article is structured as follows. Insection 2 the general methodology for single target tracking ina radar/AIS environment will be outlined. Section 3 demon-strates the working principle of the proposed scheme w.r.t.measurement data. A conclusion is given in section 4.

2. METHODOLOGY

In this section the proposed methodology for fusing radar andAIS data for single target tracking will be presented in moredetail. By designing an Interacting Multiple Model (IMM)Multi-Sensor Probabilistic Data Association (MSPDA) filterthat is conditioned on asynchronous radar and AIS measure-ments a loosely-coupled architecture was chosen.

(a) Original radar image. (b) Image after background subtractionand gray-scale conversion.

(c) Extracted target candidates (red cir-cles) at time k after blob detection.

Fig. 1: Processing chain for one radar image at time k to extract the target candidates.

2.1. Radar image based target extraction

In order to fuse radar with AIS position data, the target can-didates need to be detected and extracted from radar first, tofeed them to the filter as measurement updates. The utilizedapproach to extract radar target information is based on im-age processing instead of directly working on the radar signallevel. This may introduce additional error sources originat-ing from mapping the radar signal to image domain, but alsoyields the advantage of applying the proposed technique tomost commercial radar systems by simply interfacing to thevideo output. To extract target candidates from the currentradar image at time k, the following procedure is applied:

1. Masking the image eliminating static but undesired fea-tures, e.g., colored heading lines, blob in center, radarinformation tables.

2. Conversion of image from RGB to gray-scale (weightedaverage from color channels).

3. Blob detection with fixed range settings for convexity,circularity, inertia, size and intensity of expected tar-gets.

4. Each detected target candidate per frame is expressedin range and bearing, relative to the radar’s, i.e., ship’s,position.

The key aspect in this processing chain is certainly thescale-invariant blob detection to eventually detect target can-didates. This algorithm is well described in literature andfinds many applications in image based target detection andtracking such as described in [10]. For this work the imple-mentation provided by the OpenCV framework was used1.Figures 1a to 1c show the different radar processing stages.

2.2. AIS dynamic target data

The typical AIS data set contains numerous static and dy-namic parameters, that are distributed over different AIS

1OpenCV 3.1.0: https://github.com/Itseez/opencv.git

message types and specified in the ITU-R recommenda-tion [2]. The set of dynamic parameters always comprisesthe vessel position in longitude and latitude, course overground (COG) and speed over ground (SOG), but may alsocontain true heading and rate of turn (ROT) information. Thespecified time intervals between successive messages rangefrom 2 s to 180 s, depending on the dynamic state of the ves-sel. As was shown in [4] these reporting rates are violated ina considerable amount of cases, leading to outdated or simplymissing AIS messages.

2.3. IMM-MSPDA framework for single target tracking

In this work, an IMM-MSPDA filter was designed for singletarget tracking in an AIS/radar environment. The IMM, be-ing first proposed in [11], is generally applied to best capturerapidly changing motion dynamics by running a bank of in-teracting Kalman filters in parallel, with each filter being con-ditioned on a different process model. The final IMM stateestimate as well as the re-initialization of the Kalman filtersafter each iteration is based on a weighted combination ofthe individual state estimates, whereas the transition betweenthe models (or modes) is governed by an underlying Markovprocess. The combination with a Probabilistic Data Associ-ation (PDA) filter yields a powerful scheme for associatingclutter measurements to the expected target state in a dynam-ically challenging scenario. The basic steps of the PDA filterare comprehensively described in [12]. Essentially, each sen-sor measurement gets validated based on a validation regioncentered around the expected state of the target. The finalstate update is then based on the weighted sum of the resid-uals between validated and expected measurements, with theweights being computed from the likelihood of the measure-ment to origin from the target. In contrast to the standardPDA approach in [12] we apply Unscented Kalman Filtering(UKF) (see [15]) to compensate especially for nonlinearitiesin the radar measurement domain.

An algorithm combining both approaches to form anIMM-PDA filter in a multi-sensor environment was originally

proposed in [13], outlining a scheme to combine synchronousmeasurement updates from 2 to 3 sensors sequentially. Anextension to incorporate multiple sensors providing asyn-chronous or delayed measurements was published in [14]. Inour work, the latter is adopted to the particular scenario ofobserving high rate radar measurements and low rate AIS up-dates, both running asynchronously. In contrast to the originalalgorithm, in our implementation the standard IMM cycle iscontinued on arrival of any sensor measurement. Otherwise,if low rate AIS messages would solely trigger the update ofthe IMM model probabilities, the IMM could not adopt tochanging motion dynamics as quickly as if radar measure-ments were also used for initiating the model probabilityupdate of the IMM cycle.

2.4. UKF filter design

For the actual target tracking an Unscented Kalman fil-ter (UKF) was designed incorporating state augmentationby the process noise during state prediction and additive cor-rection steps for each of the sensors. Details on the basic ideaof the unscented transform as well as the implementationbased on state augmentation can be found in [15]. In ourparticular application the UKF was found to outperform theExtended Kalman filter (KF) (EKF) in the presence of highlynonlinear radar measurement updates, as was already dis-cussed in [6] and [16]. In the context of vessel dynamics twodominant motion scenarios were identified, that are nearlystraight-path and turn-maneuver based motion. For that rea-son, two process models were defined, namely the ConstantVelocity (CV) and the Constant Turn Rate Velocity (CTRV),assuming the former to provide best fit to straight-path andthe latter to turn-maneuver motion respectively. Further de-tails on the definition of CV and CTRV process models canbe found in [17].

Within each filter hat implements one of the modes fromabove, the predicted state xk|k−1 and its associated covari-ance will be corrected based on measurements of sensor s ∈{radar, ais}. The corresponding measurement models are ex-pressed as functions hs(xk|k−1, ε

sk), with

hs(xk|k−1, εsk) =

[xk|k−1, yk|k−1

]T+ εsk (1)

for s = ais and

hs(xk|k−1,εsk) =[√

(xk|k−1 − xs)2 + (yk|k−1 − ys)2

arctan(

yk|k−1−ys

xk|k−1−xs

) ]+ εsk

(2)

for s = radar, mapping the target position from state to radarmeasurement domain. In that context, (xs, ys) denotes theradar reference position and

(xk|k−1, yk|k−1

)the predicted

position in the target’s local ENU frame respectively. The

vector εsk ∼ N(0,Rs) captures the additive sensor measure-ment noise.

Careful attention has to be paid to the interaction of mod-els with state spaces of different dimensions within the IMMcycle. In this work the strategy from [18] is followed, whichis based on state augmentation. In this context, the extra el-ement from the CTRV state space is essentially replicated toobtain a combined IMM state estimate.

3. RESULTS

In this section the proposed algorithm for fusing AIS withradar in an IMM-MSPDA filter shall be evaluated based on adynamically challenging measurement scenario.

3.1. Baltic Sea experiments

Fig. 3: Nautical chart depicting the area of the measurementcampaign at the Baltic Sea, zooming into the selected testtrajectory. The bottom right picture shows the vessel to betracked.

For validating the proposed method a dedicated measure-ment campaign with two chartered vessels was conducted inOctober 2015. The offshore supply ship BALTIC TAUCHERII was conducting sea trial maneuvers for two successive daysin the Baltic Sea (see Fig. 3). Its transmitted AIS messageswere recorded at a shore-based AIS station at the Darßer OrtLighthouse, Germany2. Additionally, this ship was equippedwith a multi-frequency GNSS receiver, that allowed for com-putation of a PPP reference trajectory in post-processing. Asecond ship, the tug vessel AARON remained anchored in thecenter of the sea trial area, monitoring the scenery by radar atan interval of 1Hz. With this scenario the feasibility of theproposed method for maritime situation awareness w.r.t. to asingle target shall be demonstrated. For the validation of the

2Courtesy of German Federal Waterways and Shipping Administration(WSV)

−2 −1 0

·103

0

1

2

·103

ENU-E [m]

EN

U-N

[m]

radar target candidatesreferenceIMM-PDA

(a) Trajectory of tracked vessel based onradar data only, running an IMM-PDAonly.

−2 −1 0

·103

0

1

2

·103

ENU-E [m]

EN

U-N

[m]

AIS position datareferenceIMM-PDA

T2

T1

(b) Trajectory of tracked vessel based onAIS data only, running an IMM-PDAfilter.

−2 −1 0

·103

0

1

2

·103

ENU-E [m]

EN

U-N

[m]

radar target candidatesreferenceIMM-MSPDA

(c) Trajectory of tracked vessel, fusingradar and AIS data, running an IMM-MSPDA filter.

Fig. 2: Comparison of filtered vessel trajectories from two IMM-PDA filters conditioned on either radar or AIS alone and anIMM-MSPDA filter fusing both sources.

proposed filter, the subset highlighted in Fig. 3 was selecteddue to its two distinct turn maneuvers, covering 1708 s or 201valid AIS messages respectively.

3.2. Evaluation

For evaluation and to demonstrate the potential benefits ofthe proposed scheme, three different filters were tested. Atfirst, an IMM-PDA filter was conditioned on plain radar tar-get candidate data. Secondly, the AIS messages from thesame track were used as sole input to this filter. Figures 2aand 2b show the filtered trajectory in comparison to the ref-erence and original measurement updates. Thirdly, the pro-posed IMM-MSPDA filter was tested with both asynchronoussensor measurement updates. The trajectory obtained fromthis fusion process is shown in Fig. 2c. As can also be seen inTable 1, the filter being conditioned on radar image data onlycan not compete in terms of accuracy to filtered AIS positiondata. However, while the filter running on low rate AIS mes-sages is introducing a large position error during the secondturn maneuver (at label T2 in Fig. 2b) due to missing AISmessages radar can still be used for tracking as it providescontinues measurement updates. By fusing both sensors thefiltered trajectory overpasses smoothly the lack of AIS mes-sages during the turn maneuver, while it is mainly followingAIS updates otherwise. In this particular case, the maximumerror in the estimated target position was drastically reducedfrom nearly 236m to below 56m.

In Table 1 prominent statistics for the three different fil-ters are listed stressing the performance improvement fromthe proposed IMM-MSPDA filter in terms of maximum andRMS error. It is not surprising that the σ-value of the er-

Table 1: Statistics of the horizontal position error for the threedifferent filters.

mean σ(68.27%)

RMSE max.

IMM-PDAAIS only

9.6m 3.2m 36.9m 235.7m

IMM-PDARadar only

18.3m 19.1m 22.3m 75.8m

IMM-MSPDA 8.9m 7.1m 14.8m 55.6m

ror distribution, i.e., the value which bounds 68.27% of theerrors, is increasing for the fused process compared to thefiltered trajectory conditioned on AIS data only. Due to thehigh rate radar measurements more uncertainty is inferred tothe filter in times where AIS messages would actually suffice.

4. CONCLUSION

In this work, an IMM-MSPDA framework was utilized to ex-ploit the complementary nature of radar and AIS sensors in aloosely-coupled data fusion architecture. The overall aim isto provide a more robust picture of the traffic situation in thevicinity of a particular vessel, resilient to AIS faults or anoma-lies. Based on real-world measurements the benefits of theproposed scheme could be visualized for cases of missing orinsufficient AIS message updates. In future work this frame-work will be extended for multiple target tracking includingtrack initialization based on candidate extraction from radar.

5. REFERENCES

[1] United Nations, World Economic Situation andProspects 2012, chapter 2, pp. 41–66, United Nationspublication, 2012.

[2] ITU Radiocommunication Sector (ITU-R), “Techni-cal characteristics for an automatic identification sys-tem using time division multiple access in the VHF mar-itime mobile band.,” Recommendation M.1371-5, ITU,February 2014.

[3] Marco Balduzzi, Alessandro Pasta, and Kyle Wilhoit,“A Security Evaluation of AIS Automated IdentificationSystem,” in Proceedings of the 30th annual computersecurity applications conference, ASAC, New Orleans,LA, USA, December 2014.

[4] Frank Heymann, Thoralf Noack, and Paweł Banys,“Plausibility analysis of navigation related AIS param-eter based on time series,” in ENC, Vienna, Austria,2013.

[5] Lokukaluge P. Perera, Victor Ferrari, Fernando P. San-tos, Miguel A. Hinostroza, and Carlos Guedes Soares,“Experimental Evaluations on Ship Autonomous Nav-igation and Collision Avoidance by Intelligent Guid-ance,” IEEE JOURNAL OF OCEANIC ENGINEER-ING, vol. 40, APRIL 2015.

[6] Paolo Braca, Michele Vespe, Salvatore Maresca, andJochen Horstmann, “A Novel Approach to High Fre-quency Radar Ship Tracking Exploiting Aspect Diver-sity,” Geoscience and Remote Sensing Symposium(IGARSS), 2012 IEEE International, pp. 6895 – 6898,2012.

[7] Marco Guerriero, Peter Willett, Stefano Coraluppi, andCraig Carthel, “Radar/AIS Data Fusion and SAR task-ing for Maritime Surveillance,” in International Confer-ence on Information Fusion, 2008, vol. 11th.

[8] Fabio Mazzarella and Michele Vespe, “SAR Ship De-tection and Self-Reporting Data Fusion Based on Traf-fic Knowledge,” IEEE GEOSCIENCE AND REMOTESENSING LETTERS, April 2015.

[9] Witold Kazimierski and Andrzej Stateczny, “Radarand Automatic Identification System Track Fusion inan Electronic Chart Display and Information System,”THE JOURNAL OF NAVIGATION, , no. 68, pp. 1141–1154, 2015.

[10] Michael Isard and John MacCormick, “BraMBLe: ABayesian multiple-blob tracker,” in Eighth IEEE Inter-national Conference on Computer Vision, 2001, vol. 2,pp. 34–41.

[11] Henk A. P. Blom and Yaakov Bar-Shalom, “The In-teracting Multiple Model Algorithm for Systems withMarkovian Switching Coefficients,” IEEE Transactionson Automatic Control, vol. 33, 1988.

[12] Yaakov Bar-Shalom, Fred Daum, and Jim Huang, “TheProbabilistic Data Association Filter,” IEEE CONTROLSYSTEMS MAGAZINE, December 2009.

[13] Yaakov Bar-Shalom, Fred Daum, and Jim Huang, “Mul-tisensor Tracking of a Maneuvering Target in Clutter,”IEEE TRANSACTIONS ON AEROSPACE AND ELEC-TRONIC SYSTEMS, vol. AES-25, March 1989.

[14] Soonho Jeong and Jitendra K. Tugnait, “MultisensorTracking of a Maneuvering Target in Clutter with Asyn-chronous Measurements Using IMMPDA Filtering andParallel Detection Fusion,” in Proceeding of the 2004American Control Conference, Boston, Massachusetts,2004.

[15] Simon J. Julier and Jeffrey K. Uhlmann, “A NewExtension of the Kalman Filter to Nonlinear Sys-tems,” in Proc. of AeroSense: The 11th Int. Symp. onAerospace/Defence Sensing, Simulation and Controls.,1997, pp. 182–193.

[16] Zhansheng Duan and X. Rong Li, “Sequential Un-scented Kalman Filter for Radar Target Tracking withRange Rate Measurements,” in In Proc. 2005 Interna-tional Conf. on Information Fusion, 2005.

[17] Gregor Siegert, Paweł Banys, Cristina Saez Martınez,and Frank Heymann, “EKF Based Trajectory Trackingand Integrity Monitoring of AIS Data,” in IEEE/IONPosition, Location and Navigation Symposium - PLANS,Savannah, GA, April 2016, IEEE.

[18] John D. Glass, W. D. Blair, and Yaakov Bar-Shalom,“IMM Estimators with Unbiased Mixing for TrackingTargets Performing Coordinated Turns,” ProceedingsIEEE Aerospace Conference, 2013.