An Autonomous Vision-Based Target Tracking System for RotorcraftUnmanned Aerial Vehicles

H. Cheng, L. S. Lin, Z. Q. Zheng, Y. W. Guan and Z. C. Liu

Abstract— In this paper, an autonomous vision-based track-ing system is presented to track a maneuvering target for arotorcraft unmanned aerial vehicle (UAV) with an onboardgimbal camera. To handle target occlusions or loss for real-timetracking, a robust and computationally efficient visual trackingscheme is considered using the Kernelized Correlation Filter(KCF) tracker and the redetection algorithm. The states of thetarget are estimated from the visual information. Moreover,feedback control laws of the gimbal and the UAV usingthe estimated states are proposed for the UAV to track themoving target autonomously. The algorithms are implementedon an onboard TK1 computer, and extensive outdoor flightexperiments have been performed. Experimental results showthat the proposed computationally efficient visual trackingscenario can stably track a maneuvering target and is robustto target occlusions and loss.

I. INTRODUCTION

Recently, unmanned aerial vehicles (UAVs) have attractedincreasing attention from both industrial and academic com-munities [1]. Vertical takeoff and landing (VTOL) unmannedrotorcrafts equipped with visual sensors have broad applica-tions including environmental monitoring, rescue and search,surveillance, traffic control, etc [2][3][4].

UAVs that can autonomously track maneuvering targetsare demanding in a wide range of applications. Many re-search efforts have been devoted to autonomous trackingfor UAVs. However, most of current research work focuseseither on visual tracking schemes or on control laws ofUAVs [5][6]. Although various visual tracking algorithmswith high performance have been developed [5], most ofthe algorithms are computationally complicated and notsuitable for real-time tracking of UAVs with limited onboardcomputation capacities. The KCF tracker [7] is applied inour autonomous vision-based tracking system for its com-putational efficiency and impressive performance. In [8][9],visual tracking scenarios of autonomous helicopters havebeen studied via simulations. Several visual algorithms totrack the predefined targets have been implemented on UAVs[10], where simple trackers such as the color and shapefiltering are used for specific target tracking [11][12][13].In [14][15], the OpenTLD [14] and the clustering of staticadaptive correspondences (CMT) trackers robust to trackingdeformable targets are used for AR Drones to track groundtargets autonomously, where the output of the trackers isused for the PD controller and the state estimation of the

*This work is supported by NSFC-Shenzhen Robotics Projects(U1613211), the Fundamental Research Funds for the Central Universities,and Guangdong Natural Science Foundation (1614050001452).

The authors are with the School of Data and Computer Science, SunYat-sen University, China (email: chengh9@mail.sysu.edu.cn).

Fig. 1. An instant of our flight experiments. The upper left corner is animage of the onboard camera, where the green rectangle is the boundingbox indicating the current state of the target.

moving target is not considered. To track a maneuveringtarget precisely and stably, an efficient state estimator isdesirable to estimate the real-time states of the moving target.The gimbal camera is beneficial to alleviate video jiggle andhence improves the stability of the trackers [10]. A visualtracking system using a gimbal camera was proposed in [16],which provides a robust implementation on a helicopter totrack predefined targets.

There are some challenging problems for maneuveringtarget tracking of UAVs: 1) A visual tracker robust to targetocclusions and loss is necessary. 2) Accurate state estimationof the target and closed-loop control laws of the gimbal andthe UAV should be developed to stably track the maneuveringtarget. 3) Due to the limited computational capacities, highlycomputational efficiency of visual tracking, state estimationas well as control algorithms are desirable for onboardimplementation to perform real-time tracking.

These challenging problems motivate us to systematicallyinvestigate vision-based tracking of UAVs considering bothvisual trackers and the control laws. Fig. 1 illustrates ourtracking system, where a quadcopter is tracking a personautonomously using an onboard gimbal camera. The con-tributions of the paper are summarized as follow: 1) In thecase of target occlusions or loss, the status of the target, i.e.loss or not, is firstly detected based on the KCF tracker, anda computationally efficient redetection method is presented.With this scheme, the UAV can track the target againwhen it re-appears. 2) An Interacting Multi-Model ExtendedKalman Filtering (IMM-EKF) based target state estimator ispresented to estimate states of the maneuvering target, anda nonlinear feedback control law is presented to stably trackmoving targets. 3) A computationally efficient frameworkimplemented on onboard TK1 computer is presented for

ground target tracking in unstructured environments.The rest of the paper is organized as follows: In Section

II, we introduce the architecture of the vision-based trackingsystem. In Section III, we present the KCF tracker, target lossdetection and redetection schemes. Section IV addresses thetarget state estimation algorithms. The control of the gimbaland the UAV is presented in Section V. Experimental resultsare presented in Section VI. Concluding remarks and futurework are discussed in Section VII.

II. SYSTEM CONFIGURATION

High-Level Controller

Low-Level Controller (Attitude)

UAV Pose

Target State Estimation

Tracking Mode Switcher

Gimbal Sytem

Servo Controller

Visual Tracking Algorithm

Camerawith Gimbal System

Video Stream

RedetectTarget Lost?

UAV

with

Cam

era

Onb

oard

Com

pute

r

N Y

Fig. 2. Architecture of the vision-based tracking system.

A DJI Matrice100 is used as the UAV platform, which isequipped with an onboard TK1 computer and a monocularRGB gimbal camera. An overview of the system configu-ration is shown in Fig. 2. The gimbal camera mounted onthe UAV platform provides the video stream and internalangles for the onboard computer. The visual tracking al-gorithm obtains position of the target on the image plane,which is feedback to the gimbal controller. In addition,the states of the target are estimated by fusing the inertiameasurement unit (IMU) data of the UAV platform and thegimbal. A switching tracking strategy is performed based onthe estimated states. The high-level controller computes thedesired velocities of the UAV, and the low-level controllercontrols the attitude correspondingly. The frequencies of thevideo stream and the control signal are 30Hz and 10Hz,respectively.

III. A VISUAL TRACKING SCHEME

In this section, a computationally efficient visual trackingscheme robust to target occlusions and loss is presented,which consists of the KCF tracker, the target loss detectionand the redetection methods. Fig. 3 shows the configurationof the visual tracking scheme. Firstly, the KCF trackerestimates the state of the target. The status of the target, i.e.loss or not, is then detected based on the regression functionof the KCF tracker. Finally, a redetection method is presentedto track the target when it come out off ccullsions.

C������ Frame

KCF Tracker

Detector

Target Lost ?

Target Detected ?

Next Frame

Y

N

Y

N

Fig. 3. Architecture of the vision system.

A. KCF Tracker

KCF develops from Circulant Structure of Tracking-by-detection with Kernels (CSK) [17], applying online learningmethods to solve tracking problems. More precisely, it is amachine learning method without any prior knowledge. Atthe first frame, the object of interested (OOI) region is chosenmanually and the KCF tracker transforms the region into amulti-channel HOG feature descriptor. A regression functionf (z) of OOI region z is initialized by Ridge Regression withHOG descriptor. For the new frame, f (z) is evaluated onseveral regions around the last region of OOI. Finally, theregion which has max response of evaluation is consideredas the output and applied to update f (z).

To accelerate the matrix computation of Ridge Regression,KCF transforms each channel of HOG feature descriptorinto a circulant matrix by cyclic shifting. It is known thatcirculant matrix can be made diagonal by Discrete FourierTransform(DFT) [18]. Thus matrix computation, especiallymatrix inversion, can be efficiently processed in fourierdomain. Furthermore, a kernel function, which maps theregression function f (z) into non-linear space, is applied inthe KCF tracker to promote the performance of tracking.These solutions are introduced by CSK, and optimized inKCF. In this way, the process speed and mean precision ofKCF have reached 172FPS and 73.2% respectively. Moredetails of the KCF tracking algorithm may refer to [7].

B. Target Loss Detection and Redetection

Various visual trackers have been proposed to tackle il-lumination variation, scale variation and occlusion problems[19]. However, most of current algorithms can not detectthe target in the presence of full occlusions, which are oftenencountered in tracking mission of UAVs. In this paper, asimple and efficient redetection method is proposed so thatthe target can be estimated when it appears again.

The status of the target, i.e. loss or not, is estimated bythe target loss detection based on the KCF tracker. Themax response of regression function fmax(z), denotes therelevance between the OOI region and the target. When thevalue of fmax(z) is less than a threshold, it implies that the

(a) (b)

(c) (d)

Fig. 4. A moving object detector based on the Frame-Difference method.(a) the original frame; (b) the difference frame with noise; (c) the noiseis removed and the foreground is retained using the Gaussian blur; (d) thebounding boxes constructed by the moving object detector.

target may be lost. The redetection works when fmax(z) isless than a threshold. According to flight experiments, thevalues of fmax(z) vary in the range of (0, 0.5) in outdoorenvironments. The threshold is experimentally set as 0.17.

The UAV hovers and starts to search the target whenthe target loss is detected. Some classical algorithms [14]scan all pixels in the new frame to search the target, andthe computational complexity is high. In general, the targetis moving when it re-appears in the view of the camera.Hence, the target can be estimated by detecting the movingforeground, instead of searching all pixels in the new frame.

In the paper, a moving object detector based on the Frame-Difference (FD) method [20] is applied, as shown in Fig.4. In specific, the detector subtracts the last frame fromthe current frame to obtain the difference image. Althoughthe FD method is computationally efficient, it is sensitiveto noise, as shown in Fig. 4(b). The Gaussian blur ishence applied to remove the noise in the difference image.Consequently, the detector constructs the bounding boxesbased on the center of the foreground, as shown in Fig. 4(d).It is noted that the size of these boxes is the same as the initialOOI region. Finally, the regions contained by these boundingboxes are evaluated by the regression function. The region,which has the maximum value of fmax(z) and is greater thanthe threshold, is selected as the position of the target in thenew frame.

IV. GROUND MANEUVERING TARGET STATEESTIMATION

The states of the ground maneuvering target are estimatedbased on the extended Kalman Filtering.

A. Distance Estimation Method

As shown in Fig. 5, the relative distance between the UAVand the target is estimated.

The distance between the camera and the UAV center isassumed to be neglectable. Let FB denote the body frame ofUAV with axes Xb, Yb and Zb, and FC denotes the camera’s

T0

T

h

d

Xb

Yb Zb

Zc

Xc

Yc

Fb(Fc)

Fig. 5. The relationships between the UAV, the camera and the target.

reference frame with axis Xc, Yc and Zc. The relationshipsbetween the ground target T and the UAV can be shown inFig. 5. Thus, the transformation of a vector from FC to FBcan be represented by a rotation matrix RBC. The target isconsidered as a point T on the ground and is representedby a position vector pB = (xt ,yt ,zt)

T in body frame FB ofUAV. According to standard pinhole imaging model, pB canbe written as:

pB ∼ RBCK−1(u,v,1)T (1)

where the homogenous coordinate (u,v,1)T indicates theposition of the target on the image plane, and K is theintrinsic matrix of the camera. Hence, the relative distancebetween the target and UAV can be calculated by

d =hzt

√x2

t + y2t (2)

where h is the altitude of the UAV.

B. Extended Kalman Filtering

The estimated relative position between UAV and targetgiven by Eq. 2 is generally inaccurate due to observationnoise. To track the maneuvering target stably and precisely,the velocity and acceleration of the target need to be es-timated using appropriate motion models. A single motionmodel is difficult to accurately describe random movementsof the maneuvering target, the IMM-EKF [21] algorithm ispresented to estimate its states by fusing the constant velocitymodel and the current statistical model. In particular, theconstant velocity model can be written in a discrete form:

X(k+1) =[

1 t0 1

]X(k)+

[t2/

2t

]w(k) , (3)

where t is the sampling interval, X is state vector, and w isa discrete white process noise.

The current statistical model in discrete form is given as:

X(k+1) =Φ(k)X(k)+U(k) a(k)+w(k) , (4)

where Φ(k) is the state matrix, U(k) is the control matrix,a(k) is the mean of current maneuvering acceleration, andw(k) is a discrete white process noise. It is noted that theemployed model is a Singer model with an adaptive mean[22], which does not require any prior model and can handletargets with rapidly changing speeds.

V. CONTROLLER

In this section, feedback control laws are presented forboth the gimbal and the UAV with the visual feedbackinformation obtained in Section III and IV. The gimbal andthe UAV are controlled simultaneously to perform stabletracking.

A. Gimbal Controller

A closed-loop controller is developed for controlling thepitch angle and yaw angle of gimbal to keep the target inthe view of the camera. As the video stream is captured in afrequency of 30Hz, we assume that the targets accelerationwould not change too much within the time interval. TheConstant Accelerate(CA) model [22] is applied to forecastthe target’s state in the 2D image coordinate. Let u(k) denotethe targets position along the x axis on the image coordinates.The position u(k+1) at the time steps k+1 can be estimatedby the CA model as: u(k+1)

u(k+1)u(k+1)

=

1 ∆t 12 ∆t2

0 1 ∆t0 0 1

u(k)u(k)u(k)

(5)

where ∆t is the time increment between time step k and k+1.u(k) and u(k) are obtained through computing the differenceof 4 recent frames:{

u(k) = 14∆t (u(k−1)−u(k−3)+u(k−2)−u(k−4))

u(k) = 14∆t2 (u(k−1)−u(k−3)−u(k−2)+u(k−4))

(6)Similarly, the target’s position v(k) along y axis on the

image coordinates can be estimated. A PD controller is usedfor the gimbal system to compute its angular velocities inthe 2D image plane, i.e, eα(k) = u(k+1)−u0 and eβ (k) =v(k+1)−v0, where (uo,vo) is the center point of the image.

B. Switchable Tracking Strategy

To perform smooth and stable tracking of a maneu-vering target, a switchable tracking strategy consisting ofthe observing mode and the following mode is presentedconsidering the relative distance d. Switching between thetwo modes is determined by the thresholds dmin and dmax.The thresholds are calculated by a certain range of the pitchangle [θ1,θ2] and the real-time altitude of the UAV:{

dmax = h · tanθ2dmin = h · tanθ1

(7)

We use θ1 = 20◦ and θ2 = 70◦ for the thresholds, becausewitnin this range our relative distance estimator has betterresults which will be analyzied in Section VI-C.

1) Observing Mode: In the observing mode, the UAVonly adjusts its yaw angle to the target and there is no hori-zontal displacement. When the relative distance d satisfiesdmin ≤ d ≤ dmax, the observing mode works. In addition,when the targets acceleration a estimated from the IMM-EKF, the UAVs maximum acceleration amax and the currentvelocity vp of the UAV satisfy the following conditions: |a|>|amax| and a ·vp < 0, which implies that the target’s relativedistance tends to decrease to the condition of d ≤ dmax. The

UAV rotates to track the ground target while maintaining acertain height and position. A PID controller is utilized tocontrol the UAVs yaw angle.

2) Following Mode: In the following mode, the movingof UAV and the rotation of the gimbal are controlled simul-taneously to maintain the yaw angle of the gimbal in theUAVs body frame close to zero. When the relative distanced ≤ dmin and d ≥ dmax, the following mode works.

In the following mode, the nonlinear velocity controller ofthe UAV is proposed based on the Lyapunov theory. Withoutloss of generality, we assume that the UAV maintains aconstant altitude during the tracking process. The targettracking flight control is hence simplified to a planar controlproblem. The relationship between the UAV and the targetin the XY plane will be considered as shown in Fig. 6.

d

vt

σd

σt

O

Xb

Yb

T

xd

yd

Fig. 6. Relationship between the aircraft and the target in the UAV’s bodyframe XbOYb.

Let d and σd be the target’s relative position and relativeyaw angle estimated from the IMM-EKF. In the followingmode, we expect that the UAV would maintains a certaindistance to the target. Therefore, we define the distance errorand angular error as follows:{

εd = d−E(d)εσ = σd,

(8)

whereE(d) =

{dmin d ≤ dmindmax d > dmax

. (9)

It is noted that the relative position and d and relativeyaw angle σd can also be formulated as d =

√xd

2 + yd2 and

σd = arctan(yd/xd), then the dynamic error of distance εdand dynamic error of yaw angle εσ can be written as:{

εd = d =−vx cosεσ − vy sinεσ + vd cos(σt−σd)

εσ = σd =sinεσ

d vx− cosεσ

d vy +sin(σt−σd)

d vd(10)

where vx and vy are velocities of the UAV in the body frame,vt and σt are velocity and its yaw angle of the target estimatedfrom the IMM-EKF with respect to the body frame of theUAV.

We seek to control the velocity vx, vy and angular velocityω of the UAV to ensure the distance error εd and angularerror εσ converge to zero. The control law of the UAV is

designed as: vx = k1εd cosεσ + vd cos(σt−σd)cosεσ

vy = k1εd sinεσ + vd cos(σ t−σd)sinεσ

ω = k2εσ + vdd sin(σt−σd)

(11)

The stability of the feedback control system can be provedusing Lyapunovs second theory. The Lyapunov functioncandidate can be formulated as:

V1 =12(εd

2 + εσ2) (12)

Note that, V1 ≥ 0 and V1 = 0 if and only if [ εd εσ ]T =[ 0 0 ]T. The time derivative of V1 can be written as:

V1 = εd εd + εσ εσ

= vpx

(−εd cosεσ + εσ

sinεσ

d

)+vpy

(−εd sinεσ − εσ

cosεσ

d

)+vdεd cos(σv−σd)+ vdεσ

sin(σv−σd)d

(13)

By substituting vx,vy and ω into (13), we simplify the timederivative of V1 as:

V1 =−k1εd2− k2εσ

2. (14)

Equation (14) ensures that V1 ≤ 0, while k1,k2 ≥ 0. AndV1 = 0, if and only if [ εd εσ ]T = [ 0 0 ]T. Thus, thecontrol system is asymptotically stable with the designedcontrol law.

VI. EXPERIMENTAL VALIDATION

To verify the vision-based ground target tracking system,extensive flight experiments were performed in outdoor en-vironments. The performance of the visual tracking schemewith the redetection method is firstly evaluated on severalvideos collected using the UAV platform. The IMM-EKFbased state estimation of the maneuvering person is thenexperimentally verified and analyzed. Finally, experimentalresults of the visual tracking system in real-world are pre-sented. The computation is fully implemented on the onboardTK1 computer in the experiments.

A. Experimental Setup

Onboard Computer

Camera with Gimbal

Fig. 7. The experimental test-bed: DJI Matrice 100 UAV with onboardcomputer and monocular RGB gimbal camera system.

A DJI Matrice 100 UAV is used as an experimental test-bed, as shown in Fig. 7. Onboard equipments include aDJI Manifold embedded Linux computer (NVIDIA Tegra

TK1 processor consisting of a Kepler GPU with 192 CUDAcores, a NVIDIA 4-Plus-1 quad-core A15 CPU of 1.5 GHz,2 GB RAM and a WiFi interface), a monocular ZenmuseX3 gimbal camera, a GPS receiver, an Inertial MeasurementUnit (IMU), a barometer, and one downward-pointing sensormodule of a DJI Guidance visual sensing system.

B. Visual Tracking Test

The performance of the visual tracking scheme is eval-uated on several videos collected using the UAV. Threecases are considered, i.e., no occlusion, partial occlusion andfull occlusion. Full occlusion for a long term occurs in allvideos, and the target losses for at least 3s (about 90 frames).The traditional KCF fails to track the targets correctlyin the case of full occlusion. The visual tracking schemeconsisting of the KCF tracker and the redetection method isevaluated. For videos of resolution, the KCF tracker and theredetection method achieve 30FPS and 10FPS, respectively.The accuracy of the visual tracking scheme is measured byPaccuracy =

NtdNt

. The experimental results of the visual trackingscheme are summarized in Table I. Table I indicates thatthe proposed visual tracking scheme can effectively trackand redetect the target. The accuracy decreases when thebackground is complex or more variations are introduced.

Test No. Ns Nt Ntd Situation Accuracy

1 660 332 312 SB 94.87%2 930 872 838 SB 96.10%3 1169 1019 964 SB 94.60%4 1349 1222 1033 CB 84.53%5 2212 1881 1609 CB 86.07%6 840 601 459 IV,CB 76.37%7 1049 863 712 IV,CB 82.50%

TABLE I: Visual tracking results while the target with occlusion in differentscenarios. Ns is the total number of frames in each test, Nt is the number offrames containing the target, Ntd is the number of frames where the targetis correctly tracked and redetected. SB: Simple Background; CB: ComplexBackground; IV: Illumination Variation.

C. Evaluation of the Target State Estimation Test

Experiments were conducted to evaluate the accuracy ofthe target state estimation, where the measured values areused as the ground truth. The relative error is calculatedby δd = ∆d

dtrue= dest−dtrue

dtrue, where dest is the relative position

estimated by the IMM-EKF based on estimator, and dtrue isthe measured relative position. To examine the effects of thepitch and the yaw angles of the gimbal on the state estimationprecision, the relative position between the target and theUAV is fixed. Fig. 8 shows the relative distance estimationerrors δd with respect to the pitch and the yaw angles ofthe gimbal, where the greatest/smallest errors occur in thered/blue area.

Fig. 8 indicates that the accuracy of the target stateestimation decreases along with the increasing of the pitchangles. The following reasons may lead to the decreasingestimation accuracy: 1) lens distortion; 2) deviation of cal-culating actual distance in image edge; 3) inherent noiseof the gimbal camera. In general, the results show that the

Fig. 8. The pseudocolor plot of relative distance estimation errors (%).The test is under a certain range of the gimbal’s angle: −60◦ to −60◦ ofyaw angle and 20◦ to 70◦ of pitch angle. The distribution of the error canbe seen from the legend on the right side of the figure

target state estimation achieves acceptable performance in acertain range between 3% and 8% of the gimbals attitudeangles. In addition, the yaw angle between −20◦ and 20◦

and pitch angle between 30◦ and 55◦ achieves smallestestimated errors. In summary, the yaw and the pitch angle ofgimbal should be controlled within a certain range to provideacceptable localization accuracy.

D. Target Following Control Simulations

In this subsection, the simulation is presented to comparethe performance between the proposed nonlinear control lawand the PID controller.

In the simulations, the UAV is represented by a rigid body-mass point and the parameters of the UAV were set as thefollowing: the angular velocity of the yaw angle is 90 deg/s,and the max velocity of the UAV is 6m/s. The trajectoryof target is generated by a set of random way-points withvarying velocities and accelerations. The switchable trackingstrategy is considered both in proposed nonlinear controllerand the PID control law. The simulation results of the targettracking control are shown in Fig.9.

40

30

20

10

0

-5 0 5 10 15

Y (

m)

X (m)

-10

Target

Lyapunov

PID

20 25 30 35 40

(a)

0 10 20 30 40

−5

0

5

Time (s)

Lyapunov PID

0 10 20 30 40 −1

0

1

Time (s)

(b)

Fig. 9. The UAV tracks a maneuvering target with random movements.(a)trajectories; (b)velocities.

Fig.9(a) shows that the switchable tracking strategy cansteadily track the target while moves smoothly. It is notedthat the proposed nonlinear controller achieves similar per-formance compared to the PID controller under the sameswitchable tracking strategy. Fig. 9(b) illustrates that thecomputed velocity of the nonlinear controller is smoother

than the PID controller, that implies unnecessary movementsof the UAV can be reduced.

One of critical concerns of the control system is the adjust-ment of the controller’s parameters. There are nine parame-ters in the PID algorithm, and it is quite difficult to adjust theparameters to achieve satisfactory control performance. Incontrast, there are only two parameters to be adjusted in theproposed nonlinear controller, the two parameters can be setin a relatively wide range. The nonlinear controller is hencemore conveniently implemented than the PID controller inreal-world applications.

E. Target Tracking Flight Test

To systematically investigate the performance of the pro-posed vision-based tracking scenario, extensive flight ex-periments were performed for autonomous target trackingusing fully onboard computation. An un-predefined personis selected manually as the target. The UAV tracks themaneuvering person in unstructured outdoor environments.The flying altitude of the UAV is set to a height of 5 meters.The trajectory of the target person is not pre-set.

0 20 40 60 80 1000

10

20

30

40

50

60

70

80

m

m

trajectory of the UAVtrajectory of the target

A

(a)

0 20 40 60 80 100 120 140 160 180−5

0

5

time (s)

velo

city

(m

/s)

velocity Xvelocity Y

A

(b)

Fig. 10. (a) Trajectories of the UAV(red) and the target(blue). Their positioninformation is derived from the GNSS; (b)UAV’s of the x direction and ydirection velocities in the body coordinates

The flight results are shown in Fig. 10, where the totaldistance of the tracking process is 330 meters with about 3minutes. The trajectories of the target person and the UAVare obtained from the mobile phone’s GPS data and theUAV’s onboard GPS data, respectively. Fig. 10(a) shows thatthe UAV can maintain a certain distance to the target around5 meters, and can successfully track the maneuvering targetperson.

It is noted from Fig. 11(a) that the trajectory of theUAV is more stable than that of the ground target. It takesthe advantages of the proposed switchable target trackingstrategy, where the observation mode works when the targetperson frequently changes its direction. In this case, the

65 70 75 80

10

12

14

16

18

20

22

m

m

trajectory of the UAVtrajectory of the target

(a)

110 115 120 125−4

−2

0

2

4

time (s)

velo

city

(m

/s)

velocity Xvelocity Y

(b)

Fig. 11. (a) Partial enlarged views of box A in Fig. 10(a), showing theexperimental results of the switchable tracking strategy of UAVs when thetarget frequently changes its direction. (b)Partial enlarged views of box Ain Fig. 10(b)

UAV rotates to track the target person without horizontaldisplacement, which can be shown from the Fig. 11(b). Theexperimental results illustrate the UAV can stably track themaneuvering target person using the proposed vision-basedtracking scenario.

A demo video of our target tracking UAV system inoutdoor environments can be seen in https://youtu.be/7dc4etU0IHs, working entirely onboard without priorknowledge of the target.

VII. CONCLUSIONS

In this paper, the design and implementation of an au-tonomous vision-based ground target tracking system forrotorcraft UAV are presented. A visual tracking schemeintegrating the KCF tracker and the redetection method wasdesigned, which is robust to target occlusion and loss. TheIMM-EKF based estimator is presented to estimate the statesof the maneuvering target. The gimbal and the UAV arecontrolled simultaneously to perform stable tracking. Thegimbal camera is controlled to keep the target in the center ofthe frame. Moreover, a switchable tracking strategy includingthe following mode and the observing mode is proposed forUAVs, and a Lyapunov theory-based nonlinear controller wasdesigned. Extensive real-time flight experiments have beenexecuted in outdoor environments, where the computation isfully implemented on the onboard computer. Experimentalresults illustrate that the proposed real-time vision-basedtracking system achieves stable and robust tracking perfor-mance. Future research will consider autonomous vision-based obstacle avoidance and object grasping of UAVs.

REFERENCES

[1] K. P. Valavanis and G. J. Vachtsevanos, Handbook of unmanned aerialvehicles. Springer Publishing Company, Incorporated, 2014.

[2] R. He, A. Bachrach, M. Achtelik, A. Geramifard, D. Gurdan, S. Pren-tice, J. Stumpf, and N. Roy, “On the design and use of a micro airvehicle to track and avoid adversaries,” The International Journal ofRobotics Research, 2009.

[3] G. Bevacqua, J. Cacace, A. Finzi, and V. Lippiello, “Mixed-initiativeplanning and execution for multiple drones in search and rescuemissions,” in Twenty-Fifth International Conference on AutomatedPlanning and Scheduling, 2015.

[4] K. Kanistras, G. Martins, M. J. Rutherford, and K. P. Valavanis,“Survey of unmanned aerial vehicles (uavs) for traffic monitoring,” inHandbook of Unmanned Aerial Vehicles. Springer, 2015, pp. 2643–2666.

[5] A. W. Smeulders, D. M. Chu, R. Cucchiara, S. Calderara, A. Dehghan,and M. Shah, “Visual tracking: An experimental survey,” IEEE Trans-actions on Pattern Analysis and Machine Intelligence, vol. 36, no. 7,pp. 1442–1468, 2014.

[6] R. Mahony, V. Kumar, and P. Corke, “Multirotor aerial vehicles,” IEEERobotics and Automation magazine, vol. 20, no. 32, 2012.

[7] J. F. Henriques, R. Caseiro, P. Martins, and J. Batista, “High-speedtracking with kernelized correlation filters,” IEEE Transactions onPattern Analysis and Machine Intelligence, vol. 37, no. 3, pp. 583–596,2015.

[8] H. Deng, X. Zhao, and Z. Hou, “Tracking ground targets using anautonomous helicopter with a vision system,” in Image and SignalProcessing (CISP), 2010 3rd International Congress on, 2010, pp.1704 – 1708.

[9] A. Razinkova and H.-C. Cho, “Tracking a moving ground objectusing quadcopter uav in a presence of noise,” in Advanced IntelligentMechatronics (AIM), 2015 IEEE International Conference on. IEEE,2015, pp. 1546–1551.

[10] F. Lin, X. Dong, B. M. Chen, K.-Y. Lum, and T. H. Lee, “A robust real-time embedded vision system on an unmanned rotorcraft for groundtarget following,” IEEE Transactions on Industrial Electronics, vol. 59,no. 2, pp. 1038–1049, 2012.

[11] C. Teuliere, L. Eck, and E. Marchand, “Chasing a moving target from aflying uav,” in Intelligent Robots and Systems (IROS), 2011 IEEE/RSJInternational Conference on. IEEE, 2011, pp. 4929–4934.

[12] A. G. Kendall, N. N. Salvapantula, and K. A. Stol, “On-board objecttracking control of a quadcopter with monocular vision,” in UnmannedAircraft Systems (ICUAS), 2014 International Conference on. IEEE,2014, pp. 404–411.

[13] A. Chakrabarty, R. Morris, X. Bouyssounouse, and R. Hunt, “Au-tonomous indoor object tracking with the parrot ar. drone,” in Un-manned Aircraft Systems (ICUAS), 2016 International Conference on.IEEE, 2016, pp. 25–30.

[14] J. Pestana, J. L. Sanchez-Lopez, S. Saripalli, and P. Campoy, “Com-puter vision based general object following for gps-denied multirotorunmanned vehicles,” in American Control Conference (ACC), 2014.IEEE, 2014, pp. 1886–1891.

[15] M. A. Olivares-Mendez, P. Campoy, C. Martınez, and I. Mondragon,“A pan-tilt camera fuzzy vision controller on an unmanned aerial ve-hicle,” in Intelligent Robots and Systems, 2009. IROS 2009. IEEE/RSJInternational Conference on. IEEE, 2009, pp. 2879–2884.

[16] K. Krishnamoorthy, D. Casbeer, and M. Pachter, “Minimum timeuav pursuit of a moving ground target using partial information,” inUnmanned Aircraft Systems (ICUAS), 2015 International Conferenceon. IEEE, 2015, pp. 204–208.

[17] J. F. Henriques, R. Caseiro, P. Martins, and J. Batista, Exploiting thecirculant structure of tracking-by-detection with kernels, 2012.

[18] R. M. Gray et al., “Toeplitz and circulant matrices: A review,” Foun-dations and Trends R© in Communications and Information Theory,vol. 2, no. 3, pp. 155–239, 2006.

[19] Y. Wu, J. Lim, and M.-H. Yang, “Object tracking benchmark,” IEEETransactions on Pattern Analysis and Machine Intelligence, vol. 37,no. 9, pp. 1834–1848, 2015.

[20] C. Zhan, X. Duan, S. Xu, Z. Song, and M. Luo, “An improvedmoving object detection algorithm based on frame difference andedge detection,” in Image and Graphics, 2007. ICIG 2007. FourthInternational Conference on. IEEE, 2007, pp. 519–523.

[21] L. A. Johnston and V. Krishnamurthy, “An improvement to theinteracting multiple model (imm) algorithm,” IEEE Transactions onSignal Processing, vol. 49, no. 12, pp. 2909–2923, 2001.

[22] X. R. Li and V. P. Jilkov, “Survey of maneuvering target tracking.part i. dynamic models,” IEEE Transactions on Aerospace ElectronicSystems, vol. 39, no. 4, pp. 1333–1364, 2004.