Supplementary Material Need for Speed: A Benchmark for...
5
Supplementary Material Need for Speed: A Benchmark for Higher Frame Rate Object Tracking Hamed Kiani Galoogahi 1* , Ashton Fagg 2* , Chen Huang 1 , Deva Ramanan 1 and Simon Lucey 1 1 Robotics Institute, Carnegie Mellon University 2 SAIVT Lab, Queensland University of Technology {hamedk,chenh2}@andrew.cmu.edu [email protected] {deva,slucey}@cs.cmu.edu 1. Introduction This supplementary material provides additional details of the Need for Speed (NfS) dataset. The full dataset and benchmark, including all the videos, frames, annotations, Gyro and IMU raw data, evaluation codes and results (in mat files and video demos) are publicly available at http: //ci2cv.net/nfs/index.html. Frame samples of NfS: The NfS dataset consists of 100 videos. The first frame of each video is shown in Fig. 1 and Fig. 2. The target of interest is highlighted by a blue bounding box. Per tracker evaluation: Fig. 3 compares tracking higher versus lower frame rate videos (success plots) for each eval- uated method. These results are summarized by Fig. 3 (suc- cess rate at IoU > 0.50) in the main manuscript. Here, we illustrate success plots of all tracker over all overlapping thresholds. For lower frame rate tracking (30 FPS) results are reported for both with and without motion blur. AUCs are reported in the legend. This more detailed evaluation shows that all trackers achieve a significant improvement on tracking higher frame rate videos, compared to lower frame rate videos. Moreover, this evaluation shows that all track- ers are fairly robust to the presence of motion blur in lower frame rate videos. Attribute description: All 9 attributes annotated in NfS are described in Table. 1. For attribute based evaluation, please see the main manuscript. Evaluated methods: All methods evaluated in the main manuscript are summarized in Table. 2. Updated learning rates: Here, we mathematically show why we selected to update learning rate as LR new = 1 8 LR old for higher frame tracking (240 FPS). Online adaptation in all CF tracker are, generally, per- formed by updating current visual model of the target at frame f +1 as x f +1 model = x f model + ⌘x f +1 , where ⌘ is the learning rate, x f +1 model is the updated appearance model, Table 1. Attributes and their detailed description. Attr Description IV Illumination Variation - the illumination in the target region changes significantly. SV Scale Variation - the ratio of the bounding boxes of the first frame and the current frame is out of the range [1/ts, ts], ts >1 (ts=2). OCC Occlusion - the target is partially or fully occluded. DEF Deformation - non-rigid object deformation. FM Fast Motion - the motion of the ground truth is larger than tm pixels (tm=20) 1 . VC Viewpoint Change - viewpoint change caused by in-plane rotation, out-plane rotation and camera movement changes target appearance significantly. OV Out-of-View - the target is partially or fully out of the view. BC Background Clutters - the target and its surrounding background share similar color or texture. LR Low Resolution - the number of pixels inside the ground-truth bounding box is less than 400. x f model is the appearance model at the previous frame, x f +1 is the object appearance (e.g. HOG features) in current frame (i+1), and f =0, ..., F - 1 [3, 11, 6]. F is the num- ber of frames. x 0 model for the first frame is initialized as x 0 model = x 0 . The adaptation formulation at frame f +1 can be expended as: x f +1 model = x 0 + ⌘ f +1 X i=1 x i (1) The number of frames over a fixed period of time (e.g.1 second) at 240 FPS videos is 8 times more than that at 30 FPS videos. Thus, to retain the amount of visual informa- tion used to update the model in higher frame rate videos to be same as that in lower frame rate videos, we (approxi- mately) need to divide the learning rate by 8, meaning that ⌘ new = 1 8 ⌘ old . 1
Transcript of Supplementary Material Need for Speed: A Benchmark for...
Supplementary MaterialNeed for Speed: A Benchmark for Higher Frame Rate Object Tracking
Hamed Kiani Galoogahi1*, Ashton Fagg2*, Chen Huang1, Deva Ramanan1 and Simon Lucey1
1Robotics Institute, Carnegie Mellon University2SAIVT Lab, Queensland University of Technology
{hamedk,chenh2}@andrew.cmu.edu [email protected] {deva,slucey}@cs.cmu.edu
1. IntroductionThis supplementary material provides additional details
of the Need for Speed (NfS) dataset. The full dataset andbenchmark, including all the videos, frames, annotations,Gyro and IMU raw data, evaluation codes and results (inmat files and video demos) are publicly available at http://ci2cv.net/nfs/index.html.
Frame samples of NfS: The NfS dataset consists of 100videos. The first frame of each video is shown in Fig. 1and Fig. 2. The target of interest is highlighted by a bluebounding box.
Per tracker evaluation: Fig. 3 compares tracking higherversus lower frame rate videos (success plots) for each eval-uated method. These results are summarized by Fig. 3 (suc-cess rate at IoU > 0.50) in the main manuscript. Here, weillustrate success plots of all tracker over all overlappingthresholds. For lower frame rate tracking (30 FPS) resultsare reported for both with and without motion blur. AUCsare reported in the legend. This more detailed evaluationshows that all trackers achieve a significant improvement ontracking higher frame rate videos, compared to lower framerate videos. Moreover, this evaluation shows that all track-ers are fairly robust to the presence of motion blur in lowerframe rate videos.
Attribute description: All 9 attributes annotated in NfS aredescribed in Table. 1. For attribute based evaluation, pleasesee the main manuscript.
Evaluated methods: All methods evaluated in the mainmanuscript are summarized in Table. 2.
Updated learning rates: Here, we mathematically showwhy we selected to update learning rate as LR
new
=18LRold
for higher frame tracking (240 FPS).Online adaptation in all CF tracker are, generally, per-
formed by updating current visual model of the target atframe f + 1 as x
f+1model
= x
f
model
+ ⌘x
f+1, where ⌘ isthe learning rate, xf+1
model
is the updated appearance model,
Table 1. Attributes and their detailed description.Attr Description
IV Illumination Variation - the illumination in the targetregion changes significantly.
SVScale Variation - the ratio of the bounding boxes ofthe first frame and the current frame is out of therange [1/ts, ts], ts >1 (ts=2).
OCC Occlusion - the target is partially or fully occluded.DEF Deformation - non-rigid object deformation.
FM Fast Motion - the motion of the ground truth is largerthan tm pixels (tm=20)1.
VCViewpoint Change - viewpoint change caused byin-plane rotation, out-plane rotation and cameramovement changes target appearance significantly.
OV Out-of-View - the target is partiallyor fully out of the view.
BC Background Clutters - the target and its surroundingbackground share similar color or texture.
LR Low Resolution - the number of pixels inside theground-truth bounding box is less than 400.
x
f
model
is the appearance model at the previous frame, xf+1
is the object appearance (e.g. HOG features) in currentframe (i+1), and f = 0, ..., F � 1 [3, 11, 6]. F is the num-ber of frames. x
0model
for the first frame is initialized asx
0model
= x
0. The adaptation formulation at frame f + 1can be expended as:
x
f+1model
= x
0 + ⌘
f+1X
i=1
x
i (1)
The number of frames over a fixed period of time (e.g. 1second) at 240 FPS videos is 8 times more than that at 30FPS videos. Thus, to retain the amount of visual informa-tion used to update the model in higher frame rate videosto be same as that in lower frame rate videos, we (approxi-mately) need to divide the learning rate by 8, meaning that⌘
new
= 18⌘old.
1
Figure 1. Sample frames of NfS videos
References[1] L. Bertinetto, J. Valmadre, S. Golodetz, O. Miksik, and
P. H. S. Torr. Staple: Complementary learners for real-timetracking. In CVPR, June 2016. 5
[2] L. Bertinetto, J. Valmadre, J. F. Henriques, A. Vedaldi, andP. H. Torr. Fully-convolutional siamese networks for objecttracking. In ECCV, pages 850–865, 2016. 5
[3] D. S. Bolme, J. R. Beveridge, B. A. Draper, and Y. M. Lui.
Visual object tracking using adaptive correlation filters. InCVPR, pages 2544–2550. IEEE, 2010. 1
[4] M. Danelljan, G. Hager, F. Khan, and M. Felsberg. Accuratescale estimation for robust visual tracking. In BMVC, 2014.5
[5] M. Danelljan, G. Hager, F. Shahbaz Khan, and M. Felsberg.Learning spatially regularized correlation filters for visualtracking. In ICCV, pages 4310–4318, 2015. 5
Figure 2. Sample frames of NfS videos - cont.
[6] M. Danelljan, A. Robinson, F. S. Khan, and M. Felsberg.Beyond correlation filters: Learning continuous convolutionoperators for visual tracking. In ECCV, pages 472–488,2016. 1
[7] D. Held, S. Thrun, and S. Savarese. Learning to track at 100fps with deep regression networks. In ECCV, 2016. 5
[8] J. F. Henriques, R. Caseiro, P. Martins, and J. Batista. High-speed tracking with kernelized correlation filters. PAMI,
37(3):583–596, 2015. 5[9] H. Kiani Galoogahi, A. Fagg, and S. Lucey. Learning
background-aware correlation filters for visual tracking. InICCV, 2017. 5
[10] H. Kiani Galoogahi, T. Sim, and S. Lucey. Multi-channelcorrelation filters. In ICCV, pages 3072–3079, 2013. 5
[11] H. Kiani Galoogahi, T. Sim, and S. Lucey. Correlation filterswith limited boundaries. In CVPR, pages 4630–4638, 2015.
BACF SRDCF Staple
0 0.2 0.4 0.6 0.8 1Overlap threshold
0
20
40
60
80
100
Succ
ess
rate
(%) 240 FPS [49.56]
30 FPS-no MB [35.26]30 FPS-MB [34.08]
0 0.2 0.4 0.6 0.8 1Overlap threshold
0
20
40
60
80
100
Succ
ess
rate
(%) 240 FPS [47.13]
30 FPS-no MB [35.79]30 FPS-MB [35.14]
0 0.2 0.4 0.6 0.8 1Overlap threshold
0
20
40
60
80
100
Succ
ess
rate
(%) 240 FPS [45.34]
30 FPS-no MB [34.51]30 FPS-MB [33.26]
LCT DSST SAMF
0 0.2 0.4 0.6 0.8 1Overlap threshold
0
20
40
60
80
100
Succ
ess
rate
(%) 240 FPS [34.38]
30 FPS-no MB [24.88]30 FPS-MB [23.77]
0 0.2 0.4 0.6 0.8 1Overlap threshold
0
20
40
60
80
100
Succ
ess
rate
(%) 240 FPS [44.80]
30 FPS-no MB [29.46]30 FPS-MB [28.05]
0 0.2 0.4 0.6 0.8 1Overlap threshold
0
20
40
60
80
100
Succ
ess
rate
(%) 240 FPS [43.92]
30 FPS-no MB [30.32]30 FPS-MB [29.26]
KCF CFLB HCF
0 0.2 0.4 0.6 0.8 1Overlap threshold
0
20
40
60
80
100
Succ
ess
rate
(%) 240 FPS [33.34]
30 FPS-no MB [22.33]30 FPS-MB [21.77]
0 0.2 0.4 0.6 0.8 1Overlap threshold
0
20
40
60
80
100
Succ
ess
rate
(%) 240 FPS [19.92]
30 FPS-no MB [14.95]30 FPS-MB [14.27]
0 0.2 0.4 0.6 0.8 1Overlap threshold
0
20
40
60
80
100
Succ
ess
rate
(%) 240 FPS [39.50]
30 FPS-no MB [30.20]30 FPS-MB [29.54]
HDT MEEM MDNet
0 0.2 0.4 0.6 0.8 1Overlap threshold
0
20
40
60
80
100
Succ
ess
rate
(%) 240 FPS [47.80]
30 FPS-no MB [41.38]30 FPS-MB [40.32]
0 0.2 0.4 0.6 0.8 1Overlap threshold
0
20
40
60
80
100
Succ
ess
rate
(%) 240 FPS [37.53]
30 FPS-no MB [32.92]30 FPS-MB [29.65]
0 0.2 0.4 0.6 0.8 1Overlap threshold
0
20
40
60
80
100
Succ
ess
rate
(%) 240 FPS [47.34]
30 FPS-no MB [44.40]30 FPS-MB [42.90]
SFC FCNT GOTURN
0 0.2 0.4 0.6 0.8 1Overlap threshold
0
20
40
60
80
100
Succ
ess
rate
(%) 240 FPS [47.78]
30 FPS-no MB [42.32]30 FPS-MB [40.14]
0 0.2 0.4 0.6 0.8 1Overlap threshold
0
20
40
60
80
100
Succ
ess
rate
(%) 240 FPS [46.94]
30 FPS-no MB [40.57]30 FPS-MB [39.74]
0 0.2 0.4 0.6 0.8 1Overlap threshold
0
20
40
60
80
100
Succ
ess
rate
(%) 240 FPS [38.65]
30 FPS-no MB [37.79]30 FPS-MB [33.41]
Figure 3. Comparing higher frame rate tracking (240 FPS) versus lower frame rate tracking (30 FPS) for each tracker. For higher framerate tracking CF trackers are performed by updated learning rate. The results of lower frame rate tracking are plotted for both videos withand without motion blur (30 FPS-MB and 30 FPS- no MB). AUCs are reported in brackets.
Table 2. Evaluated methods.Tracker Learning FeatureBACF [9] CF HOGSRDCF [5] CF HOGStaple [1] CF + colo scores HOG + colorLCT [14] CF + random ferns HOGDSST [4] CF HOGSAMF [12] CF HOG + Color NamesKCF [8] CF HOGCFLB [10] CF pixel valuesHCF [13] CF deep featureHDT [16] CF + Hedge Algo. deep featureMEEM [18] SVM colorMDNet [15] CNN deep featureSiameseFc [2]CNN deep featureFCNT [17] CNN deep featureGOTURN [7] CNN deep feature
1[12] Y. Li and J. Zhu. A scale adaptive kernel correlation filter
tracker with feature integration. In ECCV, pages 254–265,2014. 5
[13] C. Ma, J.-B. Huang, X. Yang, and M.-H. Yang. Hierarchicalconvolutional features for visual tracking. In CVPR, pages3074–3082, 2015. 5
[14] C. Ma, X. Yang, C. Zhang, and M.-H. Yang. Long-termcorrelation tracking. In CVPR, pages 5388–5396, 2015. 5
[15] H. Nam and B. Han. Learning multi-domain convolutionalneural networks for visual tracking. 2016. 5
[16] Y. Qi, S. Zhang, L. Qin, H. Yao, Q. Huang, J. Lim, and M.-H.Yang. Hedged deep tracking. In CVPR, pages 4303–4311,2016. 5
[17] L. Wang, W. Ouyang, X. Wang, and H. Lu. Visual trackingwith fully convolutional networks. In ICCV, pages 3119–3127, 2015. 5
[18] J. Zhang, S. Ma, and S. Sclaroff. MEEM: robust trackingvia multiple experts using entropy minimization. In ECCV,pages 188–203, 2014. 5
