Struck: Structured Output Tracking with...
Transcript of Struck: Structured Output Tracking with...
![Page 1: Struck: Structured Output Tracking with Kernelsvision.stanford.edu/teaching/cs231b_spring1415/slides/struck_meng.pdf · Struck: Structured Output Tracking with Kernels Sam Hare, Amir](https://reader031.fdocuments.us/reader031/viewer/2022021722/5c61168809d3f21c6d8c83b9/html5/thumbnails/1.jpg)
Struck: Structured Output Tracking with
Kernels
Sam Hare, Amir Saffari, And Philip H. S. Torr
International Conference On Computer Vision (ICCV), 2011
![Page 2: Struck: Structured Output Tracking with Kernelsvision.stanford.edu/teaching/cs231b_spring1415/slides/struck_meng.pdf · Struck: Structured Output Tracking with Kernels Sam Hare, Amir](https://reader031.fdocuments.us/reader031/viewer/2022021722/5c61168809d3f21c6d8c83b9/html5/thumbnails/2.jpg)
Motivations
Problem: tracking-by-detection
Input: target
Output: locations over times
![Page 3: Struck: Structured Output Tracking with Kernelsvision.stanford.edu/teaching/cs231b_spring1415/slides/struck_meng.pdf · Struck: Structured Output Tracking with Kernels Sam Hare, Amir](https://reader031.fdocuments.us/reader031/viewer/2022021722/5c61168809d3f21c6d8c83b9/html5/thumbnails/3.jpg)
Performance summary
MILTLD
Struck
Y Wu, J Lim, MH Yang “Online Object Tracking: A Benchmark”, Computer Vision and Pattern
Recognition (CVPR), 2013 IEEE Conference on
![Page 4: Struck: Structured Output Tracking with Kernelsvision.stanford.edu/teaching/cs231b_spring1415/slides/struck_meng.pdf · Struck: Structured Output Tracking with Kernels Sam Hare, Amir](https://reader031.fdocuments.us/reader031/viewer/2022021722/5c61168809d3f21c6d8c83b9/html5/thumbnails/4.jpg)
Outline
Previous works
• Tracking-by-detection
• Adaptive tracking-by-detection
Methods
• Structured output tracking
• Online optimization and budget mechanism
Experiments and results
![Page 5: Struck: Structured Output Tracking with Kernelsvision.stanford.edu/teaching/cs231b_spring1415/slides/struck_meng.pdf · Struck: Structured Output Tracking with Kernels Sam Hare, Amir](https://reader031.fdocuments.us/reader031/viewer/2022021722/5c61168809d3f21c6d8c83b9/html5/thumbnails/5.jpg)
Previous Works
Tracking problem as a detection task applied over time
Separating hyperplanes with different margins.
S. Avidan. Support Vector Tracking. IEEE Trans. on PAMI, 26:1064–1072, 2004.
![Page 6: Struck: Structured Output Tracking with Kernelsvision.stanford.edu/teaching/cs231b_spring1415/slides/struck_meng.pdf · Struck: Structured Output Tracking with Kernels Sam Hare, Amir](https://reader031.fdocuments.us/reader031/viewer/2022021722/5c61168809d3f21c6d8c83b9/html5/thumbnails/6.jpg)
Previous Works
Tracking problem as a detection task applied over time
look for the image
region with the
highest SVM score
S. Avidan. Support Vector Tracking. IEEE Trans. on PAMI, 26:1064–1072, 2004.
![Page 7: Struck: Structured Output Tracking with Kernelsvision.stanford.edu/teaching/cs231b_spring1415/slides/struck_meng.pdf · Struck: Structured Output Tracking with Kernels Sam Hare, Amir](https://reader031.fdocuments.us/reader031/viewer/2022021722/5c61168809d3f21c6d8c83b9/html5/thumbnails/7.jpg)
Previous Works
Adaptive tracking-by-detection
B. Babenko, M. H. Yang, and S. Belongie. Visual Tracking with Online Multiple Instance Learning. In Proc. CVPR, 2009.
![Page 8: Struck: Structured Output Tracking with Kernelsvision.stanford.edu/teaching/cs231b_spring1415/slides/struck_meng.pdf · Struck: Structured Output Tracking with Kernels Sam Hare, Amir](https://reader031.fdocuments.us/reader031/viewer/2022021722/5c61168809d3f21c6d8c83b9/html5/thumbnails/8.jpg)
Previous Works – Adaptive Tracking-by-detection
![Page 9: Struck: Structured Output Tracking with Kernelsvision.stanford.edu/teaching/cs231b_spring1415/slides/struck_meng.pdf · Struck: Structured Output Tracking with Kernels Sam Hare, Amir](https://reader031.fdocuments.us/reader031/viewer/2022021722/5c61168809d3f21c6d8c83b9/html5/thumbnails/9.jpg)
Previous Works – Adaptive Tracking-by-detection
Adaptive tracking-by-detection
Tracking: A classification task
Learning: A update the object model.
B. Babenko, M. H. Yang, and S. Belongie. Visual Tracking with Online Multiple Instance Learning. In Proc. CVPR, 2009.
![Page 10: Struck: Structured Output Tracking with Kernelsvision.stanford.edu/teaching/cs231b_spring1415/slides/struck_meng.pdf · Struck: Structured Output Tracking with Kernels Sam Hare, Amir](https://reader031.fdocuments.us/reader031/viewer/2022021722/5c61168809d3f21c6d8c83b9/html5/thumbnails/10.jpg)
Previous Works – Adaptive Tracking-by-detection
Problem 1
What is the best
way to generate
labelled samples?
![Page 11: Struck: Structured Output Tracking with Kernelsvision.stanford.edu/teaching/cs231b_spring1415/slides/struck_meng.pdf · Struck: Structured Output Tracking with Kernels Sam Hare, Amir](https://reader031.fdocuments.us/reader031/viewer/2022021722/5c61168809d3f21c6d8c83b9/html5/thumbnails/11.jpg)
Previous Works – Adaptive Tracking-by-detection
Problem 2
Label prediction and
position estimation are
different objectives.
![Page 12: Struck: Structured Output Tracking with Kernelsvision.stanford.edu/teaching/cs231b_spring1415/slides/struck_meng.pdf · Struck: Structured Output Tracking with Kernels Sam Hare, Amir](https://reader031.fdocuments.us/reader031/viewer/2022021722/5c61168809d3f21c6d8c83b9/html5/thumbnails/12.jpg)
Main Idea
structured
output
prediction
![Page 13: Struck: Structured Output Tracking with Kernelsvision.stanford.edu/teaching/cs231b_spring1415/slides/struck_meng.pdf · Struck: Structured Output Tracking with Kernels Sam Hare, Amir](https://reader031.fdocuments.us/reader031/viewer/2022021722/5c61168809d3f21c6d8c83b9/html5/thumbnails/13.jpg)
Main Contributions
Structured output tracking
Avoid the intermediate classification step
Online learning and budgeting mechanism
Prevents too many training data
![Page 14: Struck: Structured Output Tracking with Kernelsvision.stanford.edu/teaching/cs231b_spring1415/slides/struck_meng.pdf · Struck: Structured Output Tracking with Kernels Sam Hare, Amir](https://reader031.fdocuments.us/reader031/viewer/2022021722/5c61168809d3f21c6d8c83b9/html5/thumbnails/14.jpg)
Outline
Previous work
• Tracking-by-detection
• Adaptive tracking-by-detection
Methods
• Structured output tracking
• Online optimization and budget mechanism
Experiments and results
![Page 15: Struck: Structured Output Tracking with Kernelsvision.stanford.edu/teaching/cs231b_spring1415/slides/struck_meng.pdf · Struck: Structured Output Tracking with Kernels Sam Hare, Amir](https://reader031.fdocuments.us/reader031/viewer/2022021722/5c61168809d3f21c6d8c83b9/html5/thumbnails/15.jpg)
Structured Output Tracking
tracker position
image patchBest motionssearch window
M. B. Blaschko and C. H. Lampert. Learning to Localize Objects with Structured Output
Regression. In Proc. ECCV, 2008.
![Page 16: Struck: Structured Output Tracking with Kernelsvision.stanford.edu/teaching/cs231b_spring1415/slides/struck_meng.pdf · Struck: Structured Output Tracking with Kernels Sam Hare, Amir](https://reader031.fdocuments.us/reader031/viewer/2022021722/5c61168809d3f21c6d8c83b9/html5/thumbnails/16.jpg)
Structured Output Tracking
The output space is all transformations instead of
the binary labels.
M. B. Blaschko and C. H. Lampert. Learning to Localize Objects with Structured Output
Regression. In Proc. ECCV, 2008.
![Page 17: Struck: Structured Output Tracking with Kernelsvision.stanford.edu/teaching/cs231b_spring1415/slides/struck_meng.pdf · Struck: Structured Output Tracking with Kernels Sam Hare, Amir](https://reader031.fdocuments.us/reader031/viewer/2022021722/5c61168809d3f21c6d8c83b9/html5/thumbnails/17.jpg)
Structured SVM Model
The SVM score should
correlate with overlapping
size with the best tracking
bounding box.
S. Avidan. Support Vector Tracking. IEEE Trans. on PAMI, 26:1064–1072, 2004.
![Page 18: Struck: Structured Output Tracking with Kernelsvision.stanford.edu/teaching/cs231b_spring1415/slides/struck_meng.pdf · Struck: Structured Output Tracking with Kernels Sam Hare, Amir](https://reader031.fdocuments.us/reader031/viewer/2022021722/5c61168809d3f21c6d8c83b9/html5/thumbnails/18.jpg)
Structured Output Tracking
![Page 19: Struck: Structured Output Tracking with Kernelsvision.stanford.edu/teaching/cs231b_spring1415/slides/struck_meng.pdf · Struck: Structured Output Tracking with Kernels Sam Hare, Amir](https://reader031.fdocuments.us/reader031/viewer/2022021722/5c61168809d3f21c6d8c83b9/html5/thumbnails/19.jpg)
Structured Output Tracking
![Page 20: Struck: Structured Output Tracking with Kernelsvision.stanford.edu/teaching/cs231b_spring1415/slides/struck_meng.pdf · Struck: Structured Output Tracking with Kernels Sam Hare, Amir](https://reader031.fdocuments.us/reader031/viewer/2022021722/5c61168809d3f21c6d8c83b9/html5/thumbnails/20.jpg)
Structured Output Tracking
Come back
later
![Page 21: Struck: Structured Output Tracking with Kernelsvision.stanford.edu/teaching/cs231b_spring1415/slides/struck_meng.pdf · Struck: Structured Output Tracking with Kernels Sam Hare, Amir](https://reader031.fdocuments.us/reader031/viewer/2022021722/5c61168809d3f21c6d8c83b9/html5/thumbnails/21.jpg)
Structured output SVM
A. Bordes, L. Bottou, P. Gallinari, and J. Weston. Solving multiclass support vector machines with LaRank. In Proc. ICML, 2007.
Efficient SMO optimization (CS229, EE364)
Kernels (CS229)
![Page 22: Struck: Structured Output Tracking with Kernelsvision.stanford.edu/teaching/cs231b_spring1415/slides/struck_meng.pdf · Struck: Structured Output Tracking with Kernels Sam Hare, Amir](https://reader031.fdocuments.us/reader031/viewer/2022021722/5c61168809d3f21c6d8c83b9/html5/thumbnails/22.jpg)
Structured output SVM
Gaussian kernel between image feature vectors (CS229)
Haar-like features (CS231A, CS232)
The responses of the Haar features are the input vectors of the kernel
![Page 23: Struck: Structured Output Tracking with Kernelsvision.stanford.edu/teaching/cs231b_spring1415/slides/struck_meng.pdf · Struck: Structured Output Tracking with Kernels Sam Hare, Amir](https://reader031.fdocuments.us/reader031/viewer/2022021722/5c61168809d3f21c6d8c83b9/html5/thumbnails/23.jpg)
Online optimization
![Page 24: Struck: Structured Output Tracking with Kernelsvision.stanford.edu/teaching/cs231b_spring1415/slides/struck_meng.pdf · Struck: Structured Output Tracking with Kernels Sam Hare, Amir](https://reader031.fdocuments.us/reader031/viewer/2022021722/5c61168809d3f21c6d8c83b9/html5/thumbnails/24.jpg)
Outline
Previous work
• Tracking-by-detection
• Adaptive tracking-by-detection
Methods
• Structured output tracking
• Online optimization and budget mechanism
Experiments and results
![Page 25: Struck: Structured Output Tracking with Kernelsvision.stanford.edu/teaching/cs231b_spring1415/slides/struck_meng.pdf · Struck: Structured Output Tracking with Kernels Sam Hare, Amir](https://reader031.fdocuments.us/reader031/viewer/2022021722/5c61168809d3f21c6d8c83b9/html5/thumbnails/25.jpg)
Online optimization
PROCESSNEW():
• Processes a new example
PROCESSOLD():
• Processes an existing support pattern
OPTIMIZE():
• Processes an existing support
pattern chosen at random
![Page 26: Struck: Structured Output Tracking with Kernelsvision.stanford.edu/teaching/cs231b_spring1415/slides/struck_meng.pdf · Struck: Structured Output Tracking with Kernels Sam Hare, Amir](https://reader031.fdocuments.us/reader031/viewer/2022021722/5c61168809d3f21c6d8c83b9/html5/thumbnails/26.jpg)
Budget mechanism
The number of support vectors increase over time.
Computational and storage costs grow linearly with the number of
support vectors.
![Page 27: Struck: Structured Output Tracking with Kernelsvision.stanford.edu/teaching/cs231b_spring1415/slides/struck_meng.pdf · Struck: Structured Output Tracking with Kernels Sam Hare, Amir](https://reader031.fdocuments.us/reader031/viewer/2022021722/5c61168809d3f21c6d8c83b9/html5/thumbnails/27.jpg)
Incorporating a budget
A budget (limit) of the number of supporting vectors.
Remove the support vector which results in the smallest change to the
weight vector
K. Crammer, J. Kandola, R. Holloway, and Y. Singer. Online Classification on a Budget. In NIPS, 2003.
Z. Wang, K. Crammer, and S. Vucetic. Multi-Class Pegasos on a Budget. In Proc. ICML, 2010. 2
![Page 28: Struck: Structured Output Tracking with Kernelsvision.stanford.edu/teaching/cs231b_spring1415/slides/struck_meng.pdf · Struck: Structured Output Tracking with Kernels Sam Hare, Amir](https://reader031.fdocuments.us/reader031/viewer/2022021722/5c61168809d3f21c6d8c83b9/html5/thumbnails/28.jpg)
Outline
Previous works
• Tracking-by-detection
• Adaptive tracking-by-detection
Methods
• Structured output tracking
• Online optimization and budget mechanism
Experiments and results
![Page 29: Struck: Structured Output Tracking with Kernelsvision.stanford.edu/teaching/cs231b_spring1415/slides/struck_meng.pdf · Struck: Structured Output Tracking with Kernels Sam Hare, Amir](https://reader031.fdocuments.us/reader031/viewer/2022021722/5c61168809d3f21c6d8c83b9/html5/thumbnails/29.jpg)
Experiments
• Haar-like features
• 6 different types arranged on a grid at 2 scales on a 4 x 4 grid,
resulting in 192 features
• Search radius 60, 5 radial and 16 angular divisions.
• Budget size is as low as B = 20, 50, 100, inf.
![Page 30: Struck: Structured Output Tracking with Kernelsvision.stanford.edu/teaching/cs231b_spring1415/slides/struck_meng.pdf · Struck: Structured Output Tracking with Kernels Sam Hare, Amir](https://reader031.fdocuments.us/reader031/viewer/2022021722/5c61168809d3f21c6d8c83b9/html5/thumbnails/30.jpg)
Dataset
http://vision.ucsd.edu/˜bbabenko/project_miltrack.shtml;
B. Babenko, M. H. Yang, and S. Belongie. Visual Tracking with Online Multiple Instance
Learning. In Proc. CVPR, 2009.
![Page 31: Struck: Structured Output Tracking with Kernelsvision.stanford.edu/teaching/cs231b_spring1415/slides/struck_meng.pdf · Struck: Structured Output Tracking with Kernels Sam Hare, Amir](https://reader031.fdocuments.us/reader031/viewer/2022021722/5c61168809d3f21c6d8c83b9/html5/thumbnails/31.jpg)
Overlap criterion
Jaccard similarity of bounding boxes
![Page 32: Struck: Structured Output Tracking with Kernelsvision.stanford.edu/teaching/cs231b_spring1415/slides/struck_meng.pdf · Struck: Structured Output Tracking with Kernels Sam Hare, Amir](https://reader031.fdocuments.us/reader031/viewer/2022021722/5c61168809d3f21c6d8c83b9/html5/thumbnails/32.jpg)
Results
http://www.samhare.net/research/struck
![Page 33: Struck: Structured Output Tracking with Kernelsvision.stanford.edu/teaching/cs231b_spring1415/slides/struck_meng.pdf · Struck: Structured Output Tracking with Kernels Sam Hare, Amir](https://reader031.fdocuments.us/reader031/viewer/2022021722/5c61168809d3f21c6d8c83b9/html5/thumbnails/33.jpg)
Visualization of the support vector set
![Page 34: Struck: Structured Output Tracking with Kernelsvision.stanford.edu/teaching/cs231b_spring1415/slides/struck_meng.pdf · Struck: Structured Output Tracking with Kernels Sam Hare, Amir](https://reader031.fdocuments.us/reader031/viewer/2022021722/5c61168809d3f21c6d8c83b9/html5/thumbnails/34.jpg)
Comparison
http://www.samhare.net/research/struck
![Page 35: Struck: Structured Output Tracking with Kernelsvision.stanford.edu/teaching/cs231b_spring1415/slides/struck_meng.pdf · Struck: Structured Output Tracking with Kernels Sam Hare, Amir](https://reader031.fdocuments.us/reader031/viewer/2022021722/5c61168809d3f21c6d8c83b9/html5/thumbnails/35.jpg)
Results
Struck with the smallest budget size (B = 20) outperforms
the state-of-the-art.
Average frames per second: 12 – 21.
![Page 36: Struck: Structured Output Tracking with Kernelsvision.stanford.edu/teaching/cs231b_spring1415/slides/struck_meng.pdf · Struck: Structured Output Tracking with Kernels Sam Hare, Amir](https://reader031.fdocuments.us/reader031/viewer/2022021722/5c61168809d3f21c6d8c83b9/html5/thumbnails/36.jpg)
Extensions
• Used more objection representations
• Haar-like features
• Raw pixel features
• Histogram features
• Combining multiple kernels seems to improve results, but not
significantly.
• Use key points and associated descriptors for object detection.
• Consider other machine learning algorithms.
![Page 37: Struck: Structured Output Tracking with Kernelsvision.stanford.edu/teaching/cs231b_spring1415/slides/struck_meng.pdf · Struck: Structured Output Tracking with Kernels Sam Hare, Amir](https://reader031.fdocuments.us/reader031/viewer/2022021722/5c61168809d3f21c6d8c83b9/html5/thumbnails/37.jpg)
Main Contributions
Structured output tracking
Avoid the intermediate classification step
Online learning and budgeting mechanism
Prevents too many training data
![Page 38: Struck: Structured Output Tracking with Kernelsvision.stanford.edu/teaching/cs231b_spring1415/slides/struck_meng.pdf · Struck: Structured Output Tracking with Kernels Sam Hare, Amir](https://reader031.fdocuments.us/reader031/viewer/2022021722/5c61168809d3f21c6d8c83b9/html5/thumbnails/38.jpg)
References
Sam Hare, Amir Saffari Philip H. S. Torr Struck: Structured Output Tracking
with Kernels International Conference on Computer Vision (ICCV), 2011
A. Bordes, L. Bottou, P. Gallinari, and J. Weston. Solving multiclass support
vector machines with LaRank. In Proc. ICML, 2007.
I. Tsochantaridis, T. Joachims, T. Hofmann, and Y. Altun. Large Margin
Methods for Structured and Interdependent Output Variables. JMLR, 6:1453–
1484, Dec. 2005.
K. Crammer, J. Kandola, R. Holloway, and Y. Singer. Online Classification on a
Budget. In NIPS, 2003.
P. Viola and M. J. Jones. Robust real-time face detection. IJCV, 57:137–154,
2004.
![Page 39: Struck: Structured Output Tracking with Kernelsvision.stanford.edu/teaching/cs231b_spring1415/slides/struck_meng.pdf · Struck: Structured Output Tracking with Kernels Sam Hare, Amir](https://reader031.fdocuments.us/reader031/viewer/2022021722/5c61168809d3f21c6d8c83b9/html5/thumbnails/39.jpg)
Thank you!