Kanade-Lucas-Tomasi(KLT) Feature Tracker Tracke… · Computer Vision (EEE6503) Fall 2009,...
Transcript of Kanade-Lucas-Tomasi(KLT) Feature Tracker Tracke… · Computer Vision (EEE6503) Fall 2009,...
![Page 1: Kanade-Lucas-Tomasi(KLT) Feature Tracker Tracke… · Computer Vision (EEE6503) Fall 2009, YonseiUniv. Kanade-Lucas-Tomasi(KLT) Feature Tracker Computer Vision Lab. Jae Kyu Suhr](https://reader033.fdocuments.us/reader033/viewer/2022052500/605a0defcabc266f554966ce/html5/thumbnails/1.jpg)
Computer Vision (EEE6503) Fall 2009, Yonsei Univ.
KanadeKanade--LucasLucas--TomasiTomasi (KLT)(KLT)Feature TrackerFeature Tracker
Computer Vision Lab.Computer Vision Lab.
Jae Kyu SuhrJae Kyu Suhr
![Page 2: Kanade-Lucas-Tomasi(KLT) Feature Tracker Tracke… · Computer Vision (EEE6503) Fall 2009, YonseiUniv. Kanade-Lucas-Tomasi(KLT) Feature Tracker Computer Vision Lab. Jae Kyu Suhr](https://reader033.fdocuments.us/reader033/viewer/2022052500/605a0defcabc266f554966ce/html5/thumbnails/2.jpg)
Computer Vision (EEE6503) Fall 2009, Yonsei Univ.
IntroductionIntroduction
![Page 3: Kanade-Lucas-Tomasi(KLT) Feature Tracker Tracke… · Computer Vision (EEE6503) Fall 2009, YonseiUniv. Kanade-Lucas-Tomasi(KLT) Feature Tracker Computer Vision Lab. Jae Kyu Suhr](https://reader033.fdocuments.us/reader033/viewer/2022052500/605a0defcabc266f554966ce/html5/thumbnails/3.jpg)
Computer Vision (EEE6503) Fall 2009, Yonsei Univ.
MotivationMotivation
Image sequence
![Page 4: Kanade-Lucas-Tomasi(KLT) Feature Tracker Tracke… · Computer Vision (EEE6503) Fall 2009, YonseiUniv. Kanade-Lucas-Tomasi(KLT) Feature Tracker Computer Vision Lab. Jae Kyu Suhr](https://reader033.fdocuments.us/reader033/viewer/2022052500/605a0defcabc266f554966ce/html5/thumbnails/4.jpg)
Computer Vision (EEE6503) Fall 2009, Yonsei Univ.
MotivationMotivation
Feature point detection
![Page 5: Kanade-Lucas-Tomasi(KLT) Feature Tracker Tracke… · Computer Vision (EEE6503) Fall 2009, YonseiUniv. Kanade-Lucas-Tomasi(KLT) Feature Tracker Computer Vision Lab. Jae Kyu Suhr](https://reader033.fdocuments.us/reader033/viewer/2022052500/605a0defcabc266f554966ce/html5/thumbnails/5.jpg)
Computer Vision (EEE6503) Fall 2009, Yonsei Univ.
MotivationMotivation
Feature point tracking
![Page 6: Kanade-Lucas-Tomasi(KLT) Feature Tracker Tracke… · Computer Vision (EEE6503) Fall 2009, YonseiUniv. Kanade-Lucas-Tomasi(KLT) Feature Tracker Computer Vision Lab. Jae Kyu Suhr](https://reader033.fdocuments.us/reader033/viewer/2022052500/605a0defcabc266f554966ce/html5/thumbnails/6.jpg)
Computer Vision (EEE6503) Fall 2009, Yonsei Univ.
MotivationMotivation
Tracking result
![Page 7: Kanade-Lucas-Tomasi(KLT) Feature Tracker Tracke… · Computer Vision (EEE6503) Fall 2009, YonseiUniv. Kanade-Lucas-Tomasi(KLT) Feature Tracker Computer Vision Lab. Jae Kyu Suhr](https://reader033.fdocuments.us/reader033/viewer/2022052500/605a0defcabc266f554966ce/html5/thumbnails/7.jpg)
Computer Vision (EEE6503) Fall 2009, Yonsei Univ.
3D Reconstruction results3D Reconstruction results
![Page 8: Kanade-Lucas-Tomasi(KLT) Feature Tracker Tracke… · Computer Vision (EEE6503) Fall 2009, YonseiUniv. Kanade-Lucas-Tomasi(KLT) Feature Tracker Computer Vision Lab. Jae Kyu Suhr](https://reader033.fdocuments.us/reader033/viewer/2022052500/605a0defcabc266f554966ce/html5/thumbnails/8.jpg)
Computer Vision (EEE6503) Fall 2009, Yonsei Univ.
Problem StatementProblem Statement
![Page 9: Kanade-Lucas-Tomasi(KLT) Feature Tracker Tracke… · Computer Vision (EEE6503) Fall 2009, YonseiUniv. Kanade-Lucas-Tomasi(KLT) Feature Tracker Computer Vision Lab. Jae Kyu Suhr](https://reader033.fdocuments.us/reader033/viewer/2022052500/605a0defcabc266f554966ce/html5/thumbnails/9.jpg)
Computer Vision (EEE6503) Fall 2009, Yonsei Univ.
Problem StatementProblem Statement
• I, J : 1st and 2nd grayscaled images
• I(x,y) : gray value of I at [x y]T
• Let u=[ux uy]T be a point on the 1st image I
• The goal is to find v on J, where I(u) and J(v) aresimilar. (v= u+d = [ux+dx uy+dy]T)
• d=[dx dy]T is the image velocity at uor the optical flow at u.
![Page 10: Kanade-Lucas-Tomasi(KLT) Feature Tracker Tracke… · Computer Vision (EEE6503) Fall 2009, YonseiUniv. Kanade-Lucas-Tomasi(KLT) Feature Tracker Computer Vision Lab. Jae Kyu Suhr](https://reader033.fdocuments.us/reader033/viewer/2022052500/605a0defcabc266f554966ce/html5/thumbnails/10.jpg)
Computer Vision (EEE6503) Fall 2009, Yonsei Univ.
Problem StatementProblem Statement
• Definition of the residual function ε(d).
• ωx and ωy integration window size parameterintegration window size (2ωx+1)×(2ωy+1).
(ux, uy)
wx
wy
I J
∑( - )2
![Page 11: Kanade-Lucas-Tomasi(KLT) Feature Tracker Tracke… · Computer Vision (EEE6503) Fall 2009, YonseiUniv. Kanade-Lucas-Tomasi(KLT) Feature Tracker Computer Vision Lab. Jae Kyu Suhr](https://reader033.fdocuments.us/reader033/viewer/2022052500/605a0defcabc266f554966ce/html5/thumbnails/11.jpg)
Computer Vision (EEE6503) Fall 2009, Yonsei Univ.
Pyramid ImplementationPyramid Implementation
![Page 12: Kanade-Lucas-Tomasi(KLT) Feature Tracker Tracke… · Computer Vision (EEE6503) Fall 2009, YonseiUniv. Kanade-Lucas-Tomasi(KLT) Feature Tracker Computer Vision Lab. Jae Kyu Suhr](https://reader033.fdocuments.us/reader033/viewer/2022052500/605a0defcabc266f554966ce/html5/thumbnails/12.jpg)
Computer Vision (EEE6503) Fall 2009, Yonsei Univ.
Why pyramid representation?Why pyramid representation?
• Standard KLT algorithm can deal with small pixel displacement.• Solution for this is a pyramidal implementation.• Let I0=I be the 0th level image• The pyramid representation is built recursively.
I0 → I1 → I2 → I3 → ILm … (Lm: 2~4)• IL-1 : the image at level L-1
IL : the image at level L
![Page 13: Kanade-Lucas-Tomasi(KLT) Feature Tracker Tracke… · Computer Vision (EEE6503) Fall 2009, YonseiUniv. Kanade-Lucas-Tomasi(KLT) Feature Tracker Computer Vision Lab. Jae Kyu Suhr](https://reader033.fdocuments.us/reader033/viewer/2022052500/605a0defcabc266f554966ce/html5/thumbnails/13.jpg)
Computer Vision (EEE6503) Fall 2009, Yonsei Univ.
Pyramidal feature trackingPyramidal feature tracking
• A given point u in I, find its corresponding location v=u+d.
• Corresponding point of u(u0) on the pyramidal image IL is uL
• Simple overall pyramid tracking algorithm
dLm is computed at the pyramid level Lm
dLm-1 is computedwith an initial guess of dLm at Lm-1
This continues up to the level 0
![Page 14: Kanade-Lucas-Tomasi(KLT) Feature Tracker Tracke… · Computer Vision (EEE6503) Fall 2009, YonseiUniv. Kanade-Lucas-Tomasi(KLT) Feature Tracker Computer Vision Lab. Jae Kyu Suhr](https://reader033.fdocuments.us/reader033/viewer/2022052500/605a0defcabc266f554966ce/html5/thumbnails/14.jpg)
Computer Vision (EEE6503) Fall 2009, Yonsei Univ.
• Let gL=[gLx gL
y]T be an initial guess at level L( gL is available from level Lm to level L+1 )
• Residual pixel displacement vector dL=[dLx dL
y]T
that minimizes the new image matching error function εL
• Window size (2ωx+1)×(2ωy+1) is constant for all pyramid
• gL is used to pre-translate the image patch in 2nd image J→ dL is small and therefore easy to compute using KLT algorithm
Pyramidal feature trackingPyramidal feature tracking
![Page 15: Kanade-Lucas-Tomasi(KLT) Feature Tracker Tracke… · Computer Vision (EEE6503) Fall 2009, YonseiUniv. Kanade-Lucas-Tomasi(KLT) Feature Tracker Computer Vision Lab. Jae Kyu Suhr](https://reader033.fdocuments.us/reader033/viewer/2022052500/605a0defcabc266f554966ce/html5/thumbnails/15.jpg)
Computer Vision (EEE6503) Fall 2009, Yonsei Univ.
Pyramidal feature trackingPyramidal feature tracking
• Assume that dL is computed, new initial guess gL-1 at level L-1
• Initial guess for the deepest level Lm
• The final optical flow solution d
![Page 16: Kanade-Lucas-Tomasi(KLT) Feature Tracker Tracke… · Computer Vision (EEE6503) Fall 2009, YonseiUniv. Kanade-Lucas-Tomasi(KLT) Feature Tracker Computer Vision Lab. Jae Kyu Suhr](https://reader033.fdocuments.us/reader033/viewer/2022052500/605a0defcabc266f554966ce/html5/thumbnails/16.jpg)
Computer Vision (EEE6503) Fall 2009, Yonsei Univ.
Standard KLT algorithmStandard KLT algorithm
![Page 17: Kanade-Lucas-Tomasi(KLT) Feature Tracker Tracke… · Computer Vision (EEE6503) Fall 2009, YonseiUniv. Kanade-Lucas-Tomasi(KLT) Feature Tracker Computer Vision Lab. Jae Kyu Suhr](https://reader033.fdocuments.us/reader033/viewer/2022052500/605a0defcabc266f554966ce/html5/thumbnails/17.jpg)
Computer Vision (EEE6503) Fall 2009, Yonsei Univ.
Standard KLT optical flow computationStandard KLT optical flow computation
• Goal is to find dL that minimizes the matching function εL.Same operation is performed for all levels L, drop the superscript L and define the new images A and B
• Let us change the displacement vector and the point vector
![Page 18: Kanade-Lucas-Tomasi(KLT) Feature Tracker Tracke… · Computer Vision (EEE6503) Fall 2009, YonseiUniv. Kanade-Lucas-Tomasi(KLT) Feature Tracker Computer Vision Lab. Jae Kyu Suhr](https://reader033.fdocuments.us/reader033/viewer/2022052500/605a0defcabc266f554966ce/html5/thumbnails/18.jpg)
Computer Vision (EEE6503) Fall 2009, Yonsei Univ.
Standard KLT optical flow computationStandard KLT optical flow computation
• Goal is to find that minimizes the matching function
• To find the optimum
• Let us substitute by its 1st order Taylor expansion about the point
frame difference Image gradient
![Page 19: Kanade-Lucas-Tomasi(KLT) Feature Tracker Tracke… · Computer Vision (EEE6503) Fall 2009, YonseiUniv. Kanade-Lucas-Tomasi(KLT) Feature Tracker Computer Vision Lab. Jae Kyu Suhr](https://reader033.fdocuments.us/reader033/viewer/2022052500/605a0defcabc266f554966ce/html5/thumbnails/19.jpg)
Computer Vision (EEE6503) Fall 2009, Yonsei Univ.
Standard KLT optical flow computationStandard KLT optical flow computation
![Page 20: Kanade-Lucas-Tomasi(KLT) Feature Tracker Tracke… · Computer Vision (EEE6503) Fall 2009, YonseiUniv. Kanade-Lucas-Tomasi(KLT) Feature Tracker Computer Vision Lab. Jae Kyu Suhr](https://reader033.fdocuments.us/reader033/viewer/2022052500/605a0defcabc266f554966ce/html5/thumbnails/20.jpg)
Computer Vision (EEE6503) Fall 2009, Yonsei Univ.
Standard KLT optical flow computationStandard KLT optical flow computation
( G must be invertible).
Standard LK is valid only the pixel displacement is small.
(Because of the first order Taylor approximation)
Therefore, iterative version of LK is necessary.
![Page 21: Kanade-Lucas-Tomasi(KLT) Feature Tracker Tracke… · Computer Vision (EEE6503) Fall 2009, YonseiUniv. Kanade-Lucas-Tomasi(KLT) Feature Tracker Computer Vision Lab. Jae Kyu Suhr](https://reader033.fdocuments.us/reader033/viewer/2022052500/605a0defcabc266f554966ce/html5/thumbnails/21.jpg)
Computer Vision (EEE6503) Fall 2009, Yonsei Univ.
Iterative KLT algorithmIterative KLT algorithm
![Page 22: Kanade-Lucas-Tomasi(KLT) Feature Tracker Tracke… · Computer Vision (EEE6503) Fall 2009, YonseiUniv. Kanade-Lucas-Tomasi(KLT) Feature Tracker Computer Vision Lab. Jae Kyu Suhr](https://reader033.fdocuments.us/reader033/viewer/2022052500/605a0defcabc266f554966ce/html5/thumbnails/22.jpg)
Computer Vision (EEE6503) Fall 2009, Yonsei Univ.
Iterative KLT optical flow computationIterative KLT optical flow computation
• k : iteration index
•: initial guess from the previous iteration 1,2,…,k-1
• Bk : new translated image according to
• The goal is to compute that minimize
![Page 23: Kanade-Lucas-Tomasi(KLT) Feature Tracker Tracke… · Computer Vision (EEE6503) Fall 2009, YonseiUniv. Kanade-Lucas-Tomasi(KLT) Feature Tracker Computer Vision Lab. Jae Kyu Suhr](https://reader033.fdocuments.us/reader033/viewer/2022052500/605a0defcabc266f554966ce/html5/thumbnails/23.jpg)
Computer Vision (EEE6503) Fall 2009, Yonsei Univ.
Iterative KLT optical flow computationIterative KLT optical flow computation
• One step LK optical flow computation is
• Ix, Iy are computed only once at the beginning of the iteration.
• G (2×2matrix) remains constant throughout the iteration loop.
• Only δIk needs to be recomputed at each iteration
• Once is computed, a new pixel displacement guess
• The iteration goes on until is smaller than a thresholdor reached at the maximum number of iteration.(5 are enough to reach convergence)
![Page 24: Kanade-Lucas-Tomasi(KLT) Feature Tracker Tracke… · Computer Vision (EEE6503) Fall 2009, YonseiUniv. Kanade-Lucas-Tomasi(KLT) Feature Tracker Computer Vision Lab. Jae Kyu Suhr](https://reader033.fdocuments.us/reader033/viewer/2022052500/605a0defcabc266f554966ce/html5/thumbnails/24.jpg)
Computer Vision (EEE6503) Fall 2009, Yonsei Univ.
Iterative KLT optical flow computationIterative KLT optical flow computation
• At 1st iteration, the initial guess
• Assuming that K iterations are necessary to reach convergence,the final solution for the optical flow vector
• This overall procedure is repeated at all levels L-1, L-2, … , 0
![Page 25: Kanade-Lucas-Tomasi(KLT) Feature Tracker Tracke… · Computer Vision (EEE6503) Fall 2009, YonseiUniv. Kanade-Lucas-Tomasi(KLT) Feature Tracker Computer Vision Lab. Jae Kyu Suhr](https://reader033.fdocuments.us/reader033/viewer/2022052500/605a0defcabc266f554966ce/html5/thumbnails/25.jpg)
Computer Vision (EEE6503) Fall 2009, Yonsei Univ.
![Page 26: Kanade-Lucas-Tomasi(KLT) Feature Tracker Tracke… · Computer Vision (EEE6503) Fall 2009, YonseiUniv. Kanade-Lucas-Tomasi(KLT) Feature Tracker Computer Vision Lab. Jae Kyu Suhr](https://reader033.fdocuments.us/reader033/viewer/2022052500/605a0defcabc266f554966ce/html5/thumbnails/26.jpg)
Computer Vision (EEE6503) Fall 2009, Yonsei Univ.
Feature selectionFeature selection
• Goal is to find the location of v on J corresponding to u on I.How to detect u on I
• Find the point u whose G matrix is non-singularThe minimum eigenvalue of G must larger than a threshold.
![Page 27: Kanade-Lucas-Tomasi(KLT) Feature Tracker Tracke… · Computer Vision (EEE6503) Fall 2009, YonseiUniv. Kanade-Lucas-Tomasi(KLT) Feature Tracker Computer Vision Lab. Jae Kyu Suhr](https://reader033.fdocuments.us/reader033/viewer/2022052500/605a0defcabc266f554966ce/html5/thumbnails/27.jpg)
Computer Vision (EEE6503) Fall 2009, Yonsei Univ.
Feature selectionFeature selection
1. Compute the G matrix and its minimum eigenvalue λm at every pixelin the image I.
2. Call λmax the maximum value of λm over the whole image.
3. Retain the image pixels that have a λm value larger than a threshold.
4. Retain the local maximum pixels (a pixel is kept if its λm value is larger than that of any other pixel in its 3×3 neighborhood).
5. Keep the subset of those pixels so that the minimum distancebetween any pair of pixels is larger than a given threshold distance.
![Page 28: Kanade-Lucas-Tomasi(KLT) Feature Tracker Tracke… · Computer Vision (EEE6503) Fall 2009, YonseiUniv. Kanade-Lucas-Tomasi(KLT) Feature Tracker Computer Vision Lab. Jae Kyu Suhr](https://reader033.fdocuments.us/reader033/viewer/2022052500/605a0defcabc266f554966ce/html5/thumbnails/28.jpg)
Computer Vision (EEE6503) Fall 2009, Yonsei Univ.
Feature selectionFeature selection
![Page 29: Kanade-Lucas-Tomasi(KLT) Feature Tracker Tracke… · Computer Vision (EEE6503) Fall 2009, YonseiUniv. Kanade-Lucas-Tomasi(KLT) Feature Tracker Computer Vision Lab. Jae Kyu Suhr](https://reader033.fdocuments.us/reader033/viewer/2022052500/605a0defcabc266f554966ce/html5/thumbnails/29.jpg)
Computer Vision (EEE6503) Fall 2009, Yonsei Univ.
Feature selectionFeature selection
![Page 30: Kanade-Lucas-Tomasi(KLT) Feature Tracker Tracke… · Computer Vision (EEE6503) Fall 2009, YonseiUniv. Kanade-Lucas-Tomasi(KLT) Feature Tracker Computer Vision Lab. Jae Kyu Suhr](https://reader033.fdocuments.us/reader033/viewer/2022052500/605a0defcabc266f554966ce/html5/thumbnails/30.jpg)
Computer Vision (EEE6503) Fall 2009, Yonsei Univ.
Feature selectionFeature selection
![Page 31: Kanade-Lucas-Tomasi(KLT) Feature Tracker Tracke… · Computer Vision (EEE6503) Fall 2009, YonseiUniv. Kanade-Lucas-Tomasi(KLT) Feature Tracker Computer Vision Lab. Jae Kyu Suhr](https://reader033.fdocuments.us/reader033/viewer/2022052500/605a0defcabc266f554966ce/html5/thumbnails/31.jpg)
Computer Vision (EEE6503) Fall 2009, Yonsei Univ.
Feature selectionFeature selection
![Page 32: Kanade-Lucas-Tomasi(KLT) Feature Tracker Tracke… · Computer Vision (EEE6503) Fall 2009, YonseiUniv. Kanade-Lucas-Tomasi(KLT) Feature Tracker Computer Vision Lab. Jae Kyu Suhr](https://reader033.fdocuments.us/reader033/viewer/2022052500/605a0defcabc266f554966ce/html5/thumbnails/32.jpg)
Computer Vision (EEE6503) Fall 2009, Yonsei Univ.
Feature selectionFeature selection
![Page 33: Kanade-Lucas-Tomasi(KLT) Feature Tracker Tracke… · Computer Vision (EEE6503) Fall 2009, YonseiUniv. Kanade-Lucas-Tomasi(KLT) Feature Tracker Computer Vision Lab. Jae Kyu Suhr](https://reader033.fdocuments.us/reader033/viewer/2022052500/605a0defcabc266f554966ce/html5/thumbnails/33.jpg)
Computer Vision (EEE6503) Fall 2009, Yonsei Univ.
Declaring a feature Declaring a feature ““LostLost””
• A feature point falls outside of the image.
• An image patch around the tracked point varies too much between image I and image J. (A cost function (SSD) is larger than a threshold).
![Page 34: Kanade-Lucas-Tomasi(KLT) Feature Tracker Tracke… · Computer Vision (EEE6503) Fall 2009, YonseiUniv. Kanade-Lucas-Tomasi(KLT) Feature Tracker Computer Vision Lab. Jae Kyu Suhr](https://reader033.fdocuments.us/reader033/viewer/2022052500/605a0defcabc266f554966ce/html5/thumbnails/34.jpg)
Computer Vision (EEE6503) Fall 2009, Yonsei Univ.
Flow chartFlow chart
Input 1st image
Select feature points
Input 2nd image
Track feature points
Delete lost feature points
Save feature points
Save tracked points
Replace 1st image with 2nd image
Replace 2st image with next image
Select feature pointsto replace lost feature points
Save feature points
![Page 35: Kanade-Lucas-Tomasi(KLT) Feature Tracker Tracke… · Computer Vision (EEE6503) Fall 2009, YonseiUniv. Kanade-Lucas-Tomasi(KLT) Feature Tracker Computer Vision Lab. Jae Kyu Suhr](https://reader033.fdocuments.us/reader033/viewer/2022052500/605a0defcabc266f554966ce/html5/thumbnails/35.jpg)
Computer Vision (EEE6503) Fall 2009, Yonsei Univ.
ReferencesReferences
• Bruce D. Lucas and Takeo Kanade. “An Iterative Image Registration Technique with an Application to Stereo Vision”. IJCAI, pages 674-679, 1981.
• J. Shi and C. Tomasi, "Good Features to Track," CVPR'94
• Jean-Yves Bouguet, “Pyramidal Implementation of the Lucas KanadeFeature Tracker Description of the algorithm”, Intel Corporation Microprocessor Research Labs.
• http://www.ces.clemson.edu/~stb/klt/ C++ codeKLT: An Implementation of the Kanade-Lucas-Tomasi Feature Tracker
• OpenCV: CalcOpticalFlowLK, CalcOpticalFlowPyrLK
• http://vision.ucla.edu//MASKS/labs.html Matlab codeAn Invitation to 3D Vision
![Page 36: Kanade-Lucas-Tomasi(KLT) Feature Tracker Tracke… · Computer Vision (EEE6503) Fall 2009, YonseiUniv. Kanade-Lucas-Tomasi(KLT) Feature Tracker Computer Vision Lab. Jae Kyu Suhr](https://reader033.fdocuments.us/reader033/viewer/2022052500/605a0defcabc266f554966ce/html5/thumbnails/36.jpg)
Computer Vision (EEE6503) Fall 2009, Yonsei Univ.
Thank youThank you
Q & AQ & A