Lecture30
Transcript of Lecture30
![Page 1: Lecture30](https://reader036.fdocuments.us/reader036/viewer/2022081401/5584a7f3d8b42a0b5c8b4afe/html5/thumbnails/1.jpg)
CSE486, Penn StateRobert Collins
Lecture 30:
Video Tracking: Lucas-Kanade
![Page 2: Lecture30](https://reader036.fdocuments.us/reader036/viewer/2022081401/5584a7f3d8b42a0b5c8b4afe/html5/thumbnails/2.jpg)
CSE486, Penn StateRobert Collins
Two Popular Tracking Methods
• Mean-shift color histogram tracking (last time)
• Lucas-Kanade template tracking (today)
![Page 3: Lecture30](https://reader036.fdocuments.us/reader036/viewer/2022081401/5584a7f3d8b42a0b5c8b4afe/html5/thumbnails/3.jpg)
CSE486, Penn StateRobert Collins
Lucas-Kanade Tracking
![Page 4: Lecture30](https://reader036.fdocuments.us/reader036/viewer/2022081401/5584a7f3d8b42a0b5c8b4afe/html5/thumbnails/4.jpg)
CSE486, Penn StateRobert Collins
Review: Lucas-Kanade
• Brightness constancy
• One equation two unknowns
• How to get more equations for a pixel?– Basic idea: impose additional constraints
• one method: pretend the pixel’s neighbors have the same (u,v)– If we use a 5x5 window, that gives us 25 equations per pixel!
* From Khurram Hassan-Shafique CAP5415 Computer Vision 2003
unknown flow vector
spatial gradienttemporal gradient
![Page 5: Lecture30](https://reader036.fdocuments.us/reader036/viewer/2022081401/5584a7f3d8b42a0b5c8b4afe/html5/thumbnails/5.jpg)
CSE486, Penn StateRobert Collins
Review: Lucas-Kanade (cont)
• Now we have more equations than unknowns
– The summations are over all pixels in the K x K window
– This technique was first proposed by Lucas & Kanade (1981)• described in Trucco & Verri reading
• Solution: solve least squares problem– minimum least squares solution given by solution (in d) of:
* From Khurram Hassan-Shafique CAP5415 Computer Vision 2003
![Page 6: Lecture30](https://reader036.fdocuments.us/reader036/viewer/2022081401/5584a7f3d8b42a0b5c8b4afe/html5/thumbnails/6.jpg)
CSE486, Penn StateRobert Collins
Lucas Kanade Tracking
Traditional Lucas-Kanade is typically run on small, corner-like features (e.g. 5x5) to compute optic flow.
Observation: There’s no reason we can’t use the same approach on a larger window around the object being tracked.
80x50 pixels
![Page 7: Lecture30](https://reader036.fdocuments.us/reader036/viewer/2022081401/5584a7f3d8b42a0b5c8b4afe/html5/thumbnails/7.jpg)
CSE486, Penn StateRobert Collins
Basic LK Derivation for Templates
template(model)
current frameu,v = hypothesized location of template in current frame
![Page 8: Lecture30](https://reader036.fdocuments.us/reader036/viewer/2022081401/5584a7f3d8b42a0b5c8b4afe/html5/thumbnails/8.jpg)
CSE486, Penn StateRobert Collins
Basic LK Derivation for Templates
Take partial derivs and set to zero
First order approx
Form matrix equation
solve vialeast-squares
![Page 9: Lecture30](https://reader036.fdocuments.us/reader036/viewer/2022081401/5584a7f3d8b42a0b5c8b4afe/html5/thumbnails/9.jpg)
CSE486, Penn StateRobert Collins
One Problem with this...
Assumption of constant flow (pure translation) forall pixels in a larger window is unreasonable for long periods of time.
However, we can easily generalize Lucas-Kanadeapproach to other 2D parametric motion models (like affine or projective) by introducing a “warp”function W.
2])],([))];,([([ yxTPyxWIpatchimagewithin
y
x
generalize
![Page 10: Lecture30](https://reader036.fdocuments.us/reader036/viewer/2022081401/5584a7f3d8b42a0b5c8b4afe/html5/thumbnails/10.jpg)
CSE486, Penn StateRobert Collins
Step-by-Step Derivation
PPyxWI ))];,([([
The key to the derivation is Taylor series approximation:
PP
WIPyxWI
))];,([([~~
We will derive this step-by-step. First, we need two background formula:
![Page 11: Lecture30](https://reader036.fdocuments.us/reader036/viewer/2022081401/5584a7f3d8b42a0b5c8b4afe/html5/thumbnails/11.jpg)
CSE486, Penn StateRobert Collins
Step-by-Step Derivation
First consider the expansion for a single variable p
![Page 12: Lecture30](https://reader036.fdocuments.us/reader036/viewer/2022081401/5584a7f3d8b42a0b5c8b4afe/html5/thumbnails/12.jpg)
CSE486, Penn StateRobert Collins
Step-by-Step Derivation
Note that each variable parameter pi contributes a term of the form
![Page 13: Lecture30](https://reader036.fdocuments.us/reader036/viewer/2022081401/5584a7f3d8b42a0b5c8b4afe/html5/thumbnails/13.jpg)
CSE486, Penn StateRobert Collins
Step-by-Step Derivation
Now let’s rewrite the expression as a matrix equation. For each term,we can rewrite:
So that we have:
![Page 14: Lecture30](https://reader036.fdocuments.us/reader036/viewer/2022081401/5584a7f3d8b42a0b5c8b4afe/html5/thumbnails/14.jpg)
CSE486, Penn StateRobert Collins
Step-by-Step DerivationFurther collecting the dw/dpi terms into a matrix, we can write:
PPyxWI ))];,([([ PP
WIPyxWI
))];,([([~~
which are the terms in the matrix equation:
![Page 15: Lecture30](https://reader036.fdocuments.us/reader036/viewer/2022081401/5584a7f3d8b42a0b5c8b4afe/html5/thumbnails/15.jpg)
CSE486, Penn StateRobert Collins
Example: Jacobian of Affine Warp
11
1)];,([
642
531 y
x
ppp
pppPyxW
affine warp function (6 parameters)
1000
0100
642
531
yx
yxP
PyPyxP
PyPxPx
P
W
n
yyyy
n
xxxx
yx
P
W
P
W
P
W
P
WP
W
P
W
P
W
P
W
P
W
WWPyxWLet
321
321
],[)];,([
general equation of Jacobian
![Page 16: Lecture30](https://reader036.fdocuments.us/reader036/viewer/2022081401/5584a7f3d8b42a0b5c8b4afe/html5/thumbnails/16.jpg)
CSE486, Penn StateRobert Collins
Iterate Warp I to obtain I(W([x y];P))
Compute the error image T(x) – I(W([x y]; P))
Warp the gradient with W([x y]; P)
Evaluate at ([x y]; P)
Compute steepest descent images
Compute Hessian matrix
Compute
Compute P
Update P P + P
I
P
W
P
WI
)()(P
WI
P
WI T
)))];,([(),(()( PyxWIyxTP
WI T
Until P magnitude is negligible
Source: “Lucas-Kanade 20 years on: A unifying framework” Baker and Mathews, IJCV 04
Dr. Ng Teck Khim
(Jacobian)
![Page 17: Lecture30](https://reader036.fdocuments.us/reader036/viewer/2022081401/5584a7f3d8b42a0b5c8b4afe/html5/thumbnails/17.jpg)
CSE486, Penn StateRobert Collins
Source: “Lucas-Kanade 20 years on: A unifying framework” Baker and Mathews, IJCV 04
Algorithm At a Glance
![Page 18: Lecture30](https://reader036.fdocuments.us/reader036/viewer/2022081401/5584a7f3d8b42a0b5c8b4afe/html5/thumbnails/18.jpg)
CSE486, Penn StateRobert Collins
Baker, Matthews, CMU
State of the Art Lucas Kanade Tracking
QuickTime™ and aYUV420 codec decompressor
are needed to see this picture.
Tracking facial mesh models (piecewise affine)