Motion and Optical Flow. Moving to Multiple Images So far, we’ve looked at processing a single...
-
date post
20-Dec-2015 -
Category
Documents
-
view
216 -
download
1
Transcript of Motion and Optical Flow. Moving to Multiple Images So far, we’ve looked at processing a single...
Motion and Optical FlowMotion and Optical Flow
Moving to Multiple ImagesMoving to Multiple Images
• So far, we’ve looked at processing aSo far, we’ve looked at processing asingle imagesingle image
• Multiple imagesMultiple images– Multiple cameras at one time: stereoMultiple cameras at one time: stereo– Single camera at many times: videoSingle camera at many times: video– (Multiple cameras at multiple times)(Multiple cameras at multiple times)
Applications of Multiple Applications of Multiple ImagesImages
• 2D2D– Feature / object trackingFeature / object tracking– Segmentation based on motionSegmentation based on motion
• 3D3D– Shape extractionShape extraction– Motion captureMotion capture
Applications of Multiple Applications of Multiple ImagesImages
in Graphicsin Graphics• Stitching images into panoramasStitching images into panoramas
• Automatic image morphingAutomatic image morphing
• Reconstruction of 3D models for Reconstruction of 3D models for renderingrendering
• Capturing articulated motion for Capturing articulated motion for animation animation
Applications of Multiple Applications of Multiple ImagesImages
in Biological Systemsin Biological Systems• Shape inferenceShape inference
• Peripheral sensitivity to motionPeripheral sensitivity to motion
• Looming field – obstacle avoidanceLooming field – obstacle avoidance
• Very similar applications in roboticsVery similar applications in robotics
Looming FieldLooming Field
• Pure Pure translation: translation: motion looks motion looks like it like it originates at originates at a point – a point – focus of focus of expansionexpansion
Key ProblemKey Problem
• Main problem in most multiple-image Main problem in most multiple-image methods: methods: correspondencecorrespondence
CorrespondenceCorrespondence
• Small displacementsSmall displacements– Differential algorithmsDifferential algorithms– Based on gradients in space and timeBased on gradients in space and time– Dense correspondence estimatesDense correspondence estimates– Most common with videoMost common with video
• Large displacementsLarge displacements– Matching algorithmsMatching algorithms– Based on correlation or featuresBased on correlation or features– Sparse correspondence estimatesSparse correspondence estimates– Most common with multiple cameras / Most common with multiple cameras /
stereostereo
Result of CorrespondenceResult of Correspondence
• For points in image For points in image ii displacements displacements to corresponding locations in image to corresponding locations in image jj
• In stereo, usually called In stereo, usually called disparitydisparity
• In video, usually called In video, usually called motion fieldmotion field
Computing Motion FieldComputing Motion Field
• Basic idea: a small portion of the Basic idea: a small portion of the imageimage(“local neighborhood”) shifts position(“local neighborhood”) shifts position
• AssumptionsAssumptions– No / small changes in reflected lightNo / small changes in reflected light– No / small changes in scaleNo / small changes in scale– No occlusion or disocclusionNo occlusion or disocclusion– Neighborhood is correct size: Neighborhood is correct size: apertureaperture
problemproblem
Actual and Apparent MotionActual and Apparent Motion
• If these assumptions violated, can still If these assumptions violated, can still use the same methods – use the same methods – apparent apparent motionmotion
• Result of algorithm is Result of algorithm is optical flowoptical flow (vs. (vs. ideal motion field)ideal motion field)
• Most obvious effects:Most obvious effects:– Aperture problem: can only get motion Aperture problem: can only get motion
perpendicular to edgesperpendicular to edges– Errors near discontinuities (occlusions)Errors near discontinuities (occlusions)
Computing Optical Flow:Computing Optical Flow:PreliminariesPreliminaries
• Image sequence I(x,y,t)Image sequence I(x,y,t)
• Uniform discretization along x,y,t –Uniform discretization along x,y,t –“cube” of data“cube” of data
• Differential framework: compute Differential framework: compute partial derivatives along x,y,t by partial derivatives along x,y,t by convolving with derivative of convolving with derivative of GaussianGaussian
Computing Optical Flow:Computing Optical Flow:Image Brightness ConstancyImage Brightness Constancy
• Basic idea: a small portion of the Basic idea: a small portion of the imageimage(“local neighborhood”) shifts position(“local neighborhood”) shifts position
• Brightness constancy assumptionBrightness constancy assumption
0dt
dI0
dt
dI
Computing Optical Flow:Computing Optical Flow:Image Brightness ConstancyImage Brightness Constancy
• This does not say that the image This does not say that the image remainsremainsthe same brightness!the same brightness!
• vs. : total vs. partial derivativevs. : total vs. partial derivative
• Use chain ruleUse chain rule t
I
dt
dy
y
I
dt
dx
x
I
dt
ttytxdI
),(),(
t
I
dt
dy
y
I
dt
dx
x
I
dt
ttytxdI
),(),(
dt
dIdt
dI
t
I
t
I
Computing Optical Flow:Computing Optical Flow:Image Brightness ConstancyImage Brightness Constancy
• Given optical flow Given optical flow vv(x,y)(x,y)
0)(
0
0),(),(
T
tII
t
I
dt
dy
y
I
dt
dx
x
Idt
ttytxdI
v
0)(
0
0),(),(
T
tII
t
I
dt
dy
y
I
dt
dx
x
Idt
ttytxdI
v
Image brightness constancy equationImage brightness constancy equation
Computing Optical Flow:Computing Optical Flow:DiscretizationDiscretization
• Look at some neighborhood N:Look at some neighborhood N:
0
0),(),(
want
want
N),(
T
bAv
v jiIjiIji
t
0
0),(),(
want
want
N),(
T
bAv
v jiIjiIji
t
),(
),(
),(
),(
),(
),(
22
11
22
11
nnt
t
t
nn jiI
jiI
jiI
jiI
jiI
jiI
bA
),(
),(
),(
),(
),(
),(
22
11
22
11
nnt
t
t
nn jiI
jiI
jiI
jiI
jiI
jiI
bA
Computing Optical Flow:Computing Optical Flow:Least SquaresLeast Squares
• In general, overconstrained linear In general, overconstrained linear systemsystem
• Solve by least squaresSolve by least squares
bAAAv
bAvAA
bAv
T1T
TT
want
)(
)(
0
bAAAv
bAvAA
bAv
T1T
TT
want
)(
)(
0
Computing Optical Flow:Computing Optical Flow:StabilityStability
• Has a solution unless Has a solution unless CC = = AATTAA is is singularsingular
Ny
Nyx
Nyx
Nx
nn
nn
III
III
jiI
jiI
jiI
jiIjiIjiI
2
2
22
11
2211
T
),(
),(
),(
),(),(),(
C
C
AAC
Ny
Nyx
Nyx
Nx
nn
nn
III
III
jiI
jiI
jiI
jiIjiIjiI
2
2
22
11
2211
T
),(
),(
),(
),(),(),(
C
C
AAC
Computing Optical Flow:Computing Optical Flow:StabilityStability
• Where have we encountered Where have we encountered CC before? before?
• Corner detector!Corner detector!
• C C is singular if intensity is constant is singular if intensity is constant oror if if there’s an edgethere’s an edge
• Use eigenvalues of Use eigenvalues of CC::– to evaluate stability of optical flow to evaluate stability of optical flow
computationcomputation– to find good places to compute optical flowto find good places to compute optical flow
(finding good features to track)(finding good features to track)
Computing Optical Flow:Computing Optical Flow:ImprovementsImprovements
• Assumption that optical flow is Assumption that optical flow is constant over neighborhood not constant over neighborhood not always goodalways good
• Decreasing size of neighborhood Decreasing size of neighborhood CC more likely to be singular more likely to be singular
• Alternative: weighted least-squaresAlternative: weighted least-squares– Points near center = higher weightPoints near center = higher weight– Still use larger neighborhoodStill use larger neighborhood
Computing Optical Flow:Computing Optical Flow:Weighted Least SquaresWeighted Least Squares
• Let Let WW be a matrix of weights be a matrix of weights
Wbb
WAA
Wbb
WAA
bWAAWAv
bAAAv2T12T
T1T
)(
)(
w bWAAWAv
bAAAv2T12T
T1T
)(
)(
w
Computing Optical Flow:Computing Optical Flow:ImprovementsImprovements
• What if windows are still bigger?What if windows are still bigger?
• Adjust motion model: no longer Adjust motion model: no longer constant within a windowconstant within a window
• Popular choice: affine modelPopular choice: affine model
Computing Optical Flow:Computing Optical Flow:Affine Motion ModelAffine Motion Model
• Translational modelTranslational model
• Affine modelAffine model
y
x
v
v
y
x
y
x
1
1
2
2
y
x
v
v
y
x
y
x
1
1
2
2
y
x
v
v
y
x
dc
ba
y
x
1
1
2
2
y
x
v
v
y
x
dc
ba
y
x
1
1
2
2
Computing Optical Flow:Computing Optical Flow:ImprovementsImprovements
• Larger motion: how to maintain Larger motion: how to maintain “differential” approximation?“differential” approximation?
• Solution: iterateSolution: iterate
• Even better: adjust window / smoothingEven better: adjust window / smoothing– Early iterations: use larger Gaussians to Early iterations: use larger Gaussians to
allow more motion allow more motion – Late iterations: use less blur to find exact Late iterations: use less blur to find exact
solution, lock on to high-frequency detailsolution, lock on to high-frequency detail
Computing Optical Flow:Computing Optical Flow:Lucas-KanadeLucas-Kanade
• Iterative algorithm:Iterative algorithm:1.1. Set Set = large (e.g. 3 pixels)= large (e.g. 3 pixels)
2.2. Set I’ Set I’ I I11
3.3. Set Set vv 0 0
4.4. Repeat while SSD(I’, IRepeat while SSD(I’, I22) > ) > 1.1. vv += Optical flow(I’ += Optical flow(I’ I I22))
2.2. I’ I’ Warp(I Warp(I11, , vv))
5.5. After After nn iterations, iterations,set set = small (e.g. 1.5 pixels) = small (e.g. 1.5 pixels)
Computing Optical Flow:Computing Optical Flow:Lucas-KanadeLucas-Kanade
• I’ always holds warped version of II’ always holds warped version of I11
– Best estimate of IBest estimate of I22
• Gradually reduce thresholdsGradually reduce thresholds
• Stop when difference between I’ and Stop when difference between I’ and II22 small small– Simplest difference metric = sum of Simplest difference metric = sum of
squared differences (SSD) between squared differences (SSD) between pixelspixels
Optical Flow ApplicationsOptical Flow Applications
[Feng & Perona][Feng & Perona]
Video FramesVideo Frames
Optical Flow ApplicationsOptical Flow Applications
Optical FlowOptical Flow Depth ReconstructionDepth Reconstruction
[Feng & Perona][Feng & Perona]
Optical Flow ApplicationsOptical Flow Applications
Obstacle Detection: Unbalanced Optical FlowObstacle Detection: Unbalanced Optical Flow
TemizerTemizer
Optical Flow ApplicationsOptical Flow Applications
• Collision Collision avoidance: keep avoidance: keep optical flow optical flow balanced between balanced between sides of imagesides of image
TemizerTemizer