MSU Fall 20141 Computing Motion from Images Chapter 9 of S&S plus otherwork.
-
Upload
elfrieda-watkins -
Category
Documents
-
view
213 -
download
0
Transcript of MSU Fall 20141 Computing Motion from Images Chapter 9 of S&S plus otherwork.
MSU Fall 2014 1
Computing Motion from Images
Chapter 9 of S&S plus otherwork.
MSU Fall 2014 2
General topics
Low level change detection Region tracking or matching over
time Interpretation of motion MPEG compression Interpretation of scene changes in
video Understanding human activites
MSU Fall 2014 3
Motion important to human vision
MSU Fall 2014 4
What’s moving: different cases
MSU Fall 2014 5
Image subtraction
Simple method to remove unchanging background from
moving regions.
MSU Fall 2014 6
Change detection for surveillance
MSU Fall 2014 7
Change detection by image subtraction
Closing = dilation+erosion
MSU Fall 2014 8
http://homepages.inf.ed.ac.uk/rbf/HIPR2/close.htm
MSU Fall 2014 9
What to do with regions of change?
Discard small regions Discard regions of non interesting
features Keep track of regions with
interesting features Track in future frames from motion
plus component features
MSU Fall 2014 10
Some effects of camera motion that can cause problems
MSU Fall 2014 11
Motion field
MSU Fall 2014 12
FOE and FOC
Will return to use the FOE or FOC or detection of panning to determine what the camera is doing in video tapes.
MSU Fall 2014 13
Gaming using a camera to recognize the player’s motion
Decathlete game
MSU Fall 2014 14
Decathlete game
Cheap camera replaces usual mouse for input
Running speed and jumping of the avatar is controlled by detected motion of the player’s hands.
MSU Fall 2014 15
Motion detection input device
Running (hands)Jumping (hands)
MSU Fall 2014 16
Motion analysis controls hurdling event (console)
• Top left shows video frame of player
• Middle left shows motion vectors from multiple frames
• Center shows jumping patterns
MSU Fall 2014 17
Related work
Motion sensed by crude cameras Person dances/gestures in space Kinect/Leap motion sensors System maps movement into
music Creative environment? Good exercise room?
MSU Fall 2014 18
Computing motion vectors from corresponding “points”
High energy neighborhoods are used to define points for
matching
MSU Fall 2014 19
Match points between frames
Such large motions are unusual. Most systems track small motions.
MSU Fall 2014 20
Requirements for interest points
Match small neighborhood to small neighborhood. The previous “scene” contains several highly textured neighborhoods.
MSU Fall 2014 21
Interest = minimum directional variance
Used by Hans Moravec in his robot stereo vision system.
Interest points were used for stereo matching.
MSU Fall 2014 22
Detecting interest points in I1
MSU Fall 2014 23
Match points from I1 in I2
MSU Fall 2014 24
Search for best match of point P1 in nearby window of I2
For both motion and stereo, we have some constraints on where to search for a matching interest point.
MSU Fall 2014 25
Motion vectors clustered to show 3 coherent regions
All motion vectors are clustered into 3 groups of similar vectors showing motion of 3 independent objects. (Dina Eldin)
Motion coherence: points of same object tend to move in the same way
MSU Fall 2014 26
Two frames of aerial imagery
Video frame N and N+1 shows slight movement: most pixels are same, just in different locations.
MSU Fall 2014 27
Can code frame N+d with displacments relative to frame N
for each 16 x 16 block in the 2nd image
find a closely matching block in the 1st image
replace the 16x16 intensities by the location in the 1st image (dX, dY)
256 bytes replaced by 2 bytes! (If blocks differ too much, encode
the differences to be added.)
MSU Fall 2014 28
Frame approximation
Left is original video frame N+1. Right is set of best image blocks taken from frame N. (Work of Dina Eldin)
MSU Fall 2014 29
Best matching blocks between video frames N+1 to N (motion vectors)
The bulk of the vectors show the true motion of the airplane taking the pictures. The long vectors are incorrect motion vectors, but they do work well for compression of image I2!
Best matches from 2nd to first image shown as vectors overlaid on the 2nd image. (Work by Dina Eldin.)
MSU Fall 2014 30
Motion coherence provides redundancy for compression
MPEG “motion compensation” represents motion of 16x16 pixels blocks, NOT objects
MSU Fall 2014 31
MPEG represents blocks that move by the motion vector
MSU Fall 2014 32
MPEG has ‘I’, ‘P’, and ‘B’ frames
MSU Fall 2014 33
Computing Image Flow
MSU Fall 2014 34
Motion Field & Optical Flow Field
Motion Field = Real world 3D motion
Optical Flow Field = Projection of the motion field onto the 2d image
3D motion vector
2D optical flow vector
vu,u
CCD
Slides from Lihi Zelnik-Manor
MSU Fall 2014 36
Assumptions
When does it break?
The screen is stationary yet displays motion
Homogeneous objects generate zero optical flow.
Fixed sphere. Changing light source.
Non-rigid texture motion
Slides from Lihi Zelnik-Manor
MSU Fall 2014 38
Image flow equation 1 of 2
MSU Fall 2014 39
Image flow equation 2 of 2
Estimating Optical Flow Assume the image intensity is
constant
tyxI ,, dttdyydxxI ,,
ITime = t Time = t+dt
dyydxx ,
yx,
Slides from Lihi Zelnik-Manor
MSU Fall 2014 40
Brightness Constancy Equation dttdyydxxItyxI ,,,,
dttI
dyyI
dxxI
tyxI
,,
First order Taylor Expansion
0 dtIdyIdxI tyx
Simplify notations:
Divide by dt and denote:
dtdx
u dtdy
v
tyx IvIuI Problem I: One equation, two unknowns
Slides from Lihi Zelnik-Manor MSU Fall 2014 41
Time t+dt
Problem II: “The Aperture Problem”
Time t
?Time t+dt
Where did the yellow point move to?
We need additional constraints
For points on a line of fixed intensity we can only recover the normal flow
Slides from Lihi Zelnik-Manor
MSU Fall 2014 42
Use Local InformationSometimes enlarging the aperture can help
Slides from Lihi Zelnik-Manor
MSU Fall 2014 43
Local smoothnessLucas Kanade (1984)
Assume constant (u,v) in small neighborhood
tyx IvIuI tyx Iv
uII
2
1
22
11
t
t
yx
yx
I
I
v
uII
II
bA u
Slides from Lihi Zelnik-Manor
MSU Fall 2014 44
Lucas Kanade (1984)
bA u
bAAA TT 1u
Goal: Minimize2u bA
bAAA TT u
2x2 2x1 2x1
Method: Least-Squares
Slides from Lihi Zelnik-Manor
MSU Fall 2014 45
Lucas-Kanade Solution
2
2
yyx
yxxT
III
IIIAA
We want this matrix to be invertible.i.e., no zero eigenvalues
bAAA TT 1u
Slides from Lihi Zelnik-Manor
MSU Fall 2014 46
Break-downs Brightness constancy is not satisfied
A point does not move like its neighbors what is the ideal window size?
The motion is not small (Taylor expansion doesn’t hold)
Correlation based methods
Regularization based methods
Use multi-scale estimation
Slides from Lihi Zelnik-Manor
MSU Fall 2014 47
Multi-Scale Flow Estimation
image It-1 image I
Gaussian pyramid of image It Gaussian pyramid of image It+1
image It+1image Itu=10 pixels
u=5 pixels
u=2.5 pixels
u=1.25 pixels
Slides from Lihi Zelnik-Manor
MSU Fall 2014 48
MSU Fall 2014 49
Tracking several objects
Use assumptions of physics to compute multiple smooth
paths.(work of Sethi and R. Jain)
MSU Fall 2014 50
MSU Fall 2014 51
Tracking in images over time
MSU Fall 2014 52
General constraints from physics
MSU Fall 2014 53
Other possible constraints
Background statistics stable Object color/texture/shape might
change slowly over frames Might have knowledge of objects
under survielance Objects appear/disappear at
boundary of the frame
MSU Fall 2014 54
MSU Fall 2014 55
Sethi-Jain algorithm
MSU Fall 2014 56
MSU Fall 2014 57
Total smoothness of m paths
MSU Fall 2014 58
Greedy exchange algorithm
MSU Fall 2014 59
Example data structure
Total smoothness for trajectories of Figure 9.14
MSU Fall 2014 60
Example of domain specific tracking (Vera Bakic)
Tracking eyes and nose of PC user. System presents menu (top). User moves face to position cursor to a particular box (choice). System tracks face movement and moves cursor accordingly: user gets into feedback-control loop.
MSU Fall 2014 61
Segmentation of videos/movies
Segment into scenes, shots, specific actions, etc.
MSU Fall 2014 62
Types of changes in videos
MSU Fall 2014 63
Anchor person scene at left
Street scene for news story
Scene break
From Zhang et al 1993
How do we compute the scene change?
MSU Fall 2014 64
Histograms of frames across the scene change
Histograms at left are from anchor person frames, while histogram at bottom right is from the street frame.
MSU Fall 2014 66
American sign language example
MSU Fall 2014 67
Example from Yang and Ahuja
MSU Fall 2014 68
MSU Fall 2014 69