Prakash Chockalingam Clemson University Non-Rigid Multi-Modal Object Tracking Using Gaussian Mixture...
-
Upload
michael-palmer -
Category
Documents
-
view
214 -
download
0
Transcript of Prakash Chockalingam Clemson University Non-Rigid Multi-Modal Object Tracking Using Gaussian Mixture...
![Page 1: Prakash Chockalingam Clemson University Non-Rigid Multi-Modal Object Tracking Using Gaussian Mixture Models Committee Members Dr Stan Birchfield (chair)](https://reader030.fdocuments.us/reader030/viewer/2022032708/56649e5d5503460f94b56adc/html5/thumbnails/1.jpg)
Prakash ChockalingamPrakash Chockalingam
Clemson UniversityClemson University
Non-Rigid Multi-Modal Object Tracking Using Gaussian Mixture Models
Committee MembersCommittee Members
Dr Stan Birchfield (chair)Dr Stan Birchfield (chair)Dr Robert SchalkoffDr Robert Schalkoff
Dr Brian DeanDr Brian Dean
![Page 2: Prakash Chockalingam Clemson University Non-Rigid Multi-Modal Object Tracking Using Gaussian Mixture Models Committee Members Dr Stan Birchfield (chair)](https://reader030.fdocuments.us/reader030/viewer/2022032708/56649e5d5503460f94b56adc/html5/thumbnails/2.jpg)
Tracking OverviewTracking Overview
Tracker Tasks
Feature Descriptors
Object Model
Update / LearningMechanism
TrackingFramework
Color
Gradients
Texture
Shape
Motion
Template
Contour
Active Appearance
ProbabilityDensities
Mean Shift
Pixel-wiseClassification
Optical Flow
Filtering techniques
No Update
Adaboost
Expectation Maximization
Re-weightingStrategy
Object Detection
Manual
Segmentation
Feature Points
![Page 3: Prakash Chockalingam Clemson University Non-Rigid Multi-Modal Object Tracking Using Gaussian Mixture Models Committee Members Dr Stan Birchfield (chair)](https://reader030.fdocuments.us/reader030/viewer/2022032708/56649e5d5503460f94b56adc/html5/thumbnails/3.jpg)
ApproachApproach
• Tracking Framework: Target and background is modeled as a mixture of Gaussians in a joint feature-spatial space. A strength map is computed indicating the probability of each pixel belonging to the foreground.
• Contour Extraction: Contour is extracted using a discrete implementation of level sets
• Image Segmentation: Each Gaussian (fragment) is adapted to the image data by segmenting the image.
• Update Mechanism: The parameters of all the Gaussians are updated based on tracked data
• Results
![Page 4: Prakash Chockalingam Clemson University Non-Rigid Multi-Modal Object Tracking Using Gaussian Mixture Models Committee Members Dr Stan Birchfield (chair)](https://reader030.fdocuments.us/reader030/viewer/2022032708/56649e5d5503460f94b56adc/html5/thumbnails/4.jpg)
Tracking FrameworkTracking Framework
Bayesian Formulation:Bayesian Formulation:
0: 0: 1 0: 1
arg
( | , ) ( | ) ( | ) ( | )t t t t t t t t t
t et background shape
p I p I p I p
Image data of all frames
Contour at time t Previously seen contours
*
**( | ) ( | )t t t
y R
p I p y
Assuming conditional independence among pixels,
Feature vector
* { , }
![Page 5: Prakash Chockalingam Clemson University Non-Rigid Multi-Modal Object Tracking Using Gaussian Mixture Models Committee Members Dr Stan Birchfield (chair)](https://reader030.fdocuments.us/reader030/viewer/2022032708/56649e5d5503460f94b56adc/html5/thumbnails/5.jpg)
Object ModelingObject Modeling
f1
f2
*( | )tp y ?
Gaussian Mixture Model (GMM):
*
*1
( | ) ( | , )k
t tj
prior likelihood
p j p y j
( | )tp j * * 1 **
1( | , ) exp{ ( ) ( ) ( )}
2T
t j j jp y j y y
Strength Image:
( )( ) log
( )
p xx
p x
>0 for Foreground<0 for Background
y
![Page 6: Prakash Chockalingam Clemson University Non-Rigid Multi-Modal Object Tracking Using Gaussian Mixture Models Committee Members Dr Stan Birchfield (chair)](https://reader030.fdocuments.us/reader030/viewer/2022032708/56649e5d5503460f94b56adc/html5/thumbnails/6.jpg)
Strength ImageStrength Image
GMM Linear Classifier Single Gaussian
![Page 7: Prakash Chockalingam Clemson University Non-Rigid Multi-Modal Object Tracking Using Gaussian Mixture Models Committee Members Dr Stan Birchfield (chair)](https://reader030.fdocuments.us/reader030/viewer/2022032708/56649e5d5503460f94b56adc/html5/thumbnails/7.jpg)
Strength Image (contd…)Strength Image (contd…)
…
Linear ClassifierSingle Gaussian
Individual Fragments
Final Strength Strength Without Spatial Information
![Page 8: Prakash Chockalingam Clemson University Non-Rigid Multi-Modal Object Tracking Using Gaussian Mixture Models Committee Members Dr Stan Birchfield (chair)](https://reader030.fdocuments.us/reader030/viewer/2022032708/56649e5d5503460f94b56adc/html5/thumbnails/8.jpg)
TopicsTopics
• Tracking Framework: Target and background is modeled as a mixture of Gaussians in a joint feature-spatial space. A strength map is computed indicating the probability of each pixel belonging to the foreground.
• Contour Extraction: Contour is extracted using a discrete implementation of level sets
• Image Segmentation: Each Gaussian (fragment) is adapted to the image data by segmenting the image.
• Update Mechanism: The parameters of all the Gaussians are updated based on tracked data
• Results
![Page 9: Prakash Chockalingam Clemson University Non-Rigid Multi-Modal Object Tracking Using Gaussian Mixture Models Committee Members Dr Stan Birchfield (chair)](https://reader030.fdocuments.us/reader030/viewer/2022032708/56649e5d5503460f94b56adc/html5/thumbnails/9.jpg)
Contour ExtractionContour Extraction
Implicit representation of growing region
Likelihood term(Strength image) Regularization term
Energy Functional:
(strength image) (frontier)
( ) ( ) * ( )x x
E x x G x
> 0 Inside< 0 Outside
![Page 10: Prakash Chockalingam Clemson University Non-Rigid Multi-Modal Object Tracking Using Gaussian Mixture Models Committee Members Dr Stan Birchfield (chair)](https://reader030.fdocuments.us/reader030/viewer/2022032708/56649e5d5503460f94b56adc/html5/thumbnails/10.jpg)
Contour Extraction (contd…)Contour Extraction (contd…)
( ) 0, ( ) 0y y
( ) 0, ( ) 0y y
( ) 0, ( ) 0y y
( ) 0, ( ) 0y y
(Region to be shrunk)
(Region already grown)
(Region to be grown)
(Region that need not be considered)
![Page 11: Prakash Chockalingam Clemson University Non-Rigid Multi-Modal Object Tracking Using Gaussian Mixture Models Committee Members Dr Stan Birchfield (chair)](https://reader030.fdocuments.us/reader030/viewer/2022032708/56649e5d5503460f94b56adc/html5/thumbnails/11.jpg)
Contour Extraction (contd…)Contour Extraction (contd…)
4{ ' : ' ( ), ( ') 0, ( ') 0}gx x N x x x
4{ : ' ( )x x N x ( ) 0, ( ') 0}gx x such that
x x’
xx’
4{ ' : ' ( ), ( ') 0, ( ') 0}gx x N x x x
4{ : ' ( )x x N x ( ) 0, ( ') 0}g gx x such that
Dilation
Contraction
![Page 12: Prakash Chockalingam Clemson University Non-Rigid Multi-Modal Object Tracking Using Gaussian Mixture Models Committee Members Dr Stan Birchfield (chair)](https://reader030.fdocuments.us/reader030/viewer/2022032708/56649e5d5503460f94b56adc/html5/thumbnails/12.jpg)
Contour Extraction (contd…)Contour Extraction (contd…)
Expand
Remove interior points
Contract
Remove exterior points
![Page 13: Prakash Chockalingam Clemson University Non-Rigid Multi-Modal Object Tracking Using Gaussian Mixture Models Committee Members Dr Stan Birchfield (chair)](https://reader030.fdocuments.us/reader030/viewer/2022032708/56649e5d5503460f94b56adc/html5/thumbnails/13.jpg)
Contour Extraction (contd…)Contour Extraction (contd…)
Likelihood
Final Region
![Page 14: Prakash Chockalingam Clemson University Non-Rigid Multi-Modal Object Tracking Using Gaussian Mixture Models Committee Members Dr Stan Birchfield (chair)](https://reader030.fdocuments.us/reader030/viewer/2022032708/56649e5d5503460f94b56adc/html5/thumbnails/14.jpg)
TopicsTopics
• Tracking Framework: Target and background is modeled as a mixture of Gaussians in a joint feature-spatial space. A strength map is computed indicating the probability of each pixel belonging to the foreground.
• Contour Extraction: Contour is extracted using a discrete implementation of level sets
• Image Segmentation: Each Gaussian (fragment) is adapted to the image data by segmenting the image.
• Update Mechanism: The parameters of all the Gaussians are updated based on tracked data
• Results
![Page 15: Prakash Chockalingam Clemson University Non-Rigid Multi-Modal Object Tracking Using Gaussian Mixture Models Committee Members Dr Stan Birchfield (chair)](https://reader030.fdocuments.us/reader030/viewer/2022032708/56649e5d5503460f94b56adc/html5/thumbnails/15.jpg)
Region SegmentationRegion Segmentation
Mode-seeking region growing algorithm:
do {
• Pick a seed point that is not associated to any fragment
• Grow the fragment from the seed point based on the similarity of the pixel and its neighbor’s appearance
• Stop growing the fragment if no more similar pixels are present in the neighborhood of the fragment
} until all pixels are assigned
Seed point:
3
1
( ) ( )ii
x x
Eigen values of 3x3 RGB covariance matrix
1,..., nS where ( , , )i i i ix y
![Page 16: Prakash Chockalingam Clemson University Non-Rigid Multi-Modal Object Tracking Using Gaussian Mixture Models Committee Members Dr Stan Birchfield (chair)](https://reader030.fdocuments.us/reader030/viewer/2022032708/56649e5d5503460f94b56adc/html5/thumbnails/16.jpg)
Region Segmentation (contd…)Region Segmentation (contd…)
• Pick the minimum element in S. Create a region to hold the pixel and add the neighbors in a fixed window.
• Compute Mean μj and Covariance Σj of the region.
• Likelihood:
• Grow the region as before with two additional steps: Update μj, and Σj, as a new pixel is added Remove the corresponding element in S if a pixel is added
• Continue above steps if S is not empty.
( ) ( ( ), ( , ))j j jx MD f x
Initial region
Mahalanobis distance
Configurable parameter
![Page 17: Prakash Chockalingam Clemson University Non-Rigid Multi-Modal Object Tracking Using Gaussian Mixture Models Committee Members Dr Stan Birchfield (chair)](https://reader030.fdocuments.us/reader030/viewer/2022032708/56649e5d5503460f94b56adc/html5/thumbnails/17.jpg)
Region Segmentation (contd…)Region Segmentation (contd…)
Region Growing Graph-Based Mean-Shift
![Page 18: Prakash Chockalingam Clemson University Non-Rigid Multi-Modal Object Tracking Using Gaussian Mixture Models Committee Members Dr Stan Birchfield (chair)](https://reader030.fdocuments.us/reader030/viewer/2022032708/56649e5d5503460f94b56adc/html5/thumbnails/18.jpg)
Region Segmentation (contd…)Region Segmentation (contd…)
Region Growing Graph-Based Mean-Shift
![Page 19: Prakash Chockalingam Clemson University Non-Rigid Multi-Modal Object Tracking Using Gaussian Mixture Models Committee Members Dr Stan Birchfield (chair)](https://reader030.fdocuments.us/reader030/viewer/2022032708/56649e5d5503460f94b56adc/html5/thumbnails/19.jpg)
TopicsTopics
• Tracking Framework: Target and background is modeled as a mixture of Gaussians in a joint feature-spatial space. A strength map is computed indicating the probability of each pixel belonging to the foreground.
• Contour Extraction: Contour is extracted using a discrete implementation of level sets
• Image Segmentation: Each Gaussian (fragment) is adapted to the image data by segmenting the image.
• Update Mechanism: The parameters of all the Gaussians are updated based on tracked data
• Results
![Page 20: Prakash Chockalingam Clemson University Non-Rigid Multi-Modal Object Tracking Using Gaussian Mixture Models Committee Members Dr Stan Birchfield (chair)](https://reader030.fdocuments.us/reader030/viewer/2022032708/56649e5d5503460f94b56adc/html5/thumbnails/20.jpg)
Update MechanismUpdate Mechanism
f1
f2
* *, ,,j t j t
• Update parameters of existing fragments
• Detect fragment occlusion
• Find new fragments
Initial Frame Initial Model Fragment Association
![Page 21: Prakash Chockalingam Clemson University Non-Rigid Multi-Modal Object Tracking Using Gaussian Mixture Models Committee Members Dr Stan Birchfield (chair)](https://reader030.fdocuments.us/reader030/viewer/2022032708/56649e5d5503460f94b56adc/html5/thumbnails/21.jpg)
Update Mechanism (contd…)Update Mechanism (contd…)
* * * * *, ,0: ,0(1 )j t j j t j j
( ) *,
* 0,0:
( )
0
tt
j
j t tt
e
e
Initial Model
(function of past and current values)
Weight computed by comparing Mahalanobis distance
Updating parameters of existing fragments:
![Page 22: Prakash Chockalingam Clemson University Non-Rigid Multi-Modal Object Tracking Using Gaussian Mixture Models Committee Members Dr Stan Birchfield (chair)](https://reader030.fdocuments.us/reader030/viewer/2022032708/56649e5d5503460f94b56adc/html5/thumbnails/22.jpg)
Update Mechanism (contd…)Update Mechanism (contd…)
Occluded fragments:
If a fragment is associated with less than 0.2% of the image pixels, then the fragment is declared as occluded.
Finding new fragments:
( ){ : log 0}
( )
p xT x
p x
Helps in handling self-occlusion
![Page 23: Prakash Chockalingam Clemson University Non-Rigid Multi-Modal Object Tracking Using Gaussian Mixture Models Committee Members Dr Stan Birchfield (chair)](https://reader030.fdocuments.us/reader030/viewer/2022032708/56649e5d5503460f94b56adc/html5/thumbnails/23.jpg)
Spatial AlignmentSpatial Alignment
The spatial parameters are updated using the motion vectors from Joint Lucas-Kanade approach
Lucas-Kanade Joint Lucas-Kanade
![Page 24: Prakash Chockalingam Clemson University Non-Rigid Multi-Modal Object Tracking Using Gaussian Mixture Models Committee Members Dr Stan Birchfield (chair)](https://reader030.fdocuments.us/reader030/viewer/2022032708/56649e5d5503460f94b56adc/html5/thumbnails/24.jpg)
Algorithm summaryAlgorithm summaryInitial frame:
• The user marks the object to be tracked.
• The target object and background scene are segmented based on their appearance similarity.
• The target object and background scene are modeled using a mixture of Gaussians where each Gaussian correspond to a fragment in the joint feature-spatial space
Subsequent frames:
• Update the spatial parameters of GMM using the motion vectors of Joint Lucas-Kanade
• Each pixel is classified into either foreground or background by generating a strength map using the Gaussian mixture model (GMM) of the object and background.
• The strength map is integrated into a discrete level set formulation to obtain accurate contour of the object.
• Using the tracked data, the appearance parameters of the GMM are updated.
![Page 25: Prakash Chockalingam Clemson University Non-Rigid Multi-Modal Object Tracking Using Gaussian Mixture Models Committee Members Dr Stan Birchfield (chair)](https://reader030.fdocuments.us/reader030/viewer/2022032708/56649e5d5503460f94b56adc/html5/thumbnails/25.jpg)
TopicsTopics
• Tracking Framework: Target and background is modeled as a mixture of Gaussians in a joint feature-spatial space. A strength map is computed indicating the probability of each pixel belonging to the foreground.
• Contour Extraction: Extract contour using a discrete implementation of level sets
• Image Segmentation: Each Gaussian (fragment) is adapted to the image data by segmenting the image.
• Update Mechanism: The parameters of all the Gaussians are updated based on tracked data
• Results
![Page 26: Prakash Chockalingam Clemson University Non-Rigid Multi-Modal Object Tracking Using Gaussian Mixture Models Committee Members Dr Stan Birchfield (chair)](https://reader030.fdocuments.us/reader030/viewer/2022032708/56649e5d5503460f94b56adc/html5/thumbnails/26.jpg)
Experimental ResultsExperimental Results
Elmo Sequence Monkey Sequence
![Page 27: Prakash Chockalingam Clemson University Non-Rigid Multi-Modal Object Tracking Using Gaussian Mixture Models Committee Members Dr Stan Birchfield (chair)](https://reader030.fdocuments.us/reader030/viewer/2022032708/56649e5d5503460f94b56adc/html5/thumbnails/27.jpg)
Experimental Results (Contd…)Experimental Results (Contd…)
Person Sequence Fish Sequence
![Page 28: Prakash Chockalingam Clemson University Non-Rigid Multi-Modal Object Tracking Using Gaussian Mixture Models Committee Members Dr Stan Birchfield (chair)](https://reader030.fdocuments.us/reader030/viewer/2022032708/56649e5d5503460f94b56adc/html5/thumbnails/28.jpg)
Experimental Results: Self-Occlusion Experimental Results: Self-Occlusion
Without Self-Occlusion Module With Self-Occlusion Module
![Page 29: Prakash Chockalingam Clemson University Non-Rigid Multi-Modal Object Tracking Using Gaussian Mixture Models Committee Members Dr Stan Birchfield (chair)](https://reader030.fdocuments.us/reader030/viewer/2022032708/56649e5d5503460f94b56adc/html5/thumbnails/29.jpg)
ConclusionConclusion
• A tracking framework based on modeling the object as mixture of Gaussians is proposed
• An efficient discrete implementation of level sets is employed to extract contour.
• A mode-seeking region growing algorithm is used to segment the image.
• A simple re-weighting strategy is proposed to update the parameters of Gaussians.
Future Directions:
• Incorporate shape priors.
• Utilize the extracted shapes to learn more robust priors.
• An offline or online evaluation mechanism during the initialization phase.
• Adding global information into the region segmentation process.
• Automating the object detection and initialization.
![Page 30: Prakash Chockalingam Clemson University Non-Rigid Multi-Modal Object Tracking Using Gaussian Mixture Models Committee Members Dr Stan Birchfield (chair)](https://reader030.fdocuments.us/reader030/viewer/2022032708/56649e5d5503460f94b56adc/html5/thumbnails/30.jpg)
Questions ?Questions ?
![Page 31: Prakash Chockalingam Clemson University Non-Rigid Multi-Modal Object Tracking Using Gaussian Mixture Models Committee Members Dr Stan Birchfield (chair)](https://reader030.fdocuments.us/reader030/viewer/2022032708/56649e5d5503460f94b56adc/html5/thumbnails/31.jpg)
Thank you !Thank you !