On-the-fly Specific Person Retrieval University of Oxford 24 th May 2012 Omkar M. Parkhi, Andrea...
-
Upload
amir-matlock -
Category
Documents
-
view
215 -
download
0
Transcript of On-the-fly Specific Person Retrieval University of Oxford 24 th May 2012 Omkar M. Parkhi, Andrea...
![Page 1: On-the-fly Specific Person Retrieval University of Oxford 24 th May 2012 Omkar M. Parkhi, Andrea Vedaldi and Andrew Zisserman.](https://reader036.fdocuments.us/reader036/viewer/2022062515/56649c9a5503460f94958484/html5/thumbnails/1.jpg)
On-the-fly Specific Person Retrieval
University of Oxford 24th May 2012
Omkar M. Parkhi, Andrea Vedaldi and Andrew Zisserman
![Page 2: On-the-fly Specific Person Retrieval University of Oxford 24 th May 2012 Omkar M. Parkhi, Andrea Vedaldi and Andrew Zisserman.](https://reader036.fdocuments.us/reader036/viewer/2022062515/56649c9a5503460f94958484/html5/thumbnails/2.jpg)
Overview
People“Barack Obama”“George Bush”“Courtney Cox”
Ranked ShotsTextual QueriesSearch for:
On-the-flyi.e. with no previous knowledgeor model for these queries
Large collection of
un-annotated videos
![Page 3: On-the-fly Specific Person Retrieval University of Oxford 24 th May 2012 Omkar M. Parkhi, Andrea Vedaldi and Andrew Zisserman.](https://reader036.fdocuments.us/reader036/viewer/2022062515/56649c9a5503460f94958484/html5/thumbnails/3.jpg)
Scrubs Data Set
• 12 Episodes from Seasons 1-5 and 8
• 5 hours of video data
• About 400k frames, partitioned into 5k shots
• About 300k near frontal face detections
• 768 x 576 MPEG2 format
![Page 4: On-the-fly Specific Person Retrieval University of Oxford 24 th May 2012 Omkar M. Parkhi, Andrea Vedaldi and Andrew Zisserman.](https://reader036.fdocuments.us/reader036/viewer/2022062515/56649c9a5503460f94958484/html5/thumbnails/4.jpg)
Demo
• Search for “Courteney Cox” in Scrubs dataset.
• Steps:
1. Download example images from Google
2. Train a ranking function
3. Apply ranking function to video collection
![Page 5: On-the-fly Specific Person Retrieval University of Oxford 24 th May 2012 Omkar M. Parkhi, Andrea Vedaldi and Andrew Zisserman.](https://reader036.fdocuments.us/reader036/viewer/2022062515/56649c9a5503460f94958484/html5/thumbnails/5.jpg)
DEMO
![Page 6: On-the-fly Specific Person Retrieval University of Oxford 24 th May 2012 Omkar M. Parkhi, Andrea Vedaldi and Andrew Zisserman.](https://reader036.fdocuments.us/reader036/viewer/2022062515/56649c9a5503460f94958484/html5/thumbnails/6.jpg)
On the fly person retrieval systemText Query“Courteney
Cox” Negative Training Images
Video Collection
Face Tracks
Facial Features &Descriptors
Facial Features &Descriptors
Results
ON-LINE PROCESSING
OFF-LINE PROCESSING
Google Image Search
“Courteney Cox”
Facial Features &
Descriptors
Fast Linear Classifier
Ranking
![Page 7: On-the-fly Specific Person Retrieval University of Oxford 24 th May 2012 Omkar M. Parkhi, Andrea Vedaldi and Andrew Zisserman.](https://reader036.fdocuments.us/reader036/viewer/2022062515/56649c9a5503460f94958484/html5/thumbnails/7.jpg)
Detection and Tracking
• Viola-Jones face detection on each frame• Tracking measures “connectedness” of a pair of faces by point
tracks intersecting both• Doesn’t require contiguous detections• No drift• Faces clustered into tracks
[Everingham et al. 2006, Apostoloff & Zisserman, 2007]
![Page 8: On-the-fly Specific Person Retrieval University of Oxford 24 th May 2012 Omkar M. Parkhi, Andrea Vedaldi and Andrew Zisserman.](https://reader036.fdocuments.us/reader036/viewer/2022062515/56649c9a5503460f94958484/html5/thumbnails/8.jpg)
Scrubs Data Set
• 12 Episodes from Seasons 1-5 and 8
• 5 hours of video data
• About 400k frames, partitioned into 5k shots
• 300k face detections
• 6k face tracks
• 768 x 576 MPEG2 format
![Page 9: On-the-fly Specific Person Retrieval University of Oxford 24 th May 2012 Omkar M. Parkhi, Andrea Vedaldi and Andrew Zisserman.](https://reader036.fdocuments.us/reader036/viewer/2022062515/56649c9a5503460f94958484/html5/thumbnails/9.jpg)
Detecting facial feature points
• Pictorial structure model
• Joint model of feature appearance and position
[Felzenszwalb and Huttenlocher’2004, Everingham et al. 2006]
![Page 10: On-the-fly Specific Person Retrieval University of Oxford 24 th May 2012 Omkar M. Parkhi, Andrea Vedaldi and Andrew Zisserman.](https://reader036.fdocuments.us/reader036/viewer/2022062515/56649c9a5503460f94958484/html5/thumbnails/10.jpg)
Face Appearance Representation
Affine transformation of face to canonical frame Independent photometric normalization of parts Represent gradients over circle centred on facial feature points Feature descriptor is a 3849 dimensional vector
[Everingham et al. 2006]
![Page 11: On-the-fly Specific Person Retrieval University of Oxford 24 th May 2012 Omkar M. Parkhi, Andrea Vedaldi and Andrew Zisserman.](https://reader036.fdocuments.us/reader036/viewer/2022062515/56649c9a5503460f94958484/html5/thumbnails/11.jpg)
Negative Training Images
• Combination of faces from• Random downloaded images• Labeled Faces in the Wild dataset• Caltech Faces dataset
• About 16k face detections.
Caltech 10, 000 Web Faces:http://www.vision.caltech.edu/Image_Datasets/Caltech_10K_WebFaces/Labeled Faces in the Wild:http://vis-www.cs.umass.edu/lfw/
![Page 12: On-the-fly Specific Person Retrieval University of Oxford 24 th May 2012 Omkar M. Parkhi, Andrea Vedaldi and Andrew Zisserman.](https://reader036.fdocuments.us/reader036/viewer/2022062515/56649c9a5503460f94958484/html5/thumbnails/12.jpg)
On-the-fly Person RetrievalText Query“Courteney
Cox” Negative Training Images
Video Collection
Face Tracks
Facial Features &Descriptors
Facial Features &Descriptors
Results
ON-LINE PROCESSING
OFF-LINE PROCESSING
Google Image Search
“Courteney Cox”
Facial Features &
Descriptors
Fast Linear Classifier
Ranking
![Page 13: On-the-fly Specific Person Retrieval University of Oxford 24 th May 2012 Omkar M. Parkhi, Andrea Vedaldi and Andrew Zisserman.](https://reader036.fdocuments.us/reader036/viewer/2022062515/56649c9a5503460f94958484/html5/thumbnails/13.jpg)
DEMO
![Page 14: On-the-fly Specific Person Retrieval University of Oxford 24 th May 2012 Omkar M. Parkhi, Andrea Vedaldi and Andrew Zisserman.](https://reader036.fdocuments.us/reader036/viewer/2022062515/56649c9a5503460f94958484/html5/thumbnails/14.jpg)
TRECVid 2011 (IACC.1.B)
• About 200 hours of video data.
• 8k videos.
• MPEG4, 320x240 pixels
• 130k shots,
• About 3 million face detections
• 25,535 face tracks.
![Page 15: On-the-fly Specific Person Retrieval University of Oxford 24 th May 2012 Omkar M. Parkhi, Andrea Vedaldi and Andrew Zisserman.](https://reader036.fdocuments.us/reader036/viewer/2022062515/56649c9a5503460f94958484/html5/thumbnails/15.jpg)
DEMO
![Page 16: On-the-fly Specific Person Retrieval University of Oxford 24 th May 2012 Omkar M. Parkhi, Andrea Vedaldi and Andrew Zisserman.](https://reader036.fdocuments.us/reader036/viewer/2022062515/56649c9a5503460f94958484/html5/thumbnails/16.jpg)
Facial attributes – FaceTracer project
Examples: gender: male, female age: baby, child, youth, middle age, senior race: white, black, asian smiling, mustache, eye-wear, hair colour
N. Kumar, P. N. Belhumeur and S. K. Nayar, FaceTracer: A Search Engine for Large Collections of Images with Faces,European Conference on Computer Vision (ECCV), 2010http://www.cs.columbia.edu/CAVE/projects/face_search/
Method
• person independent training set with attribute
• facial feature representation
• discriminative training of classifier for attribute
![Page 17: On-the-fly Specific Person Retrieval University of Oxford 24 th May 2012 Omkar M. Parkhi, Andrea Vedaldi and Andrew Zisserman.](https://reader036.fdocuments.us/reader036/viewer/2022062515/56649c9a5503460f94958484/html5/thumbnails/17.jpg)
DEMO
![Page 18: On-the-fly Specific Person Retrieval University of Oxford 24 th May 2012 Omkar M. Parkhi, Andrea Vedaldi and Andrew Zisserman.](https://reader036.fdocuments.us/reader036/viewer/2022062515/56649c9a5503460f94958484/html5/thumbnails/18.jpg)
Quantitative Performance - Scrubs Dataset
• Performance evaluation for 3 guest actors (Brendan Fraser, Courteney Cox and Michael J Fox)
• 12 dataset videos split into training and test sets (3 Training, 9 Testing)
• Annotations:
• Manual labeling of training and test set for each actor
• Manual labeling of positive training images from Google
• Negative training images from Caltech Faces dataset.
![Page 19: On-the-fly Specific Person Retrieval University of Oxford 24 th May 2012 Omkar M. Parkhi, Andrea Vedaldi and Andrew Zisserman.](https://reader036.fdocuments.us/reader036/viewer/2022062515/56649c9a5503460f94958484/html5/thumbnails/19.jpg)
Quantitative Performance - Scrubs Dataset
• Retrieval Average Precision (AP)
Training Examples Source Average Precision
Positive NegativeBrendan Fraser
Courteney Cox
Michael J Fox
Scrubs Scrubs 0.56 0.88 0.49
Google Scrubs 0.25 0.62 0.52
Google Caltech 0.41 0.56 0.57
Brendan Fraser Courteney Cox Michael J Fox0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1
+ve=Scrubs -ve=Scrubs+ve=Google -ve=Scrubs+ve=Google -ve=Caltech
![Page 20: On-the-fly Specific Person Retrieval University of Oxford 24 th May 2012 Omkar M. Parkhi, Andrea Vedaldi and Andrew Zisserman.](https://reader036.fdocuments.us/reader036/viewer/2022062515/56649c9a5503460f94958484/html5/thumbnails/20.jpg)
Quantitative Performance - Scrubs Dataset
• Using more training data per track
Training Examples Source # samples
per trackAverage Precision
Positive NegativeBrendan Fraser
Courteney Cox
Michael J Fox
Scrubs Scrubs Single 0.56 0.88 0.49Scrubs Scrubs Multiple 0.6 0.88 0.53
Brendan Fraser Courteney Cox Michael J Fox0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1
+ve=Scrubs -ve=Scrubs Single+ve=Scrubs -ve=Scrubs Mul-tiple
![Page 21: On-the-fly Specific Person Retrieval University of Oxford 24 th May 2012 Omkar M. Parkhi, Andrea Vedaldi and Andrew Zisserman.](https://reader036.fdocuments.us/reader036/viewer/2022062515/56649c9a5503460f94958484/html5/thumbnails/21.jpg)
Future Work
• Exploring sources for positive examples
• Better feature representations
• Combination of attributes and identities
![Page 22: On-the-fly Specific Person Retrieval University of Oxford 24 th May 2012 Omkar M. Parkhi, Andrea Vedaldi and Andrew Zisserman.](https://reader036.fdocuments.us/reader036/viewer/2022062515/56649c9a5503460f94958484/html5/thumbnails/22.jpg)
Any Questions?