Activity Recognition in Videofiles.meetup.com/4379272/Bostion Image Processing Meetup.pdf ·...
Transcript of Activity Recognition in Videofiles.meetup.com/4379272/Bostion Image Processing Meetup.pdf ·...
![Page 1: Activity Recognition in Videofiles.meetup.com/4379272/Bostion Image Processing Meetup.pdf · Document Query 2/7/2013 8. Video Indexing Process 2/7/2013 9 Blob Extraction Source Video(s)](https://reader033.fdocuments.us/reader033/viewer/2022050107/5f451be722d10d14970f3ae2/html5/thumbnails/1.jpg)
Activity Recognition in VideoShashi Kant
Cognika
www.cognika.com
February 6, 2013
![Page 2: Activity Recognition in Videofiles.meetup.com/4379272/Bostion Image Processing Meetup.pdf · Document Query 2/7/2013 8. Video Indexing Process 2/7/2013 9 Blob Extraction Source Video(s)](https://reader033.fdocuments.us/reader033/viewer/2022050107/5f451be722d10d14970f3ae2/html5/thumbnails/2.jpg)
Cognika Introduction
2/7/2013 2
MachineVision
Real-TimeSearch
![Page 3: Activity Recognition in Videofiles.meetup.com/4379272/Bostion Image Processing Meetup.pdf · Document Query 2/7/2013 8. Video Indexing Process 2/7/2013 9 Blob Extraction Source Video(s)](https://reader033.fdocuments.us/reader033/viewer/2022050107/5f451be722d10d14970f3ae2/html5/thumbnails/3.jpg)
Cognika Introduction
2/7/2013 3
MachineVision
Real-TimeSearch
Forensic“Search”
Real-TimeAlerting
![Page 4: Activity Recognition in Videofiles.meetup.com/4379272/Bostion Image Processing Meetup.pdf · Document Query 2/7/2013 8. Video Indexing Process 2/7/2013 9 Blob Extraction Source Video(s)](https://reader033.fdocuments.us/reader033/viewer/2022050107/5f451be722d10d14970f3ae2/html5/thumbnails/4.jpg)
What we do
• “Search” within FMV• By Image (OOI)
• By Video Clip
• By Text
• Real-Time
• Activity-based Searching – spatiotemporal querying
2/7/2013 4
![Page 5: Activity Recognition in Videofiles.meetup.com/4379272/Bostion Image Processing Meetup.pdf · Document Query 2/7/2013 8. Video Indexing Process 2/7/2013 9 Blob Extraction Source Video(s)](https://reader033.fdocuments.us/reader033/viewer/2022050107/5f451be722d10d14970f3ae2/html5/thumbnails/5.jpg)
Inverted Indexing
2/7/2013 5Source: developer.apple.com
![Page 6: Activity Recognition in Videofiles.meetup.com/4379272/Bostion Image Processing Meetup.pdf · Document Query 2/7/2013 8. Video Indexing Process 2/7/2013 9 Blob Extraction Source Video(s)](https://reader033.fdocuments.us/reader033/viewer/2022050107/5f451be722d10d14970f3ae2/html5/thumbnails/6.jpg)
Text Indexing Process
2/7/2013 6
Source Document
Analyze
Parser
Tokenizer
Stemmer
Tokens
Payloads
Write to IndexInvertedIndex
Analyzer
Indexed Documents
![Page 7: Activity Recognition in Videofiles.meetup.com/4379272/Bostion Image Processing Meetup.pdf · Document Query 2/7/2013 8. Video Indexing Process 2/7/2013 9 Blob Extraction Source Video(s)](https://reader033.fdocuments.us/reader033/viewer/2022050107/5f451be722d10d14970f3ae2/html5/thumbnails/7.jpg)
Vector Space Model
• Documents and Queries are “Vectors”
– Di = (wi,1, wi,2, wi,3….wi,n)
– Where wi,j is weight for “term” j in document
• Cosine Similarity = Cosine of angle between query and stored document
2/7/2013 7
![Page 8: Activity Recognition in Videofiles.meetup.com/4379272/Bostion Image Processing Meetup.pdf · Document Query 2/7/2013 8. Video Indexing Process 2/7/2013 9 Blob Extraction Source Video(s)](https://reader033.fdocuments.us/reader033/viewer/2022050107/5f451be722d10d14970f3ae2/html5/thumbnails/8.jpg)
TF-IDF Vector Space Querying
𝑑𝑗 = 𝑤1,𝑗 , 𝑤2,𝑗 … .𝑤𝑛,𝑗
𝑞 = 𝑤1,𝑞 , 𝑤2,𝑞… .𝑤𝑛,𝑞
Document
Query
2/7/2013 8
![Page 9: Activity Recognition in Videofiles.meetup.com/4379272/Bostion Image Processing Meetup.pdf · Document Query 2/7/2013 8. Video Indexing Process 2/7/2013 9 Blob Extraction Source Video(s)](https://reader033.fdocuments.us/reader033/viewer/2022050107/5f451be722d10d14970f3ae2/html5/thumbnails/9.jpg)
Video Indexing Process
2/7/2013 9
Blob Extraction
Source Video(s) Frames
Index“Documents”
Training Set
Object Classification
Metadata
Frame Extraction
BlobDescriptors
Document Construction
Inverted Index
![Page 10: Activity Recognition in Videofiles.meetup.com/4379272/Bostion Image Processing Meetup.pdf · Document Query 2/7/2013 8. Video Indexing Process 2/7/2013 9 Blob Extraction Source Video(s)](https://reader033.fdocuments.us/reader033/viewer/2022050107/5f451be722d10d14970f3ae2/html5/thumbnails/10.jpg)
Simplified Example
2/7/2013 10
Circle
Triangle
Circle <x1,y1>
Triangle <x2,y2>
Training Image Set
Frame ImageIndex Document Representation
<x4,y4>
Color, Shape, Texture, Contour
Descriptors
![Page 11: Activity Recognition in Videofiles.meetup.com/4379272/Bostion Image Processing Meetup.pdf · Document Query 2/7/2013 8. Video Indexing Process 2/7/2013 9 Blob Extraction Source Video(s)](https://reader033.fdocuments.us/reader033/viewer/2022050107/5f451be722d10d14970f3ae2/html5/thumbnails/11.jpg)
Flow Chart
StabilizationMotion
Compensation
VideoStream
IsCamera Moving?
Blob Tracking
Yes
No
Disk-basedIndex
Extract Blob
Feature Vector
Build Frameset(Sliding
Window)
In-Memory Index
2/7/2013 11
Alerting
Search
![Page 12: Activity Recognition in Videofiles.meetup.com/4379272/Bostion Image Processing Meetup.pdf · Document Query 2/7/2013 8. Video Indexing Process 2/7/2013 9 Blob Extraction Source Video(s)](https://reader033.fdocuments.us/reader033/viewer/2022050107/5f451be722d10d14970f3ae2/html5/thumbnails/12.jpg)
Sliding Window Approach
Frame-1 Frame-2 Frame-3 Frame-k... Frame-p...
Window 1
Window 2
Frame-q...
Window w...
2/7/2013 12
![Page 13: Activity Recognition in Videofiles.meetup.com/4379272/Bostion Image Processing Meetup.pdf · Document Query 2/7/2013 8. Video Indexing Process 2/7/2013 9 Blob Extraction Source Video(s)](https://reader033.fdocuments.us/reader033/viewer/2022050107/5f451be722d10d14970f3ae2/html5/thumbnails/13.jpg)
Sequences Hierarchy
Objects (e.g. Humans, Vehicles)
Events (e.g. Humans Moving, Vehicles Moving)
Activities (e.g. Persons Moving Away, Vehicles Driving away)
Scenarios (e.g. Humans Gathering around Parked Vehicles)
2/7/2013 13
![Page 14: Activity Recognition in Videofiles.meetup.com/4379272/Bostion Image Processing Meetup.pdf · Document Query 2/7/2013 8. Video Indexing Process 2/7/2013 9 Blob Extraction Source Video(s)](https://reader033.fdocuments.us/reader033/viewer/2022050107/5f451be722d10d14970f3ae2/html5/thumbnails/14.jpg)
VideoIndex
Blob Extraction
Object-Frame Matrixes
Inferred Latent SemanticGraph
Normalization to adjust for quality
Object Classification
Metadata(e.g. Date-
Time, Resolution
etc.)
Frameset
2/7/2013 14
![Page 15: Activity Recognition in Videofiles.meetup.com/4379272/Bostion Image Processing Meetup.pdf · Document Query 2/7/2013 8. Video Indexing Process 2/7/2013 9 Blob Extraction Source Video(s)](https://reader033.fdocuments.us/reader033/viewer/2022050107/5f451be722d10d14970f3ae2/html5/thumbnails/15.jpg)
What we Index
• Color histograms• Shape Descriptors• Contour Descriptors• Video Metadata (e.g. date-time, resolution etc.)• Contextual information (e.g. Geo-location etc.)
2/7/2013 15
![Page 16: Activity Recognition in Videofiles.meetup.com/4379272/Bostion Image Processing Meetup.pdf · Document Query 2/7/2013 8. Video Indexing Process 2/7/2013 9 Blob Extraction Source Video(s)](https://reader033.fdocuments.us/reader033/viewer/2022050107/5f451be722d10d14970f3ae2/html5/thumbnails/16.jpg)
Query Clip
Result Clips
2/7/2013 16
![Page 17: Activity Recognition in Videofiles.meetup.com/4379272/Bostion Image Processing Meetup.pdf · Document Query 2/7/2013 8. Video Indexing Process 2/7/2013 9 Blob Extraction Source Video(s)](https://reader033.fdocuments.us/reader033/viewer/2022050107/5f451be722d10d14970f3ae2/html5/thumbnails/17.jpg)
Query Response TimesActivity Query Mean Response Time(milliseconds)
(averaged over 5 consecutive queries)No. of Results
Parked Vehicle 762 482
Person Walking 482 891
Ingressing Vehicle 319 876
Egressing Vehicle 410 573
Moving Vehicles 890 1098
Vehicle Halting & Person Exiting 1028 73
Person Entering Vehicle & Vehicle Moving
1176 48
Persons Gathering 908 382
Sub-second Responses for Terascale & Larger possible
2/7/2013 17
![Page 18: Activity Recognition in Videofiles.meetup.com/4379272/Bostion Image Processing Meetup.pdf · Document Query 2/7/2013 8. Video Indexing Process 2/7/2013 9 Blob Extraction Source Video(s)](https://reader033.fdocuments.us/reader033/viewer/2022050107/5f451be722d10d14970f3ae2/html5/thumbnails/18.jpg)
Prototype UI
2/7/2013 18
![Page 19: Activity Recognition in Videofiles.meetup.com/4379272/Bostion Image Processing Meetup.pdf · Document Query 2/7/2013 8. Video Indexing Process 2/7/2013 9 Blob Extraction Source Video(s)](https://reader033.fdocuments.us/reader033/viewer/2022050107/5f451be722d10d14970f3ae2/html5/thumbnails/19.jpg)
Further Research
• Improved Feature Vectors (for sparse features)
• Improved Blob Classifiers
• Improved Stabilization, BG Subtraction & Motion Compensation
• “Super Resolution” Enhancements
2/7/2013 19
![Page 20: Activity Recognition in Videofiles.meetup.com/4379272/Bostion Image Processing Meetup.pdf · Document Query 2/7/2013 8. Video Indexing Process 2/7/2013 9 Blob Extraction Source Video(s)](https://reader033.fdocuments.us/reader033/viewer/2022050107/5f451be722d10d14970f3ae2/html5/thumbnails/20.jpg)
We Are Hiring!
2/7/2013 20
Machine Vision Engineers
• OpenCV, and other machine vision toolkits
• OpenGL, CUDA
• Bayesian, ANN, SVMs etc.
• Video Background desirable
Search Engineers
• Lucene, Solr, Elastic-Search
• Hadoop, Katta, ZooKeeper
• Terascale+ Real-Time Search Experience desirable