Generic Object and Action Detection with LARK …Haejong Seo (summer Internship) (3) Develop fast...
Transcript of Generic Object and Action Detection with LARK …Haejong Seo (summer Internship) (3) Develop fast...
Generic Object and Action Detection withLARK (Locally Adaptive Regression Kernels)
Haejong SeoUniversity of California, Santa Cruz
Mentor: Gary Bradski
Haejong Seo (summer Internship) (1)
MotivationHumanRobot InteractionRobotRobot Interaction
Where to look?
Is there any motion?
Where is a bottle of beer located?How big is this bottle?What pose is this ?
Is he/she waving hands to me?Is other PR2 approaching to me?
122
3
4
MotivationDOT (dominant orientation templates) by Stefan
1) can handle object detection2) run with a single webcam3) pretty fast
However, can not deal with
Haejong Seo (summer Internship) (2)
BiGGPy (binarized gradients grid pyramid) by Gary
3
1 2 4
LARK can tackle all the problems like
Motivation & Goal
static saliency object detection
spacetime saliency action detection
Haejong Seo (summer Internship) (3)
Develop fast and robust detection systems in open sourse
1
2 4
3
OutlineOutline
LARK Overview
Saliency Detection
Object Detection
Conclusion
Spacetime Saliency Detection
Action Detection
1
2
3
4
5
6
LARK (locally adaptive regression kernels)
Haejong Seo (summer Internship) (4)
●Euclidean distance vs. Geodesic distance
Image as a Surface Embedded in the Euclidean 3‐space
Arclength on the surface
Chain rule
Haejong Seo (summer Internship) (5)
LARK selfsimilarity →
Haejong Seo (summer Internship) (6)
LARK (example)
Haejong Seo (summer Internship) (7)
LARK (speedup)Step1: downsample by a factor of 4
Step 2: interpolate C = [C11, C12, C22] after computing in a lower scale
C11 C22 C12
Haejong Seo (summer Internship) (8)
0.02 sec (70 times faster)
Saliency Detection
Haejong Seo (summer Internship) (9)
LARK selfresemblance
Saliency map
thresholding
Saliency Detection (video)
Haejong Seo (summer Internship) (10)
Object Detection
Haejong Seo (summer Internship) (11)
Compute LARKCompute LARK
templatetemplate
imageimage
Object Detection (speedup)
Use saliency to reduce search space
Haejong Seo (summer Internship) (12)
Face Detection (video)
Haejong Seo (summer Internship) (13)
One template Three templates
Object Detection (video)
Haejong Seo (summer Internship) (14)
Door knob PR2
Drawing Small robot
Three templates
3D Object Detection (speedup)
Pyramid searchPyramid searchTree structure for template
Haejong Seo (summer Internship) (15)
3D Object Detection (video)
Haejong Seo (summer Internship) (18)
CD case mouse
naked organizer
3D Object Detection (video)
Haejong Seo (summer Internship) (19)
Two objects Three objects
3D LARK selfsimilarity in 3D→
Haejong Seo (summer Internship) (19)
Spacetime Saliency Detection
Haejong Seo (summer Internship) (11)
Spacetime Saliency (video)
Haejong Seo (summer Internship) (12)
Action Detection
Haejong Seo (summer Internship) (20)
template
Input video
LARKs(30~35 frames)
Action Detection (speedup)
Haejong Seo (summer Internship) (21)
spacetime saliencypyramid search
5 frames 5 frames
35 frames
7 frames of 3D LARK (3x3 (space)x5 (time))
Action Detection (video)
Haejong Seo (summer Internship) (22)
4 actions
sitting down
moving closer
boxing
waving
Code Availability
Package larks: service that trains object templates and detects objects locations and poses (available now)
→ stacks/object_recognition_experimental/larks
Haejong Seo (summer Internship) (23)
Package saliency: service that provides salient regions in images and videos (will be available)
→ cturtle/wgrospkgunreleased/sandbox/saliency
Package actiondetection: service that detect generic human actions in videos (will be available)
→ cturtle/wgrospkgunreleased/sandbox/actiondetection
Discussion & Future Work
Use partsbased detection to deal with occlusion (Steve Gould)
Use a tracking algorithm to avoid blinking effects
Improve scalability build a common tree for all the objects→
Haejong Seo (summer Internship) (24)
Learn threshold values for each object and action
Use LARK as a post filter for BiGGPy
Thank you!
Any Questions?