CVPR2010: Advanced ITinCVPR in a Nutshell: part 3: Feature Selection
CVPR2010: modeling mutual context of object and human pose in human-object interaction activities
-
Upload
zukun -
Category
Technology
-
view
644 -
download
0
Transcript of CVPR2010: modeling mutual context of object and human pose in human-object interaction activities
Modeling Mutual Context of Object
and Human Pose in Human-Object
Interaction Activities
Bangpeng Yao and Li Fei-Fei
Computer Science Department, Stanford University
{bangpeng,feifeili}@cs.stanford.edu
1
Robots interact
with objects
Automatic sports
commentary
“Kobe is dunking the ball.”
2
Human-Object Interaction
Medical care
3
Vs.
Human-Object Interaction
Playing
saxophone
Playing
bassoon
Playing
saxophone
Grouplet is a generic feature for structured objects, or interactions
of groups of objects.
(Previous talk: Grouplet)
Caltech101
HOI activity: Tennis Forehand
Holistic image based classification
Detailed understanding and reasoning
Berg & Malik, 2005 Grauman & Darrell, 2005 Gehler & Nowozin, 2009 OURS
48% 59% 77% 62%
4
Human-Object Interaction
Torso
Head
• Human pose estimation
Holistic image based classification
Detailed understanding and reasoning
5
Human-Object Interaction
Tennis
racket
• Human pose estimation
Holistic image based classification
Detailed understanding and reasoning
• Object detection
6
Human-Object Interaction
• Human pose estimation
Holistic image based classification
Detailed understanding and reasoning
• Object detection
Torso
Head
Tennis
racket
HOI activity: Tennis Forehand
• Background and Intuition
• Mutual Context of Object and Human Pose
Model Representation
Model Learning
Model Inference
• Experiments
• Conclusion
Outline
7
• Background and Intuition
• Mutual Context of Object and Human Pose
Model Representation
Model Learning
Model Inference
• Experiments
• Conclusion
Outline
8
• Felzenszwalb & Huttenlocher, 2005
• Ren et al, 2005
• Ramanan, 2006
• Ferrari et al, 2008
• Yang & Mori, 2008
• Andriluka et al, 2009
• Eichner & Ferrari, 2009
Difficult part
appearance
Self-occlusion
Image region looks
like a body part
Human pose estimation & Object detection
9
Human pose
estimation is
challenging.
Human pose estimation & Object detection
10
Human pose
estimation is
challenging.
• Felzenszwalb & Huttenlocher, 2005
• Ren et al, 2005
• Ramanan, 2006
• Ferrari et al, 2008
• Yang & Mori, 2008
• Andriluka et al, 2009
• Eichner & Ferrari, 2009
Human pose estimation & Object detection
11
Facilitate
Given the
object is
detected.
• Viola & Jones, 2001
• Lampert et al, 2008
• Divvala et al, 2009
• Vedaldi et al, 2009
Small, low-
resolution, partially
occluded
Image region similar
to detection target
Human pose estimation & Object detection
12
Object
detection is
challenging
Human pose estimation & Object detection
13
Object
detection is
challenging
• Viola & Jones, 2001
• Lampert et al, 2008
• Divvala et al, 2009
• Vedaldi et al, 2009
Human pose estimation & Object detection
14
Facilitate
Given the
pose is
estimated.
Human pose estimation & Object detection
15
Mutual Context
• Hoiem et al, 2006
• Rabinovich et al, 2007
• Oliva & Torralba, 2007
• Heitz & Koller, 2008
• Desai et al, 2009
• Divvala et al, 2009
• Murphy et al, 2003
• Shotton et al, 2006
• Harzallah et al, 2009
• Li, Socher & Fei-Fei, 2009
• Marszalek et al, 2009
• Bao & Savarese, 2010
Context in Computer Vision
~3-4%
with
context
without
context
Helpful, but only moderately
outperform better
Previous work – Use context
cues to facilitate object detection:
• Viola & Jones, 2001
• Lampert et al, 2008
16
Context in Computer Vision
Our approach – Two challenging
tasks serve as mutual context of
each other:
With
mutual
context:
Without
context:
17
~3-4%
with
context
without
context
Helpful, but only moderately
outperform better
Previous work – Use context
cues to facilitate object detection:
• Hoiem et al, 2006
• Rabinovich et al, 2007
• Oliva & Torralba, 2007
• Heitz & Koller, 2008
• Desai et al, 2009
• Divvala et al, 2009
• Murphy et al, 2003
• Shotton et al, 2006
• Harzallah et al, 2009
• Li, Socher & Fei-Fei, 2009
• Marszalek et al, 2009
• Bao & Savarese, 2010
• Background and Intuition
• Mutual Context of Object and Human Pose
Model Representation
Model Learning
Model Inference
• Experiments
• Conclusion
Outline
18
19
H
A
Mutual Context Model Representation
• More than one H for each A;
• Unobserved during training.
A:
Croquet
shot
Volleyball
smash
Tennis
forehand
Intra-class variations
Activity
Object
Human pose
Body parts
lP: location; θP: orientation; sP: scale.
Croquet
malletVolleyball
Tennis
racket
O:
H:
P:
f: Shape context. [Belongie et al, 2002]
P1
Image evidence
fO
f1 f2 fN
O
P2 PN
20
Mutual Context Model Representation
( , )e O H
( , )e A O
( , )e A H
e e
e E
w
Markov Random Field
Clique
potential
Clique
weight
O
P1 PN
fO
H
A
P2
f1 f2 fN
( , )e A O ( , )e A H ( , )e O H• , , : Frequency
of co-occurrence between A, O, and H.
21
A
f1 f2 fN
Mutual Context Model Representation
( , )e nO P
( , )e m nP P
fO
P1 PNP2
O
H• , , : Spatial
relationship among object and body parts.
( , )e nO P ( , )e m nP P( , )e nH P
bin binn n nO P O P O Pl l s s
location orientation size
( , )e nH P
e e
e E
w
Markov Random Field
Clique
potential
Clique
weight
( , )e A O ( , )e A H ( , )e O H• , , : Frequency
of co-occurrence between A, O, and H.
22
H
A
f1 f2 fN
Mutual Context Model Representation
Obtained by
structure learning
fO
PNP1 P2
O
• Learn structural connectivity among
the body parts and the object.
( , )e A O ( , )e A H ( , )e O H• , , : Frequency
of co-occurrence between A, O, and H.
• , , : Spatial
relationship among object and body parts.
( , )e nO P ( , )e m nP P( , )e nH P
bin binn n nO P O P O Pl l s s
location orientation size ( , )e nO P
( , )e m nP P
( , )e nH P
e e
e E
w
Markov Random Field
Clique
potential
Clique
weight
23
H
O
A
fO
f1 f2 fN
P1 P2 PN
Mutual Context Model Representation
• and : Discriminative
part detection scores.
( , )e OO f ( , )ne n PP f
[Andriluka et al, 2009]
Shape context + AdaBoost
• Learn structural connectivity among
the body parts and the object.
[Belongie et al, 2002]
[Viola & Jones, 2001]
( , )e OO f
( , )ne n PP f
( , )e A O ( , )e A H ( , )e O H• , , : Frequency
of co-occurrence between A, O, and H.
• , , : Spatial
relationship among object and body parts.
( , )e nO P ( , )e m nP P( , )e nH P
bin binn n nO P O P O Pl l s s
location orientation size
e e
e E
w
Markov Random Field
Clique
potential
Clique
weight
• Background and Intuition
• Mutual Context of Object and Human Pose
Model Representation
Model Learning
Model Inference
• Experiments
• Conclusion
Outline
24
25
Model Learning
H
O
A
fO
f1 f2 fN
P1 P2 PN
e e
e E
w
cricket
shot
cricket
bowling
Input:
Goals:
Hidden human poses
26
Model Learning
H
O
A
fO
f1 f2 fN
P1 P2 PN
Input:
Goals:
Hidden human poses
Structural connectivity
e e
e E
w
cricket
shot
cricket
bowling
e e
e E
w
27
Model Learning
Goals:
Hidden human poses
Structural connectivity
Potential parameters
Potential weights
H
O
A
fO
f1 f2 fN
P1 P2 PN
Input:
cricket
shot
cricket
bowling
28
Model Learning
Goals:
Parameter estimation
Hidden variables
Structure learning
H
O
A
fO
f1 f2 fN
P1 P2 PN
Input:e e
e E
w
cricket
shot
cricket
bowling
Hidden human poses
Structural connectivity
Potential parameters
Potential weights
29
Model Learning
Goals:
H
O
A
fO
f1 f2 fN
P1 P2 PN
Approach:
croquet shot
e e
e E
w
Hidden human poses
Structural connectivity
Potential parameters
Potential weights
30
Model Learning
Goals:
H
O
A
fO
f1 f2 fN
P1 P2 PN
Approach:
2
2max
2e eeE e
Ew
Joint density
of the model
Gaussian priori of
the edge number
Hill-climbing
e e
e E
w
Hidden human poses
Structural connectivity
Potential parameters
Potential weights
31
Model Learning
Goals:
H
O
A
fO
f1 f2 fN
P1 P2 PN
Approach:
( , )e O H( , )e A O ( , )e A H
( , )e nO P ( , )e m nP P( , )e nH P
( , )e OO f ( , )ne n PP f
• Maximum likelihood
• Standard AdaBoost
e e
e E
w
Hidden human poses
Structural connectivity
Potential parameters
Potential weights
32
Model Learning
Goals:
H
O
A
fO
f1 f2 fN
P1 P2 PN
Approach:
Max-margin learning
2
2,
1min
2r i
r i
w
w
• xi: Potential values of the i-th image.
• wr: Potential weights of the r-th pose.
• y(r): Activity of the r-th pose.
• ξi: A slack variable for the i-th image.
Notations
s.t. , where ,
1
, 0
i
i
c i r i i
i
i r y r y c
i
w x w x
e e
e E
w
Hidden human poses
Structural connectivity
Potential parameters
Potential weights
33
Learning Results
Cricket
defensive
shot
Cricket
bowling
Croquet
shot
34
Learning Results
Tennis
serve
Volleyball
smash
Tennis
forehand
• Background and Intuition
• Mutual Context of Object and Human Pose
Model Representation
Model Learning
Model Inference
• Experiments
• Conclusion
Outline
35
I
36
Model Inference
The learned models
I
37
Model Inference
The learned models
Head detection
Torso detection
Tennis racket detection
Layout of the object and body parts.
Compositional
Inference
[Chen et al, 2007]
* *
1 1 1 1,, , , nn
A H O P
I
38
Model Inference
The learned models
* *
1 1 1 1,, , , nn
A H O P * *
,, , ,K K K K nn
A H O P
Output
• Background and Intuition
• Mutual Context of Object and Human Pose
Model Representation
Model Learning
Model Inference
• Experiments
• Conclusion
Outline
39
40
Dataset and Experiment Setup
• Object detection;
• Pose estimation;
• Activity classification.
Tasks:
[Gupta et al, 2009]
Cricket
defensive shot
Cricket
bowling
Croquet
shot
Tennis
forehand
Tennis
serve
Volleyball
smash
Sport data set: 6 classes
180 training (supervised with object and part locations) & 120 testing images
[Gupta et al, 2009]
Cricket
defensive shot
Cricket
bowling
Croquet
shot
Tennis
forehand
Tennis
serve
Volleyball
smash
Sport data set: 6 classes
41
Dataset and Experiment Setup
• Object detection;
• Pose estimation;
• Activity classification.
Tasks:
180 training (supervised with object and part locations) & 120 testing images
0 0.2 0.4 0.6 0.8 10
0.2
0.4
0.6
0.8
1
Recall
Pre
cis
ion
Object Detection Results
Cricket bat
42
Valid
region
Croquet mallet Tennis racket Volleyball
0 0.2 0.4 0.6 0.8 10
0.2
0.4
0.6
0.8
1
Recall
Pre
cis
ion
Cricket ball
Our
Method
Sliding
window
Pedestrian
context
[Andriluka
et al, 2009]
[Dalal &
Triggs, 2006]
Object Detection Results
43
430 0.2 0.4 0.6 0.8 1
0
0.2
0.4
0.6
0.8
1
Recall
Pre
cis
ion
Volleyball
0 0.2 0.4 0.6 0.8 10
0.2
0.4
0.6
0.8
1
Recall
Pre
cis
ion
Cricket ball
0 0.2 0.4 0.6 0.8 10
0.2
0.4
0.6
0.8
1
RecallP
recis
ion
Our Method
Pedestrian as context
Scanning window detector
0 0.2 0.4 0.6 0.8 10
0.2
0.4
0.6
0.8
1
Recall
Pre
cis
ion
Our Method
Pedestrian as context
Scanning window detector
0 0.2 0.4 0.6 0.8 10
0.2
0.4
0.6
0.8
1
Recall
Pre
cis
ion
Our Method
Pedestrian as context
Scanning window detectorSliding window Pedestrian context Our method
Sm
all
ob
jec
tB
ac
kg
rou
nd
clu
tte
r
44
Dataset and Experiment Setup
• Object detection;
• Pose estimation;
• Activity classification.
Tasks:
[Gupta et al, 2009]
Cricket
defensive shot
Cricket
bowling
Croquet
shot
Tennis
forehand
Tennis
serve
Volleyball
smash
Sport data set: 6 classes
180 training & 120 testing images
45
Human Pose Estimation Results
Method Torso Upper Leg Lower Leg Upper Arm Lower Arm Head
Ramanan,
2006.52 .22 .22 .21 .28 .24 .28 .17 .14 .42
Andriluka et
al, 2009.50 .31 .30 .31 .27 .18 .19 .11 .11 .45
Our full
model.66 .43 .39 .44 .34 .44 .40 .27 .29 .58
46
Human Pose Estimation Results
Method Torso Upper Leg Lower Leg Upper Arm Lower Arm Head
Ramanan,
2006.52 .22 .22 .21 .28 .24 .28 .17 .14 .42
Andriluka et
al, 2009.50 .31 .30 .31 .27 .18 .19 .11 .11 .45
Our full
model.66 .43 .39 .44 .34 .44 .40 .27 .29 .58
Andriluka
et al, 2009
Our estimation
result
Tennis serve
modelAndriluka
et al, 2009
Our estimation
result
Volleyball
smash model
47
Human Pose Estimation Results
Method Torso Upper Leg Lower Leg Upper Arm Lower Arm Head
Ramanan,
2006.52 .22 .22 .21 .28 .24 .28 .17 .14 .42
Andriluka et
al, 2009.50 .31 .30 .31 .27 .18 .19 .11 .11 .45
Our full
model.66 .43 .39 .44 .34 .44 .40 .27 .29 .58
One pose
per class.63 .40 .36 .41 .31 .38 .35 .21 .23 .52
Estimation
result
Estimation
result
Estimation
result
Estimation
result
48
Dataset and Experiment Setup
• Object detection;
• Pose estimation;
• Activity classification.
Tasks:
[Gupta et al, 2009]
Cricket
defensive shot
Cricket
bowling
Croquet
shot
Tennis
forehand
Tennis
serve
Volleyball
smash
Sport data set: 6 classes
180 training & 120 testing images
Activity Classification Results
49
Gupta et
al, 2009
Our
model
Bag-of-
Words
83.3%
Cla
ssific
atio
n a
ccu
racy
78.9%
52.5%
0.9
0.8
0.7
0.6
0.5
No scene
information Scene is
critical!! Cricket
shot
Tennis
forehand
Bag-of-words
SIFT+SVM
Gupta et
al, 2009
Our
model
50
Conclusion
Human-Object Interaction
Next Steps
Vs.
• Pose estimation & Object detection on PPMI images.
• Modeling multiple objects and humans.
Grouplet representation
Mutual context model
Acknowledgment• Stanford Vision Lab reviewers:
– Barry Chai (1985-2010)
– Juan Carlos Niebles
– Hao Su
• Silvio Savarese, U. Michigan
• Anonymous reviewers
51