CVPR2012: discriminative virtual views for cross-view action recognition

7/31/2019 CVPR2012: discriminative virtual views for cross-view action recognition

http://slidepdf.com/reader/full/cvpr2012-discriminative-virtual-views-for-cross-view-action-recognition 1/56

Discrimina)ve‘VirtualViews’forCross-

ViewAc)onRecogni)on

RuonanLiandToddZickler

HarvardSchoolofEngineeringandAppliedSciences



Cross-ViewRecogni)on

“kick”

View1

2




“kick”

View2View1

3

?




“kick”

View2View1

4

?

x x�

�=




5

x x�

?




6

x x�W

Ourapproach:1. Dimensionexpansionfordiscrimina)vecross-viewfeatures;



OurApproach:

Cross-ViewDimensionExpansion

7

x x�

=x̂ x̂� W=

Ourapproach:1. Dimensionexpansionfordiscrimina)vecross-viewfeatures;



OurApproach:

Cross-ViewDimensionExpansion

8

x x�

=x̂ x̂� W=

Ourapproach:1. Dimensionexpansionfordiscrimina)vecross-viewfeatures;2. Exploitvariouscross-viewannota)ontypesthatwere

consideredseparatelybefore.



Case1:WeaklyLabeledTargetView

View2(TargetViewView1(SourceView

?



Case1:WeaklyLabeledTargetView

Domain2(TargetDomainDomain1(SourceDomain

• DomainAdapta)on/Cross-DomainRecogni)on

• Ben-David2007;Blitzer2006,2007;DaumeIII2007;Duan2009,2010,2011;

Jhuo2012;Kulis2011;Pan2010,2011;Saenko2010;andmanyothers.

?



Case2:UnlabeledCorrespondence

View1(SourceView View2(TargetView

• Farhadi2008,2009;Liu2011

Thesameunknownac)on

executedbythesamesubject

andsimultaneouslyobservedinbothviews



Case3:NoSupervisionintheTargetView

View1(SourceView View2(TargetView

• Gong2012;Gopalan2011;

?



Intra-ClassSimilarity

“kick” “kick”


x x�

=x̂ x̂

�



Inter-ClassSeparability

“kick” “wavehand”


x x�

=x̂ x̂

�



Intra-ClassSimilarityandInter-ClassSeparability

maxW

I (Wx; c)

W W



UnlabeledCorrespondence


Thesameunknownac)on

executedbythesamesubject

andsimultaneouslyobservedinbothviews

x x�

=x̂ x̂

�



UnlabeledCorrespondence

maxW

I (Wx; c)− γ H (Wx−Wx�)

W W



Cross-ViewTransformObjec)ve

W W

maxW





maxW

lndetΣall−1

2lndetΣClass1−

1

2lndetΣClass2−γ ln detΣ∆

Assump)ons:

• Uniformdistribu)onforclassprior;

• Gaussiandistribu)onforcross-viewfeatures.




maxW

lndetΣall−1

2lndetΣClass1−

1

2lndetΣClass2−γ lndetΣ∆




maxW

lndetΣall−1

2lndetΣClass1−

1





maxW

lndetΣall−1

2lndetΣClass1−

1


∆ =Wx−Wx�



Cross-ViewTransform

SourceView TargetView

AS



Cross-ViewTransform


AS

AT



Cross-ViewTransform


x

=

AS

AT



Cross-ViewTransform


x

=

AS

AS



Cross-ViewTransform


x

=

AS

AT

AS

AT



Cross-ViewTransform


x

=

AS

AT

A1

AS

AT



Cross-ViewTransform


x

=

AS

AT

A1

A2

AS

AT



Cross-ViewTransform


x

=

AS

AT

A1

A2

A3

AS

AT



Discrimina)veVirtualViews

x

=

AS

AT

A1

A2

A3




x

=

AS

AT

A1

A2

A3

Allrealandvirtualviews

Sourceview

Targetview




x

=

AS

AT

A1

A2

A3


Sourceview

Targetview

`Virtualviews’




x

=

AS

AT

A1

A2

A3


Sourceview

Targetview

`Virtualviews’

Cross-ViewDiscrimina)ve

‘DimensionExpansion’W



TheOverallObjec)ve

x

=

AS

AT

A1

A2A3

W

maxAT ,AS


• Close-forminterpola)onofvirtualviewtransformsfromand;

AS

AT

subject to AS AT S = I and AT A

T T = I



Solu)on

x

=

AS

AT

A1

A2A3

W

maxAT ,AS



AS

AT


T T = I



Solu)on

x

=

AS

AT

A1

A2

A3

W

maxAT ,AS



• andini)alizedwithLDA(orPCAwheneverLDAisunavailable;

AS AT

AS

AT


T T = I



Solu)on

x

=

AS

AT

A1

A2

A3

W

maxAT ,AS



• andini)alizedwithLDA(orPCAwheneverLDAisunavailable;

• Approximategradientascenton.

SO(D)

AS AT

AS

AT

A(i)�

A(i+1)�

=

R(i)opt


T T = I



Recap

W W

x

=

AS

AT

A1

A2

A3

W

• Discrimina)vecross-view`dimensionexpansion’;• Uniformlyaccountforvariouscross-viewsemi-supervisions.



IXMASDataset• 11ac)oncategoriesperformedby10actors,takenfrom5views.

View0

View1

View2

View3

View4

Check-watch Scratch-head Sit-down Wave-hand Kicking Pick-up



Case1Result:WeaklyLabeledTargetView

• ViewMixing:ASVMtrainedontheunionofsourceandtargetviews




• ViewConcatena)on:DaumeIII2007

x̂ = [xT ,0

T ,x

T ]T , x̂

� = [0T ,x

�T ,x

�T ]T




• CombiningClassifier:Schweikert2008

f (x) = β f S (x) + (1− β )f T (x)



Case2Result:UnlabeledCorrespondence

• Farhadi2008,2009




• Liu2011

V i e w

1

V i e w

2

+

e x a m p l e s

Training data vectorsM

Source view Target view

Unlabelled AcƟon Videos A1

Unlabelled AcƟon Videos A2

Vocabulary V1

Vocabulary V2

BoVWmodels M1

BoVWmodels M2

Bilingual Words

BoBWmodels

Visual Words

BiparƟteGraph

BiparƟte

Graph

ParƟƟoning



Case3Result:NoSupervisionintheTargetView




• Gopalan2011




• Non-discrimina)vevirtualviews



FourSourceViews+SingleTargetView

Case1

Case2



Summary

• A‘dimensionexpansion’transformtoproducecross-viewfeatureswhich(1arediscrimina)ve;(2encodethe

transi)onfromthesourcetothetarget;

--SeealsoGonget.al.,CVPR2012

• Accommodatevarioustypesofsemi-supervisions,including(1weaklylabeledtargetview;(2Unlabeled

correspondencebetweentwoviews;(3completely

unsupervisedtargetview.

• Applicabletogeneraldomaintransfer/adapta)onproblems.



Discrimina)ve‘VirtualViews’forCross-View

Ac)onRecogni)on

W W

x

=

AS

AT

A1

A2

A3

W

i b f i l i d



VaryingNumberofVirtualViewsand

VaryingDimensionsofVirtualViews

CVPR2012: discriminative virtual views for cross-view action recognition

Documents

Transcript of CVPR2012: discriminative virtual views for cross-view action recognition