What Helps Where – And Why? Semantic Relatedness for Knowledge Transfer Marcus Rohrbach 1,2...

What Helps Where – And Why?Semantic Relatedness for Knowledge Transfer

Marcus Rohrbach1,2 Michael Stark1,2 György Szarvas1 Iryna Gurevych1 Bernt Schiele1,2

1Department of Computer Science, TU Darmstadt 2MPI Informatics, Saarbrücken

2

Knowledge transfer for zero-shot object class recognition

CVPR 2010 | What Helps Where - And Why? Semantic Relatedness for Knowledge Transfer | Marcus Rohrbach |

Group classes by attributes[Lampert et al., CVPR `09]

Manual supervision:Object class-attribute associations



Describing using attributes[Farhadi et al., CVPR `09 & `10]

Manual supervision:Attribute labels



• animal• four legged• mammal

white paw

Unseen class(no training images) Giant panda ?

Attributes for knowledge transfer

3

Knowledge transfer for zero-shot object class recognition






• animal• four legged• mammal





white paw

Unseen class(no training images) Giant panda ? Attributes for knowledge transfer Replace manual supervision

by semantic relatednessmined from language resources

Unsupervised Transfer

WordNetAttributes for knowledge transfer

4

Attribute-based model [Lampert et al., CVPR `09]


oceanoceanspotsspots ……

Known training classes

Attribute classifiers

Unseentest classes

Class-attribute associations


[Lampert et al., CVPR `09]

Supervised:manual (human judges)

Attributeswhitewhite

5

Attribute-based model [Lampert et al., CVPR `09]


WordNet

semantic relatednessfrom language

[Lampert et al., CVPR `09]

Supervised:manual (human judges)




Unseentest classes



whitewhite

6

Direct similarity-based model


WordNet





Unseentest classes



whitewhite

7



WordNet



Unseentest classes


Classifierper class

killer whalekiller

whaleDalmatian polar

bearpolar bear

8



WordNet


Unseentest classes

most similarclasses


Classifierper class

polar bearpolar bear

killer whalekiller

whaleDalmatian……

9

Models for visual knowledge transferSemantic relatedness measuresLanguage resources

WordNet Wikipedia WWW Image search

Respective state-of-the-art measuresEvaluationConclusion

Outline


10

WordNet Lin measure

[Budanitsky & Hirst, CL `06]Wikipedia Explicit Semantic Analysis

[Gabrilovich & MarkovitchI, IJCAI `07]

Word Wide Web Hitcount (Dice coeffient)

[Kilgarriff & Grefenstette, CL `03]

Image search Visually more relevant Hitcount (Dice coeffient)

Semantic Relatedness Measures


WordNet[Fellbaum, MIT press `98]


11

WordNet Lin measure










12

WordNet Lin measure








A farm is an area of lanthe training of horses.

A hoof is the tip of a toe

Rear hooves of a horse

Hoof

Farm

Tusks are long teeth, uElephants and narwhals

Tusk

Article horse elephantFarm 3 0Hoof 2 1Tusk 0 4

… … …



Rear hooves of a horse Most evem tped ungulat

Hoof

Farm


Tusk

13

WordNet Lin measure










Rear hooves of a horse

Hoof

Farm


Tusk

Article horse elephantFarm 3 0Hoof 2 1Tusk 0 4

… … …



Rear hooves of a horse Most evem tped ungulat

Hoof

Farm


Tusk

cosine

14

WordNet Lin measure








15

WordNet Lin measure








We watched a horse race yesterday. [..] Tomorrow we go in the zoo to look at the baby elephant.

„the dance of the horse and elephant“

web search image search[http://www.flickr.com/photos/ lahierophant/2099973716/]

Incidental co-occurence

Terms refer to same entity (the image)

16

Models for visual knowledge transferSemantic relatedness measuresEvaluationAttributes Querying class-attribute associations Mining attributes

Direct similarityAttribute-based vs. direct similarity

Conclusion

Outline


17

Animals with attributes dataset [Lampert et al., CVPR `09]40 training, 10 test classes (disjoint)≈ 30.000 images totalDownsampled to 92 training images per classManual associations to 85 attributes

Image classificationSVM: Histogram intersection kernelArea under ROC curve (AUC) - chance level: 50%Mean over all 10 test classes

Experimental Setup


18

Performance of supervised approach


19

Querying: abbreviationagile

Manual supervision: detailed description“having a high degree of physical coordination”

Querying: abbreviationagile

Manual supervision: detailed description“having a high degree of physical coordination”

Performance of queried association Encouraging Below manual supervision

Image search Based on image related text

Wikipedia Robust resource

Yahoo Web Very noisy resource

WordNet Path length poor indicator of

class-attribute associations

Querying class-attribute association


20


Image search (Yahoo Img, Flickr) Based on image related text

Wikipedia Robust resource (definition texts)






the dance of the horse and elephant

image search

21



Wikipedia Robust resource (definition text)






Noise:While he watched a horse race

the leg of his chair broke.

Noise:While he watched a horse race

the leg of his chair broke.

22



Wikipedia Robust resource (definition text)






23

Mining attributes


WordNet


Attributeterms oceanoceanspotsspots ……


Unseentest classes



whitewhite

24

Mining attributes


WordNet


Attributeterms ???? ??


Unseentest classes



??

25

Part attributesLeg of a dogAttribute classifiers

Mining attributes


WordNet



Unseentest classes



flipperleg paw WordNet

26

Additional measure:Holonym patterns Only part attributesHit Counts of Patterns

[Berland & Charniak, ACL 1999] “cow’s leg” “leg of a cow” Dice coefficient

Mining attributes


While he watched a horse race the leg of his chair broke.

Leg of the horse

web search holonym patterns

Incidental co-occurence

One term likely part of other term

27

Best: Yahoo Holonyms Close to manual attributes Tailored towards part attributes

Mining attributes


28

Best: Yahoo Holonyms Close to manual attributes Tailored towards part attributes

Performance drop Reduced diversity Only part attributes

Specialized terms E.g. pilus (=hair) Coverage problem:

Image search, Wikipedia

Mining attributes


29

Models for visual knowledge transferSemantic relatedness measuresEvaluationAttributes Querying class-attribute associations Mining attributes

Direct similarityAttribute-based vs. direct similarity

Conclusion

Outline


30



WordNet


Unseentest classes

most similarclasses


Classifierper class

polar bearpolar bear

killer whalekiller

whaleDalmatian

31

Nearly all very good On par with manual supervision

attribute model (black) Clearly better than any

mined attribute-associations result

Why? Five most related classes Ranking of semantic

relatedness reliable Similar between methods

Direct Similarity


32

0 10.000 20.000

65

70

75

80

mea

n AU

C (in

%)

Number of additional training class images in test set

attributes: manually definedattributes: mined associationsattributes: mined attributesdirect similarity

Attributes vs. direct similarity


Extending the test setAdd images From known classes As negatives

More realistic setting

ResultsDirect similarity

drop in performance(orange curve)

Attribute modelsgeneralize well

0 10.000 20.000

65

70

75

80

mea

n AU

C (in

%)


attributes: manually definedattributes: queried associationsattributes: mined attributesdirect similarity

0 10.000 20.000

65

70

75

80

mea

n AU

C (in

%)



0 10.000 20.000

65

70

75

80

mea

n AU

C (in

%)



0 10,000 20,000

65

70

75

80

mea

n AU

C (in

%)



33

Models for visual knowledge transferSemantic relatedness measuresEvaluationConclusion

Outline


34

Supervision replaced with semantic relatedness Direct similarity better than attributes on par with supervised approach

Attributes: generalizes better

Semantic relatedness measures Overall best Yahoo image with hit count

Holonym patterns for web search Improvement Limited to part attributes

Conclusion


35


Attributes: generalize better



Conclusion


0 10.000 20.000

65

70

75

80

mea

n AU

C (in

%)



0 10.000 20.000

65

70

75

80

mea

n AU

C (in

%)



0 10.000 20.000

65

70

75

80

mea

n AU

C (in

%)



0 10.000 20.000

65

70

75

80

mea

n AU

C (in

%)



0 10.000 20.000

65

70

75

80

mea

n AU

C (in

%)



36


Attributes: generalize better



WordNet poor for object-attributes associations

Conclusion


patterns:dog’s legleg of the dogs

patterns:dog’s legleg of the dogs

0 10.000 20.000

65

70

75

80

mea

n AU

C (in

%)



0 10.000 20.000

65

70

75

80

mea

n AU

C (in

%)



0 10.000 20.000

65

70

75

80

mea

n AU

C (in

%)



0 10.000 20.000

65

70

75

80

mea

n AU

C (in

%)



0 10.000 20.000

65

70

75

80

mea

n AU

C (in

%)



37

Further supervision for closing the semantic gap?See us at our poster (A2, Atrium)!


Knowledge Transfer

Thank you!

Software? www.mis.tu-darmstadt.de/nlp4vision

What Helps Where – And Why? Semantic Relatedness for Knowledge Transfer Marcus Rohrbach 1,2...

Documents

Transcript of What Helps Where – And Why? Semantic Relatedness for Knowledge Transfer Marcus Rohrbach 1,2...