Relative Attributes - University of Texas at...
Transcript of Relative Attributes - University of Texas at...
![Page 1: Relative Attributes - University of Texas at Austinvision.cs.utexas.edu/381V-spring2016/slides/padmakumar... · 2016-03-05 · Relative Attributes Devi Parikh and Kristen Grauman](https://reader034.fdocuments.us/reader034/viewer/2022043006/5f8f70985964e7140d7d476a/html5/thumbnails/1.jpg)
Relative AttributesDevi Parikh and Kristen Grauman
ICCV 2011
Experiment Presentation by Aishwarya Padmakumar
![Page 2: Relative Attributes - University of Texas at Austinvision.cs.utexas.edu/381V-spring2016/slides/padmakumar... · 2016-03-05 · Relative Attributes Devi Parikh and Kristen Grauman](https://reader034.fdocuments.us/reader034/viewer/2022043006/5f8f70985964e7140d7d476a/html5/thumbnails/2.jpg)
Motivation
Image Source: https://filebox.ece.vt.edu/~parikh/relative.html
Sometimes it is hard to make a binary decision on whether an image satisfies an attribute. Comparisons are easier.
![Page 3: Relative Attributes - University of Texas at Austinvision.cs.utexas.edu/381V-spring2016/slides/padmakumar... · 2016-03-05 · Relative Attributes Devi Parikh and Kristen Grauman](https://reader034.fdocuments.us/reader034/viewer/2022043006/5f8f70985964e7140d7d476a/html5/thumbnails/3.jpg)
Sources
Code and data for following experiments obtained from :https://filebox.ece.vt.edu/~parikh/relative.html#code
![Page 4: Relative Attributes - University of Texas at Austinvision.cs.utexas.edu/381V-spring2016/slides/padmakumar... · 2016-03-05 · Relative Attributes Devi Parikh and Kristen Grauman](https://reader034.fdocuments.us/reader034/viewer/2022043006/5f8f70985964e7140d7d476a/html5/thumbnails/4.jpg)
Pubfig dataset
→ Public Figures Face Database - Images of famous public figures
→ Images of the same person are grouped into categories→ Attributes - Male, White, Young, Smiling, Chubby,
VisibleForehead, BushyEyebrows, NarrowEyes, PointyNose,BigLips, RoundFace
![Page 5: Relative Attributes - University of Texas at Austinvision.cs.utexas.edu/381V-spring2016/slides/padmakumar... · 2016-03-05 · Relative Attributes Devi Parikh and Kristen Grauman](https://reader034.fdocuments.us/reader034/viewer/2022043006/5f8f70985964e7140d7d476a/html5/thumbnails/5.jpg)
Ground Truth - Ordering of categories for each attribute
Attr1 Attr2 Attr3 Attr4
Class1 1 2 3 3
Class2 2 1 1 1
Class3 3 3 2 2
![Page 6: Relative Attributes - University of Texas at Austinvision.cs.utexas.edu/381V-spring2016/slides/padmakumar... · 2016-03-05 · Relative Attributes Devi Parikh and Kristen Grauman](https://reader034.fdocuments.us/reader034/viewer/2022043006/5f8f70985964e7140d7d476a/html5/thumbnails/6.jpg)
Evaluating Ranking
→ Obtain training pairs using category ranking→ Train SVM to get a score for each image→ Average scores of images in category to get score of
category→ Rank categories accoridng to this score→ Compare to ground truth ranking
![Page 7: Relative Attributes - University of Texas at Austinvision.cs.utexas.edu/381V-spring2016/slides/padmakumar... · 2016-03-05 · Relative Attributes Devi Parikh and Kristen Grauman](https://reader034.fdocuments.us/reader034/viewer/2022043006/5f8f70985964e7140d7d476a/html5/thumbnails/7.jpg)
Ranking PredictionC
lass
es
Attributes Rank difference
0
1
2
3
4
5
6
7
![Page 8: Relative Attributes - University of Texas at Austinvision.cs.utexas.edu/381V-spring2016/slides/padmakumar... · 2016-03-05 · Relative Attributes Devi Parikh and Kristen Grauman](https://reader034.fdocuments.us/reader034/viewer/2022043006/5f8f70985964e7140d7d476a/html5/thumbnails/8.jpg)
Varying the amount of training data
![Page 9: Relative Attributes - University of Texas at Austinvision.cs.utexas.edu/381V-spring2016/slides/padmakumar... · 2016-03-05 · Relative Attributes Devi Parikh and Kristen Grauman](https://reader034.fdocuments.us/reader034/viewer/2022043006/5f8f70985964e7140d7d476a/html5/thumbnails/9.jpg)
With original amount of training dataC
lass
es
Attributes Rank difference
0
1
2
3
4
5
6
7
![Page 10: Relative Attributes - University of Texas at Austinvision.cs.utexas.edu/381V-spring2016/slides/padmakumar... · 2016-03-05 · Relative Attributes Devi Parikh and Kristen Grauman](https://reader034.fdocuments.us/reader034/viewer/2022043006/5f8f70985964e7140d7d476a/html5/thumbnails/10.jpg)
With 80% of original training dataAttributes
Cla
sses
![Page 11: Relative Attributes - University of Texas at Austinvision.cs.utexas.edu/381V-spring2016/slides/padmakumar... · 2016-03-05 · Relative Attributes Devi Parikh and Kristen Grauman](https://reader034.fdocuments.us/reader034/viewer/2022043006/5f8f70985964e7140d7d476a/html5/thumbnails/11.jpg)
With 60% of original training dataAttributes
Cla
sses
![Page 12: Relative Attributes - University of Texas at Austinvision.cs.utexas.edu/381V-spring2016/slides/padmakumar... · 2016-03-05 · Relative Attributes Devi Parikh and Kristen Grauman](https://reader034.fdocuments.us/reader034/viewer/2022043006/5f8f70985964e7140d7d476a/html5/thumbnails/12.jpg)
With 40% of original training dataAttributes
Cla
sses
![Page 13: Relative Attributes - University of Texas at Austinvision.cs.utexas.edu/381V-spring2016/slides/padmakumar... · 2016-03-05 · Relative Attributes Devi Parikh and Kristen Grauman](https://reader034.fdocuments.us/reader034/viewer/2022043006/5f8f70985964e7140d7d476a/html5/thumbnails/13.jpg)
What sort of mistakes does it make?
![Page 14: Relative Attributes - University of Texas at Austinvision.cs.utexas.edu/381V-spring2016/slides/padmakumar... · 2016-03-05 · Relative Attributes Devi Parikh and Kristen Grauman](https://reader034.fdocuments.us/reader034/viewer/2022043006/5f8f70985964e7140d7d476a/html5/thumbnails/14.jpg)
Failure cases - Who has bigger lips?
![Page 15: Relative Attributes - University of Texas at Austinvision.cs.utexas.edu/381V-spring2016/slides/padmakumar... · 2016-03-05 · Relative Attributes Devi Parikh and Kristen Grauman](https://reader034.fdocuments.us/reader034/viewer/2022043006/5f8f70985964e7140d7d476a/html5/thumbnails/15.jpg)
Failure cases - Who has bigger lips?
Supposed to be the person on the right but certain expressions don’t indicate this.
![Page 16: Relative Attributes - University of Texas at Austinvision.cs.utexas.edu/381V-spring2016/slides/padmakumar... · 2016-03-05 · Relative Attributes Devi Parikh and Kristen Grauman](https://reader034.fdocuments.us/reader034/viewer/2022043006/5f8f70985964e7140d7d476a/html5/thumbnails/16.jpg)
Failure cases - Who looks younger?
![Page 17: Relative Attributes - University of Texas at Austinvision.cs.utexas.edu/381V-spring2016/slides/padmakumar... · 2016-03-05 · Relative Attributes Devi Parikh and Kristen Grauman](https://reader034.fdocuments.us/reader034/viewer/2022043006/5f8f70985964e7140d7d476a/html5/thumbnails/17.jpg)
Failure cases - Who looks younger?
Supposed to be the girl on the right. Other images are more obvious but expression, make-up and lighting affect this.
![Page 18: Relative Attributes - University of Texas at Austinvision.cs.utexas.edu/381V-spring2016/slides/padmakumar... · 2016-03-05 · Relative Attributes Devi Parikh and Kristen Grauman](https://reader034.fdocuments.us/reader034/viewer/2022043006/5f8f70985964e7140d7d476a/html5/thumbnails/18.jpg)
Instance Level Supervision vs Category Level Supervision
![Page 19: Relative Attributes - University of Texas at Austinvision.cs.utexas.edu/381V-spring2016/slides/padmakumar... · 2016-03-05 · Relative Attributes Devi Parikh and Kristen Grauman](https://reader034.fdocuments.us/reader034/viewer/2022043006/5f8f70985964e7140d7d476a/html5/thumbnails/19.jpg)
WhittleSearch Dataset
→ Collected by: Adriana Kovashka, Devi Parikh, and Kristen Grauman
→ Publication: Whittle Search: Image Search with Relative Attribute Feedback, in CVPR 2012
→ Dataset of shoes: 14658 images→ Provide extracted features (GIST and colour - LAB
histogram), accounting for image orientation
![Page 20: Relative Attributes - University of Texas at Austinvision.cs.utexas.edu/381V-spring2016/slides/padmakumar... · 2016-03-05 · Relative Attributes Devi Parikh and Kristen Grauman](https://reader034.fdocuments.us/reader034/viewer/2022043006/5f8f70985964e7140d7d476a/html5/thumbnails/20.jpg)
→ Category level ordering and instance level comparisons for each attribute
→ Attributes: pointy-at-the-front, open, bright-in-color, covered-with-ornaments, shiny, high-at-the-heel, long-on-the-leg, formal, sporty, feminine
→ Instance level ordering obtained via MTurk
WhittleSearch Dataset
![Page 21: Relative Attributes - University of Texas at Austinvision.cs.utexas.edu/381V-spring2016/slides/padmakumar... · 2016-03-05 · Relative Attributes Devi Parikh and Kristen Grauman](https://reader034.fdocuments.us/reader034/viewer/2022043006/5f8f70985964e7140d7d476a/html5/thumbnails/21.jpg)
Examples
Women’s Boots
Women’s flatsWomen’s stiletto
![Page 22: Relative Attributes - University of Texas at Austinvision.cs.utexas.edu/381V-spring2016/slides/padmakumar... · 2016-03-05 · Relative Attributes Devi Parikh and Kristen Grauman](https://reader034.fdocuments.us/reader034/viewer/2022043006/5f8f70985964e7140d7d476a/html5/thumbnails/22.jpg)
Instance level supervision vs category level supervisionA
ccur
acy
![Page 23: Relative Attributes - University of Texas at Austinvision.cs.utexas.edu/381V-spring2016/slides/padmakumar... · 2016-03-05 · Relative Attributes Devi Parikh and Kristen Grauman](https://reader034.fdocuments.us/reader034/viewer/2022043006/5f8f70985964e7140d7d476a/html5/thumbnails/23.jpg)
Why does training on category level information do well sometimes?→ List of categories - athletic_shoes, boots, clogs, flats,
high_heels, pumps, rain_boots, sneakers, stiletto, wedding_shoes
→ Pointy, ornaments, long on the leg seem to be attributes that we can associate with some of these classes
→ Possibly category level labels result in more well-separated data, making it easier to train an SVM (SVMs are sensitive to noise near the support vectors)
![Page 24: Relative Attributes - University of Texas at Austinvision.cs.utexas.edu/381V-spring2016/slides/padmakumar... · 2016-03-05 · Relative Attributes Devi Parikh and Kristen Grauman](https://reader034.fdocuments.us/reader034/viewer/2022043006/5f8f70985964e7140d7d476a/html5/thumbnails/24.jpg)
Some cases where the prediction differs
More Sporty: Ground truth - left; Instance based - left; Category based - right
![Page 25: Relative Attributes - University of Texas at Austinvision.cs.utexas.edu/381V-spring2016/slides/padmakumar... · 2016-03-05 · Relative Attributes Devi Parikh and Kristen Grauman](https://reader034.fdocuments.us/reader034/viewer/2022043006/5f8f70985964e7140d7d476a/html5/thumbnails/25.jpg)
Some cases where the prediction differs
More Sporty: Ground truth - left; Instance based - right; Category based - left
![Page 26: Relative Attributes - University of Texas at Austinvision.cs.utexas.edu/381V-spring2016/slides/padmakumar... · 2016-03-05 · Relative Attributes Devi Parikh and Kristen Grauman](https://reader034.fdocuments.us/reader034/viewer/2022043006/5f8f70985964e7140d7d476a/html5/thumbnails/26.jpg)
Some cases where the prediction differs
More Sporty: Ground truth - equal; Instance based - right; Category based - left
![Page 27: Relative Attributes - University of Texas at Austinvision.cs.utexas.edu/381V-spring2016/slides/padmakumar... · 2016-03-05 · Relative Attributes Devi Parikh and Kristen Grauman](https://reader034.fdocuments.us/reader034/viewer/2022043006/5f8f70985964e7140d7d476a/html5/thumbnails/27.jpg)
Usefulness of different types of features
![Page 28: Relative Attributes - University of Texas at Austinvision.cs.utexas.edu/381V-spring2016/slides/padmakumar... · 2016-03-05 · Relative Attributes Devi Parikh and Kristen Grauman](https://reader034.fdocuments.us/reader034/viewer/2022043006/5f8f70985964e7140d7d476a/html5/thumbnails/28.jpg)
Comparing GIST and colour features - instance comparisons
![Page 29: Relative Attributes - University of Texas at Austinvision.cs.utexas.edu/381V-spring2016/slides/padmakumar... · 2016-03-05 · Relative Attributes Devi Parikh and Kristen Grauman](https://reader034.fdocuments.us/reader034/viewer/2022043006/5f8f70985964e7140d7d476a/html5/thumbnails/29.jpg)
Comparing GIST and colour features - category comparisons
![Page 30: Relative Attributes - University of Texas at Austinvision.cs.utexas.edu/381V-spring2016/slides/padmakumar... · 2016-03-05 · Relative Attributes Devi Parikh and Kristen Grauman](https://reader034.fdocuments.us/reader034/viewer/2022043006/5f8f70985964e7140d7d476a/html5/thumbnails/30.jpg)
Where GIST and the combination differ
More Bright: Ground truth - right; GIST - equal; Both - right
![Page 31: Relative Attributes - University of Texas at Austinvision.cs.utexas.edu/381V-spring2016/slides/padmakumar... · 2016-03-05 · Relative Attributes Devi Parikh and Kristen Grauman](https://reader034.fdocuments.us/reader034/viewer/2022043006/5f8f70985964e7140d7d476a/html5/thumbnails/31.jpg)
Where GIST and the combination differ
More Bright: Ground truth - left; GIST - equal; Both - right
![Page 32: Relative Attributes - University of Texas at Austinvision.cs.utexas.edu/381V-spring2016/slides/padmakumar... · 2016-03-05 · Relative Attributes Devi Parikh and Kristen Grauman](https://reader034.fdocuments.us/reader034/viewer/2022043006/5f8f70985964e7140d7d476a/html5/thumbnails/32.jpg)
Where colour and the combination differ
More Bright: Ground truth - right; Colour - equal; Both - right
![Page 33: Relative Attributes - University of Texas at Austinvision.cs.utexas.edu/381V-spring2016/slides/padmakumar... · 2016-03-05 · Relative Attributes Devi Parikh and Kristen Grauman](https://reader034.fdocuments.us/reader034/viewer/2022043006/5f8f70985964e7140d7d476a/html5/thumbnails/33.jpg)
Where colour and the combination differ
More Bright: Ground truth - equal; Colour - equal; Both - left
![Page 34: Relative Attributes - University of Texas at Austinvision.cs.utexas.edu/381V-spring2016/slides/padmakumar... · 2016-03-05 · Relative Attributes Devi Parikh and Kristen Grauman](https://reader034.fdocuments.us/reader034/viewer/2022043006/5f8f70985964e7140d7d476a/html5/thumbnails/34.jpg)
Relative Descriptions
![Page 35: Relative Attributes - University of Texas at Austinvision.cs.utexas.edu/381V-spring2016/slides/padmakumar... · 2016-03-05 · Relative Attributes Devi Parikh and Kristen Grauman](https://reader034.fdocuments.us/reader034/viewer/2022043006/5f8f70985964e7140d7d476a/html5/thumbnails/35.jpg)
Relative Descriptions
![Page 36: Relative Attributes - University of Texas at Austinvision.cs.utexas.edu/381V-spring2016/slides/padmakumar... · 2016-03-05 · Relative Attributes Devi Parikh and Kristen Grauman](https://reader034.fdocuments.us/reader034/viewer/2022043006/5f8f70985964e7140d7d476a/html5/thumbnails/36.jpg)
Relative Descriptions - Pointy at the front
![Page 37: Relative Attributes - University of Texas at Austinvision.cs.utexas.edu/381V-spring2016/slides/padmakumar... · 2016-03-05 · Relative Attributes Devi Parikh and Kristen Grauman](https://reader034.fdocuments.us/reader034/viewer/2022043006/5f8f70985964e7140d7d476a/html5/thumbnails/37.jpg)
Relative Descriptions - open
![Page 38: Relative Attributes - University of Texas at Austinvision.cs.utexas.edu/381V-spring2016/slides/padmakumar... · 2016-03-05 · Relative Attributes Devi Parikh and Kristen Grauman](https://reader034.fdocuments.us/reader034/viewer/2022043006/5f8f70985964e7140d7d476a/html5/thumbnails/38.jpg)
Relative Descriptions - bright in colour
![Page 39: Relative Attributes - University of Texas at Austinvision.cs.utexas.edu/381V-spring2016/slides/padmakumar... · 2016-03-05 · Relative Attributes Devi Parikh and Kristen Grauman](https://reader034.fdocuments.us/reader034/viewer/2022043006/5f8f70985964e7140d7d476a/html5/thumbnails/39.jpg)
Relative Descriptions - covered in ornaments
![Page 40: Relative Attributes - University of Texas at Austinvision.cs.utexas.edu/381V-spring2016/slides/padmakumar... · 2016-03-05 · Relative Attributes Devi Parikh and Kristen Grauman](https://reader034.fdocuments.us/reader034/viewer/2022043006/5f8f70985964e7140d7d476a/html5/thumbnails/40.jpg)
Relative Descriptions - shiny
![Page 41: Relative Attributes - University of Texas at Austinvision.cs.utexas.edu/381V-spring2016/slides/padmakumar... · 2016-03-05 · Relative Attributes Devi Parikh and Kristen Grauman](https://reader034.fdocuments.us/reader034/viewer/2022043006/5f8f70985964e7140d7d476a/html5/thumbnails/41.jpg)
Relative Descriptions - high at the heel
![Page 42: Relative Attributes - University of Texas at Austinvision.cs.utexas.edu/381V-spring2016/slides/padmakumar... · 2016-03-05 · Relative Attributes Devi Parikh and Kristen Grauman](https://reader034.fdocuments.us/reader034/viewer/2022043006/5f8f70985964e7140d7d476a/html5/thumbnails/42.jpg)
Relative Descriptions - long on the leg
![Page 43: Relative Attributes - University of Texas at Austinvision.cs.utexas.edu/381V-spring2016/slides/padmakumar... · 2016-03-05 · Relative Attributes Devi Parikh and Kristen Grauman](https://reader034.fdocuments.us/reader034/viewer/2022043006/5f8f70985964e7140d7d476a/html5/thumbnails/43.jpg)
Relative Descriptions - formal
![Page 44: Relative Attributes - University of Texas at Austinvision.cs.utexas.edu/381V-spring2016/slides/padmakumar... · 2016-03-05 · Relative Attributes Devi Parikh and Kristen Grauman](https://reader034.fdocuments.us/reader034/viewer/2022043006/5f8f70985964e7140d7d476a/html5/thumbnails/44.jpg)
Relative Descriptions - sporty
![Page 45: Relative Attributes - University of Texas at Austinvision.cs.utexas.edu/381V-spring2016/slides/padmakumar... · 2016-03-05 · Relative Attributes Devi Parikh and Kristen Grauman](https://reader034.fdocuments.us/reader034/viewer/2022043006/5f8f70985964e7140d7d476a/html5/thumbnails/45.jpg)
Relative Descriptions - feminine
![Page 46: Relative Attributes - University of Texas at Austinvision.cs.utexas.edu/381V-spring2016/slides/padmakumar... · 2016-03-05 · Relative Attributes Devi Parikh and Kristen Grauman](https://reader034.fdocuments.us/reader034/viewer/2022043006/5f8f70985964e7140d7d476a/html5/thumbnails/46.jpg)
Varying the number of images in between
![Page 47: Relative Attributes - University of Texas at Austinvision.cs.utexas.edu/381V-spring2016/slides/padmakumar... · 2016-03-05 · Relative Attributes Devi Parikh and Kristen Grauman](https://reader034.fdocuments.us/reader034/viewer/2022043006/5f8f70985964e7140d7d476a/html5/thumbnails/47.jpg)
Leave 1/16 of the images in between (formal)
![Page 48: Relative Attributes - University of Texas at Austinvision.cs.utexas.edu/381V-spring2016/slides/padmakumar... · 2016-03-05 · Relative Attributes Devi Parikh and Kristen Grauman](https://reader034.fdocuments.us/reader034/viewer/2022043006/5f8f70985964e7140d7d476a/html5/thumbnails/48.jpg)
Leave 1/8 of the images in between (formal)
![Page 49: Relative Attributes - University of Texas at Austinvision.cs.utexas.edu/381V-spring2016/slides/padmakumar... · 2016-03-05 · Relative Attributes Devi Parikh and Kristen Grauman](https://reader034.fdocuments.us/reader034/viewer/2022043006/5f8f70985964e7140d7d476a/html5/thumbnails/49.jpg)
Leave 1/4 of the images in between (formal)
![Page 50: Relative Attributes - University of Texas at Austinvision.cs.utexas.edu/381V-spring2016/slides/padmakumar... · 2016-03-05 · Relative Attributes Devi Parikh and Kristen Grauman](https://reader034.fdocuments.us/reader034/viewer/2022043006/5f8f70985964e7140d7d476a/html5/thumbnails/50.jpg)
Leave 1/2 of the images in between (formal)
![Page 51: Relative Attributes - University of Texas at Austinvision.cs.utexas.edu/381V-spring2016/slides/padmakumar... · 2016-03-05 · Relative Attributes Devi Parikh and Kristen Grauman](https://reader034.fdocuments.us/reader034/viewer/2022043006/5f8f70985964e7140d7d476a/html5/thumbnails/51.jpg)
Thank You!