WhittleSearch: Image Search with Relative Attribute Feedback CVPR 2012 Adriana Kovashka Devi Parikh...

20
WhittleSearch: Image Search with Relative Attribute Feedback CVPR 2012 Adriana Kovashka Devi Parikh Kristen Grauman University of Texas at Austin Toyota Technological Institute Chicago (TTIC)

Transcript of WhittleSearch: Image Search with Relative Attribute Feedback CVPR 2012 Adriana Kovashka Devi Parikh...

Page 1: WhittleSearch: Image Search with Relative Attribute Feedback CVPR 2012 Adriana Kovashka Devi Parikh Kristen Grauman University of Texas at Austin Toyota.

WhittleSearch: Image Search with Relative Attribute FeedbackCVPR 2012

Adriana Kovashka Devi Parikh Kristen Grauman

University of Texas at Austin Toyota Technological Institute Chicago (TTIC)

Page 2: WhittleSearch: Image Search with Relative Attribute Feedback CVPR 2012 Adriana Kovashka Devi Parikh Kristen Grauman University of Texas at Austin Toyota.
Page 3: WhittleSearch: Image Search with Relative Attribute Feedback CVPR 2012 Adriana Kovashka Devi Parikh Kristen Grauman University of Texas at Austin Toyota.
Page 4: WhittleSearch: Image Search with Relative Attribute Feedback CVPR 2012 Adriana Kovashka Devi Parikh Kristen Grauman University of Texas at Austin Toyota.

Approach

•Dataset

•At each iteration, the top K < N ranked images

Page 5: WhittleSearch: Image Search with Relative Attribute Feedback CVPR 2012 Adriana Kovashka Devi Parikh Kristen Grauman University of Texas at Austin Toyota.

Binary Relevance Feedback

Page 6: WhittleSearch: Image Search with Relative Attribute Feedback CVPR 2012 Adriana Kovashka Devi Parikh Kristen Grauman University of Texas at Austin Toyota.

Binary Relevance Feedback

•Current reference set

•Score function

Page 7: WhittleSearch: Image Search with Relative Attribute Feedback CVPR 2012 Adriana Kovashka Devi Parikh Kristen Grauman University of Texas at Austin Toyota.

Relative Attribute Feedback

Page 8: WhittleSearch: Image Search with Relative Attribute Feedback CVPR 2012 Adriana Kovashka Devi Parikh Kristen Grauman University of Texas at Austin Toyota.

Learning Relative Attributes

•Mechanical Turk (MTurk)

Page 9: WhittleSearch: Image Search with Relative Attribute Feedback CVPR 2012 Adriana Kovashka Devi Parikh Kristen Grauman University of Texas at Austin Toyota.
Page 10: WhittleSearch: Image Search with Relative Attribute Feedback CVPR 2012 Adriana Kovashka Devi Parikh Kristen Grauman University of Texas at Austin Toyota.

Updating the scoring function from feedback•Feedback form

▫“ What I want is more/less/similarly m than image Itr ”

Page 11: WhittleSearch: Image Search with Relative Attribute Feedback CVPR 2012 Adriana Kovashka Devi Parikh Kristen Grauman University of Texas at Austin Toyota.

Three cases

Page 12: WhittleSearch: Image Search with Relative Attribute Feedback CVPR 2012 Adriana Kovashka Devi Parikh Kristen Grauman University of Texas at Austin Toyota.

Score function of Relative Attribute Feedback

•Take the intersection of all F feedback

Page 13: WhittleSearch: Image Search with Relative Attribute Feedback CVPR 2012 Adriana Kovashka Devi Parikh Kristen Grauman University of Texas at Austin Toyota.

Hybrid Feedback Approach

Page 14: WhittleSearch: Image Search with Relative Attribute Feedback CVPR 2012 Adriana Kovashka Devi Parikh Kristen Grauman University of Texas at Austin Toyota.

x denotes the Cartesian product

Page 15: WhittleSearch: Image Search with Relative Attribute Feedback CVPR 2012 Adriana Kovashka Devi Parikh Kristen Grauman University of Texas at Austin Toyota.

Experimental Results

Page 16: WhittleSearch: Image Search with Relative Attribute Feedback CVPR 2012 Adriana Kovashka Devi Parikh Kristen Grauman University of Texas at Austin Toyota.

Datasets• Shoes

▫14,658 shoe images from like.com▫attributes—‘pointy-at-the-front’, ‘open’, ‘bright-

in-color’, ‘covered-with-ornaments’, ‘shiny’, ‘highat-the-heel’, ‘long-on-the-leg’, ‘formal’, ‘sporty’, and ‘feminine’

• PubFig▫the Public Figures dataset of human faces▫772 images from 8 people and 11 attributes

• OSR▫the Outdoor Scene Recognition dataset of

natural scenes▫2,688 images from 8 categories and 6 attributes

Page 17: WhittleSearch: Image Search with Relative Attribute Feedback CVPR 2012 Adriana Kovashka Devi Parikh Kristen Grauman University of Texas at Austin Toyota.

Datasets

•Training set ▫750 triplets of images (i , j , k) from each

dataset

•Score correlation▫Normalized Discounted Cumulative Gain at

top K (NDCG@K)▫K = 50

Page 18: WhittleSearch: Image Search with Relative Attribute Feedback CVPR 2012 Adriana Kovashka Devi Parikh Kristen Grauman University of Texas at Austin Toyota.

Iteration experiments on the three datasets

Page 19: WhittleSearch: Image Search with Relative Attribute Feedback CVPR 2012 Adriana Kovashka Devi Parikh Kristen Grauman University of Texas at Austin Toyota.

Amount of feedback

Page 20: WhittleSearch: Image Search with Relative Attribute Feedback CVPR 2012 Adriana Kovashka Devi Parikh Kristen Grauman University of Texas at Austin Toyota.

Ranking accuracy with first feedback

•Attribute meaning▫“amount of perspective” on a scene is less

intuitivethan “shininess” on shoes