What makes an image memorable?cv-fall2012/slides/islam-expt.pdf · 2012-10-26 · What makes an...
Transcript of What makes an image memorable?cv-fall2012/slides/islam-expt.pdf · 2012-10-26 · What makes an...
![Page 1: What makes an image memorable?cv-fall2012/slides/islam-expt.pdf · 2012-10-26 · What makes an image memorable? P. Isola, J. Xiao, A. Torralba, A. Oliva. CVPR 2011 Islam Beltagy](https://reader034.fdocuments.us/reader034/viewer/2022042218/5ec4a83df97204754e20a029/html5/thumbnails/1.jpg)
What makes an image memorable?
P. Isola, J. Xiao, A. Torralba, A. Oliva. CVPR 2011
Islam Beltagy
![Page 2: What makes an image memorable?cv-fall2012/slides/islam-expt.pdf · 2012-10-26 · What makes an image memorable? P. Isola, J. Xiao, A. Torralba, A. Oliva. CVPR 2011 Islam Beltagy](https://reader034.fdocuments.us/reader034/viewer/2022042218/5ec4a83df97204754e20a029/html5/thumbnails/2.jpg)
Experiment
• https://picasaweb.google.com/101392440470561716618/CS395TVisualRecognition
![Page 3: What makes an image memorable?cv-fall2012/slides/islam-expt.pdf · 2012-10-26 · What makes an image memorable? P. Isola, J. Xiao, A. Torralba, A. Oliva. CVPR 2011 Islam Beltagy](https://reader034.fdocuments.us/reader034/viewer/2022042218/5ec4a83df97204754e20a029/html5/thumbnails/3.jpg)
LabelMe
• Images database and annotation tool• A helpful matlab toolbox
![Page 4: What makes an image memorable?cv-fall2012/slides/islam-expt.pdf · 2012-10-26 · What makes an image memorable? P. Isola, J. Xiao, A. Torralba, A. Oliva. CVPR 2011 Islam Beltagy](https://reader034.fdocuments.us/reader034/viewer/2022042218/5ec4a83df97204754e20a029/html5/thumbnails/4.jpg)
Agenda
• Memorability vs Object class
• Memorabiliy vs Saliency
![Page 5: What makes an image memorable?cv-fall2012/slides/islam-expt.pdf · 2012-10-26 · What makes an image memorable? P. Isola, J. Xiao, A. Torralba, A. Oliva. CVPR 2011 Islam Beltagy](https://reader034.fdocuments.us/reader034/viewer/2022042218/5ec4a83df97204754e20a029/html5/thumbnails/5.jpg)
Memorability and object classes
• Task:– Find relation between each individual object
class and memorability• Features:
– Object Presence– Object Counts– Object Area
![Page 6: What makes an image memorable?cv-fall2012/slides/islam-expt.pdf · 2012-10-26 · What makes an image memorable? P. Isola, J. Xiao, A. Torralba, A. Oliva. CVPR 2011 Islam Beltagy](https://reader034.fdocuments.us/reader034/viewer/2022042218/5ec4a83df97204754e20a029/html5/thumbnails/6.jpg)
Memorability and object classes
• Use Spearmen correlation to find correlation between class features statistics (presence, count, area) and memorability (from ground truth).
![Page 7: What makes an image memorable?cv-fall2012/slides/islam-expt.pdf · 2012-10-26 · What makes an image memorable? P. Isola, J. Xiao, A. Torralba, A. Oliva. CVPR 2011 Islam Beltagy](https://reader034.fdocuments.us/reader034/viewer/2022042218/5ec4a83df97204754e20a029/html5/thumbnails/7.jpg)
![Page 8: What makes an image memorable?cv-fall2012/slides/islam-expt.pdf · 2012-10-26 · What makes an image memorable? P. Isola, J. Xiao, A. Torralba, A. Oliva. CVPR 2011 Islam Beltagy](https://reader034.fdocuments.us/reader034/viewer/2022042218/5ec4a83df97204754e20a029/html5/thumbnails/8.jpg)
Object type (all sorted by correlation with Presence)
![Page 9: What makes an image memorable?cv-fall2012/slides/islam-expt.pdf · 2012-10-26 · What makes an image memorable? P. Isola, J. Xiao, A. Torralba, A. Oliva. CVPR 2011 Islam Beltagy](https://reader034.fdocuments.us/reader034/viewer/2022042218/5ec4a83df97204754e20a029/html5/thumbnails/9.jpg)
![Page 10: What makes an image memorable?cv-fall2012/slides/islam-expt.pdf · 2012-10-26 · What makes an image memorable? P. Isola, J. Xiao, A. Torralba, A. Oliva. CVPR 2011 Islam Beltagy](https://reader034.fdocuments.us/reader034/viewer/2022042218/5ec4a83df97204754e20a029/html5/thumbnails/10.jpg)
Memorability and object classes
• Most and Least memorable object classes are 50/1360 class. (|Correlation| >=0.2)
• Most of the object classes are not correlated with memorability, why ?
![Page 11: What makes an image memorable?cv-fall2012/slides/islam-expt.pdf · 2012-10-26 · What makes an image memorable? P. Isola, J. Xiao, A. Torralba, A. Oliva. CVPR 2011 Islam Beltagy](https://reader034.fdocuments.us/reader034/viewer/2022042218/5ec4a83df97204754e20a029/html5/thumbnails/11.jpg)
Windows (freq: 588)
door (freq: 360)
Ice field (freq: 4)
|Correlation|>=0.2
![Page 12: What makes an image memorable?cv-fall2012/slides/islam-expt.pdf · 2012-10-26 · What makes an image memorable? P. Isola, J. Xiao, A. Torralba, A. Oliva. CVPR 2011 Islam Beltagy](https://reader034.fdocuments.us/reader034/viewer/2022042218/5ec4a83df97204754e20a029/html5/thumbnails/12.jpg)
Memorability and object classes
• Most of the object classes are not correlated with memorability, why ? Because:– Not frequent enough to find a correlation
pattern (dataset limitation)– Very common on all memorability levels– Correlated with medium levels of memorability
![Page 13: What makes an image memorable?cv-fall2012/slides/islam-expt.pdf · 2012-10-26 · What makes an image memorable? P. Isola, J. Xiao, A. Torralba, A. Oliva. CVPR 2011 Islam Beltagy](https://reader034.fdocuments.us/reader034/viewer/2022042218/5ec4a83df97204754e20a029/html5/thumbnails/13.jpg)
Memorability and object classes
• What about high correlation with very low frequency ? (like “ice field”)
![Page 14: What makes an image memorable?cv-fall2012/slides/islam-expt.pdf · 2012-10-26 · What makes an image memorable? P. Isola, J. Xiao, A. Torralba, A. Oliva. CVPR 2011 Islam Beltagy](https://reader034.fdocuments.us/reader034/viewer/2022042218/5ec4a83df97204754e20a029/html5/thumbnails/14.jpg)
![Page 15: What makes an image memorable?cv-fall2012/slides/islam-expt.pdf · 2012-10-26 · What makes an image memorable? P. Isola, J. Xiao, A. Torralba, A. Oliva. CVPR 2011 Islam Beltagy](https://reader034.fdocuments.us/reader034/viewer/2022042218/5ec4a83df97204754e20a029/html5/thumbnails/15.jpg)
Filter by frequency
50
![Page 16: What makes an image memorable?cv-fall2012/slides/islam-expt.pdf · 2012-10-26 · What makes an image memorable? P. Isola, J. Xiao, A. Torralba, A. Oliva. CVPR 2011 Islam Beltagy](https://reader034.fdocuments.us/reader034/viewer/2022042218/5ec4a83df97204754e20a029/html5/thumbnails/16.jpg)
Side effect of filtration by frequency
![Page 17: What makes an image memorable?cv-fall2012/slides/islam-expt.pdf · 2012-10-26 · What makes an image memorable? P. Isola, J. Xiao, A. Torralba, A. Oliva. CVPR 2011 Islam Beltagy](https://reader034.fdocuments.us/reader034/viewer/2022042218/5ec4a83df97204754e20a029/html5/thumbnails/17.jpg)
High +ve correlation
High - ve correlation
Low correlation
Correlation after filtration
![Page 18: What makes an image memorable?cv-fall2012/slides/islam-expt.pdf · 2012-10-26 · What makes an image memorable? P. Isola, J. Xiao, A. Torralba, A. Oliva. CVPR 2011 Islam Beltagy](https://reader034.fdocuments.us/reader034/viewer/2022042218/5ec4a83df97204754e20a029/html5/thumbnails/18.jpg)
Memorability and object classes
• Most of the object classes are not correlated with memorability, why ? Because:– Not frequent enough to find a correlation
pattern (filter infrequent images)– Very common on all memorability levels– Correlated with medium levels of memorability
![Page 19: What makes an image memorable?cv-fall2012/slides/islam-expt.pdf · 2012-10-26 · What makes an image memorable? P. Isola, J. Xiao, A. Torralba, A. Oliva. CVPR 2011 Islam Beltagy](https://reader034.fdocuments.us/reader034/viewer/2022042218/5ec4a83df97204754e20a029/html5/thumbnails/19.jpg)
wallfloor
personperson sitting
ceilingshelves
chairfluorescent
tubescreen
ceiling lampbag
standbottleseats
boxtabledesk
boxescurtain
bulletin boardsconcecabinet
textbasket
paintingpicturecolumn
poledoor
flowersstaircase
benchpot
signsteps
railingfence
windowcar
sidewalkstreetlight
boatpathsand
searoad
waterground
fieldplant
grassmountain
buildingtreesky
![Page 20: What makes an image memorable?cv-fall2012/slides/islam-expt.pdf · 2012-10-26 · What makes an image memorable? P. Isola, J. Xiao, A. Torralba, A. Oliva. CVPR 2011 Islam Beltagy](https://reader034.fdocuments.us/reader034/viewer/2022042218/5ec4a83df97204754e20a029/html5/thumbnails/20.jpg)
wallfloor
personperson sitting
ceilingshelves
chairfluorescent
tubescreen
ceiling lampbag
standbottleseats
boxtabledesk
boxescurtain
bulletin boardsconcecabinet
textbasket
paintingpicturecolumn
poledoor
flowersstaircase
benchpot
signsteps
railingfence
windowcar
sidewalkstreetlight
boatpathsand
searoad
waterground
fieldplant
grassmountain
buildingtreesky
![Page 21: What makes an image memorable?cv-fall2012/slides/islam-expt.pdf · 2012-10-26 · What makes an image memorable? P. Isola, J. Xiao, A. Torralba, A. Oliva. CVPR 2011 Islam Beltagy](https://reader034.fdocuments.us/reader034/viewer/2022042218/5ec4a83df97204754e20a029/html5/thumbnails/21.jpg)
wallfloor
personperson sitting
ceilingshelves
chairfluorescent
tubescreen
ceiling lampbag
standbottleseats
boxtabledesk
boxescurtain
bulletin boardsconcecabinet
textbasket
paintingpicturecolumn
poledoor
flowersstaircase
benchpot
signsteps
railingfence
windowcar
sidewalkstreetlight
boatpathsand
searoad
waterground
fieldplant
grassmountain
buildingtreesky
![Page 22: What makes an image memorable?cv-fall2012/slides/islam-expt.pdf · 2012-10-26 · What makes an image memorable? P. Isola, J. Xiao, A. Torralba, A. Oliva. CVPR 2011 Islam Beltagy](https://reader034.fdocuments.us/reader034/viewer/2022042218/5ec4a83df97204754e20a029/html5/thumbnails/22.jpg)
Agenda
• Memorability vs Object class
• Memorabiliy vs Saliency
![Page 23: What makes an image memorable?cv-fall2012/slides/islam-expt.pdf · 2012-10-26 · What makes an image memorable? P. Isola, J. Xiao, A. Torralba, A. Oliva. CVPR 2011 Islam Beltagy](https://reader034.fdocuments.us/reader034/viewer/2022042218/5ec4a83df97204754e20a029/html5/thumbnails/23.jpg)
Saliency vs Memorability
• Experiment Design– Run Saliency algorithm (T. Liu et al. CVPR
2007) on the memorability dataset. – Find correlation between area of salient
region and memorability score– Find correlation between location of Salient
block and memorability score– Study object classes statistics on salient
regions
![Page 24: What makes an image memorable?cv-fall2012/slides/islam-expt.pdf · 2012-10-26 · What makes an image memorable? P. Isola, J. Xiao, A. Torralba, A. Oliva. CVPR 2011 Islam Beltagy](https://reader034.fdocuments.us/reader034/viewer/2022042218/5ec4a83df97204754e20a029/html5/thumbnails/24.jpg)
Saliency vs Memorability
• Corr (memorability score, salient region area) = -3.01%
• Corr (memorability score, salient region location) = 4.87%
• Why ?
![Page 25: What makes an image memorable?cv-fall2012/slides/islam-expt.pdf · 2012-10-26 · What makes an image memorable? P. Isola, J. Xiao, A. Torralba, A. Oliva. CVPR 2011 Islam Beltagy](https://reader034.fdocuments.us/reader034/viewer/2022042218/5ec4a83df97204754e20a029/html5/thumbnails/25.jpg)
Saliency vs Memorability
High memorability score
![Page 26: What makes an image memorable?cv-fall2012/slides/islam-expt.pdf · 2012-10-26 · What makes an image memorable? P. Isola, J. Xiao, A. Torralba, A. Oliva. CVPR 2011 Islam Beltagy](https://reader034.fdocuments.us/reader034/viewer/2022042218/5ec4a83df97204754e20a029/html5/thumbnails/26.jpg)
Saliency vs Memorability
Low memorability score
![Page 27: What makes an image memorable?cv-fall2012/slides/islam-expt.pdf · 2012-10-26 · What makes an image memorable? P. Isola, J. Xiao, A. Torralba, A. Oliva. CVPR 2011 Islam Beltagy](https://reader034.fdocuments.us/reader034/viewer/2022042218/5ec4a83df97204754e20a029/html5/thumbnails/27.jpg)
Saliency and Object classes
• Compare object class statistics (Presence, Area) for :– Full picture– Salient region– Non-Salient region
![Page 28: What makes an image memorable?cv-fall2012/slides/islam-expt.pdf · 2012-10-26 · What makes an image memorable? P. Isola, J. Xiao, A. Torralba, A. Oliva. CVPR 2011 Islam Beltagy](https://reader034.fdocuments.us/reader034/viewer/2022042218/5ec4a83df97204754e20a029/html5/thumbnails/28.jpg)
Filtered by frequency and sorted by correlation
![Page 29: What makes an image memorable?cv-fall2012/slides/islam-expt.pdf · 2012-10-26 · What makes an image memorable? P. Isola, J. Xiao, A. Torralba, A. Oliva. CVPR 2011 Islam Beltagy](https://reader034.fdocuments.us/reader034/viewer/2022042218/5ec4a83df97204754e20a029/html5/thumbnails/29.jpg)
Filtered by frequency and sorted by correlation
![Page 30: What makes an image memorable?cv-fall2012/slides/islam-expt.pdf · 2012-10-26 · What makes an image memorable? P. Isola, J. Xiao, A. Torralba, A. Oliva. CVPR 2011 Islam Beltagy](https://reader034.fdocuments.us/reader034/viewer/2022042218/5ec4a83df97204754e20a029/html5/thumbnails/30.jpg)
![Page 31: What makes an image memorable?cv-fall2012/slides/islam-expt.pdf · 2012-10-26 · What makes an image memorable? P. Isola, J. Xiao, A. Torralba, A. Oliva. CVPR 2011 Islam Beltagy](https://reader034.fdocuments.us/reader034/viewer/2022042218/5ec4a83df97204754e20a029/html5/thumbnails/31.jpg)
Person, Person setting
![Page 32: What makes an image memorable?cv-fall2012/slides/islam-expt.pdf · 2012-10-26 · What makes an image memorable? P. Isola, J. Xiao, A. Torralba, A. Oliva. CVPR 2011 Islam Beltagy](https://reader034.fdocuments.us/reader034/viewer/2022042218/5ec4a83df97204754e20a029/html5/thumbnails/32.jpg)
Window
![Page 33: What makes an image memorable?cv-fall2012/slides/islam-expt.pdf · 2012-10-26 · What makes an image memorable? P. Isola, J. Xiao, A. Torralba, A. Oliva. CVPR 2011 Islam Beltagy](https://reader034.fdocuments.us/reader034/viewer/2022042218/5ec4a83df97204754e20a029/html5/thumbnails/33.jpg)
Conclusion
• Found a list of object classes sorted by their relation to memorability
• Memorability is not directly related to saliency
• Saliency is related to specific list of object classes
![Page 34: What makes an image memorable?cv-fall2012/slides/islam-expt.pdf · 2012-10-26 · What makes an image memorable? P. Isola, J. Xiao, A. Torralba, A. Oliva. CVPR 2011 Islam Beltagy](https://reader034.fdocuments.us/reader034/viewer/2022042218/5ec4a83df97204754e20a029/html5/thumbnails/34.jpg)
References
• What Makes an Image Memorable? P. Isola, J. Xiao, A. Torralba, A. Oliva. CVPR 2011.
• Learning to Detect a Salient Object. T. Liu et al. CVPR 2007
• Matlab functions (http://www.mathworks.com/matlabcentral/fileexchange/) : lscatter, herrorbar, rotateticklabel
![Page 35: What makes an image memorable?cv-fall2012/slides/islam-expt.pdf · 2012-10-26 · What makes an image memorable? P. Isola, J. Xiao, A. Torralba, A. Oliva. CVPR 2011 Islam Beltagy](https://reader034.fdocuments.us/reader034/viewer/2022042218/5ec4a83df97204754e20a029/html5/thumbnails/35.jpg)
Thank You