Carl Vondrick, Antonio Torralba Adria Recasens*, Aditya...
Transcript of Carl Vondrick, Antonio Torralba Adria Recasens*, Aditya...
![Page 1: Carl Vondrick, Antonio Torralba Adria Recasens*, Aditya ...vision.cs.utexas.edu/381V-spring2016/slides/goel-expt.pdfWhere are they looking? Adria Recasens*, Aditya Khosla*, Carl Vondrick,](https://reader033.fdocuments.us/reader033/viewer/2022052001/6013ab41ed750d623a0c02e6/html5/thumbnails/1.jpg)
Where are they looking?Adria Recasens*, Aditya Khosla*,Carl Vondrick, Antonio Torralba
Presented by: Surbhi Goel
![Page 2: Carl Vondrick, Antonio Torralba Adria Recasens*, Aditya ...vision.cs.utexas.edu/381V-spring2016/slides/goel-expt.pdfWhere are they looking? Adria Recasens*, Aditya Khosla*, Carl Vondrick,](https://reader033.fdocuments.us/reader033/viewer/2022052001/6013ab41ed750d623a0c02e6/html5/thumbnails/2.jpg)
Where are they looking?
Follow the gaze of the person and identify the object being looked at
![Page 4: Carl Vondrick, Antonio Torralba Adria Recasens*, Aditya ...vision.cs.utexas.edu/381V-spring2016/slides/goel-expt.pdfWhere are they looking? Adria Recasens*, Aditya Khosla*, Carl Vondrick,](https://reader033.fdocuments.us/reader033/viewer/2022052001/6013ab41ed750d623a0c02e6/html5/thumbnails/4.jpg)
Experiments
● Dataset Visualizations○ Images in the Dataset○ Head Locations○ Gaze Locations/Length
● Model Experiments○ Qualitative Evaluation○ Visualizing Gaze Mask and Saliency Map○ Animal Gaze Following○ Extending to Short Video
![Page 5: Carl Vondrick, Antonio Torralba Adria Recasens*, Aditya ...vision.cs.utexas.edu/381V-spring2016/slides/goel-expt.pdfWhere are they looking? Adria Recasens*, Aditya Khosla*, Carl Vondrick,](https://reader033.fdocuments.us/reader033/viewer/2022052001/6013ab41ed750d623a0c02e6/html5/thumbnails/5.jpg)
Dataset Visualizations
![Page 6: Carl Vondrick, Antonio Torralba Adria Recasens*, Aditya ...vision.cs.utexas.edu/381V-spring2016/slides/goel-expt.pdfWhere are they looking? Adria Recasens*, Aditya Khosla*, Carl Vondrick,](https://reader033.fdocuments.us/reader033/viewer/2022052001/6013ab41ed750d623a0c02e6/html5/thumbnails/6.jpg)
Training Set Images
![Page 7: Carl Vondrick, Antonio Torralba Adria Recasens*, Aditya ...vision.cs.utexas.edu/381V-spring2016/slides/goel-expt.pdfWhere are they looking? Adria Recasens*, Aditya Khosla*, Carl Vondrick,](https://reader033.fdocuments.us/reader033/viewer/2022052001/6013ab41ed750d623a0c02e6/html5/thumbnails/7.jpg)
Training Set Images
![Page 8: Carl Vondrick, Antonio Torralba Adria Recasens*, Aditya ...vision.cs.utexas.edu/381V-spring2016/slides/goel-expt.pdfWhere are they looking? Adria Recasens*, Aditya Khosla*, Carl Vondrick,](https://reader033.fdocuments.us/reader033/viewer/2022052001/6013ab41ed750d623a0c02e6/html5/thumbnails/8.jpg)
Training Set Images
![Page 9: Carl Vondrick, Antonio Torralba Adria Recasens*, Aditya ...vision.cs.utexas.edu/381V-spring2016/slides/goel-expt.pdfWhere are they looking? Adria Recasens*, Aditya Khosla*, Carl Vondrick,](https://reader033.fdocuments.us/reader033/viewer/2022052001/6013ab41ed750d623a0c02e6/html5/thumbnails/9.jpg)
Heatmaps for Head Location
Train Test
![Page 10: Carl Vondrick, Antonio Torralba Adria Recasens*, Aditya ...vision.cs.utexas.edu/381V-spring2016/slides/goel-expt.pdfWhere are they looking? Adria Recasens*, Aditya Khosla*, Carl Vondrick,](https://reader033.fdocuments.us/reader033/viewer/2022052001/6013ab41ed750d623a0c02e6/html5/thumbnails/10.jpg)
Heatmaps for Gaze Location
Train Test
![Page 11: Carl Vondrick, Antonio Torralba Adria Recasens*, Aditya ...vision.cs.utexas.edu/381V-spring2016/slides/goel-expt.pdfWhere are they looking? Adria Recasens*, Aditya Khosla*, Carl Vondrick,](https://reader033.fdocuments.us/reader033/viewer/2022052001/6013ab41ed750d623a0c02e6/html5/thumbnails/11.jpg)
Heatmaps for Relative Gaze Location
Train Test
![Page 12: Carl Vondrick, Antonio Torralba Adria Recasens*, Aditya ...vision.cs.utexas.edu/381V-spring2016/slides/goel-expt.pdfWhere are they looking? Adria Recasens*, Aditya Khosla*, Carl Vondrick,](https://reader033.fdocuments.us/reader033/viewer/2022052001/6013ab41ed750d623a0c02e6/html5/thumbnails/12.jpg)
Histogram for Length of Gaze
Train Test
![Page 13: Carl Vondrick, Antonio Torralba Adria Recasens*, Aditya ...vision.cs.utexas.edu/381V-spring2016/slides/goel-expt.pdfWhere are they looking? Adria Recasens*, Aditya Khosla*, Carl Vondrick,](https://reader033.fdocuments.us/reader033/viewer/2022052001/6013ab41ed750d623a0c02e6/html5/thumbnails/13.jpg)
Observations
● Head/Gaze are concentrated for train and scattered for test
● Relative gaze is concentrated for both
● Gaze length relatively short (0.2 peak)
![Page 14: Carl Vondrick, Antonio Torralba Adria Recasens*, Aditya ...vision.cs.utexas.edu/381V-spring2016/slides/goel-expt.pdfWhere are they looking? Adria Recasens*, Aditya Khosla*, Carl Vondrick,](https://reader033.fdocuments.us/reader033/viewer/2022052001/6013ab41ed750d623a0c02e6/html5/thumbnails/14.jpg)
Model Evaluation
![Page 15: Carl Vondrick, Antonio Torralba Adria Recasens*, Aditya ...vision.cs.utexas.edu/381V-spring2016/slides/goel-expt.pdfWhere are they looking? Adria Recasens*, Aditya Khosla*, Carl Vondrick,](https://reader033.fdocuments.us/reader033/viewer/2022052001/6013ab41ed750d623a0c02e6/html5/thumbnails/15.jpg)
Good Cases
![Page 16: Carl Vondrick, Antonio Torralba Adria Recasens*, Aditya ...vision.cs.utexas.edu/381V-spring2016/slides/goel-expt.pdfWhere are they looking? Adria Recasens*, Aditya Khosla*, Carl Vondrick,](https://reader033.fdocuments.us/reader033/viewer/2022052001/6013ab41ed750d623a0c02e6/html5/thumbnails/16.jpg)
Good Cases
![Page 17: Carl Vondrick, Antonio Torralba Adria Recasens*, Aditya ...vision.cs.utexas.edu/381V-spring2016/slides/goel-expt.pdfWhere are they looking? Adria Recasens*, Aditya Khosla*, Carl Vondrick,](https://reader033.fdocuments.us/reader033/viewer/2022052001/6013ab41ed750d623a0c02e6/html5/thumbnails/17.jpg)
Bad Cases
Head fully tilted but missed
![Page 18: Carl Vondrick, Antonio Torralba Adria Recasens*, Aditya ...vision.cs.utexas.edu/381V-spring2016/slides/goel-expt.pdfWhere are they looking? Adria Recasens*, Aditya Khosla*, Carl Vondrick,](https://reader033.fdocuments.us/reader033/viewer/2022052001/6013ab41ed750d623a0c02e6/html5/thumbnails/18.jpg)
Bad Cases
Face forward but eyes tiltedNo object of attention
![Page 19: Carl Vondrick, Antonio Torralba Adria Recasens*, Aditya ...vision.cs.utexas.edu/381V-spring2016/slides/goel-expt.pdfWhere are they looking? Adria Recasens*, Aditya Khosla*, Carl Vondrick,](https://reader033.fdocuments.us/reader033/viewer/2022052001/6013ab41ed750d623a0c02e6/html5/thumbnails/19.jpg)
Bad Cases
Back facing
![Page 20: Carl Vondrick, Antonio Torralba Adria Recasens*, Aditya ...vision.cs.utexas.edu/381V-spring2016/slides/goel-expt.pdfWhere are they looking? Adria Recasens*, Aditya Khosla*, Carl Vondrick,](https://reader033.fdocuments.us/reader033/viewer/2022052001/6013ab41ed750d623a0c02e6/html5/thumbnails/20.jpg)
Observations
● Handle groups well
● Gaze location is very accurate, head location often not
● Unable to capture eye movement independent of face orientation
● Fails at a lot of back facing cases
![Page 21: Carl Vondrick, Antonio Torralba Adria Recasens*, Aditya ...vision.cs.utexas.edu/381V-spring2016/slides/goel-expt.pdfWhere are they looking? Adria Recasens*, Aditya Khosla*, Carl Vondrick,](https://reader033.fdocuments.us/reader033/viewer/2022052001/6013ab41ed750d623a0c02e6/html5/thumbnails/21.jpg)
Gaze Mask and Saliency Map
![Page 22: Carl Vondrick, Antonio Torralba Adria Recasens*, Aditya ...vision.cs.utexas.edu/381V-spring2016/slides/goel-expt.pdfWhere are they looking? Adria Recasens*, Aditya Khosla*, Carl Vondrick,](https://reader033.fdocuments.us/reader033/viewer/2022052001/6013ab41ed750d623a0c02e6/html5/thumbnails/22.jpg)
Gaze Mask and Saliency Map
● Gaze Mask incorporates the general direction of gaze
● Saliency Map incorporates the salient objects in image
● Element-wise product captures locations that satisfy both
![Page 23: Carl Vondrick, Antonio Torralba Adria Recasens*, Aditya ...vision.cs.utexas.edu/381V-spring2016/slides/goel-expt.pdfWhere are they looking? Adria Recasens*, Aditya Khosla*, Carl Vondrick,](https://reader033.fdocuments.us/reader033/viewer/2022052001/6013ab41ed750d623a0c02e6/html5/thumbnails/23.jpg)
Gaze Mask and Saliency Map
Image with Gaze Gaze Mask Saliency Map
![Page 24: Carl Vondrick, Antonio Torralba Adria Recasens*, Aditya ...vision.cs.utexas.edu/381V-spring2016/slides/goel-expt.pdfWhere are they looking? Adria Recasens*, Aditya Khosla*, Carl Vondrick,](https://reader033.fdocuments.us/reader033/viewer/2022052001/6013ab41ed750d623a0c02e6/html5/thumbnails/24.jpg)
Animal Gaze Follow
![Page 25: Carl Vondrick, Antonio Torralba Adria Recasens*, Aditya ...vision.cs.utexas.edu/381V-spring2016/slides/goel-expt.pdfWhere are they looking? Adria Recasens*, Aditya Khosla*, Carl Vondrick,](https://reader033.fdocuments.us/reader033/viewer/2022052001/6013ab41ed750d623a0c02e6/html5/thumbnails/25.jpg)
Animal Gaze Follow
![Page 26: Carl Vondrick, Antonio Torralba Adria Recasens*, Aditya ...vision.cs.utexas.edu/381V-spring2016/slides/goel-expt.pdfWhere are they looking? Adria Recasens*, Aditya Khosla*, Carl Vondrick,](https://reader033.fdocuments.us/reader033/viewer/2022052001/6013ab41ed750d623a0c02e6/html5/thumbnails/26.jpg)
Animal Gaze Follow
Works (almost) for even birds
![Page 27: Carl Vondrick, Antonio Torralba Adria Recasens*, Aditya ...vision.cs.utexas.edu/381V-spring2016/slides/goel-expt.pdfWhere are they looking? Adria Recasens*, Aditya Khosla*, Carl Vondrick,](https://reader033.fdocuments.us/reader033/viewer/2022052001/6013ab41ed750d623a0c02e6/html5/thumbnails/27.jpg)
Animal Gaze Follow
Works even when more than one salient object
![Page 28: Carl Vondrick, Antonio Torralba Adria Recasens*, Aditya ...vision.cs.utexas.edu/381V-spring2016/slides/goel-expt.pdfWhere are they looking? Adria Recasens*, Aditya Khosla*, Carl Vondrick,](https://reader033.fdocuments.us/reader033/viewer/2022052001/6013ab41ed750d623a0c02e6/html5/thumbnails/28.jpg)
Animal Gaze Follow
● Model generalizes to animals○ Initialized with ImageNet which has animal data
● Able to learn properties based on orientation of head
● Point of gaze is not always correct
![Page 29: Carl Vondrick, Antonio Torralba Adria Recasens*, Aditya ...vision.cs.utexas.edu/381V-spring2016/slides/goel-expt.pdfWhere are they looking? Adria Recasens*, Aditya Khosla*, Carl Vondrick,](https://reader033.fdocuments.us/reader033/viewer/2022052001/6013ab41ed750d623a0c02e6/html5/thumbnails/29.jpg)
Extension to a Short Video
Apply model per frame of video
![Page 30: Carl Vondrick, Antonio Torralba Adria Recasens*, Aditya ...vision.cs.utexas.edu/381V-spring2016/slides/goel-expt.pdfWhere are they looking? Adria Recasens*, Aditya Khosla*, Carl Vondrick,](https://reader033.fdocuments.us/reader033/viewer/2022052001/6013ab41ed750d623a0c02e6/html5/thumbnails/30.jpg)
Extension to a Short Video
Head detector often fails, could use temporal context to improve
![Page 31: Carl Vondrick, Antonio Torralba Adria Recasens*, Aditya ...vision.cs.utexas.edu/381V-spring2016/slides/goel-expt.pdfWhere are they looking? Adria Recasens*, Aditya Khosla*, Carl Vondrick,](https://reader033.fdocuments.us/reader033/viewer/2022052001/6013ab41ed750d623a0c02e6/html5/thumbnails/31.jpg)
Conclusions
● Can be confused with mixed orientations and back-facing
● Model generalizes well to animals
● Could be potentially extended to videos
● Could be applied to other domains?
![Page 32: Carl Vondrick, Antonio Torralba Adria Recasens*, Aditya ...vision.cs.utexas.edu/381V-spring2016/slides/goel-expt.pdfWhere are they looking? Adria Recasens*, Aditya Khosla*, Carl Vondrick,](https://reader033.fdocuments.us/reader033/viewer/2022052001/6013ab41ed750d623a0c02e6/html5/thumbnails/32.jpg)
Thank You!