EgoMon Gaze and Video Dataset for Visual Saliency Prediction
-
Upload
xavier-giro -
Category
Technology
-
view
425 -
download
0
Transcript of EgoMon Gaze and Video Dataset for Visual Saliency Prediction
Mònica Chertó Sarret, “EgoMon Gaze and Video Dataset for Visual Saliency Prediction”. ESEIAAT 11/07/2016.
EgoMon Gaze and Video Dataset for Visual Saliency
Prediction
Mònica Chertó Sarret Supervised by: Cathal Gurrin and Xavier Giró
Mònica Chertó Sarret, “EgoMon Gaze and Video Dataset for Visual Saliency Prediction”. ESEIAAT 11/07/2016.
Outline
1. Introduction2. State of the art3. EgoMon Gaze & Video Dataset4. Visual Saliency Prediction5. Conclusions and Future Works
2
Mònica Chertó Sarret, “EgoMon Gaze and Video Dataset for Visual Saliency Prediction”. ESEIAAT 11/07/2016.
1. Introduction
3
Mònica Chertó Sarret, “EgoMon Gaze and Video Dataset for Visual Saliency Prediction”. ESEIAAT 11/07/2016.
Introduction. Main goals and project planning
4
Goals February March April May June
Construct the Dataset
Run state of the art saliency estimator with a single image
Frames extraction
Run saliency estimator with the extracted frames
Compare Results
Mònica Chertó Sarret, “EgoMon Gaze and Video Dataset for Visual Saliency Prediction”. ESEIAAT 11/07/2016.
Equipment and Software. Eye tracker, Tobii Glasses
5
Mònica Chertó Sarret, “EgoMon Gaze and Video Dataset for Visual Saliency Prediction”. ESEIAAT 11/07/2016.
Equipment and Software. Tobii studio Software
6
Mònica Chertó Sarret, “EgoMon Gaze and Video Dataset for Visual Saliency Prediction”. ESEIAAT 11/07/2016.
Equipment and Software.
7
Mònica Chertó Sarret, “EgoMon Gaze and Video Dataset for Visual Saliency Prediction”. ESEIAAT 11/07/2016.
Equipment and Software.
8
Mònica Chertó Sarret, “EgoMon Gaze and Video Dataset for Visual Saliency Prediction”. ESEIAAT 11/07/2016.
Publication
9
Repositori of Egocentric-saliency in GitHub [online] Available: https://github.com/imatge-upc/egocentric-saliencyEgoMon Dataset [online] Available: https://imatge.upc.edu/web/sites/default/files/resources/1720/saliency/2016-egomon/
Mònica Chertó Sarret, “EgoMon Gaze and Video Dataset for Visual Saliency Prediction”. ESEIAAT 11/07/2016.
Outline
1. Introduction2. State of the art3. EgoMon Gaze & Video Dataset4. Visual Saliency Prediction5. Conclusions and Future Works
10
Mònica Chertó Sarret, “EgoMon Gaze and Video Dataset for Visual Saliency Prediction”. ESEIAAT 11/07/2016.
2. State of the art
11
GTEA Dataset UT Ego Dataset
GTEA (Georgia Tech Egocentric Activities) – Gaze Dataset [online] Available: http://ai.stanford.edu/~alireza/GTEA_Gaze_Website/UT (University of Texas) Ego Dataset [online] Available: http://vision.cs.utexas.edu/projects/egocentric_data/UT_Egocentric_Dataset.html
Mònica Chertó Sarret, “EgoMon Gaze and Video Dataset for Visual Saliency Prediction”. ESEIAAT 11/07/2016.
Outline
1. Introduction2. State of the art3. EgoMon Gaze & Video Dataset4. Visual Saliency Prediction5. Conclusions and Future Works
12
Mònica Chertó Sarret, “EgoMon Gaze and Video Dataset for Visual Saliency Prediction”. ESEIAAT 11/07/2016.
Acquisition. Calibration process of the Tobii Glasses
13
Video tutorial uploaded on YouTube.
Mònica Chertó Sarret, “EgoMon Gaze and Video Dataset for Visual Saliency Prediction”. ESEIAAT 11/07/2016.
Acquisition. Results of the calibration process of the Tobii Glasses
14
Mònica Chertó Sarret, “EgoMon Gaze and Video Dataset for Visual Saliency Prediction”. ESEIAAT 11/07/2016.
EgoMon Gaze & Video Dataset
15
...
7 x text files (gaze data)
7 x RAW (videos)
7 x Gaze (videos with the gaze information plotted)
13428 x frames extracted
75 x narrative images
Mònica Chertó Sarret, “EgoMon Gaze and Video Dataset for Visual Saliency Prediction”. ESEIAAT 11/07/2016.
EgoMon Gaze & Video Dataset
16
Mònica Chertó Sarret, “EgoMon Gaze and Video Dataset for Visual Saliency Prediction”. ESEIAAT 11/07/2016.
EgoMon Gaze & Video Dataset
17
INDOOR OUTDOOR
Mònica Chertó Sarret, “EgoMon Gaze and Video Dataset for Visual Saliency Prediction”. ESEIAAT 11/07/2016.
EgoMon. Oral Presentation
18
Mònica Chertó Sarret, “EgoMon Gaze and Video Dataset for Visual Saliency Prediction”. ESEIAAT 11/07/2016.
EgoMon. DCU and Albert College Park
19
Mònica Chertó Sarret, “EgoMon Gaze and Video Dataset for Visual Saliency Prediction”. ESEIAAT 11/07/2016.
EgoMon. Spanish Omelette
20
Mònica Chertó Sarret, “EgoMon Gaze and Video Dataset for Visual Saliency Prediction”. ESEIAAT 11/07/2016.
EgoMon. Playing cards
21
Mònica Chertó Sarret, “EgoMon Gaze and Video Dataset for Visual Saliency Prediction”. ESEIAAT 11/07/2016.
EgoMon. Botanic Gardens
22
Mònica Chertó Sarret, “EgoMon Gaze and Video Dataset for Visual Saliency Prediction”. ESEIAAT 11/07/2016.
EgoMon. Botanic Gardens (Narrative Clip)
23
Mònica Chertó Sarret, “EgoMon Gaze and Video Dataset for Visual Saliency Prediction”. ESEIAAT 11/07/2016.
EgoMon. Bus Ride
24
Mònica Chertó Sarret, “EgoMon Gaze and Video Dataset for Visual Saliency Prediction”. ESEIAAT 11/07/2016.
EgoMon. Walking to the Office
25
Mònica Chertó Sarret, “EgoMon Gaze and Video Dataset for Visual Saliency Prediction”. ESEIAAT 11/07/2016.
EgoMon. Privacy
26
Mònica Chertó Sarret, “EgoMon Gaze and Video Dataset for Visual Saliency Prediction”. ESEIAAT 11/07/2016.
EgoMon. Problems with the Gaze (Losses)
27
static
non-static
Mònica Chertó Sarret, “EgoMon Gaze and Video Dataset for Visual Saliency Prediction”. ESEIAAT 11/07/2016.
EgoMon. Processing, Eye Gaze data
28
Mònica Chertó Sarret, “EgoMon Gaze and Video Dataset for Visual Saliency Prediction”. ESEIAAT 11/07/2016.
EgoMon. Frame extraction
29
DURATION FRAMES EXTRACTED
TOTAL 3:43:41 13428
AVERAGE: 0:34:30 1918
1 fps
Mònica Chertó Sarret, “EgoMon Gaze and Video Dataset for Visual Saliency Prediction”. ESEIAAT 11/07/2016.
Outline
1. Introduction2. State of the art3. EgoMon Gaze & Video Dataset4. Visual Saliency Prediction5. Conclusions and Future Works
30
Mònica Chertó Sarret, “EgoMon Gaze and Video Dataset for Visual Saliency Prediction”. ESEIAAT 11/07/2016.
4. Visual Saliency Predictor.
31
Mònica Chertó Sarret, “EgoMon Gaze and Video Dataset for Visual Saliency Prediction”. ESEIAAT 11/07/2016.
Saliency Predictor. SalNet
32
Mònica Chertó Sarret, “EgoMon Gaze and Video Dataset for Visual Saliency Prediction”. ESEIAAT 11/07/2016.
EgoMon Gaze & Video Dataset
33
...
7 x text files (gaze data)
7 x RAW (videos)
7 x Gaze (videos with the gaze information plotted)
13428 x frames extracted
75 x narrative images
...13428 x saliency models
Mònica Chertó Sarret, “EgoMon Gaze and Video Dataset for Visual Saliency Prediction”. ESEIAAT 11/07/2016.
Results of the Dataset
34
Mònica Chertó Sarret, “EgoMon Gaze and Video Dataset for Visual Saliency Prediction”. ESEIAAT 11/07/2016.
Quantitative Evaluation. Comparison Metric
35
Location-based Distribution-based
AUC-Judd, sAUC, NSS SIM, CC, EMD, KL
NORMALIZED SCANPATH SALIENCY
MIT Saliency Benchmark [online] Available: http://saliency.mit.edu/results_mit300.html
Mònica Chertó Sarret, “EgoMon Gaze and Video Dataset for Visual Saliency Prediction”. ESEIAAT 11/07/2016.
Results. Quantitative Evaluation
36
Mònica Chertó Sarret, “EgoMon Gaze and Video Dataset for Visual Saliency Prediction”. ESEIAAT 11/07/2016.
Results. Qualitative Evaluation
37
Example of GOOD results
Mònica Chertó Sarret, “EgoMon Gaze and Video Dataset for Visual Saliency Prediction”. ESEIAAT 11/07/2016.
Results. Qualitative Evaluation
38
Example of BAD results
Mònica Chertó Sarret, “EgoMon Gaze and Video Dataset for Visual Saliency Prediction”. ESEIAAT 11/07/2016.
Outline
1. Introduction2. State of the art3. EgoMon Gaze & Video Dataset4. Visual Saliency Prediction5. Conclusions and Future Works
39
Mònica Chertó Sarret, “EgoMon Gaze and Video Dataset for Visual Saliency Prediction”. ESEIAAT 11/07/2016. 40
ConclusionsDataset Amount of Data Recorded
DeviceEnvironment Number of
participants
GTEA 17 sequences Tobii eye-tracker Glasses
Indoor 14
UT Ego 4 videos of 4 hours (16 h)
Looxcie wearable camera
Indoor + Outdoor 4
EgoMon 7 clean videos (4 h)7 gaze videos13428 extracted frames13428 saliency maps7 files with eye gaze data75 Narrative images
Tobii eye tracker glasses + Narrative Cip
Indoor + Outdoor 3
Mònica Chertó Sarret, “EgoMon Gaze and Video Dataset for Visual Saliency Prediction”. ESEIAAT 11/07/2016.
Future Works
Fine-tuning of saliency estimator based on the comparison metric
41
Mònica Chertó Sarret, “EgoMon Gaze and Video Dataset for Visual Saliency Prediction”. ESEIAAT 11/07/2016.
Publication
42
http://imatge-upc.github.io/egocentric-2016-saliency/
Mònica Chertó Sarret, “EgoMon Gaze and Video Dataset for Visual Saliency Prediction”. ESEIAAT 11/07/2016. 43