Measuring and Modeling Shared Visual Attention Jeffrey · PDF fileMeasuring and Modeling...

29
Measuring and Modeling Shared Visual Attention Jeffrey B. Mulligan NASA Ames Research Center Workshop on Computational and Mathematical Models in Vision May 2016 MODVIS 2016 file:///Users/jmulliga/working/public_html/home/presentations/modvis16/modvis16_title.htm 1 of 1 5/20/16, 6:31 AM https://ntrs.nasa.gov/search.jsp?R=20160008099 2018-05-04T14:47:06+00:00Z

Transcript of Measuring and Modeling Shared Visual Attention Jeffrey · PDF fileMeasuring and Modeling...

Page 1: Measuring and Modeling Shared Visual Attention Jeffrey · PDF fileMeasuring and Modeling Shared Visual Attention Jeffrey B. Mulligan NASA Ames Research Center Workshop on Computational

 

Measuring and Modeling Shared Visual Attention

Jeffrey B. Mulligan

NASA Ames Research Center

 

Workshop on Computational and Mathematical Models in VisionMay 2016

 

MODVIS 2016 file:///Users/jmulliga/working/public_html/home/presentations/modvis16/modvis16_title.htm

1 of 1 5/20/16, 6:31 AM

https://ntrs.nasa.gov/search.jsp?R=20160008099 2018-05-04T14:47:06+00:00Z

Page 2: Measuring and Modeling Shared Visual Attention Jeffrey · PDF fileMeasuring and Modeling Shared Visual Attention Jeffrey B. Mulligan NASA Ames Research Center Workshop on Computational

Patrick GontarInstitute of Ergonomics, TUMunchen

     

 My collaborator     

My collaborator file:///Users/jmulliga/working/public_html/home/presentations/modvis16/patrick.htm

1 of 1 5/20/16, 6:31 AM

Page 3: Measuring and Modeling Shared Visual Attention Jeffrey · PDF fileMeasuring and Modeling Shared Visual Attention Jeffrey B. Mulligan NASA Ames Research Center Workshop on Computational

Simulated approach toNYC/JFK with multipleproblems

Crews rated by instructor(s)

Objective measures: fuelconsumption, timespent/remaining, etc.

High-level goals: - determine what styles ofcrew coordination lead to successfuloutcomes - use results to inform

     

 Patrick's experiment     

Patrick's experiment file:///Users/jmulliga/working/public_html/home/presentations/modvis16/pg_exp2.htm

1 of 2 5/20/16, 6:31 AM

Page 4: Measuring and Modeling Shared Visual Attention Jeffrey · PDF fileMeasuring and Modeling Shared Visual Attention Jeffrey B. Mulligan NASA Ames Research Center Workshop on Computational

training   Haslbeck, A., Gontar, P. & Schubert, E. (2014). How Can Procedures and Checklists Help Pilots in Abnormal Flight Situations?In N. A. Stanton, S. J. Landry, G. Di Bucchianico & A. Vallicelli (Eds.), Advances in Human Aspects of Transportation. Part II(S. 456-461). AHFE International.

Patrick's experiment file:///Users/jmulliga/working/public_html/home/presentations/modvis16/pg_exp2.htm

2 of 2 5/20/16, 6:31 AM

Page 5: Measuring and Modeling Shared Visual Attention Jeffrey · PDF fileMeasuring and Modeling Shared Visual Attention Jeffrey B. Mulligan NASA Ames Research Center Workshop on Computational

 

    

 Eye gaze as a proxy for attention    

Eye gaze as a proxy for attention file:///Users/jmulliga/working/public_html/home/presentations/modvis16/orig_data1.htm

1 of 1 5/20/16, 6:32 AM

Page 6: Measuring and Modeling Shared Visual Attention Jeffrey · PDF fileMeasuring and Modeling Shared Visual Attention Jeffrey B. Mulligan NASA Ames Research Center Workshop on Computational

Data sorted by "UseCase" (phase of flight) and "Area"

     

 Format of the "raw" data     

Format of the raw data file:///Users/jmulliga/working/public_html/home/presentations/modvis16/raw_data1.htm

1 of 1 5/20/16, 6:32 AM

Page 7: Measuring and Modeling Shared Visual Attention Jeffrey · PDF fileMeasuring and Modeling Shared Visual Attention Jeffrey B. Mulligan NASA Ames Research Center Workshop on Computational

    

 Converting the data back to a time series    

Converting the data back to a time series file:///Users/jmulliga/working/public_html/home/presentations/modvis16/proc_data1.htm

1 of 1 5/20/16, 6:32 AM

Page 8: Measuring and Modeling Shared Visual Attention Jeffrey · PDF fileMeasuring and Modeling Shared Visual Attention Jeffrey B. Mulligan NASA Ames Research Center Workshop on Computational

82 seconds

     

 A graphical representation     

A graphical representation file:///Users/jmulliga/working/public_html/home/presentations/modvis16/proc_data3.htm

1 of 1 5/20/16, 6:33 AM

Page 9: Measuring and Modeling Shared Visual Attention Jeffrey · PDF fileMeasuring and Modeling Shared Visual Attention Jeffrey B. Mulligan NASA Ames Research Center Workshop on Computational

21.8 minutes

     

 A graphical representation     

A graphical representation file:///Users/jmulliga/working/public_html/home/presentations/modvis16/proc_data7.htm

1 of 1 5/20/16, 6:33 AM

Page 10: Measuring and Modeling Shared Visual Attention Jeffrey · PDF fileMeasuring and Modeling Shared Visual Attention Jeffrey B. Mulligan NASA Ames Research Center Workshop on Computational

     

Chi-square approach (thanks to A.Ahumada)

        

Null hypothesis: observations drawn from same distribution   

Choose time window   

Convert from time series to fixation counts

(using average fixation duration)

Chi-square approach file:///Users/jmulliga/working/public_html/home/presentations/modvis16/chisq1.htm

1 of 1 5/20/16, 6:33 AM

Page 11: Measuring and Modeling Shared Visual Attention Jeffrey · PDF fileMeasuring and Modeling Shared Visual Attention Jeffrey B. Mulligan NASA Ames Research Center Workshop on Computational

     

 Observed fixation durations     

Observed fixation durations file:///Users/jmulliga/working/public_html/home/presentations/modvis16/chisq2.htm

1 of 1 5/20/16, 6:34 AM

Page 12: Measuring and Modeling Shared Visual Attention Jeffrey · PDF fileMeasuring and Modeling Shared Visual Attention Jeffrey B. Mulligan NASA Ames Research Center Workshop on Computational

     

 A 21 second example     

A 21 second example file:///Users/jmulliga/working/public_html/home/presentations/modvis16/chisq3.htm

1 of 1 5/20/16, 6:34 AM

Page 13: Measuring and Modeling Shared Visual Attention Jeffrey · PDF fileMeasuring and Modeling Shared Visual Attention Jeffrey B. Mulligan NASA Ames Research Center Workshop on Computational

Probability of next state depends only on current state

    

 A Markov model of fixation sequences    

A Markov model of fixation sequences file:///Users/jmulliga/working/public_html/home/presentations/modvis16/markov2.htm

1 of 1 5/20/16, 6:34 AM

Page 14: Measuring and Modeling Shared Visual Attention Jeffrey · PDF fileMeasuring and Modeling Shared Visual Attention Jeffrey B. Mulligan NASA Ames Research Center Workshop on Computational

     

 Example Markov models     

Example Markov models file:///Users/jmulliga/working/public_html/home/presentations/modvis16/mat_example8.htm

1 of 1 5/20/16, 6:35 AM

Page 15: Measuring and Modeling Shared Visual Attention Jeffrey · PDF fileMeasuring and Modeling Shared Visual Attention Jeffrey B. Mulligan NASA Ames Research Center Workshop on Computational

     

Synthetic data experiment details        

10 randomly generated activities   

2 sample records for each activity   

Compare statistics of matched/unmatched pairs

Synthetic data experiment details file:///Users/jmulliga/working/public_html/home/presentations/modvis16/synth_exp1b.htm

1 of 1 5/20/16, 6:35 AM

Page 16: Measuring and Modeling Shared Visual Attention Jeffrey · PDF fileMeasuring and Modeling Shared Visual Attention Jeffrey B. Mulligan NASA Ames Research Center Workshop on Computational

1.4sec

25sec

426sec

    

 A scale-space image of Chi-squared p    

A scale-space image of Chi-squared p file:///Users/jmulliga/working/public_html/home/presentations/modvis16/scale_space_si...

1 of 2 5/20/16, 6:35 AM

Page 17: Measuring and Modeling Shared Visual Attention Jeffrey · PDF fileMeasuring and Modeling Shared Visual Attention Jeffrey B. Mulligan NASA Ames Research Center Workshop on Computational

1.4sec

25sec

426sec

    

 A scale-space image of Chi-squared p    

A scale-space image of Chi-squared p file:///Users/jmulliga/working/public_html/home/presentations/modvis16/scale_space_si...

1 of 2 5/20/16, 6:35 AM

Page 18: Measuring and Modeling Shared Visual Attention Jeffrey · PDF fileMeasuring and Modeling Shared Visual Attention Jeffrey B. Mulligan NASA Ames Research Center Workshop on Computational

     

 Histograms of p-values     

Histograms of p-values file:///Users/jmulliga/working/public_html/home/presentations/modvis16/hist_example1.htm

1 of 1 5/20/16, 6:36 AM

Page 19: Measuring and Modeling Shared Visual Attention Jeffrey · PDF fileMeasuring and Modeling Shared Visual Attention Jeffrey B. Mulligan NASA Ames Research Center Workshop on Computational

    

 Cumulative distributions of p-values    

Cumulative distributions of p-values file:///Users/jmulliga/working/public_html/home/presentations/modvis16/cum_example1.htm

1 of 1 5/20/16, 6:36 AM

Page 20: Measuring and Modeling Shared Visual Attention Jeffrey · PDF fileMeasuring and Modeling Shared Visual Attention Jeffrey B. Mulligan NASA Ames Research Center Workshop on Computational

     

  An ROC curve     

An ROC curve for discriminating matched from mismatched activities file:///Users/jmulliga/working/public_html/home/presentations/modvis16/roc_example1.htm

1 of 1 5/20/16, 6:36 AM

Page 21: Measuring and Modeling Shared Visual Attention Jeffrey · PDF fileMeasuring and Modeling Shared Visual Attention Jeffrey B. Mulligan NASA Ames Research Center Workshop on Computational

     

  AUC vs. scale     

AUC vs. scale file:///Users/jmulliga/working/public_html/home/presentations/modvis16/single_auc_plot.htm

1 of 1 5/20/16, 6:36 AM

Page 22: Measuring and Modeling Shared Visual Attention Jeffrey · PDF fileMeasuring and Modeling Shared Visual Attention Jeffrey B. Mulligan NASA Ames Research Center Workshop on Computational

     

 Between-activity correlations     

Between-activity correlations file:///Users/jmulliga/working/public_html/home/presentations/modvis16/asymp_dist.htm

1 of 1 5/20/16, 6:37 AM

Page 23: Measuring and Modeling Shared Visual Attention Jeffrey · PDF fileMeasuring and Modeling Shared Visual Attention Jeffrey B. Mulligan NASA Ames Research Center Workshop on Computational

     

  AUC vs. scale     

AUC vs. scale file:///Users/jmulliga/working/public_html/home/presentations/modvis16/activity_auc_pl...

1 of 1 5/20/16, 6:37 AM

Page 24: Measuring and Modeling Shared Visual Attention Jeffrey · PDF fileMeasuring and Modeling Shared Visual Attention Jeffrey B. Mulligan NASA Ames Research Center Workshop on Computational

     

  AUC vs. scale     

AUC vs. scale file:///Users/jmulliga/working/public_html/home/presentations/modvis16/activity_auc_pl...

1 of 1 5/20/16, 6:37 AM

Page 25: Measuring and Modeling Shared Visual Attention Jeffrey · PDF fileMeasuring and Modeling Shared Visual Attention Jeffrey B. Mulligan NASA Ames Research Center Workshop on Computational

     

  Human data     

Human data file:///Users/jmulliga/working/public_html/home/presentations/modvis16/human_auc_plo...

1 of 1 5/20/16, 6:37 AM

Page 26: Measuring and Modeling Shared Visual Attention Jeffrey · PDF fileMeasuring and Modeling Shared Visual Attention Jeffrey B. Mulligan NASA Ames Research Center Workshop on Computational

     

 Average of all crews     

Average of all crews file:///Users/jmulliga/working/public_html/home/presentations/modvis16/human_auc_plo...

1 of 1 5/20/16, 6:38 AM

Page 27: Measuring and Modeling Shared Visual Attention Jeffrey · PDF fileMeasuring and Modeling Shared Visual Attention Jeffrey B. Mulligan NASA Ames Research Center Workshop on Computational

     

What to make of the human data?        

Very low p values at long time windows   

Suggests pilot and first officer average behaviors differ   

Suggests simple Markov model is a poor description

What to make of the human data? file:///Users/jmulliga/working/public_html/home/presentations/modvis16/human_puzzle1.htm

1 of 1 5/20/16, 6:38 AM

Page 28: Measuring and Modeling Shared Visual Attention Jeffrey · PDF fileMeasuring and Modeling Shared Visual Attention Jeffrey B. Mulligan NASA Ames Research Center Workshop on Computational

     

  AUC vs. scale     

AUC vs. scale file:///Users/jmulliga/working/public_html/home/presentations/modvis16/activity_auc_pl...

1 of 1 5/20/16, 6:38 AM

Page 29: Measuring and Modeling Shared Visual Attention Jeffrey · PDF fileMeasuring and Modeling Shared Visual Attention Jeffrey B. Mulligan NASA Ames Research Center Workshop on Computational

Chi-square statistics can be used to compare scan paths 

Results suggest that shared attention in the cockpit

is limited to intervals < 1 minute 

Human activities in the cockpit are poorly described

by random Markov processes 

Real activities are likely to involve small numbers of AOIs 

THANK YOU!

     

 Conclusions     

Conclusions file:///Users/jmulliga/working/public_html/home/presentations/modvis16/modvis16_conc...

1 of 1 5/20/16, 6:39 AM


Jeffrey Walker

Jeffrey Walker

NightWalkers Written by Jeffrey Pedersen Story by Jeffrey ... · NightWalkers Written by Jeffrey Pedersen Story by Jeffrey Pedersen Peter West agooga@gmail.com (818)448-3751 WGA#

NightWalkers Written by Jeffrey Pedersen Story by Jeffrey ... · NightWalkers Written by Jeffrey Pedersen Story by Jeffrey Pedersen Peter West [email protected] (818)448-3751 WGA#

A Shared Multi-Attention Framework for Multi-Label Zero-Shot ...khoury.neu.edu/home/eelhami/publications/MultiAtt-MLZSL...Dat Huynh Northeastern University huynh.dat@husky.neu.edu

A Shared Multi-Attention Framework for Multi-Label Zero-Shot ...khoury.neu.edu/home/eelhami/publications/MultiAtt-MLZSL...Dat Huynh Northeastern University [email protected]

A Shared Multi-Attention Framework for Multi-Label Zero-Shot ......A Shared Multi-Attention Framework for Multi-Label Zero-Shot Learning Dat Huynh Northeastern University huynh.dat@husky.neu.edu

A Shared Multi-Attention Framework for Multi-Label Zero-Shot ......A Shared Multi-Attention Framework for Multi-Label Zero-Shot Learning Dat Huynh Northeastern University [email protected]

Chapter 16: Origins of shared attention in human infants · 2016. 12. 16. · 331 Chapter 16: Origins of shared attention in human infants Gedeon O. Deák and Jochen Triesch From

Chapter 16: Origins of shared attention in human infants · 2016. 12. 16. · 331 Chapter 16: Origins of shared attention in human infants Gedeon O. Deák and Jochen Triesch From

Cognitive Prediction of Reading, Math, and Attention · eling the shared and unique cognitive risk factors that pre-dict reading, math, and attention using data from a large community-based

Cognitive Prediction of Reading, Math, and Attention · eling the shared and unique cognitive risk factors that pre-dict reading, math, and attention using data from a large community-based

Attention Deficit Disorder (ADD) / Attention Deficit ... · Attention Deficit Disorder (ADD) / Attention ... Attention Deficit Hyperactivity Disorder ... Attention Deficit Disorder

Attention Deficit Disorder (ADD) / Attention Deficit ... · Attention Deficit Disorder (ADD) / Attention ... Attention Deficit Hyperactivity Disorder ... Attention Deficit Disorder

Maximising efficiency through shared services · Maximising efficiency . through shared services. ... our attention is very much on the external ... finalising the environment and

Maximising efficiency through shared services · Maximising efficiency . through shared services. ... our attention is very much on the external ... finalising the environment and

Investigating shared attention with a virtual agent using ...

Investigating shared attention with a virtual agent using ...

Jeffrey & Bryan

Jeffrey & Bryan

Jeffrey McClellan

Jeffrey McClellan

Jeffrey Donovan

Jeffrey Donovan

Towards Shared Attention through Geometric Reasoning for ... · International Conference on Humanoid Robots, Dec 2009, Paris, France. hal-01976928 Towards Shared Attention through

Towards Shared Attention through Geometric Reasoning for ... · International Conference on Humanoid Robots, Dec 2009, Paris, France. hal-01976928 Towards Shared Attention through

SHARED CARE GUIDELINE...For review September 2022 1 SHARED CARE GUIDELINE Drugs: Methylphenidate, Lisdexamfetamine, Dexamfetamine, Atomoxetine For Attention Deficit Hyperactivity Disorder

SHARED CARE GUIDELINE...For review September 2022 1 SHARED CARE GUIDELINE Drugs: Methylphenidate, Lisdexamfetamine, Dexamfetamine, Atomoxetine For Attention Deficit Hyperactivity Disorder

AI: The internet of brains · – Deep decoding: Emotional states – Media attention, shared attention in twitter – Social patterns: Sensible DTU / Copenhagen network study “Oticon

AI: The internet of brains · – Deep decoding: Emotional states – Media attention, shared attention in twitter – Social patterns: Sensible DTU / Copenhagen network study “Oticon

Jeffrey and Jeffrey v. Desmond, 1st Cir. (1995)

Jeffrey and Jeffrey v. Desmond, 1st Cir. (1995)

Jeffrey Brooker

Jeffrey Brooker

Shared Care Guideline for Methylphenidate, Atomoxetine ...€¦ · Indications Attention Deficit Hyperactivity Disorder (ADHD) in children and adolescents aged 5 to 18 years. The

Shared Care Guideline for Methylphenidate, Atomoxetine ...€¦ · Indications Attention Deficit Hyperactivity Disorder (ADHD) in children and adolescents aged 5 to 18 years. The

Jeffrey campos

Jeffrey campos

Jeffrey Dahmer

Jeffrey Dahmer