PupilWare Petra 2015

48
eric c. larson | eclarson.com PupilWare Assistant Professor Computer Science and Engineering towards pervasive cognitive load measurement using commodity devices

Transcript of PupilWare Petra 2015

Page 1: PupilWare Petra 2015

eric c. larson | eclarson.com

PupilWare

Assistant Professor Computer Science and Engineering

towards pervasive cognitive load measurement using commodity devices

Page 2: PupilWare Petra 2015

PupilWare

Suku NairSohail Rafiqi Chatchai Wangwiwattana

Ephrem Fernandez

Jasmine Kim

Team:

Page 3: PupilWare Petra 2015

sclerairispupil

pupil: regulate light

Pupillometry Primer

Page 4: PupilWare Petra 2015

Pupillometry Primer

macro changes: light reflex drug impairment concussions

Page 5: PupilWare Petra 2015

sclerairispupil

pupil: regulate lightautonomic nervous system physiological response to stress, arousal

Pupillometry Primer

Page 6: PupilWare Petra 2015

Subtle Pupillary Response

1940 1950 1960 1970 1980 1990 2000 2010 2015

1943 Blood alcohol proportional to pupil size

1947 First scientific studies of light reflex

1950 Pupil size decreases with age

1959 Pupil size projector invented, by Hess and Polt

1960 Pupil size and visual stimuli

1961-1969 Indicator of preference, fatigue

1966 Pupil size & memory

1967 Pupil size & processing difficulty

1968 Pupil size maintained during thought

1983 First digital device for capturing pupil size

1989 Apparatus for desktop imaging of pupil

1995 First handheld digital pupillometer

2005-2012 Validation of eye trackers for pupil measurements

2008 Framework for measuring digital interruption cost

Page 7: PupilWare Petra 2015

1940 1950 1960 1970 1980 1990 2000 2010 2015

1943 Blood alcohol proportional to pupil size

1947 First scientific studies of light reflex

1950 Pupil size decreases with age

1959 Pupil size projector invented, by Hess and Polt

1960 Pupil size and visual stimuli

1961-1969 Indicator of preference, fatigue

1966 Pupil size & memory

1967 Pupil size & processing difficulty

1968 Pupil size maintained during thought

1983 First digital device for capturing pupil size

1989 Apparatus for desktop imaging of pupil

1995 First handheld digital pupillometer

2005-2012 Validation of eye trackers for pupil measurements

2008 Framework for measuring digital interruption cost

Skoglund, 1943 Thompson 1947 & Delauney 1949 Birren, 1950

Page 8: PupilWare Petra 2015

1940 1950 1960 1970 1980 1990 2000 2010 2015

1943 Blood alcohol proportional to pupil size

1947 First scientific studies of light reflex

1950 Pupil size decreases with age

1959 Pupil size projector invented, by Hess and Polt

1960 Pupil size and visual stimuli

1961-1969 Indicator of preference, fatigue

1966 Pupil size & memory

1967 Pupil size & processing difficulty

1968 Pupil size maintained during thought

1983 First digital device for capturing pupil size

1989 Apparatus for desktop imaging of pupil

1995 First handheld digital pupillometer

2005-2012 Validation of eye trackers for pupil measurements

2008 Framework for measuring digital interruption cost

Hess and Polk, 1959-1960

12 mm

4 mm

4.2 mm

Page 9: PupilWare Petra 2015

1940 1950 1960 1970 1980 1990 2000 2010 2015

1943 Blood alcohol proportional to pupil size

1947 First scientific studies of light reflex

1950 Pupil size decreases with age

1959 Pupil size projector invented, by Hess and Polt

1960 Pupil size and visual stimuli

1961-1969 Indicator of preference, fatigue

1966 Pupil size & memory

1967 Pupil size & processing difficulty

1968 Pupil size maintained during thought

1983 First digital device for capturing pupil size

1989 Apparatus for desktop imaging of pupil

1995 First handheld digital pupillometer

2005-2012 Validation of eye trackers for pupil measurements

2008 Framework for measuring digital interruption cost

Hess and Polk, 1959-1960 Hess and Polk, 1960

Page 10: PupilWare Petra 2015

1940 1950 1960 1970 1980 1990 2000 2010 2015

1943 Blood alcohol proportional to pupil size

1947 First scientific studies of light reflex

1950 Pupil size decreases with age

1959 Pupil size projector invented, by Hess and Polt

1960 Pupil size and visual stimuli

1961-1969 Indicator of preference, fatigue

1966 Pupil size & memory

1967 Pupil size & processing difficulty

1968 Pupil size maintained during thought

1983 First digital device for capturing pupil size

1989 Apparatus for desktop imaging of pupil

1995 First handheld digital pupillometer

2005-2012 Validation of eye trackers for pupil measurements

2008 Framework for measuring digital interruption cost

• schizophrenia and neurotic diagnosis (Rubin 1964) • sexual orientation (Hess 1965) • political preference (Hess 1965) • pleasant taste (Hess 1965) • infant object recognition and preferential looking

(Fitzgerald et al.1967) • musical pitch interpretation (Kahneman and Beatty 1967) • fatigue, alertness, and sleep deprivation (Lowenstein and

Loewenfeld 1964, Bartlett et al. 1967)

Consensus: -pupil dilation indicated something pleasing -not a strict measure of emotion -related but not proportional to arousal

Page 11: PupilWare Petra 2015

1940 1950 1960 1970 1980 1990 2000 2010 2015

1943 Blood alcohol proportional to pupil size

1947 First scientific studies of light reflex

1950 Pupil size decreases with age

1959 Pupil size projector invented, by Hess and Polt

1960 Pupil size and visual stimuli

1961-1969 Indicator of preference, fatigue

1966 Pupil size & memory

1967 Pupil size & processing difficulty

1968 Pupil size maintained during thought

1983 First digital device for capturing pupil size

1989 Apparatus for desktop imaging of pupil

1995 First handheld digital pupillometer

2005-2012 Validation of eye trackers for pupil measurements

2008 Framework for measuring digital interruption cost

Kahneman and Beatty, 1966 Hess and Polk, 1967 Bradshaw, 1968

Eventually the processing of the brain and use of working memory came to be

understood as cognitive load

Page 12: PupilWare Petra 2015

1940 1950 1960 1970 1980 1990 2000 2010 2015

1943 Blood alcohol proportional to pupil size

1947 First scientific studies of light reflex

1950 Pupil size decreases with age

1959 Pupil size projector invented, by Hess and Polt

1960 Pupil size and visual stimuli

1961-1969 Indicator of preference, fatigue

1966 Pupil size & memory

1967 Pupil size & processing difficulty

1968 Pupil size maintained during thought

1983 First digital device for capturing pupil size

1989 Apparatus for desktop imaging of pupil

1995 First handheld digital pupillometer

2005-2012 Validation of eye trackers for pupil measurements

2008 Framework for measuring digital interruption cost

Kahneman and Beatty, 1966 Hess and Polk, 1967 Bradshaw, 1968

Eventually the processing of the brain and use of working memory came to be

understood as cognitive load klingner, 2012

Digit Span Task

Page 13: PupilWare Petra 2015

1940 1950 1960 1970 1980 1990 2000 2010 2015

1943 Blood alcohol proportional to pupil size

1947 First scientific studies of light reflex

1950 Pupil size decreases with age

1959 Pupil size projector invented, by Hess and Polt

1960 Pupil size and visual stimuli

1961-1969 Indicator of preference, fatigue

1966 Pupil size & memory

1967 Pupil size & processing difficulty

1968 Pupil size maintained during thought

1983 First digital device for capturing pupil size

1989 Apparatus for desktop imaging of pupil

1995 First handheld digital pupillometer

2005-2012 Validation of eye trackers for pupil measurements

2008 Framework for measuring digital interruption cost

Page 14: PupilWare Petra 2015

1940 1950 1960 1970 1980 1990 2000 2010 2015

1943 Blood alcohol proportional to pupil size

1947 First scientific studies of light reflex

1950 Pupil size decreases with age

1959 Pupil size projector invented, by Hess and Polt

1960 Pupil size and visual stimuli

1961-1969 Indicator of preference, fatigue

1966 Pupil size & memory

1967 Pupil size & processing difficulty

1968 Pupil size maintained during thought

1983 First digital device for capturing pupil size

1989 Apparatus for desktop imaging of pupil

1995 First handheld digital pupillometer

2005-2012 Validation of eye trackers for pupil measurements

2008 Framework for measuring digital interruption cost

Jones and Smith, 1983 Carter, 1989

Page 15: PupilWare Petra 2015

1940 1950 1960 1970 1980 1990 2000 2010 2015

1943 Blood alcohol proportional to pupil size

1947 First scientific studies of light reflex

1950 Pupil size decreases with age

1959 Pupil size projector invented, by Hess and Polt

1960 Pupil size and visual stimuli

1961-1969 Indicator of preference, fatigue

1966 Pupil size & memory

1967 Pupil size & processing difficulty

1968 Pupil size maintained during thought

1983 First digital device for capturing pupil size

1989 Apparatus for desktop imaging of pupil

1995 First handheld digital pupillometer

2005-2012 Validation of eye trackers for pupil measurements

2008 Framework for measuring digital interruption cost

Carter, 1995

$10,000 USD

$4,000 USD

Page 16: PupilWare Petra 2015

1940 1950 1960 1970 1980 1990 2000 2010 2015

1943 Blood alcohol proportional to pupil size

1947 First scientific studies of light reflex

1950 Pupil size decreases with age

1959 Pupil size projector invented, by Hess and Polt

1960 Pupil size and visual stimuli

1961-1969 Indicator of preference, fatigue

1966 Pupil size & memory

1967 Pupil size & processing difficulty

1968 Pupil size maintained during thought

1983 First digital device for capturing pupil size

1989 Apparatus for desktop imaging of pupil

1995 First handheld digital pupillometer

2005-2012 Validation of eye trackers for pupil measurements

2008 Framework for measuring digital interruption cost

Klingner et al., 2005-2012

$500+ USD

Page 17: PupilWare Petra 2015

1940 1950 1960 1970 1980 1990 2000 2010 2015

1943 Blood alcohol proportional to pupil size

1947 First scientific studies of light reflex

1950 Pupil size decreases with age

1959 Pupil size projector invented, by Hess and Polt

1960 Pupil size and visual stimuli

1961-1969 Indicator of preference, fatigue

1966 Pupil size & memory

1967 Pupil size & processing difficulty

1968 Pupil size maintained during thought

1983 First digital device for capturing pupil size

1989 Apparatus for desktop imaging of pupil

1995 First handheld digital pupillometer

2005-2012 Validation of eye trackers for pupil measurements

2008 Framework for measuring digital interruption cost

Iqbal and Bailey, 2008

Use of cognitive load for real time user awareness is

becoming more of a possibility

Page 18: PupilWare Petra 2015

can cognitive load be measured ubiquitously?

Page 19: PupilWare Petra 2015

the webcam

~15 x 15 pixels

dilations are mostly sub-pixel highly affected by noise and lighting

HD 1280x720

Page 20: PupilWare Petra 2015

Study

Can baseline cognitive load be measured, compared to a gold standard pupillometer?

Can subtle variations be measured, compared to a remote eye tracker?

Age: (mean=23, range=19-38) Sex: 7 Male, 5 Female

Color: Lighter Eyes: 7, Brown: 5

Vision: Corrected-to-normal, 2

12 participants

Page 21: PupilWare Petra 2015

Experimenter Laptop Control View

Participant

CameraiPad Application Survey & Self Report

Remote Gaze Tracker

Focal point

Calipers measurement

Pupillometer Measurement

DesignStudy

Page 22: PupilWare Petra 2015

5 Digits 6 Digits 7 Digits 8 Digits 9 Digits

x 5 iterationsover 200 iterations total

2 7 9 3 13

Digit Span Task

Page 23: PupilWare Petra 2015

Algorithm Overview

Page 24: PupilWare Petra 2015

Algorithm

grayscale conversion face detection

1280x720~350x370

Page 25: PupilWare Petra 2015

approximate eye location~125x125

Page 26: PupilWare Petra 2015

darkness threshold morphology

means of gradient

Page 27: PupilWare Petra 2015

iris bounding box histogram equalization

median filtering

Page 28: PupilWare Petra 2015
Page 29: PupilWare Petra 2015

modified starburst algorithm find strong edges

~45x45

Page 30: PupilWare Petra 2015

eliminate measurements with RANSAC fit ellipse to points

calculate distance between pupil centers

Page 31: PupilWare Petra 2015

Algorithm Overview

Page 32: PupilWare Petra 2015

Algorithm Overview

10 FPS

Page 33: PupilWare Petra 2015

Post Processing

0 10 20 30 40 50 60

15

16

14

17

13

seconds

ellip

se d

iamet

er, i

n pi

xels

Page 34: PupilWare Petra 2015

Post Processing

0 10 20 30

15

16

14

13

seconds

ellip

se d

iamet

er, i

n pi

xels

12

average

Page 35: PupilWare Petra 2015

Post Processing

0 10 20 30

1516

1413

seconds

ellip

se d

iamet

er, i

n pi

xels

121110

17

average median smooth

Page 36: PupilWare Petra 2015

Post Processing

0 10 20 30seconds

5.05.2

4.84.6

ellip

se d

iamet

er, i

n m

m

4.44.24.0

5.4average median smooth millimeters

Page 37: PupilWare Petra 2015

average median smooth millimeters

Light Eyes Dark Eyes Reflection

3 participants removed from analysis

Page 38: PupilWare Petra 2015

Baseline Results

3.00

4.00

5.00

6.00

7.00

P1 P2 P3 P4 P5 P6 P7 P8 P9

Pupillometer PupilWare GZ-C

Ŧ = GZ-C and PupilWare different (p<0.05)

Ŧ Ŧ

* = different from pupillometer (p<0.05)

* *

*

*

*

***

* *

base

line

pupi

l size

, mill

imet

ers

Page 39: PupilWare Petra 2015

Digit Span Tasks

Time

20%

10%

0%

-10%

-20% 5s 10s 15s 20s 25s 30s

Gaze Tracker PupilWare

Perc

enta

ge P

upil D

ilatio

n

Baseline Period

Listening Period

Speaking Period

Relaxation Period

mean percent difference < 4% 90th percentile < 9% difference

Page 40: PupilWare Petra 2015

20%

15%

10%

5%

0%

0s 5s 10s 15s 20s 25s 30s 0s 5s 10s 15s 20s 25s 30s

Aggregate Results

Perc

enta

ge P

upil D

ilatio

n

TimeArea = Interquartile Range

GazeTracker PupilWareBaseline

8 digit

5 digit6 digit7 digit

Kahneman and Beatty, 1968

Page 41: PupilWare Petra 2015

Per Trial Correlation1.0

0.8

0.6

0.4

0.2

0.0

-0.2

-0.4P1 P2 P3 P4 P5 P6 P7 P8 P9Rank

Ord

er C

orre

latio

n G

aze

Trac

ker v

s. P

upilW

are

Participant

Page 42: PupilWare Petra 2015

Conclusions and Future Research

+about as accurate as gaze tracking +within 0.5mm of pupillometer

-dark eyes, reflection

-complex stimuli -privacy -off axis eye and head pose -accounting for light reflex

i.e., screen brightness changes

ubiquitous?

Page 43: PupilWare Petra 2015

markers of pain

sympathetic nerve damage head injury

context aware computing

attention

Future Work and Applications

fatigue and sleep deprivation

comprehension and cognitive disability

Page 44: PupilWare Petra 2015

PupilWaretowards pervasive cognitive load measurement

using commodity devices

Thank You!

Page 45: PupilWare Petra 2015

eric c. larson | eclarson.comAssistant Professor Computer Science and Engineering

eclarson.com [email protected] @ec_larson

PupilWare

towards pervasive cognitive load measurement using commodity devices

Suku NairSohail Rafiqi Chatchai Wangwiwattana

Ephrem Fernandez

Jasmine Kim

Page 46: PupilWare Petra 2015
Page 47: PupilWare Petra 2015

Back Up Graphs

Page 48: PupilWare Petra 2015

Back Up Graphs