Post on 07-Aug-2015
eric c. larson | eclarson.com
PupilWare
Assistant Professor Computer Science and Engineering
towards pervasive cognitive load measurement using commodity devices
PupilWare
Suku NairSohail Rafiqi Chatchai Wangwiwattana
Ephrem Fernandez
Jasmine Kim
Team:
sclerairispupil
pupil: regulate light
Pupillometry Primer
Pupillometry Primer
macro changes: light reflex drug impairment concussions
sclerairispupil
pupil: regulate lightautonomic nervous system physiological response to stress, arousal
Pupillometry Primer
Subtle Pupillary Response
1940 1950 1960 1970 1980 1990 2000 2010 2015
1943 Blood alcohol proportional to pupil size
1947 First scientific studies of light reflex
1950 Pupil size decreases with age
1959 Pupil size projector invented, by Hess and Polt
1960 Pupil size and visual stimuli
1961-1969 Indicator of preference, fatigue
1966 Pupil size & memory
1967 Pupil size & processing difficulty
1968 Pupil size maintained during thought
1983 First digital device for capturing pupil size
1989 Apparatus for desktop imaging of pupil
1995 First handheld digital pupillometer
2005-2012 Validation of eye trackers for pupil measurements
2008 Framework for measuring digital interruption cost
1940 1950 1960 1970 1980 1990 2000 2010 2015
1943 Blood alcohol proportional to pupil size
1947 First scientific studies of light reflex
1950 Pupil size decreases with age
1959 Pupil size projector invented, by Hess and Polt
1960 Pupil size and visual stimuli
1961-1969 Indicator of preference, fatigue
1966 Pupil size & memory
1967 Pupil size & processing difficulty
1968 Pupil size maintained during thought
1983 First digital device for capturing pupil size
1989 Apparatus for desktop imaging of pupil
1995 First handheld digital pupillometer
2005-2012 Validation of eye trackers for pupil measurements
2008 Framework for measuring digital interruption cost
Skoglund, 1943 Thompson 1947 & Delauney 1949 Birren, 1950
1940 1950 1960 1970 1980 1990 2000 2010 2015
1943 Blood alcohol proportional to pupil size
1947 First scientific studies of light reflex
1950 Pupil size decreases with age
1959 Pupil size projector invented, by Hess and Polt
1960 Pupil size and visual stimuli
1961-1969 Indicator of preference, fatigue
1966 Pupil size & memory
1967 Pupil size & processing difficulty
1968 Pupil size maintained during thought
1983 First digital device for capturing pupil size
1989 Apparatus for desktop imaging of pupil
1995 First handheld digital pupillometer
2005-2012 Validation of eye trackers for pupil measurements
2008 Framework for measuring digital interruption cost
Hess and Polk, 1959-1960
12 mm
4 mm
4.2 mm
1940 1950 1960 1970 1980 1990 2000 2010 2015
1943 Blood alcohol proportional to pupil size
1947 First scientific studies of light reflex
1950 Pupil size decreases with age
1959 Pupil size projector invented, by Hess and Polt
1960 Pupil size and visual stimuli
1961-1969 Indicator of preference, fatigue
1966 Pupil size & memory
1967 Pupil size & processing difficulty
1968 Pupil size maintained during thought
1983 First digital device for capturing pupil size
1989 Apparatus for desktop imaging of pupil
1995 First handheld digital pupillometer
2005-2012 Validation of eye trackers for pupil measurements
2008 Framework for measuring digital interruption cost
Hess and Polk, 1959-1960 Hess and Polk, 1960
1940 1950 1960 1970 1980 1990 2000 2010 2015
1943 Blood alcohol proportional to pupil size
1947 First scientific studies of light reflex
1950 Pupil size decreases with age
1959 Pupil size projector invented, by Hess and Polt
1960 Pupil size and visual stimuli
1961-1969 Indicator of preference, fatigue
1966 Pupil size & memory
1967 Pupil size & processing difficulty
1968 Pupil size maintained during thought
1983 First digital device for capturing pupil size
1989 Apparatus for desktop imaging of pupil
1995 First handheld digital pupillometer
2005-2012 Validation of eye trackers for pupil measurements
2008 Framework for measuring digital interruption cost
• schizophrenia and neurotic diagnosis (Rubin 1964) • sexual orientation (Hess 1965) • political preference (Hess 1965) • pleasant taste (Hess 1965) • infant object recognition and preferential looking
(Fitzgerald et al.1967) • musical pitch interpretation (Kahneman and Beatty 1967) • fatigue, alertness, and sleep deprivation (Lowenstein and
Loewenfeld 1964, Bartlett et al. 1967)
Consensus: -pupil dilation indicated something pleasing -not a strict measure of emotion -related but not proportional to arousal
1940 1950 1960 1970 1980 1990 2000 2010 2015
1943 Blood alcohol proportional to pupil size
1947 First scientific studies of light reflex
1950 Pupil size decreases with age
1959 Pupil size projector invented, by Hess and Polt
1960 Pupil size and visual stimuli
1961-1969 Indicator of preference, fatigue
1966 Pupil size & memory
1967 Pupil size & processing difficulty
1968 Pupil size maintained during thought
1983 First digital device for capturing pupil size
1989 Apparatus for desktop imaging of pupil
1995 First handheld digital pupillometer
2005-2012 Validation of eye trackers for pupil measurements
2008 Framework for measuring digital interruption cost
Kahneman and Beatty, 1966 Hess and Polk, 1967 Bradshaw, 1968
Eventually the processing of the brain and use of working memory came to be
understood as cognitive load
1940 1950 1960 1970 1980 1990 2000 2010 2015
1943 Blood alcohol proportional to pupil size
1947 First scientific studies of light reflex
1950 Pupil size decreases with age
1959 Pupil size projector invented, by Hess and Polt
1960 Pupil size and visual stimuli
1961-1969 Indicator of preference, fatigue
1966 Pupil size & memory
1967 Pupil size & processing difficulty
1968 Pupil size maintained during thought
1983 First digital device for capturing pupil size
1989 Apparatus for desktop imaging of pupil
1995 First handheld digital pupillometer
2005-2012 Validation of eye trackers for pupil measurements
2008 Framework for measuring digital interruption cost
Kahneman and Beatty, 1966 Hess and Polk, 1967 Bradshaw, 1968
Eventually the processing of the brain and use of working memory came to be
understood as cognitive load klingner, 2012
Digit Span Task
1940 1950 1960 1970 1980 1990 2000 2010 2015
1943 Blood alcohol proportional to pupil size
1947 First scientific studies of light reflex
1950 Pupil size decreases with age
1959 Pupil size projector invented, by Hess and Polt
1960 Pupil size and visual stimuli
1961-1969 Indicator of preference, fatigue
1966 Pupil size & memory
1967 Pupil size & processing difficulty
1968 Pupil size maintained during thought
1983 First digital device for capturing pupil size
1989 Apparatus for desktop imaging of pupil
1995 First handheld digital pupillometer
2005-2012 Validation of eye trackers for pupil measurements
2008 Framework for measuring digital interruption cost
1940 1950 1960 1970 1980 1990 2000 2010 2015
1943 Blood alcohol proportional to pupil size
1947 First scientific studies of light reflex
1950 Pupil size decreases with age
1959 Pupil size projector invented, by Hess and Polt
1960 Pupil size and visual stimuli
1961-1969 Indicator of preference, fatigue
1966 Pupil size & memory
1967 Pupil size & processing difficulty
1968 Pupil size maintained during thought
1983 First digital device for capturing pupil size
1989 Apparatus for desktop imaging of pupil
1995 First handheld digital pupillometer
2005-2012 Validation of eye trackers for pupil measurements
2008 Framework for measuring digital interruption cost
Jones and Smith, 1983 Carter, 1989
1940 1950 1960 1970 1980 1990 2000 2010 2015
1943 Blood alcohol proportional to pupil size
1947 First scientific studies of light reflex
1950 Pupil size decreases with age
1959 Pupil size projector invented, by Hess and Polt
1960 Pupil size and visual stimuli
1961-1969 Indicator of preference, fatigue
1966 Pupil size & memory
1967 Pupil size & processing difficulty
1968 Pupil size maintained during thought
1983 First digital device for capturing pupil size
1989 Apparatus for desktop imaging of pupil
1995 First handheld digital pupillometer
2005-2012 Validation of eye trackers for pupil measurements
2008 Framework for measuring digital interruption cost
Carter, 1995
$10,000 USD
$4,000 USD
1940 1950 1960 1970 1980 1990 2000 2010 2015
1943 Blood alcohol proportional to pupil size
1947 First scientific studies of light reflex
1950 Pupil size decreases with age
1959 Pupil size projector invented, by Hess and Polt
1960 Pupil size and visual stimuli
1961-1969 Indicator of preference, fatigue
1966 Pupil size & memory
1967 Pupil size & processing difficulty
1968 Pupil size maintained during thought
1983 First digital device for capturing pupil size
1989 Apparatus for desktop imaging of pupil
1995 First handheld digital pupillometer
2005-2012 Validation of eye trackers for pupil measurements
2008 Framework for measuring digital interruption cost
Klingner et al., 2005-2012
$500+ USD
1940 1950 1960 1970 1980 1990 2000 2010 2015
1943 Blood alcohol proportional to pupil size
1947 First scientific studies of light reflex
1950 Pupil size decreases with age
1959 Pupil size projector invented, by Hess and Polt
1960 Pupil size and visual stimuli
1961-1969 Indicator of preference, fatigue
1966 Pupil size & memory
1967 Pupil size & processing difficulty
1968 Pupil size maintained during thought
1983 First digital device for capturing pupil size
1989 Apparatus for desktop imaging of pupil
1995 First handheld digital pupillometer
2005-2012 Validation of eye trackers for pupil measurements
2008 Framework for measuring digital interruption cost
Iqbal and Bailey, 2008
Use of cognitive load for real time user awareness is
becoming more of a possibility
can cognitive load be measured ubiquitously?
the webcam
~15 x 15 pixels
dilations are mostly sub-pixel highly affected by noise and lighting
HD 1280x720
Study
Can baseline cognitive load be measured, compared to a gold standard pupillometer?
Can subtle variations be measured, compared to a remote eye tracker?
Age: (mean=23, range=19-38) Sex: 7 Male, 5 Female
Color: Lighter Eyes: 7, Brown: 5
Vision: Corrected-to-normal, 2
12 participants
Experimenter Laptop Control View
Participant
CameraiPad Application Survey & Self Report
Remote Gaze Tracker
Focal point
Calipers measurement
Pupillometer Measurement
DesignStudy
5 Digits 6 Digits 7 Digits 8 Digits 9 Digits
x 5 iterationsover 200 iterations total
2 7 9 3 13
Digit Span Task
Algorithm Overview
Algorithm
grayscale conversion face detection
1280x720~350x370
approximate eye location~125x125
darkness threshold morphology
means of gradient
iris bounding box histogram equalization
median filtering
modified starburst algorithm find strong edges
~45x45
eliminate measurements with RANSAC fit ellipse to points
calculate distance between pupil centers
Algorithm Overview
Algorithm Overview
10 FPS
Post Processing
0 10 20 30 40 50 60
15
16
14
17
13
seconds
ellip
se d
iamet
er, i
n pi
xels
Post Processing
0 10 20 30
15
16
14
13
seconds
ellip
se d
iamet
er, i
n pi
xels
12
average
Post Processing
0 10 20 30
1516
1413
seconds
ellip
se d
iamet
er, i
n pi
xels
121110
17
average median smooth
Post Processing
0 10 20 30seconds
5.05.2
4.84.6
ellip
se d
iamet
er, i
n m
m
4.44.24.0
5.4average median smooth millimeters
average median smooth millimeters
Light Eyes Dark Eyes Reflection
3 participants removed from analysis
Baseline Results
3.00
4.00
5.00
6.00
7.00
P1 P2 P3 P4 P5 P6 P7 P8 P9
Pupillometer PupilWare GZ-C
Ŧ = GZ-C and PupilWare different (p<0.05)
Ŧ Ŧ
* = different from pupillometer (p<0.05)
* *
*
*
*
***
* *
base
line
pupi
l size
, mill
imet
ers
Digit Span Tasks
Time
20%
10%
0%
-10%
-20% 5s 10s 15s 20s 25s 30s
Gaze Tracker PupilWare
Perc
enta
ge P
upil D
ilatio
n
Baseline Period
Listening Period
Speaking Period
Relaxation Period
mean percent difference < 4% 90th percentile < 9% difference
20%
15%
10%
5%
0%
0s 5s 10s 15s 20s 25s 30s 0s 5s 10s 15s 20s 25s 30s
Aggregate Results
Perc
enta
ge P
upil D
ilatio
n
TimeArea = Interquartile Range
GazeTracker PupilWareBaseline
8 digit
5 digit6 digit7 digit
Kahneman and Beatty, 1968
Per Trial Correlation1.0
0.8
0.6
0.4
0.2
0.0
-0.2
-0.4P1 P2 P3 P4 P5 P6 P7 P8 P9Rank
Ord
er C
orre
latio
n G
aze
Trac
ker v
s. P
upilW
are
Participant
Conclusions and Future Research
+about as accurate as gaze tracking +within 0.5mm of pupillometer
-dark eyes, reflection
-complex stimuli -privacy -off axis eye and head pose -accounting for light reflex
i.e., screen brightness changes
ubiquitous?
markers of pain
sympathetic nerve damage head injury
context aware computing
attention
Future Work and Applications
fatigue and sleep deprivation
comprehension and cognitive disability
PupilWaretowards pervasive cognitive load measurement
using commodity devices
Thank You!
eric c. larson | eclarson.comAssistant Professor Computer Science and Engineering
eclarson.com eclarson@smu.edu @ec_larson
PupilWare
towards pervasive cognitive load measurement using commodity devices
Suku NairSohail Rafiqi Chatchai Wangwiwattana
Ephrem Fernandez
Jasmine Kim
Back Up Graphs
Back Up Graphs