Carnegie Mellon 1. Toward Exploiting EEG Input in a Reading Tutor Jack Mostow, Kai-min Chang, and...
-
date post
19-Dec-2015 -
Category
Documents
-
view
215 -
download
0
Transcript of Carnegie Mellon 1. Toward Exploiting EEG Input in a Reading Tutor Jack Mostow, Kai-min Chang, and...
![Page 1: Carnegie Mellon 1. Toward Exploiting EEG Input in a Reading Tutor Jack Mostow, Kai-min Chang, and Jessica Nelson Project LISTEN listen.](https://reader034.fdocuments.us/reader034/viewer/2022042717/56649d2e5503460f94a06044/html5/thumbnails/1.jpg)
Carnegie Mellon
1
![Page 2: Carnegie Mellon 1. Toward Exploiting EEG Input in a Reading Tutor Jack Mostow, Kai-min Chang, and Jessica Nelson Project LISTEN listen.](https://reader034.fdocuments.us/reader034/viewer/2022042717/56649d2e5503460f94a06044/html5/thumbnails/2.jpg)
Carnegie Mellon
2
Carnegie Mellon Toward Exploiting EEG Input in
a Reading Tutor
Jack Mostow, Kai-min Chang, and Jessica Nelson Project LISTEN
www.cs.cmu.edu/~listenCarnegie Mellon University
This work was supported by the Institute of Education Sciences, U.S. Department of Education, through Grants R305A080157 and R305A080628 to Carnegie Mellon University. The opinions expressed are those of the authors and do not necessarily represent the views of the Institute or U.S. Department of Education.
![Page 3: Carnegie Mellon 1. Toward Exploiting EEG Input in a Reading Tutor Jack Mostow, Kai-min Chang, and Jessica Nelson Project LISTEN listen.](https://reader034.fdocuments.us/reader034/viewer/2022042717/56649d2e5503460f94a06044/html5/thumbnails/3.jpg)
Carnegie Mellon
Motivation: peer into student’s mind• Identify mental states
– Cognitive: effort, recognition, understanding, learning, …– Affective: attention, engagement, frustration, …– Motor: speech, motion, expression, …
• Decide accordingly– What to teach next– How to teach it– …
3
![Page 4: Carnegie Mellon 1. Toward Exploiting EEG Input in a Reading Tutor Jack Mostow, Kai-min Chang, and Jessica Nelson Project LISTEN listen.](https://reader034.fdocuments.us/reader034/viewer/2022042717/56649d2e5503460f94a06044/html5/thumbnails/4.jpg)
Carnegie Mellon
Peering Into Minds: EEG (electroencephalography)
4
![Page 5: Carnegie Mellon 1. Toward Exploiting EEG Input in a Reading Tutor Jack Mostow, Kai-min Chang, and Jessica Nelson Project LISTEN listen.](https://reader034.fdocuments.us/reader034/viewer/2022042717/56649d2e5503460f94a06044/html5/thumbnails/5.jpg)
Carnegie Mellon EEG in the Lab
5
Cons:• Impractical for schools!• Wet electrodes (gel or saline)• Expert application and monitoring• Money for equipment & personnel• Magnetically-shielded room
Pros:• Controlled experimental conditions• Temporally fine-grained data• Reflects widespread brain activity• Detects relevant mental states: attention
(Marosi et al. ’02), engagement (Lutsyuk et al. ’06), frustration (Berka et al. ’06)
![Page 6: Carnegie Mellon 1. Toward Exploiting EEG Input in a Reading Tutor Jack Mostow, Kai-min Chang, and Jessica Nelson Project LISTEN listen.](https://reader034.fdocuments.us/reader034/viewer/2022042717/56649d2e5503460f94a06044/html5/thumbnails/6.jpg)
Carnegie Mellon
New Portable EEG Devices
6
Pros:• Feasible to use in schools• Inexpensive• Dry electrodes• No expert needed• Headphones and microphone
Cons:• 1-2 electrodes; not whole brain• What can such devices detect?
![Page 7: Carnegie Mellon 1. Toward Exploiting EEG Input in a Reading Tutor Jack Mostow, Kai-min Chang, and Jessica Nelson Project LISTEN listen.](https://reader034.fdocuments.us/reader034/viewer/2022042717/56649d2e5503460f94a06044/html5/thumbnails/7.jpg)
Carnegie Mellon
Detect brain states related to learning?
7
1. Can EEG detect when reading is difficult?2. Can EEG detect differences between words?3. Which EEG features detect differences best?
![Page 8: Carnegie Mellon 1. Toward Exploiting EEG Input in a Reading Tutor Jack Mostow, Kai-min Chang, and Jessica Nelson Project LISTEN listen.](https://reader034.fdocuments.us/reader034/viewer/2022042717/56649d2e5503460f94a06044/html5/thumbnails/8.jpg)
Carnegie Mellon
Easy vs Hard (Grade K-1) (GRE level)
8
We need water, land, and air to live. Earth has all these things. Water covers much of Earth. Most of this water is not safe to drink. Many people are running out of fresh drinking water.
In regard to propaganda the early advocates of universal literacy and a free press envisaged only two possibilities: the propaganda might be true, or it might be false. They did not foresee what in fact has happened…
Carnegie Mellon 1. Detect when reading is difficult?
![Page 9: Carnegie Mellon 1. Toward Exploiting EEG Input in a Reading Tutor Jack Mostow, Kai-min Chang, and Jessica Nelson Project LISTEN listen.](https://reader034.fdocuments.us/reader034/viewer/2022042717/56649d2e5503460f94a06044/html5/thumbnails/9.jpg)
Carnegie Mellon 1. Detect when reading is difficult?
• For adults / children– 10 adults in Project LISTEN lab– 11 children ages 9-10, at school
• Reading connected text / isolated words• Aloud / silently
9
![Page 10: Carnegie Mellon 1. Toward Exploiting EEG Input in a Reading Tutor Jack Mostow, Kai-min Chang, and Jessica Nelson Project LISTEN listen.](https://reader034.fdocuments.us/reader034/viewer/2022042717/56649d2e5503460f94a06044/html5/thumbnails/10.jpg)
Carnegie Mellon
Child readerAdult reader
Pilot study setup
10
Carnegie Mellon
Project LISTEN’s Reading Tutor
Reader
![Page 11: Carnegie Mellon 1. Toward Exploiting EEG Input in a Reading Tutor Jack Mostow, Kai-min Chang, and Jessica Nelson Project LISTEN listen.](https://reader034.fdocuments.us/reader034/viewer/2022042717/56649d2e5503460f94a06044/html5/thumbnails/11.jpg)
Carnegie Mellon Collecting EEG and Reading Tutor Data
11
Time Student Text07-12-2011-09:51:00.10 Kai-min We …07-12-2011-09:51:00.20 Kai-min need …07-12-2011-09:51:00.35 Kai-min need …07-12-2011-09:51:00.55 Kai-min water…07-12-2011-09:51:01.01 Kai-min land …07-12-2011-09:51:01.50 Kai-min air …
Reading Tutor LogTime Student Raw Attention07-12-2011-09:51:00.01 Kai-min -33 11 …07-12-2011-09:51:00.02 Kai-min 351 17 …07-12-2011-09:51:00.03 Kai-min 117 20…07-12-2011-09:51:00.04 Kai-min 66 1 14 …07-12-2011-09:51:00.05 Kai-min -451 26…07-12-2011-09:51:00.06 Kai-min -3 43…
MindSet (EEG) LogTime Student Text Raw07-12-2011:09:51:00.10 Kai-min We -33 …07-12-2011:09:51:00.20 Kai-min need 351 …07-12-2011:09:51:00.35 Kai-min need 117 …07-12-2011:09:51:00.55 Kai-min water 661 …07-12-2011:09:51:00.10 Kai-min land -451 …07-12-2011-09:51:01.50 Kai-min air 43 ,,,
Combined Log
![Page 12: Carnegie Mellon 1. Toward Exploiting EEG Input in a Reading Tutor Jack Mostow, Kai-min Chang, and Jessica Nelson Project LISTEN listen.](https://reader034.fdocuments.us/reader034/viewer/2022042717/56649d2e5503460f94a06044/html5/thumbnails/12.jpg)
Carnegie Mellon MindSet (EEG) Features
12
Raw EEG signal, reported at 512 Hz
Filtered EEG signal, 512 Hz
Proprietary “attention” measure, 1 Hz
Proprietary “meditation” measure, 1 Hz
Proprietary signal quality measure, 1Hz
Theta band (4-7 Hz), 8Hz
Alpha band (8-11 Hz), 8Hz
Beta band (12-29 Hz), 8Hz
Gamma band (30-100 Hz), 8Hz
Gamma+ band (101-256 Hz), 8Hz
Delta band (1-3 Hz), 8Hz
![Page 13: Carnegie Mellon 1. Toward Exploiting EEG Input in a Reading Tutor Jack Mostow, Kai-min Chang, and Jessica Nelson Project LISTEN listen.](https://reader034.fdocuments.us/reader034/viewer/2022042717/56649d2e5503460f94a06044/html5/thumbnails/13.jpg)
Carnegie Mellon
13
Machine Learning Approach
• Train classifiers to detect mental states associated with stimuli– f = Binary Logistic Regression Classifier– X = MindSet (EEG) Features (averaged over stimulus interval)– Y = Easy or hard sentences
N
11 F
Y = f( X )N
1
![Page 14: Carnegie Mellon 1. Toward Exploiting EEG Input in a Reading Tutor Jack Mostow, Kai-min Chang, and Jessica Nelson Project LISTEN listen.](https://reader034.fdocuments.us/reader034/viewer/2022042717/56649d2e5503460f94a06044/html5/thumbnails/14.jpg)
Carnegie Mellon
14
Train on: Test on:
Train on: Test on:
Trial 1Trial 2Trial 3Trial 4
Trial 1Trial 2Trial 3Trial 4
Trial 1Trial 2Trial 3Trial 4
Trial 1Trial 2Trial 3Trial 4
Reader-Specific Classifiers
Reader-Independent Classifiers
![Page 15: Carnegie Mellon 1. Toward Exploiting EEG Input in a Reading Tutor Jack Mostow, Kai-min Chang, and Jessica Nelson Project LISTEN listen.](https://reader034.fdocuments.us/reader034/viewer/2022042717/56649d2e5503460f94a06044/html5/thumbnails/15.jpg)
Carnegie Mellon Class Size Imbalance
15
• More easy sentences than hard ones
• Made the two sets equal in size using 3 approaches:1. Random Oversampling (with replacement)
![Page 16: Carnegie Mellon 1. Toward Exploiting EEG Input in a Reading Tutor Jack Mostow, Kai-min Chang, and Jessica Nelson Project LISTEN listen.](https://reader034.fdocuments.us/reader034/viewer/2022042717/56649d2e5503460f94a06044/html5/thumbnails/16.jpg)
Carnegie Mellon Class Size Imbalance
16
• More easy sentences than hard ones
• Made the two sets equal in size using 3 approaches:1. Random Oversampling (with replacement)2. Random Undersampling
![Page 17: Carnegie Mellon 1. Toward Exploiting EEG Input in a Reading Tutor Jack Mostow, Kai-min Chang, and Jessica Nelson Project LISTEN listen.](https://reader034.fdocuments.us/reader034/viewer/2022042717/56649d2e5503460f94a06044/html5/thumbnails/17.jpg)
Carnegie Mellon Class Size Imbalance
17
• More easy sentences than hard ones
• Made the two sets equal in size using 3 approaches:1. Random Oversampling (with replacement)2. Random Undersampling3. Directed Undersampling (Truncating)
![Page 18: Carnegie Mellon 1. Toward Exploiting EEG Input in a Reading Tutor Jack Mostow, Kai-min Chang, and Jessica Nelson Project LISTEN listen.](https://reader034.fdocuments.us/reader034/viewer/2022042717/56649d2e5503460f94a06044/html5/thumbnails/18.jpg)
Carnegie Mellon
Detect when reading is difficult:Reader-specific classifier accuracy
18
oral
sile
nt
both
oral
sile
nt
both
oral
sile
nt
both
adul
t child both
0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
over-sample under-sample truncate
chance
p < .05
![Page 19: Carnegie Mellon 1. Toward Exploiting EEG Input in a Reading Tutor Jack Mostow, Kai-min Chang, and Jessica Nelson Project LISTEN listen.](https://reader034.fdocuments.us/reader034/viewer/2022042717/56649d2e5503460f94a06044/html5/thumbnails/19.jpg)
Carnegie Mellon
oral
sile
nt
both
oral
sile
nt
both
oral
sile
nt
both
adult child both
0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
over-sample under-sample truncate
Detect when reading is difficult:Reader-independent classifier accuracy
19
chance
p < .05
![Page 20: Carnegie Mellon 1. Toward Exploiting EEG Input in a Reading Tutor Jack Mostow, Kai-min Chang, and Jessica Nelson Project LISTEN listen.](https://reader034.fdocuments.us/reader034/viewer/2022042717/56649d2e5503460f94a06044/html5/thumbnails/20.jpg)
Carnegie Mellon
2. Detect differences between words?
20
Easy Words
BedroomChickenStation
Hard Words
CologneChassisBrocade
Non-Words
KOFCUNWAF
Illegal String
FFSGHTNKL
![Page 21: Carnegie Mellon 1. Toward Exploiting EEG Input in a Reading Tutor Jack Mostow, Kai-min Chang, and Jessica Nelson Project LISTEN listen.](https://reader034.fdocuments.us/reader034/viewer/2022042717/56649d2e5503460f94a06044/html5/thumbnails/21.jpg)
Carnegie Mellon
21
Machine Learning Classifiers• Multinomial logistic regression classifiers• Measure classifier’s rank accuracy (Mitchell et al. ’04)
– Use classifier to rank-order possible class labels– Rank accuracy = percentile rank of correct label; 0.5 = chance– More sensitive than % correct
rank accuracy = 0.67
Classifier’s Ranking
1. Non-Word2. Illegal String3. Hard Word4. Easy Word
1.0
0.5
0.0
True label
![Page 22: Carnegie Mellon 1. Toward Exploiting EEG Input in a Reading Tutor Jack Mostow, Kai-min Chang, and Jessica Nelson Project LISTEN listen.](https://reader034.fdocuments.us/reader034/viewer/2022042717/56649d2e5503460f94a06044/html5/thumbnails/22.jpg)
Carnegie Mellon
Detect differences between words: Reader-specific rank accuracy
22
oral
sile
nt
both
oral
sile
nt
both
oral
sile
nt
both
adul
t child both
00.10.20.30.40.50.60.7
over-sample under-sample truncate
chance
p < .05
![Page 23: Carnegie Mellon 1. Toward Exploiting EEG Input in a Reading Tutor Jack Mostow, Kai-min Chang, and Jessica Nelson Project LISTEN listen.](https://reader034.fdocuments.us/reader034/viewer/2022042717/56649d2e5503460f94a06044/html5/thumbnails/23.jpg)
Carnegie Mellon
oral
sile
nt
both
oral
sile
nt
both
oral
sile
nt
both
adul
t child both
0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
over-sample under-sample truncate
Detect differences between words: Reader-independent rank accuracy
23
chance
p < .05
![Page 24: Carnegie Mellon 1. Toward Exploiting EEG Input in a Reading Tutor Jack Mostow, Kai-min Chang, and Jessica Nelson Project LISTEN listen.](https://reader034.fdocuments.us/reader034/viewer/2022042717/56649d2e5503460f94a06044/html5/thumbnails/24.jpg)
Carnegie Mellon
3. Which features detect text difficulty best?
• Train classifier using each feature in isolation• Average accuracy across subjects, CV folds
– Higher = better
DELTATHETA
MED
RAWW
AVE
GAMMA+
FILTERED
ALPHA
BETA
GAMMAATT
0.400.420.440.460.480.500.520.540.56
Reader-specific accuracy
OversampleUndersampleTruncate
Feature
0.400.420.440.460.480.500.520.540.560.58 Reader-independent accuracy
OversampleUndersampleTruncate
Feature
![Page 25: Carnegie Mellon 1. Toward Exploiting EEG Input in a Reading Tutor Jack Mostow, Kai-min Chang, and Jessica Nelson Project LISTEN listen.](https://reader034.fdocuments.us/reader034/viewer/2022042717/56649d2e5503460f94a06044/html5/thumbnails/25.jpg)
Carnegie Mellon
3. Which features detect word differences best?
• Train classifier using each feature in isolation• Average rank accuracy across subjects, CV folds
– Higher = better
BETAALP
HAMED
ATT
GAMMA
FILTERED
RAWW
AVE
THETA
GAMMA+DELT
A0.420.440.460.480.500.520.540.56
Reader-specific rank accuracy of feature
OversampleUndersampleTruncate
Feature
THETA
FILTERED
ATTMED
GAMMA+
GAMMABETA
RAWW
AVE
ALPHA
DELTA
0.42
0.46
0.50
0.54
0.58
Reader-independent rank accuracy of feature
OversampleUndersampleTruncate
Feature
![Page 26: Carnegie Mellon 1. Toward Exploiting EEG Input in a Reading Tutor Jack Mostow, Kai-min Chang, and Jessica Nelson Project LISTEN listen.](https://reader034.fdocuments.us/reader034/viewer/2022042717/56649d2e5503460f94a06044/html5/thumbnails/26.jpg)
Carnegie Mellon Which EEG features are sensitive to
which lexical properties?
26
Delta(1-3 Hz)
Theta(4-7 Hz)
Alpha(8-11 Hz)
Beta(12-29 Hz)
Gamma(30-100 Hz)
Gamma+(101-256 Hz)
ConcretenessImageabilityColorado Meaningfulness -0.08 -0.10Familiarity -0.08Age of acquisition Brown verbal frequency -0.12 -0.13 -0.09 -0.12Kucera and Francis written frequency 0.07 0.10Thorndike-Lorge frequency 0.09Number of letters 0.11 0.13 0.10 0.10 0.14 0.16
• Do EEG spectra reflect natural lexical variance among sentences?– If so, those bands may carry different information for a tutor.
• Correlate with MRC Psycholinguistic word properties (Coltheart 81)– Blank = not statistically significant; bold = passes False Discovery Rate test
![Page 27: Carnegie Mellon 1. Toward Exploiting EEG Input in a Reading Tutor Jack Mostow, Kai-min Chang, and Jessica Nelson Project LISTEN listen.](https://reader034.fdocuments.us/reader034/viewer/2022042717/56649d2e5503460f94a06044/html5/thumbnails/27.jpg)
Carnegie Mellon
4. Conclusions and Future Work
• 1-electrode EEG tells easy from hard better than chance.• Frequency bands tap different properties a tutor may use.• Detect mental states such as attention or frustration• Use longitudinal EEG in schools to:
– Instrument authentic behavior– Label data based on normal tutor use, not artificial experiments – Detect longer-term learning, not just recency effects– Combat EEG noise with “big data”– Inform tutor redesign and make student-specific models
27
![Page 28: Carnegie Mellon 1. Toward Exploiting EEG Input in a Reading Tutor Jack Mostow, Kai-min Chang, and Jessica Nelson Project LISTEN listen.](https://reader034.fdocuments.us/reader034/viewer/2022042717/56649d2e5503460f94a06044/html5/thumbnails/28.jpg)
Carnegie Mellon
28
Questions?
![Page 29: Carnegie Mellon 1. Toward Exploiting EEG Input in a Reading Tutor Jack Mostow, Kai-min Chang, and Jessica Nelson Project LISTEN listen.](https://reader034.fdocuments.us/reader034/viewer/2022042717/56649d2e5503460f94a06044/html5/thumbnails/29.jpg)
Carnegie Mellon
29
![Page 30: Carnegie Mellon 1. Toward Exploiting EEG Input in a Reading Tutor Jack Mostow, Kai-min Chang, and Jessica Nelson Project LISTEN listen.](https://reader034.fdocuments.us/reader034/viewer/2022042717/56649d2e5503460f94a06044/html5/thumbnails/30.jpg)
Carnegie Mellon
3. Which features detect word differences best?
• Train classifier using each feature in isolation• Average rank accuracy across subjects, CV folds
– Higher = better
BETAALP
HAMED
ATT
GAMMA
FILTERED
RAWW
AVE
THETA
GAMMA+DELT
A0.420.440.460.480.500.520.540.56
Reader-specific rank accuracy of feature
OversampleUndersampleTruncate
Feature
![Page 31: Carnegie Mellon 1. Toward Exploiting EEG Input in a Reading Tutor Jack Mostow, Kai-min Chang, and Jessica Nelson Project LISTEN listen.](https://reader034.fdocuments.us/reader034/viewer/2022042717/56649d2e5503460f94a06044/html5/thumbnails/31.jpg)
Carnegie Mellon
Which features predict word differences best?
THETA
FILTERED
ATTM
ED
GAMM
A+
GAMM
ABETA
RAWW
AVE
ALPHA
DELTA0.42
0.46
0.50
0.54
0.58
Reader-independent rank accuracy of feature
OversampleUndersampleTruncate
Feature