Sponsored by US Army Research Office STIR: Advanced Estimation of Cognitive Status

PLS’09 Beijing, China, September 7, 2009

Sponsored by US Army Research Office STIR: Advanced Estimation of Cognitive StatusContract No. W911NF-08-C-0121 15-SEP-2008 TO14-MAR-2009

PLS Tools in ElectroencephalographyLeonard J. Trejo

PDT InstitutePalo Alto, CA 94303, USA

The 6th International Conference on Partial Least Squares and Related Methods

Sept. 4th – 7th, 2009Beijing, China


PDT and PDT Institute• PDT

• Neuroergonomics Models and Applications • Human-System Integration• Human Performance Optimization

• Robust Biomedical Signal Processing• Embedded and Real-time Systems for Bio-Sensing

• PDT Institute• PhD/Masters/Undergraduate Training• University Partners (UC Santa Cruz, Tsinghua

University, UC San Diego, Univ. of West Florida)


When I am not working…


Outline• Problem: stress, workload, fatigue and performance• Response: Neuroergonomic models and control systems

– Create useful definitions of cognitive states– Model, estimate and control cognitive states

• Background– Multimodal sensor-state models using PLS and KPLS Algorithms– Successes and failures: fatigue / BCI / engagement / workload

• New directions– Truly multidimensional sensor-process models– PARAFAC, N-PLS

• Summary


Estimation of Cognitive States

Aroused or Overloaded

Fatigued

RestingEngaged

Working

Other States

Behavior and PerformanceRewardsystem

Executivecontrol

WorkingmemorySensation

& perception

Autonomicsystem

Other Processes

Internal Processes

Biosignals


Useful Definitions

• Engagement: selection of a task as the focus of attention and effort

• Workload: significant commitment of attention and effort to an engaged task

•Visual, Auditory, Haptic•Psychomotor•Cognitive (memory, executive)

• Overload: task demands outstrip performance capacity

• Mental Fatigue: desire to withdraw attention and effort from an engaged task associated with extended performance (~45 min)

Work-load

MentalFatigue

Non-specificFactors

Engage-ment

GeneralCognitive

Status


ElectroencaphalogramCerebral Cortex

• the outermost layers of brain• 2-4 mm thick (human)


EEG Sources

A pyramidal neuron with a soma, apical & basal dendrites and a single axon

http://en.wikipedia.org/wiki/Image:Gray627.png


EEG Sources

A pyramidal neuron with a soma, apical & basal dendrites and a single axon

C:\Program Files\DipoleSimulator\DipoleSimulator.exe


Other Elements of Sensor-State Models

Modality Effect of Workload

Heart rate Increase

Heart rate variability (and HFQRS) Decrease

Vertical and horizontal EOG (eye movements)

Increase

Blinks May decrease for intake

Pupil diameter Increase

Skin conductance, SCR, GSR Increase

EMG (frontalis, temporalis, trapezius) Increase


Successful Application 1: Mental Fatigue Black = Alert Red = Mentally Fatigued

Fz Pz

FrontalTheta

ParietalAlpha


Robust EEG-Based Classification of Mental Fatigue 2300 (Day 1) vs. 1900 Hrs (Day 2)

40

50

60

70

80

90

100

Signal-to-noise Ratio (dB)

Test

Pro

port

ion

Cor

rect

21 Channels 100100999479565012 Channels 989896886551504 Channels 87889088765450

0-3-6-9-12-15-18


Successful Application 2: BCI


Stress, Workload, Fatigue and Performance

Trejo, et al. ACI 2007


Cognitive Overload (Trejo, et al. ACI 2007)

Trejo, et al. ACI 2007Trejo, et al. ACI 2007


Stabilizing Classifiers

Spectral Normalization

EEG Bandwidth Limiter

EEG Spectral Features

Classifiers

2s ECG Epoch

MV EOG/ECG Regression

Filters

R-wave Detector

PSDEMG

FeaturesRMS20,

Burst Duration, Burst

Frequency…

Innovations in EEG Algorithm Stabilization

ECG

FeaturesR-wave detectHR,

HRV, STD-IBI,

…

EOG

FeaturesRMS20, Blinks, EMs ,

…

AVAS Thresholds

Filters

4- 20s ECG Epoch

Gauges


Multimodal Overload Patterns

0 50 100 150 200 250 300 350 400 450

50

100

RT - blue; HRstd - red

0 50 100 150 200 250 300 350 400 4500

5

10

15

Left temporalis EMG - blue; Right temporalis EMG - red

0 50 100 150 200 250 300 350 400 4500

5

10

15

Fz/theta - blue; Pz/alpha - red

0 50 100 150 200 250 300 350 400 4500

100

200

300

vEOG - blue; hEOG - redTime (s)

Val

ue


Workload-related EEG Sources

Passive viewing: theta alpha

Engaged 5: theta alpha

10.55 Hz

10.30 Hz5.79 Hz

5.79 Hz

Anterior Cingulate Inferior Parietal Precuneus


Application Summary• Individual models of engagement and fatigue:

• 90-100% accurate• Stable within a day• Stable from day to day

• Individual models of workload or effort:• 60-90% accurate• Moderately stable within a day• Unstable from day to day

• Normative models (limited data):• 50-70% accurate• Moderately stable


Recommended Directions

1. Deployable multimodal sensors (EEG, fNIR, EOG,

gaze, HRV, EMG, SCR, SpO2, BP, core body

temperature, gesture, posture facial expression, ...)

2. Multimodal experimental designs and operational tests

3. Advanced neurocognitive process models4. Multimodal sensor-process mapping algorithms

APECS


“Atomic” Decomposition “In the parlance of modern harmonic analysis (Chen and

Donoho, 2001), we performed a space/ frequency/time ‘‘atomic decomposition’’ of multidimensional data. In other words, we assume that each neural mass contributes a distinctive atom to the topographic frequency/time description of the EEG, so that the estimation of these atoms is possible by means of signal-processing techniques. Each atom will be defined by its topography, spectral content, and time profile; in other words, by its spatial, spectral, and temporal signatures.”

• Fumikazu Miwakeichi, et al, Decomposing EEG data into space–time–frequency components using Parallel Factor AnalysisNeuroImage 22 (2004) 1035–1045.

• Chen, S., Donoho, D., 2001. Atomic decomposition by basis pursuit. SIAM Rev. 43, 129– 159.


“Atomic” EEG Elements

Atoms

Molecule

Basic Sources“atoms”

CoherentSystems

“molecules”

Coherence BondsCovalent Bonds


“Molecular” EEG Processes

Coherence BondsAtoms


Familiar (bilinear) Mapping Algorithms

Factor Analysis

Principal Component Analysis (PCA)

ijjf

F

fifij ebax

1

F

f 1 af

bf

0ije


Multimodal MappingHow to generalize bilinear models to systems with more dimensions?

1. Unfolding a bilinear modela. Represent all experimental factors in one dimensionb. Observations (trials) is second dimensionc. Contrast each dimension vs. pairs of the other two

2. Multidimensional modela. Assume orthogonal factors: PARAFACb. Allow interacting factors: Tucker 3

3. Modeling approacha. Unsupervised extraction: PARAFAC, CANDECOMP, Tucker 3b. Supervised extraction: N-PLS


Unfolding a Bilinear Model

Unfolding

Dim 1 Dim 2 Dim 3

XX

X1 X2 X3


Multidimensional Modeling (Tucker 3 Model, unsupervised)

ijklmnknjm

F

nil

F

m

F

lijk egcbax

321

111

• xijk is an element of (l x m x n) multidimensional array• F1, F2, F3 are the number of components extracted on the

1st, 2nd and 3rd mode• a, b, c are elements of the A, B, C loadings matrices for the

1st, 2nd and 3rd mode • g are the elements of the core matrix G which defines how

individual loading vectors in different modes interact• eijk is an error element (unexplained variance)


PARAFAC (Parallel Factor Analysis, unsupervised)

ijkkfjf

F

fifijk ecbax

1

F

f 1 af

bf

cf

PARAFAC is a special case of the Tucker 3 model where F1= F2 = F3=F and G = I For a 3-way array:


N-way PLS(supervised)

X

frequency

workload condition

X

F

f 1 af

cfbf

F

f 1 vf

uf

time

time

max. covariance

electr

odes

EEG

Labels

af – spectral atom bf – spatial atom cf – temporal atom

vf – workload atom uf – temporal atom


Demo: Workload / PARAFAC EEG

Workl

oad co

ndtions

(e.g.

, trial

s, time)

Elec

trod

es

EEG Frequency


Summary•Successes and Failures

• Individual models of engagement and fatigue: accurate, stable• Individual models of workload or effort: variably accurate, unstable• Normative models (limited data): inaccurate, unstable

•Useful models of state-related EEG sources• “Atomic” EEG sources• “Molecular” EEG systems

•Approaches to multidimensional models and algorithms• Tradtional bilinear methods (PCA, factor analysis, ICA)• Truly multidimensional methods

•Correlated factors (Tucker 3)•Uncorrleated factors (PARAFAC, CANDECOMP, N-PLS)•Supervised algorithms (N-PLS)


Within-day Results

• Trials 1 and 5: disengaged.

• Trials 2-4, 6-8: engaged increasing numbers of enemies.

• Trials 5-8: engaged and reported status in response to command tones

Trial 2 3 4 5 6 7 81 99.95 100.00 100.00 97.78 100.00 99.89 99.662 87.28 88.94 100.00 74.06 83.56 82.783 68.56 99.95 82.83 88.89 86.164 99.95 82.50 85.22 82.645 100.00 99.84 99.786 76.00 76.087 82.44

Accuracy (% correct)Trial 2 3 4 5 6 7 81 99.95 100.00 100.00 97.78 100.00 99.89 99.662 87.28 88.94 100.00 74.06 83.56 82.783 68.56 99.95 82.83 88.89 86.164 99.95 82.50 85.22 82.645 100.00 99.84 99.786 76.00 76.087 82.44

Accuracy (% correct)

EEG/ERP/ECGEOG/EMG

Engagement and workload algorithms

Real-time alert or advisory signal

Accuracy of only EEG-based classification of engagement or mental workload levels in 18 human subjects performing a first-person shooter simulation.

Sponsored by US Army Research Office STIR: Advanced Estimation of Cognitive Status

Documents

Transcript of Sponsored by US Army Research Office STIR: Advanced Estimation of Cognitive Status