Sensing Valence and Confusion with Facial EMG Phil Davis and Hsin-Ni Ho.

Sensing Valence and Confusion with Facial EMG

Phil Davis and Hsin-Ni Ho

OutlineI. Experimental SetupII. Valence

A. Problems AddressedB. Experimental ProcedureC. ResultsD. Discussion

III. Confusion A. Problems AddressedB. Experimental ProcedureC. ResultsD. Discussion

IV. Conclusions

Experimental Setup

Electrodes attached to: Corrugator supercilii (“brow”) Zygomaticus major (“cheek”)

Grounded on arm




IV. Conclusions

Problems Addressed Can we sense valence from facial

electromyographic (EMG) activity? Intensity?

Can we sense valence from mouse behavior (pressure in particular)?

In general, what types of activity can we sense with facial EMG?

Relevant Prior Work Recent study by Larsen, Norris, Cacioppo Change in mean “cheek” EMG activity

correlated positively with valence Change in mean “brow” EMG activity

negatively correlated with valence

(CS = “brow” muscle, ZM = “cheek” muscle)




IV. Conclusions

Experimental Procedure

6 subjects General procedure:

1. Subject watches a short film clip2. Subject answers questions about film

content and experienced affective state Subject uses pressure sensitive mouse

EMG output is recorded during all stages

Film Clip Sequence Each film clip was roughly 3 minutes

1. Golf Instruction (Neutral)2. Clip from “The Champ” (Negative)

Sadness in 94.2% (Gross & Levenson)

3. Golf Instruction (Neutral)4. Clip from “Robin Williams Live” (Pos)

Amusement in 84.1% (Gross & Levenson)

Did not vary order




IV. Conclusions

Self-reported Valence Consistent with Expectations Neutral Valence: 4 subjects indicated

“neutral” affect, 2 subjects indicated low intensity positive affect

Negative Valence: All 6 reported sadness

Positive Valence: 5 of 6 reported amusement The sixth subject did not like Robin

Williams

Example of Output

Output normalized due to large scale differences

Mean Output Consistent with Prior Work

Individual Results Varied

Other Observations

For some subjects, brow activity was a good indicator of web form activity




IV. Conclusions

Discussion Highly sensitive to individual subject differences and

electrode positioning Mean output consistent with Larsen, Norris, Cacioppo Do results extend to other positive/negative states? Unable to correlate EMG output with self-reported mood

intensity EMG output may be useful as an activity recognition

sensor Unable to correlate mouse pressure or velocity with

valence Dynamic model of mouse behavior may produce better

results?




IV. Conclusions

OutlineI. Introduction and Experimental SetupII. Valence



IV. Conclusions

Problem Addressed

Does the facial expression ‘frowning’ represent the feeling of confusion?

Can we recognize the feeling of confusion with the EMG outputs?

Does the importance of understanding influence the feeling of confusion?




IV. Conclusions

Experimental Procedure

6 subjects General procedure:

1. Subject listens to an audio clip2. Subject answers questions about audio

content and experienced affective state EMG output is recorded during all

stages

Eliciting Confusion with Audio Recordings

A B

Mood control 40 s classic music

Control spoken with standard American English

Confused by accent spoken with accent

Confused by meaning

A brief paragraph with confusing content spoken with American English

Two levels of the importance of understanding:Low level importance of understanding. High level importance of understanding.

Audio clips to induce different levels of confusion:

Ratings and Measurements for feeling of Confusion

Subjective rating 5 scale Self-report confusion rating

Objective rating Test of understanding

Measurement EMG responses




IV. Conclusions

Result: EMG responses

Brow Cheek

Mean 210.14 2654

Std 214.44 5876

Difference in importance of understanding

NO DF=1, P=0.28

NO DF=1, P=0.24

Result: Average EMG responsesBrow

0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

important not important

No

rmal

ized

EM

G

mood control

1L

1W

2L

2W

3L

3W

For brow

More EMG activity when

filling the web form

listening to clips with accent

EMG in ‘web-filling’ parts increase with level of confusion in high importance of understanding

For cheek

EMG activity increase with degree of confusion ->Subject started to laugh

Cheek

0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

important not important

No

rmal

ized

EM

G

0

1L

1W

2L

2W

3L

3W

Result:Self confusion rating and test score

create the feeling of confusion consistently

Self rating confusion

Test score

Mean 3.81 2.38

Std 1.51 1.87

Difference for 6 clips

YesDF=5, P < 0.0001

YesDF=5, P < 0.0001

difference among subjects

NODF=5, P=0.42

NODF=5, P=0.33

Confusion and Test Score

0.001.002.003.004.005.006.00

1 2 3 4 5 6

Audio section

Rat

ing

and

Scor

e

Confusion

Test

Average plot




IV. Conclusions

Discussion

Huge individual difference in EMG responses -> normalize the data

More Brow activity when filling web forms -> It is more correlated to ‘the process of thinking about confusion thing’

More Brow activity when listening to clip with accent

Cheek activity increase with level of confusion -> So confused that subjects were giving up

Difference in importance of understanding




IV. Conclusions

Conclusions

Baseline difference in EMG for different muscles

Individual difference in EMG response EMG response sensitive to electrode positioning Brow EMG is negatively correlated with valence

and positively correlated with feeling of confusion

Cheek EMG is positively correlated with valence Facial EMG may be useful for activity

recognition

Sensing Valence and Confusion with Facial EMG Phil Davis and Hsin-Ni Ho.

Documents

Transcript of Sensing Valence and Confusion with Facial EMG Phil Davis and Hsin-Ni Ho.