SubvocalizationToward Hearing the Inner Thoughts of Developers

Chris ParninGeorgia Tech

Overview

Subvocal speech

+

EMG (electromyography)

=

Cognition?

Problem

QuestionsDoes a tool or language reduce cognitive effort?

Can we identify moments of learning uncertainty?

Can we distinguish cognitive processes?

Can we find news ways to interact?

Some Attempts of Measuring Cognition

fMRI

12-30 seconds tasks

While crammed in the middle of a jet engine

EEG

History of Reading?

”His heart sought out the meaning, but his voice was silent and his tongue was still”

300s AD

Kymograph

Inner Speech (Brain)

Speech Plans and Acts

fMRI processing inner speech..

BRAIN

Measure with EMG

Sokolov’s EMG Recordings (1970s)

Reasons why People Subvocalize

BoostMultiple modalities

AttentionFocus, maintain

Activationcomprehension, memory

Research Questions

When do developers subvocalize?

Can subvocalization be used to measure difficulty of programming task?

Distinguish visual or verbal cognitive processing?

Tasks

Make block fall all the way down with space key.

Make monster reappear if

missed shot.

100 200 300 400 500 600 700 800 900

Debug

Select

Click

Search

Edit

An instrumented session…

0 100 200 300 400 500 600 700 800 900

Subvocal Debug

Select Click

Search Edit

FormulatingProblem

Found useful code

Formulatingsolution

EditingFound useful code

Evaluating

Moments of interest and difficulty

Difference in tasks at frequency bands

1 5 9 13 17 21 25 29 33 37 41 45 49 53 57 61 65 69 73 77 81 85 89 93 970

10000

20000

30000

40000

50000

60000

Series1microvolts

Frequency bands

One more thing…

Silent Speech Interfaces

?

“Things Programmers Think When

Coding?”

Jorgensen

6 words(8% error)

2003

Schultz

101 words

(10% error)

2010 ICPC 2012

State of the Art

Questions?

0 100 200 300 400 500 600 700 800 900

Detect Hmm?