Spoken Language Support for Software Development

1

Spoken Language Support for Software

Development

Andrew BegelAdvisor: Susan L. GrahamComputer Science Division, EECSUniversity of California, Berkeley

2

Motivation

• Programmers conventionally use keyboard– Long hours at keyboard leads to higher risk of

RSI• Can a programmer code using speech?• Can a computer understand what the

developer says?

while (counter < limit) { }

3

Programming by Voice

• My Goal1. Find out how developers use code verbally. Use this to

develop a naturally verbalizable input form.2. Build development environment that supports verbal

authoring, navigation, modification.• Extend conventional compiler analyses to support ambiguities

generated by speech.3. Learn how developers can use voice-based

programming, and iterate design.

while counter is lessthan limit do ...

4

Challenges

Speech is inherently ambiguous.Programming tools were not designed for

ambiguity.Speech tools are poorly suited for

programming tasks.Programmers are not used to verbal software

development.

5

Talk Outline

• Introduction and Motivation Programming by Voice• Program Analyses for Ambiguous Inputs• SPEech EDitor Programming Environment• SPEED User Study• Conclusion

6

How do Programmers Speak Code?

• 10 programmers read Java code out loud (Begel ‘05)– Graduate students in Computer Science– Five knew Java, five did not– Five were native English speakers, five were not– Five were educated in U.S.A., five were not

• Read pre-written code into tape recorder– As if speaking to a sophomore-level CS undergrad

who knows Java, but does not know the program

• Most programmers spoke the same way

7

for int i equals zero i less than ten i plus plus

for (int i = 0; i < 10; i++ ) { ▌}

How do Programmers Speak Code?

8

Spoken Words Can Be Hard To Write Down

2

How Do Programmers Speak Code?

9


2 2, two, to, too


10


2 2, two, to, too

print


11


2 2, two, to, too

print print, Print


12


2 2, two, to, too

print print, Print

drop stack process


13


2 2, two, to, too

print print, Print

drop stack process drop stack processdrop stackprocessdropstack processdropstackprocess


14

Many Ways to Say the Same Thing

bar[i]


15


bar[i] bar sub i, bar of i, i from bar


16



.


17



. period, dot


18



. period, dot

}


19



. period, dot

} right brace, close the if, end method


20



. period, dot


println


21



. period, dot


println print line, print lin, print l n


22

One Utterance May Mean Many Things

object stack


23


object stack Object stack;object.stackobject(stack)

object().stack()


24



object().stack()

array sub i plus plus


25



object().stack()

array sub i plus plus array[i]++array[i++]


26

People Have Trouble Saying Some Things

System.out.println


27


System.out.println system out print linesystem dot out print line

system dot out dot print line


28




(int)foo


29




(int)foo cast foo to integer int foo

cast something to integer. that something is foo.


30

Sometimes They Describe the Code

And then there’s a class.


31



Set all the fields of that object to null.


32



Set all the fields of that object to null.

All of these are just assignment operations.


33

Design Tradeoffs

CommandLanguage

Easy to analyze,but prescriptive

NaturalLanguage

Flexible,but ambiguous

Programmingby Voice

34

Programming by Voice Related Work

Human-CentricComputer-Centric

MultipleTasks

AuthoringOnly

Arnold ‘00

Snell ‘00

Price ‘00 ‘02

Desilets ‘01 ‘04

Gray ‘03

Begel ‘05

35

A More Natural Way to Code

public class symbol implements serializable

public class Symbol implements Serializable { ▌}

36


static hash map hash table gets new hash map

public class Symbol implements Serializable { static HashMap hashtbl = new HashMap();

▌}

37


end the class

public class Symbol implements Serializable { static HashMap hashtbl = new HashMap();

}▌

38


for int i equals zero i less than ten i plus plus

for (int i = 0; i < 10; i++ ) { ▌}

39

Too Many Ambiguities

for (int i = 0; i < 10; i++ ) { ▌}

4 int eye equals 0 aye less then ten i plus plus

KW or #?Spelling of ID?

KW or ID?for int i equals zero i less than ten i plus plus

40

Sometimes It’s Non-Obvious

for (times = 8; file(2, load); times == one) {▌

}

for times equals 8 file 2 load times equals one

fore *= 8; file.tooLode.times = won ▌

4; times = ate(file).to(load).equals(1) ▌

41

Spoken Java• Semantically identical to Java• Syntactically easier to say than Java

– Methodology generalizable to any computer language

1. All punctuation has English equivalents• Open Brace, End For Loop

2. Most punctuation is optional3. Provide verbalization for all abbreviations4. Relaxed phrasing for better fit with English

• (int)foo “cast foo to integer”• foo = 6 “set foo to 6”• foo[i]++ “increment the ith element of array foo”

42

SPEED: Speech Editor• Build an editor that supports naturally verbalized

programs• SPEED: SPEech EDitor

• Based on IBM ViaVoice, Eclipse IDE, Harmonia

– Spoken Java Language for Composition– Spoken Command language for Navigation,

Editing, Template instantiation, Refactorings, Search

– Audible and visual feedback• Similar to JavaSpeak (Smith 2000)

43

Harmonia Analysis Framework

• Framework to support interactive editors– Language-based, programmer-oriented tools

• Incremental analyses– Lexing (Wagner ‘97), GLR Parsing (Wagner ‘97, Begel ‘04), Static

Semantics (Garrison ‘87, Begel, Jamison)

• C, Java, Titanium, Cool, Flex, Bison– Also, languages where indentation and CRs are

significant• Interactive Program Transformations (Boshernitsan)

• CodeLink (Toomim et. al. ‘04)

• Shorthand Editing

44

Talk Outline

• Introduction and Motivation• Programming by Voice Program Analyses for Ambiguous Inputs • SPEech EDitor Programming Environment• SPEED User Study• Conclusion

45

Traditional Compiler Analyses

LexicalAnalysis

FOR I

Parsing SemanticAnalysis

for (i = 0; i < 10; i++ ) { }

Programming languages are designed to be unambiguous

For Loop

FOR Assign Expr

I = 0

i LocalVar int

46

Ambiguity-Aware Analysesfor i equals zero ...

Handles input stream, syntactic and semantic ambiguities

LexicalAnalysis

FOR I

AmbiguousParsing

SemanticAmbiguityResolution

For Loop

FOR Assign Expr

I = 0

i LocalVar int

foureye

LocalVar ?4 EYE FOUREYE

Assign Expr

= 0

Ambig Stmt

47

Scan Input Stream

CommercialSpeech

Recognizer

HomophoneDictionary Lexical

Analysis

48

Homophones Cause Ambiguities

for i equals

for i =

4 i equals

foreeye ==

fore ayeequals foureyeequals

foriequals

Concatenated words cause them too

4

forefour

eyeaye

===

49


LexicalAnalysis

FOR I

XGLRAmbiguous

Parsing


For Loop

FOR Assign Expr

I = 0

i LocalVar int

foureye


Assign Expr

= 0

Ambig Stmt

50

XGLR Parsing [Begel 04]

< XIF FIFTY FIVE

51

XGLR Parsing [ Begel 04 ]

< XIF FIFTY FIVEKW

52


< XFIFTY FIVEIFKW

53


< XFIFTY FIVEIFKW

FIFTY 5

50 FIVE

50 5

55

ID

ID

#

#

#

#

ID

#

ID

< X

< X

< X

< X

54


FIFTY FIVEIFKW

FIFTY 5

50 FIVE

5

ID

ID

#

50#

55#

#

ID

#

ID

IFKW

IFKW

IFKW

IFKW

< X

< X

< X

< X

< XOp

55


FIFTY FIVEIFKW

FIFTY 5

50 FIVE

50 5

55

ID

ID

#

#

#

#

ID

#

ID

IFKW

IFKW

IFKW

IFKW

FIFTYIFKW

FIFTY

ID

IDIF

KW (

(

FIFTYIFKW

FIFTY

ID

IDIF

KW ..

< X

< X

< X

< X

X<Op

56


FIFTY FIVEIFKW

FIFTY 5

50 FIVE

50 5

55

ID

ID

#

#

#

#

ID

#

ID

IFKW

IFKW

IFKW

IFKW

FIFTYIFKW

FIFTY

ID

IDIF

KW (

(

FIFTYIFKW

FIFTY

ID

IDIF

KW ..

< X

< X

< X

< X

X<Op

FIVE

5

FIVE

5#

ID

#

ID< X

< X

< X

< X

57


FIFTY FIVEIFKW

FIFTY 5

50 FIVE

50 5

55

ID

ID

#

#

#

#

ID

#

ID

IFKW

IFKW

IFKW

IFKW

FIFTYIFKW

FIFTY

ID

IDIF

KW (

(

FIFTYIFKW

FIFTY

ID

IDIF

KW ..

< X

< X

< X

< X

X<Op

FIVE

5

FIVE

5#

ID

#

ID< X

< X

< X

< X

58


FIFTY FIVEIFKW

FIFTY 5

50 FIVE

50 5

55

ID

ID

#

#

#

#

ID

#

ID

IFKW

IFKW

IFKW

IFKW

FIFTYIFKW

FIFTY

ID

IDIF

KW (

(

FIFTYIFKW

FIFTY

ID

IDIF

KW ..

< X

< X

< X

< X

X<Op

FIVE

5

FIVE

5#

ID

#

ID< X

< X

< X

< X

59


55#

IFKW

FIFTYIFKW

FIFTY

ID

IDIF

KW (

(

FIFTYIFKW ID .

X<Op

FIVE

5

FIVEID

#

ID< X

< X

< X

60


55#

IFKW

FIFTYIFKW

FIFTY

ID

IDIF

KW (

(

FIFTYIFKW ID .

X<Op

FIVE

5

FIVEID

#

ID< X

< X

< X

FIFTYIFKW ID ( FIVE

ID

FIFTYIFKW ID ( FIVE

ID

FIFTYIFKW ID . FIVE

ID

FIFTYIFKW ID . FIVE

ID .

.(

(

< X

< X

< X

< X

FIFTYIFKW

FIFTY

ID

IDIF

KW (

( FIVE

5#

ID )

)

< X

< X

61


55#

IFKW

FIFTYIFKW

FIFTY

ID

IDIF

KW (

(

FIFTYIFKW ID .

X<Op

FIVE

5

FIVEID

#

ID< X

< X

< X

FIFTYIFKW ID ( FIVE

ID

FIFTYIFKW ID ( FIVE

ID

FIFTYIFKW ID . FIVE

ID

FIFTYIFKW ID . FIVE

ID .

.(

(

< X

< X

< X

< X

FIFTYIFKW

FIFTY

ID

IDIF

KW (

( FIVE

5#

ID )

)

< X

< X

Op

Op

Op

Op

Op

Op

Op

Op

Op

ID

62


55#

IFKW

FIFTYIFKW

FIFTY

ID

IDIF

KW (

(

FIFTYIFKW ID .

X<Op

FIVE

5

FIVEID

#

ID< X

< X

< X

FIFTYIFKW ID ( FIVE

ID

FIFTYIFKW ID ( FIVE

ID

FIFTYIFKW ID . FIVE

ID

FIFTYIFKW ID . FIVE

ID .

.(

(

< X

< X

< X

< X

FIFTYIFKW

FIFTY

ID

IDIF

KW (

( FIVE

5#

ID )

)

< X

< X

Op

Op

Op

Op

Op

Op

Op

Op

Op

ID

63


55#

IFKW

FIFTYIFKW

FIFTY

ID

IDIF

KW (

(

FIFTYIFKW ID .

X<Op

FIVE

5

FIVEID

#

ID< X

< X

< X

FIFTYIFKW

FIFTY

ID

IDIF

KW (

( FIVE

5#

ID )

)

< X

< X

Op

Op

Op

Op

Op

ID

64


55#

IFKW

FIFTYIFKW

FIFTY

ID

IDIF

KW (

(

FIFTYIFKW ID .

X<Op

FIVE

5

FIVEID

#

ID< X

< X

< X

FIFTYIFKW

FIFTY

ID

IDIF

KW (

( FIVE

5#

ID )

)

< X

< X

Op

Op

Op

Op

Op

ID

ID

ID

ID

ID

ID

65


55#

IFKW

FIFTYIFKW

FIFTY

ID

IDIF

KW (

(

FIFTYIFKW ID .

X<Op

FIVE

5

FIVEID

#

ID< X

< X

< X

FIFTYIFKW

FIFTY

ID

IDIF

KW (

( FIVE

5#

ID )

)

< X

< X

Op

Op

Op

Op

Op

ID

ID

ID

ID

ID

ID

66


55#

IFKW

FIFTYIFKW ID .

X<Op

FIVEID

< X

FIFTYIFKW

FIFTY

ID

IDIF

KW (

( FIVE

5#

ID )

)

< X

< X

Op

Op

Op

ID

ID

ID

ID

FIFTYIFKW

FIFTY

ID

IDIF

KW (

( FIVE

5#

ID< X

< X

Op

Op

ID

ID

67


55#

IFKW

FIFTYIFKW

FIFTY

ID

IDIF

KW (

(

FIFTYIFKW ID .

X<Op

FIVE

5

FIVEID

#

ID< X

< X

< X

FIFTYIFKW

FIFTY

ID

IDIF

KW (

( FIVE

5#

ID )

)

< X

< X

Op

Op

Op

Op

Op

ID

ID

ID

ID

ID

ID

Expr

ExprExpr

ExprFuncCall

ExprFuncCall

68

XGLR Summary

• Generalization of traditional GLR algorithm– Forks on structural and lexical ambiguity– Preserves subtree sharing when parses have

different yields– Retains efficiency when parses get out of sync

• Determine parse position w.r.t. ambiguous input

• Blender: Combined lexer and parser generator for XGLR

69

GLR Parsing Genealogy

Tomita1985

Farshi1991

Rekers1992

Wagner1997

Visser1997

van den Brand2002

Begel2004

Incremental

Scannerless

Johnstone, Scott2002

Input StreamAmbiguities

70


LexicalAnalysis

FOR I

XGLRAmbiguous

Parsing


For Loop

FOR Assign Expr

I = 0

i LocalVar int

foureye


Assign Expr

= 0

Ambig Stmt

71

Disambiguation Exampleclass Loader {

public void load() { String filetoload = null;InputStream stream =

getStream();... ▌

}}

file to load equals stream dot read string

filetoload = stream.readString();

72

Many Interpretations

file(2, lowed)

file(to, load) file(to.lode) file(to(lode)) file(toload)

(file, 2, load)

file.to.lowed file.to(load)

file.toload filetoload() filetoload

73

Incremental Semantics• What does this name mean? • What names are visible at this program

point?– Or, What can I say here?

• Visibility Graph [Garrison 1987]– Incrementally updated data structure for scopes, names

and bindings– Designed Visibility Graph algorithms for name

propagation and incremental update– Used for type checking, too

• Doesn’t <insert favorite IDE here> do this?

74

Program Context Can Helpclass Loader {

public void load() {String filetoload = null;InputStream stream =

getStream();... ▌

}}

class Loaderscope

[ load, Method, () void ]

method loadscope

[ filetoload, LocalVar, String ][ stream, LocalVar, InputStream ]

75

Program Context Can Helpclass Loader {

public void load() {String filetoload = null;InputStream stream =

getStream();... ▌

}}

class Loaderscope


method loadscope

[ filetoload, LocalVar, String ][ stream, LocalVar, InputStream ][ load, Method, () void ]

76

Semantic Disambiguation

file(2, lowed)


(file, 2, load)

file.to.lowed

file.to(load)


class Loaderscope


method loadscope


77


file(2, lowed)


(file, 2, load)

file.to.lowed

file.to(load)


class Loaderscope


method loadscope

Is “file” a visible variable name?


78


file(2, lowed)


filetoload() filetoload

class Loaderscope


method loadscope

Is “file” a visible method name?


79


filetoload() filetoload

class Loaderscope


method loadscope

Is “filetoload” a visible method name?


80

Manual Disambiguation• Some ambiguities cannot (and should not) be

automatically resolved:

print(“line”) vs. println()

if (pred1) then if (pred2) then foo() else bar()

• If ambiguities remain, ask the user how to resolve them. (e.g. [Mankoff 00])

if

foo()

if

bar()

if

if

foo()

bar()

81

Talk Outline

• Introduction and Motivation• Programming by Voice• Program Analyses for Ambiguous Inputs SPEech EDitor Programming

Environment• SPEED User Study• Conclusion

82

SPEED Editor

83

Speech Editing Model

Code TemplateInsertion

ToggleMicrophone

84


Choose FromAlternatives

85


86


87

Context-Sensitive Mouse Grid

88

What Can I Say/Type?

89

Cache Pad

90

Talk Outline

• Introduction and Motivation• Programming by Voice• Program Analyses for Ambiguous Inputs• SPEech EDitor Programming Environment SPEED User Study• Conclusion

91

Study - SPEED UsabilityGoal: Understand how SPEED can be used

by expert programmers

Hypothesis: SPEED is learnable and usable for standard programming tasks

1. Train 5 expert Java programmers on SPEED2. Create and modify code

– Build a Linked List data structure with associated algorithms

• 3 programmers used commercial speech recognizer2 programmers used human speech recognizer

92

Metrics

• Number of Commands/Dictations• Features Used

– Code Templates, Dictation, Navigation, Editing, Fixing Mistakes

• Quantity and Kinds of Mistakes– Speech Recognition, SPEED, User

93

Results

• Accuracy of commercial speech recognizers was horrible (25-50%). Human SR was much better (10-20%).

• Recognition delay was equal for both recognizers (0.5-0.75 sec)

94

Results

• Commands were easy to learn and remember.– Very few user mistakes

• Most commands spoken for code templates and editing.– GOMS analysis predicts speech will be slower

until you can get a lot of text for each utterance• Speakers were apprehensive about speaking

code instead of describing it via code templates.

95

Study Conclusion

• SPEED is learnable in a short amount of time

• Programming-by-voice is slower than typing– Programmers would not want to use it until

they had to• Programmers believed they would be

efficient enough using SPEED to remain in software engineering jobs

96

Talk Outline

• Introduction and Motivation• Programming by Voice• Program Analyses for Ambiguous Inputs• SPEech EDitor Programming Environment• SPEED User Study Conclusion

97

Contributions

1. A study of programmers to understand and design a naturally verbalizable input for programming

2. An interactive editor designed for spoken interaction

3. The use of syntax and semantics of programming for disambiguation– Enhanced lexical, syntactic, semantic analyses for

support of verbal ambiguities4. Evaluation of design and tools by studying

programmers using voice for software development

98

SPEED Next Steps

• Add more code templates– Enable users to write their own

• Add “Jump To <arbitrary code here>”• Find new ways to edit strings by voice• Integrate speech recognition with other IDE

features– GUI, code completion, debugger

99

Further SPEED Studies

• Develop methodology for short-term voice recognition studies

• Find out why programmers felt code dictation was weird.

• Evaluate more complex code editing operations by voice

• Evaluate context-sensitive mouse grid usability

100

Future of Programming by Voice1. Improved automation of semantic disambiguation

– Use ideas from NLP, Machine Learning (team styles)

2. Early pruning of ambiguities using analysis feedback3. Higher-level linguistic programming tools

– Transformations, Paraphrasing– Phonetic search, Audible feedback

4. Support more software engineering tasks by voice– Debuggers, IDEs, Comments, Code reviews

5. Design spoken variants of other formal languages– General (C, C#) Scripting (PL, OS), Design (HCI),

Command (Robotics), Domain-specific languages (SQL)

101

Any Questions?

Andrew Begel: [email protected]

Spoken Language Support for Software Development

Documents

Transcript of Spoken Language Support for Software Development