PLATEAU 2011 - Capturing and Analyzing Low-Level Events from the Code Editor

Capturing and Analyzing Low-Level Eventsfrom the Code Editor

YoungSeok Yoon ([email protected])

Institute for Software Research (ISR)

Brad A. Myers ([email protected])

Human-Computer Interaction Institute (HCII)

School of Computer Science

Carnegie Mellon University

mailto:[email protected]

mailto:[email protected]

PLATEAU 2011 2

Motivation• When trying to understand what problems the

developers have, researchers investigate…

when / how do the developers use certain features?what kinds of

problems do developers

have?

how frequently do those prob-

lems occur?

in what context does a certain event occur?

in what sequence do the developers complete cer-

tain tasks?

PLATEAU 2011 3

Motivation• When trying to evaluate existing tools…

do the develop-ers actually

use those tools?

how fre-quently do developers

use the tools?

how much time do they spend using the tools?

exactly how they

are used in detail?

do the devel-opers use the tools as ex-

pected?

what types of errors do they make?

PLATEAU 2011 4

How do we answer these questions?

• Essentially, we are collecting the usage datafrom the developers

• The usage data can be collected by:▪ asking the developers▪ observing the developers▪ mining software repositories▪ logging the developers’ behavior

• However, previous tools make it difficult or impossible to get / analyze fine-grained code editing history

PLATEAU 2011 5

Asking the developers• Interviews, surveys, focus groups

• Limitations▪ Responses may not be reliable▪ Developers perform many operations quite automatically

Maybe they might not remember the specific occasions

PLATEAU 2011 6

Observing the developers• Contextual inquiries, lab studies• Think aloud + videotape

• Limitations▪ Requires manual inspection of the videotape

Can be time-consuming and error-prone

• [Coman2008], [Ko2003], [Ko2005], ...

PLATEAU 2011 7

Mining software repositories• Has been used for studying software evolution

• Limitations▪ Cannot know exactly what happened between two

consecutive revisions we may miss some important user behavior

▪ Need to infer from the check-in comments and other clues

• [Aversano2007], [Bettenburg2009], [Kim2005], [Murphy-Hill2009], …

PLATEAU 2011 8

Eclipse Usage Data Collector (UDC)• Logging tool for aggregate data• Collects usage information from all the Eclipse users

who consented to upload their data to UDC

• Limitations▪ Does not capture the sequences of events▪ Misses some important commands executed

• e.g., UDC ignores backspace, moving cursor with arrow keys, …

▪ Command-specific parameters are not captured

PLATEAU 2011 9

Other Logging Tools• Mylyn Monitor [Kersten2006]

▪ Textual level changes are not captured▪ Focuses on more abstract user interaction data on the

entire IDE

• Syde / Replay [Hattori2011]▪ Only commands which change the source code are logged▪ AST-level, not textual level

PLATEAU 2011 10

Our new logging tool: FLUORITE

Full ofLow-levelUserOperationsRecordedInTheEditor

PLATEAU 2011 11

Our new logging tool: FLUORITE• FLUORITE is a publicly available Eclipse plug-in that captures

low-level code editing events andproduces XML log files

• This is a tool for the researchers

• Can be used to overcome the previously listed limitations

• For each event, FLUORITE logs:▪ timestamp▪ event type / command ID▪ event-specific parameters

PLATEAU 2011 12

Three types of events• FLUORITE logs three different types of events:

▪ Commands: all the user events in the code editor• (e.g., typing new text, moving cursor, copying, …)

▪ Document Changes• logged whenever the active file is modified• contain the actual inserted/deleted code, resulting code length, …• makes it possible to reproduce snapshots of the files at any point

▪ Annotations• logged when the developer manually adds annotations to help the researcher

PLATEAU 2011 13

An example

Click

PLATEAU 2011 14

An example

Shift ↓

PLATEAU 2011 15

An example

Del

PLATEAU 2011 16

An example

PLATEAU 2011 17

Resulting FLUORITE log<Command __id="2" _type="MoveCaretCommand" caretOffset="142"

docOffset="142" timestamp="3977"/><Command __id="3" _type="EclipseCommand"

commandID="eventLogger.styledTextCommand.SELECT_LINE_DOWN" timestamp="5598"/>

<DocumentChange __id="4" _type="Delete" docASTNodeCount="22" docActiveCodeLength="125" docExpressionCount="10" docLength="151" endLine="9" length="39" offset="142" startLine="8" timestamp="7186">

<text> <![CDATA[ System.out.println("Hello World!");]]> </text></DocumentChange><Command __id="5" _type="EclipseCommand"

commandID="org.eclipse.ui.edit.delete" timestamp="7202"/>

PLATEAU 2011 18







① Cursor was moved by the mouse clicking

PLATEAU 2011 19







② One line of code was selected

bySHIFT + ↓

PLATEAU 2011 20







③ “Delete” key was pressed

PLATEAU 2011 21







④ The actual code deleted by

the “delete” com-mand

PLATEAU 2011 22

FLUORITE log files• The log files are written in XML format

▪ Anyone can build their own automatic analyzer!

• Is FLUORITE practical enough to use?▪ Already has been useful for a couple of studies

• Our own exploratory study with 12 developers• Dörner’s evaluation study of Euklas system

▪ Size of the logs• Average log size: 236.8KB / hr = 9.25MB / week• Could be reduced to 1MB / week if the logs were compressed

▪ Performance• There was no measurable performance loss during our study

PLATEAU 2011 23

Built-in analyses• We also provide a log analyzer which has

several built-in analyses

• Our study focused on when and how the developers backtrack while editing code▪ The analyses were built for this purpose

PLATEAU 2011 24

Example Analysis:Command distribution report

PLATEAU 2011 25


PLATEAU 2011 26


PLATEAU 2011 27


PLATEAU 2011 28

Example Analysis:Keystroke distribution report

PLATEAU 2011 29


PLATEAU 2011 30


PLATEAU 2011 31


PLATEAU 2011 32

Example Analysis:Code size growth graph

PLATEAU 2011 33

Events View

PLATEAU 2011 34

Example Analysis:Code editing pattern detection• FLUORITE logs enable us to detect code editing patterns

composed of sequences of events• Examples (not all of them are implemented)

▪ Typo correction▪ Parameter tuning▪ Commenting out / uncommenting▪ Cutting/copying and pasting within a project▪ Manual refactoring (e.g., rename variable)

• Preliminary implementationsPattern Counts Rate Precision

Typo correction 274 / 288 13.6 / hr 95.14%Parameter tuning 52 / 98 2.6 / hr 53.06%

PLATEAU 2011 35

Conclusion• FLUORITE web page:

http://www.cs.cmu.edu/~fluorite/

• FLUORITE is publicly available, open-sourced tool for Eclipse which can be used when conducting studies

• FLUORITE turned out to be useful for our own study, and we hope that it will help you too!


PLATEAU 2011 36

Questions?• FLUORITE web page:


• Acknowledgements▪ National Science Foundation (NSF) CCF-0811610▪ Korea Foundation for Advanced Studies (KFAS)


PLATEAU 2011 37

References• [Aversano2007] Aversano, L., Cerulo, L. and Di Penta, M. 2007. How Clones are Maintained: An Empirical

Study. In Proc. 11th European Conf. on Soft. Maint. and Reengineering (CSMR’07). 81-90.• [Bettenburg2009] Bettenburg, N., Weyi, S., Ibrahim, W., Adams, B., Ying, Z. and Hassan, A. E. 2009. An

Empirical Study on Inconsistent Changes to Code Clones at Release Level. In Proc. 16th Working Conf. on Reverse Eng. (WCRE’09). 85-94.

• [Coman2008] Coman, I. D. and Sillitti, A. 2008. Automated Identification of Tasks in Development Sessions. In Proc. 16th IEEE Int’l Conf. on Program Comprehension (ICPC’08). 212-217.

• [Kersten2006] Kersten, M. and Murphy, G. C. 2006. Using task context to improve programmer productivity. In Proc. 14th ACM SIGSOFT Int’l Symp. on Foundations of Soft. Eng. (FSE’06). 1-11.

• [Kim2005] Kim, M., Sazawal, V., Notkin, D. and Murphy, G. 2005. An empirical study of code clone genealogies. In Proc. 10th Euro. Soft. Eng. Conf. & 13th ACM SIGSOFT Int’l Symp. on Foundations of Soft. Eng. (ESEC/FSE’05). 187-196.

• [Ko2003] Ko, A. J. and Myers, B. A. 2003. Development and evaluation of a model of programming errors. In Proc. IEEE Symp. on Human Centric Computing Languages and Environments (HCC’03). 7-14.

• [Ko2005] Ko, A. J., Aung, H. H. and Myers, B. A. 2005. Design requirements for more flexible structured editors from a study of programmers’ text editing. In Proc. Extended Abstracts of CHI'2005. 1557-1560.

• [Hattori2011] Hattori, L., D’Ambros, M., Lanza, M. and Lungu, M. 2011. Software Evolution Comprehension: Replay to the Rescue. In Proc. 19th IEEE Int’l Conf. on Program Comprehension (ICPC’11). 161-170.

• [Murphy-Hill2009] Murphy-Hill, E., Parnin, C. and Black, A. P. 2009. How we refactor, and how we know it. In Proc. 31st Int’l Conf. on Soft. Eng. (ICSE’09). 287-297.

PLATEAU 2011 38

BACKUP SLIDES

PLATEAU 2011 39

Longer example log<Command __id="2" _type="MoveCaretCommand" caretOffset="103" docOffset="103"

timestamp="11073"/><Command __id="3" _type="EclipseCommand"

commandID="eventLogger.styledTextCommand.SELECT_LINE_DOWN" timestamp="13001"/><DocumentChange __id="4" _type="Delete" docASTNodeCount="15" docActiveCodeLength="86"

docExpressionCount="4" docLength="112" endLine="8" length="39" offset="103" startLine="7" timestamp="14969">

<text><![CDATA[ System.out.println("Hello World!");]]></text></DocumentChange><Command __id="5" _type="EclipseCommand" commandID="org.eclipse.ui.edit.delete"

timestamp="14985"/><DocumentChange __id="6" _type="Insert" docASTNodeCount="22" docActiveCodeLength="125"

docExpressionCount="10" docLength="151" length="39" offset="103" timestamp="17564"><text><![CDATA[ System.out.println("Hello World!");]]></text></DocumentChange><Command __id="7" _type="UndoCommand" timestamp="17570"/>

PLATEAU 2011 40


PLATEAU 2011 41


Command Count (Percentage)

InsertString 5797 (31.48%)

LINE_DOWN 5693 (10.67%)

DELETE_PREV 4495 (10.48%)

MoveCaret 3586 (8.63%)

LINE_UP 2751 (8.27%)

PLATEAU 2011 42


PLATEAU 2011 43


Key Count (Percentage)

↓ 5797 (12.64%)

Backspace 5693 (12.41%)

↑ 4495 (9.80%)

→ 3586 (7.82%)

← 2751 (6.00%)

PLATEAU 2011 - Capturing and Analyzing Low-Level Events from the Code Editor

Technology

Transcript of PLATEAU 2011 - Capturing and Analyzing Low-Level Events from the Code Editor