Does a Programmer’s Activity Indicate Knowledge of...
Transcript of Does a Programmer’s Activity Indicate Knowledge of...
![Page 1: Does a Programmer’s Activity Indicate Knowledge of Code?se.inf.ethz.ch/old/teaching/2010-S/0276/slides/rueegg.pdf · Introduction Motivation DOI: Degree-Of-Interest Study Subjects](https://reader033.fdocuments.us/reader033/viewer/2022042321/5f0aab297e708231d42cc096/html5/thumbnails/1.jpg)
1
Does a Programmer’s Activity IndicateKnowledge of Code?Software Engineering Seminar 2010
Who? Jan Rüegg
From? 2007 Paper By
Thomas Fritz, University of British Columbia
Gail C. Murphy, University of British Columbia
Emily Hill, University of Delaware
When? 16th March 2010
![Page 2: Does a Programmer’s Activity Indicate Knowledge of Code?se.inf.ethz.ch/old/teaching/2010-S/0276/slides/rueegg.pdf · Introduction Motivation DOI: Degree-Of-Interest Study Subjects](https://reader033.fdocuments.us/reader033/viewer/2022042321/5f0aab297e708231d42cc096/html5/thumbnails/2.jpg)
2
Imagine You See Something LikeThis. . .What?!?
if (fabs (text->letter_spacing) > 0.1)
{
PangoAttrList *attrs;
PangoAttribute *attr;
attrs = pango_layout_get_attributes(
layout->layout);
if (attrs)
pango_attr_list_ref (attrs);
else
attrs = pango_attr_list_new ();
attr = pango_attr_letter_spacing_new (
text->letter_spacing * PANGO_SCALE);
attr->start_index = 0;
attr->end_index = -1;
...
![Page 3: Does a Programmer’s Activity Indicate Knowledge of Code?se.inf.ethz.ch/old/teaching/2010-S/0276/slides/rueegg.pdf · Introduction Motivation DOI: Degree-Of-Interest Study Subjects](https://reader033.fdocuments.us/reader033/viewer/2022042321/5f0aab297e708231d42cc096/html5/thumbnails/3.jpg)
3
To The Rescue, Git Blame!. . . really?
(Martin Nordholts) if (fabs (text->letter_spacing) > 0.1)
(Martin Nordholts) {
(Michael Natterer) PangoAttrList *attrs;
(Martin Nordholts) PangoAttribute *attr;
(Martin Nordholts)(Michael Natterer) attrs = pango_layout_get_attributes(
(Michael Natterer) layout->layout);
(Michael Natterer) if (attrs)
(Michael Natterer) pango_attr_list_ref (attrs);
(Michael Natterer) else
(Michael Natterer) attrs = pango_attr_list_new ();
(Michael Natterer)(Martin Nordholts) attr = pango_attr_letter_spacing_new (
(Martin Nordholts) text->letter_spacing * PANGO_SCALE);
(Martin Nordholts)(Martin Nordholts) attr->start_index = 0;
(Martin Nordholts) attr->end_index = -1;
(Martin Nordholts) ...
![Page 4: Does a Programmer’s Activity Indicate Knowledge of Code?se.inf.ethz.ch/old/teaching/2010-S/0276/slides/rueegg.pdf · Introduction Motivation DOI: Degree-Of-Interest Study Subjects](https://reader033.fdocuments.us/reader033/viewer/2022042321/5f0aab297e708231d42cc096/html5/thumbnails/4.jpg)
4
The Basic Question
How can we measure a Programmers Knowledgeabout source Code?
HypothesisThe more Frequently and Recently a pro-grammer has interacted with an element X,the higher is his Knowledge about X.
![Page 5: Does a Programmer’s Activity Indicate Knowledge of Code?se.inf.ethz.ch/old/teaching/2010-S/0276/slides/rueegg.pdf · Introduction Motivation DOI: Degree-Of-Interest Study Subjects](https://reader033.fdocuments.us/reader033/viewer/2022042321/5f0aab297e708231d42cc096/html5/thumbnails/5.jpg)
4
The Basic Question
How can we measure a Programmers Knowledgeabout source Code?
HypothesisThe more Frequently and Recently a pro-grammer has interacted with an element X,the higher is his Knowledge about X.
![Page 6: Does a Programmer’s Activity Indicate Knowledge of Code?se.inf.ethz.ch/old/teaching/2010-S/0276/slides/rueegg.pdf · Introduction Motivation DOI: Degree-Of-Interest Study Subjects](https://reader033.fdocuments.us/reader033/viewer/2022042321/5f0aab297e708231d42cc096/html5/thumbnails/6.jpg)
5
Outline
Example, Hypothesis
IntroductionMotivationDOI: Degree-Of-Interest
StudySubjectsQuestionnaires
ResultsSignificance of DOIProblemsImprovements
![Page 7: Does a Programmer’s Activity Indicate Knowledge of Code?se.inf.ethz.ch/old/teaching/2010-S/0276/slides/rueegg.pdf · Introduction Motivation DOI: Degree-Of-Interest Study Subjects](https://reader033.fdocuments.us/reader033/viewer/2022042321/5f0aab297e708231d42cc096/html5/thumbnails/7.jpg)
5
Outline
Example, Hypothesis
IntroductionMotivationDOI: Degree-Of-Interest
StudySubjectsQuestionnaires
ResultsSignificance of DOIProblemsImprovements
![Page 8: Does a Programmer’s Activity Indicate Knowledge of Code?se.inf.ethz.ch/old/teaching/2010-S/0276/slides/rueegg.pdf · Introduction Motivation DOI: Degree-Of-Interest Study Subjects](https://reader033.fdocuments.us/reader033/viewer/2022042321/5f0aab297e708231d42cc096/html5/thumbnails/8.jpg)
5
Outline
Example, Hypothesis
IntroductionMotivationDOI: Degree-Of-Interest
StudySubjectsQuestionnaires
ResultsSignificance of DOIProblemsImprovements
![Page 9: Does a Programmer’s Activity Indicate Knowledge of Code?se.inf.ethz.ch/old/teaching/2010-S/0276/slides/rueegg.pdf · Introduction Motivation DOI: Degree-Of-Interest Study Subjects](https://reader033.fdocuments.us/reader033/viewer/2022042321/5f0aab297e708231d42cc096/html5/thumbnails/9.jpg)
6
MotivationWho knows What?
Important forI Expertise RecommendationsI IDE with Adaptive InterfaceI Recommending group members
Mylyn Degree-Of-Interest ModelI Introduced 2005 by Mik Kersten and Gail C. MurphyI Interest level in program elementsI Original goal: An Adaptive Eclipse Interface
![Page 10: Does a Programmer’s Activity Indicate Knowledge of Code?se.inf.ethz.ch/old/teaching/2010-S/0276/slides/rueegg.pdf · Introduction Motivation DOI: Degree-Of-Interest Study Subjects](https://reader033.fdocuments.us/reader033/viewer/2022042321/5f0aab297e708231d42cc096/html5/thumbnails/10.jpg)
6
MotivationWho knows What?
Important forI Expertise RecommendationsI IDE with Adaptive InterfaceI Recommending group members
Mylyn Degree-Of-Interest ModelI Introduced 2005 by Mik Kersten and Gail C. MurphyI Interest level in program elementsI Original goal: An Adaptive Eclipse Interface
![Page 11: Does a Programmer’s Activity Indicate Knowledge of Code?se.inf.ethz.ch/old/teaching/2010-S/0276/slides/rueegg.pdf · Introduction Motivation DOI: Degree-Of-Interest Study Subjects](https://reader033.fdocuments.us/reader033/viewer/2022042321/5f0aab297e708231d42cc096/html5/thumbnails/11.jpg)
7
Mylyn DOI ModelHow Mylyn improves Eclipse. . .
How does it work?I Each Java element has its DOI value thatI Increases with each select/edit/. . .
different operations are differently weightedI Decreases over time, when not accessed
Idea Elements with higher DOI:Higher up in Eclipse lists. . .
Effect Programmer spends less time searching forimportant items
![Page 12: Does a Programmer’s Activity Indicate Knowledge of Code?se.inf.ethz.ch/old/teaching/2010-S/0276/slides/rueegg.pdf · Introduction Motivation DOI: Degree-Of-Interest Study Subjects](https://reader033.fdocuments.us/reader033/viewer/2022042321/5f0aab297e708231d42cc096/html5/thumbnails/12.jpg)
7
Mylyn DOI ModelHow Mylyn improves Eclipse. . .
How does it work?I Each Java element has its DOI value thatI Increases with each select/edit/. . .
different operations are differently weightedI Decreases over time, when not accessed
Idea Elements with higher DOI:Higher up in Eclipse lists. . .
Effect Programmer spends less time searching forimportant items
![Page 13: Does a Programmer’s Activity Indicate Knowledge of Code?se.inf.ethz.ch/old/teaching/2010-S/0276/slides/rueegg.pdf · Introduction Motivation DOI: Degree-Of-Interest Study Subjects](https://reader033.fdocuments.us/reader033/viewer/2022042321/5f0aab297e708231d42cc096/html5/thumbnails/13.jpg)
8
Empirical Study
SubjectsI 19 Industry Java ProgrammersI Two IBM development laboratory locationsI Experience between one month to twenty years
Interactive Monitoring for five weeksI Eclipse Mylyn projectI Modified to the needs of the studyI Show about one questionnaire per week
EvaluationI Analysis of the collected dataI Interviews with programmers about the results
![Page 14: Does a Programmer’s Activity Indicate Knowledge of Code?se.inf.ethz.ch/old/teaching/2010-S/0276/slides/rueegg.pdf · Introduction Motivation DOI: Degree-Of-Interest Study Subjects](https://reader033.fdocuments.us/reader033/viewer/2022042321/5f0aab297e708231d42cc096/html5/thumbnails/14.jpg)
8
Empirical Study
SubjectsI 19 Industry Java ProgrammersI Two IBM development laboratory locationsI Experience between one month to twenty years
Interactive Monitoring for five weeksI Eclipse Mylyn projectI Modified to the needs of the studyI Show about one questionnaire per week
EvaluationI Analysis of the collected dataI Interviews with programmers about the results
![Page 15: Does a Programmer’s Activity Indicate Knowledge of Code?se.inf.ethz.ch/old/teaching/2010-S/0276/slides/rueegg.pdf · Introduction Motivation DOI: Degree-Of-Interest Study Subjects](https://reader033.fdocuments.us/reader033/viewer/2022042321/5f0aab297e708231d42cc096/html5/thumbnails/15.jpg)
8
Empirical Study
SubjectsI 19 Industry Java ProgrammersI Two IBM development laboratory locationsI Experience between one month to twenty years
Interactive Monitoring for five weeksI Eclipse Mylyn projectI Modified to the needs of the studyI Show about one questionnaire per week
EvaluationI Analysis of the collected dataI Interviews with programmers about the results
![Page 16: Does a Programmer’s Activity Indicate Knowledge of Code?se.inf.ethz.ch/old/teaching/2010-S/0276/slides/rueegg.pdf · Introduction Motivation DOI: Degree-Of-Interest Study Subjects](https://reader033.fdocuments.us/reader033/viewer/2022042321/5f0aab297e708231d42cc096/html5/thumbnails/16.jpg)
9
Questionnaires
Individual Questionnaires are generatedI After 20000, 40000 and 60000 interactionsI Random questions from each ofI 20% highest DOII 20% lowest DOII 20% in the middleI Automatically generated and verified
Typical Flow of Events1. Programmer works as usual with IDE2. IDE monitors elements for about about a week3. A questionnaire pops up, asking about high and low
DOI elements4. Programmer answers
![Page 17: Does a Programmer’s Activity Indicate Knowledge of Code?se.inf.ethz.ch/old/teaching/2010-S/0276/slides/rueegg.pdf · Introduction Motivation DOI: Degree-Of-Interest Study Subjects](https://reader033.fdocuments.us/reader033/viewer/2022042321/5f0aab297e708231d42cc096/html5/thumbnails/17.jpg)
9
Questionnaires
Individual Questionnaires are generatedI After 20000, 40000 and 60000 interactionsI Random questions from each ofI 20% highest DOII 20% lowest DOII 20% in the middleI Automatically generated and verified
Typical Flow of Events1. Programmer works as usual with IDE2. IDE monitors elements for about about a week3. A questionnaire pops up, asking about high and low
DOI elements4. Programmer answers
![Page 18: Does a Programmer’s Activity Indicate Knowledge of Code?se.inf.ethz.ch/old/teaching/2010-S/0276/slides/rueegg.pdf · Introduction Motivation DOI: Degree-Of-Interest Study Subjects](https://reader033.fdocuments.us/reader033/viewer/2022042321/5f0aab297e708231d42cc096/html5/thumbnails/18.jpg)
10
Typical QuestionsWhat the Subjects were asked. . .
1 public class Calculator2 {3 public Stack<StackItem> stack;4
5 public Calculator(double operand)6 {7 stack = new Stack<StackItem>();8 stack.push(new StackNumber(operand));9 }
10
11 public double popOperand()12 {13 double result = peekOperand();14 stack.pop();15 return result;16 }17 ...
![Page 19: Does a Programmer’s Activity Indicate Knowledge of Code?se.inf.ethz.ch/old/teaching/2010-S/0276/slides/rueegg.pdf · Introduction Motivation DOI: Degree-Of-Interest Study Subjects](https://reader033.fdocuments.us/reader033/viewer/2022042321/5f0aab297e708231d42cc096/html5/thumbnails/19.jpg)
11
Typical QuestionsWhat the Subjects were asked. . .
Q1) Do you know the types of the parameters that arepassed to the invocation of constructor ’Calculator’?
Q2) Can you recall one method/constructor that callsmethod ’Calculator.popOperand()’?
![Page 20: Does a Programmer’s Activity Indicate Knowledge of Code?se.inf.ethz.ch/old/teaching/2010-S/0276/slides/rueegg.pdf · Introduction Motivation DOI: Degree-Of-Interest Study Subjects](https://reader033.fdocuments.us/reader033/viewer/2022042321/5f0aab297e708231d42cc096/html5/thumbnails/20.jpg)
12
Typical QuestionsWhat the Subjects were asked. . .
1 public class Calculator2 {3 public Stack<StackItem> stack;4
5 public Calculator(double operand)6 {7 stack = new Stack<StackItem>();8 stack.push(new StackNumber(operand));9 }
10
11 public double popOperand()12 {13 double result = peekOperand();14 stack.pop();15 return result;16 }17 ...
![Page 21: Does a Programmer’s Activity Indicate Knowledge of Code?se.inf.ethz.ch/old/teaching/2010-S/0276/slides/rueegg.pdf · Introduction Motivation DOI: Degree-Of-Interest Study Subjects](https://reader033.fdocuments.us/reader033/viewer/2022042321/5f0aab297e708231d42cc096/html5/thumbnails/21.jpg)
13
GUIHow the Subjects answered. . .
![Page 22: Does a Programmer’s Activity Indicate Knowledge of Code?se.inf.ethz.ch/old/teaching/2010-S/0276/slides/rueegg.pdf · Introduction Motivation DOI: Degree-Of-Interest Study Subjects](https://reader033.fdocuments.us/reader033/viewer/2022042321/5f0aab297e708231d42cc096/html5/thumbnails/22.jpg)
13
GUIHow the Subjects answered. . .
![Page 23: Does a Programmer’s Activity Indicate Knowledge of Code?se.inf.ethz.ch/old/teaching/2010-S/0276/slides/rueegg.pdf · Introduction Motivation DOI: Degree-Of-Interest Study Subjects](https://reader033.fdocuments.us/reader033/viewer/2022042321/5f0aab297e708231d42cc096/html5/thumbnails/23.jpg)
13
GUIHow the Subjects answered. . .
![Page 24: Does a Programmer’s Activity Indicate Knowledge of Code?se.inf.ethz.ch/old/teaching/2010-S/0276/slides/rueegg.pdf · Introduction Motivation DOI: Degree-Of-Interest Study Subjects](https://reader033.fdocuments.us/reader033/viewer/2022042321/5f0aab297e708231d42cc096/html5/thumbnails/24.jpg)
14
ResultsSignificance of DOI
![Page 25: Does a Programmer’s Activity Indicate Knowledge of Code?se.inf.ethz.ch/old/teaching/2010-S/0276/slides/rueegg.pdf · Introduction Motivation DOI: Degree-Of-Interest Study Subjects](https://reader033.fdocuments.us/reader033/viewer/2022042321/5f0aab297e708231d42cc096/html5/thumbnails/25.jpg)
15
ResultsSignificance of Recency and Frequency
![Page 26: Does a Programmer’s Activity Indicate Knowledge of Code?se.inf.ethz.ch/old/teaching/2010-S/0276/slides/rueegg.pdf · Introduction Motivation DOI: Degree-Of-Interest Study Subjects](https://reader033.fdocuments.us/reader033/viewer/2022042321/5f0aab297e708231d42cc096/html5/thumbnails/26.jpg)
16
ResultsSignificance of Recency and Frequency
![Page 27: Does a Programmer’s Activity Indicate Knowledge of Code?se.inf.ethz.ch/old/teaching/2010-S/0276/slides/rueegg.pdf · Introduction Motivation DOI: Degree-Of-Interest Study Subjects](https://reader033.fdocuments.us/reader033/viewer/2022042321/5f0aab297e708231d42cc096/html5/thumbnails/27.jpg)
17
ResultsSignificance of Recency and Frequency
![Page 28: Does a Programmer’s Activity Indicate Knowledge of Code?se.inf.ethz.ch/old/teaching/2010-S/0276/slides/rueegg.pdf · Introduction Motivation DOI: Degree-Of-Interest Study Subjects](https://reader033.fdocuments.us/reader033/viewer/2022042321/5f0aab297e708231d42cc096/html5/thumbnails/28.jpg)
18
ProblemsThreads to Validity
I Number of Subjects
I Technical Problems with IDE
I Only Java/Eclipse watched
I "Nature" of Questions
I Only IDE monitored
![Page 29: Does a Programmer’s Activity Indicate Knowledge of Code?se.inf.ethz.ch/old/teaching/2010-S/0276/slides/rueegg.pdf · Introduction Motivation DOI: Degree-Of-Interest Study Subjects](https://reader033.fdocuments.us/reader033/viewer/2022042321/5f0aab297e708231d42cc096/html5/thumbnails/29.jpg)
19
ImprovementsWhat should be Added to the DOI
Authorship Leads to more knowledge about authored programelements
Code Stability Fewer code changes lead to longer knowledge aboutprogram elements
Activity Short-term kinds of activity, like debugging, have askewing effect
![Page 30: Does a Programmer’s Activity Indicate Knowledge of Code?se.inf.ethz.ch/old/teaching/2010-S/0276/slides/rueegg.pdf · Introduction Motivation DOI: Degree-Of-Interest Study Subjects](https://reader033.fdocuments.us/reader033/viewer/2022042321/5f0aab297e708231d42cc096/html5/thumbnails/30.jpg)
20
Summary
I DOI is a good Indicator for structural codeKnowledge
I Improvements are still possibleAuthorship, Activity, . . .
OutlookI Results can be used to create
better Expertise Recommendation tools
I Should be repeated withmore Programmers, Languages
![Page 31: Does a Programmer’s Activity Indicate Knowledge of Code?se.inf.ethz.ch/old/teaching/2010-S/0276/slides/rueegg.pdf · Introduction Motivation DOI: Degree-Of-Interest Study Subjects](https://reader033.fdocuments.us/reader033/viewer/2022042321/5f0aab297e708231d42cc096/html5/thumbnails/31.jpg)
20
Summary
I DOI is a good Indicator for structural codeKnowledge
I Improvements are still possibleAuthorship, Activity, . . .
OutlookI Results can be used to create
better Expertise Recommendation tools
I Should be repeated withmore Programmers, Languages
![Page 32: Does a Programmer’s Activity Indicate Knowledge of Code?se.inf.ethz.ch/old/teaching/2010-S/0276/slides/rueegg.pdf · Introduction Motivation DOI: Degree-Of-Interest Study Subjects](https://reader033.fdocuments.us/reader033/viewer/2022042321/5f0aab297e708231d42cc096/html5/thumbnails/32.jpg)
21
Typical QuestionsWhat the Subjects were asked. . .
Q1) Can you recall the name of one class or interface that isdirectly extended, implemented by ’TYPE’, or can yourecall the name of one class that directly extends the type’TYPE’?
Q2) Do you know the types of the parameters that are passedto the invocation of method/constructor ’METHOD’?
Q3) Do you know two methods that are called bymethod/constructor ’METHOD’?
Q4) Can you recall one method/constructor that callsmethod/constructor ’METHOD’?
![Page 33: Does a Programmer’s Activity Indicate Knowledge of Code?se.inf.ethz.ch/old/teaching/2010-S/0276/slides/rueegg.pdf · Introduction Motivation DOI: Degree-Of-Interest Study Subjects](https://reader033.fdocuments.us/reader033/viewer/2022042321/5f0aab297e708231d42cc096/html5/thumbnails/33.jpg)
22
ImprovementsWhat should be Added to the DOI
Authorship More knowledge about authored program elements.
Authorship Duration More time spent creating an element ⇒ more knowledgeabout the element.
Code Stability Fewer code changes ⇒ longer knowledge about it
Code Patterns More code patterns ⇒ easier to infer knowledge
Role Important element ⇒ more knowledge
Task Locality More interaction with other programmers’ code ⇒ lessDOI-to-knowledge correspondence.
Activity Short-term kinds of activity, like debugging, have skewingeffect.
Work Experience More experience of programmer ⇒ betterDOI-to-knowledge correspondence.