An Empirical Investigation on Documentation Usage Patterns in

Maintenance Tasks

Gabriele Gerardo Massimiliano Rocco Sebastiano Bavota Canfora Di Penta Oliveto Panichella

Context and Motivations • Software Development

Case Study • User Study with 33 Partecipants

Results •Documentation usage pattern during maintenance evolution

actvity

Software Development

Maintenance tasks are generally facilitated when software

documentation is available

Past Work

The use of UML diagram significantly improves the

functional correctness of changes

E. Arisholm et al. TSE 2006

Past Work

Help to reduce the time needed to understand ho

perform maintenance tasks by approximately 20 %

Eirik Tryggeseth EMSE 1997

How such documentation is browsed by developers to perform

maintenance activities?

Empirical Study

Goal: observe the software documents browsed by Developers during maintenance tasks Quality focus: understand the documentation usage pattern during maintenance activities Perspective: researchers interested to develop tool to improve the program comprehension

• RQ1: How much time did participants spend on different kinds of artifacts?

• RQ2: How do participants navigate

different kinds of artifacts to

identify code to be changed

during the evolution task?

Research Questions

Context • Object: software artifacts from SMOS, a school

automation system developed by graduate students at the University of Salerno (Italy).

• Subjects: 33 partecipants

11 Bachelor Students 18 Master Students 4 PhD Students

Study Procedure

Maintenance tasks

Bug Fixing:

Add a new feature:

Improve existing features:

Example: Maintenance task

Monitoring Maintenance Activity

Monitoring Maintenance Activity

TIMESTAMP

Data Collection XML log

Files

Data Collection XML log

Files

0

0,1

0,2

0,3

0,4

0,5

0,6

0,7

0,8

0,9

1 R

eca

ll

Pre

cisi

on

F-M

eas

ure

Re

call

Pre

cisi

on

F-M

eas

ure

Re

call

Pre

cisi

on

F-M

eas

ure

Undergraduates Graduates ALL

Subjects Performances

0

0,1

0,2

0,3

0,4

0,5

0,6

0,7

0,8

0,9

1 R

eca

ll

Pre

cisi

on

F-M

eas

ure

Re

call

Pre

cisi

on

F-M

eas

ure

Re

call

Pre

cisi

on

F-M

eas

ure

Undergraduates Graduates ALL

Subjects Performances

Average 0.75 FM

0

0,1

0,2

0,3

0,4

0,5

0,6

0,7

0,8

0,9

1 R

eca

ll

Pre

cisi

on

F-M

eas

ure

Re

call

Pre

cisi

on

F-M

eas

ure

Re

call

Pre

cisi

on

F-M

eas

ure

Undergraduates Graduates ALL

Subjects Performances

Average 0.75 FM

+ 5% FM for Graduate

Students

RQ1

How much time did participants

spend on different kinds of artifacts?

Javadoc

Use Case

Class Diagram

Sequence Diagram

Source Code

ALL

Graduates

Undergraduates

15%

Use of each artifacts in overall tasks

60% 33%

100%

72%

RQ1: How much time did participants spend on different kinds of artifacts?

72%

13% 10% 3% 2%

72%

13% 10% 3% 2%

RQ1: How much time did participants spend on different kinds of artifacts?

Undergraduates students used Source Code and Javadoc significantly

more than Graduate students

Undergraduate Students

72%

13% 10% 3% 2%

RQ1: How much time did participants spend on different kinds of artifacts?

Graduate Students

Graduate students used Class Diagrams significantly more than Undergraduates

Perceived Importance of Software Artifacts

1

3

4

4

2

3

4

5

4

2

0 2 4 6

Use Case

Class Diagram

Sequence Diagram

Source Code

Javadoc

Graduates

Undergraduates

Perceived Importance of Software Artifacts

1

3

4

4

2

3

4

5

4

2

0 1 2 3 4 5 6

Use Case

Class Diagram

Sequence Diagram

Source Code

Javadoc

Graduates

Undergraduates

Sequence diagram represents a fair compromise between use cases (too abstracts) and class diagrams (providing useless details about an entire subsystem)..

Perceived Importance of Software Artifacts

1

3

4

4

2

3

4

5

4

2

0 1 2 3 4 5 6

Use Case

Class Diagram

Sequence Diagram

Source Code

Javadoc

Graduates

Undergraduates

Sequence diagram represents a fair compromise between use cases (too abstracts) and class diagrams (providing useless details about an entire subsystem)..

With the other sources of documentation available Javadoc became useless to identify impacted components

Summary RQ1

Summary RQ1 1) Participants spent more time to analyze low-level artifacts

as compared to high-level artifacts 2) Participants consider the most important sources of

information sequence and class diagrams 3) Undergraduate students spent a significantly higher

proportion of time on source code than graduate students that, instead, spent more time on class diagrams

Summary RQ1 1) Participants spent more time to analyze low-level artifacts

as compared to high-level artifacts 2) Participants consider the most important sources of

information sequence and class diagrams 3) Undergraduate students spent a significantly higher

proportion of time on source code than graduate students that, instead, spent more time on class diagrams

Summary RQ1 1) Participants spent more time to analyze low-level artifacts

as compared to high-level artifacts 2) Participants consider the most important sources of

information sequence and class diagrams 3) Undergraduate students spent a significantly higher

proportion of time on source code than graduate students that, instead, spent more time on class diagrams

RQ2

How do participants navigate different kinds

of artifacts to identify code to be changed

during the evolution task?

3%

Starting Point: Graduate vs Undergraduate students

12% 17%

25%

42%

Starting Point: Graduate vs Undergraduate

Undergraduate

Graduate

58%

32% 24%

26%

9%

22%

3%

17%

4% 3%

18%

8%

2%

2%

1%

1%

3%

0%

1%

0%

4%

12%

16%

12%

10%

4%

4%

1%

2%

1%

1%

5%

S

D

(SD)+

(US)+

U(SD)+

(DS)+

J

U

S(US)+

SU(SD)+

Other

Graduate

Undegraduate

S= Sequence Diagram

D= Class Diagram

U= Use Case

J= Javadoc

Most frequents patterns before reaching source code

18%

8%

2%

2%

1%

1%

3%

0%

1%

0%

4%

12%

16%

12%

10%

4%

4%

1%

2%

1%

1%

5%

0% 2% 4% 6% 8% 10% 12% 14% 16% 18% 20%

S

D

(SD)+

(US)+

U(SD)+

(DS)+

J

U

S(US)+

SU(SD)+

Other

Graduate

Undegraduate

S= Sequence Diagram

D= Class Diagram

U= Use Case

J= Javadoc

Most frequents patterns before reaching source code

18%

8%

2%

2%

1%

1%

3%

0%

1%

0%

4%

12%

16%

12%

10%

4%

4%

1%

2%

1%

1%

5%

0% 2% 4% 6% 8% 10% 12% 14% 16% 18% 20%

S

D

(SD)+

(US)+

U(SD)+

(DS)+

J

U

S(US)+

SU(SD)+

Other

Graduate

Undegraduate

More experienced participants use a more “Integrated approach”

S= Sequence Diagram

D= Class Diagram

U= Use Case

J= Javadoc

Most frequents patterns before reaching source code

Source Code

Sequence Diagram

Javadoc

Transition Graph between kinds of Software Artifacts

Source Code

Sequence Diagram

Javadoc

1) From Source Code participants in most cases “go back” to

Sequence and Class Diagrams

2) From Sequence and Class Diagrams

participants in most cases “go back” to

Source Code

3) Starting from a Use Case, participants go

ahead reading Sequence Diagrams. Only after,

they reading and writing Source Code

Transition Graph between kinds of Software Artifacts

0%

20%

11%

30%

16%

61%

41%

70%

12%

15%

60%

24%

58%

54%

0% 10% 20% 30% 40% 50% 60% 70% 80%

USDC

USD

USC

UDC

SDC

SC

DC

Graduates

Undergraduates

1) Developers never used Javadoc

Most Frequents Patterns

2) Patterns end always with Source Code

S= Sequence Diagram

D= Class Diagram

U= Use Case

J= Javadoc

C= Source Code

Summary RQ2 • 1) Participants tend to start the assigned task from source

code or from design documents , i.e., class and sequence diagrams

• 2) More experienced participants tend to follow a more

integrated approach than less experienced Ones • 3) During tasks, participants tend to go back and forth

repeatedly between source code and to design diagrams (sequence and class diagrams).

…Conclusions…

…Conclusions…

…Conclusions…

…Conclusions…

…Conclusions…

…Future Directions…