The Impact of Code Review Coverage and Participation on Software Quality
-
Upload
shane-mcintosh -
Category
Software
-
view
593 -
download
0
description
Transcript of The Impact of Code Review Coverage and Participation on Software Quality
The Impact of Code Review Coverage and Participation
on Software Quality
Shane McIntosh
Ahmed E. Hassan
Bram Adams
Yasutaka Kamei
Patch
+ + - -Patch
+ + - -
2
The modern, tool-supported code review process
GerritShane
Bram
Yasu
Patch
+ + - -
Patch
+ + - -
2
The modern, tool-supported code review process
GerritShane
Bram
Yasu
Patch
+ + - -
Patch
+ + - -
2
The modern, tool-supported code review process
Gerrit
Request review
Shane
Bram
Yasu
Patch
+ + - -
Patch
+ + - -
2
The modern, tool-supported code review process
Gerrit
Watch about NULL values at line 3!
Shane
Bram
Yasu
Patch
+ + - -
Patch
+ + - -
2
The modern, tool-supported code review process
Gerrit
Your code sucks!
Watch about NULL values at line 3!
Shane
Bram
Yasu
Patch
+ + - - Patch
+ + - -
2
The modern, tool-supported code review process
Gerrit
Your code sucks!
Watch about NULL values at line 3!
Shane
Bram
Yasu
The modern, tool-supported code review process
3
GerritShane
Bram
Yasu
The modern, tool-supported code review process
3
GerritShane
Bram
Yasu
The modern, tool-supported code review process
3
Upstream
GerritShane
Bram
Yasu
The modern, tool-supported code review process
3
Upstream
GerritShane
Bram
Yasu
Changes are automatically integrated after review
criteria are satisfied
Traditional!Code Inspections
Mandated reviewer checklists and in-person
meetings
Traditional!Code Inspections
Mandated reviewer checklists and in-person
meetings
Measurably!improves!software!quality
Traditional!Code Inspections
Mandated reviewer checklists and in-person
meetings
Modern!Code Review
Lightweight, tool-supported
Measurably!improves!software!quality
Traditional!Code Inspections
Mandated reviewer checklists and in-person
meetings
Modern!Code Review
Lightweight, tool-supported
Measurably!improves!software!quality
???
Do lax reviewing practices have an impact on software quality?
Expert code!reviewer
6
Upstream
Development
Feature developmentDefect repairing
Commit types
Reviews can be linked to commits in version control systems
6
Upstream
Development
Feature developmentDefect repairing
Commit types
Reviews can be linked to commits in version control systems
7
commit b5b46f398c1a04ab443bf22ec08c5efbf72c161e Author: Alan Alpert <[email protected]> Date: Thu Mar 1 20:09:52 2012 +1000 ! Add a hidden XandYAxis enum value For Qt 5 XandYAxis is being renamed to XAndYAxis to more consistently follow capitalization rules. Add an undocumented XandYAxis variable to ease porting. Change-Id: Id9e41dd5578373f5f557937da889a9326ff12e53 Reviewed-by: Martin Jones <[email protected]>
Gerrit
Upstream
Reviews can be linked to commits in version control systems
7
commit b5b46f398c1a04ab443bf22ec08c5efbf72c161e Author: Alan Alpert <[email protected]> Date: Thu Mar 1 20:09:52 2012 +1000 ! Add a hidden XandYAxis enum value For Qt 5 XandYAxis is being renamed to XAndYAxis to more consistently follow capitalization rules. Add an undocumented XandYAxis variable to ease porting. Change-Id: Id9e41dd5578373f5f557937da889a9326ff12e53 Reviewed-by: Martin Jones <[email protected]>
Gerrit
Upstream
Reviews can be linked to commits in version control systems
7
commit b5b46f398c1a04ab443bf22ec08c5efbf72c161e Author: Alan Alpert <[email protected]> Date: Thu Mar 1 20:09:52 2012 +1000 ! Add a hidden XandYAxis enum value For Qt 5 XandYAxis is being renamed to XAndYAxis to more consistently follow capitalization rules. Add an undocumented XandYAxis variable to ease porting. Change-Id: Id9e41dd5578373f5f557937da889a9326ff12e53 Reviewed-by: Martin Jones <[email protected]>
Gerrit
Upstream
Reviews can be linked to commits in version control systems
7
commit b5b46f398c1a04ab443bf22ec08c5efbf72c161e Author: Alan Alpert <[email protected]> Date: Thu Mar 1 20:09:52 2012 +1000 ! Add a hidden XandYAxis enum value For Qt 5 XandYAxis is being renamed to XAndYAxis to more consistently follow capitalization rules. Add an undocumented XandYAxis variable to ease porting. Change-Id: Id9e41dd5578373f5f557937da889a9326ff12e53 Reviewed-by: Martin Jones <[email protected]>
Gerrit
Upstream
Reviews can be linked to commits in version control systems
8
The Baseline
8
Product Metrics
The Baseline
8
Product Metrics
e.g., SLOC, complexity
The Baseline
8
Product Metrics
Process Metrics e.g., SLOC, complexity
The Baseline
8
Product Metrics
Process Metrics e.g., SLOC, complexity
e.g., Prior defects, churn
The Baseline
8
Product Metrics
Process Metrics
Human Factor Metrics
e.g., SLOC, complexity
e.g., Prior defects, churn
The Baseline
8
Product Metrics
Process Metrics
Human Factor Metrics
e.g., SLOC, complexity
e.g., Prior defects, churn
e.g., # Minor contributors
The Baseline
9
The Context
The Context
1,339 components254 defect-prone
v5.0.0
218 components24 defect-prone
v4.3.0170 components15 defect-prone
v5.10.0
1,337 components187 defect-prone
v5.1.0
10
ParticipationCoverage
Do lax reviewing practices have an impact on software quality?
ParticipationCoverage
Do lax reviewing practices have an impact on software quality?
13
Measuring review coverage using the proportion of reviewed changes
13
Upstream Development
13
Measuring review coverage using the proportion of reviewed changes
13
Upstream Development
Gerrit
13
Measuring review coverage using the proportion of reviewed changes
13
Upstream Development
Gerrit
7 out of 10 changes are!linked to code reviews
14
Concrete Predicted Defect Count
Defect Model
Create artificial component setting explanatory variables to median values
Artificial Comp.
500SLOC
Explanatory!Variable Value
Reviewed changes 100%
Churn 150
15
Defect Model
Calculate number of predicted defects
Artificial Comp.
500SLOC
Explanatory!Variable Value
Reviewed changes
Churn 150
Reviewed changes
Pred
icte
d D
efec
t Cou
nt
Concrete Predicted Defect Count
100%
15
Defect Model
Calculate number of predicted defects
Artificial Comp.
500SLOC
Explanatory!Variable Value
Reviewed changes
Churn 150
Reviewed changes
0
Pred
icte
d D
efec
t Cou
nt
Concrete Predicted Defect Count
100%
15
Defect Model
Calculate number of predicted defects
Artificial Comp.
500SLOC
Explanatory!Variable Value
Reviewed changes
Churn 150
Reviewed changes
0
Pred
icte
d D
efec
t Cou
nt
Concrete Predicted Defect Count
100%
16
Defect Model
Artificial Comp.
500SLOC
Explanatory!Variable Value
Reviewed changes 90%
Churn 150
Modify variable under test while keeping other explanatory variables constant
Reviewed changes
Pred
icte
d D
efec
t Cou
nt
Concrete Predicted Defect Count
16
Defect Model
Artificial Comp.
500SLOC
Explanatory!Variable Value
Reviewed changes 90%
Churn 150
Modify variable under test while keeping other explanatory variables constant
Reviewed changes
Pred
icte
d D
efec
t Cou
nt
Concrete Predicted Defect Count
16
Defect Model
Artificial Comp.
500SLOC
Explanatory!Variable Value
Reviewed changes 90%
Churn 150
1
Modify variable under test while keeping other explanatory variables constant
Reviewed changes
Pred
icte
d D
efec
t Cou
nt
Concrete Predicted Defect Count
16
Defect Model
Artificial Comp.
500SLOC
Explanatory!Variable Value
Reviewed changes 90%
Churn 150
1
Modify variable under test while keeping other explanatory variables constant
Reviewed changes
Pred
icte
d D
efec
t Cou
nt
Concrete Predicted Defect Count
● ●
● ● ● ● ● ● ● ●
● ● ● ● ● ● ●0
1
2
0.0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1.0Proportion of Reviewed Changes
Con
cret
e Pr
edic
ted
Def
ect C
ount
● Qt 5.0.0VTK 5.10.0
17
Review coverage doesn’t tell the whole story
● ●
● ● ● ● ● ● ● ●
● ● ● ● ● ● ●0
1
2
0.0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1.0Proportion of Reviewed Changes
Con
cret
e Pr
edic
ted
Def
ect C
ount
● Qt 5.0.0VTK 5.10.0
17
Review coverage doesn’t tell the whole story
Not a significant factor for Qt 5.1.0 or ITK
ParticipationCoverage
Do lax reviewing practices have an impact on software quality?
ParticipationCoverage
Do lax reviewing practices have an impact on software quality?
!Coverage is linked to
quality in two of the four studied systems, yet it doesn’t tell the whole
story
ParticipationCoverage
Do lax reviewing practices have an impact on software quality?
!Coverage is linked to
quality in two of the four studied systems, yet it doesn’t tell the whole
story
20
Measuring review participation using proportions of lax reviews
I❤️ Me
Self-approved changes
20
Measuring review participation using proportions of lax reviews
I❤️ Me
Self-approved changes
Hastily-reviewed changes
20
Measuring review participation using proportions of lax reviews
I❤️ Me
Self-approved changes
Hastily-reviewed changes
Changes without discussion
21
Measuring review participation using proportions of lax reviews
I❤️ Me
Self-approved changes
Hastily-reviewed changes
Changes without discussion
● ● ●
● ●
● ● ●
● ● ●
● ● ●
● ● ●
0
1
2
3
4
5
0.0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1.0Proportion of Self−Approved Changes
Con
cret
e Pr
edic
ted
Def
ect C
ount
22
Lax participation leads to many post-release defects in our models
● ● ●
● ●
● ● ●
● ● ●
● ● ●
● ● ●
0
1
2
3
4
5
0.0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1.0Proportion of Self−Approved Changes
Con
cret
e Pr
edic
ted
Def
ect C
ount
22
5 additional post-release defects
Lax participation leads to many post-release defects in our models
● ● ●
● ●
● ● ●
● ● ●
● ● ●
● ● ●
0
1
2
3
4
5
0.0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1.0Proportion of Self−Approved Changes
Con
cret
e Pr
edic
ted
Def
ect C
ount
22
5 additional post-release defects
Small increase leads to defects
Lax participation leads to many post-release defects in our models
ParticipationCoverage
Do lax reviewing practices have an impact on software quality?
!Coverage is linked to
quality in two of the four studied systems, yet it doesn’t tell the whole
story
ParticipationCoverage
Do lax reviewing practices have an impact on software quality?
!Coverage is linked to
quality in two of the four studied systems, yet it doesn’t tell the whole
story
!Components w/ lax
review participation are estimated to have several additional post-release
defects