Who Should Review My Code?

Who Should Review My Code?A File Location-Based Code-Reviewer Recommendation

Approach for Modern Code Review

Patanamon (Pick) Thongtanunam

Chakkrit Tantithamtavorn Raula G. Kula Norihiro

Yoshida Hajimu Iida Ken-ichi Matsumoto

[email protected] @pamon

22nd IEEE International Conference on Software Analysis, Evolution, and Reengineering

Montréal, Quebec, Canada March 3 - 6, 2015

1

mailto:[email protected]

Modern Code Review is a tool-based, and used regularly in practice nowadays at industrial and OSS projects

2

Gerrit Code Review

Rietveld

Modern Code Review is a tool-based, and used regularly in practice nowadays at industrial and OSS projects

2

Modern Code Review

Developer

3

Modern Code Review

DeveloperCode change

Gerrit Code Review

3

Modern Code Review

Reviewers

Developer

Can you review my code?

Code change

Gerrit Code Review

Invite Reviewers

3

Modern Code Review

Coding style passed :) There’s a

bug!

Reviewers

Developer

Can you review my code?

Code change

Gerrit Code Review

Invite Reviewers

3

Code-Reviewer Assignment Problem

Code change

Gerrit Code Review

Invite Reviewers

Developer

4

Code-Reviewer Assignment ProblemWho should review

my code?

Code change

Gerrit Code Review

Invite Reviewers

New Developer

4

RQ1: How do reviews with code-reviewer assignment problem impact reviewing time?

5


Studied Projects

5


Studied Projects

Sample Reviews 357 363 378 363

5


Studied Projects

Sample Reviews 357 363 378 363#Reviews w/

Assignment Problem 10% 5% 30% 4%

5


Studied Projects


Assignment Problem 10% 5% 30% 4%Reviews with code−reviewer assignment problem Reviews without code−reviewer assignment problem

Studied systems

Rev

iew

ing

time

(Day

s)

Android OpenStack Qt LibreOffice

010

2030

Reviews with code−reviewer assignment problem Reviews without code−reviewer assignment problem

Studied systems

Rev

iew

ing

time

(Day

s)


010

2030

Reviewing Time (Days)


Studied systems

Rev

iew

ing

time

(Day

s)


010

2030


Studied systems

Rev

iew

ing

time

(Day

s)


010

2030

Review w/ Assignment Problem Review w/o Assignment Problem

18 Days Longer 13 Days Longer 9 Days Longer 6 Days Longer

5


Studied Projects


Assignment Problem 10% 5% 30% 4%

Reviews with code-reviewer assignment problem significantly take 12 days longer to approve a code change.


Studied systems

Rev

iew

ing

time

(Day

s)


010

2030


Studied systems

Rev

iew

ing

time

(Day

s)


010

2030



Studied systems

Rev

iew

ing

time

(Day

s)


010

2030


Studied systems

Rev

iew

ing

time

(Day

s)


010

2030



5

6


A code-reviewer recommendation tool will be useful in distributed software development to speed up the code review process.

6


RevFinder: A File Location-based Code-Reviewer Recommendation Approach

7

RevFinder: Overview

R1 R2

Changed Files- src/video/camera.java - src/video/player.javaCode-Reviewers

Reviewed by John

Changed Files- src/devices/screen.java

Code-Reviewers

Reviewed by Pat

Review History

8

RevFinder: Overview

R1 R2


Reviewed by John


Code-Reviewers

Reviewed by Pat

R3 (New)

Changed Files

- src/video/recorder.java

Code-Reviewers

?

Review History

8

RevFinder: Overview

R1 R2


Reviewed by John


Code-Reviewers

Reviewed by Pat

R3 (New)

Changed Files


Code-Reviewers

?

RevFinder

Review History

Find reviewers who have reviewed the most similar file paths

8

RevFinder: Overview

R1 R2


Reviewed by John


Code-Reviewers

Reviewed by Pat

R3 (New)

Changed Files


Code-Reviewers

?

RevFinder

Review History

8

RevFinder: Overview

R1 R2


Reviewed by John


Code-Reviewers

Reviewed by Pat

R3 (New)

Changed Files


Code-Reviewers

?

RevFinder

Compare

Compare(FilePath_R2,FilePath_R3) = 0.3

Review History

8

RevFinder: Overview

R1 R2


Reviewed by John


Code-Reviewers

Reviewed by Pat

R3 (New)

Changed Files


Code-Reviewers

?

RevFinder



Review History

8

Compare

RevFinder: Overview

R1 R2


Reviewed by John


Code-Reviewers

Reviewed by Pat

R3 (New)

Changed Files


Code-Reviewers

?

RevFinder

Review History



RevFinder: Overview

R1 R2


Reviewed by John


Code-Reviewers

Reviewed by Pat

R3 (New)

Changed Files


Code-Reviewers

?

RevFinder

Review History


“Pat should review R3”9

RevFinder: FilePath Comparison Techniques

Longest Common Prefix

Longest Common Suffix

Longest Common Substring

Longest Common Subsequence

4 String Comparison Techniques

10





Longest Common Subsequence Recommendation List

Combine (Borda Count)


10








Voting Ranks

10








Voting Ranks

10

Bubble-up correct reviewers

Empirical Evaluation

Studied Projects

Period ~4 years ~1year ~1 year ~2 year

#Reviews 5,126 6,586 23,810 6,523#Reviewers 94 82 202 63

11


Studied Projects



Baseline Approach: ReviewBot (Balachandran ICSE’13)

R1 R2R3 (New)

JohnPat

“Recommend reviewers who have reviewed the same line of code changes”

11


Studied Projects



Baseline Approach: ReviewBot (Balachandran ICSE’13)

R1 R2R3 (New)

JohnPat

“Recommend reviewers who have reviewed the same line of code changes”

John should review

R3

11


12


Accuracy

“RQ2: Does RevFinder accurately recommend reviewers?”

12


Accuracy


Ranking Performance

“RQ3: Does RevFinder provide a better ranking of correct reviewers?”

12


Accuracy


Measure: Top-k Accuracy

Ranking Performance

“RQ3: Does RevFinder provide a better ranking of correct reviewers?”

12

Empirical Evaluation: Accuracy

Top-k Accuracy measures how many reviews does an approach can correctly recommend with a Top-k recommendation.

13



13

Top-5 Recommendation



13


Actual Reviewer



13


Actual Reviewer

Correct!



Top-

10 A

ccur

acy

0%

22.5%

45%

67.5%

90%

10%

28%41%

29%

74%69%

87%86%

RevFinder ReviewBot

RevFinder can correctly recommended 69%-87% of reviews with a top-10 recommendation. RevFinder is 4 times more accurate than ReviewBot.

13


Actual Reviewer

Correct!


Accuracy

“Does RevFinder accurately recommend reviewers?”


Ranking Performance

“Does RevFinder provide a better ranking of correct reviewers?”

14


Accuracy



Ranking Performance


14

On average, RevFinder can correctly recommend reviewers for 79% of

reviews with a top-10 recommendation.


Accuracy



Ranking Performance


14



Measure: Mean Reciprocal Rank (MRR)

Empirical Evaluation: Ranking Performance

Mean Reciprocal Rank measures an average rank of correct reviewer that an approach can recommend.

15



15

Recommendation Lists

vs

Actual Reviewer



15


vs

Actual Reviewer

MRR = 1/3



15


vs

Actual Reviewer

MRR = 1/3

MRR = 1/1



RevFinder recommended the correct code-reviewers with a median rank of 2 - 8. The code-reviewers ranking of RevFinder

is 3 times better than that of ReviewBot.15


vs

Actual Reviewer

MRR = 1/3

MRR

RevFinder 0.6 0.55 0.31 0.40

ReviewBot 0.25 0.30 0.22 0.07

Ran

k of

C

orre

ct R

evie

wer

s Android OpenStack Qt LibreOffice●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●●

●●●●●●●●

●

●

●

●●●●●

●●●

●●●●●●●●●●●●

●

●●●●●●

●●●

●●●●

●

●

●●●

●●●●●●

●

●●●●●

●

●●

●

●●●●●●●

●

●●●●

●

●●

●

●●●●●●●●●●

●

●●●

●

●●●

●

●●●●●●

●

●

●

●●

●

●●●

●

●

●●

●

●●●

●●

●●

●

●

●●●●

●

●●●●●

●●

●

●

●●

●

●●●●●●●●●●●●

●

●

●●●●

●

●●

●

●●●●●●●●●

●

●

●

●●●

●

●

●

●

●

●●●

●

●●

●

●

●●

●●●●●

●

●●●

●

●

●●

●

●●

●

●

●●●●

●

●●●●●●●

●●

●●●●●

●●

●

●

●●●●●●●●●

●

●●●●

●

●

●●●●●●●●●●●●●●

●●

●

●

●●

●

●●

●

●●

●

●

●●

●●

●●●●●●

●

●●●

●●

●

●

●

●

●●●

●

●●●●●●●●

●

●●●●●●

●●

●●

●

●●●●

●

●

●

●●●

●

●

●

●

●

●●

●

●●●

●

●

●●●

●

●

●●●●●

●

●

●

●●●●

●

●

●

●

●●●●●●●●●●

●

●

●

●●●

●●●

●●●●●●●●●●●

●

●

●

●

●●●●

●

●●

●●

●

●

●

●

●●●●●●●

●

●

●

●

●●

●●

●

●

●

●

●

●●●●●

●

●●●

●

●●●●●

●

●

●

●

●●●

●

●

●

●●●●●●●

●

●

●

●●

●

●●●●●

●●

●

●●●●

●

●●●●●

●

●

●●

●

●

●

●●●●●

●

●●●●●●

●

●●●

●●

●●●●

●

●●

●

●

●●

●

●

●●●●●

●

●

●●

●

●●●●●●●●●●

●

●●●●●

●●

●●●

●●

●●

●

●

●●

●

●

●

●

●●

●

●●

●●●●●

●1

515

50100200

RevFinder ReviewBot RevFinder ReviewBot RevFinder ReviewBot RevFinder ReviewBotApproach

Rank

RevFinder ReviewBot RevFinder ReviewBot RevFinder ReviewBot RevFinder ReviewBot

MRR = 1/1


Accuracy



Ranking Performance


16





Accuracy



Ranking Performance


16




On average, RevFinder can recommend reviewers with a median rank of 4.


Accuracy



Ranking Performance


16





We can use a similarity of previously reviewed file paths to recommend code-reviewers for new code changes.

Summary

17


my code?

Code change

Gerrit Code Review

Invite Reviewers

New Developer

4


my code?

Code change

Gerrit Code Review

Invite Reviewers

New Developer

4

5

How do reviews with code-reviewer assignment problem impact reviewing time?

Studied Projects


#Reviews w/ Assignment Problem 10% 5% 30% 4%



Studied systems

Revie

wing

tim

e (D

ays)


010

2030


Studied systems

Revie

wing

tim

e (D

ays)


010

2030



Studied systems

Revie

wing

tim

e (D

ays)


010

2030


Studied systems

Revie

wing

tim

e (D

ays)


010

2030




my code?

Code change

Gerrit Code Review

Invite Reviewers

New Developer

4 5


Studied Projects





Studied systems

Revie

wing

tim

e (D

ays)


010

2030


Studied systems

Revie

wing

tim

e (D

ays)


010

2030



Studied systems

Revie

wing

tim

e (D

ays)


010

2030


Studied systemsRe

viewi

ng ti

me

(Day

s)


010

2030



5


Studied Projects





Studied systems

Revie

wing

tim

e (D

ays)


010

2030


Studied systems

Revie

wing

tim

e (D

ays)


010

2030



Studied systems

Revie

wing

tim

e (D

ays)


010

2030


Studied systemsRe

viewi

ng ti

me

(Day

s)


010

2030




my code?

Code change

Gerrit Code Review

Invite Reviewers

New Developer

4

7

RevFinder: Overview

R1 R2


Reviewed by John

Changed Files

- src/devices/screen.java

Code-Reviewers

Reviewed by Pat

R3 (New)

Changed Files


Code-Reviewers

?

RevFinder

Review History


5


Studied Projects





Studied systems

Revie

wing

tim

e (D

ays)


010

2030


Studied systems

Revie

wing

tim

e (D

ays)


010

2030



Studied systems

Revie

wing

tim

e (D

ays)


010

2030


Studied systemsRe

viewi

ng ti

me

(Day

s)


010

2030



7

RevFinder: Overview

R1 R2


Reviewed by John

Changed Files


Code-Reviewers

Reviewed by Pat

R3 (New)

Changed Files


Code-Reviewers

?

RevFinder

Review History



my code?

Code change

Gerrit Code Review

Invite Reviewers

New Developer

4


Accuracy



Ranking Performance


15






5


Studied Projects





Studied systems

Revie

wing

tim

e (D

ays)


010

2030


Studied systems

Revie

wing

tim

e (D

ays)


010

2030



Studied systems

Revie

wing

tim

e (D

ays)


010

2030


Studied systemsRe

viewi

ng ti

me

(Day

s)


010

2030



7

RevFinder: Overview

R1 R2


Reviewed by John

Changed Files


Code-Reviewers

Reviewed by Pat

R3 (New)

Changed Files


Code-Reviewers

?

RevFinder

Review History



Accuracy



Ranking Performance


15







my code?

Code change

Gerrit Code Review

Invite Reviewers

New Developer

4

Who Should Review My Code?

Presentations & Public Speaking

Transcript of Who Should Review My Code?