III Conferência CMMI Portugal, Presentation 6: Process Improvement in High Maturity Environments,...

Portugal

Process Improvement in High Maturity

Environments

Improving the Efficiency of Code Reviews

André Ferreira

Software Quality Assurance Engineer

Critical Software S.A.

2013-10-18

2 © 2

013 C

MM

I P

ort

ug

al C

onfe

rence S

eri

es –

All R

ights

reserv

ed.

Outline

1. SPI success - evaluation and measurement

2. CMMI High Maturity Processes Areas - a framework to drive

SPI initiatives

3. SPI initiative at Critical Software – Improving the efficiency of

Code Reviews

4. Recommendations/guidelines to improve the probability of

success of SPI initiatives.

3 © 2

013 C

MM

I P

ort

ug

al C

onfe

rence S

eri

es –

All R

ights

reserv

ed.

SPI success: reports on SPI Success.

Performance Results of CMMI-Based Process Improvement –Technical Report

CMU/SEI-2006-TR-004 ESC-TR-2006-004

Performance category

Medium Improvement

# Data points

Lowest Improvement

Highest Improvement

Cost 34% 29

3% 87%

Schedule 50% 22 2% 95%

Productivity 61% 20 11% 329%

Quality 48% 34 2% 132%

Customer Satisfaction

14% 7 -4% 55%

Return on Investment

[4.0:1] 22 [1.7:1] [27.7:1]

4 © 2

013 C

MM

I P

ort

ug

al C

onfe

rence S

eri

es –

All R

ights

reserv

ed.

SPI success: evaluation of SPI Success - I

Evaluation and validity of SPI success is questionable.

Hard to establish a clear relation of SPI initiative and SPI

results. Results are not questionable by themselves, but a clear

understanding of how success relates to SPI is unclear, mainly as result of:

1. Confounding factors – not considering relevant factors, validity threat

2. Incomplete context descriptions – context of SPI not fully described

3. Evaluation validity – threats to validity not discussed

4. Measurement validity – measurement definition inconsistencies

5. Measurement scope – missing the big picture

Unterkalmsteiner et.al. (2011). Evaluation and Measurement of Software Process Improvement -

A Systematic Literature Review. IEEE Transactions on Software Engineering

5 © 2

013 C

MM

I P

ort

ug

al C

onfe

rence S

eri

es –

All R

ights

reserv

ed.


Y(SPI sucess) = f ( f1, f2, f,3)

Causation

established

6 © 2

013 C

MM

I P

ort

ug

al C

onfe

rence S

eri

es –

All R

ights

reserv

ed.


Y(SPI sucess) = f ( f1, f2, f,3)

Causation

established

Presence of

confounding factors Y’(SPI sucess) = f’ (?, ?, f1, f2, f3, ?)

7 © 2

013 C

MM

I P

ort

ug

al C

onfe

rence S

eri

es –

All R

ights

reserv

ed.

SPI success: evaluation of SPI Success - II

Evaluation strategies for SPI initiatives:

Unterkalmsteiner et.al. (2011). Evaluation and Measurement of Software Process Improvement -

A Systematic Literature Review. IEEE Transactions on Software Engineeringl

Name Percentage of studies

Pre-post Comparison 49%

Statistical Analysis 15%

Pre-post Comparisons & Survey 7%

Statistical Process Control 5%

Coat benefit analysis 3%

Not identified 14%

8 © 2

013 C

MM

I P

ort

ug

al C

onfe

rence S

eri

es –

All R

ights

reserv

ed.

CMMI High Maturity Framework Overview

Support

Pro

ce

ss

Ma

na

ge

me

nt

Pro

ject

Ma

na

ge

me

nt

CAR OPP

QPM

OPM BO

QPPOs PPBs

Defects

OIP

Corrective Actions Project QPPOs

Performance Issues

9 © 2

013 C

MM

I P

ort

ug

al C

onfe

rence S

eri

es –

All R

ights

reserv

ed.

SPI Initiative

Improving the Efficiency of

Code Reviews

10 © 2

013 C

MM

I P

ort

ug

al C

onfe

rence S

eri

es –

All R

ights

reserv

ed.

SPI Initiative – Opportunity Identification OPM.SP 1.1/OPM.SP1.2

1. Annual Review Business Objectives

Business objective: Improve organizational profit margins

QPPOs: Improve project profit margins – do the

same/better for less

2. Feedback from projects

Issues with review procedure – cumbersome - to much

time managing review records.

Lead to establish a performance baseline of total

appraisal cost for code reviews.

11 © 2

013 C

MM

I P

ort

ug

al C

onfe

rence S

eri

es –

All R

ights

reserv

ed.

SPI Initiative – Opportunity Identification OPM.SP1.2/OPM.SP 1.3

Analysis of performance data

Feedback from projects was confirmed to be an

issue – statistical analysis of a control chart.

Code review is a - fat process!

Opportunity identified:

Decrease appraisal cost of performing code

review procedure by:

Improving efficiency (less effort) but control

impact on effectiveness

12 © 2

013 C

MM

I P

ort

ug

al C

onfe

rence S

eri

es –

All R

ights

reserv

ed.

(A) Appraisal and

(P) Preventive

costs

(I) Internal

and (E)

External

failure costs

Quality

Cost

Low

High

100%

good

100%

defective

Total cost of

quality

SPI Initiative – Potential Improvement I OPM.SP 1.3

Cost of Quality (CoQ) Management model

13 © 2

013 C

MM

I P

ort

ug

al C

onfe

rence S

eri

es –

All R

ights

reserv

ed.

(A) Appraisal and

(P) Preventive

costs

(I) Internal

and (E)

External

failure costs

Quality

Cost

Low

High

100%

good

100%

defective

Total cost of

quality


Cost of Quality (CoQ) Management model

14 © 2

013 C

MM

I P

ort

ug

al C

onfe

rence S

eri

es –

All R

ights

reserv

ed.


BO

QPPOs

Processes

Pro

ce

ss M

an

ag

em

en

t

Project Gross

Margin >= Target

EBT

Rework Verification/ Code reviews

Rationale of impact on QPPOs

15 © 2

013 C

MM

I P

ort

ug

al C

onfe

rence S

eri

es –

All R

ights

reserv

ed.


BO

QPPOs

Processes

Pro

ce

ss M

an

ag

em

en

t

Project Gross

Margin >= Target

EBT

Rework Verification/ Code reviews

Rationale of impact on QPPOs

Decrease on overall CoQ will

have a positive impact in the

ability of projects to meet

Project Gross Margin (QPPO)

16 © 2

013 C

MM

I P

ort

ug

al C

onfe

rence S

eri

es –

All R

ights

reserv

ed.

SPI Initiative – Potential Improvement II OPM.SP 1.3

Hypothesis1: Less effort spent performing code reviews will

contribute to a reduction in total appraisal cost.

Appraisal and

preventive

costs

Internal and

external failure

costs

Quality

Cost

Low

High

100%

good

100%

defective

Total cost of

quality

17 © 2

013 C

MM

I P

ort

ug

al C

onfe

rence S

eri

es –

All R

ights

reserv

ed.




Appraisal and

preventive

costs

Internal and

external failure

costs

Quality

Cost

Low

High

100%

good

100%

defective

Total cost of

quality

18 © 2

013 C

MM

I P

ort

ug

al C

onfe

rence S

eri

es –

All R

ights

reserv

ed.




Appraisal and

preventive

costs

Internal and

external failure

costs

Quality

Cost

Low

High

100%

good

100%

defective

Total cost of

quality

19 © 2

013 C

MM

I P

ort

ug

al C

onfe

rence S

eri

es –

All R

ights

reserv

ed.


Hypothesis1: Less effort spent performing each code review will

contribute to a reduction in total appraisal cost

Rationale of estimation of potential benefit

(1) Assuming a 10x decrease on managing

review records -> x hours saved per review

(2) y number of reviews performed each year

Total benefit (TH1)

TH1(hours) = x*y

TH1(€) = ~4K€/year

20 © 2

013 C

MM

I P

ort

ug

al C

onfe

rence S

eri

es –

All R

ights

reserv

ed.

SPI Initiative – Potential Improvement III OPM.SP 1.3

Hypothesis 2: Less effort per code review can be used to

perform additional code reviews (re-invest in Appraisal Cost)

Motivate an increase of # code reviews in projects with little

investment in code reviews

Motivate re-investment of effort saved to perform additional

code reviews (increase code coverage)

...with the final objective of decrease Total Cost of Quality by

decreasing considerably the Internal and External Failure

costs.

21 © 2

013 C

MM

I P

ort

ug

al C

onfe

rence S

eri

es –

All R

ights

reserv

ed.



perform additional code reviews (Increase Appraisal Costs)

Appraisal and

preventive

costs

Internal and

external failure

costs

Quality

Cost

Low

High

100%

good

100%

defective

Total cost of

quality

22 © 2

013 C

MM

I P

ort

ug

al C

onfe

rence S

eri

es –

All R

ights

reserv

ed.




Appraisal and

preventive

costs

Internal and

external failure

costs

Quality

Cost

Low

High

100%

good

100%

defective

Total cost of

quality

23 © 2

013 C

MM

I P

ort

ug

al C

onfe

rence S

eri

es –

All R

ights

reserv

ed.




Appraisal and

preventive

costs

Internal and

external failure

costs

Quality

Cost

Low

High

100%

good

100%

defective

Total cost of

quality

24 © 2

013 C

MM

I P

ort

ug

al C

onfe

rence S

eri

es –

All R

ights

reserv

ed.


Estimation of potential benefit Based on performance baselines it takes 5x more to fix a

defect at ST when compared to code reviews

Projects who invest in code reviews, on average, have less

defects/LOC found in ST (corroborates the model) with a

typical distribution of [74%/24%] distribution of code reviews

VS. ST defects

Projects who invest less in code reviews have [5%/94%]

defect distribution

In the scenario of these projects with lower investment in code review

shifting to a bigger investment in code reviews, having an approximation

to a ratio of [74%/24/] we estimated potential savings of 70K€ for a year

timeframe.

TH2(€) = 70K€

25 © 2

013 C

MM

I P

ort

ug

al C

onfe

rence S

eri

es –

All R

ights

reserv

ed.

SPI Initiative – ROI OPM.SP 1.3

Estimation of potential benefit

Hypothesis 1

Tool license cost: 3.6K

Management of SPI initiative:

Deployment + Training: 3K€

ROI [ 0.4:1]

Hypothesis 2

Tool license cost: 3.6K

Management of SPI initiative:

Deployment + Training: 3K€

ROI [ 9.6:1]

26 © 2

013 C

MM

I P

ort

ug

al C

onfe

rence S

eri

es –

All R

ights

reserv

ed.

SPI Initiative – Elicit Improvements OPM.SP2.1/SP2.2

A target was given to identify approaches that could

decrease in 10x the time to manage code review

records

Several tools were identified and analyzed in a DAR

initiative – ‘smoke test’.

Crucible came as the best option as it enabled to set-up

a review in less than 10 minutes among other relevant

factors related to infrastructure.

27 © 2

013 C

MM

I P

ort

ug

al C

onfe

rence S

eri

es –

All R

ights

reserv

ed.

SPI Initiative – Pilot OPM.SP2.3/SP2.4

Criteria of success in supporting H1.

Decrease review setup/manage records is

consistent below 10m/code review

Impact on defect detection ability (def/kloc) is

non negative

Pilot lasted 3 months and involved 5 project and a

workshop with SPA area members.

28 © 2

013 C

MM

I P

ort

ug

al C

onfe

rence S

eri

es –

All R

ights

reserv

ed.

SPI Initiative – Pilot Results OPM.SP2.4

Performance baseline with pilot results was created.

Review setup/manage records was under 10m

resulting in a 12x decrease on overall effort – change

in performance tested with statistical

significance.

Defect defection rate has not affected:

when compared to existing performance baseline

(44 defect/KLOC)

Overall, Crucible supported H1 and therefore a base to

approach H2 with a higher degree of confidence.

29 © 2

013 C

MM

I P

ort

ug

al C

onfe

rence S

eri

es –

All R

ights

reserv

ed.

SPI Initiative – Deployment OPM.SP3.1/SP3.2/SP3.3

Phase currently in progress

– Evaluation of ROI will be over a 1 year period

– Monitoring cost of quality

– For the second year end beyond the NPV will

replace ROI for Cost/Benefit analysis of SPI

initiative

30 © 2

013 C

MM

I P

ort

ug

al C

onfe

rence S

eri

es –

All R

ights

reserv

ed.

SPI Recommendations

Seek to establish your governance model (GM)

Use/create a GM that supports the rationale of your

network of Objectives vs. Processes

Start small, measure the least as possible, focus on

making the model reliable.

Only with a minimal statistical confidence on cause and

effect you are able to estimate ROI of SPI

Y(Repeatable SPI sucess) = f (f1,f2,f3)

Y’(Risky SPI initiative) = f‘(?, ?, f1, f2, ?)

31 © 2

013 C

MM

I P

ort

ug

al C

onfe

rence S

eri

es –

All R

ights

reserv

ed.

You can contact me at: [email protected]

or

[email protected]

Questions?

III Conferência CMMI Portugal, Presentation 6: Process Improvement in High Maturity Environments,...

Business

Transcript of III Conferência CMMI Portugal, Presentation 6: Process Improvement in High Maturity Environments,...