Is CMMI a guarantee of performance improvement? - Isabel Margarido (Critical Software)

26
Is CMMI® a Guarantee of Performance Improvement? CMMI Portugal: 3 rd of October | Braga Isabel Lopes Margarido [email protected] João Pascoal Faria, Raul Moreira Vidal – FEUP Marco Vieira – FCTUC

description

 

Transcript of Is CMMI a guarantee of performance improvement? - Isabel Margarido (Critical Software)

Page 1: Is CMMI a guarantee of performance improvement? - Isabel Margarido (Critical Software)

Is CMMI® a Guarantee of Performance Improvement?

CMMI Portugal: 3rd of October | Braga

Isabel Lopes Margarido [email protected]

João Pascoal Faria, Raul Moreira Vidal – FEUP Marco Vieira – FCTUC

Page 2: Is CMMI a guarantee of performance improvement? - Isabel Margarido (Critical Software)

introduction

approach overview

metamodel

procedures

conclusion

agenda

3rd of October, 2011 Isabel Lopes Margarido, CMMI Portugal | Braga 2/26

Page 3: Is CMMI a guarantee of performance improvement? - Isabel Margarido (Critical Software)

CMMI benefits introduction overview procedures metamodel

conclusion

3rd of October, 2011 Isabel Lopes Margarido, CMMI Portugal | Braga 3/26

Performance Improvements over Time by Category

Performance Category Median Improvement

Cost 34%

Schedule 50%

Productivity 61%

Quality 48%

Customer Satisfaction 14%

Return on Investment 4.0 : 1

[1]

Page 4: Is CMMI a guarantee of performance improvement? - Isabel Margarido (Critical Software)

motivation

[7]

[4]

CMMI performance depends on the implementation method SCAMPI: organisation honesty, appraisal team quality, small percentage of the organisation, limited number of affirmations

[3-6]

[6]

[8]

3rd of October, 2011 Isabel Lopes Margarido, CMMI Portugal | Braga 4/26

introduction overview procedures metamodel conclusion

Page 5: Is CMMI a guarantee of performance improvement? - Isabel Margarido (Critical Software)

• Overhead [14]

implementation problems

• Metrics definition (collect and analyse data) [12][13][14] • Metrics categorization [14]

• Uncorrelated and meaningless metrics [10][14]

• Understand statistical nature of level 4 [9][10], [14]

• Tools setup [14]

• Not all projects are measurable [14]

• Lack of institutionalisation [6][14]

• Dissemination problems [14]

3rd of October, 2011 Isabel Lopes Margarido, CMMI Portugal | Braga 5/26

introduction overview procedures metamodel conclusion

Page 6: Is CMMI a guarantee of performance improvement? - Isabel Margarido (Critical Software)

problem statement

CMMI Model high variability of performance • dependence on methods used and quality of implementation

Quality of Implementation difficulties in the selection of implementation methods

Performance Indicators need for a performance evaluation method • CMMI V1.3 more focused on performance • SCAMPI does not measure performance

3rd of October, 2011 Isabel Lopes Margarido, CMMI Portugal | Braga 6/26

introduction overview procedures metamodel conclusion

Page 7: Is CMMI a guarantee of performance improvement? - Isabel Margarido (Critical Software)

purpose

objectives of the research develop an evaluation framework

select implementation methods select performance indicators

to allow early evaluation of quality of implementation of the model organisational performance impact of process improvement initiatives on organisational performance

beneficiaries

organisations implementing CMMI SEI – easily verify performance improvement from one SCAMPI to the next

introduction overview procedures metamodel conclusion

3rd of October, 2011 Isabel Lopes Margarido, CMMI Portugal | Braga 7/26

Page 8: Is CMMI a guarantee of performance improvement? - Isabel Margarido (Critical Software)

Improvement Techniques

concept Quality Principles • CMMI-DEV

Organisation • PP • PMC • …

Quality of Implementation

Operational Practices • Tools • Techniques • Procedures • …

Org. Performance

CMMI - ACQ

CMMI - SVC

CMMI - DEV PMI

ISO

TSP

Agile

Lean

Six Sigma Theory of

Constraints

introduction overview procedures metamodel conclusion

3rd of October, 2011 Isabel Lopes Margarido, CMMI Portugal | Braga 8/26

Page 9: Is CMMI a guarantee of performance improvement? - Isabel Margarido (Critical Software)

ML5

ML4

ML3

ML2 PA

SG1 SP1.1 SG2 SP2.1 …

n n

1 n 0 or 1 n

1 n 0 or 1 n

implementation methods

PI

concept

3rd of October, 2011 Isabel Lopes Margarido, CMMI Portugal | Braga 9/26

introduction overview procedures metamodel conclusion

Page 10: Is CMMI a guarantee of performance improvement? - Isabel Margarido (Critical Software)

approach overview

Metamodel

evaluation framework application Repository

Methods x CMMI practices

Performance Indicators • potential • profiles

(ML, methods)

T

A

I

L

O

R

I

N

G

Procedures

E

X

E

C

U

T

I

O

N

• Quality of Implementation • Organisation Performance • Performance Improvement

including aggregation and rating

-colour (red, yellow, green)-numerical value (optional)-time-source: (org, dep, proj)-target: (G/P, method, PI))

Semaphore

Evaluation

-thresholds

PI

Project Department

*

0..1

*

0..1

Goal/Practice

Method*

0..1

* 0..1*

0..1

Aggregation

Organisation

*

0..1

Aggregation

Process PI Product PI

Leading PI Lagging PI

Base Measure

Performance Indicator (PI)

* *

Influences4

*

*

Used to calculate4

Goal/Practice

CMMI Model

Evaluation Framework

Method

1..*

*

Evaluates

0..**Evaluates4

-ratting

Support

*

*

3rd of October, 2011 Isabel Lopes Margarido, CMMI Portugal | Braga 10/26

introduction overview procedures metamodel conclusion

Page 11: Is CMMI a guarantee of performance improvement? - Isabel Margarido (Critical Software)

approach overview

quality of implementation

degree of implementation of a practice with a given method

some performance indicators measure it

performance indicators

measure the organisation performance

measure the quality of implementation

their aggregation indicates degree of institutionalisation, necessary for generic goals and high maturity evaluation

8th of June, 2011 11/26 Isabel Lopes Margarido, SEPG Europe – Dublin, Ireland

introduction overview procedures metamodel conclusion

Page 12: Is CMMI a guarantee of performance improvement? - Isabel Margarido (Critical Software)

definitions

base measure – measure of a single property/characteristic of product, process, project or resource (attribute) [15]

performance indicator – measure that provides estimate/evaluation of an attribute derived from base measures or other derived measures [15]

leading indicator – anticipates quality, allows forecasting and diagnosis [16-17]

lagging indicator – follows an event or tendency, allows appraising [17]

methods – good practices, procedures, techniques, tools, etc. , that are part of the processes of the organisation

3rd of October, 2011 Isabel Lopes Margarido, CMMI Portugal | Braga 12/26

Attribute x

Base Measure 1

Attribute y

Base Measure 2

Derived Measure Performance Indicator

introduction overview procedures metamodel conclusion

Page 13: Is CMMI a guarantee of performance improvement? - Isabel Margarido (Critical Software)

Directly Adressed Partially Adressed Supports Not Adressed Unrated

[7]

Process PI Product PI

Leading PI Lagging PI

Base Measure

Performance Indicator (PI)

* *

Influences4

*

*

Used to calculate4

Goal/Practice

CMMI Model

Evaluation Framework

Method

1..*

*

Evaluates

0..**Evaluates4

Support

-rating

*

*

3rd of October, 2011 Isabel Lopes Margarido, CMMI Portugal | Braga 13/26

introduction overview procedures metamodel conclusion

Page 14: Is CMMI a guarantee of performance improvement? - Isabel Margarido (Critical Software)

-colour (red, yellow, green)-numerical value (optional)-time-source: (org, dep, proj)-target: (G/P, method, PI))

Semaphore

Evaluation

-thresholds

PI

Project Department

*

0..1

*

0..1

Goal/Practice

Method*

0..1

* 0..1*

0..1

Aggregation

Organisation

*

0..1

Aggregation

introduction overview repository procedures metamodel conclusion

3rd of October, 2011 Isabel Lopes Margarido, CMMI Portugal | Braga 14/26

Page 15: Is CMMI a guarantee of performance improvement? - Isabel Margarido (Critical Software)

aggregation

1 1… *

introduction overview procedures metamodel conclusion

3rd of October, 2011 Isabel Lopes Margarido, CMMI Portugal | Braga 15/26

Page 16: Is CMMI a guarantee of performance improvement? - Isabel Margarido (Critical Software)

rating and aggregation example PI: Schedule Estimation Error (%) Time: 2010

Time

Value

threshold2

0%

threshold1

threshold2

P1 (D1)

P2 (D2)

P3 (D1)

P4 (D1)

P5 (D3)

Org:

D1:

D2:

D3:

Legend: Org – organisation, D1 – department 1; P1 – project 1.

threshold1

3rd of October, 2011 Isabel Lopes Margarido, CMMI Portugal | Braga 16/26

introduction overview procedures metamodel conclusion

Page 17: Is CMMI a guarantee of performance improvement? - Isabel Margarido (Critical Software)

aggregation example

Legend: Org – organisation, D1 – department 1; P1 – project 1; alt – alternative; opt – optional; ,mandat – mandatory; ^ – AND; v – OR.

P3 P2

N/A

N/A

N/A

N/A

N/A

D2

D1

Org

PI1

PI2

PI3

PI4

PI5

M1 (alt M2) (PI1 ^ PI2)

M2 (alt M1) (PI1 ^ PI5)

M3 (opt) (PI3)

M4 (mandat) (PI4)

SP1 M4

SP2 (M1 v M2)

SP3 (M1 v M2) ^ M3

SP4 (M1 ^ M4)

P1 P4

N/A

N/A

N/A

N/A

3rd of October, 2011 Isabel Lopes Margarido, CMMI Portugal | Braga 17/26

introduction overview procedures metamodel conclusion

Page 18: Is CMMI a guarantee of performance improvement? - Isabel Margarido (Critical Software)

research status

identified some of the problems in the CMMI

gathered performance indicators and implementation methods for a subset of practices

designed the metamodel and preliminary version of the evaluation framework

introduction overview procedures metamodel conclusion

3rd of October, 2011 Isabel Lopes Margarido, CMMI Portugal | Braga 18/26

Page 19: Is CMMI a guarantee of performance improvement? - Isabel Margarido (Critical Software)

future work

analyse organisations data (PI, methods, relations between PI)

define the rationale for tailoring the PI

validate the evaluation framework in organisations

3rd of October, 2011 Isabel Lopes Margarido, CMMI Portugal | Braga 19/26

introduction overview procedures metamodel conclusion

Page 20: Is CMMI a guarantee of performance improvement? - Isabel Margarido (Critical Software)

summary

data aggregation and its analysis is particularly

important in the implementation of the GG and HML

performance indicators are useful to evaluate the quality of implementation

we also map the CMMI practices with methods that organisations can choose and adapt

3rd of October, 2011 Isabel Lopes Margarido, CMMI Portugal | Braga 20/26

introduction overview procedures metamodel conclusion

Page 21: Is CMMI a guarantee of performance improvement? - Isabel Margarido (Critical Software)

questions

3rd of October, 2011 Isabel Lopes Margarido, CMMI Portugal | Braga 21/26

Page 22: Is CMMI a guarantee of performance improvement? - Isabel Margarido (Critical Software)

[1] C. P. Team, "CMMI® for Development, Version 1.3," CMU/SEI CMU/SEI-2010-TR-033, ESC-TR-2010-033, November 2010.

[2] Gibson, Diane L., Goldenson, Dennis R., Kost, Keith, Performance Results of CMMI®-Based Process Improvement, CMU/SEI, 2006.

[3] N. Davis and J. Mullaney, "The Team Software ProcessSM (TSPSM) in Practice: A Summary of Recent Results," CMU/SEI CMU/SEI-2003-TR-014, ESC-TR-2003-014, 2003.

[4] N. Davis and J. McHale, "Relating the Team Software ProcessSM (TSPSM) to the Capability Maturity Model® for Software (SW-CMM®)," CMU/SEI-2002-TR-008, ESC-TR-2002-008, March 2003.

[5] J. McHale and D. S. Wall, "Mapping TSP to CMMI," CMU/SEI CMU/SEI-2004-TR-014, ESC-TR-2004-014, 2005.

[6] R. Radice, "Statistical Process Control in Level 4 and Level 5 Software Organizations Worldwide," presented at the Software Technology Conference, 2000.

[7] R. Charette, et al., "Understanding the Roots of Process Performance Failure," CROSSTALK The Journal of Defense Software Engineering, pp. 18-22, 2004.

[8] M. Schaeffer, "DoD Systems Engineering and CMMI," presented at the CMMI Technology Conference and User Group, 2004.

references

3rd of October, 2011 Isabel Lopes Margarido, CMMI Portugal | Braga 22/26

Page 23: Is CMMI a guarantee of performance improvement? - Isabel Margarido (Critical Software)

[8] A. Takara, et al., "Problems and Pitfalls in a CMMI level 3 to level 4 Migration Process," presented at the Sixth International Conference on the Quality of Information and Communications Technology, 2007.

[9] C. Hollenbach and D. Smith. (2002) A portrait of a CMMISM level 4 effort Systems Engineering. 52-61.

[10] B. Kitchenham, et al., "Lessons Learnt from the Analysis of Large-scale Corporate Databases," presented at the International Conference on Software Engineering, Shanghai, 2006.

[11] D. Breuker, et al., "Reliability of software metrics tools," presented at the International Conference on Software Process and Product Measurement, Amsterdam, 2009.

[12] M. C. P. A. Goulão, "Component-Based Software Engineering: a Quantitative Approach," Doctoral, Departamento de Informática, Universidade Nova de Lisboa, Faculdade de Ciências e Tecnologia, Lisboa, 2008.

[13] I. Lopes Margarido, et al., "What is wrong with the CMMI® High Maturity Levels?," in SEPG Europe, Porto, 2010.

[14] M. Philips, "CMMI V1.3 Planned Improvements," presented at the SEPG Europe 2010, Porto, Portugal, 2010.

references

3rd of October, 2011 Isabel Lopes Margarido, CMMI Portugal | Braga 23/26

Page 24: Is CMMI a guarantee of performance improvement? - Isabel Margarido (Critical Software)

images http://mitografias.files.wordpress.com/2009/08/superman-flying.jpg –adapted, 24-06-2010

http://api.ning.com/files/hpf*xOTebDs- F23o6FETZ3j*3sNiONWjfXjTJCzprPjU5bS1 WJoGgWBjMPIOiQkm3SbZ41ijncrJ4K2aT-6dM9QURwHK3led/Dissemination2.jpg -26-06-2010

http://blog.pmtech.com.br/wp-content/uploads/Square-Paradox.jpg – 29-04-2011

http://www.signsexpressshop.co.uk/prodpics/1103.gif – 29-04-2011

Benjamin Haas/Shutterstock, http://cynthiayildirim.posterous.com/how-can-we-measure-the-size-of-the-universe – 29-04-2011

http://ryanstephensmarketing.com/blog/wp-content/uploads/2009/10/one_size_fits_all.JPG

http://evolvingwe.com/wp-content/uploads/2010/11/image3.png – 29-04-2011

http://occlink.com/wp-content/uploads/process-picture.jpg

http://www.pastinyala.com/images/customised_software_product.jpg – adapted 29-04-2011

http://www.smartkids.com.br/conteudo/especiais/transito/sinalizacao/semaforo.gif – 21-01-2011

http://www.screenhog.com/sketch/LightbulbIdea.jpg – 21-04-2010

http://igraduatedwhatnow.files.wordpress.com/2009/11/thank_you_small.jpg – 02-05-2010

http://etablissements.ac-amiens.fr/0801372g/matieres/anglais/images/difficult.gif – 25-05-2011

http://3.bp.blogspot.com/-_Z2dYcXxMmA/TbLat4c6i_I/AAAAAAAAAnk/KlLdgG-dgtw/s1600/whereamigoing.jpg – adapted, 25-05-20111

http://www.articulate.com/rapid-elearning/wp-content/uploads/2008/08/summary-objectives450.gif – adapted, 25-05-2011

http://1.bp.blogspot.com/_C3jJLFkkSKs/Rzb7NQdztyI/AAAAAAAAA1A/H8nECRh_76A/s400/ponte.JPG – 06-06-2011

http://www.veryhappypig.com/blog/results.jpg -06-06-2011

http://www.stampa.unibocconi.it/immagini/LA4_economiaq20100603145905.jpg - 06-06-2011

3rd of October, 2011 Isabel Lopes Margarido, CMMI Portugal | Braga 24/26

Page 25: Is CMMI a guarantee of performance improvement? - Isabel Margarido (Critical Software)

interested in our research?

get involved!

participate in the surveys and/or in the validation

phase

share your experience and/or opinion

contact:

Isabel Lopes Margarido, [email protected]

http://paginas.fe.up.pt/~pro09003/

copyright:

partially sponsored by:

3rd of October, 2011 Isabel Lopes Margarido, CMMI Portugal | Braga 25/26

Page 26: Is CMMI a guarantee of performance improvement? - Isabel Margarido (Critical Software)

MA – Measurement and Analysis ML – maturity level n or * – many org – organisation PA – Process Area PI – performance indicator proj – project SCAMPI ℠ – Standard CMMI Appraisal Method for Process Improvement PP – Project Planning PMC – Project Monitoring and Control SEI – Software Engineering Institute SG – specific goal SP – specific practice SVC – services SW – software TSP℠ – Team Software Process V – version

acronyms

ACQ – acquisition C – case study CAR – Causal Analysis and Resolution CMMI® – Capability Maturity Model Integrated dep – department DEV – development DoD – Department of Defense (United States of America) FEUP – Faculty of Engineering, University of Porto GDM – goal driven measurement GG – generic goal GP – generic practice G/P – goal or practice ISO – International Organization for Standardization KLOC – thousand lines of code

3rd of October, 2011 Isabel Lopes Margarido, CMMI Portugal | Braga 26/26