Is CMMI a guarantee of performance improvement? - Isabel Margarido (Critical Software)
-
Upload
paula-gomes -
Category
Business
-
view
1.278 -
download
1
description
Transcript of Is CMMI a guarantee of performance improvement? - Isabel Margarido (Critical Software)
Is CMMI® a Guarantee of Performance Improvement?
CMMI Portugal: 3rd of October | Braga
Isabel Lopes Margarido [email protected]
João Pascoal Faria, Raul Moreira Vidal – FEUP Marco Vieira – FCTUC
introduction
approach overview
metamodel
procedures
conclusion
agenda
3rd of October, 2011 Isabel Lopes Margarido, CMMI Portugal | Braga 2/26
CMMI benefits introduction overview procedures metamodel
conclusion
3rd of October, 2011 Isabel Lopes Margarido, CMMI Portugal | Braga 3/26
Performance Improvements over Time by Category
Performance Category Median Improvement
Cost 34%
Schedule 50%
Productivity 61%
Quality 48%
Customer Satisfaction 14%
Return on Investment 4.0 : 1
[1]
motivation
[7]
[4]
CMMI performance depends on the implementation method SCAMPI: organisation honesty, appraisal team quality, small percentage of the organisation, limited number of affirmations
[3-6]
[6]
[8]
3rd of October, 2011 Isabel Lopes Margarido, CMMI Portugal | Braga 4/26
introduction overview procedures metamodel conclusion
• Overhead [14]
implementation problems
• Metrics definition (collect and analyse data) [12][13][14] • Metrics categorization [14]
• Uncorrelated and meaningless metrics [10][14]
• Understand statistical nature of level 4 [9][10], [14]
• Tools setup [14]
• Not all projects are measurable [14]
• Lack of institutionalisation [6][14]
• Dissemination problems [14]
3rd of October, 2011 Isabel Lopes Margarido, CMMI Portugal | Braga 5/26
introduction overview procedures metamodel conclusion
problem statement
CMMI Model high variability of performance • dependence on methods used and quality of implementation
Quality of Implementation difficulties in the selection of implementation methods
Performance Indicators need for a performance evaluation method • CMMI V1.3 more focused on performance • SCAMPI does not measure performance
3rd of October, 2011 Isabel Lopes Margarido, CMMI Portugal | Braga 6/26
introduction overview procedures metamodel conclusion
purpose
objectives of the research develop an evaluation framework
select implementation methods select performance indicators
to allow early evaluation of quality of implementation of the model organisational performance impact of process improvement initiatives on organisational performance
beneficiaries
organisations implementing CMMI SEI – easily verify performance improvement from one SCAMPI to the next
introduction overview procedures metamodel conclusion
3rd of October, 2011 Isabel Lopes Margarido, CMMI Portugal | Braga 7/26
Improvement Techniques
concept Quality Principles • CMMI-DEV
Organisation • PP • PMC • …
Quality of Implementation
Operational Practices • Tools • Techniques • Procedures • …
Org. Performance
CMMI - ACQ
CMMI - SVC
CMMI - DEV PMI
ISO
TSP
Agile
Lean
Six Sigma Theory of
Constraints
introduction overview procedures metamodel conclusion
3rd of October, 2011 Isabel Lopes Margarido, CMMI Portugal | Braga 8/26
ML5
ML4
ML3
ML2 PA
SG1 SP1.1 SG2 SP2.1 …
n n
1 n 0 or 1 n
1 n 0 or 1 n
implementation methods
PI
concept
3rd of October, 2011 Isabel Lopes Margarido, CMMI Portugal | Braga 9/26
introduction overview procedures metamodel conclusion
approach overview
Metamodel
evaluation framework application Repository
Methods x CMMI practices
Performance Indicators • potential • profiles
(ML, methods)
T
A
I
L
O
R
I
N
G
Procedures
E
X
E
C
U
T
I
O
N
• Quality of Implementation • Organisation Performance • Performance Improvement
including aggregation and rating
-colour (red, yellow, green)-numerical value (optional)-time-source: (org, dep, proj)-target: (G/P, method, PI))
Semaphore
Evaluation
-thresholds
PI
Project Department
*
0..1
*
0..1
Goal/Practice
Method*
0..1
* 0..1*
0..1
Aggregation
Organisation
*
0..1
Aggregation
Process PI Product PI
Leading PI Lagging PI
Base Measure
Performance Indicator (PI)
* *
Influences4
*
*
Used to calculate4
Goal/Practice
CMMI Model
Evaluation Framework
Method
1..*
*
Evaluates
0..**Evaluates4
-ratting
Support
*
*
3rd of October, 2011 Isabel Lopes Margarido, CMMI Portugal | Braga 10/26
introduction overview procedures metamodel conclusion
approach overview
quality of implementation
degree of implementation of a practice with a given method
some performance indicators measure it
performance indicators
measure the organisation performance
measure the quality of implementation
their aggregation indicates degree of institutionalisation, necessary for generic goals and high maturity evaluation
8th of June, 2011 11/26 Isabel Lopes Margarido, SEPG Europe – Dublin, Ireland
introduction overview procedures metamodel conclusion
definitions
base measure – measure of a single property/characteristic of product, process, project or resource (attribute) [15]
performance indicator – measure that provides estimate/evaluation of an attribute derived from base measures or other derived measures [15]
leading indicator – anticipates quality, allows forecasting and diagnosis [16-17]
lagging indicator – follows an event or tendency, allows appraising [17]
methods – good practices, procedures, techniques, tools, etc. , that are part of the processes of the organisation
3rd of October, 2011 Isabel Lopes Margarido, CMMI Portugal | Braga 12/26
Attribute x
Base Measure 1
Attribute y
Base Measure 2
Derived Measure Performance Indicator
introduction overview procedures metamodel conclusion
Directly Adressed Partially Adressed Supports Not Adressed Unrated
[7]
Process PI Product PI
Leading PI Lagging PI
Base Measure
Performance Indicator (PI)
* *
Influences4
*
*
Used to calculate4
Goal/Practice
CMMI Model
Evaluation Framework
Method
1..*
*
Evaluates
0..**Evaluates4
Support
-rating
*
*
3rd of October, 2011 Isabel Lopes Margarido, CMMI Portugal | Braga 13/26
introduction overview procedures metamodel conclusion
-colour (red, yellow, green)-numerical value (optional)-time-source: (org, dep, proj)-target: (G/P, method, PI))
Semaphore
Evaluation
-thresholds
PI
Project Department
*
0..1
*
0..1
Goal/Practice
Method*
0..1
* 0..1*
0..1
Aggregation
Organisation
*
0..1
Aggregation
introduction overview repository procedures metamodel conclusion
3rd of October, 2011 Isabel Lopes Margarido, CMMI Portugal | Braga 14/26
aggregation
1 1… *
introduction overview procedures metamodel conclusion
3rd of October, 2011 Isabel Lopes Margarido, CMMI Portugal | Braga 15/26
rating and aggregation example PI: Schedule Estimation Error (%) Time: 2010
Time
Value
threshold2
0%
threshold1
threshold2
P1 (D1)
P2 (D2)
P3 (D1)
P4 (D1)
P5 (D3)
Org:
D1:
D2:
D3:
Legend: Org – organisation, D1 – department 1; P1 – project 1.
threshold1
3rd of October, 2011 Isabel Lopes Margarido, CMMI Portugal | Braga 16/26
introduction overview procedures metamodel conclusion
aggregation example
Legend: Org – organisation, D1 – department 1; P1 – project 1; alt – alternative; opt – optional; ,mandat – mandatory; ^ – AND; v – OR.
P3 P2
N/A
N/A
N/A
N/A
N/A
D2
D1
Org
PI1
PI2
PI3
PI4
PI5
M1 (alt M2) (PI1 ^ PI2)
M2 (alt M1) (PI1 ^ PI5)
M3 (opt) (PI3)
M4 (mandat) (PI4)
SP1 M4
SP2 (M1 v M2)
SP3 (M1 v M2) ^ M3
SP4 (M1 ^ M4)
P1 P4
N/A
N/A
N/A
N/A
3rd of October, 2011 Isabel Lopes Margarido, CMMI Portugal | Braga 17/26
introduction overview procedures metamodel conclusion
research status
identified some of the problems in the CMMI
gathered performance indicators and implementation methods for a subset of practices
designed the metamodel and preliminary version of the evaluation framework
introduction overview procedures metamodel conclusion
3rd of October, 2011 Isabel Lopes Margarido, CMMI Portugal | Braga 18/26
future work
analyse organisations data (PI, methods, relations between PI)
define the rationale for tailoring the PI
validate the evaluation framework in organisations
3rd of October, 2011 Isabel Lopes Margarido, CMMI Portugal | Braga 19/26
introduction overview procedures metamodel conclusion
summary
data aggregation and its analysis is particularly
important in the implementation of the GG and HML
performance indicators are useful to evaluate the quality of implementation
we also map the CMMI practices with methods that organisations can choose and adapt
3rd of October, 2011 Isabel Lopes Margarido, CMMI Portugal | Braga 20/26
introduction overview procedures metamodel conclusion
questions
3rd of October, 2011 Isabel Lopes Margarido, CMMI Portugal | Braga 21/26
[1] C. P. Team, "CMMI® for Development, Version 1.3," CMU/SEI CMU/SEI-2010-TR-033, ESC-TR-2010-033, November 2010.
[2] Gibson, Diane L., Goldenson, Dennis R., Kost, Keith, Performance Results of CMMI®-Based Process Improvement, CMU/SEI, 2006.
[3] N. Davis and J. Mullaney, "The Team Software ProcessSM (TSPSM) in Practice: A Summary of Recent Results," CMU/SEI CMU/SEI-2003-TR-014, ESC-TR-2003-014, 2003.
[4] N. Davis and J. McHale, "Relating the Team Software ProcessSM (TSPSM) to the Capability Maturity Model® for Software (SW-CMM®)," CMU/SEI-2002-TR-008, ESC-TR-2002-008, March 2003.
[5] J. McHale and D. S. Wall, "Mapping TSP to CMMI," CMU/SEI CMU/SEI-2004-TR-014, ESC-TR-2004-014, 2005.
[6] R. Radice, "Statistical Process Control in Level 4 and Level 5 Software Organizations Worldwide," presented at the Software Technology Conference, 2000.
[7] R. Charette, et al., "Understanding the Roots of Process Performance Failure," CROSSTALK The Journal of Defense Software Engineering, pp. 18-22, 2004.
[8] M. Schaeffer, "DoD Systems Engineering and CMMI," presented at the CMMI Technology Conference and User Group, 2004.
references
3rd of October, 2011 Isabel Lopes Margarido, CMMI Portugal | Braga 22/26
[8] A. Takara, et al., "Problems and Pitfalls in a CMMI level 3 to level 4 Migration Process," presented at the Sixth International Conference on the Quality of Information and Communications Technology, 2007.
[9] C. Hollenbach and D. Smith. (2002) A portrait of a CMMISM level 4 effort Systems Engineering. 52-61.
[10] B. Kitchenham, et al., "Lessons Learnt from the Analysis of Large-scale Corporate Databases," presented at the International Conference on Software Engineering, Shanghai, 2006.
[11] D. Breuker, et al., "Reliability of software metrics tools," presented at the International Conference on Software Process and Product Measurement, Amsterdam, 2009.
[12] M. C. P. A. Goulão, "Component-Based Software Engineering: a Quantitative Approach," Doctoral, Departamento de Informática, Universidade Nova de Lisboa, Faculdade de Ciências e Tecnologia, Lisboa, 2008.
[13] I. Lopes Margarido, et al., "What is wrong with the CMMI® High Maturity Levels?," in SEPG Europe, Porto, 2010.
[14] M. Philips, "CMMI V1.3 Planned Improvements," presented at the SEPG Europe 2010, Porto, Portugal, 2010.
references
3rd of October, 2011 Isabel Lopes Margarido, CMMI Portugal | Braga 23/26
images http://mitografias.files.wordpress.com/2009/08/superman-flying.jpg –adapted, 24-06-2010
http://api.ning.com/files/hpf*xOTebDs- F23o6FETZ3j*3sNiONWjfXjTJCzprPjU5bS1 WJoGgWBjMPIOiQkm3SbZ41ijncrJ4K2aT-6dM9QURwHK3led/Dissemination2.jpg -26-06-2010
http://blog.pmtech.com.br/wp-content/uploads/Square-Paradox.jpg – 29-04-2011
http://www.signsexpressshop.co.uk/prodpics/1103.gif – 29-04-2011
Benjamin Haas/Shutterstock, http://cynthiayildirim.posterous.com/how-can-we-measure-the-size-of-the-universe – 29-04-2011
http://ryanstephensmarketing.com/blog/wp-content/uploads/2009/10/one_size_fits_all.JPG
http://evolvingwe.com/wp-content/uploads/2010/11/image3.png – 29-04-2011
http://occlink.com/wp-content/uploads/process-picture.jpg
http://www.pastinyala.com/images/customised_software_product.jpg – adapted 29-04-2011
http://www.smartkids.com.br/conteudo/especiais/transito/sinalizacao/semaforo.gif – 21-01-2011
http://www.screenhog.com/sketch/LightbulbIdea.jpg – 21-04-2010
http://igraduatedwhatnow.files.wordpress.com/2009/11/thank_you_small.jpg – 02-05-2010
http://etablissements.ac-amiens.fr/0801372g/matieres/anglais/images/difficult.gif – 25-05-2011
http://3.bp.blogspot.com/-_Z2dYcXxMmA/TbLat4c6i_I/AAAAAAAAAnk/KlLdgG-dgtw/s1600/whereamigoing.jpg – adapted, 25-05-20111
http://www.articulate.com/rapid-elearning/wp-content/uploads/2008/08/summary-objectives450.gif – adapted, 25-05-2011
http://1.bp.blogspot.com/_C3jJLFkkSKs/Rzb7NQdztyI/AAAAAAAAA1A/H8nECRh_76A/s400/ponte.JPG – 06-06-2011
http://www.veryhappypig.com/blog/results.jpg -06-06-2011
http://www.stampa.unibocconi.it/immagini/LA4_economiaq20100603145905.jpg - 06-06-2011
3rd of October, 2011 Isabel Lopes Margarido, CMMI Portugal | Braga 24/26
interested in our research?
get involved!
participate in the surveys and/or in the validation
phase
share your experience and/or opinion
contact:
Isabel Lopes Margarido, [email protected]
http://paginas.fe.up.pt/~pro09003/
copyright:
partially sponsored by:
3rd of October, 2011 Isabel Lopes Margarido, CMMI Portugal | Braga 25/26
MA – Measurement and Analysis ML – maturity level n or * – many org – organisation PA – Process Area PI – performance indicator proj – project SCAMPI ℠ – Standard CMMI Appraisal Method for Process Improvement PP – Project Planning PMC – Project Monitoring and Control SEI – Software Engineering Institute SG – specific goal SP – specific practice SVC – services SW – software TSP℠ – Team Software Process V – version
acronyms
ACQ – acquisition C – case study CAR – Causal Analysis and Resolution CMMI® – Capability Maturity Model Integrated dep – department DEV – development DoD – Department of Defense (United States of America) FEUP – Faculty of Engineering, University of Porto GDM – goal driven measurement GG – generic goal GP – generic practice G/P – goal or practice ISO – International Organization for Standardization KLOC – thousand lines of code
3rd of October, 2011 Isabel Lopes Margarido, CMMI Portugal | Braga 26/26