Identifying and Using a Project’s Key Subprocess Metrics
description
Transcript of Identifying and Using a Project’s Key Subprocess Metrics
![Page 1: Identifying and Using a Project’s Key Subprocess Metrics](https://reader035.fdocuments.us/reader035/viewer/2022070502/56814c74550346895db994e0/html5/thumbnails/1.jpg)
Identifying and Using a Project’s Key Subprocess
Metrics
Jeff S. HolmesBTS Fort Worth
![Page 2: Identifying and Using a Project’s Key Subprocess Metrics](https://reader035.fdocuments.us/reader035/viewer/2022070502/56814c74550346895db994e0/html5/thumbnails/2.jpg)
Everyone Loves a Hero
![Page 3: Identifying and Using a Project’s Key Subprocess Metrics](https://reader035.fdocuments.us/reader035/viewer/2022070502/56814c74550346895db994e0/html5/thumbnails/3.jpg)
Heroes Come Through!
Firemen – Saves Baby in Burning House Policemen – Catches Bad Guy Athlete – Hits Game Winning Homerun
![Page 4: Identifying and Using a Project’s Key Subprocess Metrics](https://reader035.fdocuments.us/reader035/viewer/2022070502/56814c74550346895db994e0/html5/thumbnails/4.jpg)
Software Engineering Heroes!
All night coding! Debugging over the weekend!
THIS SHOULD NOT BE THE NORM!
![Page 5: Identifying and Using a Project’s Key Subprocess Metrics](https://reader035.fdocuments.us/reader035/viewer/2022070502/56814c74550346895db994e0/html5/thumbnails/5.jpg)
How To Minimize “Fire Drills”?
Preventative Actions Proper wiring can prevent fires. Locking your car can prevent theft. Don’t get behind in the ball game. Understand project status earlier.
![Page 6: Identifying and Using a Project’s Key Subprocess Metrics](https://reader035.fdocuments.us/reader035/viewer/2022070502/56814c74550346895db994e0/html5/thumbnails/6.jpg)
Metrics, metrics, everywhere…
![Page 7: Identifying and Using a Project’s Key Subprocess Metrics](https://reader035.fdocuments.us/reader035/viewer/2022070502/56814c74550346895db994e0/html5/thumbnails/7.jpg)
But What is Really But What is Really Important?Important? Customer WantsCustomer Wants
– FunctionalityFunctionality– Zero DefectsZero Defects– On TimeOn Time
What software What software metrics map to metrics map to these?these?
How can we optimize How can we optimize these outputs?these outputs?
![Page 8: Identifying and Using a Project’s Key Subprocess Metrics](https://reader035.fdocuments.us/reader035/viewer/2022070502/56814c74550346895db994e0/html5/thumbnails/8.jpg)
BTS Fort Worth Approach
Selected DMAIC to Improve Process Identified Project with Two Years of Data Performed Statistical Analysis Conducted Pilot Currently in “Control” Phase
![Page 9: Identifying and Using a Project’s Key Subprocess Metrics](https://reader035.fdocuments.us/reader035/viewer/2022070502/56814c74550346895db994e0/html5/thumbnails/9.jpg)
DMAIC : Define
Identify “what is important” BTS FW Monitors
Productivity (KLOC/Hour)*
Quality (Post Release Defects/KLOC)Schedule Adherence
These are BTS FW “Big X’s”
![Page 10: Identifying and Using a Project’s Key Subprocess Metrics](https://reader035.fdocuments.us/reader035/viewer/2022070502/56814c74550346895db994e0/html5/thumbnails/10.jpg)
DMAIC : Measure
Software Development Life CycleRequirementsResources
Perfect Software!
The “Simple” View
![Page 11: Identifying and Using a Project’s Key Subprocess Metrics](https://reader035.fdocuments.us/reader035/viewer/2022070502/56814c74550346895db994e0/html5/thumbnails/11.jpg)
DMAIC : More DetailsRequirements
RequirementsResources
Perfect Software!
Design
Code
Test
Perfect RequirementsResources
Perfect Design Perfect Models
Perfect DesignPerfect ModelsResources
Perfect Code
Perfect CodeResources
![Page 12: Identifying and Using a Project’s Key Subprocess Metrics](https://reader035.fdocuments.us/reader035/viewer/2022070502/56814c74550346895db994e0/html5/thumbnails/12.jpg)
DMAIC : Subprocess Identification
BTS FW Identified Following Subprocesses Planning Phase Requirements Phase Design Phase Code Phase Test Phase Release Phase Code Inspections
![Page 13: Identifying and Using a Project’s Key Subprocess Metrics](https://reader035.fdocuments.us/reader035/viewer/2022070502/56814c74550346895db994e0/html5/thumbnails/13.jpg)
DMAIC : Measured Data BTS FW Uses Following Data:
# Requirements # Developers on the project (Resources) % Time in Planning % Time in Requirements % Time in Design % Time in Code % Time in Test % Time in Release Requirements Churn Actual Size (KLOC) Avg Defect Detection Rate (DDR) in Code Inspection
![Page 14: Identifying and Using a Project’s Key Subprocess Metrics](https://reader035.fdocuments.us/reader035/viewer/2022070502/56814c74550346895db994e0/html5/thumbnails/14.jpg)
DMAIC : Data Sources BTS FW Data
DOORs for Requirements Project Plans # Developers and % Times ClearCase for Code Size Inspection Database for DDR
![Page 15: Identifying and Using a Project’s Key Subprocess Metrics](https://reader035.fdocuments.us/reader035/viewer/2022070502/56814c74550346895db994e0/html5/thumbnails/15.jpg)
DMAIC : BTS Subprocess Metrics Planning
Design
Code
Test
Req CountTime in Phase
Req ChurnTime in Phase
Requirements
ResourcesTime in Phase
Req ChurnKLOC
Time in Phase
Time in Phase
Release Time in Phase
Code Inspections DDR
![Page 16: Identifying and Using a Project’s Key Subprocess Metrics](https://reader035.fdocuments.us/reader035/viewer/2022070502/56814c74550346895db994e0/html5/thumbnails/16.jpg)
DMAIC : BTS Subprocess Metrics Planning
Design
Code
Test
Req CountTime in Phase
Req ChurnTime in Phase
Requirements
ResourcesTime in Phase
Req ChurnKLOC
Time in Phase
Time in Phase
Release Time in Phase
Code Inspections DDR
Productivity (KLOC/Hour)
Quality (PR Defects/KLOC)
Schedule Adherence
?
?
?
?
?
![Page 17: Identifying and Using a Project’s Key Subprocess Metrics](https://reader035.fdocuments.us/reader035/viewer/2022070502/56814c74550346895db994e0/html5/thumbnails/17.jpg)
DMAIC : BTS FW Analysis Project Data
• 8 Releases since 2002• Similar work• “Stable” team
Used Step-wise Linear Regression to • Identify statistically significant factors• Develop prediction formulas for “Big X’s”
![Page 18: Identifying and Using a Project’s Key Subprocess Metrics](https://reader035.fdocuments.us/reader035/viewer/2022070502/56814c74550346895db994e0/html5/thumbnails/18.jpg)
CAUTION !!
The following slides contain statistics that could be hazardous to your health!
Persons who suffer from narcolepsy or “statisticitis” should consider leaving the room.
![Page 19: Identifying and Using a Project’s Key Subprocess Metrics](https://reader035.fdocuments.us/reader035/viewer/2022070502/56814c74550346895db994e0/html5/thumbnails/19.jpg)
DMAIC : Stepwise Linear Regression Describes the relationship between one
'predicted' variable and 'predictor' variables
Goal – get the simplest equation with the best predictive power for• Productivity – KLOC/Hour• Quality – Post Release Defect/KLOC
![Page 20: Identifying and Using a Project’s Key Subprocess Metrics](https://reader035.fdocuments.us/reader035/viewer/2022070502/56814c74550346895db994e0/html5/thumbnails/20.jpg)
DMAIC : Standard Least Squares
Model accounts for99.82% of variance.
![Page 21: Identifying and Using a Project’s Key Subprocess Metrics](https://reader035.fdocuments.us/reader035/viewer/2022070502/56814c74550346895db994e0/html5/thumbnails/21.jpg)
DMAIC : Significant Effects
Most significant effects• % Time in Req• Average DDR• Interaction between % Time in Code and Average DDR
< 0.05 is significant
![Page 22: Identifying and Using a Project’s Key Subprocess Metrics](https://reader035.fdocuments.us/reader035/viewer/2022070502/56814c74550346895db994e0/html5/thumbnails/22.jpg)
DMAIC : Standard Least Squares
Model accounts for90.62% of variance.
![Page 23: Identifying and Using a Project’s Key Subprocess Metrics](https://reader035.fdocuments.us/reader035/viewer/2022070502/56814c74550346895db994e0/html5/thumbnails/23.jpg)
DMAIC : Significant Effects
Most significant effects• % Time in Requirements• Interaction between % Time in Requirements
and Requirements Churn
< 0.05 is significant
![Page 24: Identifying and Using a Project’s Key Subprocess Metrics](https://reader035.fdocuments.us/reader035/viewer/2022070502/56814c74550346895db994e0/html5/thumbnails/24.jpg)
DMAIC : Statistically Significant # Requirements # Developers on the project % Time in Planning % Time in Requirements % Time in Design % Time in Code % Time in Test % Time in Release Requirements Churn Actual Size (KLOC) Avg Defect Detection Rate (DDR) in Code Inspection
![Page 25: Identifying and Using a Project’s Key Subprocess Metrics](https://reader035.fdocuments.us/reader035/viewer/2022070502/56814c74550346895db994e0/html5/thumbnails/25.jpg)
DMAIC : Key Subprocess Metrics Planning
Design
Code
Test
Req CountTime in Phase
Req ChurnTime in Phase
Requirements
ResourcesTime in Phase
Req ChurnKLOC
Time in Phase
Time in Phase
Release Time in Phase
Code Inspections DDR
Productivity (KLOC/Hour)
Quality (PR Defects/KLOC)
Schedule Adherence
?
?
?
?
?
![Page 26: Identifying and Using a Project’s Key Subprocess Metrics](https://reader035.fdocuments.us/reader035/viewer/2022070502/56814c74550346895db994e0/html5/thumbnails/26.jpg)
DMAIC : Variation Analysis Prediction formulas generated to identify:
• Good and bad variance• Most significant factors
NOTE: Prediction formula uses all effects from the models, not just the significant ones. Formula added in Percent Planning.
![Page 27: Identifying and Using a Project’s Key Subprocess Metrics](https://reader035.fdocuments.us/reader035/viewer/2022070502/56814c74550346895db994e0/html5/thumbnails/27.jpg)
DMAIC : Factor WeightingMetric 10% Deviation in Each Subprocess Area% Plan 10% 0 0 0 0
% Req 0 10% 0 0 0
% Code 0 0 10% 0 0
Req Churn 0 0 0 10% 0
Avg DDR 0 0 0 0 10%
Productivity 0 3% 31% 0 9%
Quality 14% 40% 0 6% 0
Cycle Time 0 3% 24% 0 9%
![Page 28: Identifying and Using a Project’s Key Subprocess Metrics](https://reader035.fdocuments.us/reader035/viewer/2022070502/56814c74550346895db994e0/html5/thumbnails/28.jpg)
DMAIC : Factors’ Effects
Metric LOC/Hr Effect PR Defect/KLOC Effect% Planning Time N/A More time = Less
defects/KLOC
% Req Time More Time = Less KLOC/Hr
Less time = More defects/KLOC
% Code Time More Time = Less KLOC/Hr
N/A
Code Insp. DDR Higher DDR = Less KLOC/Hr
N/A
2SR Req. Churn N/A Less Churn = Less defects/KLOC
![Page 29: Identifying and Using a Project’s Key Subprocess Metrics](https://reader035.fdocuments.us/reader035/viewer/2022070502/56814c74550346895db994e0/html5/thumbnails/29.jpg)
DMAIC : BTS FW Limits
Green limit indicates direction a metric can deviatefrom the average and have desired results.Red indicates direction of undesired results.
Metric Average Std Dev Lower Limit
Upper Limit
% Planning Time 18 % 8 % 10 % 26 %
% Req Time 15 % 8 % 7 % 23 %
% Code Time 21 % 12 % 11 % 33 %
Code Insp. DDR 1.89 .5 1.39 2.39
2SR Req. Churn 1.76 1.89 0 3.65
![Page 30: Identifying and Using a Project’s Key Subprocess Metrics](https://reader035.fdocuments.us/reader035/viewer/2022070502/56814c74550346895db994e0/html5/thumbnails/30.jpg)
Subprocess Metrics Notes Initial Data Left Much to be “Desired” Despite Poor Data, the Analysis
Identified:• Which Metrics and Processes Are Significant• Prediction Formulas Based on Project’s Data• Insight into Factors’ Effects• Limits for Monitoring the Factors
![Page 31: Identifying and Using a Project’s Key Subprocess Metrics](https://reader035.fdocuments.us/reader035/viewer/2022070502/56814c74550346895db994e0/html5/thumbnails/31.jpg)
DMAIC : Pilot Confirmation Used prediction formulas on other projects Compared project actuals vs. predicted. Used historical data from 5 projects. Unable to compare predicted quality versus
actual. (Predicted LOC/Hr)• These projects have not been in field long enough for
CRUD to stabilize. Interesting results found on predicted LOC/Hr.
![Page 32: Identifying and Using a Project’s Key Subprocess Metrics](https://reader035.fdocuments.us/reader035/viewer/2022070502/56814c74550346895db994e0/html5/thumbnails/32.jpg)
Project A Project B Project C Project D Project E% Plan 0% 18% -65% 10% 17%% Req -28% 10% -97% 4% 17%% Code -348% 20% 0% 43% 25%Inspection DDR 1.2 1.2 0 1.51 1.89Requirements Churn-2SR 1 1.81 0 0 0Pred LOC/Hr 92.673688 4.789660 25.540537 2.288774 2.418825Pred PR Def/KLOC 7.7196696 1.33169549 28.1019536 3.39281 0.4010827
Actual LOC/Hr 1.21 1.79 0.829 2.039 1.945LOC/Hr Delta 99% 63% 97% 11% 20%
DMAIC : Predicted vs. Actual LOC/Hr
•Projects A, B, and C projects had huge deviations.• Projects D and E were within 20%.
![Page 33: Identifying and Using a Project’s Key Subprocess Metrics](https://reader035.fdocuments.us/reader035/viewer/2022070502/56814c74550346895db994e0/html5/thumbnails/33.jpg)
DMAIC : Improve Performance? So what? How do you use this information? Does Project Management have confidence
in this analysis?
![Page 34: Identifying and Using a Project’s Key Subprocess Metrics](https://reader035.fdocuments.us/reader035/viewer/2022070502/56814c74550346895db994e0/html5/thumbnails/34.jpg)
More emphasis on statistically significant activities
Resulting in Increased Productivity
On-Time Delivery Desired Functionality Delivered
Improved Quality
DMAIC : Applying Analysis
![Page 35: Identifying and Using a Project’s Key Subprocess Metrics](https://reader035.fdocuments.us/reader035/viewer/2022070502/56814c74550346895db994e0/html5/thumbnails/35.jpg)
DMAIC : Agile Processes BTS FW Adopted Agile Practices
Iterative Development Prioritizes Requirements Negates Requirements Churn
Pair Programming Optimizes Coding and Inspection Time
Minimal Documentation Moves effort from non-statistical activities.
![Page 36: Identifying and Using a Project’s Key Subprocess Metrics](https://reader035.fdocuments.us/reader035/viewer/2022070502/56814c74550346895db994e0/html5/thumbnails/36.jpg)
DMAIC : Agile Pilot Results Productivity
0.00291 KLOC/Hr 20% improvement from 0.002399
Inspection Defect Detection Rate 1.18 Defects/Hr Detected 48% improvement from 0.8
Quality 0 Post Release Defects!
![Page 37: Identifying and Using a Project’s Key Subprocess Metrics](https://reader035.fdocuments.us/reader035/viewer/2022070502/56814c74550346895db994e0/html5/thumbnails/37.jpg)
DMAIC : Agile Pilot Results Customer Wants
Functionality – All functionality delivered Zero Defects – No customer found defects On Time – Product delivered 6 months early!
![Page 38: Identifying and Using a Project’s Key Subprocess Metrics](https://reader035.fdocuments.us/reader035/viewer/2022070502/56814c74550346895db994e0/html5/thumbnails/38.jpg)
DMAIC : Agile Monitoring Monitor Iterations Not Phases Refactoring Subprocess Monitoring Two Agile projects in-work now
![Page 39: Identifying and Using a Project’s Key Subprocess Metrics](https://reader035.fdocuments.us/reader035/viewer/2022070502/56814c74550346895db994e0/html5/thumbnails/39.jpg)
DMAIC : Agile Monitoring Monitoring
LOC per week Defects caught per week by inspection Defects caught per week by test Time spent per week Ratio of new work to correction work.
![Page 40: Identifying and Using a Project’s Key Subprocess Metrics](https://reader035.fdocuments.us/reader035/viewer/2022070502/56814c74550346895db994e0/html5/thumbnails/40.jpg)
DMAIC : Agile Metrics% Fault Work
0%
2%
4%
6%
8%
10%
12%
14%
16%
18%
20%
7/31/2005 8/7/2005 8/14/2005 8/21/2005 8/28/2005
![Page 41: Identifying and Using a Project’s Key Subprocess Metrics](https://reader035.fdocuments.us/reader035/viewer/2022070502/56814c74550346895db994e0/html5/thumbnails/41.jpg)
SummarySummary Save your “heroes” for real crises. Save your “heroes” for real crises. Understand subprocesses Understand subprocesses Monitor subprocessesMonitor subprocesses Seek to optimize key subprocessesSeek to optimize key subprocesses
![Page 42: Identifying and Using a Project’s Key Subprocess Metrics](https://reader035.fdocuments.us/reader035/viewer/2022070502/56814c74550346895db994e0/html5/thumbnails/42.jpg)
RecommendationsRecommendations Examine current project data, it Examine current project data, it
could prove to be very valuable!could prove to be very valuable! Improve data capture on important Improve data capture on important
data.data. Use the data as a guideline, but Use the data as a guideline, but
experience can never be discounted.experience can never be discounted.
![Page 43: Identifying and Using a Project’s Key Subprocess Metrics](https://reader035.fdocuments.us/reader035/viewer/2022070502/56814c74550346895db994e0/html5/thumbnails/43.jpg)
THANK YOU!THANK YOU!Jeff S. HolmesJeff S. Holmes
Principal Staff Software EngineerPrincipal Staff Software EngineerMotorola Six Sigma Black BeltMotorola Six Sigma Black Belt
Fort Worth BTS Development TeamFort Worth BTS Development TeamFort Worth, TXFort Worth, TX817-245-7053817-245-7053
[email protected]@Motorola.com