7/31/2019 Estimator_Featured Article II
1/22
Featured Article II
A Roadmap for Balanced Productivity Metrics: Are We There Yet?
By Jim Mayes (IT Metrics Strategies, August 2000)
The Balanced Approach
Philosophy
IT outsourcing continues to be a popular course of action for streamlining
businesses. These arrangements always include measurements of
performance and productivity improvement. The questions that we should be
asking are:
Why isnt it just as important to measure performance and productivity
related to in-house IT?
Why dont we develop in-sourcing contracts, as an alternative to out-
sourcing?
Whether IT is inhouse or outsourced, performance and productivity are
important.
You know how the kids in the back seat are always asking: Are we there yet?
Most companies make outsourcing decisions without even knowing their
current IT capability and are often surprised when they find out that they
were better off prior to outsourcing. Similarly, many companies embark on
software process improvement programs without knowing their existing level
of performance and without any metrics in place to measure productivity
improvement. Its like the old saying If you dont know where you are, a map
wont help. This applies to measuring progress as well. It is very difficult to
know if were there yet, if we dont know where we started and exactly
where we still need to go.
7/31/2019 Estimator_Featured Article II
2/22
Definition of BPM
Balanced Productivity Metrics (BPM) incorporates the use of both quantitative
and qualitative data for measuring performance and productivityimprovement. The primary goals of a Software Engineering Institute (SEI)
Capability Maturity Model (CMM) based process improvement program are to
improve productivity, improve quality, and reduce risk via consistent
processes. BPM focuses on the SEI CMM core measures (size, time, effort, and
defects), as well as other data collected to measure process improvement. A
multi-dimensional approach is needed to measure productivity improvement
in a way that also provides an understanding of the environmental factors
(and other significant factors) that influence project productivity.
The BPM approach is used to measure productivity with regard to software
development and maintenance. BPM is based on the principle that the
management of productivity improvement should focus on achieving a
balance of time (schedule), cost (effort), and quality (defect rate) [TCQ] 1
improvement. It is consistent with a balanced scorecard philosophy, since
these fundamental metrics components cannot be observed independently
with regard to providing a valid productivity assessment. Balancing TCQ is
much like filling a balloon. If it is filled or compressed too much, it bursts. If it
doesnt burst and it is compressed in one area, it expands in another.Likewise, if a projects schedule (time) is compressed too much, the project
will either fail or the project effort (cost), project related defects and
production defects (quality) will be disproportionably increased, which
increases maintenance effort (cost). The reverse is also true. Project
schedules can be increased too much, which does reduce effort and defects,
but may not provide the desired business value. Therefore, a balanced
measurement approach is needed.
To quote Karl Wiegers:
A risk with any metrics activity is dysfunctional measurement, in which
participants alter their behavior to optimize something that is being measured,
rather than focusing on the real organizational goal. For example, if we are
7/31/2019 Estimator_Featured Article II
3/22
measuring output productivity but not quality, expect some developers to
change their coding style to expand the volume of material they produce, or to
code quickly without regard for bugs. I can write code very fast if it doesnt
actually have to run correctly. The balanced set of measurements helps
prevent dysfunctional behavior by monitoring the groups performance in
several complementary aspects of their work that lead to project success.2
The two primary BPM reporting components for providing a balanced set of
metrics, are the:
Productivity Metrics Report (PMR), and the
Process Performance Report (PPR).
PMR Quantitative Measures
The quantitative measures associated with the Productivity Metrics Report
(PMR) are used to assess productivity improvement. The initial assessment
period is considered the baseline year. Improvement is normally expected at
the end of the second year over the baseline year. Every year thereafter,improvement is expected over the previous year.
The PMR uses a combination of metrics for calculating the BPM Score, which
measures success in achieving productivity improvement goals. The project
metrics are calculated for each software enhancement and new development
project, and then averaged for each productivity metrics component. The
maintenance metric is calculated with regard to the yearly hours required to
provide ongoing support for the total inventory of supported applications. Themetrics components to be included in calculating the productivity score are
shown in Table 1.
Metrics Component Calculation
7/31/2019 Estimator_Featured Article II
4/22
Project Schedule Duration = Total Project Months / Total Projects Units of
Product Measurement (UPM) = Months per UPM
(Function Points, Lines of Code, etc.)
Project Output Productivity = Total Project Hours / Total Projects Unit of
Measure
= Effort Hours per UPM (Could also calculate as FP per
hour, etc.)
Project Quality = Mean Time To Defect (MTTD), for each project is
calculated by dividing the number of days exposure
during the first 30 days after deployment by thenumber of defects found
= SUM (Each Projects MTTD)/ Total Number of
Projects
= Weighted Average MTTD
Maintenance Output Productivity = Total Maintenance Hours / Total
Maintenance UPM
= Effort Hours per UPM
Table 1 BPM Components and Calculations
PPR Qualitative Measures
The qualitative measures associated with the Process Performance Report
(PPR), when analyzed along with the quantitative measures, provide insightinto the overall process productivity and direction toward achieving process
improvement goals. Looking at the BPM Score which is a letter grade or
number only provides the current status or contractual measure. Unless
there is an underlying analysis behind that score, there is little information to
use for identifying the next steps. The PPR provides information that can be
used to identify problems and areas that have improved, for managing
7/31/2019 Estimator_Featured Article II
5/22
process productivity improvement activities. This report provides the
following analyses, trending, and comparisons to previous period:
Demographic data analysis related to organization profiles, project
types, size, complexity, and languages
Statistical analysis of metrics categories against previous performance
trends
Analysis of delivery performance, including requirements volatility
Significant factors impacting project results
Analysis of customer satisfaction survey results in relation to project
data
Analysis of environmental factors related to tools/methodology,technical difficulty, and personnel
Process Overview
Administration
The BPM process involves joint responsibilities. The client performancemetrics manager (client PMM) manages the process and the database and
serves as team lead for a productivity metrics team. The client PMM will
prepare the PPR and the IT metrics group will prepare the PMR, which includes
calculating the BPM Score. Change management with regard to this process
will be the responsibility of the productivity metrics team. (These roles may
vary depending on organizational structure. Additional roles are identified in
Table 2.)
Frequency
The PMR should be calculated annually for productivity scoring related to
annual software process improvement goals, or contractual agreements in the
case of outsourcing. For purposes of monitoring progress and enabling the
parties to promptly address any service or process deficiencies, quarterly BPM
Reports containing cumulative data should be created and reviewed.
7/31/2019 Estimator_Featured Article II
6/22
If, at any time the BPM Score is to be calculated, and one or more metrics or
component metrics cannot be calculated, either because the relevant data is
not available, or the applicable scoring scale has not been adopted or is not
yet in effect, then the BPM Score should be calculated disregarding such
metrics. The weights of the other metrics or component metrics within a given
metric shall be increased proportionately such that other metrics or
component metrics maintain their respective relative weights.
Data Collection and Dictionary
Data consistency is the key to having a good decision support database;
therefore, to insure consistency, the definitions must be documented. Also,
when building an internal historical project repository, the project data points
must be categorized. This will provide stratification such that apples-to-apples
project comparisons can be made based upon project type, environment, user
organization, development organization, language, etc. The data collectionrequirements should also be defined and documented. The collection of
software project data during the project is designed to track the status of
projects, enable predictive analysis, and facilitate the collection of project
closeout data required for the BPM reports. Upon completion of the project,
the data should be summarized in a detailed data collection form.
Tools/Database
BPM project data should be stored in a metrics database. The project-level
information in this database can be used for planning new projects and
evaluating risks. At a minimum, this can be done by using data from similar
historical projects for providing an analogy to the project being planned. This
is associated with the CMM Level 2 Software Project Planning key process area
and supports the software project estimation and risk management processes.
(For more information on this, see my December 2000 ITMS article, Saving
the World One Project at a Time: Planning by the Numbers.)8
Process Steps
The BPM process-flow schematic is illustrated in Figure 1. The BPM schematic
description and responsibilities are provided in Table 2. The BPM process
begins with collecting baseline project data and the development of a
7/31/2019 Estimator_Featured Article II
7/22
productivity baseline. The schematic illustrates the next steps after the
baseline has been established.
7/31/2019 Estimator_Featured Article II
8/22
Figure 1 BPM Process Flow Schematic
Balanced Productivity Metrics Process- Schematic Description
Activity Performed By1. Software Enhancement or NewDevelopment Project LaunchedWhenever new projects are launched,it must be determined if they meetthe criteria for being included in theproductivity metrics process. Theremay be some projects that werelaunched prior to the implementation
of this process that should be includedas well. Also, projects should beincluded that were not initially inscope, but changed such that theysubsequently fit the criteria.
2. Track Project Phase Effort,Schedule, and Defects The actualeffort hours, start date, and end datefor each project phase should betracked. The project phases include
planning, requirements analysis, andmain build (detailed design deployment). Defects are tracked byseverity (1critical, 2 serious, 3moderate, and 4 tolerable/cosmetic)from the beginning of system testuntil deployment. Also, defects mustbe tracked by severity, for the first 30days after deployment.
Activity
Performed By
IT project manager
7/31/2019 Estimator_Featured Article II
9/22
2. Track Project Phase Effort, Schedule, and Defects
The actual effort hours, start date, and end date for each project phase should
be tracked. The project phases include planning, requirements analysis, and
main build (detailed design deployment). Defects are tracked by severity
(1critical, 2 serious, 3 moderate, and 4 tolerable/cosmetic) from the beginning
of system test until deployment. Also, defects must be tracked by severity, for
the first 30 days after deployment.
IT project manager
3. Schedule and Perform Software (UPM) SizingFunction points, source lines of code or some other method should be used to
size the project and application.
IT project manager and technician
4. Enter Project Data into Detailed Data Collection Form
On project completion, project data is entered in the detailed data collection
form or spreadsheet. This form is sent to the IT program manager prior to the
project closeout meeting.
IT project manager
5. Review Project Data
The project information provided in the detailed data collection form should bereviewed and validated during the project closeout meeting. Agreement
should be reached on the ratings and results provided. This can provide an
opportunity for joint problem solving related to project impacts and process
improvements needed. The validated detailed data collection form should
then be forwarded to the Client performance metrics manager.
7/31/2019 Estimator_Featured Article II
10/22
Client program manager
6. Enter Detailed Project Data
The data contained in the detailed data collection form is entered into the
metrics database.
Client performance metrics manager
7. Track Ongoing Maintenance Effort Hours Actual effort
hours charged to ongoing maintenance support must be tracked.
IT project manager
8. Schedule and Perform Application (UPM) Baseline Sizing
Function points, source lines of code, or some other method should be used to
size the applications being maintained.
IT project manager and sizing specialist
9. Update Maintenance Effort and Size Spreadsheet
A data repository should be maintained containing maintenance productivity
data.
IT metrics group
10. Prepare Quarterly or Yearly BPM ReportThe PMR containing the BPM score, is created by the IT metrics group, and the
Performance Manager creates the PPR.
IT metrics group and client performance metrics manager
7/31/2019 Estimator_Featured Article II
11/22
Table 2 BPM Process Schematic Description
Critical Success Factors
The following factors are critical to the success of a Balanced Productivity
Metrics Program: Client and IT partnering relationship with open
communication paths
Senior management support and a documented policy statement
Client program management, client performance metrics management,
IT project management, and IT metrics group support, and synchronized
processes
Accurate data collection and sizing on a timely basis
Using project-level quantitative and qualitative metrics data for software
project planning and process improvement (i.e., dont just accumulate it for
contractual purposes)
Communication, training, and mentoring related to the BPM process
Scoring
Metrics Component Scores
Each BPM Metric Component will be given a letter score determined in
accordance with the scoring scale shown in Table 3, assuming a 5% yearly
improvement for each component is desired. This letter grade will be
converted to a numerical score using the scale also shown in Table 3. (The
percentages and letter grades can be adjusted based upon the yearlyimprovement desired for each TCQ category.)
Grade
7/31/2019 Estimator_Featured Article II
12/22
Numerical Score
Component Grade (calculated for each Metrics Component and then converted
to numerical score)
A
5.0
A for = 5.5% or better (exceeded)
B
4.0
B for = 4.5% to 5.4% (expected)
C
3.0
C for = 0 to 4.4%
D
2.0
D for =
7/31/2019 Estimator_Featured Article II
13/22
Metrics Component Weighting and Balanced Scoring
Business value should determine the weightings with regard to Project TCQ
and Maintenance, since this is a business decision as to which component hasthe highest priority, or if they are all equally important. Of course this will
vary from project to project, depending on the circumstances, therefore the
BPM weightings should reflect the overall philosophy for the organization.
Component/Category weightings should be established once per year, at the
beginning of each annual measurement cycle. If everything were equally
weighted, the total BPM composite numerical score would be calculated as
shown it Table 4, and the final composite BPM grade would be calculated as
shown in Table 5. Examples of BPM Score Calculations are provided in Tables
6, 7, and 8.
Component
Weight
Composite Numerical Score Calculation
Project schedule duration
25%
Project schedule numerical score (T3) x 0.25 (T4)
Project output productivity
25%
+ Project output numerical score (T3) x 0.25 (T4)
Project quality
7/31/2019 Estimator_Featured Article II
14/22
25%
+ Project quality numerical score (T3) x 0.25 (T4)
Maintenance output productivity
25%
+ Maintenance output numerical score (T3) x 0.25(T4)
Total
100%
= Composite numerical score (convert to grade in T5)
Table 4 (T4) Numerical Component Weighting and Composite Calculation
Grade
Numerical Score
Assessment
A
4.5 5.0
Exceeds standards
B
4.0 4.4
Meets standards
7/31/2019 Estimator_Featured Article II
15/22
C
3.0 3.9
Does not meet standards
D
2.0 2.9
Does not meet standards
F
0.0 1.9
Does not meet standards
Table 5 (T5) Final BPM Composite Grading
Metrics Component
Baseline Values
Year 2 Values
% Change
Component Grade (T3)
Numerical Score (T3) x Weight (T4)
Composite Score and Grade (T5)
7/31/2019 Estimator_Featured Article II
16/22
Project Schedule Duration
1.1
1.045
5% better
B
4 x .25
1
Project Output Productivity
3.6
3.42
5% better
B
4 x .25
1
Project Quality
4.5
4.73
5% better
B
4 x .25
1
Maintenance Output Productivity
7/31/2019 Estimator_Featured Article II
17/22
1.8
1.71
5% better
B
4 x .25
1
Total
4 = B (meets standards)
Table 6 Example 1: BPM Score Calculation
Metrics Component
Baseline Values
Year 2 Values
% Change
Component Grade
Component Score x Weight
7/31/2019 Estimator_Featured Article II
18/22
Composite Scoring
Project Schedule Duration
1.1
1.045
5% better
B
4 x .25
1
Project Output Productivity
3.6
3.49
3% better
C
3 x .25
0.75
Project Quality
4.5
4.73
5% better
B
4 x .25
1
7/31/2019 Estimator_Featured Article II
19/22
Maintenance Output Productivity
1.8
1.71
5% better
B
4 x .25
1
Total
3.75 = C (does not meet standards)
Table 7 Example 2: BPM Score Calculation
Metrics Component
Baseline Values
Year 2 Values
% Change
Component Grade
7/31/2019 Estimator_Featured Article II
20/22
Component Score x Weight
Composite Scoring
Project Schedule Duration
1.1
1.045
5% better
B
4 x .25
1
Project Output Productivity
3.6
3.39
6% better
A
5 x .25
1.25
Project Quality
4.5
4.73
5% better
B
4 x .25
7/31/2019 Estimator_Featured Article II
21/22
1
Maintenance Output Productivity
1.8
1.75
3% better
C
3 x .25
0.75
Total
4 = B (Meets Standards)
Table 8 Example 3: BPM Score Calculation
Conclusion
The measurement of productivity and performance is just as important for
inhouse IT as it is for an outsourced IT environment. A balanced approach, as
provided by the BPM process, is needed in order to ensure that all aspects of
productivity are assessed and that the correct behavior with regard to
performance is achieved. Some outsourcing vendors will argue that only one
measure should be used, such as effort per unit of output, but to most clients,
7/31/2019 Estimator_Featured Article II
22/22
schedule and quality are equally important. Therefore, measures should be in
place such that schedule or quality cannot be sacrificed, and to prevent
dysfunctional measurement behavior from occurring. It is beneficial to IT that
all factors be measured, due to the clients ever changing TCQ priorities. In
addition to balancing the TCQ metrics and quantitatively tracking projects, it is
also important to understand the relationship between the quantitative and
qualitative measures that influence productivity. When we trend the
quantitative and qualitative data and compare the results to previous internal
benchmarks, our corporate knowledge will be increased such that we can
make intelligent business decisions. The BPM process can help us to
understand the factors that influence our productivity, know where we are on
the map, effectively predict where we are headed, and recognize when we get
there.
Top Related