Metrics and Performance Measurement
Transcript of Metrics and Performance Measurement
-
8/20/2019 Metrics and Performance Measurement
1/47
Metrics and Performance MeasurementSystem for the Lean Enterprise
Professor Deborah Nightingale
October 24,2005
-
8/20/2019 Metrics and Performance Measurement
2/47
Overview
• Metrics and Performance measurement
•
Why measure? What is performance measure? What are goodmetrics?
• Performance measurement and Lean Transformation
• Current practices and Performance measurement
frameworks
•
Performance measurement system for the lean
enterprise
ESD.61J / 16.852J: Integrating the Lean Enterprise © Deborah Nightingale, 2005 Massachusetts Institute of Technology Page 2
-
8/20/2019 Metrics and Performance Measurement
3/47
Why Measure?
• “Performance control systems can
serve two purposes, to measure
and to motivate.”
- H. Mintzberg,
The Structure of Organizations, 1979
•
“The firm becomes what itmeasures”
- Hauser and Katz,
You are What You Measure, 2002
•
Metrics serve multiple purposes!
Strategy
Feed Decision
Back Making
Measures Outcomes Actions
Robert Nixon (1990)
ESD.61J / 16.852J: Integrating the Lean Enterprise © Deborah Nightingale, 2005 Massachusetts Institute of Technology Page 3
-
8/20/2019 Metrics and Performance Measurement
4/47
What is performance
measurement?• Performance measurement is the process of measuring
efficiency, effectiveness and capability, of an action or a
process or a system, against given norm or target.
• Effectiveness is a measure of doing the right job - the extent to
which stakeholder requirements are met.
•
Efficiency is a measure of doing the job right - howeconomically the resources are utilized when providing a given
level of stakeholder satisfaction.
• Capability is a measure of ability required to do both the job
right and right job, in the short term as well as the long term.This can be tangible, such as, resources, technology, or
intangible, such as a corporate culture.
ESD.61J / 16.852J: Integrating the Lean Enterprise © Deborah Nightingale, 2005 Massachusetts Institute of Technology Page 4
-
8/20/2019 Metrics and Performance Measurement
5/47
Characteristics of good metrics
• Metrics are meaningful, quantified measures
•
Metric must present data or information that allowsus to take action
•
Helps to identify what should be done
• Helps to identify who should do it
• Metrics should be tied to strategy and to “core”
processes - indicate how well organizational
objectives and goals are being met
• Metrics should foster process understanding and
motivate individual, group, or team action and
continual improvement.
ESD.61J / 16.852J: Integrating the Lean Enterprise © Deborah Nightingale, 2005 Massachusetts Institute of Technology Page 5
-
8/20/2019 Metrics and Performance Measurement
6/47
Broad Criteria
A “Good” Metric Satisfies Three
1. Strategic• Enable strategic planning and then drive deployment of the actions
required to achieve strategic objectives• Ensure alignment of behavior and initiatives with strategic objectives
• Focus the organization on its priorities
2. Quantitative
•
Provide a clear understanding of progress toward strategic objectives• Provide current status, rate of improvement, and probability of
achievement
• Identify performance gaps and improvement opportunities
3. Qualitative• Be perceived as valuable by your organization and the people involved
with the metric
ESD.61J / 16.852J: Integrating the Lean Enterprise © Deborah Nightingale, 2005 Massachusetts Institute of Technology Page 6
-
8/20/2019 Metrics and Performance Measurement
7/47
Metric Elements
Metric Elements
Title
Objective/purpose
Scope
Target
Formula
Units of measure
Frequency
Data source
Owner
Comments
Explanation
Use exact names to avoid ambiguity
The relation of the metric with the organizational objectives must be clear
States the areas of business or parts of the organization that are included
Benchmarks must be determined in order to monitor progress
The exact calculation of the metric must be known
What is/are the unit(s) used
The frequency of recording and reporting of the metric
The exact data sources involved in calculating a metric value
The responsible person for performance of that part of the organization,
collecting data and reporting the metric
Outstanding issues regarding the metric
Adapted from Neely, A., et al. (1995a) Performance measurement system design. 15, 80.
ESD.61J / 16.852J: Integrating the Lean Enterprise © Deborah Nightingale, 2005 Massachusetts Institute of Technology Page 7
-
8/20/2019 Metrics and Performance Measurement
8/47
Role of Performance Measurement
• Monitoring • Measuring and recording actual performance
•
Control• Identifying and attempt to close the gap between planned target
and actual performance
• Improvement•
Identify critical improvement opportunities
• Coordination• Information for decision making – Leading Indicators
•
Internal communication across processes•
External communication with stakeholders
• Motivation•
Align Behavior and encourage transformation
Source: Vikram Mahidhar
ESD.61J / 16.852J: Integrating the Lean Enterprise © Deborah Nightingale, 2005 Massachusetts Institute of Technology Page 8
-
8/20/2019 Metrics and Performance Measurement
9/47
© Deborah Nightingale, 2005 Massachusetts Institute of Technology ESD.61J / 16.852J: Integrating the Lean Enterprise Page 9
Transformation to the Lean Enterprise
Functional Enterprise
6
5
4
3
2
1
7
1
2 2
3 3
6
5
4
3
2
1
7
1
2 2
3 3
6
5
4
3
2
1
7
1 1
2 2
3 3
S a l e
s
E n g i n
e e r i n g
Manufacturing
P u r c h a s i n g
R & D
Value Delivery
Stakeholder Objectives
Customer
EmployeesSuppliers
Shareholders Community
Enterprise Leadership &Governance Processes
Enabling InfrastructureProcesses
Lifecycle Processes
Processes
S a l e s
C
U
S
T
O
M
E
R
Centers of
Excellence
CEO
Executive board
E n g i n e e r i n g
M a n u f a c t
u r i n g
R & D
Process Enterprise Lean Enterprise
Source: Vikram Mahidhar
-
8/20/2019 Metrics and Performance Measurement
10/47
Metrics Challenge Today
• Hierarchical organizational architectures givingway to networked enterprise architectures
•
The evolving structure and dynamics of networked enterprises display immensecomplexity
• Metrics response to such complexity has been adisappointment:
• Hierarchical metrics mindset still continues
•
Response to greater complexity has been a metrics explosion
• Challenge: How best to design metrics systems
for networked enterprises?ESD.61J / 16.852J: Integrating the Lean Enterprise © Deborah Nightingale, 2005 Massachusetts Institute of Technology Page 10
-
8/20/2019 Metrics and Performance Measurement
11/47
© Deborah Nightingale, 2005 Massachusetts Institute of Technology ESD.61J / 16.852J: Integrating the Lean Enterprise Page 11
ROIC = Return on Invested CapitalORPIC* = Operational Readiness per Invested Capital
Performance measurement system for
Lean Transformation
T
r an sf or m
a t i on ov
er T i m e
T
r an sf or m
a t i on ov
er T i m e
Knowledge and Behavior Metrics and Assessments
New approaches (training andintroduction of new methods)
Engagement in “LAI-venue” with like-
minded people
Enterprise simulation, Lean Now and
LAI knowledge area teams
Local efforts and new capabilities
Industry Government
Skills, training hours, certification,lean deployment, joint assessments
and efforts
New local behavior Shift in thinking and behavior
New routines and ways of doing
business
Organization and group culture change
Cycle time, quality, WIP, on-timedelivery, customer satisfaction,employee turnover and attitude,
organizational climate and LESATmaturity
Local results and visible indicators
Industry Government
New enterprise capability
Integration of processes/methods &
tools supporting transformation across
the value stream enabling new
enterprise capabilities
ROIC
Enterprise impact and results
Industry GovernmentORPIC
Adapted from : Noel Nightingale, 2004
-
8/20/2019 Metrics and Performance Measurement
12/47
Generic Enterprise Management Process
Value delivery
Management Assessment j
Shareholder Value
Customer Satisfaction
Employee Satisfaction
Other stakeholders
Strategy Formulation
- Competitive Intelligence
- Internal Assessment
Strategic and Operational Planning
Resource Allocation Plan
Strategy ExecutionCascaded Ob ectives
VSM and Project Prioritization
Performance Mgmt Process
Communication
Operations
Balanced Score Cards
Operations Management
Financial ManagementHuman Resource
Information Systems
External
Environment
ESD.61J / 16.852J: Integrating the Lean Enterprise © Deborah Nightingale, 2005 Massachusetts Institute of Technology Page 12
-
8/20/2019 Metrics and Performance Measurement
13/47
Strategic Metrics
• ROIC (Return on
Invested Capital)• Economic Value Add
(EVA)
• Net Operating Profit
• Inventory Turnover
• Revenue
• Cash flow
• Market Position
• Wall Street Expectations
Operations
Management Assessment
Value delivery
Strategy Execution
Strategy Formulation
- Competitive Intelligence
- Internal Assessment
Strategic and Operational Planning
Resource Allocation Plan
Efficiency metrics and Lagging indicators
ESD.61J / 16.852J: Integrating the Lean Enterprise © Deborah Nightingale, 2005 Massachusetts Institute of Technology Page 13
-
8/20/2019 Metrics and Performance Measurement
14/47
Tactical Metrics
• Financial Turnover
•
Budget/Cost andExpenses
• Cost of quality
• Productivity
• Supply Chain
Excellence
Operations
j
Value Delivery
Strategy Formulation
Strategy ExecutionCascaded Ob ectives
VSM and Project Prioritization
Performance Mgmt Process
Communication
• Regulatory and social
compliance
Accuracy and timeliness of reporting and controlESD.61J / 16.852J: Integrating the Lean Enterprise © Deborah Nightingale, 2005 Massachusetts Institute of Technology Page 14
-
8/20/2019 Metrics and Performance Measurement
15/47
Operational Metrics
• Safety
•
Quality
• Environment
• Cost/Manufacturing
Efficiency• Delivery
• Time to market
•
Education anddevelopment
• Time to Hire
Value Delivery
Strategy Formulation Strategy Execution
Operations
Balanced Score Cards
Operations Management
Financial Management
Human ResourceInformation Systems
ESD.61J / 16.852J: Integrating the Lean Enterprise © Deborah Nightingale, 2005 Massachusetts Institute of Technology Page 15
-
8/20/2019 Metrics and Performance Measurement
16/47
Value Delivery Metrics
•
Stock Price
• Revenue
• On time delivery
• Customer satisfaction
and loyalty
• Employee Satisfaction
• New product
Introduction
OperationsValue Delivery
Strategy Formulation Strategy Execution
Shareholder Value
Customer Satisfaction
Competition
Macro Economic Trends
ESD.61J / 16.852J: Integrating the Lean Enterprise © Deborah Nightingale, 2005 Massachusetts Institute of Technology Page 16
-
8/20/2019 Metrics and Performance Measurement
17/47
Management Process
Value delivery
Management Assessment j
Time lags - Performance
Shareholder Value
Customer Satisfaction
Employee Satisfaction
Other stakeholders
Strategy Formulation
- Competitive Intelligence
- Internal Assessment
Strategic and Operational Planning
Resource Allocation Plan
Strategy ExecutionCascaded Ob ectives
VSM and Project Prioritization
Performance Mgmt Process
Communication
Operations
Balanced Score Cards
Operations Management
Financial Management
Human Resource
Information Systems
External
Environment
ESD.61J / 16.852J: Integrating the Lean Enterprise © Deborah Nightingale, 2005 Massachusetts Institute of Technology Page 17
Ti l P f
-
8/20/2019 Metrics and Performance Measurement
18/47
Management Process
Strategy
Formulation
Strategy
Execution
Operations Reporting
Yearly
Month
Day
Quarter
Time lags - Performance
ESD.61J / 16.852J: Integrating the Lean Enterprise © Deborah Nightingale, 2005 Massachusetts Institute of Technology Page 18
-
8/20/2019 Metrics and Performance Measurement
19/47
Performance Measurement &
Management Frameworks
• Balanced Scorecard – More than 50%
companies have implemented in the US• Performance Prism
•
European Foundation QualityFramework
•
X-Matrix (EVSMA)
ESD.61J / 16.852J: Integrating the Lean Enterprise © Deborah Nightingale, 2005 Massachusetts Institute of Technology Page 19
-
8/20/2019 Metrics and Performance Measurement
20/47
© Deborah Nightingale, 2005 Massachusetts Institute of Technology ESD.61J / 16.852J: Integrating the Lean Enterprise Page 20
Balanced Scorecard/Strategy Map
Long-Term
Shareholder Value
Human capital
Information capital
Organization capitalCulture Leadership
Learning
& Growth
Operations
-Supply
-Production
-Quality
-Logistics
Customer Relationship
-Selection
-Acquisition
-Retention
-Growth
Innovation Process
-Opportunity Ident.
- R&D Portfolio
- Design/Development
- Time to market
Regulatory and Social
- Environment
- Safety and Health
- Employee development
- Community
Internal
Processes
Price Quality Availability Selection Functionality Services PartnershipCustomer
Financial Improve CostStructure
Increased Asset
Utilization
Expand Revenue
Opportunities
Enhance
Shareholder value
Growth StrategyProductivity Strategy
Adapted from : Kaplan and Norton, (2000) “Having Trouble with your strategy? Then Map it!” Harvard Business Review Sept_Oct 2000
P i P f th
-
8/20/2019 Metrics and Performance Measurement
21/47
Primary Purposes of the
Balanced Scorecard
• Align a balanced set of performance metrics
with business strategy and vision
• Provide management and work teams with
the information necessary and sufficient to
meet their objectives and goals
• Create “line-of-sight” at lower levels of the
organization• Foster and support process continuous
improvement initiatives
ESD.61J / 16.852J: Integrating the Lean Enterprise © Deborah Nightingale, 2005 Massachusetts Institute of Technology Page 21
D i d Ch t i ti f P f
-
8/20/2019 Metrics and Performance Measurement
22/47
Desired Characteristics of Performance
Measurement Systems
• Performance Measures should support the
strategic intentions of the organizations
• Managers at all levels should understand
both drivers and results of their activities.
• Explicating Cause-Effect relationships
between drivers and results
ESD.61J / 16.852J: Integrating the Lean Enterprise © Deborah Nightingale, 2005 Massachusetts Institute of Technology Page 22
Lean Enterprise Metric
-
8/20/2019 Metrics and Performance Measurement
23/47
© Deborah Nightingale, 2005 Massachusetts Institute of Technology ESD.61J / 16.852J: Integrating the Lean Enterprise Page 23
S t a k e h ol d e r
V a l u e
Lean Enterprise Metric
Interdependencies
L i f e C y c l e
P r o c e s s e s
E
n a b l i n g
I
n f r a s t r u c t u r e
P
r o c e s s e s
E n t e r p r i s e
L e a
d e r s h i p
P r o
c e s s e s
Production
Quality
Control
Logistics
Training
Change
Management
Strategic
Planning
Process
Yield/CostQuality
Financial
Management
Financial
HealthDelivery
Customer
Value
Source: Vikram Mahidhar
Process Control View
-
8/20/2019 Metrics and Performance Measurement
24/47
Process Control View –
Performance Measurement System
Process
ComparisonFunction
ing
Output Input
Action
Targets
Performance
measures
Strategy
Making
Behavior
Strategic
Operational
Strategic
Tactical
F
e e d
b a c k
Control
Evaluat Plan
Stakeholders
DecisionObjectives
Source: Vikram Mahidhar
ESD.61J / 16.852J: Integrating the Lean Enterprise © Deborah Nightingale, 2005 Massachusetts Institute of Technology Page 24
U d t di P f M t
-
8/20/2019 Metrics and Performance Measurement
25/47
Understanding Performance Measurement
Metric Cluster
Metric Set
Individual Metric
Illustrative
Structures – Causal Models
Example
Source: Vikram Mahidhar
ESD.61J / 16.852J: Integrating the Lean Enterprise © Deborah Nightingale, 2005 Massachusetts Institute of Technology Page 25
-
8/20/2019 Metrics and Performance Measurement
26/47
“…although there may be a potentially long list of non-financial
indicators, individual firms have to be selective by linking
explicitly their choice of indicators to their corporate strategy.”
No One “Right” Set of Metrics
• The balanced scorecard has to be tailored to
each specific company
• The resulting scorecard of indicators should
be driven by the firm’s strategy if it is not to
consist merely of a listing of indicators:
indicators, individual firms have to be selective by linking
“…although there may be a potentially long list of non-financial
explicitly their choice of indicators to their corporate strategy.”
R.S. Kaplan and D.P. Norton, Harvard Business Review , January-February, 75-85 (1996)
ESD.61J / 16.852J: Integrating the Lean Enterprise © Deborah Nightingale, 2005 Massachusetts Institute of Technology Page 26
Assessing a Performance
-
8/20/2019 Metrics and Performance Measurement
27/47
Assessing a Performance
Measurement System
• Does it clearly define what constitutes business excellence?
• Does it provide the information required to set aggressive yet
achievable strategic objectives and stretch goals?• Does it accurately portray our progress and probability of
achieving both long-term strategic objectives and near-termmilestones?
•
Does it identify the root causes of barriers?
• Does it focus the organization on the priority improvement needs?
• Does it drive the behavior and actions required to achieve the
objectives?• Does it align work with value?
• Is it easy to use?
•
Does it involve everyone?
ESD.61J / 16.852J: Integrating the Lean Enterprise © Deborah Nightingale, 2005 Massachusetts Institute of Technology Page 27
Metrics Will Change Over an Item’s
-
8/20/2019 Metrics and Performance Measurement
28/47
Raytheon Systems, 1998
Metrics Will Change Over an Item s
Life CycleEntity
Emerging
Growth
Mature
Declining
•
•
•
•
Product Concept
Development
Phase
•
•
•
•
•
•
•
•
•
•
•
•
Recognition
Learn
Practi
Expert
•
•
•
•
• Use
• Apply
•
• Deployment
• Teach
•
• Combine and evaluate
Phases Attributes
Business Cash flowCompetitive advantage
Market share
Critical Mass
In market
-out
Creative backlogPotential product revenue
Cost per feature
Time to market
Performance requirements
Predicted product quality
Design tocost
ProfitabilityMarket expansion rate
Volume impact on cost
Inventory
Customersupport
Core
Competency
ce
Inventory of skills and capabilities
Competitive advantage
Acquire knowledgeCycles of learning
Levels of use in organization
Leverage advantage
ESD.61J / 16.852J: Integrating the Lean Enterprise © Deborah Nightingale, 2005 Massachusetts Institute of Technology Page 28
-
8/20/2019 Metrics and Performance Measurement
29/47
t
Process and Metric Maturity Model
That
1
Busii
consci ing of management.
i i l
core processes
2Alignment
CoreProcessesManaged
Metrics reinforce & leveragei
3 HorizontalAlignment
CoreProcessesIntegrated
4 TotalAlignment
EnablingProcesses
Integrated
Allorganizational, geog )
i j
& optimize the whole
ith
i
5Holisticimulated
g gy setting(e.g., nodal influence,
LevelProcess
Metrics are ad hoc andprimarily results oriented.Little or no process focus.
which exists is primarily directedinternally toward local operations Initial Initial
ness process management, whichbegins & ends w th the customer isestablished, in control, and in the
ous think
Process metrics added &ntegrated w th resu t metrics
Metrics aligned betweenstrategy & daily activities in
Vertical
Common process language & specsIntegrated core processes allow a
seamless flow of work across processboundaries
activ ties across all corebusiness processes
Local interests are subordinatedto the good of the whole
metrics (process, results,raphic, etc.
align with strateg c ob ectives,provide competitive advantage
Support processes are integrated wand enable core business processesto provide competitive advantage.
Customer-focused processmanagement is appliedunconsc ously
OptimizingMetric-driven actions sdurin strate process toensure organizational alignmentbefore metrics are implemented
Process management has providedworld-class competitiveadvantageagile & forward looking)
MetricMaturityMaturity
ESD.61J / 16.852J: Integrating the Lean Enterprise © Deborah Nightingale, 2005 Massachusetts Institute of Technology Page 29
Ray heon Systems, 1998
-
8/20/2019 Metrics and Performance Measurement
30/47
Level One: Initial
• Enterprise does not manage its business
with a process focus• Many metrics sub-optimized by local organizational
interests rather than having them aligned with
customer interests and with the strategic
objectives of the enterprise
• Organizations measure the results of past actions
•
Results-oriented metrics cannot provide theleading indicators needed for timely corrective
action to change outcomes
ESD.61J / 16.852J: Integrating the Lean Enterprise © Deborah Nightingale, 2005 Massachusetts Institute of Technology Page 30
-
8/20/2019 Metrics and Performance Measurement
31/47
Vertical alignment is the alignment and reinforcement of strategic
objectives with supportive goals and progress measures at alllevels of the organization.
Level Two: Vertical Alignment
Definition
Vertical alignment i li i ics the a gnment and re nforcement of strateg
objectives with supportive goals and progress measures at alllevels of the organization.
• The business enterprise applies a process focus so it can
measure leading indicators of the expected process output• Defective process output is viewed as a process-capability
problem, not a people problem
•
Carefully chosen metrics ensure that all levels of theorganization align with strategic objectives
ESD.61J / 16.852J: Integrating the Lean Enterprise © Deborah Nightingale, 2005 Massachusetts Institute of Technology Page 31
-
8/20/2019 Metrics and Performance Measurement
32/47
A core process related to product development activities
might be documented, be in control (repeatable), be
consistently deployed across the organization, and havemeasurable improvement gains. If so, that process is
probably at or near Level 2 maturity. If the metrics
indicate variations in the process results, then they are
still at Level 1 because the process is not in control.
Level Two Example
), be
l
A core process related to product development activities
might be documented, be in control (repeatable
consistently deployed across the organization, and havemeasurab e improvement gains. If so, that process is
probably at or near Level 2 maturity. If the metrics
indicate variations in the process results, then they are
still at Level 1 because the process is not in control.
ESD.61J / 16.852J: Integrating the Lean Enterprise © Deborah Nightingale, 2005 Massachusetts Institute of Technology Page 32
-
8/20/2019 Metrics and Performance Measurement
33/47
Level Three: Horizontal Alignment
Two phases:
1. The global optimization of work flow across all process
boundaries. These boundaries become transparent to the flowof work. Metrics are customer-focused and assess theenterprise-level capability of a process to provide value from thecustomer’s perspective.
2. The global optimization of work flow across all organizationalboundaries that support or use a particular process. Metrics arecustomer-focused and assess how well the infrastructure
enables execution of customer-focused processes.
ESD.61J / 16.852J: Integrating the Lean Enterprise © Deborah Nightingale, 2005 Massachusetts Institute of Technology Page 33
L l Th E l
-
8/20/2019 Metrics and Performance Measurement
34/47
The enterprise may have several core customer-related processes
such as winning new business and developing new products. Also,
the enterprise may have many functions that support or executethese core processes.
Level Three Example
l
lso,
The enterprise may have several core customer-re ated processes
such as winning new business and developing new products. A
the enterprise may have many functions that support or execute
these core processes.
Level 3 characteristics include:
• Integrated core processes that customers see as seamless
• Minimized hand-offs or delays as work moves among
processes and sub-processes
• Management focus primarily on early process activities in a
product life cycle• Metrics insure local organizational interests (functional or
business unit) are subordinated to customer needs and what is
best for the entire business
ESD.61J / 16.852J: Integrating the Lean Enterprise © Deborah Nightingale, 2005 Massachusetts Institute of Technology Page 34
L l F T t l Ali t
-
8/20/2019 Metrics and Performance Measurement
35/47
Total alignment is the synergistic interaction of metrics from all
support processes with metrics from all core process to reinforcethe strategy and to drive business excellence.
Level Four: Total Alignment
Definition
Total alignment is the synergistic interaction of metrics from all
support processes with metrics from all core process to reinforcethe strategy and to drive business excellence.
• All employees clearly see where the business is headed
and how they can make a difference.
• Horizontal integration (Level 3) provides employees with
“line of sight” to customer value. Dramatic performance
improvements can occur at this level.
•
Total enterprise-level alignment (Level 4) is required to
overcome the major systemic barriers to great performance
and to embed the long-term gains into the fabric of the
organization’s culture.
ESD.61J / 16.852J: Integrating the Lean Enterprise © Deborah Nightingale, 2005 Massachusetts Institute of Technology Page 35
L l F E l
-
8/20/2019 Metrics and Performance Measurement
36/47
At Level 4, the enterprise begins asking howenabling processes create competitive
advantage for the core customer-related
processes, rather than what they do to improve
themselves.
Level Four Example
l
l
At Leve 4, the enterprise begins asking howenabling processes create competitive
advantage for the core customer-re ated
processes, rather than what they do to improve
themselves.
• Total enterprise alignment is required to overcome the major
systemic barriers to great performance
ESD.61J / 16.852J: Integrating the Lean Enterprise © Deborah Nightingale, 2005 Massachusetts Institute of Technology Page 36
L l Fi O ti i i
-
8/20/2019 Metrics and Performance Measurement
37/47
Level Five: Optimizing
•
From a process perspective, the enterprise will have muchgreater influence on the market than its size might indicate. The
agile and forward-looking enterprise will be able to foresee
events and respond to those events before they occur.
• From a metrics perspective, the enterprise will be able not only
to simulate and predict the outcome of a strategy before its
deployment, but also to predict the effect of specific metrics onthe outcome of that strategy before choosing metrics.
ESD.61J / 16.852J: Integrating the Lean Enterprise © Deborah Nightingale, 2005 Massachusetts Institute of Technology Page 37
Case Study: Nike
-
8/20/2019 Metrics and Performance Measurement
38/47
Case Study: Nike
• The study focuses on European Operations of Nike.
• The company is organized around three lines of business:Apparel – 60,000 SKU, Footwear – 25000 SKU, and Equipment 1000 SKU.
•
90% of the business comes from 6 countries in Europe.
• The product life cycles are short.
• Uncertainty of demand is an important characteristic.
•
Company was continuously improving its supply chainmanagement through a better integration of operations acrosssubsequent echelons and separate functions in the valuechain.
•
As part of these efforts management decided to assign someof their resources for improvement projects to performancemanagement. The integration of various local performanceindicators into a company-wide consistent system wasrequired.
Lohman, C., et al. (2004) “Designing a performance measurement system: A casestudy”. European Journal of Operational Research, v.156, pp.267-286.
ESD.61J / 16.852J: Integrating the Lean Enterprise © Deborah Nightingale, 2005 Massachusetts Institute of Technology Page 38
Case Study: Nike
-
8/20/2019 Metrics and Performance Measurement
39/47
Case Study: Nike
The project objectives:
• To develop a set of high-level performance metricstailored to the specific business needs for use by thesenior supply chain management team, i.e. Operations,
while including existing local metrics as much as possibleand sensible.
• To design a format, i.e. a scorecard, displaying the metric
scores at the level of Nike as well as that of the businessunits.
Lohman, C., et al. (2004) “Designing a performance measurement system: A casestudy”. European Journal of Operational Research, v.156, pp.267-286.
ESD.61J / 16.852J: Integrating the Lean Enterprise © Deborah Nightingale, 2005 Massachusetts Institute of Technology Page 39
Case Study: Nike
-
8/20/2019 Metrics and Performance Measurement
40/47
Case Study: Nike
• A structure with three layers is used for displaying the information
• High Level
• Mid Level
• Lower Level
•
Normalization
• All the metrics were normalized based on a linear 0–10 scale.
•
Usage
•
Senior supply chain management team : director Operations, thefunctional directors of Transportation, Warehousing, and CustomerService and the three business unit Operations directors. They will usethe scorecard on a monthly basis to facilitate review of theorganizations performance.
• General Manager Nike Europe to facilitate a quarterly review of
Operations
• Maintenance
• Monthly scorecard reviews
• Yearly redesign of the scorecard and its contents when launching new
business plans
Lohman, C., et al. (2004) “Designing a performance measurement system: A casestudy”. European Journal of Operational Research, v.156, pp.267-286.
ESD.61J / 16.852J: Integrating the Lean Enterprise © Deborah Nightingale, 2005 Massachusetts Institute of Technology Page 40
Case Study: Nike
-
8/20/2019 Metrics and Performance Measurement
41/47
Case Study: Nike
Developing and Embedding Performance Measurement System
• Defined a metrics dictionary with some 100 fixed metrics combined with anoverview of existing initiatives, while being much more flexible regarding
the structure.•
Used a hierarchical structure within each cluster, thereby using so-called‘‘engineered indicators’’ i.e. metrics based on two or more lower-levelmetrics.
•
Brought together all people working on parallel initiatives in the field of PMwithin the organization.
• Experimented with different ways of clustering the metrics and presentingthe scorecards.
•
Gathered the data required for actually measuring and reporting all metrics
in a reliable way.•
One person of the cross-functional initiatives group managed theperformance reporting process.
• The PM manager has authority to make changes in the definition, to makesure that metrics remain consistent between various areas and with theglobal scorecard. Lohman, C., et al. (2004) “Designing a performance measurement system: A casestudy”. European Journal of Operational Research, v.156, pp.267-286.
ESD.61J / 16.852J: Integrating the Lean Enterprise © Deborah Nightingale, 2005 Massachusetts Institute of Technology Page 41
Case Study: Air Force Sustainment
-
8/20/2019 Metrics and Performance Measurement
42/47
yDecline in Mission Capability Rates
“Material condition of an aircraft indicating it can perform at leastone and potentially all of its designated missions. Mission-capableis further defined as the sum of full mission-capable and partialmission-capable. Also called MC”. - DOD Definition of Mission Capability
S pe cial M is s ion
9 5
9 0
8 5
8 0
Fighters Bombers
SOURCE: Compiled from data in
National Research Council, Aging7 5 Avionics in Military Aircraft
(Washington, DC: National
Academy Press, 2001)7 0
6 5
6 0
5 5
5 0
Year 8 2
8 5
8 8
9 1
9 4
9 7
ESD.61J / 16.852J: Integrating the Lean Enterprise © Deborah Nightingale, 2005 Massachusetts Institute of Technology Page 42
Air Force Sustainment System
-
8/20/2019 Metrics and Performance Measurement
43/47
i ise Page 43
Air Force Sustainment System
ll l
i ision:
/
(Fli line
i
i
CSI
le
thi i
)
No
) &
le
)
il le
Pi
l
p
No
i
i
i
ia
ll
i
w i
w i
i
i
i
i ipeli
A i
© Deborah N ghtingale, 2005 Massachusetts Institute of TechnologyESD.61J / 16.852J: Integrating the Lean Enterpr
Information F ows: Process:Physica Part F ows: Status:
Ent ty: Dec
LEGEND
ALC DEPOT-LEVE L
TRANSPORTATION
BASE-LEVELBase Repair ght
Mantenance)
Base
Supply
Base Traff c Mgmt
Express Carr er
CRI
Repaired
at Depot
ComponentFails
Depot
TMO
Repairab
s staton?
(RTS
(NRTS
Possib
MICA P
Yes
(RTS
Parts ava abat base?
No &
Possible
MICA P
Yes or
ece
Part
CA NN
SMAG
Requests New
Part Issuance
Parts avai able
at De ot?
Yes
How to
Repa r?
Offs te
Locaton
Contract
Out
Non-Job
Routed
Repair
Job
Routed
Repair
Order to
CSI v
SBSS
Component
Reinsta ed
Job Routed Serv ceable Parts
Carcass
thout
Backorder
Carcassthout
Backorder
CSI
Parts
A rriv e
AWP at
Depot
AWP atBase
Non-Job Routed Serv ceable Parts
SBSS-
Initiated
Order
to Base
Non-
Condemned
CarcassServ cable
Unit
Serv cable Part Re-Enters
Sus ta n ment P ne
rcraft
CA NN
Case Study: Air Force Sustainment
-
8/20/2019 Metrics and Performance Measurement
44/47
Case Study: Air Force Sustainment
Initial Field-Study Observations
• Current metrics generally foster local optimization rather than global optimization
•
Depot maintenance metrics are primarily focused on financial andrelated measures of performance
• Metrics conflict with delivery of best customer support
•
Current metrics generally do not allow measures of progress towards the achievement of system-wide goals•
Impossible to trace incremental impact on the warfighter ofimprovement actions at local level (e.g., incremental investment inmore parts and materials)
•
Informed tradeoff decisions cannot be made at system-level
• Current metrics drive the “wrong” behavior• Big example is cannibalization (removal of a serviceable component
from one end-item waiting for parts to repair another)
ESD.61J / 16.852J: Integrating the Lean Enterprise © Deborah Nightingale, 2005 Massachusetts Institute of Technology Page 44
Regression Model:
-
8/20/2019 Metrics and Performance Measurement
45/47
Mission Capability Rate
F-16 fully-mission-capable (FMC) rate
Effect of key strategic metrics and action variables (below) on the
Fully-mission- Utilization rate 0.08
capable (FMC) rate
as a function of:
Î
Flying scheduling effectiveness 0.14
Break rate -0.06
Cannibalization rate -0.15 Adjusted R2 = 0.47 Repeat discrepancy 0.03
Unscheduled maintenance -0.12
Temperature (control variable) -0.09
Precipitation (control variable) 0.05
Active duty “dummy”* variable 1.36
National Guard “dummy”* variable -0.32
* “Dummy” variable captures the impact of special characteristics peculiar to each user group.
ESD.61J / 16.852J: Integrating the Lean Enterprise © Deborah Nightingale, 2005 Massachusetts Institute of Technology Page 45
Complex Causal Loop Structure
-
8/20/2019 Metrics and Performance Measurement
46/47
Representation
Fix Rate Total Mission Capable Rate
Aircraft Utilization Rate
Flying Scheduling Effectiveness
Cannibalization Rate Break Rates
Sustainment Costs
+
-
+ -
+
++
-
-
-
-
-
+
+
--
B
R
R
R
R
R
B
RR
R = Reinforcing loop
B = Balancing loop
width = magnitude
+/- = Sign of coefficient
Active Duty Mission Reserve Mission Guard Mission
ESD.61J / 16.852J: Integrating the Lean Enterprise © Deborah Nightingale, 2005 Massachusetts Institute of Technology Page 46
Research Findings
-
8/20/2019 Metrics and Performance Measurement
47/47
Research Findings
“Metrics Thermostat” research reveals a far more complex metrics structurethan indicated by top-down hierarchical metrics structure currently beingused
• AFMC Metrics Handbook: Presents hierarchical metrics structure (command;
mission element board (MEB); center-process metrics)
•
US Air Force Logistics Transformation Team “Balanced Scorecard” approach
Complex feedback loops show system behavioral dynamics not otherwise wellunderstood
• Increased fix rates lead to higher aircraft utilization, higher FMC rates, and increased
flying effectiveness• But this also results in increased break rates, leading to lower fix rates by tapping
limited resources, and therefore lowering FMC rates
• Model quantifies net effects of complex interactions
Model provides new insights into the prevalence and effects of “cannibalization”
not available before• Quantitative evidence provides new shows that cannibalization at field level has net
negative effect on the overall F-16 mission capability rate
• Shows how cannibalization (an emergent system property) is driving “wrong”behavior
ESD.61J / 16.852J: Integrating the Lean Enterprise © Deborah Nightingale, 2005 Massachusetts Institute of Technology Page 47