Do you know MESA?

Key Performance Indicator Assessment Model

John Horst

NIST, Smart Manufacturing Operations Management,

Intelligent Systems Division

Barry Ezell, Ph.D.

Virginia Modeling, Analysis and Simulation Center

February 23, 2016

The power of knowing what MESAKNOWS Copyright 2015, MESA International

Barry C. Ezell, Ph.D.

• Old Dominion University’s Virginia Modeling, Analysis and Simulation Center

• Chief Scientist • 26 years risk and decision analysis • Retired Army Officer, Operations Research Systems Analysis • Old Dominion University – Ph.D. Engineering Management,

University of Virginia – MS, Systems Engineering, and University of Southern Mississippi – BS, Mechanical Engineering Technology

• Security Analysis Risk Management Association, Board Member, and immediate past president, 2013-2015.

The power of knowing what MESA KNOWS Copyright 2015, MESA International

KPI-Centric Process Model

Source: John Horst, NIST

The power of knowing what MESAKNOWS Copyright 2015, MESA International

KPI Assessment Method (KAM)

KPI Characterization (Do we have effective KPIs?)

• Identify target manufacturing process (TMP)

• Identify stakeholders

• Discuss KPIs used in TMP

• Decide on KPIs for TMP

• Rank Criteria for KPI assessment

• Criteria Weighting Calc

• Set value function for each criterion

• Assess each KPI score for each criterion

• Calculate KPI score.

• Initial observations

KPI Alignment (How aligned are our KPI to critical objectives?)

• Identify critical objectives for TMP

• Link KPIs to critical objectives

• Assess strength of linkages to critical objectives

(w,m,s)

• Calculate KPI alignment to critical objectives

• Calculate KPI alignment to critical objectives

• Assess balance and alignment of KPIs

• Initial observations

The power of knowing what MESAKNOWS Copyright 2015, MESA International

Onsite Visit - example

• Introduction and discussion

• Group input activity for characterization (weights and scores)

• Initial assessment / discussion

• After today, VMASC conducts deeper analysis

– Benchmarking

– Findings & conclusions

– Short report back to you with our analysis (includes control policies)

• Follow-up in the future to see if method was useful.

The power of knowing what MESAKNOWS Copyright 2015, MESA International

KPI Assessment Process

KPI Criteria Characterization

Do we have effective KPIs? Approach: Weight and score KPIs with respect to a literature synthesized set of generalized criteria.

KPI Alignment & Balance

How well do KPIs align with objectives? Are KPIs balanced with respect to objectives? Approach: KPI characterization is an input to the KPI assessment. Output: KPI Characterization vs. Alignment KPI –Objective balance (# KPI per objective (none, #weak, #moderate, #strong)

Manufacturing Company

Preparatory Activity

Stakeholder Values and

Preference Elicitation

Activity

Manufacturing Company KPI

Characterization & Analysis

Activity

Manufacturing Company KPI

Alignment –Benefit Analysis

Activity

Manufacturing Company KPI

Decisionmaking Activity

The power of knowing what MESAKNOWS Copyright 2015, MESA International

KPI Effectiveness & Alignment

The power of knowing what MESAKNOWS Copyright 2015, MESA International

(1/2) Criteria Definitions – the degree to which…

Quantifiable the KPI's value can be numerically specified.

Relevant the KPI enables performance improvement in the target operation.

Predictive the KPI is able to predict non-steady-state operations and is accompanied by a record of the past performance values for analysis and feedback control.

Standardized a standard for the KPI exists and that standard is correct, complete, and unambiguous; also, the more broad the scope of the standard, the better, for example, plant-wide is good, corporate-wide is better, and industry-wide is best.

Verified the KPI can be shown to be true and correct with respect to an accepted standard and has been correctly implemented.

Accurate the measured value of the KPI is close to the true value.

Timely the KPI is computed and accessible in real-time, where real-time depends on the operational context, and real-time means the updated KPI is accessible close enough in time to the occurrence of the event triggering a change in any metric affecting the KPI.

The power of knowing what MESAKNOWS Copyright 2015, MESA International

(2/2) Criteria Definitions – the degree to which…

Traceable the steps to fix a problem are known, documented, and accessible, where the particular problem is indicated by values or temporal trends of the KPI.

Independent the KPI collection, transfer, computation, implementation, and reporting are performed independently from process stakeholders.

Actionable a team responsible for the KPI has the ability and authority to improve the actual value of the KPI within their own process.

Buy-in the team responsible for the target operation are willing to support the use of the KPI and perform the tasks necessary to achieve target values for the KPI.

Understandable the meaning of the KPI is comprehended by team members and management, particularly with respect to corporate goals.

Documented the documented instructions for implementation of a KPI are up-to-date, correct, and complete, including instructions on how to compute the KPI, what measurements are necessary for its computation, and what actions to take for different KPI values.

Inexpensive the cost of measuring, computing, and reporting the KPI is low.

The power of knowing what MESAKNOWS Copyright 2015, MESA International

What’s Next

• Continue system development

• Continue testing and evaluation

• Document observations, insights, and lessons learned

• Integrate into solution

The power of knowing what MESA KNOWS Copyright 2015, MESA International

Thank you

11

Do you know MESA?

Contact Information Dr. Barry Ezell Chief Scientist

Office: Virginia Modeling, Analysis and Simulation Center

Mobile: 757.638.4439

Email: bezell@odu.edu

Skype: orsa1965, barry.ezell

MESA International - 107 S. Southgate Drive - Chandler, AZ 85226 – USA - www.mesa.org

HQ Phone: +1 (480) 893-6883 F: +1 (480) 893-7775

The information contained in this document is MESA International Proprietary Information and is disclosed in confidence. This information is to be used solely for the pursuit of MESA International purposes. This information shall not be used, disclosed to others or reproduced without the express written consent of MESA International. If you believe you have received this document in error, please notify the contact above immediately and delete the document.