Impact of Dominant Metrics
description
Transcript of Impact of Dominant Metrics
Impact of Dominant Metrics
“Simple can be harder to achieve than keeping it
complex. You have to work hard to
make it simple. But it’s worth it in the end because once you get there, you
can move mountains.”
Objectives• Understand what is “Dominant Information”
• Understand who should use or who has the most to gain by using “Dominant Information”
• Identify an example where Dominant Information could have assisted you
3
Metrics vs. Dominant Metrics
Is There A Difference?
4
Case Study – ASU Dining Service• 2007, ASU implements the Best-Value process to procure Dining Services
• ASU Student Population = 72,000
• $1+ Billion Dining Service contract
• 15-Year Contract
• Results:• 62% in Revenue• $33 Million
5
Documenting Success• We will document:
• Commissions paid to University• Monthly sales per location• Conference sales• Equipment depreciation schedule• Capital improvement depreciation schedule• Average check per customer• Sales per labor hour• Annual statement of operations• Number and type of meal plan sold• Non-commissionable revenue detail• Mandatory and Voluntary meal plan rate (number and $)
6
Performance Metrics
8
Is This Dominant?• Increase sale of food by 14%• Increased money to ASU by 23%• Decreased management cost by 80%• Increased customer satisfaction by 37%
9
CRITERIA2006-2007
(Incumbent)2007-2008
(Year 1)DEVIATION
Total Revenue ($) 27,000,000$ 30,830,000$ 14%Total Commissions ($) 2,170,000$ 2,670,000$ 23%Capital Investment ($) 14,750,000$ 30,830,000$ 109%ASU Administration Effort (#) 7.0 1.5 -79%Student Satisfaction (1-10) 5.2 7.1 37%
Is This Dominant?
10
CriteriaYear 1(From
Incumbent)Year 2
(From Year 1)Year 3
(From Year 2)Year 4
(From Year 3)
Sales 14% Increase 11% Increase 24% Increase 13.5% Increase
Commission 23% Increase 6% Increase 20% Increase 22% Increase
ASU Management Requirement Reduced 79% -- -- --
Student Satisfaction 37% Increase 1% Decrease 9% Increase 3% Increase
The difficulty of making things simple can cause
us to ignore creating “dominant metrics”
11
BenchmarkingDoes It Really Make A
Difference?
12
State of Idaho Inmate Health Services• Scope of Work: Provide health services to correctional inmates across
Idaho (13 facilities – approximately 5,000 inmates). This includes Healthcare Services, Mental Health Services, and Pharmaceutical Services.
• Length of Contract: 3-Year Base Contract (opportunity for 5 total years)
• Budget Constraints: $22,900,000 (2010-2011)
• RFP Issued: 12/01/2009
• Politics: IDOC extremely dissatisfied with the incumbent
13
Analysis of Proposals
• Awarded Vendor was:– 2.8% below the IDOC’s budget– 9.4% below average of all proposals– Highest Interview– Highest Work Plan– Second highest RAVA Plan
NO CRITERIA FIRM A FIRM B FIRM C FIRM D FIRM E
1 Total Cost 85,567,680$ 66,786,240$ 83,710,560$ 66,488,400$ 65,840,160$
2 Interview 4.0 6.9 6.9 4.5 2.2
3 Risk & Value Plan 4.8 6.4 6.6 5.4 3.8
4 Work Plan 6.0 5.0 7.4 4.2 4.6
5 PPI Scores (1-10) 9.6 9.8 9.8 9.8 9.2
6 PPI Scores (#) 26 40 28 32 20
Overall Score (1,000): 709 953 951 779 601
Politics• Client was very disappointed in the best-value process since the highest-
prioritized vendor was the incumbent
• Client did NOT want to re-hire incumbent
• Client not happy with the results
• Could not understand why the better vendor did not propose within their budget
15
Moving Into Phase 2• The vendor was unaware of clients concerns with their company
• They assumed that the client was very satisfied with their past performance.
• The client put together a list of their major ‘deal-breakers’ or issues that were causing them significant displeasure in the past. These included: • Incumbents ability to staff the service properly (high turnover issues)• Lack of clear and understandable performance metrics• Incumbents inability to manage the service adequately
16
Vendor Response:• Very traditional (used vague language / didn’t clearly address the issues)
• Individuals preparing the responses did not understand the issues that had occurred in the past with the previous contract.
• Did not make the client feel comfortable they could manage the issues.
• Could not specifically state what would be done differently
• Without understanding the past issues, it is very difficult to explain what they will do differently and how they can improve.
• Vendor transfers risks/decision making back to the client (“what exactly would you like us to do” “how would you like us to handle this”)
17
Client Reaction• The vendor does not understand the new paradigm. Worried about saying
the right things to get the contract, instead of figuring out how to succeed.
• The client identifies that they have issues about trust, support, and retention
• The client identifies that the Regional Manager is a concern. • Staff members will leave if current RM is not replaced• Staff told not to talk to client or they would be fired• RM is creating an adversarial environment
18
Vendor Reaction• The vendors management dismiss these comments as typical statements
that owners make.
• Vendor management does not believe that there have been problems in the past
• If there were problems, why weren’t they informed previously?
19
How Do We Address These Issues?
20
Quick Survey• Perform a third party survey with the client staff
• Identify their overall satisfaction with• The vendor• The vendor’s management• The client
• Primary Goal: Determine if there is a problem or not!
• Within a couple of days, a survey was creased and sent out to all employees.
• No names, setup online for a quick response.
21
Survey Results• Within a week, we received 83 surveys (out of 128) from 9 different facilities
• Results confirmed that there was a significant weakness with staff satisfaction towards corporate support and regional manager
22
NO CRITERIA RESULTS
1 Overall Job Satisfaction (10 max) 6.8
2 Satisfaction of the Client (10 max) 7.3
3 Satisfaction of the Regional Manager (10 max) 3.7
4 Satisfaction with Corporate Support (10 max) 4.5
5 Total Number of Responses 83
Do We Really Have An Issue?• Surveyed 4 other institutions nationwide (similar in size)• Metrics can be used to identify any potential strengths and weaknesses
23
NO CRITERIA UNIT OVERALL CLIENT A
CLIENT B
CLIENT C
CLIENT D
CLIENT E
1 Overall Satisfaction - Job (1-10) 7.0 6.8 7.5 6.9 7.0 6.4
2 Overall Satisfaction - Job Training (1-10) 6.1 6.0 6.8 6.2 4.9 5.7
3 Overall Satisfaction - Pay / Benefits / Comp (1-10) 6.3 6.5 7.1 6.2 5.6 5.6
4 Overall Satisfaction - Site Management Team (1-10) 6.7 7.2 8.2 6.3 6.0 5.6
5 Overall Satisfaction - Regional Manager (1-10) 6.2 3.7 8.1 6.3 5.9 6.1
6 Overall Satisfaction - Corporate Management (1-10) 6.1 4.5 7.6 6.0 5.5 5.5
7 Overall Satisfaction - Client (1-10) 7.4 7.3 7.7 7.3 7.2 7.5
8 Total Number of Responses # 525 83 138 159 58 87
9 Percent of Surveys Returned % 44% 65% 66% 36% 32% 37%
Summary• Vendors don’t track their critical staff performance
• Clients don’t communicate in a ‘dominant’ manner
• Dominant metrics can minimize surprises, communication, effort
• Dominant metrics can make your operation more efficient
24
Dominant MetricsWho Benefits The Most?
25
Student Health Insurance Consortium• Create a statewide Student Health
Insurance Plan (SHIP) consortium – Boise State University (BSU)– Idaho State University (ISU)– Lewis-Clark State College (LCSC)
• 3-Year Contract | $36 Million (for 3-years)
• Measurements of Success1.Reduce internal University program
administration costs2.Maintain or increase Customer
Satisfaction (University & Students)3.Maintain or increase cost-effectiveness of
program to students
Coverage/Plan Characteristics• Consortium goal was to standardize
coverage between all three University's (to maximum extent possible). However, deviations were made as necessary (BSU athletic coverage, ISU RX Coverage, Capitated Fee, etc)
• Consortium goal was to increase plan characteristics (to provide better coverage for students)
NO CRITERIA BSU ISU LCSC CONSORTIUM
1 Deductible Per Academic Year (In-Network) $250 $250 $250 $250
2 Deductible Per Academic Year (Out-Of-Network) $500 $250 $250 $500
3 Maximum Benefit (Standard) $100,000 $50,000 $50,000 $250,000
4 In-Network Coinsurance 80% 80% 80% 80%
5 In-Network Max out of Pocket $4,000 No MOP No MOP $4,000
6 Out-Of-Network Coinsurance 50% 60% 80% 60%
7 Out-Of-Network Max out of Pocket $6,000 No MOP No MOP $6,000
8 RX Drug Coverage (Max) $400 None $500 $500*
Analysis of Proposals
Total Score: 923 916 886 831 840
NO CRITERIA FIRM A FIRM C FIRM D FIRM E FIRM F
1 Cost - Average Student Premium $1,422 $1,327 $1,365 $1,561 $1,596
2 Cost - Average Spouse & Dependent Premium $1,698 $2,668 $2,343 $2,559 $2,762
3 Average Interview Rating 6.4 6.6 5.2 6.3 6.9
4 RAVA Plan Rating 7.4 6.3 7.4 5.6 5.2
5 Work Plan Rating 6.7 7.2 6.3 5.5 5.6
6 PPI - 1-10 Rating 9.9 9.7 9.9 10.0 10.0
7 PPI - Number of projects and clients 10 17 9 10 10
Results
• Best-Value Results:– Student Premium has decreased by
2% (-$26)– Spouse & Dependent Premium has decreased by 19% (-$519)
– In general, Benefits/Coverage have been increased
– The Best-Value Process was able to stabilize premium rates for the first time in 4 years (for both Student and Spouse/Dependents)
School Premiums 2006-2007 2007-2008 2008-2009 2009-2010Average
Increase Per Year ($)
Average Increase Per
Year (%)Student $1,012 $1,182 $1,263 $1,385 $124 11%Spouse & Dependent $1,843 $2,022 $2,104 $2,220 $126 6%
• Previous Program:• Student Premiums increased $124/year (past 4 years)• Spouse & Dependent Premiums increased $126/year
Post Award Metrics• Client is extremely satisfied with award (increased benefits for decreased
costs to students)
• ASU encourages the vendor to partner:• Perform continuous education with vendor key staff• Collect and analyze student surveys (from SHIP) to document performance• Collect and analyze student surveys (from other programs)• Assist vendor in documenting performance (dominant information)• Assist vendor in documenting the project for long-term success
• Vendor states that they are comfortable and does not partner with ASU
Waiver Process• During the first year of the program, the Vendor runs into issues with the
waiver process at one of the institutions.
• Some students did not meet the minimum requirements (but were allowed to waive out), some were incorrectly denied coverage, and some students have not been notified that they do not meet the standards.
• Vendor does not know how to document the issue. ASU proposes the vendor:• Identifies what the issues are and the magnitude of the issue• Identifies why did the issues occur• What is being done to address the issues• What is being done to prevent the issues from reoccurring
• Vendor takes several months putting together a summary of the issues.
• University is frustrated and request additional audits (added cost to vendor)
Student Surveys• Vendor was required to survey students to identify their satisfaction.
• Results of initial student survey showed:• Overall Satisfaction was 4.6 (out of 10)• Number of Responses was 896
NO CRITERIA RESPONSE1 Frequency of use (how often was student health insurance used this semester) 3.1
2 Satisfaction with the level of benefits offered/provided by SHIP 4.4
3 Helpfulness of the marketing materials 4.4
4 Overall claims experience 4.7
5 Would students rather have greater benefits (10) or lower premiums (1) 5.9
6 Overall satisfaction with SHIP? 4.6
7 Percent of students that feel current healthcare needs are being met by SHIP 51%
8 Number of student responses 896
Student Surveys• ASU proposes to collect similar survey information from other
consortiums to determine if the results are about average with the industry (or above/below).
• ASU does a quick analysis of the student surveys and discovers that the dissatisfied students identified issues with claims from the previous vendor. ASU identifies that it would be beneficial to market and educate the students on the differences of this new SHIP program to their previous programs. ASU envisions a simple website that illustrates the differences (prior to completing the survey).
• Vendor does not see value in either proposal.
Post Award Metrics• Vendor is unaware of how to provide dominant performance metrics.
The Vendor submits a vast amount of data that is confusing to the client (client does not know how to interpret all of the data).
• ASU quickly reviews all of the data and prepares a summary:
• Based on the loss ratio, student premiums should not increase• University satisfaction increased by 18%• Student satisfaction decreased by 2%
NO CRITERIA ANTICIPATED ACTUAL IMPACT1 Total Enrollment 7,895 11,213 42%2 Total Premiums Collected $11,350,311 $15,926,766 40%3 Total Claims $9,080,249 $8,484,393 -7%4 Total Loss Ratio 80% 53% -34%
Dominant MetricsWhen Should You Start
Collecting?
35
The State of Hawaii• The State of Hawaii-Public Works
Division (PWD) received numerous complaints on the quality of completed construction projects.– Roofs were poorly constructed.– Contractors were slow in correcting punch
list items.– Response time to warranty work was
slow.– Painting work was so bad that the State
assumed all painting work on Oahu with their own staff (painting awards dropped from 101 projects to 5 in 1998).
• In 1998, PWD implemented the BV process on roofing and painting projects.
• Over a 4 year implementation, PWD procured 193 projects using BV (106 roofing, 33 painting, 15 mechanical, 5 electrical, and 34 general renovation projects)
Kapuaiwa Courthouse RenovationCase Study• Historical renovation
project • 60 year old, 3 story,
20,000 SF facility• One year design
• Deteriorated balusters allowed water infiltration
Kapuaiwa Courthouse RenovationCase Study• Nobody bid the job (design wouldn’t
work)
• Project was re-bid using PIPS– No specification– Requirement was to waterproof the
building for the longest possible period of time
– Budget was $800K
• Award was made using the PIPS process
Complaints• A small handful of vendors were complaining.
• Contacted the media and 19 articles were published stating:• Process was not open (awarding to the same contractors)• Process was unfair• Process was not working• Process was wasting taxpayer money• Projects took longer to complete• Process violated State Regulations
45
Audit Report• In response to the negative articles,
the State performed an audit in November 2002 on the BV process.
• The audit analyzed low-bid projects and PIPS projects.
• Results/Findings1. PIPS resulted in 3% savings of project
costs2. PIPS resulted in greater contractor
accountability3. PIPS resulted in fewer Change Orders4. PIPS has given higher quality
construction
PIPS Roofing Conclusions(Taken from Audit Report, 2002)
1. Total number of awarded PIPS roofs: 96
2. 100% would rather use the PIPS process over low-bid process (55 DOE users)
3. 100% would use a PIPS contractor again (55 DOE users)
4. PIPS average performance rating 9.6 (10 max)
5. Projects were 6% under budget (adjusting for insulation)
6. Projects finished approximately 35% faster (than LB)
7. Performance rating of PIPS vs. LB: 8.1 vs. 5.6
8. 98% were completed on time9. Contractors were almost twice as
productive ($4.5K/day vs. $2.5K/day)10. In last 4 years, there has been no
roof leaks
Outstanding Results….
48
…But PWD is Directed to go back to Low-Bid
Dominant Metrics• By the time the State had compiled and presented the information, the
decision was already made to go back to low-bid.
• Although the performance metrics were significantly higher, the Dominant Metrics were not available fast enough.
• Decision was based on politics, not performance.
49
Can ‘Domina
nt Metrics’ Assist You?
50
Objectives - RECAP• Understand what is “Dominant Information”
• Understand who should use or who has the most to gain by using “Dominant Information”
• Identify an example where Dominant Information could have assisted you
51