Outcome Based Management
-
Upload
settlementatwork -
Category
Business
-
view
1.238 -
download
2
description
Transcript of Outcome Based Management
400-45 rue Rideau Street • Ottawa
Ontario • Canada • K1N 5W8
www.bbmd.ca
Outcome-Based Management
How do you tell your performance story? Introduction to the concepts and practices of
Performance Measurement
OCASI Conference, November 5th, 2009
BBMD Consulting
2
• Leaders in Performance Measurement
• Have worked with hundreds of profit and
not-for-profit organizations and
thousands of individuals in Africa, Asia,
North America and Europe.
• Extensive Performance Measurement
experience with federal, provincial and
municipal governments (currently
supporting the PM efforts of Canadian
and Ontario Governments)
Bell, Browne, Molnar & Delicate Consulting
Objectives for today
1. Understand how government bodies are using Performance Measurement
2. Share the fundamentals of Outcome-Based Performance Measurement
3. Discuss why a non-profit agency MUST use this methodology to remain competitive
4. Explore how this might change the way your program is delivered
3
Bell, Browne, Molnar & Delicate Consulting4
Today’s Reality…EVERYONE IS DOING IT!
www.expectmore.gov
http://www.mnplan.state.mn.us/mm/
www.ratemyteacher.com
http://www.cbc.ca/canada/ottawa/st
ory/2009/10/22/ottawa-ontario-
second-career.html
Bell, Browne, Molnar & Delicate Consulting
Federal and Provincial Direction
Federal:
• Internal federal policy commits to having the WHOLE of the federal government (all programs, including internal services) apply the components of a MRRS (see: http://www.tbs-sct.gc.ca/pubs_pol/dcgpubs/mrrsp-psgrr/id-cm/id-cm-PR_e.asp?printable=True)
• CIC Settlement – the Modernized Approach
– Outcomes – The modernized approach is an outcome based approach
– Performance Measurement – results in terms of outcomes, outputs, and financial resources will be gathered and monitored to ensure activities continue to achieve expected results and link services to specific settlement outcomes
Provincial
• Results-based plan drives provincial allocation and public reporting (see http://www.gov.on.ca/ont/portal/!ut/p/.cmd/cs/.ce/7_0_A/.s/7_0_252/_s.7_0_A/7_0_252/_l/en?docid=STEL01_105128)
• Budget Transparency and Accountability Act (BTAA), British Columbia (www.bcauditor.com/PUBS/2008_09/Report2/Strengthening%20Accountability_WEB.pdf)
• LMDA has resulted in MTCU building one common performance framework for all employment services.
5
Bell, Browne, Molnar & Delicate Consulting6
Bell, Browne, Molnar & Delicate Consulting
Wh
y O
utc
om
e B
ase
d
Pe
rfo
rman
ce M
eas
ure
me
nt?
7
ALIGNMENTEveryone knows what is
important to management
(“what gets measured gets
done”)
PRIORITY
SETTINGAssists in identifying priorities
(“what gets measured gets
improved”)
RESOURCE
ALLOCATIONHelps justify the need for
resources (“what gets
measured gets funded”)
ACCOUNTABILITYClarity around accountability
obligations (“always ready to report”)
Why
Performance
Measurement?
Adv
ice
to B
oard
and
Fun
der
Tax D
olla
r S
tew
ard
ship
Answerability to the Public and Funder
Dem
onstration of Success
to Public and funder
Evid
ence to
Support D
ecis
ion M
akin
g
Bell, Browne, Molnar & Delicate Consulting
Not just about meeting funding obligations…Are you complying or competing?
High performance organizations build their own Performance Measurement Frameworks, measuring Effectiveness, Efficiency and Customer Satisfaction acrossfunded program boundaries
– How do you make sure you are as relevant as possible to your community? What is your cumulative value add (more than just the sum of your programs)
– What value can you demonstrate BEYOND what your funding body is asking for?
– How can you proactively report your progress to your city, your MP, your community, your funders?
Ministry of Training, Colleges and University
“Employment Ontario” Performance Measurement
Framework” 2008
8
Bell, Browne, Molnar & Delicate Consulting
CASE STUDY: Job ConnectHow did MTCU get to be ‘outcome based’
• Longitudinal study over a decade ago showed that going to an employment agency did notincrease your likelihood of employment
• Program objectives were to serve the ‘hard-to-serve’ – they were NOT being served
• Something had to change – Introduction of the Continuous Improvement Performance Measurement System (CIPMS)
9
Bell, Browne, Molnar & Delicate Consulting
Understanding Job Connect:Congratulations! I just gave you 1.2 million to serve 1000 ‘hard-to-serve’ clients, but
there’s more...
10
Bell, Browne, Molnar & Delicate Consulting
3
Employment Service Funding Decision MatrixEvidence of
Organizational
Capacity
Compliance
with MTCU Contract
Directed Improvement - Funding Approval with Compliance and In Year
Improvements to Overall Service Quality StandardOrganization is NOT in compliance with contract/guidelines and must submit an
action plan for achieving compliance
Organization has not achieved Overall Service Quality Standard for service
delivery nor improvement targets from last contract
OC review demonstrates there is capacity and potential for immediate
improvement
✘ ✔✘
Termination – Notice of contract termination, procedures to support
client service transferOrganization did not achieve in year service improvements to standard and/or
is not in compliance with contract/guidelines
Organization does not demonstrate necessary OC to support improvement
✘ ✘✘
Official Review – Funding Approval with Immediate Compliance
DirectiveOrganization is NOT in compliance with contract despite achieving or
surpassing Overall Service Quality Standard for service delivery and must
submit an action plan for achieving compliance
OC review demonstrates little ability for sustained or improved results
Organization did not meet improvement targets from last contract.
✔ ✘✘
Success – Funding approval, Performance Incentive and/or Award
PotentialOrganization is in compliance with contract and program guidelines and has
achieved or surpassed Overall Service Quality Standard for service delivery
OC review demonstrates ability for sustained and/or improved results
Organization has attained improvement targets identified in last contract
✔ ✔✔
MTCU Response
Overall
Service
Quality
Standard Achieved
✔
✘
✘
✘
Reached
Improvement Targets
2.
11
Bell, Browne, Molnar & Delicate Consulting
CIC, Implementing the modernized approach,
October 2009
Bell, Browne, Molnar & Delicate Consulting
CIC, Implementing
the modernized
approach, October
2009
Bell, Browne, Molnar & Delicate Consulting 14
2. PERFORMANCE MEASUREMENT FUNDAMENTALS
An intensive.
Bell, Browne, Molnar & Delicate Consulting
Performance Measurement…
is all about answering the following questions:– Why do we exist?
– What do we want to accomplish / what is our end state?
– How are we going to deliver our Programs?
– How will we know how well we are doing?
– What programs, priorities and initiatives do we need to undertake to work meaningfully towards our end state?
15
Bell, Browne, Molnar & Delicate Consulting
Some Basic Requirements for Success
• Common Glossary and Methodology
• Common understanding and agreement of the desired ‘end state’
• Establishing the “vital few” performance measures
• Building a performance report and reporting strategy
• Formalize a continuous improvement strategy
16
Bell, Browne, Molnar & Delicate Consulting
The foundation – the logic model
17
Inputs Activities OutputsShort-term
Outcomes
Intermediate
Outcomes
Long-term
Outcomes
Low Influence →← High Control
Strategic
Outcomes
RESULTS
Bell, Browne, Molnar & Delicate Consulting
Inputs, Activities, Outputs and Outcomes
18
Definition Examples
Inputs“what we need to do the work”
The human, financial, structural resources, knowledge and information necessary to perform the activities.
Activities“what we do”
Describe a collection of functions (actions, jobs, tasks) that consume inputs/resources and whose outputs deliver benefits/impacts to a client or contribute to a particular outcome/set of outcomes
Outputs“what we produce”
The direct products and/or services (“usually tangible and concrete”) generated through activities without specific reference to their ultimate purpose
Outcomes“why we do it”
The benefits or desired states (“not fully under our influence”) to which the outputs of activities contribute
Bell, Browne, Molnar & Delicate Consulting
Four Types of Outcomes
Categorized according to the degree of influence, as follows:
1. Immediate Outcomes First-level effects of, or immediate response to the outputs, e.g., changes in compliance rates or degree of customer satisfaction.
2. Intermediate Outcomes are the second order of outcomes, Benefits and changes in behaviour, decisions, policies and social action attributable to outputs to demonstrate that program objectives are being met, e.g., increased employability as a result of a training program.
19
Bell, Browne, Molnar & Delicate Consulting
3. Ultimate/Long-term Outcomes The ultimate or long-term consequences for human, economic, civic or environmental benefit, to which government policy or legislation contributes, e.g., life expectancy rates, overall economic performance. Very broad in nature, requires many departments collaboration and take some time to achieve.
All of which feed the…
4. Strategic Outcomes are those long term and enduring organizational and/or government-wide benefits or goals that the Project, Program or Organization is contributing to.
Four Types of Outcomes
20
Bell, Browne, Molnar & Delicate Consulting
WHAT CHANGES IN THE WORLD AS A RESULT OF THE WORK YOUR AGENCY IS DOING?
This first, then this, then this, then this….
Products and
Services
LESS CONTROL
(attribution difficult)
KNOWLEDGE ATTITUDES
•Awareness•Understanding•Skills
•Perceptions•Acceptance
BEHAVIOUR
•Involvement•Compliance•Action
EARLY / LATER EFFECTS
•Societal change•Socio-economic
benefits
OUTPUTS IMMEDIATE
OUTCOMES
INTERMEDIATE OUTCOMES
Long TermOUTCOMES
MORE CONTROL
(easier attribution)
Social Marketing Role
21
Bell, Browne, Molnar & Delicate Consulting
Principles of Good Outcome Statements
1. Outcomes are noun-based desired end states (e.g., “newcomers are meaningfully employed…” rather than “increase employment opportunities for newcomers …”)
2. Directional outcome statements (e.g., “Improved/increased labour market access by Ontarians…”) do not belong in logic models or outcome statements. Direction is achieved through targets not outcome statements
3. Avoid self-serving statements (e.g., “World leader in settlement services”). After all, this is not the end state desired
4. Clear cause and effect linkage to the next level(s) of outcomes
5. SIMPLE: Outcome described in a way that helps Canadians relate to the desired state that is being described
22
Bell, Browne, Molnar & Delicate Consulting 23
Exercise: Outcomes language exercise
Small group work
• Examine the bulleted list and reach consensus on which items are immediate, intermediate, long term outcomes
4. PM Concepts
Bell, Browne, Molnar & Delicate Consulting 24
Outcome ExerciseExamine the list and reach consensus on which items are immediate, intermediate, OR long term outcomes Imm / Inter / LT
4. PM Concepts
1. Newcomers and their families are healthy, safe and secure.
2. Our settlement agency is recognized and consulted in settlement-related policy development.
3. Communities know, understand and appreciate the social, cultural and economic benefits that newcomers bring.
4. A skilled, adaptable and inclusive labour force and efficient labour market.
5. Preservation of the Environment.
6. Women and Men have equal treatment under the law.
7. Newcomers have the job search skills required.
8. Canadians provide a welcoming community to facilitate the full participation of newcomers into Canadian society
9. Newcomers access affordable childcare.
10. Certainty over ownership and use of land and resources by First Nations and Inuit communities.
11. Newcomers and their family actively works towards achievement of goals
12. Newcomers are aware of and access settlement programming (e.g.: labour market language training with a bridge-to-work component)
Bell, Browne, Molnar & Delicate Consulting 25
Direct Outcomes Shared Outcomes Ultimate Outcomes
Settlement Logic Model
Bell, Browne, Molnar & Delicate Consulting
Considering these outcomes, what SHOULD we be measuring?(ok to stick post-it notes to the relevant column)
1. Outcomes Measured, Outcomes Managed
Direct Outcomes Shared Outcomes Ultimate Outcomes
Bell, Browne, Molnar & Delicate Consulting 27
Indicators and Targets
• Indicators ( The “How”)– A statistic or parameter that, tracked over time, provides information
on trends in the condition of a phenomenon (e.g., percentage of clients that are satisfied)
• Targets (The “How Much”)– Specific performance goals tied to indicators against which actual
performance will be compared (e.g., 90% client satisfaction rating)
• Standards– Standards imply a service level to which an organization is prepared to
commit, either internally, externally or both.
Bell, Browne, Molnar & Delicate Consulting
Examples - Indicators
• % of newcomers who are aware of the skills and expectations necessary to join the Canadian labour force
• % of newcomers who access to labour market language training
• % of clients who report that they have the language ability and skills needed to participate socially, culturally and economically in Canada
• % of clients who report that they feel connected to the broader community and social networks
• % of program participants who claim to create a welcoming community for newcomers
• % employed for > than 3 months
• % satisfied with service
28
Bell, Browne, Molnar & Delicate Consulting
Examples - Targets
• 80% of newcomers aware of the skills and expectations necessary to join the Canadian labour force
• 90% of newcomers have access to labour market language training
• 64% employment retained for > 3 months
• Full integration of newcomers as participating members of Canadian society (economic, education, employment, etc.) within 10 years of arrival.
29
Bell, Browne, Molnar & Delicate Consulting
CASE STUDY –Employment Ontario
Achieving the standard of overall service quality is key to stable and ongoing funding
For illustration purposes only!! Provincial Service Quality Standard
DIMENSION
MEASURE
MINIMUM PROVINCIAL STANDARD
WEIGHT
SQS
VALUE
MAXIMUM
VALUE
Customer Service (40%)
1. Customer Satisfaction (participant/employer)
85% 15% 1.28 1.5
2. Service Coordination 30% 25% 0.75 2.5
Effectiveness (50%)
3. Employed/Career Path 69% 25% 1.73 2.5
4. Training/Education 10% 10% 0.10 1.0
5. Suitability (Participant Profile)
25% 15% 0.38 1.5
Efficiency (10%) 6. Intake in assisted service
90% 5% 0.45 0.5
7. Workshop Activities/Information Sessions
90% 5% 0.45 0.5
Service Quality Standard 10.0
Customer Satisfaction (0.85 x 1.5) = 1.28
Service Coordination (0.30 x 2.5) = .75
Employed/Career Path (0.69 x 2.5) = 1.73
Training/Education (0.10 x 1.0) = 0.10
Suitability (0.25 x 1.5) = 0.38
Intake in Assisted Service (.90 x .5) = 0.45
Workshops (.90 x .5) = 0.45
Overall Service Quality Standard 5.1430
Bell, Browne, Molnar & Delicate Consulting
How to Determine “the Critical Few” Indicators
• Credible – provide a credible, independent view
• Specific - eliminate ambiguity, show relevance to the expected outcome
• Linked - clear (cause and effect) linkages to other indicators
• Reliable - scientifically and statistically sound, provide an appropriate degree of accuracy. Measure the same thing across time (allowing comparisons) and for different groups/regions
• Available – data are easily accessible or there is a low level of effort to collect and analyze
• Cost-effective – the costs for collection are aligned with the overall utility of the indicators
• Understandable – data can be easily grasped by various audiences
31
Bell, Browne, Molnar & Delicate Consulting
Performance Indicator Assessment Grid
Priority
Degree of importance/relevance in terms of measuring efficiency (“doing things well”) and/or effectiveness
(“doing the right things/having an impact”)
High 3 It is essential in order to measure efficiency and/or effectiveness
Medium 2 It is useful in measuring efficiency and/or effectiveness
Low 1 It is of little use in measuring efficiency and/or effectiveness
Data Accessibility
The degree to which data are readily available either manually and/or through an automated system
High 3 Indicator data are immediately available
Medium 2 The appropriate data are not immediately available through an automated or manual system
but could be gathered relatively easily
Low 1 The data to support this indicator are not currently available or accessible and will be difficult
to access
Level of Effort
An estimate of the level of effort required to report reliably against the indicator on a regular basis
Low 3 Minimal effort to gather, analyze, interpret and report reliably on the indicator
Medium 2 It requires a moderate degree of effort to reliably report on the indicator
High 1 It is very resource intensive to reliably report on the indicator
32
Bell, Browne, Molnar & Delicate Consulting
WHY MUST A NON-PROFIT USE THIS METHODOLOGY TO REMAIN COMPETITIVE?
Discussion.
33
Type here
Bell, Browne, Molnar & Delicate Consulting
HOW COULD THIS CHANGE THE WAY YOUR SERVICE IS DELIVERED?
Discussion.
34
Type here
Bell, Browne, Molnar & Delicate Consulting
SARAH DELICATE, BBMD (905) 922-9726 (cell), [email protected]
35