Outcome Based Management

35
400-45 rue Rideau Street • Ottawa Ontario • Canada • K1N 5W8 www.bbmd.ca Outcome-Based Management How do you tell your performance story? Introduction to the concepts and practices of Performance Measurement OCASI Conference, November 5 th , 2009

description

 

Transcript of Outcome Based Management

Page 1: Outcome Based Management

400-45 rue Rideau Street • Ottawa

Ontario • Canada • K1N 5W8

www.bbmd.ca

Outcome-Based Management

How do you tell your performance story? Introduction to the concepts and practices of

Performance Measurement

OCASI Conference, November 5th, 2009

Page 2: Outcome Based Management

BBMD Consulting

2

• Leaders in Performance Measurement

• Have worked with hundreds of profit and

not-for-profit organizations and

thousands of individuals in Africa, Asia,

North America and Europe.

• Extensive Performance Measurement

experience with federal, provincial and

municipal governments (currently

supporting the PM efforts of Canadian

and Ontario Governments)

Page 3: Outcome Based Management

Bell, Browne, Molnar & Delicate Consulting

Objectives for today

1. Understand how government bodies are using Performance Measurement

2. Share the fundamentals of Outcome-Based Performance Measurement

3. Discuss why a non-profit agency MUST use this methodology to remain competitive

4. Explore how this might change the way your program is delivered

3

Page 5: Outcome Based Management

Bell, Browne, Molnar & Delicate Consulting

Federal and Provincial Direction

Federal:

• Internal federal policy commits to having the WHOLE of the federal government (all programs, including internal services) apply the components of a MRRS (see: http://www.tbs-sct.gc.ca/pubs_pol/dcgpubs/mrrsp-psgrr/id-cm/id-cm-PR_e.asp?printable=True)

• CIC Settlement – the Modernized Approach

– Outcomes – The modernized approach is an outcome based approach

– Performance Measurement – results in terms of outcomes, outputs, and financial resources will be gathered and monitored to ensure activities continue to achieve expected results and link services to specific settlement outcomes

Provincial

• Results-based plan drives provincial allocation and public reporting (see http://www.gov.on.ca/ont/portal/!ut/p/.cmd/cs/.ce/7_0_A/.s/7_0_252/_s.7_0_A/7_0_252/_l/en?docid=STEL01_105128)

• Budget Transparency and Accountability Act (BTAA), British Columbia (www.bcauditor.com/PUBS/2008_09/Report2/Strengthening%20Accountability_WEB.pdf)

• LMDA has resulted in MTCU building one common performance framework for all employment services.

5

Page 6: Outcome Based Management

Bell, Browne, Molnar & Delicate Consulting6

Page 7: Outcome Based Management

Bell, Browne, Molnar & Delicate Consulting

Wh

y O

utc

om

e B

ase

d

Pe

rfo

rman

ce M

eas

ure

me

nt?

7

ALIGNMENTEveryone knows what is

important to management

(“what gets measured gets

done”)

PRIORITY

SETTINGAssists in identifying priorities

(“what gets measured gets

improved”)

RESOURCE

ALLOCATIONHelps justify the need for

resources (“what gets

measured gets funded”)

ACCOUNTABILITYClarity around accountability

obligations (“always ready to report”)

Why

Performance

Measurement?

Adv

ice

to B

oard

and

Fun

der

Tax D

olla

r S

tew

ard

ship

Answerability to the Public and Funder

Dem

onstration of Success

to Public and funder

Evid

ence to

Support D

ecis

ion M

akin

g

Page 8: Outcome Based Management

Bell, Browne, Molnar & Delicate Consulting

Not just about meeting funding obligations…Are you complying or competing?

High performance organizations build their own Performance Measurement Frameworks, measuring Effectiveness, Efficiency and Customer Satisfaction acrossfunded program boundaries

– How do you make sure you are as relevant as possible to your community? What is your cumulative value add (more than just the sum of your programs)

– What value can you demonstrate BEYOND what your funding body is asking for?

– How can you proactively report your progress to your city, your MP, your community, your funders?

Ministry of Training, Colleges and University

“Employment Ontario” Performance Measurement

Framework” 2008

8

Page 9: Outcome Based Management

Bell, Browne, Molnar & Delicate Consulting

CASE STUDY: Job ConnectHow did MTCU get to be ‘outcome based’

• Longitudinal study over a decade ago showed that going to an employment agency did notincrease your likelihood of employment

• Program objectives were to serve the ‘hard-to-serve’ – they were NOT being served

• Something had to change – Introduction of the Continuous Improvement Performance Measurement System (CIPMS)

9

Page 10: Outcome Based Management

Bell, Browne, Molnar & Delicate Consulting

Understanding Job Connect:Congratulations! I just gave you 1.2 million to serve 1000 ‘hard-to-serve’ clients, but

there’s more...

10

Page 11: Outcome Based Management

Bell, Browne, Molnar & Delicate Consulting

3

Employment Service Funding Decision MatrixEvidence of

Organizational

Capacity

Compliance

with MTCU Contract

Directed Improvement - Funding Approval with Compliance and In Year

Improvements to Overall Service Quality StandardOrganization is NOT in compliance with contract/guidelines and must submit an

action plan for achieving compliance

Organization has not achieved Overall Service Quality Standard for service

delivery nor improvement targets from last contract

OC review demonstrates there is capacity and potential for immediate

improvement

✘ ✔✘

Termination – Notice of contract termination, procedures to support

client service transferOrganization did not achieve in year service improvements to standard and/or

is not in compliance with contract/guidelines

Organization does not demonstrate necessary OC to support improvement

✘ ✘✘

Official Review – Funding Approval with Immediate Compliance

DirectiveOrganization is NOT in compliance with contract despite achieving or

surpassing Overall Service Quality Standard for service delivery and must

submit an action plan for achieving compliance

OC review demonstrates little ability for sustained or improved results

Organization did not meet improvement targets from last contract.

✔ ✘✘

Success – Funding approval, Performance Incentive and/or Award

PotentialOrganization is in compliance with contract and program guidelines and has

achieved or surpassed Overall Service Quality Standard for service delivery

OC review demonstrates ability for sustained and/or improved results

Organization has attained improvement targets identified in last contract

✔ ✔✔

MTCU Response

Overall

Service

Quality

Standard Achieved

Reached

Improvement Targets

2.

11

Page 12: Outcome Based Management

Bell, Browne, Molnar & Delicate Consulting

CIC, Implementing the modernized approach,

October 2009

Page 13: Outcome Based Management

Bell, Browne, Molnar & Delicate Consulting

CIC, Implementing

the modernized

approach, October

2009

Page 14: Outcome Based Management

Bell, Browne, Molnar & Delicate Consulting 14

2. PERFORMANCE MEASUREMENT FUNDAMENTALS

An intensive.

Page 15: Outcome Based Management

Bell, Browne, Molnar & Delicate Consulting

Performance Measurement…

is all about answering the following questions:– Why do we exist?

– What do we want to accomplish / what is our end state?

– How are we going to deliver our Programs?

– How will we know how well we are doing?

– What programs, priorities and initiatives do we need to undertake to work meaningfully towards our end state?

15

Page 16: Outcome Based Management

Bell, Browne, Molnar & Delicate Consulting

Some Basic Requirements for Success

• Common Glossary and Methodology

• Common understanding and agreement of the desired ‘end state’

• Establishing the “vital few” performance measures

• Building a performance report and reporting strategy

• Formalize a continuous improvement strategy

16

Page 17: Outcome Based Management

Bell, Browne, Molnar & Delicate Consulting

The foundation – the logic model

17

Inputs Activities OutputsShort-term

Outcomes

Intermediate

Outcomes

Long-term

Outcomes

Low Influence →← High Control

Strategic

Outcomes

RESULTS

Page 18: Outcome Based Management

Bell, Browne, Molnar & Delicate Consulting

Inputs, Activities, Outputs and Outcomes

18

Definition Examples

Inputs“what we need to do the work”

The human, financial, structural resources, knowledge and information necessary to perform the activities.

Activities“what we do”

Describe a collection of functions (actions, jobs, tasks) that consume inputs/resources and whose outputs deliver benefits/impacts to a client or contribute to a particular outcome/set of outcomes

Outputs“what we produce”

The direct products and/or services (“usually tangible and concrete”) generated through activities without specific reference to their ultimate purpose

Outcomes“why we do it”

The benefits or desired states (“not fully under our influence”) to which the outputs of activities contribute

Page 19: Outcome Based Management

Bell, Browne, Molnar & Delicate Consulting

Four Types of Outcomes

Categorized according to the degree of influence, as follows:

1. Immediate Outcomes First-level effects of, or immediate response to the outputs, e.g., changes in compliance rates or degree of customer satisfaction.

2. Intermediate Outcomes are the second order of outcomes, Benefits and changes in behaviour, decisions, policies and social action attributable to outputs to demonstrate that program objectives are being met, e.g., increased employability as a result of a training program.

19

Page 20: Outcome Based Management

Bell, Browne, Molnar & Delicate Consulting

3. Ultimate/Long-term Outcomes The ultimate or long-term consequences for human, economic, civic or environmental benefit, to which government policy or legislation contributes, e.g., life expectancy rates, overall economic performance. Very broad in nature, requires many departments collaboration and take some time to achieve.

All of which feed the…

4. Strategic Outcomes are those long term and enduring organizational and/or government-wide benefits or goals that the Project, Program or Organization is contributing to.

Four Types of Outcomes

20

Page 21: Outcome Based Management

Bell, Browne, Molnar & Delicate Consulting

WHAT CHANGES IN THE WORLD AS A RESULT OF THE WORK YOUR AGENCY IS DOING?

This first, then this, then this, then this….

Products and

Services

LESS CONTROL

(attribution difficult)

KNOWLEDGE ATTITUDES

•Awareness•Understanding•Skills

•Perceptions•Acceptance

BEHAVIOUR

•Involvement•Compliance•Action

EARLY / LATER EFFECTS

•Societal change•Socio-economic

benefits

OUTPUTS IMMEDIATE

OUTCOMES

INTERMEDIATE OUTCOMES

Long TermOUTCOMES

MORE CONTROL

(easier attribution)

Social Marketing Role

21

Page 22: Outcome Based Management

Bell, Browne, Molnar & Delicate Consulting

Principles of Good Outcome Statements

1. Outcomes are noun-based desired end states (e.g., “newcomers are meaningfully employed…” rather than “increase employment opportunities for newcomers …”)

2. Directional outcome statements (e.g., “Improved/increased labour market access by Ontarians…”) do not belong in logic models or outcome statements. Direction is achieved through targets not outcome statements

3. Avoid self-serving statements (e.g., “World leader in settlement services”). After all, this is not the end state desired

4. Clear cause and effect linkage to the next level(s) of outcomes

5. SIMPLE: Outcome described in a way that helps Canadians relate to the desired state that is being described

22

Page 23: Outcome Based Management

Bell, Browne, Molnar & Delicate Consulting 23

Exercise: Outcomes language exercise

Small group work

• Examine the bulleted list and reach consensus on which items are immediate, intermediate, long term outcomes

4. PM Concepts

Page 24: Outcome Based Management

Bell, Browne, Molnar & Delicate Consulting 24

Outcome ExerciseExamine the list and reach consensus on which items are immediate, intermediate, OR long term outcomes Imm / Inter / LT

4. PM Concepts

1. Newcomers and their families are healthy, safe and secure.

2. Our settlement agency is recognized and consulted in settlement-related policy development.

3. Communities know, understand and appreciate the social, cultural and economic benefits that newcomers bring.

4. A skilled, adaptable and inclusive labour force and efficient labour market.

5. Preservation of the Environment.

6. Women and Men have equal treatment under the law.

7. Newcomers have the job search skills required.

8. Canadians provide a welcoming community to facilitate the full participation of newcomers into Canadian society

9. Newcomers access affordable childcare.

10. Certainty over ownership and use of land and resources by First Nations and Inuit communities.

11. Newcomers and their family actively works towards achievement of goals

12. Newcomers are aware of and access settlement programming (e.g.: labour market language training with a bridge-to-work component)

Page 25: Outcome Based Management

Bell, Browne, Molnar & Delicate Consulting 25

Direct Outcomes Shared Outcomes Ultimate Outcomes

Settlement Logic Model

Page 26: Outcome Based Management

Bell, Browne, Molnar & Delicate Consulting

Considering these outcomes, what SHOULD we be measuring?(ok to stick post-it notes to the relevant column)

1. Outcomes Measured, Outcomes Managed

Direct Outcomes Shared Outcomes Ultimate Outcomes

Page 27: Outcome Based Management

Bell, Browne, Molnar & Delicate Consulting 27

Indicators and Targets

• Indicators ( The “How”)– A statistic or parameter that, tracked over time, provides information

on trends in the condition of a phenomenon (e.g., percentage of clients that are satisfied)

• Targets (The “How Much”)– Specific performance goals tied to indicators against which actual

performance will be compared (e.g., 90% client satisfaction rating)

• Standards– Standards imply a service level to which an organization is prepared to

commit, either internally, externally or both.

Page 28: Outcome Based Management

Bell, Browne, Molnar & Delicate Consulting

Examples - Indicators

• % of newcomers who are aware of the skills and expectations necessary to join the Canadian labour force

• % of newcomers who access to labour market language training

• % of clients who report that they have the language ability and skills needed to participate socially, culturally and economically in Canada

• % of clients who report that they feel connected to the broader community and social networks

• % of program participants who claim to create a welcoming community for newcomers

• % employed for > than 3 months

• % satisfied with service

28

Page 29: Outcome Based Management

Bell, Browne, Molnar & Delicate Consulting

Examples - Targets

• 80% of newcomers aware of the skills and expectations necessary to join the Canadian labour force

• 90% of newcomers have access to labour market language training

• 64% employment retained for > 3 months

• Full integration of newcomers as participating members of Canadian society (economic, education, employment, etc.) within 10 years of arrival.

29

Page 30: Outcome Based Management

Bell, Browne, Molnar & Delicate Consulting

CASE STUDY –Employment Ontario

Achieving the standard of overall service quality is key to stable and ongoing funding

For illustration purposes only!! Provincial Service Quality Standard

DIMENSION

MEASURE

MINIMUM PROVINCIAL STANDARD

WEIGHT

SQS

VALUE

MAXIMUM

VALUE

Customer Service (40%)

1. Customer Satisfaction (participant/employer)

85% 15% 1.28 1.5

2. Service Coordination 30% 25% 0.75 2.5

Effectiveness (50%)

3. Employed/Career Path 69% 25% 1.73 2.5

4. Training/Education 10% 10% 0.10 1.0

5. Suitability (Participant Profile)

25% 15% 0.38 1.5

Efficiency (10%) 6. Intake in assisted service

90% 5% 0.45 0.5

7. Workshop Activities/Information Sessions

90% 5% 0.45 0.5

Service Quality Standard 10.0

Customer Satisfaction (0.85 x 1.5) = 1.28

Service Coordination (0.30 x 2.5) = .75

Employed/Career Path (0.69 x 2.5) = 1.73

Training/Education (0.10 x 1.0) = 0.10

Suitability (0.25 x 1.5) = 0.38

Intake in Assisted Service (.90 x .5) = 0.45

Workshops (.90 x .5) = 0.45

Overall Service Quality Standard 5.1430

Page 31: Outcome Based Management

Bell, Browne, Molnar & Delicate Consulting

How to Determine “the Critical Few” Indicators

• Credible – provide a credible, independent view

• Specific - eliminate ambiguity, show relevance to the expected outcome

• Linked - clear (cause and effect) linkages to other indicators

• Reliable - scientifically and statistically sound, provide an appropriate degree of accuracy. Measure the same thing across time (allowing comparisons) and for different groups/regions

• Available – data are easily accessible or there is a low level of effort to collect and analyze

• Cost-effective – the costs for collection are aligned with the overall utility of the indicators

• Understandable – data can be easily grasped by various audiences

31

Page 32: Outcome Based Management

Bell, Browne, Molnar & Delicate Consulting

Performance Indicator Assessment Grid

Priority

Degree of importance/relevance in terms of measuring efficiency (“doing things well”) and/or effectiveness

(“doing the right things/having an impact”)

High 3 It is essential in order to measure efficiency and/or effectiveness

Medium 2 It is useful in measuring efficiency and/or effectiveness

Low 1 It is of little use in measuring efficiency and/or effectiveness

Data Accessibility

The degree to which data are readily available either manually and/or through an automated system

High 3 Indicator data are immediately available

Medium 2 The appropriate data are not immediately available through an automated or manual system

but could be gathered relatively easily

Low 1 The data to support this indicator are not currently available or accessible and will be difficult

to access

Level of Effort

An estimate of the level of effort required to report reliably against the indicator on a regular basis

Low 3 Minimal effort to gather, analyze, interpret and report reliably on the indicator

Medium 2 It requires a moderate degree of effort to reliably report on the indicator

High 1 It is very resource intensive to reliably report on the indicator

32

Page 33: Outcome Based Management

Bell, Browne, Molnar & Delicate Consulting

WHY MUST A NON-PROFIT USE THIS METHODOLOGY TO REMAIN COMPETITIVE?

Discussion.

33

Type here

Page 34: Outcome Based Management

Bell, Browne, Molnar & Delicate Consulting

HOW COULD THIS CHANGE THE WAY YOUR SERVICE IS DELIVERED?

Discussion.

34

Type here

Page 35: Outcome Based Management

Bell, Browne, Molnar & Delicate Consulting

SARAH DELICATE, BBMD (905) 922-9726 (cell), [email protected]

35