Impact measurement and client wellbeingAndrew Wearring & Joann Fildes

Mission Australia is client focused. We strive for client informed services that achieve positive outcomes with individuals and communities.

“How do we know what we’re doing is helping people?” and“how do we measure that impact and aggregate the results to show how much difference the organisation is making?”

Outputs are the products, services or facilities that result from an organisation’s or project’s activities.

Outcomes are the changes, benefits, learning or other effects that result from what the project or organisation makes, offers or provides.

Impact is the broader or longer-term effects of projects or organisation’s outputs, outcomes and activities.

Outputs, outcomes and impact

Our approach

Inputs, Outputs & Activities

Outcomes Impact

Organisational Strategies

• Early intervention and prevention

• Advocacy and client voice

• Evidence based best practice and case management

• Self-efficacy and optimism• Confidence and fulfillment• Participation and inclusion

• Independence

Evaluation Methods

• Input and output measurement using existing administrative systems

• Development of indicators.• Consultation with service

and organisational staff• Lessons from previous

pilots• MA Client Wellbeing Survey

pilot

• Data analysis and reflection of the Theory of Change and Practice

• Reporting back to stakeholders and practitioners

Developing impact measurement

Outcome measures must:

• Be externally credible

• Be used with the external population (to provide comparison data)

• Have communication power, internally and externally

• Demonstrate program effectiveness to funders and donors

Principles

1. Outcome hierarchies developed and implemented

2. Indicators and collection methodologies developed

3. Consultation and approval with staff across the organisation

4. MA Client Wellbeing Survey pilot developed and piloted in two services:

PHaMs and JSA Stream 4:• The Personal Helpers and Mentors service (PHaMs) is a non clinical ‐

recovery focussed support service for people experiencing ‐mental health issues.

• Jobs Services Australia, now JobActive, was the Australian Government’s employment program. Stream 4 clients were those with multiple and complex needs.

Stages

Stages of the pilot:

1. Processes for data collection developed

2. Ethics, consent and protocols developed

3. Sampling framework – all new clients in a set period

4. Training – face to face and webinars

5. Repeat measures data collection implemented

6. Data analysis and reporting

Implementing impact measurement

Client Wellbeing Survey

Mission Australia Client Wellbeing Survey

Captures subjective wellbeing through a combination of measures

• Personal Wellbeing Index (PWI)

• Single item measures on:• Housing• Financial security• Health• Self-esteem and control

• Demographics:• Collected via MA systems• Age, gender, ATSI status • Service interventions

PWI is the foundation of the survey. It measures subjective wellbeing by asking about satisfaction in:

• Standard or living• Health• Achieving• Relationships• Safety• Community• Future security• Life as a whole

Mission Australia Client Wellbeing Survey

Journey to independence

PHaMs JSA

Participants at intake

Many of our clients have very low levels of personal wellbeing

High RiskPWI 0-50

ChallengedPWI 51-69

Normal RangePWI 70-100

94%

6%

50%

25%

25%

PHaMs participants

MA saw significant improvements for the PHaMs clients across a range of domains and self-reported measures after eight months in the program

0 20 40 60 80 100

50

48

35

Personal Wellbeing Index

Wave 1 Wave 2 Wave 3

PHaMs participants

Standard of living

Health

Achievements

Future security

Life as a whole

0 20 40 60 80 100

30

31

21

31

26

51

43

39

48

44

49

49

48

45

49

Satisfaction with:

Wave 3Wave 2Wave 1

PHaMs participants

Control over own life: has less than mixed control

Housing adequacy: is not adequate

Coping: Not at all or a little

Not enough money to meet needs

0 20 40 60 80 100

82

63

77

47

59

25

47

18

41

19

35

18

Proportion self-reporting poor outcomes

Wave 3

Wave 2

Wave 1

%

PHaMs participants

0 10 20 30 40 50 60 70 80 90 100

35

71

94

Accessing appropriate mix of support services

Wave 3 Wave 2 Wave 1

%

JSA participants

Significant improvements were not seen in wellbeing of JSA Stream 4 job seekers

0 20 40 60 80 100

54

51

55

Personal Wellbeing IndexWave 1 Wave 2 Wave 3

JSA participants

Standard of living

Health

Achievements

Future security

Life as a whole

0 20 40 60 80 100

54

58

50

42

45

48

53

45

45

45

55

55

44

50

46

Satisfaction with:Wave 3Wave 2Wave 1

Mission Australia’s Client Wellbeing Pilot has been validated as a useful tool to measure and inform client journeys to independence in programs where clients receive wrap-around intensive supports, such as in the PHaMs program.

The findings of the pilot indicated that where a program has a narrow focus and a constrained delivery model such as in the Job Services Australia model, the tool is not well adapted to collect the outcomes.

1. Consider what is already known to develop a sound framework

2. Gain management endorsement and collaborate

3. Use repeat measures

Key lessons about measuring impact

Informing policy & practice

• Continuous quality improvement in our services.

• Where site results were reported and reflective practice undertaken, service improvements have been identified and implemented.

• The pilot will be continued in selected sites to inform the design of a model for the roll out of the NDIS.

• To inform a review of an outcomes measurement framework across MA for reducing homelessness and strengthening communities.

• As an iterative process with government in tender and program design.

How Mission Australia uses outcome measurement

How Mission Australia uses outcome measurement

Reflecting on practice

PHaMs sites reflected on their practice. They were provided with site specific outcome measures and questions to help them reflect on what the outcome measures mean for practice:

• Are our clients better off? • If not, what could be the reasons and are any of them reasons that we

can influence with our practice?

Feedback from the field:

“By having regular discussions the pilot has enabled us to identify: • How we can continue to increase support for participants,

carers and families, • How we can improve on group activities, external factors

which may be influencing participants’ daily lives etc.

As a team our purpose is to assist participants in pursuing the best outcomes possible towards their recovery, as such being able to participate in the Client Impact Pilot has been a valuable experience.”

PHaMs Manager

• There is no common set of indicators being used across Commonwealth and State Governments for outcomes-based funding and program delivery.

• The modified PWI approach was adopted to measure our impact on client independence. Future impact evaluation needs to measure outcomes against MA’s current goals of reducing homelessness and strengthening communities across our diverse programs and services.

• Developing outcomes measurement systems and requisite IT platforms requires staff capacity and resources.

Challenges

Outcome measurement is important. It shows us and our stakeholders how we are achieving client outcomes. It assists us to continually reflect and improve.

For more information

Andrew Wearring

t: 02 9288 0012

e: researchandpolicy@missionaustralia.com.au