2012Desiree Nangle, Evaluation Project [email protected]
SIMPLE
COMMON SENSE
PLAIN LANGUAGE
MINIMUM PAPER
USEFUL
With thanks to
The Fiscal Policy Studies InstituteSanta Fe, New Mexico
What is Results Based Accountability?
• Starts with ends and works backwards to means– Ends are how customers are better off when
the program works the way it should
• RBA is meant to get you from talk to action quickly
• Used for continuous program improvement
• Inclusive process
Results Based Accountabilityis made up of two parts:
POPULATION ACCOUNTABILITYIs about the well-being ofWHOLE POPULATIONS
For Communities – Cities – Counties – States - Nations
PERFORMANCE ACCOUNTABILITYIs about the well-being of
CUSTOMER POPULATIONSFor Programs – Agencies – Services
August, 2012
7 Performance Accountability Questions
1. Who are our customers?
2. How can we measure if our customers are better off?
3. How can we measure if we are delivering services well?
4. How are we doing on the most important of these measures?
5. Who are the partners that have a role to play in boing better?
6. What works to do better, including low-cost and no-cost ideas?
7. What do we propose to do?
7 Performance Accountability Questions
1. Who are our customers?
2. How can we measure if our customers are better off?
3. How can we measure if we are delivering services well?
4. How are we doing on the most important of these measures?
5. Who are the partners that have a role to play in boing better?
6. What works to do better, including low-cost and no-cost ideas?
7. What do we propose to do?
How much service did we deliver?
Performance Measures
How welldid we
deliver it?
How much change / effect
did we produce?
What quality of change / effect
did we produce?
Quantity Quality
Effe
ct
Effo
rt
How much did we do?
How well did we do it?
Is anyone better off?
Quantity Quality
Effe
ct
E
ffort
• Skills and Knowledge• Attitude and Opinion• Behaviour• Circumstance
• Client Satisfaction• Standards • Unit Costs• Wait List• Staff Quality• Targets
• Number Clients• Number Activities
Types of Measures Found in Each Quadrant
How much did we do? How well did we do it?
Is anyone better off?
Quantity Quality
Effe
ct
E
ffort
% of parents with more of a child focusoassessed using client self-reported perceptions in questionnaire
o % of parents with less parent to parent conflict
o % of parents with decreased litigationoAssessed via Family Court feedback questionnaire
o % of clients seeking other services
o % of parents with improved ability to co-parent
o % of parents with improved self-awareness
Number of groups Number of clients
No shows/Drop out ratesBookings for intakeIntake groupGroup to completion
Level of satisfaction Family Court has with MDF
Staff satisfaction Reconnects
Mums and Dads Forever2012
How much did we do?
Not All Performance Measures Are Created Equal
How well did we do it?
Is anyone better off?
LeastImportant
Quantity Quality
Effe
ct
E
ffort
MostImportant
Least
Most
AlsoVery Important
How much did we do?
The Matter of Control
How well did we do it?
Is anyone better off?
Quantity Quality
Effe
ct
E
ffort
LeastControl
MostControl
Program: _______________
PerformanceMeasureBaseline
Story behind the baseline
--------------------------- ---------------------------
Partners
--------------------------- ---------------------------
Three Best Ideas – What Works
1. --------------------------- 2. --------------------------- 3. ---------No-cost / low-cost
ONE PAGE Turn the Curve Report: Performance
4. --------- Off the Wall
Example of a Turn the Curve Report
Results Based Accountability
Service Description: Anglicare WA has chosen to implement Results Based Accountability (RBA) framework across all its services. This choice is centred on a strong desire to know and have evidence the impact our services have.
Defined Service Users
All Anglicare WA staff
Data Development Agenda
1. Determine population outcomes our services contribute to
2. Determine whether or not there are questions we can ask across services that use the same wording and same scale
Headline Performance Measures
% of staff who understand what RBA is
% of programs in Phase 2 of RBA implementation
% of programs in Phase 3 of RBA implementation
% of staff who support RBA implementation within Anglicare WA
Story Behind the Baselines
While all four baseline indicators increased from April 2012 to August 2012, some had greater increases than others. While the percentage of staff understanding RBA increased significantly (from 54% to 74%) since April, the increase in staff supporting RBA use in Anglicare WA had a much less significant increase of only 2.5%. Part of the reason for the smaller increase is most likely due to the smaller response rate on this survey (~25% of staff). The percentage of programs in Phase two jumped considerably and progress was also made in moving services from Phase two to Phase three since April (an 8% increase). This increase in services moving to Phase three is most likely due to changes that were made in the process based on suggestions given by Mark Friedman while he was here in June.
Partners Who Can Help Us Do Better
All Anglicare WA Programs and Staff
The Community Sector
Other Not-for-Profit Organisations
National and Federal Government
Trained RBA Trainers
Action Plan to Continue Turning the Curves
In an effort to continue improving performance on our four headline measures, Anglicare WA will:•Revise the length of the survey and possibly how often it is being sent out to staff in an effort to increase staff response rates•Continue focusing on data that services do have to begin the process of RBA implementation and work around helping them develop tools to address each program’s data development agenda•Highlight success stories and curves turned to encourage a culture of RBA monitoring among staff
RBA HEADLINE PERFORMANCE MEASURES
RBA on RBA
(roll out with services)
2012
How much did we do? How well did we do it?
Is anyone better off?
Quantity Quality
Effe
ct
E
ffort
• # of active services by phases
• # of staff involved in active services by phase
• staff satisfaction with training and support
• # RBA facilitators trained
• time to self-sustainability
• time programs spend in each stage
• validity of measures• validity of data
• % staff that understand what RBA is• % staff that support RBA within Anglicare
WA• % services implementing RBA • % services using RBA for program
improvement• % services with the capacity to use RBA
unassisted• % of curves turned
33%(48 out of 61 services) =78%
66%
81%
19 in Phase 37
75%
(1 service) = 2%
RBA on RBAUpdated: March 2013
Do staff understand what RBA is?
December 2011: 53.6% AgreeApril 2012: 68.3% Agree
August 2012: 73.5% AgreeMarch 2013: 80.9% Agree
2013
Do staff think RBA is good for Anglicare?
December 2011: 55.2% AgreeApril 2012: 60.4% Agree
August 2012: 66.3% AgreeMarch 2013: 66.1% Agree
2013
Results Based Accountability March 2013 Headline Measures Report
Service Description: Anglicare WA has chosen to implement Results Based Accountability (RBA) framework across all its services. This choice is centred on a strong desire to know and have evidence the impact our services have.
Defined Service Users
All Anglicare WA staff
Data Development Agenda
1. Determine population outcomes our services contribute to
2. Determine whether or not there are questions we can ask across services that use the same wording and same scale
Headline Performance Measures
% of staff who understand what RBA is
% of services in Phase 3 of RBA implementation
% of staff who think RBA will help improve their service
% of staff who support RBA implementation within Anglicare WA
Story Behind the Baselines
Partners Who Can Help Us Do Better
All Anglicare WA Programs and Staff
The Community Sector
Other Not-for-Profit Organisations
National and Federal Government
Trained RBA Trainers
Action Plan to Continue Turning the Curves
In an effort to continue improving performance on our four headline measures, Anglicare WA will:•…………•…………..•…………..
RBA HEADLINE PERFORMANCE MEASURES
Open-ended response
• Challenges of RBA • Opportunities with RBA
2012
Case Study
YES! Housing Project
YES! HousingTURNING THE CURVE
How Much Did We Do? How Well Did We Do It?
# of housed clients # of referrals made by DCP # of young people on the waitlist # of allocations made by DoH # of young people who disengage after being housed # of days on waitlist # of young people who complete the series of
workshops # of clients that receive a certificate for completion
of ILS workshops
% of young people with case plans % of young people who failed tenancies/eviction rate % of qualified staff who have completed core training
including:o Attending and participating in regular supervisiono Proper ways trainingo TAS Training: Tenancy Law for Community
Workers & Advocating for Public Housing Tenantso Other relevant training as identified in supervision
% of young people satisfied with the service % of HSO from DoH who are satisfied with the program % of clients referred to other agencies % of waitlist population, housed population, and
mainstreamed population who are:o Male/femaleo Indigenouso Parents
Unit Cost (cost/#of clients) Referrals from DCP are regular and timely
Is Anyone Better Off?
% of clients who access safe, secure, and stable housing at the completion of case management % of clients that demonstrate improved knowledge of ILS skills (i.e. home, work, education, parenting, tenancy
and healthy relationships) % of clients that ustilse strategies developed through case management to address identified barriers to
maintain their housing % of clients successfully linked to appropriate specialist services % of clients regularly engaging with their support worker to develop their case plans % of clients who achieve set goals
YES! Housing July – December 2012: RBA Headline Program Measures Report
Service Description: Anglicare WA’s YES! Housing program provides permanent housing for young people who are supported to establish independent living through long term housing, including public housing properties.
RBA Headline MeasuresRBA Headline Measures
Story Behind the Baselines
Graph 1: There continues to be an increase of young people on the waitlist trying to access safe, secure affordable accommodation. Unfortunately YES! Housing does not have the resources necessary to meet this ever-growing need.Graph 2: Data from graph 2 is based on the average difference from pre score to post score for the combined four ILS workshops (health, tenancy, parenting skills and home skills). The total average increase went down 1% in this past 6 months, mainly due to the health and relationships survey (deceased from a 31% increase Jan-June to 12% increase July-Dec). The Home Skills workshop increased along with Tenancy Support while the Parenting Skills results decreased in the past 6 months.Graph 3: 33.3% of clients with a case management plan mostly achieved or fully achieved their most important goal.Graph 4: 96.8% of clients engaged have a case management plan. Graph 5: 90% of clients self referred to the program or were referred through word of mouth. The remaining 10% were referred from DCP and Community agencies.
Action Plan to Continue Turning the Curves
In an effort to continue improving performance on our headline measures, YES! Housing will:•Research other options for content, presentation methods and implementation ideas from clients participating in the Health & Relationship and Parenting workshops.•Continue to implement the case management process to assist clients in achieving their most important goals through regular outreach visits.•Continue to create case plans with housed and waitlisted clients through implementing the case management process and developing strategies to engage clients who have disengaged from the program.•Develop stronger networks with DCP, DoH, Community Organisations, other housing organisations to offer support for their clients through case management process more accommodation options for clients in other housing situations to maintain a successful tenancy.•Implement strategies developed in consultation with DCP during our recent Program Review to access alternative affordable, safe, secure housing for clients on the YES! Housing waitlist by establishing and developing relationships with private real estate.•Review our current waitlist and remove ineligible clients.
Partners Who Can Help Us Do Better
Internal partners includingY-SHAC, Foyer, YPSG, Youth Services Manager, Fundraising
External partners including Department of Housing, Department for Child Protection, Foundation Housing, Community Housing Real Estate,YACWA,, young people,
Data Development Agenda
Yes! Housing will work on developing and implementing new survey tools:•to measure Independent Living Skills knowledge increase that is only post-workshop (not pre and post)•to measure the Department of Housing’s satisfaction with the work done by YES! Housing
Clients
Young people 15-17 years who are homeless or at risk of being
homeless.
Phase 1
Awareness raising
• Emails• Presentations to groups
Phase 2
Service Planning
• Scoping and readiness assessment
• Workshop • Define clients• Agree outcomes• Agree measures• Data strategy• Plot baselines
Phase 3
Service Implementation
• Data collection• Plot curves• Monitor and use
Phase 4
Normalise
• Refine• Use unassisted
Program: _______________
Performance Measure PerformanceMeasureBaseline
Story behind the baseline --------------------------- --------------------------- (List as many as needed)
Partners --------------------------- --------------------------- (List as many as needed)
Three Best Ideas – What Works 1. --------------------------- 2. --------------------------- 3. ---------No-cost / low-cost
ONE PAGE Turn the Curve Report: Performance
4. --------- Off the Wall
Four Phases of Implementation
What we do is an not a science. It can’t be measured
What does the data say?I know I should
be measuring but I don’t know how
What if my program doesn’t show success?
Check out these cool metrics!
The Task (1)
Quadrant exercise: Performance measures
• How much did we do?• How well did we do it?• Is anyone better off?
How much did we do?
How well did we do it?
Is anyone better off?
Quantity Quality
Effe
ct
E
ffort
• Skills and Knowledge• Attitude and Opinion• Behaviour• Circumstance
• Client Satisfaction• Standards • Unit Costs• Wait List• Staff Quality• Targets
• Number Clients• Number Activities
Types of Measures Found in Each Quadrant