A FOCUS ON QUALITY OUTCOMES DAO and WANADA Session 1 st May 2012 ANITA GHOSE B. App Sci; MBA; MAICD...

A FOCUS ON QUALITY OUTCOMES

DAO and WANADASession 1st May 2012

ANITA GHOSE B. App Sci; MBA; MAICD

DIRECTORAnita Ghose and Associates

AGENDA1. An overview of outcomes

2. Developing outcomes

3. Measuring outcomes

1. An Overview of Outcomes

INPUTS OUTPUTS OUTCOMES

Resources Used

Activities

Products

Participation Short Medium

What we invest

What we do

Who we reach

What are the results?

What changed?

Long-term

Eg:MoneyStaff timeFacilitiesEquipment

Eg:Provide counsellingProvide diversion servicesProvide residential rehab services

What we produce

Eg:Number of hours of service deliveredNumber of counselling sessions providedNumber of people in residential rehab services

Eg:People who secured stable housingImproved attitudes towards cannabis useIncreased knowledge of the effects of alcohol and drug useReduction in harmful use of alcohol and drugs

REFRESHER – KEY CONCEPTS


Resources Used

Activities

Products


What we invest

What we do

Who we reach


What changed?

Long-term

What we produce

Based on Productivity Commission Report 2010; United Way of America (1996)

PROCESSES – eg waiting lists, response times, client/staff ratios


Resources Used

Activities

Products


What we invest

What we do

Who we reach


What changed?

Long-term

What we produce

Based on Productivity Commission Report 2010; United Way of America (1996) Population

Inputs—the resources used by the organisation to achieve the objectives of an activity – “what we invest” eg money, staff time, facilities, equipment

Processes – the related tasks and systems of the organisation to fulfil its mission (eg. response times, staff trained, worker safety, client/staff ratio, length of wait lists)

Outputs— the direct products of the organisation’s activities or intervention (“what we produce”) and the participation (“who did we reach”). Eg number of service hours delivered, number of counselling sessions provided, number of clients in residential rehabilitation services

Outcomes—the effects or change on an individual or group as a result of the activities and outputs (“what was the result?) Eg people who secured stable housing, reduction in harmful use of alcohol and drugs, increased knowledge of the effects of cannabis

Impacts—the broader effects of an activity reflected in community wellbeing domains (what changed for the population?) Eg. reduced unemployment, increased mental health, stronger families


Resources Used

Activities

Products


What we invest

What we do

Who we reach


What changed?

Long-term

What we produce

The effects or change

as a result of

the activities

or outputs


Resources Used

Activities

Products


What we invest

What we do

Who we reach


What changed?

Long-term



What we produce



Government Agencies are moving here in OBP


Resources Used

Activities

Products


What we invest

What we do

Who we reach


What changed?

Long-term



What we produce



This isn’t being prescribed by Govt – BUT organisations do need to understand their business here

OUTCOMESThere will generally be effects, results, changes in one or more of these core areas:

KNOWLEDGE OR SKILLS

ATTITUDE OR OPINION

VALUES

BEHAVIOUR

CIRCUMSTANCE

CONDITION OR STATUS

OUTCOMES

KNOWLEDGE OR SKILLS - increased knowledge of the effects of alcohol and drug use on the body

ATTITUDE OR OPINION – improved attitudes or opinion about the dangers of cannabis use

VALUES - enhanced values on the impact of drug use on others around the person

BEHAVIOUR – reduction in the harmful use of drugs and alcohol

CIRCUMSTANCE – increased ability to secure and maintain stable housing

CONDITION OR STATUS – improved general health and wellbeing

OUTCOMES – KEY CONSIDERATIONS

Proximal/DistalAttribution/ContributionPathways approachLink to outputsReasonable numberPerverseMeasurement versus achievementPositive or negativeIntended or unintended

EXAMPLE

Community Drug Service Team

Counselling model

Proximal outcome:

Amphetamine users have reduced their drug and alcohol useAmphetamine users have an improved level of mental health and wellbeing

Distal outcome:Amphetamine users have reduced levels of offending

EXAMPLE

Mother and Child Program

Specialist support

Proximal outcome:

Mothers reduce their alcohol and drug useMothers demonstrate an increased awareness of the harmful effects of alcohol and drug use on their childMothers remain the main caregiver of their child

Distal outcome:Mothers participate more in their children’s schooling program

How efficient is the organisation?

How effective is the organisation?

Efficient and effective


Resources Used

Activities

Products

Participation Short Medium Long-term

Efficiency = Inputs Outputs

Example: the average cost per session of counselling delivered

Effectiveness = Inputs Outcome

Example: the average cost per completed as planned episode

Outcomes Frameworks

A Logic Model

Source: University of Wisconsin – Extension, Cooperative Extension, Program Development and Evaluation

Is a framework for describing the relationships between investments, activities, and results.

Depicts a program showing what the program will do and what it is to accomplish (reasoning)

Provides a series of “if-then” relationships that, if implemented as intended, lead to the desired outcomes

Provides a common approach for integrating planning, implementation, evaluation and reporting.

Applied at a program, process, multi component program, organisational level

Fully detailed logic modelFully detailed logic model


Logic model and reporting


RESULTS BASED ACCOUNTABILITY

Strong and stable families, positive family relationships

Rate of family breakdown due to alcohol and drug use

Rate of domestic violence

1. How much did we do? 2. How well did we do it?

3. Is anyone better off?

RESULT or OUTCOME

INDICATOR or BENCHMARK

PERFORMANCE MEASURE

A condition of well-being for children, adults, families or communities.

A measure which helps quantify the achievement of a result.

A measure of how well a program, agency or service system is working. Three types:

= Customer Results

Popu

latio

nPe

rfor

man

cePo

pula

tion

Perf

orm

ance

= Customer Results

Source: Mark Friedman 2005

How much did we do? How well did we do it?

Is anyone better off?

# Clients/customers served

# Activities (by type of activity)

% Common measurese.g. client/staff ratio, unit cost, clients from a target group, staff turnover, staff morale, staff fully trained, worker safety, client satisfaction (were they treated well), wait time, wait lists/turnaways, meeting specific industry standards

% Skills / Knowledge (e.g. parenting skills)

#

% Attitude / Opinion (e.g. toward drugs)

#

% Behaviour (eg drug and alcohol use)

#

% Circumstance (e.g. working, in stable housing)

#

% Activity-specific measures

e.g. timely, clients completing activity, correct and complete processes, meeting standards

Source:Mark Friedman 2005;Anglicare 2011

EFFECTEFFO

RT

QUANTITY QUALITY

How Population

&Performance Accountability

FIT TOGETHER

Contributionrelationship

Alignmentof measures

Appropriateresponsibility

THE LINKAGE Between POPULATION and PERFORMANCE

POPULATION ACCOUNTABILITY

General Physical Health Prevalence of liver diseaseStable Families and Relations Rate of alcohol and drug use Mental Health Rate of co-occurrence with mental health issues

# ofbed days

% receiving assessment withinagreed timeframe

# increased awareness of the impact of alcohol and drug use on physical health

PERFORMANCE ACCOUNTABILITY

POPULATIONRESULTS

Residential Rehab Services

Source:Mark Friedman 2005

% increased awareness of the impact of alcohol and drug use on physical health

2. Developing Outcomes

The Request

Key Assumptions

Partnership and collaboration between agency, organisation and individuals, families to identify community outcomes

Appropriate consultation framework and engagement strategies with key stakeholders

Appropriate mechanisms in place (formal and informal) to capture the information required to get the outcomes and time frames right

This is the stage of developing and identifying outcomes which is outside of scope for this workshop but is the important precursor to defining and writing outcomes

Individual outcomes for people with alcohol and drug issues

Family outcomes

Program or Service level outcomes

Organisational outcomes

Corporate – Financial; Human resource; Risk

Community outcomes

Population outcomes

procurement

Outcomes - different levels and purposes

Types of Levels

Outcomes – Responding to a Request

Key considerations:

•Statement of the desired change or result (what does it look like?)

•Interpretation – get clarity and agreed understanding across stakeholders

•Be clear about target group (eg child, mother, parent)

•Outcomes – at what level (eg individual, service level, population)

•What are you trying to assess?

•Measurable and meaningful; reasonable number

Outcomes – Responding to a Request

Key considerations:

•Attribution – circle of control and influence; proximal and distal

•System – how does this fit within the whole? Communities of practice

•Think about time frame – short term to long term; longitudinal change

•Consider context eg chronic relapsing condition

•Outcomes in the AOD sector are generally about results/effects or change in knowledge or skills; attitude or opinion; values; behaviour; circumstance; condition or status

A useful approach to writing an outcome statement

SMART objectives – specific, measurable, achievable, results oriented, time frame

Who/what Change/desired effect

In what By when

Parents of people with alcohol and drug issues participating in the parenting program

increase their knowledge of alcohol and drug use

at the completion of the program

Young people in a specialist alcohol and drug treatment centre

reduce their harmful alcohol and drug use

after the first year of the program

3. Measuring Outcomes

Indicator - is a measure for which data is available which provides a specific and reliable means to quantify the achievement of the outcome. These results indicators are what we will see if change occurs.

Ask the questions: What will we see if change occurs?What will be the evidence?

Each outcome can have a set of indicators, which, if taken together gives a reasonable approximation of the achievement of the outcome or result

Indicator – can be expressed as a # or % (quantifies the achievement of the outcome)

Eg: The percentage of participants reporting increased awareness about alcohol and drugs

The number of people who have secured stable housing since the programme’s commencement

The percentage of mothers who have retained primary care of their children since the program’s completion

Sometimes, one indicator may be sufficient to evidence the outcome’s achievement.

Often, several indicators are required. Select carefully to be meaningful and not over burdensome.

Outcome: SOBER UP CENTRE – Clients are kept safe overnight

Reduction in complaints about public drunkeness

Reduction in incidence of domestic violence Reduction in street arrests

of clients

Reduction in clients attending A&E depts in hospital

About measurement tools

There is no magical measurement tool!

There are a range of tools (eg. clinical profiling tools, instruments, surveys and questionnaires etc) that can be used but they all have limitations

Develop an “evidencing the outcome” approach that is context specific, in collaboration with the sector organisations, individuals and families which can give a reasonable approximation of the achievement of the outcome

There is always a story, assumptions and a context which needs to be explicitly stated

Reliability refers to the extent to which a measure produces accurateinformation.

Validity refers to the extent to which the measure actually measures what it isintended to measure.

Suitability refers to the criteria for selecting measures that meet the target group or intent – and are appropriate for those administering it

Relevancy refers to measures clearly related to the outcome being evaluated, providing useful information and allowing meaningful conclusions to be drawn

THIS IS A JOURNEY – CONSIDER THESE AND WORK TOWARDS

We need a baseline (pre-post)

Outcomes, Indicators, DataOutcomes, Indicators, Data

Outcomes Indicators Data collection

Sources Methods Sample Timing

Linkages and relationship, alignment, what is important, reducing administrative burdenLinkages and relationship, alignment, what is important, reducing administrative burden

Evidence based practice

SUMMARY OF THE DAY1. An overview of outcomes

2. Developing outcomes

3. Measuring outcomes

QUESTIONS

Consultants to Government and Community

A FOCUS ON QUALITY OUTCOMES DAO and WANADA Session 1 st May 2012 ANITA GHOSE B. App Sci; MBA; MAICD...

Documents

Transcript of A FOCUS ON QUALITY OUTCOMES DAO and WANADA Session 1 st May 2012 ANITA GHOSE B. App Sci; MBA; MAICD...