Using indicators to advocate Using indicators to advocate for policy and programsfor policy and programs

Dr Gill WesthorpDr Gill Westhorp

Community MattersCommunity Matters

Session OverviewSession Overview

Framing advocacy argumentsFraming advocacy arguments

Advocacy in one policy/intervention areaAdvocacy in one policy/intervention area

An approach to evidenceAn approach to evidence

Implications for advocating for policy and Implications for advocating for policy and programsprograms

AdvocacyAdvocacy

““advocare”: “to be called to stand beside”advocare”: “to be called to stand beside”

verbal support or argument for a cause or verbal support or argument for a cause or policypolicy; the function of being an advocate; the function of being an advocate

advocate: a person who supports or advocate: a person who supports or speaks in favour; a person who pleads for speaks in favour; a person who pleads for another; recommend, plead for, defendanother; recommend, plead for, defend

Australian Concise Oxford DictionaryAustralian Concise Oxford Dictionary

Framing the argumentFraming the argument

What’s the problem?What’s the problem?– Breached rights?Breached rights?– Unmet needs?Unmet needs?– ‘‘Wrong’ solution?Wrong’ solution?– Program effectiveness?Program effectiveness?– Cost-effectiveness?Cost-effectiveness?

What’s the solution?What’s the solution?

What’s the evidence?What’s the evidence?

Planning your advocacyPlanning your advocacy

To whom? - Politicians? Senior policy To whom? - Politicians? Senior policy makers? Service provision organisations?makers? Service provision organisations?For what? – An over-arching policy For what? – An over-arching policy framework? A change within an existing framework? A change within an existing policy? A particular program?policy? A particular program?What issues does the decision-maker What issues does the decision-maker need to take into account?need to take into account?What evidence will they find convincing?What evidence will they find convincing?

Matching evidence to argumentMatching evidence to argument

The ArgumentThe Argument The EvidenceThe Evidence

RightsRights UN ConventionsUN Conventions

LegislationLegislation

PolicyPolicy

NeedsNeeds Population-level statisticsPopulation-level statistics

Comparative statisticsComparative statistics

ConsultationConsultation

EffectivenessEffectiveness ResearchResearch

EvaluationEvaluation

Literature reviewLiterature review

Kinds of Evidence (R&E)Kinds of Evidence (R&E)

QualitativeQualitative QuantitativeQuantitative ReviewsReviews

InterviewsInterviews

Focus GroupsFocus Groups

EthnographyEthnography

Case studiesCase studies

Documents Documents

SurveysSurveys

QuestionnairesQuestionnaires

‘‘Objective’ o/c Objective’ o/c measuresmeasures

CostsCosts

Narrative Narrative reviewsreviews

Meta-analysisMeta-analysis

Realist Realist synthesissynthesis

What?What? How?How?

Why?Why?How much?How much? How common? How common?

Where? When?Where? When?

Mixed Evidence = Comprehensive ArgumentMixed Evidence = Comprehensive Argument

Who uses evaluation findings?Who uses evaluation findings?Internal to the program:Internal to the program:– program sponsors, directors, and practitionersprogram sponsors, directors, and practitioners– ““learning organisations”learning organisations”

External to the program:External to the program:– managers of other similar programs to learn how to managers of other similar programs to learn how to

improve their programs, improve their programs, – funding and policy bodies to figure out what to fund or how funding and policy bodies to figure out what to fund or how

to improve the operation of programs they sponsor, to improve the operation of programs they sponsor, – politicians to amend/make policies, politicians to amend/make policies, – social scientists to see what new knowledge has accrued social scientists to see what new knowledge has accrued

and incorporate it into theory and incorporate it into theory – evaluators to profit from the findings and the methods of evaluators to profit from the findings and the methods of

study. study. – civil society – as volunteers, members of Boards, for advocacy…civil society – as volunteers, members of Boards, for advocacy…

Types of useTypes of use““instrumental use” – decision-making e.g. by fundersinstrumental use” – decision-making e.g. by funders““conceptual use by local program people… learn more conceptual use by local program people… learn more about what the program is and does…about what the program is and does…mobilize support for changemobilize support for changeinfluence external to the program – “enlightenment”: influence external to the program – “enlightenment”:

“ “evaluation evidence often comes into currency evaluation evidence often comes into currency through professionals and academics and evaluators, through professionals and academics and evaluators, and it influences networks of practicing professionals and it influences networks of practicing professionals and policy wonks (Heclo,1978) infiltrates advocacy and policy wonks (Heclo,1978) infiltrates advocacy coalitions (Sabatier, 1993), alters policy paradigms coalitions (Sabatier, 1993), alters policy paradigms (Weiss, 1980), changes the policy agenda (Kingdon, (Weiss, 1980), changes the policy agenda (Kingdon, 1995), and affects belief communities within 1995), and affects belief communities within institutions (Radaelli, 1995).”institutions (Radaelli, 1995).”

Carol Weiss, 1998

Early Years Early InterventionEarly Years Early Intervention

Common risk and protective factors for (e.g.)Common risk and protective factors for (e.g.)– offendingoffending– homelessnesshomelessness– early school leavingearly school leaving– drug and alcohol abusedrug and alcohol abuse– unemploymentunemployment– mental health problems…mental health problems…

Early years brain development researchEarly years brain development research

Advocacy for early years interventionsAdvocacy for early years interventions

Child Outcomes for most effective programs:Child Outcomes for most effective programs: school completionschool completion college/university attendancecollege/university attendance employment/earningsemployment/earnings mental healthmental health drug use, tobacco usedrug use, tobacco use child abuse, removals from homechild abuse, removals from home delinquency, adult arrest for violence, jaildelinquency, adult arrest for violence, jail

Advocacy for early years interventions Advocacy for early years interventions (cont.)(cont.)

Parent outcomes for most effective programsParent outcomes for most effective programs completed high schoolcompleted high school employment/earningsemployment/earnings arrest for violencearrest for violence welfare supportwelfare support

Cost effectiveness of effective programsCost effectiveness of effective programs– Between ≈$4 and $10 saved for every $1 Between ≈$4 and $10 saved for every $1

spentspent

Early Head Start Early Head Start

Major funding program, US of A Major funding program, US of A Low income families with 0-3 year oldsLow income families with 0-3 year oldsWide geographic spread, urban and rural Wide geographic spread, urban and rural Target diverse populations Target diverse populations Services: Services: – centre based centre based – home visiting based home visiting based – mixed approachesmixed approachesEvaluation: 17 EHS programs; 3001 Evaluation: 17 EHS programs; 3001 families families Random assignment evaluation (RCT) Random assignment evaluation (RCT)

Early Head Start - ImpactsEarly Head Start - Impacts

overall positive impacts for children, overall positive impacts for children, parents and home environments across parents and home environments across approachesapproaches. . – parents more emotionally supportive of their parents more emotionally supportive of their

children, less detached, and less likely to spank children, less detached, and less likely to spank their childrentheir children ……

overall impactsoverall impacts:: effect sizes 10% to 20%; effect sizes 10% to 20%;some programs had higher levels of effects some programs had higher levels of effects (20% -50% across multiple outcomes).(20% -50% across multiple outcomes). – uused mixed approaches sed mixed approaches – African American families (more African American families (more

disadvantageddisadvantaged??) ) – families with moderate number of demographic families with moderate number of demographic

risk factorsrisk factors

Early Head Start: Works for Early Head Start: Works for Whom?Whom?

5 demographic risk factors analysed: “being a single 5 demographic risk factors analysed: “being a single parent; receiving public assistance; being neither parent; receiving public assistance; being neither employed nor in school or job training; being a employed nor in school or job training; being a teenage parent; and lacking a high school diplomateenage parent; and lacking a high school diploma……””

““The programs had only a few significant impacts on families The programs had only a few significant impacts on families with fewer than three demographic risks, and with fewer than three demographic risks, and the impacts on the impacts on families with more than three risks were unfavourablefamilies with more than three risks were unfavourable. . …… Previous research suggests that low-income families who Previous research suggests that low-income families who have experienced high levels of instability, change and risk have experienced high levels of instability, change and risk may be overwhelmed by changes that a new program may be overwhelmed by changes that a new program introduces into their lives, even though the program is introduces into their lives, even though the program is designed to help….” (p9)designed to help….” (p9)

Programs ‘not effective’ for:Programs ‘not effective’ for:Those presenting with depression, withdrawal, low self-esteem, limited Those presenting with depression, withdrawal, low self-esteem, limited parenting skills and unrealistic expectations of their children. The parenting skills and unrealistic expectations of their children. The additional presence of family violence or chemical dependency can additional presence of family violence or chemical dependency can lead to a deterioration in these families lead to a deterioration in these families (Boston program by Ayoub et (Boston program by Ayoub et al; cited in Browne, 1995).al; cited in Browne, 1995).

Poor families characterized by a high, rather than low or moderate, Poor families characterized by a high, rather than low or moderate, number of risk factors (i.e. low maternal education, unemployed head number of risk factors (i.e. low maternal education, unemployed head of household, single marital status, teenage mother, high levels of of household, single marital status, teenage mother, high levels of depressive symptoms, and low social support) depressive symptoms, and low social support) (Infant Health (Infant Health Development Program; cited in Brooks-Gunn et al, 2000).Development Program; cited in Brooks-Gunn et al, 2000).

Mothers who have experienced much grief, trauma, depression and Mothers who have experienced much grief, trauma, depression and abuse abuse (Hawaii Healthy Start;(Hawaii Healthy Start; cited in Knitzer, 2000).cited in Knitzer, 2000).

Women with few friends, little support and many problemsWomen with few friends, little support and many problems (Keys to Caregiving Program;(Keys to Caregiving Program; cited in Knitzer, 2000).cited in Knitzer, 2000).

Families in poverty experiencing significant personal stress Families in poverty experiencing significant personal stress (Farran, (Farran, 2000).2000).

Reducing child abuse in families with significant levels of domestic Reducing child abuse in families with significant levels of domestic violence violence (Nurse Home Visiting Program, Eckenrode et al, 2000)(Nurse Home Visiting Program, Eckenrode et al, 2000)

Relatively more disadvantaged families within disadvantaged areas Relatively more disadvantaged families within disadvantaged areas (British Sure Start evaluation, 2005)(British Sure Start evaluation, 2005)

Other social interventions…Other social interventions… ExamplesExamples from 1 book offrom 1 book of “ “where evidence of harm was where evidence of harm was

detected following a social intervention”: detected following a social intervention”: social work counselling for ‘problem’ boys (Cambridge-social work counselling for ‘problem’ boys (Cambridge-Somerville study); Somerville study); social work counselling for girls (Girls at Vocational social work counselling for girls (Girls at Vocational High); High); income maintenance (Negative Income Tax studies); income maintenance (Negative Income Tax studies); financial aid for ex-prisoners (Transitional Aid financial aid for ex-prisoners (Transitional Aid Research Project); Research Project); privatizing teaching (Educational Performance privatizing teaching (Educational Performance Contracting research ); Contracting research ); counselling for prisoners in California;counselling for prisoners in California;intensive social and medical services for older people intensive social and medical services for older people in New York. in New York.

Oakley (2000), Experiments in Knowing, p 309Oakley (2000), Experiments in Knowing, p 309

Hearing what we want to hearHearing what we want to hear““In all these instances, the findings of the research In all these instances, the findings of the research surprised those who had launched it, and also the surprised those who had launched it, and also the professional groups involved. The research professional groups involved. The research findings appeared ‘counter-intuitive’, in the sense findings appeared ‘counter-intuitive’, in the sense that people responsible for the various that people responsible for the various interventions believed that they were doing good, interventions believed that they were doing good, not harm. For this reason, they often found it not harm. For this reason, they often found it difficult to take on board the results of the difficult to take on board the results of the experimental evaluation, seeking instead other experimental evaluation, seeking instead other explanations (the context of the research, its explanations (the context of the research, its design, problems with the quality or the design, problems with the quality or the implementation of the intervention) which would implementation of the intervention) which would preserve the possibility that the practice under test preserve the possibility that the practice under test diddid actually work..” actually work..”

Oakley op cit p 309Oakley op cit p 309

EHS Negative ImpactsEHS Negative Impacts

NOT explained by:NOT explained by:– service modelservice model– lower access to serviceslower access to services– teenage parent-hoodteenage parent-hood– maternal depressionmaternal depression– cultural group (Black American / Hispanic)cultural group (Black American / Hispanic)– ‘‘not meeting the needs of the target group’not meeting the needs of the target group’

Policy & Methodological ProblemsPolicy & Methodological Problems

““Effective” strategy doesn’t work, and Effective” strategy doesn’t work, and sometimes is counter-productive, for the sometimes is counter-productive, for the highest risk (primary) target grouphighest risk (primary) target group

No-one knows whyNo-one knows why

Need to evaluate for whom our programs Need to evaluate for whom our programs work / don’t work and find out why.work / don’t work and find out why.

A ‘definition’: Realist EvaluationA ‘definition’: Realist Evaluation

Realistic EvaluationRealistic Evaluation

Ray Pawson and Nick TilleyRay Pawson and Nick Tilley

Sage, 1997Sage, 1997

Not “what works”, but “What works for Not “what works”, but “What works for whom, in what contexts, and how?”whom, in what contexts, and how?”

Social ProgramsSocial Programs

Social programs are real and can have real Social programs are real and can have real effects – both positive (helpful) and negative effects – both positive (helpful) and negative (harmful).(harmful).Programs are an attempt to create change.Programs are an attempt to create change.Programs ‘work’ by changing the choices that Programs ‘work’ by changing the choices that participants (individuals, organisations, participants (individuals, organisations, communities) makecommunities) makeChoice-making is always constrained - by Choice-making is always constrained - by previous experiences, attitudes, beliefs, previous experiences, attitudes, beliefs, resources available, expectations…resources available, expectations…

MechanismsMechanisms

Programs change choices by altering Programs change choices by altering reasoning reasoning ofof or or resources available toresources available to participants participants– Reasoning: (eg) beliefs, attitudes, values, ‘logic in Reasoning: (eg) beliefs, attitudes, values, ‘logic in

use’ use’ – Resources: (eg) information, skill, material resources, Resources: (eg) information, skill, material resources,

social supportsocial support– The interaction between reasoning & resources = The interaction between reasoning & resources =

program mechanismprogram mechanism

Participant response, not just program strategy, Participant response, not just program strategy, determines whether programs ‘work’determines whether programs ‘work’

ContextContext

The contexts in which programs operate The contexts in which programs operate make a difference to the outcomes they make a difference to the outcomes they achieve achieve by determining whether/ which by determining whether/ which mechanisms “fire”. mechanisms “fire”. Program contexts include features such as Program contexts include features such as – organizational contextorganizational context– program participantsprogram participants– program staffingprogram staffing– economic, geographical and historical contexteconomic, geographical and historical context– and so on…and so on…

Mechanism Outcome Domains

Group 1 2 3 4 Group 1 2 3 4Cognitive learning ? Parent as person ?

x

Therapeutic engagement ? x Parent Behaviours ?

x

Peer support x Child ?

x

Beliefs: self x x Prt-Ch-Ch Relationship ?

x

Beliefs: child ? ? xBeliefs: role as a parent ? xBeliefs: others’ judgements x x x

Mechanisms & outcomesin one parenting program

Code: = evidence in support; x = evidence against;? = contradictory evidence;blank = no evidence

Realist research programmesRealist research programmes

Single studies are apt to misleadSingle studies are apt to mislead

Repeat single studies tend to produce Repeat single studies tend to produce mixed findingsmixed findings

Series of studies are needed to identify Series of studies are needed to identify and refine CMOCsand refine CMOCs

CMOCs can change cumulatively through CMOCs can change cumulatively through series of studiesseries of studies

Realist synthesisRealist synthesis

A particular process for learning from the A particular process for learning from the literature literature

Builds program theories about ‘what works Builds program theories about ‘what works for whom in what contexts and how’for whom in what contexts and how’

Can use any previous research/reports Can use any previous research/reports that were of “good enough quality” to that were of “good enough quality” to support the conclusions they reachedsupport the conclusions they reached

Constructing CMOC’sConstructing CMOC’sContext Mechanism Outcome

Secure adult attachment

Positive relationships with service providers

Active engagement in programs

High parenting stressAvoidant adult attachment

Reduced use of support networks

Drop out from services

High demographic risk High rate of mental health issuesParent support / education-oriented home visiting service

Increased focus on crisis intervention / adult needsDecreased focus on child development in services Decreased parental supportiveness

Poorer child development outcomes

High demographic risk High rate of mental health issues‘Therapeutic’ (relationship based) model of service

Decreased depression Increased parenting sensitivity

Improved child development outcomes

Who needs what information?Who needs what information?RoleRole FunctionFunction InformationInformation

Politicians Politicians Fund Fund programsprograms

One size does not fit allOne size does not fit all

Suite of programs requiredSuite of programs required

Program types x populationsProgram types x populations

Policy staffPolicy staff Design & Design & administeradminister

Broad CMOC for program Broad CMOC for program types (middle level theory)types (middle level theory)

System System gatekeepersgatekeepers

Referral Referral C & M x local programsC & M x local programs

Assessment instrumentsAssessment instruments

Service Service providersproviders

ProvisionProvision Detailed knowledge & skillDetailed knowledge & skill

Substantive theorySubstantive theory

Implications for advocacyImplications for advocacy

Match the indicator type to the argument Match the indicator type to the argument – rights, needs, effectiveness, cost-effectivenessrights, needs, effectiveness, cost-effectiveness

Know the research Know the research (some of the bureaucrats do)(some of the bureaucrats do)

Don’t rely on single evaluationsDon’t rely on single evaluations

Check your own assumptions: what works for Check your own assumptions: what works for whom? Where and how?whom? Where and how?

Match the level of information or argument to Match the level of information or argument to the audience: ‘who needs to know what?’the audience: ‘who needs to know what?’