Prev Summit Syd09 Evaluation 090623

8/8/2019 Prev Summit Syd09 Evaluation 090623

http://slidepdf.com/reader/full/prev-summit-syd09-evaluation-090623 1/44

2nd Annual Preventative Health Summit, Sydney 22‐23 June 2009

Dr Andrew Churchill Manager Evaluation & Intelligence Development

Health IntelligenceOffice of the Chief Health OfficerPublic Health BranchVictorian Department of Human Services

23 June 2009Andrew Churchill

Evaluating Prevention

Policy

Effectiveness



Disclaimer

The opinions, comments and/or analysis expressed in this document and

presentation are those of the author and do not necessarily represent the viewsof the Victorian Minister for Health, or the Victorian Department of HumanServices, and cannot be taken in any way as expressions of Government policy.



•What evaluation questions are relevant to assessingprevention policy effectiveness

•Use the “Healthiest Country 20201” paper as a policy caseexample

•Prevention initiatives taxonomy

•Outcomes Hierarchy

•Linking the initiatives taxonomy with the outcomes hierarchy to

create a policy evaluation framework

•Uses of a framework

•Questions 1National Preventative Health Taskforce (Commonwealth of Australia 2008)

Australia: the healthiest country by 2020 A discussion paper

Evaluating Prevention Policy Effectiveness



Policy Process

Problem/Need

Drivers for Gov’t

Action Policy Problem

Identification& Description

Policy Analysis

Interventions

Interventions/Policy

Instruments

Design, Development& Deployment

MonitoringAccountability

Monitoring Evaluation

Policy & P lanning Process



Policy Process & Evaluation Questions

Problem /Need

•What are the parameters of the problem ?

•What are the unwanted impacts of the problem ?

•How does the problem cause the unwanted consequences ?

•How does this problems compare with other problems ?

•What are the determinants of the problems ?

•Who has the problem ? Why them ? Where are they located ?

•What is the size of the problem ?

•Is the problem getting worse ?




Interventions - DESIGN

•What are the candidate interventions that work or have promise ?

•What adaptation/matching is needed to align interventions with

need characteristics

•What are the interventions targeting criteria What do interventioneffects modelling predict ?

•What is the effective “dose-response” relationship at a populationlevel to meaningfully improve outcomes ?

•What new interventions innovation is required because there isno precedent for intervention ?




Interventions: Implementation

•What is the existing intervention topography and where are the gaps ?

•Do interventions meet research benchmarks for effectiveness, takinginto account local and cultural conditions ?

•What are the best geographic locations (& administrative location) forintervention delivery?

•What is the most aligned need(demand)/supply map (both geographic& social-demographic)?

•How aligned is resource configuration ? What community assets canbe built upon ?

•What soft infrastructure and other enablers are needed to support effectiveintervention design and delivery to ensure initiative investment outcomes

are realised ?




Monitoring & Accountability

•What kind of monitoring framework is needed to monitor implementedinterventions ?

•What is the policy logic of the problem’s initiative package of interventions ?

•What kinds of performance indicators are needed ?

•Where do you look for data sources for indicators ?

•Is there alignment between indicators and administrative area interventiondelivery responsibilities, so that they are held to account ?

•Is there sufficient small area intelligence to inform local planningand accountability ?

•What are the contextual factors that influence success that are

uncontrolled and which should be acknowledged ?



What is Program Logic ?

Preventative Health Outcomes Hierarchy:

Example Using National Taskforce 2020

Preventative Health Outcomes Hierarchy:

Example Using National Taskforce 2020

• An illustration of what the program

will do and what results it willproduce

• Demonstrates design &

implementation competence

• A series of “if-then” relationshipsthat, if implemented as intended,lead to the desired outcomes

• The foundation of program planningand evaluation



Program Logic : If-then relationships

A logic model comprises a series of ‘if-then’ relationships that express theprogram’s (or policy’s) explanation of

why it produces the results itclaims to … .

IF then IF then IF then IF then



Common logic model used by Center for

Disease Control and others

Inputs Outputs OutcomesActivities

Medium L-termShort

University of Wisconsin-Extension, Program Development and Evaluation



Another variant on Program Logic

Modelling… ..



Program assumptions that should be made

explicit

The beliefs we have about the program, the participants, and howthe program will work. Includes ideas about:

•the problem or existing situation•program operations•expected outcomes and benefits•the participants and how they learn, behave, their motivations•resources•external environment: influences•the knowledge base

And at the policy level,makes evaluations questions we reviewed earlier, andstatus of the answers, explicit rather than assumed……



Policy Assumptions

– that are best made explicit

Problem/Need

Drivers for Gov’tAction

Policy Problem

Identification& Description

Policy Analysis

Interventions

Interventions/PolicyInstruments

Design, Development& Deployment

MonitoringAccountability

Monitoring

Evaluation

Policy & P lanning Process

The revelation of the assumptions through a program logic analysiscan be linked back to the relevant evaluation questions and creates a

context ready for testing the assumptions/answering the evaluationquestions…………..



National P reventative Health Taskforce: Priority Interventions… ..

What do we need to do ?

Obesity• Reshape industry supply and consumer• Protect children and others from inappropriate marketing• Improve public education and information

• school, community and workplace programs• Reshape urban environments• Support primary health community education and advice• Management of overweight and obesity• Targeted approaches to disadvantaged groups• Evidence base, monitor and evaluateTobacco

• Increases in excise and customs duty & evasion reduction• Further regulate the tobacco industry• Increase education campaigns• Give smokers effective encouragement and support to quit• Access to information, treatment and services for highly disadvantaged groups• Increase understanding of change processesAlcohol• Managing both physical availability (access) and economic availability (price)

• Social marketing and public education• Curbing advertising and sponsorship• Changes to the current taxation of low-alcohol products• Improved enforcement of current legislative and regulatory measures• Removal of tax deductibility for advertising• Support brief interventions

National Preventative Health Taskforce (Commonwealth of Australia 2008)Australia: the healthiest country by 2020 A discussion paper



Intervention initiatives taxonomy

Various dimensions to choose from:…

•Topic area, e.g. obesity, tobacco, alcohol, etc

•Population inclusive, e.g. universal, targeted, etc

•Prevention continuum location, e.g. primary, secondary, early, etc

•Administrative area, e.g. Health Region, Municipal, etc

•Setting, e.g. Community, school, workplace, etc.

•Target group, e.g. age, disadvantage, at-risk, etc

•Intervention type, e.g. regulatory, educative, environments, etc



Another EG of Intervention taxonomy (USA –Tobacco

Control)

Communityprograms

Chronic diseaseprograms

School programs

Enforcement

Evaluation andSurveillance

Counter-marketing

Administration &management

Statewideprograms

Cessationprograms




Intervention Taxonomy Example



Common logic model…………………

Inputs Outputs OutcomesActivities

Medium L-termShort




Multi-component, embedded hierarchical programlogics, – a way to describe and link different

activities within a comprehensive initiative

From: University of Wisconsin-Extension, Program Development and Evaluation

Shared topic/outcomes/ultimate objectives……



Multiple Programs & Their Program Logics

Some of these

outcomeswill be shared,while others aredistinct.

Next, an“OutcomesHierarchy”approach toorganising acomplex array ofoutcomes

into a coherent and

intelligiblehierarchy orframeworkIs described.



Outcome Hierarchy Policy Foundation:

Example: Health System Performance Framework

(Adapted from the Canadian Institute for Health Information. This framework forms the basis of theAustralian National Health Performance Framework. The degree of inequality in society is a characteristicof all indicators at each tier of the framework).



National Taskforce: Proposed Tier 1 Indicators

1National Preventative Health Taskforce (Commonwealth of Australia 2008)Australia: the healthiest country by 2020 A discussion paper



National Taskforce: Proposed Tier 3 Indicators (con’t)

1National Preventative Health Taskforce (Commonwealth of Australia 2008)

Australia: the healthiest country by 2020 A discussion paper



Health System Performance Framework

(Adapted from the Canadian Institute for Health Information. This framework forms the basis of theAustralian National Health Performance Framework. The degree of inequality in society is a characteristicof all indicators at each tier of the framework).

Domains



Conceptually rotate…….

Time & Direction of Influence

Tiers 2 and 3 represent

those things that havean influence onhealth outcomes (Tier 1)

To reveal hierarchy of influence:



Relationship between health system tiers & the

outcomes hierarchy component of policy logic…..

Time & Direction of Influence

l h h



Preventative Health Outcomes Hierarchy:Example Using Summarised National Taskforce

2020 Report Outcomes



How results arise from responses to issues

Programs Taxonomy & Logics Outcomes HierarchyShort Term Medium Term Long Term

Response to issue/s Results



Preventative Health Policy Logic Model

Programs Taxonomy & Logics



If-then relationships

Underlying a logic model is a series of ‘if-then’ relationships that express theprogram’s (or policy’s) theory of whyit produces the results it claims to… .

IF then IF then IF then IF then



Preventative Health Policy Logic Model


Evaluation Framework



Each Intervention Category will be madeup of initiatives and sub-programs

Policy authors

Program implementers



Policy Design & Evaluation Evidence


E

E

E

E

E

EE

E

E

E

E

E

E

E

E

E

E = evidence of influencing relationship



Policy Logic – KPI Classification


KPI

KPI

KPI

KPI

KPI

KPI

KPI

KPI

KPIKPI

KPI

KPI

KPI

KPI

KPI

KPI

KPI

KPI

KPI

KPI

KPI

KPIKPI

KPI

KPI

KPI

KPI

KPI

KPI

KPI

KPI

Monitoring Framework



Policy Logic & Health Intelligence

Population Health Surveillance•Population Health Surveys•Population Health Statistics•Programs data & Hospital data

Evidence-Informed

Decision-Making

Research & Evaluation

Measured inequalities



Evaluating planning processes

Programs Taxonomy & Logics Outcomes HierarchyShort Term Medium Term Long Term

MissionManagementObjectives

AnnualGoals

IntermediateObjectives

StrategicObjectives

Business planning Strategic planning



Benefits of a Policy Logic Modelling Approach

• Provides a structure to guide intervention development, evaluation & monitoring

• Provides a mechanism to profile health inequalities at multiple levels

• Support performance benchmarking at multiple system levels

• Provides a robust context for assessing impact of funding and value for moneyof resource investments

• Supports accountability by providing an explanatory context that makes sense

of multiple and complex system of potential indicators

• Informs an organised basis for agreement across initiatives on the potentialimprovements that their individual and joint efforts are intended to achieve.



Questions

[email protected]



Further reading………….

Centers for Disease Control and Prevention(US) evaluation site:

http://www.cdc.gov/eval/resources.htm

University of Wisconsin-Extension Program Development and Evaluation site:

http://www.uwex.edu/ces/pdande/http://www.uwex.edu/ces/lmcourse/

Logic Model Development Guide (2004). W.K Kellogg Foundation(http://www.wkkf.org)

Funnell S (1997), Program Logic: An Adaptable Tool for Designing and EvaluatingPrograms. Evaluation News and Comment, 6(1): 5-17

Chen H-T, Rossi PH (1992), Using theory to improve program and policyevaluation. Westport, CT: Greenwood Press

http://www.dpcd.vic.gov.au/evaluation

Prev Summit Syd09 Evaluation 090623

Documents

Transcript of Prev Summit Syd09 Evaluation 090623