Laureate Professor Rob Sanson-FisherUniversity of Newcastle

Mixed Methods in Prevention and Health Services Research:

EVALUATION OF PREVENTION IN PUBLIC HEALTH

Need to ensure interventions are:

Why Evaluate?

Effective

Cost-effective

EquitableNot harmful

Egas Monitz and Walter Freeman1930s – 1950’s

Used to treat a range of mental illnesses

Not based on scientific evidence Resulted in significant harm to

thousands

Lobotomy

Efforts to improve general practice care: the last 30 years…

Staffing changes

• Move from solo to group practices

• Practice managers• Practice nurses

Training and education

• Education about smoking and alcohol in undergraduate medical curriculum

• Vocational registration • Compulsory Continuing

Professional Development

System changes

• Computerization of medical records

Financial incentives

• Practice Incentive Program (PIP)-incentives related to smoking, diabetes etc

• Practice nurse incentives • Payments for preparation of

chronic disease care plans, preventive health assessments

Policies and guidelines

• RACGP Redbook guidelines for preventive care

• Divisions of General Practice and Medicare Locals with a focus on chronic disease prevention and management

• Focus on SNAP risk factors

 Treatment changes

• Better availability of treatment options- NRT and Champix for smoking cessation

Consent rates in parentheses

Alcohol Smoking0

20

40

60

80

100

28

56

40

66

26

63

Accuracy of GP detection

Dickinson et al. 1989

Heywood et al. 1994

Paul et al. 2014 Bryant et al. (UER)

Randomly selected GPs 50 (52%) 230 (72%) 51 (63%)

Randomly selected Patients 2044 (83%) 7161(91%) 1720 (86%)

%At-risk patients

correctly detected by GP

Despite efforts and investment with the aim of improving the health of Aboriginal and Torres Strait Islander people, a notable gap still exists in health outcomes between Indigenous and non-Indigenous Australians

Chronic diseases (e.g. circulatory disease, cancer) are the main contributors to the mortality gap

Need for effective interventions for prevention of chronic diseases for this vulnerable group

Outcomes for Indigenous Australians

AIHW, 2014

Life Expectancy

Males Females Males Females0

102030405060708090

Indigenous Non-Indigenous

2005-2007 2010-2012

Age

AIHW, 2014

Mortality

2001 2002 2003 2004 2005 2006 2007 2008 2009 2010 2011 20120

200

400

600

800

1000

1200

IndigenousNon-Indigenous

Age standardised mortality

rates(per 100,000)

AIHW, 2014

Research designs must be able to answer:

Three Fundamental Questions

1

Has a change occurred?

2Did the change

occur as a result of the intervention?

3Is the degree of

change significant and important to

stakeholders?

Theories of Human Behaviour

Historically, the field was dominated by theories with an individual and interpersonal focus

Freud (psycho-analytic theory) Rogers (humanistic theory) Jung (analytical psychology)

Later models of Human behaviour have a social focus Social Learning Theory Health Belief Model Theory of Planned Behaviour

Changes in Theories of Human Behaviour Recognise the influence of health systems and social

factors on health behaviour

Shift towards multi-faceted approaches that address factors at the individual, social, and system levels

PRECEDE-PROCEED model (L Green) Behaviour Change Wheel (S Michie)

Need for system-level interventions

Health behaviour is complex

Whole-system, multilayered interventions are needed in order to change behaviour

Need a research design with the complexity and flexibility to evaluate system-level behaviour change interventions

Bias is anything that introduces systematic variation in the data, resulting in deviation from ‘truth’

Sources of bias include: Selection bias Performance bias Detection bias Attrition bias Reporting bias Contamination

Need to minimise bias

Randomised Controlled Trial (RCT)

Considered the ‘gold standard’ for evaluating intervention effectiveness

Individuals randomly intervention or control conditions

Randomisation eliminates selection bias and variation between conditions

Baseline Outcome Measurement

Follow-up Outcome Measurement

Intervention Condition

Control Condition

Randomisation

Assessed for Eligibility

Participant Consent

Advantages and Disadvantages

Advantages

Methodological rigour

Well-suited to interventions where individuals are the unit of analysis

Disadvantages

Availability of sufficient populations

Discouragement of innovative research questions

Contamination

Time available for follow-up

Threats to external validity

Ethics of randomisation

Blackbox phenomena

Randomised controlled trial

Aim: To determine the effectiveness of an intensive quit-smoking intervention on smoking rates among pregnant Aboriginal and Torres Strait Islander women (n=263)

No significant differences found between groups

Reducing smoking among pregnant Aboriginal and Torres Strait Islander Women

Eades et al. 2012

Baseline Outcome MeasurementSocial and demographic status, smoking history

Follow-up Outcome MeasurementSelf-reported smoking status (validated with urine cotinine measurement)

Between 36 weeks gestation and delivery

InterventionTailored quit smoking advice from

GP Evidence-based communication skills

Engaging significant others NRT after two failed quit attempts

Control

Usual Care

Randomisation by week

Assessed for EligibilityATSI women attending first antenatal appointment, <20 weeks gestation, current

smokers/recent quitters

Participant Consent

Key challenges: More recent quitters in the intervention group than control High rate of loss to follow-up (>30%) Lack of allocation concealment Contamination between groups Poor adherence to study protocol – high staff turnover

Reducing smoking among pregnant Aboriginal and Torres Strait Islander Women

Cluster RCT

Type of RCT Groups/settings randomly

intervention or control conditions Baseline Outcome Measurement

Groups matched

Assessed for Eligibility

Groups consent

Follow-up Outcome Measurement

Intervention Condition

Control Condition

Randomisation

Advantages and Disadvantages

Advantages

Reduces risk of contamination

Well-suited to multi-level population interventions

Methodological rigour

Disadvantages

Individuals within groups likely to be correlated

Difficulty recruiting adequate number of groups and potential imbalance

Very costly

Ethics of randomisation and non-consent

Requires long term commitment from stakeholders

Alcohol Action in Rural Communities

Matched-pair cluster RCT Aim: To evaluate the effectiveness of a community action

strategy aimed at reducing alcohol misuse and alcohol-related harm

Randomization of 20 pair-matched rural communities

Thirteen individual interventions were implemented systematically over five years

Harm reduction strategies Demand reduction strategies

Shakeshaft et al. 2013

Intervention communities showed: 20% reduction in average alcohol consumption 42% reduction in experience of alcohol fuelled verbal abuse 33% reduction in alcohol-related street offences 30% reduction in the number of people drinking at high-risk

levels

Key challenges: High cost High level of expertise required Long-term commitment required from researchers and

community partners

Alcohol Action in Rural Communities

Alternative Research Designs

Pragmatic research designs can be more useful in evaluating public health interventions

When selecting research designs, need to consider trade-offs between:

Internal validity External validity Cost effectiveness Acceptability Feasibility

Stepped Wedge

Intervention implemented sequentially in all groups

Order in which groups receive the intervention is randomized

May be implemented in multiple sites at same time

Outcomes are measured pre-intervention and post-intervention (prior to implementation at the next site) Sanson-Fisher et al. 2014

Advantages and Disadvantages

Advantages

All groups receive intervention – no ethical issues associated with withholding intervention

Flexibility in timing of intervention

Disadvantages

Increased number of measurements compared to cluster RCT

Complex statistical analysis

Does it establish causality

Interrupted Time Series

Intervention delivered to a single group/site

Multiple observations pre- and post-intervention

Group acts as own control Change in level = Change in

outcome post-intervention Change in slope = A different

trend in outcome post-intervention Sanson-Fisher et al. 2014

Advantages and Disadvantages

Advantages

Useful where outcome is obtained from routinely collected data

Multiple groups are not required

Disadvantages

Need to determine the number ofobservations required and duration of time period of measurement

Cannot be certain that change is due to the intervention and not other factors

Increased number of outcome measurements required

Multiple Baseline

Intervention implemented in all sites in a phased approach

Time of intervention implementation is randomised

Multiple observations pre- and post-intervention

Sanson-Fisher et al. 2014

Advantages and Disadvantages

Advantages

All groups receive intervention – no ethical issues associated with withholding intervention

Flexibility in timing of intervention

Disadvantages

Conceptually not usable for all outcome measures

Repeated measures required

Matched-pair multiple baseline design Aim: To reduce the burden of chronic disease amongst NSW

Aboriginal people by partnering with ACCHSs to develop, implement and evaluate improvement strategies

Chronic Care Service Enhancements Program Evaluation

1. Improving chronic disease management

Developing diabetes register and follow-up protocolTraining in diabetes careImplementation across 6 matched ACCHSs

2. Improving preventative health screeningTraining in screening protocolsRoutine health assessmentsEstablishing a reminder system Implementation across 5 matched ACCHSs

3. Increasing reach of ACCHSs

ACCHS open day (presentations, health stalls, BBQ)Tailored re-engagement strategies and care plansImplementation across 8 matched ACCHSs

4. Improving data quality

Data quality audit and trainingImplementation across 6 matched ACCHSs

Westfall et al. 2007

“Blue Highways” on the NIH Roadmap

Research designs must be able to answer:1. Has change occurred?2. Did the change occur as a result of the intervention?3. Is the degree of change significant and important to

stakeholders?

Need a research design with the complexity and flexibility to evaluate system-level behaviour change interventions

Individual RCT Cluster RCT Stepped Wedge Interrupted Time Series Multiple Baseline Design

Summary