Matt Egan.ppt [Compatibility M

7/31/2019 Matt Egan.ppt [Compatibility M

1/48

MRC/CSO Social and Public Health Sciences Unit

evaluating the health effects of organisational interventions.

M EganMRC/CSO Social and Public Health Sciences Unit24 th May 2012


2/48

Why do evaluators care about implementation?

So we can learn how to improve implementation

To help us assess the extent to which findings aregeneralisable and transferable (relationship betweencontext, implementation and outcomes).

MRC|CSO Social and Public Health Sciences Unit

So we can distinguish between theory failure andimplementation failure.

A consequence of our growing interest in complexinterventions, natural experiments and theory ledevaluation.


3/48

Theory based approaches



4/48

The Cochrane logo



5/48

Clinical trial phases (MRC guidance, 2000)



6/48

Key elements of the development andevaluation process (MRC guidance, 2008)

Feasibility/piloting1 Testing procedures2 Estimating recruitment /retention3 Determining sample size


Development1 identifying evidence base2 identifying /developing theory3 Modelling process and outcome

Evaluation1 Assessing effectiveness2 Understanding change process3 Assessing cost effectiveness

Implementation1 dissemination2 surveillance3 long term follow up


7/48

Appropriate study designsResearch question Qualitative

ResearchSurveys Case control

studiesCohortStudies

RCTs

EffectivenessDoes this work?

+ ++

Effectiveness of service delivery:How does it work ?

++ +


SalienceDoes it matter ? ++ ++

AcceptabilityWill children/parentswant to use it?

++ + +

AppropriatenessIs this the right servicefor these children?

++ ++

(Muir Gray, 1997)


8/48

Evaluating social interventions:a false alternative

Black box evaluationsof effectiveness Experimental designs Research hierarchy

Theory based evaluation Suspicious of

experimental designs(espec. RCTs)


,

process, theory of change, or differentialeffects

Focused on attributionof intervention to effect

intervention into a seriesof hypothesised causallinks.

What works, for whomand in what context?


9/48

Evaluating social interventions:a false alternative

Black box evaluationsof effectiveness Experimental designs Research hierarchy

Theory based evaluation Suspicious of

experimental designs(espec. RCTs)


,

process, theory of change, or differentialeffects

Focused on attributionof intervention to effect

intervention into a seriesof hypothesised causallinks.

What works, for whomand in what context?

Why cant you do both?


10/48

We should do both (sometimes)

Evaluators are often interested in1. What works? (i.e. effects and effectiveness)2. How does it work?/Why does it now work?3. For whom?4. In what context?


A mixed methods approach needed to answer all 4 questions

Quasi-experimental designs address What Works? (sub-(groupanalysis can sometimes help address the third and fourth questiontoo).

Often, the how, for whom and what context questions canonly be properly interpreted in the light of whether or not aparticular intervention works.


11/48

Gatehouse study

MRC|CSO Social and Public Health Sciences Unit Note: I did not work on the Gatehouse Study (M Egan).


12/48

Aim & rationale of the Gatehouse Project

School-based universal intervention

Focus on improving the school environment(rather than simply teaching health promotionto children)


Aim to promote emotional well-being & reducehealth risk behaviours related to this, byincreasing connectedness in young people

Each school given resources and expertise todevelop their own unique approach.


13/48


14/48

Implementation factorsImplementation factors Measures/observationsMeasures/observations

Understanding of intervention

Broader political &socialenvironment

Facilitator journals

Key informant interviews

School engagement


Education regioncharacteristics

Schoolcharacteristics School background audit

questionnaires

Minutes of meetings


15/48

12schools*

Year 8 students

Intervention

Cluster randomised controlled trial

1,62082%

response


s r c s

32 schools

14schools*

* 4 schools closed or threatened with closure, 2 did not agree to participate

Control

1,874

71%response


16/48

Implementation appears to be broadly

successful in terms of schools responseto the Project

Shift in opinion from burden & irrelevant to

ownership & utility strong sense of partnership in research respected, valued, worked with schools


pr v r w r r g opportunity to take risks a lens to review activities & practices

empowered, built capacity

actively seek new research opportunities to continuework


17/48

The Gatehouse Project: changes in health riskbehaviour in year 8 students after 2 years

80

100

120

p


0

20

40

60

Smoking Regularsmoking

Bingedrinking

Cannabis WeeklyCannabis

% o

f g r o u

Comparison schoolsIntervention

All analysesadjusted for

previous level of substance use in

the school


18/48

But...No change in explanatory mechanisms

- school connectedness did not change

Implementation


Explanatory mechanism

Outcome


19/48



20/48



21/48

Why am I here?



22/48

Based on the following systematic reviews Egan M, Bambra C, Thomas S, Petticrew M, Whitehead M, Thomson H.

The psychosocial and health effects of workplace reorganisation.

1. A systematic review of organisational-level interventions thataim to increase employee control . Journal of Epidemiology andCommunity Health (JECH) 2007; 61:945-54.


reorganisation. 2. A systematic review of task restructuringinterventions. JECH 2007; 61:1028-37

Bambra, C. et al. Shifting schedules - The health effects of reorganizing shift work . American Journal of Preventive Medicine,

2009

Bambra, C et al . "A hard day's night?" The effects of CompressedWorking Week interventions on the health and work-life balanceof shift workers: a systematic review . JECH, 2008; 62(9):764-77


23/48

Interventions



24/48

Workplace reorganisation reviews:notable features

Organisation change rather than individual-level organisational-level interventions intended to increase

employees opportunities to make decisions or participatein decision making.

Broad review question and inclusion criteria A ran e of health outcomes linked to stress h sical

psychological, sick days) Many study designs (experimental, prospective,

retrospective studies: best available evidence)

Mediating mechanism

Psychosocial outcome (e.g. demand, control, support)

Implementation appraisal

Health inequalities

Narrative synthesis not meta-analysis


25/48

The Cochrane logo



26/48

Psychosocialimprovement

Intervention

So what where we looking for?

Psychosocialdeterioration

Healthdeterioration

Healthimprovement

Di f f er en


Reliable/plausible data on context, implementation to help future interventionsand help us distinguish between implementation failure and theory failure .

Control

Psychosocialimprovement

Psychosocialdeterioration

Healthimprovement

Healthdeterioration

i al ef f e

c t s


27/48

Psychosocial

and healthoutcomesIntervention

What did we find?

Hardly


Hardly anything on context or implementation

(ill)matchedcontrol

differential

effects /health

inequalities

Psychosocialand healthoutcomes

and sometimes...


28/48

Nonetheless Our findings did generally fit the demand, control support model in

that

Evidence of improving psychosocial environment tended to co-occurwith health improvements. Evidence of deteriorating psychosocial environments tended to co-

occur with worsening health.


This applies particularly to changes in employee control.


29/48

Review methods: search and selection forpsychosocial reviews

65282search hits


retrieved

18Selected forparticipation

21Selected for task

restructuring

Note: similar process for shiftwork reviews


30/48

Review methods: critical appraisal checklist1. prospective2. representative sample3. appropriate comparison group4. baseline response 60%5. follow-up 80% in cohort, 60% in cross-section


6. adjustment for non-response and drop-out;7. conclusions substantiated by data8. adjustment for confounders9. no contamination

10. appropriate statistical tests.

Also emphasised controlled and prospective studies.


31/48

Involves criteria that are eitherarbitrary or vague

Implies that sources of bias areof equal important and thathigher scores mean less biased

Problems with critical appraisal


studies.

What are we appraising? Studymethods or reporting of studymethods?

How useful is evidencehierarchy?

See Cochrane Handbook


32/48

Review methods: implementation checklist



33/48

Review methods: implementation checklist



34/48

Involves criteria that are either arbitrary or vague

Implies that sources of bias are of equal importantand that higher scores mean less biased studies.

Problems with implementation checklist


of study methods?

What distinguishes good and bad implementation noestablished hierarchy

In other words, similar problems to those that relate tomethodological checklists.


35/48

Specific problems with our checklist Terminology : we use implementation to cover context,

planning, delivery, programme theory and differential

effects.

Ignorance : we didnt know what a well implemented


.

Uninformative : the checklist rewards studies thatmention certain implementation issues, but does notdistinguish between (i) good and bad implementation;

nor (ii) robust or non-robust evaluation of implementation

There wasnt much to appraise.


36/48

There wasnt much to appraise.


More than half studies reported on Motivation for intervention Employee support for intervention and thats it.


37/48

Problem not just about quantity of reporting

also quality General reliance on anecdotal evidence rather than a

planned study of implementation.

Sometimes appeared to privilege one view point e.g.either managers or employees.


Few studies discussed how implementation influencedoutcomes.

Quality of implementation reporting did not vary muchby type or date of publication , or methodologicalquality , or health outcome.


38/48

Whats the point of all that? Good reporting of implementation is a pre-condition to

being able to appraise whether the implementation was

good or bad.

We have confirmed that this pre-condition is generally


the implementation was good or bad.


39/48

Whats the point of all that? Good reporting of implementation is a pre-condition to

being able to appraise whether the implementation was

good or bad.

We have confirmed that this pre-condition is generally


the implementation was good or bad.

Is that right?


40/48

Qualitative version of the checklist (no score)



41/48

This more qualitative approach producedmore useful findings

Context: Participation interventions dont work indownsizing organisations

Mechanism(?): Task structure interventions motivated


y us n ss r as ns c ncy, pr , pr uc v y

were more likely to have negative health impacts thaninterventions motivated by stress reduction/health goals.

Differential effects: Some evidence that the positive

and negative health effects of organisational changes tothe psychosocial work environment are felt more by menand lower socio-economic groups.


42/48

Workplace psychosocial reviews(demand, control, support model)

Increase participation in the workplace 18 studies

Most robust = prospective with non-random control Some participation interventions may benefit employee

physical and (especially) psychological health as predictedb the demand-control-su ort model but ma not rotect


employees from generally poor working conditions.

Task restructuring interventions 19 studies

Most robust = prospective with non-random control Task-restructuring interventions that increase demand or

decrease control adversely affect the health of employees.Adverse effects are less likely if health goals are a planned

into the intervention


43/48

Changing shifts reviews Changing shifts

26 studies Most robust = prospective with non-random control Beneficial effects on health and worklife balance: (1)

switching from slow to fast rotation, (2) changing from


ac war rwar r a n, an s -sc u ng

shifts Compressed working week

40 studies Most robust = prospective with non-random control Little effect on self-reported health. Work-life balance

was generally improved


44/48

Key messages from review Workplaces can be made more or less healthy. Findings often supportive of psychosocial models

explaining workplace health especially control. Negative outcomes more likely in workplaces

undergoing downsizing and/or some forms of


.

Little evidence on differential effects although somesuggestion that organisational workplace change canpotentially widen or narrow health inequalities.

Barriers to robustly evaluating these kinds of workplace

interventions

These conclusions are, we believe, useful (e.g. seeBambra, 2010 and Marmot Review, 2010).


45/48

Summing up I have suggested a mixed methods approach to

evaluating the health impacts of organisational change

including quasi-experimental, qualitative andprogramme theory methods.


possible and replicable, even for complex interventions,although the approach is very challenging.

Systematic reviews can assess outcomes, mechanisms

and implementation although it takes more than justa checklist score.

Using a best available evidence approach we canderive useful findings from flawed studies and reviews.


46/48

But This is definitely a field where there is room for

improvement



47/48

But This is definitely a field where there is room for

improvement.

Im sure the rest of this conference will


emonstrate t e progress t at as a rea ybeen made, and how further improvementscan be made.


48/48

Key ReferencesPublications (Co-)Authored by Presenter Bambra et al. The psychosocial and health effects of workplace reorganisation. 2. A systematic review of

task restructuring interventions. JECH 2007; 61:1028-37 Egan M, Bambra C, Petticrew M, Whitehead M. Reviewing evidence on complex social

interventions: appraising implementation in systematic reviews of the health effects of organisational-level workplace interventions. Journal of Epidemiology & Community Health 2009; 63 :4-11.

Egan M, Bambra C, Thomas S, Petticrew M, Whitehead M, Thomson H. The psychosocial and health effectsof workplace reorganisation. 1. A systematic review of organisational-level interventions that aim toincrease employee control. JECH 2007; 61:945-54.

Other Publications Bambra, C. et al. Shifting schedules - The health effects of reorganizing shift work. American Journal of

, Bambra, C et al. "A hard day's night?" The effects of Compressed Working Week interventions on the

health and work-life balance of shift workers: a systematic review. JECH, 2008; 62(9):764-77 Bambra C, Gibson M, Sowden A, Wright K, Whitehead M, Petticrew M. Tackling the wider social

determinants of health and health inequalities: evidence from systematic reviews. Journal of Epidemiology& Community Health 2010; 64:284-91 .

Craig P, Dieppe P, Macintyre S, Michie S, Nazareth I, Petticrew M. Developing and evaluating complexinterventions. London: Medical Research Council, 2008.

MRC. A framework for the development and evaluation of RCTs for complex interventions to improvehealth. London: Medical Research Council, 2000:18.

Muir Gray JA. (1997) Evidence-based healthcare: how to make health policy and management decisions.London: Churchill Livingstone.

Patton GC, Bond L, Carlin J, Thomas L, Butler H, Glover S, et al. Promoting social inclusion in schools: acluster randomised trial. American Journal of Public Health. 2006; 96(9): 1582-7.

Thanks co-authors and funders. Thanks to Lyndal Bond for permission to use Gatehouse slides.

Matt Egan.ppt [Compatibility M

Documents

Transcript of Matt Egan.ppt [Compatibility M