Workshop 1 – Introduction16th February 2010

Introducing the systematic review

Workshop Overview Workshop will discuss:The methods of systematic reviews and the

rationale behind their use.How to systematically search for relevant

literature.The importance of developing a protocol.Inclusion and exclusion criteria and how to

filter findingsData extraction and quality assessmentSynthesising the findings.

What is a systematic review and what is its use?A systematic review provides information on the

effectiveness of interventions by identifying, appraising, and summarising the results of otherwise unmanageable quantities of primary research.

It uses a replicable, scientific and transparent approach which seeks to minimise bias.

Primarily it is used to collate data to produce a summary of the available findings of a given topic.

The approach is needed to inform policy and decision making and is useful in eliminating uncertainty regarding various interventions.

Rationale for undertaking SRWhenever a systematic review is being

considered, efforts should be made to ensure that a good quality review in the field of interest does not already exist.

If the available reviews are outdated or of poor quality, it may then become necessary to update existing reviews or conduct a new one.

In order to avoid duplicating research, it should also be established that a relevant review has not already been commissioned or is in progress.

Why review this type of research?Useful for application of knowledgePolicy-makers and practitioners may want to know:

- What people need- Whether interventions are acceptable and appropriate- Whether an intervention is effective- Why people do the things they do

Useful to generate knowledge and take stock of what is already known

Achieve an understanding beyond that gained in any individual study.

How to begin• Before beginning a systematic review it is important to ensure that

the information regarding the subject is required, is there a need for this research? Searching the relevant databases, hand searching key journals and reading across the topic is essential before beginning to collate data.

• Some key sources for published and ongoing reviews emerge from the Cochrane Library:

• The Cochrane Database of Systematic Reviews (CDSR)• Database of Abstracts of Reviews of effectiveness (DARE)• Health Technology Assessment Database (HTA)• Other useful sources to check are:• HMIC (part of the Kings Fund Library)• MEDLINE• CINAHL• Science and Social Science Citation Index• PsychInfo• Searching the databases and internet resources should also be

supplemented (if possible) with key contacts in the field who may be helpful in knowing whether any similar reviews are taking place.

Searching for data• The aim of the search is to generate as comprehensive a list as

possible of primary studies, both published and unpublished, which may be suitable for answering the questions posed in the review.

• A thorough, replicable and unbiased search strategy is crucial.• Constructing an effective combination of search terms (key words)

for searching electronic databases requires a structured approach.

• It is often helpful to produce a table of terms which can be broken up into population, interventions, outcomes and study design.

• All searches should be documented as they develop and saved electronically.

• The searches will gradually be filtered so the best available data on the chosen subject is extracted for inclusion in the review, once again this process must be documented.

• Not all papers will be relevant for inclusion, in fact most will not be, however all the information retrieved must be saved.

Developing a protocol• The first milestone of any

review is to develop a protocol.• The protocol specifies the plan

which the review will follow to identify, appraise and collate evidence

• The protocol should state precisely the main and secondary questions which will be addressed and note the inclusion and exclusion criteria of any potential papers.

• Through undertaking some preliminary reading it is possible from the protocol to develop a rough hypothesis of the findings which will help to refine the research question further.

Components of a protocol• Background• Review questions• Search strategy including

search terms and resources to be searched

• Study selection criteria and procedures

• Study quality assessment checklists and procedures

• Data extraction strategy• Synthesis of the extracted

evidence• Project timetable

Inclusion and exclusion criteriaCriteria Inclusio

n zoneExclusion zone

Population Studies that relate to mental health users

Other studies

Geographical coverage

UK only Outside of the UK

Timing of studies

1999-2010 Any work prior to 1999

Condition of interest

Studies that look at the effectiveness of alternative therapies for the treatment of mental health users.

Studies which do not look at alternative therapies.

Study design

Empirical data

Numerical or conceptual

• Both inclusion and exclusion criteria should follow logically from the review questions and they should be defined in terms of the population, the interventions, the outcomes, and the study designs of interest. Only studies that meet all of the inclusion criteria and none of the exclusion criteria should be included in the review. The review should be piloted to check that they can be reliably interpreted and that they classify the studies appropriately.

Data extraction (refer to handout)• To ensure validity the works filtered from the initial

search which met the inclusion criteria are subsequently assessed by a minimum of two separate researchers.

• A data extraction assessment form is required for this process.

• The document will cover:o Study quality - the degree to which a study employs to

minimise bias, a set of parameters in the design and conduct of a study that reflects the validity of the outcome.

o Bias o Internal validity – the degree to which the results of a study

are likely to approximate to the ‘truth’.o External validity – the extent to which the effects observed

in a study are applicable outside of the study.

What is synthesis?‘..the product of activity where some set of parts is combined or integrated into a whole.....[synthesis] involved some degree of conceptual innovation, or

employment of concepts not found in the characterisation of the parts and a means of creating

the whole’

Strike and Posner (1983)

A synthesis is not just a report of the findings of the individual studies in the review, it is a transformation of the data from the primary studies and it is the basis of the systematic review method.

Common elements of different methods of synthesisALL syntheses involved interpretation of

concepts and dataALL syntheses involve the comparison

of settings, contexts and findings of studies

ALL syntheses involve decisions to be made regarding the relevance and trustworthiness of studies to be included

Extracting data for synthesisWhat counts as the

findings of a study? - Concepts- Themes- Theories- Quotes- Numbers- Author(s)

interpretations

Where should you look?

- Abstract - Results - Discussion - Conclusion

Practical 1In groups or pairs look through the synthesis

exercise on homework evidence.Using the information try to synthesise the

studies transforming the data into a collective summary.

Remember that description is not transformation.

What to look for (Practical 1)Key concepts of data synthesisDescriptive data of the evidenceThe variability of difference (heterogeneity)

between the studies in terms of key characteristics .

The degree to which the studies are similar (homogeneity) e.g. If the results vary no more than that expected by the play of chance.

The influence of bias and the idea that some studies which go against the norm may not be published.

Stages of synthesis

Part 2 overviewWill look at:The intricacies of different methods of

synthesis.Short-hand approaches to urgent systematic

information using REAGo on to look at meta-ethnography as a review

methodLook at the idea of first, second and third

order concepts.Interpreting data

Rapid Evidence AssessmentRapid Evidence Assessment (also known as Rapid

Appraisal).Applies the principles of the systematic review in

a truncated form.Uses a very narrowly focused question,

concentrates on only specific type of evidence using limited databases and gains less information from each study.

Does not involved such a detailed quality assessment.

A trade off between quality and speed.

Examples of qualitative synthesis methods

Genre Examples Question....?Textual: empirical Realist synthesis In what situation

does it work?Narrative numerical What works?Meta-ethnography What does this mean?

E.g. Interpretative.Textual: conceptual Critical interpretive

synthesis (not raw data but the researchers interpretation of meaning)

How have investigators understood what this means? (how can social phenomena be understood).

Example of first, second and third order constructs

First order concepts

Second order concepts

Third order concepts / hypotheses

-Explanations for low levels of participation-Positional power-Power from experience-Standing/education/class-Reluctance to criticise-Expertise-Disagreements among experts-Impact of illness

- Power, status and deference

- Trust (reduces anxiety)-Dependency-Capacity of assertiveness undermined- Physical dependency

-Patients with higher educational levels and social standing may be slightly more likely to insist on participation in discharge planning- Past experience in hospital may enhance confidence to participate- Trust is undermined if experts disagree- Anxiety increases when access to medical expertise is reduced-Expertise legitimates power

Translating studies into one another

Methods and concepts

Donovan and Blake

Morgan Britten

Sample 54 patients with suspected inflammatory arthropathy

60 white and Afro-Caribbean patients treated for hypertension for at least one year

30 patients, attendees and non-attendees

Data collection Home interviews pre and post consultation: observation of consultations

Home interviews Home interviews

Setting Three rheumatology units

15 general practices Two general practices

Explanation theory (second order interpretation)

“Patients carry out a ‘cost-benefit’ analysis of each treatment, weighting up the costs/risks of each treatment against the benefits as they perceive them”

Medicine taking is influenced by cultural meanings and cultural resources

Patients may not articulate views that they do not perceive to be medically legitimated

Summarising synthesis

Syntheses of people’s perspectives and experiences.

Epistemological reasons:Crucial part of our knowledge and about

the worldHelp us consider health or social issues in

the specific context of peoples lives• Ethical reason:Important to listen to those who are the

‘targets’ of professional intervention.

Theories of change and effectiveness reviews

Introducing Theories of Change (ToC)

A theories of change model can often be useful when looking at studies concerning effectiveness. It can inform both the review question and the inclusion criteria alongside aiding the interpretation of the reviews findings.

Built around the pathway of change, a Theory of Change describes the types of interventions (a single program or a comprehensive community initiative) that bring about the outcomes depicted in the pathway of a change map. Each outcome in the pathway of change is tied to an intervention, revealing the often complex web of activity that is required to bring about change.

A Theory of Change is a specific and measurable description of a social change initiative that forms the basis for strategic planning, on-going decision-making and evaluation.

Theories of change can change!Probable intended theory

Theory of change after systematic review

Practical 2In pairs:

Read through the summaries of effectiveness studies

Develop a ‘theory of change’ that encompasses the ways in which the interventions are thought to impact on children’s healthy eating

Combine with another pair and prepare to present your findings to the rest of the group.

Summary of workshop 1Summary of workshop 1

What kind of research is included?Interviews (different degrees of structure)Questionnaire-based surveysFocus groupsCould be nested within an RCT

Numerical or statistical analysisThematic analysis

Some examples of review topicsOlder people’s views of hospital dischargeYoung people’s views on what impacts on their

motivation to learn in the classroomLay experiences of diabetes and diabetes careExperiences of being a teenage mother in the

UKExperiences of patients with coronary heart

diseaseExamining the effectiveness of new

interventions

Workshop 2 (2nd March)Will look at: the different ways of presenting an

interventions effect using statistical measurements.

Binary outcomesOdds ratioCalculating relative riskStandardised mean differenceCombining the quantitative and the qualitative.