Systematic reviews in the social sciences

Sarah Whitehead, Cardiff Institute of Society and Health

Outline

• Why do systematic reviews in the social sciences?

• Examples of types of reviews: Correlates of physical activity Effectiveness of tobacco harm reduction approaches Barriers and facilitators to tobacco harm reduction

• Overview of different types of evidence generated• Lessons learnt

Why do systematic reviews in the social sciences? Chaos in the brickyard

• Goal of scientific research to discover explanations• Research findings to be assembled like bricks to create edifices• Initially bricks carefully made and only those required for the edifice

produced• Over time, brick making became an end in itself until the land was

flooded with bricks, many of them random and useless• Now we need to search through so many bricks it’s difficult to find

proper ones for the task• “Saddest of all, sometimes no effort was made even to maintain the

distinction between a pile of bricks and a true edifice”1

1. Forscher 1963

Why do systematic reviews in the social sciences?

• Systematic reviews allow us to make sense of the chaos in the brickyard

• They allow us to: develop a comprehensive understanding of research

to date identify areas of evidence that can be translated into

policy and practice determine which areas are in need of further research

Examples of types of reviews

Example 1: correlates of physical activity

• Research question: What factors are associated with physical activity

participation among adolescent girls?

• Suitable design: Correlates review of quantitative studies reporting a

measure of physical activity and at least one potential correlate

Example 1: methods

• 4 databases searched, manual searches of journals• Inclusion criteria:

Quantitative research design Published 1999-2003 Participants in age range 10-18 years Results available for females separately Articles published in English language Any type of physical activity

Example 1: methods (cont’d)

• Potential correlates classed as demographic/biological, psychological, behavioural, social/cultural, or physical environmental

• Variables classed as related/ not related to physical activity• Strength of association noted1

• Variables only included if studied three or more times2

• Quality of physical activity measure coded2: Self-report of poor or unknown reliability/validity Self-report with acceptable reliability/validity Acceptable objective measure

1. Cohen 19882. Sallis et al. 2000

Example 1: findings• 50 published papers yielding 51 independent

samples• Majority of studies (63%) carried out in US• Mean sample size (females only) = 1,280

(range 22-9,039)• 41 (80.4%) cross-sectional in design• Majority (61%) used self-report with acceptable

reliability/validity

Example 1: findings• Demographic variables:

Non-white ethnicity (-) Age (-) SES (+)

• Psychological variables: Enjoyment (+) Perceived competence (+) Self-efficacy (+) Physical self-perceptions (+)

• Behavioural variables: Smoking (-) Organised sport

involvement (+)

• Social/cultural variables: Family/parental support (+) Father’s physical activity (+)

• No consistent trends for environmental variables

Example 1: conclusions

• Several variables related with PA across 3+ studies• Many potential correlates understudied• Several weak designs and measures of poor or

unknown validity• Most promising modifiable correlates:

Family support Positive psychology Organised sport

Example 2: effectiveness of tobacco harm reduction approaches• Research questions:

How effective are tobacco harm reduction approaches with the intention of quitting (i.e. ‘cutting down to quit’ or ‘reduction to stop smoking’), with and without assistance?

How effective are long-term harm reduction approaches without the prior intention of quitting (i.e. reducing consumption without the aim of quitting), with and without assistance?

• Suitable design: Two effectiveness reviews of experimental studies of tobacco

harm reduction approaches, e.g. randomised controlled trials, controlled trials

Example 2: methods

• 21 databases, 28 websites, range of 'snowballing’ techniques• Included populations:

People of all ages wanting to quit smoking but unable to do so abruptly / not willing to quit but wanting to reduce/ stop temporarily

Participants in cut down to quit/ reduction/ temporary abstinence interventions

• Included interventions: Pharmacotherapies licensed for cutting down/ temporary abstinence Other non-tobacco nicotine containing products, e.g. e-cigarettes Behavioural support or counselling Self-help

Example 2: methods

• Data extraction to capture: Study design, setting and participants Intervention details Methods of analyses Outcomes

• Quality appraisal to assess: Methodological quality External validity

Narrative synthesis and meta-analyses to summarise findings

Example 2: included studies and types

Cut down to quit• 11 studies

10 RCTs or quasi-RCTs 1 uncontrolled before &

after• Quality

Generally moderate• Applicability

Only one UK study

Cut down without quitting• 45 studies

29 RCTs or quasi-RCTs 3 non RCTs 2 controlled before & after 10 uncontrolled before &

after 1 secondary analysis

• 1 systematic review• Quality

Generally moderate• Applicability

5 UK studies 6 in UK-comparable

countries

Example 2: types of evidence examined

• Effectiveness of pharmacotherapies in helping people cut down to quit or cut down indefinitely without the aim of quitting

• The effectiveness of behavioural support, counselling or self-help

• Whether different approaches had differential impacts on different groups

Example 3: barriers and facilitators to tobacco harm reduction approaches

• Research question: What are the barriers and facilitators to implementing

tobacco harm reduction approaches, including users’ and providers’ perspectives?

• Suitable design: Mixed methods review of qualitative and quantitative

views and opinions studies

Example 3: methods• 21 databases, 28 websites, range of 'snowballing’ techniques• Included populations:

Smokers interested in THR or who take part in THR interventions, or those who provide opinions or experiences regarding THR approaches

Service providers, healthcare personnel and policy makers who may deliver/ commission/ refer smokers to THR interventions

• Included study types: Views studies relating to cut down to quit/ long term reduction/

temporary abstinence intervention studies previously identified Views studies relating to tobacco harm reduction approaches in

general

Example 3: methods

• Data extraction to capture: Study design, setting and participants Methods and analytical approach Outcomes / key themes

• Quality appraisal to assess: Methodological quality External validity

• Thematic synthesis of views, experiences and perspectives• Mapping of whether effectiveness studies addressed the

identified barriers and facilitators to THR approaches

Example 3: included studies and types

• 41 studies 22 cross-sectional surveys 13 qualitative studies 3 mixed methods studies 2 process evaluations 1 longitudinal study

• Quality Generally moderate

• Applicability 9 UK studies and 5 UK applicable

Example 3: types of evidence examined

• Factors that might act as barriers or facilitators to tobacco harm reduction approaches

• Smokers’/ their families’ and healthcare professionals' views and experiences on whether specific tobacco harm-reduction approaches are perceived to have a differential impact on particular groups

• Smokers’/ their families’ and healthcare professionals' views and experiences on the potential or actual unintended consequences from adopting a harm-reduction approach

Summary of different types of evidence that systematic reviews can generate

• Correlates of behaviours or outcomes• Effectiveness of interventions / intervention type

studies• Views and opinions• All can be useful in informing policy and practice

and identifying evidence gaps where further research is needed

Lessons learnt

• Develop a solid search strategy• Identify appropriate tools for the type of review

being undertaken, for example quality assessment measures and approaches to synthesising the data;

• Most importantly – work with a team of experienced information specialists

Acknowledgement

Sarah Whitehead’s role at Cardiff Institute of Society and Health is supported by the Welsh Assembly Government’s National Institute for Social Care and Health Research