META-ANALYSIS, RESEARCH SYNTHESES AND SYSTEMATIC

REVIEWS

© LOUIS COHEN, LAWRENCE MANION & KEITH MORRISON

STRUCTURE OF THE CHAPTER

• Evidence-based research• Meta-analysis• Research syntheses and systematic reviews

EVIDENCE-BASED RESEARCH

• Comes from randomized controlled trials and research syntheses

• Based on accumulated evidence of ‘what works’• Evidence-based approaches are necessary in order

to: – challenge the imposition of unproven practices – solve problems and avoid harmful procedures– create improvement that leads to more effective learning

• Looks at effect size, not only statistical significance

META-ANALYSIS

• The analysis of other analyses• Uses data from randomized controlled trials• Aggregates and combines the results of

comparable studies into a coherent account to discover main effects

• Often uses statistical processes• Looks at effect size, not only statistical

significance• Combines the results of small-scale studies• Uses transparent means to draw conclusions

CONCERNS ABOUT META-ANALYSIS

• Lacks comprehensiveness, as it makes selective use of subsets of studies.

• Misrepresents research findings.• Relies too heavily on significance tests.• Understates effect sizes as a way of supporting hypotheses.• Fails to recognize that sampling error can play a part in

creating variations in findings.• Overlooks differing/conflicting research findings.• Published research is favoured over unpublished research.• Interaction effects overlooked in favour of main effects.• Unclear how it differentiates between ‘good’ and ‘bad’

research, e.g. between rigorous and poorly constructed research.

CONCERNS ABOUT META-ANALYSIS• Difficult to draw logical conclusions from studies that

use different interventions, measurements, definitions of variables, and participants.

• Results from poorly designed studies take their place alongside results from higher quality studies.

• Multiple results from a single study are used, making the overall meta-analysis appear more reliable than it is, since the results are not independent.

• Dangerous consequences because its apparent objectivity and precision may disguise procedural invalidity in the studies.

• Risks trying to synthesize studies which are insufficiently similar to each other to permit this.

CONCERNS ABOUT META-ANALYSIS

• Fails to examine critically the evidence, methods and conclusions of previous reviews

• Overlooks the extent to which findings from research are mediated by sample characteristics

• Overlooks the importance of intervening variables in research

• Unreplicable because procedures for integrating the research findings not made explicit

AN EIGHT-STEP PROCESS FOR META-ANALYSIS

Step 1: Identify the variables for focus (dependent and independent).

Step 2: Identify all the studies which feature the variables in which the researcher is interested.

Step 3: Code each study for those characteristics that might be predictors of outcomes and effect sizes (e.g. age, gender, ethnicity).

Step 4: Estimate the effect sizes for each pair of variables (dependent and independent variable), weighting the effect-size by the sample size.

AN EIGHT-STEP PROCESS FOR META-ANALYSIS

Step 5: Calculate the mean and the standard deviation of effect-sizes across the studies, i.e. the variance across the studies.

Step 6: Determine the effects of sampling errors, measurement errors and range of restriction.

Step 7: If a large proportion of the variance is attributable to the issues in Step 6, then the average effect-size may be an accurate estimate of relationships between variables.

Step 8: If a large proportion of the variance is not attributable to the issues in Step 6, review those characteristics of interest which correlate with the study effects.

EFFECT SIZE

• The effect size is calculated as:

groupcontroltheofdeviationStandardgroup)controlofMeangroupalexperimentof(Mean

RESEARCH SYNTHESES AND SYSTEMATIC REVIEWS

• Include studies that are not solely randomized controlled trials.

• Apply explicit protocols and rigorous criteria for searching for and selecting relevant primary, usually empirical studies, and to remove bias.

• Combine qualitative and quantitative studies.

• Follow a standard set of stages.

• Are accountable, able to be replicated and updated.

• Must be relevant and useful to users.

• Must answer specific research questions.

• Are evidence-based.

RESEARCH SYNTHESES AND SYSTEMATIC REVIEWS

• Clear, transparent criteria for:

– inclusion and exclusion of studies

– standards for acceptable methodological rigour

– relevance to the topic in question

– scope of the studies included

– team approaches to reviewing in order to reduce bias

– adoption of a consistent and clearly stated approach to combining information from across different studies

– careful, relevant conclusions and recommendations

SEQUENCE FOR CONDUCTING SYSTEMATIC REVIEWS

• Formulate the problem and kinds of relevant research evidence to answer the research question or hypothesis.

• Examine the research designs in the studies used, the treatment of the main effects, the kind of research, the conceptual relevance of the studies, i.e. screen the studies.

• Search the literature.

• Gather information from studies (including developing and using a coding guide, identifying predictor and outcome variables, research designs used, sampling, context, statistics used, effect sizes).

SEQUENCE FOR CONDUCTING SYSTEMATIC REVIEWS

• Evaluate the quality of studies (decide which studies to include and exclude, identify problems in the research studies used, evaluate the suitability of the research design for the research synthesis, evaluate the quality and rigour of the research).

• Analyze and integrate the outcomes of studies.

• Interpret the evidence.

• Present the results.