Ninian PeckittFRCS FFD RCS FDS RCS FACCS

Oral and Maxillofacial Surgeon / Facial Plastic Surgeon

Adjunct Associate Professor of Engineering Assisted SurgeryMassey University, New Zealand

Cochrane Collaboration• group >28,000 volunteers

• >100 countries

• Review health care interventions tested in biomedical randomized controlled trials.[3]

• + more non-randomized, observational studies.

• systematic reviews published as "Cochrane Reviews"

• in the Cochrane Library.

Goals and PrinicplesThe goal of the collaboration is to help people make well informed decisions about health care by preparing, maintaining and ensuring the accessibility of systematic reviews of the effects of health care interventions. The principles of the Cochrane Collaboration are:

• collaboration• building on the enthusiasm of individuals• avoiding duplication• minimizing bias• keeping up to date• striving for relevance• promoting access• ensuring quality• continuity• enabling wide participation

Forest Plot - (Blobbogram)

• Common - 2 columns

• The left-hand column lists: – the names of the studies

(e.g. RCTs / epidemiol studies)

– chronological order

– from the top downwards.

Forest Plot - (Blobbogram)• The right-hand column

• a plot of the measure of effect – e.g. an odds ratio for each of these studies – (often represented by a square) – incorporating confidence intervals – represented by horizontal lines.

• The graph may be plotted on a natural logarithmic scale when using odds ratios or other ratio-based effect measures

• confidence intervals are symmetrical about the means from each study

• ensures undue emphasis is not given to odds ratios greater than 1 when compared to those less than 1

Forest Plot

Meta Analysis• combines the results of several studies

• that address a set of related research hypotheses

• In its simplest form, this is normally by identification of a common measure of effect size,

• for which a weighted average might be the output of a meta-analyses.

• Here the weighting might be related to sample sizes within the individual studies.

• More generally there are other differences between the studies that need to be allowed for, but the general aim of a meta-analysis is to more powerfully estimate the true "effect size" as opposed to a smaller "effect size" derived in a single study under a given single set of assumptions and conditions.

Advantages of Meta Analysis

• Shows if the results are more varied than expected from sample diversity

• Derivation and statistical testing of overall factors / effect size parameters in related studies

• Generalization to the population of studies

• Ability to control for between-study variation

• Including moderators to explain variation

• Higher statistical power to detect an effect than in ‘n=1 sized study sample’

• Deal with information overload: the high number of articles published each year.

• combines several studies less influenced by local findings than single studies will be.

• May show if a publication bias exists.

Steps in Meta Analysis1. Formulation of the problem

2. Search of literature

3. Selection of studies (‘incorporation criteria’)

– Based on quality criteria, e.g. the requirement of randomization and blinding in a clinical trial

– Selection of specific studies on a well-specified subject, e.g. the treatment of breast cancer.

– Decide whether unpublished studies are included to avoid publication bias

4. Decide which dependent variables or summary measures are allowed.

– Differences (discrete data)

– Means (continuous data)

Steps in Meta Analysis• Hedges' g

– is a popular summary measure for continuous data – that is standardized in order to eliminate scale differences, but incorporates

an index of variation between groups:

μt is the treatment mean, μc is the control mean, σ2 the pooled variance

• For reporting guidelines, see QUOROM statement [6][7]

Metaregression Models

5. Model selection

• simple regression

• fixed effect meta-regression

• random effects meta-regression

Simple Regression

Where yj is the effect size in study j and \β0 (intercept) the estimated overall effect size

The variables specify

• different characteristics of the study,

• specifies the between study variation

• Note that this model does not allow specification of within study variation.

Fixed Effect Meta-regression• assumes that the true effect size θ

• is normally distributed - within study variance of the effect size

• allows for within study variability

• but no between study variability

• because all studies have the identical expected fixed effect size θ,

• i.e.***Note that for the "fixed-effect" no plural is used (in contrast to "random-effects") as only ONE true effect across all datasets is assumed.***

Fixed Effect Meta-regression

• variance of the effect size in study j

• Fixed effect meta-regression ignores between study variation

• parameter estimates biased if between study variation can not be ignored

• Generalizations to the population are not possible

Random effects meta-regression

• assumption that θ in is a random variable• following a (hyper-)distribution

• Here is the variance of the effect size in study j.

• Between study variance is estimated using common estimation procedures for random effects models (restricted maximum likelihood (REML) estimators).

Applications in Modern Science• Modern statistical meta-analysis

• does more than just combine the effect sizes of a set of studies.

• test if studies show more variation than expected (e.g. sampling different research participants)

• study characteristics such as :

– Measurement– instrument used– population sampled– or aspects of the studies' design are coded.

• These characteristics are then used as predictor variables :

– to analyze the excess variation in the effect sizes– Studiesweaknesses can be corrected statistically (e.g. bias in size or codings)

Applications in Modern Science

• Meta-analysis can be done with – single-subject design – group research designs

• much research on low incidents populations– single-subject research designs

– Considerable dispute exists for the most appropriate meta-analytic technique for single subject research.[8]

Single Subject DesignContinuous assessment:

• Individual behaviour observed repeatedly over the intervention. • Insures Rx effects observed long enough to convince that Rx produces a lasting effect.

Baseline assessment:

• Before Rx is implemented, look for behavioral trends.• If Rx reverses a baseline trend (e.g., getting worse [baseline ]vx [Rx] reversed this trend)• powerful evidence suggesting (though not proving) a treatment effect.

Variability in data:

• Ability to observe how Rx changes behavior from day-to-day• Large-group statistical designs do not typically provide this information • because repeated assessments are not usually not taken • and the behavior of individuals in the groups are not scrutinized - group means are reported

Phases within Single Subject DesignPhases within single-subject design

• Baseline:– data on the dependent variable without any intervention in place

• Intervention: – introduce independent variable (the intervention)– collect data on dependent variable

• Reversal:– removes the independent variable (reversal)– collects data on the dependent variable

• Data must be Stable (steady trend / low variability) before move to the next phase

• Single-subject designs produce or approximate three levels of knowledge:

– (1) descriptive– (2) correlational– (3) causal[4]

Flexibility of Single Subject Design• highly flexible

• highlight individual differences

• in response to intervention effects[5]

• In general

– reduce interpretation bias

– for counselors when doing therapy[6]

Data InterpretationIndependent variable on the dependent variable,

• graph the data collected / visually inspect the differences between phases

• If there is a clear distinction between baseline and intervention, and then the data returns to the same trends/level during reversal, a functional relation between the variables is inferred.[7]

• Sometimes, visual inspection of the data demonstrates results that statistical tests fail to find[8][9]

• Begin with Graphic analysis– During the baseline, data are repeatedly collected and then graphed on the behavior of interest. – Visual representation of the subject’s behavior before application of the intervention– Several (e.g. 3-5 ) baseline data points for description of effects on the target behavior during intervention

• Subject as their own control

• Baseline behavior would match its behavior in the intervention phase unless the intervention does something to change it.

• This logic then holds to rule out confound

Goals

• Prediction

• Verification

• Replication

Meta-analysis of single subject research

• Meta-analysis, like all research, has the ability to change a profession.

• functional analysis < effect sizes than contingency management

• Debate meta-analysis of single-subject designs.

• The two choices being debated are

– the percentage nonoverlapping data(PND) vs.

– data points exceeding the median(PEM) method.[13][14][15]

– Noorgate and colleagues have argued that meta-analyses that analyze all linear trends in data don’t work since they don’t distinguish between effects on level and slope.[14][16]

Limitations Single Subject Designs• Preplanning of Designs often made as the data are collected.[17]

• No widely agreed upon rules for: altering phases / conflicting ideas / conduct

• Major criticisms of single-subject designs are:

– Carry-over effects: results from the previous phase carry-over into the next phase,

– Order effects: the ordering (sequence) of the intervention / treatment affects what results

– Irreversibility:

• once a change in the independent variable occurs, the dependent variable is affected• This cannot be undone by simply removing the independent variable.

– Ethical problems:

• Withdrawal of treatment - ethical and feasibility problems

Statistical Significance

• unlikely to have occurred by chance

• Stat hypothesis tests that answer the question

– Assuming that the null hypothesis is true – what is the probability of a value as extreme as

the value observed?

• Systematic Review: takes 23 months from protocol to publication

• Timespan: Hundreds / Thousands of hours.

• Problems: Production and Updating of review

• <40% data is up-to-date

Out of Date Systematic Reviews• 2009 - 2,383• 2012 - 3,149• Terrible figures made worse by increase in

spending on Cochrane

http://blog.tripdatabase.com/2013/04/a-critique-of-cochrane-collaboration.html#sthash.vioz56f0.dpuf

Funding Cochrane

• £100 million over last 7 years

• >£150 million – $250 billion US Dollars over 20 years of its existence

http://blog.tripdatabase.com/2013/04/a-critique-of-cochrane-collaboration.html#sthash.vioz56f0.dpuf

• Current system is unsustainable

• Current System is not fit for purpose. 

• The methodology

• has reduced some bias, • but resulted in a huge financial cost increase • a huge cost in opportunity

• The Tamiflu fiasco highlights a flawed methodology

http://blog.tripdatabase.com/2013/04/a-critique-of-cochrane-collaboration.html#sthash.vioz56f0.dpuf

Tamiflu - Tom Jefferson Review Leader stated:

“…I personally believe and my colleagues believe with me that Cochrane Reviews based on publications should really be a thing of the past…”

Cochrane systematic review relied on published journal articles

• Large amounts of data were missed, most of which was made available for the regulatory agencies e.g. EMA, FDA. 

• The updated, 2012, review was a huge undertaking

• Jack Cuzick - Evidence Live 2013 made a general call for reviews to be based on individual patient data (IPD)

http://blog.tripdatabase.com/2013/04/a-critique-of-cochrane-collaboration.html#sthash.vioz56f0.dpuf

Peer opinion Cochrane Methodology

• Incapable of making an accurate assessment of an interventions ‘worth’ 

• The seriousness of this challenge should not be underestimated,

• This challenge attacks at the very heart of the Cochrane Collaboration. 

http://blog.tripdatabase.com/2013/04/a-critique-of-cochrane-collaboration.html#sthash.vioz56f0.dpuf

Doing things more quickly can give you the same or similar results to the Cochrane methodology.

1)    Can we rely on the best trial?

A comparison of individual trials and systematic reviews

• Random sample of Cochrane systematic reviews

• Was largest RCT in agreement with the subsequent meta-analysis? 

• Yes - 81% of the meta-analyses examined and if the largest RCT was positive and significant it was around 95%. 

• In other words, using the largest RCT can give a broad hint as to the likely result of a subsequent meta-analysis

Is Systematic Review Really Required?

http://blog.tripdatabase.com/2013/04/a-critique-of-cochrane-collaboration.html#sthash.vioz56f0.dpuf

2)    McMaster Premium LiteratUre Service (PLUS) performed well for identifying new studies for updated Cochrane reviews.

• Authors compared the performance of McMaster Premium LiteratUre Service (PLUS) and Clinical Queries (CQs) to that of the Cochrane Controlled Trials Register, MEDLINE, and EMBASE for locating studies added during an update of reviews.

• They concluded that PLUS included less than a quarter of the new studies in Cochrane updates, but most reviews appeared unaffected by the omission of these studies. 

• In other words, you do not necessarily need to get all articles to arrive at an accurate effect size (compared to the Cochrane systematic review).

Is Systematic Review Really Required?

http://blog.tripdatabase.com/2013/04/a-critique-of-cochrane-collaboration.html#sthash.vioz56f0.dpuf

Is Systematic Review Really Required?

3)    A pragmatic strategy for the review of clinical evidence. 

• Authors compared a research strategy based on the review of a selected number of core journals, with that derived by an SR in estimating the efficacy of treatments. 

• Conclusion: “We verified in a sample of SRs that the conclusion of a research strategy based on a pre-defined set of general and specialist medical journals is able to replicate almost all the clinical recommendations of a formal SR.

http://blog.tripdatabase.com/2013/04/a-critique-of-cochrane-collaboration.html#sthash.vioz56f0.dpuf

The Future

• Reduce the cost per review. 

• Recognise laws of diminishing returns 

• Major challenge to do a modification of a systematic review in a month (or less)

rapid systematic reviews A more detailed/costly sytematic review including regulatory data and/or IPD.

• Reduce cost of Review by 90%

Is Systematic Review Really Required?

http://blog.tripdatabase.com/2013/04/a-critique-of-cochrane-collaboration.html#sthash.vioz56f0.dpuf

But might we get the wrong answer with streamlining?

Yes !

But detailed Cochrane systematic reviews also have given us the wrong answer !

Is Systematic Review Really Required?

References

1. The Cochrane Oversight Committee. Measuring the performance of The Cochrane Library. 2012

2. Allen IE, Olkin I. Estimating time to conduct a meta-analysis from number of citations retrieved. JAMA. 1999 Aug 18;282(7):634-5.

3. Cochrane Collaboration Annual Report & Financial Statements 2010/11

4. Payne D. Tamiflu: the battle for secret drug data. BMJ 2012;345:e7303

5. HAI Europe - Dr. Tom Jefferson on lack of access to Tamiflu clinical trials

6. Jefferson TO, Demicheli V, Di Pietrantonj C, Jones M, Rivetti D. Neuraminidase inhibitors for preventing and treating influenza in healthy adults. Cochrane Database Syst Rev. 2006 Jul 19;(3):CD001265

7. Jefferson T, Jones MA, Doshi P, Del Mar CB, Heneghan CJ, Hama R, Thompson MJ. Neuraminidase inhibitors for preventing and treating influenza in healthy adults and children. Cochrane Database Syst Rev. 2012 Jan 18;1:CD008965. doi: 10.1002/14651858.CD008965.pub3

8. Glasziou PP, Shepperd S, Brassey J. Can we rely on the best trial? A comparison of individual trials and systematic reviews. BMC Med Res Methodol. 2010 Mar 18;10:23. doi: 10.1186/1471-2288-10-23

9. Hemens BJ, Haynes RB. McMaster Premium LiteratUre Service (PLUS) performed well for identifying new studies for updated Cochrane reviews. J Clin Epidemiol. 2012 Jan;65(1):62-72.e1

10.Sagliocca L, De Masi S, Ferrigno L, Mele A, Traversa G. A pragmatic strategy for the review of clinical evidence. J Eval Clin Pract. 2013 Jan 15. doi: 10.1111/jep.1202

11. Greenhalgh T. Why do we always end up here? Evidence-based medicine's conceptual cul-de-sacs and some off-road alternative routes. J Prim Health Care. 2012 Jun 1;4(2):92-7.J Prim Health Care. 2012 Jun 1;4(2):92-7.