26th November 2012

Critical Paper Review

Dr Dave Brodbelt MA VetMB PhD DVA DipECVAA MRCVS Senior Lecturer, Companion Animal Epidemiology VEPH Group, Royal Veterinary College, London dbrodbelt@rvc.ac.uk

Learning Outcome

• Identify the key elements of a critical appraisal

• Distinguish validity from importance

• Evaluate limitations including

– Selection / allocation of patients

– Outcomes assessment

– Dealing with bias and confounding

– Study power

Review: Types of Study Design

Descriptive

Studies

Case Series Case Reports Surveys

Analytical

Studies

Intervention Observational

Cross

Sectional

Studies

Cohort

Studies

Case-

Control

Studies

Descriptive Studies

• Describe events

• Generate hypotheses for further evaluation

• Particularly valuable when little is known about the

condition

– E.g. Preliminary observations of a new disease

• Types of descriptive studies

– Case reports

– Case series

– Surveys

Analytical Studies

• Build on descriptive studies

• Explicit comparison between groups

• Test hypotheses

• Identify ‘determinants’ of ‘disease’ – risk factors

• Explore / identify causal relationships

• Types of analytical studies – Observational studies

– Intervention studies

Hierarchy of Evidence

Systematic reviews / Meta

analyses

Randomised Controlled

Trial

Cohort study

Case-control study

Cross sectional study

Case series

Single case reports

Editorials, Opinion

Increasing Strength

of Evidence

Weaker

Evidence

Critical Appraisal of Studies • Validity of the results – alternative explanation?

– Chance?

– Bias?

– Confounding?

• Importance of results

– What was the magnitude of the treatment effect?

– How precise are the estimates of treatment effects?

Bias – Systematic Error

• Selection bias

– Systematic error in the identification of the study

population

– Patients selected differ systematically from those not

selected

– Selection based on exposure / outcome depends on the

other

• E.g. allocation of dogs with worst ataxia to spinal

surgery and others to conservative treatment in a

non-randomised clinical trial

Bias – Systematic Error

• Information bias

– Information obtained differently between study

groups

• Systematically incorrect measurements

• Systematically incorrect recording of exposure

or outcome

– Recall Bias – patients in one group more likely to

remember exposure / risk factor

Bias – Systematic Error

• Information bias

– Observer Bias –

• Investigators assessing exposure do so with

differing accuracy for cases and controls

• E.g. interviewer bias – systematic differences

in soliciting, recording or interpreting

information from study participants

Potential confounders identified and adjusted for?

– Effect of extraneous variable wholly or partially accounts for apparent association between exposure and disease

– Health status as a confounder

• ‘Isoflurane associated with anaesthetic-death’

• Health status = confounder

Health status

Isoflurane Death

Confounders • Potential confounders identified and adjusted for

– Confounders need to be measured

– Adjust for

• Match for the confounder in the study design

• Adjust for in the analysis

– Stratify

– Multivariable regression

Importance of results

• What was the magnitude of the treatment

effect? – Greater likelihood of real effect

– More relevant to clinical practice

• How precise are the estimates of treatment

effects?

Importance of results • How precise are the estimates of treatment effects?

– The level of UNCERTAINTY in the measured results

– Confidence Intervals (CI) - 95% CI

• The range in which the true treatment effect is likely to

lie with a given degree of confidence

• “95% sure that true value in lies within the interval”

– 100 repeated samples taken from same

population → 95% of confidence intervals

calculated will include true population estimate

• Narrow CI means greater precision

Association and Causation

• Questions we should ask:

– For a study are the associations valid?

– Given the totality of evidence are the associations

causal?

• Studies often report a valid association of exposure

and outcome

• Association NOT necessarily causation

– Association – statistical dependence

– Cause – factor contributes to outcome

Association and Causation • Assess causation based on body of

evidence

– Type of study

– Strength of association

– Consistency with other work

– Specificity of association

– Temporal relationship

A Bradford Hill 1965

Association and Causation • Assess causation based on body of evidence

– Biological gradient / dose response

– Biological plausibility

– Coherence with known biology of the disease

– Supporting experimental evidence

– Analogy of a similar condition

AB Hill 1965

Appraisal Check List

• Study – original?

– There may be previous studies

– Does the study ADD to the literature?

• Who was the study about?

– How were subjects recruited?

– Who was included in the study?

– Who was excluded from the study?

Appraisal Check List

• Study Design – sensible?

– Objectives clear?

– Study design clear AND appropriate?

• What intervention or other process studied?

• Who compared to?

– What outcome measured?

Appraisal Check List

• Systemic bias avoided?

– Clinical trials – random allocation of subject?

• RCT

• Non-randomised clinical trials?

– Observational studies

• Case definitions

• Selection of control group

• Follow-up sufficiently long and complete

• Adjusted for confounders in the design

Appraisal Check List

• Was assessment blind?

– Assessment of outcome blind?

• Were preliminary statistical questions answered?

– Sample size based on pre-study power calculation?

– Study large enough?

– Sufficiently long follow-up?

Appraisal Check List

• Study Conduct

– Good response rate?

– Sufficient sample size?

– Losses to follow-up?

• Minimal

• Attempt to evaluate any major losses to follow-up

Appraisal Check List

• Analysis and presentation

– Statistical methods clear AND appropriate?

– Presentation of data appropriate?

– Sufficient analyses presented?

– Adjusting for confounders if appropriate?

– Precision / confidence intervals for main results?

• Overall assessment

– Limitations acknowledged?

– Conclusions justified?

‘Controlled trial of behavioural therapy for separation-related disorders in dogs’

• Vet Rec (2006) 158, 551-554

• Method – 55 dogs assigned to 3 groups – 2 treatment, 1 control

– Underwent behavioural therapy, or no treatment

– Outcome

• Owner assessed change in behaviour

• Video footage for duration of 6 behaviour types

• Study design?

• Appraisal?

‘Controlled trial of behavioural therapy for separation-related disorders in dogs’

• VR (2006) 158, 551-554

• Method – 55 dogs assigned to 3 groups – 2 treatment, 1 control

• Study design? Non-randomised trial

• Appraisal? – Pre-study power?

– Random allocation?(no)

– Clear outcome defined and assessed?

– Owner and investigator assessments blinded?

Cystic ovarian disease Comparison of synchronization of ovulation with timed insemination and exogenous

progesterone

The objective of this study was to compare the effectiveness of the Ovsynch and

controlled internal drug releasing (CIDR) protocols under commercial conditions

for the treatment of cystic ovarian disease in dairy cattle. A total of 401 lactating

dairy cows with ovarian cysts were alternatively allocated to two treatment

groups on the day of diagnosis. Cows in the Ovsynch group were treated with

GnRH on Day 0, PGF2alpha on Day 7, GnRH on Day 9,

• Study design?

• Limitations?

Cystic ovarian disease Comparison of synchronization of ovulation with timed insemination and exogenous

progesterone

The objective of this study was to compare the effectiveness of the Ovsynch and

controlled internal drug releasing (CIDR) protocols under commercial conditions

for the treatment of cystic ovarian disease in dairy cattle. A total of 401 lactating

dairy cows with ovarian cysts were alternatively allocated to two treatment

groups on the day of diagnosis. Cows in the Ovsynch group were treated with

GnRH on Day 0, PGF2alpha on Day 7, GnRH on Day 9, • Study design? Non-random allocation - alternate

• Limitations

• Sampling method? How selected cows?

• Clear outcome?

• Sufficiently long follow-up?

• Median survival?

• Limitations?

Outcomes over time • Percent survival at a particular time

– 1 or 2 year survival

• Median survival time

– Time at which 50% of patients have developed the outcome

• Survival curves

– Each point on the curve

represents the proportion of

patients that have

NOT developed the outcome

– Probability of surviving

– Kaplan Meier curve

• Appraisal

• Study group – all similar at start of study?

• Clear outcome?

• Blindly assessed?

• Follow-up length and completeness?

• Adjusted for co-treatments?

‘Improved survival in a retrospective cohort of 28 dogs with insulinoma’

• JSAP (2007) 48, 151-156

• Method

– 28 dogs identified with insulinoma

– Record clinical signs, lab results, clinical staging

– Outcome – survival time, complications

• Study design?

• Appraisal?

‘Improved survival in a retrospective cohort of 28 dogs with insulinoma’

• JSAP (2007) 48, 151-156

– 28 dogs identified with insulinoma

– Outcome – survival time, complications

• Study Design - Cohort

• Appraisal?

– Representative sample?

– Clear definition of exposure / risk factors?

– Objective outcome applied blindly?

– Follow-up sufficiently long and complete?

– Adjust for confounders?

Cohort Study • Group of subjects with a common defined characteristic, i.e.

exposure / risk factor status

• Prospective

– Study starts BEFORE outcome of interest occurred

– Preferred – temporal sequence ensured, data required is recorded

• Retrospective

– Study begins AFTER

outcome occurred

– Relies on

pre-existing records

often not recorded

with study in mind

OUTCOMEEXPOSURE

Yes

No

Yes

No

Time

PROSPECTIVE

starts study here

RETROSPECTIVE

starts study here

‘Risk factors for anaesthetic death in Rabbits’

• D Brodbelt, PhD thesis (2006)

• Method – Study animals

• 70 deaths, peri-operative

• 70 non-deaths – anaesthetised before or soon after each death at the same centre

– Record patient, procedure, anaesthetic management, personnel and recovery details

– Outcome – anaesthetic related death

• Study design?

• Limitations?

• Precision of estimates?

‘Risk factors for anaesthetic death in Rabbits’ • D Brodbelt, PhD thesis (2006)

– CASE – CONTROL STUDY

• 70 deaths = CASES

• 70 non-deaths requested – before or soon after each death at the same centre = MATCHED CONTROLS

– Record patient, procedure, anaesthetic management, personnel and recovery details

– Outcome – anaesthetic related death

• Limitations? – Study size?

– Selection of controls?

– Clear outcome?

– Adjusted for confounders?

• Precision of estimates?

Case-Control Study

• Identify patients with disease (Cases) then

compare to non-diseased patients (Controls)

• Look back at their exposures / risk factors

• Identify risk factors for disease

• Well suited for RARE diseases

OUTCOMEEXPOSURE

Yes

No

TIME

Study starts

here

Yes / Case

No / Control

Rabbit Multivariable

Model Risk Factor OR (95% CI) LRT

P value

Sick/Healthy

(ASA 3-5/1-2)

28.7 (2.9 – 283.7) 0.001

Duration intended

30 min plus /<30 min

4.5 (0.9-21.5) 0.034

Major /Minor Procedures 5.8 (0.9 – 39.2) 0.048

Vet: Very Familiar / Familiar or

Unfamiliar

0.14 (0.02-0.96) 0.025

What is likely to be the major weakness of this study?

‘…Investigators randomly allocated 20 patients to two

analgesic groups … postoperative analgesia was

scored.

Results. No difference detected between analgesics.

Conclusions. The new analgesic was as good as the

current standard…’

1. One – non-random allocation

2. Two – investigators were not

blinded to groups

3. Three – conclusion over-stated

Conclusions

• Study appraisal – consider main elements – to

assess validity and importance

• Major issues

– Original?

– Whom does it concern? Generalisability?

– Study design sensible?

– Systematic bias avoided / minimised?

– Assessments undertaken blindly?

– Assess preliminary statistical questions?

Further Reading

• Cockcroft and Holmes (2003) Handbook of Evidence-Based Veterinary Medicine

• Cockcroft and Holmes (2004) EBVM 1-3 In Practice

• Dohoo, I., W. Martin, et al. (2009). Veterinary Epidemiologic Research. Charlottetown, AVC Inc. 2nd edn

• Greenhalgh, T (1997) How to read a paper: Assessing the methodological quality of published papers. BMJ, 7103, 315

• Greenhalgh T (2010) How to read a paper, the basics of evidence based medicine. 4th edn. BMJ Books.

• Pfeiffer (2010) Veterinary Epidemiology: An Introduction

• Straus et al (2005) Evidence-Based Medicine, How to Practice and Teach EBM. London, Elsevier.