How to Analyze Therapy in the Medical Literature

(part 1)

Akbar Soltani. MD.MScTehran University of Medical Sciences (TUMS)

Shariati Hospitalwww.soltaniebm.com

Objectives

• What is randomized controlled trial?• How to appraise RCT using standard

checklist• Review different concepts such as

– Randomization, allocation concealment, blinding, loss to follow up, intention to treat analysis

  The hierarchy of evidence

• Systematic reviews and meta-analyses • Randomised controlled trials • Cohort studies • Case-control studies • Cross sectional surveys • Case reports• Expert opinion

Three Step Guide in Using an Article to Assess Therapy

1. Are the results of the study valid?

2. What are the results? What measures of precision of effects were reported (CIs, p-values)?

3. How can I apply these results to patient care?

Randomized control trial design

Assess Validity and Applicability to my practice setting

1.Is the study a randomized control trial (RCT)? Yes (go on) No (stop)2.Were the patients properly selected for the trial and

randomized with concealed assignment? Yes (go on) No (stop)3.Were patients and study personnel “blind” to treatment? Yes (go on) No (pause)4.Were the intervention and control groups similar at the

start? (Check “Table 1” of most studies) Yes (go on) No (stop)5.Was follow-up complete?ii. Were patients analyzed in the groups to which they

were randomized (“intention-to-treat” analysis)?

Benefits of Random Allocation (Randomization)

1.Reduces bias in those selected for treatment– guarantees treatment assignment will

not be based on patients’ prognosis

2.Prevents confounding– known and unknown potential

confounders are evenly distributed

Assess Validity and Applicability Assess Validity and Applicability to my practice settingto my practice setting

1.Is the study a randomized control trial (RCT)? Yes (go on) No (stop)2.Were the patients properly selected for the trial and

randomized with concealed assignment? Yes (go on) No (stop)3.Were patients and study personnel “blind” to treatment? Yes (go on) No (pause)4.Were the intervention and control groups similar at the

start? (Check “Table 1” of most studies) Yes (go on) No (stop)5.Was follow-up complete?ii. Were patients analyzed in the groups to which they

were randomized (“intention-to-treat” analysis)?

Ensuring Allocation Concealment

BEST – most valid technique

Central computer randomization

DOUBTFUL Envelopes, etc

NOT RANDOMIZED Date of birth, alternate days,

etc

Do Not Confuse Allocation Concealment with Blinding

• Allocation concealment seeks to prevent selection bias, protects assignment sequence before and until allocation, and can always be successfully implemented

Do Not Confuse Allocation Concealment with Blinding (Cont’d)

• Blinding seeks to prevent information bias, protects sequence after allocation, and cannot always be successfully implemented

Assess Validity and Applicability Assess Validity and Applicability to my practice settingto my practice setting1.Is the study a randomized control trial (RCT)? Yes (go on) No (stop)2.Were the patients properly selected for the trial and

randomized with concealed assignment? Yes (go on) No (stop)3.Were patients and study personnel “blind” to

treatment? Yes (go on) No (pause)4.Were the intervention and control groups similar at the

start? (Check “Table 1” of most studies) Yes (go on) No (stop)5.Was follow-up complete?ii. Were patients analyzed in the groups to which they

were randomized (“intention-to-treat” analysis)?

Placebo effect Trial in patients with chronic severe itching

0

10

20

30

40

50

60

I tching scoreCyproheptadine

HCL

Trimeprazine tartrate

No treatment

Treatment vs no treatment for itching

Placebo effect Trial in patients with chronic severe itching

0

10

20

30

40

50

60

I tching scoreCyproheptadine

HCL

Trimeprazine tartrate

Placebo

No treatment

Treatment vs no treatment vs placebo for itching

Placebo effect - attributable to the expectation that the treatment will have an effect

Blinding and Reporting

• Usually reduces differential assessment of outcomes (information bias)

• Authors should explicitly state who was blinded – and how.

• Many investigators and readers consider a randomized trial as high quality simply because it is “double-blind,” as if double-blinding is the sine qua non of an RCT.

– Trials not “double-blinded” should not automatically be deemed inferior trials.

Blinding in randomised trials: hiding who got what

• Double blinding prevents bias but is less important, than adequate allocation concealment.

• open studies are more likely to favour experimental interventions over the controls and that studies that are not double-blinded can exaggerate effect estimates by 17%

• Furthermore, in some trials, blinding cannot be successfully implemented, whereas allocation concealment can always be successfully implemented.

Assess Validity and Applicability Assess Validity and Applicability to my practice settingto my practice setting

1.Is the study a randomized control trial (RCT)? Yes (go on) No (stop)2.Were the patients properly selected for the trial and

randomized with concealed assignment? Yes (go on) No (stop)3.Were patients and study personnel “blind” to treatment? Yes (go on) No (pause)4.Were the intervention and control groups similar at the

start? (Check “Table 1” of most studies) Yes (go on) No (stop)5.Was follow-up complete?ii. Were patients analyzed in the groups to which they

were randomized (“intention-to-treat” analysis)?

Significance tests for baseline differences

Baseline data

AzathioprineAzathioprine PlaceboPlacebo

Mean ageMean age 54.754.7 54.954.9

Serum bilirubin (Serum bilirubin (mol/L)mol/L) 37.237.2 30.930.9

Stage I disease %Stage I disease % 1414 1212

Stage II disease %Stage II disease % 4444 4343

Stage III disease %Stage III disease % 1515 1515

Stage IV disease %Stage IV disease % 2727 3030

Christensen et al. Gastro 1985

Effect of azathioprine on the survival of patients with primary biliary cirrhosis

Baseline data

AzathioprineAzathioprine PlaceboPlacebo

Mean ageMean age 54.754.7 54.954.9

Serum bilirubin (Serum bilirubin (mol/L)mol/L) 37.237.2 30.930.9

Stage I disease %Stage I disease % 1414 1212

Stage II disease %Stage II disease % 4444 4343

Stage III disease %Stage III disease % 1515 1515

Stage IV disease %Stage IV disease % 2727 3030

Christensen et al. Gastro 1985

Effect of azathioprine on the survival of patients with primary biliary cirrhosis

Assess Validity and Applicability Assess Validity and Applicability to my practice settingto my practice setting

1.Is the study a randomized control trial (RCT)? Yes (go on) No (stop)2.Were the patients properly selected for the trial and

randomized with concealed assignment? Yes (go on) No (stop)3.Were patients and study personnel “blind” to treatment? Yes (go on) No (pause)4.Were the intervention and control groups similar at the

start? (Check “Table 1” of most studies) Yes (go on) No (stop)5.Was follow-up complete?5.Was follow-up complete?ii. Were patients analyzed in the groups to which they

were randomized (“intention-to-treat” analysis)?

How complete was the follow up? How many dropouts were there?

• Conventionally, a 20% drop out rate is regarded as acceptable, but this depends on the study question.

• Some regard should be paid to why participants dropped out, as well as how many.

• It should be noted that the drop out rate may be expected to be higher in studies conducted over a long period of time.

• A higher drop out rate will normally lead to downgrading, rather than rejection of a study.

Over-estimation oftreatment effect

Bias: a one-sided inclination of the mind

• Not random 40%

• Not double-blind 17%

• Duplicate information 20%

• Small trials 30%

• Poor reporting quality 25%

Assess Validity and Applicability Assess Validity and Applicability to my practice settingto my practice setting

1.Is the study a randomized control trial (RCT)? Yes (go on) No (stop)2.Were the patients properly selected for the trial and

randomized with concealed assignment? Yes (go on) No (stop)3.Were patients and study personnel “blind” to treatment? Yes (go on) No (pause)4.Were the intervention and control groups similar at the

start? (Check “Table 1” of most studies) Yes (go on) No (stop)5.Was follow-up complete?ii. Were patients analyzed in the groups to which they ii. Were patients analyzed in the groups to which they

were randomized (“intention-to-treat” analysis)?were randomized (“intention-to-treat” analysis)?

Montorri V, Guyatt G. CMAJ 2001 165(10) p1340

Intention to treat

Montorri V, Guyatt G. CMAJ 2001 165(10) p1340

Intention to treat

Montorri V, Guyatt G. CMAJ 2001 165(10) p1340

Intention to treat

High risk?

Assess Validity and Applicability Assess Validity and Applicability to my practice settingto my practice setting

1.Is the study a randomized control trial (RCT)? Yes (go on) No (stop)2.Were the patients properly selected for the trial and

randomized with concealed assignment? Yes (go on) No (stop)3.Were patients and study personnel “blind” to

treatment? Yes (go on) No (pause)4.Were the intervention and control groups similar at the

start? (Check “Table 1” of most studies) Yes (go on) No (stop)5.Was follow-up complete?ii. Were patients analyzed in the groups to which they

were randomized (“intention-to-treat” analysis)?