Medical Epidemiology Critiquing the Research Literature.

Medical Epidemiology

Critiquing the Research Literature

Critiquing the Research Literature

Fundamental questions to ask, systematically

Sample critical discussions of two research papers

– Meningococcal disease and campus bar patronage

– Diltiazem and mortality

Critiquing the Literature

Association: a statistical feature of comparisons(s), with six possible explanations: – Causation, with exposure promoting disease

– Chance

– Selection bias

– Measurement bias

– Confounding variable(s)

– Causation, with disease promoting appearance of the exposure

Always ask: are there plausible alternative explanations for the data?


Fundamental Questions– Do the results claimed by the investigator depend on (explicit or

implicit) comparison of different groups? – If so, did the research design include such comparison?

– Are the results claimed by the investigator plausibly explainable by random chance?

– Are the results the investigator claims to have found plausibly explainable by bias in selecting the study subjects?

– Are the results the investigator claims to have found plausibly explainable by bias in measuring important variables?

– Are the results the investigator claims to have found plausibly explainable by failure to control for confounding variables?

– If the conclusions pertain to cause and effect, does the study design provide assurance that the purported cause actually preceded the supposed outcome?


What one needs to know What did the researchers want to find out? What study design was used?

– Don’t take at face value the terms the researchers use to describe their own study design, as these may be misleading.

Was randomization used appropriately and successfully?

Is the population studied representative of an identifiable and interpretable population of patients?– For clinical application, is it representative of my patients?


Are subjects in the comparison group(s) similar, with respect to possibly confounding traits, to subjects in the case or experimental group?– Note that possibly confounding traits are variables related to

those being compared between groups -- health outcome(s) in clinical trials and cohort studies, past exposure(s) for case-control studies.

How are levels of exposure and end-points measured?– Are the measures valid and reliable?– Are they the same for control and case or experimental

groups?


Was statistical analysis interpretable and correct?– What do the tables and figures mean?

• Do they address the issues of scientific importance?

– Were hypothesis tests, p-values, and confidence intervals used to distinguish systematic from chance variation?

• Are they interpretable and correctly generated?

• Were they adjusted for potential confounders?

– What inferences do the authors make? • Are they logically correct?

• Do they involve assumptions that go beyond what the results imply?


How do the various results of the study fit together and with outside information and common sense.

What inferences can be made re – the study subjects, – the population from which the study subjects were

drawn, – other populations of interest?

For your presentation

May use powerpoint, transparencies, handouts. Should at least have 1 page handout of the outline of your

presentation. Mention every point that is in your written critique. Either give a quick overview of article pointing important tables

and graphs, then go through your critique. Or go through the article critiquing as you go along. Grades range from 60% to 100% (There have been lower

grades). Grade for written (½ of total) is usually very similar to oral (1/4).

Participation is different.

2 Secrets

Go to the issue where the paper was published and see if there was an accompanying editorial.(JAMA has a page number)

Go to the index of the journal for the 6-9 months after the paper was published and find the letters to the editor and the authors’ response.

Meningoccoccal Disease In Urbana-ChampaignCollege Students, January 1991 – April 1992

MALE FEMALE COMMUNITYCOLLEGE

FATALITY

19921991

DEC

JAN

FEB

MAR

APR

MAY

JUN

JUL

AUG

SEP

OCT

NOV

DEC

JAN

FEB

MAR

APR

MAY

19

19 19 20 20 20 22 18 19

Meningococcemia on Campus


CASE INVESTIGATION

– All 8 recovered isolates serogroup C

– All 9 cases patronized campus bars within 2 weeks prior to illness onset

– 8 of 9 cases patronized Bar A within 2 weeks prior to illness onset

Meningococcemia on CampusANALYTIC STUDIES

1:20 matched case-control study of last 7 cases

nasopharyngeal carriage studies of:– university health service

February 1992, May 1992, January 1993

– campus bar workers

March-May 1992, January 1993

Serogrouping of all N. meningitidis positive cultures, and multilocus enzyme electrophoresis (MEE) of all serogroup C isolates from cases and all carriage surveys


.0006 23.1 19/117 5/6 Bar A (any duration)

.002 16.726/117 5/6 14 hours/week

.034 --73/117 6/6 Any

Campus bar exposure prior two weeks

.012 7.8 23/117 4/6 Current cigarette smoking

Exact mid-pORControls Cases Exposure

CASE-CONTROL STUDY*

*Matched to cases by gender, school, year in school, calendar time of bar exposure. One matched set excluded at request of case.


0 (0.0) 5 (3.3)Others (n=152)

0 (0.0)37 (19.3)

1993

Recent bar patrons (n=192)

1 (0.2) 28 (6.1)Others (n=456)

1 (0.2)58 (14.1)

1992


2 (0.2)128 (10.6)Health center (n=1211)

0 (0.0)59 (28.0) 1993, 11 bars (n=211)

1 (7.1)6 (42.9) Bar B (n=14)

2 (9.1)7 (31.8) Bar A (n=22)

3 (3.5)32 (37.6) 1992, 5 bars (n=85)

Bar workers (n=296)

Case-Linked Strains All StrainsSource

Meningococcal Carriersnumber (%)

CARRIAGE/MEE RESULTS


.0122 1.15-3.25 1.94

Bar patronage in past 2 weeks

.0001 1.71-4.85 2.88 Type II without antibiotic

1.69-8.42 3.77 >15 Drinks vs. None

1.42-4.142.42 7-15 Drinks vs. None

1.19-2.031.56 1-6 Drinks vs. None

.0012

Alcohol consumption in past week

.31960.93-1.26 1.08 February vs. May

Wald

p-value95% CI for OR

Estimated ORFactor

Multiple Logistic Model For 1992 Health Center Carriage Data


0 (0.0) 5 (3.3)Others (n=152)

0 (0.0)37 (19.3)

1993


1 (0.2) 28 (6.1)Others (n=456)

1 (0.2)58 (14.1)

1992


2 (0.2)128 (10.6)Health center (n=1211)

0 (0.0)59 (28.0) 1993, 11 bars (n=211)

1 (7.1)6 (42.9) Bar B (n=14)

2 (9.1)7 (31.8) Bar A (n=22)

3 (3.5)32 (37.6) 1992, 5 bars (n=85)

Bar workers (n=296)

Case-Linked Strains All StrainsSource

Meningococcal Carriersnumber (%)

CARRIAGE/MEE RESULTS


CONCLUSIONS

Some cases in this outbreak may have followed transmission in Bar A.

Elevated levels of carriage of N. meningitidis in campus bar workers, and a campus bar environment, may participate in the spread of meningococcal disease among teens and young adults more generally.

Meningococcal Disease and Campus Bar Patronage

Do the results claimed by the investigator depend on (explicit or implicit) comparison of different groups?

–Yes. Inference of elevated risk associated with exposure to campus bars, and Bar A in particular, depends on claim of unusually high bar exposure among cases, which implies a reference group.–Inference of unusually high prevalence of organism in Bar A workers implies comparison to some reference “usual” prevalence in the same or a related population.

If so, did the research design include such comparison?

–Yes: explicit case-control design for bar exposure comparison, and explicit cross-sectional design using workers at other bars and McKinley clients for oropharyngeal prevalence surveys.

- Apparently no. The association of Bar A with each of the causal organism and the disease both generate quite low p-values, so neither association seems due to chance.

– But, one might argue that these p-values are misleadingly low because any one of a number of bars, restaurants or other venues might have been unusually associated with the organism or the disease. Perhaps an adjustment for multiple comparisons should have been made. However, even a conservative adjustment for as many as 25 venues would have left both associations statistically significant at the conventional 5% level.

– But, one might argue that the investigation of bar attendance was motivated by the apparently unusually consistent presence of that variable among many. Perhaps one should adjust for the number of variables in the case histories.

Are the results claimed by the investigator plausibly explainable by random chance?

However, beyond the statistical significance of either relationship individually is the further “coincidence” that both the organism and the disease were

not only each related to a campus bar,

but that

the two bars were the same bar.

Are the results claimed by the investigator plausibly explainable by random chance?

Case-control study:– Controls, matched to cases by school, year, and sex. – But one case dropped from analysis.– UI controls selected from University phone book. Many

listings inaccurate.– Appropriate Parkland controls would have been randomly

chosen 1991 2nd-year students living out-of-town. However, controls used were selected from list of 50 1992 2nd-year students living in town, selected by college administration.

– Only about 80% of targeted students were reached and gave interviews.

– The above matter only if the lost case was less likely or potential controls missed were more likely patrons of campus bars, especially Bar A, than other members of their respective groups.

Are the results the investigator claims to have found plausibly explainable by bias in selecting the study subjects?

Are the results the investigator claims to have found plausibly explainable by bias in selecting the study subjects?

Oropharyngeal prevalence survey:– McKinley patrons don’t perfectly mirror UI students. Those

with health problems, possibly taking antibiotics that lower carriage, were oversampled. (Though other infections may promote carriage.)

Bar employee survey: – Employees studied were from selected bars whose owners

cooperated, not from all bars. Even among employees of those bars, those studied were volunteers, and may not be representative of all workers from these same bars.

– A (very) few bar samples were lost in shipment to CDC. – Thus, maybe the real comparison was between McKinley

patrons who felt lousy and bar workers who felt like showing up at a meeting, whose samples made it to CDC.

These things matter only to the (unknown) extent that they are associated with carriage of N. meningitidis, particularly the case-linked strains.

Are the results the investigator claims to have found plausibly explainable by bias in measuring important variables?

Case-control study:– Though anonymity of responses was promised, controls

were not anonymous. “Underage” controls may have been reluctant to acknowledge bar patronage.

– Controls were asked to recall bar patronage as long as 5 months previously. While cases had their illness to fix memories, controls had no special reason to remember their precise behavior, and were asked about “usual” frequencies and “likely” patronage, rather than actual behavior.

– However, results from controls were comparable to other UI surveys, including the McKinley Health Center carriage surveys, and such bias would have been expected to affect all bars equally (since the study was roughly age-matched via year in school).


Case-control study: – The two week periods during which bar exposure was

examined did not sufficiently mirror the behavior of the disease under study. While there is no firm consensus on the actual duration of meningococcal disease, many believe that some cases incubate for over two weeks, so that the periods of study were too short. Further, exposure to Bar A was reported by one of the cases only <24 hours before symptom onset. This exposure should not have been counted, as it was unlikely to have caused the illness.

– But, one has to choose some reasonable period and apply it equally to cases & controls. Control of Communicable Diseases in Man, an authoritative reference, lists the incubation period for meningococcal meningitis as 2-10 days, commonly 3-4 days. By this definition, the two week period was liberal. The case patronizing Bar A the day before onset was queried in detail ten months later, and may also have been there earlier and forgotten.


Oropharyngeal prevalence surveys:– These were anonymous, and requested behavioral data were

recent.– However, samples from McKinley Health Center controls

and bar employees were handled by different lab techniques which can substantially affect results in culturing N. meningitidis.

• A comparability study was built into one of the McKinley health surveys, and showed no bias.

– Most critically, all the carriage studies were at the wrong time. All were done after 8 of the 9 cases had already occurred. Two of the three McKinley surveys and all of the bar employee surveys were done after much of the campus had been immunized. The critical 1992 surveys were done at McKinley at specific times in February and May, while the bar surveys were scattered from mid-march to mid-May.


But:

• Immunization has no documented effect on meningococcal carriage, and none was detected in our carriage studies.

• A case occurred in late January 1992, 2-3 weeks prior to the initial McKinley survey. An additional (fatal) case occurred in late April 1992, prima facie evidence that the causal organism was still present just prior to the final 1992 McKinley and bar worker surveys.

• The duration of meningococcal carriage is thought to be relatively brief (several weeks to several months), but carriage over years has been documented (although this was done prior to the period when individual substrains were tracked, so that continuing carriage of the same substrain has not been proven).

Are the results the investigator claims to have found plausibly explainable by failure to control for confounding variables?

Candidate confounders: Other variables closely related to both Bar A exposure and more strongly associated with disease than Bar A exposure.

• e.g. different forms and/or sites of close interpersonal contact between the cases or other common individuals.

– A modest but by no means exhaustive search was made, and nothing turned up.


If the conclusions pertain to cause and effect, does the study design provide assurance that the purported cause actually preceded the supposed outcome?

– No. Case-control and cross-sectional designs generally can’t do that.


Conclusions

– Ours: It is quite likely that exposure at Bar A caused several but by no means all of the cases in this cluster.

– Yours: Up to you.

Diltiazem and Mortality

Objective: To determine whether long term treatment with diltiazem in patients with prior MI would reduce total mortality (and recurrent cardiac events [MI or cardiac death]) Design: randomized, triple-blind clinical trial Eligible patients: aged 25 to 75 with documented MI

Therapy: Diltiazem or placebo, started 3-15 days after MI

Exclusions: indication for diltiazem or contraindications to diltiazem


Other outcomes assessed: one-year event rate for patients with 12 preselected covariates (24 subgroups).

Sample size estimation: 2000 total for 80% power to detect 25% reduction in total mortality from a baseline mortality of 20% and alpha of 0.05 (one sided).


Results

13,618 patients were eligible. 11,152 were excluded. 2,466 were enrolled (18%)

Baseline characteristics were similar (Table 1)

Follow up duration was 12-52 months (mean 25)

Compliance was 63% at 2 years for diltiazem and placebo (pretty good)

Only serious side effects were bradycardia and hypotension (4% in diltiazem group vs. 2% in placebo (also pretty good)


OUTCOMES

Total mortality was virtually identical (RR 1.02, CI 0.82-1.27)

Cardiac events RR 0.90, CI 0.74-1.08

Subgroup analysis was done using 1 year event rate. The only significant variable was pulmonary congestion, which had a significant interaction with treatment (p=0.0042, after Bonferroni p= 0.0042x12= 0.0504)


It seems that diltiazem was beneficial to patients without pulmonary congestion

(RR=0.77, CI 0.61-0.98)

with no correction for multiplicity,

and harmful in those with pulmonary congestion

(RR=1.41, CI 1.01-1.96).

It also seems that the higher the degree of pulmonary congestion, the higher the RR (dose response relationship).


Critique

No fault with design, randomization, blinding, bias, confounding (Typical of clinical trial in major journal)

(In that setting the problems are usually with conclusions, “leap of faith”, misplaced emphasis,contradictions, items not reported or glossed over.)

Authors claim the primary end points were total mortality and cardiac events.– But actually it was only total mortality and then later it

was changed. Also cardiac events is a less useful clinical endpoint than mortality.


Critique The authors claim that the 12 variables used as covariates

were predetermined.–But only 42% of the patients received ejection fraction measurement. Might this variable have been added after the fact?

Multiplicity 2 endpoints, 12 variables, 24 subgroups Why one year event rate? Did they try other time periods? Data monitoring committee The CI’s are very wide for both the groups with and without

pulmonary congestion.

Objectives of Subgroup Analysis

Support the main finding Check the consistency of main finding Address specific concerns re efficacy or safety in

specific subgroup Generate hypotheses for future studies

Inappropriate Uses of Subgroup Analysis

Rescue a negative trial Rescue a harmful trial Data dredging: find interesting results without a

prespecified plan or hypothesis

To Avoid Inappropriate Uses of Subgroup Analysis

Prespecify analysis plan Prespecify hypotheses to be tested based on prior

evidence Plan adequate power in the subgroups Avoid the previous pitfalls.

Problems with Subgroup Analysis

1. Low power

2. Multiplicity

3. Test for interaction

4. Comparability of the treatment groups maybe compromized

5. Over-interpretation


Critique

Data derived hypothesis

Power analysis– Should have been done using 2-sided p-values,

like the rest of the study.– They should have made provisions for a lower

outcome rate, as they should expect that.

The conclusion is not valid. The only conclusions that can be drawn are:


Diltiazem is not of major benefit (or harm) in preventing death or recurrent cardiac events after MI.

The effect of diltiazem at 1 year is different in patients with pulmonary congestion compared to those without pulmonary congestion.– The small number of events in these subgroups

and the multiplicity of comparisons done preclude a meaningful conclusion regarding the benefit of diltiazem in patients without pulmonary congestion.

– However, since the rules of evidence should be different with regard to harmful effect, we strongly caution against using diltiazem in patients with pulmonary congestion.”

Are we asking for too much?

Compare that conclusion to this one

Here is the <7 weeks category. But where is the >12 weeks category?

Went too far the other way?

Went too far the other way?

Statistically significant Dose response Biologic plausibility Consistent with animal studies Consistent with preexisting hypothesis Consistent with the hypothesis they

set out to test

So what can you do?

Be skeptical. Yes. Be cynical. Maybe. Is medical literature any better than political

commercials. Yes. Is it better than newspaper articles. Yes. In what way? The article itself almost always includes all

the facts you need to refute its own conclusion.

Medical Epidemiology Critiquing the Research Literature.

Documents

Transcript of Medical Epidemiology Critiquing the Research Literature.