1

Xiaomei Yao, R. Brian Haynes,

Nancy L. Wilczynski

Health Information Research Unit, McMaster University

Optimal sample sizes for testing search strategies for retrieving studies of treatment, diagnosis and prognosis

from MEDLINE

2

Background

• MEDLINE covers 4800 journals and contains over 13 million articles in life sciences.

• Clinicians and researchers need to retrieve the target articles quickly.

3

Background con’d

• Unique optimal search filters (“hedges”) have been developed, based on the 161 core clinical journals published in 2000 from MEDLINE. (http://www.ncbi.nlm.nih.gov/entrez/query/static/clinical.html).

• Determining optimal sample sizes for testing search strategies for retrieving studies of treatment, diagnosis and prognosis from MEDLINE.

4

Categories Types Ovid strategies Sensitivities (%) Specificities (%)

Treatment Best sensitivity clinical trial.mp. OR clinical trial.pt. OR random:.mp. OR tu.xs.

99.2 70.4

Best specificity randomized controlled trial.mp. OR randomized controlled trial.pt.

93.1 97.5

Optimization randomized controlled trial.pt. OR randomized.mp. OR placebo.mp.

95.8 95.0

Diagnosis Best sensitivity Sensitive:.mp. or diagnos:.mp. or di.fs.

98.6 74.3

Best specificity Specificity.tw. 64.6 98.4

Optimization Sensitive:.mp. or predictive value:.mp. or accuracy:.tw.

92.5 92.1

Prognosis Best sensitivity Incidence.sh. OR exp mortality OR follow-up studies.sh. OR prognos:.tw OR predict:.tw. OR course:.tw.

90.1 79.7

Best specificity prognos:.tw. OR first episode.tw. OR cohort.tw.

52.3 94.1

Optimization prognosis.sh. OR diagnosed.tw. OR cohort:.mp. OR predictor:.tw. OR death.tw. OR exp models, statistical

82.9 83.7

Table 1 - Best search strategies for 3 categories retrieval from MEDLINE

5

Method• The search terms were treated as “diagnostic

tests” and hand searching as a “gold standard”.• 49028 articles were categorized as “pass” or

“fail” studies for clinical topic areas.• Precision analysis was used to calculate the

sample size of “pass articles” needed to estimate sensitivity with 95% confidence intervals of ±0.01, ±0.02, ±0.025, ±0.03, ±0.04, ±0.05, ±0.10 (margin of error [ME]) .

6

Method con’d

• The lowest sensitivity among 3 strategy types was chosen.

• Random samples of journals for size 100, 50, 30, 20, 10, etc. were generated.

• The order of the 161 journals was ranked in terms of the number of “pass articles”.

• STATA 9.0 was used as the statistical software.

7

Results

• sample size calculations based on search sensitivity were shown in Table 2.

• Table 3 shows the numbers of “pass articles” for the 3 purpose categories in different random samples of journals from the 161 database.

8

Categories ME = ±0.01

ME = ±0.02

ME = ±0.025

ME = ±0.03

ME = ±0.04

ME = ±0.05

ME = ±0.10

Treatment 2468 617 395 275 155 99 25

Diagnosis 8786 2197 1406 977 550 352 88

Prognosis 9584 2396 1534 1065 599 384 96

Table 2 – Sample size needed for different margins of error (MEs)

9

Random journal subsets from 161 N for treatment (ME) N for diagnosis (ME) N for prognosis (ME)140 1425 (±0.02) 136 (±0.01) 159 (±0.01)

120 1200 (±0.02) 109 (±0.01) 150 (±0.01)

110 1228 (±0.02) 94 (±0.01) 153 (±0.01)

105 1027 (±0.02) 93 (±0.01) 110 (±0.01)

100 1019 (±0.02) 86 133 (±0.01)

85 1014 (±0.02) 83 128 (±0.01)

80 847 (±0.02) 70 125 (±0.01)

75 632 (±0.02) 53 73

70 402 (±0.025) 61 69

60 660 (±0.02) 72 72

50 696 (±0.02) 38 80

45 395 (±0.025) 21 30

40 374 (±0.03) 39 51

30 263 (±0.04) 24 24

25 251 (±0.04) 40 35

20 141 (±0.05) 28 22

15 191 (±0.05) 11 25

10 85 (±0.10) 11 15

5 37 (±0.10) 0 1

Table 3 - The Numbers of “pass articles” (N) with their achieved MEs for random journal subsets

10

Results con’d

• Based on table 2 and table 3, we can find the smallest journal subset for any ME.

• The scatterplot of various MEs against random journal sample sizes for treatment is shown in Figure.

• For the diagnosis and prognosis categories, we only can achieve a ME = ±0.10 because of low prevalence of “pass articles”.

11

Figure

0.0

1.0

2.0

3.0

4.0

5.0

6.0

7.0

8.0

9.1

.11

Mar

gin

of E

rror

s

0 10 20 30 40 50 60 70 80 90 100 110 120 130 140 150 160Random Journal Subsets

Treatment Category

12

Results con’d• We tested the previous optimal search

strategies for the treatment category in different random journal subsets (Table 4).

• 1, 2, and 6 top journals can achieve a ME = ±0.10, ±0.05, and ±0.025 for treatment, respectively.

• The performances of the strategy in 1, 2, and 6 top journal subsets were very similar to the result of the 161 journals.

13

Ovid search strategies

Sensitivities (%) of 100; 45; 15; 10 journal subsets,

161 journals (95% CI)

Specificities (%) of 100; 45; 15; 10 journal subsets,

161 journals (95% CI)

Precisions (%) of 100; 45; 15; 10 journal subsets,

161 journals (95% CI)

Accuracies (%) of 100; 45; 15; 10 journal subsets,

161 journals (95% CI)

Absolute differences:

100−161, 45−161, 15−161, 10-161

Absolute differences:

100−161, 45−161, 15−161, 10-161

Absolute differences:

100−161, 45−161, 15−161, 10-161

Absolute differences:

100−161, 45−161, 15−161, 10-161

Best sensitivityclinical trial.mp.

OR clinical trial.pt. OR

random:.mp. OR tu.xs.

99.0 (98.4, 99.6),99.0 (98.0, 100.0),98.4 (96.7, 100.2),

100.0 (100.0, 100.0),99.2 (98.7, 99.8)

71.5 (71.0, 71.9),66.6 (65.7, 67.5),69.3 (68.1, 70.5),75.1 (73.6, 76.6),70.4 (69.8, 70.9)

9.8 (9.2, 10.4),9.2 (8.4, 10.1),9.4 (8.1, 10.7),9.9 (7.9, 11.9),9.9 (9.3, 10.5)

72.3 (71.8, 72.8),67.7 (66.8, 68.5),70.2 (69.0, 71.3),75.8 (74.3, 77.3),71.3 (70.8, 71.8)

−0.2,−0.2,−0.8,0.8

1.1,−3.8,−1.1, 4.7

−0.1,−0.7,−0.5,0.0

1.0,−3.6,−1.1,4.5

Table 4 - Performance characteristics in random sample 100, 45, 15, 10 journal subsets, and 161

journals for treatment category

14

Best specificityrandomized controlled trial.mp. OR randomized controlled trial.pt.

93.8 (92.3, 95.3),94.4 (92.2, 96.7),90.1 (85.8, 94.3),

96.5 (92.6, 100.4),93.1 (91.5, 94.8)

97.7 (97.5, 97.9),97.3 (97.0, 97.6),97.6 (97.2, 98.0),97.4 (96.8, 97.9),

97.5 (97.3, 97.6)

56.0 (53.7, 58.4),54.5 (50.8, 58.3),55.0 (49.4, 60.5),50.0 (42.4, 57.7),

54.4 (52.0, 56.8)

97.6 (97.4, 97.7),97.2 (96.9, 97.5),97.4 (97.0, 97.8),97.3 (96.8, 97.9),

97.3 (97.1, 97.5)

0.7,1.3,

−3.0,3.4

0.2,−0.2,0.1,

−0.1

1.6,0.1,0.6,

−4.4

0.3,−0.1,0.1,0.0

Best optimization of sensitivity & specificityrandomized controlled trial.pt. OR randomized.mp. OR placebo.mp.

96.3 (95.2, 97.5),97.0 (95.3, 98.7),93.2 (89.6, 96.8),

98.8 (96.5, 101.1),95.8 (94.5, 97.1)

95.9 (95.7, 96.1),92.7 (92.2, 93.2),96.3 (95.8, 96.8),95.1 (94.4, 95.9),

95.0 (94.8, 95.3)

42.2 (40.2, 44.2),31.3 (28.7, 33.9),45.0 (40.1, 49.9),35.7 (29.6, 41.9),

38.5 (36.5, 40.5)

95.9 (95.7, 96.1),92.8 (92.4, 93.3),96.2 (95.7, 96.7),95.2 (94.5, 96.0),

95.0 (94.8, 95.3)

0.5,1.2,

−2.6,3.0

0.9,−2.3,1.3,0.1

3.7,−7.2,6.5,

−2.8

0.9,−2.2,1.2,0.2

Table 4 con’d

15

Conclusion

• Our sample size calculations suggest that search strategies calibrated in small journal subsets will be as robust as those calibrated in larger collections for some categories that have relatively much more “pass articles”, such as treatment category with 3.24% of “pass articles”.

16

Conclusion con’d

• The "top yielding journal" approach seems most economical.

• We will develop and test the new search strategies in small random and top journal subsets to see how close the estimates are.

17

Acknowledgments• The Hedges Team included Angela Eady,

Brian Haynes, Susan Marks, Ann McKibbon, Doug Morgan, Cindy Walker-Dilks, Stephen Walter, Stephen Werre, Nancy Wilczynski, and Sharon Wong.

• Chris Cotoi (our computer programmer) helped us generate a journal randomizer.