I Special Article

Guidelines for reading literature reviews

Andrew D. Oxman, MDGordon H. Guyatt, MD

One strategy for dealing with the burgeoningmedical literature is to rely on reviews of theliterature. Although this strategy is efficient,readers may be misled if the review does notmeet scientific standards. Therefore, guidelinesthat will help readers assess the scientific quali-ty of the review are proposed. The guidelinesfocus on the definition of the question, thecomprehensiveness of the search strategy, themethods of choosing and assessing the primarystudies, and the methods of combining theresults and reaching appropriate conclusions.Application of the guidelines will allow clin-icians to spend their valuable reading time onhigh-quality material and to judge the validityof an author's conclusions.

Une facon efficace de se tenir au courant de lalitterature mddicale toujours plus abondanteclest de se rabattre sur les revues gindrales. Maissi celles-ci ne se conforment pas aux normesscientifiques, elles risquent d'induire en erreur.11 est propose ici des lignes directrices afind'aider le lecteur d apprdcier la qualitd scientifi-que d'une revue gdndrale. Elles s"attachent dddterminer si la question y est bien dnoncde, larecherche bibliographique est complete, les tra-vaux retenus sont bien choisis et bien analysds,et les divers resultats sont mis en regard defacon d cerner des conclusions valables. Ensuivant ces lignes directrices le clinicien uti-lisera son temps precieux . bon escient.

From the departments of Clinical Epidemiology and Biostatisticsand of Medicine, McMaster University, Hamilton, Ont.

Dr. Guyatt is a career scientist of the Ontario Ministry ofHealth.

Reprint requests to: Dr. Gordon H. Guyatt, McMaster Universi-ty Health Sciences Centre, 3H7-1200 Main St. W, Hamilton,Ont. L8N3Z5

C linicians who are attempting to keepabreast of developments must find ways todeal with the exponentially expanding liter-

ature. Efficient strategies for finding and storingrelevant studiesl-6 and for discarding invalid orinapplicable studies7-'2 are available. However,processing the literature for an answer to a clinicalquestion remains time consuming, and it is notfeasible for clinicians to read all the primaryliterature for each of the myriad clinical issues thatconfront them daily.

One solution to this problem is the literaturereview or overview in which the primary researchrelevant to a clinical question is examined andsummarized. However, reviews, as well as primarystudies, must be read selectively and critically. Justas flawed methods in a study of diagnosis ortherapy may invalidate the results, an unscientificliterature review may come to incorrect conclu-sions. Authors of reviews do collect and analysedata from primary research, although this is some-times done subjectively and subconsciously. Thefundamental difference between a review and aprimary study is the unit of analysis, not thescientific principles that apply.

Five conflicting recommendations for manag-ing mild hypertension, quoted from the literature,are shown below.

* The available data . . . lead this reviewer toconclude that treatment of mild hypertension [90 to 104mm Hg] to achieve diastolic pressures below 90 mm Hgis the appropriate public health policy based on currentevidence.'3

* Most patients with diastolic blood pressure inthe 90 to 104 mm Hg range should be treated unlesscontraindications to drug therapy exist.... In certainpatients, vigorous dietary and behavioral modificationsmay be attempted before instituting or as an adjunct topharmacologic therapy.'4

* Non-drug measures are often effective for mildhypertension. The initial choice between thiazides andbeta-adrenoceptor blocking drugs often depends on thephysician's personal preference.... With care, the risks

CMAJ, VOL. 138, APRIL 15, 1988 697- For prescribing information see page 713

of antihypertensive therapy are considerably less thanthe benefits.15

* The benefits of drug treatment for patients withmild hypertension [diastolic blood pressure between 90and 105 mm Hg] remain unproven. Non-drug therapyhas also been insufficiently investigated.16

* At present, therefore, with the diuretic-basedtreatments principally studied in the previous trials,treatment of mild-to-moderate hypertension [diastolicblood pressure below 115 mm Hg] is of directly demon-strated value only if the stroke rate is high enough(perhaps due to age or cerebrovascular disease) forhalving it to justify the costs and trouble of therapy....Lipid-sparing antihypertensives might have more impor-tant effects on MI [myocardial infarction] than on stroke.But, in the trials reviewed, the size of the MI reductionremains uncertain [Rory Collins: unpublished observa-tions, 1987].

If one doesn't have some guidelines for assess-ing the reviews from which these recommenda-tions are taken, deciding which review to believe islike deciding which toothpaste to use. It is aquestion of taste rather than a question of science.

One does not have to look far to find otherexamples of important clinical questions for whichrecent reviews have come to different conclusions:Should clinicians avoid administering cortico-steroids because of concern about clinically impor-tant osteoporosis?17'18 What are the benefits tocritically ill patients of catheterizing the right sideof the heart?19'20 Should mild hypokalemia betreated aggressively?21'22

Clearly, the expertise of the author is not asufficient criterion of a review's credibility, sinceexperts reviewing the same topic often come todifferent conclusions. Nor is the prestige of thejournal or textbook in which the review is pub-lished a sufficient criterion. Recent surveys of themedical literature have found that the scientificquality of most published reviews, including thosein the most highly regarded journals, is poor.23-27

In this article we present a reader's guide toassessing research reviews. Similar guidelines havebeen suggested before, particularly in the psychol-ogy and social science literature.28-30 We focus onhow readers of the medical literature can decidewhether a review is worth reading and whether itsconclusions are to be believed. Our guidelines mayalso be of use to those planning to write a researchreview.

Guidelines

We have framed our guidelines as a series ofquestions (Table I). Before we address each item indetail some general comments are warranted. First,the questions are intended to be used to assessoverviews of primary studies on pragmatic ques-tions. Second, the term "primary studies" refers toresearch reports that contain original informationon which the review is based. Third, the intentionof the guidelines is to encourage efficient use of themedical literature and a healthy scepticism, not to

promote nihilism. Readers who apply these guide-lines will find that most published reviews havemajor scientific flaws.23-27 Indeed, surveys on thescientific adequacy of medical research reportshave found that most primary studies also havemajor scientific flaws.25

There is a need for improvement in thedesign, implementation and reporting of both re-views and primary studies. None the less, vastamounts of valuable information exist, and tomake informed decisions clinicians must use theresearch available. Although most published re-views do not provide strong support for theirconclusions, critical readers can discern usefulinformation and make their own inferences, whichmay or may not be the same as those of theauthors.

Were the questions and methods clearly stated?

When examining a review article readers mustdecide whether the review addresses a questionthat is relevant to their clinical practice or interests.They therefore require a clear statement of thequestions being addressed.

,able - uidehines '-: -issessi

; i-'e t5'8,1ij(S((

Tab 1e '- Fr" t e -F x Cir-t-t e .,.,

-. I.

698 CMAJ, VOL. 138, APRIL 15, 1988

I5. i.L.

Any causal question has three key elements:the population, the exposure or intervention andthe outcome. Examples of these elements in fivekey areas of clinical inquiry are presented in TableII. A clear statement of the question requiresexplicit specification of all three elements if thereader is to quickly decide whether the review isrelevant. If there is no clear statement of thequestions being addressed at the beginning of thereview the reader might as well stop. Fuzzyquestions tend to lead to fuzzy answers.

Many reviews address several questions; forexample, an article or a chapter in a textbook aboutacquired immune deficiency syndrome may reviewwhat is known about the cause, diagnosis, progno-sis, treatment and prevention of the disease. Suchreviews may be extremely helpful for readersseeking a broad overview. However, they tend toprovide little, if any, support for most of theinferences they make. Typically, an inference ispresented as a fact followed by one or morecitations. In this case the reader has no basis uponwhich to judge the strength or validity of theinferences without reading the articles that arecited. Readers seeking answers to specific clinicalquestions should not rely on reviews that addressbroad topics and encompass many questions.

In addition, an explicit statement of the meth-ods used for the research review is necessary forthe reader to make an informed assessment of thescientific rigour of the review and the strength ofthe support for the review's inferences. Unfortu-nately, this information is often lacking. In general,when a review does not state how something wasdone - for example, how it was decided whichprimary studies would be included - it is reason-able to assume that it was not done rigorously andthat a threat to the validity of the review exists.Readers looking for answers to specific clinicalquestions should seek reviews that clearly reportthe methods used. Without knowing the authors'methods the reader cannot distinguish statementsbased on evidence from those based on the opin-ions of the authors.

Were comprehensive search methods used tolocate relevant studies?

It is surprisingly difficult to locate all thepublished research in a particular area, even whenthe area is relatively circumscribed.3'-33 For exam-ple, Dickersin and associates33 found that a MED-LINE search yielded only 29% of the relevant trialson the prevention and treatment of perinatal hy-perbilirubinemia.

This problem is exacerbated by the fact thatsome of the relevant material may not even bepublished. Furthermore, the unpublished studiesmay be systematically different from those thathave appeared in peer-reviewed journals, not inthat their methods are flawed but in that theirresults are "negative". Research has suggested that

of two articles that use the same methods toinvestigate a question the study yielding positiveresults is more likely to be published than the oneyielding negative results.33-37 Research conductedby an agency that has an- investment in thetreatment being studied (such as a pharmaceuticalcompany with a new drug) may not even besubmitted for publication if its results are nega-tive. It thus behoves an author to try to determinethe extent of the "publication bias" in the areabeing reviewed.

Authors' search strategies vary widely, andexperts are no more likely than nonexperts to besystematic in their search.38 The more selective orhaphazard the authors' search for papers the morelikely it is that there will be bias in the review. Forexample, authors are likely to attend to papers thatsupport their preconceptions.

The reader needs assurance that all the perti-nent and important literature has been included inthe review. The more comprehensive the authors'search the more likely it is that all the importantarticles have been found. The reader should lookfor an explicit statement of the search strategiesused. Ideally, such strategies include the use of oneor more bibliographic databases (including a speci-fication of the key words and other aspects of thesearch strategies39), a search for reports that cite theimportant papers found through a database such asthe Science Citation Index, perusal of the refer-ences of all the relevant papers found and personalcommunication with investigators or organizationsactive in the area being reviewed (to make sureimportant published papers have not been missedand particularly to look for methodologicallyadequate studies that have not been published).

Were explicit methods used to deternine whicharticles to include in the review?

A comprehensive literature search will yieldmany articles that may not be directly relevant tothe question under investigation or that may be somethodologically weak that they do not contributevalid information. The authors must thereforeselect those that are appropriate for inclusion inthe review. When, as is often the case, this processis unsystematic, opportunities for bias develop.Thus, it is common to find two reviews of the samequestion in which different primary studies areincluded and for the choice of studies to contributeto different conclusions. For example, in two meth-odologically sophisticated and carefully conductedreviews on whether corticosteroids are associatedwith peptic ulcer the two teams of authors useddifferent criteria for choosing which studies wouldbe included in the review.40,41 This difference wasthe main reason for the remarkable result of thetwo reviews: diametrically opposed conclusionsabout whether or not the association exists.

The authors should specify how the articleswere chosen by referring to the three basic ele-

CMAJ, VOL. 138, APRIL 15, 1988 699

ments of primary studies: the population, theexposure or intervention and the outcome. Forexample, in assessing the effect of cholinomimeticagents in patients with dementia the authors couldspecify the criteria as follows.

* Population: patients with senile dementiain whom causes other than Alzheimer's diseasewere excluded.

* Intervention: oral administration of choli-nomimetic agents.

* Outcome: indicated by measurements ofboth memory and functional status.

Other methodologic criteria may be used toselect primary papers for review. In this examplethe authors may consider only studies in whichpatients were selected at random to receive thetreatment drug or a placebo and in which both theinvestigator and the patient were blind to alloca-tion.

Was the validity of the primary studies assessed?

Authors will come to correct conclusions onlyif they accurately assess the validity of the primarystudies on which the review is based. If all thestudies have basic flaws their conclusions may bequestionable even if their results are comparable.For example, if the literature on extracranial-intracranial bypass surgery for threatened strokewere reviewed before the results of a recentrandomized controlled trial42 were published, alarge number of studies with positive results but ofsuboptimal design and thus open to bias wouldhave been found. The appropriate conclusionwould have been that the procedure's effectivenesswas still open to question, despite the volume ofstudies with positive results; indeed, the subse-quent trial showed no benefit of surgical overmedical therapy.

Methodologic guidelines for studies of etiolo-gy,10'43 diagnosis,8 prognosis9 and therapy1144 areavailable. In a study of therapy one is interested inwhether the allocation to treatment was random,whether the subjects and investigators were blindto the allocation, and whether all the relevantoutcomes were monitored. Important aspects ofthe design and conduct of each primary studyshould be critiqued and the standard used in thesecritiques made explicit. Critiques should be report-ed in sufficient detail to allow readers to judge themethodologic quality of the primary studies. Al-though a study-by-study critique can be tedious,presentation of the methodologic assessment in atable may allow a rapid assessment of validity.Readers should be wary of any review that focuseson the results of studies without thoroughly dis-cussing the methods that were used to arrive at theresults.

When information about the methods or re-sults has been omitted from a published report theauthors of a review can contact the writers of thereport to obtain the missing information. A review

is strengthened if the authors have discussed theimplications of missing information and haveattempted to collect the relevant data.

Was the assessment of the primary studiesreproducible and free from bias?

Expert assessment of primary research studiesgenerally results in a level of disagreement that isboth extraordinary and distressing. For example,correlations measuring agreement about the de-cision to publish or not publish primary researchstudies are almost always less than 0.5 and averageabout 0.3,2845,46 a level not much higher than onewould expect to achieve by chance.

Not only do assessments lack reproducibility,but also they are often biased. In one study Petersand Ceci47 resubmitted previously published arti-cles from respected institutions after they substitut-ed the names of the authors and the institutionswith fictitious names. Mahoney35 submitted anarticle to different referees, varying the resultswithout altering the methods. These studies foundthat the articles that came from respected institu-tions and reported positive results were morereadily accepted. Furthermore, in Peters and Ceci'sstudy many of the articles were rejected because of"serious methodological flaws", and in Mahoney'sstudy the article was judged as having weakermethods when it described negative results.

It is even possible for authors to disagree onthe results of a study. Numerous conflicting re-views have been reported in which an author whofavoured a particular treatment classified the pri-mary study as positive, whereas an author who didnot favour the treatment classified the study asnegative. For example, Miller48 found five reviewsthat compared drug therapy plus psychotherapywith drug therapy alone for psychiatric patients. Ofthe 11 studies cited in two or more of the reviewsthe results of 6 were interpreted as positive in atleast one review and as negative in at least oneother.

Problems with reproducibility and bias canaffect two stages of the review process: the de-cision about which papers to include and judge-ment of the quality of the papers included. Suchproblems can be minimized if explicit criteria areused. However, many of the criteria will requireconsiderable judgement of the author of a review.In an example we used earlier one of the criteriafor inclusion in a review of treatment with cholino-mimetic agents for Alzheimer's disease was adefinition of the population as patients with seniledementia in whom causes other than Alzheimer'sdisease were excluded. Is a statement in the textsuch as "standard methods for diagnosing Alz-heimer's disease were used" adequate or does onerequire details of how other causes of dementiawere ruled out?

Explicit criteria offer little advantage if theycannot be reproduced by other authors. Ideally, all

700 CMAJ, VOL. 138, APRIL 15, 1988

the potential primary studies should be assessedfor inclusion by at least two authors, each blind tothe other's decision, and the extent of agreementshould be recorded. Reproducibility should bequantified with a statistical measure that quanti-tates agreement above and beyond that whichwould have occurred by chance, such as an intra-class correlation coefficient49 or a x statistic.50 Asimilar process should be used to assess thereproducibility of the criteria used to determine thevalidity of the primary studies.

Even if the criteria for study inclusion orvalidity can be reproduced there is no guaranteethat bias has not intruded. For example, if theauthors believe that a new treatment works theymay apply inclusion criteria by which studies withnegative results are systematically excluded; thevalidity of such studies that are included may bejudged more harshly. What can be done to preventthis sort of bias?

In randomized controlled trials bias is avoidedif both the patients and the clinicians are blind towhether the patients are taking the active drug or aplacebo. In an assessment of primary studies themajor possible sources of bias are related to theauthors, their institution and the results. However,one can assess the content and quality of a studythrough its methods without knowing this infor-mation; the relevant sections of the paper cansimply be "whited out" so that the reviewers areblind to the authors' institutions and results. De-cisions about study inclusion and validity ideallyshould be made under these conditions. Thisadded precaution will strengthen the review.

Was variation in the findings of the relevantstudies analysed?

Authors of reviews are certain to encountervariability in the results of studies addressing thequestion of interest. Indeed, if all the results ofprimary research were the same a review articlewould probably not be necessary. It is the authors'task to try to explain this variability.

Possible sources of variability are the studydesign, chance and differences in the three basicstudy components (the population, the exposure orintervention and the outcome).5" If randomizedcontrolled trials, before-and-after studies andstudies with historical controls are all included in areview, and if the randomized controlled trialsconsistently show results that differ systematicallyfrom those of the other studies, the study designprobably explains the differences. For example,Sacks and colleagues52 found that randomizedcontrolled trials consistently show smaller effectsthan studies that use historical controls.

A second explanation for differences in studyresults is chance. Even if two investigations usecomparable methods and the true size of theeffects is identical the play of chance will lead toapparent differences in the size. If the samples are

small, chance alone may lead to apparently largedifferences in the size of the effects. Some trials ofacetylsalicylic acid (ASA) in patients with transientischemic attacks have shown a trend in favour of aplacebo, whereas others have shown reductions inrisk of up to 50% with ASA.53 However, theconfidence intervals, which represent the upperand lower limits of the size of the effects consistentwith the observed results, overlap. Thus, althoughthe apparently discrepant results might suggesthypotheses for testing in subsequent studies, theyare all consistent with a reduction in risk ofbetween 15% and 30% with ASA.

In other instances differences in study resultsmay be so large that they cannot be explained bychance. The authors must therefore look to differ-ences in the population, exposure or interventionand outcome. In our example of cholinomimeticagents in patients with Alzheimer's disease thestudies with negative results may have included alarger number of severely affected patients thanthe studies with positive results. One might thenassume that the intervention works only in mildlyaffected patients. However, the intervention mayhave differed - that is, higher doses or differentagents may have been given in the studies withpositive results. Finally, the tests used to determinememory and functional status may have beendifferent; some tests are more responsive tochanges in patient status. Horwitz51 has document-ed many ways in which differences in the methodsof randomized controlled trials can lead to differ-ing results.

Readers of a review should be alert to whetherthese five explanations for differing study resultshave been considered and should be scepticalwhen differences are attributed to one explanationwithout adequate consideration of the others.

Were the findings of the primary studies combinedappropriately?

Meta-analysis (the use of several statisticaltechniques to combine the results of differentstudies) is becoming increasingly popular, especial-ly as a method of combining results from random-ized controlled trials. However, it remains contro-versial, and clinical readers cannot be expected tojudge the merits of a particular statistical techniqueused by the authors of a meta-analysis. Neverthe-less, there are issues that clinical readers canaddress.

The crudest form of meta-analysis, in whichthe number of studies with positive results iscompared with the number of those with negativeresults, is not satisfactory. This "vote count" ig-nores the size of the treatment effects and thesample sizes of each study. The most satisfactorymeta-analysis yields two pieces of information: themagnitude of the overall treatment effect and thelikelihood that this effect would have occurred bychance if the true effect were zero. The former may

CMAJ, VOL. 138, APRIL 15, 1988 701

be expressed as a percentage risk reduction, thelatter as a p value.

The primary advantage of meta-analysis isthat the results of different studies can be com-bined accurately and reliably to determine the bestestimate of the average magnitude of the effects ofthe exposure or intervention of interest. Before theresults are combined, however, one should consid-er whether it is appropriate to aggregate across thestudies. Study designs, or the three basic studyelements, may differ sufficiently that a statisticalcombination of the results does not make sense.Meta-analysis can be used to analyse the variationin study results to generate or test hypothesesabout the source of the differences. However, it ison strongest ground when the methods of theprimary studies are similar and the differences inthe study results can be explained by chance.

Reviews in which the results are not statisti-cally combined should state explicitly the basis forthe conclusions and should attempt to explain theconflicting results. Readers should beware of re-views that conclude that there is no effect withouthaving considered the studies' power to detect aclinically important effect. When several studies donot show a significant difference there is a tenden-cy for reviewers who have not used meta-analysisto conclude that there is no effect even whenstatistical aggegation demonstrates otherwise.Cooper and Rosenthal54 demonstrated this experi-mentally by assigning reviewers at random toeither use or not use meta-analysis to combine theresults of several studies, including some that didnot show significant results. Another investigatormade the same observation when he polled re-searchers who had conducted trials of tamoxifencitrate as adjuvant therapy for breast cancer (RoryCollins: personal communication, 1987). Most ofthe researchers concluded from the available infor-mation that tamoxifen did not produce a longerdisease-free interval; however, statistical aggrega-tion of all the available results demonstrated aclinically important, statistically significant effect.

It is important to remember that all the otherguidelines we have discussed still apply whetheror not the authors of a review have used meta-analysis.

fabie W. Guidelines tot assessaring 4-h-ausal rterence

T 4*t f{ eIatior-: -ti;-.

*i itla 1. ')et ee.rkAer1 ( _;t73, i;;

* 1, .. :.ia t o){ )tFro mrTz

Fise Ci:sponse re...latr..

--ee .elating d'nterTiadiffere r-, .tlav t;eat CL

SStl;;i- ic!ir<¢ ven]tloiais.: e. i.rop.esitC{D!SEro t -s:st^SEti VZ;

Were the reviewers' condusions supported by thedata cited?

Whether or not authors have used meta-anal-ysis, the results of individual primary studiesshould be reported in sufficient detail that readersare able to critically assess the basis for theauthors' conclusions. The method of presentingindividual study summaries will depend on thequestion addressed. For questions of treatmenteffectiveness and prevention the size of the effectsand its confidence interval give the key informa-tion. Reviews of diagnostic tests may providesensitivities, specificities and likelihood ratios (andtheir confidence intervals).8 Survival curves mayefficiently depict the main results of studies ofprognosis.

With questions of etiology and causation forwhich randomized controlled trials are not avail-able the authors can evaluate the evidence withcriteria for causal inference. Variations of thesecriteria have been presented by several investiga-tors,10'44'55'56 but common ingredients include thesize and consistency of the association between thecausal agent and the outcome and the necessity fordemonstrating the appropriate temporal relation.Our version of these criteria is presented in TableIII. The authors' comments on each of these criteriashould, of course, refer directly back to the data inthe primary studies cited.

Conclusion

A literature review is a scientific endeavour,and, as with other scientific endeavours, standardsare available for conducting the review in such away that valid conclusions are reached. Just asreaders of the clinical literature who are unable tocritically appraise the methods of primary studiesmay arrive at incorrect conclusions, readers whoare unable to assess the scienti'fic quality of areview are apt to be misled. We have offered eightguidelines for readers interested in answering aclinical question relevant to their everyday prac-tice. Application of these guidelines will allowreaders to quickly discard review articles that areirrelevant or scientifically unsound, to detect po-tential sources of bias and to be confident ofconclusions made from a systematic evaluation ofthe available research.

We thank Drs. Geoff Norman, David Streiner, David L.Sackett and Brian Hutchison, and Professor Mike Gentfor their help in developing the guidelines.

This work was supported in part by the OntarioMinistry of Health.

References

1. Haynes RB, McKibbon KA, Fitzgerald D et al: How to keepulp with the medical literature: I. Why try to keep up andhow to get started. Ann Intern Med 1986; 105: 149-153

702 CMAJ, VOL. 138, APRIL 15, 1988

2. Idem: How to keep up with the medical literature: II.Deciding which journals to read regularly. Ibid: 309-312

3. Idem: How to keep up with the medical literature: III.Expanding the number of journals you read regularly. Ibid:474-478

4. Idem: How to keep up with the medical literature: IV.Using the literature to solve clinical problems. Ibid: 636-640

5. Idem: How to keep up with the medical literature: V.Access by personal computer to the medical literature. Ibid:810-824

6. Idem: How to keep up with the medical literature: VI. Howto store and retrieve articles worth keeping. Ibid: 978-984

7. Department of Clinical Epidemiology and Biostatistics,McMaster University Health Sciences Centre: How to readclinical journals: I. Why to read them and how to startreading them critically. Can Med Assoc J 1981; 124: 555-558

8. Idem: How to read clinical journals: II. To learn about adiagnostic test. Ibid: 703-710

9. Idem: How to read clinical journals: III. To learn the clinicalcourse and prognosis of disease. Ibid: 869-872

10. Idem: How to read clinical journals: IV. To determineetiology or causation. Ibid: 985-990

11. Idem: How to read clinical journals: V. To distinguishuseful from useless or even harmful therapy. Ibid: 1156-1162

12. Idem: How to read clinical journals: VI. To learn about thequality of clinical care. Can Med Assoc J 1984; 130: 377-382

13. Labarthe DR: Mild hypertension: the question of treatment.Ann Rev Public Health 1986; 7: 193-215

14. Haber E, Slater EE: High blood pressure. In Rubenstein M,Federman DD (eds): Scientific American Medicine, 9th ed,Sci Am, New York, 1986: sect 1, VII1-VII29

15. Risks of antihypertensive therapy [E]. Lancet 1986; 2: 1075-1076

16. Sacks HS, Chalmers TC, Berk AA et al: Should mildhypertension be treated? An attempted meta-analysis of theclinical trials. Mt Sinai J Med 1985; 52: 265-270

17. Guyatt GH, Webber CE, Mewa AA et al: Determiningcausation - a case study: adrenocorticosteroids and os-teoporosis. J Chronic Dis 1984; 37: 343-352

18. Baylink DJ: Glucocorticoid-induced osteoporosis. N Engl JMed 1983; 309: 306-308

19. Swan HJC, Ganz W: Hemodynamic measurements inclinical practice: a decade in review. J Am Coll Cardiol1983; 1:103-113

20. Robin ED: The cult of the Swan-Ganz catheter: overuseand abuse of pulmonary flow catheters. Ann Intern Med1985; 103: 445-449

21. Harrington JT, Isner JM, Kassirer JP: Our national obsessionwith potassium. Am J Med 1982; 73: 155-159

22. Kaplan NM: Our appropriate concern about hypokalemia.Am JMed 1984; 77: 1-4

23. Mulrow CD: The medical review article: state of thescience. Ann Intern Med 1987; 106: 485-488

24. Sacks HS, Berrier J, Reitman D et al: Meta-analyses ofrandomized controlled trials. NEngliJMed 1987; 316: 450-455

25. Williamson JW, Goldschmidt PG, Colton T: The quality ofmedical literature. An analysis of validation assessments. InBailar JC III, Mosteller F (eds): Medical Uses of Statistics,NEJM Bks, Waltham, Mass, 1986: 370-391

26. Halvorsen KT: Combining results from independent inves-tigations: meta-analysis in medical research. Ibid: 392-416

27. Oxman AD: A Methodological Framework for ResearchOverviews, MSc thesis, McMaster U, Hamilton, Ont, 1987:23-31, 98-105

28. Light RJ, Pillemer DB: Summing Up: the Science ofReviewing Research, Harvard U Pr, Cambridge, Mass, 1984

29. Jackson GB: Methods for integrative reviews. Rev Educ Res1980; 50: 438-460

30. Cooper HM: The Integrative Research Review': a Systematic

Approach, Sage, Beverly Hills, Calif, 198431. Glass GV, McGaw B, Smith ML: Meta-Analysis in Social

Research, Sage, Beverly Hills, Calif, 198132. Poynard T, Conn HO: The retrieval of randomized clinical

trials in liver disease from the medical literature. ControlledClin Trials 1985; 6: 271-279

33. Dickersin K, Hewitt P, Mutch L et al: Perusing theliterature: comparison of MEDLINE searching with a peri-natal trials database. Ibid: 306-317

34. Simes RJ: Publication bias. The case for an internationalregistry of clinical trials. J Clin Oncol 1986; 4: 1529-1541

35. Mahoney MJ: Publication prejudices: an experimental studyof confirmatory bias in the peer review system. Cognit TherRes 1977; 1: 161-175

36. Devine EC, Cook TD: Effects of psycho-educational inter-vention on length of hospital stay: a meta-analytic reviewof 34 studies. In Light RJ (ed): Evaluation Studies ReviewAnnual, 8th ed, Sage, Beverly Hills, Calif, 1983: 417-432

37. Simes RJ: Confronting publication bias: a cohort design formeta-analysis. Stat Med 1987; 6: 11-29

38. Cooper HM: Literature searching strategies of integrativeresearch reviewers: a first survey. Knowledge 1986; 8: 372-383

39. Huth EJ: Needed: review articles with more scientific rigor.Ann Intern Med 1987; 106: 470-471

40. Messer J, Reitman D, Sacks HS et al: Association ofadrenocorticosteroid therapy and peptic-ulcer disease. NEnglJ Med 1983; 309: 21-24

41. Conn HO, Blitzer BL: Nonassociation of adrenocortico-steroid therapy and peptic ulcer. N Engl J Med 1976; 294:473-479

42. EC/IC Bypass Study Group: Failure of extracranial-intra-cranial arterial bypass to reduce the risk of ischemic stroke.Results of an international randomized trial. N Engl J Med1985; 313: 1191-1200

43. Hill AB: Principles of Medical Statistics, 9th ed, Lancet,London, 1971: 312-320

44. Chalmers TC, Smith H, Blackburn B et al: A method forassessing the quality of a randomized controlled trial.Controlled Clin Trials 1981; 2: 31-49

45. Bailar JC III, Patterson K: Journal peer review: the need fora research agenda. In Bailar JC III, Mosteller F (eds):Medical Uses of Statistics, NEJM Bks, Waltham, Mass,1986: 349-369

46. Marsh HW, Ball S: Interjudgmental reliability of reviews forthe Journal of Educational Psychology. J Educ Psychol1981; 73: 872-880

47. Peters DP, Ceci SJ: Peer-review practices of psychologicaljournals: the fate of published articles, submitted again.Behav Brain Sci 1982; 5: 187-255

48. Miller TI: The Effects of Drug Therapy on PsychologicalDisorders, PhD dissertation, U of Colorado, Boulder, 1977

49. Shrout PE, Fleiss JL: Intraclass correlations: uses in assess-ing rater reliability. Psychol-Bull 1979; 86: 420-428

50. Cohen J: A coefficient of agreement for nominal scales.Educ Psychol Meas 1960; 20: 37-46

51. Horwitz RI: Complexity and contradiction in clinical trialresearch. AmJMed 1987; 82: 498-510

52. Sacks H, Chalmers TC, Smith H: Randomized versushistorical controls for clinical trials. Am J Med 1982; 72:233-240

53. Sze PC, Pincus M, Sacks HS et al: Antiplatelet agents insecondary stroke prevention. A meta-analysis of the avail-able randomized control trials [abstr]. Clin Res 1986; 34:385A

54. Cooper HM, Rosenthal R: Statistical versus traditionalprocedures for summarizing research findings. Psychol Bull1980; 87: 442-449

55. Susser M: Reviews and commentary: the logic of Sir KarlPopper and the practice of epidemiology. Am J Epidemiol1986; 124: 711-718

56. Guyatt GH, Newhouse MT: Are active and passive smok-ing harmful? Determining causation. Chest 1985; 88: 445-451

CMAi, VOL. 138, APRIL 15, 1988 703