COCHRANE METHODS · Cochrane Methods Sally Hopewell, Co-Scientifi c Editor and Technical Editor UK...

COCHRANE METHODSEditors: Sally Hopewell, Mike Clarke and Julian PT Higgins

www.thecochranelibrary.comISSN 2044–4702

September 201 1

Cochrane Methods

Sally Hopewell, Co-Scientifi c Editor and Technical EditorUK Cochrane Centre National Institute for Health ResearchSummertown Pavilion Middle WayOxford OX2 7LG UK Tel: +44 1865 516300 [email protected]

Registered Methods Groups Adverse Eff ects Methods GroupYoon Loke, ConvenorSchool of Medicine, Health Policy and PracticeUniversity of East AngliaNorwich, NR4 7TJUKTel: +44 1603 [email protected]@cochrane.org

Applicability and Recommendations Methods GroupHolger Schünemann, ConvenorClinical Epidemiology and BiostatisticsMcMaster University1200 Main Street WHamilton Ontario L8N 3Z5CanadaTel: +1 905 5259140 ext 24931 [email protected]

Bias Methods GroupDavid Moher, ConvenorOttawa Health Research Institute501 Smyth RoadBox 208Ottawa, Ontario K1H 8L6CanadaTel: +1 613 7377600 ext [email protected]/bmg

Comparing Multiple Interventions Methods GroupTianjing Li, ConvenorDepartment of EpidemiologyJohns Hopkins Bloomberg School of Public Health615 North Wolfe Street, Mail Room W5010Offi ce: SPH, E6006Baltimore MD 21205USA+1 410 502 [email protected]

Campbell and Cochrane Economics Methods GroupJanice LeggeInstitute of Health and SocietyNewcastle UniversityBaddiley-Clark BuildingRichardson RoadNewcastle upon Tyne NE2 4AXUK+44 191 222 [email protected]

Campbell and Cochrane Equity Methods GroupErin Ueffi ng, Co-ordinatorCentre for Global HealthInstitute of Population HealthUniversity of Ottawa207-1 Stewart StreetOttawa Ontario K1N 6N5CanadaTel: +1 613 5625800 ext 1963erin.ueffi [email protected]@cochrane.org

Individual Participant Data Meta-analysis Methods GroupLarysa Rydzewska, Co-ordinatorMeta-analysis GroupMRC Clinical Trials Unit222 Euston RoadLondon NW1 2DAUKTel: + 44 207 [email protected]/cochrane/ipdmg

Information Retrieval Methods GroupAlison Weightman, Co-ConvenorHead of Library Service DevelopmentInformation ServicesCardiff UniversityCardiff CF24 0DEUK Tel: +44 2920 875693 weightmanal@cardiff [email protected]

Non-Randomised Studies Methods GroupBarney Reeves, ConvenorBristol Heart InstituteUniversity of BristolLevel 7, Bristol Royal Infi rmaryMarlborough StreetBristol BS2 8HWUKTel: +44 117 9283143 [email protected]

Patient Reported Outcomes Methods GroupDonald Patrick, ConvenorDepartment of Health ServicesSeattle Quality of Life Group\Center for Disability Policy and Research at the University of WashingtonBox 359455Seattle Washington 98195-9455USATel: +1 206 [email protected]

Julian Higgins, Co-Scientifi c EditorMRC Biostatistics Unit Institute of Public HealthUniversity of Cambridge Robinson WayCambridge CB2 0SRUKTel: +44 1223 [email protected]

Mike Clarke, Co-Scientifi c EditorAll-Ireland Hub for Trials Methodology ResearchCentre for Public HealthQueen’s University BelfastInstitute of Clinical Sciences, Block B, Royal Victoria HospitalGrosvenor RoadBelfast BT12 6BANorthern [email protected]

Prognosis Methods GroupKatrina Williams, ConvenorSydney Children’s HospitalUniversity of New South WalesSydney Children’s Community Health CentreCnr Avoc and Barker StreetRandwick, NSW 2031AustraliaTel: +61 2 93828183katrina.williams@sesiahs.health.nsw.gov.auwww.prognosismethods.cochrane.org/en/index.html

Prospective Meta-analysis Methods GroupLisa Askie, ConvenorNHMRC Clinical Trials CentreUniversity of SydneyLocked Bag 77Camperdown NSW 1450AustraliaTel: +61 2 [email protected]@cochrane.org

Qualitative Research Methods GroupJane Noyes, ConvenorCentre for Health Related ResearchSchool of Healthcare SciencesCollege of Health and Behavioural SciencesUniversity of WalesBangorWales LL57 2EFUKTel: +44 1248 [email protected]/cqrmg

Screening and Diagnostic Tests Methods GroupConstantine Gatsonis, ConvenorCenter for Statistical StudiesBrown University, Box G-HProvidence, RI 02912USA Tel: +1 401 8639183 [email protected]

Statistical Methods GroupDoug Altman, ConvenorCentre for Statistics in MedicineWolfson College University of OxfordLinton RoadOxford, OX2 6UDUKTel: +44 1865 284401 [email protected]

The opinions expressed in the newsletter do not necessarily refl ect the opinion of the editors, The Cochrane Collaboration, or anyone other than the authors of the individual articles. Cochrane Methods should be cited as: Hopewell S, Higgins J, Clarke M (editors). Cochrane Methods. Cochrane DB Syst Rev 2011 Suppl 1:1–42.

Editors

Cochrane MethodsSeptember 2011

Table of ContentsWelcome to CochraneMethods 1

Articles 2Update on the MECIR project: Methodological Expectations for Cochrane Intervention Reviews 2The role of the new Cochrane Methods Co-ordinator 2The Cochrane Risk of Bias tool for RevMan 5.1 3PROSPERO – the new international prospective register of systematic reviews 4

PublishedMethodological Research – structured abstracts and commentaries 6Bias due to changes in specified outcomes during the systematic review process 6Improving the interpretation of quality of life evidence in meta-analyses: the application of minimal important difference units 7Inclusion of methodological filters in searches for diagnostic test accuracy studies misses relevant studies 8Reporting performance of prognostic models in cancer: a review 9Indirect comparisons: a review of reporting andmethodological quality 10Reporting ofmethodologic information on trial registries for quality assessment: a study of trial records retrieved from theWHO

search portal 11Updating systematic reviews: an international survey 12A systematic examination of the citation of prior research in reports of randomized, controlled trials 13Interpretation of random-effects meta-analyses 14Confidence intervals for random-effects meta-analysis and robustness to publication bias 14

Empirical Studies within the Cochrane Collaboration 16Review production in The Cochrane Collaboration – where is it happening and why? 16Risk of Bias tool evaluation: summary results from focus groups and online surveys 19Consumer involvement in The Cochrane Collaboration 20Developing and evaluating ‘Summary of findings’ tables for Cochrane reviews – two studies 21Is multivariate meta-analysis a solution for reducing the impact of outcome reporting bias in systematic reviews? 23How do tests impact on patient health? Development of an explanatory framework 24Cochrane Methodology Review Group 27

Information from theMethods Groups 28Registered Groups 29Cochrane Adverse Effects Methods Group 29Cochrane Bias Methods Group 29Cochrane Comparing Multiple Interventions Methods Group 30Campbell and Cochrane Economics Methods Group 31Campbell and Cochrane Equity Methods Group 32Cochrane Individual Participant Data Meta-analysis Methods Group 33Cochrane Information Retrieval Methods Group 33Cochrane Patient Reported Outcomes Methods Group 34Cochrane Prognosis Methods Group 35Cochrane Prospective Meta-Analysis Methods Group 35Cochrane Qualitative Research Methods Group 36Cochrane Screening and Diagnostic Tests Methods Group 36Cochrane Statistical Methods Group 37

PossibleMethodsGroup 37Cochrane Agenda Setting and Prioritisation Methods Group 37

Campbell CollaborationMethods Groups (C2) 39Future Meetings 40

19th Colloquium of The Cochrane Collaboration 40Clinical Trials Methodology Conference 2011 40

Welcome to CochraneMethods

Welcome to the second issue of Cochrane Methods, the official annual newsletter for methodological issues within The CochraneCollaboration. Many of you will have seen and contributed to previous issues of the Cochrane Methods Groups Newsletter which wasin circulation since 1997. Last year, we redesigned and renamed the newsletter with the aim of giving greater prominence to the workof CochraneMethods Groups within the Collaboration, and to help raise their profile more widely. We hope you enjoyed the new look.

The Cochrane Collaboration is an international, independent, not-for-profit organization of more than 25,000 contributors frommorethan 100 countries, dedicated to making up-to-date, accurate information about the effects of health care readily available worldwide.Its contributors work together to produce Cochrane reviews of healthcare interventions, diagnostic tests andmethodology, publishedonline in The Cochrane Library. These help providers, practitioners, patients and the public make informed decisions about theirown health care and that of others. The role of the Cochrane Methods Groups is primarily to provide policy advice to The CochraneCollaboration on how the validity and precision of Cochrane reviews can be improved. In addition, Methods Groupsmay also carry outadditional tasks such as providing training, peer review and specialist advice, contributing to software developments, or conductingresearch aimed at improving the quality of Cochrane reviews. You can read more about individual Methods Groups on pages 27 to 38.

This issue of Cochrane Methods focuses on some challenging issues facing the methodology of Cochrane reviews and other typesof systematic review. We describe the ongoing work of the MECIR (Methodological Expectations of Cochrane Intervention Reviews)project which aims to develop minimum standards for review methodology. We report on the launch of PROSPERO, the internationalprospective register of systematic reviews which aims to capture key information about the design of a systematic review at theprotocol stage. We also welcome Jackie Chandler, the new Cochrane Methods Co-ordinator.

This year we have also introduced a new section which presents a collection of full length feature articles highlighting some ofthe methodological research currently being carried out within The Cochrane Collaboration. These include a study describing thedevelopment and evaluation of the ‘Summary of findings’ tables in Cochrane reviews, a study presenting the results of an evaluationof the ‘Risk of bias’ tool and an examination of review production within The Cochrane Collaboration. If you would like to publishfindings from a study you have done in the next issue of CochraneMethods, then please contact us.

As in previous issues, we include a series of structured abstracts and commentaries on topical methodological issues. For example,this year we highlight research into the bias that may arise when changes are made to specified outcomes during the systematicreview process and a study looking at the reporting and methodological quality of systematic reviews containing data from indirectcomparisons. We also include a commentary looking at the interpretation of random-effects meta-analysis in systematic reviews andanother examining a new method for calculating confidence intervals for random-effects meta-analysis that seeks to minimize theeffects of publication bias.

We are, as ever, very grateful to the many people who have contributed to Cochrane Methods. We should also like to thank TheCochrane Collaboration and the UK Cochrane Centre (part of the National Institute for Health Research) for providing resources toproduce it. Finally, we should verymuchwelcome your comments on the new look newsletter and your suggestions for future content.

Sally Hopewell, Julian Higgins andMike Clarke (Editors of CochraneMethods)

Copyright �c 2011 The Cochrane Collaboration. Cochrane Methods. Cochrane DB Syst Rev 2011 Suppl 1: 1–40Published by John Wiley & Sons, Ltd. 1

ArticlesUpdate on theMECIRproject: MethodologicalExpectations forCochrane InterventionReviewsJulian Higgins, Rachel Churchill, DavidTovey, Toby Lasserson and JackieChandler

Correspondence to:[email protected] Biostatistics Unit, Institute of PublicHealth, Cambridge, UK.

IntroductionSome empirical evidence, and plenty ofanecdotes, indicate that Cochrane in-tervention reviews vary substantially intheir methodological sophistication. Akey role of the Editor-in-Chief of TheCochrane Library is to work to improvethe quality and consistency of our re-views. The MECIR project was initiatedin September 2010 as a major contri-bution to this aim, and builds on ini-tial work by the Co-ordinating Editors’Board. The Methods Application and Re-viewStandardsWorkingGroup (convenedbyRachel Churchill and JulianHiggins) hasassumed responsibility along with DavidTovey for the project, working closely withthe Cochrane Editorial Unit, the MethodsExecutive and the Co-ordinating Editors’Executive. The project’s objectives arefirstly to specify some methodological ex-pectations forCochraneprotocols, reviewsand updates of reviews on the effects

of interventions, and secondly to ensurethat these methodological expectationsare implementedacross theCollaboration.In this report,wedescribeprogress todate,which relates mainly to the first objective.

Development of methodologicalexpectationsMethodological expectations are beingdeveloped in three categories. A list ofminimum standards collates key aspectsof review methodology, drawing directlyfrom Handbook guidance as well as therecent minimum standards developed bythe Institute of Medicine in the UnitedStates.1 Someof these arebeingdescribedas mandatory, and others as highly desir-able; some are of course relevant onlyin specific circumstances (such as whenmeta-analysis is performed). In additionto the minimum standards, lists of exem-plars of good practice and of commonerrors are being collated. The exemplarsof good practice provide examples of rec-ommended approaches in areas in whichno single methodology can bemandated.

Six working groups have been workingon draft methodological expectations insix different areas of review methodology(question formulation and study eligibil-ity; searching; study selection and datacollection; bias in individual studies; statis-tics; and reporting and interpretation (in-cluding publication bias)). Each workinggroup’s convenors include a methodolo-gist and a co-ordinating editor. A draftset of minimum standards and a list of ex-emplarmethods and commonerrorswerereleased for wide consultation around The

Cochrane Collaboration in June 2011. Arevised, hopefully final, version of themin-imum standards should be available intime for the Colloquium in Madrid in Oct-ober 2011; the other lists will be updatedcontinually as new items are identified.

Implementation and next steps

In creating the methodological expecta-tions, our emphasis is on sound meth-ods that can feasibly be implementedby review authors. The methodologicalexpectations will feed into a variety ofdocuments and tools. The main set ofminimum standards will become a policydocument that defines what a Cochranereview is and is not. Some mandatoryitemsmaysuggest structural or validation-check modifications for RevMan. Key out-puts will be one or more checklists forauthors, editors and peer reviewers (in-cluding methodological peer reviewers).These are likely to address an appropriateselection ofminimum standards and com-mon errors to avoid. Online resources willprovide further dissemination along withtraining tools based on common errorsand exemplar reviews; and the CochraneHandbook for Systematic Reviews of Inter-ventions and centralized trainingmaterialswill be updated as appropriate.

Reference

1. Eden J, Levit L, Berg A, Morton S, ed-itors. Finding what works in health care:standards for systematic reviews. Washing-ton, DC: Institute of Medicine, NationalAcademies Press, 2011.

The role of the newCochrane MethodsCo-ordinatorCorrespondence to:[email protected] Collaboration Secretariat.

Jackie Chandler joined the CochraneEditorial Unit as the Cochrane MethodsCo-ordinator in February 2011 from Ban-gor University, Wales. Jackie originallytrained as a nurse and has subsequentlyworked for the National Institute forHealth and Clinical Excellence, and on

research projects relating to the imple-mentation of evidence-based practice.This is a new post with strategic andsupport functions. Her role is to sup-port the development, implementationand application of methods for Cochraneintervention reviews. She represents


methods-based interests on key commit-tees within the Collaboration such as theMonitoring and Registration, Archie De-velopment and RevMan advisory com-mittees. She will support the work ofthe Methods Board, Methods Executiveand the Methods Application and Re-view Standards Working Group. A keyinitiative that Jackie will be working onis the development of the networks ofmethods-based individuals in CochraneReviewGroups (CRGs) andCentres. Meth-ods Groups will host and support thesenetworks for standard review meth-ods: question formulation; searching forstudies; statistics; assessing risk ofbias; and interpretation and applicabilityof findings. There will also be a net-work to support economic issues. Theoverarching aims of these networks willbe to:

• create or reinforce communicationamong similarmethods across CRGs;

• facilitate effective communicationand collaboration betweenMethodsGroups and CRGs;

• enable CRGs to ensure that their ed-itorial policies and methods used intheir protocols and reviews are con-sistent with methodology specifiedin the Cochrane Handbook for Sys-tematic Reviews of Interventions; and

• encourage collation of challenges inthe implementationofmethodologyspecified in the Cochrane Handbookfor Systematic Reviews of Interven-tions, so that guidance can be up-dated or improved.

This initiative will seek to identify ev-eryone engaged in methods activity inThe Cochrane Collaboration and ensure

access to methodological expertise as re-quired to support authors in complet-ing reviews. Other areas of responsibilityinclude the co-ordination of the imple-mentation of the minimum standards forreviews, collation of common errors andgood practice. Jackie will also developa website to promote and disseminatemethodological initiatives and develop-ments to internal and external audiences.This year’s Methods Symposium at theCochraneColloquium inMadridwill focuson review synthesis methods for complexinterventions. The work programme for2012 will include monitoring the impactof the minimum standards, the contin-ued development of the Risk of Bias tooland ‘Summary of findings’ table, and thedevelopment of the next major revisionof the Cochrane Handbook for SystematicReviews of Interventions.

The Cochrane Risk ofBias tool for RevMan 5.1

Lucy Turner, DavidMoher, IsabelleBoutron, Jonathan Sterne, DougAltman, Julian Higgins, LauraWeeksand Jelena Savovic

Correspondence to:[email protected]

Cochrane Bias Methods Group, OttawaHospital Research Institute, Canada.

After an extensive evaluation of the Riskof Bias tool, modifications and improve-ments have been made and an updatedversion of the tool was implemented inRevMan 5.1. Here we summarize thechanges to the tool, and provide guid-ance on some of the practical issuessurrounding the changes. There are

implications for existing reviews, as wellas for new and updated reviews. For fur-ther details of assessing risk of bias inincluded studies, please refer to Chap-ter 8 of the CochraneHandbook for System-atic Reviews of Interventions (Version 5.1,updated March 2011).

The evaluation project also identifiedsome medium- and long-term objectivesfor the continued development of the

Summary of changes.

Modification Explanation

Separation of blinding In the earlier version, biases related to blinding of participants, personnel and outcomeassessors were all assessed within a single domain (although they may have been assessedseparately for different outcomes). In the revised tool, bias related to blinding ofparticipants and personnel is now assessed separately from bias related to blinding ofoutcome assessment.

Nature of the judgement The judgements are now expressed simply as ‘Low risk’, ‘High risk’ or ‘Unclear risk’ of bias. Thequestions have been removed, along with the responses ‘Yes’ indicating low risk of biasand ‘No’ indicating high risk of bias.

Minor rewording The items have been renamed with the removal of question-based judgements:

Adequate sequence generation? Random sequence generation

Allocation concealment? Allocation concealment

Blinding? Blinding of participants and personnelBlinding of outcome assessment

Incomplete outcome data addressed? Incomplete outcome data

Free of selective outcome reporting? Selective reporting

Free of other bias? Other bias


Continued.

Modification Explanation

Insertion of categories of bias The revised tool clarifies the category of bias within which each domain falls:• selection bias (random sequence generation and allocation concealment);• performance bias (blinding of participants and personnel);• detection bias (blinding of outcome assessment);• attrition bias (incomplete outcome data);• reporting bias (selective reporting); and• other bias.

Reconsideration of eligibleissues for ‘other bias’, includingearly stopping of a trial

The guidance for the ‘other bias’ domain has been edited to strengthen the guidance thatadditional items should be used only exceptionally, and that these items should relate toissues that may lead directly to bias. In particular, themention of early stopping of a trial hasbeen removed, because (i) simulation evidence suggests inclusion of stopped early trials inmeta-analyses will not lead to substantial bias, and (ii) exclusion of stopped early trials hasthe potential to bias meta-analyses towards the null (as well as leading to loss of precision).

tool. These include developing a bankof examples of completed assessments,guidance and training materials on howto complete assessments, and provid-ing answers to some common questions.These materials will all be placed on-line when available. A Methodological

Expectations for Cochrane InterventionsReviews (MECIR) Working Group has de-veloped ‘should’ and ‘should not do’ guid-ance for assessing risk of bias. Moreover,as an established methods priority topic,work will start soon to develop an ex-tension to the tool for non-randomized

studies. Should you have any questions,there is an online forumatwww.cochrane.org/forum, or please contact the BiasMethods Group Co-ordinator, Lucy Turner([email protected]).

PROSPERO – the newinternational prospectiveregister of systematicreviews

Mike Clarke and Lesley Stewart

Correspondence to:[email protected] Hub for Trials MethodologyResearch, Queen’s University, Belfast,Northern Ireland.

Systematic reviews are promoted and ac-cepted as a gold standard for summariz-ing existing research evidence in a waythat improves precision, minimizes biasand in somesenseestablishes ‘the truthofthe matter’. Systematic reviews thereforehave great potential to influence clinicaldecision-making and health policy. Theyalso require significant investmentof timeand resource if they are to be done well.Systematic reviewsmake considerable ef-forts to address the potential for biasin individual studies, and similar influ-ences to be selective in conduct, re-porting and publication may also applyto systematic reviews. Indeed there isemerging evidence of outcome reporting

bias in systematic reviews.1 Users needto be able to make judgements aboutthe risk of bias in reviews, in the sameway as they would for the studies theycontain.

A systematic review protocol is an im-portant safeguard against bias because itsets out the objectives andmethods to beused, explicit criteria for including or ex-cluding studies, the outcomes of primaryinterest, and the methods that might beused to summarize outcome data. Thiscan help reduce the chances of post hocdecisions that might lead to bias. WithinThe Cochrane Collaboration, the value ofprotocols is understood and prospectiveregistration is fundamental to the reviewprocess. Authors approach a CochraneReview Group for approval of a topic,and then register the title formally, beforesubmitting the draft protocol for editorialreview, feedback and acceptance. How-ever, the majority of systematic reviewsin health care are not Cochrane reviews,2

and until earlier this year there was noplace to register protocol information forsuch reviews. This changed in Februarywith the launch of PROSPERO by the Cen-tre for Reviews and Dissemination (CRD),University of York, UK. The developmentof the register involved a wide consulta-

tion exercise that included many peoplefrom within The Cochrane Collaborationand was overseen by an internationaladvisory group.3

The consultation established a data set ofitems to be included in a systematic re-view registration record. This includes 22required items and 18 that are optional.The items cover data that are neededto understand the methods proposed forthe review, including its eligibility criteriaand intended analyses; as well as admin-istrative details. The information is notas comprehensive as that needed in afull protocol for a systematic review, butPROSPERO includes an option to makethe protocol itself available online.PROSPERO follows similar principles tothose adopted by theWorld Health Orga-nization for trial registries,4 including theability to register the review freeof chargeand for the register entry to be openlyaccessible online. The aim is to cap-ture the key attributes of systematic re-view design (particularly those that couldbe associated with bias) at the protocolstage; maintain an audit trail of any sub-sequent protocol amendments; and adddetails of final publications when avail-able. PROSPERO will therefore providea permanent public record of systematic


reviews that have reached the protocolstage. It will help those planning new re-views or guidelines to access informationeasily and quickly on reviews that are un-derway, minimizing unplanned and un-necessary duplication of effort. It will alsoallowpublic scrutiny and comparison, en-abling discrepancies between publishedanalyses and those planned in the regis-tered review protocol5 to bemore readilyidentified. A recent Cochrane method-ology review has highlighted how theavailability of protocols and trial registrieshas brought to light such discrepancieswithin clinical trials6 and PROSPERO willfacilitate the same rigour for systematicreviews.For more information about PROSPERO,or to register a non-Cochrane system-atic review, visit www.crd.york.ac.uk/

prospero. It is not necessary to enterdetails for Cochrane reviews becauseCRD, the Cochrane Editorial Unit andthe Cochrane Collaboration’s InformationManagement System team are workingtogether to develop a system to up-load information on Cochrane reviewsautomatically.

References

1. Kirkham JJ, Altman DG, Williamson PR.Bias due to changes in specified out-comes during the systematic review pro-cess. PLoS One 2010; 5(3): e9810.2. MoherD, Tetzlaff J, TriccoAC, SampsonM, Altman DG. Epidemiology and report-ing characteristics of systematic reviews.PLoSMedicine 2007; 4(3): e78.

3. Booth A, Clarke M, Ghersi D, Moher D,Petticrew M, Stewart L. An internationalregistry of systematic-review protocols.Lancet 2011; 377(9760): 108-9.

4. Ghersi D, Pang T. FromMexico to Mali:four years in the history of clinical trialregistration. Journal of Evidence BasedMedicine 2009; 2(1): 1-7.

5. PLoS Medicine Editors. Many reviewsare systematic but some are more trans-parent andcompletely reported thanoth-ers. PLoSMedicine 2007; 4(3): e147.

6. Dwan K, Altman DG, Cresswell L, Blun-dell M, Gamble CL, Williamson PR. Com-parison of protocols and registry entriesto published reports for randomised con-trolled trials. Cochrane Database of Sys-tematic Reviews 2011, Issue 1. Art. No.:MR000031.


Published MethodologicalResearch – Structured

Abstracts andCommentaries

Bias due to changes in specifiedoutcomes during the systematicreview process

Kirkham JJ, Altman DG, Williamson PR. PLoS ONE 2010;5(3): e9810.

Background: Bias can be introduced into a review when out-comes are added, omitted or changed between the publishedprotocol and the subsequent review.Objective: To look for discrepancies between primary outcomeslisted in protocols and in the subsequent completed reviewspublished in The Cochrane Library and then quantify the risk ofbias in a set of meta-analyses where discrepancies were found.Design: Purposive sample of new reviews from three consecutiveissues of The Cochrane Library (n = 288), paired with their respec-tive protocols to identify discrepancies in the primary outcomesreported. Protocols and reviews separately were independentlyexamined by two investigators. Inconsistencies were classifiedas inclusion or exclusion of one or more primary outcomes, withchanges classified as an upgrade from a primary to secondaryoutcome or a downgrade if the change was in the oppositedirection.Data collection and analysis: Review authors were contacted toestablish reasons and agree classification of discrepancies. Rela-tive risks were calculated for these classifications (outcomes in-cluded/excludedorprimaryoutcomesupgradedordowngraded)for the primary review comparison and compared with the meta-analyses with no discrepancies estimated. Study findings werefed back to the authors and their Cochrane Review Group.Main results: Two hundred and ninety-seven reviews were iden-tified. Protocols for nine could not be sourced from authors or TheCochrane Library. Twenty-two per cent (64/288) of the remainingreview/protocol pairings contained at least one outcome mea-sure discrepancy. Forty-eight of these were attributable to theprimary outcome measure. Only four reviews described reasonsfor change. In at least eight reviews biased discrepancies werefound, but not reported, because changes were made after theresults were known. Outcomes that were promoted in the reviewweremore likely to be significant than if therewas no discrepancy(relative risk (RR) 1.66 (95% confidence interval (CI) 1.10 to 2.49),P = 0.02).

Conclusion: Reasons for discrepancies are not reported, demon-strating an under recognition of the threat of bias andmisleadinginterpretation of results. Reporting changes in outcome specifi-cation with reasons in the review is vital.

Commentary

Prepared by PhilippaMiddleton

Correspondence to: [email protected] Research Centre for Health of Women and Babies,University of Adelaide, Australia.

In this methodological study of Cochrane reviews, Kirkham andcolleagues traced changes made to outcomes from protocol toreview stage. They found 48 discrepancies (17%) in the primaryoutcomes in a total of 288 protocol/review dyads. For primaryoutcomes, discrepanciesweredefinedaseither anupgrade (listedas a secondary outcome [or not categorized as either secondaryor primary] in the protocol and promoted to a primary outcomein the review); an inclusion (not mentioned in the protocol butlisted as a primary outcome in the review); a downgrade (listed asa primary outcome in the protocol but demoted to a secondaryoutcome in the review); or an exclusion (listed as a primaryoutcome in the protocol but not in the review).Upgrades and inclusions were identified as being of potentiallyhigher risk of bias, with 10 of the 59 protocol/review dyadsshowing a significant result in favour of the intervention for theprimary outcome and only 12 of 89 showing a non-significantresult. Thus, Kirkham and colleagues have demonstrated theimportant finding that outcome reporting bias may be a featureof systematic reviews just as it may be for randomized trials (butperhaps to a lesser degree). In eight of the 28 dyads where reviewauthors gave a reason for the discrepancy, post hoc changes toprimary outcomes were made after the review authors had readthe results of the included trials.While these findings appear convincing, further work on thecomplexities of review outcomes and their reporting is neededto understand the best ways of further reducing the potential foroutcome reporting bias in systematic reviews. What influencemight composite outcomeshaveon the risk of outcome reportingbias? Does it matter if the outcome is one of effectiveness orsafety? What is the potential impact of specifying ‘global’ primary


outcomes such as any adverse effect in the protocol and thenreporting only one or two adverse effects as primary outcomes inthe review?It is very pleasing to see that the discrepancy rates have droppedfrom 81% in 2000 in our study1 to 22% in 2006/2007. I wassurprised to see that 84 (29%) of 228 protocols in Kirkham’s studydid not specify any primary outcomes (accounting for a third of allprimary outcome discrepancies in the study), and so I looked atthe new protocols in Issue 5, 2011 of TheCochrane Library. It was arelief to see that only one of the 45 new protocols did not list anyprimary outcomes (and this was the protocol for an overview).Both these improvements are likely to be due to the increased ef-forts and investmentby TheCochraneCollaboration in improvingmethodology in recent years, including the Cochrane Handbookfor Systematic Reviews of Interventions. It is also a testament to theforesight of those who decided that Cochrane protocols shouldbe a publication in their own right – and to those who have es-tablished PROSPERO, the international database of prospectivelyregistered systematic reviews (see page 4) in health and socialcare (www.crd.york.ac.uk/prospero/).

Reference1. SilagyCA,MiddletonP,Hopewell S. Publishingprotocolsof sys-tematic reviews: comparingwhat was done to what was planned.JAMA 2002; 287(21): 2831–4.

Improving the interpretationof quality of life evidence inmeta-analyses: the application ofminimal important difference units

Johnston BC, Thorlund K, Schunemann HJ, Xie F, MuradMH, Montori VM, Guyatt GH.Health and Quality of LifeOutcomes 2010; 8: 116.

Background: Systematic reviews of randomized trials that in-clude measurements of health-related quality of life (HRQL) po-tentially provide critical information for patients and cliniciansfacing challenging healthcare decisions. When, as is most oftenthe case, individual randomized trials use different measure-ment instruments for the same construct (such as physical oremotional function), authors typically report differences betweenintervention and control in standard deviation units (so-calledstandardized mean difference (SMD) or effect size). This ap-proach has statistical limitations and it is influenced by theheterogeneity of the population and is non-intuitive for decisionmakers.Objective: To describe an alternative approach to using standarddeviationunitswhen reportingdifferencesbetween studygroupsin systematic reviews that include HRQL measures.Maincontent: Thispapersummarizesanewmethodforanalysingthe results from individual trials that report on HRQL measuresusing the minimal important difference (MID). The MID is definedas ‘‘the smallest difference in score in the outcome of interestthat informedpatients or informedproxies perceive as important,either beneficial or harmful, and which would lead the patientor clinician to consider a change in the management’’. This newmethod substitutes the MID for the usual denominator of the

SMD, the standard deviation. The mean difference is dividedby the MID that was established for the instrument used in thetrial. As a result, rather than obtaining an estimate in standarddeviation units, an estimate in MID units is obtained. This is thenstandardized by dividing the mean difference by the MID andaltering the scale on which the meta-analysis is performed. Indoing so, one also needs to account for the changes that thestandardization has on the standard error andweights associatedwith each standardized trial outcome.Conclusion: This approach provides a potential solution toboth the statistical and interpretation problems of existingmethods.

Commentary

Prepared by Roy G. Elbers

Correspondence to: [email protected] Cochrane Centre, Academic Medical Center, Amsterdam,The Netherlands.

The proposed method to replace the standard deviation (SD)by the minimal important difference (MID) as the denominatorin the formula to calculate the conventional standardized meandifference (SMD) may eliminate the problem of heterogeneityintroduced by using the SD. In addition, the presentation in MIDunits may indicate whether the observed effect is consideredimportant by patients.Johnston and colleagues applied a conventional meta-analysisusing the SMD and a meta-analysis expressed in MID units tothe data in an existing Cochrane review.1 The difference foundin the I2 statistic (26%) between both meta-analyses (I2 = 58%for the SMD analysis and I2 = 32% for the MID analysis) supportsthe hypothesis that a meta-analysis expressed in MID units re-duces heterogeneity of between study variances, compared to ameta-analysis expressed in SD units. The interpretation of resultsexpressed in MID units seems intuitive, but may be vulnerableto naıve misinterpretation. However, this misinterpretation isnot unique to the MID approach. The authors provide a rule-of-thumb to guide interpretation on the proportion of patients thatare likely to benefit from treatment.The MID approach presented by Johnston and colleagues ispromising for conducting meta-analyses of patient reported out-comes (PROs). However, review authors are challenged by iden-tifying MID values for the instruments included in their meta-analysis. For most instruments, MID values are not established. IftheMIDisestablished, evidencesupportingthesevalues is limited,because these are often based on distribution-basedmethods in-stead of preferred methods relying on patient-based and clinicalanchors.2 Fortunately, research puts more effort in establishinganchor-basedMIDvalues formeasurement instruments. As a con-sequence, updating a Cochrane review presenting an MID-unitmeta-analysis should include a re-evaluation of whether the MIDvalue used as the denominator is still the most accurate valueavailable.At this time, a combination of both conventional methods formeta-analysis and theMID approachmaybe afirst step in improv-ing the synthesis and presentation of PROs in Cochrane reviews.To facilitate reviewauthors using theMIDapproach, theCochranePRO Methods Group should summarize evidence on publishedMID values, and pilot the MID approach in Cochrane reviews to


enhance and refine the methods and interpretation of this newapproach.

References

1. Lacasse Y, Goldstein R, Lasserson TJ, Martin S. Pulmonary re-habilitation for chronic obstructive pulmonary disease. CochraneDatabase of Systematic Reviews 2006, Issue 4. Art. No.: CD003793.2. RevickiD,HaysRD,CellaD, Sloan J. Recommendedmethods fordetermining responsiveness andminimally important differencesfor patient-reported outcomes. Journal of Clinical Epidemiology2008; 61 (2): 102–9.

Inclusion of methodological filtersin searches for diagnostic test accuracystudiesmisses relevant studies

Whiting P, Westwood M, Beynon R, Burke M, Sterne JA,Glanville J. Journal of Clinical Epidemiology 2011; 64(6):602–7.

Background: Challenges in identifying test accuracy studies forsystematic reviews have led to the development of methodolog-ical search filters which aim to identify such studies in biblio-graphic databases. Previous studies have found these filters tobe insufficiently accurate as they have less than optimal levelsof sensitivity. However, search strategies designed to be highlysensitive in identifying test accuracy studies that did not includea diagnostic filter have also been shown to miss studies.Objective: To compare the performance of MEDLINE searchesusing index tests and target conditions (subject searches) withthe same searches combined with methodological filters for testaccuracy studies.Design: A relative recall reference set of 506 test accuracy studiesindexed in MEDLINE was derived from seven systematic reviewswith extensive searches. The performance of ‘subject’ and ‘fil-tered’ searches (same searches with each of 22 filters) against thereference set for each reviewwere then compared. The filters var-ied considerably in complexity and the number of search terms,the briefest consisting only of the term ‘specificity’ in the title orabstract and the longest including up to 89 terms.Data collection and analysis: The results of all searches weremanually screened to determine which of the reference set stud-ies were identified by each of the searches. The number ofreference set records missed, sensitivity, number needed to read(NNR) and precision (number of reference set studies identifiedfor every 100 records screened) were assessed. Analyses wereconducted for all reviews combined and stratified according toreview.Main results: Subject searches missed 47 of the 506 referencestudies and missed from 0 to 10 (0% to 19%) of studies withinreviews. Filtered searches missed an additional 21 to 241 studies,with searches incorporating one of the best performing filtersmissing a median of three additional relevant studies (range 0to 14) per review. Sensitivity was 91% for subject searches andranged from 43% to 87% for filtered searches. The NNR was 56(precision 2%) for subject searches and ranged from 7 to 51 (2%to 15%) for filtered searches. None of the filters offered a substan-tial improvement in precision without compromising sensitivity.

Addition of filters to subject searches reduced the NNR from 56(precision 2%) to 29 to 36 (precision 3%).Conclusions: Filtered searchesmiss additional studies comparedwith searches based on index test and target condition. None ofthe existing filters provided reductions in the NNR for acceptablesensitivity. Currently available methodological filters should notbe used to identify studies for inclusion in test accuracy reviews.

Commentary

Prepared by Anne Eisinga

Correspondence to: [email protected] Support Unit for Cochrane Systematic Reviews of DiagnosticTest Accuracy, University of Birmingham, Birmingham, UK.

Whiting and colleagues have produced an excellent and timelystudy in a fast-moving, complex area of information science relat-ing to the retrieval of studies evaluating theaccuracyofdiagnostictests. It should be of particular benefit to information specialists,Trials Search Co-ordinators (TSCs) in The Cochrane Collaborationand researchers seeking to undertake systematic reviews of di-agnostic test accuracy (DTA). More than twenty methodologicalfilters have been designed to retrieve DTA studies and severalevaluation studies have assessed their performance,1–5 of whichthis is the most recent.This study has a number of key strengths. It is wide-ranging inscope and, for the first time, enables a comparison of the sen-sitivity and precision of extensive, reproducible subject searches(of index tests and target conditions) in a variety of topics ontheir own, along with a comparison of the sensitivity and pre-cision of the full range of published methodological diagnosticfilters. The relative recall method used to conduct this research issound because of the relative ‘completeness’ of the reference setagainst which the performance of the filters is assessed, namely aset of seven DTA reviews (506 included studies). These had usedextensive searches across a range of resources without the use offilters, with inclusion criteria inwhich an index test was comparedwith a reference standard that provided sufficient data to allowcross-tabulation of results.The web table, listing the original version of each search filteralongsidehowtheauthorshave translated thefilters into theOvidinterface, is an excellent example of explicit reporting of searchstrategies. It allows the reader to assess reliability in the par-ticularly problematic area of translating strategies across searchinterfaces. It is also a useful resource for the existing publishedfilters.A variety of measures can be used in information retrieval topresent search results, including sensitivity (recall), precision (pos-itive predictive value), NNR (number needed to read to identifyone relevant study), fall-out (proportion of non-relevant recordsretrieved of all non-relevant records), F-measure (which allowsthe user to attach the same or more importance to recall asprecision). However, their interpretation can be challenging. Inthis study, the innovative graphics – paired forest plots showingoverall sensitivity and precision for subject searches and eachof the filtered searches (combined with the subject search) –are particularly helpful for assessing the relative performance ofthese strategies. Presentation of search results of performanceevaluations in order to aid interpretation by different user groupswarrants investigation.


The authors’ findings are similar to the results of previousevaluations1–4 and add to the evidence base for the recom-mendation not to use methodological filters in identifying DTAstudies for Cochrane DTA systematic reviews as they do not cur-rently have sufficient sensitivity6. However, a recent evaluationby Kastner and colleagues5 for a filter they had developed7, pro-duced dissimilar results to the previously published evaluations.Whiting and colleagues chose to do an additional separate anal-ysis of data from the five reference set reviews in the Kastnerstudy that had not used filters to identify studies for inclusion,to test for reproducibility. They present results of this additionalanalysis and a brief discussion of some suggested limitations ofthe Kastner study. A Cochranemethodology review8 is underwaythat seeks to clarify these findings further.Finally, Whiting and colleagues indicate the key areas of uncer-tainty which require further research: the impact of excluding‘missed’ studies on the review results and conclusions; combin-ing filtered searches with other methods of identifying studies,such as screening bibliographies of relevant studies to reducethe workload to more acceptable levels while trying to reducethe risk of missing relevant studies. They acknowledge that re-search into developing more sensitive filters that also achievea reduction in NNR may not be possible until improvementsin indexing and reporting become more evident. In December2010, Elsevier introduced ‘diagnostic test accuracy study’ as anindexing term (check tag) in EMBASE followingnegotiations andaformal, detailed submission by RuthMitchell, TSCCochrane RenalGroup, Anne Eisinga, UK Support Unit for Cochrane SystematicReviews of Diagnostic Test Accuracy, Julie Glanville, Co-convenorCochrane InformationRetrievalMethodsGroup,MariskaLeeflang,Co-convenor Cochrane Screening and Diagnostic Tests MethodsGroup and Marie Westwood, Kleijnen Systematic Reviews Ltd.Discussions and preparations for a formal submission to the Na-tional Library of Medicine for a publication type for DTA studiesin MEDLINE, are underway.The STARD guidelines currently recommend that authors should‘‘identify the article as a studyof diagnostic accuracy (recommendMeSH heading ‘sensitivity and specificity’)’’ in the title, abstract,or keywords. A consistent implementation of these guidelineswould greatly facilitate retrieval of DTA studies and, togetherwithimprovements in indexing, should enable future research into fil-ter design to be more rewarding. Another area of future researchworth exploring to improve search efficiency for DTA reviewscould be in semantic technology, which is showing promisingresults in a similarly complex field, economic evaluations.9

References1. Doust JA, Pietrzak E, Sanders S, Glasziou PP. Identifying studiesfor systematic reviews of diagnostic tests was difficult due to thepoor sensitivity and precision ofmethodologic filters and the lackof information in the abstract. Journal of Clinical Epidemiology2005; 58(5): 444–9.2. Mitchell R, Rinaldi F, Craig J. Performance of published searchstrategies for studies of diagnostic test accuracy (SDTAs) in MED-LINE and EMBASE [abstract]. XIII Cochrane Colloquium; 2005 Oct22-26; Melbourne, Australia: 33. Available from: www.cochrane.org/colloquia/abstracts/melbourne/O-01.htm.3. LeeflangMMG,ScholtenRJPM,RutjesAWS, Reitsma JB, BossuytPMM. Use of methodological search filters to identify diagnosticaccuracy studies can lead to the omission of relevant studies.Journal of Clinical Epidemiology 2006; 59(3): 234–40.

4. Ritchie G, Glanville J, Lefebvre C. Do published search filtersto identify diagnostic test accuracy studies perform adequately?Health Information and Libraries Journal 2007; 24(3): 188–92.5. Kastner M, Wilczynski NL, McKibbon AK, Garg AX, Haynes RB.Diagnostic test systematic reviews: bibliographic search filters(‘‘Clinical Queries’’) for diagnostic accuracy studies perform well.Journal of Clinical Epidemiology 2009; 62(9): 974–81.6. de Vet HCW, Eisinga A, Riphagen II, Aertgeerts B, Pewsner D.Chapter 7: Searching for studies. In: Cochrane Handbook for Sys-tematic Reviews of Diagnostic Test Accuracy Version 0.4 [updatedSeptember 2008]. The Cochrane Collaboration, 2008. Availablefrom: http://srdta.cochrane.org/handbook-dta-reviews7. Haynes RB, Wilczynski NL. Optimal search strategies for re-trieving scientifically strong studies of diagnosis from Medline:analytical survey. BMJ 2004; 328(7447): 1040.8. Leeflang M, McDonald S, Scholten RJ, Rutjes A, Reitsma JJB.Search strategies to identify diagnostic accuracy studies in MED-LINE and EMBASE (Protocol). Cochrane Database of SystematicReviews 2007, Issue 2. Art. No.: MR000022.9. Glanville JM, Lefebvre C, Porter B, Negosanti P. Improvingsearch efficiency for economic evaluations in major databasesusing semantic technology. Oral Presentation at the Joint Col-loquium of The Cochrane and Campbell Collaborations; 2010Oct 18–22; Keystone, Colorado, USA [abstract]. CochraneDatabase of Systematic Reviews, Supplement 2010; SupplCD000002: 46. Available from: www.cochrane.org/sites/default/files/uploads/abstract book keystone 2010.pdf

Reporting performance of prognosticmodels in cancer: a review

Mallet S, Royston P, Waters R, Dutton S, Altman DG. BMCMedicine 2010; 8: 21.

Background: Prognosis is used todirectdiagnosticpathwaysandto inform patient treatment. For accurate outcome prediction,multiple risk factors need to be considered jointly. Prognosticmodels allow this to be done systematically, reproducibly andusing evidence-based methods. Appropriate choice and use ofprognostic models in clinical practice require the use of goodmethods both for model development and for developing prog-nostic indices and risk groups from the models. In order toassess reliability and generalizability, models need to have beenvalidated and measures of model performance reported.Objective: To review published articles to assess the methodsand reporting used to develop and evaluate performance ofprognostic indices and risk groups from prognostic models.Design: A search string was used to identify articles in PubMed,published in 2005, which aimed to predict patient outcome, pre-sented new prognostic models in cancer with outcome time toan event, included two or more separate variables and analyseddata suitable for time-to-event data.Main results: Forty-seven articles were included. Cox modelswere used in 94% (n = 44) but the coefficients or hazard ratiosfor the variables were reported in only 72% (n = 34). The repro-ducibility of the derivedmodelwas assessed in 11% (n = 5) of thearticles. Aprognostic indexwasdeveloped from themodel in 81%(n = 38) of the articles, but researchers derived the prognosticindex from the final prognosticmodel in only 34% (n = 13) of thestudies; different coefficients or variables from those in the final


model were used in 50% (n = 19) of models and the methodsused were unclear in 16% (n=6) of the articles. Methods usedto derive prognostic groups were also poor, with researchers notreporting the methods used in 39% (14 of 36) of the studiesand data-derived methods likely to bias estimates of differencesbetween risk groups being used in 28% (n = 10) of the studies.Validation of their models was reported in only 34% (n = 16) ofthe studies. In 15 studies validation used data from the samepopulation and in five studies from a different population. Includ-ing reports of validation with external data from publications upto four years following model development, external validationwas attempted for only 21% (n=10) of models. Insufficient infor-mation was provided on the performance of models in terms ofdiscrimination and calibration.Conclusions: Many published prognostic models have beendeveloped using poor methods and many with poor reporting,both of which compromise the reliability and clinical relevance ofmodels, prognostic indices and risk groups derived from them.

CommentaryPrepared byWilli Sauerbrei

Correspondence to: [email protected] of Medical Biometry and Medical Informatics, Universityof Freiburg, Germany.

During recent decades, a lot of money and effort was invested inmarker research. This resulted in a very large number of publica-tions on prognostic markers and prognostic models in all areasof medicine. Unfortunately, only a few models have found theirway into clinical practice. Probably, an important reason for thisdisappointing situation is the insufficient quality of developingand reporting prognostic models. This fact is well known tomanyresearchers and a lot of criticism can be found in the literature.Empirical evidence demonstrating severeweaknesses of researchis available for many related issues, but according to my knowl-edge this is the first study providing evidence for prognosticmodels.Based on a systematic search and using some inclusion criteriathe authors assessed 47 articles published in 2005. Specific char-acteristics are reported in an accompanying paper. The numberof articles is relatively small, but this sample size is sufficient todemonstrate severe weaknesses in many parts of model devel-opment, reporting and validation. Nearly all papers (n = 44)used the Cox model (time to event was an inclusion criterion)for analysis, but only some (n = 10) checked the proportionalhazards assumption. However, it is not possible to knowwhetherthis assumptionwas not assessed or whether the assessment wasnot reported. Such types of problems, distinguishing whether animportant part was not done or just not reported, are inherent inall papers trying to derive empirical evidence about the quality ofresearch. With related guidelines (such as REMARK for prognosticmarkers), reporting should becomemuch better.Perhaps originally not considered as an aim of this study, theresults clearly demonstrate that guidance for several importantissues of analysis is missing. Unfortunately variable selectionis not considered in the paper, but many tables illustrate thata large variety of methods and measurements was used to as-sess model performance, reproducibility or how to derive riskgroups from a prognostic index. Analysts of the individual study,who are not always experts in their field, need more guidance

urgently. Obviously, that would require some agreement aboutkey principles of analysing studies, validating the results anddeci-sionswhenmarkers ormodels are ready for clinical use. This leadsdirectly to the necessity of derivingmore guidance for systematicreviews andmeta-analysis for these types of issues. A lack of guid-ance is one reason that the number of successful review projectsin prognostic research is pitifully small. On the negative side itmay be stressed that the current situation is still disappointing,on the positive side it is obvious that methodologists will findmany interesting topics for their research.

Indirect comparisons: a review ofreporting andmethodological quality

Donegan S, Williamson P, Gamble C, Tudur-Smith C. PLoSONE 2010; 5(11): e11054.

Background: The quality of an indirect comparison dependsupon the methodology selected and the validity of underlyingassumptions. The core assumption underlying methodology issimilarityof treatmenteffects. Toensure thisholds across trials theeffect of patient or trial characteristics, using subgroup analysis,sensitivity analysis or meta-regression, may indicate whether thesimilarity assumption is reasonable. Reporting recommendationsare required as published indirect comparisons become morecommon.Objective: To review systematically the quality of published in-direct comparisons to add to existing empirical data and suggestimprovements that can be made when reporting and applyingindirect comparisons.Design: A systematic review of reviews that have applied statis-tical methods to compare indirectly the clinical effectiveness oftwo interventions based on randomized trials.Data collection and analysis: Databases searched (1966–2008)were Database of Abstracts of Reviews of Effects, The CochraneLibrary, and MEDLINE. Specific, defined quality criteria were de-veloped and applied. Criteria were classified as yes, no or unclearfor each criterion. Reviews were excluded if they comparedmorethan two interventions.Main results: Forty-three reviews were included. Twenty-onewere excluded because they compared more than two treat-ments. Adequatemethodologywas used to calculate the indirectcomparison in 41 reviews. Nineteen reviews assessed the sim-ilarity assumption using sensitivity analysis, subgroup analysisor meta-regression. Eleven reviews compared trial-level charac-teristics. Twenty-four reviews assessed statistical homogeneity.Twelve reviews investigated causes of heterogeneity. Seventeenreviews included direct and indirect evidence for the same com-parison; six reviews assessed consistency. One review combinedboth evidence types. Twenty-five reviews urged caution in in-terpretation of results, and 24 reviews indicated when resultswere from indirect evidence by stating this term with the result.Quality criteria proposed are adequacy of indirect comparisonmethod, similarity, homogeneity across trials within each of thetwo trial sets involved in the indirect comparison, consistency,interpretation and reporting.Conclusion: This review concludes that underlying assumptionsare not routinely explored or reportedwhen undertaking indirectcomparisons. Quality criteria are proposed to facilitate the con-duct of indirect comparisons and their appropriate interpretation.


CommentaryPrepared by Georgia Salanti and Deborah Caldwell

Correspondence to: [email protected] of Hygiene and Epidemiology, University of IoanninaSchool of Medicine, Greece.

In the absence of randomized trials comparing all eligible treat-ments, the Cochrane Handbook for Systematic Reviews of Inter-ventions suggests review authors could consider using indirectcomparisons. For example, an indirect estimate of the ben-efit of intervention A over B can be obtained by comparingtrials of A versus C with trials of B versus C. A network meta-analysis (NMA) refers to ensembles of trial evidence in whichdirect and indirect evidence on relative treatment effects aresynthesized. A growing number of Cochrane authors are keento employ these techniques. The first Cochrane review withan NMA was published in 2009,1 and several more are in thepipeline.Several recent research articles highlight the importance of eval-uating the assumptions that underlie the models and the needfor transparent reporting. However, there is no consensus or clearguidance as to what is good practice. The paper by Donegan andcolleagues highlights this issue.The authors reviewed 43 systematic reviews that indirectly com-pare two treatments via a (single) common comparator. Theevaluation was done against predefined criteria on appropriate-ness of the statistical analysis, the evaluation of the assumptionsof similarity, heterogeneity and consistency and the adequacy ofinterpretation of the results. They suggest these criteria be usedas a starting point for developing and establishing clear guid-ance for reporting and conducting indirect comparisons. Theyfound that 41 reviews used statistically adequate methods, butonly 11 considered the assumptions underpinning an indirectcomparison. This paper adds to an increasing call for betterguidance on the conducting and reporting of indirect compar-isons and NMA. An important limitation of this study is that itfocused solely on networks with three interventions, excludingthe (most challenging) NMA that are increasingly used to enablethe simultaneous comparison ofmultiple interventions in a singleanalysis.The aim of the newly formed Cochrane Comparing Multiple In-terventions Methods Group (CMIMG) (see page 30) is to adviseThe Cochrane Collaboration on how intervention reviews andoverviews can best provide reliable comparisons of multiple in-terventions. The Methods Group aims to provide clear guidanceto Cochrane Review Groups and review authors on how, whenand why to do these analyses and how to report them. To ourknowledge, there are several research groups currently workingon the production of guidelines to ensure methodological andreporting standards for NMA are adhered to. This paper providesan important contribution to this agenda.

Reference1. Singh JA, Wells GA, Christensen R, Tanjong Ghogomu E,Maxwell L, MacDonald JK, Filippini G, Skoetz N, Francis D, LopesLC, Guyatt GH, Schmitt J, La Mantia L, Weberschock T, Roos JF,Siebert H, Hershan S, Lunn MP, Tugwell P, Buchbinder R. Ad-verse effects of biologics: a network meta-analysis and Cochraneoverview. Cochrane Database of Systematic Reviews 2011, Issue 2.Art. No.: CD008794.

Reporting of methodologicinformation on trial registries forquality assessment: a study of trialrecords retrieved from theWHO searchportal

Reveiz L, Chan AW, Krleza-Jeric K, Granados CE, Pinart M,Etxeandia I, Rada D, Martinez M, Bonfill X, Cardona AF.PLoS ONE 2010; 5(8): e12484.

Background: Critical appraisal of randomized trials relies onthe availability of adequate information about study design andconduct, but the reporting of randomized trials is often subop-timal. Trial registries have the potential to contribute importantmethodological information for critical appraisal of study results.Full protocols are a valuable source of information about thedesign and conduct of trials but are often not publicly available,so trial registries constitute the main public source of protocolinformation.

Objective: Toevaluate the reportingof keymethodological studycharacteristics in trial registries.

Design: From seven clinical trial registries, a random sample(n = 265) of actively-recruiting randomized trials was identi-fied using the World Health Organization (WHO) InternationalClinical Trials Registry Platform (ICTRP) search portal in 2008.An assessment was made of the reporting of relevant domainsin The Cochrane Collaboration’s Risk of Bias tool and otherkey methodological aspects. Two reviewers independently as-sessed each record. Primary outcomes were the proportionof registry records with adequate reporting of random se-quence generation, allocation concealment, blinding and trialoutcomes.

Main results: Weighted overall proportions in the ICTRP searchportal were 5.7% (95% CI 3.0% to 8.4%) for adequate reportingof random sequence generation, 1.4% (0% to 2.8%) for allocationconcealment, 41% (35% to 47%) for blinding and 66% (60% to72%) for trial outcomes. The proportion of adequately reportedrandomized trials was higher for registries that used specificmethodological fields for describing methods of randomizationand allocation concealment compared with registries that didnot. For other methodological aspects, weighted overall pro-portions of randomized trials with adequately reported itemswere: eligibility criteria 81%; secondary outcomes 46%; harm 5%;follow-up duration 62%; description of interventions 53%; andsample size calculation 1%. Reporting varied substantially acrossregistries, with the Australian New Zealand Clinical Trials Registryand the Clinical Trials Registry India having higher proportionsof trials with adequate reporting of most items than the oth-ers. Characteristics of the data entry fields also varied betweenregistries.

Conclusions: Trial registries currently contain limited method-ological information about registered randomized trials. To en-able adequate critical appraisal of trial results reported in journalsand registries, trial registries should consider requesting detailson key randomized trial methods to complement journal publi-cations. Full protocols remain the most comprehensive sourceof methodological information and should be made publiclyavailable.


Commentary

Prepared by Kerry Dwan

Correspondence to: [email protected] of Child Health, University of Liverpool, Liverpool, UK.

Information about study design and conduct is necessary to al-low critical appraisal of randomized trials. The Cochrane Risk ofBias tool1 includes six domains, but information is not alwaysadequately reported in the published trial reports to allow suffi-cient assessment of these domains. As protocols are not oftenpublicly available, trial registries may be a useful source of furtherinformation. However, a recent Cochrane methodology review2

has shown that discrepancies occur between protocols or trialregistry entries and published reports.Trial registration was originally introduced to identify the exis-tence of a trial rather than to include detailed information onmethods. The 20-item World Health Organization (WHO) reg-istration data set is recommended for trial registries, but thisdoes not permit full appraisal of a randomized trial, since allo-cation concealment, method of randomization and sample sizecalculations are not part of this data set. Other studies haveshown that this data set is often incompletely used by trialists3

and methodological information is inadequately described inprotocols too.4

Reveiz and colleagues evaluated the reporting of key method-ological study characteristics in trial registries listed on the WHOInternational Clinical Trials Registry Platform in randomized trialswhich were actively recruiting in 2008. The study is well con-ducted; while two reviewers independently extracted data, onlyone screened records. It illustrates the limited methodologicalinformation contained in trial registries. This may be because theregistry only provides a non-specific general field or a drop-downbox without the option to include additional information. In onecase, there was contradictory information within a single registryentry with a trial being described as blinded in the methodssection and open in the summary section. Only four recordsprovided a link to the full protocol.Trialists should be encouraged to include more detailed in-formation in trial registries. This would allow systematic re-viewers to check registries not only for relevant studies butalso for additional information to assess the risk of bias orother data not available from either trial reports or trialists. Itwould be of interest to establish whether systematic review-ers actively use trial registries as a source of information giventheir, often limited, resources. With the introduction of reg-istries for systematic reviews, it is important to learn fromthis study and build on work already undertaken on trial reg-istries, to ensure the quality of reportingwithin systematic reviewregistries.

References

1. Higgins JPT,GreenS,editors. CochraneHandbookforSystematicReviews of Interventions Version 5.1.0 [updated March 2011]. TheCochrane Collaboration 2011; Available from: www.cochrane-handbook.org.2. Dwan K, Altman DG, Cresswell L, Blundell M, Gamble CL,Williamson PR. Comparison of protocols and registry entries topublished reports for randomised controlled trials. CochraneDatabase of Systematic Reviews 2011, Issue 1. Art. No.: MR000031.

3. Moja LP, Moschetti I, Nurbhai M, Compagnoni A, Liberati A,Grimshaw JM, Chan AW, Dickersin K, Krleza-Jeric K, Moher D, SimI, Volmink J. Compliance of clinical trial registries with the WorldHealth Organization minimum data set: a survey. Trials 2009; 10:56.4. Chan AW, Hrobjartsson A, Jørgensen KJ, Gøtzsche PC, AltmanDG. Discrepancies in sample size calculations and data analysesreported in randomised trials: comparison of publications withprotocols. BMJ 2008; 337: a2299.

Updating systematic reviews: aninternational survey

Garritty C, Tsertsvadze A, Tricco AC, Sampson M, Moher D.PLoS ONE 2010; 5(4): e9914.

Background: Systematic reviews should be up to date if theyare to maintain their importance in informing healthcare policyand practice. However, little guidance is available on when andhow to update, and the updating policies of organizations thatcommission or produce systematic reviews are unclear.Objective: To describe the updating practices and policies ofagencies that commission or conduct systematic reviews.Design: An internet-based survey was administered to a purpo-sive non-random sample of 195 healthcare organizations withinthe international systematic review community. Survey resultswere analysed using descriptive statistics. The response ratewas 58% (n = 114) across 26 countries, with 70% (75/107) ofrespondents identified as producers of systematic reviews.Main results: Among responders, 79% (84/107) rated the im-portance of updating as high or very high. Fifty-seven per cent(60/106) of organizations reported having a formal policy forupdating but only 29% (35/106) referred to awritten policy docu-ment. Irregular updatingpracticeswere reportedby59% (62/105)of groups and over half (53/103) of organizational respondentsestimated thatmore than 50%of their reviewswere probably outof date. Review authors were most often deemed responsiblefor ensuring their reviews were current (42/106, 40%). Barriersto updating included resource constraints, reviewer motivation,lack of academic credit and limited publishing formats. Seventyper cent of respondents (70/100) supported the centralization ofupdating efforts across institutions and 84%were in favour of thedevelopment of a central registry of systematic reviews.Conclusions: Most organizations that commission or conductsystematic reviews consider updating important. However, up-dating practices are not regular and many organizations lack aformal written policy for updating systematic reviews.

CommentaryPrepared by Toby Lasserson

Correspondence to: [email protected] Editorial Unit, London, UK.

The comprehensive international survey of systematic reviewproducers undertaken by Garritty and colleagues helps us toplace the updating practices among Cochrane review authorsand Cochrane Review Groups into some context. The study find-ings indicate that the core expectation of updating differentiatesThe Cochrane Collaboration from most other systematic review


producers. Despite this differential expectation, many CochraneReviewGroupswhoparticipated in the survey still perceivedmorethan half of the reviews to be out of date. The study also providessome interesting insights into perceived barriers to updating.Among these is the lack of academic recognition for the workthat goes into an update and the impact this has on author mo-tivation. However, additional considerations include competingclinical or academic commitments among review authors1 andthe high threshold for an update to qualify for a new citation.Some evidence for this may be seen in the low proportion ofupdated Cochrane reviews with changed conclusions.2

Garritty and colleagues provide a clear definition of the updatingprocess which was widely accepted by the survey respondents.However, we should also consider what defines an out-of-datereview. Arbitrary thresholds based on search dates enable thesimple and quick identification of reviews meeting these crite-ria, but they do not perform well if we are looking to prioritizereviews for updating. Should a review of a therapy that is nolonger used or recommended be regarded in the same light asa review of a treatment with promising results but where largestudies are known to await inclusion? Distinguishing betweensuch systematic reviews from a large portfolio does require anunderstanding of their content and clinical context, and furtheridentifying statistical instability poses additional methodologicalchallenges that are likely to affect this process: as the authorsnoted, among the respondents there was a low uptake of formalstatistical techniques to identify when a systematic review wasready to update.To address some of the burden of updating systematic reviewsone solution proposed is to create a centralized body or networkbased on sharing resources and expertise. Future research shouldindeed consider the viability of such an endeavour, but there isalso aneed for efficientprioritization techniques that enable iden-tification of those reviews which funders and users of systematicreviews need in order to base policy and treatment decisions oncurrent, relevant evidence.

References

1. Ervin AM. Motivating authors to update systematic reviews:practical strategies from a behavioural science perspective. Pae-diatric and Perinatal Epidemiology 2008; 22 (Suppl 1): 33–7.2. French SD, McDonald S, McKenzie JE, Green SE. Investing inupdating: how do conclusions change when Cochrane system-atic reviews are updated? BMC Medical Research Methodology2005; 5: 33.

A systematic examination of thecitation of prior research in reportsof randomized, controlled trials

Robinson KA, Goodman SN. Annals of Internal Medicine2011; 154(1): 50–5.

Background: A randomized trial should not be started or inter-pretedwithout accounting for evidence frompreceding random-ized trials addressing the same question. Research has suggestedthat evidence fromprior trials is oftennot accounted for in reportsof subsequent randomized trials. There has been no systematic

study of whether published randomized trials acknowledge priorrelated research.Objective: To assess the extent to which reports of randomizedtrials cite prior trials studying the same interventions.Design: Meta-analyses published in 2004 that combined four ormore trials were identified. The extent to which each trial reportcited the trials which preceded it by more than one year wasassessed. The proportion of prior trials that were cited (priorresearch citation index or PCRI), the proportion of the total par-ticipants from prior trials that were in the cited trials (sample sizecitation index or SSCI) and the absolute number of trials citedwere calculated. The introduction and discussion sections of arandom sample of 30 randomized trials, 15 each from the lowestand highest PRCI, were reviewed. An assessment was made ofwhether the article claimed to report the first trial on the subject,referenced a systematic review, described methods to identifyprior trials or attempted to incorporate prior results.Main results: Two hundred and twenty-seven meta-analyseswere identified, comprising 1523 trials published from 1963 to2004. The median prior research citation was 0.21 (95% CI 0.18to 0.27), meaning that fewer than one quarter of relevant reportswere cited. Themedian sample size citation index (0.24; CI 0.21 to0.27) was similar, suggesting that large trials were not selectivelycited. Of the 1101 randomized trials with five or more prior trialsto cite, 254 (23%) cited none and 257 (23%) cited only one. Themedian number of cited trialswas two, irrespective of the numberof citable trials. The mean number of prior trials cited by trialspublished after 200 was 2.4 compared to 1.5 for those publishedbefore 2000 (P < 0.001).Conclusions: Across many healthcare disciplines and questions,and over four decades, less than 25% of prior randomized trialresearch was cited, representing about 25% of the participants inearlier trials. Research is needed to explore the reasons for thisand the consequences. Potential implications include ethicallyunjustifiable trials, wasted resources, incorrect conclusions andunnecessary risks for trial participants.

Commentary

Prepared by Iain Chalmers

Correspondence to: [email protected] Lind Initiative, Oxford, UK.

Askpeoplewhoarenotacademicswhether they think researchersshould assess what is known already before embarking on newresearch and they would wonder whether you had taken leaveof your senses. Is it not obvious that researchers should findout, systematically, what is already known before doing moreresearch? By not doing so, theywill not learn how to deal with theinadequacies of earlier investigations. Yet in a survey reportedby Nicola Cooper, David Jones and Alex Sutton, only 11 of 24responding authors of trial reports which had subsequently beenadded to existing systematic reviews were even aware of therelevant reviews when they designed their new studies.1

It gets worse, failure to assess what is already known can result instudies that address questions that have already been answered.For example, Fergusson and his colleagues2 showed that thebeneficial effect of aprotinin on the use of perioperative bloodtransfusion had been established after only a dozen trials hadbeen done. Yet over the subsequent decade, over 40 further trials


were reported, the most recent among them referring to only10% of the previous trials.Now, Karen Robinson and Steven Goodman have shown that thisproblem is pervasive. In their analysis of trial reports publishedover four decades, fewer than 25% of preceding trials were cited,comprising fewer than 25% of the participants enrolled in allrelevant prior trials. They conclude that ‘‘potential implicationsinclude ethically unjustifiable trials, wasted resources, incorrectconclusions, and unnecessary risks for trial participants’’.Analyses exposing this unethical scientific sloppiness are of cen-tral importance to themission of The Cochrane Collaboration andshould be used to lobby for increased resources for preparing andmaintaining systematic reviews. Reports of new research shouldbegin and end with systematic reviews.3

References1. Cooper N, Jones D, Sutton A. The use of systematic reviewswhen designing studies. Clinical Trials 2005; 2(3): 260–4.2. Fergusson D, Glass KC, Hutton B, Shapiro S. Randomizedcontrolled trials of aprotinin in cardiac surgery: could clinicalequipoise have stopped the bleeding? Clinical Trials 2005; 2(3):218–32.3. Clarke M, Hopewell S, Chalmers I. Clinical trials should beginand end with systematic reviews of relevant evidence: 12 yearsand waiting. Lancet 2010; 376(9734): 20–1.

Interpretation of random-effectsmeta-analyses

Riley RD, Higgins JP, Deeks JJ. BMJ 2011; 342: d549.

Background: Most meta-analyses use either a fixed-effect or arandom-effects model. The former assumes that all studies areestimating the same treatment effect, whereas the latter allowsfor differences in the treatment effect from study to study.Objective: To describe how to interpret a random-effects meta-analysis appropriately.Main content: The paper summarizes some empirical evidencethat many authors fail to interpret the results of a random-effectsmeta-analysis appropriately, since they refer to the mean effectacross studies as if it were a single true effect. The authors thenillustrate how this common interpretation may be misleadingbecause it does not incorporate the extent of any heterogene-ity. They propose that a prediction interval for the treatmenteffect in a new study is a suitable way to describe the extent ofbetween-study heterogeneity in a random-effects meta-analysis.Two examples are used to illustrate the ideas. The paper drawson a previousmethodological paper about random-effects meta-analysis.Conclusion: A random-effects meta-analysis can often be bestpresented using a prediction interval, which more fully revealsthe potential effect of a treatment in clinical practice.

CommentaryPrepared by Chris Cates

Correspondence to: [email protected] Sciences andEducation, StGeorge’s, Universityof London, London, UK.

One of the commonest problems I find in editing the method-ology section of Cochrane protocols is that review authors saythat they will use a fixed-effect model, unless there is significantheterogeneity, inwhich case a random-effectsmodelwill beused.This is unacceptable because the data are inspected first and thenanalysed according to what is found, effectively making a posthoc decision in choosing which method to use for meta-analysis.Richard Riley, Julian Higgins and Jon Deeks use this article to un-pack the differences between the two models of meta-analysis.The question answered by a fixed-effect model is: if there is asingle population treatment effect across all the trials, what is thebest estimate (anduncertainty) for this common treatmenteffect?Using a random-effectsmodel, we assume that there are differenttreatment effects across the studies (perhaps due to differencesin population, details of treatment or comparison groups andmeasurement of outcomes). This distribution of treatment effectsis considered to have a mean and a standard deviation, and therandom-effectsmodelwill give an estimate of these. Section 9.5.4of the Cochrane Handbook for Systematic Reviews of Interventionsgives a good summary of the different questions addressed byfixed-effect and random-effects models.1

A problem arising from this is that the random-effects confidenceinterval only describes uncertainty around the average treatmenteffect across the studies and does not tell us about the spread ofthe study results. For this reason the authors advocate consid-eration of the predictive interval, which gives us an idea aboutthe possible range of results for future studies. This interval canbe much wider than the confidence interval around the meanresult, and serves as a warning that we may have considerableuncertainty about the treatment effect that could occur in futurestudies.The method is of limited use with small numbers of studies, orwhen there is a substantial risk of bias in the included studies.However, it serves as a timely reminder that the confidence inter-val generated by a random-effects meta-analysis only relates towhere we estimate the average treatment effect to be, and doesnot encompass the spread of the results of the individual studies.

Reference

1. Higgins JPT, Green S, editors. Cochrane Handbook for Sys-tematic Reviews of Interventions 5.1.0 [updated March 2011]: TheCochrane Collaboration; 2011. Available from: www.cochrane-handbook.org.

Confidence intervals forrandom-effects meta-analysis androbustness to publication bias

Henmi M, Copas JB. Statistics in Medicine 2010; 29(29):2969–83.

Background: The DerSimonian-Laird confidence interval for theaverage treatment effect in meta-analysis is widely used in prac-tice when there is heterogeneity between studies. However, itis well known that its coverage probability (the probability thatthe interval actually includes the true value) can be substantiallybelow the target level of 95 per cent. It can also be very sensitiveto publication bias.


Objective: To propose a new confidence interval that has bettercoverage than the DerSimonian-Lairdmethod (when performinga random-effects meta-analysis) and which is less sensitive topublication bias.Main content: This paper first summarizes the current meth-ods proposed for improving the DerSimonian-Laird confidenceinterval before proposing a new approach. The key idea is tonote that fixed-effect estimates are less sensitive to biases thanrandom-effects estimates, since they put relatively more weighton the larger studies and relatively less weight on the smallerstudies. Whereas the DerSimonian-Laird interval is centred on arandom-effects estimate, the newmethod centres the confidenceinterval on a fixed-effect estimate, but allows for heterogeneityby including an assessment of the extra uncertainty induced bythe random-effects setting. In this paper properties of the result-ing confidence interval are studied by simulation and comparedwith other random-effects confidence intervals that have beenproposed in the literature.

Commentary

Prepared by Dan Jackson

Correspondence to: [email protected] Biostatistics Unit, Institute of Public Health, Cambridge, UK.

Henmi and Copas present an impressive paper and introduce anew method for performing a meta-analysis under the random-effectsmodel. They use the fixed-effect estimate of the treatmenteffect but allow for between-study heterogeneity when evaluat-ing the percentiles of the resulting pivotal statistic. Hence theirmethod may be regarded as an improvement on conventionalfixed-effectmeta-analysiswhichmakes the stronger, andperhapsimplausible, assumption that there is no between-study variationwhen constructing confidence intervals.Henmi and Copas prefer the fixed-effect estimate because it ismore robust topublicationbias than the random-effects estimate.

However they use standard estimates of between-study variance,for which publication bias also has implications.1 Henmi and Co-pas do not dwell on this consideration. Perhaps another possiblecriticism is that Henmi and Copas appear to take asymmetric fun-nel plots and publication bias as synonymous, but there are otherexplanations of funnel plot asymmetry.2 Finally, the robustnessof the fixed-effect estimate to publication bias comes at a price: ifthere is heterogeneity then this estimate is not themost efficient.We can therefore anticipate that the resulting confidence inter-vals will be wider than those using random-effects weights, andstatistical significance might be dampened or lost when usingHenmi’s and Copas’ method.Themathematics employed by Henmi and Copas is sophisticatedandmakes use of a novel gamma approximation for a conditionaldistribution of Cochran’s heterogeneity statistic. A gamma ap-proximation for the unconditional distribution of this statistic hasbeen used for some time, but a saddle point approximation andevenexactmethods arenowavailable.3 Itmaybe that these couldimprove Henmi’s and Copas’ method further. However, I have nodoubt that something very useful has been invented. I suggestthat Henmi’s and Copas’ method should become part of the stan-dard tool-kit. It is relatively straightforward to implement, andcode is provided, and I find the proposed method preferable to afixed-effect meta-analysis, although I am sure that conventionalrandom-effects methods still have a role to play.

References1. Jackson D. Assessing the implications of publication biasfor two popular estimates of between-study variance in meta-analysis. Biometrics 2007; 63(1): 187–93.2. Sterne JA, Egger M, Davey Smith G. Systematic reviews inhealth care: investigating and dealing with publication and otherbiases in meta-analysis. BMJ 2001; 323(7304): 101–5.3. Biggerstaff BJ, Jackson D. The exact distribution of Cochran’sheterogeneity statistic in one-way random effects meta-analysis.Statistics in Medicine 2008; 27(29): 6093–110.


Empirical Studies within theCochrane Collaboration

Thissectionpresentsacollectionof featurearticles highlighting some of the method-ological research currently being carried

outwithin TheCochraneCollaboration. Toregister ongoingmethodological research

within The Cochrane Collaboration pleasecontact [email protected].

Review production in The Cochrane Collaboration – where is it happeningandwhy?

Peter Gøtzsche, Britta Tendal andMike Clarke

Correspondence to:[email protected] Cochrane Centre, Copenhagen, Denmark.

Background: The Cochrane Collabo-ration is an international organization,with members from all healthcare spe-cialties and most parts of the world.Authors work on Cochrane reviews,supported mainly by Cochrane ReviewGroups, but also by Centres and Branches,with input from Methods Groups andFields, including the Cochrane ConsumerNetwork.

Objective: We are seeking to answerthe question: where do Cochrane reviewscome from, and why? As a starting point,we limited our explorations to areas of theworld (rather than areas of health care)and also to a few rather obvious possi-ble causes. We suspected that countrieswith many reviews per million inhabitantsmight be those that: had a CochraneCentre or a Branch; registered a Centreor a Branch early in the growth of TheCochrane Collaboration; had a CochraneReview Group (CRG) or a satellite of a CRG;had a tradition for performing many ran-domized trials of healthcare interventions.

Methods: We used Issue 10 2010 of theCochrane Database of Systematic Reviews,which had 6369 full reviews andprotocols,collectively called reviews in the following.The contact addresses for these reviewscovered a total of 75 countries, 28 ofwhichhosted a registered Centre or Branch.

We focused on 48 countries, whichwere the contact country for at leastfive reviews (see www.cochrane.dk/research/review production for the fulldata set).

Summary of main results: Twenty-sixof these 48 countries had a Centre orBranch, and 22 had not. Countries withat least one review per million inhabi-tants were predominantly those that hada Centre or Branch, 20 of 26 versus six of22 (P = 0.0002, Fisher’s exact test) (seeTable 1 where we also indicate if thereis a CRG or a satellite of a CRG in thecountry). Sixteen countries had a CRG orsatellite, and 15 of these countries are rep-resented in Table 1. Thus, countries withat least one review per million inhabitantswere predominantly those that had a CRGor a satellite, 15 of 16 versus 11 of 32(P = 0.0001).

Since the existence of a Centre or Branch isclosely related to the existence of a CRG orsatellite, it is not possible to separate theeffects of these two possible causal fac-tors on review production. There are alsonotable exceptions to the general picture.For example, although there is no Centreor Branch in Ireland, review productionthere is very high. This might be relatedto the fact that the UK Cochrane Centreis the reference centre in The Cochrane

Collaboration for Ireland, receives fundingfrom the Irish Health Research Board toprovide training and support in Ireland,and conducts similar activities in bothNorthern Ireland and Ireland. Thus, theUK Cochrane Centre could be consid-ered to be a UK and Ireland CochraneCentre.France and Russia are special cases ofcountries that used to have a Centre orBranch, but in which review productionis relatively rare. The French CochraneCentre was registered in 1996 and dereg-istered in 2002; the new Centre opened in2010. The review production in France iscurrently 0.43 per million inhabitants. TheRussian Branch of the Nordic CochraneCentre was registered in 1999 and dereg-istered in 2007. The review productionin Russia is 0.03 per million inhabitants.Only Turkey, Ethiopia, Indonesia, andBangladeshare lower,with0.01permillioninhabitants.Figure 1 shows the relation betweenCochrane review production and date ofregistration of the Centre or Branch inthese countries.There does not seem to be a temporalrelation between review production andthe age of the Centre or Branch. How-ever, the number of observations is small,and Bahrain and New Zealand might be


Table 1. Reviews per million inhabitant and association with Cochraneentity.

CountryReviews per million

inhabitantsCentre orBranch

Cochrane ReviewGroups or satellites

UK 35.8 yes 25 CRGs 3 satellitesAustralia 34.3 yes 5 CRGs 4 satellitesBahrain 30.0 yesNew Zealand 23.5 yes 1 CRGNetherlands 18.4 yes 2 CRGsDenmark 16.3 yes 3 CRGsIreland 15.5 no 1 CRGCanada 15.1 yes 7 CRGsSwitzerland 10.6 yesNorway 8.0 yes 1 satelliteFinland 6.0 yes 1 CRGIsrael 4.6 noSingapore 4.6 yesItaly 3.1 yes 2 CRGsAustria 2.3 noSouth Africa 2.2 yes 1 satellitePortugal 1.9 no 1 CRGGermany 1.8 yes 2 CRGsBelgium 1.6 yesSpain 1.6 yes 1 CRGUSA 1.5 yes 2 CRGs 2 satellitesUruguay 1.4 noChile 1.1 yesCosta Rica 1.1 noHong Kong 1.1 yesThailand 1.1 yes

Figure 1. Cochrane reviews or protocols for countries with a centre or branch.

exceptions (seebelow), inwhichcasetherewould be a relation with older CentresandBranchesbeingassociatedwithhigherproduction.Bahrain is the contact address for 24 re-views in a population of only 800,000people. The reason that this might bean exception is that the Director of theBahrain Branch, Zbys Fedorowicz, has de-voted much effort to Cochrane activities,including the mentorship of colleagues in

Bahrain and elsewhere, and is the contactperson for all these 24 reviews. Further-more, these reviews tend to be smallerthan the typical Cochrane review, withnarrowly focused questions, and containbetween zero and six trials (median twotrials).People in New Zealand have been activein The Cochrane Collaboration since itsearly years, which is not reflected in thecurrent status of Cochrane infrastructure

in that country, with only one CRG and arelatively recently established Branch. Atone stage, there were three CRGs in NewZealand: Musculoskeletal Injuries Group(now Bone, Joint and Muscle TraumaGroup) for seven years from 1998; the De-pression, Anxiety and Neurosis Group forsix years from 1996; and the sole remain-ing CRG, Menstrual Disorders and Subfer-tility Group, which was registered in 1996.There has also been a very active clusterof editors and authors from the Inconti-nence Group in Dunedin. This activity mayhave created a critical mass of interest inthe work of The Cochrane Collaborationin New Zealand. For example, for severalyears before the opening of its Branch oftheAustralasianCochraneCentre, thegov-ernment fundedaCochraneFellow, sittingwith the Cochrane Menstrual DisordersGroup and working with the AustralasianCochrane Centre to co-ordinate activities,in particular training for review authors inthe country. There has also long been aculture of evidence use in New Zealand,with a leadingGuidelineGroupand reviewactivity. For comparison, Denmark has asimilar sized population (5.5 million ver-sus 4.3 million in New Zealand), has threeCRGs (registered between 1996 and 2000,which are all still in the country) a Centre(registered in 1993) and has a similar num-ber of reviews permillion inhabitants: 16.3compared to 23.5 for New Zealand.Review production is also large in the UKandAustralia, aboutdouble thenumberofreviews per million inhabitants comparedto TheNetherlands, Denmark and Canada.These are followed by Switzerland, Nor-way, Finland, Singapore and Italy. One ofthe reasons for the productivity in Switzer-land might be the clusters of clinical re-search at the universities of Basel, Bern,Geneve and Zurich, and the fact that theWorldHealthOrganization (WHO) is basedin Geneva. It includes several people withstrong track records in systematic reviewswho promote evidence-based healthcareactivities and are involved in the produc-tion of The WHO Reproductive Health Li-brary,which isbasedonCochrane reviews.In contrast to Denmark (16.3 reviews permillion inhabitants), Norway (8.0 reviewsper million), and Finland (6.0 reviews permillion), review production is much rarerin Sweden (0.97 reviews per million). Thisis likely to be related to the strong tra-dition Sweden has for health technologyassessment,whichhas involvedmany spe-cialists in its work who see this as their pri-marytask, rather thanproducingCochranereviews.


Review production is also relatively lowin the USA, when considered alongsidethe size of its population (1.5 per million).The authors are not supported financiallyfor their time working on reviews and USdoctors have little time available outsidethat which is compensated, but this is un-likely to be a major reason, because thisis also the case in the UK and in othercountries. A more likely reason mightbe that many US clinicians do not realizethat a support system is in place to helpthem work on Cochrane reviews.1 For ex-ample, the US satellite of the CochraneEyes and Vision Group has successfully re-cruited US authors, who have providedfeedback that they value themethodolog-icalassistancetheyreceivedfromthesatel-lite. A second reason may be somewhatself-perpetuating in that when potentialcontributors to the work of The CochraneCollaboration who are based in the USlook at the list of authors or editors formost CRGs, they see disproportionatelyfew, or no US-based members, and mayconclude that theCochraneeffort has littleUS inputor leadership. One solution to thismight be to encourage more prominentUS clinicians to join the editorial boards ofthe 52 CRGs. A third reason could be theUS tradition for creating its own structures,even when international organizations al-ready exist, for example, most recentlythe Evidence-based Practice Centers. Afourth reason could be that the number ofCochrane entities is very small comparedto the population size.Figure 2 shows the relation betweenthe production of Cochrane reviews ina country and the production of ran-domized trials in that country (as de-fined by the publication type RandomizedControlled Trial in PubMed and the cor-respondence address for these articles).

Figure 2. Cochrane reviews or protocols related to production of randomised trials.

Contrary to what we expected, thereappears to be no relationship betweena high output of trials and a high output ofCochrane reviews. If anything, theremightbe an inverse relationship, with very highoutputs for trials in the Nordic countries(including Sweden) that are not matchedby as high an output of Cochrane reviews.Conclusions:When we set out on this ex-ploration,wewonderedif there isanythingwecan learn thatmighthelpTheCochraneCollaboration to produce reviews in moreefficient ways in the future. At the mo-ment, we remain uncertain about ways toachieve this, based on the current anal-yses, and we will continue to investigatethis and conduct more analyses. We invitecomments for a fuller article, in which wewould explore this issue, and additionalones.It is not easy to draw reliable conclu-sions about review production based onobservational data when, in addition,the most obvious predictive factors arecorrelated. One might ask, for example,whether Centres and Branches were es-tablished in countries where there was al-ready a high output of healthcare researchor systematic reviews, and a tradition forevidence-based health care? Or whetheroneormorepioneers encouraged thepro-duction of systematic reviews that wouldnot otherwise have happened, provedinstrumental in establishing a CochraneCentre, Branch or Review Group, and pro-moted evidence-based health care andresearch more generally?As we worked on this study, we realizedthat itmight be helpful to compare reviewproduction with the spread of an infec-tious disease. The enthusiasm for workingon a Cochrane review can be infectious,but the infected contacts are likely to beless contagious than the original case, and

second-order contacts even less so. Thiswould lead to higher disease prevalencein small populations than in large ones, asone or a few seed cases would have aproportionately greater effect in smallpopulations than in large populationswhen considering the number of Coc-hrane reviews permillion inhabitants. Thismechanism seems plausible, and it wouldbe expected to result in higher values forreview production in small countries. Thedata in the table agree reasonably wellwith this hypothesis. The UK is an ex-ception but this can be explained. This iswhere TheCochraneCollaboration startedand where in particular Iain Chalmersproved to be a particularly strong ini-tial seed infecting many people with the‘Cochrane virus.’ In addition, there are 25CRGs and 3 satellites in the UK.

As just indicated, clustering effects arean important issue, with these appear-ing around Centres and Branches, andaround CRGs. Claire Allen and Kiley Rich-mond from The Cochrane CollaborationSecretariat have recently investigated thisfor CRGs and found national clustering ofauthors around the editorial base. For ex-ample, when they considered all authorsof Cochrane reviews (not just the contactauthors) at the beginning of 2010,2 4.8%of those linked to the three CRGs in Den-mark are based in Denmark, while Danishauthors make up only 0.7% of the authorslinked to the 23 CRGs in the UK. Similarly,19.6% of the authors contributing to re-views published by the Danish CRGs arebased in theUK,while35.7%of theauthorscontributing to reviews published by theUK CRGs are based in the UK.

One explanation for this clustering is that,for some CRGs, the co-ordinating editoror other members of staff of the edito-rial base are co-authors on a large num-ber of the reviews produced by theirgroup. For example, Christian Gluud, Co-ordinating Editor of the Cochrane Hepato-Biliary Group, which is based in Denmark,has co-authored 64 of his group’s 222 re-views (as of January 2011), and Peter Tug-well, Co-ordinating Editor of the CochraneMusculoskeletal Group, which is based inCanada, has co-authored 49 of his group’s227 reviews. This would have more ef-fect on the Allen and Richmond type ofanalyses than on the work we have done,since our work is based on the location ofthe contact author only. It is much morelikely that someone from the CRG’s edito-rialbasewillbeaco-author, rather than thecontact author, given that being contact


author is usually far more labour-intensiveand time-consuming, not least because ofthe expectation that reviews are regularlyupdated.It seems reasonable to conclude that theexistence of Cochrane entities in a countryplays a major role for review productivity.We also note that our choice of denom-inator – million inhabitants – has limita-tions. It does not distinguish betweenlarge and small countries, does not takeGross National Product, language prob-lems or number of clinicians into account.Furthermore, there have been instancesof funding for a considerable number ofreviews in some countries, and we havenot taken intoaccount thedegreetowhichthe sameauthors co-authormany reviews.Finally, we have not looked at the intensityof training review authors.

We plan to continue our work on thisproject, and some of the issues we shouldlike to address in the future are: whethersomeone in a Centre, Branch, or CRG isdirectly responsible for the productivityin their country because they are thecontact author for multiple reviews;whether a Centre, Branch or CRG influ-ences the productivity for their countrybecause of the support they provide toauthors locally; whether the patterns wehave seen for the contact authors, aredifferent for other authors; whether the lo-cation of the editors of a CRG influences itsoutput, andtheproductivityof thecountryin which the editor is based.Acknowledgments: We thank KayDickersin, Sally Green, Mark Jeffery,Cindy Farquhar, Alvaro Atallah, Alessan-dro Liberati, Iain Chalmers, and Zbys

Fedorowicz for their comments and in-sight aswehave explored these issues. Weshould welcome comments from otherson our interpretation and plans for futureanalyses.

References

1. Dickersin K. Health-care policy. To re-form U.S. health care, start with system-atic reviews. Science 2010; 329(5991):516–7.2. Allen C, Richmond K. The Cochrane Col-laboration: International activity withinCochrane Review Groups in the firstdecade of the twenty-first century. Jour-nal of Evidence Based Medicine 2011; 4(1):2–7.

Risk of Bias tool evaluation: summary results from focus groups and onlinesurveys

Jelena Savovic, LauraWeeks, Julian Higgins, Doug Altman, Lucy Turner, DavidMoher and Jonathan Sterne

Correspondence to:[email protected] of Social and Community Medicine, University of Bristol, UK.

Objective: To obtain feedback from arangeof stakeholderswithinTheCochraneCollaboration regarding their experienceswith and perceptions of the Risk of Bias(RoB) tool and associated guidance mate-rials.

Methods: We used qualitative and quan-titative methods to evaluate the RoBtool. We held four focus groups with25 international participants, using asemi-structured format with a list ofpre-specified topics. The focus groupswere fully transcribed and analysed, andtheir results informed the developmentof questionnaires for online surveys. Weconducted two surveys, one for reviewauthors, and one for managing editorsand other Cochrane Review Group staff.We enquired about experience and per-ceptions of the RoB tool, bias domain-specific issues, incorporation of risk ofbias assessments in meta-analyses, andtraining requirements. Authors whohad not previously used the RoB toolwere only asked about training require-ments and the reasons for not usingthe tool. The surveys were distributedthroughestablishedCochranemailing lists

and administered online between 1 and22 February 2010.

Summary of main results: We received190 responses from review authors whohad used the RoB tool and 132 from au-thors who had not. Of the 58 CochraneReview Group staff who responded, 19were Managing Editors, 11 Co-ordinatingEditors, 11 Editors, and 17 other CochraneReview Group staff. Authors take on av-erage 10 to 60 minutes per study to com-plete risk of bias assessments, and 83%deemed this acceptable. Over half ofrespondents have used the RoB tool toupdate an existing review, and 93% ofthem stated they assessed risk of bias forboth existing and newly included studies.The majority of authors (84%) completethe recommended ‘Risk of bias’ table inRevMan, while 36% also include at leastone ‘Risk of bias’ figure or table. Over72% of respondents stated they often oralways included quotes from the study re-port to support their judgment, and themajority thought that this feature addstransparency and increases confidence inrisk of bias assessments. Authors reportedsome difficulties in completing each bias

domain, but the domains thought to bemost difficult were ‘Incomplete outcomedata’ and ‘Selective outcome reporting’.Nevertheless, over 90% of respondentsfelt ‘somewhat’ to ‘very confident’ in theirrisk of bias assessments.

The survey showed that authors neededclearer guidance on what to do with riskof bias assessments once completed: four-teen per cent did not incorporate theirrisk of bias assessments into review con-clusions at all, 55% include a narrativesummary, 40% conduct a sensitivity anal-ysis based on risk of bias assessments, and11% restrict theprimary analysis to studiesof low risk (authors could tick all optionsthat applied).

Almost a third of responders used a mod-ified version of the RoB tool to assessrandomized trials. Modifications includedthe addition of new domains, modifiedcriteria for Yes/Unclear/No judgements, orremoval of some domains (blinding, whennot feasible, for example). Modificationswere usually based on their own exper-tise, or following guidelines from theirCochrane Review Group. A fifth of authorstold us they used the RoB tool to assess


studies other than randomized trials (e.g.quasi-random, cohort, before-after stud-ies), and 80% of them made some modi-fications to the tool in order to use it fornon-randomizedtrials. Modificationswereusually based on the author’s own exper-tise and literature, but with no consistentor standard approach.In terms of completed training, 65% ofauthors told us they read relevant train-ing materials in their own time, 39%attended at least one workshop at asymposium or Cochrane Colloquium, and23% attended the Cochrane Collabora-tion’s standard author training. Over 92%of responders read the guidance in theCochraneHandbook for Systematic Reviewsof Interventions relating to the RoB tool,while 16% told us they read the entireCochrane Handbook (part 2). Most users(75%) thought the level of detail in theCochrane Handbook is appropriate, while22% wanted more detail. Seventy-fiveper cent of authors thought availabilityof training materials for personal use (e.g.Cochrane Handbook, other written guid-ance) was sufficient, and 68% found theavailability of training events (workshops,for example) sufficient. Most respondentsfelt they would be most likely to accessonline training, including webinars (55%).Overall, 87% of authors and 95% ofCochrane Review Group staff thought riskof bias assessments were better than past

approaches to trial quality assessment.Most liked the standardized approach(81%) and the ability to provide quotesto support judgments (74%). About athird of participants did not like the in-creased workload, and found the word-ing describing judgments of risk of bias(yes/no/unclear) confusing. Many authorsand CRG staff suggested that the wordinglow/high/unclear risk of bias should beused instead.Conclusions: Overall, respondents iden-tified positive experiences and percep-tions of the RoB tool. Revisions of thetool and associated guidance, and im-proved provision of training, may improveimplementation. Expansion of the RoBtool for non-standard randomized trialsand non-randomized studies is highlydesirable.Based on these findings, the Risk of BiasEvaluation Panel recommended changesto the tool which have already been im-plemented in RevMan 5.1 and the revisedversion of Chapter 8 of the CochraneHandbook for Systematic Reviews of In-terventions (see page 3). Authors needmore guidance regarding how to in-corporate risk of bias assessments intometa-analysis and review conclusions. Weidentified a clear need for the develop-mentof extensionsof theRoB tool to covernon-randomized studies and trial designsother than the standard two-arm parallel

design. Provision of an online bank of col-latedexamples for riskofbiasassessments,aswell asmoreweb-based training,wouldbe desirable.Our findings were initially presented tothe Risk of Bias Evaluation Panel and theManaging Editors’ Meeting at the UK-and Ireland-based Contributors to TheCochrane Collaboration Conference inCardiff, March 2010. An abstract was pre-sented in CochraneMethods in September2010, followed by an oral presentationand a workshop at the Cochrane Col-loquium in Keystone in October 2010.The full report was submitted to theCochrane Collaboration OpportunitiesFund Panel. A manuscript describing therevised RoB tool was submitted to theBMJ.1 A manuscript presenting detailedsurvey results is currently in preparationand will be submitted to a peer reviewedjournal.

Reference

1. Higgins JPT, Altman DG, Gøtzsche PC,Juni P, Moher D, Oxman AD, Savovic J,Schulz KF, Weeks L, Sterne JAC, CochraneBias Methods Group, Cochrane StatisticalMethods Group. The Cochrane Collabo-ration’s tool for assessing risk of bias inrandomized trials. BMJ (in press).

Consumer involvement in The Cochrane Collaboration

Dell Horey

Correspondence to:[email protected] Trobe University, Melbourne, Australia

Background: TheCochraneCollaborationpromotes the involvement of consumersin itsworkbut there has been little system-atic analysis of how this occurs across theorganization. Mostofwhat is knownaboutconsumer involvement has focused on ac-tivities in Cochrane Review Groups and in-clude descriptive accounts.1–5 Like all ob-servational studies, these may not be rep-resentative. Twosurveysofconsumershadlow response rates.6–7 Cochrane ReviewGroups have also been surveyed,7–8 butnot other entities. All registered entities inThe Cochrane Collaboration are requiredtopublishamodule inTheCochraneLibrarywithcontactdetailsand information about

their scope, membership and activities.Entity modules are expected to be up-dated at least annually, although this maybe beyond the resources of smaller enti-ties. Despite possible shortcomings, entitymodulesprovide themost comprehensivepicture of what entities believe importantin their work.Objective: To describe consumer partici-pation across TheCochraneCollaboration.Methods: This study used an empiri-cal evaluation approach or case studyevaluation,9 which is an appropriatemethod to investigate a complex in-tervention with no clear boundaries(consumer involvement) in a complex set-

ting (The Cochrane Collaboration and itsactivities). Case study evaluations can useamixof qualitative andquantitativemeth-ods to develop an analytical framework tointerpret findings. In this study a two-stage content analysis of all publishedentity modules in The Cochrane Libraryled to theoretical conceptualizations ofconsumer descriptions and roles. Mod-ules were downloaded on 15 December2009, transferred to Word documentsand imported into the NVivo computer-assisted data analysis program. The initialsearch of module content used the terms‘consumer’, ‘user’ and ‘patient’ to identifyhow consumers and their activities were


described. From this, conceptualization ofthe types of consumers operating in theCollaboration and the roles undertaken bythem were developed. A second analysisof module content focused on specific ac-tivities undertaken by consumers, whichwere mapped and categorized withineach role.Summary of main results: Eighty-twoentity modules were included in the anal-ysis (13 Cochrane Centres; 51 CochraneReview Groups; 12 Fields; six MethodsGroups). Information about consumerinvolvement was found most frequentlyin the modules of the Cochrane ReviewGroups (47/51, 92%). The next mostfrequent area was in Cochrane Centremodules (11/13, 84%), then Fieldmodules(9/12 75%). None of the six MethodsGroups’ modules included informationabout consumer involvement. Threemainareas of involvement for consumers in TheCochrane Collaboration were identified:decision-making in the Collaboration;choice and preparation of reviews; anddissemination and promotion of reviewsand evidence-based health care.Three types of consumers were described.Mostconsumerswerevolunteersandwereassociated with input based on direct ex-periential knowledge of a health issue.Consumer volunteer activities includedpeer reviewing, providing feedback aboutreview priorities, providing advice on howto involve other consumer members, anddisseminating review outcomes. Anothergroup of consumers can be described asconsumer facilitators. Theseoperate indif-ferent guises across the Collaboration topromote and support consumer involve-ment, for example, Consumer Editors orConsumer Co-ordinators. In some ReviewGroups and Centres this work is funded.The third type of consumer involvementcame from consumer organizations; they

werea sourceof volunteers andcontributeby helping to prioritize reviews, identifyrelevant outcomes, raise funds and/or dis-seminate review findings to consumers.Involvement in the preparation ofCochrane reviews was the most com-mon area of activity, although this var-ied considerably across Cochrane ReviewGroups. Consumer peer reviewing or ref-ereeing and consumer co-ordinationwerethe most frequently cited activities (foundin 18 modules, 35%), followed by workrelated to Plain Language Summaries (15modules, 29%). Involvement inthepromo-tion and dissemination of the work of TheCochrane Collaboration and in decision-making was commonly mentioned in themodules. Ten modules (20%) identifiedconsumers with dissemination and eightreported consumer members on boardsor reference groups (8, 16%).Conclusions: Consumers are involvedacross The Cochrane Collaboration inmany ways, but there is considerablevariation in the ways they take part. Thereare someareasof theCollaboration thatdonot involve consumers. Strategies to sup-port consumers who want to be involvedin the Collaboration need to take accountof the different practices that have de-veloped. Consumer involvement shouldbe monitored to make the most effectiveuseof this contribution to theorganizationand itswork. These findingswere includedin a discussion paper for the CochraneCollaboration Steering Group in 2010 andincluded in a poster presentation at theCochrane Colloquium in Keystone in 2010.A paper for peer-reviewed publication iscurrently in preparation.

References1. GhersiD.Making ithappen: approachesto involving consumers in Cochrane

reviews. Evaluation and the Health Pro-fessions 2002; 25(3): 270–83.

2. Skoetz N, Weingart O, Engert A. A con-sumer network for haematological malig-nancies. Health Expectations 2005; 8(1):86–90.

3. Sakala C, Gyte G, Henderson S, Neil-son JP, Horey D. Consumer-professionalpartnership to improve research: the ex-perience of the Cochrane Collaboration’sPregnancy and Childbirth Group. Birth2001; 28(2): 133–7.

4. Collier A, Johnson K, Heilig L, Leonard T,WilliamsH,Dellavalle RP.Awin-winpropo-sition: fostering US health care consumerinvolvement in the Cochrane Collabora-tion Skin Group. Journal of the Ameri-can Academy of Dermatology 2005; 53(5):920–1.

5. Shea B, Santesso N, Qualman A, Hei-berg T, Leong A, Judd M, Robinson V,Wells G, Tugwell P, Cochrane Muscu-loskeletal Consumer Group. Consumer-driven health care: building partnershipsin research. Health Expectations 2005; 8(4):352–9.

6. Horey D. It takes time to find your role:a survey of consumers in the Cochrane Col-laboration in 2002. Melbourne: CochraneConsumer Network, 2003.

7. Wale J, Colombo C, Belizan M, NadelJ. International health consumers in theCochrane Collaboration: fifteen years on.Journal of Ambulatory Care Management2010; 33(3): 182–9.

8. Kelson MC. Consumer collaboration,patient-defined outcomes and the prepa-ration of Cochrane Reviews. Health Expec-tations 1999; 2(2): 129–35.

9. Keen J, Packwood T. Qualitative re-search: Case study evaluation. BMJ 1995;311(7002): 444–6.

Developing and evaluating ‘Summary of findings’ tables for Cochrane reviews –two studies

Sarah Rosenbaum and Claire Glenton

Correspondence to:[email protected] Knowledge Centre for Health Services, Oslo, Norway.

Background: Limited time is a frequently-cited barrier to clinicians’ use of evidencein practice.1–2 Summaries of systematicreviews could help make evidence more

useful and easy to find for clinicians withlimited time.3–4 GRADE ‘Summary of find-ings’ tables offer a useful starting point bybringing the most important information

to the foreground, irrespectiveof resultsorthe lack of them, and explicitly highlight-ing the quality of the evidence for eachoutcome.5


Objectives: To develop and evaluate a‘Summary of findings’ table for use inCochrane reviews that is understandableanduseful forhealthprofessionals, accept-able to CochraneCollaboration stakehold-ers and feasible to implement.

Study 1 – Developing the‘Summary of findings’ tableArticle title: ‘‘User testing and stakeholderfeedback contributed to the developmentof understandable and useful Summary ofFindings tables for Cochrane reviews’’6

Methods:We gathered stakeholder feed-back on the format and content of a‘Summary of findings’ table from an ad-visory group of over 50 participants andtheir constituencies through e-mail con-sultations. Participants had a range ofroles in The Cochrane Collaboration, in-cluding statisticians and other methodol-ogists, review authors, editors, consumerrepresentatives, publishers, and membersof the SteeringGroup. We then conducteduser tests employing a think-aloud pro-tocol method, collecting feedback from21 health professionals and researchersin Norway and the UK. We analysed thefeedback, definedproblem areas and gen-erated new solutions in brainstormingworkshops.Results: Cochrane stakeholders wereprimarily concernedwithmaintainingpre-cision in the data representation, for in-stance by including: the basis for assumedcontrol group risks; a recognition of theimprecision of control group rates/values;reasons for GRADE quality scores; andthe duration of the intervention to ob-tain measured effect. Stakeholders werealso concerned with table productionfeasibility.User test participants, on the other hand,were worried about the dense tablebeing too complex and difficultto understand. User testing re-vealed comprehension problems,such as:

• mistaking ‘x fewer per 1000’ (ab-solute difference) for ‘x per 1000’(intervention group risk);

• confusion about what the differ-ent numbers referred to (class ref-erence);

• difficulty understanding continu-ous outcomes;

• difficulty understanding termsand abbreviations;

• uncertainty about the meaning ofempty cells or ‘no data available’.

The user test participants also suggestedways of making tables more useful forclinicians:

• describe the population and set-ting;

• describe the inclusion criteria forhigh/low risk populations;

• describe the intervention;• add recommendations.

Conclusions: We added footnotes to ac-commodate many of the stakeholders’concerns andwe implemented all but oneof the suggestions from the user test par-ticipants to increaseusefulness (wedidnotadd recommendations). However, resolv-ing the tension between achieving tableprecision and table simplicity became ourmain developmental focus. We dealt withcomprehension issues (without sacrificingprecision) by adapting the following prin-ciples, and recommend these to otherspublishing evidence in table format:

• Avoid class confusion:7

◦ use same class reference, es-pecially in number sets thatare to be compared (for in-stance, to avoid misunder-standing, we changed ‘xfewer per 1000’, to ‘x per1000’);

◦ support correct class inter-pretation by adding class la-bels (‘studies’, for example);

◦ describethescales thatwereused for continuous out-comes in close proximity tothe results.

• Avoid unfamiliar abbreviationswherever possible, even if theyhave been introduced in the text.

• Explain empty cells to make un-certainty or lack of data explicit.

• Help the reader quickly formthe correct interpretation8 of thenumbers by:

◦ using text cues where appli-cable;

◦ aligning type to make com-parison of numbers easier;

◦ layering the information vi-sually so that the most im-portantparts ‘popout’ at thereader.

Study 2 –Measuring the effect ofa ‘Summary of findings’ tableArticle title: ‘‘Summary of Findings tables inCochrane reviews improved understandingand rapid retrieval of key information’’9

Objective: To measure the effects of in-cluding a ‘Summary of findings’ table in aCochrane review (compared to the samereview with no table) on user satisfaction,understanding and time spent finding keyresults.

Methods: We randomized participantsin an evidence-based practice workshop(RCT I, 72participants) andaCochraneCol-laboration entities meeting (RCT II, 33 par-ticipants), to receive a Cochrane reviewwith orwithout a ‘Summary of findings’ ta-ble. In RCT Iwemeasureduser satisfaction.In RCT II wemeasured correct comprehen-sion and time spent finding key results.

Summary of main results: In these twosmall trials we found that inclusion of a‘Summary of findings’ table in a review im-proved understanding and rapid retrievalof key findings compared to reviews withno ‘Summary of findings’ table.

RCT I: Participants with the ‘Summary offindings’ table (n = 47) were more likelyto ‘Agree’ or ‘Strongly agree’ that it waseasy tofindresults for importantoutcomesthan (n = 25) participants without the‘Summary of findings’ table: 68% versus40% (P = 0.021).

RCT II: Participants with the ‘Summary offindings’ table (n = 18)weremore likely toanswer two questions correctly regardingresults than (n = 15) participants with-out the ‘Summary of findings’ table: 93%versus 44% (P = 0.003) and 87% versus11% (P < 0.001). Participants with the‘Summary of findings’ table spent an aver-age of 90 seconds to find key informationcompared to four minutes for participantswithout the ‘Summary of findings’ table(P = 0.002).

Implications: ‘Summary of findings’tables should be included upfront inCochrane reviews and other similar evi-dence syntheses, as they have the poten-tial for making reviews quicker to readand easier to understand correctly. Read-ers of other systematic review summaries,such as Plain Language Summaries (forconsumers) or SUPPORT summaries (forpolicy makers in low- and middle-incomecountries) may also benefit from the in-clusion of ‘Summary of findings’ tables orsimplified versions of them.10–11

Dissemination of findings: The tableformat we developed is the basis forGRADE evidence-table output. The Coc-hrane Handbook for Systematic Reviewsof Interventions now recommends that


review authors compile a ‘Summary offindings’ table, and that this should beplaced directly after the abstract. We arecurrently implementing different adapta-tions of ‘Summary of findings’ tables inother products, including Plain LanguageSummaries,10 SUPPORT summaries11 andevidence-based policy briefs.

References1. Coumou HC, Meijman FJ. How do pri-mary care physicians seek answers to clin-ical questions? A literature review. Jour-nal of the Medical Library Association 2006;94(1): 55–60.2. Ely JW, Osheroff JA, Ebell MH, Chamb-liss ML, Vinson DC, Stevermer JJ, Pifer EA.Obstacles to answering doctors’ questionsabout patient care with evidence: qualita-tive study BMJ 2002; 324(7339): 710.3. Sackett DL. Using evidence-basedmedicine to help physicians keep up-to-

date. Journal for the Serials Community1996; 9(2): 178–81.

4. Glasziou P, Haynes B. The paths fromresearch to improved health outcomes.Evidence-Based Nursing 2005; 8(2): 36–8.

5. GRADEWorkingGroup. Gradingqualityof evidence and strength of recommenda-tions. BMJ 2004; 328(7454): 1490.

6. Rosenbaum SE, Glenton C, Nylund HK,Oxman AD. User testing and stakeholderfeedback contributed to the developmentof understandable and useful Summary ofFindings tables forCochrane reviews. Jour-nal of Clinical Epidemiology 2010; 63(6):607–19.

7. Gigerenzer G, Edwards A. Simple toolsfor understanding risks: from innumeracyto insight. BMJ 2003; 327(7417): 741–4.

8. Reyna VF. A theory of medical deci-sion making and health: fuzzy trace the-ory. Medical Decision Making 2008; 28(6):850–65.

9. Rosenbaum SE, Glenton C, Oxman A.Summary-of-findings tables in Cochranereviews improved understanding andrapid retrieval of key information. Jour-nal of Clinical Epidemiology 2010; 63(6):620–6.10. Glenton C, Santesso N, Rosenbaum S,Nilsen ES, Rader T, Ciapponi A, Dilkes H.Presenting the results of Cochrane Sys-tematic Reviews to a consumer audience:a qualitative study. Medical Decision Mak-ing 2010; 30(5): 566–77.11. Rosenbaum SE, Glenton C, WiysongeCS, Abalos E, Mignini L, Young T, AlthabeF, Ciapponi A, Marti SG, Meng Q, Wang J,la Hoz Bradford AM, Kiwanuka SN, Rutebe-mberwa E, Pariyo GW, Flottorp S, OxmanAD. Evidence summaries tailored to healthpolicy-makers in low- and middle-incomecountries. Bulletin of the World Health Or-ganization 2011; 89(1): 54–61.

Thomas C Chalmers M.D.Award – 2010

The Thomas C Chalmers M.D. prize isawarded annually for the best oral orposter presentation at the CochraneColloquium. In 2010, in Keystone, thebest oral presentation was awarded to

Jamie Kirkham, Richard Riley and PaulaWilliamson for their study entitled ‘‘Ismultivariate meta-analysis a solution forreducing the impact of outcome report-ing bias in systematic reviews?’’ The

best poster presentation was awarded toLavinia Ferrante di Ruffano, Chris Hyde,Kirsten McCaffery, Patrick Bossuyt and JonDeeks for their study originally entitled‘‘What do test-treat trials measure?’’

Is multivariate meta-analysis a solution for reducing the impact of outcomereporting bias in systematic reviews?

Jamie Kirkham, Richard Riley and PaulaWilliamson

Correspondence to:[email protected] for Medical Statistics and Health Evaluation, University of Liverpool, Liverpool, UK.

Background: The prevalence and impactof outcome reporting bias (ORB) in trialsonCochrane reviewshas beenexamined.1

ORB was suspected in at least one trial in34% of 283 Cochrane reviews. However,in this situation trials may report otheroutcomes that are correlated with themissing outcome of interest; one canthen utilise amultivariatemeta-analysis to

synthesize jointly all the reported out-comesand‘borrowstrength’across them.2

This approach has the potential to reducethe impact of ORB.

Objectives: (i) To compare the perfor-mance of the multivariate meta-analysisapproach compared to the univariateapproach; (ii) To determine whether the‘borrowing of strength’ in a multivariate

meta-analysis can reduce the impact ofORB, and by howmuch.

Methods: A simulation study wasconducted considering the fixed-effectbivariate meta-analysis setting. In eachsimulation, the bias and coverage of thepooled meta-analysis result was assessed.Data were simulated for situations wherea) both outcomes were reported in all


studies, and b) outcome data were infor-matively missing.Summary of main results: Results showthat the ‘borrowing of strength’ in amultivariate meta-analysis can reducethe magnitude of the impact of ORB.Bias, mean-square error and cover-age are improved using the multi-variate meta-analysis approach, withimprovements increasing as the correla-tion between outcomes increased.

Conclusions: The reliability of meta-analysis can be improved if more atten-tion is paid to missing outcome data. If ahigh percentage of data is missing, reviewauthors should be encouraged to contactthe trialists to confirm whether an out-come was measured and analysed and,if so, to obtain the results. If this is notpossible, a multivariate meta-analysis ap-proach is a potential statistical solution forreducing the impact of ORB.

References

1. Kirkham JJ, Dwan KM, AltmanDG, Gam-ble C, Dodd S, Smyth R, Williamson PR.The impact of outcome reporting bias inrandomised controlled trials ona cohort ofsystematic reviews. BMJ 2010; 340: c365.2. RileyRD.Multivariatemeta-analysis: theeffect of ignoringwithin-study correlation.Journal of the Royal Statistical Society SeriesA 2009; 172(4): 789–811.

How do tests impact on patient health? Development of an explanatoryframework

Lavinia Ferrante di Ruffano, Chris Hyde, KirstenMcCaffery , Patrick Bossuyt and Jon Deeks

Correspondence to:[email protected] of Public Health, Epidemiology and Biostatistics, University of Birmingham, Birmingham, UK.

Background: Quantifying the impact ofan intervention on patient health is fun-damental to clinical effectiveness eval-uations, and diagnostic tests are noexception. Diagnostic accuracy forms thecurrent focus of research into the perfor-mance of tests, and it is often assumedthat improvements in health outcomesinevitably flow from improving a test’s ac-curacy. However, direct measurementsof whether a particular diagnostic testdoes in fact enhance patient health arecurrently very rare. Here, the methodol-ogy of choice is the test-treat randomizedcontrolled trial (TT-RCT), where patientsare randomized to one of two or morediagnostic strategies andhealthoutcomesare measured after the implementation ofsubsequent treatment.This test-treatment pathway is a complexprocess involving not only multiple inter-ventions (test and treatment), but multi-ple episodes of decision-making. Unliketreatment randomized trials, where the

Patient given test

Patient outcome

Test results produced

Diagnostic decision made

Treatment decision made

Treatment implemented

Figure 1. Five key components of a generic test-treatment pathway: a test is appliedto a patient in order to identify a target condition, the test result is considered alongsideother evidence to decide a diagnosis, fromwhich a course of treatment is identified andimplemented. Adapted from Bossuyt and Lijmer 1999.5

intervention is expected to influencepatient health directly, in TT-RCTs the pa-tient is separated from the interventionof interest by the intervening decision-making and treatment implementation.Consequently the influence that a testexerts on patients is mediated by theseintervening processes. Although thepresence of this indirect relationshipis well iterated in the methodologicalliterature,1−4 the exact nature of how wecan expect diagnostic tests to changehealth outcomes needs more detailedinvestigation.Objective: As part of an MRC-fundedproject to develop methods for evaluat-ing the clinical effectiveness of diagnostictests, we set out to map comprehensivelyhow a diagnostic test can change healthoutcomes. This was conducted at theUniversity of Birmingham (August 2007 toJuly2010), in co-operationwith colleaguesbased in the UK (CH), the Netherlands (PB)and Australia (KM).

Methods: A basic model of a generictest-treat pathway was initially designed,reducing the test-treat process to fivekey components (Figure 1). Character-istics of these components hypothesizedto produce changes in patient outcomeswere sought by reference to publisheddiagnostic research frameworks, identi-fied through a recent systematic review;4

these were used to annotate our basicmodel.The resulting schema was developed andrefined through examination of publishedevaluations of test-treatment interven-tions. Seventy-eight TT-RCTs were usedfor this process, identified from 8974 titlesgenerated through a systematic searchof the Cochrane Central Register of Con-trolled Trials (Issue 2, 2007). RelevantTT-RCTs measured patient outcomes af-ter administration of treatment and werepublished between January 2004 andMay2007. Searching was unrestricted by clin-ical topic and test modality, but limitedto diagnostic settings. Where the frame-work failed to explain how the test un-der evaluation altered patient outcomes,new mechanisms were constructed andthe schema updated.Summary of main findings: The finalframework outlines 15 mechanisms forwhich we have found data that demon-strate tests can impact on patient health(Table 1).Changes to mechanisms at the begin-ning of the pathway can trigger changesto those further along, and so achievechanges to health outcomes. This wasobserved to occur in four ways:


Table 1. Test-treatment mechanisms that may influence the effectiveness of a test-treat strategy.

Care PathwayComponent Mechanism Definition

Test Delivery 1. Timing of Test The rapidity of performance of a test within themanagement strategy.2. Feasibility Completion of the test process, where reasons for non-completion

are:Counter-indication (Clinician refusal to administer test)Patient acceptability (Patient refusal to have test)Technical failure (Ability of diagnostic equipment to produce data)

3. Test Process Patients’ interaction with the test procedure, potentially causingphysical or psychological harms or benefits.

Test Result 4. Interpretability After successful completion of the test process, the degree to whichtest data can be used to inform a diagnostic classification.

5. Accuracy The ability of a test to distinguish between diseased andnon-diseased patients.

6. Timing of Result The speed with which test results are available.DiagnosticDecision

7. Timing of Diagnosis The speed with which a diagnostic decision is made.8. Diagnostic Yield The degree to which the test contributes to a patient diagnosis in any

form, including:Provision of definitive diagnosisConfirmation of suspected diagnosisRuling out a working diagnosisDistinguishing between alternative diagnoses with different

treatment implicationsDifferentiated from accuracy in that it also incorporates any other

information used by a clinician to formulate a diagnostic decision(such as prior test results).

9. Diagnostic Confidence The degree of confidence clinicians have in the validity orapplicability of a test result.

TreatmentDecision

10. Therapeutic Yield The degree to which diagnostic decisions influence therapeutic plans.11. Therapeutic Confidence The certainty with which a clinician pursues a course of treatment.

TreatmentImplementation

12. Timing of Treatment The speed with which patients receive treatment.13. Treatment efficacy The ability of the treatment intervention to improve patient

outcomes.14. Adherence The extent to which patients participate in the management plan, as

advised by their physician, in order to attain the therapeutic goal.All components 15. Perceptions of testing The extent to which patients’ experience, understanding or

perception of testing modifies the potential health effects offeredby other mechanisms.

1. Direct test effects: The test itself(Table, mechanism 3; M3) cancause physical or psychologicalchanges to patients, irrespectiveof subsequent decision-making.

2. Altering clinical decisions and ac-tions: As commonly recognized,tests that are more reliable (M2)or more clearly interpreted (M4)can increase accuracy (M5) and di-agnostic confidence (M9), leadingto more appropriate classification(M8), treatment planning (M10)and a more confident approachto treatment (M11), all of whichcan improve the success of an ef-fective treatment (M13). Failedprocedures could also inviteadditional investigations (M2),

increasing total diagnostic time(M7) or decreasing diagnostic andtherapeutic yields (M8, M10)through incorrect decision-making and poor diagnosticconfidence (M9).

3. Changing the timeframes of de-cisions and actions: Tests whichare undertaken earlier (M1), pro-duce results more quickly (M6),or otherwise enable a diagnosisto be made more promptly (M7)have all been shown to improvedownstream health outcomes,though only if they succeed in ac-celerating the time to treatment(M12).

4. Altering patient or clinician per-spectives: Emotional, cognitive,

social and behavioural responsesto the test-treat process can cre-ate health consequences in manyways (M15).7 Tests that are un-acceptable may invite different,possibly less accurate tests, whileclinicians may modify clinical de-cisions for other reasons, such asthe fear of malpractice claims. Al-ternatively, patients can gain re-assurance through the impressionof a thorough investigation, a ‘di-agnostic placebo effect’, influenc-ing inclinationtofollowsuggestedtest and treatment advice (M14).

Conclusions: Diagnostic tests canchange patient health in numerous ways;however the process through whichthey achieve this is complex and often


multifaceted. Improved accuracy is com-monly considered as the sole test at-tribute capable of generating improvedpatient health. Our research empha-sizes we should move beyond this.Accuracy can indeed be a powerful trig-ger for improving health outcomes, butit is one mechanism amongst severalin a chain of interactions (see point 2.above), and consequently its potentialbenefits may be nullified by insufficien-cies in related mechanisms, such as diag-nostic yield or confidence. Equally, sig-nificant health improvements can be real-ized in the absence of superior accuracy,by improving the timeliness of diagnosisand treatment, for example. In focusingon accuracy and in decision-making wemay therefore risk missing other poten-tial benefits, such as when a test al-ters the way in which treatment is de-livered.The UK MRC recommends researchersidentify ‘the likely processes of change’8

before conducting randomized trial eval-uations of complex interventions. Ourframework can be used to do this fortest-treatment comparisons, assisting theplanning anddesignof TT-RCTsby inform-ing the selection of outcomes that captureall true effects. An accompanying checklist

hasbeendesigned toguide the systematicidentification of mechanisms responsiblefor driving changes in patient health in atest-treatment comparison. The ability toidentify which elements of the interven-tion shouldgeneratedifferences inpatienthealth will also aid the appraisal and inter-pretation of published TT-RCTs, by serv-ing as a check that key mechanisms areoperating as theorized. This approach en-courages a comprehensive understandingof the intervention, and will be of useto trialists, policy makers and guidelinedevelopers.We present this analysis as a prelimi-nary framework, open to deliberation,modification and further development asmethodological research on test evalua-tion progresses.Disseminationof findings: This research,including the accompanying checklist, iscurrentlyunder submission forpublicationin a journal.Acknowledgements: Funded by UK MRCMethodology Programme (Grant 78381).

References1. Fryback DG, Thornbury JR. The efficacyof diagnostic imaging. Medical DecisionMaking 1991; 11(2): 88–94.

2. Hunink MG, Krestin GP. Study designfor concurrent development, assessment,and implementation of new diagnosticimaging technology. Radiology 2002;222(3): 604–14.3. Lord SJ, Irwig L, Bossuyt PM. Using theprinciples of randomized controlled trialdesign to guide test evaluation. MedicalDecisionMaking 2009; 29(5): E1–12.4. Gatsonis C. Design of evaluations ofimaging technologies: development of aparadigm. Academic Radiology 2000; 7(9):681–3.5. Bossuyt PMM, Lijmer JG. Traditionalhealth outcomes in the evaluation of di-agnostic tests. Academic Radiology 1999;6(Suppl 1): S77–80.6. Lijmer JG, Leeflang M, Bossuyt PM. Pro-posals for a phased evaluation of medicaltests. Medical Decision Making 2009; 29(5):E13–21.7. Bossuyt PM,McCaffery K. Additional pa-tient outcomes and pathways in evalua-tions of testing. Medical Decision Making2009; 29(5): E30–8.8. Craig P, Dieppe P, Macintyre S,Michie S, Nazareth I, PetticrewM.Develop-ing and evaluating complex interventions:the new Medical Research Council guid-ance. BMJ 2008; 337: a1655.


Cochrane MethodologyReview Group

Mike Clarke and Nicola McDowell

Correspondence to:[email protected] Methodology Review Group, All-Ireland Hub for TrialsMethodology Research, Queen’s University, Belfast, NorthernIreland.

During 2010 and into 2011, the editorial base for the CochraneMethodology Review Group continued to be based in Dublin,Ireland and Oxford, England; with Nicola McDowell acting asManaging Editor and Sarah Chapman providing advice and sup-port for study identification. The editorial team continues to beco-ordinated by Mike Clarke and Andy Oxman. The other editorsare Paul Glasziou, Peter Gøtzsche, Gordon Guyatt (Criticism Edi-tor), PhilippaMiddleton and Karen Robinson. We are very gratefulto Peter Juni who stood down during the year. Following Mike’smove from the UK Cochrane Centre to Queen’s University Belfastin Northern Ireland, we expect that the main responsibilities forthe editorial base will transfer there later this year, to be takenup by the new staff team within the All-Ireland Hub for TrialsMethodology Research.During the 12 months to April 2011, one Cochrane methodologyreview was updated and two reviews appeared in full for the firsttime. A further four new reviews or updates are at advancedstages within the editorial process of the Review Group, andshould be published before the end of 2011.The updated review was led by Jan Odgaard-Jensen in Oslo,and tackles the evidence on the effects of randomization toprotect against selection bias in healthcare trials. This updat-ing also involved some changes to the scope and structure ofthe review. It continues to show that the effect estimates fromrandomized and non-randomized trials of the same treatmentcomparisons differ in ways that are unpredictable. The authorsconclude ‘‘it is not generally possible to predict themagnitude, oreven the direction, of possible selection biases and consequentdistortions of treatment effects from studies with non-random al-location or controlled trials with inadequate or unclear allocationconcealment’’.1

Towards the end of 2010, Vivian Welch and a team from withinthe Cochrane Equity Methods Group published their new reviewof how systematic reviews assess the effects of interventions onhealth equity. They found ‘‘there is a need for improvement inconceptual clarityabout thedefinitionofhealthequity, describingsufficient detail about analytic approaches (including subgroup

analyses) and transparent reporting of judgments required forapplicability assessments in order to assess and report effects onhealth equity in systematic reviews’’.2

The other new Cochranemethodology reviewwas funded in partby the UK National Institute for Health Research (NIHR) CochraneReviewIncentiveScheme. KerryDwanandhercolleaguesbroughttogether evidence from 16 studies that compared the content ofprotocols and registry entries for randomized trials withwhatwasdescribed in the published reports of those trials. They showedthat ‘‘discrepancies between protocols or trial registry entries andtrial reports were common, although reasons for these were notdiscussed in the reports’’ and concluded ‘‘full transparency willbe possible only when protocols are made publicly available orthe quality and extent of information included in trial registriesis improved, and trialists explain substantial changes in theirreports’’.3

The Cochrane Methodology Register (CMR) continued to growthis year with infrastructure support from the UK Cochrane Cen-tre, which is funded by the NIHR. Sarah Chapman, Anne Eisingaand Sally Hopewell at the UK Cochrane Centre work with Mike onCMR. It grewby a thousand records through the year and containsmore than 14,600 references to studies and other reports relevantto the methods of systematic reviews and other evaluations ofhealth and social care in mid 2011.If you are interested in contributing to the work of the CochraneMethodology Review Group, as an author, referee or in someother way, please contact Nic ([email protected]).

References

1. Odgaard-Jensen J, Vist GE, Timmer A, Kunz R, Akl EA,Schunemann H, Briel M, Nordmann AJ, Pregno S, Oxman AD.Randomisation to protect against selection bias in healthcaretrials. Cochrane Database of Systematic Reviews 2011, Issue 4. Art.No.: MR000012.2. Welch V, Tugwell P, Petticrew M, de Montigny J, Ueffing E,Kristjansson B, McGowan J, Benkhalti Jandu M, Wells GA, Brand K,Smylie J. How effects on health equity are assessed in systematicreviews of interventions. CochraneDatabaseofSystematicReviews2010, Issue 12. Art. No.: MR000028.3. Dwan K, Altman DG, Cresswell L, Blundell M, Gamble CL,Williamson PR. Comparison of protocols and registry entries topublished reports for randomised controlled trials. CochraneDatabase of Systematic Reviews 2011, Issue 1. Art. No.: MR000031.


Information from theMethods Groups

Methods Groups have an important function in The CochraneCollaboration. They are initiated in response to the ongoing andevolving science of systematic reviews in assessing and evalu-ating healthcare interventions. The Groups currently registeredrepresent a breadth of methodology addressing, for example,bias, non-randomized studies, qualitative research, patient re-ported outcomes and, most recently, the Comparing MultipleInterventionsMethods Group. This Group is developingmethodsfor Cochrane overviews of reviews and conventional interven-tion reviews that assess the effects of multiple interventionsfor the same clinical condition. This involves statistical meth-ods known synonymously as multiple-treatments meta-analysis,mixed-treatment comparisons and network meta-analysis.Further information is provided below.Current important and challenging areas of methodological de-velopment are how best to incorporate the assessment andreporting of harms, assessments of cost effectiveness and issuesof equity. In addition, this includes reviews that need to incor-porate qualitative and quantitative (including non-randomizeddesigns) methodology such as assessing complex interventions,and prognostic studies.The Cochrane Collaboration has set up the Methods InnovationFund to support methodological development within the Col-laboration. After a Collaboration wide consultation exercise sixoverarchingmethodological topics were identified and agreed aspriorities for methodological development by the Methods Ap-plication and Review Standards (MARS) Working Group. MethodsGroups have been asked to lead on the submissions for thesepriority topics in collaboration with other Cochrane entities andindividuals who have relevant interests and expertise. These keymethodological priorities are:

• Further development of the Risk of Bias tool, in particu-lar to address risk of bias in studies other than parallelgroup randomized trials: for example, non-randomizedstudies evaluating adverse effects, and clinical trials withnon-standard designs such as cross-over and cluster ran-domized trials.

• Work to improve the acceptance, implementation andconsistency of ‘Summary of findings’ tables in Cochranereviews, with consultation with Cochrane Review Groupsto consider refinements, alternatives or extensions to theexisting methodology.

• Investigation of issues associated with reviews of com-plex interventions, particularly in relation to quantitativeand qualitative synthesis of their results.

• Research and guidance on methods for dealing withmissing data, particularly when participants are missing

from the results presented, but potentially also whenkey statistics aremissing that prevent the straightforwardinclusion of the study in a meta-analysis (e.g. miss-ing standard deviations, correlation coefficients, samplesizes).

• Work to establish the role of overviews of reviews, inparticular with regard to the relatively new methodol-ogy ofmultiple treatments meta-analysis (network meta-analysis).

• Searching for unpublished material - to investigate theuse of licensing authorities as an alternative (or sup-plement) to standard literature searching in reviews ofpharmaceutical interventions (and possibly devices); andto develop new routine strategies for the identification ofunpublished trials and the retrieval of unpublished datafor Cochrane reviews.

It is expected that the projects will start in January 2012.A core function of many Methods Groups is to provide training.This is often done by holding workshops at the Cochrane Col-loquia. This year saw the first annual Methods Groups trainingevent funded through the TrainingWorking Group’s programme.The 2011 training event on ‘Summary of findings’ tables from theApplicability and Recommendations Methods Group was held incollaboration with the Robert Koch Institute in Germany. TheCochrane Collaboration has made a commitment to support au-thors in using GRADE and creating ‘Summary of findings’ tables.The objective of the workshop was to build an internationalnetwork of people who have the expertise to provide ‘Summaryof findings’ tables support to Review Groups and their authors.The workshop used an interactive format to develop expertise inGRADE and ‘Summary of findings’ tables. The training event for2012 will be organized by the Bias Methods Group and will beheld in the UK.There are currently 15 Methods Groups. Another Methods Group(Agenda Setting and Prioritisation) is proposed and in the pro-cess of seeking registration, details of which are below. Contactdetails for each of theMethodsGroups canbe foundon the insidefront cover of this edition.

Registered groups

Adverse Effects Methods GroupApplicability and Recommendations Methods GroupBias Methods GroupComparing Multiple Interventions Methods GroupCampbell and Cochrane Economics Methods GroupCampbell and Cochrane Equity Methods Group


Individual Participant Data Meta-analysis Methods GroupInformation Retrieval Methods GroupNon-Randomised Studies Methods GroupPatient Reported Outcomes Methods GroupPrognosis Methods GroupProspective Meta-Analysis Methods GroupQualitative Research Methods Group

Screening and Diagnostic Tests Methods GroupStatistical Methods Group

Possible group

Agenda Setting and Prioritisation

Registered GroupsCochrane Adverse Effects MethodsGroup

Su Golder, Andrew Herxheimer, Yoon Loke and Sunita Vohra

Reflections of the Cochrane Adverse Effects Methods Group: aresystematic reviewers doing enough?

Harms are defined as the ‘‘totality of possible adverse conse-quences of any intervention, therapy or medical test; they arethe direct opposite of benefits, against which they must becompared’’.1 Harms and other unintended events are inherentlyunpredictable and methods or instruments used to capture allpossible adverse effects can be problematic. Most often, harmsare collected in clinical trials as secondary outcomes, but themethods used to identify and capture harms differmarkedly fromthose used for beneficial outcomes. Since many harms are notanticipated in terms of their nature, severity, seriousness, timing,reversibility, etc., most studies cannot specify exact protocolsto capture adverse effects actively. This makes it difficult toassess the internal validity of studies being critically appraised insystematic reviews.

The potential for threats to internal validity for the collection andreporting of harms are distinct from those that apply to beneficialoutcomes. These threats to validity that are specific toharmsneedto be assessed differently and may require new criteria. Thereare also challenges to interpreting the type of biases associatedwith harms; the understanding of traditional biases associatedwith study design have been defined largely in the context ofbeneficial or primary outcomes; harms are usually considered tobe secondary outcomes and are not collected and reported in thesame way as the beneficial outcomes.

A second important threat to validity when assessing adverseeffects exists when ‘absence of evidence of harm’ is confusedwith ‘evidence of lack of harm’, as in statements such as ‘no signif-icant harm’, ‘no significant differences in adverse events betweengroups’, or ‘the drug was safe and well-tolerated’. The newlydeveloped Risk of Bias tool evaluates components that have beenempirically linked to exaggerated estimates of benefit, with a keyaim being to identify bias that predisposes to false conclusionsabout therapeutic effectiveness. In contrast, the opposite viewis important for adverse effects, where we are most concerned

about Type II errors which can induce a false sense of security(for example, wrongly concluding that there was no significantdifference in harm between drug and placebo, and erroneouslyjudging the drug to be safe).The Type II errors stem from the difficulties in rigorously defin-ing unexpected outcomes, inadequate monitoring and under-reporting, insufficient sample size or follow-up duration to mea-sure rare events, exclusion of patients with risk factors for adverseevents, and slicing up of adverse effects data into many subcat-egories (with few events in each subcategory). Review authorsneed to find and assess the specific study designs that are mostlikely to yield robust data on the adverse effects of interest, ratherthan rely on studies that cannot reliably detect adverse effects –and may yield ‘false negatives’.If the data for quantitative analysis of adverse effects are sparse,we believe that the primary focus should be on rigorous assess-ment of the potential for Type II errors and the level of uncertaintymanifest within the 95% confidence intervals, rather than deriv-ing potentially biased conclusions of ‘no significant harm’. Wetherefore propose the development of new criteria to assess thepotential for Type II errors specific to harms. They might, forexample, ask the following questions: What were the characteris-tics of the participants? Which adverse events were predefined?What specific monitoring was performed and for how long? Wecan then more formally assess claims that no harm was found.The Cochrane Collaboration seeks to improve its relevance toclinicians, policy-makers, and other decision-makers. Interven-tions, whether therapeutic or diagnostic, must be assessed forboth effectiveness and harms. The Collaboration has excelledin the former, but there is still much room for improvement inharms assessment. We must develop criteria to facilitate a morecomplete assessment of the harms of an intervention better toinform readers and users of our systematic reviews.

Reference1. Ioannidis JP, Evans SJ, Gøtzsche PC, O’Neill RT, Altman DG,Schulz K, Moher D, CONSORT Group. Better reporting of harmsin randomized trials: an extension of the CONSORT statement.Annals of Internal Medicine 2004; 141(10): 781–8.

For more references see http://aemg.cochrane.org.

Cochrane Bias Methods Group

Doug Altman, Isabelle Boutron, DavidMoher, JonathanSterne and Lucy Turner

Over the past six years the Bias Methods Group (BMG) hascontinued to raise awareness of methodological approachesto and codes of practice for dealing with bias in systematicreviews. The BMG is active within the Cochrane community:


hosting workshops and training sessions, giving presentations,conducting priority topic research and guiding methods re-views. The BMG has a membership of 141 methodologists,authors, statisticians, editors and others throughout 20 coun-tries who share an interest in bias.In recent years perhaps the BMG‘s most significant contributionto the Collaboration has been the extensive evaluation of andupdate to the Cochrane Risk of Bias (RoB) tool (see page 3).The objective of the tool is to provide an easily accessible, com-prehensive means of assessing and reporting bias in reviews.The update of the RoB tool has involved focus group ses-sions, discussion meetings and an extensive survey completedby Cochrane authors, editors and review staff. The updatedand improved version of the tool is available in the CochraneHandbook for Systematic Reviews of Interventions, Version 5.1.In line with the update, many guidance materials are availableincluding online training materials, a summary of changes doc-ument disseminated Collaboration wide and an open accessonline discussion forum. Further information on the processand conduct of the update to the tool has been submitted forpublication.In other work, since May 2010 the BMG has conducted a consid-erable number of training events, including: six workshops pri-marily featuringmethods for investigating bias, critical appraisaland outcome reporting bias; three author training sessions; anda train-the-trainer session. In addition, threewebinars havebeenhosted by BMG convenors, DavidMoher discussing PRISMA andthe launch of PROSPERO (the international site for systematicreview registration) (seepage4), and IsabelleBoutrondiscussingthe feasibility and assessment of blinding in primary studies.The Group has two accepted abstracts for workshops at theColloquium in Madrid featuring the standard investigating biasworkshop and aworkshop specific to the update of the RoB tool.Methodology research currently in progress by BMG membersinclude:

• The Cochrane Collaboration’s tool for assessing risk of biasin randomized trials (Julian Higgins);

• recommendations for examining and interpreting funnelplot asymmetry inmeta-analyses of randomized controlledtrials (Jonathan Sterne); and

• CONsolidated Standards Of Reporting Trials (CONSORT)and the quality of reporting of randomized trials (DavidMoher).

Recently published BMGmethodology reviews:

• Comparison of protocols and registry entries to publishedreports for randomized controlled trials (Kerry Dwan);

• Checking reference lists to find additional studies for sys-tematic reviews (Tanya Horsley);

• Blinded versus unblinded assessments of risk of bias instudies included in a systematic review (David Moher).

The BMG is also involved in initiatives related to the reportingof health research. Without adequate reporting, it is impossi-ble to identify and assess risk of bias in primary studies. Ourconvenors are among the founders of the EQUATOR Network(www.equator-network.org) which seeks to improve the qualityof scientific publications by promoting transparent and accu-rate reporting of health research. Our members were alsointegral to the recent 2010 update of the CONSORT Statement(www.consort-statement.org).The BMG will be holding our next meeting at the CochraneColloquium in Madrid in October 2011 and we should like toinvite all those interested to attend. We thank our currentfunders of ten years, the Canadian Institutes of Health Research,without whose support our continued progress would not bepossible. For further information about the group, please visitthe BMG website (www.ohri.ca/bmg) or contact our ResearchCo-ordinator, Lucy Turner ([email protected]).

Cochrane ComparingMultipleInterventions Methods Group

Lorne Becker, Deborah Caldwell, Julian Higgins, ChristopherSchmid and Li Tianjing

The aim of the newly formed Cochrane Comparing Multiple In-terventions Methods Group (CMIMG) is to advise The CochraneCollaboration on how intervention reviews and overviews canbest provide reliable comparisons of multiple interventions. Itwill consider howwe canmeet the needs of a healthcare decisionmaker approaching The Cochrane Library asking, ‘‘Which inter-vention should I use for this condition?’’. The CMIMG combinesa continuation of the work of the previous ‘Umbrella ReviewsWorking Group’ with expertise in the statistical methodologyfor undertaking meta-analyses of multiple interventions. Thisrapidly developing area, known variously as ‘multiple treatmentsmeta-analysis’ (MTM), ‘mixed treatment comparisons’ (MTC) or‘network meta-analysis’, allows randomized trials making differ-ent comparisons to be analysed simultaneously, while respectingthe randomization within each study.

The suggestion to form this newMethods Group arose at a meet-ing of opponents and enthusiasts of meta-analyses of multipleinterventions during the Cochrane Colloquium in Singapore in

2009. An exploratory meeting was held in Bristol, UK, in March2010 and the CMIMG was successfully registered as a formalentity of The Cochrane Collaboration in September 2010. Thegroup currently has 54 members, many of whom have alreadymade contributions as authors or peer reviewers of Cochranereviews and overviews, overviews in the Cochrane Journal ofEvidence-Based Child Health, and comparative effectiveness re-views published elsewhere. Two members of the Group servedonthepanel for theU.S. InstituteofMedicine’s reportonStandardsfor Systematic Reviews of Comparative Effectiveness Research.

During our relatively brief existence, we have held twomeetings,have established a website (cmimg.cochrane.org) with a grow-ing list of resources and have started a discussion forum on theCollaboration’s intranet. At our first official meeting, held duringthe Cochrane Colloquium in Keystone in October 2010, we heardpresentations from four different CochraneReviewGroups (CRGs)about their experiences in producing Cochrane reviews that com-pare multiple interventions and discussed a variety of strategiesfor supporting CRGs in this task. A second meeting funded bythe Cochrane Discretionary Fund and organized by the ItalianCochrane Centre was held in March 2011 in Milan. The 20 partic-ipants held a variety of roles within the Collaboration (includingCentreDirector,ManagingEditor, Trials SearchCo-ordinator, FieldCo-ordinator, Co-ordinating Editor, Methods Group Convenor).


Discussion focused on developing a set of recommendations forthe further evolution of both overviews and intervention reviewsto allow comparisons ofmultiple interventions to be takenwithina consistent and logical framework.A summary of the meeting is being developed and will be avail-able on our website. We established three working groups tocarry forward the decisions made at that meeting. One groupwill focus on logistical issues involved in undertaking, publishingand maintaining reviews that compare multiple interventions.A second will concentrate on statistical considerations. A thirdwill focus on interpretation and display of the results includingassessment of risk of bias and evaluation of quality.A list of training workshops on topics relevant to our groupis available on our website. Workshops were initially run bymembers of the Statistical Methods Group, with the first beingheld at the Cochrane Colloquium in Dublin in 2006. In 2009,the growth of interest in MTM and its application in overviews

led to the workshop being split into three (basic, intermedi-ate and advanced), and these workshops were repeated atthe Cochrane Colloquium in Keystone in 2010. Workshops atnational Cochrane meetings have also been held for severalyears: for example, at UK- and Ireland-based Contributors toThe Cochrane Collaboration meetings, at various branches ofthe Iberoamerican Cochrane Centre, at the 3rd South AsianSymposium organized by the South Asian Cochrane Centre andat the Statistical Methods Group training meeting in March2010. For the latter event, resources from the Cochrane Dis-cretionary Fund and the Training Working Group were usedto create online podcasts (http://smg.cochrane.org/teachingtraining-workshops/smg training course 2010/session5 salantiand http://smg.cochrane.org/teachingtraining-workshops/smgtraining course 2010/session6 dias).Wearealso formulatingplans forproviding individualizedsupportof ongoing or new Cochrane reviews withmultiple interventions.

Campbell and Cochrane EconomicsMethods Group

Ian Shemilt and Luke Vale

Economics is the study of the optimal allocation of resourcesfor the production of benefit to society. It is concerned withhow resources - such as professional time and skills, equipment,supplies and materials, buildings and facilities, and energy -are used to meet needs. It starts out from the simple truththat there are never enough resources to meet all needs andthat, consequently, choices need to be made between alterna-tive uses in order to make the most of the resources that areavailable.

There is increasing demand amongst healthcare decision mak-ers around the world for economic perspectives and evidencein addition to evidence for the beneficial and adverse effects ofhealthcare interventions. Decision makers looking to Cochranereviews as a source of reliable evidence to inform adoption andresource allocation decisions often find that economic evidenceis either not included at all, or is inconsistently or injudiciouslytreated, compared with the evidence for health effects. Atbest this means that Cochrane reviews lack relevance and im-pact, and at worst that they could be positively misleading.Therefore, judicious application of appropriate methods to in-corporate economic perspectives and evidence into Cochranereviews can increase their applicability as a component of thebasis for decision-making.

The Campbell and Cochrane Economics Methods Group(CCEMG) advocates that authors of Cochrane reviews considereconomic dimensions of the interventions or diagnostic testsbeing reviewed, and the health conditions they address, at thetitle registration stage. If it is judged likely that evidence forresource use, costs and/or cost-effectiveness will be consideredtobean important componentof adoptiondecisions around theworld about the interventions being reviewed, authors shouldfurther consider whether, to what extent, and how economicperspectives and evidence will be treated at different stages ofthe review process.

Minimally, economic commentaries can be incorporated intothe background and discussion sections of the review in or-der to place an economic lens on the interventions or tests

being reviewed and the health conditions they address. A moresystematic (but more resource intensive) option is to conducta systematic review of evidence for resource use, costs and/orcost-effectiveness, collected from relevant published (and un-published) economic evaluations, as a fully integrated compo-nent of the Cochrane review process. Then, to complement,present this alongside the systematic review of evidence forhealth effects. The main objective of an integrated systematicreview of economic evidence should be on helping end users tounderstand: the structure of adoption and resource allocationdecisions they are likely to face (see Chapter 15 of the CochraneHandbook for Systematic Reviews of Interventions for elaborationof this objective).The CCEMG recognizes that whilst, in principle, economicsmethods are relevant to, and have potential applications inall Cochrane reviews, the capacity and resources available toconduct reviews frequently do not allow for all available eco-nomic evidence to be considered or integrated. This has ledto a stronger focus in our methods development work overthe last 12 months on what can be achieved without ma-jor additional time and resource implications for authors andeditorial bases. For example, we have recently completed astudy sponsored by the Cochrane Editorial Unit that will leadto further recommendations and guidance on how to developeconomic commentaries for incorporation into background anddiscussion sections of intervention reviews. In parallel, we haveundertaken to host a network of Cochrane Review Group andCentre based individuals, with the aim of increasing capacity forthe application of economicmethods in reviews and to link eco-nomics methods expertise more closely with the core businessof the Collaboration.As well as running methods training workshops at theCochrane Colloquium in Keystone and at the UK- and Ireland-based Contributors to The Cochrane Collaboration Meeting inBelfast in 2010, we held a panel session on health technologyassessments (HTAs) and systematic reviews of health economicevidence at the HTAi 2010 Meeting in Dublin. We have also de-velopedan ‘Introduction toeconomicsmethods’ online learningmodule in collaboration with the UK Cochrane Centre and theUniversity of Portsmouth and worked with Wiley, the CochraneEditorial Unit and Cochrane Web Team colleagues to roll outfree access to the Health Economic Evaluations Database to allCochrane contributors via the cochrane.org intranet.


The last 12 months has seen a series of changes in the orga-nization of the CCEMG core group. Ian Shemilt left his role asthe Group’s full-time Research Co-ordinator in April, followingthe discontinuation of our principal source of funding. Ian hastaken up a new post in the Behaviour and Health ResearchUnit at the University of Cambridge, where he will design andexecute a programmeof evidence synthesis in public health. Ianwill continue to be an active contributor to the Collaborationin his roles as a CCEMG Co-convenor, member of the Meth-ods Executive and author. Whilst we will aim to minimize theinevitable major impact of the loss of our principal source offunding on CCEMG activities and outputs, we ask members ofthe Collaboration to bear with us during this transitional period.The new initial point of contact for the CCEMG is Janice Legge([email protected]).The CCEMG also has two new Co-convenors. Frikkie Booysen(Professor of Economics, University of the Free State, SA) re-places Damian Walker in taking a lead role for the CCEMG withrespect to issues relevant to low- and middle-income coun-tries. Mehmet Ugur (Jean Monnet, Reader in Political Economy,

University of Greenwich, UK) will take a new lead role for theCCEMG in the applied field of international development. Theseappointments recognize the need to ensure that the CCEMGCo-convenors Panel includes sufficient capacity and relevantexpertise to respond to the recent expansion of applied fieldscovered by The Campbell Collaboration. These include inter-national development, crime and justice, education and socialwelfare, and the recent surge of interest within both Cochraneand Campbell in the development and application of methodsfor systematic reviews and synthesis of evidence collected fromeconometric and observational studies.

For those interested in reading further about approaches to evi-dence synthesis that combine economics and systematic reviewmethods to inform policy and practice decisions, CCEMG Co-convenors have edited a published volume Evidence-BasedDeci-sionsandEconomics, which includes in-depthcoverageof the lat-est methods developments, proposals and controversies in thisfield (see http://eu.wiley.com/WileyCDA/WileyTitle/productCd-1405191538.html).

Campbell and Cochrane EquityMethods Group

Mark Petticrew, Peter Tugwell and Erin Ueffing

The Campbell and Cochrane Equity Methods Group aims toimprove the quality of Campbell and Cochrane reviews on inter-ventions to reduce socioeconomic inequalities in health and topromote their use to the wider community. Ultimately, this willhelp build the evidence base on such interventions and increaseour capacity to act on the health gap between rich and poor.Equity guidance for systematic reviewers: this year, the EquityMethods Group published equity guidance in the BMJ. This paperhighlights seven key components for equity-related systematicreviews: 1) developing a logic model; 2) defining disadvantage;3) choosing appropriate study designs; 4) identifying appropriateoutcomes; 5) understanding context and including process eval-uations; 6) analysing and presenting data; and 7) assessing theapplicability of review findings. The seven components providea framework for the upcoming CochraneHandbook for SystematicReviewsofInterventionschapteronequityandspecialpopulations,being prepared in collaboration with Cochrane colleagues.This year, we also published our first equity Cochrane Methodol-ogyReview,withsupport fromtheUKNationalHealthService. Ledby Vivian Welch, the Cochrane review examined how effects onhealth equity are assessed in systematic reviews of interventions.Funding:Our Group was part of a major funding proposal led bythe Canadian Cochrane Network and Centre. We were successfulin obtaining funding from the Canadian Institutes for Health Re-search (CIHR) for a five-year term (2010–2014), specifically fromthe Institute of Gender and Health (IGH) to support our researchprogramme.Sex and gender-based analyses: Our five-year research planincludes a significant programme on sex and gender-based anal-yses and methods for systematic reviews. As part of this pro-gramme, we collaboratedwith the Institute of Gender andHealthto launch a Cochrane Corner to present summaries of the sex andgender relevance in select Cochrane reviews. The Corner alsoincludes feature articles highlighting particular methodological

issues around the social determinants of health and systematicreviews (www.cihr-irsc.gc.ca/e/42414.html). In May 2011, we par-ticipated in a two-day meeting on gender, equity, and bias insystematic reviews, funded by CIHR.Complex interventions: In collaboration with Jeremy Grimshawof the Cochrane Effective Practice and Organisation of CareGroup and the Canadian Cochrane Centre, we were awardeda grant from CIHR to hold a meeting on methodological issuesaroundcomplex interventions in systematic reviews. Themeetingwill bring stakeholders includingpolicymakers, systematic reviewauthors, and methodologists together to form new internationalpartnerships on complex interventions. We will also be giving aworkshop on complex interventions at the Cochrane Colloquiumin Madrid in October 2011.PRISMA equity extension: Vivian Welch is leading an initiativeto develop an equity extension for the PRISMA (Preferred Report-ing Items for Systematic Reviews and Meta-Analyses) Statement.With Peter Tugwell, Mark Petticrew, and David Moher, we havesecured support from the Rockefeller Foundation for an interna-tional consensus-building conference at the Bellagio ConferenceCentre in Italy. The conference meeting will focus on the out-come of an evidence-based, globally endorsed equity extensionof PRISMA.Webinars and training: We are developing a series oftraining modules/workshops on basic equity principles andhow the differential effects of interventions on disadvantagedpopulations can be considered in systematic reviews. Ourfirst webinar module, Equity 101, was presented three timeslast year to international audiences, and the first session’sYouTube video had been viewed nearly 500 times by July 2011(www.youtube.com/watch?v=w6UNUjsNMWU). Our next mod-ule on logic models was presented in collaboration with LaurieAnderson of the Cochrane Public Health Group, and will be of-fered again throughout 2011. Please contact us to learn aboutsessions convenient for your timezone ([email protected]).Contact: The Equity Methods Group has nearly 550 mem-bers from 45 countries, and distributes two newsletters annu-ally. For more information or to join our listserv, please con-tact Erin Ueffing ([email protected]) or visit our websitewww.equity.cochrane.org.


Cochrane Individual Participant DataMeta-analysis Methods Group

Larysa Rydzewska, Jayne Tierney, Lesley Stewart,Mike Clarke andMaroeska Rovers

The Individual Participant Data (IPD) Meta-analysis MethodsGroup has 74 members (32 active, 41 passive) from 17 coun-tries, with interests spanning prevention, diagnosis, treatment,rehabilitation and prognosis in a wide range of healthcare areasincluding cancer, epilepsy, stroke, perinatal care and malaria.With this diversity in mind, The Cochrane Collaboration hasagreed to our request to change our name - replacing ‘pa-tient’ with ‘participant’. We are now the Individual ParticipantData Meta-analysis Methods Group, helpfully retaining the IPDabbreviation.As part of recent changes to the way that Methods Groupsparticipate in The Cochrane Collaboration, our core functionsare now defined as:

• providing training;• providing peer review on the IPD elements of Cochrane

reviews;• providing specialist advice on IPD;• conductingCochranemethodology reviews relevant to IPD

topics;• contributing to the Cochrane Methodology Register.

At the Cochrane Colloquium in Keystone in October 2010, weran a training workshop on when and how to use IPD in sys-tematic reviews. The training covered the process of carryingout an IPD review and provided review authors with practicalguidance, focusing on the differences between conducting anIPD review compared with an aggregate data review. We alsoplan to run a further training workshop at the Colloquium in

Madrid in October 2011, led by Catrin Tudur Smith and RichardRiley, on statistical methods for the meta-analysis of IPD. Thisworkshop will cover methods for modelling IPD, combiningIPD and aggregate data, and estimating treatment-covariateinteractions.The convenors are holding regular teleconferences to discussissues relevant to the Group, with a view to raising the profile ofIPD methodology. We encourage members to contact us withany specific issues theywould like us to consider. We also plan tomeet at theMadrid Colloquium. We encourage Groupmembersto attend and will e-mail details of the meeting and welcomesuggestions for agenda items.We previously reported that some of our members had expe-rienced difficulties in obtaining funding for IPD projects. Aftersurveying the Group, we have compiled a list of both criticismsand positive feedback received from funding bodies. From thisfeedback, we have created a list of ‘top tips’ that may be usefulwhen submitting funding applications for IPD-based projects(we plan to add this list to our website but, in the meantime, ifyou would like a copy please contact us).We would like to take this opportunity to congratulate theCo-convenors who have recently taken up new posts or respon-sibilities. Mike has stepped down as Director of the UK CochraneCentre to take up the Chair of Research Methodology at theCentre for Public Health, based at Queen’s University, Belfast inNorthern Ireland. Jayne has been made the Deputy Director ofthe MRC London Hub for Trials Methodology and Research inEngland and Maroeska has taken up the position of AssociateProfessor of Clinical Epidemiology and Evidence-Based Surgeryat the Radboud University Nijmegen Medical Centre (RUN-MC)in The Netherlands.If youwould like to join the IPDMethods Group or are interestedin finding out more, please contact us ([email protected]) orvisit www.ctu.mrc.ac.uk/cochrane/ipdmg.

Cochrane Information RetrievalMethods Group

Julie Glanville, Carol Lefebvre, Jessie McGowan, AlisonWeightman and Bernadette Coles

The Information Retrieval Methods Group was establishedfollowing a series of meetings in 2004 and in 2010 we wel-comed our 200th member to the Group. In fact, we now have206 members from over 26 countries worldwide all of whom cankeep in touch and discuss issues of interest via our e-mail discus-sion list (contact [email protected] to join). The membershiprepresents a wide variety of people, some information profes-sionals but also other review authors with an interest in searchingskills and many are active in a number of collaborative projectsdesigned to enhance or illuminate the process of identifying theevidence on which systematic reviews are based.

The Group held its Annual General Meeting during the CochraneColloquium in Keystone in 2010 and for the first time it was ajoint meeting with our colleagues in the Campbell Collaboration.Links between the two Groups were strengthened and mem-bers of the Campbell Information Retrieval Methods Group have

joined our Group and one of our Co-convenors, Carol Lefebvre,has been invited to be a member of the C2-SPECTR Task Force toadvise on the development of the Register of Studies within theCampbell Collaboration. The ‘Searching for Studies’ chapter oftheCochraneHandbook for SystematicReviewsof Interventionswasused extensively as the basis for the revised version of the Camp-bell Information Retrieval Policy Brief. Members are involved in awide variety of projects including work in the following areas:

Quality of review methodology: The Methodological Expecta-tions of Cochrane Intervention Reviews (MECIR) is a body whichis working towards establishing essential minimum standards forall Cochrane intervention reviews. Working Group 2, Searchingfor Studies Working Group (MECIR-SfS), aims to specify method-ological expectations with respect to searching for studies forCochrane protocols, full reviews and updates on the effects ofinterventions and to ensure these methodological expectationsare implemented across the Collaboration. It will cover all aspectsof identification of studies for consideration for interventionreviews.

The membership of the MECIR-SfS Working Group draws on theexpertise and includes representatives of all the major relevantstakeholders including the Cochrane Editorial Unit, Co-ordinatingEditors, information retrievalmethodologists, review authors and


editors and Trials Search Co-ordinators. The group comprisesmen and women from several countries (Australia, Canada,Ireland, the Netherlands, South Africa, Switzerland and the UK).Themembersof thegroup include:methodsco-leadsCarol Lefeb-vre and Julie Glanville (Information Retrieval Methods Group); in-formationretrievalmethodologists JessieMcGowanandMargaretSampson; Trials Search Co-ordinators Edith Leclercq (ChildhoodCancer Group), Ruth Mitchell (Renal Group), Anna Noel-Storr (De-mentia and Cognitive Improvement Group) and Doug Salzwedel(Hypertension Group).In 2006, a survey to address the issue of the quality of searchstrategies in Cochrane reviews was carried out.1 The results werefed back to The Cochrane Collaboration through the Trials SearchCo-ordinators. This survey will be updated in 2010/11. Work con-tinues on the Peer Review of Electronic Search Strategies (PRESS)project.Search filter design: Projects to design search filters to identifyreporting of adverse effects have resulted in new publications2–3

and there is also an ongoing case study systematic review to as-sess different information sources. The development of a searchfilter to identify diagnostic test accuracy studies and to explorethe value of handsearching in this area is underway. Work onexpanding and updating the web resource of search filters com-piled by the InterTASC Information Specialists’ Sub-Group (ISSG)continues.4 There are also two proposed projects in this area: todevelop search filters to identify economic evaluations in MED-LINE and EMBASE; and to elaborate and test a search filter toidentify studies relevant to low- and middle-income countries inMEDLINE.Training and support: The Group held two workshops at theCochrane Colloquium in Keystone in 2010. The first was aimed atnew authors of reviews and was entitled ‘Searching for Studies’and the second was entitled ‘Introductory/intermediate work-shop for Cochrane and Campbell Trials Search Co-ordinators’. Weare hoping that in addition to these, a workshop on the subjectof searching for studies in the area of diagnostic test accuracy willbe held under the IRMG banner in the Colloquium in Madrid inOctober 2011.The Collaboration decided some time ago to develop a set ofstandard training materials as a resource for all Cochrane trainers

to use when training review authors. This project is being takenforward by the Training Working Group. As part of this project, aset of training materials on searching for studies has been devel-oped by Miranda Cumpston (Australasian Cochrane Centre) andCarol Lefebvre (UK Cochrane Centre), based on the ‘Searchingfor studies’ chapter of the Cochrane Handbook. The materials aredesigned to be used as part of a two-day, face-to-face introduc-tory workshop for new Cochrane authors at the title registrationstage, with optional hands-on exercises and handouts.A 2013 Cochrane anniversary taskforce: It has been agreedthat in 2013 we will plan a year full of activities to celebrateCochrane’s 20th Anniversary. The Collaboration has had consid-erable influence in the sphere of information science, especiallysearch methodology, so it is fitting that we celebrate thoseachievements. Information Retrieval Methods Group membersare invited to contribute in the following ways:

• publications: to write 20th anniversary special articles oreditorials in key information science journals;

• involvement in key meetings and conferences: attendandpresent the anniversarymessages at key conferencesaroundtheworld, doingaspecial sessionaboutCochrane;

• contribute to the creation of Cochrane archives by con-tributing Cochrane paraphernalia and historical docu-ments.

Visit us at http://irmg.cochrane.org/

References1. SampsonM, McGowan J. Errors in search strategies were iden-tifiedby type and frequency. JournalofClinicalEpidemiology2006;59(10): 1057–63.2. Golder S, Loke YK, BlandM.Unpublisheddata canbeof value insystematic reviews of adverse effects: methodological overview.Journal of Clinical Epidemiology 2010; 63(10): 1071–81.3. Golder S, Loke YK. Sources of information on adverse effects: asystematic review. Health Information and Libraries Journal 2010;27(3): 176–90.4. www.york.ac.uk/inst/crd/intertasc/ [accessed 15th April 2011].

Cochrane Patient Reported OutcomesMethods Group

Donald Patrick, Gordon Guyatt and Iliana Petkova

The Cochrane Patient Reported OutcomesMethods Group (PROMG)was launched in 1993. The focus of the PROMethods Groupis on patient-important outcomes self-reported by randomizedtrial participants. These outcomesmay include symptoms, signs,health status, or broader perceptions andquality of life concernsincluded in diaries, questionnaires, or interviews. Investigatorsmay obtain PROs using anymethod of data collection, includingpaper and pencil, telephone interviews, web-based surveys andelectronic data collection.

At the last Cochrane Colloquium in Keystone in 2010 the PROMethods Group presented a workshop entitled ‘‘How to inter-pret patient reportedoutcomes inCochrane reviews’’, facilitatedby Gordon Guyatt and Donald Patrick. At our annual busi-ness meeting we reviewed collaborations with the Cochrane

Musculoskeletal Group in selecting and interpreting outcomesfor trials of arthritis and other musculoskeletal diseases. Ofparticular interest in our work with the Musculoskeletal Groupis how to estimate the effect on patient-important outcomesof surrogate measures and the best way of presenting resultsto enhance their interpretability. A broader area of inquirythat overlaps with our work with the Musculoskeletal Groupis an exploration of methods of aggregating results from dif-ferent PROs measuring similar underlying constructs. We arecurrently preparing an article that will summarize the avail-able approaches. Roy Elbers, who attended from the Nether-lands, reported on his work on Parkinson’s Disease with theMovement Disorders Group and will be our liaison.On our website, Caroline Terwee and colleagues have pro-vided reviews of PROs in the review areas which can be lo-cated at www.cochrane-pro-mg.org/index.html. In the comingyear, we shall begin modifications of the PRO chapter in theCochrane Handbook for Systematic Reviews of Interventions, andcontinue to look for an existing individual patient data set todo analyses of PROs in trials. All ideas are welcome. We


shall convene a conference call to get input from interestedparties on activities for the Cochrane Colloquium in Madrid inOctober, sometime in June. If you wish to join our MethodsGroup, please complete theMembership formon thewebsite at

www.cochrane-pro-mg.org/index.html. We thank the MAPI Re-search Trust for providing funds to support our activities and toset up our website.

Cochrane Prognosis Methods Group

KatrinaWilliams, Jill Hayden, Karel Moon, Henrica De Vet,Doug Altman, Richard Riley, SusanWoolfenden and KarelMoons

TheCochranePrognosisMethodsGroup (PMG) continues toworktowards its goal of establishing appropriate and high qualitymethods for undertaking systematic reviews and meta-analysesof prognosis studies. The PMG currently has 116 membersand receives, on a regular basis, requests to provide adviceto review authors wishing to write prognosis systematic re-views or incorporate prognosis information into their interven-tion or diagnostic reviews. Currently published information onprognosis methodology has been made available on our web-site (www.prognosismethods.cochrane.org/en/index.html). Re-search initiatives are currently underway to develop researchtools for writing prognosis systematic reviews including a frame-work for writing prognosis systematic reviews, data extractiontool, risk of bias tool, and refining of search strategies. Theseinitiatives are being driven by our convenors: Katrina Williams;Jill Hayden; Karel Moons; Henrica De Vet; Doug Altman; RichardRiley and Susan Woolfenden. Professor Karel Moons joined asa convenor in 2010 and is based at the Julius Center for HealthSciences and Primary Care, University Medical Centre, Utrecht,The Netherlands.Those who attended the PMG workshops at the Cochrane Col-loquium in Keystone in 2010: systematic reviews of prognosis:current methods and challenges (Henrica de Vet, Karel Moons,Jill Hayden and Tobias Van den Berg for the Cochrane PrognosisMethods Group); and search strategies for reviews of diagnos-tic and prognostic prediction models and review of prognosticfactors (Karel Moons, Geert-Jan Geersing, Mariska Leeflang, Wal-ter Bouwmeester) will have noted key areas where prognosis

research is being undertaken. In particular the focus is on syn-thesis in prognosis reviews including meta-analysis in predictionstudies, search strategies in prediction studies and quality assess-ment in risk prediction models and prognosis reviews.Projects that are currently being undertaken that fall under theauspices of the PMG include:PICOs (Participants, Interventions, Comparisons, Outcomes) inrespiratory child health (KatrinaWilliams, Terry Klassen and DavidTovey), funded by the Cochrane Collaboration OpportunitiesFund. This project aims to develop and test a series of 25 PICOs inthe area of respiratory child health, using questions contributedby clinicians. The Bronchiolitis PICO / clinical answer format iscurrently being finalized. Methods are also being developed forcreating PICOs about interventions for specific clinical subgroups,for including non-Cochrane evidence in PICOs, and for creatingPICOs on diagnosis and prognosis questions.Approaches to estimate and present baseline risks: recommen-dations for ‘Summary of findings’ tables (Jill Hayden, GeorgeTomlinson, Maurits van Tulder, Doug Altman) funded by theCochrane Collaboration Opportunities Fund. This project aimsto explore the relevant methodological literature, compare andidentify strengths and weaknesses of different approaches toestimate baseline risk, towards the goal of valid, feasible andunderstandable presentation of absolute measures of effect.Writing systematic reviews investigating prognosis of autismspectrum disorders (Susan Woolfenden, Katrina Williams in col-laboration with CPMG and South West Sydney Local HealthNetwork). Published search strategies and quality criteria arebeing adapted for this process. Key areas being reviewed includediagnostic stability, epilepsy and mortality, and communicationand vocation.If you are interested in joining this network or the CochranePrognosis Methods Group, please contact Greta Ridley ([email protected]).

Cochrane Prospective Meta-AnalysisMethods Group

Davina Ghersi, Jesse Berlin and Lisa Askie

Last year saw the first ever publication of a ProspectiveMeta-Analysis (PMA) protocol in The Cochrane Library, enti-tled ‘‘Misoprostol for cervical priming prior to IUD insertion innulliparous women’’1.At the CochraneColloquium in Keystone in 2010, the PMAMeth-ods Group ran a workshop entitled, ‘‘Prospective meta-analysis:a practical guide’’. This workshop looked at both the theoret-ical and practical aspects of conducting a PMA and was wellreceived by those that attended. A joint PMA and IndividualParticipant Data Methods Group Meeting was also held at theColloquium to identify areas of common interest and potentialfor collaboration on research projects. Both groups found thismeeting fruitful and will work on a proposed shared researchagenda and potential sources of funding.

During 2010, the PMA Methods Group Convenors met everythree months by teleconference. The convenors also provideadvice on request to those wanting to prepare a PMA pro-tocol, including the editors of Cochrane Review Groups andthe authors of these reviews. Advice is given by e-mail ortelephone, or through participation in the advisory commit-tees of several prospectivemeta-analyses (for example, ongoingPMAs of neonatal oxygen levels, and interventions for the pre-vention of childhood obesity). For more information aboutthe PMA Methods Group, please visit our website (http://pma.cochrane.org/).

Reference

1. Turok D, Simonsen SE, Schulz KF. Misoprostol for cervicalpriming prior to IUD insertion in nulliparous women (Protocol).Cochrane Database of Systematic Reviews 2010, Issue 1. Art.No.: CD008278.


Cochrane Qualitative ResearchMethods Group

Angela Harden, Karin Hannes, Craig Lockwood, Jane Noyes,Janet Harris and Andrew Booth

Qualitative research can addressmany different questions includ-ing those around the appropriateness and implementation ofinterventions and patient and public perspectives. The integra-tion of qualitative researchwithin Cochrane reviews of the effectsof interventions presents new challenges and opportunities forthe Collaboration and the Cochrane Qualitative Research Meth-ods Group (CQRMG) provides a network to advise on, debate andresearch solutions to these challenges.This past year has seen some significant milestones for theGroup. We have supported a number of review teams in planningCochrane reviews that will include qualitative research rangingfrom weaning protocols for critically ill patients to peer supportfor chronic diseases. Many of these have now been registeredas titles with their Cochrane Review Groups and several are go-ing through the peer review process. The first Cochrane reviewformally to integrate a synthesis of qualitative research was pub-lished by the Cochrane Consumers and Communication ReviewGroup in March 2011.1 The review set out to evaluate the effectsof interventions to notify and support consumers in situationswhere exposure to the risk of Creutzfeldt-Jakob disease (CJD)or variant CJD has occurred as a result of medical treatment.In this review, a lack of relevant trials led the authors to syn-thesize qualitative and other types of relevant research usingthematic synthesis methods2 to address questions around policyimplementation and consumer experiences.This year, we have also continued to develop our extendedguidance on including qualitative research in systematic reviewsto supplement Chapter 20: Qualitative research and Cochranereviews in the Cochrane Handbook for Systematic Reviews of Inter-ventions. We are posting each chapter of the guidance on ourwebsite as soon as it is approved by the steering group. Membersof the Group have also developed books and journal articles thatcontribute to the wider literature on the synthesis of qualitativeresearch. Our lead convenor, Jayne Noyes, published a thoughtprovoking piece on including qualitative research for the Journalof Research in Nursing.3 Andrew Booth led the production of adraft chapter on standards for reporting qualitative research foran edited ‘how to’ collection on research reporting guidelines.

Karin Hannes and Craig Lockwood published their evaluation ofonline critical appraisal tools for qualitative research4 as well ascompleting an edited Wiley-Blackwell textbook Qualitative evi-dence synthesis: choosing the right approach to be launched at theCochrane Colloquium in Madrid.Finally, we have run several training events around the world.We ran a fully-booked workshop on preparing a protocol for areview at the Colloquium in Keystone in October 2010. Membersof theGroup also delivered other qualitative synthesis workshopssuch as the Evidence Synthesis of Qualitative Research in Eu-rope workshop at Sheffield University in September 2010 andplanned for September 2011 and the Catholic University of Leu-ven’s three-day systematic review course in June 2010 and June2011. At this year’s Colloquium inMadridwe are pleased to offer anew exploratory workshop focused on how reviews that includequalitative research can best be accommodated within RevMan.Currently, reviews that include qualitative research have to housethis part of the review within appendices.Workshops at the Cochrane Colloquium in Madrid in 2011:

• how to complete a Cochrane protocol for qualitativeevidence synthesis;

• including data from protocols on qualitative researchfindings in RevMan: current version and a wish list;

• searching, sampling and selecting for qualitative evi-dence synthesis.

References

1. Ryan R, Hill S, Lowe D, Allen K, Taylor M, Mead C. Notificationand support for people exposed to the risk of Creutzfeldt-Jakobdisease (CJD) (or other prion diseases) throughmedical treatment(iatrogenically). Cochrane Database of Systematic Reviews 2011,Issue 3. Art. No.: CD007578.2. Thomas J, Harden A. Methods for the thematic synthesis ofqualitative research in systematic reviews. BMCMedical ResearchMethodology 2008; 8: 45.3. Noyes J. Never mind the qualitative feel the depth! The evolv-ing role of qualitative research in Cochrane intervention reviews.Journal of Research in Nursing 2010; 15(6): 525–34.4. Hannes K, Lockwood C, Pearson A. A comparative analysis ofthree online appraisal instruments’ ability to assess validity inqualitative research. Qualitative Health Research 2010; 20(12):1736–43.

Cochrane Screening and DiagnosticTests Methods Group

Constantine Gatsonis, Mariska Leeflang and PetraMacaskill

Over the past year, our Methods Group members have againprovided methodological leadership and support for Cochranediagnostic test accuracy reviews. The level of activity continuesto increase as the number of protocols and completed reviewsincreases. The strong links between our Methods Group andthe Support Centres (the UK Support Unit based in Birmingham,and the Continental Europe Support Unit based in Amsterdam)continue to facilitate the progress that is being made.

A notable achievement over the past year is the publication ofthe statistics chapter of the Cochrane Handbook for Diagnostic

Test Accuracy Reviews (http://srdta.cochrane.org/handbook-dta-reviews). The chapter provides background on: key conceptsunderlying the investigationofdiagnostic test accuracy; descrip-tive analyses; and statistical modelling. It provides a frameworkfor the meta-analysis of diagnostic accuracy studies that re-port results in terms that test sensitivity and specificity at achosen test threshold. Guidance is given as to whether thefocus should be on estimating a summary ROC curve, or asummary operating point (i.e. summary estimates for sensi-tivity and specificity). Guidance is also provided on the useof statistical models for the investigation of heterogeneity andtest comparisons. Worked examples are used to illustratethe modelling approaches and their interpretation. Data andprograms for these examples are provided to assist reviewauthors and their statistical collaborators. Because methodol-ogy in this area continues to evolve, it is envisaged that the


statistics chapter will be expanded over time to cover newdevelopments.The increase in the number of diagnostic protocols and reviewshas led to a sharp increase in the need for peer reviewers andeditors. Members of the Group contribute their expertise byacting as peer reviewers, and several have also joined the di-agnostic test accuracy editorial team. This involvement willhelp tohighlightmethodological topics that require clarificationor further development. We also continue to provide trainingworkshops. Seven workshops were presented on behalf of our

Group at the Cochrane Colloquium in Keystone in 2010. Thesecovered a wide range of topics including: literature searching;assessing study quality; basic methods; advanced statisticalmethods; and interpretation of results.We should especially like to thank Roger Harbord for his sub-stantial contributions over the past years. Unfortunately, heneeded to resign fromhis role as a Co-convenor in 2010 becauseof other commitments. We are very grateful for his major contri-butions to statistical methodology and the development of thediagnostic software that has been incorporated into RevMan.

Cochrane Statistical Methods Group

Doug Altman, Joseph Beyene, Steff Lewis, JoanneMcKenzieand Georgia Salanti

Statistical issues are a core aspect of much of the work of TheCochraneCollaboration. The StatisticalMethodsGroup (SMG) hasa broad scope which covers issues relating to statistical methods,training, software, and research, together with attempting toensure that adequate statistical and technical support is availableto review groups. The SMG is a forum where all statistical issuesrelated to the work of The Cochrane Collaboration are discussed.The SMG has 214 members including methodologists, authors,statisticians, editorsandotherswidelyspreadbothgeographicallyandacross theCollaboration. Toapplyto jointheSMG,pleasesendan e-mail to Georgia Salanti ([email protected]),including a short paragraph summarizing your interests in meta-analysis, andanyconnectionsyoualreadyhavewithTheCochraneCollaboration (50 words or so is plenty).

A recent highlight was a two-day course aimed at statisti-cians who provide support to Cochrane Review Groups orCentres, held from 4 to 5 March 2010 in Cardiff, UK. Thirty-four participants attended the course. A wide range of ad-vanced issues in meta-analysis and assessment of bias wereaddressed. The presentations from the course are now avail-able as pdf files of slides and audio presentations on the SMGwebsite: (http://smg.cochrane.org/teachingtraining-workshops/smg training course 2010). The SMG website will be furtherdeveloped over time to meet the needs of the SMG better. Com-ments on how it can be improved or suggestions for additionalcontent are welcome ([email protected]).

Many SMG members facilitated workshops on a range of top-ics at the Cochrane Colloquium in Keystone in 2010 and atother regional meetings and symposia. We thank all thosewho have contributed to SMG activities over the last yearand look forward to seeing many of you at the Colloquium inMadrid.

Possible Methods GroupCochrane Agenda Setting andPrioritisationMethods Group

MonaNasser, Sally Crowe, VivianWelch and Erin Ueffing

Prioritization seems to be a hot topic in The Cochrane Collabora-tion and elsewhere in health and social care research. Identifyingand prioritizing reviews that are useful and relevant to end usersis an activity worthy of investment. However, there is little con-sensus about how to approach this task, with whom to consult,reasonable expectations of a prioritization process, and how toimplement, monitor and re-evaluate the process. There havebeen some important developments within The Cochrane Col-laboration. In a survey in 2008, 29 of the 66 Cochrane entitiesreported having processes to inform the selection or prioritiza-tion of topics. This survey will be published in a series of articleson priority setting in the Collaboration in the Journal of ClinicalEpidemiology.

It seems that the logical next step is to convene a MethodsGroup that can help pool the experiences so far and appraise theevidence on different priority setting methods. To identify chal-lenges faced by The Cochrane Collaboration and issues that willbe addressed by the future Methods Group, we have organized

workshops and events over the past few years to provide theopportunity for Cochrane contributors to share their views. Theseevents were developed and organized in collaboration with theCampbell and Cochrane Equity Methods Group, the James LindAlliance, the Cochrane Public Health Group and the CochraneEditorial Unit:

• a workshop at the Cochrane Colloquium in Freiburg, Ger-many in October 2008 entitled ‘‘Ensuring relevance andbuilding enthusiasm for Cochrane reviews: determiningappropriate methods for identifying priority topics forfuture Cochrane reviews’’;

• a workshop at the Cochrane Colloquium in Singapore inOctober 2009 entitled ‘‘Priority-setting for the CochraneCollaboration: methods, challenges, and opportunities?’’;

• an exploratory meeting in Singapore in October 2009;• a specific plenary session on priority setting in Singapore

in October 2009;• a workshop at the Cochrane Colloquium in Keystone in

October 2010 entitled ‘‘Agenda setting: evidence andequity’’;

• a workshop at the UK- and Ireland-based Contributors toThe Cochrane Collaboration Meeting in Belfast in March2011.


These events highlighted several methodological challenges, in-cluding uncertainties on how to develop an equity-orientedapproach to priority setting or ways to evaluate, implement ordisseminate a prioritization process (for example, how to findauthor groups and funding for priority topics and whom to con-sult in different steps of priority-setting approaches (patients,clinicians, policy makers)). The proposed Agenda setting andPrioritisation Methods Group aims to address these challenges.It will be convened by five people: Mona Nasser (UK), SallyCrowe (UK), Vivian Welch (Canada), Alessandro Liberati (Italy),and Prathap Tharyan (India) and aims to support The CochraneCollaboration in keeping up to date on the methods of researchon agenda setting and prioritization. We define methods of pri-ority setting as methods used to set a research agenda, prioritizeand map healthcare research, healthcare interventions and out-comes of healthcare interventions. The proposedMethods Group

will attempt to provide links between Cochrane methodologistsand methodologists working in the field of priority setting. TheMethods Group has already begun specific methods initiativesincluding: developing an equity lens for agenda setting and pri-ority setting projects; starting a Cochrane methodology reviewon priority setting for comparative effectiveness research thatwill identify, describe, and evaluate different approaches to pri-oritize research or set a research agenda for conducting futurecomparative effectiveness research studies; and working withthe Educational Resources Committee on considering prioritysetting in the internal checklists for Cochrane Review Groups.

At this stage, we are still in the process of registering the newMethods Group and recruiting interested partners. If this sort ofwork appeals to you, please get in touch with us ([email protected]).


Campbell CollaborationMethods Groups (C2)

Terri Pigott

The Campbell CollaborationMethods Groupmet during the JointCochrane and Campbell Collaboration meetings in Keystone inOctober 2010. The major activity for the Campbell MethodsGroup has been a re-organization of the editorial functions. Weare pleased to announce the appointment of Emily Tanner-Smith,Research Associate at the Peabody Research Institute, VanderbiltUniversity, Nashville, Tennessee as our new Associate MethodsEditor. Wehave also formedagraduate student reviewboard thatwill be responsible for the first reviewof protocols and completedreviews that are sent through the C2 Methods Group. The gradu-ate students are studying statistics and research methodology atvarious universities including Loyola University Chicago, North-western University, and Vanderbilt University. Any interestedgraduate student should contact Terri Pigott at [email protected] are the reports from each of the subgroups of theC2 Methods Group.Economics Group: During the past year, the Campbell andCochrane EconomicsMethods Group has produced a range of ac-tivities and outputs relevant to economicsmethods for use in thepreparation and maintenance of Campbell reviews, including:

• an edited volume Evidence-Based Decisions andEconomics, published by Wiley-Blackwell/BMJ Books(http://eu.wiley.com/WileyCDA/WileyTitle/productCd-1405191538.html);

• a web-based tool and supplementary guidance for use insystematic reviews to adjust estimates of costs collectedfrom included studies to a common target currency andprice year (http://eppi.ioe.ac.uk/costconversion/default.aspx and http://ingentaconnect.com/content/tpp/ep);

• a methods training workshop on ‘Introduction to searchmethods for economics studies’ at the Colloquium inKeystone in October 2010.

Equity Group: In September 2010, the Equity Group publishedguidance on equity and systematic reviews in the BMJ1; this pa-per represents four years of workshop discussions, meetings, and

consultations with international researchers, policy makers, andpractitioners. The Equity Group is also collaborating with LaurieAnderson of the Cochrane Public Health Group around the use oflogicmodels to inform systematic reviews on equity and complexinterventions.Information Retrieval: The Information Retrieval Methods Grouphas published their new guide to information retrieval for sys-tematic reviews: Searching for studies: a guide to informationretrieval for Campbell systematic reviews. This guide can befound on the Campbell website at www.campbellcollaboration.org/news /new information retrieval guide.phpProcess and Implementation Methods: The team has focusedits effort over the last year on conducting an empirical study ofcompleted child and youth C2 reviews to inform the develop-ment of methodological guidance for process, implementationand intervention theory. This work has led to the developmentof a conceptual framework to guide the review, a comprehen-sive data extraction tool and instruction guide. The tool willbe adapted into a review author checklist and critical appraisaltools for primary studies and systematic reviews. In addition,five critical appraisal tools addressing process, implementationand intervention theory were reviewed across a core set ofdimensions.The Group is collaborating with the C2 Information RetrievalGroup (Anne Wade and David Pickup) on information retrievalissues related to process and implementation. The Group hasdeveloped a working document identifying search terms anddefinitions related to process and implementation in the majordatabases utilized by C2 reviewers.

Reference1. Tugwell P, Petticrew M, Kristjansson E, Welch V, Ueffing E,Waters E, Bonnefoy J, Morgan A, Doohan E, Kelly MP. Assessingequity in systematic reviews: realising the recommendations ofthe Commission on Social Determinants of Health. BMJ 2010; 341:c4739.


Future Meetings

19th Colloquium of The CochraneCollaboration

Madrid, Spain19 to 22 October 2011

The theme of the 19th Cochrane Colloquium is scientific evidencefor healthcare quality and patient safety. This provides a goodopportunity to celebrate the Colloquium in conjunction with theVI International Conference on Patient Safety, organized by theNational Agency for Health Care Quality at the Spanish Ministryof Health. In addition to the issues more directly related to healthquality and patient safety, there will be many opportunities tolearn together and to discuss a large variety ofmethodological is-sues and topics of great interest, directed to theusual participantsas well as to newcomers.Finally, the city itself! Madrid offers many options for enjoying itsculture, music and food – life in all of its expressions! Don’t missthis opportunity and you will be happy with your decision. Moreinformation is available at www.colloquium.info.

Clinical Trials MethodologyConference 2011

Bristol, UK4 to 5 October 2011

This interactivemeetingwill provideopportunities todiscussissues and innovation in trials methodology research andhas been sponsored by the MRC Network of Hubs for TrialsMethodology Research in the UK. The Scientific Committeehas developed an exciting two-day programme that will beof interest to statisticians, trial managers, clinicians, inves-tigators, and health economists. More information is avail-able at www.methodologyhubs.mrc.ac.uk/methodologyconference 2011.aspx.


AcknowledgementsThe editors of Cochrane Methods would like to thank all the con-tributors for making this issue possible. We should also like tothank Sarah Chapman and Jackie Chandler for their assistance inco-ordinating content production andwithwriting structured ab-stracts; also Anne Eisinga for her careful proof reading. Thanks arealso due to the Chief Scientist Office in Scotland, Health and SocialCare Public Health Agency in Northern Ireland, Health ResearchBoard in Ireland, National Institute for Health Research in Eng-land and the Welsh Assembly Government, which provide corefunding to the UK Cochrane Centre to produce this newsletterand to TheCochraneCollaboration for providing funding towardsprinting costs.

Availability of CochraneMethodsAdditional copies of Cochrane Methods may be obtained free ofcharge from the UK Cochrane Centre, which is based at:

UK Cochrane CentreNational Institute for Health ResearchSummertown PavilionMiddle WayOxford OX2 7LGUK

CochraneMethods is also available electronically via TheCochraneCollaborationwebsite atwww.cochrane.org/newslett/index.htm.

Comments and FeedbackIf you want to make sure that you receive the next issue ofCochrane Methods please contact the address below. Com-ments are welcome. Let us know what you liked and what youdid not like and any suggestions you have for the next issue.Thank you!

Maria BurgessUK Cochrane CentreNational Institute for Health ResearchSummertown PavilionMiddle WayOxford, OX2 7LGUKTel: +44 1865 [email protected]

The Cochrane LibraryThe Cochrane Library is available at www.cochranelibrary.com.It contains six databases: the Cochrane Database of SystematicReviews (CDSR), the Database of Abstracts of Reviews of Effects(DARE), the Cochrane Central Register of Controlled Trials (CEN-TRAL), and the Cochrane Methodology Register (CMR) as well astheHealthTechnologyAssessmentDatabase and theNHSEconomicEvaluation Database. In addition, The Cochrane Library containsinformation about the Collaboration, complete contact detailsfor all Cochrane entities, and links to the Cochrane Handbook forSystematic Reviewsof Interventions (formerly the CochraneReview-ers’ Handbook) and a glossary of Cochrane and methodologicalterminology. Information about how to subscribe is availablefrom:

Jennifer CoatesCochrane Library Customer Services Advisor

John Wiley & Sons Ltd1 Oldlands WayBognor RegisWest Sussex, PO22 9SAUKTel: +44 1243 [email protected]/view/0/HowtoOrder.html

Cochrane Collaboration Internet SiteA wide range of Cochrane Collaboration information is avail-able from www.cochrane.org including the abstracts from all theCochrane reviews in the current issue of The Cochrane Library, de-tails of Cochranee-mail lists, opportunities todownloadCochranesoftware (including RevMan), contact details for all Cochrane en-tities, copies of the Cochrane Methods Groups Newsletters andmuch more.

International Cochrane e-mail list: CCINFOThismoderated list offers anexcellentmeansof keeping informedabout the activities and policies of The Cochrane Collaboration.The list is used for announcements and discussion ofmatters rele-vant to theCollaborationasawhole. Tosubscribegoto the follow-ing webpage: http://lists.cochrane.org/mailman/listinfo/ccinfo.

Cochrane Centre Internet SitesThere are 14 Cochrane Centres around the world; to speak tosomeone about The Cochrane Collaboration, please contact yourlocal Centre.Australasian Cochrane Centrewww.aac.cochrane.org

Brazilian Cochrane Centrewww.centrocochranedobrasil.org

Canadian Cochrane Centrewww.ccnc.cochrane.org

Chinese Cochrane Centerwww.ebm.org.cn

Dutch Cochrane Centrewww.cochrane.nl

French Cochrane Centree-mail: [email protected]

German Cochrane Centrewww.cochrane.de

Iberoamerican Cochrane Centrewww.cochrane.es

Italian Cochrane Centrewww.cochrane.it

Nordic Cochrane Centrewww.cochrane.dk

South African Cochrane Centrewww.mrc.ac.za/cochrane

South Asian Cochrane Centrewww.cochrane-sacn.org

UK Cochrane Centrewww.cochrane.ac.uk

United States Cochrane Centerwww.cochrane.us

COCHRANE METHODS · Cochrane Methods Sally Hopewell, Co-Scientifi c Editor and Technical Editor UK...

Documents

Transcript of COCHRANE METHODS · Cochrane Methods Sally Hopewell, Co-Scientifi c Editor and Technical Editor UK...