Quality of news media reports about the effects and costs ... · cepts. In Norway, where I am...
Transcript of Quality of news media reports about the effects and costs ... · cepts. In Norway, where I am...
Quality of news media reports about the effects and costs of health interventions: Systematic review protocol 1
Qualityofnewsmediareportsabouttheeffectsandcostsofhealthinterventions:SystematicreviewprotocolOxmanMWorkingpaper,April26,2018www.informedhealthchoices.org
Quality of news media reports about the effects and costs of health interventions: Systematic review protocol 2
Colophon
Title Qualityofnewsmediareportsabouttheeffectsandcostsofhealthinterventions:Systematicreviewprotocol
Authors Oxman, Matt 1,2,3
1. University of Oxford, United Kingdom 2. Faculty of Health Sciences,Oslo Metropolitan University, Norway 3. Centre for Informed Health Choices,Norwegian Institute of Public Health, Norway
Correspondingauthor
Matt [email protected] Telephone: +47 98636788 Oslo Metropolitan University, Faculty of Health Sciences Section for Research & Development Pilestredet Campus P.O. Box 4 St. Olavs plass N-0130 Oslo, Norway
Keywords Newsmedia,Systematicreview,Unreliableinformation,KeyConcepts
Citation Oxman,M.,Qualityofnewsmediareportsabouttheeffectsandcostsofhealthinterventions:Systematicreviewprotocol.IHCWorkingPaper.Oslo:InformedHealthChoices.ISBN: 978-82-8082-971-9
Articlecategory �AboutInformedHealthChoices�Keyconceptsandglossary�Learningresources�Systematicreviews�Developmentandevaluationoflearningresources�Claimevaluationtools�Editorialsandcommentaries�Grantapplicationsþ Other
Date April26,2018
Quality of news media reports about the effects and costs of health interventions: Systematic review protocol 3
Abstract
Background:Unreliableinformationabouthealthcare,combinedwiththeina-bilitytoassessthereliabilityofsuchinformation,canleadtouninformeddeci-sionsand,ultimately,wasteandunnecessarysuffering.Journalismisaparticu-larlyimportantsourceofhealthinformation.Therehavebeenseveralscientificstudiesofthequality(reliability)ofnewsmediareportsabouttheeffectsofhealthinterventions.Theredoesnotappeartobeasystematicreviewofthesestudies.Theplannedreviewisintendedtoinform:theproductionandcon-sumptionofhealthnews;aswellasfurtherresearch,includingthedevelopmentofinterventionstohelppeopleassesshealthcareinformationinthenews,sotheycanmakewell-informeddecisions.Objectives:1)Toassesscriteriausedtomeasurethequalityofnewsmediare-portsabouttheeffectsandcostsofhealthinterventions;2)toassessthequalityofsuchreports;and3)toexplorefactorsthatmightexplainvariationinquality.Eligibility:Wewillincludescientificstudiesofprint,broadcastandonlinenewsreports(thepopulation).Atleastoneexplicitcriterionmusthavebeenusedbytheresearcherstomeasurequality(thecondition).Itmustbepossibleandsen-sibletoframetheresultsforatleastonecriterionastheproportionofreportsthatsatisfiedit(e.g.theproportionofreportsinwhicheffectestimatesareab-solute,notonlyrelative).Therewillbenolimitsonlanguage,geographyortimeperiodofthereports(thecontext).Tobeincluded,studiesmustspecify:thesamplingframe;theselectioncriteria;andtheselectiontechnique.Methods:Tworeviewerswillextractdata,assessriskofbias,andmakejudge-mentsaboutInformedHealthChoices(IHC)KeyConceptscapturedbycriteriausedinscientificstudies.Ifnecessary,athirdreviewerwillhelpreachconsen-susonanyjudgement.Meta-analyseswillbeconductedtoestimatethepreva-lenceofnewsreportsthatsatisfyrespectivecriteria.Toexplorevariation,sub-groupanalyseswillbeconductedfor:medium(broadcastvs.other);timeperi-od(decade);financialmodel(commercialvs.non-commercial);journalistspe-cialisation(healthorsciencevs.other);andcountryincomelevel(low-incomevs.other).
Quality of news media reports about the effects and costs of health interventions: Systematic review protocol 4
Background
TheproblemUnreliablehealthinformationThisprotocolisforasystematicreviewthatisultimatelyintendedtohelpad-dressamajor,globalproblem.Theproblemhasthreeparts.Thefirstisunrelia-blehealthinformation.Ineffect,thereisendlessinformationabouttheeffectsofhealthinterventions,ahealthinterventionbeinganyactionintendedtoimproveormaintainthehealthofindividualsorgroups.Thisincludesinformationabout:“modern”,“academ-ic”,“conventional”or“western”medicine;“complementary”,“alternative”,“tra-ditional”or“natural”medicine;screening;surgeryanddevices;diet,exerciseandlifestyle;andsystemsandpolicies.Someofthisinformation,ifnotmostofit,isunreliableinonewayoranother.Theinformationcanbedirectlymislead-ing,suchastheexplicitclaimthataninterventioncausesanoutcomewhenitisonlyassociatedwiththeoutcome.Oritcanbemisleadingbyomission,suchasrelativeeffectestimateswithoutabsoluteestimates,particularlywhenthebase-lineriskissmall(Woloshin,etal.,2008).Researchershavefoundunreliablelayinformationabouttheeffectsofhealthinterventionsin:
• Patientmaterials(Coulter,etal.,1999);productlabels(UnitedStatesGovernmentAccountabilityOffice,2010),
• Varioustypesofwebsites(Culver,etal.,1997;Wolfe,etal.,2002;Glenton,etal.,2005;Spencer,etal.,2016)
• Advertisements(Sansgiry,etal.,1999;Frosch,etal.,2007;Frosch,etal.,2011;Faerber&Kreling,2013;Groven&Braitwaite,2016)
• Thenews(Appendix1)Informationinthenewsmediaisofparticularinterest,forreasonsdescribedlaterinthebackgroundsection.
Quality of news media reports about the effects and costs of health interventions: Systematic review protocol 5
Finally,researchreferencedinthefollowingsubsectionshowspeopleareuna-bletoassessinformationabouttheeffectsofhealthinterventions.Logically,ifpeopleareunabletoassesssuchinformation,theyaremorelikelytospreadthatwhichisunreliable.Conversely,peoplespreadingunreliableinformation,asdocumentedinthestudiesreferencedinthissubsection,suggeststheyareunabletoassesssuchinformation.InabilitytoassesshealthinformationThesecondpartoftheproblemisthatpeopleareunabletoassessthereliabilityofinformationabouttheeffectsofhealthinterventions.AspartoftheInformedHealthChoices(IHC)project(www.informedhealthchoices.org),theabilitytoassesssuchinformationandmakeinformeddecisionsabouthealthinterven-tionshasbeenbrokendownintotheabilitytoapplyspecificconceptscalledtheIHCKeyConcepts.Forexample,peopleneedtobeabletoapplytheconceptthatassociationisnotthesameascausation.Theoriginallistincluded32concepts(Austvoll-Dahlgren,etal.,2015).Thenewestiterationincludes36(Box1)(Chalmers,etal.,2018).TheabilitytoapplyIHCKeyConceptsispartofhealthliteracyandhasbeenmeasuredbothdirectlyandindirectly.Therehavebeenlargesurveysofhealthliteracy,inEuropebytheEuropeanHealthLiteracyProjectConsortium(HLS-EUConsortium,2013)andintheUnit-edStatesbytheNationalCenterforEducationStatistics(Kutner,etal.,2006).AccordingtotheEuropeansurvey,“Healthliteracyislinkedtoliteracyanden-tailspeople’sknowledge,motivationandcompetencestoaccess,understand,appraise,andapplyhealthinformationinordertomakejudgmentsandtakede-cisionsineverydaylifeconcerninghealthcare,diseasepreventionandhealthpromotiontomaintainorimprovequalityoflifeduringthelifecourse”(HLS-EUConsortium,2013,p.7).FortheAmericansurvey,thetermwasdefinedas:“Thedegreetowhichindividualshavethecapacitytoobtain,process,andunder-standbasichealthinformationandservicesneededtomakeappropriatehealthdecisions”(Kutner,etal.,2006,p.iii).TheabilitytoapplyIHCKeyConcepts(i.e.assessthereliabilityofinformationabouttheeffectsofhealthinterventionsandmakewell-informeddecisions)islogicallyincludedinbothdefinitions.However,neithersurveyappearstohaveobjectivelymeasuredthis.TheEuropeansurveymeasuredthesubjectiveeaseof47tasks.TasksrelatedtoapplyingIHCKeyConcepts,including“judgeiftheinformationaboutillnessinthemediaisreliable”,wererankedamongstthemostdifficult(HLS-EUConsortium,2013).
Quality of news media reports about the effects and costs of health interventions: Systematic review protocol 6
Box1:ShorttitlesfortheInformedHealthChoices(IHC)KeyConceptsRecognisinganunreliablebasisforaclaim
1.1. Treatmentscanharm.1.2. Anecdotesareunreliableevidence.1.3. Associationisnotthesameascausation.1.4. Commonpracticeisnotalwaysevidence-based.1.5. Newerisnotnecessarilybetter.1.6. Expertopinionisnotalwaysright.1.7. Bewareofconflictinginterests.1.8. Moreisnotnecessarilybetter.1.9. Earlierisnotnecessarilybetter.1.10. Hopemayleadtounrealisticexpectations.1.11. Explanationsabouthowtreatmentsworkcanbewrong.1.12. Dramatictreatmenteffectsarerare.
Understandingwhethercomparisonsarefairandreliable2.1. Comparisonsareneededtoidentifytreatmenteffects.2.2. Comparisongroupsshouldbesimilar.2.3. Peoples’outcomesshouldbeanalysedintheiroriginalgroups.2.4. Comparisongroupsshouldbetreatedequally.2.5. Peopleshouldnotknowwhichtreatmenttheyget.2.6. Peoples’outcomesshouldbeassessedsimilarly.2.7. Allshouldbefollowedup.2.8. Consideralltherelevantfaircomparisons.2.9. Reviewsoffaircomparisonsshouldbesystematic.2.10. Peerreviewandpublicationdoesnotguaranteereliableinformation.2.11. Allfaircomparisonsandoutcomesshouldbereported.2.12. Subgroupanalysesmaybemisleading.2.13. Relativemeasuresofeffectscanbemisleading.2.14. Averagemeasuresofeffectscanbemisleading.2.15. Faircomparisonswithfewpeopleoroutcomeeventscanbemisleading.2.16. Confidenceintervalsshouldbereported.2.17. Donotconfuse‘statisticalsignificance’with‘importance’.2.18. Donotconfuse‘noevidenceofadifference’with‘evidenceofnodifference’.
Makinginformedchoices3.1. Dotheoutcomesmeasuredmattertoyou?3.2. Areyouverydifferentfromthepeoplestudied?3.3. Arethetreatmentspracticalinyoursetting?3.4. Dotreatmentcomparisonsreflectyourcircumstances?3.5. Howcertainistheevidence?3.6. Dotheadvantagesoutweighthedisadvantages?
Quality of news media reports about the effects and costs of health interventions: Systematic review protocol 7
TheAmericansurveysuggestedaminorityofAmericanadults(12%)had“pro-ficient”healthliteracy(Kutner,etal.,2006,p.v).However,onlyasampleofthehealthliteracyquestionsusedinthatsurveyarepubliclyavailableandnoneoftheavailablequestionsmeasuretheabilitytoassessinformationaboutcausali-ty(www.nces.ed.gov/NAAL/sample.asp).Anothersurrogatemeasureofpeople’sabilitytoapplyIHCKeyConceptsistheirexpectationsabouteffects.Inasystematicreviewofpatients’expectations,themajorityofparticipantsoverestimatedbenefitsandunderestimatedharms(Hoffmann&DelMar,2015).Asystematicreviewofclinicians’expectationsshowedcliniciansdothesame(Hoffmann&DelMar,2017).Othersubstitutesforabilityareattitudesandbeliefs.AccordingtoastudyintheUnitedKingdom,patientsandgeneralpractitionersappeartoplaceadispropor-tionateamountoftrustinthereputationoftheorganisationconductingaclini-caltrial,thequalificationsoftheresearchers,andpeer-review,versusthemeth-ods,whicharewhatinfactdeterminethereliabilityoftheresults(TheAcademyofMedicalSciences,2016).Moreover,aboutathirdoftherespondents(37%)placedahighleveloftrustindatafrommedicaltrials,whileabouttwothirds(65%)placedahighleveloftrustintheexperiencesoffriendsandfamily,i.e.anecdotalevidence.TherehaveindeedbeenstudiesdirectlymeasuringtheabilitytoapplyKeyCon-cepts.InNorway,whereIambased,Oxmanetal.testedarandomsampleof626Norwegianadults(Oxman,etal.,2017).Aboutoneinfive(19%)showedtheywereabletodistinguishbetweenanassociationandcausation.OthersurveyssuggestNorwegianstudentsinpost-secondaryschool(Pettersen,2007)andsecondaryschool(Pettersen,2005)alsostrugglewithassessingclaimsabouttheeffectsoftreatments.AnotherproductoftheInformedHealthChoicesprojectistheCLAIMEvaluationToolsdatabase(www.informedhealthchoices.org/claim-evaluation-tools)(Chalmers,etal.,2018).TheCLAIMEvaluationToolsaremultiple-choiceques-tionsdevelopedspecificallytomeasurepeople’sabilitytoapplytheIHCKeyConcepts(Austvoll-Dahlgren,etal.,2016;Austvoll-Dahlgren,etal.,2017).Be-forethedevelopmentofthedatabase,inasystematicreviewoftoolsformeas-uringsaidability,Austvoll-Dahlgrenetal.identified215discreteinstrumentsorprocedures,capturingupto15oftheconcepts,whereoffourcaptured10ormore(Austvoll-Dahlgren,etal.,2016).Avalidatedsetof26questionsmeasuringtheabilitytoapply13oftheconcepts(i.e.twoquestionsperconcept)wereusedtomeasuretheeffectsofIHClearn-ingresources,intworandomisedtrials(Nsangi,etal.,2017;Semakula,etal.,
Quality of news media reports about the effects and costs of health interventions: Systematic review protocol 8
2017).Inthefirsttrial,thecontrolgroupwas4430Ugandanprimaryschoolchildren,afterlosstofollow-up(Nsangi,etal.,2017).Inthesecondtrial,afterlosstofollow-up,thecontrolgroupwas273parentsandguardiansofchildrenparticipatingintheinterventionorcontrolgroupofthefirsttrial(Semakula,etal.,2017).Lessthanhalfoftheparticipantsinthecontrolgroupforeithertrialansweredbothquestionscorrectlyforanyofthe13includedconcepts.UninformedhealthdecisionsThecombinationofunreliableinformationabouttheeffectsofhealthinterven-tionsandpeople’sinabilitytoassesssuchinformationcanresultinuninformeddecisions.Thisisthethirdpartoftheproblem.Peopleactingonunreliablein-formation,orfailingtoactonreliableadvice,caninturnleadtowasteandun-necessarysuffering.Thisiswhytheproblemisimportant.Inasystematicreview,lowhealthliteracywasfoundtobeassociatedwithworsehealthoutcomes(Berkman,etal.,2011).Granted,whatwasmeasuredintheincludedstudies,aswiththeaforementionedsurveysofhealthliteracy,maynotincludetheobjectiveabilitytoapplyIHCKeyConcepts.Peoplecertainlyaremakingpoordecisions.Systematicreviewshavefound:worldwideoveruseofmedicalservicesmorelikelytodoharmthangood(Brownlee,etal.,2017);underuseofeffectiveservices(Glasziou,etal.,2017);andovertestingandundertestinginprimarycare(O'Sullivan,etal.,2018).Meanwhile,fortunesarespenton“alternative”medicineanddietarysupple-mentswithoutreliableevidencethattheyarebeneficialorsafe(Optum,2013;Starr,2015).TheinabilitytoapplyIHCKeyConceptsislogicallyafactorinmanyofthesedecisions.ThisreviewWhydothereview?WhileIamawareofscientificstudiesofthequalityofnewsmediareportsabouttheeffectsofhealthinterventions(Appendix1),Iwasunabletofindasystemat-icreviewofsuchprimarystudiesinEpistemonikos(www.epistemonikos.org),TripPro(www.tripdatabase.com),orGoogleScholar(www.scholar.google.com),aslaidoutinAppendix2.NeitheramIawareofasystematicreviewofhowqualityhasbeenmeasuredinsuchstudies.Zeraatkaretal.conductedaliteraturesearchbeforedevelopingtheirownmeasurementtool,namedtheQualityIndexforhealth-relatedMediaReports(QIMR)(2017).However,theyonlyassessedvalidatedtools,andtheydidnotcomparecriteriatotheIHCKeyConcepts.
Quality of news media reports about the effects and costs of health interventions: Systematic review protocol 9
Byassessingthemethodsusedtoassessqualityineligiblestudies,theplannedreviewcaninformthedevelopmentanduseofmethodsinfurtherresearch.Byprovidinganoverviewofthequalityofnewsmediareportsabouttheeffectsofhealthinterventions,thereviewcaninformthegenerationofnewresearchquestions,forexamplethegeographicfocusofthosequestions.Intwoways,thereviewcanhelpaddresstheproblemofunreliableinformationabouttheeffectsofhealthinterventions,inabilitytoassessthatinformation,anduninformeddecisionsabouttheinterventions.First,theresultscanbeusedtoraiseawarenessabouttheproblem.Second,byshowinghownewsmediare-portsabouttheeffectsofhealthinterventionstendtobeunreliable(i.e.thespe-cificcriteriathatthereportstendnottosatisfy),theresultscaninformthepri-oritisationofIHCKeyConceptswhendevelopinginterventionstohelpjournal-istsproducereliablereportsandhelpconsumersassessinformationinthenews.Whyfocusonthenewsmedia?Thenewsmediaplayaspecialroleinthedisseminationofhealthinformation.IntheUnitedKingdom,despiteanincreaseintheuseofsocialmedia,themajor-ityofadultsstillfindoutaboutsciencemostregularlythroughnewsmedia,ac-cordingtoarepresentativesurveyof1,749adults(Castell,etal.,2014).Granted,theproportionofpeoplefindingoutaboutspecificallyhealthsciencethroughthenewsmaybedifferent.Meanwhile,ChewandEysenbachfoundthatin5395tweetsabouttheH1N1outbreakin2009,mostlinksweretoanewswebsite(23%)(Chew&Eysenbach,2010).Another12%linkedtonewsblogs,feedsornichenews,comparedto1.5%thatlinkedtogovernmentandpublichealthagencies.Whilethereareotherimportantsourcesoflayinformationabouttheeffectsofhealthinterventions,includingtheminthisreviewwouldbe“mixingapplesandoranges”.Inotherwords,thequalityofinformationinthosesourcesislikelytobesignificantlydifferent.Forexample,patientmaterialsarelikelytoincludemoreinformationaswellasmorereliableinformationthannewsreports,giventhatthematerialsarepreparedbyhealthprofessionalswhoaretryingtoinformpatients,notattractanaudience.Themethodsusedtoassessinformationfromdifferentsourcesmayalsobedifferent.Furthermore,includingsourcesbesidesthenewsmediacouldmakethereviewunwieldy.Rather,thisstudycanbeusedtoinformreviewsfocusingonothersources.AsstatedintheCochraneHandbookforSystematicReviewsofInterventions,there
Quality of news media reports about the effects and costs of health interventions: Systematic review protocol 10
areprosandconstobothbroadandnarrowreviewquestions,butwiththead-ventofoverviewsofsystematicreviews,“Itmayincreasinglybeconsideredde-sirabletoplanaseriesofreviewswitharelativelynarrowscope,alongsideanOverviewtosummarizetheirfindings”(Higgins&Green,2011).
Quality of news media reports about the effects and costs of health interventions: Systematic review protocol 11
Objectives
Primary
1. Assesscriteriausedtomeasurethequalityofnewsmediareportsabouttheeffectsandcostsofhealthinterventions
2. Assessthequalityofnewsmediareportsabouttheeffectsandcostsofhealthinterventions
Secondary
3. Explorefactorsthatmightexplainvariationinqualityofnewsmediare-portsabouttheeffectsandcostsofhealthinterventions
Quality of news media reports about the effects and costs of health interventions: Systematic review protocol 12
Eligibility
CoCoPopMunnetal.notethatquestionsaboutprevalencedonotfitthepopulation,in-tervention,comparatorandoutcome(PICO)framework,usedforsystematicre-viewsaboutefficacy(Munn,etal.,2015,pp.148-149).Thereisnointerventionorexposure,noristhereanoutcomeonwhichaninterventionorexposurecanhaveaneffect.Therefore,insteadofthePICOframework,Iwillusethemne-monicCoCoPop(condition,context,andpopulation)tosetinclusioncriteria,asMunnetal.recommend.Ihaveaddedcriteriaforstudydesign,asperthePre-ferredReportingItemsforSystematicreviewandMeta-AnalysisProtocols(PRISMA-P)statement(Moher,etal.,2015).PopulationInNewsAroundtheWorld:Content,PractitionersandthePublic,ShoemakerandCohenwrite:“[Thejournalist]typicallyconstructsamethodforfulfillingthedailyjobrequirements.Heorsherarelyhasanunderlyingtheoreticalunder-standingofwhatdefiningsomethingorsomeoneasnewsworthyentails”(Shoemaker&Cohen,2006,p.7).Inotherwords,evenjournalistswillstruggletotellyouwhatexactly“news”is.Thisincludesme.InAhistoryofnews,Stephensusesthedefinition:“Newinformationaboutasub-jectofsomepublicinterestthatissharedwithsomeportionofthepublic”(Stephens,2007,p.4).Intermsofthereview,applyingsuchabroaddefinitionwouldbeproblematicinseveralways.Forexample,advertisements,pressre-leasesandjournalarticlescouldallbeconsiderednews.Insteadofusingaspecificdefinition,Iwillbepragmatic.Iwillonlyconsiderstudiesofinformationthat:1)isinnewspapersormagazines(print),radio,podcastsortelevision(broadcast)ornewswebsites(online);and2)islabelledbytheauthorsas“news”oranycommonsynonym.Intermsofonline,Iwillfo-cusondedicatednewswebsites,i.e.notsocialmediaplatforms,althoughnewsreportsappearonsuchplatforms.Iwillonlyincludestudiesofreportsaboutthepotentialhealtheffects(negativeorpositivechangesinhealthoutcomes)andthemonetarycostsofhealthinter-ventions(anyactionintendedtoimproveormaintainthehealthofanindividualorgroup).Thereportscanallbeaboutthesameconditionorintervention,ortheycanbeamix.Ifanotherwiseeligiblestudy’ssampleincludesnewsreportsabouttheeffectsofhealthinterventionsaswellasothertypesofnewsreports,Iwillextractdata
Quality of news media reports about the effects and costs of health interventions: Systematic review protocol 13
onlyabouttheformer,ifpossible.Otherwise,Iwillleaveoutthestudyandlistitinatableofexcludedstudies.Inthesametable,Iwilllistanystudiesofthequalityofotherhealthinformationinnewsreports,besidesinformationabouttheeffectsofhealthinterventions.ConditionBy“quality”,Imeananattributethatiseitherconducivetoinformeddecisionsabouthealthinterventions(e.g.presentspotentialeffectsinabsolutenumbers)ormisleading(e.g.onlypresentspotentialeffectsinrelativenumbers),orthesumofsuchattributes.Theconditionofmypopulation—newsmediareports—iseitherhavingsatisfiedatleastoneexplicitcriterionforqualityornot(e.g.ei-therpresentingpotentialeffectsinabsolutenumbersornot).InpotentiallyeligibleprimarystudiesofwhichIamaware(Appendix1),re-searchershaveuseddifferentsetsofcriteriatomeasurequality.Zeraatkaretal.conductedaliteraturesearchbeforedevelopingtheirowntooltomeasurethequalityofreportingonhealthresearch,namedtheQualityIndexforhealth-relatedMediaReports(QIMR)(2017).Theyidentifiedoneothervalidatedtoolwiththesamepurpose,namedtheIndexofScientificQuality(ISQ)(Oxman,etal.,1993).Theyalsoidentifiedandassessedinstrumentsformeasuringthequalityofpatientinformation:EnsuringQualityInformationforPatients(EQIP)(Moult,etal.,2004)andDISCERN(Charnock,etal.,1999).Zeraatkaretal.con-cludedthatbothEQIPandDISCERNhavepoorcontentrelevancetohealthnewsreports.ItwouldbeproblematictorestrictthereviewtostudiesthatuseeithertheQIMRorISQtomeasurequality.Thiswouldexcludestudiesofreportswherethehealthinformationisnotbasedonresearch—e.g.informationbasedonan-ecdotalevidence.Moreover,theQIMRwaspublishedasrecentlyas2017,andinmostofthepotentiallyeligiblestudiesofwhichIamaware,neithertheQIMRortheISQwereusedtomeasurequality.Inseveralstudies,researchershaveusedcriteriabasedonthoseusedbyMoynihanetal.(Moynihan,etal.,2000)—e.g.Smithetal.(2005).Inotherstudies,researchershavedevelopedandusedtheirowntools,e.g.Marconetal.(2017).Acriterionmaybesensibleevenifthetoolwasnotvalidated,anditmaybesensibletogroupthecriterionwithothersformeta-analysis.ContextThepopulationforthereviewislimitedtonewsintheaforementionedmedi-ums:newspapersandmagazines:radio,podcastsandtelevision;andnewsweb-sites.Iwillnotsetanylimitsintermsofgeography,languageortimeperiod.StudydesignForastudytobeeligible,theresultsmustincludedichotomousorcategoricaldataforatleastoneexplicitcriterionusedtomeasurequality.Itmustbepossi-bletodichotomisethedataaseithersatisfyingthecriterionornot,fortworea-sons:toensuretheresultscanbeeasilyinterpreted,andtosynthesisefindingsacrossstudies.
Quality of news media reports about the effects and costs of health interventions: Systematic review protocol 14
Somenuance,suchasthenumberofreportsthat“partially”satisfyacriterion,maybelostinthissynthesis.Theimportanceofsuchdetailisquestionable;ifaqualityisimportantenoughtobemeasured,thecriterionusedtomeasurethatqualityshouldbeclearlysatisfied,assumingitisrelevanttothespecificreport.Studiesthatonlyprovideaglobalqualityscore,suchasastarrating,willbeex-cludedandlistedinthetableofexcludedstudies.Globalscoresaredifficulttointerpret,sincetheydonottellyouwhichpartsofanewsreportareproblemat-ic.Furthermore,theyaredifficulttocompareacrossstudiesusingdifferentscoringsystems.Tobeincluded,studiesmustspecify:thesamplingframe(i.e.wherethenewsreportsweresampledfrom);theselectioncriteriaforthereports;andtheselec-tiontechnique.Thisexcludescasestudiesofsinglereports.Iwilllistanycasestudiesthatwouldotherwisebeeligibleinthetableofexcludedstudies.
Quality of news media reports about the effects and costs of health interventions: Systematic review protocol 15
Methods
SearchstrategyTomakesuremysearchissystematicandtransparent,IwillbasemyapproachontheguidelinedevelopedbyKableetal.(2012).Asnotedinthedescriptionofmysearchforanexistingreview(Appendix2),whilemyquestionisinterdisci-plinary,Iconsideritunlikelythatasystematicreviewdesignedtoansweritwouldhavebeenpublishedinanacademicjournalfocusedonjournalism.How-ever,Iconsideritmorelikelythataneligibleprimarystudyhasbeenpublishedinsuchajournal.Therefore,inadditiontoPubMed(www.ncbi.nlm.nih.gov/pubmed),IwillsearchGoogleScholar(www.scholar.google.com)andScopus(www.scopus.com).IwillsearchforgreyliteratureinOpenGrey(www.opengrey.ey)andGreyLiteratureReport(www.greylit.org),aswellasthesesinProQuestDissertations&Theses(GlobalFulltextplusUKandIrelandabstracts)(www.proquest.com).Iwillnotplacelimitsonlanguageortimeperiodinanyofmysearches.Inthesearchforanexistingreview,Ionlyincludedtermsforthepopulation(newsmediareports)whensearchingEpistemonikosandTripPro,sinceIcouldfilterforsystematicreviews,whichreducedtheresultstoamanageablenum-ber.Intheupcomingsearchforprimarystudies,giventhereisnotareliablefil-terIcanuseforstudydesign,Iwillalsoincludetermsforthecondition(quality)ortermsrelatedtotheanalysis,toimprovethespecificityofthesearchstrategy.IwillidentifyadditionaltermsforthePubMedsearch,includingMedicalSubjectHeadings(MeSH)terms,bylookingupthestudiesinPubMedor,ifthestudyisnotinPubMed,inanotherdatabase,andextractingrelevantterms.TheaimistoensuremysearchcapturesallstudiesofwhichIamaware,aswellassimilarstudies.Iwillalsoaddtermsforspecificcategoriesofmedia(e.g.print).Iwillconducttestsearchestoinformpragmaticdecisionsabouteditingordroppingterms,withtheaimofmakingthesearchandscreeningasefficientaspossible,whileminimisingtheriskofmissingeligiblestudies.IwillusethePubMedsearchasthebasisforsearchesinotherdatabases.Besidessearchingdatabases,Iwillcheckthereferencelistsofeligiblestudies.InScopus,Iwillscreenstudiescitingthosethatareeligible,aswellasstudiescit-ingthedevelopmentoftheISQortheQIMR.Iwilltakeadvantageofmyprofes-sionalnetwork,conductingtargeteddisseminationoftheprotocolviaemailandbroaddisseminationviaTwitter,tocrowdsourceeligiblestudies.Finally,IwillcontacttheauthorsofeligiblestudiesandthedevelopersoftheISQandQIMR,toaskaboutunpublishedstudies.
Quality of news media reports about the effects and costs of health interventions: Systematic review protocol 16
Studylanguageisnotaneligibilitycriterion,buttobepragmatic,IwillonlysearchEnglishlanguagedatabases.ToscreenanystudieswithoutanEnglishabstractforeligibility,IwilluseGoogleTranslate(www.translate.google.com).Ifnecessary,Iwillseekhelpfromsomeonefluentinthelanguagee.g.anac-quaintanceoroneoftheauthors.Toextractdatafromanon-Englishpaper,Iwillagainseekhelp.Intheexcludedstudiestable,Iwilllistanyapparentlyeli-giblestudiesfromwhichIamunabletoextractdatabecauseofalanguagebar-rierorbecauseofinadequatereportingandinabilitytoobtainmissinginfor-mationfromtheauthors.AsecondreviewerandIwillindependentlyjudgeeligibilitybyreviewingeachabstractand,ifnecessary,(sectionsof)thefulltext.Appendix3istheformwewillusetoconfirmeligibility.Anydisagreementswillbediscussedbeforemov-ingontodataextractionandassessmentofriskofbias.Athirdreviewerwillbebroughtintoarbitrateifnecessary.DataextractionAsecondreviewerandIwillindependentlyextractdataaboutthepopulation,studydesign,criteriausedtoassessthequalityofthereports,andtheresults.Wewillagaindiscussanydisagreements,bringinginathirdreviewertoarbi-trateifnecessary.Appendix4istheformwewillusetoextractdataaboutpopulationanddesign.Appendix5istheseparatespreadsheetinwhichwewillentertheindividualcriteriaandresultsforeachcriterion.Ihavedeterminedwhatdatatoextractbasedonwhatisreportedinthepoten-tiallyeligiblestudiesofwhichIamaware(Appendix1),varioustools(Vandenbroucke,etal.,2007;TheJoannaBriggsInstitute,2014;Moher,etal.,2015;Munn,etal.,2015),myexpertise,andcommonsense.Appendix6con-tainsdummytablesforallthedatatobeextracted,aswellastheriskofbiasas-sessmentsandanalyses.PopulationAsecondreviewerandIwillextractdataaboutthefollowingvariables,intermsofthesamplesofnewsreports:mediumofpublication;geographicareaofpub-lication;timeperiodofpublication;categoryofinterventionreportedon;andwhethertheoutletiscommercialornot.Intermsofmedium,radioandpod-castswillbeonecategory,inadditiontothecategories:newspapers;magazines;television;andnewswebsites.Theninepredeterminedcategoriesofinterven-tionare:“modern”medicine(aka.“academic”,“conventional”or“Western”medicine);“alternative”medicine(aka.“complementary”,“traditional”or“natu-ral”medicine);screening;surgery;devices;diet;exercise;lifestyle;andsystemsandpolicies.StudydesignForeachstudy,wewillrecord:theinstrumentusedtomeasurequality;thestatedobjectives;thesamplingframe;theselectioncriteriaandtechnique;andanyreportedsubgroupanalyses.
Quality of news media reports about the effects and costs of health interventions: Systematic review protocol 17
Criteria,responseoptionsandresultsFinally,wewillrecordeacheligiblecriterionusedtoassessthequalityofre-ports,aswellastheresponseoptions,inaspreadsheetseparatefromtheformusedtocollectdataaboutthepopulationandstudydesign(Appendix5).Wewillentertheresultsforeachcriterionintothesamespreadsheet.Wheredataareavailableforsubgroupsofnewsreports,wewillrecordtheoverallresultsandtheresultsforeachsubgroup.AssessingriskofbiasInasystematicreviewoftoolsusedtoassessthequalityofprevalencestudiesandotherobservationalresearch,Shamliyanetal.identified46scalesand51checklist,noneofwhichdiscriminatedbetweenreportingversusmethodologi-calquality,orexternalversusinternalvalidity(2010).Throughcitationsearch-es,Iidentifiedseveraltoolsdevelopedafterthereview,ofwhichIhavechosentousethelatest,developedbyMunnetal.(2014),asastartingpoint,basedonfacevalidityandsimplicity.WhileMunnetal.donotrefertothereviewbyShamliyanandcolleagues,theycomparetheirtooltoonedevelopedbyHoyetal.(2012)whodoconsidersaidreview.ThecriticalappraisaltooldevelopedbyMunnetal.includestenitems,withtheresponseoptions“Yes”,“No”,“Unclear”and“Notapplicable”:
1. Wasthesamplerepresentativeofthetargetpopulation?2. Werestudyparticipantsrecruitedinanappropriateway?3. Wasthesamplesizeadequate?4. Werethestudysubjectsandthesettingdescribedindetail?5. Wasthedataanalysisconductedwithsufficientcoverageoftheidenti-
fiedsample?6. Wereobjective,standardcriteriausedforthemeasurementofthecondi-
tion?7. Wastheconditionmeasuredreliably?8. Wasthereappropriatestatisticalanalysis?9. Areallimportantconfoundingfactors/subgroups/differencesidentified
andaccountedfor?10. Weresubpopulationsidentifiedusingobjectivecriteria?
Ofthetenitems,threeassessriskofbiasandarerelevantandapplicabletothisreview:items2,6and7.Item2assessesriskofselectionbias,whileitems6and7measuredetectionbias(aka.Informationbias).FocusingonthesetwotypesofbiasisconsistentwiththeStrengtheningtheReportingofObservationalStudiesinEpidemiology(STROBE)statement(Vandenbroucke,etal.,2007,p.814)andmethodsusedbytheCochraneMethodologyReviewGrouptosystematicallyreviewtheprevalenceofaconditioninapopulationofinformation,asopposedtopeople(Welch,etal.,2010,p.7).AsecondreviewerandIwillindependentlyassessriskofbiasineachstudy,cri-terionbycriterion,giventhatsomecriteriausedtomeasurethequalityofnewsreportsrequiremoresubjectivejudgementthanothers.Theseassessmentswillbeusedtodescribethestrengthsandweaknessesofthestudies.Ifdeemednec-essary,theywillalsobeusedtoconductasensitivityanalysis.Itisunclearhow
Quality of news media reports about the effects and costs of health interventions: Systematic review protocol 18
agreaterriskofbiaswouldimpacttheresultsofstudiestobeincludedinthisreview,soIhavenohypothesisabouttheresultsofsuchananalysis.Intermsofselectionbias,ifthenewsreportswererandomlyorsequentiallysampledfromthetargetpopulation,wewillconsidertheriskofbiasaslow.Ifthereportswerepurposivelysampled,wewillconsidertheriskofbiasashigh.Ifanyothersamplingtechniquewasused,wewillconsidertheriskofbiasasmoderateorhigh.Intermsofdetectionbias,ifacriterionrequiresminimaljudgement(e.g.acrite-rionthatrisksbepresentedinabsolutenumbers),wewillconsidertheriskofbiasaslow.Ifthecriterionrequiressubstantialjudgement(e.g.whetherthereissufficientinformationaboutpotentialharms),tworesearchersassessedthere-ports,andtheywereblindedtothejournalistandpublication,wewillconsidertheriskofbiasasmoderate.Ifthecriterionrequiressubstantialjudgement,andonlyoneresearcherassessedthereportsorresearchersassessingthereportswerenotblindedtothejournalistandpublication,wewillconsidertheriskofbiasashigh.Iftheriskofeitherselectionordetectionbiasishigh,wewillconsidertheover-allriskofbiasashigh.Iftheriskofeitherismoderate,wewillconsidertheriskasmoderate.Appendix6includesadummytablefortheriskofbiasassess-ments.Appendix7istheformthatwewillusetoassessriskofbias.Itincludesaspaceforrecordinganyconflictsofinterests.Anysuchconflictswillbedis-cussedseparatelywheninterpretingtheresultsofthereview.AnalysesCriteriausedtomeasurequalityHowqualityhasbeenmeasuredwillbeassessedbycomparingcriteriatotheIHCKeyConcepts(Box1).Eacheligiblecriterionfromeachstudywillbeen-teredintoaspreadsheet,diagonallyacrossfromthefullconceptlist(Appendix8).AsecondreviewerandIwilljudgewhichconceptsarecapturedbyeachcri-terion,representingthejudgementswithmarksinthespreadsheet.Wewilldis-cussanyagreementsandbringinathirdresearchertoarbitrate,ifnecessary.TheIHCKeyConceptsdifferfromotherlistsofcriteriaformakingcausalinfer-encesintwoways(Chalmers,etal.,2018,pp.30-31).First,thelistisdifferentinthatitisboth:a)developedusingasystematic,transparentanditerativepro-cess;andb)intendedtohelpchildren,thegeneralpublicandhealthpractition-ersmakeinformeddecisions,notjusthealthresearchers.Second,unlikeothertoolssuchaschecklistsandtipsheets,itisnotaninterventioninandofitself,limitedtoaparticularpopulationorcontext,butacomprehensiveframeworkthatcanbeusedformappingskills,developinginterventionsandevaluatingin-terventions.UsingtheIHCKeyConceptlisttoassessthecriteriausedtomeasurequalityofthenewsreportshasseveraladvantages.First,itprovidesaframeworkforgroupingcriteriafromdifferentstudies,soresultscanmoreeasilybesynthe-sised.Second,itwillshedlightonrelevantqualitiesthathavenottypicallybeenmeasuredormeasuredatall.Conversely,itmayrevealconceptscapturedby
Quality of news media reports about the effects and costs of health interventions: Systematic review protocol 19
thecriteriamissingfromtheIHCKeyConceptlist.Finally,usingthelistcanin-formtheinterpretationofresultsfromthisreview,includingwhattheresultssuggestaboutwhatconceptsaremostimportantwhendevelopinginterven-tionsforimprovingreportingontheeffectsofinterventions,orhelpingpeopleassessinformationinsuchreports.AsecondreviewerandIwilljudgewhatcriteriausedindifferentstudiescanbegrouped.Oncemore,wewilldiscussanydisagreementsandbringinathirdre-viewertoarbitrateifnecessary.QualityofnewsreportsAsfarasthedatapermits,wewillmeta-analysetheproportionsofsamplesthateithersatisfiedorfailedtosatisfyrespectivecriteria,i.e.theprevalenceofre-portssatisfyingorfailingtosatisfyagivencriterion.SeedummytablesinAp-pendix6.Wewilldichotomisecategoricaldatawherepossibleandsensible.Furthermore,wemayreframedatathatalreadyisdichotomousassatisfyingornotsatisfyingagivencriterion.Notsatisfyingacriterionwillincludeanythingotherthancompletelysatisfyingit.Inotherwords,ifnewsreports“partially”satisfyacriterionoritisunclearwhetherthereportssatisfyit,wewillconsiderthereportsasnothavingsatis-fiedthecriterion.Ifacriterionhasbeendeemedinapplicabletoreports,wewillexcludethosereportsfromthemeta-analysisforthatcriterion.Inadditiontopotentiallygivinguslargersamplesformeta-analysis,thedichotomisationwillmakeinterpretingtheresultsmorestraightforward.Acrossstudies,Ianticipatereportswillgenerallyfailtosatisfymostcriteria,de-spitedifferencesinpopulationsandmethods.Therefore,Iexpectmeanpropor-tionswith95%confidenceintervalswillbemeaningful.Iwillprepareforestplotsandvisuallyanalysetheextenttowhichtherearemeaningfuldifferencesbetweenpointestimates,andIwillconductaChi-squaretesttoseeiftheheter-ogeneityislargerthanonewouldexpectbychance.Foragivencriterion,ifthevisualinterpretationandChi-squaretestsuggestitisappropriate,Iwillconductameta-analysisusingarandomeffectsmodel.Iwillalsoconductsixsubgroupanalyses,whendataareavailable.VariationinqualityAgain,Idonotexpectalotofvariationinthequalityofreportsacrossstudies.Totheextentthatthereisvariation,Idonotexpectalotofdataforexploringit.Thatsaid,thereareseveralvariablesthatcouldlogicallyexplainvariation,evi-dencesupportingsomeofthosehypotheses,andreasonstotestthem.Table1isanoverviewof:thevariablesforwhichwewillconductsubgroupanalyses;thesubgroups;thehypothesiseddifferences;andtherationalesforthehypothesesandforconductingtheanalyses.Thevariablesare:themediuminwhichareportwaspublished;thetimeperiodinwhichareportwaspub-lished;whetherthepublishingoutletwascommercial;whetherthejournalistwasspecialisedinhealthorscience;andtheincomelevelofthecountryinwhichtheoutletissituated.
Quality of news media reports about the effects and costs of health interventions: Systematic review protocol 20
Ontheonehand,Iwillbecautiousaboutextrapolatingfindingsifthereisnodi-rectevidenceforasubpopulation(e.g.broadcastreports)andthereisreasontobeuncertainabouttheapplicabilityoftheevidencefromothersubpopulations(e.g.printandonlinereports).Ontheotherhand,subgroupanalysesarefre-quentlymisleading,asshownbySunetal.(2014).Toavoidspuriousresults,wewilluseexplicitcriteriaforassessingthecredibil-ityofanydifference,basedonSunandcolleagues’criteria:
• Thedifferenceisofpracticalimportance.• Thedifferenceisbiggerthanwewouldexpectbychance.• Thedifferenceisinthehypothesiseddirection.• Thedifferenceisunlikelytobeexplainedbyotherknownvariables.• Thedifferenceisconsistentacrossstudieswithdifferentmethods.• Thedifferenceisconsistentacrossstudieswithdifferentoutcome
measures.• Thedifferenceisconsistentacrossstudieswithdifferentlevelsofriskof
bias.Foreachexplanatoryvariable,givensufficientdata,wewillconductunivariateregressionanalyseswiththevariable(e.g.medium)astheindependentvaria-ble,andwiththeproportionofsubpopulations(e.g.reportsfromrespectivedecades)thatsatisfythecriterionasthedependentvariable.Wewillalsocon-ductmultipleregressionanalyseswithalloftheexplanatoryvariables.Wewillnoteanywithin-studysubgroupanalyseswheretheresearchershaveusedap-propriatestatisticalmethodsorprovidedthenecessarydataforustodoanap-propriatestatisticalanalysisourselves.Wewillconsiderthoseanalyseswhenmakingjudgementsaboutthecredibilityofsubgroupdifferences.Ihavedecidedagainstconductingasubgroupanalysisforthetypeofreport,i.e.whetherthereportisafeatureorabreakingnewsreport.Featuresaretypicallylongerinwordsandpreparedonlongerdeadlines.Theymightlogicallybehigherqualitythanbreakingnewsreports.However,ananalysisbytypeofre-portwouldlikelybeconfoundedbymedium,financialmodel,andspecialisation.Furthermore,howresearchershavedefinedfeatureversusbreakingnews,andhowtheyhavemeasuredlengthofreportordeadline,maybesodifferentacrossstudiesthatcombiningthedatawouldnotmakesense.
Quality of news media reports about the effects and costs of health interventions: Systematic review protocol 21
Table1:SubgroupanalysesVariable Grouping Hypothesis RationaleandimportanceMedium Broadcastvs.other(print
andonline)Broadcastreportsarelesslikelytosatisfycriteria
Broadcastallowsforfewerwords,aswellaslesstimeforwriting,giventimespentrecordingandediting.Thiswouldlogicallyhaveanegativeimpactonthequalityofreports.Iamawareofempiricalevidencethatsupportsthishypothesis(Walsh-Childers,etal.,2016).Acredibledifferenceinthehypothesiseddirectionwouldimplyconsumersshouldpaylessattentiontobroadcastreports.
Timeperiod Decades(≥2000;1990-1999;1980-1989;etc.)
Theproportionofreportssatisfyingcriteriaislikelytobesimilaracrossdecades
Ontheonehand,newsreportsmayhaveimprovedsincetheadventof“evidence-basedmedicine”(EBM)(Evidence-BasedMedicineWorkingGroup,1992).Ontheother,therearesustained,systemicbarrierstoreliablehealthjournalismsuchaslackoftime(Larsson,etal.,2003;Pettersen,2005),whichEBMdoesnotaddress.Moreover,revenuelossesmayhavelednewsmediatoinvestlessinstrategiesthatmightimprovethequalityoftheirhealthnews,suchastrainingjournalistsandeditors,andhiringandretainingspecialisedhealthandsciencereporters.However,acredibleimprovementforoneormorecriteriawouldbeanimpetusforresearchtoexplainsaidimprovement,andthatresearchcouldinformthedevelopmentofinterventionstoimprovequalityoverall.
Financialmodel Commercialvs.non-commercial
Reportspublishedbycommercialoutletsarelesslikelytosatisfycrite-ria
Thefinancialinterestsofcommercialoutletsmayleadtosensationalism,toattractaudiences(Larsson,etal.,2003;Pettersen,2005;Wallington,etal.,2010).Moreover,companiesthatsellhealthinterventionsmaybeadvertisers,disincentivisingcriticalreporting.Acredibledifferenceinthehypothesiseddirectionwouldimplyconsumersshouldsupportnon-commercialoutletsandpaylessattentiontoreportspublishedbycommercialoutlets.
Specialisation Healthandsciencejour-nalistsvs.other
Reportspreparedbyhealthorsci-encejournalistsaremorelikelytosatisfycriteria
Journalistwhohavespecialisedinhealthorsciencearemorelikelytohavereceivedtrainingincriticalappraisal.Moreover,comparedtootherjournalists,theymayhavebeeninmorecontactwithscientists,andmaybemorefamiliarwiththelanguage,contentandstructureofresearchpapers.Thereisevidencethatjournalistswithmoreadvanceddegreesaremorelikelytousescientificjournalarticlesassources,andthatjournalistswithfeweryearsofexperiencearemorelikelytosayprovidingenter-tainmentisanimportantpriorityforhealthnews(Wallington,etal.,2010).Acredibledifferenceinthehypothesiseddirectionwouldimplyconsumersshouldpaymoreattentiontoreportspreparedbyspecialisedhealthandsciencejournalists.
Countryincomelevel
Low-incomecountriesvs.other(middleandhigh-incomecountries)
Theproportionofreportssatisfyingcriteriaislikelytobesimilaracrosscountrieswithdifferentincomelevels
Whilenewsmediainmiddleandhigh-incomecountriesmayhavemoreresources,journalistsinthosecountriesstillfacesys-temicbarrierstohigh-qualityreporting(Larsson,etal.,2003;Pettersen,2005;Wallington,etal.,2010).Supportforthishypoth-esiswouldsuggestresearchersshoulddevelopinterventionsthatcanimprovethequalityofreportinginlow-incomecountriesandmiddleorhigh-incomecountriesalike.Itwouldalsoimplythoseinterventionsaddressbarriersbesidelackofresources.
Newspapertype Broadsheetvs.tabloid Reportsinbroadsheetnewspapersaremorelikelytosatisfycriteria
Reportingintabloidnewspapersmaybelessdetailedandmoresensationalthanreportinginbroadsheetnewspapers,andIamawareofevidencethatthequalityofreportsabouttheeffectsofhealthinterventionsislowerintabloids(Entwistle&Hancock-Beaulieu,1992;Robinson,etal.,2013).Acredibledifferenceinthehypothesiseddirectionwouldimplyconsumersshouldpaymoreattentiontobroadsheetnewspapers,andlesstotabloids.
Quality of news media reports about the effects and costs of health interventions: Systematic review protocol 22
PilotHavingconductedthesearch,wewillpilotforms,spreadsheetsandtables(Ap-pendixes3through6)onthefirstfiveeligiblestudies.Wewilldiscussanyis-suesandmakeanyrevisionsdeemednecessarybeforecompletingdataextrac-tion,riskofbiasassessmentsandanalysesforremainingstudies.RatingthequalityoftheevidenceIhaveadjustedtheGradingofRecommendationsAssessment,DevelopmentandEvaluation(GRADE)approachtoratingthequalityofevidence(Guyatt,etal.,2008).TheGRADEapproachinvolvesconsideringfivefactorsforloweringthecertaintyofevidence:studylimitations,inconsistencyofresults,indirectnessofevidence,imprecision,andpublicationbias.Ihaveadjustedthese,giventhatGRADEwasdevelopedforresearchabouteffects,notprevalence.Appendix9istheformwewillusetoratethequalityoftheevidence.Thefirstadjustmentisthatwewillnotconsiderpublicationbias.Iamnotawareofanyresearchdocumentingpublicationbiasforprevalenceorincidencestudiesgenerally,norspecificallyinthisarea.Besides,Iexpecttoofewstudiesandsamplesthataretoosmallforittobemeaningfultoassesstheriskofpubli-cationbias.Second,wewillnotconsiderdirectnesssinceitisinapplicable.Iwill,however,becautiousaboutextrapolatingevidenceacrosssubpopulationsspecifiedinTable1(e.g.acrossreportsfromdifferenttimeperiods),ifthereisnoevidenceforoneofthesubpopulationsspecific(e.g.reportsfrombeforeaparticulardec-ade).Thisleavesconsideringriskofbias,imprecisionandinconsistency.Intermsofimprecision,wewillusequartilesasaruleofthumb;iftheconfidenceintervalsubstantiallycrossestwoormorequartiles,wewillconsidertheretobeim-portantimprecision,followingguidancedevelopedbytheCochraneEffectivePracticeandOrganisationofCare(EPOC)group(CochraneEffectivePracticeandOrganisationofCare,2017).GRADEfactorsforincreasingthecertaintyoftheevidencearenotrelevanthere.
Quality of news media reports about the effects and costs of health interventions: Systematic review protocol 23
Ethicalconsiderationsandconflictsofinterest
Thereviewdoesnotinvolveparticularethicalchallenges.IdeclarethatIhavenorelevantconflictsofinterest.
Quality of news media reports about the effects and costs of health interventions: Systematic review protocol 24
References
Ashorkhani,M.etal.,2012.Qualityofhealthnewsdisseminatedintheprintmediaindevelopingcountries:AcasestudyinIran.BMCPublicHealth,12(627).Austvoll-Dahlgren,A.etal.,2017.Measuringabilitytoassessclaimsabouttreatmenteffects:alatenttraitanalysisofitemsfromthe‘ClaimEvaluationTools’databaseusingRaschmodelling.BMJOpen,Volume7,p.e013184.Austvoll-Dahlgren,A.,Nsangi,A.&Semakula,D.,2016.Interventionsandassessmenttoolsaddressingkeyconceptspeopleneedtoknowtoappraiseclaimsabouttreatmenteffects:asystematicmappingreview.SystematicReviews,5(215).Austvoll-Dahlgren,A.etal.,2015.Keyconceptsthatpeopleneedtounderstandtoassessclaimsabouttreatmenteffects.JournalofEvidence-BasedMedicine,Volume8,pp.112-125.Austvoll-Dahlgren,A.etal.,2016.Measuringabilitytoassessclaimsabouttreatmenteffects:thedevelopmentofthe‘ClaimEvaluationTools’.BMJOpen,Volume7,p.e013184.Berkman,N.D.etal.,2011.Lowhealthliteracyandhealthoutcomes:Anupdatedsystematicreview.AnnalsofInternalMedicine,155(2),pp.97-107.Brownlee,S.etal.,2017.Evidenceforoveruseofmedicalservicesaroundtheworld.Lancet,390(10090),pp.156-168.Castell,S.etal.,2014.PublicAttitudestoScience2014,London:IpsosMORI.Chalmers,I.etal.,2018.KeyConceptsforInformedHealthChoices:aframeworkforhelpingpeoplelearnhowtoassesstreatmentclaimsandmakeinformedchoices.BMJEvidence-BasedMedicine,23(1),pp.29-33.Charnock,D.,Shepperd,S.,Needham,G.&Gann,R.,1999.DISCERN:aninstrumentforjudgingthequalityofwrittenconsumerhealthinformationontreatmentchoices.JournalofEpidemiologyandCommunityHealth,Volume53,pp.105-111.Chew,C.&Eysenbach,G.,2010.PandemicsintheAgeofTwitter:ContentAnalysisofTweetsduringthe2009H1N1Outbreak.PLoSONE,5(11),p.e14118.CochraneEffectivePracticeandOrganisationofCare,2017.ReportingtheeffectsofaninterventioninEPOCreviews.s.l.:CochraneEffectivePracticeandOrganisationofCare.Coulter,A.,Entwistle,V.&Gilbert,D.,1999.Sharingdecisionswithpatients:istheinformationgoodenough?.BMJ,Volume318,pp.318-322.
Quality of news media reports about the effects and costs of health interventions: Systematic review protocol 25
Culver,J.D.,Gerr,F.&Frumkin,H.,1997.MedicalinformationontheInternet:Astudyofanelectronicbulletinboard.JournalofGeneralInternalMedicine,12(8),pp.466-470.Entwistle,V.&Hancock-Beaulieu,M.,1992.HealthandmedicalcoverageintheUKnationalpress.PublicUnderstandingofScience,Volume1,pp.367-382.Evidence-BasedMedicineWorkingGroup,1992.Evidence-BasedMedicine:ANewApproachtoTeachingthePracticeofMedicine.JAMA,268(17),pp.2420-2425.Faerber,A.E.&Kreling,D.H.,2013.ContentAnalysisofFalseandMisleadingClaimsinTelevisionAdvertisingforPrescriptionandNonprescriptionDrugs.JournalofGeneralInternalMedicine,29(1),pp.110-118.Frosch,D.L.etal.,2007.CreatingDemandforPrescriptionDrugs:AContentAnalysisofTelevisionDirect-to-ConsumerAdvertising.AnnalsofFamilyMedicine,5(1),pp.6-13.Frosch,D.L.,May,S.G.,Tietbohl,C.&Pagán,J.A.,2011.Livinginthe“landofno”?Consumerperceptionsofhealthylifestyleportrayalsindirect-to-consumeradvertisementsofprescriptiondrugs.SocialScience&Medicine,Volume73,pp.995-1002.Glasziou,P.etal.,2017.Evidenceforunderuseofeffectivemedicalservicesaroundtheworld.Lancet,390(10090),pp.169-177.Glenton,C.,Paulsen,E.J.&Oxman,A.D.,2005.PortalstoWonderland:Healthportalsleadtoconfusinginformationabouttheeffectsofhealthcare.BMCMedicalInformaticsandDecisionMaking,5(7).Groven,K.S.&Braitwaite,J.,2016.Happily-ever-after:Personalnarrativesinweight-losssurgeryadvertising.HealthCareforWomenInternational,37(11),pp.1221-1238.Guyatt,G.H.etal.,2008.GRADE:whatis“qualityofevidence”andwhyisitimportanttoclinicians?.BMJ,Volume336,pp.995-998.Higgins,J.&Green,S.eds.,2011.Definingthescopeofareviewquestion(broadversusnarrow).In:CochraneHandbookforSystematicReviewsofInterventions.s.l.:TheCochraneCollaboration.HLS-EUConsortium,2013.ComparativereportofhealthliteracyineightEUmembersstates,s.l.:s.n.Hoffmann,T.C.&DelMar,C.,2015.Patients’ExpectationsoftheBenefitsandHarmsofTreatments,Screening,andTests:ASystematicReview.JAMAInternalMedicine,175(2),pp.274-286.Hoffmann,T.C.&DelMar,C.,2017.Clinicians’ExpectationsoftheBenefitsandHarmsofTreatments,Screening,andTests:ASystematicReview.JAMAInternalMedicine,177(3),pp.407-419.Hoy,D.etal.,2012.Assessingriskofbiasinprevalencestudies:modificationofanexistingtoolandevidenceofinterrateragreement.JournalofClinicalEpidemiology,Volume65,pp.934-939.
Quality of news media reports about the effects and costs of health interventions: Systematic review protocol 26
Kable,A.K.,Pich,J.&Maslin-Prothero,S.E.,2012.Astructuredapproachtodocumentingasearchstrategyforpublication:A12stepguidelineforauthors.NurseEducationToday,Volume32,pp.878-886.Kutner,M.,Greenberg,E.,Jin,Y.&Paulsen,C.,2006.TheHealthLiteracyofAmerica’sAdults:ResultsFromthe2003NationalAssessmentofAdultLiteracy,Washington,DC:NationalCenterforEducationStatistics.Larsson,A.,Oxman,A.D.,Carling,C.&Herrin,J.,2003.Medicalmessagesinthemedia–barriersandsolutionstoimprovingmedicaljournalism.HealthExpectations,Volume6,pp.323-331.Marcon,A.R.,Murdoch,B.&Caulfield,T.,2017.Fakenewsportrayalsofstemcellsandstemcellresearch.RegenerativeMedicine,12(7),pp.765-775.Moher,D.etal.,2015.Preferredreportingitemsforsystematicreviewandmeta-analysisprotocols(PRISMA-P)2015statement.Systematicreviews,4(1).Moult,B.,Franck,L.S.&Brady,H.,2004.EnsuringQualityInformationforPatients:developmentandpreliminaryvalidationofanewinstrumenttoimprovethequalityofwrittenhealthcareinformation.HealthExpectations,Volume7,pp.165-175.Moynihan,R.etal.,2000.Coveragebythenewsmediaofthebenefitsandrisksofmedications.TheNewEnglandJournalofMedicine,342(22),pp.1645-1650.Munn,Z.etal.,2015.Methodologicalguidanceforsystematicreviewsofobservationalepidemiologicalstudiesreportingprevalenceandcumulativeincidencedata.InternationalJournalofEvidence-BasedHealthcare,Volume13,pp.147-153.Munn,Z.,Moola,S.,Riitano,D.&Lisy,K.,2014.Thedevelopmentofacriticalappraisaltoolforuseinsystematicreviewsaddressingquestionsofprevalence.InternationalJournalofHealthPolicyandManagement,3(3),pp.123-128.Nsangi,A.etal.,2017.EffectsoftheInformedHealthChoicesprimaryschoolinterventionontheabilityofchildreninUgandatoassessthereliabilityofclaimsabouttreatmenteffects:acluster-randomisedcontrolledtrial.Lancet,390(10092),pp.374-388.Optum,2013.Effectivenessofhomeopathyforclinicalconditions:Evaluationoftheevidence,s.l.:Optum.O'Sullivan,J.W.etal.,2018.Overtestingandundertestinginprimarycare:asystematicreviewandmeta-analysis.BMJOpen,Volume8,p.e018557.Oxman,A.D.,Austvoll-Dahlgren,A.,Garrat,A.&Rosenbaum,S.,2017.Understandingofkeyconceptsrelevanttoassessingclaimsabouttreatmenteffects:asurveyofNorwegianadults.IHCWorkingPaper.Oslo:InformedHealthChoices.Oxman,A.D.etal.,1993.AnIndexofScientificQualityforHealthReportsintheLayPress.JournalofClinicalEpidemiology,46(9),pp.987-1001.Pettersen,S.,2005.CriticalThinkinginNorwegianUpperSecondaryBiologyEducation:TheCasesofComplementary-Alternative-MedicineandHealthClaimsintheMedia.Nordina,1(2),pp.61-71.
Quality of news media reports about the effects and costs of health interventions: Systematic review protocol 27
Pettersen,S.,2005.Norwegianhealthjournalists’abilitytoreportonhealthresearch:Aconcerntoscienceeducation?.Nordina,1(1),pp.5-16.Pettersen,S.,2007.HealthClaimsandScientificKnowledge:Astudyofhowstudentsofhealthsciences,theirteachers,andnewspaperjournalistsrelatetohealthclaimsinsociety.Oslo:UniversityofOslo.Robinson,A.,Coutinho,A.,Bryden,A.&McKee,M.,2013.AnalysisofhealthstoriesindailynewspapersintheUK.PublicHealth,Volume127,pp.39-45.Sansgiry,S.,Sharp,W.T.&Sansgiry,S.S.,1999.AccuracyofInformationonPrintedOver-the-CounterDrugAdvertisements.HealthMarketingQuarterly,17(2),pp.7-18.Semakula,D.etal.,2017.EffectsoftheInformedHealthChoicespodcastontheabilityofparentsofprimaryschoolchildreninUgandatoassessclaimsabouttreatmenteffects:arandomisedcontrolledtrial.Lancet,390(10092),pp.389-398.Shamliyan,T.,Kane,R.L.&Dickinson,S.,2010.Asystematicreviewoftoolsusedtoassessthequalityofobservationalstudiesthatexamineincidenceorprevalenceandriskfactorsfordiseases.JournalofClinicalEpidemiology,Volume63,pp.1061-1070.Shoemaker,P.J.&Cohen,A.A.,2006.Part1:TheoryandMethod.In:P.J.Shoemaker&A.A.Cohen,eds.NewsAroundtheWorld.NewYork:Routledge,pp.1-34.Smith,D.E.,Wilson,A.J.&Henry,D.A.,2005.Monitoringthequalityofmedicalnewsreporting:earlyexperiencewithmediadoctor.MJA,Volume183,pp.190-193.Spencer,E.A.,Mahtani,K.R.,Goldacre,B.&Heneghan,C.,2016.Claimsforfertilityinterventions:asystematicassessmentofstatementsonUKfertilitycentrewebsites.BMJOpen,Volume6,p.e013940.Starr,R.R.,2015.TooLittle,TooLate:IneffectiveRegulationofDietarySupplementsintheUnitedStates.AmericanJournalofPublicHealth,105(3),pp.478-485.Stephens,M.,2007.AHistoryofNews.3rdEditioned.NewYork:OxfordUniversityPress.Sun,X.etal.,2014.HowtoUseaSubgroupAnalysis:Users'GuidestotheMedicalLiterature.JAMA,311(4),pp.405-411.TheAcademyofMedicalSciences,2016.TheAcademyofMedicalSciences:MedicalInformationSurvey,London:TheAcademyofMedicalSciences.TheJoannaBriggsInstitute,2014.Thesystematicreviewofprevalenceandincidencedata.Adelaide:TheJoannaBriggsInstitute.UnitedStatesGovernmentAccountabilityOffice,2010.HerbalDietarySupplements:ExamplesofDeceptiveorQuestionableMarketingPracticesandPotentiallyDangerousAdvice,s.l.:UnitedStatesGovernmentAccountabilityOffice.
Quality of news media reports about the effects and costs of health interventions: Systematic review protocol 28
Vandenbroucke,J.P.etal.,2007.StrengtheningtheReportingofObservationalStudiesinEpidemiology(STROBE):Explanationandelaboration.Epidemiology,18(6),pp.805-835.Wallington,S.F.,Blake,K.,Taylor-Clark,K.&Viswanath,K.,2010.AntecedentstoAgendaSettingandFraminginHealthNews:AnExaminationofPriority,Angle,Source,andResourceUsagefromaNationalSurveyofU.S.HealthReportersandEditors.JournalofHealthCommunication,15(1),pp.76-94.Walsh-Childers,K.,Braddock,J.,Rabaza,C.&Schwitzer,G.,2016.OneStepForward,OneStepBack:ChangesinNewsCoverageofMedicalInterventions.HealthCommunication,33(2),pp.174-187.Welch,V.etal.,2010.Howeffectsonhealthequityareassessedinsystematicreviewsofinterventions(Review).CochraneDatabaseofSystematicReviews,Issue12.Wolfe,R.M.,Sharp,L.K.&Lipsky,M.S.,2002.ContentandDesignAttributesofAntivaccinationWebSites.JAMA,287(24),pp.3245-3248.Woloshin,S.,Schwartz,L.M.&Welch,H.G.,2008.Knowyourchances:understandinghealthstatistics.Berkeley:UniversityofCaliforniaPress.Zeraatkar,D.,Obeda,M.,Ginsberg,J.S.&Hirsh,J.,2017.Thedevelopmentandvalidationofaninstrumenttomeasurethequalityofhealthresearchreportsinthelaymedia.BMCPublicHealth,17(343).