Quality of news media reports about the effects and costs of health interventions: Systematic review protocol 1

Qualityofnewsmediareportsabouttheeffectsandcostsofhealthinterventions:SystematicreviewprotocolOxmanMWorkingpaper,April26,2018www.informedhealthchoices.org

Quality of news media reports about the effects and costs of health interventions: Systematic review protocol 2

Colophon

Title Qualityofnewsmediareportsabouttheeffectsandcostsofhealthinterventions:Systematicreviewprotocol

Authors Oxman, Matt 1,2,3

1. University of Oxford, United Kingdom 2. Faculty of Health Sciences,Oslo Metropolitan University, Norway 3. Centre for Informed Health Choices,Norwegian Institute of Public Health, Norway

Correspondingauthor

Matt Oxmanmatt@mattoxman.com Telephone: +47 98636788 Oslo Metropolitan University, Faculty of Health Sciences Section for Research & Development Pilestredet Campus P.O. Box 4 St. Olavs plass N-0130 Oslo, Norway

Keywords Newsmedia,Systematicreview,Unreliableinformation,KeyConcepts

Citation Oxman,M.,Qualityofnewsmediareportsabouttheeffectsandcostsofhealthinterventions:Systematicreviewprotocol.IHCWorkingPaper.Oslo:InformedHealthChoices.ISBN: 978-82-8082-971-9

Articlecategory �AboutInformedHealthChoices�Keyconceptsandglossary�Learningresources�Systematicreviews�Developmentandevaluationoflearningresources�Claimevaluationtools�Editorialsandcommentaries�Grantapplicationsþ Other

Date April26,2018

Quality of news media reports about the effects and costs of health interventions: Systematic review protocol 3

Abstract

Background:Unreliableinformationabouthealthcare,combinedwiththeina-bilitytoassessthereliabilityofsuchinformation,canleadtouninformeddeci-sionsand,ultimately,wasteandunnecessarysuffering.Journalismisaparticu-larlyimportantsourceofhealthinformation.Therehavebeenseveralscientificstudiesofthequality(reliability)ofnewsmediareportsabouttheeffectsofhealthinterventions.Theredoesnotappeartobeasystematicreviewofthesestudies.Theplannedreviewisintendedtoinform:theproductionandcon-sumptionofhealthnews;aswellasfurtherresearch,includingthedevelopmentofinterventionstohelppeopleassesshealthcareinformationinthenews,sotheycanmakewell-informeddecisions.Objectives:1)Toassesscriteriausedtomeasurethequalityofnewsmediare-portsabouttheeffectsandcostsofhealthinterventions;2)toassessthequalityofsuchreports;and3)toexplorefactorsthatmightexplainvariationinquality.Eligibility:Wewillincludescientificstudiesofprint,broadcastandonlinenewsreports(thepopulation).Atleastoneexplicitcriterionmusthavebeenusedbytheresearcherstomeasurequality(thecondition).Itmustbepossibleandsen-sibletoframetheresultsforatleastonecriterionastheproportionofreportsthatsatisfiedit(e.g.theproportionofreportsinwhicheffectestimatesareab-solute,notonlyrelative).Therewillbenolimitsonlanguage,geographyortimeperiodofthereports(thecontext).Tobeincluded,studiesmustspecify:thesamplingframe;theselectioncriteria;andtheselectiontechnique.Methods:Tworeviewerswillextractdata,assessriskofbias,andmakejudge-mentsaboutInformedHealthChoices(IHC)KeyConceptscapturedbycriteriausedinscientificstudies.Ifnecessary,athirdreviewerwillhelpreachconsen-susonanyjudgement.Meta-analyseswillbeconductedtoestimatethepreva-lenceofnewsreportsthatsatisfyrespectivecriteria.Toexplorevariation,sub-groupanalyseswillbeconductedfor:medium(broadcastvs.other);timeperi-od(decade);financialmodel(commercialvs.non-commercial);journalistspe-cialisation(healthorsciencevs.other);andcountryincomelevel(low-incomevs.other).

Quality of news media reports about the effects and costs of health interventions: Systematic review protocol 4

Background

TheproblemUnreliablehealthinformationThisprotocolisforasystematicreviewthatisultimatelyintendedtohelpad-dressamajor,globalproblem.Theproblemhasthreeparts.Thefirstisunrelia-blehealthinformation.Ineffect,thereisendlessinformationabouttheeffectsofhealthinterventions,ahealthinterventionbeinganyactionintendedtoimproveormaintainthehealthofindividualsorgroups.Thisincludesinformationabout:“modern”,“academ-ic”,“conventional”or“western”medicine;“complementary”,“alternative”,“tra-ditional”or“natural”medicine;screening;surgeryanddevices;diet,exerciseandlifestyle;andsystemsandpolicies.Someofthisinformation,ifnotmostofit,isunreliableinonewayoranother.Theinformationcanbedirectlymislead-ing,suchastheexplicitclaimthataninterventioncausesanoutcomewhenitisonlyassociatedwiththeoutcome.Oritcanbemisleadingbyomission,suchasrelativeeffectestimateswithoutabsoluteestimates,particularlywhenthebase-lineriskissmall(Woloshin,etal.,2008).Researchershavefoundunreliablelayinformationabouttheeffectsofhealthinterventionsin:

• Patientmaterials(Coulter,etal.,1999);productlabels(UnitedStatesGovernmentAccountabilityOffice,2010),

• Varioustypesofwebsites(Culver,etal.,1997;Wolfe,etal.,2002;Glenton,etal.,2005;Spencer,etal.,2016)

• Advertisements(Sansgiry,etal.,1999;Frosch,etal.,2007;Frosch,etal.,2011;Faerber&Kreling,2013;Groven&Braitwaite,2016)

• Thenews(Appendix1)Informationinthenewsmediaisofparticularinterest,forreasonsdescribedlaterinthebackgroundsection.

Quality of news media reports about the effects and costs of health interventions: Systematic review protocol 5

Finally,researchreferencedinthefollowingsubsectionshowspeopleareuna-bletoassessinformationabouttheeffectsofhealthinterventions.Logically,ifpeopleareunabletoassesssuchinformation,theyaremorelikelytospreadthatwhichisunreliable.Conversely,peoplespreadingunreliableinformation,asdocumentedinthestudiesreferencedinthissubsection,suggeststheyareunabletoassesssuchinformation.InabilitytoassesshealthinformationThesecondpartoftheproblemisthatpeopleareunabletoassessthereliabilityofinformationabouttheeffectsofhealthinterventions.AspartoftheInformedHealthChoices(IHC)project(www.informedhealthchoices.org),theabilitytoassesssuchinformationandmakeinformeddecisionsabouthealthinterven-tionshasbeenbrokendownintotheabilitytoapplyspecificconceptscalledtheIHCKeyConcepts.Forexample,peopleneedtobeabletoapplytheconceptthatassociationisnotthesameascausation.Theoriginallistincluded32concepts(Austvoll-Dahlgren,etal.,2015).Thenewestiterationincludes36(Box1)(Chalmers,etal.,2018).TheabilitytoapplyIHCKeyConceptsispartofhealthliteracyandhasbeenmeasuredbothdirectlyandindirectly.Therehavebeenlargesurveysofhealthliteracy,inEuropebytheEuropeanHealthLiteracyProjectConsortium(HLS-EUConsortium,2013)andintheUnit-edStatesbytheNationalCenterforEducationStatistics(Kutner,etal.,2006).AccordingtotheEuropeansurvey,“Healthliteracyislinkedtoliteracyanden-tailspeople’sknowledge,motivationandcompetencestoaccess,understand,appraise,andapplyhealthinformationinordertomakejudgmentsandtakede-cisionsineverydaylifeconcerninghealthcare,diseasepreventionandhealthpromotiontomaintainorimprovequalityoflifeduringthelifecourse”(HLS-EUConsortium,2013,p.7).FortheAmericansurvey,thetermwasdefinedas:“Thedegreetowhichindividualshavethecapacitytoobtain,process,andunder-standbasichealthinformationandservicesneededtomakeappropriatehealthdecisions”(Kutner,etal.,2006,p.iii).TheabilitytoapplyIHCKeyConcepts(i.e.assessthereliabilityofinformationabouttheeffectsofhealthinterventionsandmakewell-informeddecisions)islogicallyincludedinbothdefinitions.However,neithersurveyappearstohaveobjectivelymeasuredthis.TheEuropeansurveymeasuredthesubjectiveeaseof47tasks.TasksrelatedtoapplyingIHCKeyConcepts,including“judgeiftheinformationaboutillnessinthemediaisreliable”,wererankedamongstthemostdifficult(HLS-EUConsortium,2013).

Quality of news media reports about the effects and costs of health interventions: Systematic review protocol 6

Box1:ShorttitlesfortheInformedHealthChoices(IHC)KeyConceptsRecognisinganunreliablebasisforaclaim

1.1. Treatmentscanharm.1.2. Anecdotesareunreliableevidence.1.3. Associationisnotthesameascausation.1.4. Commonpracticeisnotalwaysevidence-based.1.5. Newerisnotnecessarilybetter.1.6. Expertopinionisnotalwaysright.1.7. Bewareofconflictinginterests.1.8. Moreisnotnecessarilybetter.1.9. Earlierisnotnecessarilybetter.1.10. Hopemayleadtounrealisticexpectations.1.11. Explanationsabouthowtreatmentsworkcanbewrong.1.12. Dramatictreatmenteffectsarerare.

Understandingwhethercomparisonsarefairandreliable2.1. Comparisonsareneededtoidentifytreatmenteffects.2.2. Comparisongroupsshouldbesimilar.2.3. Peoples’outcomesshouldbeanalysedintheiroriginalgroups.2.4. Comparisongroupsshouldbetreatedequally.2.5. Peopleshouldnotknowwhichtreatmenttheyget.2.6. Peoples’outcomesshouldbeassessedsimilarly.2.7. Allshouldbefollowedup.2.8. Consideralltherelevantfaircomparisons.2.9. Reviewsoffaircomparisonsshouldbesystematic.2.10. Peerreviewandpublicationdoesnotguaranteereliableinformation.2.11. Allfaircomparisonsandoutcomesshouldbereported.2.12. Subgroupanalysesmaybemisleading.2.13. Relativemeasuresofeffectscanbemisleading.2.14. Averagemeasuresofeffectscanbemisleading.2.15. Faircomparisonswithfewpeopleoroutcomeeventscanbemisleading.2.16. Confidenceintervalsshouldbereported.2.17. Donotconfuse‘statisticalsignificance’with‘importance’.2.18. Donotconfuse‘noevidenceofadifference’with‘evidenceofnodifference’.

Makinginformedchoices3.1. Dotheoutcomesmeasuredmattertoyou?3.2. Areyouverydifferentfromthepeoplestudied?3.3. Arethetreatmentspracticalinyoursetting?3.4. Dotreatmentcomparisonsreflectyourcircumstances?3.5. Howcertainistheevidence?3.6. Dotheadvantagesoutweighthedisadvantages?

Quality of news media reports about the effects and costs of health interventions: Systematic review protocol 7

TheAmericansurveysuggestedaminorityofAmericanadults(12%)had“pro-ficient”healthliteracy(Kutner,etal.,2006,p.v).However,onlyasampleofthehealthliteracyquestionsusedinthatsurveyarepubliclyavailableandnoneoftheavailablequestionsmeasuretheabilitytoassessinformationaboutcausali-ty(www.nces.ed.gov/NAAL/sample.asp).Anothersurrogatemeasureofpeople’sabilitytoapplyIHCKeyConceptsistheirexpectationsabouteffects.Inasystematicreviewofpatients’expectations,themajorityofparticipantsoverestimatedbenefitsandunderestimatedharms(Hoffmann&DelMar,2015).Asystematicreviewofclinicians’expectationsshowedcliniciansdothesame(Hoffmann&DelMar,2017).Othersubstitutesforabilityareattitudesandbeliefs.AccordingtoastudyintheUnitedKingdom,patientsandgeneralpractitionersappeartoplaceadispropor-tionateamountoftrustinthereputationoftheorganisationconductingaclini-caltrial,thequalificationsoftheresearchers,andpeer-review,versusthemeth-ods,whicharewhatinfactdeterminethereliabilityoftheresults(TheAcademyofMedicalSciences,2016).Moreover,aboutathirdoftherespondents(37%)placedahighleveloftrustindatafrommedicaltrials,whileabouttwothirds(65%)placedahighleveloftrustintheexperiencesoffriendsandfamily,i.e.anecdotalevidence.TherehaveindeedbeenstudiesdirectlymeasuringtheabilitytoapplyKeyCon-cepts.InNorway,whereIambased,Oxmanetal.testedarandomsampleof626Norwegianadults(Oxman,etal.,2017).Aboutoneinfive(19%)showedtheywereabletodistinguishbetweenanassociationandcausation.OthersurveyssuggestNorwegianstudentsinpost-secondaryschool(Pettersen,2007)andsecondaryschool(Pettersen,2005)alsostrugglewithassessingclaimsabouttheeffectsoftreatments.AnotherproductoftheInformedHealthChoicesprojectistheCLAIMEvaluationToolsdatabase(www.informedhealthchoices.org/claim-evaluation-tools)(Chalmers,etal.,2018).TheCLAIMEvaluationToolsaremultiple-choiceques-tionsdevelopedspecificallytomeasurepeople’sabilitytoapplytheIHCKeyConcepts(Austvoll-Dahlgren,etal.,2016;Austvoll-Dahlgren,etal.,2017).Be-forethedevelopmentofthedatabase,inasystematicreviewoftoolsformeas-uringsaidability,Austvoll-Dahlgrenetal.identified215discreteinstrumentsorprocedures,capturingupto15oftheconcepts,whereoffourcaptured10ormore(Austvoll-Dahlgren,etal.,2016).Avalidatedsetof26questionsmeasuringtheabilitytoapply13oftheconcepts(i.e.twoquestionsperconcept)wereusedtomeasuretheeffectsofIHClearn-ingresources,intworandomisedtrials(Nsangi,etal.,2017;Semakula,etal.,

Quality of news media reports about the effects and costs of health interventions: Systematic review protocol 8

2017).Inthefirsttrial,thecontrolgroupwas4430Ugandanprimaryschoolchildren,afterlosstofollow-up(Nsangi,etal.,2017).Inthesecondtrial,afterlosstofollow-up,thecontrolgroupwas273parentsandguardiansofchildrenparticipatingintheinterventionorcontrolgroupofthefirsttrial(Semakula,etal.,2017).Lessthanhalfoftheparticipantsinthecontrolgroupforeithertrialansweredbothquestionscorrectlyforanyofthe13includedconcepts.UninformedhealthdecisionsThecombinationofunreliableinformationabouttheeffectsofhealthinterven-tionsandpeople’sinabilitytoassesssuchinformationcanresultinuninformeddecisions.Thisisthethirdpartoftheproblem.Peopleactingonunreliablein-formation,orfailingtoactonreliableadvice,caninturnleadtowasteandun-necessarysuffering.Thisiswhytheproblemisimportant.Inasystematicreview,lowhealthliteracywasfoundtobeassociatedwithworsehealthoutcomes(Berkman,etal.,2011).Granted,whatwasmeasuredintheincludedstudies,aswiththeaforementionedsurveysofhealthliteracy,maynotincludetheobjectiveabilitytoapplyIHCKeyConcepts.Peoplecertainlyaremakingpoordecisions.Systematicreviewshavefound:worldwideoveruseofmedicalservicesmorelikelytodoharmthangood(Brownlee,etal.,2017);underuseofeffectiveservices(Glasziou,etal.,2017);andovertestingandundertestinginprimarycare(O'Sullivan,etal.,2018).Meanwhile,fortunesarespenton“alternative”medicineanddietarysupple-mentswithoutreliableevidencethattheyarebeneficialorsafe(Optum,2013;Starr,2015).TheinabilitytoapplyIHCKeyConceptsislogicallyafactorinmanyofthesedecisions.ThisreviewWhydothereview?WhileIamawareofscientificstudiesofthequalityofnewsmediareportsabouttheeffectsofhealthinterventions(Appendix1),Iwasunabletofindasystemat-icreviewofsuchprimarystudiesinEpistemonikos(www.epistemonikos.org),TripPro(www.tripdatabase.com),orGoogleScholar(www.scholar.google.com),aslaidoutinAppendix2.NeitheramIawareofasystematicreviewofhowqualityhasbeenmeasuredinsuchstudies.Zeraatkaretal.conductedaliteraturesearchbeforedevelopingtheirownmeasurementtool,namedtheQualityIndexforhealth-relatedMediaReports(QIMR)(2017).However,theyonlyassessedvalidatedtools,andtheydidnotcomparecriteriatotheIHCKeyConcepts.

Quality of news media reports about the effects and costs of health interventions: Systematic review protocol 9

Byassessingthemethodsusedtoassessqualityineligiblestudies,theplannedreviewcaninformthedevelopmentanduseofmethodsinfurtherresearch.Byprovidinganoverviewofthequalityofnewsmediareportsabouttheeffectsofhealthinterventions,thereviewcaninformthegenerationofnewresearchquestions,forexamplethegeographicfocusofthosequestions.Intwoways,thereviewcanhelpaddresstheproblemofunreliableinformationabouttheeffectsofhealthinterventions,inabilitytoassessthatinformation,anduninformeddecisionsabouttheinterventions.First,theresultscanbeusedtoraiseawarenessabouttheproblem.Second,byshowinghownewsmediare-portsabouttheeffectsofhealthinterventionstendtobeunreliable(i.e.thespe-cificcriteriathatthereportstendnottosatisfy),theresultscaninformthepri-oritisationofIHCKeyConceptswhendevelopinginterventionstohelpjournal-istsproducereliablereportsandhelpconsumersassessinformationinthenews.Whyfocusonthenewsmedia?Thenewsmediaplayaspecialroleinthedisseminationofhealthinformation.IntheUnitedKingdom,despiteanincreaseintheuseofsocialmedia,themajor-ityofadultsstillfindoutaboutsciencemostregularlythroughnewsmedia,ac-cordingtoarepresentativesurveyof1,749adults(Castell,etal.,2014).Granted,theproportionofpeoplefindingoutaboutspecificallyhealthsciencethroughthenewsmaybedifferent.Meanwhile,ChewandEysenbachfoundthatin5395tweetsabouttheH1N1outbreakin2009,mostlinksweretoanewswebsite(23%)(Chew&Eysenbach,2010).Another12%linkedtonewsblogs,feedsornichenews,comparedto1.5%thatlinkedtogovernmentandpublichealthagencies.Whilethereareotherimportantsourcesoflayinformationabouttheeffectsofhealthinterventions,includingtheminthisreviewwouldbe“mixingapplesandoranges”.Inotherwords,thequalityofinformationinthosesourcesislikelytobesignificantlydifferent.Forexample,patientmaterialsarelikelytoincludemoreinformationaswellasmorereliableinformationthannewsreports,giventhatthematerialsarepreparedbyhealthprofessionalswhoaretryingtoinformpatients,notattractanaudience.Themethodsusedtoassessinformationfromdifferentsourcesmayalsobedifferent.Furthermore,includingsourcesbesidesthenewsmediacouldmakethereviewunwieldy.Rather,thisstudycanbeusedtoinformreviewsfocusingonothersources.AsstatedintheCochraneHandbookforSystematicReviewsofInterventions,there

Quality of news media reports about the effects and costs of health interventions: Systematic review protocol 10

areprosandconstobothbroadandnarrowreviewquestions,butwiththead-ventofoverviewsofsystematicreviews,“Itmayincreasinglybeconsideredde-sirabletoplanaseriesofreviewswitharelativelynarrowscope,alongsideanOverviewtosummarizetheirfindings”(Higgins&Green,2011).

Quality of news media reports about the effects and costs of health interventions: Systematic review protocol 11

Objectives

Primary

1. Assesscriteriausedtomeasurethequalityofnewsmediareportsabouttheeffectsandcostsofhealthinterventions

2. Assessthequalityofnewsmediareportsabouttheeffectsandcostsofhealthinterventions

Secondary

3. Explorefactorsthatmightexplainvariationinqualityofnewsmediare-portsabouttheeffectsandcostsofhealthinterventions

Quality of news media reports about the effects and costs of health interventions: Systematic review protocol 12

Eligibility

CoCoPopMunnetal.notethatquestionsaboutprevalencedonotfitthepopulation,in-tervention,comparatorandoutcome(PICO)framework,usedforsystematicre-viewsaboutefficacy(Munn,etal.,2015,pp.148-149).Thereisnointerventionorexposure,noristhereanoutcomeonwhichaninterventionorexposurecanhaveaneffect.Therefore,insteadofthePICOframework,Iwillusethemne-monicCoCoPop(condition,context,andpopulation)tosetinclusioncriteria,asMunnetal.recommend.Ihaveaddedcriteriaforstudydesign,asperthePre-ferredReportingItemsforSystematicreviewandMeta-AnalysisProtocols(PRISMA-P)statement(Moher,etal.,2015).PopulationInNewsAroundtheWorld:Content,PractitionersandthePublic,ShoemakerandCohenwrite:“[Thejournalist]typicallyconstructsamethodforfulfillingthedailyjobrequirements.Heorsherarelyhasanunderlyingtheoreticalunder-standingofwhatdefiningsomethingorsomeoneasnewsworthyentails”(Shoemaker&Cohen,2006,p.7).Inotherwords,evenjournalistswillstruggletotellyouwhatexactly“news”is.Thisincludesme.InAhistoryofnews,Stephensusesthedefinition:“Newinformationaboutasub-jectofsomepublicinterestthatissharedwithsomeportionofthepublic”(Stephens,2007,p.4).Intermsofthereview,applyingsuchabroaddefinitionwouldbeproblematicinseveralways.Forexample,advertisements,pressre-leasesandjournalarticlescouldallbeconsiderednews.Insteadofusingaspecificdefinition,Iwillbepragmatic.Iwillonlyconsiderstudiesofinformationthat:1)isinnewspapersormagazines(print),radio,podcastsortelevision(broadcast)ornewswebsites(online);and2)islabelledbytheauthorsas“news”oranycommonsynonym.Intermsofonline,Iwillfo-cusondedicatednewswebsites,i.e.notsocialmediaplatforms,althoughnewsreportsappearonsuchplatforms.Iwillonlyincludestudiesofreportsaboutthepotentialhealtheffects(negativeorpositivechangesinhealthoutcomes)andthemonetarycostsofhealthinter-ventions(anyactionintendedtoimproveormaintainthehealthofanindividualorgroup).Thereportscanallbeaboutthesameconditionorintervention,ortheycanbeamix.Ifanotherwiseeligiblestudy’ssampleincludesnewsreportsabouttheeffectsofhealthinterventionsaswellasothertypesofnewsreports,Iwillextractdata

Quality of news media reports about the effects and costs of health interventions: Systematic review protocol 13

onlyabouttheformer,ifpossible.Otherwise,Iwillleaveoutthestudyandlistitinatableofexcludedstudies.Inthesametable,Iwilllistanystudiesofthequalityofotherhealthinformationinnewsreports,besidesinformationabouttheeffectsofhealthinterventions.ConditionBy“quality”,Imeananattributethatiseitherconducivetoinformeddecisionsabouthealthinterventions(e.g.presentspotentialeffectsinabsolutenumbers)ormisleading(e.g.onlypresentspotentialeffectsinrelativenumbers),orthesumofsuchattributes.Theconditionofmypopulation—newsmediareports—iseitherhavingsatisfiedatleastoneexplicitcriterionforqualityornot(e.g.ei-therpresentingpotentialeffectsinabsolutenumbersornot).InpotentiallyeligibleprimarystudiesofwhichIamaware(Appendix1),re-searchershaveuseddifferentsetsofcriteriatomeasurequality.Zeraatkaretal.conductedaliteraturesearchbeforedevelopingtheirowntooltomeasurethequalityofreportingonhealthresearch,namedtheQualityIndexforhealth-relatedMediaReports(QIMR)(2017).Theyidentifiedoneothervalidatedtoolwiththesamepurpose,namedtheIndexofScientificQuality(ISQ)(Oxman,etal.,1993).Theyalsoidentifiedandassessedinstrumentsformeasuringthequalityofpatientinformation:EnsuringQualityInformationforPatients(EQIP)(Moult,etal.,2004)andDISCERN(Charnock,etal.,1999).Zeraatkaretal.con-cludedthatbothEQIPandDISCERNhavepoorcontentrelevancetohealthnewsreports.ItwouldbeproblematictorestrictthereviewtostudiesthatuseeithertheQIMRorISQtomeasurequality.Thiswouldexcludestudiesofreportswherethehealthinformationisnotbasedonresearch—e.g.informationbasedonan-ecdotalevidence.Moreover,theQIMRwaspublishedasrecentlyas2017,andinmostofthepotentiallyeligiblestudiesofwhichIamaware,neithertheQIMRortheISQwereusedtomeasurequality.Inseveralstudies,researchershaveusedcriteriabasedonthoseusedbyMoynihanetal.(Moynihan,etal.,2000)—e.g.Smithetal.(2005).Inotherstudies,researchershavedevelopedandusedtheirowntools,e.g.Marconetal.(2017).Acriterionmaybesensibleevenifthetoolwasnotvalidated,anditmaybesensibletogroupthecriterionwithothersformeta-analysis.ContextThepopulationforthereviewislimitedtonewsintheaforementionedmedi-ums:newspapersandmagazines:radio,podcastsandtelevision;andnewsweb-sites.Iwillnotsetanylimitsintermsofgeography,languageortimeperiod.StudydesignForastudytobeeligible,theresultsmustincludedichotomousorcategoricaldataforatleastoneexplicitcriterionusedtomeasurequality.Itmustbepossi-bletodichotomisethedataaseithersatisfyingthecriterionornot,fortworea-sons:toensuretheresultscanbeeasilyinterpreted,andtosynthesisefindingsacrossstudies.

Quality of news media reports about the effects and costs of health interventions: Systematic review protocol 14

Somenuance,suchasthenumberofreportsthat“partially”satisfyacriterion,maybelostinthissynthesis.Theimportanceofsuchdetailisquestionable;ifaqualityisimportantenoughtobemeasured,thecriterionusedtomeasurethatqualityshouldbeclearlysatisfied,assumingitisrelevanttothespecificreport.Studiesthatonlyprovideaglobalqualityscore,suchasastarrating,willbeex-cludedandlistedinthetableofexcludedstudies.Globalscoresaredifficulttointerpret,sincetheydonottellyouwhichpartsofanewsreportareproblemat-ic.Furthermore,theyaredifficulttocompareacrossstudiesusingdifferentscoringsystems.Tobeincluded,studiesmustspecify:thesamplingframe(i.e.wherethenewsreportsweresampledfrom);theselectioncriteriaforthereports;andtheselec-tiontechnique.Thisexcludescasestudiesofsinglereports.Iwilllistanycasestudiesthatwouldotherwisebeeligibleinthetableofexcludedstudies.

Quality of news media reports about the effects and costs of health interventions: Systematic review protocol 15

Methods

SearchstrategyTomakesuremysearchissystematicandtransparent,IwillbasemyapproachontheguidelinedevelopedbyKableetal.(2012).Asnotedinthedescriptionofmysearchforanexistingreview(Appendix2),whilemyquestionisinterdisci-plinary,Iconsideritunlikelythatasystematicreviewdesignedtoansweritwouldhavebeenpublishedinanacademicjournalfocusedonjournalism.How-ever,Iconsideritmorelikelythataneligibleprimarystudyhasbeenpublishedinsuchajournal.Therefore,inadditiontoPubMed(www.ncbi.nlm.nih.gov/pubmed),IwillsearchGoogleScholar(www.scholar.google.com)andScopus(www.scopus.com).IwillsearchforgreyliteratureinOpenGrey(www.opengrey.ey)andGreyLiteratureReport(www.greylit.org),aswellasthesesinProQuestDissertations&Theses(GlobalFulltextplusUKandIrelandabstracts)(www.proquest.com).Iwillnotplacelimitsonlanguageortimeperiodinanyofmysearches.Inthesearchforanexistingreview,Ionlyincludedtermsforthepopulation(newsmediareports)whensearchingEpistemonikosandTripPro,sinceIcouldfilterforsystematicreviews,whichreducedtheresultstoamanageablenum-ber.Intheupcomingsearchforprimarystudies,giventhereisnotareliablefil-terIcanuseforstudydesign,Iwillalsoincludetermsforthecondition(quality)ortermsrelatedtotheanalysis,toimprovethespecificityofthesearchstrategy.IwillidentifyadditionaltermsforthePubMedsearch,includingMedicalSubjectHeadings(MeSH)terms,bylookingupthestudiesinPubMedor,ifthestudyisnotinPubMed,inanotherdatabase,andextractingrelevantterms.TheaimistoensuremysearchcapturesallstudiesofwhichIamaware,aswellassimilarstudies.Iwillalsoaddtermsforspecificcategoriesofmedia(e.g.print).Iwillconducttestsearchestoinformpragmaticdecisionsabouteditingordroppingterms,withtheaimofmakingthesearchandscreeningasefficientaspossible,whileminimisingtheriskofmissingeligiblestudies.IwillusethePubMedsearchasthebasisforsearchesinotherdatabases.Besidessearchingdatabases,Iwillcheckthereferencelistsofeligiblestudies.InScopus,Iwillscreenstudiescitingthosethatareeligible,aswellasstudiescit-ingthedevelopmentoftheISQortheQIMR.Iwilltakeadvantageofmyprofes-sionalnetwork,conductingtargeteddisseminationoftheprotocolviaemailandbroaddisseminationviaTwitter,tocrowdsourceeligiblestudies.Finally,IwillcontacttheauthorsofeligiblestudiesandthedevelopersoftheISQandQIMR,toaskaboutunpublishedstudies.

Quality of news media reports about the effects and costs of health interventions: Systematic review protocol 16

Studylanguageisnotaneligibilitycriterion,buttobepragmatic,IwillonlysearchEnglishlanguagedatabases.ToscreenanystudieswithoutanEnglishabstractforeligibility,IwilluseGoogleTranslate(www.translate.google.com).Ifnecessary,Iwillseekhelpfromsomeonefluentinthelanguagee.g.anac-quaintanceoroneoftheauthors.Toextractdatafromanon-Englishpaper,Iwillagainseekhelp.Intheexcludedstudiestable,Iwilllistanyapparentlyeli-giblestudiesfromwhichIamunabletoextractdatabecauseofalanguagebar-rierorbecauseofinadequatereportingandinabilitytoobtainmissinginfor-mationfromtheauthors.AsecondreviewerandIwillindependentlyjudgeeligibilitybyreviewingeachabstractand,ifnecessary,(sectionsof)thefulltext.Appendix3istheformwewillusetoconfirmeligibility.Anydisagreementswillbediscussedbeforemov-ingontodataextractionandassessmentofriskofbias.Athirdreviewerwillbebroughtintoarbitrateifnecessary.DataextractionAsecondreviewerandIwillindependentlyextractdataaboutthepopulation,studydesign,criteriausedtoassessthequalityofthereports,andtheresults.Wewillagaindiscussanydisagreements,bringinginathirdreviewertoarbi-trateifnecessary.Appendix4istheformwewillusetoextractdataaboutpopulationanddesign.Appendix5istheseparatespreadsheetinwhichwewillentertheindividualcriteriaandresultsforeachcriterion.Ihavedeterminedwhatdatatoextractbasedonwhatisreportedinthepoten-tiallyeligiblestudiesofwhichIamaware(Appendix1),varioustools(Vandenbroucke,etal.,2007;TheJoannaBriggsInstitute,2014;Moher,etal.,2015;Munn,etal.,2015),myexpertise,andcommonsense.Appendix6con-tainsdummytablesforallthedatatobeextracted,aswellastheriskofbiasas-sessmentsandanalyses.PopulationAsecondreviewerandIwillextractdataaboutthefollowingvariables,intermsofthesamplesofnewsreports:mediumofpublication;geographicareaofpub-lication;timeperiodofpublication;categoryofinterventionreportedon;andwhethertheoutletiscommercialornot.Intermsofmedium,radioandpod-castswillbeonecategory,inadditiontothecategories:newspapers;magazines;television;andnewswebsites.Theninepredeterminedcategoriesofinterven-tionare:“modern”medicine(aka.“academic”,“conventional”or“Western”medicine);“alternative”medicine(aka.“complementary”,“traditional”or“natu-ral”medicine);screening;surgery;devices;diet;exercise;lifestyle;andsystemsandpolicies.StudydesignForeachstudy,wewillrecord:theinstrumentusedtomeasurequality;thestatedobjectives;thesamplingframe;theselectioncriteriaandtechnique;andanyreportedsubgroupanalyses.

Quality of news media reports about the effects and costs of health interventions: Systematic review protocol 17

Criteria,responseoptionsandresultsFinally,wewillrecordeacheligiblecriterionusedtoassessthequalityofre-ports,aswellastheresponseoptions,inaspreadsheetseparatefromtheformusedtocollectdataaboutthepopulationandstudydesign(Appendix5).Wewillentertheresultsforeachcriterionintothesamespreadsheet.Wheredataareavailableforsubgroupsofnewsreports,wewillrecordtheoverallresultsandtheresultsforeachsubgroup.AssessingriskofbiasInasystematicreviewoftoolsusedtoassessthequalityofprevalencestudiesandotherobservationalresearch,Shamliyanetal.identified46scalesand51checklist,noneofwhichdiscriminatedbetweenreportingversusmethodologi-calquality,orexternalversusinternalvalidity(2010).Throughcitationsearch-es,Iidentifiedseveraltoolsdevelopedafterthereview,ofwhichIhavechosentousethelatest,developedbyMunnetal.(2014),asastartingpoint,basedonfacevalidityandsimplicity.WhileMunnetal.donotrefertothereviewbyShamliyanandcolleagues,theycomparetheirtooltoonedevelopedbyHoyetal.(2012)whodoconsidersaidreview.ThecriticalappraisaltooldevelopedbyMunnetal.includestenitems,withtheresponseoptions“Yes”,“No”,“Unclear”and“Notapplicable”:

1. Wasthesamplerepresentativeofthetargetpopulation?2. Werestudyparticipantsrecruitedinanappropriateway?3. Wasthesamplesizeadequate?4. Werethestudysubjectsandthesettingdescribedindetail?5. Wasthedataanalysisconductedwithsufficientcoverageoftheidenti-

fiedsample?6. Wereobjective,standardcriteriausedforthemeasurementofthecondi-

tion?7. Wastheconditionmeasuredreliably?8. Wasthereappropriatestatisticalanalysis?9. Areallimportantconfoundingfactors/subgroups/differencesidentified

andaccountedfor?10. Weresubpopulationsidentifiedusingobjectivecriteria?

Ofthetenitems,threeassessriskofbiasandarerelevantandapplicabletothisreview:items2,6and7.Item2assessesriskofselectionbias,whileitems6and7measuredetectionbias(aka.Informationbias).FocusingonthesetwotypesofbiasisconsistentwiththeStrengtheningtheReportingofObservationalStudiesinEpidemiology(STROBE)statement(Vandenbroucke,etal.,2007,p.814)andmethodsusedbytheCochraneMethodologyReviewGrouptosystematicallyreviewtheprevalenceofaconditioninapopulationofinformation,asopposedtopeople(Welch,etal.,2010,p.7).AsecondreviewerandIwillindependentlyassessriskofbiasineachstudy,cri-terionbycriterion,giventhatsomecriteriausedtomeasurethequalityofnewsreportsrequiremoresubjectivejudgementthanothers.Theseassessmentswillbeusedtodescribethestrengthsandweaknessesofthestudies.Ifdeemednec-essary,theywillalsobeusedtoconductasensitivityanalysis.Itisunclearhow

Quality of news media reports about the effects and costs of health interventions: Systematic review protocol 18

agreaterriskofbiaswouldimpacttheresultsofstudiestobeincludedinthisreview,soIhavenohypothesisabouttheresultsofsuchananalysis.Intermsofselectionbias,ifthenewsreportswererandomlyorsequentiallysampledfromthetargetpopulation,wewillconsidertheriskofbiasaslow.Ifthereportswerepurposivelysampled,wewillconsidertheriskofbiasashigh.Ifanyothersamplingtechniquewasused,wewillconsidertheriskofbiasasmoderateorhigh.Intermsofdetectionbias,ifacriterionrequiresminimaljudgement(e.g.acrite-rionthatrisksbepresentedinabsolutenumbers),wewillconsidertheriskofbiasaslow.Ifthecriterionrequiressubstantialjudgement(e.g.whetherthereissufficientinformationaboutpotentialharms),tworesearchersassessedthere-ports,andtheywereblindedtothejournalistandpublication,wewillconsidertheriskofbiasasmoderate.Ifthecriterionrequiressubstantialjudgement,andonlyoneresearcherassessedthereportsorresearchersassessingthereportswerenotblindedtothejournalistandpublication,wewillconsidertheriskofbiasashigh.Iftheriskofeitherselectionordetectionbiasishigh,wewillconsidertheover-allriskofbiasashigh.Iftheriskofeitherismoderate,wewillconsidertheriskasmoderate.Appendix6includesadummytablefortheriskofbiasassess-ments.Appendix7istheformthatwewillusetoassessriskofbias.Itincludesaspaceforrecordinganyconflictsofinterests.Anysuchconflictswillbedis-cussedseparatelywheninterpretingtheresultsofthereview.AnalysesCriteriausedtomeasurequalityHowqualityhasbeenmeasuredwillbeassessedbycomparingcriteriatotheIHCKeyConcepts(Box1).Eacheligiblecriterionfromeachstudywillbeen-teredintoaspreadsheet,diagonallyacrossfromthefullconceptlist(Appendix8).AsecondreviewerandIwilljudgewhichconceptsarecapturedbyeachcri-terion,representingthejudgementswithmarksinthespreadsheet.Wewilldis-cussanyagreementsandbringinathirdresearchertoarbitrate,ifnecessary.TheIHCKeyConceptsdifferfromotherlistsofcriteriaformakingcausalinfer-encesintwoways(Chalmers,etal.,2018,pp.30-31).First,thelistisdifferentinthatitisboth:a)developedusingasystematic,transparentanditerativepro-cess;andb)intendedtohelpchildren,thegeneralpublicandhealthpractition-ersmakeinformeddecisions,notjusthealthresearchers.Second,unlikeothertoolssuchaschecklistsandtipsheets,itisnotaninterventioninandofitself,limitedtoaparticularpopulationorcontext,butacomprehensiveframeworkthatcanbeusedformappingskills,developinginterventionsandevaluatingin-terventions.UsingtheIHCKeyConceptlisttoassessthecriteriausedtomeasurequalityofthenewsreportshasseveraladvantages.First,itprovidesaframeworkforgroupingcriteriafromdifferentstudies,soresultscanmoreeasilybesynthe-sised.Second,itwillshedlightonrelevantqualitiesthathavenottypicallybeenmeasuredormeasuredatall.Conversely,itmayrevealconceptscapturedby

Quality of news media reports about the effects and costs of health interventions: Systematic review protocol 19

thecriteriamissingfromtheIHCKeyConceptlist.Finally,usingthelistcanin-formtheinterpretationofresultsfromthisreview,includingwhattheresultssuggestaboutwhatconceptsaremostimportantwhendevelopinginterven-tionsforimprovingreportingontheeffectsofinterventions,orhelpingpeopleassessinformationinsuchreports.AsecondreviewerandIwilljudgewhatcriteriausedindifferentstudiescanbegrouped.Oncemore,wewilldiscussanydisagreementsandbringinathirdre-viewertoarbitrateifnecessary.QualityofnewsreportsAsfarasthedatapermits,wewillmeta-analysetheproportionsofsamplesthateithersatisfiedorfailedtosatisfyrespectivecriteria,i.e.theprevalenceofre-portssatisfyingorfailingtosatisfyagivencriterion.SeedummytablesinAp-pendix6.Wewilldichotomisecategoricaldatawherepossibleandsensible.Furthermore,wemayreframedatathatalreadyisdichotomousassatisfyingornotsatisfyingagivencriterion.Notsatisfyingacriterionwillincludeanythingotherthancompletelysatisfyingit.Inotherwords,ifnewsreports“partially”satisfyacriterionoritisunclearwhetherthereportssatisfyit,wewillconsiderthereportsasnothavingsatis-fiedthecriterion.Ifacriterionhasbeendeemedinapplicabletoreports,wewillexcludethosereportsfromthemeta-analysisforthatcriterion.Inadditiontopotentiallygivinguslargersamplesformeta-analysis,thedichotomisationwillmakeinterpretingtheresultsmorestraightforward.Acrossstudies,Ianticipatereportswillgenerallyfailtosatisfymostcriteria,de-spitedifferencesinpopulationsandmethods.Therefore,Iexpectmeanpropor-tionswith95%confidenceintervalswillbemeaningful.Iwillprepareforestplotsandvisuallyanalysetheextenttowhichtherearemeaningfuldifferencesbetweenpointestimates,andIwillconductaChi-squaretesttoseeiftheheter-ogeneityislargerthanonewouldexpectbychance.Foragivencriterion,ifthevisualinterpretationandChi-squaretestsuggestitisappropriate,Iwillconductameta-analysisusingarandomeffectsmodel.Iwillalsoconductsixsubgroupanalyses,whendataareavailable.VariationinqualityAgain,Idonotexpectalotofvariationinthequalityofreportsacrossstudies.Totheextentthatthereisvariation,Idonotexpectalotofdataforexploringit.Thatsaid,thereareseveralvariablesthatcouldlogicallyexplainvariation,evi-dencesupportingsomeofthosehypotheses,andreasonstotestthem.Table1isanoverviewof:thevariablesforwhichwewillconductsubgroupanalyses;thesubgroups;thehypothesiseddifferences;andtherationalesforthehypothesesandforconductingtheanalyses.Thevariablesare:themediuminwhichareportwaspublished;thetimeperiodinwhichareportwaspub-lished;whetherthepublishingoutletwascommercial;whetherthejournalistwasspecialisedinhealthorscience;andtheincomelevelofthecountryinwhichtheoutletissituated.

Quality of news media reports about the effects and costs of health interventions: Systematic review protocol 20

Ontheonehand,Iwillbecautiousaboutextrapolatingfindingsifthereisnodi-rectevidenceforasubpopulation(e.g.broadcastreports)andthereisreasontobeuncertainabouttheapplicabilityoftheevidencefromothersubpopulations(e.g.printandonlinereports).Ontheotherhand,subgroupanalysesarefre-quentlymisleading,asshownbySunetal.(2014).Toavoidspuriousresults,wewilluseexplicitcriteriaforassessingthecredibil-ityofanydifference,basedonSunandcolleagues’criteria:

• Thedifferenceisofpracticalimportance.• Thedifferenceisbiggerthanwewouldexpectbychance.• Thedifferenceisinthehypothesiseddirection.• Thedifferenceisunlikelytobeexplainedbyotherknownvariables.• Thedifferenceisconsistentacrossstudieswithdifferentmethods.• Thedifferenceisconsistentacrossstudieswithdifferentoutcome

measures.• Thedifferenceisconsistentacrossstudieswithdifferentlevelsofriskof

bias.Foreachexplanatoryvariable,givensufficientdata,wewillconductunivariateregressionanalyseswiththevariable(e.g.medium)astheindependentvaria-ble,andwiththeproportionofsubpopulations(e.g.reportsfromrespectivedecades)thatsatisfythecriterionasthedependentvariable.Wewillalsocon-ductmultipleregressionanalyseswithalloftheexplanatoryvariables.Wewillnoteanywithin-studysubgroupanalyseswheretheresearchershaveusedap-propriatestatisticalmethodsorprovidedthenecessarydataforustodoanap-propriatestatisticalanalysisourselves.Wewillconsiderthoseanalyseswhenmakingjudgementsaboutthecredibilityofsubgroupdifferences.Ihavedecidedagainstconductingasubgroupanalysisforthetypeofreport,i.e.whetherthereportisafeatureorabreakingnewsreport.Featuresaretypicallylongerinwordsandpreparedonlongerdeadlines.Theymightlogicallybehigherqualitythanbreakingnewsreports.However,ananalysisbytypeofre-portwouldlikelybeconfoundedbymedium,financialmodel,andspecialisation.Furthermore,howresearchershavedefinedfeatureversusbreakingnews,andhowtheyhavemeasuredlengthofreportordeadline,maybesodifferentacrossstudiesthatcombiningthedatawouldnotmakesense.

Quality of news media reports about the effects and costs of health interventions: Systematic review protocol 21

Table1:SubgroupanalysesVariable Grouping Hypothesis RationaleandimportanceMedium Broadcastvs.other(print

andonline)Broadcastreportsarelesslikelytosatisfycriteria

Broadcastallowsforfewerwords,aswellaslesstimeforwriting,giventimespentrecordingandediting.Thiswouldlogicallyhaveanegativeimpactonthequalityofreports.Iamawareofempiricalevidencethatsupportsthishypothesis(Walsh-Childers,etal.,2016).Acredibledifferenceinthehypothesiseddirectionwouldimplyconsumersshouldpaylessattentiontobroadcastreports.

Timeperiod Decades(≥2000;1990-1999;1980-1989;etc.)

Theproportionofreportssatisfyingcriteriaislikelytobesimilaracrossdecades

Ontheonehand,newsreportsmayhaveimprovedsincetheadventof“evidence-basedmedicine”(EBM)(Evidence-BasedMedicineWorkingGroup,1992).Ontheother,therearesustained,systemicbarrierstoreliablehealthjournalismsuchaslackoftime(Larsson,etal.,2003;Pettersen,2005),whichEBMdoesnotaddress.Moreover,revenuelossesmayhavelednewsmediatoinvestlessinstrategiesthatmightimprovethequalityoftheirhealthnews,suchastrainingjournalistsandeditors,andhiringandretainingspecialisedhealthandsciencereporters.However,acredibleimprovementforoneormorecriteriawouldbeanimpetusforresearchtoexplainsaidimprovement,andthatresearchcouldinformthedevelopmentofinterventionstoimprovequalityoverall.

Financialmodel Commercialvs.non-commercial

Reportspublishedbycommercialoutletsarelesslikelytosatisfycrite-ria

Thefinancialinterestsofcommercialoutletsmayleadtosensationalism,toattractaudiences(Larsson,etal.,2003;Pettersen,2005;Wallington,etal.,2010).Moreover,companiesthatsellhealthinterventionsmaybeadvertisers,disincentivisingcriticalreporting.Acredibledifferenceinthehypothesiseddirectionwouldimplyconsumersshouldsupportnon-commercialoutletsandpaylessattentiontoreportspublishedbycommercialoutlets.

Specialisation Healthandsciencejour-nalistsvs.other

Reportspreparedbyhealthorsci-encejournalistsaremorelikelytosatisfycriteria

Journalistwhohavespecialisedinhealthorsciencearemorelikelytohavereceivedtrainingincriticalappraisal.Moreover,comparedtootherjournalists,theymayhavebeeninmorecontactwithscientists,andmaybemorefamiliarwiththelanguage,contentandstructureofresearchpapers.Thereisevidencethatjournalistswithmoreadvanceddegreesaremorelikelytousescientificjournalarticlesassources,andthatjournalistswithfeweryearsofexperiencearemorelikelytosayprovidingenter-tainmentisanimportantpriorityforhealthnews(Wallington,etal.,2010).Acredibledifferenceinthehypothesiseddirectionwouldimplyconsumersshouldpaymoreattentiontoreportspreparedbyspecialisedhealthandsciencejournalists.

Countryincomelevel

Low-incomecountriesvs.other(middleandhigh-incomecountries)

Theproportionofreportssatisfyingcriteriaislikelytobesimilaracrosscountrieswithdifferentincomelevels

Whilenewsmediainmiddleandhigh-incomecountriesmayhavemoreresources,journalistsinthosecountriesstillfacesys-temicbarrierstohigh-qualityreporting(Larsson,etal.,2003;Pettersen,2005;Wallington,etal.,2010).Supportforthishypoth-esiswouldsuggestresearchersshoulddevelopinterventionsthatcanimprovethequalityofreportinginlow-incomecountriesandmiddleorhigh-incomecountriesalike.Itwouldalsoimplythoseinterventionsaddressbarriersbesidelackofresources.

Newspapertype Broadsheetvs.tabloid Reportsinbroadsheetnewspapersaremorelikelytosatisfycriteria

Reportingintabloidnewspapersmaybelessdetailedandmoresensationalthanreportinginbroadsheetnewspapers,andIamawareofevidencethatthequalityofreportsabouttheeffectsofhealthinterventionsislowerintabloids(Entwistle&Hancock-Beaulieu,1992;Robinson,etal.,2013).Acredibledifferenceinthehypothesiseddirectionwouldimplyconsumersshouldpaymoreattentiontobroadsheetnewspapers,andlesstotabloids.

Quality of news media reports about the effects and costs of health interventions: Systematic review protocol 22

PilotHavingconductedthesearch,wewillpilotforms,spreadsheetsandtables(Ap-pendixes3through6)onthefirstfiveeligiblestudies.Wewilldiscussanyis-suesandmakeanyrevisionsdeemednecessarybeforecompletingdataextrac-tion,riskofbiasassessmentsandanalysesforremainingstudies.RatingthequalityoftheevidenceIhaveadjustedtheGradingofRecommendationsAssessment,DevelopmentandEvaluation(GRADE)approachtoratingthequalityofevidence(Guyatt,etal.,2008).TheGRADEapproachinvolvesconsideringfivefactorsforloweringthecertaintyofevidence:studylimitations,inconsistencyofresults,indirectnessofevidence,imprecision,andpublicationbias.Ihaveadjustedthese,giventhatGRADEwasdevelopedforresearchabouteffects,notprevalence.Appendix9istheformwewillusetoratethequalityoftheevidence.Thefirstadjustmentisthatwewillnotconsiderpublicationbias.Iamnotawareofanyresearchdocumentingpublicationbiasforprevalenceorincidencestudiesgenerally,norspecificallyinthisarea.Besides,Iexpecttoofewstudiesandsamplesthataretoosmallforittobemeaningfultoassesstheriskofpubli-cationbias.Second,wewillnotconsiderdirectnesssinceitisinapplicable.Iwill,however,becautiousaboutextrapolatingevidenceacrosssubpopulationsspecifiedinTable1(e.g.acrossreportsfromdifferenttimeperiods),ifthereisnoevidenceforoneofthesubpopulationsspecific(e.g.reportsfrombeforeaparticulardec-ade).Thisleavesconsideringriskofbias,imprecisionandinconsistency.Intermsofimprecision,wewillusequartilesasaruleofthumb;iftheconfidenceintervalsubstantiallycrossestwoormorequartiles,wewillconsidertheretobeim-portantimprecision,followingguidancedevelopedbytheCochraneEffectivePracticeandOrganisationofCare(EPOC)group(CochraneEffectivePracticeandOrganisationofCare,2017).GRADEfactorsforincreasingthecertaintyoftheevidencearenotrelevanthere.

Quality of news media reports about the effects and costs of health interventions: Systematic review protocol 23

Ethicalconsiderationsandconflictsofinterest

Thereviewdoesnotinvolveparticularethicalchallenges.IdeclarethatIhavenorelevantconflictsofinterest.

Quality of news media reports about the effects and costs of health interventions: Systematic review protocol 24

References

Ashorkhani,M.etal.,2012.Qualityofhealthnewsdisseminatedintheprintmediaindevelopingcountries:AcasestudyinIran.BMCPublicHealth,12(627).Austvoll-Dahlgren,A.etal.,2017.Measuringabilitytoassessclaimsabouttreatmenteffects:alatenttraitanalysisofitemsfromthe‘ClaimEvaluationTools’databaseusingRaschmodelling.BMJOpen,Volume7,p.e013184.Austvoll-Dahlgren,A.,Nsangi,A.&Semakula,D.,2016.Interventionsandassessmenttoolsaddressingkeyconceptspeopleneedtoknowtoappraiseclaimsabouttreatmenteffects:asystematicmappingreview.SystematicReviews,5(215).Austvoll-Dahlgren,A.etal.,2015.Keyconceptsthatpeopleneedtounderstandtoassessclaimsabouttreatmenteffects.JournalofEvidence-BasedMedicine,Volume8,pp.112-125.Austvoll-Dahlgren,A.etal.,2016.Measuringabilitytoassessclaimsabouttreatmenteffects:thedevelopmentofthe‘ClaimEvaluationTools’.BMJOpen,Volume7,p.e013184.Berkman,N.D.etal.,2011.Lowhealthliteracyandhealthoutcomes:Anupdatedsystematicreview.AnnalsofInternalMedicine,155(2),pp.97-107.Brownlee,S.etal.,2017.Evidenceforoveruseofmedicalservicesaroundtheworld.Lancet,390(10090),pp.156-168.Castell,S.etal.,2014.PublicAttitudestoScience2014,London:IpsosMORI.Chalmers,I.etal.,2018.KeyConceptsforInformedHealthChoices:aframeworkforhelpingpeoplelearnhowtoassesstreatmentclaimsandmakeinformedchoices.BMJEvidence-BasedMedicine,23(1),pp.29-33.Charnock,D.,Shepperd,S.,Needham,G.&Gann,R.,1999.DISCERN:aninstrumentforjudgingthequalityofwrittenconsumerhealthinformationontreatmentchoices.JournalofEpidemiologyandCommunityHealth,Volume53,pp.105-111.Chew,C.&Eysenbach,G.,2010.PandemicsintheAgeofTwitter:ContentAnalysisofTweetsduringthe2009H1N1Outbreak.PLoSONE,5(11),p.e14118.CochraneEffectivePracticeandOrganisationofCare,2017.ReportingtheeffectsofaninterventioninEPOCreviews.s.l.:CochraneEffectivePracticeandOrganisationofCare.Coulter,A.,Entwistle,V.&Gilbert,D.,1999.Sharingdecisionswithpatients:istheinformationgoodenough?.BMJ,Volume318,pp.318-322.

Quality of news media reports about the effects and costs of health interventions: Systematic review protocol 25

Culver,J.D.,Gerr,F.&Frumkin,H.,1997.MedicalinformationontheInternet:Astudyofanelectronicbulletinboard.JournalofGeneralInternalMedicine,12(8),pp.466-470.Entwistle,V.&Hancock-Beaulieu,M.,1992.HealthandmedicalcoverageintheUKnationalpress.PublicUnderstandingofScience,Volume1,pp.367-382.Evidence-BasedMedicineWorkingGroup,1992.Evidence-BasedMedicine:ANewApproachtoTeachingthePracticeofMedicine.JAMA,268(17),pp.2420-2425.Faerber,A.E.&Kreling,D.H.,2013.ContentAnalysisofFalseandMisleadingClaimsinTelevisionAdvertisingforPrescriptionandNonprescriptionDrugs.JournalofGeneralInternalMedicine,29(1),pp.110-118.Frosch,D.L.etal.,2007.CreatingDemandforPrescriptionDrugs:AContentAnalysisofTelevisionDirect-to-ConsumerAdvertising.AnnalsofFamilyMedicine,5(1),pp.6-13.Frosch,D.L.,May,S.G.,Tietbohl,C.&Pagán,J.A.,2011.Livinginthe“landofno”?Consumerperceptionsofhealthylifestyleportrayalsindirect-to-consumeradvertisementsofprescriptiondrugs.SocialScience&Medicine,Volume73,pp.995-1002.Glasziou,P.etal.,2017.Evidenceforunderuseofeffectivemedicalservicesaroundtheworld.Lancet,390(10090),pp.169-177.Glenton,C.,Paulsen,E.J.&Oxman,A.D.,2005.PortalstoWonderland:Healthportalsleadtoconfusinginformationabouttheeffectsofhealthcare.BMCMedicalInformaticsandDecisionMaking,5(7).Groven,K.S.&Braitwaite,J.,2016.Happily-ever-after:Personalnarrativesinweight-losssurgeryadvertising.HealthCareforWomenInternational,37(11),pp.1221-1238.Guyatt,G.H.etal.,2008.GRADE:whatis“qualityofevidence”andwhyisitimportanttoclinicians?.BMJ,Volume336,pp.995-998.Higgins,J.&Green,S.eds.,2011.Definingthescopeofareviewquestion(broadversusnarrow).In:CochraneHandbookforSystematicReviewsofInterventions.s.l.:TheCochraneCollaboration.HLS-EUConsortium,2013.ComparativereportofhealthliteracyineightEUmembersstates,s.l.:s.n.Hoffmann,T.C.&DelMar,C.,2015.Patients’ExpectationsoftheBenefitsandHarmsofTreatments,Screening,andTests:ASystematicReview.JAMAInternalMedicine,175(2),pp.274-286.Hoffmann,T.C.&DelMar,C.,2017.Clinicians’ExpectationsoftheBenefitsandHarmsofTreatments,Screening,andTests:ASystematicReview.JAMAInternalMedicine,177(3),pp.407-419.Hoy,D.etal.,2012.Assessingriskofbiasinprevalencestudies:modificationofanexistingtoolandevidenceofinterrateragreement.JournalofClinicalEpidemiology,Volume65,pp.934-939.

Quality of news media reports about the effects and costs of health interventions: Systematic review protocol 26

Kable,A.K.,Pich,J.&Maslin-Prothero,S.E.,2012.Astructuredapproachtodocumentingasearchstrategyforpublication:A12stepguidelineforauthors.NurseEducationToday,Volume32,pp.878-886.Kutner,M.,Greenberg,E.,Jin,Y.&Paulsen,C.,2006.TheHealthLiteracyofAmerica’sAdults:ResultsFromthe2003NationalAssessmentofAdultLiteracy,Washington,DC:NationalCenterforEducationStatistics.Larsson,A.,Oxman,A.D.,Carling,C.&Herrin,J.,2003.Medicalmessagesinthemedia–barriersandsolutionstoimprovingmedicaljournalism.HealthExpectations,Volume6,pp.323-331.Marcon,A.R.,Murdoch,B.&Caulfield,T.,2017.Fakenewsportrayalsofstemcellsandstemcellresearch.RegenerativeMedicine,12(7),pp.765-775.Moher,D.etal.,2015.Preferredreportingitemsforsystematicreviewandmeta-analysisprotocols(PRISMA-P)2015statement.Systematicreviews,4(1).Moult,B.,Franck,L.S.&Brady,H.,2004.EnsuringQualityInformationforPatients:developmentandpreliminaryvalidationofanewinstrumenttoimprovethequalityofwrittenhealthcareinformation.HealthExpectations,Volume7,pp.165-175.Moynihan,R.etal.,2000.Coveragebythenewsmediaofthebenefitsandrisksofmedications.TheNewEnglandJournalofMedicine,342(22),pp.1645-1650.Munn,Z.etal.,2015.Methodologicalguidanceforsystematicreviewsofobservationalepidemiologicalstudiesreportingprevalenceandcumulativeincidencedata.InternationalJournalofEvidence-BasedHealthcare,Volume13,pp.147-153.Munn,Z.,Moola,S.,Riitano,D.&Lisy,K.,2014.Thedevelopmentofacriticalappraisaltoolforuseinsystematicreviewsaddressingquestionsofprevalence.InternationalJournalofHealthPolicyandManagement,3(3),pp.123-128.Nsangi,A.etal.,2017.EffectsoftheInformedHealthChoicesprimaryschoolinterventionontheabilityofchildreninUgandatoassessthereliabilityofclaimsabouttreatmenteffects:acluster-randomisedcontrolledtrial.Lancet,390(10092),pp.374-388.Optum,2013.Effectivenessofhomeopathyforclinicalconditions:Evaluationoftheevidence,s.l.:Optum.O'Sullivan,J.W.etal.,2018.Overtestingandundertestinginprimarycare:asystematicreviewandmeta-analysis.BMJOpen,Volume8,p.e018557.Oxman,A.D.,Austvoll-Dahlgren,A.,Garrat,A.&Rosenbaum,S.,2017.Understandingofkeyconceptsrelevanttoassessingclaimsabouttreatmenteffects:asurveyofNorwegianadults.IHCWorkingPaper.Oslo:InformedHealthChoices.Oxman,A.D.etal.,1993.AnIndexofScientificQualityforHealthReportsintheLayPress.JournalofClinicalEpidemiology,46(9),pp.987-1001.Pettersen,S.,2005.CriticalThinkinginNorwegianUpperSecondaryBiologyEducation:TheCasesofComplementary-Alternative-MedicineandHealthClaimsintheMedia.Nordina,1(2),pp.61-71.

Quality of news media reports about the effects and costs of health interventions: Systematic review protocol 27

Pettersen,S.,2005.Norwegianhealthjournalists’abilitytoreportonhealthresearch:Aconcerntoscienceeducation?.Nordina,1(1),pp.5-16.Pettersen,S.,2007.HealthClaimsandScientificKnowledge:Astudyofhowstudentsofhealthsciences,theirteachers,andnewspaperjournalistsrelatetohealthclaimsinsociety.Oslo:UniversityofOslo.Robinson,A.,Coutinho,A.,Bryden,A.&McKee,M.,2013.AnalysisofhealthstoriesindailynewspapersintheUK.PublicHealth,Volume127,pp.39-45.Sansgiry,S.,Sharp,W.T.&Sansgiry,S.S.,1999.AccuracyofInformationonPrintedOver-the-CounterDrugAdvertisements.HealthMarketingQuarterly,17(2),pp.7-18.Semakula,D.etal.,2017.EffectsoftheInformedHealthChoicespodcastontheabilityofparentsofprimaryschoolchildreninUgandatoassessclaimsabouttreatmenteffects:arandomisedcontrolledtrial.Lancet,390(10092),pp.389-398.Shamliyan,T.,Kane,R.L.&Dickinson,S.,2010.Asystematicreviewoftoolsusedtoassessthequalityofobservationalstudiesthatexamineincidenceorprevalenceandriskfactorsfordiseases.JournalofClinicalEpidemiology,Volume63,pp.1061-1070.Shoemaker,P.J.&Cohen,A.A.,2006.Part1:TheoryandMethod.In:P.J.Shoemaker&A.A.Cohen,eds.NewsAroundtheWorld.NewYork:Routledge,pp.1-34.Smith,D.E.,Wilson,A.J.&Henry,D.A.,2005.Monitoringthequalityofmedicalnewsreporting:earlyexperiencewithmediadoctor.MJA,Volume183,pp.190-193.Spencer,E.A.,Mahtani,K.R.,Goldacre,B.&Heneghan,C.,2016.Claimsforfertilityinterventions:asystematicassessmentofstatementsonUKfertilitycentrewebsites.BMJOpen,Volume6,p.e013940.Starr,R.R.,2015.TooLittle,TooLate:IneffectiveRegulationofDietarySupplementsintheUnitedStates.AmericanJournalofPublicHealth,105(3),pp.478-485.Stephens,M.,2007.AHistoryofNews.3rdEditioned.NewYork:OxfordUniversityPress.Sun,X.etal.,2014.HowtoUseaSubgroupAnalysis:Users'GuidestotheMedicalLiterature.JAMA,311(4),pp.405-411.TheAcademyofMedicalSciences,2016.TheAcademyofMedicalSciences:MedicalInformationSurvey,London:TheAcademyofMedicalSciences.TheJoannaBriggsInstitute,2014.Thesystematicreviewofprevalenceandincidencedata.Adelaide:TheJoannaBriggsInstitute.UnitedStatesGovernmentAccountabilityOffice,2010.HerbalDietarySupplements:ExamplesofDeceptiveorQuestionableMarketingPracticesandPotentiallyDangerousAdvice,s.l.:UnitedStatesGovernmentAccountabilityOffice.

Quality of news media reports about the effects and costs of health interventions: Systematic review protocol 28

Vandenbroucke,J.P.etal.,2007.StrengtheningtheReportingofObservationalStudiesinEpidemiology(STROBE):Explanationandelaboration.Epidemiology,18(6),pp.805-835.Wallington,S.F.,Blake,K.,Taylor-Clark,K.&Viswanath,K.,2010.AntecedentstoAgendaSettingandFraminginHealthNews:AnExaminationofPriority,Angle,Source,andResourceUsagefromaNationalSurveyofU.S.HealthReportersandEditors.JournalofHealthCommunication,15(1),pp.76-94.Walsh-Childers,K.,Braddock,J.,Rabaza,C.&Schwitzer,G.,2016.OneStepForward,OneStepBack:ChangesinNewsCoverageofMedicalInterventions.HealthCommunication,33(2),pp.174-187.Welch,V.etal.,2010.Howeffectsonhealthequityareassessedinsystematicreviewsofinterventions(Review).CochraneDatabaseofSystematicReviews,Issue12.Wolfe,R.M.,Sharp,L.K.&Lipsky,M.S.,2002.ContentandDesignAttributesofAntivaccinationWebSites.JAMA,287(24),pp.3245-3248.Woloshin,S.,Schwartz,L.M.&Welch,H.G.,2008.Knowyourchances:understandinghealthstatistics.Berkeley:UniversityofCaliforniaPress.Zeraatkar,D.,Obeda,M.,Ginsberg,J.S.&Hirsh,J.,2017.Thedevelopmentandvalidationofaninstrumenttomeasurethequalityofhealthresearchreportsinthelaymedia.BMCPublicHealth,17(343).