RoB2.0: A revised tool to assess risk of bias in ... · Current Cochrane tool for risk of bias in...
Transcript of RoB2.0: A revised tool to assess risk of bias in ... · Current Cochrane tool for risk of bias in...
RoB 2.0:Arevisedtooltoassessriskofbiasinrandomizedtrials
MatthewPageUniversityofBristol,UK
WithspecialthankstoJulianHiggins,Jelena Savović,Asbjørn Hróbjartsson,IsabelleBoutron,BarneyReeves,RoyElbers,JonathanSterne
Overview
• ReminderoftheCochraneriskofbiastoolforrandomizedtrials• Theneedforanewtool• Developmentofthenewtool• Keyinnovationstothetool• Someexcerptsfromthetool• Someunresolvedissues
3
BMJ 2011; 343: d5928
4
Foam dressings for venous leg ulcers
CurrentCochranetoolforriskofbiasinrandomizedtrials
• Sixsourcesofbias(withoptional‘Other’)
• Foreachsource,• Freetexttodescribewhathappened• Judgement:Lowrisk/Unclearrisk/Highriskofbias
• Somesourcesofbiascanberepeatedfordifferentendpoints
CurrentCochranetoolforriskofbiasinrandomizedtrials
• CochraneRoB toolisverywidelyused(Jørgensen 2016)• 100outof100Cochranereviewsfrom2014(100%)• 31outof81non-Cochranereview(38%)
• >2700citationsfromnon-Cochranesources
• Thescientificdebateonriskofbiashascontinued
• Evaluationstudiesofthetool• Userexperience:surveyandfocusgroups(Savovic 2014)• Inter-agreementstudies(e.g.Hartling2009&2013)• Actualuseinreviewsandpublishedcomments(Jørgensen2016)
Someissuesraisedwithexistingtool
• Usedsimplistically
• Usedinconsistently (domainsaddedorremoved)
• Modestagreement rates
• Only5-10%oftrialsinCochranereviewsarescoredasLowriskofbias
• overuseof“unclearrisk”?
• RoB judgementsaredifficult forsomedomains,particularlyincompleteoutcomedataandselectivereporting
• Challengeswithunblindedtrials
• Notwellsuitedtocross-overtrialsorcluster-randomizedtrials
• Notwellsetuptoassessoverallriskofbias
Funding
• Therevisedtoolforrandomizedtrials(RoB2.0)wassupportedbytheUKMedicalResearchCouncil NetworkofHubsforTrialsMethodologyResearch(MR/L004933/1- N61)
RoB2.0:developmentchronology
• RevisionoftheRoBtoolstartedinMay2015• 1st DevelopmentmeetingheldinBristolinAugust2015• 1st ‘workingdraft’ofthetoolcompletedJanuary2016• PilotingphaseFeb– March2016• Revised‘workingdraft’• 2nd DevelopmentmeetingheldinBristolon21-22April2016• Developmentoffurtherguidanceandpiloting• ReleasedforSeoulColloquium
RoB2.0:contributors
• Coregroup:• JulianHiggins,Jelena Savović,MatthewPage,AsbjørnHróbjartsson,Isabelle
Boutron,BarneyReeves,RoyElbers,JonathanSterne• WorkingGroupmembers:
• DougAltman,NatalieBlencowe,MikeCampbell,ChristopherCates,RachelChurchill,MarkCorbett,NickyCullum,FrancoisCurtin,AmyDrahota,SandraEldridge,JonathanEmberson,BrunoGiraudeau,JeremyGrimshaw,ShareaIjaz,SallyHopewell,AsbjørnHróbjartsson,PeterJüni,JamieKirkham,TobyLasserson,TianjingLi,StephenSenn,SashaShepperd,IanShrier,NandiSiegfried,LesleyStewart,PennyWhiting
• And:HenningKeinke Andersen,MikeClarke,JonDeeks,GeraldineMacDonald,RichardMorris,MonaNasser,Nishith Patel,JaniRuotsalainen,HolgerSchünemann, JayneTierney
Keyinnovations
• Result-focussed assessments• Fixed(inclusive)biasdomains,notmodifiable• “Signallingquestions”tofacilitateriskofbiasjudgements• Newresponseoptionsforriskofbias,without‘Unclear’option• Formaloverall riskofbiasjudgement
• Somerethinkingoftheassessment:• Importantdistinctionbetweeneffectsofinterest• Selectivereportingfocusesonreportedresult
RoB1.0 RoB2.0
Randomsequencegeneration(selectionbias) Biasarisingfromtherandomization
processAllocationconcealment(selectionbias)
Blindingofparticipantsandpersonnel(performancebias)
Biasduetodeviationsfromintendedinterventions
Incompleteoutcomedata(attritionbias) Biasduetomissingoutcomedata
Blindingofoutcomeassessment(detectionbias) Biasinmeasurementoftheoutcome
Selectivereporting(reportingbias) Biasinselectionofthereportedresult
Otherbias N/A
N/A Overallbias
Fundingandvestedintereststobeaddressed,butnotwithinthispartofthewiderframeworkWorkinggroupledbyAsbjørn Hróbjartsson and
IsabelleBoutron
Signallingquestionsandjudgements
• Signallingquestionsareintroducedtomakethetooleasier(andmoretransparent)• ‘Yes’,‘Probablyyes’,‘Probablyno’,‘No’,‘Noinformation’
• Riskofbiasjudgementsfollowfromanswerstosignallingquestions(canbeover-ridden)• ‘Lowriskofbias’,‘Someconcerns’,‘Highriskofbias’
• Achangeintheinterpretationofthejudgements,sothata‘Highriskofbias’judgementinonedomainputsthewholestudyathighriskofbias
• Overallriskofbiasjudgementcanthenbecompletedautomatically(canbeover-ridden)
Overallriskofbiasjudgement
Lowriskofbias Thestudyisjudgedtobeatlowriskofbias foralldomainsforthisresult.
Someconcerns Thestudyisjudgedtobeatsomeconcernsinatleastonedomainforthisresult.
Highriskofbias Thestudyisjudgedtobeathighriskofbias inatleastonedomainforthisresult.ORThestudyisjudgedtohavesomeconcernsformultipledomains inawaythatsubstantiallylowersconfidenceintheresult.
riskofbias.info
Someexcerptsfromthetool
Examplealgorithm
4.1Wereoutcomeassessorsawareoftheinterventionreceivedby
studyparticipants?
4.2Wastheassessmentoftheoutcomelikelyto
beinfluencedbyknowledgeofintervention
received?
Highrisk
Someconcerns
Lowrisk
Lowrisk
Y/PY/NI
N/PN
Y/PY
N/PN
NI
Biasarisingfromtherandomizationprocess
Biasarisingfromtherandomizationprocess
• Currenttoolincludestwoseparatedomains:• sequencegeneration• allocationconcealment(bothunder“selectionbias”)
• Botharerelatedtorandomization/allocationofparticipatesintotreatmentarms
• Failuretoimplementeitherprocessadequatelycreatesopportunitiesforeithertheenrolmentintothestudyortheallocationofenrolledparticipantsintogroupstobeinfluencedbyprognosticfactors
• Theendresultisthesame– unbalanced(biased)distributionofpatientsbetweengroups(notafaircomparison,confounding)
Ø ItmakessensetocombineSGandACintoasingledomain
Biasarisingfromtherandomizationprocess
• EvaluationstudiesoftheuseoftheRoB toolinCochraneshowthatreviewersoftenconsiderbaselineimbalanceas“Otherbias”
• Butthisisrelatedtothesuccessofrandomization
Ø Itmakessensetoincludebaselineimbalanceinthesamebiasdomain
• Indicatorsthatrandomizationwasnotperformedadequately:• unusuallylargedifferencesbetweeninterventiongroupsizes;• asubstantialexcessinstatisticallysignificantdifferencesinbaselinecharacteristics;
• asubstantialexcessinclinicallyimportantdifferencesinbaselinecharacteristics
Biasarisingfromtherandomizationprocess
1.1Wastheallocationsequencerandom?1.2Wastheallocationsequenceconcealeduntil
participantswererecruitedandassignedtointerventions?
1.3Weretherebaselineimbalancesthatsuggestaproblemwiththerandomizationprocess?
Randomizationmethods
Additionalevidenceofproblems
Biasduetodeviationsfromintendedinterventions
Theeffectofinterest
• Thecurrenttoolhasverylittletosayaboutsituationsinwhichblindingisnotfeasible• (otherthantoclassifyasnotblindhencehighriskofbias)
• Issuesofperformancebiasverydifferentfordifferenteffectsofinterest,yetpoorlyaddressedincurrentRoB tool
Theeffectofinterest
• Thecurrenttoolhasverylittletosayaboutsituationsinwhichblindingisnotfeasible• (otherthantoclassifyasnotblindhencehighriskofbias)
• Issuesofperformancebiasverydifferentfordifferenteffectsofinterest,yetpoorlyaddressedincurrentRoB tool
• effectofassignmenttointervention• e.g.doesreferraltophysicaltherapyincreasepost-operativemobility?(thequestionofinteresttoahospitalmanageraboutwhethertointroduceareferralprogramme)
• effectofstartingandadheringtointervention• e.g.doesattendingaphysicaltherapyprogramincreasepost-operativemobility?(thequestionofinteresttoanindividualaboutwhethertoattendphysicaltherapy)
Theeffectofinterest
• Wheninterestedineffectofassignment tointervention• Deviationsfromintendedinterventionarenotimportantprovidingthesedeviationsreflectusualpractice
• e.g.itisusualpracticeforsomereferredpatientstonotattendphysicaltherapy,ortocompleteonlysomesessions
• thisdifferstobehaviourthatreflectsexpectationsofadifferencebetweeninterventionandcomparator
• Wheninterestedineffectstartingandadheringtointervention• Deviationssuchaspooradherence,poorimplementationandco-interventionsmayleadtoriskofbias
• Wethereforehavedifferenttoolsforthesetwoeffectsofinterest
Biasduetodeviationsfromintendedinterventions
Effectofassignment tointervention2.1.Wereparticipantsawareoftheirassignedinterventionduring
thetrial?2.2.Werecarersandtrialpersonnelawareofparticipants'assigned
interventionduringthetrial?2.3.IfY/PY/NIto2.1or2.2:Weretheredeviationsfromthe
intendedinterventionbeyondwhatwouldbeexpectedinusualpractice?
2.4.IfY/PYto2.3:Werethesedeviationsfromintendedinterventionunbalancedbetweengroupsand likelytohaveaffectedtheoutcome?
2.5Wereanyparticipantsanalysedinagroupdifferentfromtheonetowhichtheywereassigned?
2.6IfY/PY/NIto2.5: Wastherepotentialforasubstantialimpact(ontheestimatedeffectofintervention)ofanalysingparticipantsinthewronggroup?
Blinding
Deviationsreflectusual
practice?
Firstprincipleof
ITT
Biasduetodeviationsfromintendedinterventions
Effectofstartingandadheringtointervention2.1.Wereparticipantsawareoftheirassignedintervention
duringthetrial?2.2.Werecarersandtrialpersonnelawareofparticipants'
assignedinterventionduringthetrial?2.3.IfY/PY/NIto2.1or2.2:Wereimportantco-interventions
balancedacrossinterventiongroups?2.4.Wastheinterventionimplementedsuccessfully?2.5.Didstudyparticipantsadheretotheassignedintervention
regimen?2.6.IfN/PN/NIto2.3,2.4or2.5:Wasanappropriateanalysis
usedtoestimatetheeffectofstartingandadheringtotheintervention?
Blinding
Specificdeviations
Overcomebyanalysis?
Biasduetomissingoutcomedata
Missingoutcomedata
• Whencompleteoutcomedataforallparticipantsisnotavailableforyourreview• attrition- losstofollowup,withdrawals,othermissingdata• exclusions– someavailabledatanotincludedinreport
• Considerations• howmuchdataismissingfromeachgroup?(includenumbersinyourdescription)
• whyisitmissing?• howwerethedataanalysed?
Source:CochraneTraininghttp://training.cochrane.org/resource/assessing-risk-bias-included-studies
Biasduetomissingoutcomedata
3.1.Wereoutcomedataavailableforall,ornearlyall,participantsrandomized?
3.2.IfN/PN/NIto3.1:Aretheproportionsofmissingoutcomedataandreasonsformissingoutcomedatasimilaracrossinterventiongroups?
3.3.IfN/PN/NIto3.1:Isthereevidencethatresultswererobusttothepresenceofmissingoutcomedata?
Anymissingdata?
Amountandreasons?
Resultsrobust?
Biasinmeasurementoftheoutcome
Biasinmeasurementoftheoutcome
• Systematicdifferencesbetweengroupsinhowoutcomesareassessed
• Someoutcomesaremorepronetobiasthanothers• Patient-reportedoutcome(e.g.pain,qualityoflife)• Observer-reportedinvolvingjudgement(e.g.clinicalexamination)
• Observer-reportednotinvolvingjudgement(e.g.all-causemortality)
Biasinmeasurementoftheoutcome
4.1.Wereoutcomeassessorsawareoftheinterventionreceivedbystudyparticipants?
4.2.IfY/PY/NIto4.1:Wastheassessmentoftheoutcomelikelytobeinfluencedbyknowledgeofinterventionreceived?
Blinding?
Assessmentinfluenced?
Biasinselectionofthereportedresult
Selectivereporting
• Currenttooltakesabroadapproachtoselectivereporting• Anyevidenceofitinthetrialreports?
38
Results
• Selectivenon-reportingbiasestheresultofthemeta-analysiswhichcannotincludethetrialthatomittedtheoutcome;itdoesnotbiasthetrialresult
• Thisissimilartopublicationbias(non-reportingofastudy)
WeincludeonlyselectionofthereportedresultintheRoB 2.0tool
...andconsiderselectivenon-reporting inotherways
Biasinselectionofthereportedresult
Trialresultisbiasedbecauseithasbeenselectedonthebasisoftheresultsfrommultiple:• Outcomemeasurements
• Scales• Definitionsof/criteriaforanevent• Timepoints
• Analyses• Unadjustedvsadjustedmodels• Differentsetsofcovariatesinadjustedmodels• Finalvaluesvschangefrombaselinevsanalysisofcovariance• Continuousscaleconvertedtocategoricaldatawithdifferentcut-
points
Biasinselectionofthereportedresult
Arethereportedoutcomedatalikelytohavebeenselected,onthebasisoftheresults,from...
5.1....multipleoutcomemeasurements(e.g.scales,definitions,timepoints)withintheoutcomedomain?
5.2...multipleanalysesofthedata?
Selectiveoutcomereporting
Selectiveanalysisreporting
Piloting
• RoB 2.0hasundergonemultiplephasesofpiloting• informeddevelopmentandrefinement• moreisalwayswelcome
• Formalstudiesofinter-rateragreementnotyetperformed
• Fullguidanceavailableatriskofbias.info• initialdraft,subjecttominorrefinements
Someunresolvedissues
• Howmanyresultstoassessperstudy?• Howtointegrateintodatacollectionprocess?• Howtopresentassessmentsinareview?
• Implementation• RoB 2.0willneedcarefulconsiderationtomaketheprocessefficientformultipleoutcomes
• DiscussionsinitiatedwithRevMan andCovidence teamatSeoulColloquium
Concludingremarks
• WebelieveRoB 2.0offersconsiderableadvantagesovertheexistingtool
• Onceprogrammedintosoftware,weexpectthetoolwillbeeasytouseandintegrateintotheinterpretationofresults
• WeareextremelygratefultoallthosewhohavecontributedtothedevelopmentofRoB 2.0
• RoB 2.0isavailableatriskofbias.info