Developing and implementing clinical practice guidelines.
10
Quality in Health Care 1995;4:55-64 Effectiveness Bulletin Developing and implementing clinical practice guidelines Jeremy Grimshaw, Nick Freemantle, Sheila Wallace, Ian Russell, Brian Hurwitz, Ian Watt, Andrew Long, Trevor Sheldon This review is based on Effective Health Care, Bulletin No 8 Health Services Research Unit, University of Aberdeen, Aberdeen AB9 2ZD Jeremy Grimshaw, programme director Sheila Wallace, research fellow NHS Centre for Reviews and Dissemination, University of York, York YOl 5DD Nick Freemantle, research fellow Ian Watt, clinical liaison officer Trevor Sheldon, director Department of Health Sciences, University of York, York YOl 5DD Ian Russell, professor of health sciences Department of Primary Health Care, University College, London WClE 6AU Brian Hurwitz, senior lecturer Nuffield Institute for Health, University of Leeds, Leeds LS2 9PL Andrew Long, senior lecturer Correspondence to: Dr J Grimshaw, Health Services Research Unit, University of Aberdeen, Drew Kay Wing, Polwarth Building, Foresterhill, Aberdeen AB9 2ZD The Effective Health Care bulletins have con- centrated on providing systematic reviews of the research evidence on clinical and cost effectiveness to help inform decision makers and clinicians in the NHS. However, providing information by itself is rarely sufficient to stimulate corresponding change in practice. Various implementation strategies can be used to promote the use of research evidence. One approach which has received growing attention and support is the development and implemen- tation of clinical practice guidelines.' 2 These are "systematically developed statements to assist practitioner and patient decisions about appropriate health care for specific clinical circumstances."3 Some previous bulletins argued for the incorporation of the evidence on effectiveness into guidelines,4 5and it has been argued that healthcare commissioners should purchase guidelines or protocols rather than simple procedures.6 This paper examines the evidence on whether practice guidelines can change the behaviour of health professionals and how best to introduce them into clinical practice. The characteristics of high quality guidelines and how purchasers might use guidelines in commissioning are also considered. Evaluating implementation of guidelines Grimshaw and Russell undertook a systematic review of rigorous evaluations of clinical guidelines published between 1976 and 1992.7 In this paper the review has been updated to include another 32 studies either previously unidentified or published up until June 1994. To identify evaluations of clinical guidelines the DHSS-DATA,5 Embase,9 Medline,'0 and SIGLE" databases were searched, all since 1975; published bibliographies of related topics12-26 were searched; and citations in articles were reviewed. Further references were provided by colleagues. Investigations of clinical guidelines were reviewed in depth if they were intended primarily for medical staff, if they used rigorous study designs to evaluate the effectiveness of guidelines in terms of the process of care or outcome for patients, and if they reported sufficient data for statistical analysis. Evaluations were grouped according to a hierarchy of study designs which reflects the reliability with which they are likely to assess implementation of guidelines (table). Generally, randomised controlled trials provide the best evidence of the effectiveness of implementation. However, in behavioural research simple randomised control trials may be more susceptible to a range of biases than in other types of research.27 28 One of the most reliable trial designs for these types of interventions is one in which each participating doctor simultaneously experiences both guidelines for some conditions and the status quo for others in a balanced incomplete block design (for example, Norton and Dempsey29). Although less reliable, randomised controlled trials randomising doctors either individually or in groups and randomised controlled crossover trials (for example, Landgren et al'0) are, with the balanced incomplete block design, considered to provide the most reliable (grade I) evidence. Before and after studies with non-randomised controls, which compare changes in the targeted behaviour with a control group of activities performed by the same doctors but not targeted by guidelines (for example, De Vos Meiring and Wells3") may provide useful though less reliable results. These studies and simple randomised controlled trials in which patients are randomised are considered together as providing grade II evidence. Time series techniques have been used to analyse before and after studies (for example, Kosecoff et al'2). Before and after studies controlled by data from other sites where non-randomised controls are selected in the belief that they may experience changes similar to those of the study populations (for example, Thompson et al") can also be useful. Studies with this design were included if control and study sites were apparently similar; baseline character- istics and performance in control and study sites were similar; and data collection was contemporaneous in study and control sites during both phases of the study. These two designs are considered to provide grade III evidence. Simple uncontrolled before and after studies were not reviewed as secular trends or sudden changes make it impossible to attribute observed changes to the intervention. Do guidelines influence clinical practice? In all, we identified 91 studies which fulfilled the stated criteria, covering a wide range of 55
-
Upload
nguyenkiet -
Category
Documents
-
view
217 -
download
1
Transcript of Developing and implementing clinical practice guidelines.
Quality in Health Care 1995;4:55-64
Effectiveness Bulletin
Developing and implementing clinical practiceguidelines
Jeremy Grimshaw, Nick Freemantle, Sheila Wallace, Ian Russell, Brian Hurwitz, Ian Watt,Andrew Long, Trevor SheldonThis review is based on Effective Health Care, Bulletin No 8
Health ServicesResearch Unit,University ofAberdeen,Aberdeen AB9 2ZDJeremy Grimshaw,programme directorSheila Wallace, researchfellow
NHS Centre forReviews andDissemination,University ofYork,York YOl 5DDNick Freemantle,research fellowIan Watt, clinical liaisonofficerTrevor Sheldon, director
Department ofHealthSciences, University ofYork, York YOl 5DDIan Russell, professor ofhealth sciences
Department ofPrimary Health Care,University College,London WClE 6AUBrian Hurwitz, seniorlecturerNuffield Institute forHealth, University ofLeeds, Leeds LS2 9PLAndrew Long, seniorlecturerCorrespondence to:Dr J Grimshaw,Health Services ResearchUnit, University ofAberdeen, Drew Kay Wing,Polwarth Building,Foresterhill, AberdeenAB9 2ZD
The Effective Health Care bulletins have con-centrated on providing systematic reviews ofthe research evidence on clinical and costeffectiveness to help inform decision makersand clinicians in the NHS. However, providinginformation by itself is rarely sufficient tostimulate corresponding change in practice.Various implementation strategies can be usedto promote the use of research evidence. Oneapproach which has received growing attentionand support is the development and implemen-tation of clinical practice guidelines.' 2 Theseare "systematically developed statements toassist practitioner and patient decisions aboutappropriate health care for specific clinicalcircumstances."3 Some previous bulletinsargued for the incorporation of the evidence oneffectiveness into guidelines,4 5and it has beenargued that healthcare commissioners shouldpurchase guidelines or protocols rather thansimple procedures.6This paper examines the evidence on
whether practice guidelines can change thebehaviour of health professionals and how bestto introduce them into clinical practice. Thecharacteristics of high quality guidelines andhow purchasers might use guidelines incommissioning are also considered.
Evaluating implementation of guidelinesGrimshaw and Russell undertook a systematicreview of rigorous evaluations of clinicalguidelines published between 1976 and 1992.7In this paper the review has been updated toinclude another 32 studies either previouslyunidentified or published up until June 1994.To identify evaluations of clinical guidelinesthe DHSS-DATA,5 Embase,9 Medline,'0 andSIGLE" databases were searched, all since1975; published bibliographies of relatedtopics12-26 were searched; and citations inarticles were reviewed. Further references wereprovided by colleagues. Investigations ofclinical guidelines were reviewed in depth ifthey were intended primarily for medical staff,if they used rigorous study designs to evaluatethe effectiveness of guidelines in terms of theprocess of care or outcome for patients, and ifthey reported sufficient data for statisticalanalysis.
Evaluations were grouped according to ahierarchy of study designs which reflects thereliability with which they are likely to
assess implementation of guidelines (table).Generally, randomised controlled trialsprovide the best evidence of the effectivenessof implementation. However, in behaviouralresearch simple randomised control trials maybe more susceptible to a range of biases thanin other types of research.27 28 One of the mostreliable trial designs for these types ofinterventions is one in which each participatingdoctor simultaneously experiences bothguidelines for some conditions and the statusquo for others in a balanced incomplete blockdesign (for example, Norton and Dempsey29).Although less reliable, randomised controlledtrials randomising doctors either individuallyor in groups and randomised controlledcrossover trials (for example, Landgren et al'0)are, with the balanced incomplete blockdesign, considered to provide the most reliable(grade I) evidence. Before and after studieswith non-randomised controls, which comparechanges in the targeted behaviour with acontrol group of activities performed by thesame doctors but not targeted by guidelines(for example, De Vos Meiring and Wells3") mayprovide useful though less reliable results.These studies and simple randomisedcontrolled trials in which patients arerandomised are considered together asproviding grade II evidence. Time seriestechniques have been used to analyse beforeand after studies (for example, Kosecoff etal'2). Before and after studies controlled bydata from other sites where non-randomisedcontrols are selected in the belief that they mayexperience changes similar to those of thestudy populations (for example, Thompsonet al") can also be useful. Studies with thisdesign were included if control and study siteswere apparently similar; baseline character-istics and performance in control and studysites were similar; and data collection wascontemporaneous in study and control sitesduring both phases of the study. These twodesigns are considered to provide grade IIIevidence. Simple uncontrolled before and afterstudies were not reviewed as secular trends orsudden changes make it impossible to attributeobserved changes to the intervention.
Do guidelines influence clinical practice?In all, we identified 91 studies which fulfilledthe stated criteria, covering a wide range of
55
Grimshaw, Freemantle, Wallace, Russell, Hurwitz, Watt, Long, Sheldon
Ezidence of effectiveness ofguidelines from igorous studies of introducing guidelines
A uthlos Setting Clinical area End user Interventions to promote use ofinvolvement in guidelinesguidelinedevelopment?
Design of trial Effect on Effect onor study process outcome
Studies of clinical care
Grade I evidence: Balanced incomplete block designs, randomized controlled trials randonmising doctors, and crossover tooaksMcDonald1976i4Sanazaro andWorth I9785
IHopkins ct al1980A"
1Linn 198X0
US ambulatory Various medical Nocare conditionsUS hospital care I Paediatric, No
2 surgical, and4 medicalconditions
US emergency Hypotensive Noroom care shock
US hospital care Management of Nobums
McDonald et al US ambulatory Various medical No1980 i care conditions
Sommers et al US ambulatory Unexplained1984"' care anaemia
Norton andDempsey 1985
Palmer et al1985'
Putnam andCurry 19854'
(i) Yes(ii) No
Canadian general Cystitis and Yespractice vaginitis
US ambulatory 4 Medical and Yescare 4 paediatric
conditionsCanadian general 5 Medical (i) Yespractice conditions (ii) No
Winickoff et al US ambulatory Hypertension Nol98542 care
McAlister et al Canadian general Hypertension No1986 ' practice
Wirtschafter et al19864-
Vinicor ct al198745
US community Neonatalhospitals respiratory
distresssyndrome
No
US ambulatory Diabetes mellitus Nocare
Putnam and Canadian general Hypertension (i) YesCurry 1989" practice (ii) No
Mazzuca ct al US ambulatory Diabetes mellitus No1 9904 care
Lomas ct al19914S
Canadian Caesareanobstetric care section
No
Margolis et al Israeli paediatnic 3 Paediatnrc No199249 care conditionsNorth of England UK general 5 Paediatric (i) YesStudy of practice conditions (ii) NoStandards andPerformance inGeneral Practice1992'0Emslie et al UK general Infertilitv Yes19935' practice
Jones et al 199352 UK general Dyspepsia Yespractice
Soumerai et al US hospital care Blood transfusion No19935
Local guidelines implemented by Crossovercomputer generated remindersNational guidelines approved by RCT-Drlocal staff and placed in patientsnotes
Local guidelines distributed by RCT-Drbrief educational programme andimplemented by general remindersNational guidelines distributed by RCT-Drspecific educational programmeand implemented by generalremindersLocal guidelines implemented by Crossovercomputer generated reminderssupported by bibliographiccitations on requestGuidelines implemented by RCT-Draggregated feedback:(i) developed by end users(ii) developed without end usersGuidelines implemented by BIBaggregated feedback on baselinecomplianceGuidelines implemented with BIBaggregated feedback on baselinecomplianceGuidelines distributed by mail and RCT-Drpersonal educational package, andimplemented with aggregatedfeedback on baseline compliance:(i) developed by end users for 2conditions(ii) developed without end users for2 conditionsNational guidelines modified RCT-Drlocally and implemented withpatient specific feedback outwiththe consultation and physicianspecific feedbackProvincial guidelines distributed by RCT-Drmail and implemented withcomputer generated patientspecific feedbackLocal guidelines: RCT-Dr(i) distributed by specificeducational programme(ii) (i) plus implemented withguidelines embedded in papermedical recordLocal guidelines: RCT-Dr
(i) distributed by individualpatient education(ii) distributed by intensivephysician education andimplemented by aggregatedfeedback, hotline to diabeticspecialists, and computergenerated reminder(iii) (i) + (ii)(i) guidelines developed with end RCT-Drusers(ii) guidelines developed withoutend users, distributed by targetedmailingLocal guidelines distributed by RCT-Drseminar and implemented by:(i) computer generated reminder(ii) (i) + clinical materials(iii) (ii) + diabetes patienteducation serviceProvincial consensus development RCT-Drconference guidelines distributedby mailing to targeted cliniciansand implemented by:(i) aggregated feedback(ii) local opinion leaderLocal guidelines implemented by PBIBcomputerised protocolGuidelines implemented by BIBfeedback on baseline compliance:(i) guidelines developed by endusers(ii) guidelines developed withoutend usersGuidelines distributed by mail and RCT-Drimplemented with guidelinesembedded in paper medical recordGuidelines distributed by targeted RCT-Drmailing and implemented by ageneral reminder of the guidelinesLocal guidelines distributed by RCT-Drlecture, printed materials, andeducational outreach visits toindividual doctors
+++
++±+ ++±
+ ++
(i)(ii) ++
++
(i) ++++(ii) +++
+ 0
(i) ()(ii) +
(i) +(ii) +(iii) ++
(i) ()(ii) 0
(i) +(ii) +
(iii) +
(i) 0(ii) ++
(i+
(i) +(ii) 0
(i) ++++(ii) 0
++
56
Developing and implementing clinical practice guidelines
Authors Setting Clinical area End user Interventions to promote use of Design of trial Effect on Effect oninvolvement in guidelines or study process outcomeguidelinedevelopment?
Anderson et al US hospital care Prevention of No NIH consensus development RCT-Dr (i) +++199454 deep vein
thrombosis
McDonald1976"5
Coe et al 197756
Restuccia 19825
Rogers et al198258
Barnett et al198359
Thomas et al198360Brownbridge et al198661
Weingarten et al199062
Barnett et al197863
Kosecoff et al198732
Lomas et al198964
Durand-Zaleskiet al 199265
Sherman et al199266
conference national guidelines:(i) distributed by continuingmedical education(ii) distributed by continuingmedical education andimplemented by quality assuranceprogramme
(ii) +++
Grade II evidence: randomised controlled trial randomising patients, and before and after studies controlled by activity not targeted by guidelines
US ambulatory Diabetes and No Local guidelines implemented by RCT-Ptcare other medical computer generated reminders
conditionsUS ambulatory Hypertension Yes Guidelines implemented by RCT-Ptcare computer generated remindersUS hospital care General medical No National guidelines implemented RCT-Pt
conditions by nurse coordinator providing:(i) direct feedback to attending
physician(ii) feedback to physician adviser(iii) feedback at nurse's discretion
US ambulatory Hypertension, Yes Local guidelines implemented by RCT-Ptcare obesity, and renal computer generated reminders
diseaseUS ambulatory Hypertension No Local guidelines implemented by RCT-Ptcare computer generated patient
specific feedbackUS ambulatory Diabetes No Local guidelines implemented by RCT-Ptcare computer generated remindersUK general Hypertension Yes Local guidelines discussed with CBA-Apractice participants and implemented by
paper based or computerisedprotocol
US hospital care Chest pain No Local guidelines implemented with RCT-Ptpatient specific reminder
Grade III evidence: before and after studies controlled by site and time series analyses
US ambulatory Streptococcal Yes Local guidelines "determined" by TSAcare sore throat clinic staff and implemented by
patient specific feedbackUS hospital care Breast cancer, No NIH consensus development TSA
caesarean conference guidelines distributedsection, coronary by publication in professionalartery bypass journals, no further attempt atgrafting implementation
Canadian Caesarean No Provincial consensus development TSAobstetric care section conference guidelines distributed
by publication in professionaljournals and mailing to targetedclinicians, no further attempt atimplementation
French hospital Hypovolaemia No National consensus development TSAcare conference guidelines distributed
by specific educational meetingsand implemented with monthlyfeedback and discussion ofcompliance
US hospital care Localised No NIH consensus development TSAprostatic conference guidelines distributedcarcinoma by publication in professional
journals and mailing to targetedclinicians, no further attempt atimplementation
Studies ofpreventive care
Grade I evidence: Balanced incomplete block designs, randomised controlled trials randomising doctors, and crossover trials
Cohen et al US ambulatory 8 Preventive No Local guidelines distributed by RCT-Dr198267 care tasks intensive educational intervention
and placed in patients' notesMcDonald et al US ambulatory 9 Preventive No Local guidelines implemented by RCT-Dr1984, 199266 66 care tasks and 6 computer generated reminders
laboratory testsWinickoff et al US community Colorectal cancer Yes Local guidelines distributed by Crossover198470 health plan screening specific educational programme
and implemented by comparativefeedback of individual doctorperformance
Cohen et al US ambulatory 13 Preventive No Local guidelines distributed by BIB19857' care tasks mail, residents received credit at
university book shop after readingguidelines
Tierney et al US ambulatory 11 Preventive No Local guidelines distributed by BIB198672 care tasks internal mail and implemented by:
(i) computer generated reminderwithin consultation(ii) patient specific computergenerated feedback
Cheney and US ambulatory 12 Preventive No National guidelines implemented RCT-DrRamsdell 198713 care tasks by age-sex specific checklist placed
in patient's notesCohen et al 1987, US ambulatory Smoking No National guidelines distributed by RCT-Dr198974 75 care cessation specific educational programme
and implemented by:(i) patient specific reminders(ii) nicotine gum(iii) {i -- {i
0
(i) ++(ii) 0(iii) ++
+ ++
++++ ++
++
++
0
0 +++++
0++
0
++ ++
++ _
"Modest"improve-ment incompliance(i) ++(ii) +
++ _
(i) ++++(ii) ++++(iii) +++++
57
Grimshaw, Freemantle, Wallace, Russell, Hurwitz, Watt, Long, Sheldon
Authors Setting Clinical area End user Interventions to promote use of Design of trial Effect on Effect oninvolvement in guidelines or studio process outcomeguidelinedevelopment?
Wilson et al Canadian family Smoking No National guidelines: RCT-Dr (i) +++++ (i) 019887 practice cessation
Cummings et al US ambulatory Smoking19897' care cessation
McPhee ct al US ambulatory 7 Preventive198978 care tasks
Turner et a!19908J
McPhee et al19918/)Ornstein et al199181
Cowan et al199282
Dietrich ct al199283
US ambulatory 6 Preventivecare tasks
US familypractice
US ambulatorycare
US ambulatorycare
US ambulatorycare
Headrick et al US ambulatory19928' care
11 Preventivetasks
5 Preventivetasks
No
No
No
No
No
7 Preventive care Notasks
10 Preventive Notasks
Cholesterol
Litzelman et al US ambulaton, 3 Preventive199385 care tasks
Mayefsky and US ambulatoryFoye 199386 care
No
No
Well child care No
(i) implemented by nicotine gum(ii) distributed by specificeducational intervention andimplemented with nicotine gumNational guidelines distributed by RCT-Drspecific educational programmeand implemented with reminders
National guidelines implemented RCT-Drby:
(i) audit and feedback(ii) cancer screening reminders(iii) patient education
National guidelines implemented RCT-Drby computer generated reminder(control group). In addition studygroup patients received reminders
National guidelines implemented RCT-Drby computer generated reminders
National guidelines distributed by RCT-Drintensive educational interventionand implemented by:(i) physician computer generated
reminder(ii) patient reminder(iii..) (i + (ii
National guidelines placed in RCT-Drpatients' notes
National guidelines:(i) distributed by specific
educational intervention(ii) implemented by practicefacilitator(...i) (i) + (ii)
RCT-Dr
National guidelines distributed by RCT-Drlecture and implemented by:(i) guidelines placed in notes(ii) patient specific computergenerated prompts
National guidelines modified RCT-Drlocally and implemented byrequiring physicians to respond tocomputer generated reminders
National guidelines implemented RCT-Drby individual physician feedback
\(i + I., +
(ii) ..... (ii) +
++ +
(i) +(ii) ++(iii) ++
++
++
(i) +(ii) +
(iii) +
+_
(i) ++(ii) ++(iii) ++
(i) +(ii) +
+_
++ _
Grade II evidence: randonmised controlled trial randoinising patients, and before and after studies controlled by activity not targeted by guidelines
US hospital care Antenatal care
Rodnev ct a! US family1 9838' practice
2 Adultimmunizations
McDowell ct al Canadian family Influenza198688 practice vaccination
Prislin et al198684
Becker et al198990
Chambers et al19899'
McDowell et al198992
US familypractice
2 Preventivetasks
US ambulatory 9 Preventivecare tasks
US family Mammographymedicine
Canadian family Blood pressurepractice screening
McDowell et al Canadian family Cervical198983 practice screening
No
No
No
No
No
National guidelines discussed RCT-Ptlocally and implemented bycomputer generated reminders
Local guidelines distributed by CBA-Aeducational programme andimplemented by redesign ofmedical record to encourage twoadult immunisations
National guidelines implemented RCT-Ptby:
(i) computer generated reminderto the doctor(ii) telephone reminder to thepatient(iii) reminder letter to patientLocal guidelines distributed by RCT-Ptspecific educational conference andimplemented by flowsheet placedin patients' notes
National guidelines implemented RCT-Ptby:(i) reminder to physician(ii) reminders to physician andpatient
National guidelines implemented RCT-Ptby computer generated reminders
National guidelines implemented RCT-Ptby:
(i) computer generated reminderto the doctor(ii) telephone reminder to thepatient(iii) reminder letter to patient
No
No
No National guidelines implementedby:
(i) computer generated reminderto the doctor(ii) telephone reminder to thepatient(iii) reminder letter to patient
RCT-Pt
Morgan ct al1 97828
++
++
(i) +±(ii) +++(iii) +++
(i) +(ii) ++
(i) +(ii) +(iii) ++
(i) 0(ii) +++(iii) +++
58
Developing and implementing clinical practice guidelines
Authors Setting Clinical area End user Interventions to promote use of Design of trial Effect on Effect oninvolvement in guidelines or study process outcomeguidelinedevelopment?
Rosser et al Canadian family Smoking No National guidelines implemented RCT-Pt199194 practice cessation by:
(i) computer generated reminderto the doctor(ii) telephone reminder to thepatient(iii) reminder letter to patient
Lilford et al UK hospital care Antenatal care Yes Local guidelines implemented by: RCT-Pt19929' (i) structured paper record
(ii) interactive computerisedquestionnaire
Rosser et al Canadian family Tetanus No National guidelines implemented RCT-Pt199296 practice vaccination by:
(i) computer generated reminderto the doctor(ii) telephone reminder to thepatient(iii) reminder letter to patient
Grade III evidence: before and after studies controlled by site and time series analyses
Thompson et al US prepaid Investigations in Yes Local guidelines distributed by CBA-Dr19833 health plan "routine" intensive educational programme
physical and implemented with feedback onexaminations performance
Robie 198897 US ambulatory 6 Preventive No National guidelines distributed by CBA-Drcare tasks lecture implemented by chart
remindersSchreiner et al US ambulatory 4 Preventive No National guidelines implemented CBA-Dr198898 care tasks by patient specific remindersNattinger et al US ambulatory Mammography No National guidelines implemented CBA-Dr198999 care by:
(i) audit and feedback(ii) visit-based intervention(including patient reminder andcompleted request form)
Tape and US ambulatory 8 Preventive No National guidelines distributed by CBA-DrCampbell care tasks continuing medical education and1993100 implemented by flowsheet in paper
record (control group). In addition,study group received computergenerated reminders.
(i) +++(ii) +++++(iii) ++++
(i) +(ii) +
(i) ++(ii) ++(iii) +++
++
(i) ++(ii) ++
Studies ofprescribing, laboratory and radiological investigationsGrade I evidence: balanced incomplete block designs, randomized controlled trials randomising doctors, and crossover trials
Marton et al US ambulatory Biochemistry and No Local guidelines: RCT-Dr (i) 01985101 care drug monitoring (i) distributed by educational (ii) 0
investigations materials (iii) ++++(ii) implemented by individualphysician feedback(iii) (i) + (ii)
Chassin and US hospital care Pelvimetry in Yes Local guidelines distributed by RCT-Dr +++McCue 198610'2 pregnancy educational meetings and mailed
educational materialsLandgren et al Australian Antibiotic No Local guidelines distributed by Crossover +++198830 hospital care prophylaxis in anti-advertising campaign and
surgery implemented with aggregatedfeedback of baseline compliance
Avorn et al US residential Psychoactive No Local guidelines distributed by RCT-Dr ++++1992'°3 care drugs lectures to non-medical staff,
educational materials to all doctors,and educational outreach visits tohigh prescribing doctors
Bearcroft et al UK general Chest x ray No Local guidelines distributed by RCT-Dr +1994'°4 practice examination postOakeshott et al UK general 4 x ray No National guidelines distributed by RCT-Dr ....1994105 practice investigations local department
Grade II evidence: randomised controlled trial randomising patients, and before and after studies controlled by activity not targeted by guidelines
Eisenberg et al US hospital care Biochemistry No Local guidelines implemented by CBA-A 01977106 investigations physician specific feedback about
over utilisationDe Vos Meiring UK general 9 Radiological No Local guidelines distributed by CBA-A ...
and Wells 19903' practice investigations mailing to targeted clinicians, nofurther attempt at implementation
Grade III evidence: before and after studies controlled by site and time series analyses
Brook and US ambulatory Injectable No State guidelines distributed by TSA ....
Williams 1976107 care antibiotic targeted mailing and practice visitsLohr and Brook prescribing and implemented with financial1980108 incentives (payment was denied
unless care complied withguidelines)
Wong et al US hospital care Biochemistry Yes Local guidelines: TSA (i) 01983'°" investigations (i) distributed by mailed (ii) +++++
educational materials(ii) (i) and implemented by changein request form
Fowkes et al UK accident and Skull x ray No National guidelines distributed by TSA ....
1984"1 emergency care examinations for specific educational programme,patients with implemented by structured headhead injuries injury card
59
G0ilGnshaw, Freemlantle, Wallace, Ru3ssell, H1luitz, Watt, Long, Sheldon
Au/thors SettiitO (hCli/ 0rca iHtd user I~Itelveltiiols to P)r(Ioi(t ltys of Dcsiigii of toio/ hI/cct on fPect OttI~llvolve}1lent bi gllidelbies or1 tStlICIN Proe'( ss oultcol)167(111,idelbledcvc'/opuiic'iti
Novich ct a!l US hospital care Biochemistrs No Local guidelines implemented by: TSA (i) ++++1985 '. investigations (i) requirement for general (it) ±++++
justification for test(ii) requirement for specificjustification for test
Fosskcs at i/ UK hospital care Biochemistrs and No Local guidelines distributed bs TSA ++++1 98() haematologs specific educational meeting and
ins estigations implemented by swseeklv feedbackFos kes et al UK hospital care PreoperatiC No National guidelines implemented CBA-Dr (i) ++1986' chest X rav by: ii ++
examinations (i) utilisation reviews committee iii +(ii) feedback on individual (is ++compliance(iii) structured chest s ras requestform(iv) review of requests byradiographer
Ray ct all 1 861 US outpatient Diazepam No State guidelines distributed by C BA-Dr ++practice prescribing educational outreach visit
Asorn et a/ US hospital care Dosage of Yes Local guidelines distributed by 'ISA .+++1 988 intravenous lecture and printed materials, in
antibiotics advertisements and posters,implemented through a structuredordering form
Barcford and UK hospital care Haematological No Local guidelines distributed by TSA +++Hasling 1990' tests specific educational programme
and implemented by aggregatedfeedback and cancellation ofinappropriate expensive tests
Clarke and UK accident and Skull X ras No Local guidelines distributed by TSA +++Adams 1990'' emergency care requests in specific educational programme,
patients 55 ith implemented by general remindershead injuries
Escritt et il/ US obstetric care Prophylactic No Local guidelines approved bs TSA +++++199(o0> antibiotics for senior medical staff, distributed by
complicated pamphlet and departmentalcaesarean section meetings, and implemented
through a structured ordering formRaisch eit / US health Anti-ulcer No Local guidelines distributed bs CBA-Dr 01990l"'l maintenance treatment educational outreach visit
organisationGama et 0il UK hospital care Cardiac enzymes No Local guidelines distributed by TSA ++++1992"'' specific educational programmeSoumeraetcici! US hospital care Antibiotics No Local guidelines implemented with TSA +
19931 'a structured ordering form
Designs of trial or study: BIB balanced incomplete block design; PBIB partially balanced incomplete block design; RCT-Dr trial randomized by individual doctor,team, unit, or hospital; RCT-Pt trial randomised by patient; CBA-A before and after studs controlled by untargeted activity; CBA-Dr before and after studs controlledby other doctors; TSA time series analysis.Effect sizes (average effect size of significant results): - not measured; 0 no significant effect; + absolute effect between 0-9" ,; ++ absolute effect between 10 19",;+++ absolute effect between 20-29"/.; ++++ absolute effect between 30-39"/0;+++++ absolute effect >39"'0.
clinical settings and tasks.'8 "' They comprised35 studies of clinical care, 34 of preventivecare, and 22 of prescribing or the use ofradiological or laboratory investigations. Only14 studies were based in the United Kingdom,including four studies of clinical care (commonpaediatric conditions,' infertility managementand referral,'t dyspepsia, ' and hyper-tension't), one study of preventive care(antenatal care"), and nine studies ofinvestigations (of which six were radio-logical).3' 11t 1(' 110 12 113It 1itOf the 87 studies which examined effects on
the process of care, as measured by adherenceto recommendations of practice guidelines, 81reported significant improvements. Twelve ofthe 17 studies which assessed patient outcomereported significant improvements. All 14studies based in the United Kingdom notedsignificant improvements in compliance withguidelines; the only United Kingdom study tomeasure patient outcome also reportedsignificant improvement.>" The evidence fromthose studies considered to provide the mostreliable evidence confirmed these findings: 43out of 44 grade I studies reported significantchanges in process and eight out of 11 showedsignificant changes in outcome. The evidencefrom these rigorous evaluations strongly
suggests that properly developed guidelinescan change clinical practice and may lead tochanges in patient outcome.
Introducing guidelines into practiceAlthough guidelines can be used to help tochange clinical practice, their adoption and useis not automatic and will depend to a greatextent on how they are developed andimplemented. The behavioural factors whichinfluence adherence with guidelines are verycomplex. For example, it is often assumed thatguidelines developed by the clinicians who willultimately use them (end users) improves theirimplementation, owing partly to a perceptionof increased ownership. However only two4i 50of the four studies identified3 ""16A supportedthis. Guidelines produced locally by pro-fessional end users may at times be seen as lesscredible than those produced by locallyrespected clinicians (opinion leaders) ornational experts."3 Some interventions basedon the passive receipt of information (forexample, publication in professional journalsand mailing to relevant groups) influenceprofessionals' awareness32 and knowledge ofguidelines.64 Three studies in the UnitedKingdom of local general practitioners'guidelines for radiological investigations found
60
Developing and implementing clinical practice guidelines
improved compliance after targeted mailingwithout any supporting implementationstrategy.31 104 105 However, these approachesare usually insufficient to change professionalbehaviour.
Educational interventions, requiring moreactive participation by professionals (includingtargeted seminars, educational outreach visits,and the involvement of opinion leaders) aremore likely to lead to changes in behaviour.There is some evidence of the effectiveness ofeducational outreach visits by trained staffmeeting professionals in their practice settingsin influencing prescribing behaviour (forexample, Avorn et al'03) and of the role ofopinion leaders (professionals identified bytheir colleagues as influential).48 Because theseinterventions require the investment ofvaluable resources rigorous evaluation of theircost effectiveness is important before theirwidespread use.
Implementation strategies are more likely tobe effective when they operate directly on theconsultation between the professional and thepatient. Such strategies include restructuringmedical records (for example, Emslie et al,5'Cheney and Ramsdell,73 Rodney et al"),patient specific reminders during theconsultation (for example, McDonald eta168 69) and patient mediated interventions (forexample, McPhee et al78) (whose aim is toaffect professional practice through informingpatients). Strategies operating outside theconsultation that have been rigorouslyevaluated include patient specific feedback (forexample, Tierney et al'2) and aggregatedfeedback on compliance with guidelines (forexample, Durand-Zaleski et al65), financialincentives,'07 108 explicit marketing,30 andprofessional peer review."'3
Several studies compared different edu-cational and implementation strategies(table).44 45 47 48 54 57 72 74 76 78 81 83 84 88 90 92-9699 101 109 111 113 This research suggests thateducational interventions requiring activeprofessional participation and implementationstrategies that are closely related to clinicaldecision making are more likely to lead tosuccessful implementation. In other words,implementation strategies which are nearer theend user and integrated into the process ofhealthcare delivery are more likely to beeffective. However, insufficient evidence existsto reach firm conclusions about the relativeeffectiveness of different educational andimplementation strategies in differentcontexts.
Desirable attributes of clinical practiceguidelinesStrong evidence exists that well implementedclinical guidelines can be used to change theprocess and outcome of care. However, this isonly potentially worthwhile if the changesresult in improved quality of care and efficientuse of resources. Guidelines are referred to asvalid if, when followed, they lead toimprovement in patient outcome at acceptablecosts. Validity depends on how well evidenceis identified, synthesised, and incorporated
into the guideline and, therefore, on how andby whom the guideline is developed.'22
AVOIDING BIAS
Guidelines are commonly developed by"expert" panels uninformed by the results offormal systematic literature reviews of theresearch evidence. This approach relies tooheavily on panel members' opinions andknowledge of published work. Unfortunately,published expert recommendations often lagbehind available evidence'23 and may reflectindividual enthusiasms rather than theknowledge base. Such bias may be overcomeif those developing guidelines adopt asystematic approach to identifying andsynthesising evidence.'24 Guidelines based onreviews that identify, synthesis, and interpretevidence systematically are, therefore, morelikely to be valid. Since local groups may lackthe resource and skills needed to developevidence based guidelines'25 care must betaken to ensure that any increased acceptabilityowing to local development of guidelines is notachieved at the expense of their potential toimprove patient outcomes.Many methods can be used to translate
evidence into practice recommendations,including peer groups, nominal groups, Delphitechniques, and consensus conferences.'26-7 Allhave potential biases and there is little evidenceon their relative merits.
CONSIDERING COST EFFECTIVENESS
Practice guidelines should not be solelyconcerned with clinical effectiveness butshould also pay regard to the costs oftreatments if they are to maximiseimprovements in health status.6 128-130 Theyshould explicitly take into account the costs ofinterventions so that the limited resourcesavailable are used efficiently; guidelines thatignore the issue of cost effectiveness mightrecommend practices resulting in largeincreases in cost for little correspondingimprovement in health. Unfortunately,development of guidelines has largely ignoredthe issue of costs. Since costs of treatmentsmay vary across sites the local development ofguidelines will need to consider local factorswhich may influence cost effectiveness.
INDICATING THE EVIDENCE BASE
Guidelines should clearly indicate the basis ofeach recommendation and the degree to whichit is supported by good research evidence. Thetarget patient population and circumstancesunder which the recommendations apply alsoshould be clearly stated. Clarity of definitions,language, and format is also essential.Unfortunately, few published guidelines giveenough details about development for theirvalidity to be confidently assessed.'3' Thosedeveloping guidelines should provide enoughinformation to allow potential users to make aninformed judgment about validity andrelevance to specific circumstances.'32 A guidefor structured guideline abstracts, encouragingthe publication of details of development isnow available.'33
61
Grimshaw, Freemantle, Wallace, Russell, Hurwitz, Watt, Long, Sheldon
MAINTAINING VALIDITY
Those developing guidelines should specifyhow their guidelines should be monitored toidentify major changes in knowledge and howthe guidelines should be routinely reviewed toincorporate new knowledge. Guidelines shouldalso be piloted in several sites to ensure theirapplicability and relevance, and the experienceof patients as well as professional users shouldbe taken into account.
Guidelines and commissioningThe evidence discussed in the previoussections shows that properly developed,evidence based guidelines accompanied by aneffective implementation strategy can helppromote cost effective health care. Thereforeclinical practice guidelines may be a usefulfocus for healthcare commissioning.The purchaser's role in this process may
involve identifying the best evidence ofeffectiveness and cost effectiveness or validnational guidelines and prioritising areas forintroducing local guidelines. Purchasers alsohave a role in sponsoring the development oflocal guidelines, incorporating them incontracts and service specifications, andsupporting providers in implementing them.By itself, however, commissioning is unlikely tobe sufficient to implement guidelines.Purchasers also have a role in monitoring andevaluating the development and implemen-tation of local guidelines. This may includesome form of review of utilisation6 or audit tomeasure the extent to which recommendedstandards are actually achieved. Purchasersand providers need to agree on criteria forreviewing practice based on guidelines. Ideally,relevant data on patient outcomes should becollected and analysed routinely to explorehow implementation of guidelines may beinfluencing quality of care.
A limited number of guidelines can beassimilated by healthcare professionals or
provider organizations at any one time. Localactivities should be coordinated to prioritisethe guidelines that professional groups are
asked to implement. Greater priority should begiven to introducing guidelines which addressimportant local need, where rigorous nationalguidelines or research evidence are readilyavailable, and where current practice divergesfrom best practice, thus indicating the potentialfor significant gains in health. Local develop-ment of guidelines should be both adequatelyresourced and multidisciplinary (includingrepresentatives of all key clinical disciplinesand providers and purchasers) as successfullyimplementing guidelines normally requireschanges in the behaviour of more than oneprofessional group.'34 135 Public involvement indeveloping guidelines may enhance implemen-tation, especially when public expectationsinfluence practice. Clinical audit groups may
be well placed to coordinate and resource the
development of local guidelines if encouragedto develop skills in leading and facilitating localgroups developing guidelines.
Clinicians' concerns about the legal status ofguidelines and potential litigation resulting
from non-compliance may be a major barrierto implementing guidelines. However, com-pliance with, or deviation from, a clinicalguideline is unlikely to prove conclusive in amedical negligence action, unless it can beshown that the guideline is so well establishedthat no responsible doctor acting withreasonable skill would fail to comply with it.Therefore, medicolegal issues do not, inprinciple, represent a barrier to implemen-tation. 136
Research issuesGuidelines are not always the most appropriatetool to promote cost effective health care. Itwould be useful to study the optimalcontribution that guidelines can make.Research is needed to identify the most costeffective methods for developing valid andreliable guidelines, which should includeresearch into methods for deriving recommen-dations and incorporating costs. It is alsoimportant to develop valid instruments forcritically appraising guidelines.'3' Research isrequired of the effects of different formats andstyles of guidelines on their adoption and ofbarriers to adopting guidelines. Although someevidence exists on the effectiveness of differenteducational and implementation strategiesthese are still poorly understood in the UnitedKingdom. Future research will be able to buildon the reviews from the new CochraneCollaboration on Effective ProfessionalPractice, which will provide up to date system-atic reviews of the evidence on the effectivenessof different approaches to implementation. 137
We thank the following for their useful comments on a draft ofthe bulletin: Richard Baker, Richard Grol, Allen Hutchinson,Jonathan Lomas, Andy Oxman, and Mary Ann Thompson. TheChief Scientist Office of the Scottish Office Home and HealthDepartment fund the Health Services Research Unit. TheDepartment ofHealth funds the Effective Health Care Bulletinsand the Centre for Reviews and Dissemination. However, theviews expressed are those of the authors and not necessarily thefunding bodies.
1 NHS Management Executive. Improving clinical effective-ness. Leeds: Department of Health, 1993. (EL(93) 115.)
2 NHS Executive. Improving the effectiveness of the NHS.Leeds: Department of Health, 1994. (EL(94)74.)
3 Field MJ, Lohr KN, eds. Guidelines for clinical practice.From development to use. Washington, DC: NationalAcademy Press, 1992.
4 Song F, Mason JM, Ibbotson S, Freemantle N, Long AF,Sheldon TA. Issues of quality in providing services forsubfertile couples. Quality in Health Care 1992;1:202-5.
5 Sheldon TA, Freemantle N, Song F, Mason JM, Long AF,Thakker Y, et al. Surgical interventions for glue ear: whatform will a quality service take? Quality in Health Care1992;1:266-70.
6 Sheldon TA, Borowitz M. Changing the measure ofqualityin the NHS: from purchasing activity to purchasingprotocols. Quality in Health Care 1993;2:149-50.
7 Grimshaw JM, Russell IT. Effect of clinical guidelines onmedical practice: a systematic review of rigorousevaluations. Lancet 1993;342: 1317-22.
8 Departments of Health and Social Security Library andInformation Service. DHSS-DA TA. London: Depart-ments of Health and Social Security Library andInformation Service, 1983.
9 Elsevier Science Publishers. Elsevier biomedical bibliographicdatabases (Embase). Amsterdam: Elsevier SciencePublishers BV, 1992.
10 National Library of Medicine. Grateful Med. Bethesda,National Library of Medicine, 1989.
11 EAGLE Technical Committee. System for information ongrey literature (SIGLE). The Hague: EAGLE TechnicalCommittee, 1980.
12 Haynes RB, Davis DA, McKibbon A, et al. A criticalappraisal ofthe efficacy of continuing medical education.JAMA 1984;251:61-4.
13 Haynes RB, Walker CJ. Computer-aided qualityassurance: a critical appraisal. Arch Intern Med 1987;147:1297-301.
62
Developing and implementing clinical practice guidelines
14 Lomas J, Haynes RB. A taxonomy and critical review oftested strategies for the application of clinical practicerecommendations: from "official" to "individual"clinical policy. Am Jf Prev Med 1987;4:77-94.
15 Soumerai SB, McLaughlin TJ, Avorn J. Improving drugprescribing in primary care: a critical analysis of theexperimental literature. Milbank Q 1989;67:268-317.
16 Lomas J. Words without action? The production,dissemination, and impact of consensus recommen-dations. Annu Rev Public Health 1991;12:41-65.
17 Mugford M, Banfield P, O'Hanlon M. Effects of feedbackof information on clinical practice: a review. BMJ1991 ;303:398-402.
18 Agency for Health Care Policy and Research (AHCPR).Annotated bibliography: information dissemination to healthcare practitioners and policy makers. Rockville, Maryland:US Department of Health and Human Services, PublicHealth Service, AHCPR, 1992. (AHCPR PublicationNo 92-0030.)
19 Davis DA, Thomson MA, Oxman AD, et al. Evidence forthe effectiveness of CME. A review of 50 randomizedcontrolled trials. JAMA 1992;268: 1111-7.
20 Grol R. Implementing guidelines in general practice care.Quality in Health Care 1992;1: 184-91.
21 Axt-Adam P. van der Wouden JC, van der Does E.Influencing behaviour of physicians ordering laboratorytests: a literature study. Med Care 1993;31:784-94.
22 Buntinx F, Winkens R, Grol R, et al. Influencing diagnosticand preventive performance in ambulatory care by feed-back and reminders. A review. Fam Pract1993;10:219-28.
23 Woolf SH. Practice guidelines: a new reality in medicine.III. Impact on patient care. Arch Intern Med 1993;153:2646-55.
24 Johnston ME, Langton KB, Haynes RB, et al. Effects ofcomputer-based clinical decision support systems onclinician performance and patient outcome: a criticalappraisal of research. Ann Intern Med 1994;120: 135-42.
25 Haines A, Jones R. Implementing findings of research.BMJ 1994;308: 1488-92.
26 Wensing M, Grol R. Single and combined strategies forimplementing changes in primary care: a literaturereview. International Journal of Quality in Health Care1994;6:115-32.
27 Russell I, Grimshaw J. The effectiveness of referralguidelines: a review of the methods and findings ofpublished evaluations. In: Roland M, Coulter A, eds.Hospital referrals. Vol 1. Oxford: Oxford University Press,1992:179-211.
28 Morgan M, Studney DR, Barnett GO, et al. Computerizedconcurrent review of prenatal care. QRB Qual Rev Bull1978;4:33-6.
29 Norton PG, Dempsey U. Self-audit: its effect on qualityof care. FamPract 1985;21:289-91.
30 Landgren FT, Harvey KJ, Mashford ML, et al. Changingantibiotic prescribing by educational marketing. Med JAust 1988;149:595-9.
31 De Vos Meiring P, Wells IP. The effect of radiologyguidelines for general practitioners in Plymouth. ClinRadiol 1990;42:327-9.
32 KosecoffJ, Kanouse DE, Rogers WH, et al. Effects of theNational Institutes of Health Consensus DevelopmentProgram on physician practice. JAMA 1987;258:2708-13.
33 Thompson RS, Kirz HL, Gold RA. Changes in physicianbehavior and cost savings associated with organizationalrecommendations on the use of "routine" chest x rays andmultichannel blood tests. Prev Med 1983;12:385-96.
34 McDonald CJ. Protocol-based computer reminders, thequality of care and the non-perfectability of man. N EnglJ7Med 1976;295:1351-5.
35 Sanazaro PJ, Worth RM. Concurrent quality assurance inhospital care. Report of a study by private initiative inPSRO. N EnglJ Med 1978;298:1171-7.
36 Hopkins JA, Shoemaker WC, Greenfield S, et al.Treatment of surgical emergencies with and without analgorithm. Arch Surg 1980;115:745-50.
37 Linn BS. Continuing medical education. Impact onemergency room burn care. JAMA 1980;244:565-70.
38 McDonald CJ, Wilson GA, McCabe GP. Physicianresponse to computer reminders. JAMA 1980;244:1579-81.
39 Sommers LS, Sholtz R, Shepherd RM, et al. Physicianinvolvement in quality assurance. Med Care 1984;22:1115-38.
40 Palmer RH, Louis TA, Hsu LN, et al. A randomizedcontrolled trial of quality assurance in sixteenambulatory care practices. Med Care 1985;23:751-70.
41 Putnam RW, Curry L. Impact of patient care appraisal onphysician behaviour in the office setting. Can Med AssocJf 1985;132:1025-9.
42 Winickoff RN, Wilner S, Neisuler R, et al. Limitations ofprovider interventions in hypertension quality assurance.Am I Public Health 1985;75:43-6.
43 McAlister NH, Covvey HD, Tong C, et al. Randomisedcontrolled trial of computer assisted management ofhypertension in primary care. BMJ 1986;293:670-4.
44 Wirtschafter DD, Sumners J, Jackson JR, et al. Continuingmedical education using clinical algorithms: acontrolled-trial assessment of effect on neonatal care. AmJf Dis Child 1986;140:791-7.
45 Vinicor F, Cohen SJ, Mazzuca SA, et al. Diabeds: arandomized trial of the effects of physician and/or patienteducation on diabetes patient outcomes. Journal ofChronic Diseases 1987;40:345-56.
46 Putnam RW, Curry L. Physicians' participation inestablishing criteria for hypertension management in theoffice: will patient outcomes be improved? Can MedAssocJ7 1989;140:806-9.
47 Mazzuca SA, Vinicor F, Einterz RM, et al. Effects of theclinical environment on physicians' response topostgraduate medical education. American EducationalResearch Journal 1990;27:473-88.
48 Lomas J, Enkin M, Anderson GM, et al. Opinion leadersvs audit and feedback to implement practice guidelines.Delivery after previous cesarean section. JAMA 1991;265:2202-7.
49 Margolis CZ, Warshawsky SS, Goldman L, et al.Computerized algorithms and pediatricians'management of common problems in a communityclinic. Acad Med 1992;67:282-4.
50 North of England Study of Standards and Performance inGeneral Practice. Medical audit in general practice:effects on doctors' clinical behaviour and the health ofpatients with common childhood conditions. BMJ 1992;304:1480-8.
51 Emslie CJ, Grimshaw J, Templeton A. Do clinicalguidelines improve general practice management andreferral of infertile couples? BMJ 1993;306: 1728-31.
52 Jones RH, Lydeard S, Dunleavey J. Problems withimplementing guidelines: a randomised controlled trialof consensus management of dyspepsia. Quality in HealthCare 1993;2:217-21.
53 Soumerai SB, Salem-Schatz S, Avorn J, et al. A controlledtrial of educational outreach to improve bloodtransfusion practice. JAMA 1993;270:961-6.
54 Anderson FA, Wheeler HB, Goldberg RJ, et al. Changingclinical practice: prospective study of the impact ofcontinuing medical education and quality assuranceprograms on use of prophylaxis for venous thrombo-embolism. Arch Intern Med 1994;154:669-77.
55 McDonald CJ. Use of a computer to detect and respondto clinical events: its effects on clinician behavior. AnnIntern Med 1976;84:162-7.
56 Coe FL, Norton E, Oparil S, et al. Treatment of hyperten-sion by computer and physician: a prospective controlledstudy. Journal of Chronic Diseases 1977;30:81-92.
57 Restuccia JD. The effect of concurrent feedback inreducing inappropriate hospital utilization. Med Care1982;20:46-62.
58 Rogers JL, Haring OM, Wortman PM, et al. Medicalinformation systems: assessing impact in the areas ofhypertension, obesity and renal disease. Med Care1982;20:63-74.
59 Barnett GO, Winickoff RN, Morgan MM, et al. Acomputer-based monitoring system for follow up ofelevated blood pressure. Med Care 1983;21:400-9.
60 Thomas JC, Moore A, Qualls PE. The effect on cost ofmedical care for patients treated with an automatedclinical audit sysem.JMed Syst 1983;7:307-13.
61 Brownbridge G, Evans A, Fitter M, et al. An interactivecomputerized protocol for the management of hyper-tension: effects on the general practitioner's clinicalbehaviour. J7R Coll Gen Prac 1986;36:198-202.
62 Weingarten S, Ermann B, Bolus R, et al. Early "step-down" transfer of low risk patients with chest pain: acontrolled interventional trial. Ann Intern Med1990;113:283-9.
63 Barnett GO, Winickoff MD, Dorsey MD, et al. Qualityassurance through automated monitoring andconcurrent feedback using a computer based medicalinformation system. Med Care 1978;XVI:962-70.
64 Lomas J, Anderson GM, Domnick-Pierre K, et al. Dopractice guidelines guide practice? The effect of aconsensus statement on the practice of physicians. NEnglJ Med 1989;321:1306-11.
65 Durand-Zaleski I, Bonnet F, Rochant H, et al. Usefulnessof consensus conferences: the case of albumin. Lancet1992;340: 1388-90.
66 Sherman CR, Potosky AL, Weis KA, et al. The ConsensusDevelopment Program detecting changes in medicalpractice following a consensus conference on thetreatment of prostate cancer. Int J Technol Assess HealthCare 1992;8:683-93.
67 Cohen DI, Littenberg B, Wetzel C, et al. Improvingphysician compliance with preventive medicineguidelines. Med Care 1982;XX: 1040-5.
68 McDonald CJ, Hui SL, Smith DM, et al. Reminders to
physicians from an introspective computer medicalrecord: a two year randomized trial. Ann Intern Med1984;100: 130-8.
69 McDonald CJ, Hui SL, Tierney WM. Effects of computerreminders for influenza vaccination on morbidity duringinfluenza epidemics. MD Comput 1992;9:304-12.
70 Winickoff RN, Coltin KL, Morgan MM, et al. Improvingphysician performance through peer comparisonfeedback. Med Care 1984;22:527-34.
71 Cohen SJ, Weinberger M, Hui SL, et al. The impact ofreading on physicians' non-adherence to recommendedstandards of medical care. Soc Sci Med 1985;21:909-14.
72 Tierney WM, Hui SL, McDonald CJ. Delayed feedbackof physician performance versus immediate reminders to
perform preventive care. Med Care 1986;24:659-66.73 Cheney C, Ramsdell JW. Effect of medical records'
checklists on implementation of periodic healthmeasures. Am J7Med 1987;83:129-36.
74 Cohen SJ, Christen AG, Katz BP, et al. Counselingmedical and dental patients about cigarette smoking: theimpact of nicotine gum and chart reminders. Am J7 PublicHealth 1987;77:313-6.
63
Grinmoshazv, Freemoantle, Wallace, Russell, Hurwitz, Watt, Loag, Sheldon
75 Cohen SJ, Stookey GK, Katz BP, et al. Encouragingprimary care physicians to help smokers quit: arandomized, controlled trial. Ann Intern Med 1989;110:648-52.
76 Wilson DM, Taylor DW, Gilbert JR, et al. A randomizedtrial of a family physician intervention for smokingcessation.JAMA 1988;260:1570-5.
77 Cummings SR, Coates TJ, Richard RJ, et al. Trainingphysicians in counseling about smoking cessation: arandomized trial of the 'Quit for Life' program. ArniIntern Med 1989;110:640-7.
78 McPhee SJ, Bird JA, Jenkins CNH, et al. Promoting cancerscreening: a randomized, controlled trial of threeinterventions. Arch Intern Med 1989;149:1866-72.
79 Turner RC, Waivers LE, O'Brien K. The effect ofpatient-carried reminder cards on the performance ofhealth maintenance measures. Arcch Intern Med1990;150:645-7.
80 McPhee SJ, Bird JA, Fordham D, ct al. Promoting cancerprevention activities by primary care physicians. Resultsof a randomized, controlled trial. JAMA 1991;266:538-44.
81 Ornstein SM, Garr DR, Jenkins RG, et al. Computer-generated physician and patient reminders. Tools toimprove population adherence to selected preventiveservices. _J Farn IPract 199 1;32:82-90.
82 Cowan JA, Heckerling PS, Parker JB. Effect of a fact sheetreminder on performance of the periodic healthexamination: a randomized controlled trial. Amn 7 Pre'Med 1992;8:104-9.
83 Dietrich AJ, O'Connor GT, Keller A, et al. Cancer:improving early detection and prevention: a communitypractice randomised trial. BMJ 1992;304:687-91.
84 Headrick LA, Speroff T, Pelecanos HI, et al. Efforts toimprove compliance with the National CholesterolEducation Program guidelines. Results of a randomizedcontrolled trial [see comments]. Arch Intern Med1992;152:2490-6.
85 Litzelman DK, Dittus RS, Miller ME, et al. Requiringphysicians to respond to computerized remindersimproves their compliance with preventive careprotocols. ] Gen, Intern Med 1993;8:311-7.
86 Mayefsky JH, Fove HR. Use of a chart audit: teaching wellchild care to pediatric house officers. Med Eduxc 1993;27:170-4.
87 Rodney WM, Chopisvsky P, Quan M. Adult immunization:the medical record design as a facilitator for physiciancompliance. 7ontarnal M~ff1Iedical Educatnon 1983;58:576-80.
88 McDowell I, Newell C, Rosser W. Comparison of threemethods of recalling patients for influenza vaccination.Can Med Assoc 1986;135:991-7.
89 Prislin MD, Vandenbark MS, Clarkson QD. The impactof a health screening flow sheet on the performance anddocumentation of health screening procedures. FamilyMedicine 1986;18:290-2.
90 Becker DM, Gomez EB, Kaiser DL, et al. Improvingpreventive care at a medical clinic: how can the patienthelp? Ain7Prev Med 1989;5:353-9.
91 Chambers CV, Balaban DJ, Carlson BL, et a!. Micro-computer-generated reminders improving the com-pliance of primary care physicians with mammographyscreening guidelines.I Fant Pract 1989;29:273-80.
92 McDowell I, Newell C, Rosser W. A randomized trial ofcomputerized reminders for blood pressure screening inprimary care. Med Care 1989;27:297-305.
93 McDowell 1, Newell C, Rosser W. Computerizedreminders to encourage cervical screening in familypractice. _7 Fain Pract 1989;28:420-4.
94 Rosser WW, McDowell I, Newell C. Use of reminders forpreventive procedures in family medicine. Can MedAssoc_J 1991;145:807-14.
95 Lilford RJ, Kelly M, Baines A, et al. Effect of usingprotocols on medical care: randomised trial of threemethods of taking an antenatal history. BMJ 1992;305:1181-4.
96 Rosser WW, Hutchison BG, McDowell I, et al. Use ofreminders to increase compliance with tetanus boostervaccination. Can MedAssoc 1992;146:911-7.
97 Robie PW. Improving and sustaining outpatient cancerscreening by medicine residents. South Med .7 1988;81:902-5.
98 Schreiner DT, Petrusa ER, Rettie CS, et al. Improvingcompliance with preventive medicine procedures in ahouse staff training program. South/ M1ed 7 1988;81:1553-7.
99 Nattinger AB, Panzer RJ, Janus J. Improving utilization ofscreening mammography in primary care practices. ArchIntern Med 1989;149:2087-92.
1(00 Tape TG, Campbell JR. Computerized medical recordsand preventive health care: success depends on manyfactors. Ain .7 Med 1993;94:619-25.
101 Marton Kl, Tul V, Sox HC. Modifying test-orderingbehavior in the outpatient medical clinic. A controlledtrial of two educational interventions. Arch Intern Med1985;145:816-21.
102 Chassin MR, MeCue SM. A randomized trial of medicalquality assurance: improving physicians' use ofpelvimetry.JAMA 1986;256:1012-6.
103 Avorn J, Soumerai SB, Everitt DE, et al. A randomizedtrial of a program to reduce the use of psychoactive drugsin nursing homes. N EniglJ Med 1992;327:168-73.
104 Bearcroft PWP, Small JH, Flower CDR. Chestradiography guidelines for general practitioners: apractical approach. ClinfRadiol 1994;49:56-8.
105 Oakeshott P, Kerry SM, Williams JE. Randomizedcontrolled trial of the effect of the Royal College ofRadiologists' guidelines on general practitioners'referrals for radiographic examination. Brf 7 Gen Pract1994;44:197-200.
106 Eisenberg JM, Williams SV, Garner L, et al. Computer-based audit to detect and correct overutilization oflaboratory tests. Med Care 1977;15:915 21.
107 Brook RH, Williams KN. Effect of medical care resiesw onthe use of injections: a study of the New MexicoExperimental Medical Care Review Organization. AnnIntern Med 1976;85:509-15.
108 Lohr KN, Brook RH. Quality of care in episodes ofrespiratory illness among Medicaid patients in NewsMexico. Ann Intere Med 1980;92:99 106.
109 Wong ET, McCarron MM, Shaw ST Jr. Ordering oflaboratory tests in a teaching hospital: can it beimproved?_JAMA 1983;249:3076-80.
11 0 Fowkes FGR, Evans RC, Williams LA, it a!. Implemen-tation of guidelines for the use of skull radiographs inpatients with head injuries. Lancet 1984;ii:795--i.
Ill Novich M, Gillis L, Tauber Al. The laboratory testjustified: an effective means to reduce routine laboratorytesting. Anietican .jouotal l/' alinicl Pr/thologs', 1985;84:756-9.
11 2 Fowkes FGR, Hall R, Jones JH, ct al. Trial of strategy forreducing the use of laboratory tests. AIAl]_ 1986;292:883-5.
113 Fowkes FGR, Davies ER, Evans KT1, it al. Multicentretrial of four strategies to reduce use of a radiological test.Lancet 1986;i:367-70.
114 Ray WA, Blazer DG, Schaffner W, Lt al. Reducing longterm diazepam prescribing in office practice: a controlledtrial of educational visits. _7AMA 1986;256:2536-9.
115 Avorn J, Soumerai SB, Taylor W, ct al. Reduction ofincorrect antibiotic dosing through a structurededucational order form. ArlL/ Inltern AIled 1988;148:1720-4.
116 Bareford D, Hayling A. Inappropriate use of laboratoryservices: long term combined approach to modifsrequest patterns. BM7 1990,301:1305-7.
11 7 Clarke JA, Adams JE. The application of clinical guidelinesfor skull radiography in the accident and emergencydepartment: theory and practice. C/lin1 Rat/t(l1990;41:152-5.
118 Everitt DE, Soumerai SB, Avorn J, et al. Changing surgicalantimicrobial prophylaxis practices through educationtargeted at senior department leaders. InfLct C(ontio HospEpideniiol 1990;11:578-83.
119 Raisch DW, Bootman JL, Larson LN, ct al. Improvingantiulcer agent prescribing in a health maintenanceorganization. Am]_f Hosp Phann 1990;47: 1766-73.
120 Gama R, Nightingale PG, Ratcliffe JG. Effect ofeducational feedback on clinicians' requesting of cardiacenzymes. Antn C/in Biocheni 1992;29:224-5.
121 Soumerai SB, Avorn J, Taylor WC, et al. Improving choiceof prescribed antibiotics through concurrent remindersin an educational order form. Med Care 1993;31:552 8.
122 Grimshaw J, Russell I. Achieving health gain throughclinical guidelines. I. Developing scientifically alidguidelines. Quali't i'n Health Care 1993;2:243-8.
123 Antman EM, Lau J, Kupelnick B, et al. A comparison ofresults of meta-analsses of randomized control trials andrecommendations of clinical experts: treatments formyocardial infarction. JAMA 1992;268:240-8.
124 Mulrow CD. Systematic review: rationale for systematicreviews. BM] 1994;309:597-9.
125 Russell IT, GrimshawJM, Wilson BJ. Epidemiological andstatistical issues in medical audit. Proceedings of the RosalSociety of Edinburgh 1993glOlB:77-103.
126 Fink A, KosecoffJ, Chassin M, it al. Consensus methods:characteristics and guidelines for use. Ain 7 I'/tbLic Hea/lth1984;74:979-83.
127 Woolf SH. Practice guidelines, a new reality in medicine.II. Methods of developing guidelines. Aral Intent Mid1992;lS2:946-52.
128 Williams A. How should information on cost effectivenessinfluence clinical practice? In: OwLcointL into pilctiLL.London: BMJ (in press).
129 DrummondMF, Maynard A, eds. PlumL/ncgsianL/cicp)oci'l/tgLiii effecive health care. Edinburgh: ChurchillLivingstone, 1993.
130 Eddy DM. A nianiual fin assessing h/ealt/ p JLiLtl.S doinddesig7ning practice policies: the explicit appr(aclt.Philadelphia: American College of Physicians, 1992.
131 Cluzeau F, Littlejohns P, Grimshaw J. Appraising clinicalguidelines: towards a "Which" guide for purchasers.Quality in Health Care 1994;3:121-2.
132 Grimshaw J. Guidelines [letter]. BM_7 1994;308: 1511.133 Hayward RS, Wilson MC, Tunis SR, et al. More
informative abstracts of articles describing clinicalpractice guidelines. Ann Intern Med 1993;118:731 -7.
134 Lomas J. Making clinical policy explicit. Legislative policymaking and lessons for developing practice guidelines.Ini7] Techno/Assess Health Care 1993;9:11-25.
135 McNicol M, Layton A, Morgan G. Team working: the kesto implementing guidelines? Quality/ itt Health C'arc 1993;2:215-6.
136 Hurwitz B. Clinical guidelines: proliferation andmedicolegal significance. Qualirs' iii Health (,Cai 1994;3:37-44.
137 Freemantle N, Grilli R, Grimshaw, JM, Oxman AD.Implementing the findings of medicine. The CochraneCollaboration on Effectise Professional Practice. Ouaitrsit Health Carc 1995;4:45 7.
64
