Improving HPV Vaccination Rates: AStepped-Wedge Randomized TrialRebecca B. Perkins, MD, MSc,a Aaron Legler, MS,b Emily Jansen, MPH,c Judith Bernstein, MSN, PhD,d Natalie Pierre-Joseph, MD,b

Terresa J. Eun, AB,e Dea L. Biancarelli, MS,d Thomas J. Schuch, MD, MPH,f Karin Leschly, MD,g Anny T.H.R. Fenton, PhD,h

William G. Adams, MD, MPH,b Jack A. Clark, PhD,d Mari-Lynn Drainoni, PhD,d,i,j Amresh Hanchate, PhDk

abstractOBJECTIVES: To evaluate the effectiveness of a stepped-wedge randomized trial of Developmentof Systems and Education for Human Papillomavirus Vaccination (DOSE HPV), a multilevelintervention.

METHODS: DOSE HPV is a 7-session program that includes interprofessional provider education,communication training, data feedback, and tailored systems change. Five primary carepediatric and/or family medicine practices completed interventions between 2016 and 2018;all chose to initiate vaccination at ages 9 to 10. We compared vaccination rates in thepreintervention, intervention, and postintervention periods among 9- to 17-year-olds usingrandom-effects generalized linear regression models appropriate for stepped-wedge design,accounting for calendar time and clustering of patients by providers and clinic. Outcomesincluded (1) the likelihood that eligible patients would receive vaccination during clinic visits;(2) the likelihood that adolescents would complete the series by age 13; and (3) thecumulative effect on population-level vaccine initiation and completion rates. Postinterventionperiods ranged from 6 to 18 months.

RESULTS: In the intervention and postintervention periods, the adjusted likelihood of vaccinationat an eligible visit increased by .10 percentage points for ages 9 to 10 and 11 to 12, andcompletion of the vaccine series by age 13 increased by 4 percentage points (P , .001for all comparisons). Population-level vaccine initiation coverage increased from 75%(preintervention) to 84% (intervention) to 90% (postintervention), and completion increasedfrom 60% (preintervention) to 63% (intervention) to 69% (postintervention).

CONCLUSIONS: Multilevel interventions that include provider education, data feedback, tailoredsystems changes, and early initiation of the human papillomavirus vaccine series may improvevaccine series initiation and completion beyond the conclusion of the intervention period.

WHAT’S KNOWN ON THIS SUBJECT: Multilevel interventionstargeting providers appear to increase human papillomavirusvaccine series initiation rates. Limited research exists on theeffects of multilevel interventions on series completion, thesustainability of increases, and the impact of lowering the routineage of vaccination.

WHAT THIS STUDY ADDS: Multilevel interventions targeting theinterprofessional team that include provider education, datafeedback, tailored systems changes, and early initiation of thehuman papillomavirus vaccine series may be successful inimproving vaccine series initiation and completion beyond theconclusion of the intervention period.

To cite: Perkins RB, Legler A, Jansen E, et al. ImprovingHPV Vaccination Rates: A Stepped-Wedge RandomizedTrial. Pediatrics. 2020;146(1):e20192737

dDepartment of Health Law, Policy and Management, School of Public Health, cContinuing Medical EducationOffice, iEvans Center for Implementation and Improvement Sciences, jSections of Infectious Diseases and kGeneralInternal Medicine, Department of Medicine, and aDepartments of Obstetrics and Gynecology and bPediatrics andAdolescent Medicine, School of Medicine, Boston University and Boston Medical Center, Boston, Massachusetts;eDepartment of Sociology, Stanford University, Stanford, California; fSouth Boston Community Health Center,Boston, Massachusetts; gEast Boston Neighborhood Health Center, Boston, Massachusetts; and hCenter forOutcomes, Research, and Evaluation, Maine Medical Center Research Institute, Portland, Maine

PEDIATRICS Volume 146, number 1, July 2020:e20192737 ARTICLE by guest on March 6, 2021www.aappublications.org/newsDownloaded from

Data from the United States and othercountries demonstrate that humanpapillomavirus (HPV) vaccinationprofoundly reduces vaccine-typeoncogenic HPV infections, genitalwarts, and precancers of the cervix,with emerging data revealingreduction in HPV-related cancers.1–3

The Centers for Disease Control andPrevention (CDC) recommend routineHPV vaccine initiation between theages of 11 and 12 years. Vaccinationcan begin at age 9 and is alsorecommended for all adolescentsaged 13 to 26 years who were notvaccinated on time.4–6 Initiatingvaccination at an earlier age ispreferable to vaccinating in olderadolescence because vaccination ismost effective before sexual debut,7

and no waning of protection isobserved.8 HPV vaccination ratesremain below national goals of 80%of eligible adolescents completing thevaccine series,9 however, because68.1% of adolescents aged 13 to17 years initiated and 51.1%completed the vaccine series in2018.9,10 Only 39.9% of adolescentscompleted the series on time, definedas before their 13th birthday.11

Research has identified barriers toHPV vaccination,12 includinglack of effective providerrecommendation13–16 and lack ofreminder and recall systems, leadingto missed opportunities for bothseries initiation and completion.17,18

Multilevel interventions showpromise for raising HPV vaccinationrates by using techniques includingprovider education andcommunication training, dataassessment and feedback, and qualityimprovement methods.19–26 Increasesin vaccination rates varied bystudy type and setting, andpostintervention follow-up did notdemonstrate sustainability afterintervention completion.22

In 2011 and 2012, we pilotedDevelopment of Systems andEducation for Human PapillomavirusVaccination (DOSE HPV), a multilevel

performance improvementcontinuing medical education (PICME) intervention that includesprovider education andcommunication training, dataassessment and feedback, andtailored systems interventions toimprove HPV vaccination rates amongsocioeconomically vulnerablepopulations.23,27 PI CME is a learningformat consisting of longitudinalpractice improvement, for whichphysicians may earn continuingmedical education credit.Participation requirements for PICME align with maintenance ofcertification part IV requirements forboard-certified pediatric, internalmedicine, and family medicinephysicians. For nurses, nursingprofessional development (ie, contacthours) is provided for participation inlive meetings. The pilot studyinvolved 2 pediatric practices, useda pre- and postevaluation design, andwas conducted when a 3-doseschedule was the nationalrecommendation. To better estimateintervention effectiveness andprovide short-term sustainabilitydata, we performed a stepped-wedgecluster randomized trial of the DOSEHPV intervention at 5 additional sites.We chose the stepped-wedge designas the best study design to separatesecular trends from interventioneffects as well as incentivizeparticipation by clinical sites becauseall sites received the intervention. Wereport on the intervention’s effects onHPV vaccine series initiation andcompletion up through 18 monthsafter intervention completion.

METHODS

DOSE HPV Intervention and Setting

Interventions took place inMassachusetts in the Bostonmetropolitan area and included 5clinical sites serving primarily low-income and minority patients. Onesite was a safety-net hospital; theothers were independent federally

qualified health centers. Sites wereselected before the interventionperiod on the basis of 2014 site-specific data indicating HPVvaccination rates below nationaltargets.9 The DOSE HPV interventionis a provider-focused program thatincluded 7 sessions, 1 hour each,performed at approximately monthlyintervals over 6 to 8 months between2016 and 2018 (Fig 1). Sessions wereheld during regular staff meetings tomaximize attendance, and food wasserved. Intervention sessions wereinterprofessional, including clinicleadership, primary care providers,and nursing staff. The interventioneducates providers and activatesthem to make tailored systemschanges within their health caresettings. The intervention included 3core components: interprofessionaleducation and communicationtraining for health care providers,data assessment and feedback, andcreation of an action plan thatincluded both provider- and systems-level changes. All sessions werefacilitated by the intervention team,and sessions at which site-specificaction plans were developed were ledby clinical champions andparticipating providers.

In session 1, the study teamprovided health center–specificpreintervention data on HPV vaccineinitiation and completion rates. Forsession 2, a physician expert providededucation on HPV-associated cancers,effectiveness of vaccination, andevidence-based interventions.Sessions 3 and 4 were focused onmotivational interviewing training.28

During session 5, the study team usedquality improvement methods tocollaboratively identify barriers toHPV vaccination and create site-developed action plans, includingsystems- and provider-level changesto improve performance. Sessions 6and 7 were facilitated discussionsreviewing progress on implementingaction plans, defining near-termaction steps, and specifying task

2 PERKINS et al by guest on March 6, 2021www.aappublications.org/newsDownloaded from

ownership to address obstacles andaccelerate progress. Although the5 sites developed action plansspecific to their unique practiceenvironments, all decided to routinelyrecommend HPV vaccination beforeage 11 (Fig 1). The data reviewmotivated this change because clinic-level vaccination rates revealed that,although most patients receivedvaccination at well-child visits,a substantial proportion of theirprimary care populations remainedunvaccinated primarily because of

attrition in well-child care amongolder adolescents.

Data Collection and Analysis

The trial used a stepped-wedgecluster randomized trial design: thestudy team randomized clinical sites(clusters) by generated number, andinterventions occurred in order oftheir random number, staggered by∼6 months during the 3 year studyperiod.29 Interventions began onApril 7, 2016, and ended on March15, 2018. Electronic medical record

data were used to evaluate theintervention’s effect on HPVvaccination rates. The studypopulation included adolescents aged9 to 17 years with an assignedprimary care provider and $1 visit tothe primary care site at any timeduring the 3-year study period. Thestudy period was divided intopreintervention, intervention, andpostintervention periods for each site.The intention for the stepped-wedgedesign was for all preinterventionperiods to begin on the same date

FIGURE 1Intervention time line and content with adolescent vaccination by calendar month. In this figure, we describe the time line of each intervention, as well asthe percentage of adolescents with visits per calendar month who initiate the HPV vaccine series before or at the index visit. The granular representationof data details the individual action plan components at each site, and also notes changes in vaccination rates as they relate to the timing of the DOSEHPV intervention as well as release of the national 2-dose recommendations. EHR, electronic health record; PCP, primary care provider. a Nationalrecommendations for 2-dose schedule were published on December 16, 2016, in the Morbidity and Mortality Weekly Report. b Denotes action plancomponents that were discontinued. Recalls for patients overdue for vaccine were discontinued because of high staff effort and low patient response.Standing orders were discontinued because of staff turnover. EHR prompt was discontinued at site 5 because of PCP workflow issues. All other actionplan components were continued by clinical sites in the postintervention period. c Sites 4 and 5 transitioned to a different electronic medical recordsystem during the preintervention period, leading to a 1-month delay in preintervention data collection in site 4 and an 8-month delay in site 5.

PEDIATRICS Volume 146, number 1, July 2020 3 by guest on March 6, 2021www.aappublications.org/newsDownloaded from

(October 1, 2015), 6 months beforethe intervention start date at clinicalsite 1. However, because ofchangeover of electronic medicalrecord systems, preinterventionperiods began later at sites 4 and 5(November 1, 2015, and June 1, 2016,respectively; Fig 1). The interventionperiods began on the date of thefirst session and ended on thedate of the last session at eachsite. The postintervention periodsbegan 1 day after the lastintervention session at each siteand ended on September 30, 2018,for all sites. Differential pre- andpostintervention periods in thestepped-wedge design serve tocontrol for secular trends andallow comparisons within sitesover time as well as betweensites.29

All analyses accounted for thenational change from a 3-dose to a 2-dose schedule; dose eligibilitymirrored recommendations on thedate of the clinic visit. BeforeDecember 16, 2016, all adolescentsrequired 3 doses to complete theseries; visit dates $2 months afterthe first dose and $4 months afterthe second dose were consideredeligible. After December 2016, the 3-dose schedule was applied only toadolescents who initiated the serieson or after their 15th birthday.Adolescents who initiated vaccinationbefore their 15th birthday wereconsidered complete after 2 doses,with visit dates $5 months afterreceipt of dose 1 considered theminimum interval conferringeligibility for dose 2, consistent withCDC recommendations. Analysesconsidering intervals of $6 monthsand $12 months for dose 2 eligibilityyielded similar results.

Our primary outcome was thelikelihood that a patient who was duefor an HPV vaccine dose wouldreceive vaccination at an eligibleclinic visit. All visits to the primarycare clinical site by an adolescentwho was eligible to receive dose 1, 2,

or 3 on the date of the clinic visitwere eligible, including nurse-onlyand problem-focused visits. Visits tospecialty departments or emergencydepartments were not eligible. Weperformed subset analyses focusingon 9- to 10-year-olds, 11- to 12-year-olds, and 13- to 17-year-olds toexamine intervention effects on early,on-time, and catch-up vaccination.Our secondary outcome was rate of“on-time” completion, defined ascompletion before the 13th birthday,for children aged 12 and 13 years onthe visit date. We also examinedpopulation-level vaccine initiationand completion rates estimated forpatients who visited a study site atany time during the preintervention,intervention, or postinterventionperiods.

We used longitudinal generalizedlinear models to estimate the effect ofthe intervention on the primary andsecondary outcomes.29,30 Our maincovariate of interest was a categoricalindicator of the 3 study periods:preintervention (0), intervention (1),and postintervention (2). For theprimary outcome analysis, the unit ofobservation was a patient outpatientvisit, and we included all eligiblepatient visits during the study period.We regressed the indicator of receiptof an eligible vaccination (0 or 1) onthe period indicator as well aspotential confounding variables,including child age, sex, race andethnicity; English as primarylanguage; insurance coverage type;and indicator of receipt of tetanus ormeningitis vaccination during theoutpatient visit. We included dummyindicators of calendar time by quarterto capture secular temporal trends.We used a patient-level random-effects specification to capture theclustering of all visits and individualunobserved patient effects; becauseeach patient is affiliated witha provider and clinic, patient-levelrandom effects also capturesystematic variation by providerand clinic. We obtained

heteroscedasticity-consistent robustSEs.31 Similar regression models wereestimated for the secondaryoutcomes by using the appropriateeligible cohort noted above. Asa check on the robustness of ourfindings, we estimated the sameregression models with a patient-level fixed-effects specificationinstead of random effects; with fixedeffects, only within-patient variationacross visits is used to obtainestimates, avoiding potentialconfounding from unobserved factorsthat may be correlated with patient-level variables.31 Fixed-effects modelsyielded similar results; random-effects models are reported in the textand tables. The Boston UniversityInternal Review Board approved allintervention and evaluationprotocols.

RESULTS

Patient Characteristics

A total of 16 136 individuals wereincluded in our analyses (Table 1).Patient demographics reflected thepopulations served by the clinicalsites in the preintervention,intervention, and postinterventionperiods. Patients had a mean age of12 years (range: 9–17 years), wereequally divided between male andfemale individuals, and were raciallydiverse (12.0% to 15.3% white,19.6% to 29.8% black, 37.1% to55.0% Hispanic or Latino, 13.4% to17.8% other). Approximately half(42.7% to 56.0%) spoke English astheir primary language, and .80%had Medicaid or other subsidizedinsurance. Differences indemographics between the pre- andpostintervention periods relate to thestepped-wedge design. Clinical sites1, 2, and 3 had greater proportions ofHispanic and non-Hispanic blackpatients and longer postinterventionperiods, whereas clinical sites 4 and 5had more non-Hispanic whitepatients and longer preinterventionperiods.

4 PERKINS et al by guest on March 6, 2021www.aappublications.org/newsDownloaded from

Vaccination Rates at InterventionSites by Calendar Time and StudyPeriod

In Fig 1, we describe the stepped-wedge design in detail by calendarmonth. The action plan componentsenacted by each site are noted, as arewhether these changes werecontinued in the postinterventionperiod. The timing of preintervention,intervention, and postinterventionphases is delineated for each site,along with the unadjusted vaccineinitiation prevalence stratified byages 9 to 10, 11 to 12, and 13 to17 years. The monthly vaccineinitiation prevalence is the proportionof adolescents with an eligible visit atthe site in that month that initiatedthe HPV vaccine series either beforeor during the visit. Initiation rates for9- and 10-year-olds generallyincreased during the interventionperiods, generally from ,20% to.50%. More gradual improvementswere noted for older age groups.

At each intervention site, weevaluated unadjustedpreintervention, intervention, andpostintervention population coverageof HPV vaccine series initiation andcompletion, defined as the proportionof all 9- to 17-year-old patients withat least 1 visit to the clinical site inthe preintervention, intervention, orpostintervention periods with either1 HPV vaccine dose or a completeHPV vaccine series, respectively. For

all clinics combined, population-levelseries initiation coverage increasedfrom 75% (preintervention) to 84%(intervention) to 90% (postintervention),and series completion increased from60% (preintervention) to 63%(intervention) to 69% (postintervention)(P, .001 for all comparisons). Increasesoccurred at all clinical sites, withinitiation rates exceeding 80% for allsites by the postintervention period(Fig 2 A and B). Because HPV vaccinationis routinely recommended at ages 11 to12, this group was analyzed separately.Series initiation increased from 83%(preintervention) to 89% (intervention)to 93% (postintervention), and seriescompletion increased from 54%(preintervention) to 53% (intervention)to 69% (postintervention) (P , .001for all comparisons except completionin the intervention period, P = .40);individual clinic performance isdetailed in Fig 2 C and D.

Adjusted models that accounted forclustering, covariates of interest, andcalendar time were used to furtherexplore the effects of the intervention.Adjusted models indicated that thelikelihood of receiving vaccination atan eligible visit for children aged 9 to10 increased by 12.7 percentagepoints (intervention) and 27.3percentage points (postintervention)compared with the preinterventionperiod (P , .001 for all comparisons;Table 2). For ages 11 to 12, increaseswere 16.2 percentage points

(intervention) and 17.8 percentagepoints (postintervention periods; P ,.001 for all comparisons; Table 2). Forages 13 to 17, the likelihood ofvaccination increased by 8.1percentage points in the interventionperiod (P , .001; Table 2) but did notremain significantly elevated in thepostintervention period. Hispanic andnon-Hispanic black patients, patientswhose primary language was notEnglish, and those with subsidizedinsurance were more likely to receivevaccination than were non-Hispanicwhite, English-speaking, and privatelyinsured patients. Receipt of tetanusand/or meningitis vaccination wasstrongly associated with receipt ofHPV vaccination at the same visit.Over time, the likelihood ofvaccination at an eligible visitcontinued to increase for ages 9 to 10but decreased for older age groups.This may represent a ceiling effect, as.90% of patients aged 11 and olderwho attended clinic visits in thepostintervention periods had alreadyinitiated the vaccine series (Fig 1).Visits appearing as “missedopportunities” in older age groups inlater years may represent factors notamenable to clinical practice changes;for example, patients who either wereineligible or whose parentsconsistently refused vaccination. Wefound some indirect evidence fora ceiling effect: the proportion ofeligible visits with a previous missedopportunity rose from 35% to 71%

TABLE 1 Demographic Characteristics of Adolescents Aged 9–17 With at Least 1 Visit to Clinic Sites During the Study Period (N = 16 136)

Preintervention Period Intervention Period P Postintervention Period P

Sex, n (%)Male 6907 (53.5) 5370 (51.7) .0060 10 358 (50.1) ,.001Female 6000 (46.5) 5016 (48.3) 10 336 (49.9)

Race, n (%)Non-Hispanic white 1973 (15.3) 1111 (10.7) ,.001 2468 (12.0) ,.001Non-Hispanic black 3843 (29.8) 2819 (27.1) ,.001 4062 (19.6) ,.001Hispanic 4792 (37.1) 4807 (46.3) ,.001 11 382 (55.0) ,.001Other 2299 (17.8) 1649 (15.9) ,.001 2782 (13.4) ,.001

Age, mean (SD) 12.1 (2.43) 12.0 (2.31) .0105 12.2 (2.23) ,.001English is primary language, n (%) 7228 (56.0) 5094 (49.1) ,.001 8825 (42.7) ,.001Public health insurance, n (%) 10 446 (80.9) 8900 (85.7) ,.001 18 494 (89.4) ,.001Private health insurance, n (%) 2461 (19.1) 1486 (14.3) 2200 (10.6)

P value is listed once for binary variables. P values indicate statistical significance of differences in demographic variables comparing the preintervention period to the intervention andpostintervention periods.

PEDIATRICS Volume 146, number 1, July 2020 5 by guest on March 6, 2021www.aappublications.org/newsDownloaded from

from the preintervention to thepostintervention periods, and adjustedmodels indicated that patients withprevious missed opportunities wereless likely to be vaccinated at futurevisits (P , .001; data not shown).

A secondary outcome was on-timecompletion of the HPV vaccine series,by using the CDC definition of seriescompletion before the 13th birthdayand adjusted for visit date relative tonational recommendations for a 2-doseschedule. Adjusted models indicatea 4.3 and 4.7 percentage point increasein the likelihood of on-time completionin both the intervention and

postintervention periods, respectively,compared with the preinterventionperiod (P , .001; Table 3). Male sexand Hispanic ethnicity were alsopositively associated with on-timeseries completion. Over time, thelikelihood of an adolescent aged 12 to13 completing vaccination at or beforetheir eligible clinic visit increased (P ,.05 for the third quarter of 2016onward compared with the fourthquarter of 2015; Table 3).

DISCUSSION

This stepped-wedge randomizedtrial of DOSE HPV, a multilevel

intervention, was shown toimprove vaccination rates duringthe intervention period, andimprovements were sustainedduring follow-up periodsranging from 6 to 18 months.Our findings contribute to thebody of evidence indicating thatmultilevel interventions appearpromising as a means of raisingvaccination rates.19–23 Becausemultilevel interventions requiresubstantial investments of personneland time in the short-term,demonstrating that interventioneffects continue in the

FIGURE 2Impact of intervention on vaccine series initiation and completion. In this figure, we detail changes in vaccine series initiation and completion rates at allintervention sites combined (all sites) as well as individual intervention sites (sites 1–5) in the preintervention, intervention, and postinterventionperiods. A and B, Data on all adolescents ages 9 to 17. C and D, Data on 11- to 12-year-olds, the age for which routine vaccination is recommended. (Notethat clinic 5 had only a 6-month postintervention period, limiting ability to detect change.) Total population prevalence is examined in these outcomes;therefore, patients vaccinated before the intervention were excluded from primary outcomes (likelihood of vaccination at a given visit) but wereincluded here.

6 PERKINS et al by guest on March 6, 2021www.aappublications.org/newsDownloaded from

postintervention period is importantwhen clinical and policy decision-makers consider upfront costs.

Maintaining gains in vaccination ratesmay depend on the sustainability of

intervention components.32 TheDOSE HPV intervention activatesproviders to create tailored systemschanges, which may allowimprovements in HPV vaccinationprovision to be sustained afterintervention completion. Health

intervention literature indicates thatsystems-level changes require fewerongoing resources and can be moresustainable than provider-focusedefforts, especially as participatingproviders may leave the practiceafter intervention completion.33,34

However, research also demonstratesbenefits to educating providers35 aswell as developing interventions inpartnership with stakeholders toensure relevance and capacity.33,34

Viewing the DOSE HPV intervention

through this lens, educationalsessions and communicationstraining may have activated providersin the short-term, and allowingparticipants to develop tailoredsystems changes to address barriersmay have promoted sustainability bybuilding engagement and aligningefforts with existing clinicalprocesses.32,36,37 System-wide effortsto routinely recommend HPVvaccination at younger ages wereconsidered to be easy and successful

TABLE 2 Change in Likelihood of Adolescent Receiving Vaccination at an Eligible Visit During Intervention and Postintervention Periods Compared WithPreintervention Period

Variable Ages 9–10 Ages 11–12 Ages 13–17

Percentage Point Change inLikelihood of Vaccinationa

P Percentage Point Change inLikelihood

of Vaccinationa

P Percentage Point Change inLikelihood

of Vaccinationa

P

Study periodPreintervention (referenceb), % 20.2 — 45.6 — 42.5 —

Intervention 112.7 ,.001 116.2 ,.001 18.1 ,.001Postintervention 127.3 ,.001 117.8 ,.001 11.1 .49

SexFemale (reference), % 33.4 — 57.6 — 47.8 —

Male 13.1 .013 11.6 .04 26.1 ,.001RaceNon-Hispanic white (reference), % 30.2 — 52.4 — 40.9 —

Non-Hispanic black 12.7 .08 15.5 ,.001 14.0 .005Hispanic 18.3 ,.001 17.9 ,.001 15.7 ,.001Other 12.2 .18 14.2 .01 12.5 .11

Primary languageOther (reference), % 36.3 — 59.3 — 47.9 —

English 22.7 .006 22.4 .016 26.0 ,.001InsurancePrivate (reference), % 32.5 — 56.0 — 42.9 —

Public 12.8 .03 12.7 .048 12.4 .05Received meningitis or tetanusvaccines at visitNo (reference), % 34.5 — 40.7 — 37.3 —

Yes 114.5 ,.001 148.0 ,.001 140.0 ,.001Calendar time2015 fourth quarter (reference), % 21.2 — 80.4 — 61.8 —

2016 first quarter 0.0 .93 22.2 .23 26.0 .0012016 second quarter 22.6 .03 26.8 .001 24.1 .032016 third quarter 110.8 ,.001 25.6 .007 23.3 .12016 fourth quarter 113.9 ,.001 218.1 ,.001 28.4 ,.0012017 first quarter 117.6 ,.001 221.2 ,.001 214.0 ,.0012017 second quarter 118.5 ,.001 223.3 ,.001 218.7 ,.0012017 third quarter 127.3 ,.001 222.8 ,.001 218.0 ,.0012017 fourth quarter 120.5 ,.001 232.8 ,.001 221.7 ,.0012018 first quarter 121.9 ,.001 237.0 ,.001 228.2 ,.0012018 second quarter 117.7 ,.001 241.8 ,.001 231.7 ,.0012018 third quarter 119.8 ,.001 241.8 ,.001 233.1 ,.001

Eligible patients are those who had not completed the vaccine series and for whom a sufficient amount of time had elapsed since the previous dose to be eligible for a subsequent dose.Eligibility was adjusted for national vaccine schedule recommendations at date of visit and age at vaccine initiation. —, reference.a Percentage point change refers to an increase or decrease in percentage of adolescents fulfilling criteria (eg, from 20.2% completing series on time preintervention among those aged 9to 10 years to 20.2% 1 27.3%, a total of 47.5% completing the series postintervention).b Reference refers to the value against which other categories of that variable are compared. For example, the preintervention period is the reference value for the study period.Percentage point changes estimate the change in the intervention and postintervention periods compared with the preintervention period.

PEDIATRICS Volume 146, number 1, July 2020 7 by guest on March 6, 2021www.aappublications.org/newsDownloaded from

changes by providers38 and appearedto be a key factor underlyingpopulation-level gains over time.Initiating vaccination at age 9 or 10can increase the number ofopportunities to complete vaccinationon time.39 High rates of vaccinationamong 9- and 10-year-olds in thepostintervention period indicatedthat early vaccination was acceptableto parents. Providers attributedhigh parental acceptance of earlyvaccination to a preference forfewer shots per visit as well as anattenuated connection betweenvaccination and sexual activity atyounger ages.38 The ability of sites to

implement and sustain this change isdemonstrated by increases in earlyand on-time vaccine series initiationand completion that persisted throughthe postintervention periods.Researchers of other studies have alsodemonstrated that initiatingvaccination before age 11 years canincrease on-time vaccination rates;therefore, this may be a crucialcomponent of future interventions.40,41

This study has several limitations,including a limited geographicsampling area and implementation inclinics serving primarily low-income,minority, urban populations.

Although the clinical sites were allresource constrained, they did haverelatively high HPV vaccine initiationrates at baseline (75%), indicatingthat providers and parents werealready accustomed to vaccinating.Two of our sites transitioned to a newelectronic medical record system,limiting our ability to identifypatients who might receive primarycare at the intervention sites but whohad not presented for care in .12months; this in turn could lead tooverestimation of population-levelvaccination rates. Also, we could notidentify immunocompromisedadolescents who required a 3-doseschedule.

HPV vaccination rates continued toincrease nationally during the studyperiod, likely facilitated by therelease of the 2-dose schedule inDecember 2016. Although thestepped-wedge design is intended tocontrol for secular changes, fullydisentangling intervention effectsfrom contextual factors is difficult.To be able to statistically estimatethe effect of the intervention fromthat of the 2-dose schedule, wewould need to separately estimatethe DOSE HPV effect among those on3-dose and 2-dose schedulesseparately. Because our interventionfocused primarily on youngeradolescents, all of whom wereaffected by the 2-dose scheduleswitch, we cannot perform this typeof stratified analysis and primarilyuse the stepped-wedge design toassess for secular trends. Theregression models indicate that theintervention increased the likelihoodof vaccine receipt and seriescompletion after controlling forcalendar time. Despite theselimitations, this trial is one of thefirst to indicate effectiveness ofa multilevel intervention leading toimprovements in HPV vaccinationrates 6 to 18 months beyondcompletion of the intervention. Iflong-term gains are replicated inother populations, these types of

TABLE 3 Change in Likelihood of Adolescent Completing Vaccine Series by the 13th Birthday DuringIntervention and Postintervention Periods Compared With Preintervention Period

Variable Percentage Point Change in Likelihoodof Vaccinationa

P

Study periodPreintervention (referenceb), % 61.1 —

Intervention 14.3 ,.001Postintervention 14.7 ,.001

SexFemale (reference), % 62.7 —

Male 14.0 ,.001RaceNon-Hispanic white (reference), % 57.4 —

Non-Hispanic black 0.0 .93Hispanic 112.1 ,.001Other 11.6 .50

Primary languageOther (reference), % 65.1 —

English 10.3 .79InsurancePrivate (reference) 64.3 —

Public 20.6 .71Calendar time2015 fourth quarter (reference), % 59.4 —

2016 first quarter 12.7 .032016 second quarter 12.0 .172016 third quarter 14.8 .0012016 fourth quarter 14.6 .0022017 first quarter 16.1 ,.0012017 second quarter 17.2 ,.0012017 third quarter 110.3 ,.0012017 fourth quarter 19.0 ,.0012018 first quarter 111.5 ,.0012018 second quarter 112.1 ,.0012018 third quarter 116.7 ,.001

Eligible adolescents are those aged 12 to 13 years on visit date; definition of completion is adjusted for national vaccineschedule recommendations on visit date (eg, 3 doses required to be considered complete for visits occurring beforeDecember 16, 2016). —, not applicable.a Percentage point change refers to increase or decrease in percentage of adolescents fulfilling criteria (eg, from 61.1%completing series by the 13th birthday to 61.1% 1 4.7%, a total of 65.8%, postintervention).b Reference refers to the value against which other categories of that variable are compared. For example, thepreintervention period is the reference value for the study period. Percentage point changes estimate the change in theintervention and postintervention periods compared with the preintervention period.

8 PERKINS et al by guest on March 6, 2021www.aappublications.org/newsDownloaded from

interventions could be uniquelyuseful tools to improve HPVvaccination rates in the UnitedStates.

CONCLUSIONS

Multilevel interventions that includeinterprofessional provider education,data feedback, and tailored systemschanges that focus on initiating HPVvaccination before age 11 may besuccessful in improving HPV vaccineseries initiation and completion

beyond the conclusion of theintervention period. Replication ofthese findings in settings with lowerbaseline vaccination rates, in othergeographic regions, and among ruralpopulations and those with privateinsurance is an important area forfuture research.

ACKNOWLEDGMENTS

We thank the providers and clinicstaff who participated in theintervention.

ABBREVIATIONS

CDC: Centers for Disease Controland Prevention

DOSE HPV: Development ofSystems andEducation for HumanPapillomavirusVaccination

HPV: human papillomavirusPI CME: Performance Improve-

ment Continuing MedicalEducation

Dr Perkins conceptualized, designed, and delivered component interventions of the study, supervised data collection, drafted the initial manuscript, and revised and

reviewed the manuscript; Dr Bernstein conceptualized the study, delivered core curricular components on motivational interviewing, and reviewed the manuscript;

Drs Joseph, Adams, Clark, Fenton, and Drainoni conceptualized and designed the study and critically reviewed and revised the manuscript; Dr Hanchate and Mr

Legler conceptualized and designed the study methodology and statistical analyses and conducted all data analyses; Drs Schuch and Leschly oversaw

implementation of interventions and critically reviewed and revised the manuscript; Ms Eun and Ms Biancarelli critically reviewed and revised the manuscript; Ms

Jansen delivered component interventions of the study, performed data analysis during active intervention, and critically reviewed and revised the manuscript; and

all authors approved the final manuscript as submitted and agree to be accountable for all aspects of the work.

This trial has been registered at www.clinicaltrials.gov (identifier NCT02812732).

DOI: https://doi.org/10.1542/peds.2019-2737

Accepted for publication Apr 8, 2020

Address correspondence to Rebecca B. Perkins, MD, MSc, Department of Obstetrics and Gynecology, Boston University School of Medicine and Boston Medical

Center, 85 E Concord St, 6th Floor, Boston, MA 02118. E-mail: rbperkin@bu.edu

PEDIATRICS (ISSN Numbers: Print, 0031-4005; Online, 1098-4275).

Copyright © 2020 by the American Academy of Pediatrics

FINANCIAL DISCLOSURE: The authors have indicated they have no financial relationships relevant to this article to disclose.

FUNDING: Supported by an American Cancer Society Research Scholar grant (grant 128607-RSG-15-150-01-CPHPS). No commercial support was obtained.

POTENTIAL CONFLICT OF INTEREST: Dr Joseph has received grant funding from Merck; the other authors have indicated they have no potential conflicts of interest

to disclose.

REFERENCES

1. Drolet M, Bénard É, Boily MC, et al.Population-level impact and herdeffects following human papillomavirusvaccination programmes: a systematicreview and meta-analysis.Lancet Infect Dis. 2015;15(5):565–580

2. Luostarinen T, Apter D, Dillner J, et al.Vaccination protects against invasiveHPV-associated cancers. Int J Cancer.2018;142(10):2186–2187

3. World Health Organization. Humanpapillomavirus vaccines: WHO positionpaper, October 2014. Wkly EpidemiolRec. 2014;89(43):465–491

4. Markowitz LE, Dunne EF, Saraiya M,et al; Centers for Disease Control andPrevention (CDC). Humanpapillomavirus vaccination:recommendations of the AdvisoryCommittee on Immunization Practices(ACIP) [published correction appearsin MMWR Recomm Rep. 2014;63(49):1182]. MMWR Recomm Rep. 2014;63(RR):1–30

5. Meites E, Kempe A, Markowitz LE. Use ofa 2-dose schedule for humanpapillomavirus vaccination - updatedrecommendations of the AdvisoryCommittee on Immunization Practices.

MMWR Morb Mortal Wkly Rep. 2016;65(49):1405–1408

6. Meites E, Szilagyi PG, Chesson HW,Unger ER, Romero JR, Markowitz LE.Human papillomavirus vaccination foradults: updated recommendations ofthe Advisory Committee onImmunization Practices. MMWR MorbMortal Wkly Rep. 2019;68(32):698–702

7. Gertig DM, Brotherton JM, Budd AC,Drennan K, Chappell G, Saville AM.Impact of a population-based HPVvaccination program on cervicalabnormalities: a data linkage study.BMC Med. 2013;11:227

PEDIATRICS Volume 146, number 1, July 2020 9 by guest on March 6, 2021www.aappublications.org/newsDownloaded from

8. Guevara A, Cabello R, Woelber L, et al.Antibody persistence and evidenceof immune memory at 5yearsfollowing administration of the 9-valentHPV vaccine. Vaccine. 2017;35(37):5050–5057

9. HealthyPeople 2020. Immunization andinfectious diseases. Available at:https://www.healthypeople.gov/2020/topics-objectives/topic/immunization-and-infectious-diseases/objectives.Accessed July 2, 2019

10. Walker TY, Elam-Evans LD, Yankey D,et al. National, regional, state, andselected local area vaccinationcoverage among adolescents aged13–17 years - United States, 2018.MMWR Morb Mortal Wkly Rep. 2019;68(33):718–723

11. National Committee for QualityAssurance. Immunizations foradolescents (IMA). Available at: https://www.ncqa.org/hedis/measures/immunizations-for-adolescents/.Accessed August 8, 2019

12. Holman DM, Benard V, Roland KB,Watson M, Liddon N, Stokley S. Barriersto human papillomavirus vaccinationamong US adolescents: a systematicreview of the literature. JAMA Pediatr.2014;168(1):76–82

13. Perkins RB, Clark JA, Apte G, et al.Missed opportunities for HPVvaccination in adolescent girls:a qualitative study. Pediatrics. 2014;134(3). Available at: www.pediatrics.org/cgi/content/full/134/3/e666

14. Vadaparampil ST, Kahn JA, Salmon D,et al. Missed clinical opportunities:provider recommendations for HPVvaccination for 11–12 year old girls arelimited. Vaccine. 2011;29(47):8634–8641

15. Rosenthal SL, Weiss TW, Zimet GD, Ma L,Good MB, Vichnin MD. Predictors of HPVvaccine uptake among women aged 19-26: importance of a physician’srecommendation. Vaccine. 2011;29(5):890–895

16. Albright K, Barnard J, O’Leary ST, et al.Noninitiation and noncompletion of HPVvaccine among English- and Spanish-speaking parents of adolescent girls:a qualitative study. Acad Pediatr. 2017;17(7):778–784

17. Beavis AL, Levinson KL. Preventingcervical cancer in the United States:

barriers and resolutions for HPVvaccination. Front Oncol. 2016;6:19

18. Perkins RB, Chigurupati NL, Apte G,et al. Why don’t adolescents finish theHPV vaccine series? A qualitative studyof parents and providers. Hum VaccinImmunother. 2016;12(6):1528–1535

19. Brewer NT, Hall ME, Malo TL, Gilkey MB,Quinn B, Lathren C. Announcementsversus conversations to improve HPVvaccination coverage: a randomizedtrial. Pediatrics. 2017;139(1):e20161764

20. Paskett ED, Krok-Schoen JL, Pennell ML,et al. Results of a multilevelintervention trial to increase humanpapillomavirus (HPV) vaccine uptakeamong adolescent girls. CancerEpidemiol Biomarkers Prev. 2016;25(4):593–602

21. Fisher-Borne M, Preiss AJ, Black M,Roberts K, Saslow D. Early outcomes ofa multilevel human papillomavirusvaccination pilot intervention infederally qualified health centers. AcadPediatr. 2018;18(2S):S79–S84

22. Gilkey MB, Dayton AM, Moss JL, et al.Increasing provision of adolescentvaccines in primary care: a randomizedcontrolled trial. Pediatrics. 2014;134(2).Available at: www.pediatrics.org/cgi/content/full/134/2/e346

23. Perkins RB, Zisblatt L, Legler A, Trucks E,Hanchate A, Gorin SS. Effectiveness ofa provider-focused intervention toimprove HPV vaccination rates in boysand girls. Vaccine. 2015;33(9):1223–1229

24. Zimet G, Dixon BE, Xiao S, et al. Simpleand elaborated clinician reminderprompts for human papillomavirusvaccination: a randomized clinical trial.Acad Pediatr. 2018;18(2S):S66–S71

25. Krantz L, Ollberding NJ, Beck AF, CarolBurkhardt M. Increasing HPVvaccination coverage through provider-based interventions. Clin Pediatr(Phila). 2018;57(3):319–326

26. Dempsey AF, Pyrznawoski J, Lockhart S,et al. Effect of a health careprofessional communication trainingintervention on adolescent humanpapillomavirus vaccination: a clusterrandomized clinical trial. JAMA Pediatr.2018;172(5):e180016

27. National Cancer Institute; ResearchTested Intervention Program. DOSE HPV:

development of systems and educationfor HPV vaccination. Available at:https://rtips.cancer.gov/rtips/programDetails.do?programId=25930477. Accessed August 1, 2019

28. Joseph NP, Bernstein J, Pelton S, et al.Brief client-centered motivational andbehavioral intervention to promote HPVvaccination in a hard-to-reachpopulation: a pilot randomizedcontrolled trial. Clin Pediatr (Phila).2016;55(9):851–859

29. Hemming K, Haines TP, Chilton PJ,Girling AJ, Lilford RJ. The steppedwedge cluster randomised trial:rationale, design, analysis, andreporting. BMJ. 2015;350:h391

30. Twisk JWR, de Vente W. The analysis ofrandomised controlled trial data withmore than one follow-up measurement.A comparison between differentapproaches. Eur J Epidemiol. 2008;23(10):655–660

31. Cameron AC, Trivedi PK.Microeconometrics: Methods andApplications. New York: CambridgeUniversity Press; 2005

32. Walling EB, Benzoni N, Dornfeld J, et al.Interventions to improve HPV vaccineuptake: a systematic review. Pediatrics.2016;138(1):e20153863

33. Swerissen H, Crisp BR. Thesustainability of health promotioninterventions for different levels ofsocial organization. Health Promot Int.2004;19(1):123–130

34. Wiltsey Stirman S, Kimberly J, Cook N,Calloway A, Castro F, Charns M. Thesustainability of new programs andinnovations: a review of the empiricalliterature and recommendations forfuture research. Implement Sci. 2012;7:17

35. Leung SOA, Akinwunmi B, Elias KM,Feldman S. Educating healthcareproviders to increase humanpapillomavirus (HPV) vaccination rates:a qualitative systematic review. VaccineX. 2019;3:100037

36. Smulian EA, Mitchell KR, Stokley S.Interventions to increase HPVvaccination coverage: a systematicreview. Hum Vaccin Immunother. 2016;12(6):1566–1588

37. Whelan J, Love P, Pettman T, et al.Cochrane update: predicting

10 PERKINS et al by guest on March 6, 2021www.aappublications.org/newsDownloaded from

sustainability of intervention effects in

public health evidence: identifying key

elements to provide guidance. J Public

Health (Oxf). 2014;36(2):347–351

38. Biancarelli DL, Drainoni M-L, Perkins RB.

Provider experience recommending

HPV vaccination before age 11 years.

J Pediatr. 2020;217:92–97

39. Uddin SG, O’Connor KS, Ashman JJ.Physician office visits by children forwell and problem-focused care: UnitedStates, 2012. NCHS Data Brief. 2016;(248):1–8

40. St Sauver JL, Rutten LJF, Ebbert JO,Jacobson DJ, McGree ME, Jacobson RM.Younger age at initiation of the humanpapillomavirus (HPV) vaccination series

is associated with higher rates of on-

time completion. Prev Med. 2016;89:

327–333

41. Goleman MJ, Dolce M, Morack J. Quality

improvement initiative to improve

human papillomavirus vaccine

initiation at 9 years of age. Acad

Pediatr. 2018;18(7):769–775

PEDIATRICS Volume 146, number 1, July 2020 11 by guest on March 6, 2021www.aappublications.org/newsDownloaded from

DOI: 10.1542/peds.2019-2737 originally published online June 15, 2020; 2020;146;Pediatrics 

Amresh HanchateAnny T.H.R. Fenton, William G. Adams, Jack A. Clark, Mari-Lynn Drainoni and

Pierre-Joseph, Terresa J. Eun, Dea L. Biancarelli, Thomas J. Schuch, Karin Leschly, Rebecca B. Perkins, Aaron Legler, Emily Jansen, Judith Bernstein, Natalie

Improving HPV Vaccination Rates: A Stepped-Wedge Randomized Trial

ServicesUpdated Information &

http://pediatrics.aappublications.org/content/146/1/e20192737including high resolution figures, can be found at:

Referenceshttp://pediatrics.aappublications.org/content/146/1/e20192737#BIBLThis article cites 34 articles, 4 of which you can access for free at:

Subspecialty Collections

_subhttp://www.aappublications.org/cgi/collection/vaccine:immunizationVaccine/Immunizationbhttp://www.aappublications.org/cgi/collection/infectious_diseases_suInfectious Diseasesubhttp://www.aappublications.org/cgi/collection/quality_improvement_Quality Improvemente_management_subhttp://www.aappublications.org/cgi/collection/administration:practicAdministration/Practice Managementfollowing collection(s): This article, along with others on similar topics, appears in the

Permissions & Licensing

http://www.aappublications.org/site/misc/Permissions.xhtmlin its entirety can be found online at: Information about reproducing this article in parts (figures, tables) or

Reprintshttp://www.aappublications.org/site/misc/reprints.xhtmlInformation about ordering reprints can be found online:

by guest on March 6, 2021www.aappublications.org/newsDownloaded from

DOI: 10.1542/peds.2019-2737 originally published online June 15, 2020; 2020;146;Pediatrics 

Amresh HanchateAnny T.H.R. Fenton, William G. Adams, Jack A. Clark, Mari-Lynn Drainoni and

Pierre-Joseph, Terresa J. Eun, Dea L. Biancarelli, Thomas J. Schuch, Karin Leschly, Rebecca B. Perkins, Aaron Legler, Emily Jansen, Judith Bernstein, Natalie

Improving HPV Vaccination Rates: A Stepped-Wedge Randomized Trial

http://pediatrics.aappublications.org/content/146/1/e20192737located on the World Wide Web at:

The online version of this article, along with updated information and services, is

by the American Academy of Pediatrics. All rights reserved. Print ISSN: 1073-0397. the American Academy of Pediatrics, 345 Park Avenue, Itasca, Illinois, 60143. Copyright © 2020has been published continuously since 1948. Pediatrics is owned, published, and trademarked by Pediatrics is the official journal of the American Academy of Pediatrics. A monthly publication, it

by guest on March 6, 2021www.aappublications.org/newsDownloaded from