Brad Therrell, NNSGRC M. Ramachandran, Georgia Julie Miller, Nebraska

or or What Do I Think About the What Do I Think About the

Performance Performance Evaluation and Evaluation and Assessment Assessment Scheme?Scheme?

Newborn Screening Self Assessment Newborn Screening Self Assessment and Improved System Quality and Improved System Quality

NEWBORN SCREENING PERFORMANCE EVALUATION AND ASSESSMENT - HRSA Cooperative Agreement

Purpose:

To support improvement of newborn screening (NBS) systems, and ultimately patient services, by developing a Program Evaluation and Assessment Scheme (PEAS).

Activities:

1. Develop multi-disciplinary project team(s)

2. Collect and assess PEAS tools/activities currently available

3. Create a comprehensive PEAS for the NBS system

4. Validate the usability of PEAS within the community

5. Update and periodically reevaluate based on user input

Discussion, Debate, … Consensus

Final Product

PEAS PROJECT OVERSIGHT/ADVICE COMMITTEE

Committee MembersWilliam Becker Association of Public Health Laboratories E. Stephen Edwards American Academy of Pediatrics Charlie Homer National Initiative for Children’s Healthcare Quality Alex Kemper University of Michigan (Evaluation Specialist) Christopher Kus Association of Maternal and Child Health ProgramsKelly R. Leight CARES Foundation, Inc. – Lay Advocate

(CAH Research, Education, and Support) Trish Mullaley Genetic Alliance – Lay Advocate

(PKU and Allied Disorders) Patricia McLaughlin Association of Women’s Health, Obstetric and

Neonatal Nurses (AWHONN) David Ross Public Health Informatics Institute

Project Overall Direction (POD)Brad Therrell NNSGRC Michele Puryear HRSA Marie Mann HRSA Harry Hannon CDC Marion Schwartz Follow-up/Education Working Group (Contractor) Carol Southard Laboratory Working Group (Contractor)

LABORATORY WORKING GROUP

Project Co-Manager:

Carol Southard New Jersey – NBS Laboratory Director

Members:

Lisa Bates Florida – NBS Laboratory Director Roger Eaton Massachusetts – Regional NBS Laboratory Director Cheryl Hermerath Oregon – Regional NBS Laboratory DirectorGary Hoffman Wisconsin – NBS Laboratory DirectorEldridge Hutcheson Texas – Laboratory Operations ManagerMark McCann Minnesota – NBS Laboratory Director John Sherwin California – NBS Laboratory Director

Oversight:

Brad Therrell NNSGRC Marie Mann HRSA Harry Hannon CDC

Ex Officio:Donna Williams NNSGRC – Texas NBS Laboratory QA

Project Co-Manager:Marion Schwartz New Jersey – Follow-up Coordinator

Members:Sharon Anderson New Jersey – Practicing Nurse Louis Bartoshesky Delaware – Physician Consultant Mary Ann Gordon Maine – Consumer Advocate Penny Hatcher Minnesota – Follow-up Supervisor (DBS/NHS) Pam King Oklahoma – Follow-up Supervisor Marcia Lavochkin New Hampshire – Follow-up Coordinator Fred Lorey California – Development/Evaluation Director Julie Miller Nebraska – Program Manager Charles Myers Louisiana – Program Manager Ellie Mulcahy Maine – Program Manager Sheila Neier Washington – Follow-up Supervisor Sonya Ross Maryland – Sickle Cell Disease Association

Observers:Irene Forsman HRSA – Newborn Hearing Screening John Eichwald CDC – Newborn Hearing Screening

OversightBrad Therrell Texas – NNSGRC Marie Mann HRSA – Contract Officer

FOLLOW-UP/EDUCATION WORKING GROUP

NEWBORN SCREENING PERFORMANCE EVALUATION AND ASSESSMENT

SELF-ASSESSMENT PERFORMANCE CHECKLISTWith Example References/Activities for Improvement

I. CROSS-CUTTINGII. PRE-ANALYTICALIII. ANALYTICALIV. POST-ANALYTICALREFERENCES

PERFORMANCE INDICATOR FINDINGS EXAMPLES/REFERENCES

Yes In Prep No

Philippine PEAS Philippine PEAS –– Poster 44 Poster 44 –– C. Padilla C. Padilla Brazil PEAS Brazil PEAS –– In Development In Development –– J. J. BotlerBotler

Georgia Newborn Screening Program

Screening Laboratory

All metabolic cases.Emory UniversityPediatrics Dept.

Division of Med.Genetics

All SCD casesGrady Hospital

Locate the childKeep under observation

Confirmatory TestingManagement

Feedback to Program

Reference TestingMedical College

of Georgia, Augusta

Sickle- cellFoundationCounseling

Hemoglobin Genetic Testing

Family Studies

HospitalsPediatric clinics

Health Dept.

Genetic Adv.Committee

Voice Responsesystem

AdministrationBudget

Director

PROGRAM EVALUATION AND ASSESSEMENT SCHEME (P.E.A.S)

SCHEME AIMS AT PROVIDING:SCHEME AIMS AT PROVIDING:

““Precise Program DefinitionPrecise Program Definition””

““Flexible program evaluationFlexible program evaluation””

A tool to A tool to ““refine and enhancerefine and enhance”” systemsystem

A A ““system assessmentsystem assessment”” tooltool

PROGRAM EVALUATION AND ASSESSEMENT SCHEME (P.E.A.S)

1.1. As a planning tool.As a planning tool.2.2. As an Evaluation tool.As an Evaluation tool.

Criteria:

SpecificMeasurableAchievableRealisticTime-bound

Criteria:

SSpecificMMeasurableAAchievableRRealisticTTime-bound

Components:Components:

GENERALGENERALPREPRE--ANALYTICALANALYTICALANALYTICALANALYTICALPOSTPOST--ANALYTICALANALYTICAL

I General Considerations:

Addresses: EducationInformation SystemsMonitoring timely and Universal ScreeningProgram Administration and Financing.Contingency plan

1)1) As a planning tool As a planning tool –– a very good check lista very good check list2)2) As an Evaluation tool As an Evaluation tool –– some issuessome issuesSS –– YesYes

MM ––YesYesAA –– May beMay beRR –– Not in Some areasNot in Some areasTT –– YesYes

Planning

25

75

YES NO IP

Perc

enta

ge

Evaluation

75

25

YES NO IP

Perc

enta

ge

1. As a Planning Tool : 1. As a Planning Tool : ““Exhaustive ListExhaustive List””2. As an Evaluation Tool: helps identify issues2. As an Evaluation Tool: helps identify issues

CommentsCommentsA A ““Mandatory data setMandatory data set”” –– Uniform formatUniform formatCompatibility with different systemsCompatibility with different systemsFire walls/alternatives?Fire walls/alternatives?Different wings of the program have different time line/Different wings of the program have different time line/--Priorities. Priorities. Vital records are completed very late. Monitoring screening to Vital records are completed very late. Monitoring screening to completion is not always possible. completion is not always possible.

-- Factual approachFactual approach-- Probability approachProbability approach

SS –– YesYesMM –– YesYesAA –– Yes/ May beYes/ May beRR –– Yes/ May beYes/ May beTT -- YesYes

Computerized Information system33

10 9

YES NO IP

78

1

YES NO IP

EducationGeneral considerations

Monitoring of Timely and Universal Screening

1.Planning tool : Useful/addresses all 1.Planning tool : Useful/addresses all possibilitiespossibilities

2.Monitoring tool : Useful2.Monitoring tool : Useful

S S –– Very SpecificVery SpecificM M –– YesYesA A –– Yes/NoYes/NoRR –– YesYesTT ---- YesYes

CommentsCommentsInitial Screening may be monitored to completionInitial Screening may be monitored to completionSubsequent screening Subsequent screening ––VitiatedVitiated by participation from private labsby participation from private labsAssignment of responsibilities/MonitoringAssignment of responsibilities/Monitoring

78

1

YES NO IP

Program Administration/Financial1.Planning tool : Useful/has limitations1.Planning tool : Useful/has limitations2.Monitoring tool: Idealistic2.Monitoring tool: Idealistic

SS –– Not Very SpecificNot Very SpecificMM –– Yes but not fullyYes but not fullyAA –– May beMay beR R –– No/some unrealisticNo/some unrealisticTT -- DonDon’’t Knowt Know

Comments:Needs revision in the light of Expanded screening/

“Fee systems’.Can be revised to make it more realistic and

meaningful.

35

15

2

YES NO IP

II PRE-ANALYTICAL CONSIDERATIONSAddressesAddresses

PersonnelPersonnelPrenatal EducationPrenatal EducationScreening Process/Special collection Screening Process/Special collection Lab safetyLab safety

276 416

1,0651,683

3,091

6,531

Date Of Birth Date Collected Time Collected Time Of Birth Birth Weight IncompleteDemography

Missing Demography

Num

ber o

f Spe

cim

ens

138

256

YES NO IP

SS –– YesYesMM –– YesYesAA –– May be/not alwaysMay be/not alwaysRR –– IdealisticIdealisticTT -- YesYes

Comments:Comments:-- Very Exhaustive Very Exhaustive -- More useful as a monitoring toolMore useful as a monitoring tool-- Program has only recommendatory roleProgram has only recommendatory role-- No actual control over preNo actual control over pre--analytic variancesanalytic variances-- Talks exhaustively about documentation of every Talks exhaustively about documentation of every

prepre--analytic activityanalytic activity

III. ANALYTICAL CONSIDERATIONSAddresses:Addresses:

TestingTestingInstrumentationInstrumentationlab. Supplieslab. SuppliesWorking EnvironmentWorking Environment

87

16

2

YES NO IP

SS –– Very SpecificVery SpecificMM –– Yes Yes AA –– Yes Yes RR –– YesYesTT -- YesYes

As a Planning tool As a Planning tool –– A very Comprehensive check listA very Comprehensive check listAs a Monitoring tool As a Monitoring tool –– Excellent check list Excellent check list

COMMENTSCOMMENTS-- Scores both as a planning and monitoring tool.Scores both as a planning and monitoring tool.-- More of the environmental More of the environmental controlscontrols monitored at the labmonitored at the lab--levellevel

IV POST- ANALYTICAL CONSIDRATIONSAddresses:Addresses:

Screening Test resultsScreening Test resultsShortShort--term followterm follow--upupFollowFollow--up support activitiesup support activities

88

44

26

YES NO IP

SS –– Exhaustively SpecificExhaustively SpecificMM –– NoNoAA –– Not in full Not in full RR –– Not practical in certain areasNot practical in certain areasTT –– YesYes

Comments:Comments:1.1. Needs to be made more realistic and achievableNeeds to be made more realistic and achievable2.2. Good details are lost by an Good details are lost by an ““idealistic approachidealistic approach””..3.3. Medical Management protocolsMedical Management protocols-- Can this be monitored by screening program?Can this be monitored by screening program?4.4. long term followlong term follow--up up ––VagueVague--Intentional flexibility?Intentional flexibility?

LOOK ! I KNEW THEY SHOULD HAVE GOT A BETTER LABORATORY

Lab.

NewbornScreeing Program

Using the PEAS for program development in Nebraska

Julie Miller, Program Manager

May 2007 NBS & Genetics Symposium

Background

Nebraska’s NBS program administered out of Dept. Health & Human ServicesLaboratory Services are contracted (Pediatrix)Follow-up (Short-term) is within the Dept.Some Treatment (long-term) is paid for by Dept. via contracts with sub-specialty clinics & metabolic foods companies

Who should complete the self-assessment tool?

Program Manager in our case, with technical lab evaluation done by the Laboratory Director

In some states will be Lab Director and Follow-up Director and/or Medical Director

Should be whomever knows the system the best

How we used it:

1) Completed the PEAS 2) Asked for program staff review3) Distributed copies to all members of the Nebr. NBS Advisory Committee (NBSAC)4) Reviewed missing or “in progress”elements at quarterly NBSAC meeting

Advisory Committee1) Reviewed and discussed

whether or not they agreed with the initial assessmentareas needing work

2) Considered resources available to address weaknessesimpact of elements not being in placeurgency of getting missing or weaker elements in place

3) Considered strategies for addressing themwork groups to concurrently address thingsprioritizing and using existing program staff to address one at a timecontract with outside help (hiring freeze)find volunteers to help

Advisory Committee1) reviewed and discussed

Whether they agreed with the initial assessmentAreas needing work

2) Considered resources available, impact of elements not being in place, urgency of getting missing or weaker elements in place

3) Considered strategies for addressing them:Work groups to concurrently address thingsPrioritizing and using existing program staff to address one at a timeContract with outside help (hiring freeze)Find volunteers to help

Work to be done…

Education “Plan”Informatics “Plan”Communications PR “Plan”Contingency PlanProcedures to “ensure” EI and LT/FU connectedShort-term Follow-up “Evaluation Plan”Long-term follow-up

Work to be done…

Education Plan Informatics PlanCommunications PR PlanContingency PlanProcedures to ensure EI and LT/FU connectedShort-term Follow-up Evaluation PlanLong-term follow-up

How it has helped:

Director of HHS was informed of Committee discussion, and the progress on development of an “All Hazards Preparedness Plan” for NBS. Supportive of use of other Dept. resources, e.g. pan-flu coordinators, development of table-top exercises etc.

Other benefits

Helped us stand back and take a comprehensive look.A lot getting done, but often not in a planned manner, with evaluation built in.Helps focus efforts.

Acknowledgements:Advisory Committee

Khalid Awad, MDLawrence Bausch, MDJohn Colombo, MDKevin Corley, MDJeanne EggerDavid Gnarra, MDJames Harper, MD Vice ChairKathryn Heldt, RDMary KisickiRichard Lutz, MD, ChairBev MortonHoward Needleman, MDSamuel Pirruccello, MDChristine Reyes, MDWilliam Rizzo, MDKathy Rosssiter, MSN, APRNJill Skrabal, RDCorri StearnesDouglas Stickle, Ph.D.William Swisher, MDB.J. Wilson, MDHobart Wiltse, MD

Nebraska NBS Staff:Krystal Baumert, Follow-up CoordinatorKaren Eveans, Follow-up SpecialistMike Rooney, Adm. Asst./PKU Foods program

Nebraska HHS Administration:JoAnn Schaefer, MD, Chief Medical Officer & Department Director Jackie Miller, DDS, Director Health ServicesPaula Eurek, Administrator, Family Health