Building a Performance Improvement System in a Large

Urban Public Health Department: Linkages and Learning Collaboratives

MLC Open ForumWashington, DC

September 16th, 2010

Dawn Marie Jacobson, MD, MPHDirector, Performance ImprovementLos Angeles County Department of Public Health

Presentation Objectives• Provide an overview of performance

improvement efforts at Los Angeles County Department of Public Health

• Discuss linkages required for efficient analysis and reporting – Data– Standards/Benchmarks– Reports

• Describe the DPH Performance Improvement Learning Collaborative (PILC)

Quality Improvement—LAC DPH

Quality Improvement DivisionOffice of the Medical Director

Organizational Development and Training

Nursing Administration

Health Education Administration

Public Health Investigation

Physician Administration

Oral Health

Quality ImprovementFunctions

1. Performance Improvement

2. Professional Practice

3. Science Review

4. Service Quality

Gunzenhauser JD, Eggena ZP, Fielding JE, Smith KN, Jacobson DM, Bazini-Barakat N. The Quality Improvement Experience in a High-Performing Local Health Department, Los Angeles County. Journal of Public Health Management and Practice, 2010, 16 (1): 39-48

Performance Improvement—LAC DPH

1. Strategic Planning – determine priorities and goals

2. Performance Measurement– data management– reporting

3. Performance Improvement Projects– Modified IHI Method for Improvement– Other tools (RCA, Fishbone diagrams, etc)

Key Elements

1. Strategic Planning: Determine Priorities and Goals

• What are the priority public health issues in Los Angeles County?

• What are the behaviors and outcomes related to these issues that we want for people who live in LA County?

• How can we measure these conditions?

SPA/SPA/ProgramProgram

Strategic PlanStrategic Plan

DPH Strategic Plan

County Strategic PlanCounty Strategic Plan

LAC DPH—Strategic Planning

– Quality Improvement Division• Public Health Measures required Mission and Vision

Statements, Goal Setting, and Evidence-Based Strategies (2004-2007)

– Office of Planning• Department-Level Strategic Plan (2008-2011)

– Division and Program Level Strategic Plans• Office of Planning may provide technical assistance

2. Performance Measurement: Public Health Measures

• The LAC DPH approach based on Mark Friedman’s “Results Accountability”

• 32 operational units identified population health indicators linked to program performance measures to follow over time

• Healthy People 2010 objectives often identified and used as the “Standard” to achieve over time

• Organized by Essential Services of Public Health/NACCHO Standards/Accreditation Domains

Public Health Measures

POPULATION INDICATORS(measures of population-level

health outcomes and behaviors)

PERFORMANCE MEASURES(measures of program

effort and output)AND

Public HealthMeasures

Selecting Indicators and Measures

EffectiveStrategi

es

Strategy 1

Strategy 2

Population

Goals

Goal 1

Population

Indicators

Indicator

Indicator

Performance

Goals

Goal 1

Goal 2

Performance

Measures

Measure 1

Measure 2

Population Health Program Performance

Healthy People 2010/20

20

Federal, State, or

Local Guidelines

Community GuideClinical GuideOther Sources

NACCHOStandards

StrategicPlan

Type (ranked)Research findings: syntheses, systematic reviews, meta-analyses

Research findings: individual studies (quantitative and qualitative)

Performance data such as program evaluation or peer review reports

Demonstrated to be effective in computer modeling, simulations, or exercises

Consensus recommendations of recognized experts either local or national

Anecdotal accounts such as practices of other public health jurisdictions alleged to be effective, clinical narratives, or case reports

Decision-Making in Public Health: Evidence Review

Tier 1 Evidence

Type (unranked)

Philosophical or conceptual bases such as an ethical framework or a professional code of conduct

Regulations, laws, or public policies

Grant requirements

Community preferences

Necessary because of the political climate

Best hunches

Decision-Making in Public Health: Other Rationale

Tier 2 Other Rationale

Public Health Measures:Population Indicators

• Longer life span• Increased quality of life• Increased health equity• Less disease• Less premature death• Healthier choices• Safer environment• Healthier homes

POPULATION-LEVELHEALTH OUTCOMES

& BEHAVIORS

Public Health Measures: Performance Measures

Quantity Quality

Input / Effort

How Much Did We Do?

(#)

How Well Did We Do

It? (%)

Output / Effect

How Much

Change? (#)

Quality of Change?

(%)

1. Who are our clients?2. Which services do we

provide to our clients?3. What evidence-based

strategies will lead to positive change in our clients?

4. How can we measure if our clients are better off?

5. How can we measure if we are delivering services well?

Performance Measures

• Policies Created• People Informed• Partners Engaged• Surveillance Performed• Investigations Completed• Increased Access to Services• Client satisfaction

MEASURES OF PROGRAM

EFFORT & OUTPUT

Population Goal To reduce morbidity and mortality from vaccine-preventable diseases by improving immunization levels

Population IndicatorPercentage of children, ages 19-35 months, who are fully immunized with

one of the series of the Advisory Committee on Immunization Practices (ACIP)recommended vaccines

Effective, Evidence-Based Strategies (selected subset)1. Change provider behavior through systems change—Provider

recall/reminder systems in clinics2. Change provider behavior through education—multi-component

interventions with education3. Increase demand and access to immunizations—reduce out-of-pocket

costsPerformance Goal (NACCHO Standard 9)

Performance MeasurePercent of Immunization Program public and nonprofit clinic partners whoroutinely meet the Standards for Pediatric Immunization Practices for

provider andclient recall/reminder systems

Example: Immunization Program

Public Health Measures:Data Management

• Standardized spreadsheet for reporting data• Labeling System

– Population Indicators= letters• organized by population goals

– Program Performance Measures= numbers• organized by Accreditation domains

• Data Documentation• Standard Documentation

Public Health Measures: Data Measurement Worksheet

– Type of measure (PI or PM)– Measure name and description– How calculated– Target – Data source (Name, govt level, dept, program)– Data collection instrument– Data collection plan– NACCHO Standard (if applicable)

Common Data Sources

Population Indicators

Examples: – Los Angeles County Health Survey– LA FANS– Disease specific surveillance

systems– Vital Records– CA Health Interview Survey– OSHPD (Healthcare Utilization data)– BRFSS– YRBS– National Immunization Survey

Program PerformanceMeasures Examples:

– Casewatch (STD, AIDS)– RASSCLE (lead surveillance)– EHMIS– TRIMS (TB control)– vCMR (outbreak reporting

and investigation)– Syndromic surveillance– Clinic utilization data– Contracts and grants

management– Project-specific databases

Common Standard Sources

Population Indicators

Examples:– Healthy People– State of CA plans – County of LA plans (e.g.

Commission on HIV)– Internal DPH

Program PerformanceMeasuresExamples:

– Healthy People– CDC guidelines– State of CA guidelines or

mandates– Grant-specific guidelines– Professional associations– Internal DPH

Example: Immunization Program

Example: Immunization Program

Public Health Measures: Data Management

• Data collected two times per year• Data analyzed and reported one time per year• Option to update content of Public Health

Measures one time per year• Public Health Measures database in

development

Public Health Measures: Reporting

National Efforts1. CDC Guidelines or

Performance Measures

State Efforts1. State Performance

Measures2. Mandates and

Regulations

County Efforts1. Performance Counts!2. County Progress Report

Department Efforts1. Annual Performance

Report2. PI Project Reports

LAC DPH Annual Performance Report

• Internal report of a subset of Population Indicators and Performance Measures

• Includes:– Department-Level Report Card – Program-Level Performance Snapshots

• In-Person Progress Review with Director and Health Officer

Public Health Report Card

Public Health Report Card

Public Health Measures: Reporting

• Future– Linked to strategic plan objectives– More frequent reports using an automated database– Portfolio of services by NACCHO Standards– Accreditation Preparation

Total

NS1

NS2 NS3 NS4 NS5

NS6 NS7 NS8 NS9

NS10

NS11

PIs 224

PMs 736 51 (7%)

100 (14%)

110 (15%)

34(5%)

22 (3%)

36 (5%)

142 (19%)

92 (13%)

61 (8%)

36 (5%)

87 (12%)

3. Performance Improvement Projects

Since repeated measurement by itself is not enough toimprove public health practice....

What are common processes in our Department? How can we share best practices in common processes?

What support do staff need to use PI methods (e.g. rapid cycle tests) in practice?

How do we spread a successful PI approach throughout the Department?

Performance Improvement Projects

A Learning Collaborative Approach*• Create an internal performance improvement learning

collaborative (PI LC) of a diverse group of DPH units• Teams represent 8 of 32 department Divisions/Programs• Teams learn and work together for a 10 month period• Teams apply common PI methods to improve a priority area

selected by their respective units

*This project is part of the “Building the Evidence for Quality Improvement in Public Health ” grant program funded by the Robert Wood Johnson Foundation.The RAND Corporation is providing training and evaluation support.

PI Method and Tools:The IHI Model for Improvement... Plus

1. Set the Aim• Population health improvement• Customer or service improvement

2. Measure Performance• Population Indicators• Program Performance Measures

3. Map the Process4. Make Changes for Improvement

• Evidence Review and Best Practices• PDSA cycles

5. Apply other Tools (RCA, Fishbone diagrams, etc.)

Set the Aim: PI LC Team Aims• Improve Provider/Contractor Performance

– Office of AIDS Policy and Programs– Children’s Health and Disability Prevention Program– Tobacco Control and Prevention Program– Substance Abuse Prevention and Control– Emergency Preparedness and Response

• Inform and Engage Community Stakeholders– Office of Senior Health– Acute Communicable Disease Control– Office of Women’s Health

Measure Performance: PI LC Project Metrics• Population IndicatorsExamples:

– % of children who qualify for the CHDP program who receive needed follow- up care

– Community incident rates of reptile associated salmonella– Community rates of CVD among women

• Program MeasuresExamples:

– % of CHDP forms with a condition needing referral that have a referral identified

– % of Early Childhood Education providers receiving the photonovela intervention

– # of new callers to hotline per week

Percent of current adult smokers and current youth smokers in Los Angeles County (2001-2006)

14.3% 13.9% 14.3%14.5% 14.4%11.8%

0%

5%

10%

15%

20%

2001 2002 2003 2004 2005 2006Calendar Year

Perc

ent

TCPP-A Percent of adults TCPP-B Percent of youth

HP 2010 Standard(16%)

HP 2010 Standard(12%)

Data not collected where missing

Data Sources: 1) Los Angeles County Health Survey (LACHS), LAC DPH, OHAE and 2) Youth Risk Behavior Survey (YRBS), CDC, NCCDPHP

Number of jurisdictions adopting a legislative-based policy that prohibits smoking in outdoor areas (00-01 to 09-10)

0 1 2 1

5 5

10

6 6

3

0

2

4

6

8

10

12

00-01 01-02 02-03 03-04 04-05 05-06 06-07 07-08 08-09 09-10Fiscal Year

Num

ber

TCPP 5-1 Number of Jurisdictions

DPH and State of CA Standard(4 jurisdictions per 3-year funding cycle)

1st Funding Cycle = 4 3rd Funding Cycle = 152nd Funding Cycle =

Data Source: Internal Records, LAC DPH, TCPP

Map the Process: Examples

Make Changes for Improvement:PDSA and Other Tools

Average score January 2010(scale of 1-4: 4= strongly agree,

1= strongly disagree)

Average score April 2010(scale of 1-4: 4= strongly agree,

1= strongly disagree)

1. Task Force meetings are well facilitated. 3.67 3.8

1. Task Force meetings provide an effective learning forum for my campaign.

3.63 3.7

1. The ideas/strategies discussed at the meetings are helpful to my campaign.

3.75 3.9

1. Ideas/strategies shared by other task force members are helpful to my campaign.

3.71 3.8

1. Information provided by TCPP staff (e.g. billing, announcements, campaign strategies, etc.) are useful.

3.62 3.9

PDSA cycles to improve subcontractor satisfaction with Task Force meetings (complete)

Make Changes for Improvement:PDSA and Other Tools

Average score April 2010(scale of 1-4: 4= very helpful, 1= not helpful)

Community assessment 3.5

Policy campaign strategy 3.8

Coalition building/ broadening 3.6

Policy campaign implementation and policy adoption 3.6

Policy implementation and enforcement 3.4

PDSA cycles to improve subcontractor training for communitycampaigns to pass tobacco policies and ordinances (in progress)

Pareto Chart:Adult smoking rate by SPA

2007

0

5

10

15

20

25

2007

HP 2010 goal

PI LC Evaluation Metrics• Improve project team metrics• Increase staff knowledge and use of QI methods

– % of senior managers reporting they are aware of the 4-step Model for Improvement

– % of senior managers who say they are proficient in selected quality improvement methods and tools

• Disseminate QI methods across the Department– % of DPH Division/Program Directors who report using

rapid-cycle PDSA to improve performance in a priority area each year

– % of staff who report they are encouraged to take risks when implementing QI projects

Based on: 1) Senior Manager Survey of QI culture, QI knowledge and readiness for change; 2) Key Informant Interviews of DPH Executives; 3) Monthly reports from the 8 PI LC project teams; 4) DPH Annual QI Report Card

PI LC Early Lessons Learned• Successes

– Team engagement with learning sessions– Improved understanding of internal processes and links

to key measures with process mapping– Individual team coaching

• Challenges– Competing priorities (e.g., H1N1 response)– Doing rapid small scale cycles is a very new concept– Lack of readily available, validated measures and best

practice tools for team aim areas

Performance Improvement Projects: Future Plans

• Have all operational divisions and units working on at least one PI project each year

• PI to track PI projects• Inclusion of key PI project measures in the

Public Health Measures• Reporting PI project results

– Monthly for selected programs– Yearly summary for all others

Annual TimelineAnnual Timeline

Jan

Feb Mar Apr May June July Aug Sep Oct Nov Dec

Add/Drop//Modify

Public HealthMeasures

DataUpdat

e

Data Analysis and Review

PrepareReports

DataUpdate

Data Analysisand Review 

Prepare DPHand CEOPerformanceReports

Progress Reviews with Health Officer

Performance Improvement Training

Performance Improvement Projects

Summary

• A large health department needs to link many sources of data, standards, and reporting processes to build an efficient performance improvement system– This takes time to do properly– Best with department-wide participation – Need to communicate effectively across levels of

government and understand a wide variety of unit demands

Summary

• A learning collaborative approach is essential to explore common processes and small tests of change– Brings PI champions together which generates

enthusiasm– Maximizes learning and sharing– Promotes a culture of openness and transparency– Creates a “centralized” opportunity for technical

assistance and coaching

Questions and Discussion