Update onPerformance Measures Pilot

and Development of the Cancer Plan Index

Presented byDeborah Porterfield, MD, MPH

RTI International

Presented atCCC Program Directors Meeting

Atlanta, GA, May 13, 2008

RTI International is a trade name of Research Triangle Institute

3040 Cornwallis Road ■ P.O. Box 12194 ■ Research Triangle Park, NC 27709Phone 919-541-1293 e-mail dporterfield@rti.orgFax 919-541-6683

Outline

Performance measures pilotBackground and purposeMethodsExamples of preliminary results and issues raised in

the pilotNext steps

Cancer Plan IndexBackground and goalsMethods and where we areNext steps

Background: Performance Measures

Since 1998, the Division of Cancer Prevention and Control (DCPC) has provided funding to states, tribal organizations, and jurisdictions to conduct comprehensive cancer control (CCC) in their areas.

Evaluation was part of the Building Blocks and has been stressed by CDC from the beginning

In 2003 the DCPC initiated a process of developing an evaluation framework and performance measures for all CCC programs

Now CDC is asking programs to submit information on performance measures

Developed by CDC to assess recipient’s progress toward reaching required Program Activities One assessment tool for the entire US, including

Tribes and Territories, and USAPI Jurisdictions

Performance is judged by CDC through various methods including a Performance Measurement Assessment tool

“Measures of Effectiveness” are to be listed on the work plan and can be whatever programs decide. CDC does not “judge” them in any way.

Purpose of Performance Measurement

To establish accountability for public investments in cancer prevention and control;

To document the outcomes and accomplishments of funded programs and partners; and,

To facilitate a quality improvement process at the program level.

Further Purposes of Performance Measures for CDC

General information on CCC nationallyEx: The number of sectors, organizations and

individuals meeting around the CCC tableEx: Funds and in-kind other than from CDC

General knowledge of the program to help in technical assistanceEx: Sharing of satisfaction surveysEx: Proportion of interventions that are truly

“evidence-based”

Further Purposes of Performance Measures for CDC

Monitoring of the most important activitiesEx: Reviewing data at specified intervalsEx: Coalition representing “all” of your people

Documenting outcomes that we can be used to show that CCC “works”

At this point, results from the Performance Measurement worksheet are not being used for funding decisions

Methods

Advisory workgroupWithin CCC, both Program Evaluation and

Scientific Support teams represented

Framework based on CCC logic model

Draft performance measures

Extensive review

Development of a worksheet

Pilot of worksheet in 2008

Preliminary Results and Issues Raised

Year 01 Performance Measures

Response Rate

48 of 51 worksheets for states and D.C.

6 of 6 worksheets for tribes

8 of 10 worksheets for USAPI jurisdictions

Total of 62 of 67 worksheets were received

Update on Analysis

Focused review of USAPI jurisdictions and tribes already complete

Data entry underway for states

Numbers presented today represent 45 of the states, tribal organizations and USAPI jurisdictions

Response Burden

Median of 10 hours to complete the worksheet (range of 3–100 hours)

Some Potential Performance Measures Appeared to Go Smoothly

1.3 Assessment of members’ satisfaction with aspects of the partnership was conducted

62% of programs YES

2.1 Assessment of cancer burden using specified data sources and appropriate periodicity was conducted

96% of programs YES

6.1 The program has a written evaluation plan 64% of programs YES

And Some Additional “Informational” Measures Appeared to Go Smoothly

3.1 Amount of non-CDC funds Mean $2.3 million

Median $158 k

3.2 Amount of in-kind contributions Mean $642 k

Median $31 k

And Some Questions Need Work

5.1 Proportion of partner organizations implementing interventions directly related to the plan

How to know?

What counts?

5.2 Proportion of plan objectives being addressed

How to know?

5.3 Proportion of interventions that are evidence based

Definition of evidence based needs more discussion

7.1 Number and list of policy changes Need criteria for what “counts” — perhaps also in-process activities?

General and Measure-specific Revisions

Refinements in instructions

Calculation of proportionsEven if the performance measures is a proportion,

the worksheet will not ask it as such

Suggestions for additional sectors in 1.1

Next Steps

Analysis of all worksheets

Engage CCC Branch to review the data and suggest revisions to the worksheet

Continued involvement of programs in refinement of measures and worksheets for Year 02

Ongoing work of the CDC/RTI workgroup to refine the worksheet

Determination of benchmarks or standards

Inclusion of revisions in next funding announcement in fall/winter 2009

Conclusions

Performance measurement is an important tool for public health agencies to establish accountability and a culture of quality improvement.

With further refinement of the measures and evolution of a system of routine measurement, this activity has the potential to stimulate improvements in CCC at the program level and to document nationwide progress in CCC efforts

Cancer Plan Index

Background

A State Plan Index was developed by the Obesity program at CDC to review existing plans and provide guidance for new plan development

2 articles published in Preventing Chronic Disease

DCPC interested to adapt this methodology for cancer plans

Goals of the Cancer Plan Index Activity

Describe the current status of plans

Inform guidance from the DCPC to programs on CCC plans

Provide information to programs who are currently writing or revising their plans

Provide information for one of the performance measures in the current funding announcement

Methods for the Cancer Plan Index

Advisory workgroup

Program volunteers’ extensive review

Workgroup reworked components and added items

(currently) Review by larger group of program volunteers

Next Steps

Pilot in summer 2008

Review of all existing cancer plans

Outputs:Report with descriptive information about current

plansRevised instrument that will be available to

programs Guidance from CCC on plan content

Future performance measure for CCC programs

Questions?