Objectives and Indicators for MCH Programs MCH in Developing Countries.
Measuring Outcomes: Potential Approaches in Evaluating Public Health William M. Sappenfield, MD, MPH...
-
Upload
moses-wright -
Category
Documents
-
view
216 -
download
0
Transcript of Measuring Outcomes: Potential Approaches in Evaluating Public Health William M. Sappenfield, MD, MPH...
Measuring Outcomes: Potential Approaches in Evaluating Public Health
William M. Sappenfield, MD, MPHMCH EPI Program Team Leader
Division of Reproductive Health, CDC
Carolyn B. Slack, MS, RNDirector, MCH Division
Columbus Health Department
Framework for Program Evaluation in Public Health
Adopted: MMWR 48(RR11) Sept 99 Summarizes the essential elements Provides framework Clarifies the steps Reviews standards Addresses misconceptions
Essential ShiftsShifts in Public Health Planning and Policy
EFFORTEFFORT RESULTS RESULTS
RESULTS ACCOUNTABILITYRESULTS ACCOUNTABILITY
ACCOUNTABILITY ACCOUNTABILITY BUDGETS BUDGETS
Premises of Reinventing Government What gets measured gets done. If you don't measure results, you can't tell success
from failure. If you can't see success, you can't reward it. If you can't reward success, you're probably
rewarding failure. If you can't see success, you can't learn from it. If you can't recognize failure, you can't correct it. If you can demonstrate success, you can win
public support.
Osborne and Gaebler (1992: chapter 5, "Results-Oriented Government")
Planning
Doing
Assessment
Strategies
Plan
Evaluation
Monitoring
Implement
Being Effective in Public Health
Program Evaluation StepsEngage
Stakeholders
Describe the Program
Focus the Evaluation
DesignGather
Credible Evidence
Ensure Use and Share Lessons
Learned
Justify Conclusions
Standards: Utility
Feasibility Propriety Accuracy
Step 1: Engage Stakeholders
Fostering input, participation, and power-sharing among those persons who have an investmentThose involved in program operationsThose served or affected by the programPrimary users of the evaluation
Step 2: Describing the Program
NeedExpected effectsActivitiesResources
Stage of development
Context Logic model
Scrutinizing the features of the program being evaluated, including its purpose and place in a larger context.
Columbus Health Department Perinatal Program
Program MapSelf Referral Outreach Visit Pregnancy Test
Risk Assessment
Service Plan
F/U Visit
Delivery
F/U Visit
No Show Outreach
Waiting List
Referral
Columbus Health Department Perinatal Program
Program MapSelf Referral
Outreach VisitPregnancy Test
Risk Assessment
Service Plan
F/U Visit
Delivery
F/U Visit
Nursing
Service Plan
Coordination
Medical
Treatment
Social
Counseling
Nutrition
WIC
Substance Abuse
Intervention
Food Stamps
Counseling
Columbus Health Department Perinatal Program
Program MapSelf Referral
Outreach VisitPregnancy Test
Risk Assessment
Service Plan
F/U Visit
Delivery
F/U Visit
Nursing
Service Plan
Coordination
Medical
Treatment
Social
Counseling
Nutrition
WIC
Substance Abuse
Intervention
Food Stamps
Prenatal Care Entry
Reduced Risk
Reduced Stress
Quit Rates
Weight Gain
Counseling
Columbus Health Department Perinatal Program
Program MapSelf Referral
Outreach VisitPregnancy Test
Risk Assessment
Service Plan
F/U Visit
Delivery
F/U Visit
Nursing
Service Plan
Coordination
Medical
Treatment
Social
Counseling
Nutrition
WIC
Substance Abuse
Intervention
Food Stamps
Prenatal Care Entry
Reduced Risk
Reduced Stress
Quit Rates
Weight GainReduced
LBW
Counseling
Step 3: Focusing the Evaluation Design
Purpose:Gain insightChange practiceAssess impactInquiry affects participants
Components: Users Uses Questions Methods Agreements
Planning in advance where the evaluation is headed and what steps will be taken; iterative process.
Columbus Health Department Perinatal Program
Program MapSelf Referral
Outreach VisitPregnancy Test
Risk Assessment
Service Plan
F/U Visit
Delivery
F/U Visit
Nursing
Service Plan
Coordination
Medical
Treatment
Social
Counseling
Nutrition
WIC
Substance Abuse
Intervention
Food Stamps
Prenatal Care Entry
Reduced Risk
Reduced Stress
Quit Rates
Weight GainReduced
LBW
Counseling
Step 4: Gathering Credible Evidence
Compiling information that stakeholders perceive as trustworthy and relevant for answering their questions.IndicatorsSources of evidenceQuality of informationQuantity of evidenceLogistics
Step 5: Justifying Conclusions
Making claims regarding the program that are warranted on the basis of the data.Stakeholder standardsAnalysis and synthesis of findingsInterpretation of the factsJudgments Recommendations
Step 6: Ensuring Use and Sharing Lessons Learned
Ensuring that stakeholders are aware, findings are used in decisions, and participants have a beneficial experience. Designed for usePreparation for findingsFeedback from all partiesFollow-up of usersDissemination processAdditional Uses
Program Evaluation StepsEngage
Stakeholders
Describe the Program
Focus the Evaluation
DesignGather
Credible Evidence
Ensure Use and Share Lessons
Learned
Justify Conclusions
Standards: Utility
Feasibility Propriety Accuracy
Measuring Outcomes: Potential Approaches in Evaluating Public Health
Understand the Theory of Action Choose the Outcome(s) to Measure Choose the Standard or Comparison
Group
Measuring Outcomes: Potential Approaches in Evaluating Public Health
Understand the Theory of Action Choose the Outcome(s) to Measure Choose the Standard or Comparison
Group
What is the Intervention?
Bloom, 1968. The evaluation of primary prevention programs in Greenfield and Miller, Comprehensive Mental Health: The Challenge of Evaluation, University of Wisconsin Press.
“We are generally asked to evaluate an outcome of an undefined program having unspecified objectives on an often vaguely delineated recipient group whose level or variety of pathology is virtually impossible to assess, either before or after their exposure to the program.”
What is the Intervention?
Domino Approach:
What happens in the sequence to make the whole chain of events occur?
What is the Theory of Action?
Columbus Health Department Perinatal Program
Program MapSelf Referral
Outreach VisitPregnancy Test
Risk Assessment
Service Plan
F/U Visit
Delivery
F/U Visit
Nursing
Service Plan
Coordination
Medical
Treatment
Social
Counseling
Nutrition
WIC
Substance Abuse
Intervention
Food Stamps
Prenatal Care Entry
Reduced Risk
Reduced Stress
Quit Rates
Weight GainReduced
LBW
Counseling
Measuring Outcomes: Potential Approaches in Evaluating Public Health
Understand the Theory of Action Choose the Outcome(s) to Measure Choose the Standard or Comparison
Group
Choose the Outcome(s) to Measure
Example: Smoking Cessation Ask Advise Assess Assist Arrange Stages of Change Cessation Attempts Cessation During Pregnancy Improved Low Birth Weight
Choose the Outcome(s) to Measure
Relates to the Theory of Action Sufficient Evidence Measurable Accurate and Reliable Feasible in a Public Health Setting Affordable
Measuring Outcomes: Potential Approaches in Evaluating Public Health
Understand the Theory of Action Choose the Outcome(s) to Measure Choose the Standard or Comparison
Group
Choose the Standard or Comparison Group
Standard Internal Comparison
– Randomized
– Systematic External Comparison
Choose a Comparison GroupInternal
Starting Event
Pretest Posttest
Inter vention
•Randomize
•Select •Individual
•Groups
Measuring Outcomes: Potential Approaches in Evaluating Public Health
Understand the Theory of Action Choose the Outcome(s) to Measure Choose the Standard or Comparison
Group
CityMatCH Urban MCH Leadership Confluence
Measuring Outcomes: Potential Approaches in Evaluating Public Health
Case Study: “Show Me the Results!”
Midstate Health Department Perinatal Program
Program MapSelf Referral
Outreach VisitPregnancy Test
Risk Assessment
Service Plan
F/U Visit
Delivery
F/U Visit
Nursing
Service Plan
Coordination
Medical
Treatment
Social
Counseling
Nutrition
WIC
Substance Abuse
Intervention
Food Stamps
Prenatal Care Entry
Reduced Risk
Reduced Stress
Quit Rates
Weight GainReduced
LBW
Counseling