Presented by:Presented by:

Elaine A. Borawski, Ph.D.Elaine A. Borawski, Ph.D.Center for Health Promotion ResearchCenter for Health Promotion Research

Dept. of Epidemiology and BiostatisticsDept. of Epidemiology and Biostatistics

Case Western Reserve UniversityCase Western Reserve University

School of MedicineSchool of Medicine

CFHS Evaluation Workshop, January 2000CFHS Evaluation Workshop, January 2000

EVALUATIONEVALUATION

PRESENTATION OVERVIEWPRESENTATION OVERVIEW

Demand for outcome evaluations and Demand for outcome evaluations and program accountability. program accountability.

Different types of evaluations.Different types of evaluations. Developing goals and objectives.Developing goals and objectives. How to operationalize (measure) How to operationalize (measure)

objectives.objectives. Preparing your CMH grant application. Preparing your CMH grant application.

DemandDemand

Increasing demand for agencies to not Increasing demand for agencies to not only be accountable, but demonstrate only be accountable, but demonstrate that what they do is effective.that what they do is effective.

The number of people served is no The number of people served is no longer sufficient evidence of longer sufficient evidence of accountability.accountability.

Funding sources (Federal, State, Local Funding sources (Federal, State, Local and Foundations) tying money to and Foundations) tying money to outcomes.outcomes.

HAVE NO FEARHAVE NO FEAR

No need to fear evaluations.No need to fear evaluations. A good evaluation can be your best tool A good evaluation can be your best tool

for program development and for program development and refinement.refinement.

You know your programs the best.You know your programs the best. If you believe your program works you If you believe your program works you

just need to figure out how to just need to figure out how to measure it. measure it.

Include evaluation right from the start.Include evaluation right from the start.

Types of EvaluationsTypes of Evaluations

Two Basic Types of Evaluations:Two Basic Types of Evaluations:

Program Monitoring and Program Monitoring and Accountability Accountability

Impact/Outcome AssessmentImpact/Outcome Assessment

Monitoring/AccountabilityMonitoring/Accountability

Addresses Two Key Questions:Addresses Two Key Questions:

Is program reaching appropriate Is program reaching appropriate target population?target population?

Is delivery of program/services Is delivery of program/services consistent with program design consistent with program design specifications (are we doing what specifications (are we doing what we said?) we said?)

Purpose of MonitoringPurpose of Monitoring

AccountabilityAccountability• Have a responsibility to demonstrate that Have a responsibility to demonstrate that

you are on top of your project and can you are on top of your project and can document you are doing what you said you document you are doing what you said you would do.would do.

Program ManagementProgram Management• No matter how well planned a program No matter how well planned a program

may be, unexpected results and unwanted may be, unexpected results and unwanted side effects emerge -- to discover this early side effects emerge -- to discover this early on is critical. on is critical.

Types of Program Types of Program MonitoringMonitoring

Program CoverageProgram Coverage (is program (is program reaching its intended target reaching its intended target

population?)population?)

Program/Service DeliveryProgram/Service Delivery (congruence between program plan (congruence between program plan and actual program implementation)and actual program implementation)

Fiscal MonitoringFiscal Monitoring (how is money (how is money spent? Do benefits justify the cost?)spent? Do benefits justify the cost?)

Legal/Regulatory Monitoring Legal/Regulatory Monitoring

Program CoverageProgram Coverage Essential to your proposal is defining your Essential to your proposal is defining your

target population.target population. But, you must also have a procedure for But, you must also have a procedure for

determining the extent to which targets determining the extent to which targets actually participate.actually participate.

Two issues: coverage and bias.Two issues: coverage and bias. Coverage: does overall participation reach Coverage: does overall participation reach

the expected levels of participation?the expected levels of participation? Bias: Are some subgroups covered more Bias: Are some subgroups covered more

densely than others?densely than others?

Program CoverageProgram Coverage

Proposal Points:Proposal Points:• Clearly define your target population.Clearly define your target population.• Clearly describe how you will keep track Clearly describe how you will keep track

of program participants (record keeping)of program participants (record keeping)• Describe how you will periodically Describe how you will periodically

monitor participation rates and patterns.monitor participation rates and patterns.• Describe how will you keep track of Describe how will you keep track of

those who drop out of program/service those who drop out of program/service (attrition rates).(attrition rates).

Delivery of ServicesDelivery of Services

When a program fails to show an impact it When a program fails to show an impact it may be due to failure to deliver may be due to failure to deliver intervention or service in the ways specified intervention or service in the ways specified in the program design.in the program design.

Three types of implementation failures:Three types of implementation failures:• No treatment/not enough treatmentNo treatment/not enough treatment• Wrong treatment deliveredWrong treatment delivered• Treatment is unstandardized/uncontrolled Treatment is unstandardized/uncontrolled

or varies across target populations.or varies across target populations.

Service DeliveryService Delivery

Proposal Points:Proposal Points:• Clearly define each service/program Clearly define each service/program

componentcomponent• What are the tangible features of the What are the tangible features of the

program and its setting?program and its setting?• Describe how you will monitor each Describe how you will monitor each

component’s implementation.component’s implementation.

Fiscal/LegalFiscal/Legal

Fiscal MonitoringFiscal Monitoring• Can you match your dollars to specific Can you match your dollars to specific

participants or programs?participants or programs?• How will you assess the cost/benefit of the How will you assess the cost/benefit of the

program?program? Legal/Regulatory MonitoringLegal/Regulatory Monitoring

• Do you need to comply to any legal or Do you need to comply to any legal or regulatory requirements?regulatory requirements?

• How will you monitor and report these?How will you monitor and report these?

Impact/Outcome Impact/Outcome AssessmentAssessment

Does the intervention or Does the intervention or

program produce the program produce the

intended results?intended results?

PREREQUISITES PREREQUISITES

Objectives must be clearly defined and Objectives must be clearly defined and well articulated well articulated beforebefore program begins. program begins.

The evaluation plan (and how The evaluation plan (and how outcomes will be measured) must be outcomes will be measured) must be on board when your program begins.on board when your program begins.

The program/service/intervention must The program/service/intervention must be sufficiently well implemented. be sufficiently well implemented.

If you wait….If you wait….

You may miss the most significant You may miss the most significant change your program makes.change your program makes.

You may find you can’t measure the You may find you can’t measure the program as you’ve defined it.program as you’ve defined it.

You may miss the opportunity to make You may miss the opportunity to make changes in your program that could changes in your program that could benefit your participants.benefit your participants.

Some data cannot be collected Some data cannot be collected retrospectively.retrospectively.

Where to start?Where to start?

Evaluate your program goals and Evaluate your program goals and objectives.objectives.• Are they realistic and attainable? Are they realistic and attainable? • Are they measurable?Are they measurable?• Can you realistically collect the data Can you realistically collect the data

needed?needed?• Do the goals and objectives actually Do the goals and objectives actually

match the program plan?match the program plan? If not, revise your goals and objectives.If not, revise your goals and objectives.

Doomed to FailDoomed to Fail

Program: Early intervention of Program: Early intervention of pregnant teen mothers, to assure pregnant teen mothers, to assure prenatal care and reduce premature prenatal care and reduce premature and low birthweight births.and low birthweight births.

Goal: To decrease rate of Goal: To decrease rate of prematurity and low birthweight prematurity and low birthweight births in the county.births in the county.

Goals and ObjectivesGoals and Objectives

Goals: Abstract, idealized statements Goals: Abstract, idealized statements of desired outcomes.of desired outcomes.• Increase the rate of prenatal care among Increase the rate of prenatal care among

teenage mothers.teenage mothers. Objectives: Operationalization of Objectives: Operationalization of

desired outcomes with a measurable desired outcomes with a measurable criteria for success. criteria for success. • Participants will meet at least 75% of Participants will meet at least 75% of

scheduled prenatal appointments. scheduled prenatal appointments.

Program vs. Impact GoalsProgram vs. Impact Goals

Program Goal:Program Goal: Participants will meet Participants will meet at least 75% of scheduled prenatal at least 75% of scheduled prenatal appointments appointments

Impact Goal:Impact Goal: Participants will have Participants will have significantly higher number of significantly higher number of prenatal visits when compared to prenatal visits when compared to other teen mothers from the same other teen mothers from the same geographic and sociodemographic geographic and sociodemographic backgrounds.backgrounds.

Better than status quoBetter than status quo

Your goals need to be defined in terms of Your goals need to be defined in terms of both program objectives and impact both program objectives and impact objectives.objectives.

For strongest impact assessments, you For strongest impact assessments, you need a comparable population who did need a comparable population who did not receive the treatment/program not receive the treatment/program (controls).(controls).

This data may be derived from existing This data may be derived from existing data sources.data sources.

Types of Study DesignsTypes of Study Designs

Ideal: Randomized, case/control studyIdeal: Randomized, case/control study• One group = control = alternative One group = control = alternative

treatment or delayed treatment.treatment or delayed treatment.• Other group = case/experimental = Other group = case/experimental =

receive treatment/interventionreceive treatment/intervention• Outcomes observed in both groupsOutcomes observed in both groups• Any differences attributed to treatment or Any differences attributed to treatment or

intervention.intervention.

Often difficult to attain, particularly Often difficult to attain, particularly randomization.randomization.

Control SamplesControl Samples

Compare against reported data.Compare against reported data. No treatment controls not necessary.No treatment controls not necessary. Compare against a group who will Compare against a group who will

get program at later time (delayed get program at later time (delayed intervention).intervention).

Compare against group who gets a Compare against group who gets a different type of different type of program/intervention.program/intervention.

One Shot DesignOne Shot Design

PretestPretest Post-test Post-test

Initial Initial

ChangeChange

CaseCase XX22

INTERVENTIONINTERVENTION

Pre/Post Test Design Pre/Post Test Design (no (no control)control)

PretestPretest Post-test Post-test

Initial Initial

ChangeChange

CaseCase X X11 XX22

INTERVENTIONINTERVENTION

Case-Control with Pre/PostCase-Control with Pre/Post

PretestPretest Post-test Post-test

Initial Initial

ChangeChange

CaseCase X X11 XX22

ControlsControls Y Y11 YY22

INTERVENTIONINTERVENTION

Sustained ChangeSustained Change

PretestPretest Post-test Post- Post-test Post-

test test Initial Initial Sustained Sustained

ChangeChange Change Change

CaseCase X X11 XX22 XX33

ControlsControls Y Y11 YY22 YY33

INTERVENTIONINTERVENTION

What will this tell us?What will this tell us?

With a case/control comparison, and With a case/control comparison, and pre/post test design, we can make pre/post test design, we can make the assumption that any change the assumption that any change that occurs in the intervention that occurs in the intervention group (and not the controls) is due group (and not the controls) is due to our program intervention.to our program intervention.

Any element of the design left out Any element of the design left out adds question to this assumption.adds question to this assumption.

Program ExampleProgram Example

Teen Pregnancy Prevention ProgramTeen Pregnancy Prevention Program Primarily school-based programsPrimarily school-based programs Target population: 7th gradersTarget population: 7th graders Includes both safer sex and Includes both safer sex and

abstinence centered education abstinence centered education programs.programs.

Classes conducted with same gender Classes conducted with same gender and mixed classes.and mixed classes.

Teen Pregnancy Teen Pregnancy PreventionPrevention

Goals:Goals:• To reduce the proportion of students To reduce the proportion of students

who engage in sexual activity through who engage in sexual activity through continued abstinence or postponing continued abstinence or postponing sexuality (measured as both intention sexuality (measured as both intention and actual behavior).and actual behavior).

• To increase communication between To increase communication between parents and children on subjects of parents and children on subjects of dating and sexual relationships.dating and sexual relationships.

Teen Pregnancy Teen Pregnancy PreventionPrevention

• To increase knowledge about To increase knowledge about transmission of AIDS/HIV.transmission of AIDS/HIV.

• To reduce supportive beliefs of To reduce supportive beliefs of stereotypic sexual behaviors.stereotypic sexual behaviors.

• To increase knowledge and To increase knowledge and understanding of linkages between understanding of linkages between substance use and sexual behavior.substance use and sexual behavior.

• To increase student’s sense of personal To increase student’s sense of personal control in resisting sexual urges and control in resisting sexual urges and advances.advances.

Program GoalsProgram Goals

80% of abstinent students will report 80% of abstinent students will report continued abstinence from the beginning continued abstinence from the beginning to the end of the program period. to the end of the program period.

50% of sexually active students will 50% of sexually active students will report being abstinent during the report being abstinent during the program period.program period.

But what % would have remained But what % would have remained abstinent without the program?abstinent without the program?

Study DesignStudy Design

Participating schools were matched on risk Participating schools were matched on risk behaviors (from previously collected data).behaviors (from previously collected data).

For each matched pair, intervention school For each matched pair, intervention school received program in fall, control school in received program in fall, control school in spring.spring.

Pre and posttest questionnaires were Pre and posttest questionnaires were designed to measure the six goals.designed to measure the six goals.

Identifiers were used to match individual Identifiers were used to match individual pre and posttest data.pre and posttest data.

Study DesignStudy Design

Pre and Posttest were given in both Pre and Posttest were given in both intervention and control schools intervention and control schools during same time period.during same time period.

First posttest is given after First posttest is given after intervention is provided.intervention is provided.

Intervention posttest also included Intervention posttest also included specific questions concerning specific questions concerning satisfaction with program.satisfaction with program.

Study DesignStudy Design

Control posttests included questions Control posttests included questions about any outside programs that about any outside programs that students may have attended. students may have attended.

Booster classes are provided to one-Booster classes are provided to one-half the intervention schools.half the intervention schools.

Second posttest is given four months Second posttest is given four months after first posttest to test sustained after first posttest to test sustained change.change.

Your ProposalYour Proposal

Clearly define your evaluation plan.Clearly define your evaluation plan. Clearly define goals, program objectives Clearly define goals, program objectives

and impact measures. and impact measures. Expect to designate 8-10% of your Expect to designate 8-10% of your

budget to program monitoring and budget to program monitoring and impact assessment.impact assessment.

Plans for technical and programmatic Plans for technical and programmatic assistance for all grant recipients is assistance for all grant recipients is being planned.being planned.