Basic Principles of Program Evaluation

Corinne Datchi-Phillips, Ph.D.CEBP Learning Institute

May 26th, 2010

Breakout Session Objectives1. Participants will increase their understanding of

the purpose and goals of program evaluation.2. Participants will be able to identify what questions

they may ask to improve their correctional programming.

3. Participants will become knowledgeable about the goals of different types of program evaluation.

4. Participants will learn about existing resources they may use to guide the evaluation of their services.

5. Participants will be introduced to the recommendations of the CEBP regarding data collection in program evaluation.

Basic Principles of Program Evaluation

The goal and purposeof program evaluation

Program Evaluation• Purpose– To provide administrators with

information about program activities and outcomes

– To contribute to the delivery of quality service to people in need

Program Evaluation• Goals:– Improve correctional treatment and

supervision programs (enhance effectiveness and quality of service);

– Inform program-level decisions; – Help public officials meet the

obligation of using resources effectively by identifying and providing cost-effective programs.

Program Evaluation• Goals– To inform decisions by identifying• The need for specific services• The likelihood that these services will be

utilized• What amount of service is necessary to

meet the needs that have been identified

Program Evaluation• Goals– To inform decisions about the selection and

implementation of correctional treatment and supervision programs • Whether to continue offering a program if it is

successful• Whether to discontinue a program because it

fails to meet a need in the community• How to alter a program to make it more useful

to participants (Continuous quality improvement)– What are the limitations of the program? How can

these limitations be addressed?

Program Evaluation• Important questions about program

implementation:– Is a program implemented as planned?

• Deviations from the original plan of a program may have harmful effects or may result in no or minimal benefit.

–What are the outcomes of a program?• Reduction of criminogenic needs• Increased public safety (reduction of

recidivism rates)

Program Evaluation• Important question about

resource utilization:–What programs produce the most

favorable outcomes?• When programs are equally effective

with this type of client with this type of problem in this type of setting, the least expensive program may be selected.

Basic Principles of Program Evaluation

Common types of program evaluation

Program Evaluation• Different types

1. Need Evaluation2. Implementation Evaluation3. Outcome Evaluation4. Cost-effectiveness Analysis

Need Evaluation• Basic first step that precedes

program planning and implementation

• Goal– To identify and measure critical

unmet needs within a community• Nature and magnitude of social,

psychological, and behavioral problems• Community-based resources and

services already available to address the problems

Implementation Evaluation

• After a program has been developed and begun–What parts of the program have

been implemented or not–What are barriers to implementation–Who is being served by the program–Match between program

implementation and program plan

Outcome Evaluation• After satisfactory assessment of

program implementation– Different levels of complexity:

• Are program participants performing or functioning better ?

• Are program participants performing or functioning better than those not receiving the services?

• Does the program cause a positive change in the performance or functioning of the participants? (Causal relationships are more difficult to establish)

Cost-effectiveness Analysis

• After it has been established that a program has been implemented effectively with good outcomes– Cost-effectiveness analysis answers the

question of efficiency.• A successful program that requires a large

amount of resources may not be a viable option.• Among the programs that have a similar positive

impact on a specific social and behavioral problem, which ones cost less?– Only compare programs that are designed to achieve

the same outcomes with the same group of clients

Basic Principles of Program Evaluation

Resources

Resources• Robert Wood Johnson

Foundation’s Guide to Evaluation Primers– http://www.rwjf.org/pr/product.jsp?id

=18657

• W.K. Kellogg Foundation’s Evaluation Handbook– http://www.wkkf.org/knowledge-cent

er/resources/2010/W-K-Kellogg-Foundation-Evaluation-Handbook.aspx

Resources• The Evaluation Exchange

periodical, Harvard University– http://www.hfrp.org/evaluation/the-e

valuation-exchange/overview

• Center for Disease Control’s Framework for Program Evaluation in Public Health– http://www.cdc.gov/eval/framework.

htm

Basic Principles of Program Evaluation

Integrating program evaluation with routine practices:

The importance of ongoing data collection

Integrating evaluation into routine practices

• Program evaluation as a part of program management– Requires the consistent, ongoing collection and

analysis of information that will inform program decision making• Detailed description of the program (target population;

problem to be addressed; goals and objectives; program activities; mechanisms of change)

• After determining which questions you would like to answer about a specified program, identify what information you will need to answer these questions

• Develop a plan for collecting the information– Collect the information from multiple sources and

perspectives (clients, staff, documents, observations)

Data Gathering• CEBP recommendations (2009)– Basic data elements necessary to

perform program outcome evaluations• A survey of current community

corrections practices in the IDOC, recommendations p. 57, Appendix B p. 71 http://education.indiana.edu/Default.aspx?alias=education.indiana.edu/cebp-ri

Data Gathering• CEBP recommendations (2009)– Basic data elements that will help

answer the question “what works?” (effects of community corrections programs)• With whom?

– Type and severity of problems– Human differences (age, sex, etc.)– Cultural differences

• Under what conditions?– Treatment dosage– Combination of services/programs

Data Gathering• CEBP recommendations (2009)– Specification of community corrections

programming• Target population; problem to be addressed;

goals and objectives; program activities; mechanisms of change

– Effectiveness of community corrections programming• Program completion rates (criteria)• Measures of behavioral change (improvement

in performance and functioning)• Recidivism rates

Basic Principles of Program Evaluation

Conclusions & Questions

Program Evaluation• Essential to evaluate correctional

practices– To identify practices that are useful,

effective, and feasible under particular conditions

– To improve programming that address the needs of community corrections clients and communities

– To support accountability in community corrections• Using scientific methods and knowledge to inform

decisions and actions

Program Evaluation• Common concerns about program evaluation

– Belief that the evaluation will limit providers’ capacity to be flexible in their implementation of programs• Flexibility within broad structure of the program

– Belief that negative evaluation will result in termination of a program• Will more likely inform refinement of the program

– Belief that information will be misused (e.g., threat to promotion of staff)• Program evaluation is distinct from individual staff

assessment

Program Evaluation• Common concerns about program

evaluation– Belief that staff’s experiential knowledge will

not be taken into consideration• Staff’s observations constitute data that can

inform questions about program implementation.

– Belief that staff and administrators will no longer be able to make decisions about the way the program is offered• Program evaluation provides information that staff

and administrators can use to make changes in the way the program is offered to improve delivery.

Program Evaluation

Questions & Comments