Program Evaluation

The use of scientific methods to judge and improve the planning, monitoring,

effectiveness, and efficiency of health, nutrition, and other human service programs

Why Evaluate a Program?

• See table 10-1, page 309, Boyles and Morris

Types of Program Evaluation

• Process evaluation

• Impact or outcome evaluation

• Fiscal or efficiency evaluation

Process Evaluation

• Evaluate process objectives

• Provides information for why program may or may not have reached its outcome objectives

• If program is delivered from a variety of sites, provides information on why some sites may have been more successful than others

Six Steps for Program Evaluation

• 1. Determine objectives of program

• Evaluate for:

• a. Appropriateness of objectives

• b. Effectiveness in meeting objectives

• c. Efficiency of program

• d. Side effects of program

Steps in Program Evaluation

• 2. Determine characteristics to be measured

• Measurements should be:

• Valid

• Reliable

• Precise

Steps in Program Evaluation

• 3. Measure characteristics

Steps in Program Evaluation

• 4. Make comparisons

• May use:

• Control groups

• Similar groups

• Standards

• Pre vs post measurements

Steps in Program Evaluation

• 5. Draw conclusions

• 6. Make recommendations

Common Biases Introduced During Evaluations

• Selection

• Testing

• History

• Maturation

• Halo effect

Evaluation Design

• 1. Experimental design

• 2. Quasi-experimental design

• 3. Non-experimental design

Steps for experimental design

• 1. Experimental and control groups randomly assigned

• 2. Each group measured

• 3. Intervention or program provided

• 4. Groups measured again--if experimental group improved more than control, program was successful

Examples of Designs of True Experiments

• Pre-test post-test control group design

• R O X O

• R O O

Examples of Designs of True Experiments

• After only control group

• R XO

• R O

Examples of Experimental Design

• Solomon 4 group

• R O X O

• R O O

• R X O

• R O

Quasi-experimental design

• Steps similar to experimental, but rigid control not met.

• Random selection may not be done

• Subjects may be volunteers

• Nonequivalent control groups may be used

Nonexperimental design

• Random selection not used

• No control group or nonequivalent control group used

Examples of Non-experimental design

• After only or one-shot case study

• X O

• Nonequivalent control group study

• X O

• O

• Pre-test-Post-test design

• O X O

Fiscal or Efficiency Evaluations

• Cost-benefit analysis

• Cost-effectiveness analysis

Cost benefit analysis

• Decision making framework used in allocating resources among competing uses.

• Both costs and benefits are expressed in dollars

Costs

• Direct Costs --Cash expenditures

• Indirect Costs

– All other costs such as

– Spillover effects

– Costs to client

– Costs to organization not covered by program

• Opportunity costs

• Intangible costs--grief, suffering pain

Benefits

• All costs that would be avoided if the program were in effect

• Direct benefits--values of resources which the program saves

• Negative benefits

• Indirect benefits--other costs averted

• Intangible benefits--happiness, bonding from breastfeeding

Discount rate

• Based on deferred benefits

Cost-Benefit Analysis of Attending School

• Direct costs

• Indirect costs

• Intangible costs

• Direct benefits

• Indirect benefits

• Intangible benefits

• Discount rates

Cost effectiveness analysis

• Determines the most efficient way of meeting a predetermined set of objectives

• Costs measured in dollars

• Effectiveness measured by outcomes, e.g.. lives saved, increase in birth weight, etc

Communicating Evaluation Results

• See pages 322-326