Program Evaluation How to Effectively Evaluate Your Program Carol Pilcher Department of Entomology...
-
Upload
hunter-mcgregor -
Category
Documents
-
view
215 -
download
0
Transcript of Program Evaluation How to Effectively Evaluate Your Program Carol Pilcher Department of Entomology...
Program Evaluation How to Effectively Evaluate
Your Program
Carol Pilcher Department of Entomology
Iowa State University
Logic ModelUniversity of Wisconsin-Extension
Purpose of Logic Model
• Visual display to connect inputs, outputs and outcomes
“Helps people get on the same page”“Describes a program and why it works”“Roadmap to help you know where you
are going”
Logic Model
Inputs Outputs Outcomes-Impact
Situation
Assumptions External Factors
Priorities
What We
Invest
What We Do
Who We
Reach
What the
Short Term
Results Are
What the
Medium Term
Results Are
What the
Ultimate Impact
Is
Adapted From University of Wisconsin-Extension
Situation
• What is the problem or issue?– Background information, grant
proposal
• Why is this a problem?– Background information, grant
proposal
• Who has a stake in the problem?– Identify stakeholders
Priorities
• Determine the focus of your program– Establish priorities based on:
• Resources• Experience• Situation
– Objectives and Timeline, grant proposal
Logic Model
Inputs Outputs Outcomes-Impact
SITUATION
Assumptions External Factors
PRIORITIES
What We
Invest
What We Do
Who We
Reach
What the
Short Term
Results Are
What the
Medium Term
Results Are
What the
Ultimate Impact
Is
Inputs
• What resources will you invest?– Staff
• Number of staff and hours available
– Time• Hours reported
– Money• Allocated funds
Logic Model
Inputs Outputs Outcomes-Impact
SITUATION
Assumptions External Factors
PRIORITIES
What We
Invest
What We Do
Who We
Reach
What the
Short Term
Results Are
What the
Medium Term
Results Are
What the
Ultimate Impact
Is
Outputs-Activities
• What will you do?– Conduct workshops, meetings– Deliver services– Develop products, resources
Outputs-Activities
• Examples
Thirty-nine food safety programs were held including hand-washing demonstrations and food safety during the RAGBRAI intra-state cycling trek
Starting with a pilot of 4 districts during the 2001-2002 school year, the School Integrated Pest Management Program has trained more than 30 districts statewide.
Output-Participation
• Who will you reach?– Participants– Clients– Agencies– Decision makers– Customers
Output-Participation
• Examples
Attendance at Crop Advantage Meetings2002—933 2003—720 2004—1735
5,982 individuals were involved in parent education programs
Output-Reactions
• Reactions about course materials• Reactions about program
Output-Reactions
• Example
Of the participants that completed the survey (n=120), 98% indicated that the information presented was useful
for their farm operation
Strongly Agree
Agree Undecided Disagree Strongly Disagree
The information presented today was useful for my farm operation.
___ ___ ___ ___ ___
Logic Model
Inputs Outputs Outcomes-Impact
SITUATION
Assumptions External Factors
PRIORITIES
What We
Invest
What We Do
Who We
Reach
What the
Short Term
Results Are
What the
Medium Term
Results Are
What the
Ultimate Impact
Is
Outcomes-ImpactsShort Term Results
• Amount of learning– How much was learned during the
activity?– KASA
• Knowledge, Attitudes, Skills, Awareness
Outcomes-ImpactsKASA
• Example
Of those who returned evaluation surveys, 78% reported moderate to significantly improved understanding of the
factors affecting groundwater and surface water contamination of pesticides.
Please circle the best response for the following questions
Low Level
1
2 3 4 High Level
5
Prior to today’s program indicate your level of knowledge about: Understanding the factors affecting groundwater and surface water contamination of pesticides
1 2 3 4 5
After today’s program indicate your level of knowledge about: Understanding the factors affecting groundwater and surface water contamination of pesticides
1 2 3 4 5
Outcomes-ImpactsMedium Term Results
• Changes in actions– Behavioral changes– Changes in practices– Changes in decision making
• ExampleYear One
As a result of today’s training, will you use a drift-reduction nozzle in your sprayer to assist in reducing drift
–Adopted Prior To Training, –Plan to Adopt, –Choose Not to Adopt, –Does Not Apply
Year Two
As a result of last year’s program, do you use a drift-reduction nozzle in your sprayer to assist in reducing drift
–I Adopted –I Did Not Adopt –Does Not Apply
Outcomes-ImpactsChange in Actions
Outcomes-ImpactsChange in Actions
• Examples
As a result of the program, 50% of the participants said they would use drift reduction nozzles in their sprayers to reduce drift.
This post-training evaluation examined if participants had indeed successfully implemented new pesticide safety activities. According to the respondents, 85% of the participants said that they now use drift reduction nozzles in their sprayers.
Ninety-eight percent of the food safety program participants reported adopting one of more of the recommended food handling practices.
Outcomes-ImpactsUltimate Impacts
• Changes in conditions– Social/Health– Environment– Economic
Outcomes-ImpactsChanges in economic conditions
• Example
Approximately, 4.3 million acres were managed with the IPM practices of improved disease resistant varieties, use of safe and effective fungicides as recommended by a forecasting model, and use of the wheat midge risk map and midge scouting. A total of $105.7 million was gained by producers using these IPM practices in 2003.
Outcomes-ImpactsUltimate Impacts
IPM Focus Areas
Production Agriculture
Residential and Public Areas
Natural Resources and Recreational Environments
IPMImpactAreas
Environmental Impacts (Reduce environmental
risk)
Health Impacts(Reduce risks to health)
Economic Impacts(Cost/Benefit)
Logic Model
Inputs Outputs Outcomes-Impact
SITUATION
Assumptions External Factors
PRIORITIES
What We
Invest
What We Do
Who We
Reach
What the
Short Term
Results Are
What the
Medium Term
Results Are
What the
Ultimate Impact
Is
Evaluation
Answers the “so what” question.
What difference does your program make for the participants?
Determines the worth of a program.