Evaluation in the Field: Putting Concepts into Action Janet Myers, PhD MPH Richard Vezina, MPH CAPS...
-
date post
20-Dec-2015 -
Category
Documents
-
view
219 -
download
2
Transcript of Evaluation in the Field: Putting Concepts into Action Janet Myers, PhD MPH Richard Vezina, MPH CAPS...
![Page 1: Evaluation in the Field: Putting Concepts into Action Janet Myers, PhD MPH Richard Vezina, MPH CAPS HIV Prevention Conference April 21, 2006.](https://reader036.fdocuments.us/reader036/viewer/2022062407/56649d445503460f94a212bf/html5/thumbnails/1.jpg)
Evaluation in the Field: Evaluation in the Field:
Putting Concepts into Putting Concepts into ActionAction
Janet Myers, PhD MPHJanet Myers, PhD MPHRichard Vezina, MPHRichard Vezina, MPHCAPS HIV Prevention ConferenceCAPS HIV Prevention ConferenceApril 21, 2006April 21, 2006
![Page 2: Evaluation in the Field: Putting Concepts into Action Janet Myers, PhD MPH Richard Vezina, MPH CAPS HIV Prevention Conference April 21, 2006.](https://reader036.fdocuments.us/reader036/viewer/2022062407/56649d445503460f94a212bf/html5/thumbnails/2.jpg)
Overview
Warm-up Where does Evaluation fit? Approaches to Evaluation Examples Q&A: Evaluating Your Programs
![Page 3: Evaluation in the Field: Putting Concepts into Action Janet Myers, PhD MPH Richard Vezina, MPH CAPS HIV Prevention Conference April 21, 2006.](https://reader036.fdocuments.us/reader036/viewer/2022062407/56649d445503460f94a212bf/html5/thumbnails/3.jpg)
Warm-Up
![Page 4: Evaluation in the Field: Putting Concepts into Action Janet Myers, PhD MPH Richard Vezina, MPH CAPS HIV Prevention Conference April 21, 2006.](https://reader036.fdocuments.us/reader036/viewer/2022062407/56649d445503460f94a212bf/html5/thumbnails/4.jpg)
Who here is …
Evaluator?
Service Provider?
Administrator?
![Page 5: Evaluation in the Field: Putting Concepts into Action Janet Myers, PhD MPH Richard Vezina, MPH CAPS HIV Prevention Conference April 21, 2006.](https://reader036.fdocuments.us/reader036/viewer/2022062407/56649d445503460f94a212bf/html5/thumbnails/5.jpg)
4 Questions:
What are the benefits of evaluating your programs?
What are the challenges to evaluating your programs?
What needs (besides $) do you have in order to plan/conduct evaluation?
What resources do you have/use for evaluation?
![Page 6: Evaluation in the Field: Putting Concepts into Action Janet Myers, PhD MPH Richard Vezina, MPH CAPS HIV Prevention Conference April 21, 2006.](https://reader036.fdocuments.us/reader036/viewer/2022062407/56649d445503460f94a212bf/html5/thumbnails/6.jpg)
Where does Evaluation fit?
![Page 7: Evaluation in the Field: Putting Concepts into Action Janet Myers, PhD MPH Richard Vezina, MPH CAPS HIV Prevention Conference April 21, 2006.](https://reader036.fdocuments.us/reader036/viewer/2022062407/56649d445503460f94a212bf/html5/thumbnails/7.jpg)
Mission
Goals
Objectives
Activities
Outcomes
Impact
Program & Process EvaluationIMPLEMENTATION
Evaluation
Program&
EvaluationPLANNING
Outcome
![Page 8: Evaluation in the Field: Putting Concepts into Action Janet Myers, PhD MPH Richard Vezina, MPH CAPS HIV Prevention Conference April 21, 2006.](https://reader036.fdocuments.us/reader036/viewer/2022062407/56649d445503460f94a212bf/html5/thumbnails/8.jpg)
Mission
Provides the vision
How this work makes a difference in the world
Broadest scope
![Page 9: Evaluation in the Field: Putting Concepts into Action Janet Myers, PhD MPH Richard Vezina, MPH CAPS HIV Prevention Conference April 21, 2006.](https://reader036.fdocuments.us/reader036/viewer/2022062407/56649d445503460f94a212bf/html5/thumbnails/9.jpg)
Goals (Ross & Mico, 1980; McKenzie & Smeltzer 2001)
“A future event toward which a committed endeavor is directed”
Simple & concise
2 basic components
“Who will be affected”
“What will change as a result of the program”
![Page 10: Evaluation in the Field: Putting Concepts into Action Janet Myers, PhD MPH Richard Vezina, MPH CAPS HIV Prevention Conference April 21, 2006.](https://reader036.fdocuments.us/reader036/viewer/2022062407/56649d445503460f94a212bf/html5/thumbnails/10.jpg)
Objectives
Specific steps that contribute to a goal. Often several objectives per goal.
Good objectives are SMART:
SS – specific
MM – measurable
AA – attainable
RR – realistic
TT – time-bound
![Page 11: Evaluation in the Field: Putting Concepts into Action Janet Myers, PhD MPH Richard Vezina, MPH CAPS HIV Prevention Conference April 21, 2006.](https://reader036.fdocuments.us/reader036/viewer/2022062407/56649d445503460f94a212bf/html5/thumbnails/11.jpg)
Good Objectives Show… (McKenzie & Smeltzer
2001)
What will change: Outcome that will be achieved
When will it change: Conditions under which the outcomes will be observed
How much change: Criteria for deciding whether the outcomes has been achieved
Who will change: Target population
![Page 12: Evaluation in the Field: Putting Concepts into Action Janet Myers, PhD MPH Richard Vezina, MPH CAPS HIV Prevention Conference April 21, 2006.](https://reader036.fdocuments.us/reader036/viewer/2022062407/56649d445503460f94a212bf/html5/thumbnails/12.jpg)
Activities
Internal: administrative, etc. External: the services you provide
to clients Based on your goals/objectives
![Page 13: Evaluation in the Field: Putting Concepts into Action Janet Myers, PhD MPH Richard Vezina, MPH CAPS HIV Prevention Conference April 21, 2006.](https://reader036.fdocuments.us/reader036/viewer/2022062407/56649d445503460f94a212bf/html5/thumbnails/13.jpg)
Outcomes Changes that occur in people being served
by your program
Attribution: To the best extent possible, show that change is a result of your program (but note…causality is difficult)
Standards are typically different for evaluation (vs. research)
To assess, you need at least 2 time points (pre- and post-) and/or a comparison group
![Page 14: Evaluation in the Field: Putting Concepts into Action Janet Myers, PhD MPH Richard Vezina, MPH CAPS HIV Prevention Conference April 21, 2006.](https://reader036.fdocuments.us/reader036/viewer/2022062407/56649d445503460f94a212bf/html5/thumbnails/14.jpg)
Impact
The scope of the program’s effects, the duration of its outcomes and the extent of its influence on the broader context (for example, HIV incidence)
Attribution: Can be more challenging to show causality, because looking for more diffuse effects
Usually broad and long-term
Typically not in the scope of program evaluation
![Page 15: Evaluation in the Field: Putting Concepts into Action Janet Myers, PhD MPH Richard Vezina, MPH CAPS HIV Prevention Conference April 21, 2006.](https://reader036.fdocuments.us/reader036/viewer/2022062407/56649d445503460f94a212bf/html5/thumbnails/15.jpg)
Approaches to Evaluation
![Page 16: Evaluation in the Field: Putting Concepts into Action Janet Myers, PhD MPH Richard Vezina, MPH CAPS HIV Prevention Conference April 21, 2006.](https://reader036.fdocuments.us/reader036/viewer/2022062407/56649d445503460f94a212bf/html5/thumbnails/16.jpg)
Why do we evaluate?
To determine if objectives are being met
To improve quality of the program
To decide how to change content
To identify the effects of the program
![Page 17: Evaluation in the Field: Putting Concepts into Action Janet Myers, PhD MPH Richard Vezina, MPH CAPS HIV Prevention Conference April 21, 2006.](https://reader036.fdocuments.us/reader036/viewer/2022062407/56649d445503460f94a212bf/html5/thumbnails/17.jpg)
Process vs. Outcome Evaluation Process
Demographics (Who’s being trained?) Reaction to content (“Smile Sheets”) Service units delivered
Outcome Changes in knowledge/attitudes/beliefs Changes in behavior Impact on patients/clients
![Page 18: Evaluation in the Field: Putting Concepts into Action Janet Myers, PhD MPH Richard Vezina, MPH CAPS HIV Prevention Conference April 21, 2006.](https://reader036.fdocuments.us/reader036/viewer/2022062407/56649d445503460f94a212bf/html5/thumbnails/18.jpg)
Process Evaluation can help us…
Create better learning environment
Improve presentation skills
Show accountability
Reflect the target populations
Track service units
![Page 19: Evaluation in the Field: Putting Concepts into Action Janet Myers, PhD MPH Richard Vezina, MPH CAPS HIV Prevention Conference April 21, 2006.](https://reader036.fdocuments.us/reader036/viewer/2022062407/56649d445503460f94a212bf/html5/thumbnails/19.jpg)
Outcome Evaluation can help us…
Show the program’s effects
Allow for comparisons over time
Provide specific guide points for improving programs
Show accountability
![Page 20: Evaluation in the Field: Putting Concepts into Action Janet Myers, PhD MPH Richard Vezina, MPH CAPS HIV Prevention Conference April 21, 2006.](https://reader036.fdocuments.us/reader036/viewer/2022062407/56649d445503460f94a212bf/html5/thumbnails/20.jpg)
Planning Your Evaluation (1)
Figure out your questions: What will this be used for?
Consider your Resources Staffing Time Materials $$$
Choose Methods Quantitative: Surveys, pre/post tests, etc. Qualitative: Interviews, focus groups, etc.
![Page 21: Evaluation in the Field: Putting Concepts into Action Janet Myers, PhD MPH Richard Vezina, MPH CAPS HIV Prevention Conference April 21, 2006.](https://reader036.fdocuments.us/reader036/viewer/2022062407/56649d445503460f94a212bf/html5/thumbnails/21.jpg)
Planning Your Evaluation (2)
Select Indicator(s) Relevant Measurable Improvable
Instrument/Tool Development Don’t reinvent the wheel!
Analysis: Get answers to your questions
Reporting: Formal & Informal
![Page 22: Evaluation in the Field: Putting Concepts into Action Janet Myers, PhD MPH Richard Vezina, MPH CAPS HIV Prevention Conference April 21, 2006.](https://reader036.fdocuments.us/reader036/viewer/2022062407/56649d445503460f94a212bf/html5/thumbnails/22.jpg)
Examples