What is the true impact of evaluation ? JM Clinton; R McNeill *& S.K. Appleton-dyer* Centre for...
-
Upload
gerald-bossom -
Category
Documents
-
view
213 -
download
0
Transcript of What is the true impact of evaluation ? JM Clinton; R McNeill *& S.K. Appleton-dyer* Centre for...
![Page 1: What is the true impact of evaluation ? JM Clinton; R McNeill *& S.K. Appleton-dyer* Centre for Program Evaluation The Australasian Evaluation Conference:](https://reader035.fdocuments.us/reader035/viewer/2022062712/56649c695503460f94917101/html5/thumbnails/1.jpg)
What is the true impact of evaluation ?
JM Clinton; R McNeill *& S.K. Appleton-dyer*
Centre for Program EvaluationThe Australasian Evaluation Conference:
Evaluation and Influence
Sydney Sept. 2011
![Page 2: What is the true impact of evaluation ? JM Clinton; R McNeill *& S.K. Appleton-dyer* Centre for Program Evaluation The Australasian Evaluation Conference:](https://reader035.fdocuments.us/reader035/viewer/2022062712/56649c695503460f94917101/html5/thumbnails/2.jpg)
The journey
What is the true impact of evaluation?
![Page 3: What is the true impact of evaluation ? JM Clinton; R McNeill *& S.K. Appleton-dyer* Centre for Program Evaluation The Australasian Evaluation Conference:](https://reader035.fdocuments.us/reader035/viewer/2022062712/56649c695503460f94917101/html5/thumbnails/3.jpg)
The evidnece
The evidence is confusing
Do we add value?
![Page 4: What is the true impact of evaluation ? JM Clinton; R McNeill *& S.K. Appleton-dyer* Centre for Program Evaluation The Australasian Evaluation Conference:](https://reader035.fdocuments.us/reader035/viewer/2022062712/56649c695503460f94917101/html5/thumbnails/4.jpg)
• Changes organisations - Williams
• Develops learning environment - Preskill
• Evaluation willingness and capacity leads to increased program success -Clinton
• Influences individual, interpersonal & the collective-Marks & Henry
• Empowers –Fetterman;
• Evaluation attributes impact on organisations-Appleton
![Page 5: What is the true impact of evaluation ? JM Clinton; R McNeill *& S.K. Appleton-dyer* Centre for Program Evaluation The Australasian Evaluation Conference:](https://reader035.fdocuments.us/reader035/viewer/2022062712/56649c695503460f94917101/html5/thumbnails/5.jpg)
Evaluation research methods
What's our standard?
![Page 6: What is the true impact of evaluation ? JM Clinton; R McNeill *& S.K. Appleton-dyer* Centre for Program Evaluation The Australasian Evaluation Conference:](https://reader035.fdocuments.us/reader035/viewer/2022062712/56649c695503460f94917101/html5/thumbnails/6.jpg)
Our impact
KNOW THY IMPACT!
![Page 7: What is the true impact of evaluation ? JM Clinton; R McNeill *& S.K. Appleton-dyer* Centre for Program Evaluation The Australasian Evaluation Conference:](https://reader035.fdocuments.us/reader035/viewer/2022062712/56649c695503460f94917101/html5/thumbnails/7.jpg)
Click to edit Master title style
This study
![Page 8: What is the true impact of evaluation ? JM Clinton; R McNeill *& S.K. Appleton-dyer* Centre for Program Evaluation The Australasian Evaluation Conference:](https://reader035.fdocuments.us/reader035/viewer/2022062712/56649c695503460f94917101/html5/thumbnails/8.jpg)
The programs & the evaluation
5yr evaluation of a district-wide plan in NZ, aimed at mobilizing a community to prevent the onset of Type II Diabetes.
*10 action areas: Over a 150 different initiatives over a 4 year period
Nelson Marlborough Nutrition & Physical Activity: Collaborate as a community to reverse the tide of obesity.
*5 Action Areas: Over a 100 different initiatives over a 3 year period
EVALUATION METHODS: An adapted form of the CDC Evaluation Framework Standard
dimensions
Multiple forms of data collection :reports, surveys, case studies
interviews, observations and internal data
Key information from 292 initiatives was coded using scoring rubric
![Page 9: What is the true impact of evaluation ? JM Clinton; R McNeill *& S.K. Appleton-dyer* Centre for Program Evaluation The Australasian Evaluation Conference:](https://reader035.fdocuments.us/reader035/viewer/2022062712/56649c695503460f94917101/html5/thumbnails/9.jpg)
A research lens and the evaluation
THIS STUDY: Analysed year by year
and over 4 years. Determine relationships:
Factor analysis, Path models
• Scrubbed the data• Determined the strength & validity of the dimension• Identified outcomes• Looked for causal relationships
![Page 10: What is the true impact of evaluation ? JM Clinton; R McNeill *& S.K. Appleton-dyer* Centre for Program Evaluation The Australasian Evaluation Conference:](https://reader035.fdocuments.us/reader035/viewer/2022062712/56649c695503460f94917101/html5/thumbnails/10.jpg)
MEASURES
ProgramKPI
Adaptation
Degree of implementation
OutcomeProgress
Sustainability
ProcessOrganisational development
Collaboration
Evaluation S
TAN
DA
RD
S
ET
TIN
G
SCORE EXEMPLAR
KPI 1-3 Only 30% achieved. Only ‘easy’ KPIs indicator achieved
4-7 Achieved 40% mix of ‘hard and easy’ indicators
8-10 100% achieved mixture of levels of difficulty
![Page 11: What is the true impact of evaluation ? JM Clinton; R McNeill *& S.K. Appleton-dyer* Centre for Program Evaluation The Australasian Evaluation Conference:](https://reader035.fdocuments.us/reader035/viewer/2022062712/56649c695503460f94917101/html5/thumbnails/11.jpg)
THE MODEL
![Page 12: What is the true impact of evaluation ? JM Clinton; R McNeill *& S.K. Appleton-dyer* Centre for Program Evaluation The Australasian Evaluation Conference:](https://reader035.fdocuments.us/reader035/viewer/2022062712/56649c695503460f94917101/html5/thumbnails/12.jpg)
So what does this mean?
SUSTAINABILITY:
KPI XXX
ADAPTATION --IVE X
DOIX
ODX
COLLABORATION XXX
EVALUATION XXX
PROGRESS:
KPI XXX
ADAPTATION X
DOI XXX
OD XXX
COLLABORATION –IVE
XXX
EVALUATION XXX
![Page 13: What is the true impact of evaluation ? JM Clinton; R McNeill *& S.K. Appleton-dyer* Centre for Program Evaluation The Australasian Evaluation Conference:](https://reader035.fdocuments.us/reader035/viewer/2022062712/56649c695503460f94917101/html5/thumbnails/13.jpg)
The outcome variables
SUSTAINABILITY- WEIGHTING VARIES ACCORDING TO: TIME: DOI, COLLABORATION PROGRAM:OD, COLLABORATION, EVALUATION
PROGRESS- WEIGHTING VARIES ACCORDING TO: TIME:ADAPTATION,OD, DOIPROGRAM: OD, COLLABORATION
![Page 14: What is the true impact of evaluation ? JM Clinton; R McNeill *& S.K. Appleton-dyer* Centre for Program Evaluation The Australasian Evaluation Conference:](https://reader035.fdocuments.us/reader035/viewer/2022062712/56649c695503460f94917101/html5/thumbnails/14.jpg)
EVALUATION IS AN EQUATION
Program Outcome
ImpProcess
Dosage
Intervention
Adaptation
Evaluation influence
CONTEXT
![Page 15: What is the true impact of evaluation ? JM Clinton; R McNeill *& S.K. Appleton-dyer* Centre for Program Evaluation The Australasian Evaluation Conference:](https://reader035.fdocuments.us/reader035/viewer/2022062712/56649c695503460f94917101/html5/thumbnails/15.jpg)
Final word
EVALUATION IS AN INTERVENTION
EVALUATION STANDARD METHODS
EVALUATION ADDS VALUE
EVALUATORS - KNOW THY IMPACT