Introduction Evaluation Design Design... · Key Tasks ofEvaluation Design The design of an...
Transcript of Introduction Evaluation Design Design... · Key Tasks ofEvaluation Design The design of an...
![Page 1: Introduction Evaluation Design Design... · Key Tasks ofEvaluation Design The design of an evaluation has to: • describe the evaluand and derive the work tasks and challenges, •](https://reader034.fdocuments.us/reader034/viewer/2022042318/5f07ee3f7e708231d41f798e/html5/thumbnails/1.jpg)
Introduction
to
Evaluation Design
Prof. Dr. Wolfgang Meyer
Evaluation Design
One Day Course – Part A
© 2015 Prof. Dr. Wolfgang Meyer, CEval
![Page 2: Introduction Evaluation Design Design... · Key Tasks ofEvaluation Design The design of an evaluation has to: • describe the evaluand and derive the work tasks and challenges, •](https://reader034.fdocuments.us/reader034/viewer/2022042318/5f07ee3f7e708231d41f798e/html5/thumbnails/2.jpg)
Course Content
� 09.00-10.00 First Input (Part A):
Introduction to Evaluation Designs
� 10.00-10.30 Coffee break
� 10.30-12.00 Second Input (Part B):
Investigation Designs
� 12.00-13.00 Lunch break
� 13.00-14.30 Third Input (Part C):
Data Collection Designs
� 14.30-15.00 Coffee break
� 15.00-16.00 Fourth input (Part D):
Practicability of Evaluation Designs
2
© 2015 Prof. Dr. Wolfgang Meyer, CEval
![Page 3: Introduction Evaluation Design Design... · Key Tasks ofEvaluation Design The design of an evaluation has to: • describe the evaluand and derive the work tasks and challenges, •](https://reader034.fdocuments.us/reader034/viewer/2022042318/5f07ee3f7e708231d41f798e/html5/thumbnails/3.jpg)
Content of Part A
� What is Evaluation Design?
� Different Design Types
� Different Kinds of Evaluations and Different Kinds of
Evaluation Design
� Lessons Learnt
3
© 2015 Prof. Dr. Wolfgang Meyer, CEval
![Page 4: Introduction Evaluation Design Design... · Key Tasks ofEvaluation Design The design of an evaluation has to: • describe the evaluand and derive the work tasks and challenges, •](https://reader034.fdocuments.us/reader034/viewer/2022042318/5f07ee3f7e708231d41f798e/html5/thumbnails/4.jpg)
What is Evaluation Design?
� Design is the creation of a plan for the construction of a
system (Wikipedia)
� Research Design = „A research design is simply a plan for
conducting research” (Pell Institute Evaluation Toolkit)
An Evaluation Design is simply a plan for conducting an
Evaluation
4
© 2015 Prof. Dr. Wolfgang Meyer, CEval
![Page 5: Introduction Evaluation Design Design... · Key Tasks ofEvaluation Design The design of an evaluation has to: • describe the evaluand and derive the work tasks and challenges, •](https://reader034.fdocuments.us/reader034/viewer/2022042318/5f07ee3f7e708231d41f798e/html5/thumbnails/5.jpg)
Evaluation vs. Research Design
5
Formulating the Research Problem
Defining Test of Theory
Defining basic population and sampling
Planning and Preparing Data Collection
Data Collection Process
Data Analysis
Reporting / Publication
Scientific Discussion
Interpreting the Evaluation Task
Defining Assessment Criteria
Planning Stakeholder Involvement
Planning and Preparing Data Collection
Data Collection Process
Data Analysis
Reporting / Internal Exchange
Utilization of Results
© 2015 Prof. Dr. Wolfgang Meyer, CEval
Defining the Evalandum
Assessment Criteria
Stakeholder Involvement
Utilization
![Page 6: Introduction Evaluation Design Design... · Key Tasks ofEvaluation Design The design of an evaluation has to: • describe the evaluand and derive the work tasks and challenges, •](https://reader034.fdocuments.us/reader034/viewer/2022042318/5f07ee3f7e708231d41f798e/html5/thumbnails/6.jpg)
Differences to Research Designs
Evaluation Designs are different to Research Designs because
� They are not limited to a single research question
� They are performing more reactive than active
� They have to handle assessments in an objective way
� They are more or less involved in a political process
� They include the management of social processes
(stakeholder involvement, exchange with commissioners)
� They include the management of utilization
6
© 2015 Prof. Dr. Wolfgang Meyer, CEval
![Page 7: Introduction Evaluation Design Design... · Key Tasks ofEvaluation Design The design of an evaluation has to: • describe the evaluand and derive the work tasks and challenges, •](https://reader034.fdocuments.us/reader034/viewer/2022042318/5f07ee3f7e708231d41f798e/html5/thumbnails/7.jpg)
Key Tasks of Evaluation Design
� The design of an evaluation has to:
• describe the evaluand and derive the work tasks and challenges,
• give advices for formulating objectives and tasks of the
evaluation (including how evaluation criteria are developed),
• stipulates the manner in which the evaluation is to be conducted
(including data collection, analyses, assessment , interpretation),
• show how to identify stakeholders and how they will be
included (selection, contacting, informing, deciding etc.),
• develop strategies to prepare the use of evaluation results
7
© 2015 Prof. Dr. Wolfgang Meyer, CEval
1
2
3
4
5
![Page 8: Introduction Evaluation Design Design... · Key Tasks ofEvaluation Design The design of an evaluation has to: • describe the evaluand and derive the work tasks and challenges, •](https://reader034.fdocuments.us/reader034/viewer/2022042318/5f07ee3f7e708231d41f798e/html5/thumbnails/8.jpg)
Three Levels of Design
8
Evaluation Design
Investigation Design
Collection Design
Depends on kinds of
Evaluations
Depends on kinds of
Information needs
Depends on kinds of
Methods
© 2015 Prof. Dr. Wolfgang Meyer, CEval
![Page 9: Introduction Evaluation Design Design... · Key Tasks ofEvaluation Design The design of an evaluation has to: • describe the evaluand and derive the work tasks and challenges, •](https://reader034.fdocuments.us/reader034/viewer/2022042318/5f07ee3f7e708231d41f798e/html5/thumbnails/9.jpg)
Elements of Evaluation Design
9
Evaluation Design
© 2015 Prof. Dr. Wolfgang Meyer, CEval
Planning
Defining Evalandum
Developing Criteria
1
2
Imple-
mentation
Data Collection
Stakeholder
Involvement
3
4
Utili-
zation
Preparing
Use
5
Evaluation
Designs
An Evaluation is a project with the same management tasks
![Page 10: Introduction Evaluation Design Design... · Key Tasks ofEvaluation Design The design of an evaluation has to: • describe the evaluand and derive the work tasks and challenges, •](https://reader034.fdocuments.us/reader034/viewer/2022042318/5f07ee3f7e708231d41f798e/html5/thumbnails/10.jpg)
Typical kinds of Evaluations
Ex-ante On-going Ex-post
10
Evaluation
Designs
Baseline formativ summativ
Time
Metaevaluation Metaevaluation Metaevaluation
Participatory Approaches
Scope
© 2015 Prof. Dr. Wolfgang Meyer, CEval
![Page 11: Introduction Evaluation Design Design... · Key Tasks ofEvaluation Design The design of an evaluation has to: • describe the evaluand and derive the work tasks and challenges, •](https://reader034.fdocuments.us/reader034/viewer/2022042318/5f07ee3f7e708231d41f798e/html5/thumbnails/11.jpg)
Specifics I: Defining Evalandum
Ex-ante On-going Ex-post
11
Evaluation
Designs
Baseline formativ summativ
Time
Metaevaluation Metaevaluation Metaevaluation
Participatory Approaches
Scope
© 2015 Prof. Dr. Wolfgang Meyer, CEval
1
Comparing
Detecting
Anticipating Balancing
![Page 12: Introduction Evaluation Design Design... · Key Tasks ofEvaluation Design The design of an evaluation has to: • describe the evaluand and derive the work tasks and challenges, •](https://reader034.fdocuments.us/reader034/viewer/2022042318/5f07ee3f7e708231d41f798e/html5/thumbnails/12.jpg)
Specifics II: Defining Criteria
Ex-ante On-going Ex-post
12
Evaluation
Designs
Baseline formativ summativ
Time
Metaevaluation Metaevaluation Metaevaluation
Participatory Approaches
Scope
© 2015 Prof. Dr. Wolfgang Meyer, CEval
Standardized
Specialized
Potentials Essentials
2
![Page 13: Introduction Evaluation Design Design... · Key Tasks ofEvaluation Design The design of an evaluation has to: • describe the evaluand and derive the work tasks and challenges, •](https://reader034.fdocuments.us/reader034/viewer/2022042318/5f07ee3f7e708231d41f798e/html5/thumbnails/13.jpg)
Specifics III: Data Collection
Ex-ante On-going Ex-post
13
Evaluation
Designs
Baseline formativ summativ
Time
Metaevaluation Metaevaluation Metaevaluation
Participatory Approaches
Scope
© 2015 Prof. Dr. Wolfgang Meyer, CEval
Quantitative
Qualitative
Prospective Retrospective
3
![Page 14: Introduction Evaluation Design Design... · Key Tasks ofEvaluation Design The design of an evaluation has to: • describe the evaluand and derive the work tasks and challenges, •](https://reader034.fdocuments.us/reader034/viewer/2022042318/5f07ee3f7e708231d41f798e/html5/thumbnails/14.jpg)
Specifics IV: Involving Stakeholders
Ex-ante On-going Ex-post
14
Evaluation
Designs
Baseline formativ summativ
Time
Metaevaluation Metaevaluation Metaevaluation
Participatory Approaches
Scope
© 2015 Prof. Dr. Wolfgang Meyer, CEval
Mediation
Empowerment
Designing Analysing
4
![Page 15: Introduction Evaluation Design Design... · Key Tasks ofEvaluation Design The design of an evaluation has to: • describe the evaluand and derive the work tasks and challenges, •](https://reader034.fdocuments.us/reader034/viewer/2022042318/5f07ee3f7e708231d41f798e/html5/thumbnails/15.jpg)
Specifics V: Using Results
Ex-ante On-going Ex-post
15
Evaluation
Designs
Baseline formativ summativ
Time
Metaevaluation Metaevaluation Metaevaluation
Participatory Approaches
Scope
© 2015 Prof. Dr. Wolfgang Meyer, CEval
Generalizing
Steering
Planning Learning
5
![Page 16: Introduction Evaluation Design Design... · Key Tasks ofEvaluation Design The design of an evaluation has to: • describe the evaluand and derive the work tasks and challenges, •](https://reader034.fdocuments.us/reader034/viewer/2022042318/5f07ee3f7e708231d41f798e/html5/thumbnails/16.jpg)
Demands for Evaluation Designs
16
� Evaluation should anticipate
� Aim for Project Potentials
� Prospective Data („what
may happen“)
� Participation in Designing
Evaluations
� Using Results for Planning
Purposes
� Evaluation should balance
� Aim for Project Essentials
� Retropective Data („what
happened“)
� Participation in Analysing
Evaluation results
� Using Results for Learning
Purposes
� Evaluation should compare
� Aim for Comparisons
� Quantitative Data for comparative Analysis
� Participation for including interests
� Using Results for Strategic Decisions
� Evaluation should detect specifics
� Aim for Specifics
� Qualitative Data for In-depth Analysis
� Participation for empowering people
� Using Results for Project Steering
Evaluation
Designs
![Page 17: Introduction Evaluation Design Design... · Key Tasks ofEvaluation Design The design of an evaluation has to: • describe the evaluand and derive the work tasks and challenges, •](https://reader034.fdocuments.us/reader034/viewer/2022042318/5f07ee3f7e708231d41f798e/html5/thumbnails/17.jpg)
Summary
� Evaluation Designs are plans for conducting an
evaluation
� Evaluation Desings have five tasks
• Defining the Evalandum
• Defining Assessment Criteria
• Collecting Data
• Involving Stakeholders
• Preparing the Use of Evaluation Results
� There are different demands for Evaluation Designs
• Formativ (more interactive, communicative , involved)
• Summativ (more reactive, distant, outside the process)
• Comparative and aggregated (Metaevaluation)
• Specific and case-oriented (Participatory Approaches)
17
© 2015 Prof. Dr. Wolfgang Meyer, CEval
Evaluation
Designs