IDEAS Delhi, Workshop – DAC April 2005 1 Independent Evaluation « Quality Assessment of SDC’s...
-
Upload
scot-mason -
Category
Documents
-
view
213 -
download
0
Transcript of IDEAS Delhi, Workshop – DAC April 2005 1 Independent Evaluation « Quality Assessment of SDC’s...
IDEAS Delhi, Workshop – DAC April 2005
1
Independent Evaluation
« Quality Assessment of SDC’s External Evaluation Reports »
Anne BichselEvaluation + Controlling Division
IDEAS Delhi, Workshop – DAC April 2005
2
Content
1. Objectives of the Quality Assessment
2. Procedures and Method
3. Findings
4. Recommendations
IDEAS Delhi, Workshop – DAC April 2005
3
1. Objectives ot the Quality Assessment
• To assess the quality of SDC‘sexternal evaluation reports
• To contribute to improving future performance and achieving a higher evaluation quality of SDC‘s external evaluations
IDEAS Delhi, Workshop – DAC April 2005
4
2. Procedures and Method (I)
• Step I: Random Sample of 12 external evaluation reports
• Step 2: devise a list of evaluation standards on the basis of- „DAC Minimum Sufficient Evaluation Standards“- SEVAL Standards- Key questions in Approach Paper
IDEAS Delhi, Workshop – DAC April 2005
5
2. Procedures and Method (II)
List of evaluation standards with 4 categories
• Utility (9 standards): readable, accessible, timely evaluations with a good summary
• Feasibility (3 standards): realistic well thought-out evaluations
• Propriety (4 standards): ethical aspect of evaluations
• Accuracy (7 standards): proper scientific methods and procedures
IDEAS Delhi, Workshop – DAC April 2005
6
2. Procedures and Method (III)
• Step 3: review documents, conduct interviews with desk managers and evaluators
• Step 4: work through all the standards for the sample of evaluations. Result: 12 fact sheets
• Step 5: analysis and comparison
• Step 6: conclusions and recommendations
IDEAS Delhi, Workshop – DAC April 2005
7
3. Findings (I)
Strengths of external evaluation reports
• Identification and participation of stakeholders
• Timely reporting
• Cost effectiveness
• Complete and balanced assessment
IDEAS Delhi, Workshop – DAC April 2005
8
3. Findings (II)
Weaknesses of external evaluations reports
• Selection procedures of evaluation team
• Formal written agreement
• Comprehensive and clear reporting
• Description of evaluation purpose and objectives
• Description of evaluation procedures and methods
• Making findings available
IDEAS Delhi, Workshop – DAC April 2005
9
4. Recommendations (I)
Draft good, realistic and comprehensive TORs
• Clear purpose
• Clear, focused, concise and understandable objectives with three to four questions
• Agreement on TORs among SDC divisions
• Realistic expectations in terms of resources and timeframe
• Beware of questions at the impact level
IDEAS Delhi, Workshop – DAC April 2005
10
4. Recommendations (II)
Ensure more competitive and open selection procedures
• Aim at balance in desired knowhow and independence of evaluators
• Place more weight on evaluation knowhow (methods)
• More open and transparent selection procedures
• More competition
• Careful team building in matching personalities, skills, division of labor
IDEAS Delhi, Workshop – DAC April 2005
11
4. Recommendations (III)
Improve the conditions for the utilisation of external evaluations
• Accompany evaluations closely for maximum learning potential
• Continuity on the part of desk managers
• Reports need to be more accessible to a wider audience
• Enhance the quality of evaluation reports
• Make results of external evaluations more widely available
IDEAS Delhi, Workshop – DAC April 2005
12
4. Recommendations (IV)
Enhance the conditions for high quality external evaluations
Evaluations are a challenging business, hence:Provide support and training for desk managers
• Training in evaluation
• Support in drafting TORs and commissioning evaluations
• Support in obtaining high quality reports
IDEAS Delhi, Workshop – DAC April 2005
13
In sum: less is more !
IDEAS Delhi, Workshop – DAC April 2005
14
Follow-up by the Core Learning Group: Evaluation & Controlling Net
H-Bereich O-Bereich E-Bereich M-Bereich F-Bereich A-Bereich
ChristophJakob
KunoSchläfli
Peter Meier
Regula Bäbler
MarkusGlatz
IvoAngehrn
AlexandreKobel
Direktor DEZA
Stellv. Direktor
Gerhard SiegfriedSamuel WältyAnne Bichsel
Sekretariat (je 50%)Regula HerlanChrista Rohner
jeweilige Bereichsleitung
IDEAS Delhi, Workshop – DAC April 2005
15
Steps of an Independent Evaluation
COSTRAComité Stratégique
•TORs&contracts•Implementation•Draft Report•Debriefing •Final Report
•Approach Paper •Core Learning Partners (CLP)
•Approach Paper •Core Learning Partners (CLP)
•Agreement at Completion Point•Dissemination to Broad Learning Partners (BLP)
•Agreement at Completion Point•Dissemination to Broad Learning Partners (BLP)
•Senior Management Standpoint discussed in COSTRA
•Senior Management Standpoint discussed in COSTRA
I.
II.
III.
IV.
•COSTRA Decision•COSTRA Decision0.
IDEAS Delhi, Workshop – DAC April 2005
16
Measures to improve quality
Communicated evaluation results / recommendations in all departments
Clarified and communicated the palette of instruments
Developing toolkit for commissioners of evaluations Intraweb: Description of Process / examples Principles to ensure successful Use of
Evaluations Improving training courses: curriculum and
availability (1 day training Ext. Evals) Backstopping and Quality Control by the E&C
Net
IDEAS Delhi, Workshop – DAC April 2005
17
Independent Evaluation
External Evaluation
External Review
Self-Evaluation
Expert Opinion
Palette of Instruments
IDEAS Delhi, Workshop – DAC April 2005
18
Independent Evaluationssince 2001
2 – 5 annually
outside line management( E&C Division)
- issues of interest across departments- selected country programs
aim for the same
DAC Executive
Registration
ExternalEvaluations
5 – 10 annually
within
- to generate knowledge within the department- of interest across divisions
standards
Summary
in data management
ExternalReviews
according to operational needs
line management
- ongoing operations - in the context of program cycle management
system (DMS)
IDEAS Delhi, Workshop – DAC April 2005
19
IDEAS Delhi, Workshop – DAC April 2005
20