Quality Assessment of MFA’s evaluations

Rita Tesselaar

Policy and operations Evaluation Department

Netherlands Ministry of Foreign Affairs

Content

• Purpose and scope

• Quality criteria for evaluations

• Assessment method

• Findings in a nutshell

• Issues assessment method

Purpose and scope

• Purpose: to provide MFA with information for decision making on actions to improve the quality of evaluations

• Scope: evaluation product (not only the report)

Criteria for evaluations

• Government regulation on evaluations

• Evaluation criteria: effectiveness and efficiency

• Quality criteria: validity, reliability, usefulness (result orientation)

Assessment method

• Indicators for quality criteria validity, reliability and usefulness

• Rating (4-points scale)• Data collection

– review of documents – interviews

indicators validity

• Reason and purpose: clear description• Context: setting -policy and institutional• Evaluation object: clear description,

intervention logic, result levels, demarcation • Evaluation questions: definition of evaluation

criteria, clearness, relevance• Methods: specification, use of indicators,

explanation of limitations• Consistency: analyses, conclusions, lessons

Indicators reliability

• Representativeness: account for sample studied

• Information sources: identification, verification, triangulation of sources

• Evaluation team: independence• Evaluation process: undisturbed operation • Quality control: involvement stakeholders,

internal or external monitoring committee

Indicators usefulness• Relevance of results to its intended use:

robustness, unambiguous, timely, realism • Completeness: questions answered, results

explained • Presentation: summary, account of methods

and process, distinction between conclusions and recommendations

Findings (nutshell)

• Effectiveness criteria partly addressed – focus on process and outputs and less on

effects

• Efficiency criteria gets little attention• Quality is variable• Evaluations are largely designed from a

future oriented managerial perspective

Issues assessment method

• Differences in significance of indicators

• Subjective element in rating of indicators