documents.wfp.orgdocuments.wfp.org/stellent/groups/public/documents/rep… · Web...
Transcript of documents.wfp.orgdocuments.wfp.org/stellent/groups/public/documents/rep… · Web...
Decentralized evaluation for evidence-based decision makingWFP Office of Evaluation
Decentralized Evaluation Quality Assurance System (DEQAS)
Quality Checklist for Decentralized Evaluation Terms of ReferenceVersion April 2017
[Title of the decentralized evaluation]
Overall
General Comments/Status
Title: Clear without being too long, reflecting the subject of the evaluation
Length: does not exceed 15 pages (excluding annexes) Accessibility: o TOR written in a clear and understandable languageo TOR adequately emphasises the strategic and/or
operational focus of the decentralized evaluation.o TOR provides a good substantive overview of the
subject of the evaluation. o TOR provides sufficient information to stakeholders
on how the evaluation will unfoldo TOR provides sufficient information to the evaluation
T O R Q C V e r s i o n N o v e m b e r 2 0 1 5 P a g e 1 | 16
team on what is expected from them.
Editing
o Template has been followed and all its elements are included in the TOR (titles)
o Acronyms are spelt out the first time they are used. o Paragraphs and pages are numbered electronically. o Cross-references are used.o Tables and diagrams are used as relevant and are
numbered.o When data or quotes are used, TOR provides sources
of the data/quotes (either directly below the table/graph or in footnotes)
1. Introduction
Overall: The introduction should provide key information about the purpose of the TOR and the facts relevant to the subject of the decentralized evaluation.
Expected Content Assessment criteria Comments/Status
Purpose of TOR Factual information about
the subject of the evaluation (i.e. activity/operation/thematic area/country portfolio/transfer modality/pilot
Expected content is provided.
Clearly indicates 1. Subject of evaluation2. Name of commissioning
office3. Period covered by the
T O R Q C – V e r s i o n N o v e m b e r 2 0 1 5 P a g e 2 | 16
project/other being evaluated)
The period covered by and scope of the evaluation
evaluation4. Scope of the evaluation
2. Reasons for the Evaluation
Overall: Concise explanation of why the evaluation is being conducted, and for who.
Expected Content Assessment criteria Comments/Status
2.1 Rationale
Specify why the evaluation is to be undertaken
Specify why it is needed at this time
Specify how the evaluation will be used by the WFP commissioning office and other stakeholders
Valid rationale Clarity on how the timing
of the evaluation meets the stated needs in decision-making processes
2.2 Objectives
Specify the objectives of the evaluation (e.g. accountability and learning)
Nuance the standard text in an additional
Standard text on accountability and learning used
Additional objectives added where appropriate and/or
T O R Q C – V e r s i o n N o v e m b e r 2 0 1 5 P a g e 3 | 16
paragraph to specify whether more weight is placed on accountability or on learning and why
the accountability and learning objectives contextualised
Text has been nuanced as required if more emphasis on one objective
2.3 Stakeholders and Users
Specify the key internal and external stakeholders and what their stakes/interests are in the evaluation
Specify intended primary and secondary users of the evaluation, both internal to WFP and external where applicable
Indicate how beneficiaries’ perspectives (men, women, boys and girls) will be included in the evaluation process, including in relation to accountability to affected populations, and how this will be addressed
All relevant stakeholders, including beneficiaries, are identified/listed
Demonstrates understanding of stakeholder interests and concerns
Identification of users is linked to the reasons for and objectives of the evaluation
Beneficiaries are identified as stakeholders
Clear analysis of how WFP commitments to accountability to affected populations will be upheld
Stakeholder analysis is gender-responsive
T O R Q C – V e r s i o n N o v e m b e r 2 0 1 5 P a g e 4 | 16
3. Context and Subject of the Evaluation
Overall: Comprehensive description of the evaluation context and information on the nature of the evaluation subject
Expected Content Assessment criteria Comments/Status
3.1 Context
Inclusion of information about the context within which the subject of the evaluation has been/is being implemented, including: Poverty, food security Government policies,
priorities and institutional capacity
Humanitarian issues, including social protection programme/s, migration patterns and host community/social tensions
Gender equality and empowerment of women (GEEW) dimensions of the context
International assistance in the area
Key external events Other WFP work in the
area and Work of other key actors
Contextual information is focussed and concise.
Information is relevant and important to understanding the context for the subject of the evaluation
The section focuses on trend data that is relevant, is coherent with scope and important to the subject of the evaluation.
T O R Q C – V e r s i o n N o v e m b e r 2 0 1 5 P a g e 5 | 16
3. Context and Subject of the Evaluation
3.2 Subject of the Evaluation
Type of activity/operation/ thematic area/transfer modality/pilot project/other subject of evaluation
Geographic scope of the evaluation subject
Relevant dates: Approval date; start date; expected end date
Main partners (Government; NGOs; Bilateral; Multilateral)
Planned outputs at design
1. Beneficiary numbers (planned and revised) disaggregated by sex and age/activity
2. Amount of transfers (food, cash, vouchers), outputs of technical assistance (TA)/capacity development activities)
3. Other outputs
Information is relevant and important to understanding the subject of the evaluation:
1. What it is?2. When it was designed? 3. What are the key inputs ($
value, technical assistance (TA)/capacity development activities)
4. What are the key objectives outcomes and planned activities?
5. What are the planned outputs (beneficiaries, C&V $)
6. What is the total level of funding to the evaluation subject over the period under evaluation? Availability of this information imperative for all DE TOR
7. Who is involved in the implementation?
Highlights relevant issues from past evaluations and
T O R Q C – V e r s i o n N o v e m b e r 2 0 1 5 P a g e 6 | 16
3. Context and Subject of the Evaluation
If existing, logic model or logical framework should be mentioned. If not available it should also be mentioned
Resources (% funded of total requirements) and key donors. If subject funded from pooled funds, show resource allocated
Key objectives, intended outcomes and activities
Other (from WFP and/or other actors) relevant preceding/ concurrent activities/ interventions/operations
Any changes in planned implementation in terms of coverage, budget, planned beneficiaries and explanations
Gender equality and empowerment of women (GEEW) dimensions as relevant
Maps/graphs for illustration
reviews that are relevant to the evaluation
Gender equality and empowerment of women (GEEW) dimensions explained
Differences between original design and final design are explained if appropriate
T O R Q C – V e r s i o n N o v e m b e r 2 0 1 5 P a g e 7 | 16
4. Evaluation Approach
Overall: Clarity on the scope of the evaluation, evaluation criteria and key questions, availability and quality of data available to support it, including any significant gaps; as well as the overall approach and aspects of methodology which the evaluation team will incorporate in their response to the TOR and in their proposed methodology.
Expected Content Assessment Criteria Comments/Status
4.1 Scope
The defined scope should narrow down and specify what will be included and not included in the evaluation, in relation to the activities of the subject of the evaluation
Specify the focus of the evaluation including: time frame, limitations, geographic areas, operations and types of activities and specific target groups (including women and girls)
Clarity on what will be covered and what will not in relation to the contents/coverage of the subject of evaluation
Clarity on duration/period that will be covered by the evaluation
Clear justification for the scope and focus, including selection of activities and areas
Clear justification for the target groups covered by the evaluation
4.2. Evaluation Criteria and Questions
Evaluation criteria selected from the list and
Clear explanation of selected evaluation
T O R Q C – V e r s i o n N o v e m b e r 2 0 1 5 P a g e 8 | 16
justified as appropriate and consistent with evaluation purpose and objectives
Specification of evaluation questions which address the selected criteria
GEEW issues mainstreamed throughout evaluation questions
criteria and justification in case some criteria are not included.
Evaluation questions are clear and relevant to the subject of the evaluation
Evaluation questions sufficiently address the selected evaluation criteria
GEEW is effectively mainstreamed throughout questions
4.3 Data Availability
Identify the main sources of information/data available to the evaluation team
List any gaps in the data and proposed ways for the evaluation team to deal with them
Clear indication of the data available and their sources
Limitations explained and included in the TOR
Suggested mitigation measures
4.4 Methodology
Standard text provided in the template.
Present the overall
Standard text used. Clear delineation of the
overall methodology for T O R Q C – V e r s i o n N o v e m b e r 2 0 1 5 P a g e 9 | 16
methodology for the evaluation outlining the criteria, impartiality requirement, and the methods that will be employed (quantitative, qualitative or mixed)
Identify key risks and appropriate mitigation/ management measures for the methodology approach proposed for further refinement during inception as appropriate.
Specify how gender issues will be addressed by gender-responsive evaluation methodology, tools and data analysis techniques.
the evaluation outlining the data collection methods and overall approach to ensure impartiality
Key risks and appropriate mitigation/management measures for proposed approach, satisfactorily identified;
Sufficient detail to understand how GEEW issues will be addressed by the methodology
4.5 Quality Assurance
Standard text provided in the template of TOR.
Identify additional measures to assure the quality of the process and product, and thus
Standard text used. Quality standards are
included in the TOR. Quality assurance is built
into the whole process by briefly showing how
T O R Q C – V e r s i o n N o v e m b e r 2 0 1 5 P a g e 10 | 16
increase the credibility and impartiality of the evaluation.
different products will be quality assured
Requirements are spelled out for the evaluation team to ensure validity and accuracy of data `
5 Phases and Deliverables
Overall: Clear statement of the overall timing and key deliverables for the evaluation.
Expected Content Assessment Criteria Comments/Status
Clarify the timing of the five evaluation phases and the deliverables including milestones
Specify if other products, in addition to the evaluation report, will be required (e.g. Power Point; Free-standing Summary, Video clip or if suggestions for innovative products are requested from the evaluation team).
Suggested table in annex 2 of the ToR Template is used
Reasonable amount of time for each of the phases/steps is provided for
Clearly identified deliverables with stated responsible persons
6 Organization of the EvaluationT O R Q C – V e r s i o n N o v e m b e r 2 0 1 5 P a g e 11 | 16
Overall: Clear statement on the overall organisation of the evaluation including its management arrangements, the evaluation team and required composition/competencies as well as anticipated lines of communication, timing of key deliverables and key considerations, including security, to be considered when planning to undertake the evaluation.
Expected Content Assessment Criteria Comments/Status
6.1 Evaluation Conduct
Indicate how the evaluation will be conducted and led, including key aspects of communication
Specify ethical considerations for the conduct of the evaluation
Include the evaluation schedule
Specific tasks and outputs are identified for team members in the evaluation schedule.
Specifies team structure and relationship with WFP Evaluation Manager
Consideration of ethical issues that may relate to the subject of the evaluation
Evaluation schedule is logical and provides adequate time for all phases of the evaluation (including consultations on draft documents), and annex 2 in the TOR is used
6.2 Team composition and competencies
Indicate how the evaluation team will be composed (including gender, nationality, and if appropriate cultural background)
The balance of competencies of the evaluation team are consistent with the subject of the evaluation and include, for each
T O R Q C – V e r s i o n N o v e m b e r 2 0 1 5 P a g e 12 | 16
Key competencies and knowledge of evaluation team, including the appropriateness of the skills combination for the evaluation; language skills; and leadership skills for the team leader
Specify gender expertise on the team and appropriate skills to assess gender dimensions
member, expertise in one or more topics of the evaluation
Inclusion of gender parameters for evaluation team
Address fluency in the language/s spoken
Gender expertise is included
Leadership, analytical and communication competencies are spelt out for the team leader
6.3 Security Considerations
Specify any security considerations that may be relevant
Security considerations are relevant to the context for the evaluation
Security considerations are relevant to the nature of the contracting arrangements with WFP
7 Roles and Responsibilities of Stakeholders
Overall: Clear statement of expectations in terms of involvement of internal and external stakeholders and their respective roles and responsibilities
Expected Content Assessment Criteria Comments/StatusT O R Q C – V e r s i o n N o v e m b e r 2 0 1 5 P a g e 13 | 16
Describe roles of key stakeholders in the evaluation process (WFP Country/Area/Sub Office; WFP Regional Bureau; WFP Headquarters; National government stakeholders (relevant Ministries, departments, units); Partner UN agencies; Implementing partners); Office of Evaluation, Internal Evaluation Committee, Evaluation Reference Group.
Describe responsibilities of key stakeholders clearly.
All relevant stakeholders are identified, demonstrating impartiality.
Clear understanding of stakeholder interests and concerns.
Indirect stakeholders who may have a role are identified.
Clarity on measures to ensure impartiality and credibility of the evaluation, as they related to each stakeholder
Internal Evaluation Committee and the Evaluation Reference Group have been established and their roles and responsibilities have been set out
8. Communication and Budget
Overall: Clear statement of communication arrangements throughout the process, including how the report will be disseminated and the roles and responsibilities for communication and sharing the report.
T O R Q C – V e r s i o n N o v e m b e r 2 0 1 5 P a g e 14 | 16
Expected Content Assessment Criteria Comments/Status
8.1 Communication
Specify intentions in relation to the Communication and Learning Plan, and how this will be developed
Determine communication-related roles and responsibilities
Specify the need for translation and the language of each report, if not English
Communication-related roles and responsibilities have been set out
Communication and learning plan is well explained and taken into account in timeline and budget
Requirements for translation clearly set out
8.2 Budget
Overall: Any budgetary guidelines or limitations are clearly set out.
Expected Content Assessment Criteria Comments/Status
Standard text provided in the template of the TOR.
Based on the contracting approach chosen, state how the evaluation budget will be arrived at, and how total cost will be broken down
Clarify whether the budget will include the cost of
Standard text used Clarity on the contracting
method to be used (procurement through open tender, LTA/SLA or HR action)
Clarity on the cost elements that will/should be included in the evaluation budget such as
T O R Q C – V e r s i o n N o v e m b e r 2 0 1 5 P a g e 15 | 16
workshops or special communication efforts if needed.
of workshops or special communication efforts included, as appropriate.
Annexes Comment Comments/Status
Map(s) Evaluation
Timeline/schedule Evaluation committee
membership Evaluation reference group
membership Acronyms Technical annexes
Assessment criteria: Complete and necessary
T O R Q C – V e r s i o n N o v e m b e r 2 0 1 5 P a g e 16 | 16