Baastel Macro Results Framework Assessment Appendix Final · 2018-04-27 · AFESD Arab Fund for...
Transcript of Baastel Macro Results Framework Assessment Appendix Final · 2018-04-27 · AFESD Arab Fund for...
www.baastel.com
Creating a Macro Results Framework for the Middle East and North Africa
Transition Fund “Feasibility Assessment”
Final Report
Appendix
Submitted by: Le Groupe‐conseil baastel ltée
Requested by: The World Bank
June, 2014
www.baastel.com
Le Groupe‐conseil baastel ltée
85 Victoria Street
Gatineau (Québec) Canada, J8X 2A3
Tel: +(1) 819‐595‐1421
Fax: +(1) 819‐595‐8586
www.baastel.com
Contact: [email protected]
“Creating a Macro Results Framework for the Middle East and North Africa Transition Fund” Final Report Appendix
i
TABLE OF CONTENTS Acronyms ............................................................................................................................... iii
1. Introduction ................................................................................................................ 1
2. Overview of M&E Practices across the ISAs .................................................................... 1
2.1. African Development Bank ............................................................................................ 1
2.2. European Bank for Reconstruction and Development ...................................................... 2
2.3. European Investment Bank ............................................................................................ 2
2.4. International Finance Corporation .................................................................................. 3
2.5. Islamic Development Bank ............................................................................................ 3
2.6. Organisation for Economic Co‐operation and Development .............................................. 4
2.7. World Bank .................................................................................................................. 4
2.8. Others ......................................................................................................................... 5
3. Case Studies .................................................................................................................5
3.1. The Global Environment Facility ..................................................................................... 5
3.2. Global Agriculture and Food Security Program ............................................................... 12
3.3. Neighbourhood Investment Facility............................................................................... 17
3.4. Conclusion................................................................................................................... 20
4. Options ..................................................................................................................... 21
4.1. Option A1 “Enhanced” Status Quo ................................................................................ 21
4.2. Option B1 – Revise and Use the DFID‐based Logic Model ................................................ 27
4.3. Option B2 – Developing Project Cluster/Nesting Model ................................................... 31
4.4. Option B3 – Reporting Against a MENA Transition Fund Theory of Change – “Hybrid” Option
37
4.5. Others ........................................................................................................................ 41
Annex 1. Methodology ........................................................................................................... 42
Annex 2. Glossary ..................................................................................................................46
Annex 3. List of Documents Reviewed .................................................................................... 48
Annex 4. List of Persons Met/Interviewed ................................................................................ 52
Annex 5. Addendum .............................................................................................................. 55
“Creating a Macro Results Framework for the Middle East and North Africa Transition Fund” Final Report Appendix
ii
“Creating a Macro Results Framework for the Middle East and North Africa Transition Fund” Final Report Appendix
iii
ACRONYMS
Acronym Definition
€ Euros
$ Dollars
ADB Asian Development Bank
AfDB African Development Bank
AFESD Arab Fund for Economic and Social Development
AMF Arab Monetary Fund
AS Advisory Services (IFC)
CAREC Central Asia Regional Economic Cooperation
CEO Chief Executive Officer
CPPR Country Portfolio Performance Reviews
CSI Core Sector Indicator (AfDB)
CSO Civil Society Organization
CU Coordination Unit
DAC Development Assistance Committee (OECD)
DFATD Department of Foreign Affairs, Trade and Development (Canada)
DFID Department for International Development (UK)
DIME Development Impact Evaluation Initiative
DOTS Development Outcome Tracking System
EBRD European Bank for Reconstruction and Development
EC European Commission
EIB European Investment Bank
ENP European Neighbourhood Policy
ENPI European Neighbourhood and Partnership Instrument
EO Evaluation Office (GEF)
EU European Union
EvD Evaluation Department (EBRD)
ESIAF Economic and Social Impact Assessment Framework (EIB)
FAO Food and Agriculture Organization
FIG Finance Institutions Group (NIF)
GAFSP Global Agriculture and Food Security Program
GDPRD Global Donor Platform for Rural Development
GEF Global Environment Facility
GPB Global Program Branch
HS Highly Satisfactory
HU Highly Unsatisfactory
ICR Implementation Completion and Results Report
IDB Inter‐American Development Bank
IE Impact Evaluation
IEG Independent Evaluation Group (WB)
IFAD International Fund for Agricultural Development
IFC International Finance Corporation
IFI International Financial Institutions
IMF International Monetary Fund
“Creating a Macro Results Framework for the Middle East and North Africa Transition Fund” Final Report Appendix
iv
IS Investment Services (IFC)
ISA Implementation Support Agency
IsDB Islamic Development Bank
ISR Implementation Status Report
KPI Key Performance Indicator
M&E Monitoring and Evaluation
MDG Millennium Development Goals
MENA Middle East and North Africa
MOPAN Multilateral Organisation Performance Assessment Network
MS Moderately Satisfactory
MU Moderately Unsatisfactory
NGO Nongovernmental Organization
NIF Neighbourhood Investment Facility
OCE Office of the Chief Economist
OECD Organization for Economic Cooperation and Development
OPEC Organization of the Petroleum Exporting Countries
OPS Overall Performance Studies (GEF)
PAR Project Appraisal Reports
PCR Project Completion Reports
PDO Project Development Objective
PIR Project/Program Implementation Report
PISR Project Implementation Support Report
ReM Results Measurement Framework (EIB)
ROtl Review of Outcomes to Impacts (GEF)
RRS Results Reporting System (AfDB)
S Satisfactory
SC Steering Committee
STAP Scientific and Technical Advisory Panel
TC Transition Country
TIMS Transition Impact Monitoring System
U Unsatisfactory
UK United Kingdom
UNDP United Nations Development Programme
UNEP United Nations Environment Programme
USA United States of America
WB World Bank
WFP World Food Programme
“Creating a Macro Results Framework for the Middle East and North Africa Transition Fund” Final Report Appendix
1
1. INTRODUCTION his document represents the Appendix to the Final Report of the Feasibility Assessment on Creating a
Macro Results Framework (herein referred to as “Macro Framework” for brevity) for the Middle East and
North Africa (MENA) Transition Fund (also referred to as “the Fund”). This document presents a more
complete discussion on key aspects important for understanding the context and making an informed
decision on the feasibility of developing a Macro Framework for the Fund. In brief, the purpose of this document is
to provide the Steering Committee (SC) members with further details on the following aspects of the completed
Feasibility Assessment:
1. Overview of Monitoring and Evaluation (M&E) Practices across the Implementation Support Agencies
(ISAs);
2. Case Studies of Practices in Funds with Similarities to the MENA Transition Fund;
3. Options for the Way Forward, in particular a detailed discussion of Variations on Creating a Macro
Framework for the Fund.
2. OVERVIEW OF M&E PRACTICES ACROSS
THE ISAS his overview of data collection and M&E practices across the different ISAs could be useful (e.g. as a
baseline) for the next consultancy if the decision is taken to develop a Macro Framework for the Fund. As
important partners involved in project M&E under the Fund, the development of a Macro Framework for
the Fund should take into account, to the extent possible, the particular systems of the ISAs involved and
their internal procedures and capabilities.
2.1. African Development Bank
In recent years, the African Development Bank (AfDB) has been making noticeable efforts to better focus project
supervision on results, for example with the introduction of the new supervision report format and supervision
assessment, and by entrusting sector departments with the responsibility to proactively track Core Sector
Indicators (CSIs). The Bank introduced guidelines and a results‐oriented format in 2008. A new automated Results
Reporting System (RRS) is expected to be launched in 2014. This system will link expected results (as per the
logical frameworks in Project Appraisal Reports (PAR)) to progress toward results (as documented in supervision
reports) and results achieved (as reported in Project Completion Reports (PCR)). Thus, AfDB is moving forward
with more rigorous procedures and M&E systems that will track results internally. However, and as noted in AfDB’s
Management Response to the 2011 Multilateral Organisation Performance Assessment Network (MOPAN)
assessment on organizational performance, “many of these reforms will take some time before they fully yield
T
T
“Creating a Macro Results Framework for the Middle East and North Africa Transition Fund” Final Report Appendix
2
their results”.1 Some deficiencies in terms of monitoring and reporting on key results have to do with reporting
that is focused on “processes” rather than “change”.
As confirmed through interviews held with representatives of AfDB, in reporting data for the Fund activities, AfDB
simply transfers information collected by its field staff through the Bank’s internal instrument for project reporting
(i.e. supervision reports and PCR) to the standard Progress Report requested by the Fund Coordination Unit (CU),
which is considered efficient as this way the Progress Report does not constitute an additional reporting burden.
2.2. European Bank for Reconstruction and Development
At the European Bank for Reconstruction and Development (EBRD), the transition impact of every project is
planned for as part of the process of choosing, preparing and appraising projects, in addition to an independent
Evaluation Department (EvD) which closes the cycle. All projects are scrutinized by the credit department to
ensure sound banking principles and by the Office of the Chief Economist (OCE) on their development/transition
impact. The EBRD’s policy requires 100% compliance and the control systems do not allow for diverting from this
policy. Each project document specifies benchmarks for “transition impact” outcomes that are monitored over
time during project implementation. EBRD policy requires annual or bi‐annual reviews of existing projects through
the preparation of monitoring reports, which assess developments in a project’s transition impact, environment
and financial performance. EBRD’s efforts to measure the wider development impact of its transition activities are
noteworthy. To assess the transition impact of its operations, the EBRD has developed a framework that captures
and monitors key transition aspects from the project design stage through to project closure. The OCE monitors
projects under the Transition Impact Monitoring System (TIMS). The TIMS also records assessments of risks to
transition in each project, and how these risks change over the project’s life.2
As stated above, EBRD has very strict procedures that all projects must abide by. Hence, the reporting done
against the project results framework imposed by the Fund reflects EBRD procedures/standards, which are in
general deemed both sufficient and relevant for the kind of activities financed by the Fund.
2.3. European Investment Bank
To further strengthen its assessment, measurement and reporting on the results and impacts of its operations, the
European Investment Bank (EIB) launched in 2012 a new Results Measurement framework (ReM) to replace its
Economic and Social Impact Assessment framework (ESIAF). One intention with this new ReM framework is to
harmonize, to the extent possible, results indicators with other International Financial Institutions (IFIs) and
thereby simplify client reporting requirements for co‐financed operations.3
The EIB’s ReM rests on three pillars:
Pillar 1: An assessment of a project’s consistency with EIB mandate objectives and its
contribution to European Union (EU) priorities and country development objectives, going
beyond a focus on eligibility alone.
1 AfDB. (n.d.). MOPAN Management Response (Draft 23 November). 2 EBRD. (2010). Transition Impact. [online].
3 EIB. (2013).Measuring development results. The EIB Results Measurement (ReM) framework.
“Creating a Macro Results Framework for the Middle East and North Africa Transition Fund” Final Report Appendix
3
Pillar 2: A series of sector‐specific standardized indicators to capture a project’s economic,
social, environmental and institutional outcomes. It continues to measure project quality and its
ability to achieve the expected results.
Pillar 3: An assessment of the EIB additionality over market alternatives in terms of financial
product, technical, structuring and sector contribution and standards and assurance.4
As noted above under Pillar 2, EIB uses standardized core indicators for the M&E of its entire portfolio and for
different sectors. It is expected that EIB will make use of these standardized core indicators to report for the Fund
activities and will appreciate the flexibility offered to conduct “business‐as‐usual”.
2.4. International Finance Corporation
The International Finance Corporation (IFC) has a Development Outcome Tracking System (DOTS) to track the
development results of all its active operations continuously throughout their project lives. For investment services
(IS) projects, the overall development outcome rating is a synthesis of four performance categories that are
informed by the indicators agreed with the client: financial performance, economic performance, environmental
and social performance and private sector development impact. For advisory services (AS), the overall
development outcome rating is a synthesis of the overall strategic relevance, effectiveness (as measured by
project outputs, outcomes, and impacts), and efficiency of the services. At project completion, intended results
are compared with achieved results. Some results—medium‐term outcomes and longer‐term impacts—may be
unknown at project completion but can be examined post‐completion.
Each IFC industry department has identified a menu of standard, sector‐specific indicators that capture the
sector’s typical development impacts. In addition, IFC has some wide corporate indicators that are mandatory for
tracking particular impacts related to environmental & social improvements and corporate governance issues. By
standardizing indicators, IFC is able to compare performance and aggregate developmental reach by industry
department and across the corporation as a whole for a particular time frame.5 At the outset of a project, IFC's
staff members identify standardized indicators with baselines and targets. They track progress throughout
implementation and until project closure. This systematic tracking allows IFC to monitor projects, learn from what
does and does not work, and improve performance. The system facilitates data‐aggregation, comparison,
analysis, and reporting on key output, outcome and impact indicators and efficiency of the services. To meet its
monitoring obligations, the IFC requires companies to submit regular reports on financial as well as social and
environmental performance, and information on factors that might materially affect the enterprise. In addition,
IFC monitors progress through it corporate scorecard and reports once a year on the development results of its
interventions in an Annual Report.
2.5. Islamic Development Bank
Evidence collected during interviews suggests that there are various types/levels of monitoring at the Islamic
Development Bank (IsBD). The IsDB has a monitoring tool at the portfolio level which looks like a dashboard of all
projects under implementation. To report on project progress IsDB possesses a Project Implementation Support
Report (PISR). The PISR highlights the challenges/issues the projects are facing in their implementation. The IsDB
4 Idem.
5 5 IFC. (2014). IFC Development Results and Impact. [online].
“Creating a Macro Results Framework for the Middle East and North Africa Transition Fund” Final Report Appendix
4
also sporadically conducts Country Portfolio Performance Reviews (CPPRs). The report that is generated goes to
the President directly, requesting guidance, changes and approval, and even in some cases termination of projects
that are not up to par. IsDB can also count on a “results unit” internally to look at all the PISR submitted. Data from
the different performance review tools are entered directly into database software that produces automated
reports as needed. These many layers of monitoring and reporting are particularly evident for IsDB project in
Yemen “Preparation and Implementation Support Project for the Special Industrial Zone Project in Hodeida”. For
this project, the project implementation team has to report at the Transition Country (TC) ministry level, to the
IsDB every quarter, and to the Fund twice a year.
Initiatives taken in recent years to strengthen project monitoring and reporting and direct it towards expected
outcomes include enhancing the IsDB’s field presence, increasing supervision budgets, and designing a supervision
report format that includes a focus on results. Information collected during interviews alluded that at the IsDB,
results‐based logical frameworks are mandatory in all projects and that core indicators are mainstreamed into
these results‐based logical frameworks. Data collected suggests that the IsDB reports on its own key performance
indicators (KPIs) in the Progress Report submitted to the Fund. The KPIs are collected through supervision
undertaken by the implementation unit at the project level.
2.6. Organisation for Economic Co‐operation and Development
The Organisation for Economic Co‐operation and Development (OECD) is generally known for being a forum in
which governments can work together to share experiences and seek solutions to common problems. OECD
usually provides analyses and recommendations on a variety of subjects and matters and is less thought of as a
“direct implementer” of projects in developing countries. Maybe it is for this reason, that OECD has comparatively
fewer demanding rules in terms of M&E vis‐a‐vis the other ISAs involved with the Fund, which are for the most
part quite used to dealing with such projects and tracking performance rigorously. Based on discussions held with
OECD representatives, the organization shows great flexibility with regards to the method and template for
project progress reporting. Thus, for the interventions funded by the Fund, the OECD uses only the Progress
Report template provided in Fund Operations Manual (p.37) and this reporting imposes no major burden on them.
2.7. World Bank
Generally speaking, the designs of the World Bank (WB) operational activities incorporate a framework for M&E.
During the project life, the Bank monitors and evaluates its own contribution to results using this framework.
Project teams compile information and data regularly in Implementation Status Reports (ISRs) and when projects
are complete, project teams are also required to complete an Implementation Completion and Results Report
(ICR). The latter document includes for example results achieved, problems encountered, and lessons learned.
These are later reviewed by the Independent Evaluation Group (IEG). In brief, the WB requires performance
reporting on a regular basis for all projects, and institutional procedures and systems are well in place to ensure the
tracking of progress and achievements of projects supported.
“Creating a Macro Results Framework for the Middle East and North Africa Transition Fund” Final Report Appendix
5
2.8. Others
M&E systems and procedures of the following ISAs were not looked at for this assessment in‐depth as the
representatives did not respond to the request to participate:
Arab Fund for Economic and Social Development (AFESD)
Arab Monetary Fund (AMF)
International Monetary Fund (IMF)6
Organization of the Petroleum Exporting Countries (OPEC) Fund for International Development
3. CASE STUDIES he case study will describe, for each fund, the structure of the logic model: the results chain from
outputs/activities to outcomes and impacts, related theory of change and results aggregation strategies.
This section will also describe the organization of the monitoring and reporting function: the
implementation arrangements by which results are collected as well as roles and responsibilities of the
various organizations involved in the funds/case studies.
3.1. The Global Environment Facility
The Global Environment Facility (GEF) is a financial mechanism for international cooperation, established in 1991,
based on partnerships, that provides new and additional grant and concessional funding to meet the agreed
incremental costs of measures to achieve agreed global environmental benefits ‐‐ in six focal areas7 ‐‐ in
developing countries and countries with economies in transition8. The GEF is a partnership involving 183 member
countries, 10 agencies (the WB, United Nations Development Programme (UNDP), United Nations Environment
Programme (UNEP), the regional development banks, Food and Agriculture Organization (FAO), the International
Fund for Agricultural Development (IFAD), and the United Nations Industrial Development Organization),
recipient countries, nongovernmental organizations (NGO), and the private sector.
GEF’s governance architecture is set out in the Instrument for the Establishment of the Restructured Global
Environmental Facility and consists of an Assembly, a Council, the Secretariat, the Implementing and Executing
agencies and the Scientific and Technical Advisory Panel (STAP). The Assembly and Council are the governing
bodies empowered to take decision. The others, while an integral part of governance, have only an advisory role.
As financial mechanism for four international environmental Conventions (Biodiversity, Climate Change,
Persistent Organic Pollutants, and Combating Desertification) the GEF helps fund initiatives that assist developing
6 It should be noted that representatives from the IMF said they were keen to participate in the assessment, but had nothing of direct relevance to share at the time of the data collection. 7These areas are: biological diversity, climate change mitigation and adaptation, international waters, land degradation, ozone layer depletion, and persistent organic pollutants, with sustainable forest management cross‐cutting relevant focal areas. 8 GEF. (2013). GEF Website. Investing in our Planet. [online].
T
“Creating a Macro Results Framework for the Middle East and North Africa Transition Fund” Final Report Appendix
6
countries in meeting the obligations of the Conventions. GEF also collaborates closely with other treaties,
protocols and agreements9.
3.1.1. Overview of the GEF Results Framework
The following presents an overview of the GEF results framework, referring to its logic model and theory of change
as well as the results aggregation strategy, the role of evaluations, and finally, the roles and responsibilities for
M&E between GEF partners.
Logic Model and Theory of Change
GEF developed in 2012, 21 years after the inception of GEF, a fund level theory of change (please see Figure 1
below). The general framework theory of change seems to be meant to capture lessons learned.
Figure 1. General Framework for GEF Theory of Change10
Aggregation and Reporting
GEF, monitoring take place on three levels:
1. Project and program level: progress toward results, including outcomes, implementation issues, and
portfolio‐wide trends is based on information submitted by the GEF Agencies through the project or
program implementation reports (PIR) and focal area tracking tools;
2. Portfolio level: The aggregation of findings across the portfolio of projects and programs can be done by
focal area, type, theme, or issue. Portfolio monitoring tools are intended to roll up indicators from the
9 Pérez del Castillo, Carlos (GEF). (2009) OPS4 Technical Document # 5: Governance of the GEF.
10 GEF EO. (2012). Annual Thematic Evaluations Report (2012).
“Creating a Macro Results Framework for the Middle East and North Africa Transition Fund” Final Report Appendix
7
individual project level to the portfolio level and track overall portfolio performance in focal areas. Each
focal area has developed its own tracking tool to meet its unique needs, which brings uniformity. It is a
requirement of the GEF that the implementing agencies fill these tracking tools to report to the GEF;
3. National and global level: mainly of global environmental status, stress, trends, and benefits, based on
independent data gathering and analysis by national bureaus of statistics and/or international bodies and
organizations; these monitoring exercises occur mostly between replenishment periods when reassessing
and redesigning indicators for country allocations under any GEF system for allocation of resources, and
overall when formulating the focal area strategies.11
In brief, there is no single, overarching macro framework per se for GEF. GEF results are organized using what can
be called a “Project Portfolio” approach composed of results frameworks for each “focal” area of work against
which each project in that focal area has to report.12 This is a key element of GEF’s results aggregation process. As
stated, aggregation at the project and portfolio level of the GEF requires the completion of a PIR. However,
reporting of results at this level is based on a narrative and a self‐evaluation using a standard set of categories,
much like the ratings in the standard Progress Report for the MENA Transition Fund, except that the rating for the
GEF range from Highly Satisfactory (HS) to Highly Unsatisfactory (HU). Table 1 below is an excerpt from a PIR.
Table 1. Excerpt from GEF PIR13
OVERALL RATING OF IMPLEMENTATION PROGRESS PREVIOUS RATING
NEW RATING
U MS
Outputs and Outcomes
Component 1: Most of the products established to achieve the outcomes of this component have been executed according to plan. However, there are significant delays in the demarcation of the Marine Protected Area, currently just 20% of the area has been demarcated while the project life is already almost 50% complete. Also, the Government has issued a law to consolidate the protected areas management category of the national system. If this regulation is approved, the outputs related with the Integrated Management Plan updating and demarcation could be affected. In conclusion, four (4) of the twelve (12) outputs planned for the component show delays in their implementation.
The consequences to such a broad‐based, generalized, subjective approach to results aggregation is that the
discussion on results achievement at the fund level is limited. Given the self‐evaluative nature of the ratings, there
is also a risk that the results quoted above could be due to bias. Secondly, the narrative approach to performance
reporting may contribute to the perception by key informants that GEF imposes a reporting burden on projects.
In addition, GEF has developed performance tracking tools14 per work or focal area as follow:
Biodiversity Tracking Tool;
GEF International Waters Tracking Tool;
GEF Climate Change Mitigation Tracking Tool;
GEF Climate Change Adaptation Tracking Tool;
GEF Persistent Organic Pollutants Tracking Tool;
GEF Land Degradation Tracking Tool;
GEF Sustainable Forest Management/REDD Tracking Tool.
11 GEF. (2010). M&E Policy.
12 For more details, please see GEF. (2010). Focal Area Framework and LDCF/SCCF Framework. Excerpts from GEF‐5 Programming
Document at the Sixth Meeting for the Fifth Replenishment of the GEF Trust Fund, April 7, 2010. 13 Inter‐American Development Bank (IDB). (2012). Project Implementation Report (PIR) FY2012.
14 For more details on these tracking tools, please refer to GEF Website: http://www.thegef.org/gef/tracking_tools.
“Creating a Macro Results Framework for the Middle East and North Africa Transition Fund” Final Report Appendix
8
These tracking tools can be conceptualized as a results framework per area of work or focal area. The results that
are tracked are both “hard” indicators such as, “Landscape/seascape area directly covered by the project (ha)”15
and the standardization and quantification of “soft” indicators related to institutional reform, see excerpt below.16
The main benefit of basing fund‐level aggregation on focal area results is that reporting at higher levels is
facilitated.
Evaluation
The GEF fund is typical of OECD Development Assistance Committee (DAC)‐based17 evaluation standards where
performance and evaluation reporting take place at every level (project, country, and agency)18. Established 23
years ago, the scope and reach of the GEF means that it is possible to report frequently and reliably on results up
to impact level. Reflecting its longevity and resourcing, the GEF Evaluation Office (EO) publishes annual results at
the fund and country levels, as well as by theme19. The EO has three key functions which permits the
dissemination of M&E standards fund‐wide: the setting of minimum standards; quality‐control and oversight on
the design of M&E systems at the program and project levels; as well as the sharing of evaluation evidence20.
Roles and Responsibilities for M&E
M&E are a shared responsibility in the GEF. The GEF Council provides the overall framework, starting with
agreement on objectives and corporate and focal area results frameworks. The GEF Secretariat proposes to the
Council how these objectives and results should be monitored, and the GEF EO proposes to the Council how these
should be evaluated. Emerging environmental and development trends, and GEF results and performance within
the context of these trends, are reported on in the overall performance study prepared by the GEF EO as one of
the key documents of the fund replenishment process. Based on this information, the GEF Council makes strategic
and policy‐level decisions. The GEF Agencies and their partners execute project, program, and portfolio M&E. The
GEF STAP provides advice on indicators, targets, and evaluation approaches. The GEF EO collaborates with the
independent evaluation units of the GEF Agencies to enhance collective capacity to fulfill evaluation needs
effectively and efficiently.21
Consistent with the decentralized organization of the GEF, implementing agencies are expected to use their
own M&E systems. Each GEF Agency has its own system of governance and rules and regulations governing the
implementation of activities, as well as the M&E of these activities. However, the GEF Council can decide on which
partners it collaborates with and can require minimum standards and minimum procedures to be applied in
activities that it funds.22
The decentralized roles and responsibilities for M&E are counter‐balanced by an independent GEF EO that reports
directly to the GEF Council. In addition, the M&E Office has a separate budget and its own program of work, both
to be approved by the Council.23 Roles and responsibilities for M&E at the GEF are summarized in Error! Reference
source not found. and graphical form in Figure 2 below.
15 GEF. (2011). GEF‐5 Tracking Tool for Biodiversity Focal Area.
16 Idem.
17 GEF. (2010). GEF M&E Policy.
18 Binnendijk, Annette. (2000). Results Based Management in the Development Co‐operation Agencies: a Review of Experience.
19 GEF. EO. (2013). Independent Evaluation Office Website. [online].
20 GEF. EO. (2013). Independent Evaluation Office Website. [online].
21 GEF. (2010). GEF M&E Policy.
22 Idem.
23 De Crombrugghe, Dominique, et al. (2009) Peer Review: the Evaluation Function of the Global Environment Facility (GEF), Final
Report.
“Creating a Macro Results Framework for the Middle East and North Africa Transition Fund” Final Report Appendix
9
Table 2. Key Roles and Responsibilities of GEF Partners in M&E25
Partner Key Roles and Responsibilities in M&E
GEF Council Policy making on M&E Oversight of M&E functions Enabling environment for M&E
GEF EO Independent GEF evaluation Oversight of project and program evaluations Oversight of the relevance, performance, and overall quality of monitoring systems Set minimum requirements for GEF M&E Evaluative knowledge sharing and dissemination
GEF Secretariat Set results frameworks at focal area and corporate levels GEF portfolio monitoring across Agencies and focal areas Report on and incorporate lessons from portfolio monitoring Review of GEF M&E requirements in project and program proposals Coordinate partnership knowledge management activities
Agency GEF operational units
Monitor the Agency GEF portfolio Report Agency’s project, program, and portfolio progress, results, learning, and lessons Ensure monitoring at the project and program levels Adaptive management of project and program implementation Systematically involve national partners and share project M&E information at the national level
Agency evaluation units
Project, program, and/or corporate Agency independent evaluations Mainstream GEF into relevant Agency evaluations
STAP Advice on scientific/technical matters in M&E Support to scientific and technical indicators Support knowledge management and information sharing
GEF operational focal points
Collaborate on M&E at project, program, and portfolio levels
Others (NGOs and civil society organizations (CSOs), private sector, community members)
Participate in monitoring activities and mechanisms Provide views and perceptions to evaluations
24 Volonte, Claudio. (GEF EO). (2011). The GEF M&E Policy. PPT Presentation.
25 GEF. (2010). GEF M&E Policy.
Figure 2. Flow Chart of M&E Reporting in the GEF
24
“Creating a Macro Results Framework for the Middle East and North Africa Transition Fund” Final Report Appendix
10
Another counter‐balance to the shared responsibility for M&E at the GEF is strong knowledge management of the
M&E function at fund‐level. As per its mandate, the EO publishes guidelines on M&E in the form of policy
documents26. In addition to policy statements on M&E, the EO also produces technical guidelines and specific
tools27. A good example of the relatively high‐quality advice given on M&E is the Review of Outcomes to Impacts
(ROtI) methodology document28, which also reflects the predictable shift away from monitoring to evaluation29
(i.e. a focus on outcomes and impacts), given the longevity of GEF. Another key aspect of EO activity is the role of
facilitator in the exchange of M&E perspectives within the GEF “family” (i.e. consultants, implementing agencies,
national recipients, and so on). The GEF EO has sponsored such events as an Evaluators’ Summit and an
International Conference on Evaluation, Environment and Sustainable development.30 The key consequence of
these knowledge management activities, widely disseminated digitally, is the possibility to aggregate results at
the fund level.
3.1.2. Strengths, Weaknesses and Opportunities for
the MENA Transition Fund
Each of the funds discussed in this case study section have strengths and weaknesses in terms of learning lessons
for the potential development of a Macro Framework for the MENA Transition Fund. However, each fund also
have “checks and balances” to offset the weaknesses of their respective approaches.
The weakness of GEF is its performance reporting at project level which seems to yield a limited amount in the
form of aggregated results. The “check and balance” against this is the development of what is effectively a “layer”
of results frameworks per area of work or focal area and by developing, ex ante, results categorizations for a fund‐
level theory of change. Yet, perhaps because of the narrative format of the PIR, this particular tool and the
tracking tools are at the same time perceived as imposing a reporting burden on projects.
In terms of strengths, fund‐level governance of M&E at GEF is strong with reporting lines for M&E going directly to
the GEF Council. The appetite for the production of (self‐described) “cutting‐edge” M&E knowledge at fund‐level
is also strong. The “check and balance” against a central approach to the M&E function is a strong knowledge
management function that widely distributes M&E publications produced at the fund‐level. In terms of value for
money, a “top‐down”, central approach where evaluation knowledge is produced at the fund‐level can mean
increased administrative costs at the fund level. Value for money is questionable with respect to an ex‐ante
approach to results aggregation. This process requires primary research.
There are many aspects of the M&E regime at GEF which can be adopted by the MENA Transition Fund, albeit
within a solid understanding of the key differences. The following are a few points to consider:
The MENA Transition Fund could continue with its “devolved” M&E function at the project level,
but put more emphasis on building sustainability and capacity for M&E on the part of ISAs and
TCs. Strengthening the M&E capacity of line ministries in TCs will indirectly strengthen fund‐
level reporting while improving capacity;
26 GEF. EO. (2013). Independent Evaluation Office Website. [online].
27 Please see, for example, GEF. (2011) GEF‐5 Tracking Tool for Biodiversity Focal Area.
28 GEF EO. (2009). The RoTI Handbook: Towards enhancing the impacts of environmental projects, Methodological Paper #2.
29 De Crombrugghe, Dominique, et al. (2009). Peer Review: the Evaluation Function of the Global Environment Facility (GEF),
Final Report. 30 Idem.
“Creating a Macro Results Framework for the Middle East and North Africa Transition Fund” Final Report Appendix
11
Careful planning of the project‐level performance reporting regime to avoid costly “re‐
engineering” of Fund‐level aggregated results;
Adopting a “layered” results framework approach to facilitate aggregation of the diverse type of
projects funded;
GEF produces M&E research on impacts (and other topics) in addition to report on information
coming from the projects only. These additional pieces are not currently necessary for the MENA
Transition Fund. Keeping things simpler and more focused at this early stage could allow the
MENA Transition Fund to keep costs at a more manageable level compared to GEF.
Notwithstanding the greater sectoral breadth that the MENA Transition Fund covers, there are lessons to be
consider: for example, there is a decentralized approach to M&E at the GEF that is offset by: an independent M&E
function at the fund level; the distribution of high‐quality M&E guidance regarding policy and technical issues; as
well as a focus on knowledge management.
The approach to aggregation of results with respect to developing a theory of change could also be instructive for
the MENA Transition Fund. “Aggregation” is the classification of project results identified as important to bringing
about the desired impact(s). The development of a theory of change for a focal area is a combination of inductive
(“bottom up”) and deductive (“top down”) reasoning. It is inductive in that the results frameworks for projects in a
focal area are analyzed and a properly‐designed theory of change is developed for each. It is deductive in that,
using some taxonomy, an internationally‐accepted standard for that focal area is preferred, intended project
results are then categorized, based on projects’ theory of change. GEF provides many valuable technical insights
on the development of an aggregation method using both inductive and deductive reasoning. However, the
“downside” of an inductive approach as developed by the GEF is that projects’ results frameworks and theory of
change have to be analyzed and, depending on analysis findings, redesigned retrospectively. The approach is
therefore time/cost intensive, most likely requiring primary research. Also, compared to MENA Transition Fund,
GEF has a critical mass of completed projects from which to draw conclusions about project commonalities
through an impact analysis.
There are other factors which militate against wholesale adoption of the GEF M&E approach by the MENA
Transition Fund. Since 2003, GEF has focused on the measurement of impacts and lessons learned. This is to be
expected, given the longevity of and resources channeled through the GEF. However, the MENA Transition Fund is
on the other end of the spectrum. It is a facility barely out of the inception stage, with a fraction of funds in
operations compared to GEF. In other words, the focus of the MENA Transition Fund would be on results “lower”
down the results chain: activities; outputs and outcomes. In this, the GEF has relatively little guidance to offer,
except that implementing agencies should use their own systems and that the GEF only has a quality assurance
and standards‐setting role with respect to these. Moreover, the functions created by the GEF took years to
develop, are costly to maintain and were set up for a much larger fund with a long term horizon. Thus, the
strategies and methodologies used by the GEF would need to be carefully adapted for potential consideration in
the context of the MENA Transition Fund in particular as the latter could not bear a comparability cost footprint.
“Creating a Macro Results Framework for the Middle East and North Africa Transition Fund” Final Report Appendix
12
3.2. Global Agriculture and Food Security Program
The Global Agriculture and Food Security Program (GAFSP) is a multilateral mechanism to assist in the
implementation of pledges made by the G20 in Pittsburgh in September 2009. The GAFSP is a financial
intermediary fund (FIF) under external governance, established by the WB Group to support inclusive, strategic
and country‐led agriculture and food security investment plans in poor countries. GAFSP makes funding available
for countries that exhibit both need and significant readiness to use funds well for increased long‐term food
security. To date, Australia, the Bill & Melinda Gates Foundation, Canada, Ireland, Japan, the Netherlands, South
Korea, Spain, United Kingdom (UK), and the United States of America (USA) have contributed resources to the
GAFSP. These ten donors have pledged a total of US$ equivalent 1.4 billion to the Public and Private Sector
Windows. The resources received from the donors as of December 31, 2013 amounted to approximately US
equivalent $1.2 billion. Eligible countries must demonstrate a high level of need and submit a comprehensive,
technically sound project proposal along with their national agricultural investment plans. GAFSP (public and
private sector windows) has allocated funds for the following countries: Bangladesh, Burkina Faso, Burundi,
Cambodia, Cote d’Ivoire, Ethiopia, Haiti, Honduras, the Gambia, Kenya, Kyrgyz Republic, Liberia, Malawi, Mali,
Mongolia, Nepal, Nicaragua, Niger, Pakistan, Rwanda, Senegal, Sierra Leone, Tajikistan, Tanzania, Togo, Uganda,
Vanuatu, Yemen, and Zambia. Each GAFSP grant focuses on filling a specific need for the nation’s food security
and agricultural growth.
In terms of governance, GAFSP is similar if not identical to that of the MENA Transition Fund. In GAFSP, the WB
plays three roles: (1) Trustee (to carry out fiduciary responsibilities as set out in the Trust Fund Administration
Agreement), (2) CU (a small team to support the GAFSP SC and Technical Advisory Committee and facilitate
communications between the SC and the other partners of GAFSP), and (3) potential Supervising Entity of GAFSP
funded projects, if selected by the SC. IFC, the WB’s private sector arm, manages the Private Sector Window of
GAFSP. Allocations are determined by a SC composed of a balance of representatives from donor and recipient
countries, CSOs, and multilateral organizations, following advice from a technical committee of independent
experts. 31
3.2.1. Overview of GAFSP Results Framework
The following is a discussion of the GAFSP results framework, referring to its logic model and theory of change as
well as the results aggregation strategy. The section also looks at the role of evaluation and the roles and
responsibilities of the different partners involved.
Logic Model and Theory of Change
Figure 3 below is a depiction of the GAFSP’s objectives and areas of intervention which shows the intervention
logic and how the fund expects to deliver results (i.e. theory of change).
31 GASFP. (2011). GAFSP Website. [online].
“Creating a Macro Results Framework for the Middle East and North Africa Transition Fund” Final Report Appendix
13
Figure 3. GAFSP Logic Model and Theory of Change32
Aggregation and Reporting
The aggregation strategy for the GAFSP includes the development of common indicators to enable monitoring of
results across projects: “Each GAFSP financed project will select from the menu of core indicators (…) at least one
output or intermediary outcome indicator for the results areas of focus of the GASFP project. These indicators will
be included in the respective project results frameworks. This will allow for aggregation of results across
projects.”33 An excerpt of the “menu” of indicators appears in Table 3 below.
32 GAFSP. (2013). GAFSP Annual Report.
33 GAFSP. (2011) Monitoring and Evaluation Plan.
“Creating a Macro Results Framework for the Middle East and North Africa Transition Fund” Final Report Appendix
14
Table 3. Sample of “Menu” Approach to the Fund‐level Results, GAFSP34
Results Area Menu of Common Project Level Progress Indicators (Output and Intermediate Outcomes)
Higher‐yielding technologies adopted
Technology generated
1. Number of collaborative research or extension sub‐projects implemented 2. Number of client days of training to raise agricultural productivity provided to scientists, extension agents, agro‐dealers, farmers, community members etc (disaggregated by gender) 3. Number of client days of extension services provided to farmers, community members etc (disaggregated by gender) 4. Number of farmers who have adopted the technology being promoted 5. Number of additional hectares which have adopted the technology being promoted
In developing these core indicators for the GAFSP, the CU leveraged on the work of others to diminish the burden
of this task. They reviewed the IDA core sector indicators and core indicators listed in the publication by the Global
Donor Platform for Rural Development, FAO, and the World Bank (GDPRD/FAO/World Bank 2008). The selection
of core indicators was a participatory process. An earlier draft was circulated to the Supervising Entities and their
comments have been incorporated before moving forward. GAFSP document mention that these indicators were
selected using the following principles:
1. “The indicators are deliberately set at the output and intermediate outcome levels and have been
selected in order to have parsimonious but wide applicability to GAFSP‐funded interventions.
2. The indicators do not provide an added burden for individual project M&E frameworks, rather provide
indicators that would be included anyway, but in a form that is readily comparable across different GAFSP
projects.”35
There are two noteworthy aspects of the example above. First, it is the specificity of the results indicator to the
GAFSP component or area of work, which is, possible, to a certain extent, because of the sectoral focus on
agriculture of the GAFSP. Secondly, given that it may take many years of project operation before it is possible to
report on progress made on higher‐level results and the associated indicators, GAFSP focuses on aggregated
results reporting at the output and intermediate outcome level.
Evaluation
GAFSP SC selected the WB's Development Impact Evaluation Initiative (DIME) to implement in‐depth impact
evaluations for certain GAFSP Public Sector Window projects. DIME is a global program hosted in the WB's
Development Research Group. Its purpose is to increase the use of impact evaluation (IE) in the design and
implementation of public policy and increase institutional capacity and motivation for evidence‐based policy. The
GAFSP M&E Plan notes that all GAFSP projects will undergo an independent evaluation as well as an IE. The
following table outlines the IE policy for the GAFSP.
34 Idem.
35 Idem.
“Creating a Macro Results Framework for the Middle East and North Africa Transition Fund” Final Report Appendix
15
Table 4. Impact Evaluation Policy for GAFSP36
Type of IEs Target Indicative Cost Implementation Funding
In‐depth IEs (experimental or quasi‐experimental IE)
Approximately 10‐30% of all GAFSP projects selected according to lesson‐learning priorities
$800,000 for each IE CU to centrally contract a specialized organization to carry out all in‐depth IEs
To be funded separately by the SC
Rapid IEs (non‐experimental IE)
All GAFSP projects (except for those that have been chosen to carry out the in‐depth IEs)
$50,000 for each IE Each project to contract a specialized organization to carry out its rapid IE
To be funded from each GAFSP‐awarded Grant amount
Roles and Responsibilities for M&E
Similar to the GEF and to the MENA Transition Fund, GAFSP delegates project implementation to the
“supervising entity”. As a matter of fact, GAFSP has to a great extent “pushed” M&E capacity down to the
national government level, and provided support for this explicitly in the design of the program. Supervising
agencies are to use their own systems to conduct project‐level supervision. This combined with strengthened
national government capacity of M&E systems means increased donor coordination at the country level for fund‐
level results management. As a counterbalance, each supervising entity that receives funding under GAFSP have
to report on progress and results for all activities to the GAFSP CU at the WB, which then consolidate reporting
and submit as an annual implementation results report to the GAFSP SC, with copy to the Technical Advisory
Committee.37
The potential weakness of this approach has to do with “The accuracy of all reporting is the responsibility of the
originating supervising entity, and not the GAFSP CU, whose role is consolidation and not quality control of
individual contributions.”38, which is similar to the role of the MENA Transition Fund CU. However, an off‐setting
factor is an explicit mention of M&E as a specific (therefore, presumably fundable) component of GAFSP, under
Technical Assistance, Institution‐Building, and Capacity Development.39 In addition, as mentioned above, the
evaluation function is delegated to the DIME.
Below is a summary table describing the roles and responsibilities for M&E between the GAFSP supervising
entities; CU, SC, and the DIME.
Table 5. Roles and Responsibilities for M&E, GAFSP
Partner Key Roles and Responsibilities in M&E (not exhaustive)
Supervising Entities for the Public Sector Window (Asian Development Bank (ADB), AfDB, FAO, IDB, IFAD, WB, World Food Programme (WFP)
Follow their own guidelines on project M&E during all phases of the project cycle Projects under implementation: Submits supervising entity’s own progress report (or a disclosable version) every 6 months Projects under preparation: Submits a brief progress update with updated milestone dates every 6 months Completes geographic information form Submits supervising entity’s own completion report (or a disclosable version) Submits supervising entity’s own independent evaluation report (or a disclosable version) Submits rapid IE report (if the project is not selected for an in‐depth impact
36 Idem.
37 GAFSP. (2009) Framework Document for a Global Agriculture and Food Security Program (GAFSP).
38 Idem.
39 Idem.
“Creating a Macro Results Framework for the Middle East and North Africa Transition Fund” Final Report Appendix
16
Partner Key Roles and Responsibilities in M&E (not exhaustive)
evaluation)
Supervising Entity for the Private Sector Window (IFC)
Follows its own guidelines on project M&E during all phases of the project cycle To develop M&E Plan for the Private Sector Window including any policy on IEs Submits program level annual development impact indicators based on IFC’s DOTS to CU for onward transmission to the SC
CU Prepares template TOR for rapid IEs Contracts specialized organization to geo‐code submitted geographic information and carry out mapping of GAFSP projects Contracts specialized organization to carry out in‐depth IEs Submits GAFSP annual progress report to the SC Discloses relevant M&E documents on the GAFSP website (including updating the progress of GAFSP core indicators every six months) Carry out other dissemination activities such as organize global and regional workshops and liaise with relevant organizations
SC Reviews annual progress report and provides overall guidance on project implementation
DIME Responsible for IEs
3.2.2. Strengths, Weaknesses and Opportunities for
the MENA Transition Fund
Overall, key informants rated the GAFSP macro results framework as having many strengths:
“supports planning and approval of project proposals;
tells the story of the fund at the highest level;
provides evidence to make the business case for financing;
macro results are tied to Millennium Development Goals (MDGs) and international commitments
on food security;
incentivizes the partners to go into certain programming areas that are underserved;
provides a basis to “adjust the dial” on incoming proposals by tweaking the Call for Proposals to
ensure the kinds of proposals you require to achieve the full range of results will be produced”.40
A strength of GAFSP is the devolved nature of the M&E function from the “centre” (i.e. Fund Level). The self‐
described role of the GAFSP CU at the fund level is that of coordinator for reporting. There is therefore reason to
believe that the administration cost of the M&E function at fund level is less compared to a more centralized
approach such as that of GEF. On a US$1.2B fund, there are five people in the secretariat, with a low overhead
ratio).41
The weakness of a more devolved approach is that of the accuracy of information reported. With respect to
GAFSP, a “check and balance” against a “hands‐off” approach is to support partners’ M&E capacity by making
M&E systems development and knowledge management a fundable program component. This “push” of the M&E
capacity to the country level means that M&E systems development is sustainable as both the national
government and the ISAs are implicated in supporting it past the life of the fund. Therefore, the GAFSP approach
to M&E is also good value for money as M&E capacity development is not an administration cost (i.e. fund
“overhead”) but part of program spending. Another “check and balance” is the provision of results reporting
40 DFATD. (2014). MENA Transition Fund Results Feasibility Study: Questions and Answers for an Interview with Baastel Ltd.
41 DFATD. (2014). Op. cit.
“Creating a Macro Results Framework for the Middle East and North Africa Transition Fund” Final Report Appendix
17
standards at the project level as per the “menu” approach to fund‐level results reporting. This allows both ISAs to
continue to use their own systems but also to aggregate results at the Fund‐level.
A significant aspect for consideration by the MENA Transition Fund is the priorities of the DIME program which
aligns well with the MENA Transition Fund objectives. The four pillars of DIME are: Governance (local
development, public sector governance, and legal justice); Shared Prosperity (Finance, Private Sector, and
Agriculture and Food Security); Infrastructure (Transport, Urban Development, Energy and Mining; and Water);
Human Development; and Climate Change. In addition, the DIME website also cites the cross‐cutting theme of
Fragile and Conflict Situations as another area of work. However, and as outlined in Table 4 above, IE are costly to
implement.
In sum, with respect to what can be applied to the MENA Transition Fund, the approach where: a) the M&E
function is devolved; b) but with offsetting capacity development for M&E “pushed” to country level via project
spending; c) while developing standards (such as a “menu” of results), should be given serious consideration.
3.3. Neighbourhood Investment Facility
The Neighbourhood Investment Facility (NIF) represents the “counter‐factual” case. That is to say, it is an example
of a funding facility that does not have a macro‐level results framework. Thus, in some aspects, NIF represents the
status quo for the MENA Transition Fund.
Launched in May 2008, the NIF has been designed to finance capital‐intensive infrastructure projects in partner
countries covered by the European Neighbourhood Policy (ENP) as well as to support their private sector. For the
2007‐2013 period, the European Commission (EC) has earmarked a total amount of €767 million for the NIF, which
are complemented by direct contributions from Member States kept in a trust fund managed by the EIB. To date,
the NIF contributes €678.9 million to infrastructure and private sector projects, leveraging a total project volume
of more than €19.3 billion. Financing and implementing large infrastructure projects requires considerable
financial resources. The NIF aims to create a partnership, pooling together grant resources from the EC and the EU
Member States and using them to leverage loans from European finance institutions as well as own contribution
from the ENP partner countries.42
The organization of the NIF follows the principles of the three‐tier governance architecture agreed between the
EC and the Member States. The partner countries submit projects to the European finance institutions to seek
finance. Between them, the European finance institutions designate a Lead Finance Institution, which presents the
project to the Finance Institutions Group (FIG). The FIG is a group made up of all the eligible European finance
institutions and chaired by the EC. The Lead Finance Institutions then present the NIF grant requests to the
Operational Board for final or provisional approval. The lead administrative body is the NIF Secretariat.43
3.3.1. Overview of NIF Results Framework
The following is a discussion of the NIF results framework, referring to its logic model and theory of change as well
as the results aggregation strategy, and roles and responsibilities for M&E.
Logic Model and Theory of Change
42 Idem.
43 NIF. (2013). Activity Report: 2008‐2012 Overview of Activities.
“Creating a Macro Results Framework for the Middle East and North Africa Transition Fund” Final Report Appendix
18
The NIF logic model depicted in Figure 4 below was developed retrospectively for the purposed of the mid‐term
evaluation, c0nducted in late 2013.
Figure 4. NIF Logic Model44
Aggregation and Reporting
In the mid‐term evaluation the aggregation strategy used was that project results and objectives were analyzed
retrospectively and evaluated for “fit” against the retrospectively designed intervention logic model. The
evaluators noted in their report under the efficiency section that, “The Governance would benefit of a feed‐back
system (monitoring, evaluation and lesson learning) from projects to inform decision making and strategies.” They
also noted the following, “The establishment of a comprehensive monitoring for all projects is an appreciable
effort which would require adequate resources and management capacities. The monitoring should not be
considered a duplication of Finance Institutions’ monitoring as its focus would be on the grant component and
achievement of blending and EC objectives. Costs would be appreciable, but contained, likely less than 1% of the
value of the grant. Benefits in terms of aid effectiveness and improvement of the instruments would be significant
(…) Improved reporting should not significantly accrue the cost and the time for Financing Institutions and
partners, but may help to structure the information for an improved understanding and assessment of NIF
achievements.”45
To strengthen the process, the evaluators made the following recommendations which are noteworthy:
44 NIF. (2013). Mid‐Term Evaluation of the Neighbourhood Investment Facility under the European Neighbourhood and Partnership
Instrument (ENPI) 2007‐2013. 45 DRN‐ECDPM‐ECORYS‐PARTICIP. (2013). Mid‐Term Evaluation of the Neighbourhood Investment Facility under the European
Neighbourhood and Partnership Instrument (ENPI) 2007‐2013.
“Creating a Macro Results Framework for the Middle East and North Africa Transition Fund” Final Report Appendix
19
“Develop a result based framework for all NIF projects with harmonized impact and performances
indicators (e.g. a system of impact and performance indicators should be developed to allow data
management and analysis);
Monitor and measure NIF induced changes at project, regional and portfolio levels;
Strengthen evaluation arrangements at project level.”46
Roles and Responsibilities for M&E
“NIF has not developed specific and harmonized procedures for monitoring the facility and has not established a
mandatory result‐based framework for following up the interventions.”47 Project monitoring is based on periodic
(annual) assessment of progress on delivery of specified project results towards achievement of project objectives.
Under the Joint management implementation modality, responsibilities for management and implementation
(and hence monitoring) are delegated by the EC to the financing institution, although the EC maintains the
“prerogative of control and verification”.48
Project monitoring and follow‐up are delegated to the Lead Finance Institution. NIF mentions its intention is to
avoid duplicating tasks already performed by a Finance Institution for which procedures have been positively
assessed (i.e. are supported by result‐based systems). Then the EC relies on external independent audits and
evaluation foreseen by the Lead Finance Institution whenever it is appropriate.
Table 6 below briefly summarizes the main roles and responsibilities of the different partners.
Table 6. Key Roles and Responsibilities for M&E ‐ NIF49
Partner Key Roles and Responsibilities in M&E
Strategic Board Oversees NIF’s activities Determines general strategic orientations
Operational Board Approve (or reject) projects submitted
FIG Discuss technical aspects of projects presented with the EC
Finance Institutions Present projects to the FIG
Beneficiaries Country’s systems are used whenever possible
3.3.2. Strengths, Weaknesses, and Opportunities for
the MENA Transition Fund
As mentioned above, NIF represents in many ways the status quo for MENA Transition Fund. The following
strengths and weaknesses for the NIF model are thus also those which potentially apply to the MENA Transition
Fund.
In terms of value for money, there are some potential cost‐efficiencies achieved in the short term; however, in the
longer term, the lack of a fund‐level macro framework can be less efficient, as noted by the evaluators during the
mid‐term evaluation. Among the main reasons for this is that the lack of harmonized procedures or ways to
aggregate results at a macro level can hinder the ability to assess whether the fund is achieving its intended results
or not over time, and may impede the ability to take corrective actions in time or to learn from experience over
46 DRN‐ECDPM‐ECORYS‐PARTICIP. (2013). Op. cit.
47 EC. (2013). Development and Cooperation – EuropeAid. Promoting investment through the Neighbourhood Investment Facility
(NIF). [online]. 48 Idem.
49 EuropeAid. (2011). Neighbourhood Investment Facility (NIF). PPT Presentation.
“Creating a Macro Results Framework for the Middle East and North Africa Transition Fund” Final Report Appendix
20
time. Overall, the mid‐term evaluation report seems to point towards the argument that, moving forward, a
macro‐results framework is necessary to measure the impact of NIF and must be supported institutionally by
reporting coordination (“harmonization”) and an M&E system at fund‐level. This likely increases costs, but also
provide important benefits as it helps to structure the information for an improved understanding and assessment
of the achievements.
3.4. Conclusion
This section provides a brief synthesis of lessons learned relevant to the development of a Macro Framework for
the MENA Transition Fund.
The development of a Macro Framework is more often than not a sign of maturity of the fund. As was mentioned
above it is reasonable to expect that GEF would focus on outcomes and impacts given its longevity. On the other
hand GAFSP, a relatively recent fund, focuses on outputs and intermediate outcomes. NIF made a tactical decision
not to develop a fund‐level results framework at inception, as in the case of the MENA Transition Fund. However,
the authors of the mid‐term evaluation suggested that a Macro Framework be developed for NIF as a tool to track
and assess performance overall.50
With respect to the aggregation strategy for a Macro Framework, the GEF methodology of a “layered” approach
to fund‐level results framework is instructive. Essentially, there is no overarching results framework; rather there is
a series of results frameworks per work area. This provides a certain reporting flexibility, but most importantly,
allows to aggregate results from vast sectors by categorizing projects under certain “pillars”. Instructive as well is
the GEF “top down/bottom up” approach of developing fund‐level results based on an analysis of GEF projects.
There are also lessons to be learned from GAFSP strategy for results aggregation which involves requiring projects
to choose from a “menu” of outcome and intermediate results specific to the area of work, notwithstanding the
narrower sectoral focus of the GAFSP as compared to the MENA Transition Fund.
For the case studies that do have a Macro Framework (i.e. GAFSP and GEF), with respect to the governance of
M&E at fund level, the approaches are fairly similar but with different emphases. Both are similar in a sense that
results coordination function is filled by the provision of standard reporting tools and templates. The emphasis
placed on specific M&E functions is determined by the perceived role placed by fund management, regarding
M&E. GEF, compared to GAFSP, emphasizes knowledge management or, more precisely, knowledge production;
publishing several documents on M&E policy, technical issues and the results of primary research. GEF also
communicates M&E best practices through global conferences such as International Conference on Evaluation,
Environment and Sustainable Development in Alexandria, Egypt. On the other hand, GAFSP self‐described role is
less as a producer of cutting‐edge evaluation practice and more as a coordinator of fund‐level reporting and a
developer of capacity for M&E, delegating important functions to the implementing agencies and funding capacity
for M&E and knowledge management at the country level. Also, GAFSP seemingly chooses to concentrate on
intermediate results, delegating impact assessments to DIME. Importantly, GAFSP has “pushed” M&E capacity to
the country level and provided explicit support for this in program design.
50 DRN‐ECDPM‐ECORYS‐PARTICIP. (2013). Op. cit.
“Creating a Macro Results Framework for the Middle East and North Africa Transition Fund” Final Report Appendix
21
4. OPTIONS his section presents in details the following options: a) the “enhanced” status quo (i.e. with some potential
minor changes to consider that might improve on the current process); b) moving forward with developing
a Macro Framework for the Fund. This latter option entails various options: B1 ‐ Department for
International Development (DFID)’s model for Macro Framework; B2‐ Project cluster/nesting model for
aggregation of results at a macro level, and; B3 ‐ an hybrid option. These options reflect a major effort to capture
the numerous stakeholders’ expectations and to translate them into concrete options to help guide the way
forward.
4.1. Option A1 “Enhanced” Status Quo
This section presents some slight alterations to the status quo suggesting how the information collected by the
CU, through the Progress Report, could eventually provide interesting information over time at the Fund level, and
might contribute to addressing the need expressed by some partners who want to be able to report on the Fund’s
impact, without changing the way the monitoring and reporting is currently structured. The current Progress
Report that ISAs are required to complete twice a year could better contribute to aggregated reporting at the
Fund‐level over time, with some very minor adjustments.
4.1.1. Portfolio Reporting ‐ Overview
Project portfolio reporting could be a powerful tool to ensure that the Fund resources work together effectively
and deliver results in the TCs. Portfolio reporting would let SC members look holistically across the projects
supported to understand the portfolio’s strategic alignment and its potential to deliver expected results. Using the
current tools for reporting set out by the Fund, the aggregation of findings across the portfolio of projects could be
done by country, focal area/pillars, and overall project performance. Such a simple portfolio monitoring system
would allow the aggregation of some information sets, instead of looking at each project as an isolated event. This
would permit development of an overview of how projects with different objectives are contributing as a set to the
overall contribution of the MENA Transition Fund.
Focal Areas/Pillars Portfolio Reporting
Aggregation of information in this way would allow, in addition to country portfolio reporting (as shown in Figure
5), an assessment of whether the overall portfolio of projects approved consistently covers all areas of results (i.e.
pillars) set out by the Fund (exemplified in Figure 6). In addition, given the fact that projects submitted for
financing are obliged to respond to TCs’ priorities, it could also provide information about which “pillars” are
contributing most towards meeting these. This could eventually help balance the overall portfolio to most
effectively address TC priorities over time.
T
“Creating a Macro Results Framework for the Middle East and North Africa Transition Fund” Final Report Appendix
22
Figure 6
Figure 6. MENA Transition Fund Approved Portfolio by Pillars, YR X (by amount ~ Total amount) – Fictional
Data
This approach to aggregate data would not provide a clear portrait of the macro results achieved by projects
funded at an overall regional level. Nevertheless, it could prove useful for managing investments and ensuring that
thematic areas are covered in proportion to the contribution they make, thereby enabling donors and ISAs to
assess what types of projects TC value more on the basis of results achieved; thereby ensuring better targeting of
resources for enhancing impact over time in the region.
To facilitate this process moving forward, this information about pillars could be transposed under section A of the
Progress Report template as demonstrated by yellow highlights below52:
Table 7. Snapshot of Revisions to Improve the Progress Reporting Template
A. Basic Project Information
ActivityName:
Country(ies)Name: Name of Implementation SupportAgency(ies):
Total Amount Approved by theTransitionFund(US$):
Additional Funds Leveraged, if any(US$):
Steering Committee ApprovalDate:
ProjectImplementationStartDate: ProjectClosingDate:
NameofISAProjectLeader:EmailofISAProjectLeader:
Pillar(s)towhichActivityResponds(aspertheproposalapprovedbytheSteeringCommittee)
Investing in SustainableGrowth.
Enhancing EconomicGovernance
Inclusive Development andJobCreation
Competitiveness andIntegration
Primary:
Secondary:
Primary:
Secondary:
Primary:
Secondary:
Primary:
Secondary:
51 MENA Transition Fund. (2013). MENA Transition Fund Annual Report December 31, 2013.
52 Since the report was submitted, there has already been some movement along the lines suggested.
Figure 5. Percent of Funding Allocated by Country
Figure 5. Percent of Funding Allocated by Country51
“Creating a Macro Results Framework for the Middle East and North Africa Transition Fund” Final Report Appendix
23
Potential Additional Revisions to Reinforce the Process of Focal Area/Pillars Portfolio Reporting
Developing clearer results statements under each pillar could further improve the effectiveness of the results
reporting process and assist the work in term of aggregating data. This could simply be done by creating sub‐
results category. Based on the broad definitions given for each pillar, these sub‐categories of results could look like
the following:
Table 8. Examples of Sub‐Categories for MENA Transition Fund Objectives53
Pillars
Sub‐category
Pillar 1: Investing in Sustainable Growth
Pillar 2: Enhancing Economic Governance
Pillar 3: Inclusive Development and Job
Creation
Pillar 4: Competitiveness and
Integration
Innovation and Technology
Good Governance (e.g. transparency, corruption, accountability, integrity, etc.)
Youth employability Trade and Negotiation
Business, private sector development, Investment, and Competition
Finance (e.g. asset recovery, financial management, audit, banking system, etc.)
Female labor participation
Infrastructure
Energy Justice (e.g. regulatory reforms, consumer protection, etc.)
Integration of people with disability
Transport
Urban & rural planning Partnership with Civil Society (e.g. open government
Aging people (e.g. pension reform)
Private Sector Development
Then based on the sub‐category defined, it would be possible to report in the Fund Annual Report, for example, a
graph as depicted in the figure below:
53 The below is provided as an example and would have to be refined based on the current portfolio of projects approved and
the priorities defined by the partners involved in the Fund.
“Creating a Macro Results Framework for the Middle East and North Africa Transition Fund” Final Report Appendix
24
Figure 7. Snapshot of Transition Objectives Pursued by Fund Projects, 2013‐2015 – Fictional Data
This would enhance the reporting of relevant data to partners. For example, by presenting to ISAs and TCs a set of
“macro results” statements to which their project could align, these types of graphs could render an even more
precise view about how the Fund is supporting TCs in their reform processes. Another main benefit from this
refinement would be to potentially better “adjust the dial” for incoming proposals by slightly modifying and
improving the Call for Proposals to ensure the kinds of proposals required to achieve the full range of results will be
targeted. It could also help for improving understanding of future resource needs and improve alignment and
leveraging of resources. Moreover, it could enhance coordination between partners (e.g. donors, TC, ISAs, and
other actors in the region) and catalyze synergy by more clearly showing what type of projects are currently
underway/supported by the Fund. Finally, it could potentially incentivize more partners to join if they feel they
have a vested interest in the kind of support the Fund provides. Even though all of the above represent interesting
benefits for reporting in an aggregated way at the Fund level, this type of reporting would not catalyze enhanced
reporting on the contribution of the Fund in terms of development outcomes as requested by key donors.
Overall Portfolio Performance
Another aspect which could be “aggregated” easily at the Fund‐level is the overall project performance. During
project implementation, each project is required to monitor the expected achievements of objectives as follows:
Satisfactory (S): The project is likely to achieve almost all or exceed its major objectives
efficiently without any significant shortcomings.
Moderately Satisfactory (MS): The project is likely to achieve the majority of its major
objectives efficiently with moderate shortcomings.
Innova
tion and Tec
hnology
Business, P
SD, Inve
stmen
t, and Competition
Energy
Urban & rural p
lanning
Good gove
rnan
ce
Finan
ce
Justice
Partnersh
ip with Civil Society
Youth employa
bility
Fem
ale labour participation
Peo
ple with disab
ility
Aging peo
ple
Trade an
d Neg
otiation
Infrastructure
Transp
ort
PSD
60%
30%
5% 5%
25%
40%
20%15%
70%
30%
0% 0%
40%
10%
25% 25%25%
40%
20%15%
40%
10%
25% 25%
60%
30%
5% 5%
60%
30%
5% 5%
40%
10%
25% 25%
60%
30%
5% 5%
25%
40%
20%15%
25%
40%
20%15%
2013 2014 2015
Pillar 1 Pillar 2 Pillar 3 Pillar 4
“Creating a Macro Results Framework for the Middle East and North Africa Transition Fund” Final Report Appendix
25
Moderately Unsatisfactory (MU): The project is not likely to achieve at least half of its major
objectives efficiently with moderate shortcomings.
Unsatisfactory (U): The project is not likely to achieve most or any of its major objectives
efficiently.
Systematic analysis of data provided under this category of the Progress Report could enable the Fund to
continuously monitor the quality/performance of the projects under implementation (see the tables below for
examples). Based on the assumption that each project defines a Project Development Objective (PDO), which is
expected to contribute to the transition/reform process in its TC and measures progress towards achieving
relevant results against robust indicators, then it would be possible to report on the extent to which the Fund is
supporting “transformative” projects that will benefit the transition phase in TCs.
Table 9. Example of a Summary of Performance Ratings Table ‐ Fictional Data
Rating FY1 (40 projects) FY2 (50 projects) FY=n (*n project)
Progress towards achievement of objective (PDO)
S 27% 18% X%
MS 65% 74% X%
MU 5% 0% X%
U 3% 8% X%
Implementation progress
S 20% 18% X%
MS 67% 66% X%
MU 1% 16% X%
U 12% 0% X%
Table 10. Example of Performance Ratings by Country for YR X54
Country Number of Projects
Progress Towards Achievement of Objectives %
Implementation Progress %
S MS MU U S MS MU U
Egypt XX % % % % % % % %
Jordan XX % % % % % % % %
Libya XX % % % % % % % %
Morocco XX % % % % % % % %
Tunisia XX % % % % % % % %
Yemen XX % % % % % % % %
54 These tables are presented merely as suggestions and for example purposes only.
“Creating a Macro Results Framework for the Middle East and North Africa Transition Fund” Final Report Appendix
26
Table 11. Example of Performance Ratings by Pillars for YR X
Pillars Number of Projects
Progress Towards Achievement of Objectives %
Implementation Progress %
S MS MU U S MS MU U
Pillar 1 XX % % % % % % % %
Pillar 2 XX % % % % % % % %
Pillar 3 XX % % % % % % % %
Pillar 4 XX % % % % % % % %
Tables above could help to assess which thematic area (i.e. pillars) are generally more/less successful at delivering
key results or potentially facing more/less difficulty during implementation; thus, helping identify opportunities in
the future. Over time, the systematic use of this type of aggregate reporting through the Progress Reports
submitted by ISAs, would allow SC members to assess what approaches or interventions contribute most
effectively to achieving specific development objectives, and would thereby help to identify good practices for
replication.
Revisions to Reinforce the Process of Portfolio Performance Reporting
As discussed above, ratings and rating definitions are given for progress towards the achievement of objectives in
the current Progress Report. In addition to this existing information, it would be possible to add definitions for the
ratings for overall implementation progress (as shown below), which would enable the introduction of a
harmonized system for ISA reporting on this aspect. For example:
Possible Definitions for Implementation Progress Ratings
Satisfactory (S): Implementation of most components is in substantial compliance with the
original/formally revised plan except for only a few that are subject to remedial action.
Moderately Satisfactory (MS): Implementation of some components is in substantial
compliance with the original/formally revised plan with some components requiring remedial
action.
Moderately Unsatisfactory (MU): Implementation of some components is not in substantial
compliance with the original/formally revised plan with most components requiring remedial
action.
Unsatisfactory (U): Implementation of most components is not in substantial compliance with
the original/formally revised plan.
Not applicable (NA): Implementation has not yet started.
The inclusion of this information could also help pave the way for more evaluative analysis at a later stage. For
example, a comparison of the results on progress on implementation with those on progress towards
achievements could reveal areas in which the original project concepts (notably their explicit or implicit theories of
change) have or have not proved correct.
“Creating a Macro Results Framework for the Middle East and North Africa Transition Fund” Final Report Appendix
27
4.1.1. Steps Required in Moving Forward with this
“Enhanced” Status Quo Option
As mentioned, there are some small steps required in order to develop this option and these mostly require only
slight modifications to the current progress reporting system, specifically:
1. Information about pillars to which projects contribute could be included under section A of the Progress
Report template to facilitate focal areas/pillars portfolio reporting55;
2. Ratings and definitions for overall implementation progress could be added to the Progress Report
template. This would harmonize how ISAs report on this aspect and allow aggregation at the portfolio
level.
In addition, developing clearer results statements (i.e. sub‐results categories) under each pillar could further
improve the effectiveness of the results reporting process and assist in term of aggregating data. However, this
could take some time to become effective, as all partners would need to agree on a set of results sub‐categories.
As exemplified in GEF case study, given the fragility of any system based on self‐reported monitoring systems, it
could be useful to also consider how targeted mid‐term evaluations, or even formative evaluations (e.g. country,
project, sector or thematic evaluations56) might be used to refine and calibrate the monitoring‐based system. The
issue of the need to verify self‐reported data through some system of cross‐checking/quality assurance should also
be given some further thoughts.
Finally, the SC members may wish to consider building on the above by setting targets for performance. This
aspect is not developed/suggested further in this section purposefully, as given the context and stage of the Fund,
this may not be the most efficient and effective way forward.
4.2. Option B1 – Revise and Use the DFID‐based Logic Model
This variation for developing a Macro Framework involves using the logic model and theory of change developed
by DFID. The following is the logic model for the MENA Transition Fund interventions as it appears in the business
case document for UK involvement in the Fund.
Figure 8. Intervention Logic ‐ DFID Model57
55 Since the report was submitted, there has already been some movement along the lines suggested.
56 Please refer to the Glossary of terms in Annex 2 for more details on each type of evaluation mentioned.
57 DFID. (n.d.) Business Case and Intervention Summary. Middle East and North Africa (MENA) G8 Deauville Transition Fund.
“Creating a Macro Results Framework for the Middle East and North Africa Transition Fund” Final Report Appendix
28
As outlined in this DFID model, the Fund is primarily aimed at catalyzing institutional reforms that will enable
better policies and in turn drive improved economic performance. In addition, the agreed aim of the Fund is to
improve the development and implementation of government policy in the TCs, focusing on four priority thematic
areas: 1. investing in sustainable growth; 2. inclusive development and job creation; 3. enhancing economic
governance; and 4. competitiveness and integration. These thematic areas will all contribute to the outcome of
building effective policies and institutions that promote inclusive and sustainable growth as demonstrated below.
Figure 9. Theory of Change Diagram ‐ DFID Model58
To measure these different articulations of results, DFID proposes a “top‐down”” approach and has already
defined indicators and sources to track these at the Fund‐level as displayed in Figure 10 below. This model is
categorized as following a “top‐down” approach since the indicators used to track achievement of results do
not take into account the project‐level results frameworks (and indicators) already in use. Rather, the
58 Idem.
“Creating a Macro Results Framework for the Middle East and North Africa Transition Fund” Final Report Appendix
29
indicators for each results level have been defined without considering the work on the ground (i.e. the
project‐level).
Figure 10. Preliminary Logical Framework ‐ DFID Model59
Of note is that many stakeholders are skeptical that at this stage, the long‐term effect of the MENA Transition
Fund can be measured. In practice, the Fund is likely to continue to support many projects in a variety of sectors
within each thematic area; thus, the impact and outcome level of this proposed results framework might be too
general and broad to give an accurate picture of the contribution of the Fund in the region. The gap between
outputs and impacts might be too vast given the contribution of the Fund in the region and the large diversity of
projects it funds. Some have argued that the “macro” impact of the Fund should be limited to helping strengthen
59 Idem.
Indicators Sources
Annual real GDP growth in priority countries
IMF Regional Economic Outlook (projections for 2012
and 2013)
Unemployment (and female unemployment) rate
IMF Regional Update 2011; Government statistics or World Bank Development Indicators
(latest available data)
Percentage of population living on less than $2 per day (2005 prices)
World Development Indicators (latest year available)
Government Effectiveness ScoreWorld Bank Governance Indicators: Government
Effectiveness Score (annual)
Foreign direct investment (BOP, current $) total of all positive flows
World Bank, World Development Indicators
Open Budget Index Open Budget Index
Volume of funds leveraged directly as a result of technical assistance ($m)
WB Coordination Unit Six monthly report
Volume of technical assistance provided ($m) disaggregated by Implementing Support Agency (ISA)
WB Coordination Unit Six monthly report
Average percentage progress against agreed work plan (disaggregated by ISA)
WB Coordination Unit Six monthly report
Client's imprression of technical assistance effectiveness (based on Client survey, average rating on a scale of 0-10, 10 being very favourable)
WB Coordination Unit Six monthly report
80%
World Bank coordination unit
working effectively
Average time from approval to first disbursement (Number of months from approval to first disbursement) disaggregated by ISA
WB coordination unit six monthly report
Number of countries receiving at least 10% of cumulated allocated technical assistance (over achievement will be regarded as anything over 10%)
WB coordination unit six monthly report
20%Percentage of allocated technical assistance being managed by Regional ISAs
WB coordination unit six monthly report
IMPACT
Inclusive and sustainable growth
in transition countries
OUTCOMEEffective policies and institutions
OUTPUTS
Effective technical assistance which
meets country needs
“Creating a Macro Results Framework for the Middle East and North Africa Transition Fund” Final Report Appendix
30
capacity to deliver reforms. In order to minimize the reporting burden, current results reporting may need to focus
on intermediate outcomes such as institutional reform and capacity building. However, others consider that it
remains important to specify the long‐term results of the MENA Transition Fund using a logic model and theory of
change.
4.2.1. Steps Required In Moving Forward with this
Option
To build a Macro Framework following the model above, indicators would have to be developed and vetted by
partners. DFID stated in the business case document that it would like to work with the SC and other donors, to
see if the Macro Framework for the Fund can be strengthened by the addition of sector specific indicators that are
included in Figure 11 below.60
Figure 11. Possible Additional Indicators to be Considered for the MENA Transition Fund Pillars
Moreover, this option would entail, even if in a limited way, a bureaucratic burden on the Fund structure and
reporting that perhaps only a few stakeholders are interested in. This means that the roles and responsibilities for
M&E reporting in the Fund Operations Manual will likely need to be revised to fit the new “system”. Data collection
on the new indicators should represent only a slight additional burden since indicators are collected through
secondary sources, and provided that the approach is implemented with a recognition that higher‐level results
cannot and should not be measured frequently.
Of note also is that the lower level (i.e. output) of this framework uses the current Progress Report as a basis.
Again, given the fragility of any system based, to some extent, on self‐reported monitoring systems, it could be
useful to also consider how targeted mid‐term evaluations or even formative evaluations (e.g. country, project,
sector or thematic evaluations) might be used to refine and calibrate the monitoring‐based system. To ensure
integrity of individual project reporting, a “Quality Assessment Group” could be helpful.
60 Idem.
•No of micro loans (of which to women)
•No of SME loans
•No of procedures to legally start and operate a company
•Time required to complete each procedure (calendar days)
Investing in Sustainable Growth
•Estimated no of jobs created indirectly
•No of people trained (of which women or youth)
•Estimated no of people who have indirectly benefitted from the Transition Fund (calculated for each programme then quality assured by Coordination Unit)
Inclusive Development and Job Creation
•Public Expenditure Financial Assessment ScoresEnhancing Economic
Governance•Revenue Watch Index
•No of documents to export
•Time to export (days)
Competitiveness and Integration
“Creating a Macro Results Framework for the Middle East and North Africa Transition Fund” Final Report Appendix
31
4.3. Option B2 – Developing Project Cluster/Nesting Model
In this model derived from DFATD61, the concept is to have two sets of results frameworks for monitoring and
reporting; one at the project level and the other one at the Fund‐level. The relationship between the two follows a
“nesting” model. Briefly, nesting refers to the process of inserting/creating detailed project level results
frameworks that sit within a larger macro Fund‐level results framework. Each nested project results framework
depicts in greater detail a specific component that is part of a broader Fund‐level objective. For example, in Figure
12 below, projects are categorized based on the pillar to which they contribute, which also represent, to some
extent, the immediate outcomes in the macro Fund‐level results framework.
Figure 12. DFATD Nesting Model ‐ Linkages
These two sets of results frameworks are linked together because ever though the outcomes in the project results
frameworks vary depending on the nature of the project, they always link directly to the pillar to which they
contribute. However, the PDO are more specific and granular in terms of who, what and where than the outcomes
in the macro Fund‐level results framework as exemplified in Figure 13 below.
61 This option is adapted from documents provided by DFATD and collected during the interview phase. To look at the
originally proposed DFATD model, please refer to Annex 5.
“Creating a Macro Results Framework for the Middle East and North Africa Transition Fund” Final Report Appendix
32
Figure 13. Project Cluster/Nesting Model Linkages – Example62
The rationale for this model is that the Fund finances projects of short duration, low budget and limited scope.
This type of project may realistically only achieve their defined output/intermediate results by the end of the Fund‐
financed support (or even later). The collective impact of the Fund projects on a country or area of work, (i.e. pillar)
will take longer to manifest itself than is covered by the project results system. In order to take into account this
lag in measurement, ISAs and TCs still need to identify the PDO and the “pillar” to which their project contributes,
but cannot in the short‐term realistically measure impact results achievement. Thus, individually, each project
results framework would convey its respective results and output information against the macro Fund‐level result
framework. This way is then recognized that it is not the ISAs’ role to monitor and report on the progress towards
or on the achievement of the “pillars” or the intermediate and ultimate outcomes of the macro Fund‐level results
framework, but their project level reporting can be aggregated and categorized accordingly for this type of
reporting at the higher levels. The expected outcomes in the macro Fund‐level result framework would describe
broad changes for the entire Fund which would allow reporting on the “impact” or at least the “intermediate
outcomes” of the Fund. Reporting on the immediate outcomes would reflect the clusters/pillars of the overall Fund
as visible below in Figure 14, which represents the linkages (for pillar 2 and 3 only) between the project results
framework (through their PDO) and the Fund‐level results framework.
62 Project‐level results frameworks are based on information as found in the Project Progress Report (2013).
“Creating a Macro Results Framework for the Middle East and North Africa Transition Fund” Final Report Appendix
33
Figure 14. Linkages between Macro Fund‐level Outcomes (2&3) and PDO63
Fund‐level
Ultimate Outcome (Fund's impact):
To improve the lives of citizens in TCs; and to support the transformation currently underway in TCs
Intermediate Outcomes (Fund's Objectives)
To foster sustainable inclusive economic growth To strengthen governance and public institutions
Immediate Outcomes (Fund's Pillars)
Investing in sustainable growth/Strengthening enabling environment
Inclusive development and job creation Enhancing economic governance Competitiveness and integration
Project‐level
PDO:
Improving employability of graduates and unemployed youth through increasing access to career guidance, job search, and on the job training. (Jordan: Support to Building Active Labor Market Program)
(Morocco: Strengthening Micro‐Entrepreneurship for Disadvantaged Youth in the Informal Sector)
To improve individual employability and SME capabilities for graduates and firms participating in a job‐intensive SME support pilot scheme and to inform related policies and programs. (Yemen: Enterprise Revitalization and Employment Pilot Project)
To enhance the capacity of targeted accountability institutions to provide access to information and to improve enforcement of the anti‐corruption law. (Yemen: Accountability Enhancement Project)
Contribute to the strengthening of government transparency, accountability and public participation (Morocco: New Governance Framework Implementation Support Project)
To mobilize domestic resources to foster sustainable economic growth and income redistribution (Tunisia: Enhancing Domestic Resource Mobilization Through Effective Tax Systems)
This model is regarded as following a “bottom‐up” approach because to measure the results in the macro Fund‐level results framework, the indicators
would come from the individual project results framework.
63 Based on DFATD. (2013). CIDA‐PAD – Annex B: Logic Model for MENA Transition Fund. This figure is adapted from DFATD’s preliminary logic model posited for the Fund. To
look at the originally proposed DFATD model, please refer to the Appendix, Annex 5.
“Creating a Macro Results Framework for the Middle East and North Africa Transition Fund” Final Report Appendix
34
The main advantage of this option is to build from the “bottom‐up”; thus, allowing ISAs to continue to report their own indicators in the project results
framework and not change the way they conduct the monitoring and reporting for their project. This type of monitoring and reporting is possible through the
development of consolidated indicators.64 Consolidated indicators are project level indicators that are clustered together with other related indicators to form
one unit that will allow aggregation of related data at the macro Fund‐level. Table 12 exemplifies how this could be done for pillar 1 and 2 of the Fund.
Table 12. Linkages from Project Results Framework Outcomes to Macro Framework – Pillars 1 &2
Pillars Macro Fund‐level Consolidated Indicator65 Project level indicators66
Investing in Sustainable Growth Number of policies, laws, agreements or regulations promoting sustainable growth that are implemented as result of the MENA Transition Fund assistance
1) A comprehensive energy pricing and fuel switching strategy for Egypt is developed 2) Acceptance and adoption of the SME development strategy in Libya 3) Number of policy reforms identified in view of strengthening the legal framework for PPPs in Tunisia
Inclusive Development and Job Creation Number of people who have benefitted from the MENA Transition Fund
1) Number of stakeholders participating in training 2) Number of direct green jobs created 3) Number of women entrepreneur assisted 4) Number of youth micro‐entrepreneur who receive post‐creation follow‐up support (of which female %)
The process is also shown in Figure 15 below.
64 Of note is that DFATD developed guidance on how to select indicators at the program (i.e. fund level) and how to strategically select other indicators from the project level.
DFATD’s Global Program Branch (GPB) also has a Tip Sheet on consolidated indicators which provides guidance on the key steps required to conduct this process. DFATD’s initial model is discussed further in Annex 5. 65 The below is posited merely as an example and would have to be refined based on the current portfolio of projects approved.
66 MENA Transition Fund. (2013). MENA Transition Fund Project Progress Report.
“Creating a Macro Results Framework for the Middle East and North Africa Transition Fund” Final Report Appendix
35
Figure 15. Diagram Illustrating Different Requirement for Indicators and their Data at Different Levels67
67 Adapted from DFATD. (n.d.) Master Slide EDRMS No. 6065211. To look at the originally proposed DFATD model, please refer to the Appendix, Annex 5.
Indicators at the Ultimate and
Intermediate Outcomes to report
on the Impact of the Fund
Consolidated Indicators at the
Immediate Outcome level to
report on the results achievement
under each pillars
Project level indicators as set out in
the individual project results
framework
“Creating a Macro Results Framework for the Middle East and North Africa Transition Fund” Final Report Appendix
36
4.3.1. Steps Required In Moving Forward with this
Option
This option requires the creation of project clusters/categorizations. An even simpler way would be to decide that
the “cluster” simply reflects the “pillars” defined by the Fund as each project is already required to identify a
“pillar” for which its project align in the Fund proposal form.
This option also requires the creation of appropriate consolidated indicators to measure the results at the macro
Fund‐level. This step would require extensive analysis and comparison of the projects approved to build from
these the Fund‐level consolidated indicators. DFATD’s Global Program Branch (GPB) has a Tip Sheet on
consolidated indicators which provides guidance on the key steps required to conduct this process and they are as
follow:
1. “Review portfolio of projects, by sector (multidisciplinary team);
2. Look for project level indicators that meet the following criteria:
a. Dealing with the same sector;
b. Measuring similar change, at the same level;
c. Same of comparable units of measure;
d. Same/similar units of analysis that could be grouped into a broader unit;
e. Same of similar context which speak to the expected result
3. Using the project level indicators that fulfilled the criteria in Step 2, try and group the units of analysis
from the project indicators into a single broader (generic) category;
4. Add a unit of measure and context (that is related to the expected result from your program) to your unit
of analysis and your indicator is complete.”68
The main burden of this option has to do with the “mapping” of the current project level indicators to the macro
level and do the same process for all newly approved projects to the defined consolidated indicators once these
have been defined. A temporary ad hoc technical group at Fund level could lead this task or this could be
delegated to the next consultancy (to be engaged if the Macro Framework option is selected). If it is decided to
move forward with developing a Macro Framework for the Fund, then given there are consultants coming on
stream to guide this process, it could be their task to kick‐start this process and define/posit categories and
possible consolidated indicators based on the current universe of project level indicators. While doing so, they
could map the current project level indicators to these newly created consolidated indicators. They could also
develop simple guidance material and tools that would guide this process for future project design and ongoing
reporting, which would ease the reporting burden on the consolidated indicators when the time comes to produce
the Annual Report.
Potential Additional Steps to Reinforce the Process
Eventually, it could be useful to provide in the proposal form a “menu” of results statements reflecting the
categories developed and even a “menu” of indicators for the implementers (e.g. ISAs and TCs) that they could
select from. Should this avenue be pursued, it could be made clear that these “menu items” would only be posited
as suggestions and not represent a straight‐jacket; thus keeping with the principles of flexibility and appropriation
set out by the Fund at its inception. When the same outcome is shared by more than one project, harmonizing or
standardizing the indicator within and across pillars would help to compare and aggregate results for different
68 DFATD. Coherence Division‐RBM Technical Unit. (2014). GPB Tip Sheet. DFATD Consolidated Indicators.
“Creating a Macro Results Framework for the Middle East and North Africa Transition Fund” Final Report Appendix
37
sectors, countries, and the Fund as a whole. As noted above, some guidance material to assist ISAs and TCs in this
process into the future could be developed. This could be developed by the consultants that might be engaged to
assist in the process of developing the Macro Framework itself. This would have the added benefit of addressing
the challenge of standardization of terminology use with regards to ISA reporting.
4.4. Option B3 – Reporting Against a MENA Transition Fund Theory of Change – “Hybrid” Option
This variation combines the DFID and project cluster/nesting approaches together. Both approaches focus on
results, as well as themes in the realm of governance, capacity building or institutional reform. Thus, it is possible
to construct a combined “top down/bottom up” approach, focusing on the intermediate outcomes related to
capacity building, institutional reform and/or governance.
Figure 16. Top‐down and Bottom‐up Approach
However, instead of categorizing projects as per the project cluster/nesting model, the aim is to categorize results,
as per the GEF methodology. In addition, instead of developing a categorization of project results according to an
analysis of their own activities and declared expected results, the categorization could be based on a Fund‐level
logic model and theory of change, building on that developed by DFID. Based on the previous, individual work‐
area results framework could be derived from a single logic model and theory of change for the Fund. This could
be done by combining the “top down/bottom up” methodology applied by DFID and the project cluster/nesting
model respectively.
In going this route, it needs to be well understood that the macro objectives results respond to different
information needs and imperatives than the results at the ISA‐project level. In order to reflect fully the
decentralized and diverse reality, the possibility of a single, fully integrated system in which country and project
goals cascade neatly from pre‐defined broader objectives should likely be discarded. To reconcile potentially
competing needs, an “empirical” approach could be adopted. The macro level elements of the Fund needs to be
developed based on extensive consultation and maybe even pilot testing, while the results (outcomes and
outputs) are defined by the TC and ISAs and reflect the diversity of country needs.
Thus, it is possible to construct a combined “top down/bottom up” approach, focusing on the intermediate
outcomes related to capacity building, institutional reform and/or governance.
“Creating a Macro Results Framework for the Middle East and North Africa Transition Fund” Final Report Appendix
38
Figure 17. Revised Logic Model for the MENA Transition Fund
Given the revised (albeit provisional) model, the proposed logic model and theory of change portrayed in Figure 17
can be summarized as follows.
Technical assistance from the Fund increases beneficiary countries’ institutions (specifically line
ministries) capacity for administrative reform such as: M&E, Knowledge Management,
Information Management, Public Financial Management, Policy and Planning and Grants and
Contributions Management.
More efficient line ministries affect the ability of national government to implement reforms in
the key areas of a) increased service delivery in social and economic sectors and b) governance
(anti‐corruption justice, etc...)
Such key institutional reforms lead to full participation of national country citizens in the
economic and social life of the country (i.e. economic and social inclusion), which in turn yields
sustainable (i.e. broad‐based) economic growth.
The intermediate results of two Fund projects were categorized using concepts taken from the revised, provisional
logic model and theory of change described above. The results are depicted in Table 13 below.
Table 13. Proposed Categorization of MENA Transition Fund Project Results
MENA Trust Fund Project
MENA Transition Fund Pillar Intermediate Result
Indicator
Intermediate Outcome (DFID Logic Model, revised)
Enterprise Revitalization and Employment Pilot Project (Yemen)
Primary:
Investing in Sustainable Growth
Successfully facilitate internships and business services to private enterprises
Number of firms requesting business services
Institutional Reform (Service Delivery)
Secondary:
Inclusive Development and Job Creation
Competitiveness and Integration
Number of beneficiaries completing internships
Regional Affordable Housing Project (Morocco & Tunisia)
Primary: Investing in
Sustainable Growth
Evaluation of Government Housing Policy and Programs
# of reports completed
Institutional Reform (Policy and Planning)
Secondary: Inclusive Development
and Job Creation Enhancing Economic
Governance Competitiveness and
Integration
“Creating a Macro Results Framework for the Middle East and North Africa Transition Fund” Final Report Appendix
39
As can be seen, the results of two relatively different projects, one having to do with service to firms and
employment training and the other targeting affordable housing, can both be classified using the concepts
contained in the revised DFID logic model and theory of change. This points to the possibility of developing a
Macro Framework for the Fund.
In order to avoid the risk of banality, a “layered” approach to developing a Macro Framework for Fund could be
adopted. This means that, instead of developing a single overarching results framework, a series of results
framework per pillar/work area is developed. This should provide a certain level of specificity that would address
the risk of an overly‐generalized Macro Framework. A “layered” results framework approach means flexibility of
reporting horizontally and vertically (i.e. by country and/or pillars/area of work). Focusing on results related to
governance and/or institutional reform offers another dimension of reporting, one that cross‐cuts both area of
work and country as displayed in below.
Figure 18. Layered Approach
Aggregation Strategies
In order to discuss possible aggregation strategies for a Fund Macro Framework, it is useful to make a distinction
between two types of results aggregation.
1. Count: This is defined as the categorization of results according to concepts such as administrative or
institutional reform. This aggregation method is described above in the GEF case study. Using this
strategy, it is possible to make statements like the following, “25% of MENA Transition Fund projects
used policy redesign as a tactic for institutional reform”; “30% of MENA Transition Fund Projects
enhanced M&E Systems as part of administrative reform”.
2. Sum: This involves the mathematical addition of results metrics. See an excerpt below from DFATD’s Tip
Sheet regarding the consolidation of indicators which exemplifies this methodology.69
69 DFATD. Coherence Division‐RBM Technical Unit. (2014). GPB Tip Sheet. DFATD Consolidated Indicators.
“Creating a Macro Results Framework for the Middle East and North Africa Transition Fund” Final Report Appendix
40
Referring again to Table 13 above, it is possible to employ the “Count” and the “Sum” aggregation strategy to the
hypothetical Fund Macro Framework defined in this section.
Count. Referring to the column “Intermediate Outcome” one would count the number of results
categories in this column.
Sum. Referring to the column “Indicator” and using some change in wording, it is possible to add similar
results (beneficiary counts, for example) from different projects together.
4.4.1. Steps Required In Moving Forward with this
Option
As this variation is a combination of both the DFID and project cluster/nesting approach, steps in moving forward
with this option are similar to those described above. To recap, this option entails:
1. Refine/elaborate logic model and theory of change;
2. Categorization of current Fund project results, using the new logic model and theory of change to provide
“bottom‐up” validation;
3. Define/build consensus on indicators for each level of results in the logic model and theory of change;
4. Develop pillar/area of work results frameworks;
5. Revise the M&E reporting structure/arrangements to fit the new “system”;
6. Consider how audits and evaluations might be used to refine and calibrate the monitoring‐based system.
As alluded to above, the DFID logic model and theory of change, while a good start, requires further elaboration.
Some of the relationships between results need to be further specified. For example, examining DFID logic model,
there is an implied but under‐specified relationship between social/economic inclusion and sustainable economic
growth. The demonstration model attempts to give more specificity to the relationship between social/economic
inclusion and sustainable growth. Also, the demonstration logic model makes a distinction between
administrative and institutional reform, positing that administrative reform is a precondition of key institutional
EXAMPLE– How an Existing Consolidated Indicator Might Be “Unpacked” or Reflected at Project Level
Consolidated Indicator: #/total of birth attendants trained in emergency obstetrical care (F/M).
A program may have three different projects, each with a different indicator related to the training of midwives:
Project A: #/total of midwives trained in the management of common complications during
delivery in country X (f/m) Project B: #/total of midwives that received refresher training on how to recognize signs of more
severe complications in country Y (f/m) Project C: #/total of nurse practitioners trained in emergency obstetrics (f/m)
In this case, for the purposes of rolling up data against a common agency standard, DFATD created
the consolidated program‐level indicator: #/total of birth attendants trained in emergency obstetrical care (F/M).
“Creating a Macro Results Framework for the Middle East and North Africa Transition Fund” Final Report Appendix
41
reform such as governance improvement or service to the public. These hypothesized relationships between
expected results must be first rooted in the current literature and the practice of donors and ISAs70.
Once this is done, the categorization of current Fund project results, using the new logic model and theory of
change, is carried out, providing “bottom‐up” validation. In addition, as mentioned earlier, consensus results
indicators will need to be developed for the logic model. These will form the basis of results framework to be
developed per area of work, i.e. pillar.
In order to reduce reporting burden, several implementation strategies can be employed. To posit two examples:
1. Choice: Projects are given a choice of common, fund‐level indicators from which to choose and against
which they must report (i.e. a menu);
2. Prioritization: Careful selection of which indicators projects have to report on frequently and which,
because of their slow‐changing nature (i.e. intermediate outcomes or impacts), will only need to be
reported on at the end of the project. In the case of slow‐changing indicators, only require subjective
evaluations (Satisfactory/Unsatisfactory) of progress; Reserve measurement of slow‐changing, higher
level results (such as catalytic effects) at fund‐level for evaluations;
In addition to the above, the “hybrid approach”, to some degree, relies on subjective evaluations of results
progress at project level and therefore project‐level audits should likely be conducted to verify the subjective
evaluations. Furthermore, as was noted, slower‐changing fund‐level results at outcome and impact level should
likely be measured less frequently, as there is a lag in effect. A related issue is the use of evaluation at project level.
A RoTI approach71 as per the GEF, focusing on key issues, can provide some indications of impacts at fund‐level.
4.5. Others
Possible interesting resources to consult, if decided to move forward in developing a Macro Framework for the
Fund, are posited below:
The WB’s Approach to Public Sector Management (2011‐2020), which is interested in how to
support delivery of reforms;
High Level Forum on Aid Effectiveness in Busan (2011) and the Cairo Consensus on Capacity
Building (2011), which set internationally agreed frameworks for strengthening capacity
development through technical assistance which emphasizes the importance of country
ownership, transparency and knowledge transfer.
Other case studies: Central Asia Regional Economic Cooperation (CAREC) Program72, EU
Platform on Blending.
70 It is recognized that this represents a significant step and one which may not find extensive support.
71 Please refer to GEF EO. (2009). The RoTI Handbook: Towards enhancing the impacts of environmental projects, Methodological
Paper #2. 72 Please refer to http://www.carecprogram.org/index.php?page=carec‐results‐framework to see CAREC’s three levels Results
Framework.
“Creating a Macro Results Framework for the Middle East and North Africa Transition Fund” Final Report Appendix
42
ANNEX 1. METHODOLOGY
This independent Feasibility Assessment was carried out from Mid‐February to April 2014. The following section
describes the assessment questions, approach and methods, and examines some constraints and limitations.
The approach and methodology presented in this Annex are also of utmost importance to clarify and determine
the scope of the mandate. These have been presented and discussed during early stages with the CU’s Program
Manager.
Approach & Questions
In relation to the objectives mentioned in the main report, the following aspects were covered: the current project‐
specific results framework; the motivation and challenges for establishing a Macro Framework for the Fund; a
review of best practices in funds of similar nature, and; the main options and key next steps in establishing a Macro
Framework for the Fund.
A general introduction to each of these and some sample core questions are presented below:
The current project‐specific results framework: An assessment and mapping of the status quo,
in order to identify the strengths and weaknesses of the processes associated with the
monitoring and reporting function of the Fund;
o What are the results that are currently targeted and measured through the Fund?
o How and when is data collected by the CU? By individual ISAs?
o What is the capacity of implementing partners to conduct M&E of their individual
projects, data gathering information technology (IT) systems in place (or not), etc.?
Motivation and challenges: An assessment of the extent to which a Macro Framework could
meet the objectives set forth by the stakeholders working with the Fund and would be consistent
with requirements, needs, priorities, and partners’ and donors’ policies. This refers more
specifically to the opportunities, challenges, and risks.
o What do stakeholders involved in the Fund see as possible benefits and challenges for
establishing a Macro Framework? What do stakeholders involved in the Fund suggest
about the possible motivations for establishing a Macro Framework? What would be the
value (or lack thereof) for it?
o What are (or could be) the strengths of having a Macro Framework when managing a
fund?
o What are (or could be) the weaknesses of having a Macro Framework when managing a
fund?
Review of best practices in funds of similar nature: A brief study of the options, challenges and
risks based on the experience of funds of similar nature.
o How do these funds operate in terms of monitoring and reporting? What are the M&E
systems in use?
o Where evidence exists, how effective is their framework? What are the challenges?
o How could each option be used in the context of the MENA Transition Fund (i.e. what
can we learn)?
“Creating a Macro Results Framework for the Middle East and North Africa Transition Fund” Final Report Appendix
43
Based on answers to the previous questions and the assessment made, the recommendations – which will be more
fully exposed in the final report ‐ will cover the following aspects:
What recommendations can be made on the way forward? Where are the opportunities and
where is the resistance? What are the pitfalls and what are the key success factors in the way
forward?
What options is there for the MENA Transition Fund?
What are the main steps required for creating a Macro Framework for the MENA Transition
Fund?
Matrix
The main element guiding the Feasibility Assessment approach and methodology is the matrix, which is presented
below at the end of this Annex. The matrix links overall questions to more detailed sub‐questions distilled from the
TORs, which would be measured through the different data gathering and analysis methods and from varying
data sources, as presented in the following sub‐sections.
Methods
The two main data gathering and analysis method, or lines of inquiry for the Feasibility Assessment were
document review and interviews with key stakeholders. In very rare cases, surveys were sent to stakeholders to
replace the interview and to accommodate informants. Everyone involve with the Fund was contacted: TCs,
donors, and ISAs, the CU. Key focal points in each country and at each ISA were provided by the CU and the
consultant reached to all of them. However, not all of them responded.
Document Review
The document review was the first step in the mandate and allowed the consultant to prepare for interviews.
Further document were also gathered from the interviews. A complete list of document reviewed by consultant
can be found in Annex 3.
Interviews
Significant time has been spent in conducting interviews with members of the SC. These included representatives
of TCs, donors, and ISAs. Stakeholders were contacted mostly through telephone/Skype interviews. A complete
list of persons interviewed during the Feasibility Assessment can be found in Annex 4.
Key Deliverables and Calendar of Activities
Deliverables/Key Activities Date of Submission
Kick off Conference Call February 12, 2014
Interview with Key Informants February 24 to March 28, 2014
First Draft Report The week of April 7, 2014
WB/CU Comments on 1st Draft Report By April 14, 2014
Second Draft Report By April 25, 2014
WB/CU Comments on 2nd Draft Report By April 28, 2014
Final Report By May 1, 2014
“Creating a Macro Results Framework for the Middle East and North Africa Transition Fund” Final Report Appendix
44
EVALUATION QUESTIONS AND SUB-QUESTIONS INDICATORS DATA COLLECTION METHODS
SOURCES OF INFORMATION
1. Management/Operations Overview What are the results that are currently targeted and measured through the MENA Transition Fund?
Document Review
MENA Transition Fund Operation Manual Progress Reports submitted ISAs policies and procedures (Financial Procedures Agreements (FPAs) signed with recipient entities)
Interviews Recipient entities ISAs Coordination Unit Trustee Steering Committee
2. Efficiency: How effective is the current project-oriented results framework of the Fund?
Costs/Resource Allocation Demonstrating Results/Impact Simplicity/Users Satisfaction
2.1. Efficiency of Resource Allocation and Utilization How and when (frequency) is data collected? Who is involved (i.e. how many persons/for how many days)?
Interviews Recipient entities ISAs Coordination Unit Trustee
Do you consider that the project have access to appropriate financial resources to report and conduct M&E appropriately/according to the agreements signed/set forth in the MENA Fund Operation Manual? What is the ratio of resources invested in M&E to date versus those planned?
Interviews Recipient entities ISAs Coordination Unit Trustee Steering Committee
2.2. Efficiency in Demonstrating Results How are indicators defined, and does this vary between ISA? Do definitions vary by country?
Interviews ISAs
To what extent do the current annual/progress reports provide a picture of results/progress at the macro level? What do these annual/progress reports say about the MENA Transition Fund impact? What could the annual/progress reports have said about the MENA Transition Fund impact? How can these documents be revised /enhanced to better allow for capturing effects at a more macro level?
Document Review MENA Transition Fund Annual Report Interviews Coordination Unit
Steering Committee
2.3. Simplicity/User-friendliness Do the modalities and M&E procedures constitute challenges? Interviews Recipient entities
ISAs Are the definitions of roles in terms of M&E and management clear? How do individual ISAs and transition countries use the results data in their respective work programs and strategies? How do the results influence policy decisions/actions?
Interviews Recipient entities ISAs
Would you have any recommendation to improve the efficiency of the M&E process (both in terms of costs and utilisation of findings generated)?
3. Relevance What do stakeholders involved in the MENA transition Fund suggest about the possible motivations for establishing a macro/results framework? What would be
Interviews Recipient entities ISAs
“Creating a Macro Results Framework for the Middle East and North Africa Transition Fund” Final Report Appendix
45
the value (or lack thereof) for it? Coordination Unit Trustee Steering Committee
What do stakeholders involved in the MENA transition Fund see as possible challenges for establishing a macro results framework?
Interviews Recipient entities ISAs Coordination Unit Trustee Steering Committee
Would you make any recommendation concerning the future or opportunity to pursue or not the establishment of such macro-aggregate results framework for the MENA Transition Fund?
Interviews Recipient entities ISAs Coordination Unit Trustee Steering Committee
What are the strengths (general, based on the experience of other similar Funds) of having a macro results framework when managing a Fund?
Document Review Studies and Assessments of M&E frameworks and the performance and governance of international funds
Interviews/Surveys Funds targeted for Case Study What are the weaknesses (general, based on the experience of other similar Funds) of having a macro results framework when managing a Fund? Challenges, risks, costs
Document Review Studies and Assessments of M&E frameworks and the performance and governance of international funds
Interviews/Surveys Funds targeted for Case Study Is there value for money of such a macro-aggregated results framework based on the experience of other similar Funds?
Value added v.s. Costs Extent to which Funds of similar nature uses information generated for decision-making
Document Review Studies and Assessments of M&E frameworks and the performance and governance of international funds
Interviews/Surveys Funds targeted for Case Study 4. Recommendations Opportunities
Threats Resources implications Possible options Next Steps
4.1. What are the prospects for success in establishing a macro results framework for the MENA Transition Fund?
Existing capacity & resistance Costs
To what extent is there sufficient ownership of the process to sustain the implementation of the macro results framework?
Information collected through interviews about “relevance”
Generally speaking, has the Fund planned enough financial resources to cover recurring costs of this macro results framework?
Interview Coordination Unit Steering Committee Trustee
4. What recommendations can be made on the way forward regarding structuring and implementing a macro-aggregate results framework?
IT system requirements Main actions required/Next Steps
What capability exists to set up cost-efficient IT-systems? What do other Funds of similar nature use as IT-system? With what costs?
Interview Funds targeted for Case Study WB IT staff
Document Review Similar Funds website What are the main steps required for creating a macro/aggregate results framework for the MENA Transition Fund?
Document Review Funds targeted for Case Study Interviews RBM & M&E experts
What are the pitfalls in the development of a macro/aggregate results framework for the MENA Transition Fund? What are the key success factors?
“Creating a Macro Results Framework for the Middle East and North Africa Transition Fund” Final Report Appendix
46
ANNEX 2. GLOSSARY73
Term Definition
Activity Actions taken or work performed through which inputs, such as funds, technical assistance and other types of resources are mobilized to produce specific products and services
Activity indicator Measure what happens during implementation. Often they are expressed as a set of completion or milestone events taken from an activity plan, and may measure the time and/or cost required to complete them (examples: date by which building site is completed, cost of developing textbooks)
Beneficiary The individuals, groups, or organizations, whether targeted or not, that benefit, directly or indirectly, from the development intervention.
Country evaluation Evaluation of one or more donor’s or agency’s portfolio of development interventions, and the assistance strategy behind them, in a partner country.
Evaluation The systematic and objective assessment of an on‐going or completed project, program or policy, its design, implementation and results. The aim is to determine the relevance and fulfillment of objectives, development efficiency, effectiveness, impact and sustainability. An evaluation should provide information that is credible and useful, enabling the incorporation of lessons learned into the decision–making process of both recipients and donors. Evaluation also refers to the process of determining the worth or significance of an activity, policy or program. An assessment, as systematic and objective as possible, of a planned, on‐going, or completed development intervention.
Impact Positive and negative, primary and secondary long‐term effects produced by a development intervention, directly or indirectly, intended or unintended.
Impact indicator Measure the longer‐term and more widespread development changes in the society, economy or environment to which the project contributes. Often these are captured via national sector or subsector statistics (examples: reductions in percent of the population living below the poverty line, declines in infant mortality rates, reductions in urban pollution emission rates)
Indicator A variable that allows the verification of changes in the development intervention or shows results relative to what was planned
Input The financial, human, and material resources used for the development intervention
Input indicator Measure quantities of physical, human or financial resources provided to the project, often expressed in dollar amounts or amounts of employee time (examples: number of machines procured, number of staff‐months of technical assistance provided, levels of financial contributions from the government or co financiers)
Lessons learned Generalizations based on evaluation experiences with projects, programs, or policies that abstract from the specific circumstances to broader situations. Frequently, lessons highlight strengths or weaknesses in preparation, design, and implementation that affect performance, outcome, and impact.
Logical Framework Management tool used to improve the design of interventions, most often at the project level. It involves identifying strategic elements (inputs, products and services, expected results, impact) and their causal relationships, indicators, and the assumptions or risks that may influence success and failure. It thus facilitates planning, execution and evaluation of a development intervention.
Milestone A significant point or event in the project
73 OECD. (2002). Glossary of Key Terms in Evaluation and Results Based Management.
“Creating a Macro Results Framework for the Middle East and North Africa Transition Fund” Final Report Appendix
47
Monitoring A continuing function that uses systematic collection of data on specified indicators to provide management and the main stakeholders of an ongoing development intervention with indications of the extent of progress and achievement of objectives and progress in the use of allocated funds.
Outcome The likely or achieved short‐term and medium‐term effects of an intervention’s products and services. The post‐intervention state of the target group or the social conditions that an intervention is expected to have changed
Outcome indicator Measure relatively direct and short‐to‐medium term effects of project products and services on intermediary organizations or on the project beneficiaries (clients, customers) ‐‐ such as the initial changes in their skills, attitudes, practices or behaviors. Often measures of the clients’ preferences and satisfaction with product/service quality are also considered as expected results (example: percent of clients satisfied with quality of health clinic services)
Output The products, capital goods and services which result from a development intervention; may also include changes resulting from the intervention which are relevant to the achievement of expected results.
Output indicator Track the most immediate results of the project ‐‐ that is, the physical quantities of goods produced or services delivered (examples: kilometers of highway completed, number of classrooms built)
Program A development program is a time bound intervention involving multiple activities that may cut across sectors, themes and/or geographic areas
Project evaluation Evaluation of an individual development intervention designed to achieve specific objectives within specified resources and implementation schedules, often within the framework of a broader program.
Results The output, outcome or impact (intended or unintended, positive and/or negative) of a development intervention
Results chain The causal sequence for a development intervention that stipulates the necessary sequence to achieve desired objectives beginning with inputs, moving through activities and products and services, and culminating in expected results, impacts, and feedback.
Sector evaluation Evaluation of a cluster of development interventions in a sector within one country or across countries, all of which contribute to the achievement of a specific development goal.
Stakeholder Agencies, organizations, groups or individuals who have a direct or indirect interest in the development intervention or its evaluation.
Thematic evaluation Evaluation of a selection of development interventions, all of which address a specific development priority that cuts across countries, regions, and sectors.
“Creating a Macro Results Framework for the Middle East and North Africa Transition Fund” Final Report Appendix
48
ANNEX 3. LIST OF DOCUMENTS REVIEWED
General Documentation about the MENA Transition Fund and the DP
(2011). Deauville Partnership. IFI Statement.
(2013). MENA Transition Fund Fourth Steering Committee Meeting December 4 ‐ 5, 2013 (Brussels, Belgium). Meeting
Minutes.
DFID. (n.d.) Business Case and Intervention Summary. Middle East and North Africa (MENA) G8 Deauville Transition
Fund.
G8 Research Group. (2013). 2012 G8 Camp David Final Compliance Report.
G8 UK. (2013). The Deauville Partnership with Arab Countries in Transition open economies, inclusive
growth.[https://www.gov.uk/government/uploads/system/uploads/attachment_data/file/183533/DP_pamphlet_FS
__SOS___CH_signoff.pdf].
G8 UK. (2014). The Deauville Partnership with Arab Countries in Transition. Progress Report 2013.
IMF. (2013). Arab Countries in Transition: Economic Outlook and Key Challenges. Deauville Partnership Ministerial
Meeting April 19, 2013 Washington DC.
MENA Transition Fund. (2013). Terms of Reference. Feasibility Assessment on Creating a Macro Results Framework
for the Deauville Partnership’s Transition Fund.
MENA Transition Fund. (2013). MENA Transition Fund Operations Manual.
MENA Transition Fund. CU. (2013). MENA Transition Fund Project Progress Report.
MENA Transition Fund. CU. (2013). MENA Transition Fund Annual Report December 31. 2013.
[https://www.menatransitionfund.org/sites/mena_trans_fund/files/documents/MENA%20Transition%20Fund%20
Annual%20Report%20December%202013%20FINAL.pdf].
MENA Transition Fund. Trustee. (2013). Financial Report.
MENA‐OECD Investment Programme. (2013). Challenges for Investment, Growth and Job Creation in the MENA
Region: Consequences for the MENA‐OECD Investment Programme. PPT.
Documents Collected through Interviews
DFATD. (n.d.) Master Slide EDRMS No. 6065211.
DFATD. Coherence Division‐RBM Technical Unit. (2014). GPB Tip Sheet. DFATD Consolidated Indicators.
DFATD. (2014). MENA Transition Fund: a proposal for the clustering of projects. Unpublished.
DFATD. (2014). CIDA PAD – Annex B: Logic Model for Mena Transition Fund. Unpublished.
Ministry of Foreign Affairs. DANIDA. TAS Department. (2013). Danish Arab Partnership Programme. MENA
Transition Fund. Desk Appraisal Report.
“Creating a Macro Results Framework for the Middle East and North Africa Transition Fund” Final Report Appendix
49
Documents Related to the ISAs
AfDB.(n.d.). MOPAN Management Response (Draft 23 November).
AfDB. (2008). Multilateral Development Banks’ Common Performance Assessment System 2007 Report.
AfDB. (2011). Monitoring and evaluation frameworks and the performance and governance of international funds.
AfDB. Quality Assurance and Results Department (ORQR). (n.d.). AfDB’s Action Plan for Quality and Results:
Recent Progress Communicating for and about Results. PPT.
AfDB. Quality Assurance and Results Department (ORQR). (2010). Bank Group Results Measurement Framework.
AfDB. Quality Assurance and Results Department (ORQR). (2012). Staff Guidance on Project Completion Reporting
and Rating.
AfDB. North Africa Regional Department (ORNA) Department. (2012). The Bank Group’s Participation in the Middle
East And North Africa Transition Fund.
BESLEY, T., DEWATRIPONT, M. & GURIEV, S. (n.d.). Transition and transition impact. A review of the concept and
implications for the EBRD.
BINNENDIJK, A.(2000). Results Based Management in the Development Co‐Operation Agencies: A Review of
Experience. Background Report.
DFID. (2011). Multilateral Aid Review: Assessment of International Finance Corporation (IFC).
DFID. (2013). Multilateral Aid Review ‐ Update 2013 progress rating: European Bank for Reconstruction and
Development (EBRD).
EBRD. Office of the Chief Economist. (2009). Transition Impact Retrospective: Results of EBRD Operations 2005 to
2009.
EIB. (2013). Measuring development results. The EIB Results Measurement (ReM) framework.
EIB. (2012). Report on results of EIB operations outside the EU.
IFC. (n.d.). DOTS Indicator Framework.
IFC. (2012). IFC ROAD MAP FY13‐15. Creating Innovative Solutions in Challenging Times.
Multilateral Organisation Performance Assessment Network (MOPAN) Secretariat. (2012). Performance
Assessment of Organisational Effectiveness and Reporting on Development Results. African Development Bank
(AfDB). Volume I.
Multilateral Organisation Performance Assessment Network (MOPAN) Secretariat. (2012). Performance
Assessment of Organisational Effectiveness and Reporting on Development Results. World Bank. Volume I.
World Bank. (n.d.). The World Bank Operations Manual.
Documents Reviewed for Case Studies
De Crombrugghe, D., et al. (2009). Peer Review: the Evaluation Function of the Global Environment Facility (GEF),
Final Report.
“Creating a Macro Results Framework for the Middle East and North Africa Transition Fund” Final Report Appendix
50
DFATD. (2014). MENA Transition Fund Results Feasibility Study: Questions and Answers for an Interview with Baastel
Ltd. Unpublished.
EuropeAid. (2011). Neighbourhood Investment Facility (NIF). PPT Presentation.
GAFSP. (2009) Framework Document for a Global Agriculture and Food Security Program (GAFSP).
GASFP. (2011). Fact Sheet: GAFSP Public Sector Window.
GAFSP. (2011). Monitoring and Evaluation Plan.
GAFSP. (2013). GAFSP Annual Report.
GEF. (2005). Project Implementation Review 2004 – Overview Report – World Bank.
[http://www.thegef.org/gef/sites/thegef.org/files/documents/GEF.C.25.ME_.Inf4_.pdf].
GEF. (2010). Focal Area Framework and LDCF/SCCF Framework. Excerpts from GEF‐5 Programming Document at the
Sixth Meeting for the Fifth Replenishment of the GEF Trust Fund, April 7, 2010.
GEF. (2010). M&E Policy.
GEF. (2011). GEF‐5 Tracking Tool for Biodiversity Focal Area.
GEF. (2012). Annual Performance Report: 2012.
GEF EO. (2007). Final Report on a Proposed Approach to GEF Impact Evaluation. Impact Evaluation Information
Document No. 2.
GEF EO. (2009). The RoTI Handbook: Towards enhancing the impacts of environmental projects, Methodological
Paper #2.
GEF EO. (2012). Annual Thematic Evaluations Report (2012).
IDB. (2012). Project Implementation Report (PIR) FY2012.
Le Groupe‐conseil baastel ltée. (2005). Overview of Monitoring Approaches in the GEF Family.
Le Groupe‐conseil baastel ltée. (2008). Catalogue of Results‐Based Management Practices of Key Organizations.
NIF. (2013). Activity Report: 2008‐2012 Overview of Activities.
NIF. (2013). Mid‐Term Evaluation of the Neighbourhood Investment Facility under the European Neighbourhood and
Partnership Instrument (ENPI) 2007‐2013.
Pérez del Castillo, Carlos (GEF). (2009) OPS4 Technical Document # 5: Governance of the GEF.
Vivideconomics. (2011). Monitoring and evaluation frameworks and the performance and governance of international
funds.
VOLONTE, C. (GEF EO). (2011). The GEF M&E Policy. PPT Presentation.
Sitography
EBRD. (2010). Measuring transition impact. [http://www.ebrd.com/pages/research/analysis/impact.shtml].
EC. (2013). Development and Cooperation – EuropeAid. Promoting investment through the Neighbourhood
Investment Facility (NIF). [http://ec.europa.eu/europeaid/where/neighbourhood/regional‐
cooperation/irc/investment_en.htm].
“Creating a Macro Results Framework for the Middle East and North Africa Transition Fund” Final Report Appendix
51
GASFP. (2011). GAFSP Website. [http://www.gafspfund.org/].
GEF. (2013). GEF Website. Investing in our Planet. [http://www.thegef.org/gef/].
GEF EO. (2013). Independent Evaluation Office Website. [http://www.thegef.org/gef/eo_office].
IFC. (2014). IFC Development Results and Impact.
[http://www.ifc.org/wps/wcm/connect/Topics_Ext_Content/IFC_External_Corporate_Site/IDG_Home/Monitoring_
Tracking_Results/DOTS_Investment/].
MENA Transition Fund. (2013). MENA Transition Fund Website. [https://www.menatransitionfund.org/]
The White House. Office of the Press Secretary. (2012). Fact Sheet: G‐8 Action on the Deauville Partnership with
Arab Countries in Transition. [http://www.whitehouse.gov/the‐press‐office/2012/05/19/fact‐sheet‐g‐8‐action‐
deauville‐partnership‐arab‐countries‐transition].
US Department of State. (n.d.). Deauville Partnership with Arab Countries in Transition.
[http://www.state.gov/e/eb/ecosum/2012g8/deauville/].
WB. (2011). Financial Intermediary Funds. [http://fiftrustee.worldbank.org/index.php?type=page&ft=fifoverview].
Newspaper
EBRD. (2014). EBRD to lead the IFI Coordination Platform of the Deauville Partnership in 2014.
[http://www.ebrd.com/pages/news/press/2014/140106.shtml].
European Commission. (2012). EU launches new Platform for blending funds in external cooperation.
[http://ec.europa.eu/europeaid/news/2012‐12‐12‐platform‐blending‐funds_en.htm].
G8 Information Centre. (2012). Fact Sheet: G‐8 Action on the Deauville Partnership with Arab Countries in Transition.
[http://www.g8.utoronto.ca/summit/2012campdavid/g8‐transition‐factsheet.html]
Others
(2013). Memorandum regarding IFIs Harmonized Development Results Indicators for Private Sector Investment
Operations.
DFID. (2011). Briefing Note Indicators and VFM in Governance Programming.
EuropeAid. DG DevCo. (2013).Results Study.
OECD. (2002). Glossary of Key Terms in Evaluation and Results‐Based Management.
[http://www.oecd.org/dac/evaluation/2754804.pdf].
ZALL KUSEK, J. & R. C. RIST. (2004). Ten Steps to a Results‐Based Monitoring and Evaluation System. A Handbook
for Development Practitioners.
“Creating a Macro Results Framework for the Middle East and North Africa Transition Fund” Final Report Appendix
52
ANNEX 4. LIST OF PERSONS
MET/INTERVIEWED
Donors
Country Organization Name Position
Canada DFATD
Gina Watson Deauville Partnership Coordinator
Yvetta Pass Manager, Results‐Based Management Technical Unit
François‐Philippe Dubé Analyst, Policies and Planning, Europe and Middle East Bureau Development
Andrew Spezowka Director, a.j. Food Security Division, Global Issues and Development Branch
Denmark Ministry of Foreign Affairs of Denmark
Pernille Mortensen Head of Section, Department For Middle East And North Africa
Jørgen Georg Jensen Advisor
France
Ministry of Finance and Industry
Cécile Pot Deputy
Ministry of Economy and Finance
Stephane Tabarie Civil Administrator, Treasury Directorate General
Kuwait Ministry of Finance Ishaq Abdulkarim Director, International Economic Co‐operation Department
Saudi Arabia Saudi Fund for Development
Fahad A. Alnowaiser Director General of Export Credit Dept
Turkey Undersecretariat of Treasury
Zafer Düzenli Senior Treasury Expert
DG for Foreign Economic Relations, Multilateral Development Banks and Economic Co‐operation Department
UAE Ministry of Finance H.E.Younis Alkhoori Undersecretary
UK DFID
Helen Winterton Head, Middle East and North Africa Department
Tim Williams Senior Governance Adviser, Middle East
“Creating a Macro Results Framework for the Middle East and North Africa Transition Fund” Final Report Appendix
53
Country Organization Name Position
and North Africa Department
US US Treasury and International Affairs
Andrew Baukol Deputy Assistant Secretary for Africa and Middle East
Mary Svenstrup Deputy Director, Middle East Office
Elizabeth Shortino Director for Middle East North Africa
ISAs
Organization Name Position
AfDB Jabob Kolster Regional Director for North Africa
EBRD
Adrian Keler Principal Manager, Donor Co‐Financing
Frederic Lead, Transition Team
Akhil Patel RBM Expert
Alan Rousso Managing Director for External Action and Political Affairs
EIB Julien Serre Policy and Strategy Advisor
IFC Luke Haggarty Regional Head, IFC Advisory Services, Middle East and North Africa
IMF Mark Horton Assistant Director, IMF Middle East and Central Asia Department
IsDB Amadou Thierno Diallo
Director, Office of the Vice President Operations
OECD Andreas Schaal Head of Sherpa Office
Nejla Saula Legal Adviser at the OECD Sherpa Office
World Bank (WB)
Preeti S. Ahuja Manager, Development Effectiveness, MENA region
Fareeba Mahmood Senior Operations Officer (MNADE)
Franck Bousquet Coordinator, MENA region
Victoria Gyllerup Operations Officer, Development Effectiveness, MENA region
“Creating a Macro Results Framework for the Middle East and North Africa Transition Fund” Final Report Appendix
54
TCs
Country Organization Name Position
Jordan
Ministry of Planning and International Cooperation
Saleh Al‐Kharasbeh Secretary General
Zeina Toukan Director of International Cooperation
Morocco Ministry of Economy and Finance
Abderrazak Yassir Director, Treasury and External Finance
Tunisia Ministry of Development and International Cooperation
Kalthoum Hamzaoui General Director of Multilateral Cooperation
Yemen Ministry of Planning & International Cooperation
Mohammed AlHawri Deputy Minister
CU
Organization Name Position
World Bank
Hayat Al‐Harazi Program Officer, MENA
Yogita Mumssen Program Manager, MENA
Jonathan Walters Director, Regional Programs & Partnership, MENA
Pauline Zwaans Operations Officer, AES
“Creating a Macro Results Framework for the Middle East and North Africa Transition Fund” Final Report Appendix
55
ANNEX 5. ADDENDUM
The following represents DFATD’s clarification on its initial conception of a Project Cluster/Nesting Model.
Parallel Nesting Model
The DFATD model is a parallel nesting model. The draft results feasibility assessment for an overall results
framework document portrays this model as if it were a linear model where projects feed upward to the macro
results framework (refer to figure 7 in the main report), whereas in the DFATD nesting model, project results
frameworks relate to the macro results framework in parallel as follows:
The ultimate outcome (impact) of the project would be a more granular expression of the ultimate outcome
(impact) of the fund. In other words, the meaning of the ultimate outcome does not change. At the project level
and fund level, we are talking about a change in state for beneficiaries, ie., improving the life of people. Except
that at the project level, the ultimate outcome describes an improvement of the life of the specific project
beneficiaries (improved life of the people in the geographic location of the project), where as a the fund level, the
ultimate outcome describes improvement of beneficiaries of a portfolio of projects in the same sectoral or
thematic area (improved lives of people in five geographic locations).
The intermediate outcome (outcome) of the project would be a more granular expression of the intermediate
outcome (outcome) of the fund. In other words, the meaning of the intermediate outcome (outcome) does not
change. At the project level and fund level, we are talking about changes in performance, behavior, practice or
even access. Except that at the project level, the intermediate outcome describes an improvement in the
performance, practice, and behavior of the intermediaries (e.g., ministry, etc.) directly associated with a specific
project. Whereas at the fund level, the intermediate outcome describes broader changes in performance,
behavior, and practice for intermediaries of a portfolio of project in the same sectoral or thematic area.
The output of the project would be a more granular expression of the output in the fund. In other words, the
meaning of the outputs does not change. At the project level and fund level, we are talking about products and
services, for example technical assistance provided. Except that at the project level you would specify what type
of technical assistance.
Figure 19. Nesting Model Linkages ‐ DFATD's Model
“Creating a Macro Results Framework for the Middle East and North Africa Transition Fund” Final Report Appendix
56
Indicators
The report only focuses on the consolidated indicator idea from our tip sheet on how to select indicators at the
program level. However, our criteria for selecting indicators at the program (fund level) includes guidance on how
to strategically selected other indicators from the project level. This includes identifying common indicators, that
are being used to develop standards, such as a menu of commonly used indicators. Usually if you have several
projects trying to achieve similar changes in different countries, you will find one or two indicators that are more
or less the same. In the case of consolidated indicators, please note that there are no consolidated indicators at
the project level. Consolidated indicators are created at the fund level. They are based on project level indicators.
In addition the pyramid diagram on consolidated indicators (page 36 of the Appendix) that came from our input
misrepresents the concept. In referring to a consolidated indicator at the meso level, a link must be created
between a consolidated indicator at the meso level to several (for example, 2‐3 indicators) at the project level.
However, there are other issues with the pyramid as presented, such as the mixing of levels, treating the projects
as if they were leading to an immediate outcome at the meso level, which is not the case. In other words every
indicator in the pyramid is at the same outcome level, for example, intermediate outcomes at project (micro),
program (meso) and corporate level (macro). Since in the context of the fund, they only have two layers ‐ micro
and macro, it would be a two layered pyramid. The diagram in the report is adapted from the following diagram:
Figure 20. Diagram Illustrating Different Requirement for Indicators & their Data at Different Levels (Simplified)74
The DFATD parallel nesting model requires a full indicator selection menu to be effective. This is explained in a
document that we provided to the consultant as way of explaining the DFATD nesting model in its entirety.
74 DFATD. Master Slide EDRMS No. 6065211. Adapted from diagram found in: http://www.qp.org.nz/monitoring/monitor‐
tools.php.
Founded in 1989, Baastel’s mission is to provide decision‐makers, managers and implementation partners with the knowledge, tools and skills necessary to promote effective and efficient sustainable development. Baastel has earned its reputation as a consultancy
firm fully committed to providing sound advice to development partners on how to strengthen the impacts of their policy and development efforts around the world
North American Office 85 Victoria Street Gatineau QC J8X 2A3 CANADA Tel: + 1 819 595‐1421 Fax: + 1 819 595‐8586 European Office Le Groupe‐conseil baastel sprl Rue de la Croix de fer 23, 1° B‐1000 Brussels BELGIUM Tel: + 32 (0)2 893 0032 Fax: + 32 (0)2 503 3183
www.baastel.com