IW SCS Impact Evaluation IW SCS Impact Evaluation Phases and ProductsPhases and Products

Aaron ZazuetaAaron Zazueta

Reference Group MeetingReference Group Meeting

Bangkok, ThailandBangkok, Thailand

September 27, 2010September 27, 2010

OverviewOverview

2

Contents

Evaluation Phases and Products

Stakeholder Involvement

Management of the Evaluation

Evaluation Timeline

Overview of Evaluation Phases and ProductsOverview of Evaluation Phases and Products• Phase 1: Development of theory of change/ evaluation framework Phase 1: Development of theory of change/ evaluation framework

of GEF support for the main SCS IW project strands.of GEF support for the main SCS IW project strands.– Tool to track which countries are adopting policies and programs that lead Tool to track which countries are adopting policies and programs that lead

to investments and behaviors that improve environmental statusto investments and behaviors that improve environmental status

• Phase 2: Data Gathering. Using theory of change, triangulate Phase 2: Data Gathering. Using theory of change, triangulate data gathering along three lines of inquiry:data gathering along three lines of inquiry:– Portfolio analysis: Portfolio analysis:

• Broad picture of GEF support at various levels; map of interventions and outcomes

– Examination of regional dimensions of GEF support, including:Examination of regional dimensions of GEF support, including:• Synthesis of environmental trends, regional institutional context, and progress in

governance architecture (regional, national, local)

– Country and thematic case studies, including:Country and thematic case studies, including:• Assess appropriateness and effectiveness of approach, and whether approach

engages key stakeholders and is cast at the appropriate scale• Assess progress towards impact and likelihood of intermediate stages.• Identify opportunities for counterfactual analysis

• Phase 3: Analysis and SynthesisPhase 3: Analysis and Synthesis– Assess progress in reducing environmental stressAssess progress in reducing environmental stress– Assess factors and conditions contributing to/hindering reaching Assess factors and conditions contributing to/hindering reaching

intermediate stagesintermediate stages– Assess likelihood of preventing or delaying environmental degradation; Assess likelihood of preventing or delaying environmental degradation;

identify what additional steps are neededidentify what additional steps are needed– Identify lessons learned and knowledge productsIdentify lessons learned and knowledge products

3

Products of the EvaluationProducts of the Evaluation

Development of theory of change Development of theory of change evaluation framework for GEF IW SCS evaluation framework for GEF IW SCS supportsupport

• SCS theory of change should indicate SCS theory of change should indicate how how and why and why GEF support is expected to catalyze GEF support is expected to catalyze actions that address transboundary actions that address transboundary environmental priorities, including ensuring environmental priorities, including ensuring the sustainability of goods and services the sustainability of goods and services provided by the SCS.provided by the SCS.

4

Product: Assessment of the Context of Product: Assessment of the Context of GEF Support in the SCSGEF Support in the SCS

Characterization of SCS political, social, economic, and Characterization of SCS political, social, economic, and environmental services, and of the transboundary environmental services, and of the transboundary factors affecting themfactors affecting them

• Services including fisheries, waste disposal, land Services including fisheries, waste disposal, land reclamation and land-use conversion, tourism and reclamation and land-use conversion, tourism and recreation, natural disaster protection, transportation, and recreation, natural disaster protection, transportation, and oil and mineral exploration.oil and mineral exploration.

• Drivers affecting the ecosystem’s capacity to continue Drivers affecting the ecosystem’s capacity to continue providing servicesproviding services

• Assess how conflicting territorial claims have affected Assess how conflicting territorial claims have affected coastal resource use and managementcoastal resource use and management

Situate GEF support in the SCS relative to other Situate GEF support in the SCS relative to other institutionsinstitutions

• Detailed inventory of (GEF and non-GEF) programs and Detailed inventory of (GEF and non-GEF) programs and projects in the SCSprojects in the SCS

• Assess significance of GEF’s role relative to other Assess significance of GEF’s role relative to other institutionsinstitutions5

Product: Assessment of the Relevance, Product: Assessment of the Relevance, Effectiveness, and Efficiency of GEF Support Effectiveness, and Efficiency of GEF Support

in the SCSin the SCS

• Portfolio Analysis of GEF SCS SupportPortfolio Analysis of GEF SCS Support

• Broad picture of GEF support and progress Broad picture of GEF support and progress towards impact.towards impact.

• Assess whether projects have addressed the key Assess whether projects have addressed the key elements of the GEF IW approach and develop elements of the GEF IW approach and develop hypotheses on the factors and conditions that hypotheses on the factors and conditions that affect impact. affect impact.

• Approach elements include:Approach elements include:– Foundational activitiesFoundational activities– DemonstrationsDemonstrations– InvestmentsInvestments

6

Product: Assessment of the Relevance, Effectiveness, Product: Assessment of the Relevance, Effectiveness, and Efficiency of GEF Support in the SCS (continued)and Efficiency of GEF Support in the SCS (continued)

Analysis of GEF Contributions to SCS Transboundary Analysis of GEF Contributions to SCS Transboundary Environmental GovernanceEnvironmental Governance

• Evolution of regional environmental governanceEvolution of regional environmental governance

• Role of the GEF in supporting transboundary governanceRole of the GEF in supporting transboundary governance

• Effectiveness of current governance mechanisms and Effectiveness of current governance mechanisms and processes in processes in – Reaching agreement Reaching agreement – Engaging stakeholdersEngaging stakeholders– Coordinating action to reduce environmental stress and address Coordinating action to reduce environmental stress and address

critical factors affecting SCS servicescritical factors affecting SCS services

Country and Thematic Case Studies on Progress toward Country and Thematic Case Studies on Progress toward ImpactImpact

• Country case studies to better assess effectiveness and Country case studies to better assess effectiveness and contextual factors affecting progress towards impactcontextual factors affecting progress towards impact

• Assess F-D-I causal sequence and whether support has Assess F-D-I causal sequence and whether support has catalyzed investments, reforms, and behavioral changescatalyzed investments, reforms, and behavioral changes

• Identify steps needed to ensure sustainability of SCS servicesIdentify steps needed to ensure sustainability of SCS services7

Product: Assessment of Environmental Impact Product: Assessment of Environmental Impact and Likelihood of Improving SCS Social, and Likelihood of Improving SCS Social,

Economic, Environmental ServicesEconomic, Environmental Services

Progress in Reducing Transboundary Environmental StressProgress in Reducing Transboundary Environmental Stress

• Review extent to which approaches and technologies Review extent to which approaches and technologies promoted are reducing environmental stress and riskspromoted are reducing environmental stress and risks

• Quantify stress reduction achievements and use remote Quantify stress reduction achievements and use remote sensing and GIS, where availablesensing and GIS, where available

Identification of Intermediate Steps and Actions Needed inIdentification of Intermediate Steps and Actions Needed inorder to Reach Impactorder to Reach Impact

• Given IW TOC, what actions still need to happen to reach Given IW TOC, what actions still need to happen to reach impact?impact?

• Are there any actors who need to be (better) involved?Are there any actors who need to be (better) involved?

Assessment of Likelihood of Delaying or Preventing Assessment of Likelihood of Delaying or Preventing Permanent Damage to the Environmental Services Permanent Damage to the Environmental Services Provided by the SCSProvided by the SCS

• Feasibility of expert group assessment will be exploredFeasibility of expert group assessment will be explored8

Product:Product: Identify Lessons from the Identify Lessons from the SCS Evaluation Applicable to GEF SCS Evaluation Applicable to GEF Support to International WatersSupport to International Waters

• Interact with stakeholders to draw Interact with stakeholders to draw useful lessonsuseful lessons

• Draw lessons from the pastDraw lessons from the past

• Identify opportunities for the futureIdentify opportunities for the future– I W Task force / 2I W Task force / 2ndnd Reference group Reference group

meetingmeeting

• Identify missed opportunitiesIdentify missed opportunities

9

Stakeholder InvolvementStakeholder Involvement• Three groups are designed as vehicles for Three groups are designed as vehicles for

stakeholder input and to support the evaluation. stakeholder input and to support the evaluation. These groups will:These groups will:– Comment on the approach paper draft and reportComment on the approach paper draft and report– Provide suggestions on making evaluation useful to operationsProvide suggestions on making evaluation useful to operations– Help establish contact with appropriate contactsHelp establish contact with appropriate contacts– Help identify and facilitate access to informationHelp identify and facilitate access to information

• Technical Advisory Group (TAG): Technical Advisory Group (TAG): Consisting of 6 scientific Consisting of 6 scientific and technical specialists, the TAG will provide quality assurance support and technical specialists, the TAG will provide quality assurance support on methodological, scientific, and technical issues.on methodological, scientific, and technical issues.

• IW Task Force: IW Task Force: Composed of IW focal area coordinators, the GEF Composed of IW focal area coordinators, the GEF Secretariat, and STAP, the IW Task Force will provide input into the Secretariat, and STAP, the IW Task Force will provide input into the selection of knowledge products as well as the water body to be evaluated; selection of knowledge products as well as the water body to be evaluated; it will also comment on the approach paper and evaluation report and it will also comment on the approach paper and evaluation report and facilitate ongoing communication with GEF Agencies.facilitate ongoing communication with GEF Agencies.

• Reference Group:Reference Group: This group will consist of 15-20 people, This group will consist of 15-20 people, including representatives from the GEF Secretariat, GEF Agencies, key including representatives from the GEF Secretariat, GEF Agencies, key staff involved in SCS GEF projects, and some non-GEF stakeholder staff involved in SCS GEF projects, and some non-GEF stakeholder institutions. This group will play a critical role in the evaluation follow up.institutions. This group will play a critical role in the evaluation follow up.10

Management of EvaluationManagement of EvaluationThe core evaluation team will consist of:The core evaluation team will consist of:

Aaron Zazueta, Senior Evaluation Officer, Aaron Zazueta, Senior Evaluation Officer, GEFEO. GEFEO. azazueta@thegef.org

Neeraj Negi, Evaluation Officer, GEFEO. Neeraj Negi, Evaluation Officer, GEFEO. nnegi1@thegef.org

Jeneen Garcia, Research Analyst, GEFEO. Jeneen Garcia, Research Analyst, GEFEO. jgarcia2@thegef.org

Several consultants will also be hired to Several consultants will also be hired to produce subreports and gather data.produce subreports and gather data.

11

Evaluation TimelineEvaluation TimelineDate Activity

Nov 2009 Upstream consultations on evaluation questions and water body to evaluate

Dec 10, 2009

Portfolio review and selection of candidate water catchments

Sep 2010 GEF IW theory of change for large marine ecosystemsSep 29/30, 2010

Reference Group Meeting to discuss approach paper

Nov 2010 Portfolio Analysis and Mapping of SCS institutional actors and programs

Mar 2011 Characterization of the social, economic and ecological services of the SCS

Jun 2011 Analysis of GEF contributions to transboundary environmental governance

Aug 2011 Country case studiesOct 2011 Assessment of likelihood of transboundary environmental

impactsFeb 15, 2012

Meeting of the Reference Group to discuss comments on the draft

Apr 2012 Report and Council paper preparedJul/Nov 2012

Publication and learning products

12

Thank you.Thank you.

• For more information on this For more information on this evaluation, please contact Aaron evaluation, please contact Aaron Zazueta, azazueta@thegef.orgZazueta, azazueta@thegef.org

13