The Application of the CDM M,E&R Framework at the National and Regional Level

The Application of the CDM M,E&R Framework at

the National and Regional Level

Le Groupe Conseil baastel Ltee

5th Annual CDM Conference

Jamaica – December 8th, 2010

OutlineOutline

• What is the Regional CDM M,E&R Framework and Why is it Important?

• The Process of Establishing the Framework• How to Monitor using the Framework• Putting the Framework into Practice – now and potential

for the future • Considerations and Next Steps

• What is the Regional CDM M,E&R Framework and Why is it Important?

• The Process of Establishing the Framework• How to Monitor using the Framework• Putting the Framework into Practice – now and potential

for the future • Considerations and Next Steps

What is the CDM M, E &R Framework?

Why is it Needed?

Why the need for a CDM M,E&R Framework?Why the need for a CDM M,E&R Framework?

The CDM Strategy/Framework is based on a variety of contributing sources including Country Work Plans (CWPs), partner projects and programmes, either nationally or regionally and sectoral programming

The CDM Strategy/Framework is based on a variety of contributing sources including Country Work Plans (CWPs), partner projects and programmes, either nationally or regionally and sectoral programming

What is the M,E&R Framework? Why is it Important?What is the M,E&R Framework? Why is it Important?

• Developed in 2009-2010, the CDM Monitoring, Evaluation and Reporting (M,E&R) Framework outlines the overall strategy and plan for monitoring and reporting of the Enhanced Regional CDM Strategy and Framework

• Impetus for the framework is the need to track/report on progress of Disaster Management interventions related to the CDM results (outputs/outcomes) in the Caribbean Region

• Uses a utility-based Results Based Management approach for tracking progress in order to reduce monitoring burden and improve data and information being collected and shared

• Developed in 2009-2010, the CDM Monitoring, Evaluation and Reporting (M,E&R) Framework outlines the overall strategy and plan for monitoring and reporting of the Enhanced Regional CDM Strategy and Framework

• Impetus for the framework is the need to track/report on progress of Disaster Management interventions related to the CDM results (outputs/outcomes) in the Caribbean Region

• Uses a utility-based Results Based Management approach for tracking progress in order to reduce monitoring burden and improve data and information being collected and shared

What is the M,E&R Framework? Why is it Important?What is the M,E&R Framework? Why is it Important?

• The purpose is to enable CDEMA to measure ongoing performance of progress being made towards Regional CDM expected results, facilitate reporting from the national and regional levels, assist in meeting accountability standards with development partners, and facilitate the learning agenda

• Overall, it serves as a tool for decision-making as well as to inform strategic planning, programming and learning– At country level– At regional level

• The purpose is to enable CDEMA to measure ongoing performance of progress being made towards Regional CDM expected results, facilitate reporting from the national and regional levels, assist in meeting accountability standards with development partners, and facilitate the learning agenda

• Overall, it serves as a tool for decision-making as well as to inform strategic planning, programming and learning– At country level– At regional level

Core of the Model – CDM Impact and OutcomesCore of the Model – CDM Impact and Outcomes

Core of the Model – CDM OutputsCore of the Model – CDM Outputs

Monitoring the CDM FrameworkMonitoring the CDM Framework

• Development of a Performance Measurement Framework (PMF) as the core monitoring tool and guide

• PMF guides planning of the systematic compilation of data on progress and results. It outlines the main elements of the monitoring system and ensures that performance information is collected regularly/on time

• Development of a Performance Measurement Framework (PMF) as the core monitoring tool and guide

• PMF guides planning of the systematic compilation of data on progress and results. It outlines the main elements of the monitoring system and ensures that performance information is collected regularly/on time

Salient Features of the CDM MER Framework - PMF ElementsSalient Features of the CDM MER Framework - PMF Elements

Definition of ‘Indicators’ in the CDM Strategy/Framework Context

An indicator is a representative aspect or element of a CDM Output (or Outcome) in which change over time is measurable and the change attributable to activities (i.e.

projects & programmes) and is used to demonstrate progress towards the attainment of the expected Result

(i.e. Output/Outcome)

It can be used to ‘detect’, ‘point to’, or indicate some change (or lack thereof) over time in relation to a given

expected Result

Definition of ‘Indicators’ in the CDM Strategy/Framework Context

An indicator is a representative aspect or element of a CDM Output (or Outcome) in which change over time is measurable and the change attributable to activities (i.e.

projects & programmes) and is used to demonstrate progress towards the attainment of the expected Result

(i.e. Output/Outcome)

It can be used to ‘detect’, ‘point to’, or indicate some change (or lack thereof) over time in relation to a given

expected Result


Progress towards attainment ofCDM Outputs due to completed

activities


“Keeping it Manageable”

• Guidance of Review Committee (CDEMA, NDCs, Development and Sectoral Partners)

• Limited set of indicators for each Output– To get process going (delayed monitoring &

baseline already when mandate began in 2009)

– To catalyze buy-in to M, E & R Framework

– To keep monitoring feasible

– Recalling monitoring & reporting burden on NDOs

“Keeping it Manageable”

• Guidance of Review Committee (CDEMA, NDCs, Development and Sectoral Partners)

• Limited set of indicators for each Output– To get process going (delayed monitoring &

baseline already when mandate began in 2009)

– To catalyze buy-in to M, E & R Framework

– To keep monitoring feasible

– Recalling monitoring & reporting burden on NDOs


• Data Requirements, Sources and Tools:• Data requirements for each output and outcome

indicator are essential to respond to the indicator’s measurement of each result. For each indicator, data requirements are indicated

• The source is where or who this data will come from – these include CDEMA CU, NDOs, Key Sectors and other Ministries, etc.

• Tools: Nine simple tools have been developed and are contained in Volume II: M,E&R Framework Manual and Tools

• Data Requirements, Sources and Tools:• Data requirements for each output and outcome

indicator are essential to respond to the indicator’s measurement of each result. For each indicator, data requirements are indicated

• The source is where or who this data will come from – these include CDEMA CU, NDOs, Key Sectors and other Ministries, etc.

• Tools: Nine simple tools have been developed and are contained in Volume II: M,E&R Framework Manual and Tools


• Responsibility: Core responsibility of CDEMA CU. They also will depend heavily on the NDO for provision of data

• Frequency: Annually, for the majority of output/outcome indicators. Collection at more frequent intervals would not necessarily result in significant or demonstrable changes due to the regional nature of programming

• Collection Methods: Two primary methods of collection: a) document review; and b) questionnaire (via electronic means, telephone or in-person, etc.)

When a web-based platform is established this method

will change to solely web-based questionnaire, considerably reducing the data collection burden

• Responsibility: Core responsibility of CDEMA CU. They also will depend heavily on the NDO for provision of data

• Frequency: Annually, for the majority of output/outcome indicators. Collection at more frequent intervals would not necessarily result in significant or demonstrable changes due to the regional nature of programming

• Collection Methods: Two primary methods of collection: a) document review; and b) questionnaire (via electronic means, telephone or in-person, etc.)

When a web-based platform is established this method

will change to solely web-based questionnaire, considerably reducing the data collection burden

Salient Features of the CDM MER Framework – Key LinkagesSalient Features of the CDM MER Framework – Key Linkages

• Linkages between and to National CWPs, sectoral programming and other projects/programmes– Data can be gleaned from the M&E frameworks of these to

contribute to Regional CDM monitoring and reporting.

• While all data needs are not satisfied, there is considerable synergy. Conceptually it looks like:

• Linkages between and to National CWPs, sectoral programming and other projects/programmes– Data can be gleaned from the M&E frameworks of these to

contribute to Regional CDM monitoring and reporting.

• While all data needs are not satisfied, there is considerable synergy. Conceptually it looks like:

Linkages/Mapping to Hyogo Framework of Action (HFA)Linkages/Mapping to Hyogo Framework of Action (HFA)

• Monitoring slated to take place through the M,E&R Framework facilitates this global monitoring as outputs and accompanying indicators are either directly or partially linked to the Priorities for Action (PFA) indicators

• Reporting to HFA in the CDM context can be done via:• National PFA Indicators & Regional PFA Indicators

• Monitoring slated to take place through the M,E&R Framework facilitates this global monitoring as outputs and accompanying indicators are either directly or partially linked to the Priorities for Action (PFA) indicators

• Reporting to HFA in the CDM context can be done via:• National PFA Indicators & Regional PFA Indicators

Mapping to Hyogo Framework of Action (HFA)Mapping to Hyogo Framework of Action (HFA)

Strategic Goal 2: Development and strengthening of institutions, mechanisms and capacities to build resilience to hazardsInd.3: Dedicated and sufficient resources are available for planned activities to reduce disaster risks

RegionalInd.7: Percentage of national dedicated resources available for disaster risk reduction

Outcome 1, Ind.2: % of Caribbean countries with sufficient resources available for planning CDM interventions

Output 1.1 Ind. 2: % of national budget provided to NDO for routine operations

Outcome 2, Ind.3: # of male and female graduates in CDM/DRM and CDM specialists in each country

PFA 2: Identify, assess and monitor disaster risks and enhance early warning

Ind.1: National and local risk assessments based on hazard data and vulnerability information are available and include risk assessments for keySectors

RegionalInd.5: Number of national strategies of data provision for disaster risk reduction, including climate change accounted for in risk analysis

Ind.2.3.1: # of countries that have a community level Hazard and Vulnerability Assessment (HVA) processes that incorporates the use of local knowledge

Ind. 2.3.2: # of countries that have national level HVA that incorporates local community knowledge in the conduct of and to determine priority hazards

Ind.3.3.1: Level of satisfaction of lead agencies for key sectors with hazard/disaster knowledge resources available for their CDM planning and for decision-making

How Did We Get Here?

Participatory ProcessParticipatory Process

Process for Establishing the M,E&R FrameworkProcess for Establishing the M,E&R Framework

PHASE 1: Selection of Indicators

PHASE 1: Selection of Indicators

PHASE 2: Identification

of Data Requirements


of Data Requirements


of Data Sources


of Data Sources

Development of CDM

Outcomes and Outputs

Development of CDM

Outcomes and Outputs

PHASE 4: Tool Design and Collection

Process of Data

PHASE 4: Tool Design and Collection

Process of Data

PHASE 5: Country

Missions and Data

Collection

PHASE 5: Country

Missions and Data

Collection

Final M,E&R Framework

Final M,E&R Framework

Regional Baseline Report

Regional Baseline Report

Country Baseline Reports

Country Baseline Reports

Technical Report & Excel Database

(Raw Data)

Technical Report & Excel Database

(Raw Data)

Volume II: M,E&R Tools

Volume II: M,E&R Tools

Current Phase

Establishing the M,E&R FrameworkEstablishing the M,E&R Framework

• It has been an iterative process in that it has involved various stakeholders including NDOs, donor agencies, sector representatives, etc.

• Began with RBM workshops in December 2008 and February 2009 and Pre-Board Workshops in April 2009

• Work since December 2009 has focused on Phase 5 of the process with country visits and collection of baseline data

• This led to the development of a regional baseline study, country baseline reports, the regional technical report and the overall M,E&R Framework (Volume I) and Data Collection Tools for Monitoring (Volume II)

• It has been an iterative process in that it has involved various stakeholders including NDOs, donor agencies, sector representatives, etc.

• Began with RBM workshops in December 2008 and February 2009 and Pre-Board Workshops in April 2009

• Work since December 2009 has focused on Phase 5 of the process with country visits and collection of baseline data

• This led to the development of a regional baseline study, country baseline reports, the regional technical report and the overall M,E&R Framework (Volume I) and Data Collection Tools for Monitoring (Volume II)

CDM Regional Baseline Data and ReportCDM Regional Baseline Data and Report

• The baseline is critical to the M,E&R Framework and now provides a ‘starting point’ against which measurement can begin

• Baseline data was collected in late 2009/early 2010 from 16 countries against an agreed set of core indicators using a Baseline Assessment Tool (BAT)

Some Sample Baseline Data Follows

• The baseline is critical to the M,E&R Framework and now provides a ‘starting point’ against which measurement can begin

• Baseline data was collected in late 2009/early 2010 from 16 countries against an agreed set of core indicators using a Baseline Assessment Tool (BAT)

Some Sample Baseline Data Follows

Outcome 1: Enhanced Institutional Support for CDM Program Implementation

Outcome 1: Enhanced Institutional Support for CDM Program Implementation

Output 1.1: National Disaster Organizations are Strengthened for Supporting CDM Implementation and a CDM Program is Developed for Implementation at the National Level

Indicator 1.1.1: # of countries with drafted/enacted disaster legislation8 of 15 countries (53.3%) state that they have a drafted or enacted legislation, 6 of these 8 (75%) state that it mandates and defines their status. This is not applicable to 7 countries.

Output 1.1: National Disaster Organizations are Strengthened for Supporting CDM Implementation and a CDM Program is Developed for Implementation at the National Level

Indicator 1.1.1: # of countries with drafted/enacted disaster legislation8 of 15 countries (53.3%) state that they have a drafted or enacted legislation, 6 of these 8 (75%) state that it mandates and defines their status. This is not applicable to 7 countries.

Output 1.6: Capacity for Monitoring, Evaluation and Reporting Being Built

Output 1.6: Capacity for Monitoring, Evaluation and Reporting Being Built

14 respondents

NDO ratings of ability to respond to

CDM-related reporting requests

Output 2.2: Utilization of Information Communication Technology (ICT) and other Infrastructure for fact-based Policy and Decision-Making

is Established and Strengthened

Output 2.2: Utilization of Information Communication Technology (ICT) and other Infrastructure for fact-based Policy and Decision-Making

is Established and Strengthened

15 respondents

NDO rating of ICT inter-

operability between

CDEMA CU and countries

Output 3.1: CDM is Recognized as the Roadmap for Building Resilience and Decision-Makers in the Public and Private Sectors Understand

and Take Action on Disaster Risk Management

Output 3.1: CDM is Recognized as the Roadmap for Building Resilience and Decision-Makers in the Public and Private Sectors Understand

and Take Action on Disaster Risk Management

13 respondents 14 respondents 13 respondents 12 respondents

80% 83.3% 12.5% 66.7%

Whether sector disaster plans are congruent with NDP

Output 3.4: Prevention, Mitigation, Preparedness, Response, Recovery and Rehabilitation Procedures Developed and Implemented in

Tourism, Health, Agriculture and Education

Output 3.4: Prevention, Mitigation, Preparedness, Response, Recovery and Rehabilitation Procedures Developed and Implemented in

Tourism, Health, Agriculture and Education

Sectors rating of level of adequacy of institutional

framework, by sector

Output 4.2: Improved Coordination and Collaboration between Community Disaster Organizations and other Research/Data Partners including Climate Change

Entities for Undertaking CDM

Output 4.2: Improved Coordination and Collaboration between Community Disaster Organizations and other Research/Data Partners including Climate Change

Entities for Undertaking CDM

12 Respondents

Community accessibility to hazard maps and relevance of

mapping

Output 4.4: Standardized Holistic and Gender –Sensitive Community Methodologies for Natural and Anthropogenic Hazard Identification and Mapping, Vulnerability and

Risk Assessments and Recovery and Rehabilitation Procedures Developed and Applied in Selected Communities

Output 4.4: Standardized Holistic and Gender –Sensitive Community Methodologies for Natural and Anthropogenic Hazard Identification and Mapping, Vulnerability and

Risk Assessments and Recovery and Rehabilitation Procedures Developed and Applied in Selected Communities

Indicator 4.4.5: Evidence that shelter management policies & procedures address women’s protection (by country)Baseline data suggests that 8 of 15 countries (53.3%) have shelter management policies and/or procedures that address women’s protection.

Indicator 4.4.6: Evidence of sex-disaggregated data collection at country level on damages, loss, etc.National data collection and reporting on damages and losses after disasters is disaggregated by sex in 9 of 15 countries (60%). One additional country noted that this kind of disaggregation was being planned and tools are in development.

Indicator 4.4.5: Evidence that shelter management policies & procedures address women’s protection (by country)Baseline data suggests that 8 of 15 countries (53.3%) have shelter management policies and/or procedures that address women’s protection.

Indicator 4.4.6: Evidence of sex-disaggregated data collection at country level on damages, loss, etc.National data collection and reporting on damages and losses after disasters is disaggregated by sex in 9 of 15 countries (60%). One additional country noted that this kind of disaggregation was being planned and tools are in development.

Sex-Disaggregated Data

Technical Report – Raw Data by CountryTechnical Report – Raw Data by Country

To compliment the baseline report, a secondary technical supplementary document was developed. The contents of this report include:

i. Explanatory notes on data analysis and aggregation of data

ii. Results of each indicator by output (broken down by country) as a disaggregation of data

iii.Explanatory notes and references for each indicator relevant to the transfer of data to any web-based platform or database

To compliment the baseline report, a secondary technical supplementary document was developed. The contents of this report include:

i. Explanatory notes on data analysis and aggregation of data

ii. Results of each indicator by output (broken down by country) as a disaggregation of data

iii.Explanatory notes and references for each indicator relevant to the transfer of data to any web-based platform or database

CDM Regional Baseline Data and ReportCDM Regional Baseline Data and Report

• In addition, Country Baseline Reports were also developed for each of the 16 countries covered

• In addition, Country Baseline Reports were also developed for each of the 16 countries covered

LimitationsLimitations

Like any baseline, there are challenges and limitations. Two of the issues that are most critical to keep in mind are:

Economies of Scale - Because data presented regionally, assumes indicators speak equally across countries – they do not. Analysis limited in terms of country size, population, frequency of disasters, poverty rates, etc.

Understanding of Terminology/Concepts

a)CDM: discourse around CDM during baseline data

collection is still in process of evolution and therefore

understood differently between stakeholders outside of

NDO. This limited uniform responses and required an

enhancement.

Like any baseline, there are challenges and limitations. Two of the issues that are most critical to keep in mind are:

Economies of Scale - Because data presented regionally, assumes indicators speak equally across countries – they do not. Analysis limited in terms of country size, population, frequency of disasters, poverty rates, etc.


a)CDM: discourse around CDM during baseline data

collection is still in process of evolution and therefore

understood differently between stakeholders outside of

NDO. This limited uniform responses and required an

enhancement.

Limitations and ResolutionsLimitations and Resolutions


b)Climate Change discourse also evolving and various

levels of understanding. Since no common agreed

upon criteria for integrating climate change, provision of

precise data was limited and required an enhancement.

c)Gender: understanding of gender mainstreaming not

understood and defined consistently and various levels

of understanding. Since no common agreed upon

criteria for gender mainstreaming, provision of precise

data was limited and required an enhancement.


b)Climate Change discourse also evolving and various

levels of understanding. Since no common agreed

upon criteria for integrating climate change, provision of

precise data was limited and required an enhancement.

c)Gender: understanding of gender mainstreaming not

understood and defined consistently and various levels

of understanding. Since no common agreed upon

criteria for gender mainstreaming, provision of precise

data was limited and required an enhancement.

Limitations and ResolutionsLimitations and Resolutions

• Based on this process, several issues have been resolved through the final M,E&R Framework:

a) Indicators: certain indicators (and benchmarks) have been revised and enhanced to improve data being collected and facilitate data collection processes in the future

b) Tools: based on experiences the BAT and subsequent data analysis, a revised set of user-friendly monitoring tools have been developed

• Based on this process, several issues have been resolved through the final M,E&R Framework:

a) Indicators: certain indicators (and benchmarks) have been revised and enhanced to improve data being collected and facilitate data collection processes in the future

b) Tools: based on experiences the BAT and subsequent data analysis, a revised set of user-friendly monitoring tools have been developed

Improvements for Monitoring of Cross-Cutting Themes - GenderImprovements for Monitoring of Cross-Cutting Themes - Gender

• ISDR and the HFA require that countries mainstream gender in risk management. Aside from this requirement, mainstreaming of gender in disaster management and risk reduction is crucial as women and men are affected differently by disasters and hazards

• The M,E&R Framework integrates gender as a cross-cutting theme across the four outcome areas through 11 gender-specific or gender-integrated indicators (e.g. integration of gender issues into policies/plans; evidence that shelter policies/plans & procedures address women’s protection, etc)

• Indicators and data requirements for these, as well as others, have been revised to include criteria for more comprehensive assessment of gender in CDM monitoring

• ISDR and the HFA require that countries mainstream gender in risk management. Aside from this requirement, mainstreaming of gender in disaster management and risk reduction is crucial as women and men are affected differently by disasters and hazards

• The M,E&R Framework integrates gender as a cross-cutting theme across the four outcome areas through 11 gender-specific or gender-integrated indicators (e.g. integration of gender issues into policies/plans; evidence that shelter policies/plans & procedures address women’s protection, etc)

• Indicators and data requirements for these, as well as others, have been revised to include criteria for more comprehensive assessment of gender in CDM monitoring

Cross-Cutting Themes - GenderCross-Cutting Themes - Gender

Example of revised data requirements for gender indicators – Tool A

Cross-Cutting Themes – Climate ChangeCross-Cutting Themes – Climate Change

•The CDM Strategy/Framework addresses the cross-cutting issue of Climate Variability and Change and key considerations were integrated into the outputs of the Strategy and Framework in 2007.

•In the current PMF, there are a total of 15 indicators which either directly address climate change and adaptation or where there is overlap and synergy between measuring aspects related to CDM and those related to climate change

•Data requirements have now been refined to ensure more comprehensive assessment of climate change elements

•The CDM Strategy/Framework addresses the cross-cutting issue of Climate Variability and Change and key considerations were integrated into the outputs of the Strategy and Framework in 2007.

•In the current PMF, there are a total of 15 indicators which either directly address climate change and adaptation or where there is overlap and synergy between measuring aspects related to CDM and those related to climate change

•Data requirements have now been refined to ensure more comprehensive assessment of climate change elements

Cross-Cutting Themes – Climate ChangeCross-Cutting Themes – Climate Change

How to Monitor Using the Framework

Putting it Into Practice

Process for Monitoring CDMProcess for Monitoring CDM

• Challenges in monitoring at the regional level due to:a) Multiple and diverse stakeholders with different needs,

systems and processes

b) Lack of harmonized programming/monitoring which involves multiple monitoring requirements

c) Lack of existing ICT capabilities present in the region to accommodate monitoring at a regional level

• As a result, two phase approach is necessary toward functional and sustainable system:• Phase 1: addresses immediate steps and processes that can

be implemented in context of current reality• Phase 2: processes once an enhanced ICT system in place

• Challenges in monitoring at the regional level due to:a) Multiple and diverse stakeholders with different needs,

systems and processes

b) Lack of harmonized programming/monitoring which involves multiple monitoring requirements

c) Lack of existing ICT capabilities present in the region to accommodate monitoring at a regional level

• As a result, two phase approach is necessary toward functional and sustainable system:• Phase 1: addresses immediate steps and processes that can

be implemented in context of current reality• Phase 2: processes once an enhanced ICT system in place

Phased Approach to M,E&R FrameworkPhased Approach to M,E&R Framework

• Phase I: – Use of Excel database with all raw data (keep as is)

or transfer to another chosen in-house and stand alone off the shelf software (e.g. File Maker, Access)

– In early 2011, disseminate/administer data collection tools. These can be done electronically or submitted in paper form

• Phase II: Expansion and enhancement of an ICT platform within specified time frame so data can be uploaded directly

• Phase I: – Use of Excel database with all raw data (keep as is)

or transfer to another chosen in-house and stand alone off the shelf software (e.g. File Maker, Access)

– In early 2011, disseminate/administer data collection tools. These can be done electronically or submitted in paper form

• Phase II: Expansion and enhancement of an ICT platform within specified time frame so data can be uploaded directly

Phase 1: Indirect Process for Data CollectionPhase 1: Indirect Process for Data Collection

Steps to Monitor in Phase ISteps to Monitor in Phase I

Monitoring – Data Collection ToolsMonitoring – Data Collection Tools

• In total there are 9 tools that have been developed – all in Word format with Active Controls in order to simplify completion of data forms

• In total there are 9 tools that have been developed – all in Word format with Active Controls in order to simplify completion of data forms


Keeping it simple:

• Check Boxes: data entry persons select appropriate selection. In some cases, data entry persons will need to select more than one choice. In these instances, instructions follow the question.

• Option Button: option buttons are used for all benchmarks where data entry persons can only select one choice. If a choice is changed, the option control will automatically change.

Keeping it simple:

• Check Boxes: data entry persons select appropriate selection. In some cases, data entry persons will need to select more than one choice. In these instances, instructions follow the question.

• Option Button: option buttons are used for all benchmarks where data entry persons can only select one choice. If a choice is changed, the option control will automatically change.


• CDEMA CU is responsible for overall monitoring and so there are more tools specific to the CU as data which is submitted to the CU is done so for analysis at the regional level by output

– Tool B-1: Outputs: This tool is used for indicators where CDEMA CU is responsible for data collection directly and is developed in a similar way as the tools for NDOs and sectors

– Tool B-2: Roll up from country and sectors: This tool is used for the aggregation or consolidation of data collected through the NDO, Key Sectors or other stakeholders. It is intended as a data capture sheet prior to input into a database

– Tool B-3: Outcome and Impact: This tool is used to capture data of all four outcomes as well as the three indicators for impact

– Tool B-4: Document checklist : This tool serves as a checklist for documents that require review to validate indicators at all levels

• CDEMA CU is responsible for overall monitoring and so there are more tools specific to the CU as data which is submitted to the CU is done so for analysis at the regional level by output

– Tool B-1: Outputs: This tool is used for indicators where CDEMA CU is responsible for data collection directly and is developed in a similar way as the tools for NDOs and sectors

– Tool B-2: Roll up from country and sectors: This tool is used for the aggregation or consolidation of data collected through the NDO, Key Sectors or other stakeholders. It is intended as a data capture sheet prior to input into a database

– Tool B-3: Outcome and Impact: This tool is used to capture data of all four outcomes as well as the three indicators for impact

– Tool B-4: Document checklist : This tool serves as a checklist for documents that require review to validate indicators at all levels

Data Collection Tools: Example Tool B-1Data Collection Tools: Example Tool B-1

Data Tools: Example Tool AData Tools: Example Tool A

Summary of Monitoring ProcessSummary of Monitoring Process

a) From National to Regional

b) From Sectoral to Regional



1. From National Level to Regional Analysis1. From National Level to Regional Analysis

• Three monitoring processes for aggregation from National Levels (PS) to Regional Level:

a) Direct Aggregation from National Data (single criteria/data)

b) From National to Regional Level using Benchmarks (single criteria/data)

c) Other roll-up

• Three monitoring processes for aggregation from National Levels (PS) to Regional Level:

a) Direct Aggregation from National Data (single criteria/data)


c) Other roll-up

a) Monitoring Indicator with Single Criteria (National to Regional)

1b. From National Level to Regional Analysis with Benchmark1b. From National Level to Regional Analysis with Benchmark

• An example of this is Indicator 1.1.5a: Quality of CDM Strategy/Framework/CWP in CDEMA PS, where data is collected through the NDO, but where CDEMA CU is responsible for analysing with a corresponding benchmark scale

• In Tool A, the NDO is asked whether there is a country strategy, framework or CWP. If yes, they proceed to the next question which provides a list of the four CDM phases. Based on the total responses, a grade can be given on the benchmark scale

• An example of this is Indicator 1.1.5a: Quality of CDM Strategy/Framework/CWP in CDEMA PS, where data is collected through the NDO, but where CDEMA CU is responsible for analysing with a corresponding benchmark scale

• In Tool A, the NDO is asked whether there is a country strategy, framework or CWP. If yes, they proceed to the next question which provides a list of the four CDM phases. Based on the total responses, a grade can be given on the benchmark scale

1b. From National Level to Regional Analysis1b. From National Level to Regional Analysis



1b. From National Level to Regional Analysis with Benchmark1b. From National Level to Regional Analysis with Benchmark

1c. Roll up that Requires Other Analysis1c. Roll up that Requires Other Analysis

• In a minority of cases there are indicators which require multiple criteria (questions and sub-questions) (e.g. % of budget provided to NDO for routine operations)

• Instead of aggregation, three pieces of data requirements are needed

• Data from a and b come from a Tool, but c is calculated by CDEMA CU:

• In a minority of cases there are indicators which require multiple criteria (questions and sub-questions) (e.g. % of budget provided to NDO for routine operations)

• Instead of aggregation, three pieces of data requirements are needed

• Data from a and b come from a Tool, but c is calculated by CDEMA CU:

Country

(a)Annual National Budget ($US)

(b)Annual NDO Budget for Operations ($US)

(c)% of Budget for operations (b/a=c)

Anguilla

Antigua/Barbuda

Bahamas

Barbados

From Sectoral to Regional AnalysisFrom Sectoral to Regional Analysis

• Roll up from sectoral is similar to roll up from national, but includes one additional step - aggregating from the sector level to the country level prior to regional level consolidation

• Roll up from sectoral is similar to roll up from national, but includes one additional step - aggregating from the sector level to the country level prior to regional level consolidation

Process for Monitoring single criteria from sectoral to regional

From Sectoral to Regional Analysis (Multiple Criteria – No Benchmark)

From Sectoral to Regional Analysis (Multiple Criteria – No Benchmark)

• As opposed to multiple criteria and aggregation of national level to regional level, data collection process from the sectoral level is different as four sectors or more are considered per country before aggregation can be derived.

• The process involves:

• Data collection through Key Sector Tools (C1) then submitted to NDO for onward submission to CDEMA CU

• Document reviewed and verified, if applicable to methodology

• Data rolled up by each sector and country (# of sectors per country which states “yes”)

• Data entry in Database (e.g. Excel or Access for Phase I)

• As opposed to multiple criteria and aggregation of national level to regional level, data collection process from the sectoral level is different as four sectors or more are considered per country before aggregation can be derived.

• The process involves:

• Data collection through Key Sector Tools (C1) then submitted to NDO for onward submission to CDEMA CU

• Document reviewed and verified, if applicable to methodology

• Data rolled up by each sector and country (# of sectors per country which states “yes”)

• Data entry in Database (e.g. Excel or Access for Phase I)

From Sectoral to Regional (Multiple Criteria – No Benchmark)

From Sectoral to Regional (Multiple Criteria – No Benchmark)

The data above (fictional) reveals that in 3 countries, the health sector integrates CDM & all hazards in sector policies, while in 2 countries CDM is less integrated.

1=Yes; No=2

From Sectoral to Regional (Benchmark)

From Sectoral to Regional (Benchmark)

• Similar to monitoring indicators from national to regional, benchmarks are used: e.g. 3.3.1a,b,c,d: Level of satisfaction of lead agencies for key sector with hazard/disaster knowledge resources available for CDM planning.

• Similar to monitoring indicators from national to regional, benchmarks are used: e.g. 3.3.1a,b,c,d: Level of satisfaction of lead agencies for key sector with hazard/disaster knowledge resources available for CDM planning.

Summary of Monitoring ProcessSummary of Monitoring Process





Potential for Auto-Generation of ReportingPotential for Auto-Generation of Reporting

• CDM data entered into a simple database could then be an effective tool for simplifying reporting for a variety of purposes (CDM Report, HFA).

• Reporting will be straightforward once the following elements are completed:• Entering of baseline data and subsequent data collection

annually from PSs and Sectors through the current CDM Framework;

• System capture of mapping of linkages of data and indicators from HFA and other related monitoring frameworks already completed (e.g. PAHO, some CWPs); and

• Mapping of new linkages of data and indicators from other and future programmes and projects.

• CDM data entered into a simple database could then be an effective tool for simplifying reporting for a variety of purposes (CDM Report, HFA).

• Reporting will be straightforward once the following elements are completed:• Entering of baseline data and subsequent data collection

annually from PSs and Sectors through the current CDM Framework;

• System capture of mapping of linkages of data and indicators from HFA and other related monitoring frameworks already completed (e.g. PAHO, some CWPs); and

• Mapping of new linkages of data and indicators from other and future programmes and projects.

Potential for Auto-Generation of ReportingPotential for Auto-Generation of Reporting

For Phase 1 this can easily be accomplished through Access, File Maker) or other programmes/databases where multiple data informs multiple reports at the same time based on queries

For Phase 1 this can easily be accomplished through Access, File Maker) or other programmes/databases where multiple data informs multiple reports at the same time based on queries

Phase 2: Direct Process for Data CollectionPhase 2: Direct Process for Data Collection

Monitoring-related FindingsMonitoring-related Findings

• Results from baseline show that the majority (83.2%) see the benefit in having data updated on a web-based platform; 85.2% are amenable to use web-based platform.

• Outside NDOS, 94.8% of stakeholders would be amenable to being contacted by the NDO rather than self-input data.

• Results from baseline show that the majority (83.2%) see the benefit in having data updated on a web-based platform; 85.2% are amenable to use web-based platform.

• Outside NDOS, 94.8% of stakeholders would be amenable to being contacted by the NDO rather than self-input data.

Phased Approach and Potential for M,E&R Framework with Enhanced Database

Phased Approach and Potential for M,E&R Framework with Enhanced Database

Concluding Considerations Concluding Considerations

• The M,E&R Framework provides a coherent vision for monitoring and reporting where all stakeholders are contributing towards the same CDM expected results

• It requires active participation from CDEMA CU and NDOs• Can also enhance collaboration

• Enormous potential for reducing the monitoring and reporting• Opportunity to link with ICT restructuring; without it,

potential shall not be realized

• The M,E&R Framework provides a coherent vision for monitoring and reporting where all stakeholders are contributing towards the same CDM expected results

• It requires active participation from CDEMA CU and NDOs• Can also enhance collaboration

• Enormous potential for reducing the monitoring and reporting• Opportunity to link with ICT restructuring; without it,

potential shall not be realized

Next StepsNext Steps

For the CU:

• Selection of database for Phase I• Migrating of Baseline data and systematization of PMF

elements• Systematize linkages from CWPs (where these linkages

have been completed) and other tools (e.g. PAHO) and enter available data

• Systematize linkages to HFA (to simplify reporting)• Establish (and then systematize) linkages from existing

projects/programmes

For the CU:

• Selection of database for Phase I• Migrating of Baseline data and systematization of PMF

elements• Systematize linkages from CWPs (where these linkages

have been completed) and other tools (e.g. PAHO) and enter available data

• Systematize linkages to HFA (to simplify reporting)• Establish (and then systematize) linkages from existing

projects/programmes

Next StepsNext Steps

For the CU:• Begin annual monitoring process in early 2011• Training workshop for monitoring

For the CU:• Begin annual monitoring process in early 2011• Training workshop for monitoring

Thank You

Questions?

The Application of the CDM M,E&R Framework at the National and Regional Level

Documents

Transcript of The Application of the CDM M,E&R Framework at the National and Regional Level