Monitoring and Evaluation (M&E) System for the Comprehensive Africa Agriculture Development...

MONITORING AND EVALUATION (M&E) SYSTEMFOR THE

COMPREHENSIVE AFRICA AGRICULTURE DEVELOPMENT

PROGRAMME (CAADP)

6TH CAADP PARTNERSHIP PLATFORM

BIRCHWOOD HOTEL, JOHANNESBURG, SOUTH AFRICA

APRIL 21-23, 2010

BABATUNDE OMILOLACOORDINATOR, REGIONAL STRATEGIC ANALYSIS AND KNOWLEDGE SUPPORT SYSTEM

(RESAKSS)

INTERNATIONAL FOOD POLICY RESEARCH INSTITUTE (IFPRI)

Outline• What is CAADP?

–Principles

–Processes and implementation

• Why an overarching M&E System for CAADP?

• Indicators to monitor and evaluate

• Operationalizing the CAADP M&E Framework

–Data collection, management and analysis

–M&E outputs, formats and reporting

–Roles and responsibilities of partners and collaborators in implementing CAADP M&E framework

What is CAADP?

• The Comprehensive Africa Agriculture Development Programme (CAADP) is the African Union (AU)/New Partnership for Africa’s Development (NEPAD) vision and strategy for the development of African agriculture.

• CAADP puts agriculture at top of priorities of African countries

• CAADP is African-owned and African-led

CAADP Principles

• Employ agriculture-led growth to achieve MDG1 of halving poverty and hunger by 2015

• Pursuit of 6% average annual sector growth at national level

• Allocation of 10% of national budgets to agriculture sector

• Exploitation of regional complementarities and cooperation to boost growth

• Policy efficiency, dialogue, review and accountability (evidence-based policymaking)

• Partnerships and alliances to include farmers, agribusiness, civil society

CAADP Process• The principles are achieved through:

– the strategic functions of CAADP, – the guidance and involvement of the Regional Economic

Communities (RECs), and– the national roundtable process

• These activities surround four key pillars, led by Africa-based technical institutions: – Pillar 1: extending the area under sustainable land management

(University of Zambia)– Pillar 2: Improving rural infrastructure and trade-related capacities

for market access (Conference of Ministers of Agriculture of West and Central Africa (CMA/AOC))

– Pillar 3: Increasing food supply and reducing hunger (University of KwaZulu-Natal)

– Pillar 4: Agricultural research, technology dissemination and adoption (Forum for Agricultural Research in Africa (FARA))

Why an overarching M&E system for CAADP?• A mechanism by which processes put in place, commitments and

investments made, performance of agriculture sector, and anychanges in poverty, hunger and food and nutrition security areregularly and transparently measured against stated targets and, ifnecessary, can lead to the revision of processes, policies, investmentsand practices in order for CAADP to stay on track– Are countries achieving the targeted growth rates? (6%)– Are countries investing at the targeted level? (10%)– Are these investments having their intended impacts on poverty and hunger?

• To bring cohesion across the different systems being developed to track specific components of CAADP, including the individual CAADP pillar M&E systems, APRM and MAF

• To inform the review processes established by CAADP-PP (mutual, peer and progress reviews)

• To further inform policy-making and dialogue

Idea behind CAADP M&E framework

Greater/better distributedpoverty reduction & food and nutrition

security outcomes

Accelerated agricultural growth & Greater market access

More enabling policies &Greater/more efficient allocation of

agricultural investments

Global level

Commitments

Africa-wide level

Declarations

Decisions

Regional level

Early actions

8

7

6

1

2

3

5

P 4

P 3P 2

4

P 1

National level

Roundtable

Processes

Other factors

Other factors

Other factors

More effective

t

r

a

c

k

i

n

g

r

e

p

o

r

t

i

n

g

a

n

a

l

y

s

i

s

Key questions for CAADP M&E• Delivering on commitments and achieving stated targets

– Have commitments and targets been met so far?

• Effectiveness of interventions (processes, policies, investments)– How effective have different types of interventions been in any achievements

realized so far? What factors have shaped the achievements?

– What are the trade-offs and complementarities, if any, among different types of interventions?

• Consistency of planned interventions with initial targets– What are the projected impacts if interventions proceed as planned?

– Are the projected impacts compatible with the CAADP targets?

– If not, what adjustments are needed to get it on track?

• Exploring better interventions– Could greater or better distributed impacts be obtained by reconfiguring the

interventions?

– What are the different interventions that can lead to these outcomes?

Main challenge and achievements to date Main challenge

– Need indicators that are standardized, consistent and measurable across different countries and regions for cross-country comparisons and learning

• Main achievements– CAADP M&E working group established, beginning with a workshop at the AU

(Addis Ababa, Dec 3-4, 2007)

– Draft M&E framework developed and presented at CAADP PP meeting in Seychelles, March 2008

– Framework based on CAADP principles and economic theory to address issue of attribution/causality (i.e. inputs outputs outcomes impacts)

– Developed data collection formats with which ReSAKSS nodes are working with network of partners to collect data on indicators

– 5th CAADP PP in Abuja mandated that the draft M&E framework be validated, November 9-10, 2009

– M&E framework validated in collaboration with AU/NEPAD and other stakeholders in South Africa, March 1-3, 2010

– Revised version of the validated framework and implementation now ready

– First Comprehensive CAADP M&E report based on the framework now ready

Indicators to monitor and evaluate

• Input indicators: what is the overall level of effort invested?

– CAADP processes, policies, institutions, investments, etc.

• Output indicators: what is the level of provision, coverage, and utilization of services?– Access to infrastructure and services, adoption of technologies, etc.

• Outcome indicators: what is the effect on outcomes that affect goals?– Yields, production, wages, prices, trade, etc.

• Impact indicators: what is the ultimate effect on goals?

– Growth, income, poverty, food security, hunger, etc.

• Conditioning indicators: how confident are we that any observed changes is due to the intervention?– Total budgetary resources, climate, natural disasters, wars, etc.

Pillar 1

Land & water management

Pillar 4

Research &

Technology

Pillar 3

Food security

Pillar 2

Markets & trade

Programscaling up and

out, etc.

Key Indicators:

Credible and relevant evidence used in design of investment program

Inclusive participation of stakeholders in program design

Investment program aligned with CAADP principles and targets

Investment program technically reviewed

Mechanisms in place for implementation and M&E of the program

Input indicators: CAADP roundtable process

Program implementation

and M&E

Roundtable conference

held & compact signed

National compact

developed & discussed Stock taking

and gap analysis completed

Cabinet memo discussed &

approved

Steering & Technical

Committeesappointed

CAADP process

launched

National Focal Point appointed

Input indicators:enabling conditions (other processes, policies, institutions)

Key Indicators:

• Policies for private sector development (property rights, access to credit, contract enforcement, licensing, competition, …)

• Policies on equity(access of poor and vulnerable groups to resources, markets, food, and nutrition)

• Governance (political

stability, accountability, government effectiveness, regulatory quality, rule of law, control of corruption)

• Harmonized policies and strategies

• Commitments met

Private Sector

Donors

Others

National Policy Processes/Events

CAADP, SWAP, MTEF, Exp Reviews, Donor

harmonization, Elections, Law, etc.

Gov’t

Regional Level Africa-wide level

Actors

RECs, Reg. Orgs., etc.

Actors

AU, Int’l Orgs., etc.

Global level

Actors

G-8, G-20, WTO, etc.

Process

Summit, Reviews,

etc.

Process

Assembly, Summit,

etc.

Process

Conventions, etc.

Input indicators:investments and disaggregation

AGRICULTURAL & RELATED

INVESTMENTS

Space

Province, district, rural/urban, Agro-

ecology

Agriculture

Research, extension, irrigation, input support,

markets, ...

Sub-sector

Crops, livestock, fishery, forestry

Economic

Salaries, capital, operations and maintenance, …

Commodity

Staples, traditional, high value, export, ...

Source

Government, Donors, Private

Sector

Other

Gender, socio-economic groups

CAADP Pillar

1, 2, 3, 4

Sector

Agriculture, roads, education, health,

water & sanitation, ...

Output indicators:coverage and utilization of services

Investment /Intervention

Provision /Coverage (e.g.)

Utilization (e.g.) Disaggregation

Research Number of technologies dev’d

Area under technology

Commodity, gender, space

Extension Extension-farmer ratio

Number of visits received per year

Gender, space

Irrigation Capacity of irrigation (irrigable area)

Area under irrigation Commodity, gender, space

Farm support Quantity of support Area under input Commodity, gender, space

Feeder roads Length or density of roads

Space

Market Distance to nearest market

Share of output sold Commodity, gender, space

Post harvest Capacity of storage Capacity utilized Commodity, gender, space

…

Outcome indicators:agricultural sector performance

AGRICULTURAL SECTOR

PERFORMANCE

Use of factors (land, labor, capital) and inputs by:

sub-sector, commodity, gender, socio-economic group, Space

Growth returns to different types of investments by:

sub-sector, commodity, space

Sub-sector growth and contribution to

AgGDP by:

Space

Commodity growth, contribution to

AgGDP by:

Space

Productivity of factors (land, labor, capital)

and inputs by:

sub-sector, commodity, gender, socio-economic

group, space

Sector growth and contribution to overall GDP by:

Space

Production, trade and prices by:

sub-sector, commodity, space

Impact indicators

INCOME, POVERTY,

FOOD AND NUTRITION SECURITY, HUNGER

Returns to different types of investments by:

gender, socio-economic group, space

Decomposition by:

sector (agriculture, services, industry); sub-sector (crops, livestock,

fishery, forestry);commodity (staples,

high value, export, etc.)

Distribution by:

gender, socio-economic group,

space

Returns to commodity growth by:

gender, socio-economic group, Space

Unit costs by:

gender, socio-economic group,

space

Returns to sub-sector growth by:

gender, socio-economic group, Space

Operationalization of CAADP M&E Framework• Successful implementation of the CAADP M&E system ultimately

depends on the extent to which sufficient information on theindicators and outputs can be generated on a regular basis and in atimely fashion consisting of:

– Linked country level and regional teams, working under clearly defined roles and using shared data standards and protocols

– standardization and harmonization of the core set of data and indicators across countries that will enable cross-country comparisons and contribute to peer and mutual reviews of CAADP at regional and continental levels

– Regular collection, measurement, analysis, documentation and processing of data at national and regional levels

– Roles and responsibilities of different partners at the national, regional, continental and international levels in terms of data collection, management, analysis, and reporting

– Timely publication of these indicators and related monitoring reports

Process, policy or intervention area Indicator/Definition

Enabling environment

1. Political and economic governance 1a. % of population satisfied with political governance by: (i) gender; (ii) rural/urban; (iii) age

group; (iv) sector

1b. Macroeconomic management: (i) deficit to GDP; (ii),revenue to GDP; (iii) inflation rate; (iv) debt

to GDP

2. Policies for private sector development 2a. Percent of population with access to agricultural and rural finance and credit

2b. Value of commercial loans for agricultural sector as % of: (i) value of total loans; (ii) AgGDP

CAADP Country implementation process

3. Stage in county roundtable process and quality of participation 3a. Number of countries at major stages of the process

3b. Composition (e.g. institution, gender, expertise) of participants

Commitments and financing

4. Donor commitments and disbursements 4a. Total ODA commitments as % of AgGDP

4b. Share of ODA disbursed for (i) agricultural R&D; (ii) value chains; (iii) emergency food aid

5. Government spending and investment in Agricultural research and

development

5a. Expenditures on the agricultural sector as % of: (i) total government spending; (ii) AgGDP

5b. Expenditures on agricultural R&D as % of AgGDP

6. Private sector investments 6a. Total investment in agricultural sector as % of AgGDP

6b. Total investment in agricultural value chains as % of AgGDP

Agricultural sector performance

7. Capacity 7a. Number of professionals as per 1000 farmers

7b. Composition of professionals as % by: (i) gender; (ii) education attainment (PhD, MS, BS,

Diploma, etc.)

8. Agricultural growth and sources of growth 8a. Percent of area or output under improved technologies: (i) improved genetic material; (ii)

fertilizer; (iii) irrigation

8b. Productivity of major commodities (tone-equivalent per unit factor)

8c. Real AgGDP growth rate (%)

8d. % contribution to AgGDP growth of: (i) sub-sectors (crops, livestock, forestry, fishery); (ii) major

commodities

9. Agricultural trade performance 9a. Value of total agricultural exports by: (i) as % of AgGDP; (ii) share of value-added in total

exports; (iii) ratio to value of total agricultural imports; (iv) % contribution by sub-sectors and

major commodities

9b. Domestic and export-import parity prices by major commodities

CAADP goals

10. Poverty, hunger and food and nutrition security 10a. Poverty rate (P1) and gap (P2) by rural/urban

10b. Proportion of population below minimum dietary energy consumption (H1) by: (i) gender; (ii)

rural/urban; (iii) age

10c. Nutrition diversity by: (i) gender; (ii) rural/urban; (iii) age

CAADP M&E Minimum Core Set of Indicators

Data collection, management and analysis

Key question Tools Data

Delivering on commitments

• Trends• Simple correlations

• National surveys• Expert opinion surveys

Effectiveness of interventions

• Econometric methods• General equilibrium models

• National surveys• Targeted surveys• Expert opinion surveys

Consistency with initial targets

• Simulation models• Participatory approaches

• Assessment of effectiveness• Expert opinion surveys

Exploring better interventions

• Simulation models• Participatory approaches

• Assessment of effectiveness and consistency• Expert opinion surveys

Details in (www.resakss.org/publications/DiscussionP4&7.pdf)

Methodology:delivering on commitments

• Trends, situation, and simple correlation analyses to monitor progress (i.e. no attribution to interventions)– CAADP Roundtable process

– Donor commitments

– 10% government agricultural expenditure

– 6% agricultural GDP growth

– Halving poverty and hunger, etc.

• Data– National household surveys, accounts, etc.

– International public datasets (AfDB, World Bank, FAO, etc.)

– Targeted surveys (CAADP national focal points)

– Targeted expert opinion surveys

– …

Methodology:effectiveness of interventions (1)

• Main challenge is attribution

• Two complementary approaches– Before and after treatment

• Baselines (2003, compact signed, program implemented)

• Mid-term, end of project, long after project

– With and without treatment

• Treatment: direct beneficiaries; indirect beneficiaries through information/technology transfers, etc.

• Controls: can only be affected through general equilibrium effects, e.g. prices, wages, etc.

• For poverty (pov), impact of intervention (INV) measured by Average Treatment effect of the Treated (ATT):

ATTj = Ej [ povbefore, j – povafter, j | INVj =1] – Ei [povbefore, i – povafter, i | INVi =0]

Methodology:effectiveness of interventions (2)

• Underlying relationships to be estimated– Intervention decision making and placement

– Household access to and utilization of services due to intervention

– Household production, marketing and consumption decisions

• Techniques– Econometric methods (double-difference, instrumental variables,

matching) to assess direct impacts (mid-term, end of project, long after project)

– General equilibrium modeling to assess economy-wide impacts

• Data– National surveys

– Targeted household, market and other surveys to fill gaps

– Expert opinion surveys

– Case studies (selected countries and/or programs)

Methodology:consistency with initial targets

• Apply models developed for stocktaking and gap analysis

• What are the projected impacts if interventions proceed as planned?

– Use mid-term estimated ex-post impacts and parameters and growth patterns to project ex-ante impacts over period when target is expected to be achieved

Are the projected impacts compatible with the CAADP targets?– Compare above ex-ante impacts with initial targets

If not, what adjustments are needed to get it on track?– Use data and information from experts to identify plausible scenarios

– Simulate impacts under different scenarios to reach initial targets

Methodology:exploring better interventions

• Could greater or better distributed impacts be obtained by reconfiguring the interventions?– Even if interventions are consistent with the initial targets, can

simulate impacts under different composition of investments to identify (in)efficiencies in implementation

– Use estimated ex-post impacts and parameters from cross-country reviews and information from experts to identify plausible scenarios

– Value of indicators associated with simulated impacts of the desirable scenarios can be used as guidelines to set new targets

What are the different interventions that can lead to these outcomes?– Composition of investments associated with desirable scenarios from

preceding analysis

M&E outputs, formats and reporting • The primary output of the CAADP M&E system will be produced in the

form of a report, which will be produced at the national, regional andAfrica-wide levels on at least an annual basis and deal with addressing theissues of enabling environment and progress with delivery ofcommitments and achieving stated growth and poverty-reduction targets.

• Widespread dissemination of the CAADP M&E outputs, as well as thedata, tools, methods, and knowledge on agricultural and ruraldevelopment in general, will be done through the ReSAKSS website(www.resakss.org)

• Reporting mechanism to be institutionalized and implemented on regularbasis (semi-annual, annual or bi-annual depending on indicator)

• Various other media and presentations to review M&E information and results of analyses:

– CAADP PP (Africa wide)

– CAADP advisory councils (REC level)

– ReSAKSS steering committee meetings (REC level)

– Other regional- and country-level policy dialogues (Roundtables, etc)

http://www.resakss.org/





CAADP M&E: ReSAKSS Websitewww.resakss.org


Roles and responsibilities of partners and collaborators in implementing CAADP M&E Framework

Thank you.

Monitoring and Evaluation (M&E) System for the Comprehensive Africa Agriculture Development...

Technology

Transcript of Monitoring and Evaluation (M&E) System for the Comprehensive Africa Agriculture Development...