Enzo Caputo

Independent Consultant

Brussels 8-9 July 2014

Quantitative and qualitative tools for the implementation of the 3 STEP approach

2

Focus of the presentation

• Mixing two evaluation methods in 3 STEP approach

• Fixing the Intervention Logic (global and sectoral)

• Identifying the EQs

• Analytical tools

• Contribution analysis

• Policy impact analysis (causality analysis of the targeted outcomes)

• Combination of STEP 1 and 2 in STEP 3

3

Mixing two evaluation methods in 3 STEP approach

4

The global Evaluation Framework

Level 2-Direct Output

Direct effects of the Aid/BS

inputs

Level 1-Inputs

Aid/BS, Other specific inputs

Level 3-Induced Outputs

Gov. & Civil Society

strategy/ action

Level 4-Outcomes

Short-medium term develop.

results

Level 5-Impact

Longer term development

resultsContext

Context

5 levels:• Specific importance is given to the context• Two evaluation processes ( & ) distinguished &combined• With a shared level: the “induced outputs”

The 3 STEP approach does not propose a new evaluation method, but the combination of two existing methods. Each can be applied widely or narrowly, brilliantly or imperfectly, with abundant resources or limited means. Such differences in quality do not depend on the 3 STEP approach.

5

EVALUATION PROCESS

Figure 2 - GBS/ SBS Comprehensive Evaluation Framework: the Three-step approach

GOVERNMENT POLICY & SPENDING ACTIONS (STRATEGY)

Inputs to Government Policy & Spending Actions

1.1. GBS / SBS inputs

2.1. Direct Outputs. Improvement in the relationship between external assistance and the national budget & policy process

3. Induced Outputs. Positive changes in the financing and institutional framework for public spending and public policy, and in public policy management and service delivery

4. Outcomes. Positive responses by beneficiaries – service users and economic actors – to Government policy management and service delivery.

5. Impacts. Sustainable Growth & Poverty Reduction

Transfer of Funds to Budget

Policy dialogue and

related conditionality TA/capacity building GBS/ SBS aligned to

government policies and systems and harmonised across donors

Increased size and share of external assistance funds made available through the national budget

Increased size and share of budget available for discretionary spending

Increased predictability of external funds

Policy dialogue and conditionalities, coordinated, consistent with and conducive for government strategy

TA/capacity building coordinated, consistent with and conducive for government strategy

External assistance as a whole more harmonised & aligned

1.2. Various Government inputs

2.2. Other effects by various Government inputs

xxxx Domestic Revenue Funding and Domestic Policy Inputs

1.3. Other external assistance programmes

2.3. Other effects by other external assistance

- xxxx

xxxx

Improved fiscal discipline and macroeconomic management.

Strengthened PFM and

procurement systems. Improved public policies’

design and public policy processes.

Increased funding for

discretionary spending resulting in increased quantity and quality of goods and services provided by the public sector.

Enhanced allocative and

operational efficiency of public expenditure.

Improved budget process,

including better links between government and parliament

Increased use and appreciation by the beneficiaries of the goods and services provided by the public sector.

Positive response by the

general economy to the improvements in government initiatives targeted by BS arrangements.

Increased business

confidence deriving from more effective macroeconomic and regulatory policies.

Improved general

confidence of people and enhanced democratic accountability, particularly over the budget process.

Enhanced & sustainable economic growth

Reductions in

income & non-income poverty

Empowerment &

social inclusion of poor people and disadvantaged groups (including women)

Other areas,

according to specific partnership frameworks and priorities (e.g. improvements in democracy, human rights, environment protection, ….)

Various features of the “entry conditions” Overall aid framework Existing learning processes and tools

Government capacity to implement reforms; Extent of political commitment to reform processes

Capacity of public sector Nature of demand for Govt services Strength of domestic accountability

Global economic development Asset endowments Responses to changing incentives

EX

TE

RN

AL

FA

CT

OR

S, C

ON

TE

XT

FE

AT

UR

ES

AN

D F

EE

D B

AC

K P

RO

CE

SSE

S

6

Fixing the Intervention Logic (global and sectoral)

POTENTIAL FOR SUSTAINABLE AND

INCLUSIVE GROWTH & POVERTY REDUCTION

TOWARD

POSITIVE RESPONSES BY BENEFICIARIES – service

users and economic actors – to government policy management and

service delivery

INPUTS of BSencompassing:• Education,• Health,• Water,• Private sector / Employment creation,• Urban renewal, • Science &Technology ,• Legislative,• Justice.

OTHER EFFECTS BY VARIOUS OTHER GOVERNMENT INPUTS

Inputs Induced OutputsDirect Outputs Outcomes Impact

IMPROVED PUBLIC POLICIES, PUBLIC SECTOR INSTITUTIONS & PUBLIC

SPENDING PROCESS

OTHER EFFECTS BY OTHER EXTERNAL ASSISTANCE

OVERALL & SECTOR SPECIFIC IMPROVEMENTS IN THE RELATIONSHIP

BETWEEN EXTERNAL ASSISTANCE AND THE NATIONAL BUDGET AND POLICY

PROCESSES

Main government programmes and other

SPECIFIC INPUTS

INPUTS of other external assistance programmes

(EU / EU MS, and others), e.g.: PSPPD, TDCA

dialogue facility, Other donors (health, education…).

SA-E

U S

TRAT

EGIC

PAR

TNER

HSI

P &

O

THER

EXT

ERN

AL F

ACTO

RS, C

ON

TEXT

FEA

TURE

S AN

D F

EED

BACK

PRO

CESS

ES

Inputs to government policies and spending actions Government policies and spending actions (strategy)

• Various features of the “entry conditions”,• Overall aid framework,• Existing learning processes and tools,• Government capacity to implement reforms,

• Extent of political commitment to reform processes,• Capacity of public sector,• Nature of demand for public services,• Strength of domestic accountability,

• Global economic development,• Foreign capital inflows,• Responses to changing incentives ,• …

IMPROVED PUBLIC SERVICE DELIVERY IN THE INTERVENTION SUB-SECTORS

Inputs Induced OutputsDirect Outputs

Transfer of funds to the National Treasury (just shy of 1 bln € or

75% of EU funds)

Policy dialogue and performance

indicators

Capacity development inputs incl. technical

assistance

Policy dialogue architecture (including TDCA, Strategic partnership, sector

policy dialogue related to SBS, negotiation of conditions & KPIs) allows interaction and synergies between high level political dialogue & policy support.

Increased size and share of external assistance funds made available through

the national budget.

Increased size and share of budget available for discretionary spending, to

facilitate innovation and institutional development.

SBS strengthens the alignment with GoSA systems, helps reducing

transaction costs and is conducive to harmonisation with other donors.

Capacity Development, TA and complementary actions are connected

to budget support according to the actual needs and feed the dialogue.

Domestic revenue funding and domestic policy inputs…

PFM and procurement systems, incl. decentralised level, are strengthened (namely with respect to flexibility for

policy innovation, allocative and operational efficiency).

Improved public policy formulation and execution processes, incl. mainstreaming of innovative initiatives and pilots, evidence-based policy making, M&E and accountability.

Public institutions involved, namely (though not only) at decentralised level, are enabled to plan and implement the

relevant policies.

Enhanced interaction between GoSA, the CSOs and the Private Sector in the

policy processes.

Mainstream

ing of lessons from innovation

Social sectors (Water, Education, Health)• Enhanced and sustainable provision of public services at both central and decentralised levels,• Increased responsiveness of services to the needs of the population,• Piloting of different models of service delivery.

Private sector / Employment:ECF, RCF, Innovation & Poverty alleviation.

Governance: Access to Justice; Legislature.Enhanced equitable access and use of public goods in the

Water sector.

Private sector / employment:

• Increased employment (esp.

among HDP),• Increased

business confidence, and

• Increased private sector investment and production.

Governance: • Improved access to justice by the poor/marginalised groups,• Enhanced oversight and orientation capacity of the Parliament,•Enhanced participatory democracy.

Enhanced equitable access and use of public goods in the Education & Health

sectors.

Enhanced sustainable and inclusive economic growth.

Reductions in income poverty & non-income poverty.

Empowerment & social inclusion of poor people & historically disadvantaged people (incl. women).

Consolidated governance and democracy.

Development results

8

Identifying the EQs

EQ 2:Policy Dialogue,

TA and H&A

EQ 1:Relevance /

design

EQ 4:Public Spending /

Budget management

EQ 6-7-8-9:Achievement

and determinants

of the outcomes in the focal and other sectors

EQ 10:Sustainability

of theachievements

EQ 3: Financial inputs

Other factors

EQ 5:Policy formulation & implementation

processes

Other factors

10

Typical EQs for Step 11. Level 1: relevance and coherence of the design to the

context, government priorities, EU strategic framework

2. Level 2: direct effects on the aid framework and aid-based opportunities

a. Non-financial inputs: Policy dialogue structure and modalities, Technical Assistance (type and modalities), Harmonisation and Alignment (PAF, government ownership, coordination, reduction of transaction costs).

b. Financial inputs: discretionary funds, increased gov. control on aid

3. Level 3: induced effects on PFM and policy outputs, incl.:a. PFM (resource balance, strategic allocations, transparency)

b. Policy outputsi. Process (strategic dev., decentralisation, CSOs participation, …)

ii. Service delivery (economic and social services)

11

Typical EQs for Step 24. Levels 4 and 5: Outcomes and Impacts and determining

factors:

4. Focal sector Aa. Quantitative and qualitative assessment of the development

outcomes targeted by the BS programmes (evolution over the period of the evaluation)

b. Causality analysis of the determining factors of such outcomes, including both context- and policy-related factors.

5. Focal Sector Ba. As above

b. As above

6. Focal Sector C

………………………

12

Contribution Analysis

13

EVIDENCE: Fact finding - Correlation and Validation of the Contribution, in Step 1

Fact Finding: Documentary and field research :• Opportunity framework: political economy and national dynamics,

international context, strategic partnerships and motivation towards international partnerships.

• Identify facts related to changes in the targeted Gov. policies and service delivery (strategies and implementation of programmes...)

• Identify facts related to BS action (through funds, dialogue, capacity building components)

Establishing correlations and validating: BS Induced outputs:• Qualitative and quantitative correlations between BS implementation

and Government Policy & Delivery change• Validation of the correlations through qualitative analyses and

COUNTERFACTUAL

14

How to find what for Step 1: examples

Discretionary expenditure:• Additional resources available for new allocations (neither recurrent, nor development

expenditure from previous years) named in various ways (additional, fiscal space..)

Strategic budgeting• Trends in budget allocations and variation in the strategic sectors.

Strategies, laws, institutional building• New strategies and plans...

Improvements in service delivery• Data and trends on public works, construction of schools, number of teachers...

Policy dialogue:• Policy Dialogue in the BS process (identification, design, M&E, disbursement):

acquiring documents and meeting minutes, assessing consistency, alignment of PAF and quality of indicators.

• Policy Dialogue in the Gov. led structures. Records, stakeholders perceptions.• Informal Policy Dialogue: records and memory: Tunisia on privatisation; SA on water.

Capacity Development support (Gov. ownership, coordination, complementarity…)

15

Evidence of BS contribution: examples (1/2)

• Funds have increased government control of aid and discretionary resources in the budget (level 2).

• Funds have contributed to resource balance and have facilitated macro-economic stabilisation policies (level 3).

• Funds and dialogue have facilitated matching expenditure and policy priorities (level 3).

• Dialogue has built on a solid framework (level 2) and has strengthened the decision capacity and/or the knowledge base of the policy process (level 3).

• TA was provided through coordinated and owned mechanisms (level 2) and helped identify solutions for policy development (level 3).

16

• PAF were not really aligned with government key indicators, as the DPs emphasised their own priorities (levels 2 & 3).

• Some indicators in PAF helped the government achieve its own targets thanks to the establishment of a political and economic partnership and/or an external discipline and/or supplementary technical support (level 3).

Confirm the evidence by counterfactual arguments:• would the identified effects (especially induced outputs) have

been achieved (at a comparable or better level) without BS?• or with the support of a project outside the scope of BS?

Counterfactuals in Step 1 are based on hypothetical alternative scenarios, to be built case by case with the help of informed persons, or using the experience of other non-BS programmes.

Evidence of BS contribution: examples (2/2)

17

Policy Impact (causality) Analysis

18

Outcome assessmentDocumentary research (Policy and Research documents and statistics):

• A clear and exhaustive framework of the quantitative and qualitative data related to the development outcomes targeted by BS at macro and sectoral level.

• Important: not a global assessment of the government policies, but a targeted data collection.

Critical assessment of the data on Outcomes (interviews and cross-checking):

• Check data (sources, comparability).• Understand the data.

19

Causality analyses for Step 2 (types)

Use quantitative methods and / or qualitative assessments to identify the determinants (including both context- and policy-related factors) of the development outcomes:

• Statistical regression of some outcome indicators (dependent variables) against some context- and policy-related factors (independent variables).Recommended when there are clear quantified outcomes and policy outputs with time series covering the period considered.

At sectoral level (e.g. education), and/or at macro level to measure the impact of different sector policies on global outcomes (e.g. on HD index, GNI p.c.).Among the independent variable, put both supported and non-supported policies and other ‘context’ factors: e.g. demography).

• Double difference (to measure different evolution of “treated" areas compared to similar "untreated" areas.Recommended when policies supported are implemented in selected areas and similar areas are excluded.

May be more or less formalised (with stronger of weaker selection biases).

20

Causality analyses for Step 2 (examples)

Education:

DEP. VAR = Exam Pass Rates

IND. VAR = Classes, N. Teachers, Books, Desks etc. Strong correlation between Pass Rates and Books+Desks. The qualitative work shows that there is a causality link and how.

Economic Reform:

DEP. VAR = Human Dev. Index and GDP p.c. growth

IND. VAR = Fix Capital (inv.); FDI; Quasi-money (Fin. Market); Export; Savings; Eco Reform Index (Chinn). Strong correlation of FDI, QM, EXP and Eco Ref Index, confirmed by other analyses (Panel and Cluster) and literature.

Regression show correlations between DEP. and IND. VAR, but not causalities. Ex. correlation between potatoes’ consumption and GDP level is strong (at least in the ‘80s) but there was not a causality relation. Qualitative work to ascertain the nature of the correlation and the ways it works.

21

Causality analyses for Step 2 (other types)

• Other types of cross-country statistical analyses (cluster, panel data).For instance in a geopolitical region, to identify the specific development patterns and historical drivers of development.

• Quantitative checks and qualitative assessments always necessary to integrate and understand the statistical correlations.

• Political-economy analyses in a historical perspective, and literature reviews, using existing studies and resource persons.To assess broad and complex policy issues, such as the key factors of employment and unemployment at country level during a period, or some historical development drivers.

There is a limited time: either fulfil quick statistical assessments or use existing evaluations, policy studies and literature, while keeping a high qualitative rigorous approach.

22

Combination of STEP 1 and 2 results in STEP 3Theoretical possibilities

23

Analysis of the Transitive Relations between BS and development outcomes in STEP 3

Theoretically possible STEP 3 conclusions

X = Impact of the policies supported through BS (WEAK-MODERATE-STRONG)

Y =

BS

Cont

ribu

tion

to

poli

cies

and

pu

blic

ex

pen

ditu

re (

WEA

K-M

OD

ERAT

E-ST

RO

NG

)

A

B

C

AS Contribution of BSw Impact of Policies

BM Contribution of BSM Impact of Policies

CW Contribution of BSS Impact of Policies

DS Contribution of BSM Impact of Policies

EM Contribution of BSS Impact of Policies

FS Contribution of BSS Impact of Policies

………………………………

D F

E

HG

I