Evaluating the Impacts That Impact Evaluations Don’t Evaluate

Book launch @ IDS

14 October 2015

Stephen Devereux & Keetie Roelen

Disaggregating causal complexity and social dynamics

Theory of change: Development interventions

Inputs Resources (goods or services) to be delivered to

programme beneficiaries – e.g. food, livestock, microcredit, school bursaries, health insurance

Process The modality by which resources are delivered –

e.g. cash transfer, fertiliser subsidy, fee-free education, school feeding scheme

Outcomes Direct consequences of the intervention – e.g.

giving food to hungry people is expected to have higher food consumption as one outcome

Impacts (short-term)

Attributable changes in wellbeing due to the intervention – such as higher asset ownership or reduced malnutrition rates

Impacts (long-term)

Sustainable and attributable improvements in wellbeing – such as reduced intergenerational transmission of poverty

Theory of change: school feeding schemes

Inputs Process Outcomes Impacts (short-term) Impacts

(long-term)

Food to under-nourished and under-educated children

School meals scheme

Higher food consumption by children

Reduced child malnutrition

Higher income and reduced poverty for children in adulthood

Higher school attendance rates

Better child performance at school

Interrogating the theory of change

Were undernourished and under-educated children reached by the school feeding scheme?

Are school meals additional to food consumed at home?

Did school attendance rates improve after free school meals were introduced?

Did children who received school meals perform better in examinations?

Do children who received free school meals earn higher incomes and escape poverty in adult life?

Social dynamics in development programmes

Level Social relationships

Intra-household

Female ↔ Male

Older generation ↔ Younger generation

Biological children ↔ Non-biological children

Beneficiary ↔ Recipient

Intra-community Beneficiaries ↔ Non-beneficiaries

Social dynamics in development programmes

Level Social relationships

Intra-household

Female ↔ Male

Older generation ↔ Younger generation

Biological children ↔ Non-biological children

Beneficiary ↔ Recipient

Intra-community Beneficiaries ↔ Non-beneficiaries

Programme actors Beneficiaries ↔ Programme staff

Economic actors Beneficiaries ↔ Traders

Political actors Beneficiaries ↔ Local politicians

Assessing intended and unintended impacts

Positive Neutral

Positive

‘Double success’ [+/+] Intervention achieved its objectives and also had unplanned, beneficial social consequences

Success [+/=] Intervention achieved its material objectives and had no discernible social consequences

Neutral

‘Qualified success’ [=/+] Intervention did not achieve its objectives, but recorded improved social indicators

No impact [=/=] Intervention effectively never happened; it had no discernible intended or unintended impacts

Assessing intended and unintended impacts

Intended (material) impacts

Unintended (social) impacts Positive Neutral Negative

Positive

‘Double success’ [+/+] Intervention achieved its objectives and also had unplanned, beneficial social consequences

Success [+/=] Intervention achieved its material objectives and had no discernible social consequences

‘Mixed result’ [+/–] Intervention achieved its material objectives, but had unplanned negative social consequences

Neutral

‘Qualified success’ [=/+] Intervention did not achieve its objectives, but recorded improved social indicators

No impact [=/=] Intervention effectively never happened; it had no discernible intended or unintended impacts

‘Qualified failure’ [=/–] Intervention did not achieve its objectives and left beneficiaries worse off on social indicators

Negative

‘Failure plus’ [–/+] Intervention left its beneficiaries worse off in material terms but better off on social indicators

Failure [–/=] Intervention left its beneficiaries worse off in terms of its intended material indicators

‘Double failure’ [–/–] Intervention left its beneficiaries worse off in both material and social indicators

Assessing intended and unintended impacts

Intended (material) impacts

Unintended (social) impacts Positive Neutral Negative

Positive

‘Double success’ [+/+] Intervention achieved its objectives and also had unplanned, beneficial social consequences

Success [+/=] Intervention achieved its material objectives and had no discernible social consequences

‘Mixed result’ [+/–] Intervention achieved its material objectives, but had unplanned negative social consequences

Neutral

‘Qualified success’ [=/+] Intervention did not achieve its objectives, but recorded improved social indicators

No impact [=/=] Intervention effectively never happened; it had no discernible intended or unintended impacts

‘Qualified failure’ [=/–] Intervention did not achieve its objectives and left beneficiaries worse off on social indicators

Negative

‘Failure plus’ [–/+] Intervention left its beneficiaries worse off in material terms but better off on social indicators

Failure [–/=] Intervention left its beneficiaries worse off in terms of its intended material indicators

‘Double failure’ [–/–] Intervention left its beneficiaries worse off in both material and social indicators