Using non-experimental methods to evaluate a

Conditional Cash Transfer program:

The case of Bolsa Alimentação

• International Food Policy Research Institute– Pedro Olinto

• Bolsa Alimentação (Ministry of Health, Brazil)– Eduardo Nilson

• London School of Hygiene & Tropical Medicine– Saul Morris

• Emory University– Rafael Flores

• Instituto Materno-Infantil de Pernambuco– Ana Claudia Figueró

• Instituto de Estudos do Trabalho e Sociedade– Alinne Veiga

The program• Bolsa Alimentação is a Brazilian federal

government program designed to reduce nutritional deficiencies and infant mortality

• Beneficiaries– (i) pregnant women,– (ii) mothers breastfeeding a child up to 6 mo – (iii) children of age six months up to seven years

• Beneficiary selection:– Municipal quota based on projected number

malnourished children– Municipal authorities select beneficiary households

• R$15 (US$6.25) /beneficiary /month (up to R$45)

• Mother signs “Charter of responsibilities”

Evaluation design: main features

• Non-experimental• Ex-post only comparison of beneficiary

and non-beneficiary households• Non-beneficiary households were

registered by local authorities as beneficiaries, but excluded due to:– Data transmission error– “Special characters” in the name– Discrepancy family records Bolsa Escola

Evaluation design: main features

• Evaluation limited to municipalities– In the Northeast region of the country– With at least 40 excluded households– Where transfers had been made for 6

months

• Fieldwork undertaken exactly 6 mo after first transfer in each of the 4 survey municipalities

Pairing of beneficiaries and excluded

• Pairing separate for each class of beneficiary– Pregnant women– “Lactating mothers” (up to 6 mo post-partum)– Children

• For each “would-be beneficiary”, the most similar similar actual beneficiary was identified, based on– Municipality of residence*– Sex*– Socio-economic status (based on reported income;

rent; water, electricity & gas bills, family size)– Age

Pairing (2)

• Pairing algorithm: “nearest neighbor” with calipers– Pairwise distance calculated based on Euclidean

distances– Relatively importance of age and SES chosen

subjectively– No replacement of beneficiaries– “Excludeds” matched in a random order– Two beneficiaries per “excluded”, to maximize

powerExcluded Benef.1 Benef.2 Total

Pregnant 25 25 25 75Lactating 18 18 18 54Child 463 463 457 1383

Total 506 506 500 1512

Problems with the pairing

• Turned out that program registers had not recorded all the young children in the hhs– Expected no. under-7s: 1.3 per hh– Revealed no. under-7s: 1.9 per hh

• Individual-level pairing not so good for household-level analyses, since same beneficiary hh paired to more than one excluded hh

  Bolsa Alimentação beneficiary households

 Excluded

households

 P-value

Educational level of women aged 15-49:

Incomplete primary

 79.0%

(646/818)

 79.3%

(252/318)

 0.92

Flooring material:EarthFloor tilesOther (mostly cement)

 8.0% (57/716)10.8% (77/716)

81.3% (582/716)

 9.6% (27/282)

11.0% (31/282)79.4%

(224/282)

   

0.70Water source: public network 50.5%

(360/713)50.4%

(142/282)0.97

Telephone 9.2% (66/717) 11.0% (31/282) 0.39

Family size, mean (s.d.):Total

0.0   – 6.9 y7.0 – 13.9 y

 5.4 (2.1)1.9 (0.9)0.8 (1.0)

 5.8 (2.3)1.8 (0.9)1.1 (1.1)

 0.0030.02

<0.001Bolsa Escola beneficiary 26.8%

(192/717)55.7%

(157/282)<0.001

Duration of exposure to Bolsa Alimentação at the time of the survey, months, mean (s.d.):

 5.9 (0.3)

 N/A

 N/A

Socio-economic and demographic characteristics of Bolsa Alimentação beneficiary and excluded

households in the evaluation sample

Marginal propensity to consume food

out of Bolsa Alimentação transfers

• OLS model:– R$0.70 for every R$1.00 transferred

• IV model (controlling for possible endogeneity of number of beneficiaries):– R$0.57 for every R$1.00 transferred

Impact on dietary diversity

Dependent variables:

-1 -2 -3 -4

Dummy = 1 if beneficiary of BA 1.901 1.879 0.089 0.088[0.585]**[0.587]**[0.024]**[0.024]**

Dummy = 1 if beneficiary of BE -0.4 -0.9 0.001 -0.03[0.692][0.549] [0.029] [0.022]

Log of household population -0.25 -0.013[0.893] [0.037]

Share of members aging 0 to less than 7 -4.21 -0.198[2.466] [0.099]*

Share of members aging 7 to less than 15 -3.68 -0.217[2.733] [0.112]

Share of members aging 15 to less than 19 -6.19 -0.295[3.206] [0.138]*

Share of members aging 61 and up 3.518 0.152[3.730] [0.137]

Constant 29.5 27 3.357 3.232[1.678]**[0.556]**[0.068]**[0.024]**

Observations 1005 1005 1005 1005Robust standard errors in brackets* significant at 5%; ** significant at 1%

# of food items ln(# food items)

  Weight-for-age Z-score

             

Age group:   Bolsa Alimentacao   Difference

    Benef’y Excl’ded   Unadjusted Adjusted

             

0.0 - 23.9 momean -0.68 -0.40 mean -0.28 -0.25

  s.d. 1.11 1.15 s.e. 0.13 0.13

  n 315 103 signif P=0.030 P=0.058

           

24.0 - 47.9 momean -0.75 -0.65 mean -0.10 -0.11

  s.d. 1.06 1.02 s.e. 0.10 0.10

  n 430 141 signif P=0.33 P=0.29

           

48.0 - 83.9 momean -0.77 -0.74 mean -0.03 -0.08

  s.d. 1.04 0.90 s.e. 0.08 0.08

  n 596 238 signif P=0.68 P=0.31

             

Totalmean -0.74 -0.64 mean -0.10 -0.13

  s.d. 1.06 1.00 s.e. 0.06 0.06

  n 1,341 482 signif P=0.068 P=0.024

Longitudinal analysisFractional Polynomial (0 1)

Co

mp

on

en

t+re

sid

ua

l fo

r w

eig

ht

age0 12 24 36 48 60

2

20.6

Associations between length of exposure to Bolsa Alimentação and weight gain in children aged 0–36 months. Panel analysis based on routinely recorded weight data:

random effects model.------------------------------------------------------------------------------ WEIGHT | Coef. Std. Err. z P>|z| [95% Conf. Interval]-------------+---------------------------------------------------------------- AGE | .0835284 .0050511 16.54 0.000 .0736286 .0934283 ln(AGE) | 1.941647 .0445841 43.55 0.000 1.854264 2.029031 MALE | .5057401 .0886546 5.70 0.000 .3319804 .6794998 BA_BASELINE | -.1923509 .1034153 -1.86 0.063 -.3950412 .0103395 BA_DURATION | -.0305449 .0074744 -4.09 0.000 -.0451945 -.0158953 BE | -.0201923 .0475438 -0.42 0.671 -.1133765 .0729919 _cons | 3.187013 .1141902 27.91 0.000 2.963205 3.410822-------------+---------------------------------------------------------------- sigma_u | 1.0507184 sigma_e | .61632495 rho | .74400885 (fraction of variance due to u_i)------------------------------------------------------------------------------

Differential weight gain associated with each month of exposure to Bolsa

Alimentação, estimated for children of ages 6, 12, 18, 24, 30, and 36 mo, in a

random effects model that allows program effect to interact both with age

and the natural log of age------------------------------------------------------------------------------ weight | Coef. Std. Err. z P>|z| [95% Conf. Interval]-------------+---------------------------------------------------------------- 6 months | -.0324806 .0137262 -2.37 0.018 -.0593834 -.0055778------------------------------------------------------------------------------ 12 months | -.0457275 .0095092 -4.81 0.000 -.0643653 -.0270897------------------------------------------------------------------------------ 18 months | -.0402928 .0084215 -4.78 0.000 -.0567987 -.0237869------------------------------------------------------------------------------ 24 months | -.0272094 .0088983 -3.06 0.002 -.0446498 -.009769------------------------------------------------------------------------------ 30 months | -.009935 .0122145 -0.81 0.416 -.0338749 .0140049------------------------------------------------------------------------------ 36 months | .0099903 .0176865 0.56 0.572 -.0246747 .0446553------------------------------------------------------------------------------

Lessons learnt: methodology

• Very detailed understanding of beneficiary selection process resulted in robust non-experimental identification

• Success critically dependent on getting into the field at the right time

• Individual matching ensured that beneficiary and non-beneficiary groups were very similar

• Routinely collected weight data permitted longitudinal analysis, quality much better than expected

• However, lack of conventional baseline allows controversy over results to rage indefinitely!

Lessons learnt: results

• Strikingly high proportion of transfer spent on food, suggesting good targeting

• Diversity of family diet improved markedly, as in similar programs

• Vaccination coverage not improved because already excellent

• Weight gain effect probably illustrates dangers of linking targeting and/or program graduation to nutritional status outcomes

• For nutrition, too many beneficiaries over 2 yo, damage likely to be irreversible