Impact Evaluation Case Studies in climate change adaptation & mitigation issues and evidence concerning Brazil (i.e., compared to what? over and over) Alexander Pfaff Duke University prepared for The World Banks DIME-AADAPT workshop Brasilia -- November18, 2009 Evaluations & Climate Policy NE Brazil & climate adaptation: - provide forecasts w/in rainfed agric. - design water allocation institutions Amazonia & climate mitigation: - target protected areas for REDD $ ? - locate new transport infrastructure? Evaluations Differ By Policy Area NE Brazil & climate adaptation: - find and construct good data - no data pre-pilot? make it up! Amazonia & climate mitigation: - find wrong data but ask right Q - policy guidance requires a new Q ? Measuring Development Effectiveness in Environment: progress & constraints Alex Pfaff, Duke University, April 10 th 2008 Is There A State Of The Art? Yes. In fact, two different ones: -randomizing: challenge is to do it -if cant: best use of existing data? Question: compare project to..? Need to construct the best control. > problem: non-random participation; dont compare to average outcome > challenge: issues vary by setting - which choices make non-random? - can have data on what variables? Ex: stoves and climate mitigation Q: health benefits enter the CBA? - nonlinear health gains -> go bigger - constrained by capital (and beliefs) Q: micro loans so invest in bigger? - NGO could sell the carbon credits - but coal stoves improve health !? Indoor Air Quality stoves policy who uses old? who uses new? randomizing yet choose location? Q: will I ever find good data? A: yes, if good by design (intent?) - you are considering randomization - uncorrelated motivations can work A: in being cruel, Nature kind to us - as long as rain varies (time/space) - As naturally occurs in groundwater Groundwater Arsenic Information study choices test wells uncorrelated choose info? NE Brazil & Adaptation (rainfed) Q: most vulnerable raise resilience? - adapted to ongoing shocks regime - ENSO forecasts can help in principle Use & construct random variation: - learn how education helps as f(rain) - randomize if/how provide forecasts Used Newspaper Article After Forecast Release (forecast (vs. model) reflected user concerns!!) Providing seasonal rainfall forecasts to rainfed farmers Information Treatments: we give no information (control group, track all other forecast sources) reading of a new article from recent newspaper reading along with a form of an interaction about the information (pilot involved simple discussion with staff) Behavioral Survey: before & after (in Novembers) Drought, Vulnerability & Responses: basic characteristics, including movement & knowledge of climate using rain as instrument (gauges back in time though fairly sparse) migration & remittances (changes the composite income risk face) focus on income-generating choices in agriculture, parcel by parcel as is Roberta, e.g., consider non-climate factors driving decisions plans looking forward for the season, including repeated plantings Why November? trying to take out the noise in a cross-sectional comparison of groups in November households are starting land preparation & have a plan thus we compare households post-forecast choices with OWN plan NE Brazil & Adaptation (irrigated) Q: what institutions allocate water? - Cear has a new Integration Canal - Cear has participatory committees Q: how learn effects of institutions? - with asymmetric technical access? - using trust instead of rights/sales? Adaptation Water Institutions equitable efficient shifts? simply too little data w/real stakeholders try decision experiments Examining Bargaining With Asymmetric Information: - decision experiments done in Ceara in Fortaleza & Valley - find typical bargaining results when have same information - asymmetries clearly exploited, even using a sharing norm Examining Contracts vs. Trust In Sharing Of Surplus: - trust in sharing is one current regime for water reallocation - we find more trust than neoclassical but contracts are better These experiments could not be the basis for policies. They do offer ideas and processes for doing pilots Study Fake Institutions -> Real Field Pilots? would be forest anyway? find similar parcels show where impact high REDD policy protection? pay? can rule out land use Costa Rica deserves global gratitude as THE pioneer: - simply getting the idea into the heads of others is critical - demonstrating that payments (vs PAs) are feasible is too - always expected learning & demonstrates shifted design But lessons learned from Costa Rica are misleading: - as the protected areas all forested, all had a large impact - had deforestation without payments, then none with, so Protection & Eco-payments CAN have large impacts, yet that does NOT mean ANY policy has impact Costa Rica Takeaway: policy design matters !! CR Payments Summary & Policy Implications 00-05 about 2% of enrolled land would have been deforested; this (low) impact is over twice as high as the results for 97-00: despite net reforestation there was actually more gross deforestation targeting of benefits removed a bias against targeting threats/impact further, within that average, different implementation affected impact if such shifts can improve / affect forest impacts of payments, surely intentional focus on indicators of threat can do so too! yet targeting of threat could lead payments away from the poor & generally earning REDD $ will be blended with other goals bundled policy package story implies impact without targets; the KEY point is for the world to understand the lessons right! CR Parks Summary (impacts within borders): PSM vs. CM (in constructing controls, similarity metrics differ) Park Effects on % Deforestation (4 matches to each park pixel) (Andam et al. PNAS paper examines 1963 forward with same story but average estimated impact is higher as pressure higher earlier) 3.5% Treated Observations with the same match: -1.99* -9.38**Using All of the Untreated (Naive) -1.37** -0.05Propensity Score Matching (PSM) **Covariate Matching (CM) 33.4% Similarity between Control Groups Adjust Diff. in Means Differences in Means Compare Estimation Strategies Amazonia & Mitigation / REDD $ PAs are not randomly distributed: - no reason to expect they would be - lots of reasons for location (land P) Thus wrong data for evaluation: - yet skeptical C market asks impact - that focus can yield right answer International Scientific Conference. Amazon in Perspective - Integrated Science for a Sustainable Future The Chico Mendes Extractive Reserve International Scientific Conference. Amazon in Perspective - Integrated Science for a Sustainable Future 2007 Evidence of External Pressures on Chico Mendes Extractive Reserve Chico Mendes Extractive Reserve International Scientific Conference. Amazon in Perspective - Integrated Science for a Sustainable Future Matching Applied To InterOceanic Highway Region Protected Areas (Chico Mendes Extractive Reserve & Acre / InterO Protected Areas) (impacts on 1989 2000 deforestation & 2000 2007 deforestation)METHOD 1989 2000 Deforestation 2000 2007 DeforestationMendes Rest of PAs Mendes Average Rates (compare means) 7%8% 10% Regression (full data set) 7%1% (insig)8%1.5% Matching (compare means) 7%0.2% (insig)7%0.2% (insig) International Scientific Conference. Amazon in Perspective - Integrated Science for a Sustainable Future Matching applied to Brazilian Amazon Protected Areas (Federal, State, Indigenous & deforestation)METHODFCU(federal)SCU(state)IL(indigenous) Average Rates (compare means)4.9%4.0%5.6% Regression (full data set)1.9%0.4%2.2% Matching (compare means)2.2%0.4%4.6% Matching (regression)2.4%0.2%3.1% Amazonia & Mitigation / REDD $ Roads not randomly distributed: - Federal paved linking urban points - State unpaved respond to demand Here maybe asking the wrong Q? - development benefits > avg. loss? - even if so, CBA differs by location! Impacts Targeting Variable: distance to San Jose over 85km, deforestation reduction is essentially zero under 85km, deforestation reduction greater than average Impacts Targeting Variable: distance to national roads over 7.53km, deforestation reduction is insignificant under 7.53km, deforestation reduction > 2 * average Impacts Targeting Variable: slope over 7.12 degrees, deforestation reduction is insignificant under 7.12 degrees, deforestation reduction > 5 * average Back To CR Parks: guiding next investments? Figure 5 (matching on all covariates) measured road-investment impacts, exact matching, explaining deforestation rates (fraction cleared) by column, i.e. by start-of-period (1985) distance to the nearest road Distance to road 1993 (kilometers) Distance to road 1985 (kilometers) Figure 8 (matching on all covariates, means & post-regression) measured road-investment impacts, exact matching, explaining deforestation rates (fraction cleared) by cell, i.e. by start-and-end-of-period (1968 & 1975) distances Distance to road 1975 (kilometers) Distance to road 1968 (kilometers) International Scientific Conference. Amazon in Perspective - Integrated Science for a Sustainable Future