Survey Methodologies: Measurement Experiments with the...

Survey Methodologies: Measurement Experiments with the Living Standards Measurement Study (LSMS) November 19, 2014

Sydney Gourlay

Living Standards Measurement Study, DECRG

• Started in 1980s

• Purpose: Measure poverty plus study household behavior, welfare, interactions with government policies: determinants of outcomes, and linkages among assets/ characteristics of households/livelihood

sources/government interventions. • Currently 104 surveys available online with more to come @ worldbank.org/lsms

Living Standards Measurement Study (LSMS) Surveys

worldbank.org/lsms

LSMS Integrated Surveys on Agriculture

• Multi-topic, nationally representative panel household surveys conducted in eight sub-Saharan countries

• Include collection of detailed data on agriculture as well as various indicators, including: – HH composition – Education – Health – Labor – Migration – Consumption – HH enterprises – Community characteristics, prices – Credit Use – Etc…

• LSMS: More than “just” household surveys

• Methodological research gains ground

– Improving Measurement of Agricultural Productivity through Methodological Validation and Research

• Key stakeholders and donors in support of the methodological mission

– FAO and the Global Strategy to Improve Agriculture and Rural Statistics

– UK Aid from the British People

The Methodological Research Mission

UK Aid-funded research agenda includes methodological validation of:

1. Land area measurement 2. Soil fertility 3. Water resources 4. Labor inputs 5. Skill measurement 6. Production of continuous and extended-harvest crops 7. Computer-assisted personal interviewing for agricultural data


• No such thing as a simple question – How big is your plot? – How much income did you earn last year? – How much did you harvest last season? – Is so-and-so a household member?

• Who you ask and how you ask matter.

• Data collection methods can significantly impact policy relevant outcomes.


Methodological Research

Agricultural Land

Agricultural Production

Household Consumption

Other Areas


Agricultural Land



Other Areas

• Land Area Measurement

• Soil Fertility Testing

• How big is [PLOT]?

The answer will impact yield estimates, land inequality figures, etc.

• Primary Methodological Options: – Farmer self-reported estimate

– Compass and rope (aka traversing) -> the gold standard

– GPS

– Remote sensing (?)

Land Area Measurement

Farmer self-reported estimate

• Cheap

• Less missingness

• But…

• Subjective

• Complicated by traditional units

• Potential ulterior motives for over/under estimation

Compass and rope (aka traversing)

• Traditional gold standard for accuracy

• Eliminates subjectivity

• But…

• Time/labor intensive (leading to higher costs)

• Requires travel to plot

GPS

• Significantly quicker than traversing with advantages of objective measurement

• But…

• Questions of accuracy on small plots (?)

• Requires travel to plot

Remote sensing (?)

• Potential to eliminate plot visits

• But…

• Resolution limitations

• Feasibility of plot boundary identification

Land Area Measurement

Farmer Estimates: Good Enough?

LSMS-ISA Co.

GPS (acres)

Farmer SR (acres)

Mean Bias (SR – GPS)

Bias as % of GPS

Malawi 0.98 1.04 0.06 6.1%

Uganda 2.41 2.17 -0.25 -10.4%

Tanzania 2.67 2.35 -0.32 -12.0%

Niger 4.75 5.14 0.39 8.2%

Excerpt from WB Policy Research Working Paper 6550, “From Guesstimates to GPStimates: Land Area Measurement and Implications for Agricultural Analysis”. Calogero Carletto, Paul Winters, and Sydney Gourlay.

When you take a closer

look - strong evidence of systematic bias

Farmer Estimates:

Good Enough?

• Not exactly.

• Kilic, et al use multiple imputation to illustrate that self-reported estimates can be used to compute improved area measurements where GPS measurements are missing – Self-reported area estimates are statistically significant predictors of

observed GPS areas in both Tanzania 2010/11 and Uganda 2009/10

• See: Kilic, Zezza, Carletto, and Savastano. “Missing(ness) in Action:

Selectivity Bias in GPS-Based Land Area Measurements.” World Bank Policy Research Working Paper No. 6490 http://dx.doi.org/10.1596/1813-9450-490

Farmer Estimates: Good for Nothing?

http://dx.doi.org/10.1596/1813-9450-6490

http://dx.doi.org/10.1596/1813-9450-6490

http://dx.doi.org/10.1596/1813-9450-6490

http://dx.doi.org/10.1596/1813-9450-6490

http://dx.doi.org/10.1596/1813-9450-6490

http://dx.doi.org/10.1596/1813-9450-6490

Objective – Explain the dramatic differences between farmer self-reported plot areas and GPS measurements observed in the first wave of the LSMS panel through a small-scale land area verification exercise.

Land Area Validation Exercise: Nigeria

**Observations without both GPS and SR in Wave 2 were dropped, *p<0.1; ** p<0.05; *** p<0.01

Partnerships

National Bureau of Statistics

Tercile (CR) N* Panel Wave 2 Validation Exercise Panel - Validation

SR GPS SR GPS SR - SR GPS - GPS

1 157 1.153 0.433 0.627 0.300 0.525** 0.133***

2 161 3.007 0.742 1.132 0.812 1.875*** -0.07

3 152 4.074 1.975 2.531 2.456 1.543*** -0.481***

Total 470 2.709 1.021 1.396 1.150 1.313*** -0.129***

Land Area Validation Exercise: Nigeria

So far, we have sees that GPS and farmer self-

reported estimates are not simple substitutes, but how does GPS stack up to

the gold standard?

Objective – Test various measurement methodologies in order to:

• Validate the data quality associated with each method • Determine the costs and benefits of each method • Assess the feasibility and necessity of implementing each

method in national household surveys • Document best practices in data collection based on our

findings from experiments in Ethiopia and beyond

Partnerships • Central Statistical Agency of Ethiopia • World Agroforestry Centre (ICRAF)

Status • Fieldwork completed early March 2014 • Soil testing and data analysis underway

Land and Soil Experimental Research (LaSER): Ethiopia

Elevation Rainfall AEZ

85 EAs 12 HH Each

LaSER: Ethiopia

Methodologies tested:

Land area • Traversing (i.e., compass and rope) • GPS measurement (Garmin) • GPS measurement (Android tablet) • Farmer self-reported area • Clinometer • Farmer self-reported incline

Completed during the post-planting visit on up to two fields per household

Soil fertility • Spectral Soil Analysis • Conventional Soil Analysis • Farmer self-reported soil quality

Samples collected during the post- planting visit, processed at regional labs and shipped to ICRAF Nairobi for analysis

Maize production

• Crop-cutting using a 4m x 4m subplot and 2m x 2m subplot

• Farmer self-reported harvest

Completed by field teams when alerted by household


1018 households

interviewed

1799 fields selected

for objective

measurement

and soil testing

3791 soil samples

collected*

205 fields with crop-

cutting

*2 samples were collected from each field (different depths and sampling procedures), an additional sample was collected on fields with crop-cutting.


Level (CR)

Ethiopia

N SR GPS CR Avg GPS - Avg CR / Avg CR

1 (< 0.25 acres) 986 0.25 0.09 0.09 3%

2 (0.25 - 1 acre) 556 0.65 0.53 0.52 2%

3 (> 1 acre) 174 1.27 1.58 1.56 1%

Total 1716 0.48 0.39 0.38 2% 0

20

40

60

80

100

(< 0.25 acres) (0.25 - 1 acre) (> 1 acre)

CR Plot Area

Measurement Duration (Minutes)

ET GPS ET CR

• Despite similar measurements, field experience revealed slow satellite acquisition (resulting in several “0” areas)


Alternative Tablet GPS Measurements (Ethiopia Land and Soil Experimental Research)

Acres

Level (CR)

LaSER

N CR

Garmin Handheld

GPS Alternative Tablet GPS

Mean bias: (Tablet -

Garmin)/Garmin

1 (< 0.05 acres) 45 0.02 0.02 0.02 -1.1%

2 (< 0.15 acres) 45 0.09 0.09 0.10 4.0%

3 (< 0.35 acres) 57 0.24 0.24 0.25 1.9%

4 (< 0.75 acres) 36 0.50 0.51 0.49 -4.1%

5 (< 1.25 acres) 33 0.94 0.96 0.94 -1.8%

6 (>= 1.25 acres) 18 1.71 1.73 1.76 2.0%

Total 234 0.42 0.43 0.42 -0.3%

*unweighted means

Much of the difference between GPS and SR estimates are explained by plot and household factors.

Difference between GPS and CR much harder to explain due to small discrepancy.

Land and Soil Experimental Research

(LaSER): Ethiopia

Determinants of Bias (SR-GPS) OLS Regression, Error Clustered on Enumerator ID

GPS Area (acres) -0.336*** GPS Area2 0.006 Per : Area Ratio (GPS) -0.043 Distance from dwelling 0.011** Number cultivated plots in HH -0.008** Slope (clinometer) 0.004 Soil Quality (SR):

Fair -0.076** Poor -0.065

Title or Certificate of Ownership 0.086*** Can sell or use as collateral -0.032

Treecover:

Partial 0.04 Heavy 0.182*

HH Head Characterisitcs:

Female -0.001 Yrs education 0.014* Age -0.001 Literate -0.068*

Constant 0.286*** N 1716 R2 0.154 Enumerator Fixed Effects *p<.1; ** p<.05; *** p<.01

Preliminary results (please do not quote)

Why Soils? • Research questions:

– Inverse-productivity relationship – Gender issues in land allocation – Effective and appropriate input use – Impact of climate change and farmer response

• Objective: assess logistical feasibility of plot level

soil testing and analytical and policy value


Currently, soil data comes largely in the form of spatial data (but at relatively low resolution for small Africa plots) and coarse subjective farmer assessments.


Image source: ISRIC, http://www.isric.org/data/soil-property-maps-africa-1-km

• Soil results…TBD! Samples are currently with ICRAF Nairobi for testing.

• Analysis of various agricultural relationships with the plot-level data

• Identify best subjective questions to predict laboratory results.

• Average soil collection time: approx. 38 minutes per plot


Challenges:

• LOGISTICS

– Soils can’t be bagged for more than 5-7 days depending on moisture levels (leading to high drive time/fuel costs)

– Local labs to conduct initial processing before shipping

– Labeling, labeling, labeling

• Timing with crop: when can you sample?

• Mobile teams and crop-cutting

Next Steps:

• Soil Analysis at ICRAF Nairobi

• Data Analysis Capacity Building Workshop

• Analysis and Research Papers

• Ethiopia-specific handbook on soil sampling for household surveys


Insert photo from ICRAF labs

• Carletto, Savastano, and Zezza (2013). "Fact or Artifact: The Impact of Measurement Errors on the Farm Size–Productivity Relationship." Journal of Development Economics, 103C, 254-261.

• Carletto, Gourlay and Winters. “From Guesstimates to GPStimates: Land Area Measurement and Implications for Agricultural Analysis.” World Bank Policy Research Working Paper No. 6550 – Expected submission to Journal of African Economies: June 2014.

• Kilic, Zezza, Carletto, and Savastano. “Missing(ness) in Action: Selectivity Bias in GPS-Based Land Area Measurements.” World Bank Policy Research Working Paper No. 6490 – Expected submission to Agricultural Economics: June 2014.

• Kilic, Kim, and Yang. “Exploring the Promise of Fractional Imputation to Predict Missing GPS-Based Area Measurements in Household Surveys: Evidence from Malawi.” – Draft manuscript expected by December 2014.

Sourcebook: Land Area Measurement in Household Surveys: Empirical Evidence & Practical Guidance for Effective Data Collection.

References and Other On-Going Research: Land


Agricultural Land



Other Areas

• Production of continuous crops & use of mobile phones

• Seed identification

• Ag labor

Objectives • Validate methods for measuring cassava productivity

by evaluating the data quality and costs vs. benefits for each method

• Assess the feasibility and necessity of implementing each method in national household surveys

• Document best practices in data collection

Partnerships • Ministry of Agriculture and Natural Resources,

Zanzibar • Office of the Chief Government Statistician, Zanzibar

Status • Fieldwork: June 2013 – June 2014 • Data cleaning and analysis underway

Cassava Productivity: Zanzibar


Land area • Traversing (i.e. compass-and-rope)

• GPS measurement

• Farmer self-reported area

Cassava

production

• Crop-cutting with balance scales

• Crop diaries with enumerator visits twice a week

• Crop diaries with telephone calls twice a week

• Farmer self-reported harvest (12-month recall)

• Farmer self-reported harvest (6-month recall)



Level (CR) N GPS CR Bias Mean Bias /

Mean CR

1 (< 0.05 acres) 52 0.04 0.04 0.00 -3%

2 (< 0.15 acres) 646 0.11 0.11 0.00 2%

3 (< 0.35 acres) 835 0.24 0.23 0.01 3%

4 (< 0.75 acres) 328 0.51 0.49 0.02 3%

5 (< 1.25 acres) 64 0.93 0.93 0.00 0%

6 (>= 1.25 acres) 20 1.91 1.81 0.09 5%

Total 1945 0.27 0.27 0.01 3%

Difference in mean GPS and mean SR: 148%

Fieldwork Challenges:

• Differing incentive structures across treatment arms (for example, mobile phones)

• Connecting with farmers while not in the field (timing of phone prompting)

• Know the country context (for example, Ramadan)


Ethiopian Institute for Agricultural Research (EIAR)

Central Statistical Agency (CSA)

Diversity Arrays Technology (Dart)

With support from:

International Food Policy Research Institute (IFPRI)

Bill & Melinda Gates Foundation (BMGF)

DNA Fingerprinting for Varietal Identification - Pilot

34

Why are varietal identification and adoption crucial?

• Considerable public investment in crop improvement in Ethiopia

• Tracking individual variety uptake is critical to ensuring the Ethiopian

crop improvement program remains targeted and capable of

accomplishing its goals

• This requires having a highly responsive system of monitoring the spread

of improved varieties


35

• Traditional methods of estimating varietal diffusion include farmer identification (ID) of varietal use through HH surveys, with recent improvements to incorporate expert identification and consultation

• Challenges with farmer ID:

– Farmers may not be able to identify individual varieties in the field

– Prevalence of local names that cannot be matched to the official variety names

• Scientists have long discussed using DNA fingerprinting technology to verify farmer ID estimates, but this is the world’s first field application pilot with preliminary results


36

Objectives of the Pilot Project in Ethiopia:

1. Test the feasibility of estimating varietal adoption levels in specific pilot zones using DNA fingerprinting

• Wheat and Maize grain samples from farmer fields

• Reference library of breeder seed

2. Test the technical feasibility : Can DNA fingerprinting discriminate

across improved varieties grown by farmers?

3. Test the logistical feasibility: Can the institutions partner for the collection and analysis of hundreds of data and grain samples and maintain identity preservation?


Pilot survey covered a total of 108 Enumeration Areas in three zones in Oromia Region.

Field Visits with CSA enumerators for sample collection • Grain (seed) samples for DNA fingerprinting from AgSS crop cuts

– It is possible to use existing CSA crop cuts to collect grain samples to be fingerprinted?

– Logistically easier to collect and transport seed samples, including in potential scale-up

• Additional short survey asking farmers to identify the variety they were growing – This information is to see if farmer’s variety identification matches DNA fingerprinting results

Data management and analysis • Linking genetic analysis, short survey and AgSS data for additional analysis of seed use,

inputs, etc.


After lab analysis at DART(Australia), preliminary results for wheat find:

• 62% of the farmers reported using adopted improved wheat varieties

• Genetic analysis found that 96% of the farmers actually use improved wheat

• Only 9.3% of the farmers were able to correctly identify the improved wheat varieties cultivated in their fields


Preliminary results for maize slightly more encouraging:

• 56% of the farmers reported using adopted improved maize varieties

• Genetic analysis found that 64% of the farmers actually use improved maize

• Only 47% of the farmers were able to correctly identify the improved maize varieties cultivated in their fields (all for hybrid users)


Farmer reports consistently underestimate the use of improved varieties compared to DNA results with little knowledge of what specific varieties are being used…

Significant implications for monitoring varietal adoption rates, effectiveness of distribution programs, etc.


• Objective – Develop and validate methods on improving data collection on the quantity and demographics of family labor in farming in low-income settings

• Most small-holders do not hire labor

- Characterizing farm labor productivity becomes a challenge

Measuring labor in farm households in Africa

• 7 day recall for each type of labor

• 12 month for each type (recall)

• LSMS-ISA: intensive recall


DAYS /

WEEK

HOURS /

DAY

PLOT ID HH

ROSTER ID

CODE #3

WEEKS DAYS /

WEEK

HOURS /

DAY

HH

ROSTER ID

CODE #4

WEEKS

LA N D P R EP A R A T ION

A N D P LA N T IN G

HH

ROSTER ID

CODE #1

WEEKS DAYS /

WEEK

HOURS /

DAY

HH

ROSTER ID

CODE #2

WEEKS DAYS /

WEEK

HOURS /

DAY

• Challenges:

– What is the truth? Need a benchmark/gold standard

– The price of getting the truth

– Alternatives that are feasible for large household survey efforts


• Musoma District in the Mara region, Tanzania

• A district with high prevalence of maize

• 18 EAs

• 36 households per EA


Three alternative survey designs:

• Control: Standard agricultural labor module – labor reported in the aggregate by recall for the

entire season

• “Truth” Treatment: Weekly interviews for labor module for the duration of the main season.

• “Practical” Treatment: Weekly phone surveys for labor module for the duration of the main season.


Fieldwork Challenges: – Some normal

• Mobile phones (charger, coverage) • Measuring plot size

– Some linked to field work during the ag season • Transport during rainy season • Adjusted interview times to accommodate household labor

activities • Uncertainty of start of season (determined by rains)

– Some quite unexpected • Free Mason panic in Mara region…


Punchline…TBD

Analysis is underway!



Agricultural Land



Other Areas

• Issues of recall • Food eaten away

from home • Consumption

module design (x2)

• Consumption expenditures are the core indicator to measure poverty, inequality and living standards in low income countries

• large national representative surveys used to compare welfare over time or across countries:

– Different countries use different methods

– Even within countries survey design changes over time

• Trends in poverty over time, or differences across countries could be due to differences in design, rather than reflecting any true change.

Motivation – Consumption Research

• Wide variety of consumption modules … – Differences across countries and over time

• Differences and weaknesses along many dimensions … – Diary vs. recall – Length of reference period – Specificity of list

• Nomenclature; Open vs. close; Prompting – Consumption vs. acquisition – Household vs. individual – Non-standard units of measure – Food consumed away from home (FAFH) – Valuation of consumed own-production

• Method does matter and lack of standards results in poor/non-comparable data

Challenges in Measuring Household Consumption

51

Some previous attempts to assess differences…

• Papua New Guinea: Diaries result in 26% more food consumption

• El Salvador: Long recall list results in 31% more consumption than shorter aggregated list

• Indonesia: Long recall yields 20% more consumption but no re-ranking of households

• Ghana: For every day added to recall period, total purchases fall by 2.9%, plateau of 20-25% loss

• Russia: Individual diaries gave 6-11% higher expenditure than a household diary

52

Testing Consumption Measures in Tanzania

Module Consumption measurement

1 Long list (58 items) 14 day

2 Long list (58 items) 7 day

3 Subset list (17 items) 7 day

4 Collapsed list (11 items) 7 day

5 Long list (58 items) “Usual 12 month”

6 HH diary with frequent visits

7 HH diary with infrequent visits (by literacy

status)

8 Personal diary with daily visits

• Compare benchmark (personal diary) with 7 alternatives:

• 2 household diaries,

• 5 recall modules

53

SHWALITA: Survey of Household Welfare and Labour in Tanzania

• Tanzania: 168 villages, in 7 districts

• Mix urban/rural EAs; agro-climactic zones

• Per EA: randomly sample 3 HHs for each module, i.e. N=500 for each of 8 modules


54


(1) Personal diary omitted Ln total

1. Recall: Long, 14 day -0.161***

2. Recall: Long, 7 day -0.039

3. Recall: Subset, 7 day -0.071*

4. Recall: Collapse, 7 day -0.283***

5. Recall: Long usual 12 month -0.207***

6. Diary: HH, frequent -0.173***

7. Diary: HH, infrequent -0.136***

Number of households 4,025

Cik = βkMk + eik

Regressions of per capita consumption expenditure

55

Isolating effect of Recall Period & Length of List (per capita consumption)


(1) Long, 7 day omitted Ln total

Length of recall period

1.Long, 14 day -0.121***

(0.037)

5.Usual 12 month -0.168***

(0.037)


Length of list

3.Subset, 7 day -0.032

(0.036)

4.Collapse, 7 day -0.244***

(0.036)


56

There are important differences in means, but these may not result in:

– Differences in inequality

– Differences in poverty


57

Gini coefficient


Total per capita

1. Long 14 day 0.512***

2. Long 7 day 0.483

3. Subset 7 day 0.467

4. Collapse 7 day 0.484

5. Long Usual 12 month 0.537***

6. HH diary Frequent 0.427

7. HH diary Infrequent 0.419*

8. Personal diary 0.450

58


62.8

54.9 55.1

66.8 64.6

59.5

55.6

47.5

1. Recall: Long, 14day

2. Recall: Long, 7day

3. Recall: Subset, 7day

4. Recall: Collapse,7 day

5. Recall: LongUsual 12 month

6. Diary: HH,Frequent

7. Diary: HH,Infrequent

8. Diary: Personal

Poverty headcount rate (%) at $1.25/person/day by module type


0

10

20

30

40

50

60

70

80

Long list,14 day

Long list,7day

Subset list,7day

Usualmonth

Diary,HH freq

Diary,HH infreq

Diary,Indiv

Hunger Prevalence: (% of people in households with energy availability less than energy requirements)

60

COSTS: – Personal diaries are

9 times more expensive than recall

– Household diaries are 3-7 times more expensive (depending on degree of supervision)

– Recall is cheaper, but which module to choose?


49.8 48.6 41.0 41.6

76.0

0

10

20

30

40

50

60

70

80

Min

ute

s t

o c

om

ple

te

co

nsu

mp

tio

n s

ecti

on

61

Recommendations & key take-aways: • “usual” month or collapsed list approach (modules 4 &

5) perform badly (high cognitive demands) • The long list, 14 day recall has much lower mean,

higher inequality and higher poverty. • All considered: Tie between 7 day recall with full or

subset list (modules 2 or 3). • Time stamps tell us that reducing the list saves only 8

minutes of interview time: doesn’t seem worth loss in detail


62

All Questionnaire modules & diaries are available at:

www.edi-africa.com/research/shwalita

Also, see:

Beegle, K, J. De Weerdt, J. Friedman and J. Gibson. 2012. “Methods of Household Consumption Measurement through Surveys: Experimental Results from Tanzania.” Journal of Development Economics 98: 3-18


http://www.edi-africa.com/research/shwalita



LSMS team is currently engaged with BPS to analyze the consumption module(s) of the SUSENAS

Historically, SUSENAS has:

• Long Version: o 229 food items with 6 questions each

o Ideally took 2 hours, in reality, it took 45-60 minutes average

o Conducted 1x/3 years to 2005, 1x/year to 2010, 4x/year current

o Usage: Poverty Line, pce/poverty head count

• Short Version: o 21 food buckets with 1 question asking total expenses

o Took 30 minutes in average

o Conducted 1x/year until 2010

o Usage: pce/poverty head count

Indonesian SUSENAS & LSMS

Task: improve the design of the consumption module while

minimizing damage to

comparability over time.

• 6 randomized groups – Gold standard: 7-day diary, daily visits – Status quo (1 visit) – Status quo with prompting (1 visit) – Shorter list (1 visit) – Shortest list (1 visit) – Shorter list, with bounding (2 visits)

• Funding from WB and BPS • Fieldwork, using dedicated teams, in Sept-Oct,

2014

Indonesian SUSENAS & LSMS

Food away from home is growing in the developing world, but very little information is collected in household surveys.

Improving FAFH in Peru: Establishment survey and lab analysis

0.75

0.77

0.79

0.81

0.83

0.85

0.87

0.89

2006 2007 2008 2009 2010 2011 2012 2013

Incidence in Peru

Source: ENAHO 2006-2013. Household-level statistics

• We know – 44% of households eat at least 1 meal outside of home – 4 meals per week – Type of establishment – Type of meal (breakfast, lunch, dinner) – Cost

• We DO NOT know – Meal contents – Caloric/nutritional information of meals

Improving FAFH in Peru

• Sample design – Metropolitan Lima and Callao (48 districts) – Representative survey of almost 1800 formal food establishments – Stratified by 5 socio-economic strata

• Implementation – Phase 1 (DONE)

• Identify 3 most frequently consumed meals per establishment • For each: collect detailed information on ingredients

– Phase 2 (DONE) • Laboratory analysis - caloric and nutritional information for top 50

menus (10 per strata – same menus)


• Implementation (cont’d) – Phase 3 (UNDERWAY)

• Poverty analysis – Calculate unit caloric cost of each meal/item

– Poverty simulations - poverty lines, consumption, poverty estimates

• Nutritional analysis – By strata (are all meals created equal?)

– Eating-in vis a vis Eating-out

– Phase 4: Feeding back to the ENAHO and extensions • Updated ENAHO survey module


Although analysis is still underway, preliminary results suggest that after including FAFH roughly

20% of the ‘at-home’ poor leave poverty



Agricultural Land



Other Areas

• Respondent selection in asset identification

• Skills Measurement

• CAPI

Objective – Develop and validate methods for the measurement of functioning and skills in agriculture

Skills Measurement: Kenya


Cognitive - Raven - Forward and backward digit span - Math exercises - Reading test

Non-cognitive - Entrepreneurship (initiative, risk taking, tenacity, aspiration) - The Big Five - Mental Health and Self esteem (CESD)

Technical Skills - Knowledge test - Self-assessment - What farmers do Adapted to local inputs and outputs with local expert(s).

Partnerships

Paris School of Economics (Karen Macours & Rachid Laajaj)

Status

Fieldwork ongoing since January 2014

Data analysis underway

Skills Measurement: Kenya

900 farmers

96 villages

16 sub-locations

Timeline:

Pilot November 2013

Training January 2014

Fieldwork January – March 2014

Analysis April – present

• Siaya province in Western Kenya

• Quantitative pilot was programmed in Blaise; piloted on 120 farmers

• Two visits: January and March 2014

• Sample equally divided between women and men

• Survey experiment implemented in collaboration with the UN Evidence and Data for Gender Equality (EDGE) initiative & Uganda Bureau of Statistics

• Dual objective

– Provide an assessment of the effects of respondent selection decisions on the analysis of individual asset ownership & control

– To inform the 2015 operations in EDGE pilot countries & the UN guidelines on measuring individual asset ownership & control

Methodological Experiment on Measuring Asset Ownership from a Gender Perspective (MEXA)

• 140 Enumeration Areas (EAs) selected with probability proportional to size across Uganda – Actual EAs Visited: 137

• Rural/Urban EA Split: 60/40 percent

• HH listing in each EA for random selection of sample HHs

• 20 HHs randomly selected in each EA, 4 randomly allocated to each treatment arm in each EA prior to field work


Treatment Arms:


Arm Who? How? What?

1 “Most Knowledgeable” Household Member

Alone Assets Owned Exclusively/ Jointly by Household Members

2 Randomly Selected Member of Principal Couple

Alone Assets Owned Exclusively/ Jointly by Household Members

3 Principal Couple

Together Assets Owned Exclusively/ Jointly by Household Members

4 Adult (18+) Household Members

Alone, Simultaneous

Assets Owned Exclusively/ Jointly by Household Members

5 Adult (18+) Household Members

Alone, Simultaneous

Assets Owned Exclusively/ Jointly by Respondent

Scope of data collection: • (Individual-Level) Demographics • Core Assets

– Dwelling & Residential Land – Agricultural Land – Non-Agricultural Land & Other Real Estate – Livestock – Non-Farm Enterprises & Assets – Agricultural Equipment – Consumer Durables – Financial Assets & Liabilities – Valuables


• Implementing agency: Uganda Bureau of Statistics (UBoS)

• Implementation period: March-August 2014

• Use of Android tablets & computer-assisted personal interviewing (CAPI) application designed in Survey Solutions

• 7 mobile field teams

• Arms 4 & 5 interviews attempted to be conducted simultaneously

• Female (male) respondents attempted to be matched with female (male) enumerators


Stay tuned…Findings will be presented at the Kitakyushu Forum on Asian Women in Japan December 3-6, 2014.


Computer-Assisted Personal Interviewing Software Development – SURVEY SOLUTIONS

Objectives – Develop a CAPI application suitable for collecting agricultural data as part of a complex integrated household survey

Partnerships • Computational Tools Unit, Development Research Group, The World Bank

(DECCT)

• Food and Agricultural Organization of the United Nations

Status • Beta version released in September 2013

• Survey Solutions 3.0 released Nov 10th, 2014

• New and improved features under development

Package Elements • Designer, a web-based console for constructing

CAPI questionnaires • Interviewer, an Android-based application for

enumerators to collect data from assigned respondents

• Supervisor, an interface for supervisors to assign respondents and to review completed questionnaires

• Headquarters, a portal for aggregating survey data and reporting on survey progress.

Unique features include: • Survey management system • Fluid questionnaire navigation • Intuitive toolkit for collaborative questionnaire

development

Computer-Assisted Personal Interviewing Software Development – SURVEY SOLUTIONS

Join Arthur Shaw this afternoon for a hands-on tutorial of Survey Solutions


Agricultural

Land



Other Areas

• Uganda: Crop cutting - Selected districts in Eastern Region

– soils,

– maize crop-cutting on pure and intercropped plots,

– variety identification

– use of high resolution remote sensing imagery for land area measurement and and yield estimation

• Malawi: Cassava Production & Productivity

– Fieldwork: Nov/Dec 2014 – Oct/Nov 2015

• Zanzibar, Tanzania: Measurement of Banana Production

• Water quality in partnership with WHO (?)

• Future research could include…

– Rainfall measurement

– Validation of big data

In the pipeline…

The search for objective measurements that are both scalable and more in line with gold standard precision continues…

For further information, please visit worldbank.org/lsms or contact me at [email protected]

Thank you!

worldbank.org/lsms

worldbank.org/lsms

mailto:[email protected]

Survey Methodologies: Measurement Experiments with the...

Documents

Transcript of Survey Methodologies: Measurement Experiments with the...