A Neighborhood Survey in the Nation’s Capital: Balancing Rigor, Resources, and Respect

1

A Neighborhood Survey in the Nation’s Capital: Balancing Rigor, Resources, and Respect

www.dcpni.org@dcpni

www.urban.org@urbaninstitute

Eastern Evaluation Research Society37th Annual Conference

April 28, 2014

2

Today’s Panelists

Isaac CastilloDirector of Data and Evaluation

DC Promise Neighborhood Initiative

@isaac_outcomes

Samantha GreenbergData and Evaluation Specialist


@eval_revolution

Maia WoluchemResearch Assistant

Metropolitan Housing and Communities Policy Center

Urban Institute

Megan GallagherSenior Research Associate Metropolitan Housing and Communities Policy Center

Urban Institute

April 28, 2014 DC Promise Neighborhood Initiative and The Urban Institute

3

A Neighborhood Survey in the Nation’s Capital

• The DC Promise Neighborhood Initiative’s Neighborhood Survey

• It Takes a Neighborhood to Design a Survey• Getting Community Residents Excited About a

Neighborhood Survey.• Data Collection in the 21st Century


DC Promise Neighborhood Initiative and The Urban Institute

4

The DC Promise Neighborhood Initiative’s Neighborhood Survey:

When Community, Research, Evaluation, and Rigor All Come Together

April 28, 2014



Twitter: @Isaac_outcomes

5

As part of the White House Neighborhood Revitalization Initiative, Promise Neighborhoods align federal funding streams that invest in transforming neighborhoods of concentrated poverty into neighborhoods of opportunity.

What is the Promise Neighborhood Approach?

The INSPIRATION….Harlem Children’s Zone (HCZ) inspired the U.S. Department of Education’s Promise Neighborhoods program, which launched in 2010.


6

What is DCPNI?

• DCPNI is one of 12 communities awarded funding from the U.S. Department of Education.

• DCPNI’s focus is on the Kenilworth-Parkside neighborhood of Washington, DC.

• Founded as an independent nonprofit in 2012 shortly after receiving federal funding.


7

Kenilworth-Parkside by the Numbers

• 5,725 people – 1,800 children• 98% African American, 55% are female• 50% of adults live in poverty• 49% of K-P residents experience food insecurity• 70% of K-P residents have a high school

education or higher (compared to 88% in DC)• 85% of households with children are headed by

single females


8

DCPNI’s Vision

To end intergenerational poverty in the

Kenilworth-Parkside community.


9

CARING ADULTSin their lives as parents, mentors, teachers, coaches & resources

SAFE PLACESIn which to live, grow and learn

A HEALTHY STARTand healthy future

AN EFFECTIVE EDUCATIONthat equips children and parents with marketable skills

AN OPPORTUNITY TO GIVE BACK to one’s community

THE FIVE PROMISES


10

What does DCPNI do?

FundCommunity-based organizations and schools to provide effective

and research based programming

CoordinateAll services provided in the Promise Neighborhood to

maximize effectiveness and avoid duplication

ConnectCommunity residents and

students to available services provided by partners

TrainEveryone in the Promise Neighborhood to utilize

information and advocate for themselves and their neighbors


11

DCPNI’s Work at a Glance

Fund

Coordinate

Provide

Train

Schools

Partners

Parents

Children

CARING ADULTS

SAFE PLACES

HEALTHY START

EFFECTIVE EDUCATION

GIVE BACK

=+


12

How Does DCPNI Measure Progress on Each Promise?

• DCPNI has 15 federal indicators that we are required to track by the Department of Education.

• Developing additional indicators on issues that are unique to Kenilworth-Parkside. – Teen birth rate– Mother’s educational attainment– Food insecurity

• These indicators help us determine if we are progressing towards the Promises.


13

Sample of DCPNI Indicators

• Parents read to their 0-5 year old children.

• Increased feelings of safety among community residents.

• Children ages 0-5 that have a medical home.


These data points need to be collected directly from community residents.

SAFE PLACES

HEALTHY START

14

The Need for a Neighborhood Survey

• Collect information directly from community residents.

• Limited other publically (or privately) available data sets for the Kenilworth-Parkside population.

• Desire to measure community level change over time.

• What little data that was available was old, limited to parts of the community, or collected using low rigor methods.


15

The Most Significant Challenges

• Very short timeframe – needed to complete entire process (from survey design to final analysis of data) in 9 months.

• Low levels of literacy in the community (and low comfort level with technology).

• High levels of suspicion of outsiders (and even suspicion of those from another part of the community).

• High levels of distrust of government and government-like entities.



16

It Takes a Neighborhood to Design a Survey: The Methodology Behind the Kenilworth-

Parkside Neighborhood Survey

April 28, 2014

Megan GallagherSenior Research Associate Metropolitan Housing and Communities Policy Center

Urban Institute


17

Methodology Overview

April 28, 2014

• Instrument Design• IRB Review• Sampling• Sampling frame• Strata, or key groupings• Random selection

• Response Rates • Population Weights


18

Instrument Design

April 28, 2014

• Create a baseline assessment of the neighborhood from which to assess change

• Many considerations, including • Department of Education Government Performance and

Results Act (GPRA) indicators• DCPNI programmatic priorities• DCHA Choice Neighborhoods data needs• Urban Institute suggestions• Resident interests and concerns• Instrument length, nature of questions


19

Instrument Design

April 28, 2014

• Topics include:• Housing• Neighborhood amenities• Access to Food • Neighborhood Supports • Neighborhood Conditions and Safety• Household Composition• Travel, Education, Employment and Public Assistance• Child Health, Education, and Well-being


20

IRB Review

April 28, 2014

• Institutional Review Board (IRB) makes sure that research minimizes risks to human subjects

• Reviewed and approved the plan for:• Requiring confidentiality pledges from staff• Obtaining consent from research subject• Roles for community resident and City Year corps

member• Providing incentives for completed surveys• Keeping data confidential


21

Sampling

April 28, 2014

• Used a list of addresses for 2 census tracts that make up Kenilworth Parkside

• Separated them into 3 sub-neighborhoods, or strata• Group 1: Kenilworth Courts• Group 2: KPRMC, Mayfair, Paradise, Lotus Square and

Victory Square • Group 3: Eastland Gardens and Paradise

• Randomly select addresses within each sub-neighborhood (SAS Proc Surveyselect)

• Oversampled Group 1


22

Response Rate

April 28, 2014

• Eligible households• 872 addresses randomly selected• 115 unoccupied• 872 – 115 = 757 eligible households

• Completed surveys• 444 completes

• Response rate• 444 / 757 = 59% response rate


23

Population Weights

April 28, 2014

Sub-neighborhood Population

% of Population

Completed Surveys

% of Surveys Weight

Group 1: Kenilworth Courts

283 12% 133 30% =12%/30%=.3857

Group 2: KPRMC, Mayfair, Paradise, Lotus Square and Victory Square

1620 68% 217 49% =68%/49%=1.4012

Group 3: Eastland Gardens and Paradise

476 20% 94 21% =20%/21%=.9731

Total 2379 100% 444 100% 1.0000


24

Doing the Impossible: Getting Community Residents Excited About

a Neighborhood Survey

April 28, 2014



Twitter: @eval_revolution


25April 28, 2014

Be inclusive

Gain community support

Build relationships

Be culturally sensitive

Add unique value

Goals for Kenilworth-Parkside Survey:


26April 28, 2014

Survey Administration Teams:

Kenilworth-Parkside

Community Resident

• Navigated the neighborhood• Greeted person at door• Introduced survey• Documented households visited • Handled incentives

• Consent process• Asked survey questions• Documented answers on tablet• Handled technology

Step 1: Build an Inclusive Process

27

Community residents:• Familiar with neighborhood• Have relationships with survey takers• Knowledgeable about community history and assets



28

City Year corps team members :• Versed in technology• Able to offer confidentiality (not from K-P)• Experienced serving D.C. communities



29

Community residents:• Recruitment and screening• Background checks• Mandatory training session

City Year Corps Team:• Partnership with City Year• Recruitment at full team event• Mandatory training session

Hiring Process


30

NEIGHBORHOOD SURVEY TRAINING30

• Structure of each work day

• Safety precautions• How to introduce

survey• How to document

responses from potential survey respondents

Training Process


31

Kenilworth-Parkside

Community Resident

Advantages of Two-person Teams• Increased likelihood of “open doors”• Decreased # of survey days needed• Easy navigation of community• Division of labor• Two people available to answer questions• Balance between confidentiality and trust• Increased safety


32

Challenges of Two-person Teams

• Massive recruiting and hiring efforts • DCPNI staff capacity• Difficult to ensure fidelity • Downtime during surveys• Answering questions about survey when

approached on street

Kenilworth-Parkside

Community Resident


33

Step 2: Prioritize cultural sensitivity

• Survey questions tested with community

• Community feedback encouraged throughout

• Survey administrators participated in focus group

• Concerns from community members noted for next survey


34

Step 3: Add Unique Value

• I heart data booklet provides data snapshot

• Data now exists on topics for which there are no other sources

• Ongoing conversations with the K-P community about data


35April 28, 2014 DC Promise Neighborhood Initiative and The Urban Institute

Step 4: Build Relationships

36April 28, 2014 DC Promise Neighborhood Initiative and The Urban Institute

• Impacts of Kenilworth-Parkside neighborhood survey:

Inclusive data collection

Gained community support

Built relationships

Culturally sensitive survey

Unique data collected

Results: Community Support for Survey:


37

Data Collection in the 21st Century: The Use of Tablet Computers for a

Neighborhood Survey

April 28, 2014



Urban [email protected]


38

Challenges for Our Survey

• Several teams approaching hundreds of doorsteps over several points in time

• Each respondent was different from the last• Each interviewer is different from the last• Presenting several challenges– Monetary costs– Confidentiality– Standardization– Flexibility– Ease

April 28, 2014


39

Paper vs Plastic

April 28, 2014

Feature Paper Surveys Tablet Surveys

MoneyPrinting supplies,

associated materials, postsurvey dataentry

$200 per tablet, KeySurveysubscription

ConfidentialityMaintaining security of

documents in field, postsurvey

Tablet’s secure connection and privacy capabilities

Standardization Interview cues

Flexibility May involve skipping pages Skip logic, conditionality

Ease


40

Program

• Used KeySurvey as our survey platform

– Allowed ability to build conditional sections and complicated skip patterns

– Allowed personalization based on household traits• Cost-reducing in terms of time, money and flexibility

April 28, 2014


41

Back-End v. Front-End• Level one: 159 questions and hundreds of

answer choices• Level two: Variable names

April 28, 2014


42

Back-End v. Front-End

• Level Three: Skip patterns– Sometimes a simple skip:

April 28, 2014


43


• Sometimes it’s something else entirely:

April 28, 2014


44


• Everything was automated, making things easy for everyone—

• Setting:

April 28, 2014


45

Back-End v. Front-End (Consent)

April 28, 2014


46

Back-End v. Front-End (Personalization)

April 28, 2014


47

Back-End v. Front-End• Once programmed, survey looked like this—

April 28, 2014


48


April 28, 2014


49

Administration:Who conducted the survey?

• AmeriCorps members and college volunteers– Read aloud the questions to the householder– Householder responds verbally

• Why?– Literacy rates among the community– Unfamiliarity with the technology– Standardization of questions and answers

April 28, 2014


50

Training

• Back-end was sophisticated but front-end was very user friendly.

• Training session has some ground rules:– Trust the programming– Don’t be pushy

• And some golden rules:– Read the exact wording of the question as presented– Obtain consent – Work with your partner

April 28, 2014


51

Partnerships in Action

• Morning Briefing (DCPNI Offices) – Community Resident:• Receive ten addresses, gift cards (4), survey receipts,

door hangers, and a walkie talkie– Interviewer:• Receive same ten addresses and a tablet computer

• Walk– Resident: Fill out tracking form to document

address condition

April 28, 2014


52

Partnerships in Action• Doorstep

– Community Resident:• Explain survey and present opportunity for giftcard• Answer questions householder may have• Document what happens

– Interviewer: • Obtain consent • Administer survey

• Administration (Interviewer & Community Resident)– At a comfortable place– Resident Role

April 28, 2014


53

Partnerships in Action

• Post-Survey– Community Resident:• Offer respondent choice of incentives• Everyone signs receipt

• What if no one is home? – Resident:• Leave door hanger• Document address so that another

team could make attempt

April 28, 2014


54

Tablets are not Without Challenges

• What isn’t done by hand must be done by computer– Careful time-intensive programming necessary

• Technical difficulties– Potential for tablet failure (need for backup tablets)

• Post-launch errors are data errors– Examples of missing answers or misplaced variables– Typing answers in leads to some data quality errors

April 28, 2014


55

But the Benefits?

• Benefits greater than challenges (in our case)• Data/Tablet Security– End of day wipe and upload to UI SFTP– Confidential drives at UI– Cerberus for tablet security

• Substantial results– Over four weekends in October and November,

got 444 responses, a response rate of 59% overall

April 28, 2014


56

Audience Q&A

April 28, 2014



@isaac_outcomes



@eval_revolution



Urban Institute

Megan GallagherResearch Associate


Urban Institute

A Neighborhood Survey in the Nation’s Capital: Balancing Rigor, Resources, and Respect

Presentations & Public Speaking

Transcript of A Neighborhood Survey in the Nation’s Capital: Balancing Rigor, Resources, and Respect