1
A Neighborhood Survey in the Nation’s Capital: Balancing Rigor, Resources, and Respect
www.dcpni.org@dcpni
www.urban.org@urbaninstitute
Eastern Evaluation Research Society37th Annual Conference
April 28, 2014
2
Today’s Panelists
Isaac CastilloDirector of Data and Evaluation
DC Promise Neighborhood Initiative
@isaac_outcomes
Samantha GreenbergData and Evaluation Specialist
DC Promise Neighborhood Initiative
@eval_revolution
Maia WoluchemResearch Assistant
Metropolitan Housing and Communities Policy Center
Urban Institute
Megan GallagherSenior Research Associate Metropolitan Housing and Communities Policy Center
Urban Institute
April 28, 2014 DC Promise Neighborhood Initiative and The Urban Institute
3
A Neighborhood Survey in the Nation’s Capital
• The DC Promise Neighborhood Initiative’s Neighborhood Survey
• It Takes a Neighborhood to Design a Survey• Getting Community Residents Excited About a
Neighborhood Survey.• Data Collection in the 21st Century
April 28, 2014 DC Promise Neighborhood Initiative and The Urban Institute
DC Promise Neighborhood Initiative and The Urban Institute
4
The DC Promise Neighborhood Initiative’s Neighborhood Survey:
When Community, Research, Evaluation, and Rigor All Come Together
April 28, 2014
Isaac CastilloDirector of Data and Evaluation
DC Promise Neighborhood Initiative
Twitter: @Isaac_outcomes
5
As part of the White House Neighborhood Revitalization Initiative, Promise Neighborhoods align federal funding streams that invest in transforming neighborhoods of concentrated poverty into neighborhoods of opportunity.
What is the Promise Neighborhood Approach?
The INSPIRATION….Harlem Children’s Zone (HCZ) inspired the U.S. Department of Education’s Promise Neighborhoods program, which launched in 2010.
April 28, 2014 DC Promise Neighborhood Initiative and The Urban Institute
6
What is DCPNI?
• DCPNI is one of 12 communities awarded funding from the U.S. Department of Education.
• DCPNI’s focus is on the Kenilworth-Parkside neighborhood of Washington, DC.
• Founded as an independent nonprofit in 2012 shortly after receiving federal funding.
April 28, 2014 DC Promise Neighborhood Initiative and The Urban Institute
7
Kenilworth-Parkside by the Numbers
• 5,725 people – 1,800 children• 98% African American, 55% are female• 50% of adults live in poverty• 49% of K-P residents experience food insecurity• 70% of K-P residents have a high school
education or higher (compared to 88% in DC)• 85% of households with children are headed by
single females
April 28, 2014 DC Promise Neighborhood Initiative and The Urban Institute
8
DCPNI’s Vision
To end intergenerational poverty in the
Kenilworth-Parkside community.
April 28, 2014 DC Promise Neighborhood Initiative and The Urban Institute
9
CARING ADULTSin their lives as parents, mentors, teachers, coaches & resources
SAFE PLACESIn which to live, grow and learn
A HEALTHY STARTand healthy future
AN EFFECTIVE EDUCATIONthat equips children and parents with marketable skills
AN OPPORTUNITY TO GIVE BACK to one’s community
THE FIVE PROMISES
April 28, 2014 DC Promise Neighborhood Initiative and The Urban Institute
10
What does DCPNI do?
FundCommunity-based organizations and schools to provide effective
and research based programming
CoordinateAll services provided in the Promise Neighborhood to
maximize effectiveness and avoid duplication
ConnectCommunity residents and
students to available services provided by partners
TrainEveryone in the Promise Neighborhood to utilize
information and advocate for themselves and their neighbors
April 28, 2014 DC Promise Neighborhood Initiative and The Urban Institute
11
DCPNI’s Work at a Glance
Fund
Coordinate
Provide
Train
Schools
Partners
Parents
Children
CARING ADULTS
SAFE PLACES
HEALTHY START
EFFECTIVE EDUCATION
GIVE BACK
=+
April 28, 2014 DC Promise Neighborhood Initiative and The Urban Institute
12
How Does DCPNI Measure Progress on Each Promise?
• DCPNI has 15 federal indicators that we are required to track by the Department of Education.
• Developing additional indicators on issues that are unique to Kenilworth-Parkside. – Teen birth rate– Mother’s educational attainment– Food insecurity
• These indicators help us determine if we are progressing towards the Promises.
April 28, 2014 DC Promise Neighborhood Initiative and The Urban Institute
13
Sample of DCPNI Indicators
• Parents read to their 0-5 year old children.
• Increased feelings of safety among community residents.
• Children ages 0-5 that have a medical home.
April 28, 2014 DC Promise Neighborhood Initiative and The Urban Institute
These data points need to be collected directly from community residents.
SAFE PLACES
HEALTHY START
14
The Need for a Neighborhood Survey
• Collect information directly from community residents.
• Limited other publically (or privately) available data sets for the Kenilworth-Parkside population.
• Desire to measure community level change over time.
• What little data that was available was old, limited to parts of the community, or collected using low rigor methods.
April 28, 2014 DC Promise Neighborhood Initiative and The Urban Institute
15
The Most Significant Challenges
• Very short timeframe – needed to complete entire process (from survey design to final analysis of data) in 9 months.
• Low levels of literacy in the community (and low comfort level with technology).
• High levels of suspicion of outsiders (and even suspicion of those from another part of the community).
• High levels of distrust of government and government-like entities.
April 28, 2014 DC Promise Neighborhood Initiative and The Urban Institute
DC Promise Neighborhood Initiative and The Urban Institute
16
It Takes a Neighborhood to Design a Survey: The Methodology Behind the Kenilworth-
Parkside Neighborhood Survey
April 28, 2014
Megan GallagherSenior Research Associate Metropolitan Housing and Communities Policy Center
Urban Institute
DC Promise Neighborhood Initiative and The Urban Institute
17
Methodology Overview
April 28, 2014
• Instrument Design• IRB Review• Sampling• Sampling frame• Strata, or key groupings• Random selection
• Response Rates • Population Weights
DC Promise Neighborhood Initiative and The Urban Institute
18
Instrument Design
April 28, 2014
• Create a baseline assessment of the neighborhood from which to assess change
• Many considerations, including • Department of Education Government Performance and
Results Act (GPRA) indicators• DCPNI programmatic priorities• DCHA Choice Neighborhoods data needs• Urban Institute suggestions• Resident interests and concerns• Instrument length, nature of questions
DC Promise Neighborhood Initiative and The Urban Institute
19
Instrument Design
April 28, 2014
• Topics include:• Housing• Neighborhood amenities• Access to Food • Neighborhood Supports • Neighborhood Conditions and Safety• Household Composition• Travel, Education, Employment and Public Assistance• Child Health, Education, and Well-being
DC Promise Neighborhood Initiative and The Urban Institute
20
IRB Review
April 28, 2014
• Institutional Review Board (IRB) makes sure that research minimizes risks to human subjects
• Reviewed and approved the plan for:• Requiring confidentiality pledges from staff• Obtaining consent from research subject• Roles for community resident and City Year corps
member• Providing incentives for completed surveys• Keeping data confidential
DC Promise Neighborhood Initiative and The Urban Institute
21
Sampling
April 28, 2014
• Used a list of addresses for 2 census tracts that make up Kenilworth Parkside
• Separated them into 3 sub-neighborhoods, or strata• Group 1: Kenilworth Courts• Group 2: KPRMC, Mayfair, Paradise, Lotus Square and
Victory Square • Group 3: Eastland Gardens and Paradise
• Randomly select addresses within each sub-neighborhood (SAS Proc Surveyselect)
• Oversampled Group 1
DC Promise Neighborhood Initiative and The Urban Institute
22
Response Rate
April 28, 2014
• Eligible households• 872 addresses randomly selected• 115 unoccupied• 872 – 115 = 757 eligible households
• Completed surveys• 444 completes
• Response rate• 444 / 757 = 59% response rate
DC Promise Neighborhood Initiative and The Urban Institute
23
Population Weights
April 28, 2014
Sub-neighborhood Population
% of Population
Completed Surveys
% of Surveys Weight
Group 1: Kenilworth Courts
283 12% 133 30% =12%/30%=.3857
Group 2: KPRMC, Mayfair, Paradise, Lotus Square and Victory Square
1620 68% 217 49% =68%/49%=1.4012
Group 3: Eastland Gardens and Paradise
476 20% 94 21% =20%/21%=.9731
Total 2379 100% 444 100% 1.0000
DC Promise Neighborhood Initiative and The Urban Institute
24
Doing the Impossible: Getting Community Residents Excited About
a Neighborhood Survey
April 28, 2014
Samantha GreenbergData and Evaluation Specialist
DC Promise Neighborhood Initiative
Twitter: @eval_revolution
DC Promise Neighborhood Initiative and The Urban Institute
25April 28, 2014
Be inclusive
Gain community support
Build relationships
Be culturally sensitive
Add unique value
Goals for Kenilworth-Parkside Survey:
DC Promise Neighborhood Initiative and The Urban Institute
26April 28, 2014
Survey Administration Teams:
Kenilworth-Parkside
Community Resident
• Navigated the neighborhood• Greeted person at door• Introduced survey• Documented households visited • Handled incentives
• Consent process• Asked survey questions• Documented answers on tablet• Handled technology
Step 1: Build an Inclusive Process
27
Community residents:• Familiar with neighborhood• Have relationships with survey takers• Knowledgeable about community history and assets
Step 1: Build an Inclusive Process
April 28, 2014 DC Promise Neighborhood Initiative and The Urban Institute
28
City Year corps team members :• Versed in technology• Able to offer confidentiality (not from K-P)• Experienced serving D.C. communities
Step 1: Build an Inclusive Process
April 28, 2014 DC Promise Neighborhood Initiative and The Urban Institute
29
Community residents:• Recruitment and screening• Background checks• Mandatory training session
City Year Corps Team:• Partnership with City Year• Recruitment at full team event• Mandatory training session
Hiring Process
April 28, 2014 DC Promise Neighborhood Initiative and The Urban Institute
30
NEIGHBORHOOD SURVEY TRAINING30
• Structure of each work day
• Safety precautions• How to introduce
survey• How to document
responses from potential survey respondents
Training Process
April 28, 2014 DC Promise Neighborhood Initiative and The Urban Institute
31
Kenilworth-Parkside
Community Resident
Advantages of Two-person Teams• Increased likelihood of “open doors”• Decreased # of survey days needed• Easy navigation of community• Division of labor• Two people available to answer questions• Balance between confidentiality and trust• Increased safety
April 28, 2014 DC Promise Neighborhood Initiative and The Urban Institute
32
Challenges of Two-person Teams
• Massive recruiting and hiring efforts • DCPNI staff capacity• Difficult to ensure fidelity • Downtime during surveys• Answering questions about survey when
approached on street
Kenilworth-Parkside
Community Resident
April 28, 2014 DC Promise Neighborhood Initiative and The Urban Institute
33
Step 2: Prioritize cultural sensitivity
• Survey questions tested with community
• Community feedback encouraged throughout
• Survey administrators participated in focus group
• Concerns from community members noted for next survey
April 28, 2014 DC Promise Neighborhood Initiative and The Urban Institute
34
Step 3: Add Unique Value
• I heart data booklet provides data snapshot
• Data now exists on topics for which there are no other sources
• Ongoing conversations with the K-P community about data
April 28, 2014 DC Promise Neighborhood Initiative and The Urban Institute
35April 28, 2014 DC Promise Neighborhood Initiative and The Urban Institute
Step 4: Build Relationships
36April 28, 2014 DC Promise Neighborhood Initiative and The Urban Institute
• Impacts of Kenilworth-Parkside neighborhood survey:
Inclusive data collection
Gained community support
Built relationships
Culturally sensitive survey
Unique data collected
Results: Community Support for Survey:
DC Promise Neighborhood Initiative and The Urban Institute
37
Data Collection in the 21st Century: The Use of Tablet Computers for a
Neighborhood Survey
April 28, 2014
Maia WoluchemResearch Assistant
Metropolitan Housing and Communities Policy Center
Urban [email protected]
DC Promise Neighborhood Initiative and The Urban Institute
38
Challenges for Our Survey
• Several teams approaching hundreds of doorsteps over several points in time
• Each respondent was different from the last• Each interviewer is different from the last• Presenting several challenges– Monetary costs– Confidentiality– Standardization– Flexibility– Ease
April 28, 2014
DC Promise Neighborhood Initiative and The Urban Institute
39
Paper vs Plastic
April 28, 2014
Feature Paper Surveys Tablet Surveys
MoneyPrinting supplies,
associated materials, postsurvey dataentry
$200 per tablet, KeySurveysubscription
ConfidentialityMaintaining security of
documents in field, postsurvey
Tablet’s secure connection and privacy capabilities
Standardization Interview cues
Flexibility May involve skipping pages Skip logic, conditionality
Ease
DC Promise Neighborhood Initiative and The Urban Institute
40
Program
• Used KeySurvey as our survey platform
– Allowed ability to build conditional sections and complicated skip patterns
– Allowed personalization based on household traits• Cost-reducing in terms of time, money and flexibility
April 28, 2014
DC Promise Neighborhood Initiative and The Urban Institute
41
Back-End v. Front-End• Level one: 159 questions and hundreds of
answer choices• Level two: Variable names
April 28, 2014
DC Promise Neighborhood Initiative and The Urban Institute
42
Back-End v. Front-End
• Level Three: Skip patterns– Sometimes a simple skip:
April 28, 2014
DC Promise Neighborhood Initiative and The Urban Institute
43
Back-End v. Front-End
• Sometimes it’s something else entirely:
April 28, 2014
DC Promise Neighborhood Initiative and The Urban Institute
44
Back-End v. Front-End
• Everything was automated, making things easy for everyone—
• Setting:
April 28, 2014
DC Promise Neighborhood Initiative and The Urban Institute
45
Back-End v. Front-End (Consent)
April 28, 2014
DC Promise Neighborhood Initiative and The Urban Institute
46
Back-End v. Front-End (Personalization)
April 28, 2014
DC Promise Neighborhood Initiative and The Urban Institute
47
Back-End v. Front-End• Once programmed, survey looked like this—
April 28, 2014
DC Promise Neighborhood Initiative and The Urban Institute
48
Back-End v. Front-End
April 28, 2014
DC Promise Neighborhood Initiative and The Urban Institute
49
Administration:Who conducted the survey?
• AmeriCorps members and college volunteers– Read aloud the questions to the householder– Householder responds verbally
• Why?– Literacy rates among the community– Unfamiliarity with the technology– Standardization of questions and answers
April 28, 2014
DC Promise Neighborhood Initiative and The Urban Institute
50
Training
• Back-end was sophisticated but front-end was very user friendly.
• Training session has some ground rules:– Trust the programming– Don’t be pushy
• And some golden rules:– Read the exact wording of the question as presented– Obtain consent – Work with your partner
April 28, 2014
DC Promise Neighborhood Initiative and The Urban Institute
51
Partnerships in Action
• Morning Briefing (DCPNI Offices) – Community Resident:• Receive ten addresses, gift cards (4), survey receipts,
door hangers, and a walkie talkie– Interviewer:• Receive same ten addresses and a tablet computer
• Walk– Resident: Fill out tracking form to document
address condition
April 28, 2014
DC Promise Neighborhood Initiative and The Urban Institute
52
Partnerships in Action• Doorstep
– Community Resident:• Explain survey and present opportunity for giftcard• Answer questions householder may have• Document what happens
– Interviewer: • Obtain consent • Administer survey
• Administration (Interviewer & Community Resident)– At a comfortable place– Resident Role
April 28, 2014
DC Promise Neighborhood Initiative and The Urban Institute
53
Partnerships in Action
• Post-Survey– Community Resident:• Offer respondent choice of incentives• Everyone signs receipt
• What if no one is home? – Resident:• Leave door hanger• Document address so that another
team could make attempt
April 28, 2014
DC Promise Neighborhood Initiative and The Urban Institute
54
Tablets are not Without Challenges
• What isn’t done by hand must be done by computer– Careful time-intensive programming necessary
• Technical difficulties– Potential for tablet failure (need for backup tablets)
• Post-launch errors are data errors– Examples of missing answers or misplaced variables– Typing answers in leads to some data quality errors
April 28, 2014
DC Promise Neighborhood Initiative and The Urban Institute
55
But the Benefits?
• Benefits greater than challenges (in our case)• Data/Tablet Security– End of day wipe and upload to UI SFTP– Confidential drives at UI– Cerberus for tablet security
• Substantial results– Over four weekends in October and November,
got 444 responses, a response rate of 59% overall
April 28, 2014
DC Promise Neighborhood Initiative and The Urban Institute
56
Audience Q&A
April 28, 2014
Isaac CastilloDirector of Data and Evaluation
DC Promise Neighborhood Initiative
@isaac_outcomes
Samantha GreenbergData and Evaluation Specialist
DC Promise Neighborhood Initiative
@eval_revolution
Maia WoluchemResearch Assistant
Metropolitan Housing and Communities Policy Center
Urban Institute
Megan GallagherResearch Associate
Metropolitan Housing and Communities Policy Center
Urban Institute
Top Related