REDUCING CELL PHONE COVERAGEBIAS IN GEOGRAPHICALLY TARGETEDRDD SAMPLES BY WEIGHTING FORRESIDENTIAL MOBILITY
Michael Sanderson, RachelMartonik, Tara Merry, StephenImmerwahr, Andy Weiss,Michael Battaglia, JoshAppelbaum
Acknowledgements
NYC Department of Health and Mental HygieneMichael SandersonStephen Immerwahr
Abt-SRBITara MerryAndy WeissRachel MartonikJosh Appelbaum
Battaglia Consulting Group, LLCMichael Battaglia
2
Outline
1. RDD telephone sample coverage2. In area, Out of frame problem for local area surveys3. Potential scope and bias4. Weighting recent movers as a proxy5. Conclusions and recommendations
3
RDD telephone sample coverage
Dual frame (landline, cell) RDD telephone samplesprovide excellent coverage for national surveys2.6% non-telephone households Interruption in phone service adjustment (Keeter, 1995;
Frankel et al, 1999; Srinath et al, 2009)
Local area dual frame studies have less coveragethan nationalNon-telephone households (varies by location)Cell phone only (CPO) with numbers from outside area,
which we’re calling “In-area, out-of-frame”
4
In area, Out of frame problem for local areasurveysCellular RDD sample for local area surveys excludeCPO with out-of-area telephone numbers. “In-area, out-of-frame” users moved into an area and did
not change their cell number Easy to not change Growing problem Magnified in some areas
Varies by local area, can be hard to quantify
Solutions are limited Cast wider net and screen on geography Use of appended sample data (billing ZIP code)
5
In area, out of frame problem for local areasurveys: potential scope in NYC Limited research on local area frame noncoverage NY state residents with non-NY state cell phone ~13.5%
(Pierannunzi et al, 2013)
NYC is 35.7% Cell Phone Only (HVS)Assuming NYS estimate applies, 4.8% missing from dual frameVaries within borough
3.2% of NYC adults moved to NYC in the past 12 months(ACS 2011-2013)Undercoverage accumulates
6
Research Questions
What is the potential scope and bias of this problem?
Can weighting on mover status reduce bias fromexclusion of “in area, out-of-frame” households?
Hypotheses Non-coverage is related to mover status ACS mobility question can be used to determine if recent movers are
under-represented Recent movers covered by the frame may be a reasonable proxy
for recent movers that are excluded from the frame Weighting by mover status reduces frame coverage error
7
NYC Community Health Survey (CHS)
Used to understand and improve the health of NewYorkers, to allocate funds and evaluate health initiatives
Annual RDD health surveillance survey conducted among a cross-section of non-institutionalized adults in NYC. Conducted since2002
BRFSS-based methodology
~10,000 interviews/year
Provides estimates at the city, borough (county), and neighborhoodlevels
8
Mobility question
Mobility questions from ACS included in 2014 CHSDid you live in this house or apartment 1 year ago?
IF NO:Where were you living 1 year ago? Was it a house
or apartment:
1 In one of the five boroughs of New York City
2 Outside of New York City but in New York State,
3 In the U.S. but not in New York State (excludes
Puerto Rico), OR
4 Outside of the U.S. (includes Puerto Rico)
9
Analysis
Are recent movers underrepresented in the sample?
How are recent movers different from non-movers? telephone use demographics health measures
Does weighting by mover status reduce bias in healthestimates? health measures overall health measures among recent movers
10
Representation of Recent Movers (unweighted)
Recent movers are somewhat under-represented insample, particularly in Manhattan.
% of NYC residents/respondents who lived outside of NYC 1 year ago:ACS vs. CHS
Bronx Bklyn Mhtn QueensStatenIsland Total
ACS PUMS2.2% 2.3% 6.5% 2.7% 1.3% 3.2%
2014 CHS(unweighted)
1.6% 2.1% 1.9% 2.3% 1.0% 1.9%
11
Phone usage among recent movers in sample(unweighted)Recent movers are highly likely to be CPOs, especially in
Manhattan Vs. city-wide CPO rate = 35.7%Phone use by borough among CHS respondent who lived outside NYC 1year ago
Bronx Bklyn Mhtn QueensStatenIsland Total
Dual Users 40.0% 37.2% 37.5% 44.4% 83.3% 41.6%
CPO 60.0% 53.5% 62.5% 55.6% 16.7% 55.5%
LLO 0.0% 9.3% 0.0% 0.0% 0.0% 2.9%
12
Who are recent movers? (weighted)
6.6%
8.6%
32.8%
24.0%
46.1%
12.5%
16.3%
44.1%
50.1%
53.4%
Student
Unemployed
College degree
Under 30
Male
Movers Non-movers
13
Who are recent movers? (weighted)
35.7%
23.4%
13.3%
47.9%
40.7%
39.2%
30.4%
20.4%
63.0%
57.0%
Non-Eng. at home
Non-Eng. Int
Asian
Foreign born
Nvr md. or living with partner
Movers Non-movers
14
Who are recent movers? Health profiles(weighted)
15
13.9%
58.4%
38.0%
9.7%
13.5%
18.4%
42.6%
53.1%
9.4%
28.7%
Smoker
Overweight/obese
PA past 7 days
Didn't get care
Uninsured
Movers Non-movers
Who are recent movers? Health profiles(weighted)
16.2%
33.5%
64.1%
32.8%
49.2%
65.9%
Binge drinking
Condom at last sex
Ever had HIV test
Movers Non-movers
16
New weight: Adding mobility
Standard weighting for CHS:Design weights – base sample weights, probability of selectionRaking to demographic variables Neighborhood level: Sex, age, race/ethnicity (Census 2010
Summary Files 1 and 2) Borough level: education, marital status, # of adults, presence of
children (ACS PUMS) Borough level: telephone usage group (NYC Housing and
Vacancy Survey)
Added citywide weighting variable: recent moversRaking to 5 mobility groups: non-movers, past 12 month movers
within NYC, movers to NYC from NY state, movers to NYC fromUS, movers to NYC from outside US
17
Recent Movers Weighting Adjustment
Usual CHS weighting makes up for most recent moverunder-representation Except for movers from another state
% of recent movers from outside NYC
ACSPUMS
2014 CHSunweighted
2014 CHSStandard
2014 CHSMoverweight
NY State 0.5% 0.4% 0.6% 0.5%
Other state 1.6% 0.6% 0.8% 1.5%
Outside U.S. (includes PR) 1.1% 0.9% 1.2% 1.1%
Total Movers 3.2% 1.9% 2.6% 3.2%
18
Impact of Weighting for Movers: Overall
13.99%
57.95%
38.33%
13.94%
13.95%
58.00%
38.34%
13.89%
Smoker
Overweight/obese
PA past 7 days
Uninsured
Old wt Mover wt
19
Impact of Weighting for Movers: Overall
16.63%
34.08%
64.15%
13.74%
16.66%
33.93%
64.10%
13.67%
Binge drinking
Condom at last sex
Ever had HIV test
Depression diagnosis
Old wt Mover wt
20
Impact of Weighting for Movers: Movers only
18.4%
40.9%
56.0%
27.3%
19.5%
42.6%
53.1%
28.7%
Smoker
Overweight/obese
PA past 7 days
Uninsured
Old wt Mover wt
21
Impact of Weighting for Movers: Movers only
31.7%
50.8%
67.7%
11.2%
32.8%
49.2%
65.9%
9.1%
Binge drinking
Condom at last sex
Ever had HIV test
Depression diagnosis
Old wt Mover wt
22
Summary & Conclusions
Recent movers are differentDemographic and health differencesIn-area movers in frame may be similar to those missed by the
frame
ACS provides data on the size of the recent moverpopulation Usual weighting made up for most recent mover under-
representation Review ACS mover data for your geographic area Local area surveys with large recent mover and/or CPO
populations may benefit from mover adjustment
Think about adding mobility questions to your survey
23
Future Directions
Other potential solutions:
Targeted cellular sample (MSG) NYC: 4.8% noncoverage x 6.2M = 300K adults Vs. 74K in MSG sample Sampled in 2015 for comparison
Future research Compare recent movers with in-area vs. out-of-area telephone
numbers (if both groups are available) Seek other sources of data to estimate scope of problem
24
Thank You!Keeter, S. (1995). Estimated Noncoverage Bias from a Phone Survey.Public Opinion quarter, 59, 196-217.
Frankel, M.R., Srinath, K.P., Battaglia, M.P., Hoaglin, D.C., Wright, R.A.,and Smith, P.J. (1999). Reducing Nontelephone Bias in RDD Surveys.Proceedings of the American Statistical Association, Section on SurveyResearch Methods, Alexandria, VA, 934-939.
Srinath, K.P., Frankel, M.R., Hoaglin, D.C., and Battaglia, M.P. (2009).Compensating for Noncoverage of Nontelephone Households inRandom-Digit Dialing Surveys: A Comparison of Adjustments Based onPropensity Scores and Interruptions in Telephone Service. Journal ofOfficial Statistics, Vol. 25, No. 1
25
Top Related