REDUCING CELL PHONE COVERAGEBIAS IN GEOGRAPHICALLY TARGETEDRDD SAMPLES BY WEIGHTING FORRESIDENTIAL MOBILITY

Michael Sanderson, RachelMartonik, Tara Merry, StephenImmerwahr, Andy Weiss,Michael Battaglia, JoshAppelbaum

Acknowledgements

NYC Department of Health and Mental HygieneMichael SandersonStephen Immerwahr

Abt-SRBITara MerryAndy WeissRachel MartonikJosh Appelbaum

Battaglia Consulting Group, LLCMichael Battaglia

2

Outline

1. RDD telephone sample coverage2. In area, Out of frame problem for local area surveys3. Potential scope and bias4. Weighting recent movers as a proxy5. Conclusions and recommendations

3

RDD telephone sample coverage

Dual frame (landline, cell) RDD telephone samplesprovide excellent coverage for national surveys2.6% non-telephone households Interruption in phone service adjustment (Keeter, 1995;

Frankel et al, 1999; Srinath et al, 2009)

Local area dual frame studies have less coveragethan nationalNon-telephone households (varies by location)Cell phone only (CPO) with numbers from outside area,

which we’re calling “In-area, out-of-frame”

4

In area, Out of frame problem for local areasurveysCellular RDD sample for local area surveys excludeCPO with out-of-area telephone numbers. “In-area, out-of-frame” users moved into an area and did

not change their cell number Easy to not change Growing problem Magnified in some areas

Varies by local area, can be hard to quantify

Solutions are limited Cast wider net and screen on geography Use of appended sample data (billing ZIP code)

5

In area, out of frame problem for local areasurveys: potential scope in NYC Limited research on local area frame noncoverage NY state residents with non-NY state cell phone ~13.5%

(Pierannunzi et al, 2013)

NYC is 35.7% Cell Phone Only (HVS)Assuming NYS estimate applies, 4.8% missing from dual frameVaries within borough

3.2% of NYC adults moved to NYC in the past 12 months(ACS 2011-2013)Undercoverage accumulates

6

Research Questions

What is the potential scope and bias of this problem?

Can weighting on mover status reduce bias fromexclusion of “in area, out-of-frame” households?

Hypotheses Non-coverage is related to mover status ACS mobility question can be used to determine if recent movers are

under-represented Recent movers covered by the frame may be a reasonable proxy

for recent movers that are excluded from the frame Weighting by mover status reduces frame coverage error

7

NYC Community Health Survey (CHS)

Used to understand and improve the health of NewYorkers, to allocate funds and evaluate health initiatives

Annual RDD health surveillance survey conducted among a cross-section of non-institutionalized adults in NYC. Conducted since2002

BRFSS-based methodology

~10,000 interviews/year

Provides estimates at the city, borough (county), and neighborhoodlevels

8

Mobility question

Mobility questions from ACS included in 2014 CHSDid you live in this house or apartment 1 year ago?

IF NO:Where were you living 1 year ago? Was it a house

or apartment:

1 In one of the five boroughs of New York City

2 Outside of New York City but in New York State,

3 In the U.S. but not in New York State (excludes

Puerto Rico), OR

4 Outside of the U.S. (includes Puerto Rico)

9

Analysis

Are recent movers underrepresented in the sample?

How are recent movers different from non-movers? telephone use demographics health measures

Does weighting by mover status reduce bias in healthestimates? health measures overall health measures among recent movers

10

Representation of Recent Movers (unweighted)

Recent movers are somewhat under-represented insample, particularly in Manhattan.

% of NYC residents/respondents who lived outside of NYC 1 year ago:ACS vs. CHS

Bronx Bklyn Mhtn QueensStatenIsland Total

ACS PUMS2.2% 2.3% 6.5% 2.7% 1.3% 3.2%

2014 CHS(unweighted)

1.6% 2.1% 1.9% 2.3% 1.0% 1.9%

11

Phone usage among recent movers in sample(unweighted)Recent movers are highly likely to be CPOs, especially in

Manhattan Vs. city-wide CPO rate = 35.7%Phone use by borough among CHS respondent who lived outside NYC 1year ago

Bronx Bklyn Mhtn QueensStatenIsland Total

Dual Users 40.0% 37.2% 37.5% 44.4% 83.3% 41.6%

CPO 60.0% 53.5% 62.5% 55.6% 16.7% 55.5%

LLO 0.0% 9.3% 0.0% 0.0% 0.0% 2.9%

12

Who are recent movers? (weighted)

6.6%

8.6%

32.8%

24.0%

46.1%

12.5%

16.3%

44.1%

50.1%

53.4%

Student

Unemployed

College degree

Under 30

Male

Movers Non-movers

13

Who are recent movers? (weighted)

35.7%

23.4%

13.3%

47.9%

40.7%

39.2%

30.4%

20.4%

63.0%

57.0%

Non-Eng. at home

Non-Eng. Int

Asian

Foreign born

Nvr md. or living with partner

Movers Non-movers

14

Who are recent movers? Health profiles(weighted)

15

13.9%

58.4%

38.0%

9.7%

13.5%

18.4%

42.6%

53.1%

9.4%

28.7%

Smoker

Overweight/obese

PA past 7 days

Didn't get care

Uninsured

Movers Non-movers

Who are recent movers? Health profiles(weighted)

16.2%

33.5%

64.1%

32.8%

49.2%

65.9%

Binge drinking

Condom at last sex

Ever had HIV test

Movers Non-movers

16

New weight: Adding mobility

Standard weighting for CHS:Design weights – base sample weights, probability of selectionRaking to demographic variables Neighborhood level: Sex, age, race/ethnicity (Census 2010

Summary Files 1 and 2) Borough level: education, marital status, # of adults, presence of

children (ACS PUMS) Borough level: telephone usage group (NYC Housing and

Vacancy Survey)

Added citywide weighting variable: recent moversRaking to 5 mobility groups: non-movers, past 12 month movers

within NYC, movers to NYC from NY state, movers to NYC fromUS, movers to NYC from outside US

17

Recent Movers Weighting Adjustment

Usual CHS weighting makes up for most recent moverunder-representation Except for movers from another state

% of recent movers from outside NYC

ACSPUMS

2014 CHSunweighted

2014 CHSStandard

2014 CHSMoverweight

NY State 0.5% 0.4% 0.6% 0.5%

Other state 1.6% 0.6% 0.8% 1.5%

Outside U.S. (includes PR) 1.1% 0.9% 1.2% 1.1%

Total Movers 3.2% 1.9% 2.6% 3.2%

18

Impact of Weighting for Movers: Overall

13.99%

57.95%

38.33%

13.94%

13.95%

58.00%

38.34%

13.89%

Smoker

Overweight/obese

PA past 7 days

Uninsured

Old wt Mover wt

19

Impact of Weighting for Movers: Overall

16.63%

34.08%

64.15%

13.74%

16.66%

33.93%

64.10%

13.67%

Binge drinking

Condom at last sex

Ever had HIV test

Depression diagnosis

Old wt Mover wt

20

Impact of Weighting for Movers: Movers only

18.4%

40.9%

56.0%

27.3%

19.5%

42.6%

53.1%

28.7%

Smoker

Overweight/obese

PA past 7 days

Uninsured

Old wt Mover wt

21

Impact of Weighting for Movers: Movers only

31.7%

50.8%

67.7%

11.2%

32.8%

49.2%

65.9%

9.1%

Binge drinking

Condom at last sex

Ever had HIV test

Depression diagnosis

Old wt Mover wt

22

Summary & Conclusions

Recent movers are differentDemographic and health differencesIn-area movers in frame may be similar to those missed by the

frame

ACS provides data on the size of the recent moverpopulation Usual weighting made up for most recent mover under-

representation Review ACS mover data for your geographic area Local area surveys with large recent mover and/or CPO

populations may benefit from mover adjustment

Think about adding mobility questions to your survey

23

Future Directions

Other potential solutions:

Targeted cellular sample (MSG) NYC: 4.8% noncoverage x 6.2M = 300K adults Vs. 74K in MSG sample Sampled in 2015 for comparison

Future research Compare recent movers with in-area vs. out-of-area telephone

numbers (if both groups are available) Seek other sources of data to estimate scope of problem

24

Thank You!Keeter, S. (1995). Estimated Noncoverage Bias from a Phone Survey.Public Opinion quarter, 59, 196-217.

Frankel, M.R., Srinath, K.P., Battaglia, M.P., Hoaglin, D.C., Wright, R.A.,and Smith, P.J. (1999). Reducing Nontelephone Bias in RDD Surveys.Proceedings of the American Statistical Association, Section on SurveyResearch Methods, Alexandria, VA, 934-939.

Srinath, K.P., Frankel, M.R., Hoaglin, D.C., and Battaglia, M.P. (2009).Compensating for Noncoverage of Nontelephone Households inRandom-Digit Dialing Surveys: A Comparison of Adjustments Based onPropensity Scores and Interruptions in Telephone Service. Journal ofOfficial Statistics, Vol. 25, No. 1

msander1@health.nyc.gov

t.merry@srbi.com

25