Navigating Reliability of Small Area Data Lessons from New York … · 2018. 10. 26. · NYC Count...

Presentation for

American Community Survey

Data Users Conference

May 29, 2014

Joel A. Alvarez & Joseph J. Salvo

Population Division

Navigating Reliability of Small Area Data

Lessons from New York City

© 2012 Population Reference Bureau. All rights reserved. www.prb.org

Reliability of ACS Poverty Data

Across Tracts of Varying Size

PRB | INFORM, EMPOWER, ADVANCE

What are acceptable levels of relative reliability

when using ACS estimates?

Remember:

Income of $60,000

with CV of 20 percent

has MoE of +/-$20,000

“CVs of 10-12 percent or less are often accepted as a reasonable

standard of precision for an estimate.”

– Using the American Community Survey: Benefits and Challenges, 2007

“While there is no hard-and-fast rule, for the purposes of this

handbook, estimates with CVs of more than 15 percent are

considered cause for caution when interpreting patterns in the

data.”

– Census Bureau’s ACS Compass Handbook Series, 2009

E New York

Canarsie

Great Kills

St Albans

Bay Rdg

S Ozone Pk

Rosedale

Steinway

Flatlands

Queens Vlg

Forest Hls

Bellerose

Jamaica

Whitestone

Hunts Pt

Astoria

Middle Vlg

Westerleigh

Flatbush

Murray Hl

College Pt

Arden Hts

Borough Pk

Flushing

Baisley Pk

Ridgewood

Hollis

Jackson Hts

Laurelton

S Jamaica

Crown Hts N

W Vlg

Bedford Richmond Hl

Maspeth

Elmhurst

Midwood

Co-Op City

Glendale

Dyker Hts

Greenpoint

Madison

Bensonhurst W

E Flushing

Woodhaven

Gravesend

Auburndale

Brownsville

Kew Gdns Hls

E Hrlm N

Woodside

Cambria Hts

Ozone Pk

Clinton

Rego Pk

Corona

E Vlg

Ocean Hl

Clinton Hl Stuyvesant Hts

Kew Gdns

E Hrlm S

N Corona

E Elmhurst

Pelham Pkwy

Springfield Gdns N

Crown Hts S

Bayside Bayside Hls

Rossville Woodrow

Far Rckwy Bayswater

Lindenwood Howard Bch

Stapleton Rosebank

Seagate Coney Is

N Side S Side

New Dorp Midland Bch

Pk Slope Gowanus

Oakwood Oakwood Bch

E Flatbush Farragut

Woodlawn Wakefield

Jamaica Ests Holliswood

New Brighton Silver Lake

Mott Haven Prt Morris

Rugby Remsen Vlg

Briarwood Jamaica Hls

Springfield Gdns S Brookville

Allerton Pelham Gdns

Fresh Mdws Utopia

Prospect Lffrts Gdns Wingate

Charleston Richmond Vly

Tottenville

Hunters Pt Sunnyside

W Maspeth

Schuylerville Throgs Nck

Edgewater Pk

Old Town Dongan Hls

S Bch

Douglas Mnr Douglaston Little Nck

Ft Totten Bay Ter Clearvw

N Riverdale Fieldston Riverdale

Sheepshead Bay Gerritsen Bch

MN Bch

Pomonok Flushing Hts

Hillcrest

New Springville Bloomfield

Travis

Todt Hl Emerson Hl Heartlnd Vlg Lghthouse Hl

Annadale Huguenot

Prince's Bay Eltingvl

Mariner's Hbr Arlington Prt Ivory

Graniteville

Sunset Pk W

Oakland Gdns

Homecrest

E Williamsburg

Bensonhurst E

Upper W Side

Sunset Pk E

Bath Bch

Wash Hts S

Wash Hts N

E Tremont

Lower E Side

Queensboro Hl

Erasmus

Ft Greene

Yorkville

Mt Hope

Norwood

Lincoln Sq

Belmont

V Cortlandt Vlg

Bronxdale

Rikers Island

Old Astoria

Brighton Bch

Chinatown

Ocean Pkwy S

Hamilton Hts

Morningside Hts

Windsor Ter

Central Hrlm S

Longwood

Parkchstr

Williamsburg

Kingsbridge Hts

Prospect Hts

W Brighton

Gramercy

Manhattanville

Fordhm S

DUMBO Vinegar Hl

Dwntwn BK Boerum Hl

Starrett City

E New York (PA Ave)

Willmsbridge Olinville

Midtown Midtown S

Cypress Hls City Line

Westchstr Unionprt

Elmhurst Maspeth

E Conc Conc Vlg

Clarmnt Bathgate

W Farms Bronx Riv

Upper E Side Carnegie Hl

Turtle Bay E Midtown

University Hts Morris Hts

Spuyten Duyvil Kingsbridge

Morrisania Melrose

Bedfrd Pk Fordhm N

Marble Hl Inwood

Soundview Bruckner

Murray Hl Kips Bay

Lenox Hl Roosevelt Is

Kensington Ocean Pkwy

Battery Pk City Lower MN

Melrose S Mott Haven N

BK Hts Cobble Hl

Stuy Town Cooper Vlg

Eastchstr Ednwld Baychstr

Carroll Gdns Columbia St

Red Hook

Grymes Hl Clifton Fox Hls

Glen Oaks Floral Pk

New Hyde Pk

W New Brighton New Brighton

St George

Van Nest Morris Pk

Westchtr Sq

Grasmere Arrochar

Ft Wadsworth

Pelham Bay Country Club

City Island

Queensbridge Ravenswood

LIC

Hammels Arverne

Edgemere

Soundview Castle Hl Clason Pt

Harding Pk

Breezy Pt Belle Hbr Rckwy Pk

Broad Channel

Hudson Yrds Chelsea Flat Iron Union Sq

SoHo TriBeCa Civic Ctr

Little Italy

Grgtwn Marine Pk Bergen Bch Mill Basin

Neighborhood Tabulation Areas (NTAs)

New York City

Neighborhood Tabulation Areas or NTAs, are

aggregations of census tracts that are subsets of

New York City's 55 Public Use Microdata Areas

(PUMAs). Primarily due to these constraints,

NTA boundaries and their associated names may

not definitively represent neighborhoods.

LAGUARDIA AIRPORT

JOHN F. KENNEDY INTERNATIONAL AIRPORT

Bronx

Queens

Manhattan

Brooklyn

Staten

Island

West Village

NTAs are composed

of Census Tracts

NTAs

NYC Count = 188

Median Population = 36,600

Median # HUs = 15,100

Source: U.S. Census Bureau, American Community Survey, 2006-2010 Summary File

Population Division-New York City Department of City Planning

0

10

20

30

40

Nonrelative Population Below Poverty Level 1 Unit, Detached

Housing

Census Tracts

Neighborhoods

Median Coefficients of Variation for Selected Variables

NYC Census Tracts and Neighborhoods (NTAs), 2008-2012 ACS

Co

eff

icie

nt

of

Vari

ati

on

Ratio of NTA CV to

Census Tract CV = 0.34 0.31 0.33

37.3

11.8

36.3

9.0

28.6

12.6

Note:

CVs for tract estimates of zero were omitted.

For NTA aggregation, MOEs associated with tract estimates of zero were converted to zero. Source: U.S. Census Bureau, 2008-2012 American Community Survey-Summary File


Coefficients of Variation (CV) for Population Living Below Poverty

New York City Census Tracts & Neighborhoods (NTAs), 2008-2012

Source: U.S. Census Bureau, 2008-2012 American Community Survey-Summary File Population Division-New York City Department of City Planning

CV

Tract Measure of Size (TMOS) Sampling Rates

2009 and 2013 American Community Surveys

Source: U.S. Census Bureau, American Community Survey, ACS Design and Methodology Report 2009 and 2013


2009 Average of 2013 Average of Ratio of 2013

Tract Measure of Size Post-reduction Post-reduction Sampling Rate to

In Housing Units Sampling Rates Sampling Rates 2009 Sampling Rate

0

0

5,000

10,000

15,000

20,000

25,000

30,000

35,000

40,000

Tra

ct

Co

un

t

Mean 1,780

Median 1,691

SD 803

Range 13,585

Sum 130,038,080

Count 73,057

1st Quartile 1,239

3rd Quartile 2,232

US Tract Housing Unit Summary Stats

Tract Measure of Size (TMOS) Strata Tract Counts

United States, 2006-2010 ACS



1.1% 0.9%

12.1%

51.3%

of all

Census

Tracts

33.5%

0.9% 0.1%

Effect of New ACS Sample Design on Sample Size & Weight Variability

New York City Housing Estimates*, 2006-2010



*Excludes tracts with estimates of zero and those in parks, airports, and Rikers Island Neighborhood Tabulation Areas (NTAs)

Estimate

Unweighted

Sample

Percent in

Sample

Average

Weight

Average

Weight

by Tract

SD of

Tract

Weight

2006-2010 data 3,342,799 200,343 6.0 16.7 16.3 2.6

Simulated New Design 3,342,799 233,390 7.0 14.3 13.7 6.9

Percent Change - 16.5 16.5 -14.2 -16.0 160.9

Housing Units

0

5

10

15

20

25

30

Pe

rc

en

t In

cre

ase

in

M

oE

Percent Increase in MoEs (When Aggregating Tract Estimates) Attributable to

Differential Weights Used in Old vs. New ACS Sample Design, 2006-2010

New York City Neighborhood Tabulation Areas (NTAs)



Note - In this analysis tract MoEs for estimates of zero were adjusted to zero.

Tracts in parks, airports, and Rikers Island NTAs were excluded.

New Design

Average

=4.8%

Old Design

Average

=0.9%

Old

D

esig

n

Ne

w D

esig

n

Bronx

New Avg=4.0%

Old Avg=0.9%

Brooklyn

New Avg=3.9%

Old Avg=0.8%

Manhattan

New Avg=7.0%

Old Avg=0.7%

Queens

New Avg=5.3%

Old Avg=0.9%

Staten Is.

New Avg=4.1%

Old Avg=1.0%

Fresh Meadows

Utopia (26%)

Born in China = 331,616

Percent Change in Coefficients of Variation Due to Differential Weights

For Married Couple Families Variable, 2006-2010

New York City Neighborhood Tabulation Areas





Decrease of -20% or more (0)

-19.9% to -10.0% (0)

-9.9% to -5.0% (0)

-4.9% to -2.0% (0)

-1.9% to 1.9% (53)

2.0% to 4.9% (86)

5.0% to 9.9% (37)

10.0% to 19.9% (11)

Increase of 20% or more (1 NTA)

Percent Change in CV


Percent of Housing Units in Sample

Less than 5% (11)

5.0% to 7.4% (25)

7.5% to 9.9% (87)

10.0% to 12.4% (41)

12.5% to 14.9% (16)

15% or more (8 NTAs)

Percent of Housing Units in the Sample According to New ACS Sample Design

New York City Neighborhoods (NTAs), 2006-2010



Note - In this analysis tracts with Housing Unit estimates of zero and

tracts in parks, airports, and Rikers Island were excluded.

Bronx

6.8%

+18.7%

New York City

% of HUs in Sample = 7.0%

% Change in Sample Size

w/ New Design = +16.5%

Queens

8.2%

+30.9%

Manhattan

4.6%

-17.8%

Brooklyn

8.0%

+29.6%

Staten

Island

7.2%

+15.4%


Percent Change in CVs Due to Differential Weights and Sample Size Changes

For Married Couple Families Variable, 2006-2010

New York City Neighborhood Tabulation Areas





Decrease of -20% or more (9)

-19.9% to -10.0% (39)

-9.9% to -5.0% (51)

-4.9% to -2.0% (37)

-1.9% to 1.9% (21)

2.0% to 4.9% (5)

5.0% to 9.9% (7)

10.0% to 19.9% (7)

Increase of 20% or more (12 NTAs)

Percent Change in CV

Suggestions for the Census Bureau

• Provide more guidance on acceptable levels of reliability

(examples)

• More guidance on aggregation of summary level ACS data

• Reconsider the impact of new sample design on data

aggregation

• Longer-term: Press for further increases in sample size and/or

endorse new small area geography

Data Links

• DCP Population Division website:

www.nyc.gov/population

• New York City Census FactFinder:

maps.nyc.gov/census/

Navigating Reliability of Small Area Data Lessons from New York … · 2018. 10. 26. · NYC Count...

Documents

Transcript of Navigating Reliability of Small Area Data Lessons from New York … · 2018. 10. 26. · NYC Count...