Making Use of Big Data
-
Upload
econsult-solutions -
Category
Economy & Finance
-
view
289 -
download
0
Transcript of Making Use of Big Data
![Page 1: Making Use of Big Data](https://reader031.fdocuments.us/reader031/viewer/2022011721/587d710d1a28ab32318b7d5b/html5/thumbnails/1.jpg)
Making Use of Big Data What You Can Learn from Detailed Real Estate Data
October 20, 2015
PETER ANGELIDES, PHD, AICPPRINCIPALECONSULT SOLUTIONS, INC.PHILADELPHIA, PA
![Page 2: Making Use of Big Data](https://reader031.fdocuments.us/reader031/viewer/2022011721/587d710d1a28ab32318b7d5b/html5/thumbnails/2.jpg)
TAKEAWAYS• Lots of data, and more coming. Some
of it is very cool.
• Canned programs can help, and may be all you need
• Custom analysis can answer very specific questions, but is difficult and expensive
![Page 3: Making Use of Big Data](https://reader031.fdocuments.us/reader031/viewer/2022011721/587d710d1a28ab32318b7d5b/html5/thumbnails/3.jpg)
Overview
• Trends in Big Data availability
• Examples of planning-related Big Data usage
• Real Estate- specific data sources and uses
![Page 4: Making Use of Big Data](https://reader031.fdocuments.us/reader031/viewer/2022011721/587d710d1a28ab32318b7d5b/html5/thumbnails/4.jpg)
Source: SINTEF, http://www.sciencedaily.com/releases/2013/05/130522085217.htm
More than
90% of the world’s data has
been generated since 2011
![Page 5: Making Use of Big Data](https://reader031.fdocuments.us/reader031/viewer/2022011721/587d710d1a28ab32318b7d5b/html5/thumbnails/5.jpg)
Creation of Big Data
Source: IBM 2012, https://www.ibm.com/annualreport/2013/bin/assets/2013_ibm_annual.pdf
![Page 6: Making Use of Big Data](https://reader031.fdocuments.us/reader031/viewer/2022011721/587d710d1a28ab32318b7d5b/html5/thumbnails/6.jpg)
Big Data
• Trend towards big, “open” data
• Data available from municipal sources, corporate sources, and more
• Uses are varied and creative
![Page 7: Making Use of Big Data](https://reader031.fdocuments.us/reader031/viewer/2022011721/587d710d1a28ab32318b7d5b/html5/thumbnails/7.jpg)
Federal/Regional Sources
• Census (LEHD, Decennial, ACS, PUMS)
• Bureau of Labor Statistics
• Department of Transportation
• World Health Org. or World Bank
• MPOs (i.e. DVRPC)
![Page 8: Making Use of Big Data](https://reader031.fdocuments.us/reader031/viewer/2022011721/587d710d1a28ab32318b7d5b/html5/thumbnails/8.jpg)
Municipal Sources
• Individual Office Datasets
- Property Assessments
- Recorder of Deeds
- Police (Crime Incident Data)
- 311 calls
• Munistats (municipal tax rates in PA)
![Page 9: Making Use of Big Data](https://reader031.fdocuments.us/reader031/viewer/2022011721/587d710d1a28ab32318b7d5b/html5/thumbnails/9.jpg)
Municipal SourcesNew Trend: Citywide Open Data
![Page 10: Making Use of Big Data](https://reader031.fdocuments.us/reader031/viewer/2022011721/587d710d1a28ab32318b7d5b/html5/thumbnails/10.jpg)
Private SourcesData Collected by Private Companies
• Uber
• Mastercard
![Page 11: Making Use of Big Data](https://reader031.fdocuments.us/reader031/viewer/2022011721/587d710d1a28ab32318b7d5b/html5/thumbnails/11.jpg)
Private SourcesData for Purchase
• REIS/ CoStar/ IRR (Real Estate)
• NETS (Businesses)
• ESRI (Geo and census)
• STR (Industry)
• Nielson (Consumer data)
![Page 12: Making Use of Big Data](https://reader031.fdocuments.us/reader031/viewer/2022011721/587d710d1a28ab32318b7d5b/html5/thumbnails/12.jpg)
EXAMPLE 1 - PHIPhiladelphia Housing Index
• Assessment Data
• Transaction Data
• Many uses
![Page 13: Making Use of Big Data](https://reader031.fdocuments.us/reader031/viewer/2022011721/587d710d1a28ab32318b7d5b/html5/thumbnails/13.jpg)
Case Study: ESI Philadelphia Housing Index
Philadelphia House Sales
Dataset
Philadelphia Property
Assessment Data (OPA)
Philadelphia Transaction Data
(Recorder of Deeds)
Philadelphia Geographic Data
(Census, ESI, OpenDataPhilly)
![Page 14: Making Use of Big Data](https://reader031.fdocuments.us/reader031/viewer/2022011721/587d710d1a28ab32318b7d5b/html5/thumbnails/14.jpg)
![Page 15: Making Use of Big Data](https://reader031.fdocuments.us/reader031/viewer/2022011721/587d710d1a28ab32318b7d5b/html5/thumbnails/15.jpg)
![Page 16: Making Use of Big Data](https://reader031.fdocuments.us/reader031/viewer/2022011721/587d710d1a28ab32318b7d5b/html5/thumbnails/16.jpg)
Data Merge ProblemsHuman Error/ Dataset Differences
VS
OPA Entries
248 Krams Ave
230 Furley St
9906 Bustleton Ave, Unit C13
1737-39 Chestnut St
Philadox Entries
248 Krams St
230 W Furley St
9906 Bustleton Ave, Unit C-13
1737 Chestnut St
![Page 17: Making Use of Big Data](https://reader031.fdocuments.us/reader031/viewer/2022011721/587d710d1a28ab32318b7d5b/html5/thumbnails/17.jpg)
Data Merge ProblemsTime Lag in Data Entry
Screenshot from Philadox Deed
![Page 18: Making Use of Big Data](https://reader031.fdocuments.us/reader031/viewer/2022011721/587d710d1a28ab32318b7d5b/html5/thumbnails/18.jpg)
Case Study: ESI Philadelphia Housing Index
Philadelphia and Comparative House Price Indices 2003 – 2014 Q3 (2003 = 100)
2003
2003
2003
2003
2004
2004
2004
2004
2005
2005
2005
2005
2006
2006
2006
2006
2007
2007
2007
2007
2008
2008
2008
2008
2009
2009
2009
2009
2010
2010
2010
2010
2011
2011
2011
2011
2012
2012
2012
2012
2013
2013
2013
2013
2014
2014
90
110
130
150
170
190
210
Philadelphia Philadelphia Metro Area National Average
![Page 19: Making Use of Big Data](https://reader031.fdocuments.us/reader031/viewer/2022011721/587d710d1a28ab32318b7d5b/html5/thumbnails/19.jpg)
Case Study: ESI Philadelphia Housing Index
![Page 20: Making Use of Big Data](https://reader031.fdocuments.us/reader031/viewer/2022011721/587d710d1a28ab32318b7d5b/html5/thumbnails/20.jpg)
Case Study: ESI Philadelphia Housing Index
Each dot on the map represents one sale. The color indicates the price per square foot, following the classification shown on the chart below.
More than $200
$150 to $200
$101 to $150
$51 to $100
$26 to $50
$25 or Less
473
406
849
930
456
497
Count of Sales by Price per Square Foot2014, Q3
![Page 21: Making Use of Big Data](https://reader031.fdocuments.us/reader031/viewer/2022011721/587d710d1a28ab32318b7d5b/html5/thumbnails/21.jpg)
Case Study: ESI Philadelphia Housing Index
Subregion House Price Indices 2000 – 2014 Q3 (2003 = 100)
2003
2003
2003
2003
2004
2004
2004
2004
2005
2005
2005
2005
2006
2006
2006
2006
2007
2007
2007
2007
2008
2008
2008
2008
2009
2009
2009
2009
2010
2010
2010
2010
2011
2011
2011
2011
2012
2012
2012
2012
2013
2013
2013
2013
2014
2014
2015
100
120
140
160
180
200
220
240
260
280
Far Northeast Northeast NorthNorthwest Lower North/River Center City/UniversityWest/Southwest South Citywide
Color of Subregion on chart corresponds to color of Subregions on map. Quarterly changes noted on map.
-0.1%
+1.0%-1.3%
+1.6%
+2.8%+1.2%
+0.3%
+5.2%
![Page 22: Making Use of Big Data](https://reader031.fdocuments.us/reader031/viewer/2022011721/587d710d1a28ab32318b7d5b/html5/thumbnails/22.jpg)
EXAMPLE 2 – Open Space
• Similar to PHI
• “Return on Environment” done with DVRPC
• 230,000 arms length transactions
![Page 23: Making Use of Big Data](https://reader031.fdocuments.us/reader031/viewer/2022011721/587d710d1a28ab32318b7d5b/html5/thumbnails/23.jpg)
Case Study: Value of Open Space
$16.3 billion added to the value of
southeastern Pennsylvania’s housing stock
$240 millionIn annual property and transfer tax revenue for
local governments
$133 millionIn costs avoided as a result of the natural provision of
environmental services
$577 millionIn annual benefit for
residents who recreate on protected open space
$795 millionIn avoided medical costs as a
result of recreationthat takes place on
protected open space
6,900 jobscreated on or as a resultof protected open space in the five county region
![Page 24: Making Use of Big Data](https://reader031.fdocuments.us/reader031/viewer/2022011721/587d710d1a28ab32318b7d5b/html5/thumbnails/24.jpg)
Case Study: Value of Open Space
![Page 25: Making Use of Big Data](https://reader031.fdocuments.us/reader031/viewer/2022011721/587d710d1a28ab32318b7d5b/html5/thumbnails/25.jpg)
Case Study: Value of Open Space
![Page 26: Making Use of Big Data](https://reader031.fdocuments.us/reader031/viewer/2022011721/587d710d1a28ab32318b7d5b/html5/thumbnails/26.jpg)
EXAMPLE 3 – LEHDCommuting Patterns
• Works at any geography
• Stratified by income
- $0 - $1,250
- $1250 - $3,333
- >$3,333
![Page 27: Making Use of Big Data](https://reader031.fdocuments.us/reader031/viewer/2022011721/587d710d1a28ab32318b7d5b/html5/thumbnails/27.jpg)
CENSUS - LEHD (ON THE MAP)
Source: U.S. Census 2015, http://onthemap.ces.census.gov/
![Page 28: Making Use of Big Data](https://reader031.fdocuments.us/reader031/viewer/2022011721/587d710d1a28ab32318b7d5b/html5/thumbnails/28.jpg)
CENSUS - LEHD (ON THE MAP)
Source: U.S. Census 2015, http://onthemap.ces.census.gov/
![Page 29: Making Use of Big Data](https://reader031.fdocuments.us/reader031/viewer/2022011721/587d710d1a28ab32318b7d5b/html5/thumbnails/29.jpg)
MUNISTAT
Sources: Munistat - PA DCED (2015), Econsult Solutions (2015)
Combined Resident and School District Earned Income Tax
![Page 30: Making Use of Big Data](https://reader031.fdocuments.us/reader031/viewer/2022011721/587d710d1a28ab32318b7d5b/html5/thumbnails/30.jpg)
LEHD (ON THE MAP) – EIT TAXES
Sources: LEHD 2015 http://onthemap.ces.census.gov, PA DCED (2015) https://www.dced.state.pa.us/systems-and-websites/
Estimated Resident EIT Payments to Sampleville Township under One Percent Resident EIT, County Summary
County of Workplace
Sampleville Resident
Jobs subject to
EIT
Earnings of Sampleville
Resident Jobs subject to EIT
Estimated EIT Payments to Sampleville
Estimated EIT Currently Paid
to Other Municipalities
Percent of Estimated EIT Currently Paid
to Other Municipalities
Bucks 12,152 491,559,798 4,915,598 1,839,721 37%
Montgomery 3,113 148,242,619 1,482,426 1,295,707 87%
Delaware 660 31,330,978 313,310 102,300 33%
Chester 546 25,159,330 251,593 143,338 57%
Northampton 135 5,884,209 58,842 60,458 100%
Other Counties 381 14,555,996 145,560 149,590 100%
Total PA 16,986 716,732,930 7,167,328 3,591,114 50%
![Page 31: Making Use of Big Data](https://reader031.fdocuments.us/reader031/viewer/2022011721/587d710d1a28ab32318b7d5b/html5/thumbnails/31.jpg)
EXAMPLE 4 – ACS / PUMS• Census – 10 years
• ACS
• PUMS
![Page 32: Making Use of Big Data](https://reader031.fdocuments.us/reader031/viewer/2022011721/587d710d1a28ab32318b7d5b/html5/thumbnails/32.jpg)
• Only available in summarized tables
• Small sample size
• Conducted yearly
• Moredetailed data
• Only available as raw data
• Large sample size• Conducted every 10 years• Contains only basic data
PUMS
DECENNIAL
ACS
•Large sample size•Conducted every 10 years•Contains basic, accurate data•Available in summarized tables
Decennial Census
•Small sample size•Conducted yearly•More detailed, less accurate data•Available in summarized tables
ACS (American Community
Survey)
•Raw version of data used to create ACS tables•More data to manipulate but harder to work with•Can be used to create demographic multipliersPUMS (Public
Use Microdata Sample)
SUMMARY OF CENSUS DATA TYPES
![Page 33: Making Use of Big Data](https://reader031.fdocuments.us/reader031/viewer/2022011721/587d710d1a28ab32318b7d5b/html5/thumbnails/33.jpg)
CENSUS - DECENNIAL
• Collected once every 10 years
• Intended to provide accurate population count and basic demographic data
• Sent to every U.S. household
Source: http://www.census.gov/2010census/about/
![Page 34: Making Use of Big Data](https://reader031.fdocuments.us/reader031/viewer/2022011721/587d710d1a28ab32318b7d5b/html5/thumbnails/34.jpg)
CENSUS – AMERICAN COMMUNITY SURVEY (ACS)
• Collected continuously and available in 1, 3, and 5 year aggregations
• Sent to small sample of U.S. households (roughly 3.5 million per year), so all data are estimates
• Intended to provide more detailed but less accurate data.
Source: http://www.census.gov/programs-surveys/acs/guidance.html
![Page 35: Making Use of Big Data](https://reader031.fdocuments.us/reader031/viewer/2022011721/587d710d1a28ab32318b7d5b/html5/thumbnails/35.jpg)
AMERICAN FACTFINDER (ACS AND DECENNIAL)
Source: http://factfinder.census.gov/faces/nav/jsf/pages/searchresults.xhtml?refresh=t
![Page 36: Making Use of Big Data](https://reader031.fdocuments.us/reader031/viewer/2022011721/587d710d1a28ab32318b7d5b/html5/thumbnails/36.jpg)
AMERICAN FACTFINDER (ACS AND DECENNIAL)
Source: http://factfinder.census.gov/faces/nav/jsf/pages/searchresults.xhtml?refresh=t
![Page 37: Making Use of Big Data](https://reader031.fdocuments.us/reader031/viewer/2022011721/587d710d1a28ab32318b7d5b/html5/thumbnails/37.jpg)
CENSUS – PUBLIC USE MICRODATA SAMPLE (PUMS)
• Raw version of the data used to create ACS estimates
• Contains raw individual responses instead of data aggregated to geographic area
• Gives much more information but large and more complex to use
Source: https://www.census.gov/programs-surveys/acs/technical-documentation/pums.html
![Page 38: Making Use of Big Data](https://reader031.fdocuments.us/reader031/viewer/2022011721/587d710d1a28ab32318b7d5b/html5/thumbnails/38.jpg)
![Page 39: Making Use of Big Data](https://reader031.fdocuments.us/reader031/viewer/2022011721/587d710d1a28ab32318b7d5b/html5/thumbnails/39.jpg)
![Page 40: Making Use of Big Data](https://reader031.fdocuments.us/reader031/viewer/2022011721/587d710d1a28ab32318b7d5b/html5/thumbnails/40.jpg)
![Page 41: Making Use of Big Data](https://reader031.fdocuments.us/reader031/viewer/2022011721/587d710d1a28ab32318b7d5b/html5/thumbnails/41.jpg)
DEMOGRAPHIC MULTIPLIERS WITH PUMS
• Raw data allows for creation of demographic multipliers based on averages from ACS data
• Vital to development impact assessment
![Page 42: Making Use of Big Data](https://reader031.fdocuments.us/reader031/viewer/2022011721/587d710d1a28ab32318b7d5b/html5/thumbnails/42.jpg)
DEMOGRAPHIC MULTIPLIERS EXAMPLE
Unit Type BedroomsNumber of Units
Public School Students/Hou
sing Unit
Total New Public School
StudentsSingle Family – Age Targeted 3 50 0.15 8 Single Family 3 20 0.29 6 Single Family 4 100 0.95 95 Townhouse – Owner 3 110 0.29 32 Townhouse - Rental 3 20 0.64 13 Mixed Use Condominium 3 40 0.29 12 Apartments 1 150 0.07 11 Apartments 2 130 0.33 43 Total Housing Units 620 218
Estimated Number of New Public School Students Resulting from a Sample Residential Development
Source: ESI (2015), ACS 5-year 2013 (https://www.census.gov/programs-surveys/acs/data/pums.html)
![Page 43: Making Use of Big Data](https://reader031.fdocuments.us/reader031/viewer/2022011721/587d710d1a28ab32318b7d5b/html5/thumbnails/43.jpg)
DEMOGRAPHIC MULTIPLIERS EXAMPLE
Source: ESI (2015), ACS 5-year 2013 (https://www.census.gov/programs-surveys/acs/data/pums.html)
Revenue Deficit
Revenue Surplus
![Page 44: Making Use of Big Data](https://reader031.fdocuments.us/reader031/viewer/2022011721/587d710d1a28ab32318b7d5b/html5/thumbnails/44.jpg)
Other Examples• CyclePhilly
• WindyGrid
• SEPTA / PAT Cards
• Rich Blocks Poor Blocks (census data)
• Azavea / Philly Police Crime Map
![Page 45: Making Use of Big Data](https://reader031.fdocuments.us/reader031/viewer/2022011721/587d710d1a28ab32318b7d5b/html5/thumbnails/45.jpg)
Use: DVRPC/ CyclePhilly
• Smartphone app CyclePhilly collects data on bicycle trips from users
• DVRPC used data to analyze biking patterns in Philadelphia
Source: http://www.cyclephilly.org/
![Page 46: Making Use of Big Data](https://reader031.fdocuments.us/reader031/viewer/2022011721/587d710d1a28ab32318b7d5b/html5/thumbnails/46.jpg)
![Page 47: Making Use of Big Data](https://reader031.fdocuments.us/reader031/viewer/2022011721/587d710d1a28ab32318b7d5b/html5/thumbnails/47.jpg)
Use: Chicago WindyGridGIS Application for use by city officials that integrates city building and spatial data, 911 and 311 calls, and public tweets in real-time.
Source: Harvard Data-Smart City Solutions, http://datasmart.ash.harvard.edu/
![Page 48: Making Use of Big Data](https://reader031.fdocuments.us/reader031/viewer/2022011721/587d710d1a28ab32318b7d5b/html5/thumbnails/48.jpg)
Use: New Transit Cards
![Page 49: Making Use of Big Data](https://reader031.fdocuments.us/reader031/viewer/2022011721/587d710d1a28ab32318b7d5b/html5/thumbnails/49.jpg)
Thank You
Questions?
Peter Angelides, PhD, AICPEconsult Solutions215-717-2777econsultsolutions.com