Big data overwiew, Татьяна Матвиенко/Александр Павленко, Senior Java/BigData Developer, DataArt
BigData Visualization€¦ · BigData Visualization Author: Esri Subject: 2014 Esri International...
Transcript of BigData Visualization€¦ · BigData Visualization Author: Esri Subject: 2014 Esri International...
![Page 1: BigData Visualization€¦ · BigData Visualization Author: Esri Subject: 2014 Esri International Developer Summit -- Presentation Keywords: 2014 Esri International Developer Summit](https://reader031.fdocuments.us/reader031/viewer/2022013010/5eda48d4b3745412b571158f/html5/thumbnails/1.jpg)
BigData Visualization @SpatialAgent
@mraad
![Page 2: BigData Visualization€¦ · BigData Visualization Author: Esri Subject: 2014 Esri International Developer Summit -- Presentation Keywords: 2014 Esri International Developer Summit](https://reader031.fdocuments.us/reader031/viewer/2022013010/5eda48d4b3745412b571158f/html5/thumbnails/2.jpg)
BigData ?
![Page 3: BigData Visualization€¦ · BigData Visualization Author: Esri Subject: 2014 Esri International Developer Summit -- Presentation Keywords: 2014 Esri International Developer Summit](https://reader031.fdocuments.us/reader031/viewer/2022013010/5eda48d4b3745412b571158f/html5/thumbnails/3.jpg)
” Big data is like teenage sex: everyone talks about it, nobody really knows how to do it, everyone thinks everyone else is doing it, so everyone claims they are doing it… - Dan Ariely
![Page 4: BigData Visualization€¦ · BigData Visualization Author: Esri Subject: 2014 Esri International Developer Summit -- Presentation Keywords: 2014 Esri International Developer Summit](https://reader031.fdocuments.us/reader031/viewer/2022013010/5eda48d4b3745412b571158f/html5/thumbnails/4.jpg)
![Page 5: BigData Visualization€¦ · BigData Visualization Author: Esri Subject: 2014 Esri International Developer Summit -- Presentation Keywords: 2014 Esri International Developer Summit](https://reader031.fdocuments.us/reader031/viewer/2022013010/5eda48d4b3745412b571158f/html5/thumbnails/5.jpg)
Hadoop Distributed File System (HDFS)
Yet Another Resource Negotiator (YARN)
MapReduce
Commodity Servers
Hadoop Basic Stack
![Page 6: BigData Visualization€¦ · BigData Visualization Author: Esri Subject: 2014 Esri International Developer Summit -- Presentation Keywords: 2014 Esri International Developer Summit](https://reader031.fdocuments.us/reader031/viewer/2022013010/5eda48d4b3745412b571158f/html5/thumbnails/6.jpg)
The Zoo
• Hive - Ad Hoc Query - “SQL” to MapReduce • Pig - High Level Data Analysis Language • Impala - MPP SQL Engine • Mahout - Machine Learning Toolbox • HBase - Columnar KeyValue Database • Cascading - Flow Data Analysis • Avro - Data Serializer • Zookeeper - Centralized State Management
![Page 7: BigData Visualization€¦ · BigData Visualization Author: Esri Subject: 2014 Esri International Developer Summit -- Presentation Keywords: 2014 Esri International Developer Summit](https://reader031.fdocuments.us/reader031/viewer/2022013010/5eda48d4b3745412b571158f/html5/thumbnails/7.jpg)
![Page 8: BigData Visualization€¦ · BigData Visualization Author: Esri Subject: 2014 Esri International Developer Summit -- Presentation Keywords: 2014 Esri International Developer Summit](https://reader031.fdocuments.us/reader031/viewer/2022013010/5eda48d4b3745412b571158f/html5/thumbnails/8.jpg)
![Page 9: BigData Visualization€¦ · BigData Visualization Author: Esri Subject: 2014 Esri International Developer Summit -- Presentation Keywords: 2014 Esri International Developer Summit](https://reader031.fdocuments.us/reader031/viewer/2022013010/5eda48d4b3745412b571158f/html5/thumbnails/9.jpg)
GIS Tools For Hadoop
• Geometry API • Point / Line / Polygon • Operations - Contains, Intersect, Buffer • I/O - WKT, GeoJSON, Shape
• Hive Spatial UDF • ST_POINT, ST_CONTAINS
• GeoProcessing Extensions
![Page 10: BigData Visualization€¦ · BigData Visualization Author: Esri Subject: 2014 Esri International Developer Summit -- Presentation Keywords: 2014 Esri International Developer Summit](https://reader031.fdocuments.us/reader031/viewer/2022013010/5eda48d4b3745412b571158f/html5/thumbnails/10.jpg)
Hello, MapReduce !
![Page 11: BigData Visualization€¦ · BigData Visualization Author: Esri Subject: 2014 Esri International Developer Summit -- Presentation Keywords: 2014 Esri International Developer Summit](https://reader031.fdocuments.us/reader031/viewer/2022013010/5eda48d4b3745412b571158f/html5/thumbnails/11.jpg)
Density Analysis - Cell Count
![Page 12: BigData Visualization€¦ · BigData Visualization Author: Esri Subject: 2014 Esri International Developer Summit -- Presentation Keywords: 2014 Esri International Developer Summit](https://reader031.fdocuments.us/reader031/viewer/2022013010/5eda48d4b3745412b571158f/html5/thumbnails/12.jpg)
MapReduce Recap
• Map • Extract • Filter • Transform
• Reduce • Group By • Aggregate
![Page 13: BigData Visualization€¦ · BigData Visualization Author: Esri Subject: 2014 Esri International Developer Summit -- Presentation Keywords: 2014 Esri International Developer Summit](https://reader031.fdocuments.us/reader031/viewer/2022013010/5eda48d4b3745412b571158f/html5/thumbnails/13.jpg)
Cell Count
function map(lineno,text) { (x,y) = tokenize(text) if(inGrid(x,y)){ (cellX,cellY) = toCell(x,y) emit((cellX,cellY),1) } }
function reduce((cellX,cellY),iterator){ sum = 0 for( one in iterator){ sum = sum + one } emit((cellX,cellY), sum) }
![Page 14: BigData Visualization€¦ · BigData Visualization Author: Esri Subject: 2014 Esri International Developer Summit -- Presentation Keywords: 2014 Esri International Developer Summit](https://reader031.fdocuments.us/reader031/viewer/2022013010/5eda48d4b3745412b571158f/html5/thumbnails/14.jpg)
Coolest UX…
![Page 15: BigData Visualization€¦ · BigData Visualization Author: Esri Subject: 2014 Esri International Developer Summit -- Presentation Keywords: 2014 Esri International Developer Summit](https://reader031.fdocuments.us/reader031/viewer/2022013010/5eda48d4b3745412b571158f/html5/thumbnails/15.jpg)
![Page 16: BigData Visualization€¦ · BigData Visualization Author: Esri Subject: 2014 Esri International Developer Summit -- Presentation Keywords: 2014 Esri International Developer Summit](https://reader031.fdocuments.us/reader031/viewer/2022013010/5eda48d4b3745412b571158f/html5/thumbnails/16.jpg)
In Action Demo
![Page 17: BigData Visualization€¦ · BigData Visualization Author: Esri Subject: 2014 Esri International Developer Summit -- Presentation Keywords: 2014 Esri International Developer Summit](https://reader031.fdocuments.us/reader031/viewer/2022013010/5eda48d4b3745412b571158f/html5/thumbnails/17.jpg)
MapReduce Is Hard…
![Page 18: BigData Visualization€¦ · BigData Visualization Author: Esri Subject: 2014 Esri International Developer Summit -- Presentation Keywords: 2014 Esri International Developer Summit](https://reader031.fdocuments.us/reader031/viewer/2022013010/5eda48d4b3745412b571158f/html5/thumbnails/18.jpg)
Thinking Of Data As Water
![Page 19: BigData Visualization€¦ · BigData Visualization Author: Esri Subject: 2014 Esri International Developer Summit -- Presentation Keywords: 2014 Esri International Developer Summit](https://reader031.fdocuments.us/reader031/viewer/2022013010/5eda48d4b3745412b571158f/html5/thumbnails/19.jpg)
![Page 20: BigData Visualization€¦ · BigData Visualization Author: Esri Subject: 2014 Esri International Developer Summit -- Presentation Keywords: 2014 Esri International Developer Summit](https://reader031.fdocuments.us/reader031/viewer/2022013010/5eda48d4b3745412b571158f/html5/thumbnails/20.jpg)
To Cell GroupBy
count
X,Y Collection
Cell Count
R M
Source Sink
Filter
Cascading Pipeline
![Page 21: BigData Visualization€¦ · BigData Visualization Author: Esri Subject: 2014 Esri International Developer Summit -- Presentation Keywords: 2014 Esri International Developer Summit](https://reader031.fdocuments.us/reader031/viewer/2022013010/5eda48d4b3745412b571158f/html5/thumbnails/21.jpg)
![Page 22: BigData Visualization€¦ · BigData Visualization Author: Esri Subject: 2014 Esri International Developer Summit -- Presentation Keywords: 2014 Esri International Developer Summit](https://reader031.fdocuments.us/reader031/viewer/2022013010/5eda48d4b3745412b571158f/html5/thumbnails/22.jpg)
Spatial Join !
![Page 23: BigData Visualization€¦ · BigData Visualization Author: Esri Subject: 2014 Esri International Developer Summit -- Presentation Keywords: 2014 Esri International Developer Summit](https://reader031.fdocuments.us/reader031/viewer/2022013010/5eda48d4b3745412b571158f/html5/thumbnails/23.jpg)
Cascading In Action
![Page 24: BigData Visualization€¦ · BigData Visualization Author: Esri Subject: 2014 Esri International Developer Summit -- Presentation Keywords: 2014 Esri International Developer Summit](https://reader031.fdocuments.us/reader031/viewer/2022013010/5eda48d4b3745412b571158f/html5/thumbnails/24.jpg)
How About No Programming ? What About SQL ?
![Page 25: BigData Visualization€¦ · BigData Visualization Author: Esri Subject: 2014 Esri International Developer Summit -- Presentation Keywords: 2014 Esri International Developer Summit](https://reader031.fdocuments.us/reader031/viewer/2022013010/5eda48d4b3745412b571158f/html5/thumbnails/25.jpg)
Hive and Impala
![Page 26: BigData Visualization€¦ · BigData Visualization Author: Esri Subject: 2014 Esri International Developer Summit -- Presentation Keywords: 2014 Esri International Developer Summit](https://reader031.fdocuments.us/reader031/viewer/2022013010/5eda48d4b3745412b571158f/html5/thumbnails/26.jpg)
drop table if exists zipcodes; create external table if not exists zipcodes( id int, lon double, lat double ) row format delimited fields terminated by '\t' lines terminated by '\n' stored as textfile location '/user/cloudera/zipcodes';
![Page 27: BigData Visualization€¦ · BigData Visualization Author: Esri Subject: 2014 Esri International Developer Summit -- Presentation Keywords: 2014 Esri International Developer Summit](https://reader031.fdocuments.us/reader031/viewer/2022013010/5eda48d4b3745412b571158f/html5/thumbnails/27.jpg)
SELECT T.X-180+0.5 AS LON, T.Y-90+0.5 AS LAT, COUNT(*) AS POPULATION FROM ( SELECT FLOOR(LON+180) AS X, FLOOR(LAT+90) AS Y FROM ZIPCODES ) T GROUP BY T.X,T.Y;
![Page 28: BigData Visualization€¦ · BigData Visualization Author: Esri Subject: 2014 Esri International Developer Summit -- Presentation Keywords: 2014 Esri International Developer Summit](https://reader031.fdocuments.us/reader031/viewer/2022013010/5eda48d4b3745412b571158f/html5/thumbnails/28.jpg)
Hive and Impala In Action
![Page 29: BigData Visualization€¦ · BigData Visualization Author: Esri Subject: 2014 Esri International Developer Summit -- Presentation Keywords: 2014 Esri International Developer Summit](https://reader031.fdocuments.us/reader031/viewer/2022013010/5eda48d4b3745412b571158f/html5/thumbnails/29.jpg)
ArcGIS & Hadoop
![Page 30: BigData Visualization€¦ · BigData Visualization Author: Esri Subject: 2014 Esri International Developer Summit -- Presentation Keywords: 2014 Esri International Developer Summit](https://reader031.fdocuments.us/reader031/viewer/2022013010/5eda48d4b3745412b571158f/html5/thumbnails/30.jpg)
AIS DATA
• 14.8 Million data points • 1 Month • MMSI, Zulu Time, Lat, Lon, Vessel ID, Draught • Port of Miami - Free
![Page 31: BigData Visualization€¦ · BigData Visualization Author: Esri Subject: 2014 Esri International Developer Summit -- Presentation Keywords: 2014 Esri International Developer Summit](https://reader031.fdocuments.us/reader031/viewer/2022013010/5eda48d4b3745412b571158f/html5/thumbnails/31.jpg)
DEMO Steps
• GP Toolbox • Track Assembly • Hex Generation • Density Analysis
![Page 32: BigData Visualization€¦ · BigData Visualization Author: Esri Subject: 2014 Esri International Developer Summit -- Presentation Keywords: 2014 Esri International Developer Summit](https://reader031.fdocuments.us/reader031/viewer/2022013010/5eda48d4b3745412b571158f/html5/thumbnails/32.jpg)
Import Job
AIS CSV
Import Partitioner /ais/YYYY/MM/dd/HH/UUID.csv
HDFS MapReduce
![Page 33: BigData Visualization€¦ · BigData Visualization Author: Esri Subject: 2014 Esri International Developer Summit -- Presentation Keywords: 2014 Esri International Developer Summit](https://reader031.fdocuments.us/reader031/viewer/2022013010/5eda48d4b3745412b571158f/html5/thumbnails/33.jpg)
![Page 34: BigData Visualization€¦ · BigData Visualization Author: Esri Subject: 2014 Esri International Developer Summit -- Presentation Keywords: 2014 Esri International Developer Summit](https://reader031.fdocuments.us/reader031/viewer/2022013010/5eda48d4b3745412b571158f/html5/thumbnails/34.jpg)
![Page 35: BigData Visualization€¦ · BigData Visualization Author: Esri Subject: 2014 Esri International Developer Summit -- Presentation Keywords: 2014 Esri International Developer Summit](https://reader031.fdocuments.us/reader031/viewer/2022013010/5eda48d4b3745412b571158f/html5/thumbnails/35.jpg)
![Page 36: BigData Visualization€¦ · BigData Visualization Author: Esri Subject: 2014 Esri International Developer Summit -- Presentation Keywords: 2014 Esri International Developer Summit](https://reader031.fdocuments.us/reader031/viewer/2022013010/5eda48d4b3745412b571158f/html5/thumbnails/36.jpg)
![Page 37: BigData Visualization€¦ · BigData Visualization Author: Esri Subject: 2014 Esri International Developer Summit -- Presentation Keywords: 2014 Esri International Developer Summit](https://reader031.fdocuments.us/reader031/viewer/2022013010/5eda48d4b3745412b571158f/html5/thumbnails/37.jpg)
![Page 38: BigData Visualization€¦ · BigData Visualization Author: Esri Subject: 2014 Esri International Developer Summit -- Presentation Keywords: 2014 Esri International Developer Summit](https://reader031.fdocuments.us/reader031/viewer/2022013010/5eda48d4b3745412b571158f/html5/thumbnails/38.jpg)
WebMaps
![Page 39: BigData Visualization€¦ · BigData Visualization Author: Esri Subject: 2014 Esri International Developer Summit -- Presentation Keywords: 2014 Esri International Developer Summit](https://reader031.fdocuments.us/reader031/viewer/2022013010/5eda48d4b3745412b571158f/html5/thumbnails/39.jpg)
/ShippingVolume
/ShippingRank
/ShippingGlobe
/ShippingTracks
http://coolmaps.esri.com/BigData
![Page 40: BigData Visualization€¦ · BigData Visualization Author: Esri Subject: 2014 Esri International Developer Summit -- Presentation Keywords: 2014 Esri International Developer Summit](https://reader031.fdocuments.us/reader031/viewer/2022013010/5eda48d4b3745412b571158f/html5/thumbnails/40.jpg)
Return of KillerApps ?
![Page 42: BigData Visualization€¦ · BigData Visualization Author: Esri Subject: 2014 Esri International Developer Summit -- Presentation Keywords: 2014 Esri International Developer Summit](https://reader031.fdocuments.us/reader031/viewer/2022013010/5eda48d4b3745412b571158f/html5/thumbnails/42.jpg)