Data - Lifeline of the Modern World · 2017-03-05 · Data - Lifeline of the Modern World...
Transcript of Data - Lifeline of the Modern World · 2017-03-05 · Data - Lifeline of the Modern World...
![Page 1: Data - Lifeline of the Modern World · 2017-03-05 · Data - Lifeline of the Modern World Introduction - Data is an integral part of modern world 6SPIFMKHEXETPE]WMRXSHE]mW[SVPH How](https://reader033.fdocuments.us/reader033/viewer/2022043001/5f78e87e049fa11d3b0c5f19/html5/thumbnails/1.jpg)
Data - Lifeline of the Modern World
Ishaan Shrivastav Redlands High School
![Page 2: Data - Lifeline of the Modern World · 2017-03-05 · Data - Lifeline of the Modern World Introduction - Data is an integral part of modern world 6SPIFMKHEXETPE]WMRXSHE]mW[SVPH How](https://reader033.fdocuments.us/reader033/viewer/2022043001/5f78e87e049fa11d3b0c5f19/html5/thumbnails/2.jpg)
Data - Lifeline of the Modern World
Introduction - Data is an integral part of modern world
Role big data plays in today s world
How big data can make an impact in the future
Tools to store, analyze, and visualize big data
My big data project to analyze social media data
![Page 3: Data - Lifeline of the Modern World · 2017-03-05 · Data - Lifeline of the Modern World Introduction - Data is an integral part of modern world 6SPIFMKHEXETPE]WMRXSHE]mW[SVPH How](https://reader033.fdocuments.us/reader033/viewer/2022043001/5f78e87e049fa11d3b0c5f19/html5/thumbnails/3.jpg)
Types of Data
Environment: noise, CO2 humidity temperature air-pressure air-quality water-quality rain-gauges ozone-concentration
Transportation: Flight-status vehicle/plane-location airport-queue traffic-condition passenger-count parking-meters
Public Safety/Health:
Police ambulance crime-data diseases-incidences hospital-location vaccination-records
Public Records: census parcels population tax-records birth/death/marriage records
Communication: phones-calls emails satellite-transmission internet
![Page 4: Data - Lifeline of the Modern World · 2017-03-05 · Data - Lifeline of the Modern World Introduction - Data is an integral part of modern world 6SPIFMKHEXETPE]WMRXSHE]mW[SVPH How](https://reader033.fdocuments.us/reader033/viewer/2022043001/5f78e87e049fa11d3b0c5f19/html5/thumbnails/4.jpg)
Who is producing Big Data?
Social media
Twitter, Instagram, Facebook, WhatsApp, Youtube, …
Scientific Data
Satellite Imagery, Health metrics, Space observations
![Page 5: Data - Lifeline of the Modern World · 2017-03-05 · Data - Lifeline of the Modern World Introduction - Data is an integral part of modern world 6SPIFMKHEXETPE]WMRXSHE]mW[SVPH How](https://reader033.fdocuments.us/reader033/viewer/2022043001/5f78e87e049fa11d3b0c5f19/html5/thumbnails/5.jpg)
Who is producing Big Data?
Modern Live Devices (Internet of Things)
Mobile phones, Smart meters, Weather sensors, In-built car sensors
Corporations
Financial transactions, Sales
![Page 6: Data - Lifeline of the Modern World · 2017-03-05 · Data - Lifeline of the Modern World Introduction - Data is an integral part of modern world 6SPIFMKHEXETPE]WMRXSHE]mW[SVPH How](https://reader033.fdocuments.us/reader033/viewer/2022043001/5f78e87e049fa11d3b0c5f19/html5/thumbnails/6.jpg)
Data Explosion in Social Media
By 2020!!
30 Billion Mobile Devices
40 Trillion Gigabytes of
Data
10 Million mobile apps
All for 8 Billion people
![Page 7: Data - Lifeline of the Modern World · 2017-03-05 · Data - Lifeline of the Modern World Introduction - Data is an integral part of modern world 6SPIFMKHEXETPE]WMRXSHE]mW[SVPH How](https://reader033.fdocuments.us/reader033/viewer/2022043001/5f78e87e049fa11d3b0c5f19/html5/thumbnails/7.jpg)
Data - Lifeline of the Modern World
Introduction - Data is an integral part of modern world
Role big data plays in today’s world
What Big Data can be used for in the Future
Tools to Store, Analyze, and Visualize data
My Big Data project to analyze social media interaction
![Page 8: Data - Lifeline of the Modern World · 2017-03-05 · Data - Lifeline of the Modern World Introduction - Data is an integral part of modern world 6SPIFMKHEXETPE]WMRXSHE]mW[SVPH How](https://reader033.fdocuments.us/reader033/viewer/2022043001/5f78e87e049fa11d3b0c5f19/html5/thumbnails/8.jpg)
Role Big Data Plays in Today’s World
● Transportation
○ Checking flight status, identifying plane location, finding traffic
conditions
● Public Safety
○ Police Stations, Fire Stations, and Surveillance all use data coming from
different sources and make real-time decisions for public safety
![Page 9: Data - Lifeline of the Modern World · 2017-03-05 · Data - Lifeline of the Modern World Introduction - Data is an integral part of modern world 6SPIFMKHEXETPE]WMRXSHE]mW[SVPH How](https://reader033.fdocuments.us/reader033/viewer/2022043001/5f78e87e049fa11d3b0c5f19/html5/thumbnails/9.jpg)
Role Big Data Plays in Today’s World
● Energy Usage
○ Electricity, Gas, and Smart meters produce large volume of data that is
analyzed by utility companies for best utilization of resources
● Weather
○ Prediction for weather (temperature, precipitation etc.) and natural
disasters (tornadoes, hurricanes, tsunami) are made by analyzing large
volume of scientific data coming from sensors around the world.
![Page 10: Data - Lifeline of the Modern World · 2017-03-05 · Data - Lifeline of the Modern World Introduction - Data is an integral part of modern world 6SPIFMKHEXETPE]WMRXSHE]mW[SVPH How](https://reader033.fdocuments.us/reader033/viewer/2022043001/5f78e87e049fa11d3b0c5f19/html5/thumbnails/10.jpg)
Data - Lifeline of the Modern World
Introduction - Data is an integral part of modern world
Role big data plays in today s world
How Big Data can be used in the Future
Tools to Store, Analyze, and Visualize data
My Big Data project to analyze social media interaction
![Page 11: Data - Lifeline of the Modern World · 2017-03-05 · Data - Lifeline of the Modern World Introduction - Data is an integral part of modern world 6SPIFMKHEXETPE]WMRXSHE]mW[SVPH How](https://reader033.fdocuments.us/reader033/viewer/2022043001/5f78e87e049fa11d3b0c5f19/html5/thumbnails/11.jpg)
How Big Data can be used in the Future
Self driven cars will communicate with each other about road conditions,
traffic and gas prices data and making intelligent decision based on them.
Dating sites will find the perfect soulmate based on data available about
members interest, movies they like, donation they make to charities,
association with different organization etc. :)
Better healthcare management
Chronic disease like diabetes and high blood pressure patients will be wearing
devices that will be constantly sending their health related data to doctors office
so they can make appropriate adjustments
![Page 12: Data - Lifeline of the Modern World · 2017-03-05 · Data - Lifeline of the Modern World Introduction - Data is an integral part of modern world 6SPIFMKHEXETPE]WMRXSHE]mW[SVPH How](https://reader033.fdocuments.us/reader033/viewer/2022043001/5f78e87e049fa11d3b0c5f19/html5/thumbnails/12.jpg)
Data - Lifeline of the Modern World
Introduction - Data is an integral part of modern world
Role big data plays in today s world
What Big Data can be used for in the Future
Tools to Store, Analyze, and Visualize data
My Big Data project to analyze social media interaction
![Page 13: Data - Lifeline of the Modern World · 2017-03-05 · Data - Lifeline of the Modern World Introduction - Data is an integral part of modern world 6SPIFMKHEXETPE]WMRXSHE]mW[SVPH How](https://reader033.fdocuments.us/reader033/viewer/2022043001/5f78e87e049fa11d3b0c5f19/html5/thumbnails/13.jpg)
Tools to Store, Analyze and Visualize Big Data
![Page 14: Data - Lifeline of the Modern World · 2017-03-05 · Data - Lifeline of the Modern World Introduction - Data is an integral part of modern world 6SPIFMKHEXETPE]WMRXSHE]mW[SVPH How](https://reader033.fdocuments.us/reader033/viewer/2022043001/5f78e87e049fa11d3b0c5f19/html5/thumbnails/14.jpg)
Storing Big Data
Distributed file systems
Hadoop, cloud data storage (S3, azure blob storage)
Distributed Data Management systems
Elasticsearch, CouchDB, DynamoDB
![Page 15: Data - Lifeline of the Modern World · 2017-03-05 · Data - Lifeline of the Modern World Introduction - Data is an integral part of modern world 6SPIFMKHEXETPE]WMRXSHE]mW[SVPH How](https://reader033.fdocuments.us/reader033/viewer/2022043001/5f78e87e049fa11d3b0c5f19/html5/thumbnails/15.jpg)
Analyzing Big Data
Elasticsearch
Microsoft Power BI
SAP HANA, Teradata
An example of analyzing big data stored in elasticsearch using different
techniques
Police department wants to improve public safety by taking citizen petrol help
for non-violent crimes such as vandalism
![Page 16: Data - Lifeline of the Modern World · 2017-03-05 · Data - Lifeline of the Modern World Introduction - Data is an integral part of modern world 6SPIFMKHEXETPE]WMRXSHE]mW[SVPH How](https://reader033.fdocuments.us/reader033/viewer/2022043001/5f78e87e049fa11d3b0c5f19/html5/thumbnails/16.jpg)
Analyzing Big Data
Crime_incidents {
“Type” : “vandalism”, “Time” : “15:30”, “Date” :12/31/2016” “Location” : { “Lat” : 34.0656
“Long” : 117.16645 }
“Case_solved” : “No”, “Officer_assigned” : “James Smith” }
![Page 17: Data - Lifeline of the Modern World · 2017-03-05 · Data - Lifeline of the Modern World Introduction - Data is an integral part of modern world 6SPIFMKHEXETPE]WMRXSHE]mW[SVPH How](https://reader033.fdocuments.us/reader033/viewer/2022043001/5f78e87e049fa11d3b0c5f19/html5/thumbnails/17.jpg)
Spatial analysis ● Find statistics for number of vandalism within 10 miles of city hall
● {
"query": {
"bool" : {
"must" : {
"match_all" : {}
},
"filter" : {
"geo_distance" : {
"distance" : 10miles",
"pin.location" : {
"lat" : 40,
"lon" : -70
}
}
}
}
}
}
![Page 18: Data - Lifeline of the Modern World · 2017-03-05 · Data - Lifeline of the Modern World Introduction - Data is an integral part of modern world 6SPIFMKHEXETPE]WMRXSHE]mW[SVPH How](https://reader033.fdocuments.us/reader033/viewer/2022043001/5f78e87e049fa11d3b0c5f19/html5/thumbnails/18.jpg)
Analyzing Big Data
● Filter data
○ Find all vandalism type crimes that occurs query : { term : { Type : vandalism } }
![Page 19: Data - Lifeline of the Modern World · 2017-03-05 · Data - Lifeline of the Modern World Introduction - Data is an integral part of modern world 6SPIFMKHEXETPE]WMRXSHE]mW[SVPH How](https://reader033.fdocuments.us/reader033/viewer/2022043001/5f78e87e049fa11d3b0c5f19/html5/thumbnails/19.jpg)
Analyzing Big Data
● Aggregation
○ Find when do vandalism crime happens most GET /_search { "query": { term : { Type : vandalism } }, "aggs" : { "tweets_over_time" : { "date_histogram" : { "field" : "Time", "interval" : "60m" } }
![Page 20: Data - Lifeline of the Modern World · 2017-03-05 · Data - Lifeline of the Modern World Introduction - Data is an integral part of modern world 6SPIFMKHEXETPE]WMRXSHE]mW[SVPH How](https://reader033.fdocuments.us/reader033/viewer/2022043001/5f78e87e049fa11d3b0c5f19/html5/thumbnails/20.jpg)
Total Query
GET /_search {
"query": {
"bool" : {
"must" : {
"match_all" : {
query : { term : { Type : vandalism } } },
"filter" : {
"geo_distance" : {
"distance" : 10mi", "pin.location" : {
"lat" : 40,
"lon" : -70
}
}
}
}
}
}
![Page 21: Data - Lifeline of the Modern World · 2017-03-05 · Data - Lifeline of the Modern World Introduction - Data is an integral part of modern world 6SPIFMKHEXETPE]WMRXSHE]mW[SVPH How](https://reader033.fdocuments.us/reader033/viewer/2022043001/5f78e87e049fa11d3b0c5f19/html5/thumbnails/21.jpg)
Visualizing Big Data
ArcGIS
Mapping tool that can use data to show trends across space
Tableau
Tool that can create charts and diagrams to display trends in data
Kibana
![Page 22: Data - Lifeline of the Modern World · 2017-03-05 · Data - Lifeline of the Modern World Introduction - Data is an integral part of modern world 6SPIFMKHEXETPE]WMRXSHE]mW[SVPH How](https://reader033.fdocuments.us/reader033/viewer/2022043001/5f78e87e049fa11d3b0c5f19/html5/thumbnails/22.jpg)
Data - Lifeline of the Modern World
Introduction - Data is an integral part of modern world
Role big data plays in today s world
What Big Data can be used for in the Future
Tools to Store, Analyze, and Visualize data
My Big Data project to analyze social media interaction
![Page 23: Data - Lifeline of the Modern World · 2017-03-05 · Data - Lifeline of the Modern World Introduction - Data is an integral part of modern world 6SPIFMKHEXETPE]WMRXSHE]mW[SVPH How](https://reader033.fdocuments.us/reader033/viewer/2022043001/5f78e87e049fa11d3b0c5f19/html5/thumbnails/23.jpg)
My Big Data Project:
Goal: To discover social pattern based on big data available in social media
Project: To see if there was a correlation between the number of tweets and
scoreline during the SuperBowl game
Software used:
Storage and analysis: Elasticsearch including logstash extension
![Page 24: Data - Lifeline of the Modern World · 2017-03-05 · Data - Lifeline of the Modern World Introduction - Data is an integral part of modern world 6SPIFMKHEXETPE]WMRXSHE]mW[SVPH How](https://reader033.fdocuments.us/reader033/viewer/2022043001/5f78e87e049fa11d3b0c5f19/html5/thumbnails/24.jpg)
My Big Data Project:
Storage
Twitter feeds
Use Logstash pluggin in Elasticsearch to store live twitter feeds into elasticsearch during
super bowl
Filter twitter feeds based on keywords for NFL teams (patriots and falcons)
1.7 million tweets
![Page 25: Data - Lifeline of the Modern World · 2017-03-05 · Data - Lifeline of the Modern World Introduction - Data is an integral part of modern world 6SPIFMKHEXETPE]WMRXSHE]mW[SVPH How](https://reader033.fdocuments.us/reader033/viewer/2022043001/5f78e87e049fa11d3b0c5f19/html5/thumbnails/25.jpg)
My Big Data Project:
Analysis
Used Elasticsearch search and aggregation queries to find tweets for different teams during
different time of the game
Search Query to find certain tweets Search query to find number of tweets per quarter per
team
GET /_search
{
"query": {
"term" : {"text":"patriots"}
}
}
![Page 26: Data - Lifeline of the Modern World · 2017-03-05 · Data - Lifeline of the Modern World Introduction - Data is an integral part of modern world 6SPIFMKHEXETPE]WMRXSHE]mW[SVPH How](https://reader033.fdocuments.us/reader033/viewer/2022043001/5f78e87e049fa11d3b0c5f19/html5/thumbnails/26.jpg)
My Big Data Project:
0
100000
200000
300000
400000
500000
600000
Quarter 1 Quarter 2 Quarter 3 Quarter 4
Diagrammtitel
Patriots Falcons