IoT Arhitecture, database and big data - ACEIoT · Gateway –Centric IoT Cloud (1/2) IoT...
Transcript of IoT Arhitecture, database and big data - ACEIoT · Gateway –Centric IoT Cloud (1/2) IoT...
![Page 1: IoT Arhitecture, database and big data - ACEIoT · Gateway –Centric IoT Cloud (1/2) IoT infrastructure will provide the opportunities to take services, workloads, applications and](https://reader034.fdocuments.us/reader034/viewer/2022042220/5ec5e8587252a11a7e401d64/html5/thumbnails/1.jpg)
IoT Arhitecture, database and big
data
23th- 27th April,
2018
Rwanda
Abdur Rahim
Open IoT
FBK CREATE-NET
![Page 2: IoT Arhitecture, database and big data - ACEIoT · Gateway –Centric IoT Cloud (1/2) IoT infrastructure will provide the opportunities to take services, workloads, applications and](https://reader034.fdocuments.us/reader034/viewer/2022042220/5ec5e8587252a11a7e401d64/html5/thumbnails/2.jpg)
IoT Architecture and platform
![Page 3: IoT Arhitecture, database and big data - ACEIoT · Gateway –Centric IoT Cloud (1/2) IoT infrastructure will provide the opportunities to take services, workloads, applications and](https://reader034.fdocuments.us/reader034/viewer/2022042220/5ec5e8587252a11a7e401d64/html5/thumbnails/3.jpg)
Platform overview 3
![Page 4: IoT Arhitecture, database and big data - ACEIoT · Gateway –Centric IoT Cloud (1/2) IoT infrastructure will provide the opportunities to take services, workloads, applications and](https://reader034.fdocuments.us/reader034/viewer/2022042220/5ec5e8587252a11a7e401d64/html5/thumbnails/4.jpg)
Two main platforms
IoT Gateway platform (Gateway-Centric
Cloud-Bring Cloud functionalities in Gateway)
IoT integrated Cloud/PaaS platform (Cloud-
centric IoT- Bring IoT functionalities in Cloud)
![Page 5: IoT Arhitecture, database and big data - ACEIoT · Gateway –Centric IoT Cloud (1/2) IoT infrastructure will provide the opportunities to take services, workloads, applications and](https://reader034.fdocuments.us/reader034/viewer/2022042220/5ec5e8587252a11a7e401d64/html5/thumbnails/5.jpg)
Gateway – Centric IoT Cloud (1/2)
IoT infrastructure will provide the opportunities to take services,
workloads, applications and large amounts of data and deliver
it all to the edge of the network.
Processing and storage of data close to users/near to sources
To distribute data to move it closer to the end-users to eliminate latency,
numerous hop, and support mobile computing and data streaming
Creating dense geographical distribution
![Page 6: IoT Arhitecture, database and big data - ACEIoT · Gateway –Centric IoT Cloud (1/2) IoT infrastructure will provide the opportunities to take services, workloads, applications and](https://reader034.fdocuments.us/reader034/viewer/2022042220/5ec5e8587252a11a7e401d64/html5/thumbnails/6.jpg)
Gateway- Centric IoT Cloud (2/2)
This approach are useful when service is provisioned from the
data coming from same location
Highly real time applications
Reduce network traffic and cost
Supporting end-users security
Data process and service execute locally (distributed cloud
processing, sub-work flow, data aggregation locally)
![Page 7: IoT Arhitecture, database and big data - ACEIoT · Gateway –Centric IoT Cloud (1/2) IoT infrastructure will provide the opportunities to take services, workloads, applications and](https://reader034.fdocuments.us/reader034/viewer/2022042220/5ec5e8587252a11a7e401d64/html5/thumbnails/7.jpg)
What is gateway platform?7
![Page 8: IoT Arhitecture, database and big data - ACEIoT · Gateway –Centric IoT Cloud (1/2) IoT infrastructure will provide the opportunities to take services, workloads, applications and](https://reader034.fdocuments.us/reader034/viewer/2022042220/5ec5e8587252a11a7e401d64/html5/thumbnails/8.jpg)
IoT Gateway centric architecture 8
Driving the functionalities to the edge
![Page 9: IoT Arhitecture, database and big data - ACEIoT · Gateway –Centric IoT Cloud (1/2) IoT infrastructure will provide the opportunities to take services, workloads, applications and](https://reader034.fdocuments.us/reader034/viewer/2022042220/5ec5e8587252a11a7e401d64/html5/thumbnails/9.jpg)
Intelligent gateway9
Data acquisition, integration and rules activation,
providing dynamic intelligence at the edge
![Page 10: IoT Arhitecture, database and big data - ACEIoT · Gateway –Centric IoT Cloud (1/2) IoT infrastructure will provide the opportunities to take services, workloads, applications and](https://reader034.fdocuments.us/reader034/viewer/2022042220/5ec5e8587252a11a7e401d64/html5/thumbnails/10.jpg)
Industrial IoT gateway10
![Page 11: IoT Arhitecture, database and big data - ACEIoT · Gateway –Centric IoT Cloud (1/2) IoT infrastructure will provide the opportunities to take services, workloads, applications and](https://reader034.fdocuments.us/reader034/viewer/2022042220/5ec5e8587252a11a7e401d64/html5/thumbnails/11.jpg)
Intel IoT Gateway11
![Page 12: IoT Arhitecture, database and big data - ACEIoT · Gateway –Centric IoT Cloud (1/2) IoT infrastructure will provide the opportunities to take services, workloads, applications and](https://reader034.fdocuments.us/reader034/viewer/2022042220/5ec5e8587252a11a7e401d64/html5/thumbnails/12.jpg)
Gateway comparison summary 12
https://www.loriot.io/gateways.html
![Page 13: IoT Arhitecture, database and big data - ACEIoT · Gateway –Centric IoT Cloud (1/2) IoT infrastructure will provide the opportunities to take services, workloads, applications and](https://reader034.fdocuments.us/reader034/viewer/2022042220/5ec5e8587252a11a7e401d64/html5/thumbnails/13.jpg)
Kura architecture 13
http://eclipse.githu
b.io/kura/doc/intr
o.html
![Page 14: IoT Arhitecture, database and big data - ACEIoT · Gateway –Centric IoT Cloud (1/2) IoT infrastructure will provide the opportunities to take services, workloads, applications and](https://reader034.fdocuments.us/reader034/viewer/2022042220/5ec5e8587252a11a7e401d64/html5/thumbnails/14.jpg)
Open source gateway14
Kura and camel integration
https://camel.apache.org/kura.html
![Page 15: IoT Arhitecture, database and big data - ACEIoT · Gateway –Centric IoT Cloud (1/2) IoT infrastructure will provide the opportunities to take services, workloads, applications and](https://reader034.fdocuments.us/reader034/viewer/2022042220/5ec5e8587252a11a7e401d64/html5/thumbnails/15.jpg)
Agile micorservice Gateway15
![Page 16: IoT Arhitecture, database and big data - ACEIoT · Gateway –Centric IoT Cloud (1/2) IoT infrastructure will provide the opportunities to take services, workloads, applications and](https://reader034.fdocuments.us/reader034/viewer/2022042220/5ec5e8587252a11a7e401d64/html5/thumbnails/16.jpg)
Cloud-centric IoT PaaS platform
Bring IoT data in the cloud
Processing and computing the data and deploy
management tools in cloud
This approach this good if service are provided
among objects located in multiple location
Giving several example (market platform, project
platform..)
![Page 17: IoT Arhitecture, database and big data - ACEIoT · Gateway –Centric IoT Cloud (1/2) IoT infrastructure will provide the opportunities to take services, workloads, applications and](https://reader034.fdocuments.us/reader034/viewer/2022042220/5ec5e8587252a11a7e401d64/html5/thumbnails/17.jpg)
Example of IoT platform
![Page 18: IoT Arhitecture, database and big data - ACEIoT · Gateway –Centric IoT Cloud (1/2) IoT infrastructure will provide the opportunities to take services, workloads, applications and](https://reader034.fdocuments.us/reader034/viewer/2022042220/5ec5e8587252a11a7e401d64/html5/thumbnails/18.jpg)
Bosch IoT Cloud18
https://www.bosch-si.com/products/
![Page 19: IoT Arhitecture, database and big data - ACEIoT · Gateway –Centric IoT Cloud (1/2) IoT infrastructure will provide the opportunities to take services, workloads, applications and](https://reader034.fdocuments.us/reader034/viewer/2022042220/5ec5e8587252a11a7e401d64/html5/thumbnails/19.jpg)
Most known IoT cloud platform19
![Page 20: IoT Arhitecture, database and big data - ACEIoT · Gateway –Centric IoT Cloud (1/2) IoT infrastructure will provide the opportunities to take services, workloads, applications and](https://reader034.fdocuments.us/reader034/viewer/2022042220/5ec5e8587252a11a7e401d64/html5/thumbnails/20.jpg)
Architecture
![Page 21: IoT Arhitecture, database and big data - ACEIoT · Gateway –Centric IoT Cloud (1/2) IoT infrastructure will provide the opportunities to take services, workloads, applications and](https://reader034.fdocuments.us/reader034/viewer/2022042220/5ec5e8587252a11a7e401d64/html5/thumbnails/21.jpg)
Architecture: functional domains and
components
Functional domain Component Technology choice
Application platform Orchestrator Deis
Execution environments Docker
RAD Node-RED
UI manager Elastic search, Kibana (Freeboard, D3.js)
IoT platform IoT bridge Specific protocols/libraries
Pre-Process FI-WARE CEPHEUS
Sensor registry Not selected yet
Sensor discovery Not selected yet
Big data platform Data streaming Orion
Storage manager MongoDB
Data analytic FlinkML, Mlib
Security & privacy Identity manager OpenAM/Gluu
Authorization manager OpenAM/Gluu
Privacy manager OpenAM/Gluu
![Page 22: IoT Arhitecture, database and big data - ACEIoT · Gateway –Centric IoT Cloud (1/2) IoT infrastructure will provide the opportunities to take services, workloads, applications and](https://reader034.fdocuments.us/reader034/viewer/2022042220/5ec5e8587252a11a7e401d64/html5/thumbnails/22.jpg)
Big data platform
orchestrator
App
compileApp
Read
manifest
Request
sensorStart
app
deployIoT PF
Manifest
reader
App Data
stream
Status
manager
Web
GUI
WorkersBig data
runtime
Stream
broker
Analytic
![Page 23: IoT Arhitecture, database and big data - ACEIoT · Gateway –Centric IoT Cloud (1/2) IoT infrastructure will provide the opportunities to take services, workloads, applications and](https://reader034.fdocuments.us/reader034/viewer/2022042220/5ec5e8587252a11a7e401d64/html5/thumbnails/23.jpg)
IoT platform
Sensor
finder
Sensor
registry
IoT
brokerIoT
bridge
Subscribe
Raw
data
data
Conn. req.Big
data PF
stream
sensor.
req.
Status
manager
Web
GUI
Pre-
process
![Page 24: IoT Arhitecture, database and big data - ACEIoT · Gateway –Centric IoT Cloud (1/2) IoT infrastructure will provide the opportunities to take services, workloads, applications and](https://reader034.fdocuments.us/reader034/viewer/2022042220/5ec5e8587252a11a7e401d64/html5/thumbnails/24.jpg)
PaaS platform
Developer
App +
manifest
WAZIUP Cloud platform
orchestrator
Cloud execution environment
Se
rvic
e
Se
rvic
e
Se
rvic
eApp user
Service
Service
Service
create push
deploy
deploy
Local deployment
App
Gateway Sensor
Local
PC
uses
With/Without internet
24
![Page 25: IoT Arhitecture, database and big data - ACEIoT · Gateway –Centric IoT Cloud (1/2) IoT infrastructure will provide the opportunities to take services, workloads, applications and](https://reader034.fdocuments.us/reader034/viewer/2022042220/5ec5e8587252a11a7e401d64/html5/thumbnails/25.jpg)
App deploy: local and global
Developer
App +
manifes
t
WAZIUP Cloud platform
orchestrator
Cloud execution environment
Se
rvic
e
Se
rvic
e
Se
rvic
eApp user
Service
Service
Service
create push
deploy
deploy
Local deployment
Ap
p
Gateway Sensor
Local
PC
uses
![Page 26: IoT Arhitecture, database and big data - ACEIoT · Gateway –Centric IoT Cloud (1/2) IoT infrastructure will provide the opportunities to take services, workloads, applications and](https://reader034.fdocuments.us/reader034/viewer/2022042220/5ec5e8587252a11a7e401d64/html5/thumbnails/26.jpg)
Data flows 1
WAZIUP Cloud platform
Cloud execution environment
Se
rvic
e
Se
rvic
e
Se
rvic
eApp user
access
Gateway
![Page 27: IoT Arhitecture, database and big data - ACEIoT · Gateway –Centric IoT Cloud (1/2) IoT infrastructure will provide the opportunities to take services, workloads, applications and](https://reader034.fdocuments.us/reader034/viewer/2022042220/5ec5e8587252a11a7e401d64/html5/thumbnails/27.jpg)
Data flow 2
WAZIUP Cloud platform
orchestrator
Cloud execution environment
Se
rvic
e
Se
rvic
e
Se
rvic
eApp user
Service
Service
Service
access
deploy
deploy
![Page 28: IoT Arhitecture, database and big data - ACEIoT · Gateway –Centric IoT Cloud (1/2) IoT infrastructure will provide the opportunities to take services, workloads, applications and](https://reader034.fdocuments.us/reader034/viewer/2022042220/5ec5e8587252a11a7e401d64/html5/thumbnails/28.jpg)
Data flows 2
WAZIUP Cloud platform
Cloud execution environment
Se
rvic
e
Se
rvic
e
Se
rvic
eApp user
Service
Service
access
Gateway
![Page 29: IoT Arhitecture, database and big data - ACEIoT · Gateway –Centric IoT Cloud (1/2) IoT infrastructure will provide the opportunities to take services, workloads, applications and](https://reader034.fdocuments.us/reader034/viewer/2022042220/5ec5e8587252a11a7e401d64/html5/thumbnails/29.jpg)
Databases requirements
• Run in gateway (low resources)
• Run in Cloud
• Run in local PC
• Synchronize in best-effort
• Support queries from local app
![Page 30: IoT Arhitecture, database and big data - ACEIoT · Gateway –Centric IoT Cloud (1/2) IoT infrastructure will provide the opportunities to take services, workloads, applications and](https://reader034.fdocuments.us/reader034/viewer/2022042220/5ec5e8587252a11a7e401d64/html5/thumbnails/30.jpg)
Big data technologies
• Data processing:
• Spark
• Flink
• Data broker:
• Kafka
• Orion
• Data Mining:
• Spark Mlib
• H20.ai
• Data Storage
• MongoDB
• Apache HBase
![Page 31: IoT Arhitecture, database and big data - ACEIoT · Gateway –Centric IoT Cloud (1/2) IoT infrastructure will provide the opportunities to take services, workloads, applications and](https://reader034.fdocuments.us/reader034/viewer/2022042220/5ec5e8587252a11a7e401d64/html5/thumbnails/31.jpg)
Databases
Two basic types
NoSQL
SQL
31
![Page 32: IoT Arhitecture, database and big data - ACEIoT · Gateway –Centric IoT Cloud (1/2) IoT infrastructure will provide the opportunities to take services, workloads, applications and](https://reader034.fdocuments.us/reader034/viewer/2022042220/5ec5e8587252a11a7e401d64/html5/thumbnails/32.jpg)
Data model
Key/Value
Memcached, Dynamo
Tabular
Big Table
Document Orieneted
MongoDB, CouchdB, JSON stores
32
![Page 33: IoT Arhitecture, database and big data - ACEIoT · Gateway –Centric IoT Cloud (1/2) IoT infrastructure will provide the opportunities to take services, workloads, applications and](https://reader034.fdocuments.us/reader034/viewer/2022042220/5ec5e8587252a11a7e401d64/html5/thumbnails/33.jpg)
NoSQL
A form of database management system
that is non-relational
System are often schema less, avoid joins
& are easy to scale
33
![Page 34: IoT Arhitecture, database and big data - ACEIoT · Gateway –Centric IoT Cloud (1/2) IoT infrastructure will provide the opportunities to take services, workloads, applications and](https://reader034.fdocuments.us/reader034/viewer/2022042220/5ec5e8587252a11a7e401d64/html5/thumbnails/34.jpg)
But why Choose NoSQL
Amount of data stored is on the
up and up
The data we store is more
complete than before
All the data is need to be easy to
be able to add/remove servers
without any disruption of services
34
![Page 35: IoT Arhitecture, database and big data - ACEIoT · Gateway –Centric IoT Cloud (1/2) IoT infrastructure will provide the opportunities to take services, workloads, applications and](https://reader034.fdocuments.us/reader034/viewer/2022042220/5ec5e8587252a11a7e401d64/html5/thumbnails/35.jpg)
MongoDB
Document base
Schema-less
Highly scalable
Easy replication & sharing
35
![Page 36: IoT Arhitecture, database and big data - ACEIoT · Gateway –Centric IoT Cloud (1/2) IoT infrastructure will provide the opportunities to take services, workloads, applications and](https://reader034.fdocuments.us/reader034/viewer/2022042220/5ec5e8587252a11a7e401d64/html5/thumbnails/36.jpg)
Basic of Big data and database
![Page 37: IoT Arhitecture, database and big data - ACEIoT · Gateway –Centric IoT Cloud (1/2) IoT infrastructure will provide the opportunities to take services, workloads, applications and](https://reader034.fdocuments.us/reader034/viewer/2022042220/5ec5e8587252a11a7e401d64/html5/thumbnails/37.jpg)
BIG data is always not well understood
Smart Data
![Page 38: IoT Arhitecture, database and big data - ACEIoT · Gateway –Centric IoT Cloud (1/2) IoT infrastructure will provide the opportunities to take services, workloads, applications and](https://reader034.fdocuments.us/reader034/viewer/2022042220/5ec5e8587252a11a7e401d64/html5/thumbnails/38.jpg)
Big data: 3V’s (volume)
Volume
Large data size• What does mean size?
– Not gigabytes
– Most likely not a few terabytes
– Possibly not 10’s of terabytes
– Probably 100’s of terabytes
– Definitely petabytes
![Page 39: IoT Arhitecture, database and big data - ACEIoT · Gateway –Centric IoT Cloud (1/2) IoT infrastructure will provide the opportunities to take services, workloads, applications and](https://reader034.fdocuments.us/reader034/viewer/2022042220/5ec5e8587252a11a7e401d64/html5/thumbnails/39.jpg)
Big Data: 3V’s (velocity)
Velocity
real-time
near-real time
streaming data flow
![Page 40: IoT Arhitecture, database and big data - ACEIoT · Gateway –Centric IoT Cloud (1/2) IoT infrastructure will provide the opportunities to take services, workloads, applications and](https://reader034.fdocuments.us/reader034/viewer/2022042220/5ec5e8587252a11a7e401d64/html5/thumbnails/40.jpg)
Big Data: 3V’s (variety)
Variety diverse data (structure and
unstructured, diverse data models and
query languages, diverse data sources
![Page 41: IoT Arhitecture, database and big data - ACEIoT · Gateway –Centric IoT Cloud (1/2) IoT infrastructure will provide the opportunities to take services, workloads, applications and](https://reader034.fdocuments.us/reader034/viewer/2022042220/5ec5e8587252a11a7e401d64/html5/thumbnails/41.jpg)
Some make it 4V’s
![Page 42: IoT Arhitecture, database and big data - ACEIoT · Gateway –Centric IoT Cloud (1/2) IoT infrastructure will provide the opportunities to take services, workloads, applications and](https://reader034.fdocuments.us/reader034/viewer/2022042220/5ec5e8587252a11a7e401d64/html5/thumbnails/42.jpg)
IoT in BIG data
IoT presents challenges in combination of all BIG data characteristics (3Vs/4Vs)
Most challenging IoT applications impact both Velocity & Volume and sometimes also Variety (situation and context)
Today..
GE each day gathers 50 million pieces of data from 10 million sensors, off equipment worth $1 trillion
A wearable sensor produces about 55 million data points pro day (challenge for storage), whereas some medical wearable's (like ECG) produce up to 1000 events per second (challenge for real-time processing)
![Page 43: IoT Arhitecture, database and big data - ACEIoT · Gateway –Centric IoT Cloud (1/2) IoT infrastructure will provide the opportunities to take services, workloads, applications and](https://reader034.fdocuments.us/reader034/viewer/2022042220/5ec5e8587252a11a7e401d64/html5/thumbnails/43.jpg)
IoT real-time Big data
Real-time Big Data (Fast data)
Real-time Data coming from (mainly)
Monitoring systems (e.g. Nagios)
Sensors
Stationary sensors (environment)
Embedded (e.g. in mobile devices)
Wearable sensors (e.g. HR monitors)
extreme velocity
mobile data streams
extreme resource
constraints
EMERGING
GREAT BUSINESS OPPORTUNITIES
e.g. there should be about 250 Million Wearable Health & Fitness Sensing Devices by 2017.
The market for sports and fitness apps will cross $400 million in 2016
![Page 44: IoT Arhitecture, database and big data - ACEIoT · Gateway –Centric IoT Cloud (1/2) IoT infrastructure will provide the opportunities to take services, workloads, applications and](https://reader034.fdocuments.us/reader034/viewer/2022042220/5ec5e8587252a11a7e401d64/html5/thumbnails/44.jpg)
BIG Data is nothing without BIG
business value insight
IoT is a huge opportunity for BIG data
..and opposite
IoT without BiG DATA is first generation IoT 1.0
IoT with BIG DATA is the third generation IoT 3.0 (vision of future world)
=+
IoT BIG Data BIG Value
![Page 45: IoT Arhitecture, database and big data - ACEIoT · Gateway –Centric IoT Cloud (1/2) IoT infrastructure will provide the opportunities to take services, workloads, applications and](https://reader034.fdocuments.us/reader034/viewer/2022042220/5ec5e8587252a11a7e401d64/html5/thumbnails/45.jpg)
© 2016 Abdur Rahim Biswas
IoT and Big Data
Content
Big data introduction
IoT big data requirements and platform
IoT big data technologies and tools
![Page 46: IoT Arhitecture, database and big data - ACEIoT · Gateway –Centric IoT Cloud (1/2) IoT infrastructure will provide the opportunities to take services, workloads, applications and](https://reader034.fdocuments.us/reader034/viewer/2022042220/5ec5e8587252a11a7e401d64/html5/thumbnails/46.jpg)
IoT BIG data applications
Deep understanding (observe of behavior of many thing””, gain important insight
Health example (understanding the cause of diseases/comorbidities/indicators)
Real-time actionable insight (Real-time analytic, detect and react in real-time)
Health example (real-time fall detection and potential reaction for aging population)
Performance optimization (configuration, energy, health-care)
Health example (Improve overall healthcare efficiency)
Proactive and predictive functional applications
Health example (proactive and prediction identification of diagnostic in healthcare applications (before thing occur)
![Page 47: IoT Arhitecture, database and big data - ACEIoT · Gateway –Centric IoT Cloud (1/2) IoT infrastructure will provide the opportunities to take services, workloads, applications and](https://reader034.fdocuments.us/reader034/viewer/2022042220/5ec5e8587252a11a7e401d64/html5/thumbnails/47.jpg)
Deep understanding applications
challenges
This vision boils down to solve multiple challenges:
to store all the events (Velocity & Volume Challenge);
to run analytical queries over the stored events; (Velocity & Volume Challenge)
to perform analytics (data mining and machine learning) over the data to gain insights (Velocity & Volume & Variety Challenge);
![Page 48: IoT Arhitecture, database and big data - ACEIoT · Gateway –Centric IoT Cloud (1/2) IoT infrastructure will provide the opportunities to take services, workloads, applications and](https://reader034.fdocuments.us/reader034/viewer/2022042220/5ec5e8587252a11a7e401d64/html5/thumbnails/48.jpg)
Real-time actionable Insights
Real-time detection and action represent multiple challenges
How to make reliable knowledge and decision from BIG Data? (Veracity and Verity)
How to process (real-time process and data interpretation) the streaming/real-time events on the fly (Velocity challenges)
How to store the events in the operational database (Velocity challenges)
How to correlate streaming events with store data in the operational database (velocity and Volume challenges)
![Page 49: IoT Arhitecture, database and big data - ACEIoT · Gateway –Centric IoT Cloud (1/2) IoT infrastructure will provide the opportunities to take services, workloads, applications and](https://reader034.fdocuments.us/reader034/viewer/2022042220/5ec5e8587252a11a7e401d64/html5/thumbnails/49.jpg)
Performance optimization challenges
When the “old way” of processing data just doesn’t work effectively
How we store the diverse set of BIG data, from mobile/sensors/server? (much data
How we move that much data
How we extract, load & transform that much data
How we explore and analyze that much data
How we process and get meaningful insights from that much data
![Page 50: IoT Arhitecture, database and big data - ACEIoT · Gateway –Centric IoT Cloud (1/2) IoT infrastructure will provide the opportunities to take services, workloads, applications and](https://reader034.fdocuments.us/reader034/viewer/2022042220/5ec5e8587252a11a7e401d64/html5/thumbnails/50.jpg)
Prediction challenges
Context
Long-period of time
![Page 51: IoT Arhitecture, database and big data - ACEIoT · Gateway –Centric IoT Cloud (1/2) IoT infrastructure will provide the opportunities to take services, workloads, applications and](https://reader034.fdocuments.us/reader034/viewer/2022042220/5ec5e8587252a11a7e401d64/html5/thumbnails/51.jpg)
IoT Big data platform requirements
CognitiveScalable
Real-time Unified view
![Page 52: IoT Arhitecture, database and big data - ACEIoT · Gateway –Centric IoT Cloud (1/2) IoT infrastructure will provide the opportunities to take services, workloads, applications and](https://reader034.fdocuments.us/reader034/viewer/2022042220/5ec5e8587252a11a7e401d64/html5/thumbnails/52.jpg)
Data
sourceData
source
Architecture of IoT Big data platform
![Page 53: IoT Arhitecture, database and big data - ACEIoT · Gateway –Centric IoT Cloud (1/2) IoT infrastructure will provide the opportunities to take services, workloads, applications and](https://reader034.fdocuments.us/reader034/viewer/2022042220/5ec5e8587252a11a7e401d64/html5/thumbnails/53.jpg)
Major big data analytic platform
![Page 54: IoT Arhitecture, database and big data - ACEIoT · Gateway –Centric IoT Cloud (1/2) IoT infrastructure will provide the opportunities to take services, workloads, applications and](https://reader034.fdocuments.us/reader034/viewer/2022042220/5ec5e8587252a11a7e401d64/html5/thumbnails/54.jpg)
IoT Big data framework- INTEL
![Page 55: IoT Arhitecture, database and big data - ACEIoT · Gateway –Centric IoT Cloud (1/2) IoT infrastructure will provide the opportunities to take services, workloads, applications and](https://reader034.fdocuments.us/reader034/viewer/2022042220/5ec5e8587252a11a7e401d64/html5/thumbnails/55.jpg)
Traditional methods Big data
Centralize Distributed
More power More machines
Summarize data Keep all data
Transform and store Transform on demand
Pre-define schema Flexible/no-schema
Move data toward compute Move compute towards data
Less data/more complex algorithms More data/simple algorithms
Philosophical differences of Big data analytic
![Page 56: IoT Arhitecture, database and big data - ACEIoT · Gateway –Centric IoT Cloud (1/2) IoT infrastructure will provide the opportunities to take services, workloads, applications and](https://reader034.fdocuments.us/reader034/viewer/2022042220/5ec5e8587252a11a7e401d64/html5/thumbnails/56.jpg)
© 2016 Abdur Rahim Biswas
IoT and Big Data
Content
Big data introduction
IoT big data requirements and platform
IoT big data technologies and tools
![Page 57: IoT Arhitecture, database and big data - ACEIoT · Gateway –Centric IoT Cloud (1/2) IoT infrastructure will provide the opportunities to take services, workloads, applications and](https://reader034.fdocuments.us/reader034/viewer/2022042220/5ec5e8587252a11a7e401d64/html5/thumbnails/57.jpg)
BIG data tools
![Page 58: IoT Arhitecture, database and big data - ACEIoT · Gateway –Centric IoT Cloud (1/2) IoT infrastructure will provide the opportunities to take services, workloads, applications and](https://reader034.fdocuments.us/reader034/viewer/2022042220/5ec5e8587252a11a7e401d64/html5/thumbnails/58.jpg)
Hadoop ecosystem- well known tools
![Page 59: IoT Arhitecture, database and big data - ACEIoT · Gateway –Centric IoT Cloud (1/2) IoT infrastructure will provide the opportunities to take services, workloads, applications and](https://reader034.fdocuments.us/reader034/viewer/2022042220/5ec5e8587252a11a7e401d64/html5/thumbnails/59.jpg)
Hadoop: The disruptive technology at
the core of Big data
![Page 60: IoT Arhitecture, database and big data - ACEIoT · Gateway –Centric IoT Cloud (1/2) IoT infrastructure will provide the opportunities to take services, workloads, applications and](https://reader034.fdocuments.us/reader034/viewer/2022042220/5ec5e8587252a11a7e401d64/html5/thumbnails/60.jpg)
Batch and stream processing
Hadoop MapReduce Batch Processing
Storm Streaming Stream Processing
![Page 61: IoT Arhitecture, database and big data - ACEIoT · Gateway –Centric IoT Cloud (1/2) IoT infrastructure will provide the opportunities to take services, workloads, applications and](https://reader034.fdocuments.us/reader034/viewer/2022042220/5ec5e8587252a11a7e401d64/html5/thumbnails/61.jpg)
Stream Processing
Handles data at high velocity
If Hadoop is the ocean, streams are the firehose
Processing in near real-time
![Page 62: IoT Arhitecture, database and big data - ACEIoT · Gateway –Centric IoT Cloud (1/2) IoT infrastructure will provide the opportunities to take services, workloads, applications and](https://reader034.fdocuments.us/reader034/viewer/2022042220/5ec5e8587252a11a7e401d64/html5/thumbnails/62.jpg)
Storm
![Page 63: IoT Arhitecture, database and big data - ACEIoT · Gateway –Centric IoT Cloud (1/2) IoT infrastructure will provide the opportunities to take services, workloads, applications and](https://reader034.fdocuments.us/reader034/viewer/2022042220/5ec5e8587252a11a7e401d64/html5/thumbnails/63.jpg)
Storm
Storm is a distributed data processing system whose processing is based on elements called spouts and bolts
The topology consists out of spouts which are message originators for the rest of the topology
The processing elements in the topology are called bolts which can be interconnected with an internal pub/sub mechanism
Bolts can also deliver their results to other systems such as DBMS’s, legacy systems or applications.
![Page 64: IoT Arhitecture, database and big data - ACEIoT · Gateway –Centric IoT Cloud (1/2) IoT infrastructure will provide the opportunities to take services, workloads, applications and](https://reader034.fdocuments.us/reader034/viewer/2022042220/5ec5e8587252a11a7e401d64/html5/thumbnails/64.jpg)
Lambda architecture combine
![Page 65: IoT Arhitecture, database and big data - ACEIoT · Gateway –Centric IoT Cloud (1/2) IoT infrastructure will provide the opportunities to take services, workloads, applications and](https://reader034.fdocuments.us/reader034/viewer/2022042220/5ec5e8587252a11a7e401d64/html5/thumbnails/65.jpg)
Complex Architectures Using Many Big
Data Technologies