Hortonworks Data in Motion Webinar Series - Part 6, Edge Intelligence IoT MiniFi
-
Upload
hortonworks -
Category
Technology
-
view
3.049 -
download
2
Transcript of Hortonworks Data in Motion Webinar Series - Part 6, Edge Intelligence IoT MiniFi
1 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Harnessing Data-in-Motion with Hortonworks DataFlow
Edge intelligence for IoT with Apache MiNiFi (subproject of Apache NiFi)
Joe PercivallSoftware Engineer
Anna YongProduct Marketing
2 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Agenda• Apache NiFi Refresher
• Intro to Apache MiNiFi
• Demo
• Next Step
• How MiNiFi can be used – use cases
3 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
EMBRACE AN OPEN APPROACH
MASTER THE VALUE OF DATA
EVERY BUSINESS IS A DATA BUSINESS
4 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Constrained High-latency Localized context
Hybrid – cloud / on-premises Low-latency Global context
CoreInfrastructure
Harnessing Data in Motion
RegionalInfrastructureSources
5 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Hortonworks DataFlow Manages Data in Motion
CoreInfrastructureSources
Constrained High-latency Localized context
Hybrid – cloud / on-premises Low-latency Global context
RegionalInfrastructure
6 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Requirements for Data in Motion
Perishable Insights
ConnectivityPrioritization
Security
AdaptabilityExtensibility
Scalability
Provenance
Real-Time
7 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Connecting Data Between Ecosystems Without Coding: 170+ Processors
Hash
Extract
Merge
Duplicate
Scan
GeoEnrich
Replace
ConvertSplit
Translate
Route Content
Route Context
Route Text
Control Rate
Distribute Load
Generate Table Fetch
Jolt Transform JSON
Prioritized Delivery
Encrypt
Tail
Evaluate
Execute
HL7
FTP
UDP
XML
SFTP
HTTP
Syslog
HTML
Image
AMQP
MQTT
All Apache project logos are trademarks of the ASF and the respective projects.
Fetch
8 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Apache NiFi
Guaranteed delivery Data buffering
‒ Backpressure‒ Pressure release
Prioritized queuing Flow specific QoS
‒ Latency vs. throughput‒ Loss tolerance
Data provenance
Recovery / recording a rolling log of fine-grained history
Designed for extension Visual command and control Flow templates Policy based security Clustering
Key Features
9 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Agenda• Apache NiFi Refresher
• Intro to Apache MiNiFi
• Demo
• Next Step
• How MiNiFi can be used – use cases
10 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Simplified Ingestion ExampleLet’s consider the needs of a courier service
Physical Store
Gateway Server
Mobile Devices
Registers
Server Cluster
Distribution Center Core Data Center at HQ
Server Cluster
On Delivery Routes
Trucks Deliverers
Delivery Truck: Creative Stall, https://thenounproject.com/creativestall/Deliverer: Rigo Peter, https://thenounproject.com/rigo/Cash Register: Sergey Patutin, https://thenounproject.com/bdesign.by/Hand Scanner: Eric Pearson, https://thenounproject.com/epearson001/
11 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Courier service from the perspective of NiFi
Physical Store
Gateway Server
Mobile Devices
Registers
Server Cluster
Distribution Center Core Data Center at HQ
Server Cluster
Trucks Deliverers
Delivery Truck: Creative Stall, https://thenounproject.com/creativestall/Deliverer: Rigo Peter, https://thenounproject.com/rigo/Cash Register: Sergey Patutin, https://thenounproject.com/bdesign.by/Hand Scanner: Eric Pearson, https://thenounproject.com/epearson001/
NiFi NiFi NiFi NiFi NiFi NiFi
On Delivery Routes
12 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Courier service from the perspective of NiFi & MiNiFi
Physical Store
Gateway Server
Mobile Devices
Registers
Server Cluster
Distribution Center Core Data Center at HQ
Server Cluster
Trucks Deliverers
Delivery Truck: Creative Stall, https://thenounproject.com/creativestall/Deliverer: Rigo Peter, https://thenounproject.com/rigo/Cash Register: Sergey Patutin, https://thenounproject.com/bdesign.by/Hand Scanner: Eric Pearson, https://thenounproject.com/epearson001/
Client Libraries
Client Libraries
MiNiFi
MiNiFi NiFi NiFi NiFi NiFi NiFi NiFi
Client Libraries
On Delivery Routes
13 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Created to more effectively collect data at the edge
14 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Apache NiFi
Guaranteed delivery Data buffering
‒ Backpressure‒ Pressure release
Prioritized queuing Flow specific QoS
‒ Latency vs. throughput‒ Loss tolerance
Data provenance
Recovery / recording a rolling log of fine-grained history
Designed for extension Visual command and control Flow templates Policy based security Clustering
Key Features
15 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Guaranteed delivery Data buffering
‒ Backpressure‒ Pressure release
Prioritized queuing Flow specific QoS
‒ Latency vs. throughput‒ Loss tolerance
Data provenance
Different from Apache NiFi Design and Deploy Warm re-deploys
Key Features
Apache NiFi MiNiFi
Recovery / recording a rolling log of fine-grained history
Designed for extension
16 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Command and Controlvs.
Design and Deploy
17 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Different MiNiFi Distributions
Java– <40MB binary distribution– Requires Java 1.8– More feature complete– Targeted for any systems that can run a JVM (ie. Servers, Raspberry Pi)
C++– 600KB code size and static data ~50KB– Dynamic heap of ~1MB based on use-case– Targeted for resource constrained environments (ie. edge IoT devices)
Both use same config format and use NiFi terminology
Different focuses depending on requirements
18 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Typical Simple ExampleTail a rolling file -> Site to Site
19 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Typical Simple ExampleTail a rolling file -> Site to Site
20 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Simple Config.ymlTail a rolling file -> Site to Site
21 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
NiFi and MiNiFi, JAVA and C++
22 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
NiFi vs MiNiFi Java Processes
NiFi Framework
Components
MiNiFi
NiFi Framework
User Interface
Components
NiFi
23 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
NiFi Java Processes
Bootstrap
NiFi
UI
bootstrap.conf
nifi.properties
flow.xml.gzreads &modifies
reads
reads
starts
NiFi MiNiFi
24 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
MiNiFi Java Processes
MiNiFi
Bootstrap
ConfigurationChange Notifier(s)
bootstrap.conf
nifi.properties
flow.xml.gzreads
reads
starts
config.ymltransforms
reads
into
NiFi MiNiFi
25 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Same Extensible framework (nars)
NiFi standard processors are bundled– TailLog– UpdateAttribute– Route on content and attributes– PutEmail– ….
Allows MiNiFi Java to use NiFi processors
26 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
MiNiFi C++ Features Same design and deploy configuration Initial set of processors
– TailFile– GetFile– GenerateFlowFile– LogAttribute– ListenSyslog
Site to Site Client implementation in C++ for talking to NiFi instances
27 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Agenda• Apache NiFi Refresher
• Intro to Apache MiNiFi
• Demo
• Next Step
• How MiNiFi can be used – use cases
28 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Simple Config.ymlTail a rolling file -> Site to Site
29 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Courier service from the perspective of NiFi & MiNiFi
Physical Store
Gateway Server
Mobile Devices
Registers
Server Cluster
Distribution Center Core Data Center at HQ
Server Cluster
Trucks Deliverers
Delivery Truck: Creative Stall, https://thenounproject.com/creativestall/Deliverer: Rigo Peter, https://thenounproject.com/rigo/Cash Register: Sergey Patutin, https://thenounproject.com/bdesign.by/Hand Scanner: Eric Pearson, https://thenounproject.com/epearson001/
Client Libraries
Client Libraries
MiNiFi
MiNiFi NiFi NiFi NiFi NiFi NiFi NiFi
Client Libraries
On Delivery Routes
30 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Demo: Digital Temp & Humidity Sensor, Raspberry Pi Zero, MiNiFi
Raspberry Pi Zero
with MiNiFi(link) USB hub for
power & WiFi
Digital Temp & Humidity Measurement Sensor
31 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Agenda• Apache NiFi Refresher
• Intro to Apache MiNiFi
• Demo
• Next Step
• How MiNiFi can be used – use cases
32 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
MiNiFi Command and Control
Design Flow at a centralized place, deploy on the edge Version control of flows
– Align with NiFi SDLC work
Agent status monitoring Bi-directional command and control
Currently a feature proposal, initial version being architected
https://cwiki.apache.org/confluence/display/MINIFI/MiNiFi+Command+and+Control
33 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Agenda• Apache NiFi Refresher
• Intro to Apache MiNiFi
• Demo
• Next Step
• How MiNiFi can be used – use cases
34 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
CoreInfrastructure
Hortonworks DataFlow Use CasesRegional
InfrastructureSources
Dataflow Management• On-ramp into Hadoop• Log Collection / Splunk Optimization• Cyber Security• IoT Ingestion• Deliver data into stream processing engines
Real-time Event Processing (Kafka, Storm)
Move data between from on-prem and cloud environments
35 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Apache MiNiFi Use Case: Distributed Systems: Retail POS
100’s or 1000’s or more terminals to gather: Real-time pricing info Real-time inventory updates Real-time offers Traceable logs of transactions Log of user-interactions for trouble-shooting
devices Predictive help desk
Apache MiNiFi Use Case: Distributed Systems: Connected Car
MiNiFi
SD Card
Wi-Fi
DSRC
LTE
Raw data via CAN, Ethernet
Car Wireless Gateway
Buffer offline delay-tolerant data
1
Real-time data
1
4
Wi-Fi AP
DSRC RSU
eNodeB
5
5
Possibly modify route or adjust sampling rate and prepare to flush delay-tolerant buffers when SDRC RSU or Wi-Fi AP in range
2
Data Storage
Decision-Making Server
Map Database
3
Hortonworks maximizes intelligence with platforms that deliver actionable insights from connected car data
37 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
GENIVI Alliance: Open Source, In-Vehicle Infotainment Software
The GENIVI Alliance is a nonprofit industry alliance committed to driving the broad adoption of specified, open source, In-Vehicle Infotainment software.
38 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Apache MiNiFi Use Cases: Distributed Systems of Devices
Routers Security cameras Cable modems ATM
Fleet of Trucks Manufacturing Line Security Appliance Point of Sale
Weather detection system Thermostats Utility/Power meters Fleet of Ships
Any distributed system of devices with data to be collected
39 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Questions?
40 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Learn more and join us!
Apache NiFi sitehttp://nifi.apache.org
Subproject MiNiFi sitehttp://nifi.apache.org/minifi/
Subscribe to and collaborate [email protected]@nifi.apache.org
Submit Ideas or Issueshttps://issues.apache.org/jira/browse/NIFIhttps://issues.apache.org/jira/browse/MINIFI
Follow us on Twitter@apachenifi
41 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Questions?
Hortonworks Community Connection:Data Ingestion and Streaminghttps://community.hortonworks.com/
42 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Thank you!