Apache NiFi- MiNiFi meetup Slides

41
Apache NiFi – MiNiFi Taking Dataflow Management to the Edge Joe Percivall - @JPercivall

Transcript of Apache NiFi- MiNiFi meetup Slides

Page 1: Apache NiFi- MiNiFi meetup Slides

Apache NiFi – MiNiFiTaking Dataflow Management to the EdgeJoe Percivall - @JPercivall

Page 2: Apache NiFi- MiNiFi meetup Slides

2 © Hortonworks Inc. 2011 – 2016. All Rights Reserved

About Me• Software Engineer at Hortonworks

• Apache NiFi committer and PMC member

• Github: github.com/JPercivall

Page 3: Apache NiFi- MiNiFi meetup Slides

3 © Hortonworks Inc. 2011 – 2016. All Rights Reserved

Agenda• Why create MiNiFi?

• MiNiFi 0.0.1-Java

• Demo

• Prospective plans

Page 4: Apache NiFi- MiNiFi meetup Slides

4 © Hortonworks Inc. 2011 – 2016. All Rights Reserved

Agenda• Why create MiNiFi?

• MiNiFi 0.0.1-Java

• Demo

• Prospective plans

Page 5: Apache NiFi- MiNiFi meetup Slides

5 © Hortonworks Inc. 2011 – 2016. All Rights Reserved

• Web-based User Interface for creating, monitoring, & controlling data flows

• Directed graphs of data routing and transformation

• Highly configurable - modify data flow at runtime, dynamically prioritize data

• Easily extensible through development of custom components

• Data Provenance tracks data through entire system

[1] https://nifi.apache.org/

Apache NiFiDataflow

Page 6: Apache NiFi- MiNiFi meetup Slides

6 © Hortonworks Inc. 2011 – 2016. All Rights Reserved

Apache NiFiKey Features

• Guaranteed delivery• Data buffering

- Backpressure- Pressure release

• Prioritized queuing• Flow specific QoS

- Latency vs. throughput- Loss tolerance

• Data provenance• Supports push and pull

models

• Recovery/recording a rolling log of fine-grained history

• Visual command and control

• Flow templates• Pluggable/multi-role

security• Designed for extension• Clustering

Page 7: Apache NiFi- MiNiFi meetup Slides

7 © Hortonworks Inc. 2011 – 2016. All Rights Reserved

Simplified ExampleLet’s consider the needs of a courier service

Physical Store

Gateway Server

Mobile Devices

Registers

Server Cluster

Distribution Center Core Data Center at HQ

Server Cluster

On Delivery Routes

Trucks Deliverers

Delivery Truck: Creative Stall, https://thenounproject.com/creativestall/Deliverer: Rigo Peter, https://thenounproject.com/rigo/Cash Register: Sergey Patutin, https://thenounproject.com/bdesign.by/Hand Scanner: Eric Pearson, https://thenounproject.com/epearson001/

Page 8: Apache NiFi- MiNiFi meetup Slides

8 © Hortonworks Inc. 2011 – 2016. All Rights Reserved

Courier service from the perspective of NiFi & MiNiFi

Physical Store

Gateway Server

Mobile Devices

Registers

Server Cluster

Distribution Center Core Data Center at HQ

Server Cluster

On Delivery Routes

Trucks Deliverers

Delivery Truck: Creative Stall, https://thenounproject.com/creativestall/Deliverer: Rigo Peter, https://thenounproject.com/rigo/Cash Register: Sergey Patutin, https://thenounproject.com/bdesign.by/Hand Scanner: Eric Pearson, https://thenounproject.com/epearson001/

Client Libraries

Client Libraries

MiNiFi

MiNiFi NiFi NiFi NiFi NiFi NiFi NiFi

Client Libraries

Page 9: Apache NiFi- MiNiFi meetup Slides

9 © Hortonworks Inc. 2011 – 2016. All Rights Reserved

Apache NiFi MiNiFiKey Features

• Guaranteed delivery• Data buffering

- Backpressure- Pressure release

• Prioritized queuing• Flow specific QoS

- Latency vs. throughput- Loss tolerance

• Data provenance

• Recovery/recording a rolling log of fine-grained history

• Designed for extension

• Design and Deploy• Warm re-deploys

Page 10: Apache NiFi- MiNiFi meetup Slides

10 © Hortonworks Inc. 2011 – 2016. All Rights Reserved

Apache NiFi MiNiFiKey Features

• Guaranteed delivery• Data buffering

- Backpressure- Pressure release

• Prioritized queuing• Flow specific QoS

- Latency vs. throughput- Loss tolerance

• Data provenance

• Recovery/recording a rolling log of fine-grained history

• Designed for extension

• Design and Deploy• Warm re-deploys

Page 11: Apache NiFi- MiNiFi meetup Slides

11 © Hortonworks Inc. 2011 – 2016. All Rights Reserved

Visual Command and Controlvs.

Design and Deploy

Page 12: Apache NiFi- MiNiFi meetup Slides

12 © Hortonworks Inc. 2011 – 2016. All Rights Reserved

Created to more effectively collect data at the edge

Page 13: Apache NiFi- MiNiFi meetup Slides

13 © Hortonworks Inc. 2011 – 2016. All Rights Reserved

Agenda• Why create MiNiFi?

• MiNiFi 0.0.1-Java

• Demo

• Prospective plans

Page 14: Apache NiFi- MiNiFi meetup Slides

14 © Hortonworks Inc. 2011 – 2016. All Rights Reserved

NiFi vs MiNiFi Java Processes

NiFi Framework

Components

MiNiFi

NiFi Framework

User Interface

Components

NiFi

Page 15: Apache NiFi- MiNiFi meetup Slides

15 © Hortonworks Inc. 2011 – 2016. All Rights Reserved

NiFi Java Processes

Bootstrap

NiFi

UI

bootstrap.conf

nifi.properties

flow.xml.gzreads &modifies

reads

reads

starts

NiFi MiNiFi

Page 16: Apache NiFi- MiNiFi meetup Slides

16 © Hortonworks Inc. 2011 – 2016. All Rights Reserved

MiNiFi Java Processes

MiNiFi

Bootstrap

ConfigurationChange Notifier(s)

bootstrap.conf

nifi.properties

flow.xml.gzreads

reads

starts

config.ymltransforms

reads

into

NiFi MiNiFi

Page 17: Apache NiFi- MiNiFi meetup Slides

17 © Hortonworks Inc. 2011 – 2016. All Rights Reserved

Same Extensible framework (nars)

In minifi-0.0.1, the nifi-0.6.1 standard processors are bundled (~20mb)– Tailing a Log– UpdateAttribute– Routing by content or attributes– PutEmail

Allows MiNiFi to use NiFi processors

Page 18: Apache NiFi- MiNiFi meetup Slides

18 © Hortonworks Inc. 2011 – 2016. All Rights Reserved

MiNiFi 0.0.1-Java

Declarative configuration of processing flows through a YAML configuration file Exporting of provenance events to another NiFi instance via a Reporting Task over Site to

Site Flow change configuration watcher implementations that provide reloading a NiFi

instance when receiving an updated flow over REST or changes on a file system Providing a mechanism to query an instance's status

<40mb binary distribution

Release Notes

Page 19: Apache NiFi- MiNiFi meetup Slides

19 © Hortonworks Inc. 2011 – 2016. All Rights Reserved

Simple Config.ymlTail a rolling file -> Site to Site

Page 20: Apache NiFi- MiNiFi meetup Slides

20 © Hortonworks Inc. 2011 – 2016. All Rights Reserved

But what about the NiFi.properties values?Can omitted for default values

Page 21: Apache NiFi- MiNiFi meetup Slides

21 © Hortonworks Inc. 2011 – 2016. All Rights Reserved

Provenance Reporting

Site to Site Reporting Task JSON formatted provenance events Configured via config.yml Optional

“SiteToSiteProvenanceReportingTask” in NiFi 0.7.0

Page 22: Apache NiFi- MiNiFi meetup Slides

22 © Hortonworks Inc. 2011 – 2016. All Rights Reserved

A bit more complex Config.ymlTail a rolling File -> Secure Site to Site with Provenance

Page 23: Apache NiFi- MiNiFi meetup Slides

23 © Hortonworks Inc. 2011 – 2016. All Rights Reserved

MiNiFi Toolkit

Convert NiFi templates to config.yml

Validate config.yml files

CLI to facilitate config.yml building

Page 24: Apache NiFi- MiNiFi meetup Slides

24 © Hortonworks Inc. 2011 – 2016. All Rights Reserved

Config Change Notifiers Two implementations

– RestChangeNotifier• Http(s)

– FileChangeNotifier

Configured in bootstrap.conf

Page 25: Apache NiFi- MiNiFi meetup Slides

25 © Hortonworks Inc. 2011 – 2016. All Rights Reserved

Change notifier update

MiNiFi

Bootstrap

ConfigurationChange Notifiers

1. Initial state– Both running

Page 26: Apache NiFi- MiNiFi meetup Slides

26 © Hortonworks Inc. 2011 – 2016. All Rights Reserved

Change notifier update

MiNiFi

Bootstrap

ConfigurationChange Notifiers

user creates new configuration2. User sends update through

notifier– HTTP(S) post request– Change watched file

Page 27: Apache NiFi- MiNiFi meetup Slides

27 © Hortonworks Inc. 2011 – 2016. All Rights Reserved

Change notifier update

MiNiFi

Bootstrap

ConfigurationChange Notifiers

3. Bootstrap validation– Basic validation– Rest notifier will respond

accordingly– Results logged

validate new configuration

user creates new configuration

Page 28: Apache NiFi- MiNiFi meetup Slides

28 © Hortonworks Inc. 2011 – 2016. All Rights Reserved

Change notifier update

MiNiFi

Bootstrap

ConfigurationChange Notifiers

config.ymlsaves new

4. Bootstrap saves and transforms

– Copy old config.yml to a swap file

validate new configuration

user creates new configuration

nifi.properties

flow.xml.gz

transforms into

Page 29: Apache NiFi- MiNiFi meetup Slides

29 © Hortonworks Inc. 2011 – 2016. All Rights Reserved

Change notifier update

MiNiFi

Bootstrap

ConfigurationChange Notifiers

nifi.properties

flow.xml.gz

attempt restart

config.ymlsaves new

reads

transforms into

5. Bootstrap attempts restart– MiNiFi reads in the new

nifi.properties and flow.xml.gz

validate new configuration

user creates new configuration

Page 30: Apache NiFi- MiNiFi meetup Slides

30 © Hortonworks Inc. 2011 – 2016. All Rights Reserved

Change notifier update

6. Success or Fail– Successful restart continue

processing– Failure, rollback to old

config– Existing Data is mapped or

orphaned

MiNiFi

Bootstrap

ConfigurationChange Notifiers

nifi.properties

flow.xml.gz

attempt restart

config.ymlsaves new

reads

transforms into

validate new configuration

user creates new configuration

Page 31: Apache NiFi- MiNiFi meetup Slides

31 © Hortonworks Inc. 2011 – 2016. All Rights Reserved

minifi.sh flowStatus

Components Instance System Diagnostics

Get flow status at the command line

Page 32: Apache NiFi- MiNiFi meetup Slides

32 © Hortonworks Inc. 2011 – 2016. All Rights Reserved

Agenda• Why create MiNiFi?

• MiNiFi 0.0.1-Java

• Demo

• Prospective plans

Page 33: Apache NiFi- MiNiFi meetup Slides

33 © Hortonworks Inc. 2011 – 2016. All Rights Reserved

Courier service from the perspective of NiFi

Physical Store

Gateway Server

Mobile Devices

Registers

Server Cluster

Distribution Center Core Data Center at HQ

Server Cluster

On Delivery Routes

Trucks Deliverers

Delivery Truck: Creative Stall, https://thenounproject.com/creativestall/Deliverer: Rigo Peter, https://thenounproject.com/rigo/Cash Register: Sergey Patutin, https://thenounproject.com/bdesign.by/Hand Scanner: Eric Pearson, https://thenounproject.com/epearson001/

Client Libraries

Client Libraries

MiNiFi

MiNiFi NiFi NiFi NiFi NiFi NiFi NiFi

Client Libraries

Page 34: Apache NiFi- MiNiFi meetup Slides

34 © Hortonworks Inc. 2011 – 2016. All Rights Reserved

Agenda• Why create MiNiFi?

• MiNiFi 0.0.1-Java

• Demo

• Prospective plans

Page 35: Apache NiFi- MiNiFi meetup Slides

35 © Hortonworks Inc. 2011 – 2016. All Rights Reserved

Prospective Plans MiNiFi 0.0.1-Cpp

– Close to a vote to release– Code itself is 1.2mb without optimization– Data size

• ~20mb for dynamic RAM for heap• Static ~50kb

Configurable Status Reporters– minifi.sh flowStatus -> regular status update

• MQTT?• HTTP?• S2S?

Handle component Annotation Data– UpdateAttribute advanced rules

Page 36: Apache NiFi- MiNiFi meetup Slides

36 © Hortonworks Inc. 2011 – 2016. All Rights Reserved

Major Feature - Centralized Command and Control

Design at a centralized place, deploy on the edge– Flow deployment– NAR deployment– Agent deployment

Version control of flows Agent status monitoring Bi-directional command and control

Centralized management console with a UI

Page 37: Apache NiFi- MiNiFi meetup Slides

37 © Hortonworks Inc. 2011 – 2016. All Rights Reserved

Questions?

Page 38: Apache NiFi- MiNiFi meetup Slides

38 © Hortonworks Inc. 2011 – 2016. All Rights Reserved

Thank you!

Page 39: Apache NiFi- MiNiFi meetup Slides

39 © Hortonworks Inc. 2011 – 2016. All Rights Reserved

Learn more and join us!

Apache NiFi sitehttp://nifi.apache.org

Subproject MiNiFi sitehttp://nifi.apache.org/minifi/

Subscribe to and collaborate [email protected]@nifi.apache.org

Submit Ideas or Issueshttps://issues.apache.org/jira/browse/NIFI

Follow us on Twitter@apachenifi

Page 40: Apache NiFi- MiNiFi meetup Slides

40 © Hortonworks Inc. 2011 – 2016. All Rights Reserved

Matured at NSA 2006-2014

Brief history of the Apache NiFi Community

• Contributors from Government and several commercial industries

• Releases on a 6-8 week schedule

• Apache NiFi 1.0.0. release on the horizon• Zero-Master Clustering

Code developed at NSA

2006

Today

Achieved TLP

status in just 7 months

July 2015

Code available open source

ASL v2

November 2014

Page 41: Apache NiFi- MiNiFi meetup Slides

41 © Hortonworks Inc. 2011 – 2016. All Rights Reserved

MiNiFi differentiation

Let me get the key parts of NiFi close to where data begins and provide bidrectional communication

NiFi lives in the data center. Give it an enterprise server or a cluster of them. MiNiFi lives close to where data is born and may be a guest on that device or system