Apache NiFiApache NiFi21st century open-source data flows21st century open-source data flows

München, 2018-05-18Frank Thiele

ContentContentIntroduction

Technology

Functionality

Live demo

Summary

Apache NiFiApache NiFiBased on NSA project NiagaraFilesAutomation of data flows between applicationsAvailable under Apache License since 2014Development taken over by Hortonworks (2015)Release v1.0.0, August 2016Latest v1.6.0, April 2018

Apache NiFi – What is it?Apache NiFi – What is it?...comparable to Ab Initio or Talend Data Integration...or Apache Camel

Data Flow Management Platform...more than a classical ETL Framework

It models your data flowIt models your data flow

... and more complex data flows... and more complex data flows

Apache NiFi – Use casesApache NiFi – Use casesWorkflow modeling with data flowsConnects different technology stacksMass data processingCentralization of complex data flowsAccountability of data flowsIOT, Telecommunication, Banking, ...ETL – Extract, Transform, Load

ContentContentIntroduction

Technology

Functionality

Live demo

Summary

ConceptsConceptsFlow-based programmingFlowFiles, processors, controllers, groups and connectionsPass-by-reference between processorsDedicated repositories as storageHappy and error paths are modeled equivalently

Flow-based programmingFlow-based programming

FlowFilesFlowFilesThe data bucket of NiFiMeta data container

UUIDAttributesContent link

Best Practice: Modify and read attributes, not contentTransferred from one processor to another – by reference

Repositories – FlowFileRepositories – FlowFileStorage for meta dataWrite-ahead log for data consistency/persistenceSnapshots/Checkpoints for restoringCopy-on-writeSwapping supported

Repositories – ContentRepositories – ContentStorage for actual data of a FlowFileCan be scaled over partitions and cluster nodesContent is immutable (write-once, copy-on-write)

Repositories – ProvenanceRepositories – ProvenanceStorage for history of the FlowFilesCan be scaled over partitions and cluster nodesAny FlowFile can be viewed at any point in timeApache Lucene Index

ContentContentIntroduction

Technology

Functionality

Live demo

Summary

FlexibilityFlexibilityHigh number of available processors (ca. 250)Freely programmable processors for customizationsMany programming languages supported in addition to Java(e.g. Python, Lua, JS)Template supportConfiguration changes of the stream without downtimeNiFi Registry for configuration management

NiFi RegistryNiFi RegistrySince v1.5, so January 2018Configuration management

NiFi Registry UINiFi Registry UI

NiFi Registry integrationNiFi Registry integration

Further advantagesFurther advantagesOpen source (active development by Hortonworks)Paradigm shift from file and content to events and attributesAdvanced backpressure controlMulti-tenancyUnit and integration testsReduced I/O overhead

ContentContentIntroduction

Technology

Functionality

Live demo

Summary

Live demoLive demoUI OverviewStream exampleFeature examples

Data provenanceBack pressureError handlingETL with Jolt

UI overviewUI overview

Stream exampleStream example

Processor configurationProcessor configuration

Data provenanceData provenance

Data provenance – GraphData provenance – Graph

Data provenance – DetailsData provenance – Details

Data provenance – AttributesData provenance – Attributes

Data provenance - Content 1Data provenance - Content 1

Data provenance – Content 2Data provenance – Content 2

Back pressureBack pressure

Error handlingError handling

ETL with JOLTETL with JOLT

ContentContentIntroduction

Technology

Functionality

Live demo

Summary

Apache NiFi – Why again?Apache NiFi – Why again?Used in many productive systems (e.g. NSA, Slovak Telekom, ...)Solves many things already (UI, Tracing, Transactional, ...)Easily customizableOpen Source

Reduce your OPEXCan be extended by YOUIs extended and fixed by Community

It scalesCloud ready

Apache NiFi – Use casesApache NiFi – Use casesWorkflow modeling with data flowsReduce latency of your dataCentralization of complex data flowsBig Data and BI data flowsIntegration of new/different technologiesAccountability and lineageComplex Event Processing*ETL*

ContactContactFrank Thiele (frank.thiele@tngtech.com)