Apache NiFi 1.0 in Nutshell

ApacheNiFi 1.0inNutshellKojiKawamura– SoftwareEngineerArti Wadhwani – TechnicalSupportEngineer2016October27

2 ©HortonworksInc.2011–2016.AllRightsReserved

AgendaWhat’sNiFi

NiFi 1.0Enhancements

NiFi ontheedge

Commonissues

What’sNext?

AgendaWhat’sNiFi

NiFi ontheedge

Commonissues

What’sNext?

November 2014NiFi is donated to the Apache Software Foundation (ASF) through NSA’s Technology Transfer Program and enters ASF’s incubator.

2006NiagaraFiles (NiFi) was first incepted at the National Security Agency (NSA)

ABriefHistory

July 2015NiFi reaches ASF top-level project status

” NiFi islikedigging irrigationditchesasthewaterflows,ratherthanbuildingoutasprinklersysteminadvance."

“NiFiは事前にスプリンクラーを配備するというより、水が流れるのに合わせて用水路を整備するようなもんさ”

https://mail-archives.apache.org/mod_mbox/nifi-users/201604.mbox/%3C2FCCBD60-0A79-42F1-9F9B-A121591C826E@apache.org%3E

What’sApacheNiFi?

NiFi isatoolfor

DataFlowManagement

StoreData

ProcessandAnalyzeData

AcquireData

SimplisticViewofDataFlows:Easy,Definitive

Dataflow

RealisticViewofDataflows:Complex,Convoluted

StoreData

ProcessandAnalyzeData

AcquireData

StoreDataStoreData

StoreData

AcquireData

Dataflow

NiFi 1.0has170+Processors,30%IncreasefromNiFi 0.7

Extract

Duplicate

GeoEnrich

Replace

ConvertSplit

Translate

RouteContent

RouteContext

RouteText

ControlRate

DistributeLoad

GenerateTableFetch

JoltTransformJSON

PrioritizedDelivery

Encrypt

Evaluate

Execute

Syslog

AllApacheproject logosaretrademarksoftheASFandtherespectiveprojects.

DeeperEcosystemIntegration– NewProcessors

Processor Description

Publish/ConsumeKafka TwoNARs, withkafka 0.9/0.10clientlibraries, respectively

JoltTransformJson Manipulate JSONdataonthefly,withapreviewfunctionality

GenerateTableFetch Incremental fetch+parallelfetchagainstsourcetablepartitions

PutHiveQL IngesttoHivetables

SelectHiveQL Select fromHivetables

PutHiveStreaming ingeststreamingdatatoHive,leverageHivestreamingAPI

CovertAvroToORC Formatconversation,AvrotoORC

Publish/ConsumeMQTT MQTTisapopularprotocol inIoTworld

SOURCES REGIONALINFRASTRUCTURE

COREINFRASTRUCTURE

DataMovementManagement

ConstrainedHigh-Latency

Localized Context

Hybrid – Cloud/On-PremiseLow-Latency

Global Context

HortonworksDataFlow(HDF)

§ Constrained§ High-latency§ Localizedcontext

§ Hybrid– cloud/on-premises§ Low-latency§ Globalcontext

SOURCES REGIONALINFRASTRUCTURE

COREINFRASTRUCTURE

FlowManagement

DetailedBreakDownofRequirements

Ã Req 1:Acquire datafromvariousWearableDevice’sCloudInstances

Ã Req 2:Move DatafromCustomerCloudInstancestoon-premiseinstance

Ã Req 3:Performintelligent Routing &Filtering ofdata.Theroutingandfilteringruleswillbeoftenchangedatrun-time.

Ã Req 4:Deliver thedatadatatovariousdownstreamsystems.Newdownstreamappsshouldwillalwaysappearandthedatashouldbefedtoitwhenitcomesonline.

Ã Req 5:Parse thedevicedatatostandardizedformatthatdownstreamsysemcanunderstand

Ã Req 6:Enrich thedatawithcontextualinformationincludingpatient/customerinfo(age,gender,etc..)

Ã Req 7:Recognizethepatternwhentherestingheartrateexceedsacertainthreshold(theinsight),andthencreateanalert/notification.

Ã Req 8: RunaOutlierdetectionmodelonstreamingheartratethatcomesin.Ifthescoreisabovecertainthreshold,alertontheheartrate.

StreamProcessing&Analytics

AgendaWhat’sNiFi

NiFi ontheedge

Commonissues

What’sNext?

NiFi 1.0:ModernizedUI

ModernizedUI– CompleteInterfaceRedesign

ConnectComponentstodesignyourdataflow

Component What for?Processor Purposebuiltprocessingunite.g. GetXXX,PutXXXInputPort Receivingdata endpointbtwProcessGroups(local/remote)OutputPort ExposingdataendpointbtwProcessGroups(local/remote)ProcessGroup Musthave,todesignwellstructureddataflowRemoteProcessGroup EnabledatatransferbtwNiFi deploymentsviaSite-to-SiteFunnel Bundlemultiplerelationships intooneTemplate SharepartofdataflowLabel Usefultovisuallygroupprocessors,anddescription

Fromlefttoright

DataProvenance

NiFi 1.0:MultitenantAuthorization

NiFi 0.x- AuthorizationModel

Ã Previouslyhadrolebasedauthorization– DataflowManager(DFM)– Monitor– Provenance– Admin– Proxy– NiFi

Ã Limitation- Allornothingmodel– DFMcanchangeeverything,Monitorcanchangenothing– Can’tgiveauserabilitytomodify/viewonlycertaincomponents– WouldrequirestandingupmultipleNiFiinstances

NiFi 1.0- AuthorizationModel

Ã NiFi 1.0introducesanewdelegatedauthorizationmodel

Ã Authorizeeachrequestbasedonuseridentity,action,andresource– Exampleforuser1modifyingpropertiesonprocessor1:

• UserIdentity:user1• Action:WRITE• Resource:processor1 (uuid)

Ã Ifauthorizersaysresourcenotfound,parentischecked…ifparentisn’tfound,parent’sparentischecked,andsoon…

NiFi 1.0– NiFiManagedAuthorizervs.ExternalAuthorizer

Ã ManagedAuthorizer– Filebasedpersistence

• Couldbebeextendedtootherpersistencemechanisms– NiFiUItomanagepolicies– NiFicontrolsauthorizationlogic

Ã ExternalAuthorizer– Rangerintegration– RangerUItomanagepolicies– Rangercontrolsauthorizationlogic

NiFi 1.0– ManagingUsers

Ã ClickingthenewusericonallowstheadmintocreateUsersandGroups– IndividualUserscanbegrouped– Groupscanbeassigned

members

Ã ClickingtheeditusericonallowstheadmintoupdateaspecificUser/Group

NiFi 1.0– UIOverviewUsersIconinGlobalMenuusedtoaccess

Users/Groups

LockIconinGlobalMenuusedto

accessGlobalpolicies

NiFi 1.0– UIOverview

LockIconinpaletteusedtoaccess

policiesforcurrentlyselectedcomponent

SelectionContext

NiFi 1.0– OverridingComponentPolicies

Ã ComponentinheritpoliciesfromtheclosestancestorProcessGroupwithpoliciesdefined

Ã View/Modifypolicieshandledindependently

Ã ClickOverridetodefineanewpolicy,thenaddUsersandGroups

Ã NewUsersandGroupsoverridetheinheritedpolicies(whitelisting)

NiFi 1.0- Multi-TenancyExample

Ã CreateaGroupforTeam1andaGroupforTeam2

Ã GiveTeam1view&modifyforProcessGroup1

Ã GiveTeam2view&modifyforProcessGroup2

Ã AuserfromTeam1wouldsee:

Can’tseethenameofthegroupandcan’tright-clicktoconfigure thegroup, butcanenterthegroup

NiFi 1.0– Revisions

Ã Revisionpercomponent

Ã Supportsconcurrenteditingofdifferentcomponentswithoutneedforrefreshing

NiFi 1.0:ZeroMasterClustering

NiFi 0.x:NCM(NiFi ClusterManager)

ExternalDataSource

Distributionmechanismdependsondatasource

OtherNiFi

InteractwithNCM

Site-to-Site:GettopologyfromNCMThentransferdatap2p

Primary

NiFi 1.0:ZMC(ZeroMasterClustering)

ExternalDataSource

Distributionmechanismdependsondatasource

OtherNiFi

Interactwithanynode

Site-to-Site:Gettopologyfromoneofnodes

Thentransferdatap2pZookeeper

Primary

Coordinator

ZookeeperelectsClusterCoordinatorandPrimarynode

Anynodecanfail

NiFi 1.0:AndMore!

FoundationalWorkforSDLCÃ Deterministictemplateexport

– Deterministicordering,templatexmlfile

– Versioncontrolofthetemplate

– CollaborativeSDLCeffort

Ã Variableregistry

– Phaseoneimplementation

– In-memoryvariableregistry

– Thesamekeyreferencedinatemplate,mappedtodifferentenvironmental

specificvalues

EntertheTLSToolkit

⬢ Command-linetooltoautomatecertificategenerationandconfiguration

⬢ Self-containedcertificateauthority(CA)forcertificatesigning

⬢ Keystore&truststoregeneration

⬢ Clientcertificategeneration

⬢ Automaticallyupdatesnifi.properties⬢ UnderpinsAmbariTLSintegration

RESTAPI

Framework

Proc CS ReportTask

ExtensionAPI

S2SAPI

S2SClientLibraries

Site-to-SiteRefactoring– S2SHTTP(S)ProtocolthroughProxyServer

Socketprotocol:TCP

HDF2.0:HTTP(s)protocol

HTTPproxy

AgendaWhat’sNiFi

NiFi ontheedge

Commonissues

What’sNext?

EdgeIntelligencewithApacheMiNiFi

Ã GuaranteeddeliveryÃ Databuffering

‒ Backpressure‒ Pressurerelease

Ã PrioritizedqueuingÃ FlowspecificQoS

‒ Latencyvs.throughput‒ Losstolerance

Ã Dataprovenance

Ã Recovery/recordingarollinglogoffine-grainedhistory

Ã Designedforextension

DifferentfromApacheNiFiÃ DesignandDeployÃ Warmre-deploys

KeyFeatures

NiFivs.MiNiFiJavaProcessor,SmallerFootprint~40MB

NiFiFramework

Components

MiNiFi

NiFiFramework

UserInterface

Components

AgendaWhat’sNiFi

NiFi ontheedge

Commonissues

What’sNext?

Commonissues

Ã NiFi RepoconfigurationissuesÃ NiFi SSLconfigurationorcertificateissuesÃ ExecuteStreamCommand ProcessorgettingstuckÃ OutOfMemory IssueswithNCMorprocessors.

BestPractices

Ã DebugLoggingincaseofProcessorissuesÃ CorePropertiesandJVMtuning:

https://community.hortonworks.com/articles/7882/hdfnifi-best-practices-for-setting-up-a-high-perfo.html

UnderstandinghealthviaNiFi UI

StatusBar

ProcessorDetails

NiFi SummaryPage

SystemInformation

AgendaWhat’sNiFi

NiFi ontheedge

Commonissues

What’sNext?

What’sNext

Ã Frameworkextension– Distributeddatadurability(HA

data)– Configurationmanagementflows

(SDLC)Ã EnhancedUserExperience

– Template/ExtensionRegistry– VariableRegistry

Ã Deeperecosystemintegration

Ã CentralCommandandControlÃ NativeAgent(GA)

NiFi MiNiFi

https://cwiki.apache.org/confluence/display/NIFI/Product+requirements

Nifi productrequirements Search!

ThankYou

Apache NiFi 1.0 in Nutshell

Technology

Transcript of Apache NiFi 1.0 in Nutshell

Apache NiFi Toronto Meetup

NJ Hadoop Meetup - Apache NiFi Deep Dive

Introduction to Apache NiFi - RequiTestrequitest.com/uploads/3/4/0/7/34077138/introtonifi.pdf · 2016-03-03 · Introduction to Apache NiFi . What is Apache NiFi? An Open Source Data

Data ingestion and distribution with apache NiFi

APACHE edge with Apache NiFi & MiNiFi Running …...Pierre Villard // @pvillard31 Customer Engineer @ Committer and PMC member for Apache NiFi (in the community since 2015) Twitter/Github

Integrating Apache Spark and NiFi for Data Lakes

UN Global Trusted Platform Data, Applications and Services · •Web Scrapping •Public APIs •Apache NiFi 17. Apache NiFi ... •Supports R, Python Integrated Development Environment

Dataflow with Apache NiFi - Crash Course - HS16SJ

Taking DataFlow Management to the Edge with Apache NiFi/MiNiFi

Dataflow with Apache NiFi - Apache NiFi Meetup - 2016 Hadoop Summit - San Jose

Big Data Systems Infrastructure · * Implementing Apache Nifi Cluster * Simple Integration Project with Apache Nifi Intro to Python Programming * Ingesting Data into Hadoop * Implementing

Building Data Pipelines for Solr with Apache NiFi

Apache NiFi: Ingesting Enterprise Data At Scale

Apache NiFi and MiNiFi: Edge to Core Meetup

Design a Dataflow in 7 minutes with Apache NiFi/HDF

What will be new in Apache NiFi 1.2.0

BigData Techcon - Beyond Messaging with Apache NiFi

Introduction to Apache NiFi And Storm

Beyond Messaging Enterprise Dataflow powered by Apache NiFi

Hands-on with Apache NiFi and MiNiFi - Berlin Buzzwords...Apache NiFi / Integration, or ingestion, Frameworks NiFi End user facing dataflow management tool