Apache NiFi 1.0 in Nutshell

ApacheNiFi 1.0inNutshellKojiKawamura– SoftwareEngineerArti Wadhwani – TechnicalSupportEngineer2016October27

2 ©HortonworksInc.2011–2016.AllRightsReserved

AgendaWhat’sNiFi

NiFi 1.0Enhancements

NiFi ontheedge

Commonissues

What’sNext?


AgendaWhat’sNiFi


NiFi ontheedge

Commonissues

What’sNext?


November 2014NiFi is donated to the Apache Software Foundation (ASF) through NSA’s Technology Transfer Program and enters ASF’s incubator.

2006NiagaraFiles (NiFi) was first incepted at the National Security Agency (NSA)

ABriefHistory

July 2015NiFi reaches ASF top-level project status


” NiFi islikedigging irrigationditchesasthewaterflows,ratherthanbuildingoutasprinklersysteminadvance."

“NiFiは事前にスプリンクラーを配備するというより、水が流れるのに合わせて用水路を整備するようなもんさ”

https://mail-archives.apache.org/mod_mbox/nifi-users/201604.mbox/%[email protected]%3E

What’sApacheNiFi?


NiFi isatoolfor

DataFlowManagement


StoreData

ProcessandAnalyzeData

AcquireData

SimplisticViewofDataFlows:Easy,Definitive

Dataflow


RealisticViewofDataflows:Complex,Convoluted

StoreData

ProcessandAnalyzeData

AcquireData

StoreDataStoreData

StoreData

StoreData

AcquireData

AcquireData

AcquireData

Dataflow


NiFi 1.0has170+Processors,30%IncreasefromNiFi 0.7

Hash

Extract

Merge

Duplicate

Scan

GeoEnrich

Replace

ConvertSplit

Translate

RouteContent

RouteContext

RouteText

ControlRate

DistributeLoad

GenerateTableFetch

JoltTransformJSON

PrioritizedDelivery

Encrypt

Tail

Evaluate

Execute

HL7

FTP

UDP

XML

SFTP

HTTP

Syslog

Email

HTML

Image

AMQP

MQTT

AllApacheproject logosaretrademarksoftheASFandtherespectiveprojects.

Fetch


DeeperEcosystemIntegration– NewProcessors

Processor Description

Publish/ConsumeKafka TwoNARs, withkafka 0.9/0.10clientlibraries, respectively

JoltTransformJson Manipulate JSONdataonthefly,withapreviewfunctionality

GenerateTableFetch Incremental fetch+parallelfetchagainstsourcetablepartitions

PutHiveQL IngesttoHivetables

SelectHiveQL Select fromHivetables

PutHiveStreaming ingeststreamingdatatoHive,leverageHivestreamingAPI

CovertAvroToORC Formatconversation,AvrotoORC

Publish/ConsumeMQTT MQTTisapopularprotocol inIoTworld


SOURCES REGIONALINFRASTRUCTURE

COREINFRASTRUCTURE

DataMovementManagement

ConstrainedHigh-Latency

Localized Context

Hybrid – Cloud/On-PremiseLow-Latency

Global Context


HortonworksDataFlow(HDF)

§ Constrained§ High-latency§ Localizedcontext

§ Hybrid– cloud/on-premises§ Low-latency§ Globalcontext

SOURCES REGIONALINFRASTRUCTURE

COREINFRASTRUCTURE


FlowManagement

DetailedBreakDownofRequirements

Ã Req 1:Acquire datafromvariousWearableDevice’sCloudInstances

Ã Req 2:Move DatafromCustomerCloudInstancestoon-premiseinstance

Ã Req 3:Performintelligent Routing &Filtering ofdata.Theroutingandfilteringruleswillbeoftenchangedatrun-time.

Ã Req 4:Deliver thedatadatatovariousdownstreamsystems.Newdownstreamappsshouldwillalwaysappearandthedatashouldbefedtoitwhenitcomesonline.

Ã Req 5:Parse thedevicedatatostandardizedformatthatdownstreamsysemcanunderstand

Ã Req 6:Enrich thedatawithcontextualinformationincludingpatient/customerinfo(age,gender,etc..)

Ã Req 7:Recognizethepatternwhentherestingheartrateexceedsacertainthreshold(theinsight),andthencreateanalert/notification.

Ã Req 8: RunaOutlierdetectionmodelonstreamingheartratethatcomesin.Ifthescoreisabovecertainthreshold,alertontheheartrate.

StreamProcessing&Analytics


AgendaWhat’sNiFi


NiFi ontheedge

Commonissues

What’sNext?


NiFi 1.0:ModernizedUI


ModernizedUI– CompleteInterfaceRedesign


ConnectComponentstodesignyourdataflow

Component What for?Processor Purposebuiltprocessingunite.g. GetXXX,PutXXXInputPort Receivingdata endpointbtwProcessGroups(local/remote)OutputPort ExposingdataendpointbtwProcessGroups(local/remote)ProcessGroup Musthave,todesignwellstructureddataflowRemoteProcessGroup EnabledatatransferbtwNiFi deploymentsviaSite-to-SiteFunnel Bundlemultiplerelationships intooneTemplate SharepartofdataflowLabel Usefultovisuallygroupprocessors,anddescription

Fromlefttoright


DataProvenance


NiFi 1.0:MultitenantAuthorization


NiFi 0.x- AuthorizationModel

Ã Previouslyhadrolebasedauthorization– DataflowManager(DFM)– Monitor– Provenance– Admin– Proxy– NiFi

Ã Limitation- Allornothingmodel– DFMcanchangeeverything,Monitorcanchangenothing– Can’tgiveauserabilitytomodify/viewonlycertaincomponents– WouldrequirestandingupmultipleNiFiinstances


NiFi 1.0- AuthorizationModel

Ã NiFi 1.0introducesanewdelegatedauthorizationmodel

Ã Authorizeeachrequestbasedonuseridentity,action,andresource– Exampleforuser1modifyingpropertiesonprocessor1:

• UserIdentity:user1• Action:WRITE• Resource:processor1 (uuid)

Ã Ifauthorizersaysresourcenotfound,parentischecked…ifparentisn’tfound,parent’sparentischecked,andsoon…


NiFi 1.0– NiFiManagedAuthorizervs.ExternalAuthorizer

Ã ManagedAuthorizer– Filebasedpersistence

• Couldbebeextendedtootherpersistencemechanisms– NiFiUItomanagepolicies– NiFicontrolsauthorizationlogic

Ã ExternalAuthorizer– Rangerintegration– RangerUItomanagepolicies– Rangercontrolsauthorizationlogic


NiFi 1.0– ManagingUsers

Ã ClickingthenewusericonallowstheadmintocreateUsersandGroups– IndividualUserscanbegrouped– Groupscanbeassigned

members

Ã ClickingtheeditusericonallowstheadmintoupdateaspecificUser/Group


NiFi 1.0– UIOverviewUsersIconinGlobalMenuusedtoaccess

Users/Groups

LockIconinGlobalMenuusedto

accessGlobalpolicies


NiFi 1.0– UIOverview

LockIconinpaletteusedtoaccess

policiesforcurrentlyselectedcomponent

SelectionContext


NiFi 1.0– OverridingComponentPolicies

Ã ComponentinheritpoliciesfromtheclosestancestorProcessGroupwithpoliciesdefined

Ã View/Modifypolicieshandledindependently

Ã ClickOverridetodefineanewpolicy,thenaddUsersandGroups

Ã NewUsersandGroupsoverridetheinheritedpolicies(whitelisting)


NiFi 1.0- Multi-TenancyExample

Ã CreateaGroupforTeam1andaGroupforTeam2

Ã GiveTeam1view&modifyforProcessGroup1

Ã GiveTeam2view&modifyforProcessGroup2

Ã AuserfromTeam1wouldsee:

Can’tseethenameofthegroupandcan’tright-clicktoconfigure thegroup, butcanenterthegroup


NiFi 1.0– Revisions

Ã Revisionpercomponent

Ã Supportsconcurrenteditingofdifferentcomponentswithoutneedforrefreshing


NiFi 1.0:ZeroMasterClustering


NiFi 0.x:NCM(NiFi ClusterManager)

NCM

Node1

Node2

ExternalDataSource

Chunk

Chunk

Chunk

Distributionmechanismdependsondatasource

WebUI

OtherNiFi

InteractwithNCM

Site-to-Site:GettopologyfromNCMThentransferdatap2p

Primary


NiFi 1.0:ZMC(ZeroMasterClustering)

Node1

Node2

Node3

ExternalDataSource

Chunk

Chunk

Chunk

Distributionmechanismdependsondatasource

WebUI

OtherNiFi

Interactwithanynode

Site-to-Site:Gettopologyfromoneofnodes

Thentransferdatap2pZookeeper

Primary

Coordinator

ZookeeperelectsClusterCoordinatorandPrimarynode

Anynodecanfail


NiFi 1.0:AndMore!


FoundationalWorkforSDLCÃ Deterministictemplateexport

– Deterministicordering,templatexmlfile

– Versioncontrolofthetemplate

– CollaborativeSDLCeffort

Ã Variableregistry

– Phaseoneimplementation

– In-memoryvariableregistry

– Thesamekeyreferencedinatemplate,mappedtodifferentenvironmental

specificvalues

34 ©HortonworksInc.2011–2016.AllRightsReserved©HortonworksInc.2011–2016.AllRightsReserved�X

EntertheTLSToolkit

⬢ Command-linetooltoautomatecertificategenerationandconfiguration

⬢ Self-containedcertificateauthority(CA)forcertificatesigning

⬢ Keystore&truststoregeneration

⬢ Clientcertificategeneration

⬢ Automaticallyupdatesnifi.properties⬢ UnderpinsAmbariTLSintegration


JVM

RESTAPI

NiFi

Framework

Proc CS ReportTask

ExtensionAPI

S2SAPI

JVM

S2SClientLibraries

Site-to-SiteRefactoring– S2SHTTP(S)ProtocolthroughProxyServer

Socketprotocol:TCP

HDF2.0:HTTP(s)protocol

HTTPproxy


AgendaWhat’sNiFi


NiFi ontheedge

Commonissues

What’sNext?


EdgeIntelligencewithApacheMiNiFi

Ã GuaranteeddeliveryÃ Databuffering

‒ Backpressure‒ Pressurerelease

Ã PrioritizedqueuingÃ FlowspecificQoS

‒ Latencyvs.throughput‒ Losstolerance

Ã Dataprovenance

Ã Recovery/recordingarollinglogoffine-grainedhistory

Ã Designedforextension

DifferentfromApacheNiFiÃ DesignandDeployÃ Warmre-deploys

KeyFeatures


NiFivs.MiNiFiJavaProcessor,SmallerFootprint~40MB

NiFiFramework

Components

MiNiFi

NiFiFramework

UserInterface

Components

NiFi


AgendaWhat’sNiFi


NiFi ontheedge

Commonissues

What’sNext?


Commonissues

Ã NiFi RepoconfigurationissuesÃ NiFi SSLconfigurationorcertificateissuesÃ ExecuteStreamCommand ProcessorgettingstuckÃ OutOfMemory IssueswithNCMorprocessors.


BestPractices

Ã DebugLoggingincaseofProcessorissuesÃ CorePropertiesandJVMtuning:

https://community.hortonworks.com/articles/7882/hdfnifi-best-practices-for-setting-up-a-high-perfo.html


UnderstandinghealthviaNiFi UI

StatusBar

ProcessorDetails


NiFi SummaryPage


SystemInformation


AgendaWhat’sNiFi


NiFi ontheedge

Commonissues

What’sNext?


What’sNext

Ã Frameworkextension– Distributeddatadurability(HA

data)– Configurationmanagementflows

(SDLC)Ã EnhancedUserExperience

– Template/ExtensionRegistry– VariableRegistry

Ã Deeperecosystemintegration

Ã CentralCommandandControlÃ NativeAgent(GA)

NiFi MiNiFi

https://cwiki.apache.org/confluence/display/NIFI/Product+requirements

Nifi productrequirements Search!


ThankYou

Apache NiFi 1.0 in Nutshell

Technology

Transcript of Apache NiFi 1.0 in Nutshell