Turbine: A distributed-memory dataflow engine for extreme ...
Go With The Dataflow - NLNOG...Go With The Dataflow Apache NiFi [email protected] Telecom...
Transcript of Go With The Dataflow - NLNOG...Go With The Dataflow Apache NiFi [email protected] Telecom...
![Page 2: Go With The Dataflow - NLNOG...Go With The Dataflow Apache NiFi arien.vijn@linklight.nl Telecom Architecture • Network Elements (NE) are managed by Elements Management Systems](https://reader036.fdocuments.us/reader036/viewer/2022062602/5ec95a7ee92aff32af2fde06/html5/thumbnails/2.jpg)
Telecom Architecture• Network Elements (NE) are managed by
Elements Management Systems (EMS)
• EMSes communicate with Network Management Systems (NMS)
• NMSes communicate with the OSS
• OSS communicates with the BSS
• Between systems are standardised interfaces, except for the South Bound Interface (SBI) of the EMS
Fulfilment
Assurance
Network Elements (NE / NEL)
Elements Management Systems (EMS / EML)
Network Management System(s)(NMS / NML)
Operational Support System (OSS / SML)
Business Support System (BSS / BML)
![Page 3: Go With The Dataflow - NLNOG...Go With The Dataflow Apache NiFi arien.vijn@linklight.nl Telecom Architecture • Network Elements (NE) are managed by Elements Management Systems](https://reader036.fdocuments.us/reader036/viewer/2022062602/5ec95a7ee92aff32af2fde06/html5/thumbnails/3.jpg)
“The good thing about standards is that there are so many to choose from.”
― Andrew S. Tanenbaum
![Page 4: Go With The Dataflow - NLNOG...Go With The Dataflow Apache NiFi arien.vijn@linklight.nl Telecom Architecture • Network Elements (NE) are managed by Elements Management Systems](https://reader036.fdocuments.us/reader036/viewer/2022062602/5ec95a7ee92aff32af2fde06/html5/thumbnails/4.jpg)
Reality• This architecture is not really implemented
Fulfilment
Assurance
NMS
Network Elements (NE)
EMS
BSS
OSS
![Page 5: Go With The Dataflow - NLNOG...Go With The Dataflow Apache NiFi arien.vijn@linklight.nl Telecom Architecture • Network Elements (NE) are managed by Elements Management Systems](https://reader036.fdocuments.us/reader036/viewer/2022062602/5ec95a7ee92aff32af2fde06/html5/thumbnails/5.jpg)
Reality• This architecture is not really implemented
• Humans solve the missing parts and shortcomings
Fulfilment
Assurance
NMS
Network Elements (NE)
EMS
BSS
OSS
![Page 6: Go With The Dataflow - NLNOG...Go With The Dataflow Apache NiFi arien.vijn@linklight.nl Telecom Architecture • Network Elements (NE) are managed by Elements Management Systems](https://reader036.fdocuments.us/reader036/viewer/2022062602/5ec95a7ee92aff32af2fde06/html5/thumbnails/6.jpg)
Reality• This architecture is not really implemented
• Humans solve the missing parts and shortcomings
• Often using Microsoft Excel
Fulfilment
Assurance
NMS
Network Elements (NE)
EMS
BSS
OSS
+
![Page 7: Go With The Dataflow - NLNOG...Go With The Dataflow Apache NiFi arien.vijn@linklight.nl Telecom Architecture • Network Elements (NE) are managed by Elements Management Systems](https://reader036.fdocuments.us/reader036/viewer/2022062602/5ec95a7ee92aff32af2fde06/html5/thumbnails/7.jpg)
Reality• Only partly implemented
• Humans form the north and south bound interfaces over multiple layers
• Often using Microsoft Excel
• Or Python, PERL, awk, bash...
Fulfilment
Assurance
NMS
Network Elements (NE)
EMS
BSS
OSS
+
![Page 8: Go With The Dataflow - NLNOG...Go With The Dataflow Apache NiFi arien.vijn@linklight.nl Telecom Architecture • Network Elements (NE) are managed by Elements Management Systems](https://reader036.fdocuments.us/reader036/viewer/2022062602/5ec95a7ee92aff32af2fde06/html5/thumbnails/8.jpg)
Reality• Only partly implemented
• Humans form the north and south bound interfaces over multiple layers
• Often using Microsoft Excel
• Or Python, PERL, awk, bash...
• Often in organisational silos
• Transport, delivery, IP, Consumer, B2B, etc.
Fulfilment
Assurance
NMS
Network Elements (NE)
EMS
BSS
OSS
![Page 9: Go With The Dataflow - NLNOG...Go With The Dataflow Apache NiFi arien.vijn@linklight.nl Telecom Architecture • Network Elements (NE) are managed by Elements Management Systems](https://reader036.fdocuments.us/reader036/viewer/2022062602/5ec95a7ee92aff32af2fde06/html5/thumbnails/9.jpg)
Reality• Many systems
• Lots of data
• Little information
NMS
Network Elements (NE)
EMS
BSS
OSS
Data
? 2
dozensdozens
>300k
information
![Page 10: Go With The Dataflow - NLNOG...Go With The Dataflow Apache NiFi arien.vijn@linklight.nl Telecom Architecture • Network Elements (NE) are managed by Elements Management Systems](https://reader036.fdocuments.us/reader036/viewer/2022062602/5ec95a7ee92aff32af2fde06/html5/thumbnails/10.jpg)
Transform Data Into Information
![Page 11: Go With The Dataflow - NLNOG...Go With The Dataflow Apache NiFi arien.vijn@linklight.nl Telecom Architecture • Network Elements (NE) are managed by Elements Management Systems](https://reader036.fdocuments.us/reader036/viewer/2022062602/5ec95a7ee92aff32af2fde06/html5/thumbnails/11.jpg)
Transform Data Into Information• Dozens of systems
• Some open, some closed
• Tons of Network Elements
• NEs are off-limits due to change control
NMS
Network Elements (NE)
EMS
BSS
OSS
Data
information
? 2
dozensdozens
>300k
![Page 12: Go With The Dataflow - NLNOG...Go With The Dataflow Apache NiFi arien.vijn@linklight.nl Telecom Architecture • Network Elements (NE) are managed by Elements Management Systems](https://reader036.fdocuments.us/reader036/viewer/2022062602/5ec95a7ee92aff32af2fde06/html5/thumbnails/12.jpg)
Transform Data Into Information• Dozens of systems
• Some open, some closed
• Tons of Network Elements
• NEs are off-limits due to change control
• Initial contract for 36 workdays NMS
Network Elements (NE)
EMS
BSS
OSS
Data
information
? 2
dozensdozens
>300k
36 workdays
![Page 13: Go With The Dataflow - NLNOG...Go With The Dataflow Apache NiFi arien.vijn@linklight.nl Telecom Architecture • Network Elements (NE) are managed by Elements Management Systems](https://reader036.fdocuments.us/reader036/viewer/2022062602/5ec95a7ee92aff32af2fde06/html5/thumbnails/13.jpg)
Transform Data Into Information• R?
NMS
Network Elements (NE)
EMS
BSS
OSS
information
? 2
dozensdozens
>300k
36 workdays
NMS
Network Elements (NE)
EMS
BSS
OSS
information
? 2
dozensdozens
>300k
36 workdays
![Page 14: Go With The Dataflow - NLNOG...Go With The Dataflow Apache NiFi arien.vijn@linklight.nl Telecom Architecture • Network Elements (NE) are managed by Elements Management Systems](https://reader036.fdocuments.us/reader036/viewer/2022062602/5ec95a7ee92aff32af2fde06/html5/thumbnails/14.jpg)
Transform Data Into Information• R?
• Python?
NMS
Network Elements (NE)
EMS
BSS
OSS
information
? 2
dozensdozens
>300k
36 workdays
![Page 15: Go With The Dataflow - NLNOG...Go With The Dataflow Apache NiFi arien.vijn@linklight.nl Telecom Architecture • Network Elements (NE) are managed by Elements Management Systems](https://reader036.fdocuments.us/reader036/viewer/2022062602/5ec95a7ee92aff32af2fde06/html5/thumbnails/15.jpg)
Transform Data Into Information• R?
• Python?
• Go with the pros
NMS
Network Elements (NE)
EMS
BSS
OSS
information
? 2
dozensdozens
>300k
36 workdays
![Page 16: Go With The Dataflow - NLNOG...Go With The Dataflow Apache NiFi arien.vijn@linklight.nl Telecom Architecture • Network Elements (NE) are managed by Elements Management Systems](https://reader036.fdocuments.us/reader036/viewer/2022062602/5ec95a7ee92aff32af2fde06/html5/thumbnails/16.jpg)
Transform Data Into Information• Python?
• R?
• Go with the pros
• Apache NiFi
• Based on NSA's Niagara Files project
• Flow-based programming
• and a little Python NMS
Network Elements (NE)
EMS
BSS
OSS
information
? 2
dozensdozens
>300k
36 workdays
+
![Page 17: Go With The Dataflow - NLNOG...Go With The Dataflow Apache NiFi arien.vijn@linklight.nl Telecom Architecture • Network Elements (NE) are managed by Elements Management Systems](https://reader036.fdocuments.us/reader036/viewer/2022062602/5ec95a7ee92aff32af2fde06/html5/thumbnails/17.jpg)
![Page 18: Go With The Dataflow - NLNOG...Go With The Dataflow Apache NiFi arien.vijn@linklight.nl Telecom Architecture • Network Elements (NE) are managed by Elements Management Systems](https://reader036.fdocuments.us/reader036/viewer/2022062602/5ec95a7ee92aff32af2fde06/html5/thumbnails/18.jpg)
Apache NiFi• Flow-based programming
• Bit like Lego Mindstorms
![Page 19: Go With The Dataflow - NLNOG...Go With The Dataflow Apache NiFi arien.vijn@linklight.nl Telecom Architecture • Network Elements (NE) are managed by Elements Management Systems](https://reader036.fdocuments.us/reader036/viewer/2022062602/5ec95a7ee92aff32af2fde06/html5/thumbnails/19.jpg)
Apache NiFi• Flow-based programming
• Bit like Lego Mindstorms
• Drag processors on the canvas
• Put in a few settings
![Page 20: Go With The Dataflow - NLNOG...Go With The Dataflow Apache NiFi arien.vijn@linklight.nl Telecom Architecture • Network Elements (NE) are managed by Elements Management Systems](https://reader036.fdocuments.us/reader036/viewer/2022062602/5ec95a7ee92aff32af2fde06/html5/thumbnails/20.jpg)
Apache NiFi• Flow-based programming
• Bit like Lego Mindstorms
• Drag processors on the canvas
• Put in a few settings
• Connect the processors
• Run...
• Add, change and work on it while the data flows.
![Page 21: Go With The Dataflow - NLNOG...Go With The Dataflow Apache NiFi arien.vijn@linklight.nl Telecom Architecture • Network Elements (NE) are managed by Elements Management Systems](https://reader036.fdocuments.us/reader036/viewer/2022062602/5ec95a7ee92aff32af2fde06/html5/thumbnails/21.jpg)
Count The Number of Speakers
![Page 22: Go With The Dataflow - NLNOG...Go With The Dataflow Apache NiFi arien.vijn@linklight.nl Telecom Architecture • Network Elements (NE) are managed by Elements Management Systems](https://reader036.fdocuments.us/reader036/viewer/2022062602/5ec95a7ee92aff32af2fde06/html5/thumbnails/22.jpg)
Count The Number of Speakers• We start with the GenerateFlowFile
processor
![Page 23: Go With The Dataflow - NLNOG...Go With The Dataflow Apache NiFi arien.vijn@linklight.nl Telecom Architecture • Network Elements (NE) are managed by Elements Management Systems](https://reader036.fdocuments.us/reader036/viewer/2022062602/5ec95a7ee92aff32af2fde06/html5/thumbnails/23.jpg)
Count The Number of Speakers• We start with the GenerateFlowFile
processor
• We create an attribute called url and give it the value: https://nlnog.net/nlnog-day-2019/
![Page 24: Go With The Dataflow - NLNOG...Go With The Dataflow Apache NiFi arien.vijn@linklight.nl Telecom Architecture • Network Elements (NE) are managed by Elements Management Systems](https://reader036.fdocuments.us/reader036/viewer/2022062602/5ec95a7ee92aff32af2fde06/html5/thumbnails/24.jpg)
Count The Number of Speakers• We start with the GenerateFlowFile
processor
• We create an attribute called url and give it the value: https://nlnog.net/nlnog-day-2019/
• Second processor gets the web page
![Page 25: Go With The Dataflow - NLNOG...Go With The Dataflow Apache NiFi arien.vijn@linklight.nl Telecom Architecture • Network Elements (NE) are managed by Elements Management Systems](https://reader036.fdocuments.us/reader036/viewer/2022062602/5ec95a7ee92aff32af2fde06/html5/thumbnails/25.jpg)
Count The Number of Speakers• We start with the GenerateFlowFile
processor
• We create an attribute called url and give it the value: https://nlnog.net/nlnog-day-2019/
• Second processor gets the web page.
• Connect the processors
![Page 26: Go With The Dataflow - NLNOG...Go With The Dataflow Apache NiFi arien.vijn@linklight.nl Telecom Architecture • Network Elements (NE) are managed by Elements Management Systems](https://reader036.fdocuments.us/reader036/viewer/2022062602/5ec95a7ee92aff32af2fde06/html5/thumbnails/26.jpg)
Count The Number of Speakers• Start the first processor
• One FlowFile in the queue between the processors
![Page 27: Go With The Dataflow - NLNOG...Go With The Dataflow Apache NiFi arien.vijn@linklight.nl Telecom Architecture • Network Elements (NE) are managed by Elements Management Systems](https://reader036.fdocuments.us/reader036/viewer/2022062602/5ec95a7ee92aff32af2fde06/html5/thumbnails/27.jpg)
Count The Number of Speakers• Start the first processor
• One FlowFile in the queue between the processors
• We can view the contents and the history of the FlowFiles at any time
![Page 28: Go With The Dataflow - NLNOG...Go With The Dataflow Apache NiFi arien.vijn@linklight.nl Telecom Architecture • Network Elements (NE) are managed by Elements Management Systems](https://reader036.fdocuments.us/reader036/viewer/2022062602/5ec95a7ee92aff32af2fde06/html5/thumbnails/28.jpg)
Count The Number of Speakers• Add the third processor and connect the
second
• Start the second processor
• Flowfile goes through
• Contains the HTML data of the NLNOG website
![Page 29: Go With The Dataflow - NLNOG...Go With The Dataflow Apache NiFi arien.vijn@linklight.nl Telecom Architecture • Network Elements (NE) are managed by Elements Management Systems](https://reader036.fdocuments.us/reader036/viewer/2022062602/5ec95a7ee92aff32af2fde06/html5/thumbnails/29.jpg)
Count The Number of Speakers• Third processor: ExecuteStreamCommand
• Invokes a simple Python script that extracts the names of the speakers
• stdin to stdout
![Page 30: Go With The Dataflow - NLNOG...Go With The Dataflow Apache NiFi arien.vijn@linklight.nl Telecom Architecture • Network Elements (NE) are managed by Elements Management Systems](https://reader036.fdocuments.us/reader036/viewer/2022062602/5ec95a7ee92aff32af2fde06/html5/thumbnails/30.jpg)
Count The Number of Speakers• Third processor: ExecuteStreamCommand
• Invokes a simple Python script that extracts the names of the speakers
• stdin to stdout
• Last processor:
• Counts the lines
![Page 31: Go With The Dataflow - NLNOG...Go With The Dataflow Apache NiFi arien.vijn@linklight.nl Telecom Architecture • Network Elements (NE) are managed by Elements Management Systems](https://reader036.fdocuments.us/reader036/viewer/2022062602/5ec95a7ee92aff32af2fde06/html5/thumbnails/31.jpg)
Count The Number of Speakers• Third processor: ExecuteStreamCommand
• Invokes a simple Python script that extracts the names of the speakers
• stdin to stdout
• Last processor:
• Counts the lines
• 12!
![Page 32: Go With The Dataflow - NLNOG...Go With The Dataflow Apache NiFi arien.vijn@linklight.nl Telecom Architecture • Network Elements (NE) are managed by Elements Management Systems](https://reader036.fdocuments.us/reader036/viewer/2022062602/5ec95a7ee92aff32af2fde06/html5/thumbnails/32.jpg)
Back to the Telecom Company• This is a very simple example
• NiFi is used for complex data extraction tasks
• Its graphical user interface helps with visualising the complexity
• It is used to combine data from many different systems
• We use it a lot to create time series data from systems that don't keep history
• Perhaps it is useful for you too
NMS
Network Elements (NE)
EMS
BSS
OSS
Information
? 2
dozensdozens
>300k
+
![Page 33: Go With The Dataflow - NLNOG...Go With The Dataflow Apache NiFi arien.vijn@linklight.nl Telecom Architecture • Network Elements (NE) are managed by Elements Management Systems](https://reader036.fdocuments.us/reader036/viewer/2022062602/5ec95a7ee92aff32af2fde06/html5/thumbnails/33.jpg)
Thank you!