Fast and Simplified Streaming, Ad-Hoc and Batch Analytics with FiloDB and Spark Streaming
Using Streaming Views for Real Time Analytics€¦ · Using Streaming Views for Real Time Analytics...
Transcript of Using Streaming Views for Real Time Analytics€¦ · Using Streaming Views for Real Time Analytics...
![Page 1: Using Streaming Views for Real Time Analytics€¦ · Using Streaming Views for Real Time Analytics Rafael Borges . #PIWorld ©2019 OSIsoft, LLC 3 Today’s Agenda •Introduction](https://reader033.fdocuments.us/reader033/viewer/2022043017/5f39ecb81cd26f671049ce81/html5/thumbnails/1.jpg)
#PIWorld ©2019 OSIsoft, LLC
1
Using Streaming Views for Real Time Analytics
Rafael Borges
![Page 2: Using Streaming Views for Real Time Analytics€¦ · Using Streaming Views for Real Time Analytics Rafael Borges . #PIWorld ©2019 OSIsoft, LLC 3 Today’s Agenda •Introduction](https://reader033.fdocuments.us/reader033/viewer/2022043017/5f39ecb81cd26f671049ce81/html5/thumbnails/2.jpg)
#PIWorld ©2019 OSIsoft, LLC
#PIWorld ©2019 OSIsoft, LLC
3
Today’s Agenda
• Introduction
•What? Why? How?
•OSIsoft and CRISP-DM
•Architecture & Data Flow
•Live Demo
![Page 3: Using Streaming Views for Real Time Analytics€¦ · Using Streaming Views for Real Time Analytics Rafael Borges . #PIWorld ©2019 OSIsoft, LLC 3 Today’s Agenda •Introduction](https://reader033.fdocuments.us/reader033/viewer/2022043017/5f39ecb81cd26f671049ce81/html5/thumbnails/3.jpg)
#PIWorld ©2019 OSIsoft, LLC
#PIWorld ©2019 OSIsoft, LLC
4
Before we begin…
•This is a Tech Talk, not a traditional lab or lecture
• It’s informal and interactive
•There will be a Q&A session at the and so we can discuss the content that was presented.
• It will be very dense!
![Page 4: Using Streaming Views for Real Time Analytics€¦ · Using Streaming Views for Real Time Analytics Rafael Borges . #PIWorld ©2019 OSIsoft, LLC 3 Today’s Agenda •Introduction](https://reader033.fdocuments.us/reader033/viewer/2022043017/5f39ecb81cd26f671049ce81/html5/thumbnails/4.jpg)
#PIWorld ©2019 OSIsoft, LLC
#PIWorld ©2019 OSIsoft, LLC
5
What, Why and How
•What? • Stream live data to an external server in order to
process it with more powerful analytical tools
•Why? • Because there are platforms that are more capable
for specific tasks and Spark is perfect for HPC
•How? • Using Integrators’ Streaming View, Kafka and Spark
![Page 5: Using Streaming Views for Real Time Analytics€¦ · Using Streaming Views for Real Time Analytics Rafael Borges . #PIWorld ©2019 OSIsoft, LLC 3 Today’s Agenda •Introduction](https://reader033.fdocuments.us/reader033/viewer/2022043017/5f39ecb81cd26f671049ce81/html5/thumbnails/5.jpg)
#PIWorld ©2019 OSIsoft, LLC
6 6
OSIsoft and CRISP-DM⠀
![Page 6: Using Streaming Views for Real Time Analytics€¦ · Using Streaming Views for Real Time Analytics Rafael Borges . #PIWorld ©2019 OSIsoft, LLC 3 Today’s Agenda •Introduction](https://reader033.fdocuments.us/reader033/viewer/2022043017/5f39ecb81cd26f671049ce81/html5/thumbnails/6.jpg)
#PIWorld ©2019 OSIsoft, LLC
#PIWorld ©2019 OSIsoft, LLC
Cross-industry Standard Process for Data Mining
OSIsoft and CRISP-DM
• Business Understanding
• Data Understanding
• Data Preparation
• Modeling
• Evaluation
• Deployment
7
Source: Kenneth Jensen / Wikipedia
![Page 7: Using Streaming Views for Real Time Analytics€¦ · Using Streaming Views for Real Time Analytics Rafael Borges . #PIWorld ©2019 OSIsoft, LLC 3 Today’s Agenda •Introduction](https://reader033.fdocuments.us/reader033/viewer/2022043017/5f39ecb81cd26f671049ce81/html5/thumbnails/7.jpg)
#PIWorld ©2019 OSIsoft, LLC
#PIWorld ©2019 OSIsoft, LLC
OSIsoft and CRISP-DM
8
Data Ingress Products
• Interfaces
• Connectors
• Custom Code
![Page 8: Using Streaming Views for Real Time Analytics€¦ · Using Streaming Views for Real Time Analytics Rafael Borges . #PIWorld ©2019 OSIsoft, LLC 3 Today’s Agenda •Introduction](https://reader033.fdocuments.us/reader033/viewer/2022043017/5f39ecb81cd26f671049ce81/html5/thumbnails/8.jpg)
#PIWorld ©2019 OSIsoft, LLC
#PIWorld ©2019 OSIsoft, LLC
OSIsoft and CRISP-DM
9
Data Exploration Products
• PI Vision
• PI Datalink
• Power BI
• Python / R Scripts
![Page 9: Using Streaming Views for Real Time Analytics€¦ · Using Streaming Views for Real Time Analytics Rafael Borges . #PIWorld ©2019 OSIsoft, LLC 3 Today’s Agenda •Introduction](https://reader033.fdocuments.us/reader033/viewer/2022043017/5f39ecb81cd26f671049ce81/html5/thumbnails/9.jpg)
#PIWorld ©2019 OSIsoft, LLC
#PIWorld ©2019 OSIsoft, LLC
OSIsoft and CRISP-DM
10
Data Wrangling Products
• PI Datalink
• PI Integrators
• PI Dev Technologies
![Page 10: Using Streaming Views for Real Time Analytics€¦ · Using Streaming Views for Real Time Analytics Rafael Borges . #PIWorld ©2019 OSIsoft, LLC 3 Today’s Agenda •Introduction](https://reader033.fdocuments.us/reader033/viewer/2022043017/5f39ecb81cd26f671049ce81/html5/thumbnails/10.jpg)
#PIWorld ©2019 OSIsoft, LLC
#PIWorld ©2019 OSIsoft, LLC
OSIsoft and CRISP-DM
11
Third-Party Products
• Python (SciPy / NumPy)
• R Scripts
• Spark
• Jupyter
• Flink
• Hadoop
• …
![Page 11: Using Streaming Views for Real Time Analytics€¦ · Using Streaming Views for Real Time Analytics Rafael Borges . #PIWorld ©2019 OSIsoft, LLC 3 Today’s Agenda •Introduction](https://reader033.fdocuments.us/reader033/viewer/2022043017/5f39ecb81cd26f671049ce81/html5/thumbnails/11.jpg)
#PIWorld ©2019 OSIsoft, LLC
12 12
Architecture & Data Flow⠀
![Page 12: Using Streaming Views for Real Time Analytics€¦ · Using Streaming Views for Real Time Analytics Rafael Borges . #PIWorld ©2019 OSIsoft, LLC 3 Today’s Agenda •Introduction](https://reader033.fdocuments.us/reader033/viewer/2022043017/5f39ecb81cd26f671049ce81/html5/thumbnails/12.jpg)
#PIWorld ©2019 OSIsoft, LLC
#PIWorld ©2019 OSIsoft, LLC
13
Architecture & Data Flow
PI Server
Kafka Server
Spark Cluster
Data Generation
PI Integrator for BA
Domain
Node-RED
![Page 13: Using Streaming Views for Real Time Analytics€¦ · Using Streaming Views for Real Time Analytics Rafael Borges . #PIWorld ©2019 OSIsoft, LLC 3 Today’s Agenda •Introduction](https://reader033.fdocuments.us/reader033/viewer/2022043017/5f39ecb81cd26f671049ce81/html5/thumbnails/13.jpg)
#PIWorld ©2019 OSIsoft, LLC
14 14
Data Generation⠀
![Page 14: Using Streaming Views for Real Time Analytics€¦ · Using Streaming Views for Real Time Analytics Rafael Borges . #PIWorld ©2019 OSIsoft, LLC 3 Today’s Agenda •Introduction](https://reader033.fdocuments.us/reader033/viewer/2022043017/5f39ecb81cd26f671049ce81/html5/thumbnails/14.jpg)
#PIWorld ©2019 OSIsoft, LLC
#PIWorld ©2019 OSIsoft, LLC
15
Data Generation
![Page 15: Using Streaming Views for Real Time Analytics€¦ · Using Streaming Views for Real Time Analytics Rafael Borges . #PIWorld ©2019 OSIsoft, LLC 3 Today’s Agenda •Introduction](https://reader033.fdocuments.us/reader033/viewer/2022043017/5f39ecb81cd26f671049ce81/html5/thumbnails/15.jpg)
#PIWorld ©2019 OSIsoft, LLC
#PIWorld ©2019 OSIsoft, LLC
16
Data Generation
![Page 16: Using Streaming Views for Real Time Analytics€¦ · Using Streaming Views for Real Time Analytics Rafael Borges . #PIWorld ©2019 OSIsoft, LLC 3 Today’s Agenda •Introduction](https://reader033.fdocuments.us/reader033/viewer/2022043017/5f39ecb81cd26f671049ce81/html5/thumbnails/16.jpg)
#PIWorld ©2019 OSIsoft, LLC
#PIWorld ©2019 OSIsoft, LLC
17
Data Generation
PI Server
Kafka Server
Spark Cluster
Data Generation
PI Integrator for BA
Domain
Node-RED
PI W
eb
AP
I
![Page 17: Using Streaming Views for Real Time Analytics€¦ · Using Streaming Views for Real Time Analytics Rafael Borges . #PIWorld ©2019 OSIsoft, LLC 3 Today’s Agenda •Introduction](https://reader033.fdocuments.us/reader033/viewer/2022043017/5f39ecb81cd26f671049ce81/html5/thumbnails/17.jpg)
#PIWorld ©2019 OSIsoft, LLC
#PIWorld ©2019 OSIsoft, LLC
18
Data Generation – PI Web API
def send_data(timestamp, data, variable_info):
body = get_body(timestamp, data, variable_info)
connection = HTTPSConnection(host, context=ssl._create_unverified_context())
connection.request("POST", url, headers=get_header(), body=body)
response = connection.getresponse()
![Page 18: Using Streaming Views for Real Time Analytics€¦ · Using Streaming Views for Real Time Analytics Rafael Borges . #PIWorld ©2019 OSIsoft, LLC 3 Today’s Agenda •Introduction](https://reader033.fdocuments.us/reader033/viewer/2022043017/5f39ecb81cd26f671049ce81/html5/thumbnails/18.jpg)
#PIWorld ©2019 OSIsoft, LLC
#PIWorld ©2019 OSIsoft, LLC
19
Data Generation – WebId 2.0 (2017+)
def generate_web_id_by_path(path):
marker = "Ab" #Because I know it's an Attribute
owner_marker = "E" #Because I know it's an Element
if path[0:2] == "\\\\":
path = path[2:]
encoded_path_bytes = path.upper().encode('utf-8')
encoded_string = b64encode(encoded_path_bytes).decode()
encoded_path = encoded_string.strip('=')\
.replace('+', '-').replace('/', '_')
return "P1{}{}{}".format(marker, owner_marker, encoded_path)
![Page 19: Using Streaming Views for Real Time Analytics€¦ · Using Streaming Views for Real Time Analytics Rafael Borges . #PIWorld ©2019 OSIsoft, LLC 3 Today’s Agenda •Introduction](https://reader033.fdocuments.us/reader033/viewer/2022043017/5f39ecb81cd26f671049ce81/html5/thumbnails/19.jpg)
#PIWorld ©2019 OSIsoft, LLC
20 20
PI Integrator for BA Advanced Edition⠀
![Page 20: Using Streaming Views for Real Time Analytics€¦ · Using Streaming Views for Real Time Analytics Rafael Borges . #PIWorld ©2019 OSIsoft, LLC 3 Today’s Agenda •Introduction](https://reader033.fdocuments.us/reader033/viewer/2022043017/5f39ecb81cd26f671049ce81/html5/thumbnails/20.jpg)
#PIWorld ©2019 OSIsoft, LLC
#PIWorld ©2019 OSIsoft, LLC
21
PI Integrator for BA Advanced Edition
![Page 21: Using Streaming Views for Real Time Analytics€¦ · Using Streaming Views for Real Time Analytics Rafael Borges . #PIWorld ©2019 OSIsoft, LLC 3 Today’s Agenda •Introduction](https://reader033.fdocuments.us/reader033/viewer/2022043017/5f39ecb81cd26f671049ce81/html5/thumbnails/21.jpg)
#PIWorld ©2019 OSIsoft, LLC
22 22
Apache Kafka⠀
![Page 22: Using Streaming Views for Real Time Analytics€¦ · Using Streaming Views for Real Time Analytics Rafael Borges . #PIWorld ©2019 OSIsoft, LLC 3 Today’s Agenda •Introduction](https://reader033.fdocuments.us/reader033/viewer/2022043017/5f39ecb81cd26f671049ce81/html5/thumbnails/22.jpg)
#PIWorld ©2019 OSIsoft, LLC
#PIWorld ©2019 OSIsoft, LLC
23
Apache Kafka
![Page 23: Using Streaming Views for Real Time Analytics€¦ · Using Streaming Views for Real Time Analytics Rafael Borges . #PIWorld ©2019 OSIsoft, LLC 3 Today’s Agenda •Introduction](https://reader033.fdocuments.us/reader033/viewer/2022043017/5f39ecb81cd26f671049ce81/html5/thumbnails/23.jpg)
#PIWorld ©2019 OSIsoft, LLC
#PIWorld ©2019 OSIsoft, LLC
PI Server
Kafka Server
Spark Cluster
Data Generation
PI Integrator for BA
Domain
Node-RED
24
Apache Kafka
Producer
Consumer Consumer
![Page 24: Using Streaming Views for Real Time Analytics€¦ · Using Streaming Views for Real Time Analytics Rafael Borges . #PIWorld ©2019 OSIsoft, LLC 3 Today’s Agenda •Introduction](https://reader033.fdocuments.us/reader033/viewer/2022043017/5f39ecb81cd26f671049ce81/html5/thumbnails/24.jpg)
#PIWorld ©2019 OSIsoft, LLC
25 25
Apache Spark⠀
![Page 25: Using Streaming Views for Real Time Analytics€¦ · Using Streaming Views for Real Time Analytics Rafael Borges . #PIWorld ©2019 OSIsoft, LLC 3 Today’s Agenda •Introduction](https://reader033.fdocuments.us/reader033/viewer/2022043017/5f39ecb81cd26f671049ce81/html5/thumbnails/25.jpg)
#PIWorld ©2019 OSIsoft, LLC
#PIWorld ©2019 OSIsoft, LLC
26
Apache Spark
![Page 26: Using Streaming Views for Real Time Analytics€¦ · Using Streaming Views for Real Time Analytics Rafael Borges . #PIWorld ©2019 OSIsoft, LLC 3 Today’s Agenda •Introduction](https://reader033.fdocuments.us/reader033/viewer/2022043017/5f39ecb81cd26f671049ce81/html5/thumbnails/26.jpg)
#PIWorld ©2019 OSIsoft, LLC
#PIWorld ©2019 OSIsoft, LLC
27
Apache Spark
![Page 27: Using Streaming Views for Real Time Analytics€¦ · Using Streaming Views for Real Time Analytics Rafael Borges . #PIWorld ©2019 OSIsoft, LLC 3 Today’s Agenda •Introduction](https://reader033.fdocuments.us/reader033/viewer/2022043017/5f39ecb81cd26f671049ce81/html5/thumbnails/27.jpg)
#PIWorld ©2019 OSIsoft, LLC
#PIWorld ©2019 OSIsoft, LLC
28
Apache Spark
![Page 28: Using Streaming Views for Real Time Analytics€¦ · Using Streaming Views for Real Time Analytics Rafael Borges . #PIWorld ©2019 OSIsoft, LLC 3 Today’s Agenda •Introduction](https://reader033.fdocuments.us/reader033/viewer/2022043017/5f39ecb81cd26f671049ce81/html5/thumbnails/28.jpg)
#PIWorld ©2019 OSIsoft, LLC
29 29
Node-RED⠀
![Page 29: Using Streaming Views for Real Time Analytics€¦ · Using Streaming Views for Real Time Analytics Rafael Borges . #PIWorld ©2019 OSIsoft, LLC 3 Today’s Agenda •Introduction](https://reader033.fdocuments.us/reader033/viewer/2022043017/5f39ecb81cd26f671049ce81/html5/thumbnails/29.jpg)
#PIWorld ©2019 OSIsoft, LLC
#PIWorld ©2019 OSIsoft, LLC
PI Server
Kafka Server
Spark Cluster
Data Generation
PI Integrator for BA
Domain
Node-RED
30
Node-RED
Producer
Consumer Consumer
![Page 30: Using Streaming Views for Real Time Analytics€¦ · Using Streaming Views for Real Time Analytics Rafael Borges . #PIWorld ©2019 OSIsoft, LLC 3 Today’s Agenda •Introduction](https://reader033.fdocuments.us/reader033/viewer/2022043017/5f39ecb81cd26f671049ce81/html5/thumbnails/30.jpg)
#PIWorld ©2019 OSIsoft, LLC
#PIWorld ©2019 OSIsoft, LLC
31
Node-RED
![Page 31: Using Streaming Views for Real Time Analytics€¦ · Using Streaming Views for Real Time Analytics Rafael Borges . #PIWorld ©2019 OSIsoft, LLC 3 Today’s Agenda •Introduction](https://reader033.fdocuments.us/reader033/viewer/2022043017/5f39ecb81cd26f671049ce81/html5/thumbnails/31.jpg)
#PIWorld ©2019 OSIsoft, LLC
32 32
Conclusion⠀
![Page 32: Using Streaming Views for Real Time Analytics€¦ · Using Streaming Views for Real Time Analytics Rafael Borges . #PIWorld ©2019 OSIsoft, LLC 3 Today’s Agenda •Introduction](https://reader033.fdocuments.us/reader033/viewer/2022043017/5f39ecb81cd26f671049ce81/html5/thumbnails/32.jpg)
#PIWorld ©2019 OSIsoft, LLC
#PIWorld ©2019 OSIsoft, LLC
33
Conclusion
• It’s easy to send data out of the PI System • PI Web API
• PI Integrator for BA • Not only for Kafka, but any platform!
• It’s easy to do real-time analytics with PI data
• It’s easy to bring data back to the PI System
![Page 33: Using Streaming Views for Real Time Analytics€¦ · Using Streaming Views for Real Time Analytics Rafael Borges . #PIWorld ©2019 OSIsoft, LLC 3 Today’s Agenda •Introduction](https://reader033.fdocuments.us/reader033/viewer/2022043017/5f39ecb81cd26f671049ce81/html5/thumbnails/33.jpg)
#PIWorld ©2019 OSIsoft, LLC
Speaker Info
34
• Rafael Borges
• Technology Enablement Engineer
• OSIsoft
![Page 34: Using Streaming Views for Real Time Analytics€¦ · Using Streaming Views for Real Time Analytics Rafael Borges . #PIWorld ©2019 OSIsoft, LLC 3 Today’s Agenda •Introduction](https://reader033.fdocuments.us/reader033/viewer/2022043017/5f39ecb81cd26f671049ce81/html5/thumbnails/34.jpg)
#PIWorld ©2019 OSIsoft, LLC
Questions?
Please wait for
the microphone
State your
name & company
Please remember
DOWNLOAD THE MOBILE APP
35