Streaming Analytics, Data Lakes and PI Integrators...Streaming Analytics, Data Lakes and PI...
Transcript of Streaming Analytics, Data Lakes and PI Integrators...Streaming Analytics, Data Lakes and PI...
EMEA USERS CONFERENCE 2017 LONDON #OSISOFTUC ©2017 OSIsoft, LLC
Presented by
Streaming Analytics,
Data Lakes and PI
Integrators
Matt Ziegler
Daniele Farris
EMEA USERS CONFERENCE 2017 LONDON #OSISOFTUC ©2017 OSIsoft, LLC
A singular data system that does all this and more
3
Reduce
maintenance
costs by up to
25%
Eliminate
70% of
breakdowns
Reduce
downtime up
to 50%
Cut
unplanned
outages by
up to 50% Reduce
scheduled
repairs by up
to 12%
Reduce capital
investment by
3-5%
EMEA USERS CONFERENCE 2017 LONDON #OSISOFTUC ©2017 OSIsoft, LLC
Presentation Goals
• Explore common Enterprise Data Architectures including
– Lambda & Kappa
– Data Lakes
• Clarify the role the PI System plays in each architecture
• Discuss how using the PI System helps maximize the effectiveness of scarce machine learning and analytics talent
• Demonstrate PI Integrator for Business 2017 with Kafka in a machine learning and streaming scenario
4
EMEA USERS CONFERENCE 2017 LONDON #OSISOFTUC ©2017 OSIsoft, LLC
Data Scientist is the sexiest job of 21st century, but…
5
Source: http://www.forbes.com/sites/gilpress/2016/03/23/data-preparation-most-time-consuming-least-enjoyable-data-science-task-survey-says/#5481f6037f75
EMEA USERS CONFERENCE 2017 LONDON #OSISOFTUC ©2017 OSIsoft, LLC
Data Scientist is the sexiest job of 21st century, but…
6
Source: http://www.forbes.com/sites/gilpress/2016/03/23/data-preparation-most-time-consuming-least-enjoyable-data-science-task-survey-says/#5481f6037f75
EMEA USERS CONFERENCE 2017 LONDON #OSISOFTUC ©2017 OSIsoft, LLC
7
Critical Assumption – Knowledge workers will have the appropriate knowledge
of processes and equipment to apply the correct context at the time of analysis.
EMEA USERS CONFERENCE 2017 LONDON #OSISOFTUC ©2017 OSIsoft, LLC
Challenge: Map raw data to the physical world
8
BGE003 FI-111 02T100 AF_FLOW3 02:T103 AT401
ACEDemo.Unit1.Output
TI-178 B352_W778 0_CMP_SVLV_PCT
02F104 CD:F161
GE01_DT 409510395_Wind Speed QI-109 GE01_DT Cooling Fan-711.Feed Rate
1-8.Net
Volume 03LBA32CT0
01-2
DC.SJ.ITLoad.P
WR
TI-145 FR2001
TI-178
GE04_OS
FT9001
FT9001 FR5001
AF_NOISE
DC.SJ.PUE TI-102 DC.Zero DY-108 DC.SJ.C1.Z3.R3.PDU1.PF GE01_A_DT
GE04_Status
02:F101.C
FIC-144 02F100 fasttag
FI-151 0_ENG_AUX_STS
AQUA2-SI005.PV GE02_Energy 0_ENG_MODE_STS FI-151 02T103.Q DC.Z1R
0_CMP_HDR
GE05_Energy C1:14AT5 AC03.Air Flow FeedBin.Cmt Boiler Cold Reheat Pressure
B737_FG117 DC.TimeLoad
D-110.Tank Pressure.PV GE04_DT QI-121 GE03_V_WIN
DC.Rk07R DC.Srv06R GE04_Energy
TI-121 FT9001
FAC.OAK.Power-Kh-Val.PV
fic1001.C GE02_OT
02F102.1HR
AVG BGT001
PI-111 facility_output
DM-05:BW.R
DC.SJ.C1.Z1.R1.Rk06.S2.O03.PWR QI-111FinalProductBin.On
94:GRDIDX.ProdID Boiler-209.Fuel Gas Flow
GE01_A_DT
fic1001.C
02:F101.C
FR2001 TIC-121
Aso AT401 DC.Srv01R
TI-178
asset1_output Active Meters
GE01_A_DT aso AF_NOISE PI-115 DM-05:BW.R
FI-101 Volume
0_CLR_FINAL_OUT_B_TMP F506_E990 339511775_Clear Sky Global Horiz GE01_DT
FI-111
% CO2 GE05_ES T
GE01_CON AlarmTest.Input.
Float32.1
DC.NY.Actual.PWR.day.Tot
80-5.Net Volume GE01_A_DT
45-2.Net Volume
Boiler-209.Fuel Gas
Flow DC.Srv01R
94:GRDIDX.Tr
trigger
AC09.Power
403511195_Wind
Speed
DC.C2Z1.Pwr.Rippl
e GE01_A_DT
AF_NOISE
1-16.Net Volume CB1992_MS
0_CMP_FLOW_TOTAL
FT9001
fasttag
FeedBin.Cmt
DC.Zone1.Number
FT9001
AlarmTest.Input.Float32.1
FT9001
DC.Srv01R Boiler-
125.Fuel Gas
Volume
1-13.Net
Volume B045_FG978
AT401
Anacortes Refinery.Alkylation.Asset
Problems B210_FG005.KPIExcursion
D-110.Tank
Pressure.P
V Boiler Feed
Pump #1
94:BW.R TI-101 F723_E889
369512185-Temp Compressor-439.Feed Rate
DC.CH.DCE FIC-172:210
FI-121 AF_FLOW3
0_ENG_MODE_STS
GE03_Q
DY-131:166 GE01_TD
Boiler-334.Feed Rate
GE04_OSAsset1.Problems QI-122 FI-151
DC.SJ.SiteRealTim
eITLoad.PR FT9001
Weather
Conditions
Relative Humidity: 34%
Current Temp: 85 F
High: 92
Low: 57 F
Wind: 8 mph/N
Crude Furnace
Draft Pressure: -0.5 WC
Stack Temp: 316 F
Oxygen: 2.5%
Outlet Temp: 840 F
Cold Oil Velocity: 6 ft/sec
Crude Desalter
Operating Pressure: 110 psi
Charge Rate: 14 gph
Mix Valve Pressure: 8 psi
Water Rate: 8%
fic1001.C FR5001
AQUA2-TI-201.PV
TIC-181
BGE003 FI-111 02T100
AF_FLOW3 02:T103
ACEDemo.Unit1.Output
TI-178 B352_W778 0_CMP_SVLV_PCT
02F104 CD:F161
AQUA2-SI005.PV GE02_Energy
0_ENG_MODE_STS FI-151 02T103.Q
DC.Z1R 0_CMP_HDR_SUC_PRS
DM-05:BW.R
DC.SJ.C1.Z1.R1.R
k06.S2.O03.PWR
QI-111FinalProductBin.On
94:GRDIDX.ProdID Boiler-
209.Fuel Gas Flow
FI-101 bf5e1d1d-
39c9-4b5b-b3d3-
c2ce05fa3a26 DM-05:BW.R AT401
AT401
DailyTrigger FrqPrbCost_ER
GE01_DT 409510395_Wind Speed
QI-109 GE01_DT Cooling Fan
AlarmTest.Input.Float32.10
364511575-AC Power
DY-101 02:F101.C
0_CLR_FINAL_OUT_B_TMP F506_E990
339511775_Clear Sky Global
Horiz GE01_DT
80-13.Net
DY-131 DC.SJ.PUE
AlarmTest.Input.Float32.1
GE01_DT
FI-101 bf5e1d1d-39c9-
4b5b-b3d3-c2ce05fa3a26 DM-05:BW.R AT401
4-36.NetVolumeAC04.Air Flow
02T100 03LBB02CT001-2
DC.SJ.SiteRealTim
eITLoad.PR FT9001
02F102.1HRAVG BGT001
PI-111 facility_output
AQUA2-SI005.PV GE02_Energy
0_ENG_MODE_STS FI-151
02T103.Q DC.Z1R 0_CM P_ H D R _SU C _ PRS
403511195_Wind Speed
GE02_Energy
FI-101 bf5e1d1d-39c9- 4b5b-b3d3-c2ce05f
EMEA USERS CONFERENCE 2017 LONDON #OSISOFTUC ©2017 OSIsoft, LLC
Challenge: Find events in raw data streams
9
What happened
during the event?
Event Attribute Value
Name Ex 20151215-0002
Start 15-Dec-2016
10:35:02
End 15-Dec-2016 11:47:26
Asset Boiler-352
Excursion Type High Violation
Fuel Gas
Flow.Avg 37.12 k sft3/h
myKPI.Max 47.19 bbl/d
Downtime Excursion
Batch
EMEA USERS CONFERENCE 2017 LONDON #OSISOFTUC ©2017 OSIsoft, LLC
01010101010101
01010101010101
01010101010101
01010101010101
01010101010101
01010101010101
01010101010101
01010101010101
01010101010101
01010101010101
01010101010101
01010101010101
01010101010101
01010101010101
01010101010101
01010101010101
01010101010101
01010101010101
01010101010101
01010101010101
01010101010101
01010101010101
01010101010101
01010101010101
01010101010101
01010101010101
01010101010101
01010101010101
01010101010101
01010101010101
PI System collects, organizes, and enriches data
PI Data
Archive
PI Asset Framework
Asset Analytics
Event Frames
Notifications Automation or
Control Systems
New Sensor
Technology
Remote and
Mobile Assets
PI ProcessBook
PI Manual Logger
PI DataLink
PI Vision
PI Server PI Interfaces & PI Connectors
SQL (OLEDB, JDBC, ODBC)
Web API (REST)
OPC (DA, HDA)
SDK (.NET-based)
PI Visualization Suite
PI System Access
EMEA USERS CONFERENCE 2017 LONDON #OSISOFTUC ©2017 OSIsoft, LLC
Challenge: Bring trustworthy data to advanced analytics
11
Financial
Inventory
Maintenance
Logs
Data Warehouse
Data Lake
Machine
Learning
Business
Intelligence
Tools
Stream Processing
GIS
Data stores Advanced Analytics Components
PI System can integrate across
advanced analytics architectures
and
deliver authoritative operational data
PI System PI Integrator
EMEA USERS CONFERENCE 2017 LONDON #OSISOFTUC ©2017 OSIsoft, LLC
Challenge: Bring trustworthy data to advanced analytics
12
Operational
Financial
Inventory
Maintenance
Logs
Data Warehouse
Data Lake
Machine
Learning
Business
Intelligence
Tools
Stream Processing
GIS
Culprits preventing trust:
Raw data replication
Manual edits to data
Data arrives out-of-order
Impact:
Unrealistic predictions
Two databases not synchronized
Analytics disconnected from physics
Incorrect calculations
Combine data to:
• Report
• Slice and dice
• Predict
• Train algorithms
Data stores Advanced Analytics Components
EMEA USERS CONFERENCE 2017 LONDON #OSISOFTUC ©2017 OSIsoft, LLC
14
Add speed and scale to
existing architectures
EMEA USERS CONFERENCE 2017 LONDON #OSISOFTUC ©2017 OSIsoft, LLC
15
Reference design pattern for
Industrial IoT Applications and
Streaming Systems and
Greenfield Data Systems
EMEA USERS CONFERENCE 2017 LONDON #OSISOFTUC ©2017 OSIsoft, LLC
16
Shift from single vendor integrated
solutions to best of breed customer
integrated solutions
Be aware of the transfer of risk
from vendors to open source, IT,
SIs
EMEA USERS CONFERENCE 2017 LONDON #OSISOFTUC ©2017 OSIsoft, LLC
PI System – An Integrated Kappa Architecture System
17
400+
Industr
ial
Protoc
ols
Optimized Time
Series Sequential
Store
Integrated Real-
Time Analytics
Critic
al
Event
s
Integrate
d
Visualiza
tion &
Program
ming
Access
EMEA USERS CONFERENCE 2017 LONDON #OSISOFTUC ©2017 OSIsoft, LLC
Product Portfolio (Infrastructure & Advanced Integrations)
18
PI Connectors &
PI Interfaces
(400+ Industrial
Protocols)
PI Data Archive
(Optimized Time
Series Sequential
Store)
Asset Framework
(AF) Analytics
(Integrated Real-
Time Analytics)
Event
Frames (EF)
(Critical
Events)
PI Vision, PI
DataLink, PI
System
Access
(Integrated
Visualization &
Programming
Access)
PI Server PI Integrators
PI Integrator for Business Analytics
PI Integrator for SAP HANA
PI Integrator for Microsoft Azure
External Systems
EMEA USERS CONFERENCE 2017 LONDON #OSISOFTUC ©2017 OSIsoft, LLC
PI Integrators augment the existing set of PI capabilities like
visualization, data access, and analytics with capabilities that
make it easy to interact with non-OSIsoft tool sets.
20
EMEA USERS CONFERENCE 2017 LONDON #OSISOFTUC ©2017 OSIsoft, LLC
PI Integrators – Climbing the Value Ladder
21
Real-time visibility
Monitoring
Process Optimization
Benchmarking
System Optimization
• BI App (i.e. Tableau, Spotfire, Lumira)
• PI Integrator for Business Analytics
• PI Integrator for SAP HANA
• Machine Learning (Azure ML, R)
• PI Integrator for Business Analytics
• PI Integrator for SAP HANA
• HMI
Disparate assets or interacting one-by-one Interacting with common assets as a fleet
Real-time & historical view across any plant
asset
Fleet-wide performance comparison
Large scale multi-variate analysis
Co
mp
lexi
ty
• PI ProcessBook • PI Coresight • PI Datalink
EMEA USERS CONFERENCE 2017 LONDON #OSISOFTUC ©2017 OSIsoft, LLC
Business
Intelligence
& Data
Warehouses
Available Today
PI Integrator for Business
Analytics
• Microsoft SQL Server, Oracle
• Hadoop (HDFS/HIVE)
PI Integrator for SAP HANA
Available
Cloud Platforms
• Microsoft Azure
• HANA Cloud Platform (5/2017)
Streaming
Systems
Real-Time GIS
PI Integrator for Esri ArcGIS
• Situational Awareness
• Real-Time Geoprocessing
• Import ESRI features (assets)
Considered (2018)
Stream Systems
• AWS Kinesis
• Esri GeoEvent Server
(Kafka)
2015-2016 2017 Future
New Integration
Patterns
Research
Enable partners and customers
to build applications and interact
programmatically using PI
Integrator Framework.
PI Integrator
Framework
Planned (Q4 2017)
• Process Scale out*
• SSL / HTTPS
Research
Enable business process
orchestration with PI System
data – workflow, asset sync,
transaction-like data, MES
Considered (2018)
More Platforms
• AWS Redshift
• Teradata
Planned (2H 2017)
Stream Systems
• Azure Event Hubs, IoT Hub
• Apache Kafka
• SAP SDS (Available 5/2017)
Planned (2018)
• All Integrators on common
Framework (ESRI)
• Node Scale Out and HA
Research
IoT Platform Integration with 3rd
parties
Aug 2017
EMEA USERS CONFERENCE 2017 LONDON #OSISOFTUC ©2017 OSIsoft, LLC
Integration Patterns
Tables
Files
Databases
Streams Other Patterns
Metadata
Programming
On-Demand
Workflow & Transactions
Files
Queues
Messaging
External Analytics Engines
EMEA USERS CONFERENCE 2017 LONDON #OSISOFTUC ©2017 OSIsoft, LLC
Tables vs Streams
• Business Intelligence
• Human readable
• Batch / Bulk Process
• Normalized data
• Regularly scheduled
• Large data, few messages
• In-line (Streaming) Analytics
• Computer readable
• Specific Data / Targeted Process
• Raw or “Packages” of data
• Triggered
• Small data, many messages
EMEA USERS CONFERENCE 2017 LONDON #OSISOFTUC ©2017 OSIsoft, LLC
Data Modeling
26
Data Modeling • Reference the entire
scope of the PI System using PI Asset Framework
• Normalize data across similar and dissimilar assets and equipment using templates, categories, and pattern searches
• Enrich datasets with location, status, calculations, and other metadata
EMEA USERS CONFERENCE 2017 LONDON #OSISOFTUC ©2017 OSIsoft, LLC
Message Designer
27
Message Designer • Variety of message
output formats such as JSON
• Customize the output schema, message triggering, and message content
• Filter erroneous data
• Replay historical data
EMEA USERS CONFERENCE 2017 LONDON #OSISOFTUC ©2017 OSIsoft, LLC
Soft Measurements – Ideal Use Cases for Streaming
• Predicting Soft Measurements
– Deschutes Brewery – Fermentation Transition
– [Pharma Company] – Lights Out Manufacturing
– Quality
• Enabling Data Lakes
– Customize how data is stored & processed
29
EMEA USERS CONFERENCE 2017 LONDON #OSISOFTUC ©2017 OSIsoft, LLC
Transition occurs between infrequent manual measurements
30
Fermentation
Diacetyl
Rest Maturation
Filling Free Rise Cooling Ready to
Transfer
Emptying
Empty
Constraints • One manual density measurement per vessel every 8-10
hours • Large capital expenditure not an option
Impact • Up to 72 hours lost in production
Options • $750k for inline density meters • Manually predict transition in spreadsheets
9 3 1
2 4
5
6
7
8
Time (hours)
Ap
pa
ren
t D
eg
ree
of
Fe
rme
nta
tio
n
Transition
Too late
Can this be predicted?
Soft Measurement
EMEA USERS CONFERENCE 2017 LONDON #OSISOFTUC ©2017 OSIsoft, LLC
SQL Data Warehouse
On Premises
Machine Learning
Power BI
INGEST PREPARE DATA SOURCES
ANALYZE PUBLISH CONSUME
Cortana
Web/LOB Dashboards
SQL Data Warehouse
PI System
On Cloud
Azure Data Factory (Orchestration)
Predictions as Future Data (to PI 2015)
PI Integrator for Microsoft Azure
Deschutes Brewery Operationalizing Architecture
31
EMEA USERS CONFERENCE 2017 LONDON #OSISOFTUC ©2017 OSIsoft, LLC
Automate and Optimize
32
Lab Measurements
• Viable Cell Count (VCC)
• Sugar Concentration
• API Concentration
• …
Online Measurements
• Agitation Speed
• pH
• Spectroscopy
• …
Goals: Optimize Cell Production, Eliminate Coming to the Lab
Technique: Use indirect measurements and machine learning
to predict outcomes of lab measurements
EMEA USERS CONFERENCE 2017 LONDON #OSISOFTUC ©2017 OSIsoft, LLC
What success looks like for our customers
Customer What
was done
Why
it matters
PI Integrator
Type
Industry
category
Marathon Oil Centralized, trusted
source for IT and OT
data sets
• Decreased time to analyze holistic data set from 3 months
to 3 weeks
• Decreased unplanned well downtime
SAP HANA Oil & Gas
White House
Utility District
Sped up identification
of water leaks across a
large rural area
• Optimized services team’s workflow to save $30,000 per
year
• Saved $900,000 in 2 years by preventing ongoing water
leaks
Esri ArcGIS Water
CEMEX Democratizing data for
decision makers
• Reduced time to begin analysis from months to minutes
• Reduced product variations significantly
Business
Analytics
Metals,
Mining, &
Materials
Deschutes Predicting events in
brewing process
across all brands
• Avoided $750,000 cost to automate density
measurements
• Saving on average 48 hours of production time per batch of
beer
Microsoft
Azure
Food &
Bev
Archer
Daniels
Midland
Prescribe operating
parameters that
extend equipment’s life
• Losing $50,000 every time machine fails, and 28 machines
failing at a high and unexpected rate
• Need advanced analytics approach to analyze 100s of
variables that could be influencing failure
Business
Analytics
Food &
Bev
EMEA USERS CONFERENCE 2017 LONDON #OSISOFTUC ©2017 OSIsoft, LLC
Announcing Community Technology Preview
• PI Integrator for Business Analytics 2017 CTP (Open)
• PI Integrator for Microsoft Azure 2017 CTP (RCIP)
34
EMEA USERS CONFERENCE 2017 LONDON #OSISOFTUC ©2017 OSIsoft, LLC
Reference Architecture for Streaming Analytics
36
Apache Kafka
PREPARE DATA SOURCES
STREAM
PI System PI Integrators Apache Spark
ANALYZE
Predictions as Future Data
Open Source Integration Pattern (Lambda Architecture) Primary Use Cases – • Deliver modeled input data to
CEP calculations and write predictions to operational system of record
• Fill a data like and maintain 1:1 fidelity of raw data or projected data from operational system of record
• Deliver aligned array data at scheduled intervals to R or Spark based compute engines
EMEA USERS CONFERENCE 2017 LONDON #OSISOFTUC ©2017 OSIsoft, LLC
38
Questions
Please wait for the
microphone before asking
your questions
Please remember to…
Complete the Online Survey
for this session
State your
name & company
EMEA USERS CONFERENCE 2017 LONDON #OSISOFTUC ©2017 OSIsoft, LLC
Matt Ziegler
Product Manager
OSIsoft
Daniele Farris
System Engineer
OSIsoft
39