Post on 27-May-2020
Standards
Certification
Education & Training
Publishing
Conferences & Exhibits
Reinventing the Role of
the SCADA Historian
Distributed Redundancy, Centralized Access
2013 ISA Water / Wastewater and Automatic Controls Symposium
August 6-8, 2013 – Orlando, Florida, USA
Speakers:
Blair Sooley, M. B. A., P. Eng. – Trihedral Engineering
2013 ISA WWAC Symposium
Aug 6-8, 2013 – Orlando, Florida, USA 2
Presenter
Blair Sooley is a Pre-Sales Account Manager
with Trihedral Engineering
• Electrical Eng. - Dalhousie Univ., Canada
• M.B.A. - St. Mary’s University, Canada
• 18 years in SCADA and process monitoring
• Specialized in high-level solution design
2013 ISA WWAC Symposium
Aug 6-8, 2013 – Orlando, Florida, USA 3
Presentation Outline
• Analysis of the historian
• Limitations and threats
• Currently employed protection schemes
• Improving efficiency, redundancy and synchronization
• Redefining the role of the historian
• Examples of advanced historian deployments
• Selecting a database format
• Servicing data requests
• Cost control
2013 ISA WWAC Symposium
Aug 6-8, 2013 – Orlando, Florida, USA 4
Historians Don’t Take Vacations
• Ever-increasing demands
– Reporting
– Upset analysis
– Process improvement
• Storage has become cheap
• Focus – meet demands rather than high availability
Why not focus on high-availability?
The answer is Simple. Because it’s Hard.
2013 ISA WWAC Symposium
Aug 6-8, 2013 – Orlando, Florida, USA 5
Analysis of the Typical Historian
Separate Working (real-time)/Historical (long-term) databases
Key steps to a Historian’s life
STEP 1: Periodic sampling of real-time DB
STEP 2: Comparison to set of rules
STEP 3: Copy to database or disregard
STEP 4: Service data requests
2013 ISA WWAC Symposium
Aug 6-8, 2013 – Orlando, Florida, USA 6
Limitations of Current Paradigm
• Don’t forget to have 1-2 summary slides at the end
Process EFFECT
Periodic sampling of real-time
database whether or not data
changed
Significant loading of CPU. Worsens
as system scales
Storage to business DB’s, such as
SQL and Oracle.
Platform mismatch. Such DB’s are
designed for many concurrent
connections and complex data
analysis. SCADA requires fast access,
simple format, few concurrent
connections and time-based datasets.
Data logged to single DB. DB is
queried from various sources.
No redundancy. Potential for data loss
due to single point of failure.
2013 ISA WWAC Symposium
Aug 6-8, 2013 – Orlando, Florida, USA 7
Understanding the Threats
• Natural disasters
• Equipment malfunctions
• Open networks make software more accessible.
Technology makes it more difficult to protect.
1US Department of Homeland Security (DHS), US Department of Energy (DOE) (2006). Introduction SCADA Security for Managers and Operators [Presentation].
Retrieved from http://energy.gov/sites/prod/files/oeprod/DocumentsandMedia/Introduction_to_SCADA_Security_for_Managers_and_Operators.pdf
DHS & DOE - Cyber threat sources1
1. Strategic Information Warfare (e.g. terrorism)
2. Direct Cyber Attacks (e.g. disgruntled employees)
3. General Cyber Attacks (for notoriety/fame)
2013 ISA WWAC Symposium
Aug 6-8, 2013 – Orlando, Florida, USA 8
Current Protection Schemes
RAID (Redundant Array of Inexpensive Disks)
Basic – RAID 1 (mirroring) and RAID 2 (striping)
Complex – RAID 10 (mirroring and striping)
• Concerns
– Single controller = Single point of failure
– SATA (Serial Advanced Technology Attachment) randomly
oriented disks cause performance degradation.
Number of Disks in Array Average Degradation vs. Single Disk
2 33% longer to process data read request
5 67% longer to process data read request
2013 ISA WWAC Symposium
Aug 6-8, 2013 – Orlando, Florida, USA 9
Current Protection Schemes
Offline Backup
Removable storage survives catastrophic failure.
• Concerns
– No synchronization - Current data only at the moment of creation
– Backup and restore take time
– Systems may not support online backup/restore
– Keeping up with media evolution (floppy, tape, ZIP, CD, USB)
2013 ISA WWAC Symposium
Aug 6-8, 2013 – Orlando, Florida, USA 10
Current Protection Schemes
Redundant Independent Historians
Data storage in two locations for fault tolerance
• Concerns
– No synchronization – differing data if sampling at different times
– No data backfilling after server outage
– Trends and reports differ based on which Historian is queried
2013 ISA WWAC Symposium
Aug 6-8, 2013 – Orlando, Florida, USA 11
Evolution of the Historian
11
Must become Faster, More scalable, Fault tolerant
New Historian RoleGuide the process of storage and
retrieval by dedicated sub processes
Old Historian RoleStore to and retrieve from a
single DB
2013 ISA WWAC Symposium
Aug 6-8, 2013 – Orlando, Florida, USA 12
Identify Wasted Effort
Historian servers are typically overloaded with unnecessary
tasks.
Play note
Is this the
correct note?
Disregard
No
Record
Yes
Switch to
different note
Record song• Periodic evaluation of
specific set of real-time
variables
• Unchanged values are
sampled and disregarded
• Repeat
Analogy – An Overworked Musician
2013 ISA WWAC Symposium
Aug 6-8, 2013 – Orlando, Florida, USA 13
Improve Efficiency by
Reducing Wasted Actions
Event-driven execution registers specific events of interest
with the SCADA System.
• SCADA notifies when a
specific event occurs
• Quicker response
• Able to handle a greater
number of events.
= Lightly loaded resources
Analogy Revisited
A Well-rested Musician
Play C note
‘Play ‘C’ Note’
event
Wait for
next event
2013 ISA WWAC Symposium
Aug 6-8, 2013 – Orlando, Florida, USA 14
Identify Synchronization Problems
Unsynchronized Historians may record at different times.
• Historian data may differ
• Summary data differs
based on DB queried;
• Trends
• Reports
• Calculations
Analogy
2 musicians play the same song with
slightly different arrangements
2013 ISA WWAC Symposium
Aug 6-8, 2013 – Orlando, Florida, USA 15
Improve Synchronization through
Assigned Leadership
Assigning Leader (primary) and follower (backup) positions
guarantees synchronization.
• Primary defines data to
be recorded
• Backup copies actions
explicitly and in sync
The leader selects the arrangement and
provides direction for the follower
Leader Follower
2013 ISA WWAC Symposium
Aug 6-8, 2013 – Orlando, Florida, USA 16
Multi-Level Redundancy
Allows Greater Data Security
Any number of redundant data storage locations can be
synchronized in this way.
Imagine this horn section had no leader!
In an orchestra, the 1st Chair is the section leader. 2nd and maybe 3rd Chairs are
appointed to ensure section members have a hierarchy of leaders to follow.
1st Chair Etc.2nd Chair 3rd Chair
1st Chair absent?
2nd Chair assumes
leadership position
2013 ISA WWAC Symposium
Aug 6-8, 2013 – Orlando, Florida, USA 17
Data Backfill
Mitigates Data Loss
Synchronization ensures data can be backfilled to a data
storage location that has been out of service.
A recording of one musician can be copied to fill in where
another musician was unavailable to record.
1st Chair Etc.2nd Chair 3rd Chair
Data is
backfilled
from primary
recording
2013 ISA WWAC Symposium
Aug 6-8, 2013 – Orlando, Florida, USA 18
Synchronization and Data Backfill
Georgetown, KY
Geographically separated data storage. Each plant
provides local storage and backup remote storage.
Real-time synchronization
Bi-directional backfill
Secure VPN IPSec tunnel
Minimal equipment
Fault tolerant
2013 ISA WWAC Symposium
Aug 6-8, 2013 – Orlando, Florida, USA 19
Traditional Historian
With Scalability comes Complexity
The Historian role includes recording, prioritization, data
longevity management and servicing data requests.
• Powerful servers required
• Limited scalability
Traditional Historians
Each server performs all functions
Server 1 Server 2
2013 ISA WWAC Symposium
Aug 6-8, 2013 – Orlando, Florida, USA 20
Distributed Data Storage Management
A Scalable Alternative
Lightly loaded SCADA workstations offer opportunity for
distributed task assignment.
• Storage and retrieval
activities reassigned to
computers with excess
CPU and storage
• Distribute actions by
functional area
• Dedicated Historian
computers eliminated
Distributed Task Assignment
Conducting the Orchestra
Conductor
directs
activities
Each orchestra section
assumes responsibility
for its own part
2013 ISA WWAC Symposium
Aug 6-8, 2013 – Orlando, Florida, USA 21
Redefining RAID for Historian Use
Drives on SCADA computer network are clustered to form a
data storage network of subparts with central read access.
Clustered data striping and mirroring (RAID 10)
Greater fault tolerance than standard RAID due to dedicated controllers
Plant 1
primary
storage
Plant 1
backup
storage
Mirroring
(RAID 1)
Plant 2
primary
storage
Plant 2
backup
storage
Striping
(RAID 5)
2013 ISA WWAC Symposium
Aug 6-8, 2013 – Orlando, Florida, USA 22
Distributed Historian Storage
Ocala, FL
6 distributed storage locations
Small local storage, large central
Backup local computers eliminated
If the centralized DB is unavailable, data requests are broken into sub-requests for servicing
by each of the remote sites. The responses are combined and delivered to the requestor.
2013 ISA WWAC Symposium
Aug 6-8, 2013 – Orlando, Florida, USA 23
Eliminating a new
Single Point of Failure
Identify hierarchy to eliminate the Historian as a SPOF.
Help, the conductor is sick! – Alas, the show must go on.
The leadership position is assumed based on an established hierarchy such
that the orchestra is never without leadership.
Conductor Etc.Backup
Conductor
2nd Backup
Conductor
Conductor
unavailable?
Backup conductor
assumes leadership
position
2013 ISA WWAC Symposium
Aug 6-8, 2013 – Orlando, Florida, USA 24
Enhanced Fault Tolerance
Data Buffering at the Source
Distributed historian storage and autonomy allows greater
fault tolerance by buffering local data at remote assets.
Buffered data storage at
remote asset
Centralized
data storage
Data
Backfill
• No data loss after network
or central computer outage
• Replace local HMI with
integrated SCADA node
• Lightly loaded local node
easily handles logging.
2013 ISA WWAC Symposium
Aug 6-8, 2013 – Orlando, Florida, USA 25
Data Buffering at the Source
Ontario Power Generation, ON – (plan)
Existing central DBs provide collocated redundancy
Expansion plan
• Add local buffering at 26 remote generation facilities
• Data compression on backfill limits network traffic
2013 ISA WWAC Symposium
Aug 6-8, 2013 – Orlando, Florida, USA 26
Database Format Selection
The Historian controls the process. As long as it can R/W
data to the format, the storage format should not matter.
Database considerations
• Native SCADA formats may
be faster
• 3rd party software access
(reports, CMMS, etc.)
• Some formats provide better
security against hacking
• Cost
2013 ISA WWAC Symposium
Aug 6-8, 2013 – Orlando, Florida, USA 27
Combining Database Formats
Gainesville Regional Utilities, FL
Single SCADA application with mixed historical
database formats.
• Data sync and backfill across
differing DB formats
• Native DB’s high speed
required by operations team
• Existing SQL DB leveraged
by engineering team
2013 ISA WWAC Symposium
Aug 6-8, 2013 – Orlando, Florida, USA 28
Servicing Data Requests
Historian = single point of contact for servicing data requests.
• Storage mechanism is
irrelevant to requestor
• Historian knows where all
data is stored
• Complex requests (min,
max, ave., totals, counts)
can be derived from raw
data on demand
Taking Requests
From the Audience
Conductor
determines if his
orchestra has the
capabilities to
service the
request.
Only specific orchestra
sections are required to
meet request
2013 ISA WWAC Symposium
Aug 6-8, 2013 – Orlando, Florida, USA 29
Cost Control
Expectation
Faster/more robust/scalable historian = Increased cost
Actual
Reduced infrastructure = Reduced cost
$ SAVINGS OPPORTUNITIES $
Eliminate dedicated historian servers
Exchange server hardware for workstation hardware
Eliminate collocated server redundancy
Use native DB formats over expensive business formats
2013 ISA WWAC Symposium
Aug 6-8, 2013 – Orlando, Florida, USA 30
Summary
Problem
• Scalability and efficiency concerns plague existing systems
• Demands are increasing. Historical data protection has become critical.
• Current protection schemes are inadequate
Solution
• Synchronization and data backfill are key to data quality and service
availability
• Role of the new Historian is to manage synchronization, to guarantee
data backfill to all replicated storage notes, and to protect the integrity
of the data storage
• Cost control requires realization of underutilized infrastructure. Highly
organized management of distributed architecture increases reliability.
2013 ISA WWAC Symposium
Aug 6-8, 2013 – Orlando, Florida, USA 31
Thank you!