Control Center Virtualization for the Cisco Connected ... · PDF fileCisco Connected Pipeline...

Control Center Virtualization for the Cisco Connected PipelineDesign GuideApril 2016

Connected Refineries 1.0 Cisco Reference Documentii

ALL DESIGNS, SPECIFICATIONS, STATEMENTS, INFORMATION, AND RECOMMENDATIONS (COLLECTIVELY,

"DESIGNS") IN THIS MANUAL ARE PRESENTED "AS IS,WITH ALL FAULTS. CISCO AND ITS SUPPLIERS DISCLAIM

ALL WARRANTIES, INCLUDING, WITHOUT LIMITATION, THE WARRANTY OF MERCHANTABILITY, FITNESS FOR

A PARTICULAR PURPOSE AND NONINFRINGEMENT OR ARISING FROM A COURSE OF DEALING, USAGE, OR

TRADE PRACTICE. IN NO EVENT SHALL CISCO OR ITS SUPPLIERS BE LIABLE FOR ANY INDIRECT, SPECIAL,

CONSEQUENTIAL, OR INCIDENTAL DAMAGES, INCLUDING, WITHOUT LIMITATION, LOST PROFITS OR LOSS OR

DAMAGE TO DATA ARISING OUT OF THE USE OR INABILITY TO USE THE DESIGNS, EVEN IF CISCO OR ITS SUP-

PLIERS HAVE BEEN ADVISED OF THE POSSIBILITY OF SUCH DAMAGES.

THE DESIGNS ARE SUBJECT TO CHANGE WITHOUT NOTICE. USERS ARE SOLELY RESPONSIBLE FOR THEIR

APPLICATION OF THE DESIGNS. THE DESIGNS DO NOT CONSTITUTE THE TECHNICAL OR OTHER PROFES-

SIONAL ADVICE OF CISCO, ITS SUPPLIERS OR PARTNERS. USERS SHOULD CONSULT THEIR OWN TECHNICAL

ADVISORS BEFORE IMPLEMENTING THE DESIGNS. RESULTS MAY VARY DEPENDING ON FACTORS NOT

TESTED BY CISCO.

The Cisco implementation of TCP header compression is an adaptation of a program developed by the University of California,

Berkeley (UCB) as part of UCB’s public domain version of the UNIX operating system. All rights reserved. Copyright © 1981,

Regents of the University of California.

Cisco and the Cisco logo are trademarks or registered trademarks of Cisco and/or its affiliates in the U.S. and other countries. To

view a list of Cisco trademarks, go to this URL: http://www.cisco.com/go/trademarks. Third-party trademarks mentioned are the

property of their respective owners. The use of the word partner does not imply a partnership relationship between Cisco and any

other company. (1110R).

Any Internet Protocol (IP) addresses and phone numbers used in this document are not intended to be actual addresses and phone

numbers. Any examples, command display output, network topology diagrams, and other figures included in the document are

shown for illustrative purposes only. Any use of actual IP addresses or phone numbers in illustrative content is unintentional and

coincidental.

Control Center Virtualization for the Cisco Connected Pipeline Design Guide

© 2016 Cisco Systems, Inc. All rights reserved.

http://www.cisco.com/go/trademarks

Design Guide

C O N T E N T S

Document Objective and Scope vii

Contributors vii

Connected Pipeline Overview 1-1

Executive Summary 1-1

The Oil and Gas Value Chain 1-2

Pipeline Management Systems 1-5

SCADA System Design Principles 1-5

Availability 1-9

Security 1-9

Management 1-11

Baseline Integrated SCADA System 1-12

Connected Pipeline Architecture 2-1

Control Center Overview 2-2

Schneider Electric Pipeline Management Solutions 2-2

Production Environment 2-3

Test and Development 2-4

Test System 2-4

Development System 2-4

Decision Support System/Industrial Demilitarized Zone 2-4

Control Center Availability and Security 2-5

Availability 2-5

Security 2-5

Multi-Service Traffic (Non-Operational Applications) 2-6

Operational Communications Network Overview 2-7

Use Case: SCADA Real-time /Near Real-time Operations 2-10

SCADA to Pipeline 2-11

Control Center-to-Control Center Communications 2-11

Connected Pipeline System Design 3-1

BLISS Design 3-1

Functional Data Center Design 3-2

Network 3-2

iiiControl Center Virtualization for the Connected Pipeline

Contents

Compute 3-3

Storage 3-3

Security 3-3

Security Design 3-3

Segmentation Design 3-3

Control Center Segmentation and Zones 3-3

Compute Segmentation 3-6

Virtual Switch (vSwitch) 3-6

Storage Segmentation 3-7

File System Management 3-7

Fiber Channel Zoning and VSANs 3-7

LUN Masking 3-8

Summary 3-8

Firewall and Industrial DMZ Design 3-8

Overview 3-9

Firewall Deployment Options 3-9

Industrial DMZ Concepts 3-10

IDMZ and DSS Design 3-11

Remote Access Overview 3-12

Inter-Control Center Firewall Considerations 3-14

Pipeline Communication Firewall Considerations 3-14

Intra-Control Center Firewall Considerations 3-15

Summary 3-15

Network Infrastructure Security 3-15

Shutdown Unused Ports 3-15

Trunk Ports 3-15

DHCP Snooping 3-16

Dynamic ARP Inspection (DAI) 3-16

Port Security 3-16

Traffic Storm Control 3-17

Infrastructure Management 3-17

Control Plane Policing (CoPP) 3-17

Availability 3-17

Network Redundancy 3-18

Port Channel 3-19

Virtual Port Channel 3-19

ASA Redundancy 3-20

Summary 3-20

Compute Redundancy 3-20

Fabric Redundancy 3-20

ivControl Center Virtualization for the Connected Pipeline

Design Guide

Contents

End Host Mode 3-21

SCADA Server Connectivity and Redundancy (Virtual Machines) 3-21

Storage Redundancy 3-22

Hardware Redundancy 3-22

Redundant SAN 3-23

Multipath 3-23

Summary 3-23

Schneider Electric OASyS Application Design Application Redundancy 3-24

Virtual Machine Redundancy 3-24

SQL SAN Storage Hardware Mirroring 3-24

Virtual Guest Storage 3-24

Microsoft SQL Server-Based Historian 3-24

Operator Workstation Connectivity 3-25

Network Management 3-25

Cisco UCS Manager 3-25

Cisco Adaptive Security Device Manager 3-26

Out of Band Management 3-26

Role-Based Access Control 3-27

Summary 3-27

Time Synchronization 3-27

System Implementation Considerations 4-1

Server Sizing and Storage Sizing 4-1

Scalability Considerations 4-1

System Components 5-1

A P P E N D I X A Glossary A-1

vControl Center Virtualization for the Connected Pipeline

Design Guide

Preface

This Cisco Control Center Virtualization for the Connected Pipeline System Cisco Validated Design (CVD) documents best practice design and implementation of safe, highly available, and secure oil and gas pipeline infrastructure and applications. This CVD identifies customer use cases, maps those use cases to relevant architectures, and describes how Cisco and partner technology are leveraged to deliver unprecedented value for our customers. This CVD:

• Describes a Low Level Design (LLD) detailing Control Center Virtualization for the Connected Pipeline System. It will provide guidance supporting Supervisory Control and Data Acquisition (SCADA) principles, unified networking and powerful compute for the Control Center.

• Documents best practices from real world implementations, detailing the designs and architectures that are mapped back to the customer use cases.

• Addresses real-life customer deployment scenarios by providing a solution that supports implementation of a scalable, secure, and redundant operational network supporting both industrial and multi-service applications.

• Details support for implementing Control Center application virtualization, secure remote access, the Industrial Demilitarized Zone (IDMZ), and cyber-security.

• Specifies topology, Quality of Service (QoS), high availability, security services, network management services, and Control Center virtualization implementations.

• Provides information about enforcing cyber-security best practices that follow the recognized Industrial Control System (ICS) security standards and guidelines including International Society of Automation 99(ISA99)/International Electrotechnical Commission (IEC) 62443, the National Institute of Standards and Technology (NIST) Cyber Security Framework, and the Purdue Model of Control.

• Documents suggested equipment and technologies, system level configurations, and recommendations. It also describes caveats and considerations that pipeline operators should understand as they implement best practices.

Although this CVD focuses on midstream transport pipelines, the technologies, use cases, and principles are applicable for gathering and distribution pipelines.

viControl Center Virtualization for the Connected Pipeline

Design Guide

REVIEW DRAFT—CISCO CONF IDENT IAL

Document Objective and ScopeIn this initial release, Cisco has partnered with Schneider Electric to provide architecture, design, and technologies for the Control Centers, Operational telecoms network, and the pipeline stations. Cisco provides infrastructure expertise with its unified compute and networking security platforms while Schneider Electric provides the PMS (PMS) leadership with its OASyS Dynamic Network of Applications (DNA) SCADA system hardware and software.

The release will focus on the Control Center environment and security architecture to support pipeline operators. It is recommended that the reader become familiar with the following joint Cisco/Schneider Electric white papers:

• Integrated Enterprise SCADA System Architectures for Safe and Efficient Pipeline Operations at the following URL:

– http://www.cisco.com/c/dam/en/us/solutions/collateral/industry-solutions/dlfe-683318406.pdf

• Converged Telecommunication Architectures for Effective Integrated Pipeline Operations at the following URL:

– http://www.cisco.com/c/dam/en/us/solutions/collateral/industry-solutions/dlfe-683318407.pdf

As with any architecture and design program, functional requirements, use cases, and architectures evolve. Therefore, this CVD will evolve and will be updated in future phases.

ContributorsJason Greengrass, Solutions Architect, IVSG, Cisco Systems

Rik Irons-McLean, Lead Architect Oil and Gas, IoE Vertical Solutions Group, Cisco Systems

Brian Malkinson, Global Solution Architect, Pipeline Management Systems, Schneider Electric

viiControl Center Virtualization for the Connected Pipeline

Design Guide

http://www.cisco.com/c/dam/en/us/solutions/collateral/industry-solutions/dlfe-683318406.pdf

http://www.cisco.com/c/dam/en/us/solutions/collateral/industry-solutions/dlfe-683318407.pdf

ContDesign Guide

C H A P T E R 1
Connected Pipeline Overview
This chapter includes the following major topics:

• Executive Summary, page 1-1

• The Oil and Gas Value Chain, page 1-2

• Pipeline Management Systems, page 1-5

• SCADA System Design Principles, page 1-5

Executive SummaryThis chapter provides a high level overview of the end-to-end Oil and Gas value chain and where pipeline solutions fit into this chain. It also provides an overview of the emergence of virtualization technologies into these environments. This document is written for an industry with a number of key trends:

• Health and Safety—The health and safety of employees continues to be of major importance for organizations. The industry looks to improve overall worker safety while specifically providing a safe working environment for remote or unaccompanied workers.

• Environmental Safety and Compliance—Solutions must meet or exceed industry standards or regulations such as the Pipeline and Hazardous Materials Safety Administration (PHMSA), with increased attention to safety and compliance in regulations a major design factor for telemetry and SCADA systems today.

• An Aging Workforce—Worker age and skill sets have changed. As younger workers with more of an IT-based skill set join the workforce, being able to train and provide remote expertise and consultation to new workers is essential.

• Predictive Automation and Process—Through Big Data, fog or edge compute, and analytics and cloud-based services, sensors are able to provide real-time information on such measures as temperature, vibration, pressure, flow, and current. Combining this with statistical models provides predictive methods for maintenance of equipment and streamlining of processes. The Internet of Things (IoT) has focused on connecting the unconnected through wireless and wired networks, and previously inaccessible data is now available for use.

• Security—As technology evolves, more devices are connected to the network, attackers use increasingly sophisticated methods, and OT and IT technologies begin to converge, protecting assets, people, and intellectual property from cyber and physical threats becomes ever more important.

1-1rol Center Virtualization for the Connected Pipeline

Chapter 1 Connected Pipeline Overview The Oil and Gas Value Chain

It is essential to understand that a single technology cannot enable the industry to meet these requirements. Only a properly architected, secure integration of a number of technologies and applications will keep workers safe, improve efficiencies, reduce cost, and continue to drive innovation.

The Oil and Gas Value ChainAt a high level, the Oil and Gas value chain starts with discovering resources through exploration, and then the development, production, processing, transportation/storage, refining, and marketing/retail of hydrocarbons. This value chain is normally grouped into the upstream, midstream, and downstream areas, as shown in Figure 1-1.

Figure 1-1 Oil and Gas Value Chain

• Upstream—Upstream includes the initial exploration, evaluation and appraisal, development, and production of sites. This is referred to as Exploration and Production (E&P). These activities take place onshore and in the ocean. Upstream focuses on finding wells, determining how best and how deeply to drill, and determining how to construct and operate wells to achieve the best return on investment.

• Midstream—Midstream primarily focuses on the transport and storage of hydrocarbons via transmission pipelines, tankers, tank farms, and terminals, providing links between production and processing facilities, and processing and the end customer. Crude oil is transported downstream to the refinery for processing into the final product.

Midstream also includes the processing of natural gas. Although some of the needed processing occurs as field processing near the source, the complete processing of gas takes place at a processing plant or facility, reaching there typically from the gathering pipeline network. For the wholesale markets, natural gas must first be purified by removal of Natural Gas Liquids (NGLs) such as butane, propane, ethane, and pentanes, before being transported via pipeline, or turned into Liquid Natural Gas (LNG) and shipped. The gas can be used real-time or stored. The NGLs will be leveraged downstream for petrochemical or liquid fuels, or turned into final products at the refinery.

• Downstream—Downstream is concerned with the final processing and delivery of product to wholesale, retail, or direct industrial customers. The refinery treats crude oil and NGL and then converts them into consumer and industrial products through separation, conversion, and

Upstream Midstream Downstream Upstream Midstream Downstream

Explore Develop Produce Storage & Transportation Refine Marketing

Marketing Process Transport

Offshore Shipping Trading

Research & Development, Engineering, High Performance Compute

Office Facilities, Call Center, Data Center

Key Oil Gas 3

76

49

8

1-2Control Center Virtualization for the Connected Pipeline

Design Guide


purification. Modern refinery and petrochemical technology can transform crude materials into thousands of useful products including gasoline, kerosene, diesel, lubricants, coke, and asphalt. Downstream also includes gas distribution pipeline networks.

A visual overview of the value chain is shown in Figure 1-2.

Figure 1-2 Oil and Gas System

Transmission pipelines are the key transport mechanism for the oil and gas industry and operate continuously outside of scheduled maintenance windows. Pipelines provide an efficient, safe, and cost-effective way to transport processed or unprocessed oil, gas, and raw materials and products both on- and offshore. It is essential that they operate as safely and efficiently as possible, and, where problems occur, they must be able to rapidly restore normal operation to meet environmental, safety, and quality requirements.

Oil and Gas pipelines (Figure 1-3) comprise operating process, safety, and energy management functions geographically spread along the pipeline for a set of stations. Stations vary in size and function, but typically include large compressor or pump stations, mid-size metering stations, Pipeline Inspection Gauge (PIG) terminal stations, and smaller block valve stations. Each process and application must be linked with the applications and processes at other stations, and at the Control Centers (main and backup) through an operational field telecoms infrastructure. The process must be done in a reliable and efficient way, avoiding communications outages and data losses. The Control Centers should also be securely connected to the enterprise through a Wide Area Network (WAN) to allow users to improve operational processes, streamline business planning, and optimize energy consumption.


Design Guide


Figure 1-3 Example Pipeline Station Distribution

Oil and Gas pipeline management is challenging, with pipelines often running over large geographical distances, through harsh environments, and with limited communications and power infrastructure available. In addition, pipelines must comply with stringent environmental regulations and operate as safely as possible, and address growing cyber and physical security threats.

Key pipeline requirements, however, have not changed. Pipeline integrity, safety, security, and reliability are essential elements that help operators meet demanding delivery schedules and optimize operational costs.

At the same time, new operational and multi-service applications are enhancing the way assets and personnel operate. Modern cathodic detection, distributed acoustic leak detection, landslip/earthquake detection, intrusion detection, and physical security applications allow operators to reduce downtime, optimize production, and decrease energy and maintenance costs. Real-time operational data access allows incidents to be identified and addressed quickly, or prevented from occurring in the first place.

Challenges must be addressed through a secure communications strategy to ensure operators can confidently rely on remote data, video, and collaboration solutions for safety and security in addition to operations.

Communications architectures, technologies, solutions, and management for process, energy, security, and multi-service applications (Figure 1-4) must be robust, flexible, and scalable. They should be based on open standards, allowing operations from field device to Control Center, and Control Center to enterprise by combining real-time process and business control automation, information management, energy management, and security with global supervision.

MCC CP BCC M C

P T CP M B B B B B B B B B B B B B

Main/Backup Control Center

Metering/PIG Sta�on

Compressor / Pump Sta�on

Block Valve Sta�on

Pipeline Length

B

B

B

Terminal Sta�on

Component Function

Control Center Monitoring and control of the pipeline system

Compressor station Provides pressure for gas pipelines to keep flow moving

Pump station Provides pressure for oil pipelines to keep flow moving

Metering station Simultaneous, con�nuous analysis of quality and quan�ty being transferred in a pipeline

PIG station Cleaning and inspec�ng the pipeline and flowlines

Terminal station Where product will be delivered to end customer

Block valve station Isolate a segment of the line for leaks or maintenance

MCC BCC

Control Centers may be part of pipeline or in different geographic loca�ons

37

64

99


Design Guide

Chapter 1 Connected Pipeline Overview Pipeline Management Systems

Figure 1-4 High Level Pipeline Architecture

Pipeline Management SystemsReal-time monitoring and control through sharing and collection of data to a centralized PMS is critical for ensuring that the product is transported safely and efficiently. A PMS combines operational SCADA with real-time applications specific to the oil and gas industry, host-based leak detection, and historical flow measurement.

A well-designed PMS uses a hardware and software architecture that allows functions to be mobile, scalable, flexible, and robust. It also permits distribution of processing among different SCADA system components to optimize overall performance of the PMS.

These integrated applications provide pipeline operators:

• Real-time/near real-time control and supervision of operations along the pipeline through a SCADA system based in one or more Control Centers

• Accurate measurement of flow, volume, and levels to ensure correct product accounting

• Ability to detect and locate pipeline leakage, including time, volumes, and location distances

• Integrated security systems for personnel, the environment, and infrastructure using video surveillance, access control, and Intrusion Detection Systems (IDS)

• Ensured safe operations through instrumentation and safety systems

• Energy management system to visualize, manage, and optimize energy consumption within the main stations

SCADA System Design Principles Pipeline requirements will vary depending on customer preference and project, so it is essential that any communications solution be scalable and modular; elements must be able to be interchanged without affecting fundamental architectural and operational functions.

Pipeline

Main Sta�on (xN) (Compressor /

Pump)

Main Sta�on (xN) (Metering / PIG /

Terminal)

Block Valve Sta�on (xN)

Converged IT and OT Opera�onal Field Telecoms

WAN

Ope

ra�o

nal

Appl

ica�

ons

Mul

�ser

vice

Ap

plic

a�on

s

Ope

ra�o

nal

Appl

ica�

ons

Mul

�ser

vice

Ap

plic

a�on

s

Ope

ra�o

nal

Appl

ica�

ons

Mul

�ser

vice

Ap

plic

a�on

s

Corporate Office / Business Domain

Main Control Center Backup Control Center

Ope

ra�o

nal

Appl

ica�

ons

Mul

�ser

vice

Ap

plic

a�on

s

Virt

ualiz

ed

Data

Cen

ter

Ope

ra�o

nal

Appl

ica�

ons

Mul

�ser

vice

Ap

plic

a�on

s

Virt

ualiz

ed

Data

Cen

ter

37

65

00


Design Guide

Chapter 1 Connected Pipeline Overview SCADA System Design Principles

The Control Center consists of operational and non-operational domains (Figure 1-5), which are divided into secure zones based on IEC 62443-3-3, Functional Requirement (FR) 5, Restricted data flow. This segments the control system architecturally via zones and conduits to limit the unnecessary flow of data.

Figure 1-5 Control Center Segmentation Domains

SCADA system environments within an architecture include operational zones for:

• Production—Domain controllers, real-time, historical and leak detection servers, operator workstations

• Test—Non-production replica of the operational SCADA system, allowing software and system changes to be validated prior to production without disrupting production

• Development—Development area for reports, displays and database changes

• Training—Training area built upon the environment that a pipeline controller lives in every day, via a fully functional SCADA control system together with accurate simulation of the pipelines they control

• Decision Support/Industrial DMZ (L3.5):

– Decision support domain controllers and real-time historical and remote access services

– Isolates the operational system from any external systems or users

– Synchronized with real-time and historical data from the Production system to provide a secure method of providing real-time data to external users

• With firewalls creating a DMZ, secure services are enabled to provide the external environments exposure to the data.

• Functions may include secure File Transfer Protocol (FTP) connections and long term historians

And non-operational zones:

• Multi-service—Non-production applications which support the operation of the pipeline such as physical security, voice, Public Address and General Alarm (PAGA) safety announcements, video, and wireless

These building blocks are core components of a SCADA architecture design that can be deployed in one or more Control Centers. Customer philosophy and individual requirements will dictate how these zones are created and into which sites they are deployed. These are typically based on an internal risk assessment to ensure safe and reliable operations. Risk assessment guidelines to help determine philosophy and architectural design are available in the IEC 62443-3-2 standard Security for Industrial Automation and Control Systems - Security Risk Assessment and System Design.

RealTime (Core)ProductionSCADA

Primary

SCADA

Backup

Decision SupportIDMZ

Training

Test


Training

Non-RealTime (Core)

DevelopmentReplica of Production

Location may vary

Mai

n C

ontr

ol

Cen

tre

Bac

kup

Con

trol

C

entr

e


Primary

SCADA

Backup


MultiservicePhysical Security Voice Services

Emergency

AnnouncementsWireless

Remote Expert Data Access

Ope

ratio

nal D

omai

ns

Non

-Ope

ratio

nal

Dom

ains

37

65

01


Design Guide


To allow appropriate data flows to take place between zones for system operation, a communications and security infrastructure is implemented (see Figure 1-6), which provides conduits to transport only permitted data.

Figure 1-6 Control Center Communication Building Blocks

This network segmentation defined in IEC 62443-3-3, System Requirement (SR) 5.1, Network Segmentation, focuses on separating control system networks from non-control system networks, and also critical control systems from non-critical control systems. Options exist for logical segmentation, physical segmentation, or both. Again, the choice is based on customer philosophy. Guidance is available for the risk assessment to determine the philosophy in IEC 62433-3-2.

In addition to segmentation, boundary control needs to be established between each of these areas. Guidance on boundary control is found in IEC 62443-3-3, SL 5.2.

To ensure the required high availability and security, the SCADA PMS has traditionally been deployed on physically separate stacks (compute, storage, and network) with boundaries at the firewall level (see Figure 1-7). This ensures strict policy enforcement between zones, and leverages logical segmentation inside zones to differentiate between traffic types.

Figure 1-7 SCADA Deployment Physical Separation

37

65

02

WAN to Pipeline

SCADA A VPN

Multiservice VPN

Production

Test

IDMZ• Decision Support

• Remote Access

SCADA B VPN

Multiservice

Non-Operational Domain

Operational Field Telecoms

Training

Development

RealTime (Core)

Non-RealTime (Core)

Communications and Security Infrastructure

FC Switch

L3 SwitchSync

Storage

ASA F/W

L3 WAN Routers

L3 Switch

Dual FC SAN

Dual FC SAN

Dual FC SAN

Enterprise Enterprise

Test A Server

Test B Server

Development

Remote Access/DMZ

Production Server A

Production Server B

DSS Server

Dual FC SAN

Test/DevelopmentDSS/IDMZ

Production

WAN

Layer 2Modular

Training

Positioning undecided. Could fit in either environment

Routers

L2 Switch

UCS C Server

Virtual

IP

10.1.1.1

10.1.1.2 10.1.1.3

VM VM

37

65

03


Design Guide


Virtualization has transformed the data center environment over the past decade, allowing consolidation of multiple standalone bare-metal servers and applications onto smaller and more powerful nodes. The primary business push for virtualization was the aim to achieve better server usage from powerful and smaller physical servers, thus leading to an improved Total Cost of Ownership (TCO) and additional operational efficiencies (such as power, cooling, and space).

The deployment of full or partial virtualized solutions has increased in recent years, and industrial security standards and guidelines such as IEC 62443, NERC-CIP, IEC 62351, and the NIST Cybersecurity Framework have an increasing focus on the deployment of virtualization technologies.

This means that the core SCADA Control Center architectural building blocks are implemented, with areas of shared infrastructure as shown in Figure 1-8.

Figure 1-8 SCADA Deployment Leveraging the Consolidated Architecture

The choice of deployment depends on end user requirements and philosophy. However, any deployment must focus on high availability, redundancy, and security. A holistic security strategy should include risk assessment, change management, recovery processes, training, information protection, and incident response—these will determine choice.

For the purposes of this CVD, the consolidated deployment will be used. This will allow the core architectural principles to be leveraged, and provides best practice design guidance and validation, including virtualization. This also ensures that high availability, reliability, and security requirements are met as per the following design guidance sections.

Security concepts and enforcement are drawn directly from IEC 62443 to support high availability and reliability. When any potential impacts may occur (such as scope for misconfiguration), they will be highlighted in the document.

37

65

14

VM

VM

Fabric Ports

Ethernet Ports

FC Links

Virtual

IP

10.1.1.1

10.1.1.3

10.1.1.2

Production A

Test A

DSS/IDMZ B

DSS/IDMZ A

Test B

Storage B

WAN

Production B

Dev B

Dev A

L2 Switch

L3 WAN Routers

Storage A

UCS Chassis A

UCS Chassis B

ASA Firewalls

Dual FC SAN

Dual FC SAN

L2 Switch

Fabric Interconnect

Storage

ASA F/W

Routers

UCS Chassis

UCS Blade Server

Enterprise

Fabric Interconnection


Design Guide


The jointly architected and validated approach to pipeline management and telecommunications offers many realizable benefits. Solution integration quality and interoperability are maximized, while design and testing time are minimized. End users have a single point of reference accountable for integration and operational success from hardware, software, security, and management perspectives throughout a project life cycle. The jointly architected design will provide maximum benefit for current operations, and be a platform for future application enablement and integration.

The key elements of this jointly architected and validated design will be discussed in detail in the following sections.

AvailabilityThe system design must encompass a highly available architecture. The pipeline operator must have control of the pipeline 24 hours a day and 365 days of the year. Any loss of visibility or communications will either enforce a shutdown of the process resulting in loss of revenue, or in a worst case scenario, not provide the ability to shut down the pipeline under a catastrophic safety incident such as a major leak.

No Single Point of Failure (SPOF) should occur on any critical system component of the SCADA system design. A critical component is any component whose failure directly and adversely affects the overall performance of the SCADA system or its ability to continue performing the critical SCADA functions of monitoring and control. The SCADA system uses modular components so that the failure of a single component does not render other components inoperative.

Within this design, redundancy is provided for all critical SCADA functions for monitoring and control. Components comprising the standby capability continuously receive updated data, as appropriate, to provide a hot-standby capability in case of a hardware- or software-initiated failover. As an example, a hot server or critical SCADA application will have a standby equivalent within a Control Center and updates will be passed from this server/application to a backup Control Center if deployed.

The SCADA system connects to the telecommunication networks in such a way that a failure of these networks does not affect the ability of the SCADA system to perform its critical functions for monitoring and control. The logic within controllers and the safety systems along the pipeline will still operate if the Control Center loses connectivity to the pipeline stations; however, the ability to control and monitor the pipeline would be lost. Therefore, it is critical that communications to a Control Center are maintained.

SecuritySecurity, safety, and availability are tightly aligned within an industrial security framework. When discussing industrial network security, customers are concerned with how to keep the environment safe and operational.

Historically, industrial control systems were seen as isolated from the outside world and utilized proprietary technologies and communications. Security was seen as more of a security-by-obscurity approach. Security outside of physical security wasn't a primary concern. With the modernization of control systems moving towards Consumer-Off-the-Shelf (COTS) products leveraging standardized protocols and connecting to public networks, the process domain now, more than ever, depends on a security framework and architecture. By using more IT-centric products and technologies, and providing connectivity to the enterprise and outside world, new cyber-attacks from both inside and outside the operational environment can potentially occur.

Security incidents can be categorized as either malicious or accidental:


Design Guide


• Malicious acts are deliberate attempts to impact a service or cause malfunction or harm. An example is a disgruntled employee planning to intentionally affect a process by loading a virus onto a server used within the operational control domain or taking control of a process by spoofing a Human Machine Interface (HMI).

• Accidental incidents are probably more prevalent in these environments. Someone may accidentally incorrectly configure a command on a piece of networking equipment, or connect a network cable to an incorrect port. These may be accidental, such as human error, but could be malicious as well while compromising the safety of people, processes, and the environment.

It is recommended to follow an architectural approach to securing the control system and process domain. Recommended models would be the Purdue Model of Control Hierarchy, International Society of Automation 95 (ISA95) and ISA99/IEC 62443. To help adhere to the requirements of IEC 62443 and achieve a robust solution for security and compliance, it is essential to use an end-to-end approach with technologies designed to operate together, while minimizing risk and operational complexities, as shown in Figure 1-9.

Figure 1-9 IEC Foundational Requirements

Figure 1-9 highlights the seven Foundational Requirements (FRs) defined in the ISA-62443 series of documentation:

• Identification, Authentication & Control (IAC) (ISA-62443-3-3 FR 1)—Identify and authenticate all users (humans, software processes and devices) before allowing them to access to the control system.

• Use Control (UC) (ISA-62443-3-3 FR 2)—Enforce the assigned privileges of an authenticated user to perform the requested action on the IACS and monitor the use of these privileges.

• System Integrity (SI) (ISA-62443-3-3 FR 3)—Ensure the integrity of the IACS to prevent unauthorized manipulation.

• Data Confidentiality (DC) (ISA-62443-3-3 FR 4)—Ensure the confidentiality of information on communication channels and in data repositories to prevent unauthorized disclosure.

• Restricted Data Flow (RDF) (ISA-62443-3-3 FR 5)—Segment the control system via zones and conduits to limit the unnecessary flow of the data

• Timely Response to Events (TRE) (ISA-62443-3-3 FR 6)—Respond to security violations by notifying the proper authority, reporting needed evidence of the violation and taking timely corrective action when incidents are discovered.

Defense in depth Detec�on in depth

Data Confiden�ality

Restricted Data Flow

Timely Response to Events

Resource Availability

Iden�fica�on & Authen�ca�on

Control

Use Control System Integrity IEC 62443-3-3 Founda�onal Requirements

ICS

“It’s more than just a bunch of boxes, it’s solu�ons that work together”

Cisco Solu�ons

Network security

Content security

Access security

37

65

04


Design Guide


• Resource Availability (RA) (ISA-62443-3-3 FR 7)—Ensure the availability of the control system against the degradation or denial of essential services.

Defense in Depth is a common term that denotes the multiple layers required to incorporate a security framework. Security isn't just about the technologies to prevent incidents, but also incorporates people, processes, training, and continual assessment. In addition to Defense in Depth, it is also essential to Detect in Depth to ensure malicious activity can be discovered by more than one medium.

Security is not a one-off incident and response; it must be treated in a life cycle manner. This involves everything from awareness through to response. Figure 1-10 highlights the security life cycle approach in the Cisco Risk Control Framework. These concepts are built into the proposed architecture.

Figure 1-10 Cisco Risk Control Framework

ManagementManagement and diagnostics involves tools, applications, and devices used to monitor and maintain the Control Center architecture. Although a typical pipeline network does not drastically change after deployment, the network needs to be maintained and managed. Historically, these functions have not been incorporated into the automation and control systems, but this paradigm is changing because infrequent updates are more prevalent in today’s pipeline.

Business practices and levels of expertise may dictate the management practice and roles and responsibilities of the system. Companies are merging the boundaries between Information Technology (IT) and Operational Technology (OT) where these have historically remained separate domains. Staff may have differing levels of knowledge to support the applications and infrastructure of the Control Center environment. As an example, if more IT-aware personnel are maintaining the Control Center environment, they must be aware of the business requirements that dictate the security and availability aspects of the operating environment.

Pipeline architectures have differing availability requirements (24 hour per day continuous operations), Service Level Agreements (SLA), and processes compared to the typical enterprise domain. If personnel managing the system are more operationally focused, then it is important to understand the networking technologies and tools used to administer the system. It is therefore critical that staff training that is in conjunction with validated and documented processes is established so that any misconfiguration incidents are restricted.

The management systems follow the Fault, Configuration, Accounting, Performance and Security (FCAPS) model to help provide and maintain the availability and security of the Control Center architecture. FCAPS management all help to facilitate and monitor the health of the system.

37

65

05


Design Guide


• Fault Management—Detect, isolate, and notify administrators of any issues or faults within the Control Center to help aid with a timely fault resolution.

• Configuration Management—Configure, manage, and maintain the infrastructure and applications for the Control Center. This includes inventory management, software management (upgrades and patch management), configuration management (backups, archiving, and comparisons), and well documented, validated procedures to support change and replacement management.

• Accounting Management—Account management is usually defined for billed networks in order to provide accounting information. Within the context of this system, the A will stand for accounting as it relates to user access and permissions to provide Role-Based Access and Control (RBAC).

• Performance Management—Collect, analyze, and report the application and infrastructure performance of the Control Center and report on any thresholds that may be exceeded.

• Security Management—Manage, monitor, log, and control the security of the infrastructure to identify, defend, and prevent any security threats or breaches.

Although not explicitly defined in each of these areas, logging and audit trails need to be maintained to understand all aspects of FCAPS for the Pipeline Control Center.

Baseline Integrated SCADA System The design guide and validation effort follows Schneider Electric's best practice deployment model for achieving a holistic integrated solution for the Control Center SCADA.

Baseline Integrated SCADA System (BLISS) combines Schneider Electric's Enterprise OASyS SCADA, and its advanced applications with Cisco compute and networking, to provide a fully integrated holistic PMS for the Control Center. BLISS offers a validated solution to provide fast, secure, and efficient deployment either as part of a pipeline Greenfield deployment or as an upgrade to an existing pipeline.


Design Guide

ContDesign Guide

C H A P T E R 2
Connected Pipeline Architecture
This chapter includes the following major topic:

• Control Center Overview, page 2-2

The Connected Pipeline System delivers a forward-looking, flexible, and modular architecture that enables customers to build the components into an existing system or for a Greenfield deployment. Throughout the architecture, high availability and security are key deliverables. The end-to-end infrastructure provides the following:

• High Availability—Redundancy and reliability mechanisms at the physical, data, and network layer, including robust differentiated QoS and device level redundancy

• Multi-Service Support—Operational and non-operational applications coexisting on a communications network, with mechanisms to ensure the right applications operate in the right way at the right time

• Multi-Level Security—Protect against both physical and cyber-attacks, and non-intentional security threats

• Integrated Management—Network, security, and administration management, from the instrumentation or sensor to the Control Center application

• Open Standards—Based on IP, with the ability to transparently integrate and transport traditional or older serial protocols, and ensure interoperability between current and future applications

The Operational Telecom Network (OTN) and pipeline station architecture are outside the scope of this document, although a high level description of the architecture is included. This document will prioritize the virtualization and operations of the OASyS DNA SCADA software and supporting applications into an operational Control Center architecture based on BLISS.


Chapter 2 Connected Pipeline Architecture Control Center Overview

Figure 2-1 Cisco/Schneider Electric Connected Pipeline Reference Architecture

The architecture is based on a three-tier building block approach as defined in the joint Cisco Schneider Electric reference architecture (see Figure 2-1).

• Control Center—Virtualized, geographically separate, redundant Control Centers

• Operational Telecoms Network—End-to-end communication from field device to Control Center application, for operational and multi-service applications

• Pipeline Stations—Local area networks inside the stations for operational and multi-service applications

Control Center OverviewThe SCADA system provides operators who are located in a control room the ability to monitor, operate, and control the operations along the pipeline. The SCADA system consists of multiple environments within the architecture.

Schneider Electric Pipeline Management SolutionsSchneider Electric's Enterprise Pipeline Management System (EPLMS) consists of multiple services and applications to facilitate safe and efficient operations, as shown in Figure 2-2:

L1 Basic

Control

L2 Supervisory

Control

L3 Opera�onal

Control

L3.5 Industrial

DMZ

L4-5 Office & External

Domain

L2.5 Protec�on

IEC62443 ISA99

Horizontal Inter-Zone, Intra-Zone, Inter-System Security

Process Control Power Safety Systems


Mul�service

Sta�on WAN, Aggrega�on & Security

Process Domain

Metering / PIG / Terminal Sta�on M

eter

ing

PIG

Syst

ems

Gas

Qua

lity

Mul�service Sta�on WANAggrega�on & Security

Process Domain

SCADA & Opera�onal Business Systems

Engineer Worksta�ons

Applica�on Servers

Domain Controller

Instrumenta�on / Sensors Instrumenta�on Instrumenta�on Instrumenta�on

Quantum Quantum

MiCom c264

SIL3 Controller

SIL3 Controller

GTW RI/O GTW RI/O

Historian Operator Sta�on Historian PACIS

Operator Historian Operator Sta�on

Wireless

Mobile Worker

IP Voice

Access Control

CCTV

RFID

Controller Controller Controller

Historian Historian Historian

SCADA Primary

Leak Detec�on

Physical Security

Operator Worksta�ons

SCADA Backup

Historian Repor�ng

Metering Systems

Main Control Center

Video Opera�ons

Access Opera�ons

Video Storage

Incident Response

Engineering

(virt

uali

-virt

ualiz

ed)

Mul�service Process Domain


Quantum

Instrumenta�on

CeCCe Internet Edge 3rd Party Support

Voice

WLAN Controller

Call Manager

PAGA

Magelis

ION Metering

SEPAM Protec�on

TeSys T Motor Mgt

Al�var Drive

MiCOM Feeder

Protec�on

Magelis

RI/O

ScadaPack

SIL3 Op�on No SIL Op�on

Wireless op�on

Crew Welfare / Infotainment

Domain Controller

Engineering

Leak Detec�on

Database SCADA

Real-�me

SCADA Historical

Leak Detec�on

Applica�on Test

Real-�me SCADA Zone Development Test

Decision Support

Remote Access

Domain Controllers

(Indu

stria

l DM

Z)

Backup Control Center

Mobile Worker

IP Voice

Access Control

CCTV

RFID

WAN Connec�on & Security WAN Connec�on & Security

Mobile Worker

IP Voice

Access Control

CCTV

RFID


,

WAN Connec�on & Security

Phys

ical

Sec

urity

SCAD

A &

Bus

ines

s Sys

tem

s

Voic

e &

PAG

A

Wire

less

Deci

sion

Sup

port

& ID

MZ

Internet

WANNetworks

3G/LTE, WiMax, 900Mhz RF Mesh, Satellite, MicrowaveConverged Opera�onal Field Telecoms WirelessConverged Opera�onal Field Telecoms Wired

DWDM, Ethernet, IP/MPLS, MPLS - TP

37

65

10


Design Guide


Figure 2-2 Schneider Electrics Pipeline Management Systems

• RealTime SCADA—Schneider Electric's OASyS DNA transcends the traditional SCADA environment by incorporating the workflow needs of customers in real-time. OASyS DNA is an infrastructure product that adapts to the diverse and changing needs of an enterprise. From the field to the enterprise, OASyS DNA allows access to operational and historical data securely at anytime from anywhere.

• Oil and Gas Application Suite—Schneider Electric's RealTime Oil and Gas Suite works with the proven Schneider Electric OASyS DNA SCADA system to centralize delivery of key oil and gas pipeline information, enhancing a company's operational environment. Critical data is received for improving pipeline operations and meeting business goals. Schneider Electric offers up-to-the-minute metering and flow totaling; and calculates and monitors line pack, tank storage, hydraulic profiles, and compressor and pump performance in real-time.

• Leak Detection—The main strength of Schneider Electric's SimSuite Pipeline lies in its ability to accurately model the pipeline more completely than other available solutions. The leak-detection application uses a combination of methods to detect and locate leaks. Leaks can occur anywhere on the pipeline; they can vary in size; and they can be caused by fatigue, corrosion, equipment failure, or theft. Large and small leaks can be detected using multiple mass-balance calculations. Pressure-drop calculations can be used to locate the leak.

• Measurement Data—The Schneider Electric Measurement Advisor, empowered with Schneider Electric's advanced measurement user interface, provides the efficient and accurate means to configure devices and collect, validate, modify, and reconcile oil and gas measurement data. Part of the Schneider Electric suite of oil and gas solutions, Schneider Electric Measurement Advisor is the high-mileage solution that gathers measurements for multiple pipelines that interface with various Ethernet in the First Mile (EFM) polling engines, SCADA systems, chart integrators, third-parties, and manual input. Schneider Electric Measurement Advisor allows the precision required at every step to achieve process-wide accuracy.

Production EnvironmentThe Production Environment enables real-time operations of the pipeline network. It consists of real-time servers, operator workstations, historical servers, support servers, leak detection servers, and any ancillary services such as domain controllers. The following is a brief description of the server and its function within the production environment:

• Real Time Servers—Real-Time Service shall provide the monitoring and control function for the SCADA system.

• Domain Controller—Dedicated for SCADA, areas of responsibility, and user roles

37

65

07


Design Guide


• Logging Server—OASyS Logs, data playback, and windows events

• Deployment Server—Display and database commissioning system

• Historical Servers—Provide long-term storage of Real-Time measurement, event, alarming and other relevant data monitored or generated by the SCADA system

• Operator Work Stations—Interface for operating, controlling, and monitoring the pipeline

• Leak Detection Server

Test and DevelopmentThis section describes the test and development system.

Test System

Test systems are a non-production replica of the Production SCADA system. This provides an environment for the following functions:

• Non-production replica of the Production SCADA system

• Operating system patch, application change validation prior to production without affecting the Production SCADA system

Development System

The development system, which provides at least one development server per OASyS DNA system, has the following functions:

• Baseline application binary deployment

• Source code repository for customer developed programs, reports and scripts

• Database change control and auditing

• Storage and editing of baseline and customer-specific displays

Decision Support System/Industrial Demilitarized ZoneThe Decision Support System (DSS) isolates the operational system from any external systems or users. It is updated with real-time and historical data from the live system to provide a secure method of providing real-time data to external users. With firewalls creating an Industrial Demilitarized Zone (IDMZ), secure services are enabled to provide the external environments exposure to the data. The following describes the servers within the DSS and their functions:

• IDMZ Domain Controller

• DSS Historical Server

• Remote Access Server

The IDMZ will facilitate extra services to assist with the day-to-day operations in maintaining the integrity of the system. Such functions as patch management servers, secure FTP servers, and anti-virus servers will be housed within the IDMZ environment.


Design Guide


Control Center Availability and SecurityThis section describes control center availability and security.

Availability

Within a SCADA system design, it is recommended that no SPOF should exist. To provide the level of availability required, multiple layers of resiliency are built into the Control Center and communications architecture so that a failure of a single component does not disable the ability to control the pipeline. Redundancy is provided for all critical components and functions of the SCADA system. The Control Center will ensure availability of the IACS system against degradation or denial of essential services. High availability is recommended for the network, server, storage and application components. Depending on the risk assessment, customer philosophy, and cost of infrastructure, the chosen architecture may or may not adhere to all of the following guidelines recommended within a Control Center:

• Dual servers (physical or virtual) for all critical components

• Dual application instances in a hot/standby state machine configuration

• Dual networking platforms, including routers, switches, firewalls, and Fabric Interconnects

• Storage redundancy incorporating Redundant Array Of Independent Disks (RAID), controller, and chassis redundancy where appropriate

• Dual Storage Area Networks (SANs)

• Multiple data communication paths across the network

• Enhanced resiliency technologies provide fast failover from active to standby components

• Constant update of information between hot/standby application instances

• QoS to provide prioritization of key network traffic and data

• Process and procedures to guide quick replacement of failed devices in situations where redundancy may have not been provided

Because of the critical nature of supporting a pipeline infrastructure, at least one Backup Control Center is recommended, but situations may dictate more or less. This backup provides redundancy for events that may take down a primary Control Center. Communications may be lost with the remote pipeline or a natural disaster may occur that renders a Control Center unavailable to the pipeline stations. This Backup Control Center should be geographically separate from the Primary Control Center. The real-time and historical data stored on the components located at the backup system are kept synchronized with the primary system located in the Primary Control Center, at all times. In the event of a failure, the backup system is capable of running the entire operations for the pipeline.

Security

After availability, the integrity and confidentiality of the control system are the next most important. Integrity protects data and systems from intentional or accidental alteration. Confidentiality helps ensure that data cannot be accessed by unauthorized users. The following key security concepts, which are described in Chapter 3, “Connected Pipeline System Design,” will help provide availability of the system and support the networking infrastructure:

• Data Confidentiality and Privacy. Ensure the confidentially of information on the communications network and in storage. This may include methods such as segmentation, protecting against unauthorized access, and encryption of the data.


Design Guide


• Provide Network Segmentation and Isolation Techniques using Zones and Conduits. Within the Control Center, isolate each of the environments into zones that share a common set of security requirements or functions. Security issues seen in one zone need to be restricted from affecting or cross-pollinating into other zones. The security zones can be segmented using physical separation such as specific devices and interfaces or logically such as with Virtual Local Area Networks (VLANs).

• Restricted communications between zones using Firewalls, Access Lists, and Intrusion Protection System (IPS) devices where available. VLANs restrict traffic within a Layer 2 networking instance and the Layer 3 gateway can be used to apply policy to allow or restrict traffic between security zones.

• Threat detection and mitigation using the Firewalls, IPS devices, and segmentation.

• Access control to restrict access to resources to which a user or device is not authorized.

• Manage, monitor, log, and control the security of the infrastructure to identify, defend, and prevent any security threats or breaches.

IDMZ/DSS to isolate the operational system from any external systems or users and provide authorized remote access to the process control domain from external sources.

Figure 2-3 highlights the zones and conduits seen within the BLISS Control Center architecture. Each of the colors depicts an environment, its zone, and conduit. Details of the specific implementation are described in Chapter 3, “Connected Pipeline System Design.”

Figure 2-3 BLISS Zones and Conduits

Multi-Service Traffic (Non-Operational Applications)Multiple services will be deployed to support pipeline communications, physical security, and business enabling applications within the pipeline architecture. Segmentation of the multi-service applications from the operational communications is a common requirement. Regulatory demands, security concerns, and confidence of the operator to house the multi-service on the same infrastructure as the SCADA servers will drive the Control Center multi-service architecture. The two deployment models for enabling multiple services within the Control Center architectures for pipelines are as follows:

37

65

02

WAN to Pipeline

SCADA A VPN

Multiservice VPN

Production

Test


• Remote Access

SCADA B VPN

Multiservice



Training

Development

RealTime (Core)

Non-RealTime (Core)



Design Guide


• The first will house the multiple services on a separate infrastructure from the SCADA environment, as depicted in Figure 2-4.

• The second will house the multiple services on the same physical infrastructure as the SCADA environment, as depicted in Figure 2-5.

Figure 2-4 Segmented Multi-Service from SCADA Option 1

Figure 2-5 Segmented Multi-Service from SCADA Option 2

This version of the architecture will support Option 1 where multiple services will be passed to a totally separate infrastructure, outside of the operational environment.

Operational Communications Network OverviewThe Operational Telecoms architecture provides a multi-service environment that encompasses operational services (such as SCADA and process applications) and non-operational services (such as CCTV and Voice) that enable business efficiency and security along the pipeline. See Figure 2-6.

37

65

08

WAN to Pipeline

SCADA A VPN

Multiservice VPN

Production IDMZ• Decision Support

• Remote Access

SCADA B VPN

Multiservice



RealTime (Core)

37

65

09

WAN to Pipeline

SCADA A VPN

Multiservice VPN

Production IDMZ• Decision Support

• Remote Access

SCADA B VPN

Multiservice

Non Operational Domain


RealTime (Core)


Design Guide


Figure 2-6 End-to-End Communications Architecture Overview

Connectivity is required for communications between Control Centers, between the Control Centers and the pipeline, and for any inter-station communication along the pipeline. The key requirements of availability and security for the services, as mentioned in the Control Center overview are primary considerations for the architecture. Multiple secure paths through the network are provisioned to support this communication. Services are segmented, isolated, and prioritized so that SCADA networks and multi-service traffic will not affect each other under normal operations, security incidents, or network congestion.

The communications network can be built using various networking technologies. Factors that influence the communications architecture include power and space availability at the various sites, capital and operational costs, and the customer's preferred technology. Figure 2-7 provides an overview of the various technologies that may contribute to a pipeline communications architecture key requirements.

L1 Basic

Control

L2 Supervisory

Control

L3 Opera�onal

Control

L3.5 Industrial

DMZ

L4-5 Office & External

Domain

L2.5 Protec�on

IEC62443 ISA99

Horizontal Inter-Zone, Intra-Zone, Inter-System Security



Mul�service


Process Domain

Metering / PIG / Terminal Sta�on

Met

erin

g

PIG

Sys

tem

s

Gas

Qua

lity

Mul�service Sta�on WANAggrega�on & Security

Process Domain

SCADA & Opera�onal Business Systems

Engineer Worksta�ons

Applica�on Servers

Domain Controller

Instrumenta�on / Sensors Instrumenta�on Instrumenta�on Instrumenta�on

Quantum Quantum

MiCom c264

SIL3 Controller

SIL3 Controller

GTW RI/O GTW RI/O



Wireless

Mobile Worker

IP Voice

Access Control

CCTV

RFID

Controller Controller Controller

Historian Historian Historian

SCADA Primary

Leak Detec�on

Physical Security

Operator Worksta�ons

SCADA Backup

Historian Repor�ng

Metering Systems

Main Control Center

Video Opera�ons

Access Opera�ons

Video Storage

Incident Response

Engineering

(virt

uali

-virt

ualiz

ed)

Mul�service Process Domain


Quantum

Instrumenta�on

CeCCe Internet Edge 3rd Party Support

Voice

WLAN Controller

Call Manager

PAGA

Magelis

ION Metering

SEPAM Protec�on

TeSys T Motor Mgt

Al�var Drive

MiCOM Feeder

Protec�on

Magelis

RI/O

ScadaPack

SIL3 Op�on No SIL Op�on

Wireless op�on

Crew Welfare / Infotainment

Domain Controller

Engineering

Leak Detec�on

Database SCADA

Real-�me

SCADA Historical

Leak Detec�on

Applica�on Test

Real-�me SCADA Zone Development Test

Decision Support

Remote Access

Domain Controllers

(Indu

stria

l DM

Z)


Mobile Worker

IP Voice

Access Control

CCTV

RFID

WAN Connec�on & Security WAN Connec�on & Security

Mobile Worker

IP Voice

Access Control

CCTV

RFID


,


Phys

ical

Sec

urity

SCAD

A &

Bus

ines

s Sys

tem

s

Voic

e &

PAG

A

Wire

less

Dec

isio

n Su

ppor

t & ID

MZ

Internet

WANNetworks

3G/LTE, WiMax, 900Mhz RF Mesh, Satellite, MicrowaveConverged Opera�onal Field Telecoms WirelessConverged Opera�onal Field Telecoms Wired

DWDM, Ethernet, IP/MPLS, MPLS - TP

37

65

10


Design Guide


Figure 2-7 Pipeline Communications Architecture Technology Overview

The colors in Figure 2-7 signify the following:

• Green—Meets the requirements shown in column headings

• Amber—Meets requirements, but limitations exist

• Red—Does not meet requirements

As detailed earlier, various types of stations will run along the pipeline. A pipeline segment will be bookended with larger stations such as a main terminal or a pumping/compressor station. In between these larger facilities intermediate stations, such as metering stations and block valve stations, will exist. Regardless of the roles, the stations still have very similar requirements, with environmental or physical factors affecting the architecture within a station. Power may be limited at some of the stations, especially those unmanned and in remote locations such as the block valves.

Services at the stations include:

• Process control for the transportation of oil along the pipeline

• Intelligent power and motor control to manage and control the energy supply at larger stations

• Voice over IP (VoIP) for traditional enterprise voice services and a PAGA system

• Video surveillance as a component of a perimeter security system

N/A

Technology Bandwidth Latency Distance Reliability<50ms Re-

convergenceQoS

Skillsets for

Deploy/Operate

Multiservice

Support

Ethernet

MPLS

DWDM

3G

LTE

Satellite

WiMax

37

65

33


Design Guide


Figure 2-8 Compressor/Pump Station

These services lend themselves to providing a multi-service and operational infrastructure within the station environment. Security and availability are key considerations again at the station level. Segmentation of services, as previously defined, will be built into the architecture. Security policies at the network edge will be implemented that protect against unauthorized access, which is particularly important where the stations are unmanned and prevent cross-pollination of traffic between security zones.

QoS classifications, markings, and policies will provide the prioritization of resources for the services at the station and for traffic traversing the operational network to other stations or for Control Center communication.

Use Case: SCADA Real-time /Near Real-time OperationsThis section describes SCADA-to-Pipeline and Control Center-to-Control Center communications.

Figure 2-9 depicts SCADA real-time/near real-time operations.



Mul�service


Process Domain

Instrumenta�on / Sensors Instrumenta�on Instrumenta�on

Quantum Quantum

MiCom c264

SIL3 Controller

SIL3 Controller

GTW RI/O GTW RI/O



Wireless

Mobile Worker

IP Voice

Access Control

CCTV

RFID

Magelis

ION Metering

SEPAM Protec�on

TeSys T Motor Mgt

Al�var Drive

MiCOM Feeder

Protec�on

RI/O Crew Welfare / Infotainment


37

65

11


Design Guide


Figure 2-9 SCADA Real-time/Near Real-time Operations

SCADA to Pipeline

Operators in the control room rely on consistent reliable data from the pipeline in real-time to make critical decisions for the transport of the product. The control system must be able to poll, collect, store, and display information from Intelligent Electronic Devices (IEDs), including Remote Terminal Units (RTUs), Programmable Logic Controllers (PLCs), substation controllers, and measurement instruments. Real-time communications between the pipeline and the control system must be available, reliable and secure.

The operational SCADA traffic is segmented from any other traffic that may be traversing the network and should be prioritized over non-operational less critical traffic. This segmentation must be carried through the architecture from the instrumentation reporting data within the pipeline stations into the Control Center. Prioritization within the SCADA system is also a requirement. In an application such as leak detection, information must be relayed to a decision point within the system as a priority above all other SCADA traffic.

Unsolicited traffic is sent at irregular intervals as events occurs from the RTU/Controllers to the Active Control Center. Solicited traffic is originated from the Control Center, either through an application or initiated by an operator.

Control Center-to-Control Center Communications

A backup Control Center provides disaster recovery within the system. The real-time and historical data stored on the components located in the backup system are kept synchronized with the primary system located in the primary Control Center at all times. Data is continuously updated and availability of communications is a priority. A variety of communication takes place between OASyS DNA Control


Primary

SCADA

Backup


Training

Main Control Center

RealTime(Core)ProductionSCADA

Primary

SCADA

Backup


RTU/Controller

Application

Server

Instrumentation

Pressure, Temperature,

Differential Pressure

Ope

ra�o

nal W

AN

Cont

rol C

ente

r St

a�on

Meter Station

RTU/Controller

Application

Server

Instrumentation



Block Valve Station

RTU/Controller

Application

Server

Instrumentation



Meter Station


Polling unsolicited

Polling standard

RTU to instrument

RTU to server

Server to instrument

Backup polling standard

CC to CC replication

37

65

13


Design Guide


Centers. For example, the OASyS RealTime Servers pass configuration information and real-time SCADA data. Control Centers will also have one or more domain controllers that need to communicate Active Directory information to each other.

The system provides secure available communications for these components between the Control Centers. Each OASyS DNA system within an installation must have a properly configured firewall or router acting as a boundary between the control network and any external LAN or WAN as per IEC 62443-3-3, Foundational Requirements, SR 5.2, Zone Boundary Protection. This configuration restricts communications to only those ports required for domain controller communications and OASyS DNA functionality. Firewall access controls must be configured to control traffic bi-directionally.


Design Guide

ContDesign Guide

C H A P T E R 3
Connected Pipeline System Design

• BLISS Design, page 3-1

• Security Design, page 3-3

• Availability, page 3-17

BLISS DesignFigure 3-1 provides an overview of the SCADA system physical architecture. The BLISS system follows a single layer data center design. The Control Center is built around the Cisco Unified Compute System (UCS), Nexus 3524 Ethernet-collapsed Aggregation/Access Networking Platform, Cisco Adaptive Security Appliance (ASA), and the EMC VNX series storage arrays.

Figure 3-1 SCADA System Physical Architecture Overview

Inside, Outside, DMZ(DSS)

BLISS BaseLine Integrated

SCADA System

Enterprise

Fiber Channel

Twinax

Ethernet

Dual ASA 55X5-x Firewalls

Ac�ve/standby

Dual Nexus 3524 vPC Connec�vity with ASA &

UCS

Dual UCS Fabric Interconnects

2 x Direct A�ached Storage arrays to UCS Fabric Interconnects

2 x UCS Chassis with B200 M4

servers

WAN Routers

2x B200 M4 servers per SCADA Zone Each server

housed in a different Chassis

Fabric

Interconnection

37

65

25


Chapter 3 Connected Pipeline System Design BLISS Design

In the context of the Schneider Electric BLISS, the UCS B series servers will house each of the production, DSS, test, development, and training environments using virtualized servers. These services and applications will run on Microsoft Hyper-V virtual-enabled machines. Each physical server will house an environment dedicated to it with a backup version of the server housed on another physical server in another UCS chassis. Two storage arrays will support redundancy and will be directly connected to the UCS system at the 6248 Fabric Interconnects. A pair of Nexus 3524 switches will extend the Layer 2 domain to a pair of ASA firewalls where Layer 3 gateway functionality will enable policy management for any traffic that needs to traverse outside of the Layer 2 domain. The operator workstations will have connectivity into the environment through a pair of Catalyst switches that will have connectivity to the Nexus 3524 switches.

Functional Data Center DesignThe Control Center design, as shown in Figure 3-2, can be broken into four key functional blocks:

• Network—LAN design, Layer 2 and Layer 3 Networking

• Compute—UCS B series, Hypervisor, vSwitch

• Storage—Fiber Channel, SANs (directly attached to Fabric Interconnects)

• Network Security—IDMZ, Decision Support System

Figure 3-2 Functional Data Center Design

Network

The Networking Layer connects to the WAN Provider Edge (PE) router terminating the L3VPN. The Nexus 3524 Layer 2 switch is serving as both an access and aggregation node. Although the ASA is stated to be a security device, it is also serving as a Layer 3 gateway within this architecture to enforce networking policy and provide termination of the Layer 2 domain. The 3850 switches will extend the networking architecture to provide network connectivity for the operator workstations.

Netw

ork

Com

pute

Sto

rage

2x ASR 902

2x Nexus 3524

2x Catalyst 3850

2 x EMC VNX 3200

Storage Array

Directly Attached

Storage to Fabric

Interconnects

2 x UCS 6248UP Fabric

Interconnects

2 x UCS 5108

16 x UCS B200 M4 (8 per chassis)

Security

2 x ASA 55x5-X

37

65

16


Design Guide

Chapter 3 Connected Pipeline System Design Security Design

Compute

The Compute Building block includes several systems. The Unified Compute System features redundant 6248 Fabric Interconnects, UCS 5108 blade chassis, and B series blade servers. The hypervisor in this architecture will be Microsoft Hyper-V. Within this architecture, the Hyper-V vSwitch provides a virtual access switching layer that is applied to the Virtual Machines on the physical servers that can apply QoS, Access Control Lists (ACLs), and enhanced security deeper into the architecture.

Storage

The storage layer includes the 6248 Fabric Interconnects for connectivity to the EMC storage. This is directly connected from the Fabric Interconnects to the storage using Fiber Channel. EMC is used here in this release; however, the solution is storage agnostic.

Security

Finally, the ASA firewalls have the capability to not only provide traditional firewall services such as stateful inspection, access control, and remote access, but with Sourcefire can also provide IPS, Intrusion Detection System (IDS), and anomaly detection. In this phase, IPS and IDS will not be enabled. Only traditional firewall services will be validated.

Security DesignThis section describes security design, which includes segmentation design, compute segmentation, and storage segmentation.

Segmentation DesignIn Oil and Gas Pipeline architectures, the ability to securely restrict and isolate services to preserve the integrity of the traffic is paramount. Intentional and accidental cross-pollination of traffic between untrusted entities must be restricted. Securing a perimeter for a SCADA service can be achieved using isolation techniques at the various levels of the infrastructure. The security zones can be segmented using physical separation, such as specific devices and interfaces, or logically, such as with VLANs. VLANs restrict traffic within a Layer 2 networking instance and an ASA firewall can be used to apply policy to allow or restrict traffic between security zones. With logical association of virtual firewalls, VLANs, and L3VPN instances, physically-separated interfaces of equipment and security policies, perimeters and zones can be defined. These techniques are not just restricted to traditional networking platforms, but also must be followed throughout the architecture into the compute and storage areas. These techniques may be physical or logical: physical in the segmentation of physical equipment or logical where comfort and familiarity with virtualized techniques exists or where it makes sense from a business perspective.

Control Center Segmentation and Zones

To address the requirement to segment services to promote a dedicated infrastructure per service, the Connected Pipeline 1.0 architecture will promote physical and logical path isolation techniques. The BLISS architecture will take ownership of the security and segmentation from the firewalls. Multi-service traffic will be terminated on the WAN routers outside of the BLISS domain, placed onto


Design Guide


a separate interface, and passed to the relevant domain. Figure 3-3 represents the network segmentation for a Control Center. The WAN routers are not incorporated as components of the BLISS architecture. The WAN/Communications network should be agnostic to the BLISS.

Figure 3-3 BLISS Segmentation

Layer 2 Network Segmentation

VLANs will provide the segmentation on a per-zone or system level within the Control Center. The production environment seen here will have its own dedicated VLAN as will the test, development and IDMZ/DSS, training, and DSS. The production environment will incorporate all of the servers required to operate the real-time environment and include the operator workstations.

802.1q trunking and Virtual Port Channel (vPC) will be enabled to extend VLAN segmentation on the shared links between the servers, Nexus 3524, and firewalls. 802.1q trunking and vPC provide a method to allow the VLANs that are logically separated to share a set of physical links and connect to two separate Nexus switches (Figure 3-4, option 1). The data path for communications is still contained within the VLAN and segmentation is still maintained.

37

65

17

WAN to Pipeline

SCADA A VPN

Multiservice VPN

Production

Test


• Remote Access

SCADA B VPN

Multiservice



Training

Development

RealTime (Core)

Non-RealTime (Core)



Design Guide


Figure 3-4 Layer 2 Network Segmentation Options

The Connected Pipeline architecture recognizes that further segmentation above pure logical segmentation may be a business- or risk-associated requirement. Therefore, a second option for the segmentation may include physical segmentation where the business warrants it.

An example is the production environment that has requirements for an extra level of segmentation in the networking architecture. Physical isolation using dedicated non-shared ports may be deployed for this zone. This may also be a requirement for the DSS environment. Therefore, dedicated physical ports from the UCS system through to the 3524 switches and the ASA firewalls could be deployed for the production environment.

802.1q trunking of VLANs will be configured on shared ports for the less essential services. For example, the test, development, and training environments will use this method.

Figure 3-4, Option 2 provides a view of the physical/logical segmentation seen in the design at the networking level. The production servers (Gray) housed on separate UCS chassis hit the Fabric Interconnects via a unified fabric. The VLAN is then extended through the switch and to the ASA firewall on its own dedicated physical cabling and port. The test, training, and development zones are transported using logically separate VLANs, but over a shared physical medium using 802.1q trunking.

When looking to use segmentation with more of a physical rather than logical approach, be aware of the port availability on the infrastructure components, specifically the firewalls.

Layer 3 Network Segmentation

Any inter-VLAN routing will require explicit configuration on the firewall. The firewall is acting as the Layer 3 gateway for the servers and will therefore enforce policies for any inter-zone or inter-system traffic. Access lists, rules, and policies will be configured at the firewall to allow or deny communication

The firewalls will have routing enabled to the WAN-facing routers and will provide access to the Enterprise through an Industrial DMZ.

Summary

• VLANs provide the logical segmentation for the network.

• 802.1q trunks and vPC provide shared infrastructure for the logical segmentation.

• Further segmentation above logical segmentation can be provided with dedicated physical ports as described.

x8 x8

x8x8

Fabric Ports

All Zones Shared, dot1q

37

65

18

x8 x8

x8x8

Fabric Ports

Dedicated Physical ports Production ZoneDot1q VLANs for Test, Development and Training Zones

Production

Domain

Test, Development

DomainProduction

Domain

Test, Development

Domain


Design Guide


• Policy enforcement is at the ASA firewall where the Layer 2 is terminated and the ASA provides the Layer 3 gateway functionality for the Control Center zones.

Compute SegmentationFigure 3-5 provides an overview of the compute and storage setup. Each environment will have its own set of dedicated servers or server. In the production environment, it is recommended to have a pair of physical servers to provide redundancy. To enhance redundancy, each of these physical servers will be housed in a separate B series chassis. The virtual machines in these environments/zones all belong to the same VLAN and no requirement exists to house a virtual machine/server outside of its dedicated VLAN.

Figure 3-5 Compute and Storage Setup Overview

Virtual Switch (vSwitch)

From a networking perspective, each of the machines in a zone has the capability to see each other as they are in a single VLAN instance. A vSwitch is an existence of a switch housed on the hypervisor to provide typical network switching functions for the Virtual Machines. This is very similar to a standard access switch that would house regular physical compute. A Hyper-V virtual switch has the capabilities to tie a VLAN to a Virtual Machine providing segmentation at the hypervisor. If intra-VLAN traffic requires policies, then ACLs can be applied at this layer to enforce policy within a zone.

Virtual Network Interface Cards (vNICS)

In conjunction with a vSwitch, the Cisco Virtual Interface Cards (VICs) have the capability to create virtual adapters or vNICs. These allow multiple vNICs to be created rather than physical interfaces. This allows separation of traffic where logical segmentation of data traffic from the Virtual Machines can be separated from the infrastructure management traffic.

37

65

19

Fiber Channel

Unified Fabric FI

Connection

Ethernet

UCS B series Physical Server

Production Domain Active

• VM Real Time SCADA

• VM Historical Cluster• VM Logging, Eng Server

• VM Domain Control

• VM LDS

• VM RAS


Production Domain Standby




• VM LDS

• VM RAS


Decision Support System Active


• VM Historical Cluster• Vm Domain Control

• VM RAS


Decision Support Standby


• VM Historical Cluster• VM Domain Control

• VM RAS

Boot Storage for

Standby Servers

Boot Storage for

Active Servers

SAN A FC Link x1

to each FI-A

SAN B FC Link to

FI-B

x2

x2

x2

x2

Production/Historical

SQL DBs

Production

DomainDecision

Support Sys

DSS Historical SQL

DBs

x8

x8 x8

x8


Design Guide


Summary

• Each environment will have a set of dedicated servers or server.

• In the Production Environment, it is recommended to have a pair of physical servers to provide redundancy.

• Physical pairs of servers will be spread between two UCS Chassis. Chassis redundancy provides an extra level of comfort over pure fabric redundancy.

• Virtual Machines are connected to a vSwitch where VLAN, QoS, and security policies are applied.

• Segmentation is carried through the architecture from the Virtual Machine through the vSwitch, UCS system, Nexus 3524 to the firewalls using VLANs.

Storage SegmentationThis section describes file system management, fiber channel zoning and VSANs, and LUN masking.

File System Management

Each Virtual Machine will have its own dedicated storage for its running environment housed on the storage array. The Microsoft Hyper-V hypervisor's cluster file system management creates a unique Virtual Hard Disk (VHD) per Virtual Machine, ensuring that multiple Virtual Machines cannot access the same VHD sub-directory within the New Technology File System (NTFS) volume, and thus isolating one tenant's VHD from another.

Fiber Channel Zoning and VSANs

Segmentation is enhanced with the configuration of Fiber Channel zoning and Virtual Storage Area Networks (VSAN) in a Fiber Channel storage network. These technologies allow for the physical SAN fabric to be segmented into smaller logical domains.

The architecture will deploy a pair of VSANs. The VSANs will be tied to fabric paths across the infrastructure. This enables the fiber channel redundancy through the use of a Fabric A and Fabric B, which are physically and logically separate.

This partitioning of fabric services greatly reduces network instability by containing fabric reconfigurations and error conditions within an individual VSAN. The strict traffic segregation provided by VSANs helps ensure that the control and data traffic of a specified VSAN are confined within the VSAN's own domain, increasing SAN security and availability. Fiber Channel zoning is deployed to take security to the next layer by providing access control between devices in a VSAN; for example, in the architecture, it allows control and storage policy of storage communications between a host and a storage controller/chassis.

Depending on the security policy and risk assessment, multiple VSANs can be deployed per SCADA zone. Similar to the networking architecture where the networking traffic had its own physical links, the production environment may require its own set of VSANs tied to separate physical SANs. Care must be taken to accommodate the correct equipment to ensure port availability for this architectural approach. This, though, is outside the scope of this CVD.

Figure 3-6 highlights the design within scope. This is the traditional deployment where a VSAN A will be tied to Fabric A, and the VSAN B will be tied to Fabric B. Both storage arrays will be connected using these VSANs/Fabrics.


Design Guide


Figure 3-6 Fiber Channel Zoning Design

LUN Masking

Logical Unit Number (LUN) masking should be configured as an extra level of security at the storage level over VSANs and FC zoning. LUN masking creates an authorization process that restricts storage LUN access to specific hosts on the shared SAN. The storage controllers usually create LUN masking to enforce access policies between hosts and particular LUNS on the storage system.

Summary

• Dual Storage Arrays, each serving different hosts and physical servers on each UCS Chassis

• Segmentation on the storage system using VSANs, FC Zoning, and LUNs

• Standard VSAN A/B mapping to UCS Fabric A and B

• The strict traffic segregation provided by VSANs helps ensure that the control and data traffic of a specified VSAN are confined within the VSAN's own domain, increasing SAN security and availability

• An option to enhance segmentation with extra VSANs to isolate the production storage network from the non-production environments is outside the scope of this document, but is acknowledged as a consideration

Firewall and Industrial DMZ DesignThis section describes firewall deployment options, IDMZ concepts, IDMZ, and DSS design.

37

65

20

Fiber Channel

Unified Fabric FI

Connection

Ethernet






• VM LDS

• VM RAS






• VM LDS

• VM RAS





• VM RAS





• VM RAS

Boot Storage for

Standby Servers

Boot Storage for

Active Servers

SAN A FC Link x1

to each FI-A

SAN B FC Link to

FI-B

x2

x2

x2

x2


SQL DBs

Production

DomainDecision

Support Sys

DSS Historical SQL

DBs

x8

x8 x8

x8


Design Guide


Overview

The firewalls, as shown in Figure 3-7, provide the following foundational security functions within the pipeline architecture:

• A security boundary between the WAN for connection to the pipeline and the Backup Control Center within the process control network

• Architecture for communicating with the external business and enterprise domain using an IDMZ

• An Intra-Control Center policy between zones and segments as explained in the Network segmentation section earlier

• An IDMZ for operational data that can be accessed by the enterprise, and a secure staging area for patching and anti-virus services

Figure 3-7 Firewall and IDMZ Design

Firewall Deployment Options

The choice of firewall deployment will depend on customer philosophy and user requirements, as detailed earlier. The section highlights three deployment options; this CVD, however, will focus on the consolidated firewall deployment.

Single Consolidated Firewall

The Single Consolidated Firewall provides policy for all communications between zones in the Control Center, including the policies for the DSS/IDMZ, WAN, and any inter-zonal communications between environments. Remote access policies are enabled on these firewalls to provide the access into the process domain. All services are deployed as a single pair of ASA devices, providing a single point of administration.

WAN to Pipeline

RealTime/ Historian Deployment Servers

Production

IDMZDecision Support System

RAS

SCADA VPN

BLISSBaseline Integrated

SCADA System

WAN/Communications Network

Inside

100

DMZ

50

Outside 0

Enterprise 0

Enterprise

Inter CC Replication and SCADA Pipeline traffic

Remote Client Service to RAS

and Secure Remote Access

Replicated RealTime

and Historian tp DSS

for access via RCS

37

65

21


Design Guide


Multi-Context Firewalls

Multi-Context Firewalls can be thought of as virtual instances of firewalls on the same physical device. Each context provides an independent device with its own security policy, interfaces, and administration. Within the Control Center architecture, this would allow differing policies and administration for the DSS/IDMZ from the rest of the Control Center, akin to a totally separate firewall. The limitations to the deployment are detailed below:

• Redundancy/failover supports active/active mode only

• Multiple Context mode does not support the following features:

– RIP

– OSPFv3 (OSPFv2 is supported)

– Multicast routing

– Threat Detection

– Unified Communications

– QoS

The ASA family has supported multi-context firewalls since its initial release; however, no virtualization support for Remote Access has existed with multi-context configured until now. From 9.5.2, multi-context-based virtualization has support for VPN Remote Access connections to the ASA. For more information, please refer to the ASA: Multi-Context Mode Remote-Access (AnyConnect) VPN at the following URL:

• http://www.cisco.com/c/en/us/support/docs/security/asa-5500-x-series-firewalls/200353-ASA-Multi-Context-Mode-Remote-Access-A.html

Note If the supported features are not sufficient to support remote access VPN, a separate set of firewalls would need to be deployed. This may lead the customer to physically separate firewalls if administrative separation is required since multiple physical firewalls would need to be deployed (two sets of firewalls).

Physically Separate Firewalls

Multiple physical firewalls may be deployed. A separate pair of firewalls are deployed for the IDMZ/DSS and remote access, and a second pair of firewalls provide the policy for the remaining zones and WAN communications. This could have differing administration and administrators.

Industrial DMZ Concepts

Services and data need to be exchanged between the untrusted Enterprise network and the trusted Process Control Network (PCN). This exposes near real-time and historical information to the Enterprise and allows for better business decisions. Systems located in the Industrial DMZ bring all the data together for company personnel in a near real-time system, allowing the ability to forecast and change daily operations that will generate the most revenue for the company.

In aligning with standards such as IEC 62443/ISA99 and ISA 95, a requirement exists in the Process Control Domain (PCD) to provide strict policy enforcement between the trusted Levels 1-3 of the PCD and the untrusted Levels 4-5 of the Enterprise/business domain. No direct communications are allowed between the Enterprise and PCD. The IDMZ commonly referred to as Level 3.5 provides a point of access and control for the access and exchange of data between these two entities. The IDMZ


Design Guide


architecture provides termination points for the Enterprise and the process domain and then has various servers, applications, and security policies to broker and police communications between the two domains.

The following are key guidelines and concepts for the IDMZ:

• No direct communications should occur between the Enterprise and the PCD.

• The IDMZ needs to provide secure communications between the Enterprise and the PCD using mirrored or replicated servers and applications.

• The IDMZ provides for remote access services from the external networks into the PCD.

• The IDMZ should a security barrier to prevent unauthorized communications into the PCD and, therefore, create security policies to explicitly allow authorized communications.

IDMZ and DSS Design

Within the context of the Connected Pipeline architecture, the IDMZ houses the DSS, which provides proxy or brokering services for communications between the enterprise and PCD.

A historical and real-time server is located within the IDMZ that interacts with servers within the Level 3 PCN production environment. The information from the PCN is replicated to the servers in the IDMZ so that services such as Remote Client Server can facilitate the passing of real-time and historical information to the IDMZ.

Remote access servers such as a remote desktop gateway are also situated in the IDMZ. This allows for a secure method to communicate with the PCD for remote workers or contractors that may require access. Other services would typically include operating system patch server, anti-virus server, and secure FTP server/gateway: in fact, any service that requires information or interaction with Level 4 and above to assist with the day-to-day operations of the Pipeline Control Center.

Policy

The firewalls are recommended to follow the practice of denying any service unless explicitly configured. All services must be identified and rules put in place to allow the communications through the firewall. This requires that any TCP/UDP port numbers are identified so that this explicit configuration can be created.

Security Levels

The interfaces on the ASA require security levels to define the level of trust associated with that particular interface. The security level is set for values of 0-100 where the higher the value, the more trusted the interface. Implicit communications are allowed between higher to lower security levels. However, it is recommended to explicitly configure policies to overwrite the implicit nature of security levels. Therefore, within the Control Center firewall design, the security levels are really nothing other than an identifier of trust for an interface.

VLAN Segmentation

VLAN segmentation, as explained earlier, is a common component in the security framework to assist with isolating services in the Control Center architecture. This can be extended into the IDMZ architecture too. In creating several VLANs within the IDMZ and DSS environment, the servers can be isolated so that, if compromised, the servers within the VLAN container can be restricted from impacting other servers within the IDMZ.


Design Guide


Remote Access Overview

The business and security practices of a customer may influence the chosen method for remote access. The methods discussed in this section focus on using remote desktop services to access the environment. This is not an exclusive list and the client-based VPN service using Cisco AnyConnect could be used to offer another method to access the IDMZ or process domain.

Schneider Electric Remote Client Service

The Schneider Electric Remote Client Service (RCS) provides a means for secure remote access to the SCADA applications. The client is required to be a member of the control system domain. See Figure 3-8.

Figure 3-8 Schneider Remote Client Service Design

• The client communicates with the RCS server in the IDMZ through ports explicitly configured on the firewall. The corporate network is prevented from seeing any other server.

• For individual users remotely accessing the Decision Support SCADA system applications, user access management is handled through a secured RCS server after establishing a VPN connection to the SCADA system. The client initiates a connection using an Operating System client and is authenticated for application access via an instance of Active Directory.

• The RCS server queries data from the RealTime and Historical services running within the Decision Support/IDMZ on behalf of the user.

• The Historical and RealTime services in the Decision Support/IDMZ, in turn, receive their data from the operational instances of the Historical and RealTime services running in the Production Environment through replication services.

• All communication takes place between the client and the RCS server over a single port with SSL-encrypted TCP/IP.

ENTERPRISE

Produc�on

DMZ Decision Support and RAS

Remote Access Jump Servers, RCS & RDP

Cisco ASA

Leve

ls 1

-3

Leve

l 3.5

Le

vels

4-5

mote Accessmp Servers, RCS &P

Replica�on to DSS Systems

from RealTime and Historical

RCS Service with Authorized Users

User exXOS Client

37

65

22

RealTime/ Historian Produc�on Servers


Design Guide


Clientless VPN Remote Desktop Protocol Plug-in

The ASA has the capabilities to terminate remote VPN connectivity with use of a web browser and SSL encryption. This is termed as a clientless VPN because no forced download or requirement exists for the end user to have a specific client installed for VPN access (Cisco also offers a client-based VPN using Cisco AnyConnect). See Figure 3-9.

Figure 3-9 Clientless VPN RDP Plug-in

• The user, if outside of the corporate network, creates a VPN session to the corporate firewall. (Policy, authentication, and authorization would all have to be worked with the enterprise IT team to allow this connection.)

• The user opens a browser and enters the IP address of the IDMZ Firewall. Once connected, the user is authenticated against the Active Directory in the IDMZ.

• After authentication, the user is presented with a web portal and a preconfigured bookmark based on the user's security profile. In this instance, this is an Remote Desktop Protocol (RDP) bookmark to a server in the IDMZ or could be as seen an RDP session into the Decision Support SCADA system.

• The Remote Desktop session is opened from the firewall to the defined server in the bookmark and the RDP session is now presented back to the user.

• No direct access to the environment exists.

Remote Desktop Gateway Service

An alternative to using the clientless VPN and RDP plug-in is to position a Remote Desktop Gateway in the IDMZ. This server can broker RDP communication between the client outside of the IDMZ to servers in the IDMZ or into the PCD. See Figure 3-10.

Produc�on

ENTERPRISE INTERNET

3rd Party

Remote Worker

Cisco ASA

Leve

ls 1

-3

Leve

l 3.5

Le

vels

4-5

Business Domain

on

Cisi

Woooooooooooooooooooooooooooooooooorrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrkkkkkkkkkkekkkkkekekeekekekekekekekkkkeeekekkkkkkkeeekkkkkkekeeekkkkkkekekkkkkkkkkkkkkkkkkkkkk

innnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnnn

IDMZ Decision Support and RAS

IIINNNTTTEERNET

Outside VPN Session

Clientless VPN Session RDP Session

37

65

23



Design Guide


Figure 3-10 Remote Desktop Gateway

The user, if outside of the corporate network, creates a VPN session to the corporate firewall. (Policy, authentication and authorization would all have to be worked with the enterprise IT team to allow this connection.)

The user opens the RDP client from a computer. The user enters the address of the specific host to which they wish to connect and the address of the gateway server. (The firewall will only allow access to the gateway server in the IDMZ so the direct access to the server would be denied.)

The gateway server authenticates the user and, once authenticated, brokers RDP communication between the client and the server in the PCD or IDMZ.

Inter-Control Center Firewall Considerations

An overview of Control Center-to-Control Center communications can be found in Use Case: SCADA Real-time /Near Real-time Operations, page 2-10. Services for replication of data between the primary and backup Control Center and supporting services such as Active Directory need to be explicitly configured. This configuration restricts communications to only those ports required for Domain Controller communications and OASyS DNA functionality. Firewall Access Controls must be configured to control traffic bi-directionally.

Other services such as remote access may be required between Control Centers and should be configured if warranted.

Pipeline Communication Firewall Considerations

To allow communication between the production environment and the pipeline devices, explicit IP addresses and TCP/UDP port numbers should be identified for the protocols, services, and devices that require communication. Depending on the SCADA protocols used, this could be different for every customer.

Produc�on

DMZ Decision Support and RAS

Remote Access Jump Servers, RCS & RDP

ENTERPRISE INTERNET

3rd Party

Remote Worker

Cisco ASA

Leve

ls 1

-3

Leve

l 3.5

Le

vels

4-5

Business Domain

c�on

Jump Servers, RCS &RRDP

IIIIIINNNNNNTTTTTTEEEEEERNRNRNRNRNRNETETETETETET

Outside VPN Session SSL Session

RDP Session

37

65

24



Design Guide


Intra-Control Center Firewall Considerations

The firewall is the policy point for configuration of any intra-Control Center communication. In keeping with the segmentation policy, any inter-VLAN traffic should be explicitly configured on the firewall if communications is required between zones.

Summary

• Firewall policies for the SCADA Zone, IDMZ/DSS (including Remote Access and Enterprise interaction), and Development and Test environments.

• Traffic and route policies applied at the ASA Firewall to prevent inter-zone traffic, unless explicitly configured.

• Operational Telecoms WAN to SCADA and vice versa: ACLs will be explicitly configured to permit traffic between the WAN and the SCADA zone at the ASA.

• Replication between Control Centers will require specific ports to be opened at the firewalls of both Control Centers. The Schneider Electric Oil and Gas OASyS DNA 7.6 Firewall Requirements document contains the specific ports required between Domains (Zones) and physically-separated locations.

• Using the instances of replicating historical and real-time data, the servers in the SCADA zone will be explicitly permitted to allow traffic to the DSS in the IDMZ.

• RCS port will be open to allow corporate users to access the DSS for the purpose of reading the historical and real-time data.

• Clientless VPN and RDP Gateway services are alternatives for remote access to the RCS from Schneider Electric.

• DSS/IDMZ could be deployed with extra VLAN segmentation within the environment to provide further isolation of servers.

Network Infrastructure SecurityTo provide extra levels of security, basic infrastructure mechanisms exist to help protect the switching infrastructure. The guidelines described in this section should be followed.

Shutdown Unused Ports

Place any ports not being used into a shutdown state. For the purpose of a switch, add the switchport VLAN command with an unused VLAN (not VLAN 1) so that if a port is accidentally activated, it will not affect any deployed VLANs.

Trunk Ports

The following best practices should be practiced or order to prevent such attacks as VLAN hopping.

When configuring trunk ports, turn DTP off and explicitly configure trunking on the infrastructure ports. Use a dedicated VLAN ID on all trunk ports. Do not use VLAN 1 for anything and configure all tagged mode for the native VLAN on trunks. Only add the VLANs that are required on the trunk ports.


Design Guide


DHCP Snooping

If servers or workstations in the architecture are using Dynamic Host Configuration Protocol (DHCP), then DHCP snooping and Dynamic ARP Inspection (DAI) should be considered.

DHCP snooping is a security feature that acts like a firewall between untrusted hosts and trusted DHCP servers. The DHCP snooping feature performs the following activities:

• Validates DHCP messages received from untrusted sources and filters out invalid messages

• Rate-limits DHCP traffic from trusted and untrusted sources

• Builds and maintains the DHCP snooping binding database, which contains information about untrusted hosts with leased IP addresses

• Uses the DHCP snooping binding database to validate subsequent requests from untrusted hosts

Other security features, such as DAI, also use information stored in the DHCP snooping binding database. Refer to the "Configuring DHCP" chapter of the Consolidated Platform Configuration Guide, Cisco IOS XE 3.7E and Later (Catalyst 3850 Switches) at the following URL:

• http://www.cisco.com/c/en/us/td/docs/switches/lan/catalyst3850/software/release/37e/consolidated_guide/b_37e_consolidated_3850_cg/b_37e_consolidated_3850_cg_chapter_01000001.html

Dynamic ARP Inspection (DAI)

DAI is a security feature that validates ARP packets in a network. DAI intercepts, logs, and discards ARP packets with invalid IP-to-MAC address bindings. This capability protects the network from some man-in-the-middle attacks. Please refer to the "Configuring Dynamic ARP Inspection" chapter of the Consolidated Platform Configuration Guide, Cisco IOS XE 3.7E and Later (Catalyst 3850 Switches) at the following URL:


Port Security

Port security limits the amount of MACs on a particular interface. This helps to prevent threats such as MAC attacks. Port security should be enabled on access ports. Port security with dynamically learned and static MAC addresses can be used to restrict a port's ingress traffic by limiting the MAC addresses that are allowed to send traffic into the port. When secure MAC addresses are assigned to a secure port, the port does not forward ingress traffic that has source addresses outside the group of defined addresses. Port security should be enabled on the 3850 for the workstation connectivity. Port security is currently not available on the Nexus 3000 Series.

Details of the port security feature and configuration guidelines can be found in the "Configuring Port-Based Traffic Control" chapter of the Consolidated Platform Configuration Guide, Cisco IOS XE 3.7E and Later (Catalyst 3850 Switches) at the following URL:



Design Guide

http://www.cisco.com/c/en/us/td/docs/switches/datacenter/nexus3000/sw/security/7x/b_Cisco_n3k_Security_Config_7x/b_Cisco_n3k_Security_Config_7x_chapter_01000.html


http://www.cisco.com/c/en/us/td/docs/switches/lan/catalyst3850/software/release/37e/consolidated_guide/b_37e_consolidated_3850_cg/b_37e_consolidated_3850_cg_chapter_01000001.html







Chapter 3 Connected Pipeline System Design Availability

Traffic Storm Control

A traffic storm occurs when packets flood the LAN, creating excessive traffic and degrading network performance. The traffic storm control feature can be used to prevent disruptions on Ethernet interfaces by a broadcast, multicast, or unknown unicast traffic storm.

For guidance and configuration for this feature, refer to “Configuring Traffic Storm Control” of the Cisco Nexus 3000 Series NX-OS Layer 2 Switching Configuration Guide, Release 7.x at the following URL:

• http://www.cisco.com/c/en/us/td/docs/switches/datacenter/nexus3000/sw/layer2/7x/b_Cisco_Nexus_3000_Layer_2_Switching_Config_7x/b_Cisco_Nexus_3000_Layer_2_Switching_Config_7x_chapter_01100.html

For port-based traffic control on the 3850, refer to the “Configuring Port-Based Traffic Control” chapter of the Consolidated Platform Configuration Guide, Cisco IOS XE 3.7E and Later (Catalyst 3850 Switches) at the following URL:


Infrastructure Management

Most of the networking management protocols that are used are insecure (such as SNMP, Telnet, and FTP). The secure variants of these protocols (SSH in place of Telnet, SNMP V3) should be used.

Use OOB management to manage the infrastructure. The preferred method is to run the management infrastructure separately from the main Control Center infrastructure. If this is not possible, use a dedicated VLAN for management.

Protect the user access and privilege levels of all the networking devices. AAA should be enabled. The preferred method would be to use a Terminal Access Controller Access-Control System Plus (TACACS+).

Control Plane Policing (CoPP)

To protect the control plane from being overwhelmed, the Cisco devices have Control Plane Policing (CoPP) features. In the case of the Nexus 3524 switch, the device segregates different packets destined for the control plane into different classes. Once these classes are identified, the Cisco NX-operating system device polices the packets, which ensures that the supervisor module is not overwhelmed.

For guidance for CoPP on a Nexus 3524 switch, refer to the “Configuring Control Plane Policing” chapter in the Cisco Nexus 3000 Series NX-OS Security Configuration Guide, Release 7.x at the following URL:

• http://www.cisco.com/c/en/us/td/docs/switches/datacenter/nexus3000/sw/security/7x/b_Cisco_n3k_Security_Config_7x/b_Cisco_n3k_Security_Config_7x_chapter_01011.html

AvailabilityA foundational requirement for the Control Center implementation is to provide a highly available infrastructure. This section discusses some of the technologies required to implement this requirement. The goal of this section is to provide the technologies that will be validated in the design so that those implementing a Control Center architecture can make informed decisions for the best implementations that will address their business requirements and objectives.


Design Guide

http://www.cisco.com/c/en/us/td/docs/switches/datacenter/nexus3000/sw/layer2/7x/b_Cisco_Nexus_3000_Layer_2_Switching_Config_7x/b_Cisco_Nexus_3000_Layer_2_Switching_Config_7x_chapter_01100.html




Figure 3-11 highlights the key redundancy principles described in Chapter 2, “Connected Pipeline Architecture.”

Figure 3-11 Key Redundancy Principles

• Dual servers for all critical components

• Dual application instances in a hot/standby

• Dual networking platforms, including routers, switches, firewalls, and Fabric Interconnects

• Storage redundancy incorporating RAID, controller, and chassis redundancy where appropriate

• Dual SANs

• Multiple data communication paths across the network

• Using enhanced resiliency technologies provide fast failover from active to standby components

• Constant update of information between hot/standby, mirrored servers, and applications

• QoS to provide prioritization of key network traffic and data

• Process and procedures to guide quick replacement of failed devices in situations where redundancy may have not been provided

Network RedundancyFigure 3-12 depicts network redundancy.

Inside, Outside, DMZ(DSS)

BLISS BaseLine Integrated

SCADA System

Enterprise

Fiber Channel

Twinax

Ethernet

Dual ASA 55X5-x Firewalls

Ac�ve/standby

Dual Nexus 3524 vPC Connec�vity with ASA &

UCS

Dual UCS Fabric Interconnects

2 x Direct A�ached Storage arrays to UCS Fabric Interconnects

2 x UCS Chassis with B200 M4

servers

WAN Routers

2x B200 M4 servers per SCADA Zone Each server

housed in a different Chassis

Fabric

Interconnection

37

65

25


Design Guide


Figure 3-12 Network Redundancy

Port Channel

A port channel bundles individual links into a channel group to create a single logical link that provides the aggregate bandwidth of up to eight physical links. If a member port within a port channel fails, traffic previously carried over the failed link switches to the remaining member ports within the port channel.

Port channels are configured as highlighted in Figure 3-12 at the Firewalls and the Fabric Interconnects.

Virtual Port Channel

vPCs allow multiple links to be used between a port channel-attached device and a pair of participating switches. The two Nexus 3524 switches act as vPC peer endpoints and look like a single logical entity to the port channel-attached devices. vPC should be enabled to provide link redundancy and optimal link bandwidth utilization between the Nexus 3K nodes and the following devices:

• ASA

• 6248 Fabric Interconnects

A vPC provides the following benefits:

• Allows a single device to use a port channel across two upstream devices

• Eliminates Spanning Tree Protocol (STP) blocked ports

• Provides a loop-free topology

• Uses all available uplink bandwidth

• Provides fast convergence if either the link or a switch fails

• Provides link-level resiliency

• Assures high availability

Details for the vPC feature and configuration can be found in "Configuring Virtual Port Channels" in the Cisco Nexus 3548 Switch NX-OS Interfaces Configuration Guide, Release 6.x at the following URL:

• http://www.cisco.com/c/en/us/td/docs/switches/datacenter/nexus3548/sw/interfaces/602_a1_1/b_N3548_Interfaces_Config_602_A1_1/b_N3548_Interfaces_Config_602_A1_1_chapter_0110.html#concept_090B6DECCB594BC7AE8368433FFB3B3B

ASA Firewall (Active)

ASA Firewall ( Standby)

3524 Switch PairvPC Enabled

Dual Fabric InterconnectsPort channel Enabled Uplinks

WAN

ASA Firewalls Port channels Enabled Uplinks

37

65

26


Design Guide

http://www.cisco.com/c/en/us/td/docs/switches/datacenter/nexus3548/sw/interfaces/602_a1_1/b_N3548_Interfaces_Config_602_A1_1/b_N3548_Interfaces_Config_602_A1_1_chapter_0110.html#concept_090B6DECCB594BC7AE8368433FFB3B3B




ASA Redundancy

Two ASA firewalls in the consolidated architecture provide the policy point for all communications in the Control Center. Layer 3 interfaces, as previously mentioned, are enabled on the firewall and act as the Layer 3 gateway to police traffic between all of the zones in the Control Center and between the Control Center and the WAN. The firewalls are deployed as an active/standby pair.

Summary

• Link and platform redundancy should exist at all levels

• Link aggregation and resiliency should be enabled with port channels and vPC

• Firewalls should be in active/standby mode

Compute RedundancyThis section includes information about Fabric redundancy, end host node, and SCADA server connectivity and redundancy.

Fabric Redundancy

A fully redundant Cisco Unified Computing System consists of two independent fabric planes: Fabric A and Fabric B. Each plane consists of a central Fabric Interconnect (Cisco UCS 6200 Series Fabric Interconnects) connected to an I/O module (Fabric Extender) in each blade chassis. The two Fabric Interconnects are completely independent from the perspective of the data plane

All network endpoints, such as HBAs, vNICs, and management entities such as Cisco Integrated Management Controllers (IMCs; formerly referred to as baseboard management controllers, or BMCs), are dual-connected to both fabric planes. See Figure 3-13.

Figure 3-13 Fabric Redundancy

37

65

27

Nexus 3K Ethernet

Switches

Ethernet uplinks

Fabric Interconnect

A

Fabric Interconnect

B

Unified Ports 8x10g

per Fabric

Adapter Adapter

vHBA

vNIC

vNIC

vNIC

B200 Server B200 Server

Fabric Extender Fabric Extender


Design Guide


End Host Mode

In end-host mode, Cisco UCS presents an end host to an external Ethernet network. The external LAN sees the Cisco UCS Fabric Interconnect as an end host with multiple adapters. The uplink ports appear as server ports. This mode is the default and recommended configuration if the upstream device is Layer 2 switching.

End-host mode features include the following:

• STP is not run on both the uplink ports and the server ports.

• MAC address learning occurs only on the server ports; MAC address movement is fully supported.

• Links are active/active regardless of the number of uplink switches.

• The system is highly scalable because the control plane is not occupied.

More details related to fabric redundancy and end host mode can be found in Cisco Unified Computing System Ethernet Switching Modes at the following URL:

• http://www.cisco.com/c/en/us/solutions/collateral/data-center-virtualization/unified-computing/whitepaper_c11-701962.html

SCADA Server Connectivity and Redundancy (Virtual Machines)

Within the production environment, pairs of B series servers, one in each of two chassis, will exist. Each of these servers will be running duplicate Virtual Machines for each service: for example, two Virtual Machines for the real-time service and two Virtual Machines for the historical. These Virtual Machine pairs will be configured to run on separate physical servers that will be located on separate UCS Chassis.

These two virtual servers will be configured with a single vNIC, which is not the norm for this application. Dual NICs and NIC teaming are normally applied. This is not required in this virtual environment where the Hyper-V virtual switch is deployed. The virtual switch will provide a single vNIC to each virtual server. The vSwitch will have redundant connectivity to the UCS Fabric Interconnects through the use of Fabric Failover.

From an addressing standpoint, each server will be configured with an IP within the same address space, but will communicate with the other SCADA components through its virtual IP. Figure 3-14 gives a representation of the design.

Figure 3-14 SCADA Server Connectivity and Redundancy Design

Virtual

IP

10.1.1.1

10.1.1.2 10.1.1.3

VM VM

Hyper-V

Vswitch

Hyper-V

Vswitch

Nexus 3000

6248 Fabric

Interconnects

2 x B200 servers in

Separate Chassis

Redundant VMs

37

65

28


Design Guide

http://www.cisco.com/c/en/us/solutions/collateral/data-center-virtualization/unified-computing/whitepaper_c11-701962.html


Storage RedundancyThe storage redundancy follows the same consistent approach to the other areas of availability within the architecture consisting of hardware and path redundancy. See Figure 3-15.

Figure 3-15 Storage Redundancy Design

Hardware Redundancy

Redundancy is carried throughout the architecture at the hardware level. We will take the example of a production environment in Figure 3-15 to explain the hardware redundancy modeled.

Server Redundancy

Two physical UCS B200 servers will house the hot and standby instances of the production application Virtual Machines. A hot instance will exist in one physical server and its standby instance will exist in a separate physicals server. These two servers are located in different UCS Chassis, as depicted in the figure, promoting chassis redundancy. Both servers are equipped with dual-port adapters.

UCS Fabric Interconnects

Two 6200 series Fabric Interconnects exist in the architecture. These provide the management and communications for the UCS B series and 5108 UCS chassis. The pair of 6200 Fabric Interconnects form a single, highly available management domain. The pair of fabrics interface with the SAN and the Ethernet network and provide active/active access to both the storage and Ethernet networks while acting as an active passive mode for the management of the UCS system.

Storage Chassis and Controllers

Two storage chassis will exist in the architecture. These will be equipped with dual controllers to provide redundancy within a chassis.

37

65

29

Fiber Channel

Unified Fabric FI

Connection

Ethernet






• VM LDS

• VM RAS






• VM LDS

• VM RAS





• VM RAS





• VM RAS

Boot Storage for

Standby Servers

Boot Storage for

Active Servers

SAN A FC Link x1

to each FI-A

SAN B FC Link to

FI-B

x2

x2

x2

x2


SQL DBs

Production

DomainDecision

Support Sys

DSS Historical SQL

DBs

x8

x8 x8

x8


Design Guide


Redundant SAN

Within storage networking, it is a standard best practice to provide two completely separate SAN fabrics, providing two distinct paths between the compute and the storage. Within this design, a minimum of two VSANs will be created on the UCS system that map to the separate fabrics. Each of dual fabrics in this architecture are directly attached to the storage arrays. One fabric will provide connectivity for Fabric A and one for Fabric B. VSANs, as described earlier, extend SAN networking concepts to create logical segmentation on the same physical infrastructure. Figure 3-16 highlights FC port channels. This is the same concept as in Ethernet port channels where multiple physical links are bundles together to provide one logical link.

It is worth noting that both Fabrics and SANs are active and passing traffic between the hosts and storage. This is an active/active, not active/standby, model.

Figure 3-16 Redundant SAN Design

Multipath

Multipath software is loaded on top of an operating system and provides a layer that sits between the operating system and storage adapters. The multipath software closely monitors all physical paths and storage presentations while load-balancing SAN traffic across the available HBAs and fabric paths. This software layer also accomplishes failover should an HBA, path, or upstream device fail.

Summary

• UCS provides storage redundancy starting at the server with dual port adapters per host.

• Two physical servers house the hot and standby instances of the production application Virtual Machines. A hot instance in one physical server and its standby instance will occur in a separate physical server.

• Each server in a server pair is housed in a separate UCS Chassis.

• Connectivity will occur through a redundant fabric to a pair of Fabric Interconnects.

• Two storage chassis will be directly connected to the Fabric Interconnects using a redundant SAN A/B architecture.

• The guest operating systems and storage are pinned to a specific host to ensure physical redundancy and dedicated server resources.

37

65

30

SAN A FC Link x2

FC Port Channel

SAN A FC Link x2

FC Port Channel

SAN B FC Link x2

FC Port Channel

SAN B FC Link x2

FC Port Channel

Fabric A Fabric B

8x 10g 8x 10g

8x 10g8x 10g

Unified Fabric F1

Connection’s

Unified Fabric F1

Connection’s


Design Guide


• A RAID 1 hardware mirroring solution is used to mirror all historical databases onto online disks available to the SCADA system.

• All historical databases on disks are mirrored.

• Two-node clustering is configured for the historical servers' redundancy.

Schneider Electric OASyS Application Design Application RedundancyThis section includes information about Virtual Machine redundancy, SQL SAN storage hardware mirroring, virtual guest storage, Microsoft SQL server-based Historian, and operator workstation connectivity.

Virtual Machine Redundancy

As the required redundancy mechanisms are built into the OASyS software and applications, no requirement exists to configure Virtual Machine redundancy such as Hype-V live migration in the Schneider Electric and Cisco Control Center architecture. The OASyS applications are provided in state machine-aware pairs and replicate required memory segments between the virtual guests. System monitors check critical components for failures and take the least intrusive course of action to recover from any failure

SQL SAN Storage Hardware Mirroring

A RAID 1 hardware mirroring solution is used to mirror all historical databases onto online disks available to the SCADA system. All historical databases on disks are mirrored, so that a failure of a disk does not cause a failure of a particular historical database. When a disk fails, the mirrored disk has the same copy of data as the master disk. This disk makes available historical data for the failed disk to the SCADA system. Once the failed disk is replaced, the SAN software automatically copies the information from the mirror disk back onto the new disk.

Virtual Guest Storage

Similar to historical data storage, the Virtual Hosts and Virtual Guests are stored on a storage array on the SAN. A RAID 5 set of disks is used to boot the host operating system and another RAID 5 group of disks is used to boot the guest operating systems.

Microsoft SQL Server-Based Historian

As the SCADA system historian is not critical to safe operations, a Microsoft Hot/Standby Cluster is used to share SQL-based data between the two OASyS Historical servers at one site. Microsoft SQL Server Cluster provides both automatic and manual failover capability for SQL Server services to another node in case the active node is down. The active node can be down due to an operating system or a hardware failure, in which case the automatic failover to an available node on the same cluster can occur and the users will start using the application through the failover node. SQL Server services can be manually moved to a different node at times when a planned maintenance like an operating system upgrade or patch maintenance is required on the active node. Figure 3-17 illustrates the combined architecture of the OASyS Historian and Microsoft SQL Cluster.


Design Guide


Figure 3-17 OASyS Historian and Microsoft SQL Cluster Combined Architecture

Operator Workstation Connectivity

The operator workstations are connected within the SCADA zone. These stations will be configured with dual NICs and connected to traditional Catalyst 3850 stackable switches. Generally, the workstations will be in a different physical location so it will be required to extend the VLAN from the 3524 Nexus switches to the Catalyst. The NICs on these workstations will operate in an active/standby configuration and each NIC will be connected to a different stackable 3850. See Figure 3-18.

Figure 3-18 Operator Workstation Connectivity

Network ManagementManagement and diagnostics involves tools, applications, and devices used to monitor, configure and maintain the Control Center architecture. Although a typical pipeline network does not drastically change after deployment, the network needs to be maintained and managed. The business practices and levels of expertise, however, may dictate the management practice, roles, and responsibilities of the system. This section describes the platforms, technologies, and practices used.

Cisco UCS Manager

Cisco UCS Manager resides as embedded software on the Cisco UCS Fabric Interconnects, Fabric Extenders, servers, and adapters. No external management server is required, thereby simplifying administration and reducing capital expenses for the management environment. The communication between the manager on the Fabric Interconnect and the subsidiary functions found in the Fabric Extenders, chassis, servers, and adapters are built in and automatic.

37

65

31

Virtual IP Address

Operator Workstation

ACTIVE NODE

Operating

System Drive

Operating

System Drive

PASSIVE NODE

HBA (Host Bus Adapter) HBA (Host Bus Adapter)

SAN (Storage Area Network)

Data Files Log Files Quorum Drives

37

65

32


Design Guide


UCS Manager provides an intuitive GUI, a CLI, and a robust API to manage all system configuration and operations. Cisco UCS Manager enables storage, networking, and server maintenance staff to collaborate easily on defining service profiles for applications while preserving subject matter expertise and roles. Key features include:

• Supports Cisco UCS B-Series blade and C-Series rack servers, UCS Mini, Cisco Composable Infrastructure and Cisco HyperFlex hyperconverged infrastructure

• Unified, embedded, policy-driven management programmatically controls server, network, and storage resources, so that they can be efficiently managed at scale through software

• Works with HTML 5, Java, or CLI GUIs

• Auto Discovery can detect, inventory, manage, and provision system components that are added or changed

• An open XML API facilitates integration with third-party systems management tools

• Role-based administration builds on existing skills and supports collaboration across disciplines. For more information, see “Cisco UCS Manager” at the following URL:

– http://www.cisco.com/c/en/us/products/servers-unified-computing/ucs-manager/index.html

Cisco Adaptive Security Device Manager

Cisco Adaptive Security Device Manager (ASDM) can be accessed directly with a Web browser from any Java plug-in-enabled computer on the network, providing security administrators with rapid and secure access to their Cisco ASA Firewalls. ASDM allows the user to configure, monitor, and troubleshoot Cisco firewall appliances with a user-friendly application. Ideal for small or simple deployments, the Cisco ASDM provides the following:

• Setup wizards for configuring and managing Cisco firewall devices

• Powerful real-time log viewer and monitoring dashboards that provide an at-a-glance view of firewall appliance status and health

• Troubleshooting features and powerful debugging tools such as Packet Trace and Packet Capture

Out of Band Management

A separate Out of Band (OOB) network should be deployed to provide a dedicated infrastructure to support the management traffic. The OOB network segment hosts console servers, network management stations, AAA servers, analysis and correlation tools, NTP, FTP, Syslog servers, network compliance management, and any other management and control services. An OOB management network should be deployed using the following best practices:

• Provide network isolation

• Enforce access control

• Prevent data traffic from transiting the management network

• Enforce secure use of network management traffic (SSH, SNMP v3)

If an OOB network is not viable, then a dedicated VLAN should be used for the management network following the same best practices above and classifying/prioritizing management traffic using QoS.


Design Guide

http://www.cisco.com/c/en/us/products/servers-unified-computing/ucs-manager/index.html


Role-Based Access Control

Role-Based Access Control (RBAC) is a method of restricting or authorizing system access for users based on a user's role. This role consists of privileges that will be assigned to the user. These privileges can be very granular; for example, a user can be restricted from accessing particular commands and strings. This becomes very useful in the Control Center environment, especially in the UCS domain. With the consolidated nature of the UCS system, varying users with differing expertise can access the platform. Using RBAC, we can create server, storage, and network administration privileges and roles. The UCS system can even take this to the next level using Locales. For example, a user with the Server Administrator role for the Test and Development environment could update server configurations in that environment, but could not update server configurations in the Production environment unless the locales assigned to the user include the access privilege.

Summary

• OOB Management should be used where possible. Dedicated VLANs should be used if In-band Management is used.

• Use secure methods to provide extra security for network management traffic. SSH and SNMP v3 are examples.

• Provide visibility of events, faults, and performance of the network to the operators in the Control Center using Syslog and SNMP.

• Enable RBAC for access to the infrastructure components.

Time SynchronizationAll components of the SCADA system architecture must be time synchronized. Time synchronization is critical for event correlation and analysis of events in the pipeline network. Therefore, all infrastructure components must be enabled with Network Time Protocol (NTP) and configured to a central source. A Network Time Server (NTS) is leveraged for Windows time synchronization at each site (for example, one at the Primary Control Center and one at the Secondary Control Center). The NTS can serve accurate time to any system running an NTP or Simple Network Time Protocol (SNTP) client and can support 200,000 network clients (such as workstations, servers, and routers) with an NTP timestamp accuracy of <10 microseconds. Each time the client will require access to the IP address of the network time server.

The NTS derives accurate time from the atomic clocks aboard the GPS satellite system. To obtain the time, it uses the Code Division Multiple Access (CDMA) mobile telecommunications network used by many cellular telephones. This means the antenna can be conveniently located anywhere a cell phone signal is available.


Design Guide

ContDesign Guide

C H A P T E R 4
System Implementation Considerations

• Server Sizing and Storage Sizing, page 4-1

• Scalability Considerations, page 4-1

Server Sizing and Storage SizingTraditionally, SCADA Host software has been the mechanism for viewing graphical displays, alarms, and trends. Control from the SCADA host itself only became available when control elements for remote instruments were developed. These systems, which were isolated from the outside world and were the domain of operators, technicians and engineers, had the responsibility to monitor, maintain, and engineer processes and SCADA elements. With IT advancements, this is no longer the case. Many different stakeholders now take advantage of real-time access to the data that the SCADA host software generates.

Accounting, maintenance management, and material purchasing requirements are performed or partly performed from data derived from the SCADA system. Consequently, the SCADA host is driven to be an enterprise entity providing data to a number of different users and processes. This has encouraged SCADA host software development to adopt standards and mechanisms to support interfacing to these systems. Key to sizing of a SCADA system is the overall quantity of pipeline network assets to be controlled.

• Servers and storage should be considered

• Number of devices, points polled per device, polling interval, and the applications on top of SCADA determine the CPU and memory requirements

• Number of physical domain controllers, virtual hosts, virtual guests, virtual guest backups, and built-in expansion need sizing to determine hard drive requirements

• Microsoft SQL storage requirements for online data and backups

• Memory sizing for performance of all virtual guests

• Number of remote connected users determines resources allocated to the remote access servers

Scalability ConsiderationsInformation retrieval and communications over the network WAN for the SCADA system should not be a major concern; however, how the traffic traverses the Control Center network should be understood. The Primary Production system doesn't just facilitate communications with the pipeline network, but, as


Chapter 4 System Implementation Considerations Scalability Considerations

described, also requires communications with the enterprise and the Secondary Production system. The communications with the backup Control Center, from a bandwidth utilization perspective, can be very different based on the applications and the type of pipeline. For example, 20Mb/s on a gas pipeline is contrasted with 100 to 200 Mbps for replication on a LNG pipeline driven mainly by software applications running on top of OASyS DNA such as the Liquids Management System (LMS). The ASA as the security device is in line of these flows of data. Therefore, the following should be considered:

• ASA sizing

• ASA throughput degrades when enabling IPS/IDS and other computation-intensive features

• The traffic traversing the different points in the network

• Gas and liquid pipelines will have differing replication between the Control Center based on the applications

• The customers' traffic profile

• Number of devices, points polled per device, polling interval, and the applications on top of SCADA may drive replication data between Primary and Secondary Sites

• Data replicated to the IDMZ from the Operational environment

To obtain guidance to ASA firewall scalability when comparing models, see the “Compare Models” chapter in Cisco ASA 5500-X Series Firewalls at the following URL:

• http://www.cisco.com/c/en/us/products/security/asa-5500-series-next-generation-firewalls/models-comparison.html


Design Guide

http://www.cisco.com/c/en/us/products/security/asa-5500-series-next-generation-firewalls/models-comparison.html

ContDesign Guide

C H A P T E R 5
System Components
Table 5-1 details Cisco products used in the Cisco Connected Pipeline system.

Table 5-2 details Schneider Electric products used in the Cisco Connected Pipeline system.

Table 5-1 Cisco Products

Cisco Product Software Release Description

ASA 55x5-X 9.2(3)4 Firewall

Nexus 3524 6.0(2)A4(5) Network Switch

Cisco UCS 2.2(5a) Unified Compute Infrastructure, Fabric Interconnects

B200 M4 2.2(5a) Server

ASR 902 15.5(2)S WAN-facing router

Table 5-2 Schneider Electric Products

Component Role SW Version

Windows Server 2012 R2

MS SQL Server 2012 SP1

Visual Studio 2013 Professional

OASyS DNA Elk SP4 ML 7.7.1

OASyS DNA OGP

• LMS: R4.1.1

• Measurement: R5.6,

• Realtime Gas: R5.2

• Gas Day Operations: R5.4

• OGX: CR2

• Liquids clients:

– LibAPI.Installer.1.0.19

– WebClientInstaller2013.1.0.5

– Liquid.Installer.NET45-1.0.28

7.6

EMC VNXe3200 Storage 3.1.1.5395470


ContDesign Guide

C H A P T E R A
Glossary
Table A-1 lists acronyms and initialisms used in this document.

Table A-1 Acronyms and Initialisms

Term Definition

AAA Authentication, Authorization, and Accounting

ACL Access Control List

ARP Address Resolution Protocol

ASA Cisco Adaptive Security Appliance

ASDM Cisco Adaptive Security Device Manager

BLISS Baseline Integrated SCADA System

CDMA Code Division Multiple Access

CoPP Control Plane Policing

COTS Consumer off the Shelf

CVD Cisco Validated Design

DAI Dynamic ARP Inspection

DHCP Dynamic Host Configuration Protocol

DNA Dynamic Network of Applications

DSS Decision Support System

E&P Exploration and Production

EFM Ethernet in the First Mile

EPLMS Schneider Electric's Enterprise PipeLine Management System

FCAPS Fault, Configuration, Accounting, Performance and Security

FTP File Transfer Protocol

HBA Host Bus Adaptor

HMI Human Machine Interface

IAC Identification, Authentication & Control

ICS Industrial Control System

IDMZ Industrial Demilitarized Zone

A-1rol Center Virtualization for the Connected Pipeline

Chapter A Glossary

IDS Intrusion Detection System

IEC International Electrotechnical Commission

IED Intelligent Electronic Device

IMC Cisco Integrated Management Controller

IoT Internet of Things

IPS Intrusion Protection System

ISA International Society of Automation

LLD Low Level Design

LNG Liquid Natural Gas

LUN Logical Unit Number

NFTS New Technology File System

NGL Natural Gas Liquid

NIST National Institutes of Standards and Technology

NTS Network Time Server

OOB Out of Band

OTN Operational Telecom Network

PAGA Public Address and General Alarm

PCD Process Control Domain

PCN Process Control Network

PHMSA Pipeline and Hazardous Materials Safety Administration

PIG Pipeline Inspection Gauge

PLC Programmable Logic Controller

PMS Pipeline Management System

QoS Quality of Service

RAID Redundant Array of Independent Disks

RBAC Role-Based Access and Control

RCS Schneider Electric Remote Client Service

RDF Restricted Data Flow

RDP Remote Desktop Protocol

RTU Remote Terminal Unit

SAN Storage Area Network

SCADA Supervisory Control and Data Acquisition

SLA Service Level Agreements

SNTP Simple Network Time Protocol

SPOF Single Point of Failure

STP Spanning Tree Protocol

Table A-1 Acronyms and Initialisms (continued)

Term Definition

A-2Control Center Virtualization for the Connected Pipeline

Design Guide

Chapter A Glossary

TACACS+ Terminal Access Controller Access-Control System Plus

TCO Total Cost of Ownership

TRE Timely Response to Events

VHD Virtual Hard Disk

VIC Virtual Interface Card

VLAN Virtual Local Area Network

vNIC Virtual Network Interface Card

VoIP Voice over IP

vPC Virtual Port Channel

VSAN Virtual Storage Area Network

WAN Wide Area Network

Table A-1 Acronyms and Initialisms (continued)

Term Definition

A-3Control Center Virtualization for the Connected Pipeline

Design Guide

Control Center Virtualization for the Cisco Connected ... · PDF fileCisco Connected Pipeline...

Documents

Transcript of Control Center Virtualization for the Cisco Connected ... · PDF fileCisco Connected Pipeline...