REPORT D2.3 SCADA system architectureskth.diva-portal.org/smash/get/diva2:495729/FULLTEXT01.pdf ·...

(Vital Infrastructure, NetworKs, INformation

and Control System ManaGement)

Project co-funded by the European Community within the 7th Framework Programme.

REPORT D2.3 SCADA system architectures

PROJECT TITLE: Vital Infrastructure, Networks, Information and Control Systems Management

PROJECT ACRONYM: VIKING

GRANT AGREEMENT

NUMBER: 225643

PROJECT START DATE: 01.11.2008

DURATION: 36 MONTHS

PROJECT CO-ORDINATOR: GUNNAR BJÖRKMAN, ABB (ABB) DE

PROJECT MEMBERS: ABB (1) (ABB) DE

EIDGENOESSISCHE TECHNISCHE HOCHSCHULE ZÜRICH (2) (ETHZ) CH

E.ON (4) (E.ON) DE

ASTRON INFORMATICS LTD. (5) (Astron) HU

KUNGLIGA TEKNISKA HÖGSKOLAN (6) (KTH) SE

UNIVERSITY OF MARYLAND FOUNDATION (8) (USMF) US

MML ANALYS & STRATEGI (9) (MML) SE

DOCUMENT IDENTIFIER: D2.3

ISSUE: 1.0

ISSUE DATE: 2010-05-19

PREPARED: 2010-05-19

APPROVED TECHNICAL LEAD/KTH

DISSEMINATION STATUS: PUBLIC

Grant Agreement Number 225643 Report number: D2.3 Issue: 1.0

VIKING Consortium

www.vikingproject.eu Page: 2

History Chart

Issue Date Changed Page (s) Cause of Change Implemented by

1.0 2010-05-19 All First version Teodor Sommestad

Gunnar Björkman

Authorization

No. Action Name Signature Date

1 Prepared Teodor Sommestad

Gunnar Björkman

2010-05-19

3 Approved Technical Lead 2010-05-19

The information in this document is subject to change without notice.

All rights reserved.

The document is proprietary of the VIKING consortium members listed on the front page of this document. The document is supplied on the express understanding that it is to be treated as confidential and may not be used or disclosed to others in whole or in part for any purpose except as expressly authorized in terms of Grant Agreement number 225643.

The VIKING consortium makes no warranty for the information contained in this document; neither does it assume any legal liability or responsibility for the accuracy completeness or usefulness of this information.

Company or product names mentioned in this document may be trademarks or registered trademarks of their respective companies.


VIKING Consortium


Distribution List

This document is distributed as below.

Additional copies held by unnamed recipients will not be updated.

Electronic Copy Number Name Address

1 European Commission EU, Brussels

2 Gunnar Björkman ABB AG, Mannheim

3 Pontus Johnson KTH, Stockholm

4 - 9 VIKING consortium members


VIKING Consortium


Table of Contents

1 Preface ...........................................................................9

1.1 Overview of the VIKING project .......................................................... 9

1.2 Purpose of deliverable 2.3 ................................................................ 12

1.3 Scope ........................................................................................... 13

1.4 Document overview......................................................................... 13

2 Method and theoretical framework....................................14

2.1 Literature review ............................................................................ 14

2.1.1 Security literature focusing on SCADA systems ................................... 14

2.1.2 SCADA and control system literature ................................................. 15

2.1.3 Market surveys............................................................................... 15

2.2 Metamodel..................................................................................... 16

2.2.1 Scope and definition........................................................................ 16

2.2.2 Metamodel rationale........................................................................ 17

2.3 Interviews with vendors................................................................... 17

2.3.1 Scope and limitations ...................................................................... 18

2.3.2 Baseline models.............................................................................. 19

2.3.3 Respondent description.................................................................... 19

2.3.4 Qualitative validation....................................................................... 21

2.3.5 Quantitative estimates of commonality .............................................. 21

3 Services and data flows...................................................23

3.1 Services ........................................................................................ 24

3.1.1 Front-end ...................................................................................... 24

3.1.2 Phasor measurement unit ................................................................ 25

3.1.3 Phasor data concentrator ................................................................. 25

3.1.4 RTU/Gateway ................................................................................. 26

3.1.5 SCADA server................................................................................. 26

3.1.6 Inter Control centre Communication server......................................... 27

3.1.7 Application server ........................................................................... 27

3.1.8 Historian........................................................................................ 29

3.1.9 Operator HMI client......................................................................... 30


VIKING Consortium


3.1.10 HMI Server .................................................................................... 31

3.1.11 Remote client (e.g. Critix) ................................................................ 31

3.1.12 Application software for the office ..................................................... 31

3.1.13 Office applications software .............................................................. 31

3.1.14 Web browser.................................................................................. 32

3.1.15 Office servers and power applications outside of the control room.......... 32

3.1.16 Data engineering server................................................................... 32

3.1.17 Data engineering client .................................................................... 33

3.1.18 Remote engineers’ workstation for substations.................................... 34

3.2 Data flows ..................................................................................... 34

3.2.1 PMU Measurements......................................................................... 34

3.2.2 Process measurements to control center ............................................ 35

3.2.3 SCADA sends commands to process .................................................. 36

3.2.4 Front-end data to real-time database................................................. 37

3.2.5 Phasor data sent to real-time database .............................................. 37

3.2.6 SCADA server sends command to process .......................................... 38

3.2.7 Data sent to emergency centre(s) ..................................................... 38

3.2.8 Commands and status ..................................................................... 40

3.2.9 Process data to/from other control centres ......................................... 40

3.2.10 Operator commands to process and manual changes and mark-ups of data42

3.2.11 Operator views and alarms sent to HMI.............................................. 43

3.2.12 A server hosts several HMIs ............................................................. 44

3.2.13 Terminal services............................................................................ 44

3.2.14 Mail websites etc ............................................................................ 45

3.2.15 Read operator views and change parameters of power applications ........ 46

3.2.16 Process data to power applications .................................................... 47

3.2.17 Set points to generators .................................................................. 47

3.2.18 Historian used to update application server data.................................. 48

3.2.19 Historical data sent to historian......................................................... 49

3.2.20 External programs using API in the SCADA server ............................... 50

3.2.21 External programs using APIs in the historian ..................................... 51

3.2.22 Operator HMIs interface the historian ................................................ 52

3.2.23 Web browsers used to access the historian ......................................... 53

3.2.24 The historians API is used by office desktops’ applications .................... 54

3.2.25 External programs interface the application server directly.................... 55


VIKING Consortium


3.2.26 Viewing and changing engineering data.............................................. 56

3.2.27 NIS/GIS applications updates engineering data ................................... 56

3.2.28 Data interchange with other DE systems ............................................ 57

3.2.29 Define structure, views etc ............................................................... 58

3.2.30 Disturbance records sent for analysis................................................. 59

3.2.31 Settings and parameters for RTU and substation equipment.................. 60

4 Control center architecture patterns..................................61

4.1 Zones and placement of services....................................................... 61

4.1.1 Demilitarized zone architecture ......................................................... 63

4.1.2 Single firewall architecture ............................................................... 66

4.1.3 Isolated zone architecture ................................................................ 68

4.1.4 Variants ........................................................................................ 69

4.2 Services added or subtracted............................................................ 73

4.2.1 Thin HMI client ............................................................................... 74

4.2.2 HMI server gives access to operator client in the office......................... 74

4.2.3 No separate server for remote HMIs .................................................. 75

4.2.4 SCADA replica in the DMZ ................................................................ 75

4.2.5 An interface server in the DMZ.......................................................... 76

4.2.6 Separate inter control center communication front-end ........................ 77

4.2.7 External DMS system ...................................................................... 78

4.3 Bundling of services ........................................................................ 78

4.3.1 One HMI for Data engineering and Operators stations .......................... 80

4.3.2 Data engineering directly in the SCADA server .................................... 80

4.3.3 Phasor data concentrator and front-end coupled.................................. 80

4.3.4 Distributed SCADA, power application and historian ............................. 81

5 Identified cross-boundary data flows.................................82

5.1 Substation communication ............................................................... 82

5.2 Inter control center communication ................................................... 82

5.3 Data sent to emergency control center............................................... 82

5.4 Historical data going out from the control center ................................. 83

5.5 Remote operator stations................................................................. 83

5.6 Suppliers dialling in ......................................................................... 83

5.7 Office services updating control center historian .................................. 84


VIKING Consortium


5.8 Office services writing to and reading from SCADA server ..................... 84

5.9 Office services updating control center power application server ............ 84

5.10 Data engineering updated by external systems ................................... 84

5.11 Updates from quality assurance zones ............................................... 85

5.12 Office and internet applications used in the control center..................... 85

6 Conclusions ...................................................................86

6.1 Clear patterns exists in SCADA system architectures............................ 86

6.2 The architecture is partly contingent on the type of utility..................... 86

6.3 The share of legacy systems are substantial ....................................... 87

6.4 It is common that several types of data are exchanged with SCADA systems 87

6.5 The DMZ is not always a “real” DMZ .................................................. 88

7 References ....................................................................89

8 Appendix A –Substation ICT architectures..........................92

8.1 The Single RTU architecture ............................................................. 92

8.2 The field bus/process bus based architectures..................................... 93

8.3 Data flows and relevant communication protocols................................ 95

8.4 References..................................................................................... 97

9 Appendix B – Power System Communication Networks ........98

9.1 Communication media ..................................................................... 98

9.2 Communication Topologies and Network scale..................................... 99

9.3 Communication protocols ................................................................101

9.4 Further reading .............................................................................103

10 Appendix C – Questionnaire........................................... 104


VIKING Consortium


Table of Figures

Figure 1 – Examples of SCADA System vulnerabilities.............................................. 10

Figure 2 – Models used in VIKING, this report focus on the SCADA system. ................ 11

Figure 3 – Deliverable 2.3 in relation to other VIKING-deliverables. ........................... 12

Figure 4 – Architecture metamodel and annotated example...................................... 16

Figure 5 – Focus of this report.............................................................................. 18

Figure 6 – The focus of chapter 3. ........................................................................ 23

Figure 7 – Overview over services and data flows described in this section. ................ 24

Figure 8 – Section 4.1 in relation to the architectural metamodel. ............................. 61

Figure 9 – The demilitarized zone architecture........................................................ 64

Figure 10 – The two zone architecture................................................................... 67

Figure 11 – The isolated architecture. ................................................................... 69

Figure 12 – The separate front-endzone architecture. .............................................. 70

Figure 13 – The back-office zone. ......................................................................... 71

Figure 14 – The quality assurance zone. ................................................................ 72

Figure 15 – Added or removed services and the implication this. ............................... 73

Figure 16 – Thin client in the architecture. ............................................................. 74

Figure 17 – HMI server hosts operator clients located in the control center. ................ 74

Figure 18 – No separate server for remote HMIs. .................................................... 75

Figure 19 – SCADA replica in the DMZ................................................................... 76

Figure 20 – Interface server in the DMZ. ............................................................... 77

Figure 21 – Inter control center front-endand its placement. Alternative a) is in the DMZ architecture, alternative b) is in the other architectures. .......................................... 77

Figure 22 – External DMS system. ........................................................................ 78

Figure 23 – The focus of this section – how services are coupled to each other............ 79

Figure 24 – A single HMI application for data engineering and operators. ................... 80

Figure 25 – A single server for for data engineering and SCADA................................ 80

Figure 26 – front-endimplementing the phasor data concentrator.............................. 80

Figure 27 – Distributed server architecture. ........................................................... 81

Figure B.28 – Common Physical Communication Topologies (a) Star, (b) Ring, (c) Bus (d) Meshed 99

Figure B29 – An example network showing a control system, communication network.101

Figure 30: An example network showing protocols. ................................................102


VIKING Consortium


1 Preface

The VIKING project is an EU financed Framework 7 Collaborative STREP Project and is part of themes 4, ICT, and 10, Security. VIKING stands for Vital Infrastructure, NetworKs, INformation and Control Systems ManaGement and will be executed between November 1, 2008 and October 31, 2011 by a consortium of industrial and academic partners.

1.1 Overview of the VIKING project

The main objectives of VIKING are:

To investigate the vulnerability of SCADA systems and the cost of cyber attacks on society

To propose and test strategies and technologies to mitigate these weaknesses To increase the awareness for the importance of critical infrastructures and the

need to protect them

Society is increasingly dependent on the proper functioning of the electric power system, which in turn supports most other critical infrastructures: water and sewage systems; telecommunications, internet and computing services; air traffic, railroads and other transportation. Many of these other infrastructures are able to operate without power for shorter periods of time, but larger power outages may be difficult and time consuming to restore. Such outages might thus lead to situations of non-functioning societies with devastating economical and humanitarian consequences. For this reason, this consortium has decided to concentrate its research to the systems for transmission and distribution of electric power. We anticipate that most of the results will be applicable to the protection of other critical infrastructures.

The operation and management of the electric power system depend on computerized industrial control systems. Keeping these systems secure and resilient to external attacks as well as to internal operational errors is thus vital for uninterrupted service. However, this is challenging since the control systems are extremely complex. Yet, the systems are operating under stringent requirements on availability and performance: If control and supervision are not done in real-time, the power network may come to a collapse.

These computerized control system, normally called SCADA standing for System for Control And Data Acquisition, includes functions for remote collection of vast amounts of data from measurements placed in strategic points, e.g. power stations, in the geographically widely spread processes and for the remote control of process devices. Many SCADA systems include computerized models of the process which enables simulation of alternatives process states and of optimization. Due to legal and environmental constraints, e.g. for building of new high voltage power lines or power stations, the primary process itself is difficult to expand which in its turn leads to higher and higher utilization of the existing transmission and generation resources. The process is, in other words, operated closer to its physical limits. Those the SCADA systems are becoming increasingly critical for the operation of the process and therefore are becoming a critical component for the availability and security of the supervised infrastructure.

The objective of the VIKING project is to develop, test and evaluate methodologies for the analysis, design and operation of resilient and secure industrial control systems for critical infrastructures. Methodologies will be developed with a particular focus on


VIKING Consortium


increased robustness of the control system. As mentioned, the focus is on power transmission and distribution networks. The project combines a holistic management perspective—in order to counteract sub-optimization in the design—with in-depth analysis and development of security solutions adapted to the specific requirements of networked control systems.

The traditional approach to verify the security of SCADA systems has been ad-hoc testing of existing commercial SCADA system in laboratory environments. The systems to be examined have been installed in different labs, e.g. Idaho National Labs, and tested by skillful people searching for cyber attacks vulnerabilities. The focus in these tests has been on the protection of the central computer system of the SCADA system, since the central computer system has most connections to the external environment through office networks and Internet.

In the VIKING project we will take an alternative and complementary approach to SCADA system security. Firstly we will study the whole control system from the measurement points in the process itself over the communication network to the central computer system as illustrated in the following picture with the yellow exclamation marks indicating potential targets for cyber attacks.

Figure 1 – Examples of SCADA System vulnerabilities

Secondly, and more importantly, we take a model-based approach to investigating SCADA system vulnerability. Models are defined for the SCADA system, for the electrical process as well as of for the society that is dependent on the electricity supply. The society models are used to evaluate the economic consequences coming from disturbances in the electricity supply. The power system models are in turn used to evaluate the effects on the electricity supply by SCADA system misbehavior. Finally, SCADA system models are employed to assess the effect on SCADA system behavior by


VIKING Consortium


cyber attacks. Based on analysis performed on these models, VIKING will propose mitigation actions to be taken to decrease or to eliminate these risks. The results of the project will be evaluated on a test-bed that can be configured to simulate the critical infrastructure of a power network and a range of attacks.

The modeling approach is indicated in the following picture.

Figure 2 – Models used in VIKING, this report focus on the SCADA system.

With this approach the project hopes to achieve the following research results.

Estimates of the security risk and consequences (in terms of monetary loss for the society) based on threats trees, graphical system architecture and society models

Comparable, quantitative results for IT security for different control system solutions and implementations

Use of existing model based application as application level Intrusion Detection Systems to detect manipulation of data

Use of innovative and existing communication solutions to secure power system communication

Help with identifying ”weak spots” and how to mitigate them An environment for performing what-if analyses of the security risk impact of

different architecture solutions.

This report, D2.3 of the VIKING project, address the SCADA system and models of it.

From security requirements to social costs

SCADA system

Power network

Societal cost

Attack


VIKING Consortium


1.2 Purpose of deliverable 2.3

The aim of deliverable 2.3 in the VIKING is to catalogue architecture patterns or reference architectures, i.e. commonly deployed solutions, for SCADA systems. These patterns are represented as a set of descriptions that capture the vast majority of SCADA systems’ architecture on a high level. The patterns developed in this report focus on:

Software services in SCADA systems and software services which SCADA systems exchange data with.

Data flows among these services. How services are placed in different security zones (network zones).

The purpose of the SCADA architecture patterns is to clarify how SCADA systems are commonly designed by employing a stringent model framework. Internal in the project the SCADA patterns will be used to develop SCADA system design models that reflect some typical systems deployed in industry. These models will be used in other work packages and deliverables in the VIKING project. In particular, the patterns is an important input to deliverable 3.1 (Architecture based vulnerability assessment).

To support deliverable 3.1 the architecture patterns presented in this report should be a partial instantiation of the metamodel presented in deliverable 2. 1 (Threats and vulnerabilities – initial report). For a more detailed description of that metamodel, the reader is referred to deliverable 2.1 (Threats and vulnerabilities – initial report) of the VIKING project and the upcoming deliverable 2.2 (Threats and vulnerabilities – final report) which will complement deliverable 2.1 with probabilistic data on attacks’ likelihood of succeeding.

Figure 3 – Deliverable 2.3 in relation to other VIKING-deliverables.

Deliverable 2.1 will be used in task 3.1 (Architecture-based vulnerability assessment) to enable analysis of SCADA system vulnerabilities and risks. In task 3.1 the architecture descriptions described in this report will be analyzed using the probabilistic vulnerability/attack models described in deliverable 2.1 and 2.2.


VIKING Consortium


1.3 Scope

This report will focus on describing the services in and around SCADA systems, the data flows and how services are placed in different zones. It will focus on the state-of-practice and only briefly discuss the difference to state-of-art. The reason for this is that the purpose of the VIKING project starts off with the motivation to effectively enhance security in an operating legacy system environment. A state of the art solution can thus only be achieved over a very long period of time by changing different parts of the existing solution, not through a green field approach. Furthermore, emphasis is placed on control centers and data exchange with other systems and zones. Substation communication, the internal workings of SCADA systems/EMS (Energy Management Systems)/DMS (Distribution Management Systems) and the internal workings of substations are not the focus of this report. This does not mean that these aspects are unimportant from a security perspective. Tentative diagrams for substation architectures and wide area communication architectures are given in Appendix A. Some issues of the internal workings of EMS are addressed in other reports of the VIKING project.

1.4 Document overview

This report gives a qualitative and quantitative overview over architectures used in the installed base of SCADA systems. Chapter two of this report will describe the theoretical framework and method used to survey these architectures. This includes a description of the metamodel used throughout the survey, literature studied on this topic and the interviews with SCADA system vendors.

Chapter three gives an overview of the reference model developed throughout this project. A set of services and data flows are described here. Chapter four continues with a describing how these services and data flows are instantiated in different architectures. First it is described how services are placed in zones in the general case. Thereafter special solutions where services are added or subtracted are described. Lastly, in chapter four different ways to bundle services into software components are described.

Chapter five summarize the architectures from the viewpoint of cross-boundary data flows, i.e. data flows that cross the boundary to the SCADA system’s zone. Chapter six concludes the findings in this report.

This report focuses on services associated with the central part of the SCADA system, primarily those of the control center. In appendix A and B an overview of substation automation systems and control center to substation communication is given.


VIKING Consortium


2 Method and theoretical framework

The architectures presented in this report have been identified through a literature study and number of interviews with SCADA system vendors. This chapter will describe the literature review, the metamodel used to document the architectures and the interview procedure as well as the respondents.

2.1 Literature review

This study was initiated with a review of literature where state-of-practice and state-of-art for SCADA system architectures where discussed. The literature found through this review is presented below under the categories:

Security literature SCADA and control system literature Market surveys

Literature found in all these categories describes SCADA systems from one viewpoint or another. In some cases there is also qualitative or quantitative statements indicating how common different variants are. Although this literature was used as input to the models described in this report they did not describe the state-of-practice to the amount of detail given in this report (hence the need for interviewing vendors). A more detailed description of the reference material found under these categories is given below.

2.1.1 Security literature focusing on SCADA systems

Architectures can be described from different viewpoints and with different views [1]. When security is to be assessed from the model some aspects are more important than others. Descriptions of SCADA system architectures given in security related literature can therefore be expected to be more relevant for the VIKING project than models described elsewhere.

A number of texts on SCADA security have been published in recent years. This includes standards and guidelines such as NERC CIP [2], NISTs special publication 800-82 [3], ISA-99 [4], IEC’s standards on secure substation and SCADA communication[5,6,7,8,9], NICSS practice guide for firewall deployment[10], the Secure Security Procurement language[11], and much more. Academic writing on this topic can be found in journals (e.g. [12,13,14,15,16,17,18,19,20]), and in greater quantity in conferences proceedings ( e.g. [21,22,23,24,25]). A number of textbooks (e.g. [26] and [27]) also exist on this topic.

Practically all of these provide schematic descriptions of SCADA systems and describe common vulnerabilities in these. For instance, in [3] NISTs special publication 800-82 and ISA-99 network diagrams over typical SCADA systems are given, and in [4] several aspects of SCADA communication is described.

While these references describe SCADA systems they do so to a limited extent. The descriptions included in these texts are intended to be general introductions to SCADA systems. Consequently these publications focus on some central aspects and disregard many variations that exist. Most commonly the architectures are described in terms of which physical nodes and links that exist. The logical distribution and division of services/functions and what data and data flow that is present are more implicitly addressed. It is also difficult to combine the descriptions into one coherent architecture


VIKING Consortium


description. Both because the metamodels used are unclear and because the references contain a mixture of descriptions of typical architectures, vulnerable architectures, and best-practice solutions.

The work in this category that appears to be most closely related to the work presented in this report is the reference model presented in [28]. In this reference model four levels are identified: oversight entity, system and plant control centers, SCADA field equipment and infrastructure equipment. An object-role model over the services (objects) in these domains is outlined in the report and some security issues associated with the relations are discussed. In relation to [28] this report further detail the services, their relationship to each other (data flows in between these), and the levels (zones) related to these systems.

2.1.2 SCADA and control system literature

There are plenty of textbooks, articles and reports that describe SCADA systems in general. This includes:

Textbooks such as [29,30,31,32,33,34] Articles such as [35], [36] and [37,38] Reports such as [39], [40], [41] and [42].

These references include both schematic overviews of SCADA systems and detailed descriptions of their components. Some of them also provide empirical data on how common different solutions, services, or configurations are. None of the references found does however provide an encompassing description of common architectures. For example, the quantitative data given in [36] is limited to Finnish distribution utilities. Moreover, many references describing SCADA systems are dated several years back. Other are stated as state-of-practice descriptions or likely future architectures, see for example [38] and [42]. Their accuracy with respect to the SCADA systems now installed is thereby questionable and worth investigating.

The descriptions given in SCADA and control system literature have been used as an input to this work. The models presented in this report are also overall aligned with the descriptions given in literature. The interface reference model described in [41] was for example used as input when services where identified. From a particular viewpoint (described in section 2.2) this work has investigated how common different variants are.

2.1.3 Market surveys

Quantitative data on how common different architectures are is rarely described in academic literature or standards. It is however a common component in market surveys.

The market trend digests produced by Newton Evans Research Company1 does for example provide data on what security strategies utilities apply, what security measures that are being implemented, how common outage management functionality is and how common different protocols are in different regions. Also other actors produce market surveys, e.g. [43].

These market surveys have also been used as an input to this work, although to a limited extent. None of the surveys found has had the encompassing scope of this work, but

1 Official website: http://www.newton-evans.com/


VIKING Consortium


rather surveyed what is a common solution for a certain components in the SCADA system.

2.2 Metamodel

In order to describe the SCADA reference architectures in this report an explicit metamodel has been developed. A metamodel is the model language and describes what the architecture model can, and should, contain. The vast majority of the literature found throughout the literature survey did not clearly state the scope and content of the metamodel used. This section describe the metamodel used for the survey used in this report and gives a rationale for it.

2.2.1 Scope and definition

The architecture models described in this report include a set of services that has been identified. A service in these models is realized by one or more software component(s). The components as such are however not explicitly regarded as entities in their own right here. Examples of services include, “Engineering stations (HMI)” and “Data Engineering Server”. A service can be physically distributed on different machines, for example spread over multiple computers. It can also be replicated in the architecture; it is for example common that multiple Data Engineering Clients are instantiated in the architecture. Neither of computers is considered explicitly.

Figure 4 – Architecture metamodel and annotated example.

A service belongs to a zone. A zone in this metamodel has a close relationship to a network zone in computer networking. It should however rather be seen as a security domain as defined in [44], i.e. “...a set of security elements subject to a common security policy defined and enforced by a single security policy authority”. In the notation used a zone is represented as a frame around the elements (services) included in it. If there is a tunnel (e.g. a virtual private network, VPN) that makes it possible to join one zone from another zone this is also represented. In that case this represented by a block-arrow between the zones.

Data can be exchanged between services. The element data flow is included to represent this. A data flow goes between two services and data is written to either one or both


VIKING Consortium


sides (services). A line connecting two services in the notation used represents a data flow that goes between them. If an arrow is at the end this end is written to. In this report focus is on data flows that hold an apparent and direct business value is included. This means that data flows that only provide infrastructure services, e.g. the DNS and ARP, are not included.

2.2.2 Metamodel rationale

The metamodel used in this study only represent a subset of the information needed to assess security. VIKING deliverable 2.1 presents a metamodel for security analysis. As can be seen in Figure 4 the vast majority of the elements from this model are not covered here. Examples of elements that are excluded are authentication functionality, authorised users, persons, protocols, operating systems, hardware, physical zones and intrusion detection functionality. Consequently, the metamodel also excludes many concepts that are discussed in SCADA security literature.

The Purpose of this work is however to focus on what is specific for SCADA systems. That is foremost the division of services and corresponding dataflow. Of course, a larger scoped metamodel could have been used but due to limited time and resources in the project this was not prioritized. In addition, as technical details, and in particular details related to security solutions, are often confidential this type of information is more difficult to both get hold of and publish. Hence, a further argument for limiting the detail level was to avoid difficulties were when collecting empirical data on architectures.

The elements that are included in this metamodel is however central to security analysis. Classical security models such as the Bell-LaPadula model [45] and Biba model [46] focus on data flows between security levels. Security levels can be seen as the security zones used in this metamodel. Security domains are also described as a powerful concept in [44], and the implications of interactions and relationships between domains are explicitly discussed.

Modelling notations specifically developed to support security analysis also support the simplification of architecture models made here. In e.g. threat modeling procedures described in [47] and [48] actors, data flows, processes, and trust boundaries are included which is closely related to the data flows, services and zones used in this report. The elements included here are also included in deliverable D2.1 “Vulnerability models” of the VIKING project. A zone in this model is in D2.1 a “network zone”.

A further argument for the metamodel used here is that it is similar to other metamodels with a closely related purpose. In [49] the “Unified logical architecture for the smart grid” is depicted through a set of boxes (services) and arrows (data flows) over different domains.

2.3 Interviews with vendors

The reviewed literature was in combination with the experience possessed by consortium members used to create a set of draft models over SCADA systems. These models complied with the metamodel presented in section 2.2 and outlined the software services, the data flows between these, and the services their placement in different zones. As this baseline model was developed from the consortiums experience it was expected to be biased towards architectures and solutions commonly implemented by ABB. To validate that these architectures are prevalent in the industry, identify other


VIKING Consortium


architectures and collect data on how common different architectures are a number of SCADA system vendors where contacted.

This section will describe:

the scope and delimitations of this survey, the baseline models, the respondents interviewed, how the qualitative descriptions of architectures were validated, how quantitative estimates of how common these architectures are was

collected

Each of these is addressed in a subsection below.

2.3.1 Scope and limitations

The architectures and architecture-variants presented in report comply with the metamodel described in section 2.2. The data flows and services covered in these architecture diagrams are those that offer SCADA system functionality or services closely related to this functionality. In simpler terms, this study covers the services located in SCADA system control centers and the services that exchange data with these services. This means for instance, that services located in the office are included if they exchange data with the SCADA system; other services exchanging data with the services located in the office are however not included.

A key feature of most control systems is that they are implemented with some degree of redundancy. Redundancy is often implemented for both infrastructure, e.g. as network connections and network devices, and for central services such as the SCADA server. The redundancy typically implemented for services in SCADA systems are briefly discussed in chapter 3.Redundant services is however not included in the architecture descriptions and not explicitly covered during the interviews with vendors. Chapter 3 also indicates which services share infrastructure software (i.e. machine and operating system) although it is not included in the metamodel.

Figure 5 – Focus of this report.

The focus of this report and the interviews with suppliers has been placed on services located in control centers and their relationship to other services. This means that substation automation systems, e.g. the communication between intelligent electronic devices, are not included. The focus on the control center and the absence of technical details in the metamodel also means that the technical realization of the communication between substations and the control center falls out of scope of this report. While these


VIKING Consortium


two parts of the SCADA system is not fully detailed in this report an overview of them is given in appendices. An overview of substation automation systems is described in Appendix A and an overview of substation communication solutions is given in Appendix B. The architectures described in these appendices are based on a literature review and has not been empirically validated. This is not the case for the descriptions given in chapters 3, 4 and 5. The content of these chapters has been validated.

2.3.2 Baseline models

Prior to the interviews with SCADA system vendors a set of architecture models over SCADA systems were developed. These where drawn from descriptions given in literature and processed by members within the consortium.

The literature used as input to this process is described in section 2.1. Consortium members from the Royal Institute of Technology (KTH) and ABB was involved in creating these baseline models. The intention was to create generic models over SCADA system architectures. However, as ABB was the only SCADA system vendor involved in this process the baseline was expected to be biased towards the architectures and solutions that ABB offers.

The baseline models were used to produce a questionnaire used throughout the interviews with SCADA system vendors. During the interviews this baseline model was presented to the respondents and deviations, amendments, and comments to this baseline model were noted. The baseline model can be found in appendix C of this report. In appendix D the questionnaire used throughout the interviews, is shown.

2.3.3 Respondent description

A substantial effort has been placed on finding suitable respondents. The consortium contacted the following vendors with a request to meet for an interview: ABB, Siemens, PSI, Netcontrol, Areva and General Electric. When this report is issued the first four has accepted to meet and also been interviewed. In Table 1 an overview of these vendors and their typical customers is given.

Table 1 –Overview of vendors in this study.

Vendor Product Typical customer

ABB ABB Network manager Transmission utilities or larger distribution utilities worldwide.

ABB MicroScada Small or medium sized distribution utilities in Europe, the Middle East or Asia.

Siemens About 10 (including legacy systems)

Transmission and distribution utilities worldwide.

PSI PSIcontrol Transmission and distribution utilities in the German region or Russia.

Netcontrol Netcon 3000 Small or medium sized distribution utilities in Sweden or Finland

ABB have a long history in the market of SCADA systems. The product Network Manager and its ancestors ABB Ranger and ABB Spider are installed in about 500 control centers around the world. Network manager is primarily used on the transmission level or in


VIKING Consortium


larger distribution utilities. Ranger-based systems are mainly installed in the US and Australian markets and Spider-based systems are mainly installed in the European, African, and Asian markets.

ABB MicroSCADA is another ABB-product. ABB MicroScada is targeting distribution utilities and is mainly installed to operate power networks on the distribution level. More than 1000 MicroScada systems are installed in control centers in the world today, primarily in the European and Asian market.

Siemens are also seasoned in this market. Around ten different products or product versions are included in the installed base of Siemens system. Their systems are installed on all voltage levels throughout the world, but the respondents interviewed in this study do not have great experience from the US market.

PSI is a SCADA system vendor with the base in the German market. Their SCADA system solution is in use on both the transmission level, sub-transmission level and on the distribution level. While the vast majority of PSI’s systems are installed in Germany, some installations also exist in other countries and in other continents, e.g. in France, Switzerland, Rwanda and Russia.

Netcontrol is as ABB MicroScada focusing on the distribution market. Their customer base is small to medium sized utilities in Sweden and Finland. Around 110 Netcontrol systems are running in control centers in these countries.

Some vendors market multiple products and have a clear division between different regional markets. Therefore multiple interviews were held with different respondents at ABB and Siemens. An overview of the respondents, their experience and the focus of the interview with them is given in Table 2.

Table 2 –Respondents in interviews.

Vendor Respondent Experience working with SCADA systems

Experience working with the vendor and/or systems

Focus of interview

ABB Neela Mayur - - Network Manager, US market

ABB Gunnar Bjoerkman

34 34 Network Manager, rest of the world

ABB Mikael Molander 21 21 ABB MicroSCADA Siemens Hans-Joachim

Diehl 34 35 All

Gilbert Suter 38 10 (38 with Landis &

Gyr)

Distribution

Erich Wuregler Distribution Netcontrol Mats Elmgren 41 7 Distribution Lars-Gunnar Lif 33 15 Distribution PSI Wolfgang Fischer 22+ 22 All Joachim 18+ 18 All


VIKING Consortium


ABB has gone to the market with two solutions: ABB Network Manager and ABB MicroSCADA. Neela Mayur represented ABB Network Manager for the American market and Australian market. This also includes the previous installations of the product RANGER. Gunnar Bjoerkman represented ABB Network Manager for Europe and the rest of the world, which includes previous installations of the products known as SPIDER. Mikael Molander represented ABB for the product MicroSCADA.

Siemens also market multiple SCADA system products. To get an overview of these three respondents were interviewed. Gilbert Suter and Erich Wuregler are based in Switzerland and the interview focused on Siemens solutions for power distribution (lower voltage levels). Hans-Joachim Diehl confirmed the output from the interview with Gilbert Suter and Eric Wuregler. Hans-Joachim Diehl also complemented this interview with variants and data applicable to Siemens solutions on transmission and sub-transmission.

Netcontrol was represented by two respondents who gave an overview of the vendor’s architectures as a whole. As Netcontrol’s customers are power distribution utilities this was the focus of the interview.

Also PSI was represented by two respondents who gave an overview of the vendor’s architectures as a whole.

2.3.4 Qualitative validation

As outlined by the questionnaire in Appendix D the interviews were initiated by an overview of the services and services included in the baseline model. This was followed by a presentation of three different network zone models and how services are placed in these zones. From this follows that certain data flows will cross over zones and some will not.

Throughout this procedure respondents were encouraged to give comment to these architecture descriptions and comment unfamiliar or missing components. This includes the services included in the descriptions, the data flows that go between these, the zones included in the descriptions and how the services were placed in different zones.

This part of the interview lasted between one to three hours. Three hours was spent by PSI whose SCADA system has, in some aspect, a different architecture2 than the baseline architecture and approximately one hour for the other respondents.

All comments and deviations where noted throughout the interviews. Once documented in an electronic and readable format the respondents has had the opportunity to review and comment these interview protocols.

2.3.5 Quantitative estimates of commonality

The second part of the interviews was dedicated to quantifying how common different elements are in the installed base of systems. As outlined by the questionnaire in Appendix D this includes quantifying how common different zone models are, how common different data flows are and quantifying how common it is that services share one machine.

Although archival records (e.g. specifications from delivered projects) might be obtainable for these quantities they were all made based on the respondents’ own

2 PSI’s SCADA system is “distributed” unlike the other vendor’s SCADA systems.


VIKING Consortium


experience. The respondents in this study have a substantial experience from both SCADA systems in general and the products offered by the vendor they represented. This, and the difficulty to persuade respondents to retrieve archival data for these quantities, are the primary reasons for relying on the respondents own experience and judgment.

It should be noted that many of the estimates obtained where in the form “about half of the systems”, “the vast majority of systems” or “only some installation has that”. These estimates are also, according to the respondents themselves, associated with a substantial uncertainty in some cases. In this report these estimates are presented together with the components and variants they are associated to, see chapter 3 and chapter 4. When they are presented, they are presented as they were expressed by the respondents.

These estimates are also presented independently of each other. There is however often a clear dependency between the architectures. For example, modern SCADA system architectures often have the demilitarized zone architecture. Also, it is modern SCADA system architectures that use quality assurance zones and the modern SCADA systems exchange data with office systems more often than old legacy systems. Although these dependencies apply to many of the variants they are not surveyed in this work and is described in this report. Further work could explore the relationship between different patterns and how common different combinations of architecture variants are.


VIKING Consortium


3 Services and data flows

This chapter will present a number of services and data flows present in SCADA system environments. As described in section 2.3.1 this work aims to identify the services that located within the control center, or is directly interfacing services in the control center. In relation to the metamodel (cf. Figure 6) this chapter will first describe the services in the reference model used throughout this study, thereafter the most common data flows are describe

Chapter 3 thus provides a list with descriptions of the building blocks services and data flows. The actual architectural patterns describing how the services are placed in zones and how this influence the data flows is described in chapter 0. Chapter 0 also describes replicas of these services and any special solutions identified in this study.

Writes to

Service

Zone

Dataflow

1..*

1..1

1..22..2

Goes between

Belong to

0..*0..*

0..*

0..*

Can join

Figure 6 – The focus of chapter 3.

Figure 7 gives an overview of the services and data flows described in this chapter.


VIKING Consortium


Data Engineering server

[29] Define data structure, views etc

[27] NIS/GIS applications updates engineering data

Engineering stations (HMI)

[26] Viewing and changing engineering data

[28] Data interchange with other DE systems


Office Servers and power applications located

outside the control center

SCADA Server

Control center application Server

DMS(Application

Server)

GMS (Application

Server)

[16] Process data to power applications

EMS(Application

Server)

[17] Set points to generators

Historian

[20] External programs using APIs in the SCADA server

[21] External services using APIs in the historian

[19] Historical data sent to historian

[23] Web browsers used to access historian

[22] Operator HMIs interface the historian

[24] The historians API is used by office desktops’

applications

[25] External programs interfaceapplication server directly

Application software for the office

(e.g. browsers, mail clients)

[10] Operator commands to process and manual

changes and markups of data

Operator HMI Client

[11] Operator views and alarms

sent to HMI

Remote client(e.g. Citrix client)

[14] Mail, websites etc

HMI Server (e.g. VNC, Terminal Services or Citrix)

[13] Terminal services

[12] A server hosts several HMIs

Office application software

(e.g. spreadsheets)

[3] Front end sends commands to process

Front End

RTU/SCS

[6] SCADA serversends commands to

front end

[4] Front end data to real-time database

[2] Process measurements to

control center

Phasor data concentrator

Phasor measurement unit

[1] PMU Measurements

[5] Phasor data sent to real-time database

Inter control center

communication server

[9] Process data to/from other control centers

[8] Data shared with inter control center server

[7] Data sent to emergency center(s)

[15] Read operator views and change parameters

of power applications


[29] Disturbance records

[30] Settings and parameters

Web browser

[18] Historian is used to update application server data

Remote engineers’

workstations

Figure 7 – Overview over services and data flows described in this section.

3.1 Services

This section will describe the services (boxes) depicted in Figure 7. How these services are placed in zones, replicas of these services and services only used by some vendors are described in chapter 4. Data flows (arrows) are described in section 3.2.

3.1.1 Front-end

The front-end servers are responsible for the communication with the Remote Terminal Units (RTU) and Substations Control Systems (SCS) systems. They receive process data in raw format using standardized RTU communication protocols like IEC 60870-5-101/104, DNP3.0 or MODBUS. However, a big variety of proprietary protocols are also still in use in operational system due to the very long life time of RTUs. In addition to data acquisition, the Front ends receive commands and setpoints from SCADA servers


VIKING Consortium


and send these to the RTUs and SCS for execution. Traditionally point-to-point connections to the RTU/SA systems have been used but today there is clear trend to move to TCP/IP based protocols like 60870-5-104 and DNP over TCP/IP. For TCP/IP based protocols the front-end application is sometimes implemented in the SCADA servers and commercial routers are used to interface the SCADA zone to the Wide Area Network.

In some cases, like for the collection of local disturbance data, file transfer types of formats are received and sent to applications in the SCADA servers.

Normally the front-ends perform pre-processing of process data for the SCADA servers like conversion of raw digital formats to engineering values and translation of hardware addresses from the RTUs/SCSes to logical database addresses in the SCADA servers. The converted process data is transmitted to the SCADA servers (both online and standby) with very short time delays.

Front-ends are also responsible for the supervision of RTU communication lines and reconfiguration to redundant access paths if available. Statistics for the communication lines are collected like bit errors, number of retries and time out errors. Fronts Ends are normally redundant and support line-by-line failovers for point-to-point connections or complete failover between the two redundant front-end servers. The most common configuration is that the communication load is shared between the two Front Ends in a pair on a line-by-line basis but a single front-end is dimensioned to handle the whole traffic in case of failure of the other server. An alternative configuration is that one front-end handles the whole traffic and the standby is prepared to take over if the online server fails.

In some SCADA configurations data concentrators are used. These are servers that are placed geographically outside of the central control system in the communication network structure and are used to concentrate low speed RTU traffic to higher capacity communication lines. In many cases protocol conversion and front-end functionality are performed in the data concentrators.

3.1.2 Phasor measurement unit

In the recent years a new generation of measurement devices have been introduced. These are called Phasor Measurement Units or PMUs. Through the use of GPS a very accurate time measurement can be done and process values previously not possible to measure are now attainable. One example of this if phase angle measurement, especially phase angle difference over power lines, which is a measure of the electrical stability of the network. Measurement are stored locally in the PMUs with very accurate time stamps and sent to phasor data concentrators (see next section) where special applications to calculate and present network stability exist.

PMU generate substantially higher volumes of data and required more bandwidth than traditional measurements.

3.1.3 Phasor data concentrator

The phasor data concentrator collects data from PMUs using PMU protocols IEEE 1344-1995 or C37.118 and sends this data either to special applications in the SCADA servers or to dedicated applications servers for evaluation of network stability. The phasor data concentrator is normally implemented in the same computer as the front-ends. Some vendors implement the phasor applications directly in the front-ends.


VIKING Consortium


The values from PMUs are also used in traditional State Estimators to improve the accuracy of this application. Through the use of PMU, data values previously only possible to calculate are now possible to measure directly.

3.1.4 RTU/Gateway

A Remote Terminal Unit (RTU) acts as an interface and communication unit between field instruments and a SCADA master. The pioneer generations did very little more than converting the information from the instruments to the SCADA language, send the information through or vice versa. RTUs are gradually improved, with strong communication capabilities in mind. Modern RTU include web servers for configuration and support a large number of communication protocols and acts today as a Gateway between substation control equipment and the central control system.

3.1.5 SCADA server

The process information from the front-end servers are sent to the SCADA servers and stored in a real-time database. The real-time database maintains a SCADA model of the supervised process and is using the real-time information to update this model to an image of the current process state. The time difference between the real process state and the information in the real-time database is normally in the range of a few seconds. The reason that virtually all major SCADA vendors use proprietary real-time databases is that the event throughput requirements at process disturbances and for operator picture call-up could not (and cannot?) be met with the technology available in commercial relational databases.

The main task of SCADA is to monitor the process data for significant changes and to alert operators about these changes. Such a significant change can be a breaker opening initiated by line protection or a voltage measurement over an alarm limit. Significant changes are collected chronologically in Event and Alarm lists including local time stamps. Event lists record all what happens in the process, all operator actions and all events in the control system, while Alarm lists collects the most important events and requires an active acknowledgment from the operators to confirm their attention. The operator can, via the operator HMI clients, make different types of extracts from event and alarm lists.

SCADA servers normally give the operators and system engineers the possibility to define their own calculations. Such calculations can be defined, stopped and resumed online in the SCADA servers. Such calculation can be started either cyclically or started on events and can be used to calculate not directly measured values, e.g. the total generation in a power station with several generators. A related functionality is the possibility to create command sequences that can be initiated automatically or manually by the operators.

The operators use process displays on the workstations to supervise the status of the real-time process and to command breakers or isolators or to send new setpoints to local processing units. Automatic commands for "close-loop” regulations exist but are rare, mostly the operators close the loop of active regulation.

The SCADA servers include functionality for system self supervision. All devices like servers, workstations and switches/routers are continuously supervised and if a problem is discovered in one device, actions are initiated to restore lost features. If a hardware or software problem is found in one of the SCADA or application servers, a failover is initiated, i.e. the online execution is transferred to a standby computer that is kept on a


VIKING Consortium


continuous “hot” standby state either by keeping the databases synchronized or by parallel execution.

Information in the real-time database of the SCADA servers might be of interest to office user in order to create reports or make their own dedicated applications. In this case a DeMilitarized Zone (DMZ) is normally set up between the office zone and the SCADA zone. To make the real-time data available for office user in a secure way a replica of the real-time database are included in the DMZ. Such a replica database is continuously kept in an updated state by the SCADA servers.

Some vendors use the HMI Clients to store the real-time SCADA model. Received real-time data from the Front Ends is sent to all workstations and the Event and Alarm handling and other types of SCADA functionality is done locally on the Clients. The major advantage with this concept is that no data has to be fetched from a central SCADA server at picture presentation which will give very good picture call-up times. Also the scalability of the control system is easier because new operator work places will not load down the system significantly. On the other hand synchronization of the Clients at start-up and at operator entries is more complicated to secure that all Clients show the same process state

However, the SCADA functionality in these two types of configurations is more or less the same; it is the distribution of programs between SCADA and HMI Clients that differs.

3.1.6 Inter Control centre Communication server

Control systems often communicate with each other. This could be regional control systems that send process data from their regions to a National Control Centre or neighbouring countries that need to exchange process information for important cross-country power lines. Standardized protocols have been defined for this type of communication and the protocols commonly used today are TASE.1 (ELCOM) and TASE.2 (ICCP). Of these two the TASE.2 is gaining in importance and is currently dominating the market for new control systems. These protocols support transmission of process data and alarms, sending of commands and set-points and handling time stamped data.

The inter control centre communication is normally handled in dedicated servers. These servers access data from the real-time data base and convert this data to one of the TASE protocols and handle the actual communication traffic to the other centre.

Commercial products for secure protocols, like Secure MMS, are sometimes only available under certain operating system, e.g. on Windows, and might require that the Inter Control centre Communication Servers are split up in two computers, one for real-time database access and one for the actual communication.

3.1.7 Application server

Pure SCADA systems, as described under the SCADA servers section, are only used as data acquisition and process control systems. All measurements are acquired independently of each other and the process knowledge, i.e. how various objects of the process are interconnected are only shown in pictures. The same SCADA system can be used independently of the monitored process because the basic SCADA functions, i.e. data acquisition and control and event and alarm reporting, are used in all types of process monitoring. The main purpose of a SCADA system is to maintain a process type independent SCADA model of measured values and statuses.


VIKING Consortium


If the SCADA system should include a more intelligent understanding of the monitored process, models of the various process objects must be introduced, i.e. a process model is required. These models are, of course, different depending on the type of process that is supervised. There is a difference in behaviour between a compressor in a gas pipeline and a transformer in an electrical network. The process model is connected to the SCADA model through database references to create a process model including real-time measurements.

It is also important to define the connectivity of the process, i.e. how the various process devices are connected to each other, for example, on which bus a certain circuit breakers is coupled. This static connectivity is combined with real time statuses of breaker and isolator states to create a dynamically updated “bus-branch” model of the current process state.

Using the real-time updated dynamic topological model as a base it is possible to implement advanced applications. These applications will be unique for each process type based on different physical characteristics of the process. Perhaps the most clear examples of such applications exists in the area of SCADA systems for power grids, where for a long time such applications have been used based on relatively simple mathematical models of the electricity grid, i.e. Ohms and Kirchhoff’s laws. It is not the purpose of this report to discuss the different types of applications in detail but a brief description how these applications work can provide a better understanding of their use and importance.

By using a non-complete real-time measurement vector (not all points in the network have measurements) the complete power flow state of the network can be calculated (state estimation) provided that the network is observable, i.e. enough measurements are available. This calculated state defines all voltages and phase angels in all nodes and can be used to calculate the active and reactive power flows for the whole network. The resulting network state can also be used to simulate new situations in the network, for example, after disconnecting a transformer for maintenance or for other operational changes in the grid. These studies are normally done in a parallel database, a so-called study database, in order not to disturb real-time operation. Load flow calculations can automatically be done for all possible errors in the network (N-1 analysis) and warnings or recommendations for grid changes to avoid dangerous situations can be issued before the contingencies actually occur.

The electrical applications make it possible to calculate the optimal flows in the network in order to minimize, for example, the active losses which could mean substantial economical savings. Similarly, the optimal production schedules with regard to the best use of resources (nuclear power, oil, gas, water) can be calculated and applications for automatic control of generating units according to the approved schedules are available.

Process models and applications can also be used to achieve realistic operator training environments with instructor consoles and operators to be trained in normal and disturbed process situations.

In the list below typical electrical power applications in SCADA system for transmission and distribution networks are listed. In some SCADA systems these applications are not realized inside the SCADA system itself but are external applications from other vendors or partners that are interfaced to the SCADA system with more or less tight integration schemes. This will be further described in paragraph 4.1.15 “Power application outside of the control room”. The following list shows the most important electrical applications grouped according normally used system classes.


VIKING Consortium


Energy Management Systems (EMS) o State Estimator o Bad Data Detection o Operator Load Flow o Contingency Analysis o Short Circuit Calculation o Optimal Power Flow o Equipment Outage Scheduling o Dynamic Stability Calculations

Generation Management Systems (GMS) o Unit Commitment o Hydro Thermal Optimization o Automatic Generation Control (AGC) o Production Costing o Reserve Monitoring o Transaction Scheduling

Distribution Management Systems (DMS) o Dynamic Network Coloring and Tracing o Load Calibration o Load Flow o Short Circuit Analysis o Volt/Var Optimization o Switch Order Management o Fault Isolation and System Restoration o Outage Management o Crew Management

Operator Training Simulators (OTS)

3.1.8 Historian

An important task for SCADA systems is to record and archive all incoming data from the process and registration of all operator actions. For this purpose the SCADA systems use special archiving applications and servers based on a commercial relational database such as Oracle. Connected to the relational database, there is a variety of data mining, reporting and programming tools available for different types of users ranging from the normal control room operators to system users and application experts. An alternative solution is to interface commercial available archiving systems to the SCADA system.

The historian can be used for;

Storage of process data, events/alarms, operator actions, calculation and applications results, etc.

Storage of planned (future) data. The historian is sometimes used as an interface for schedules (time series data) from planning and optimization functions, normally used in the office environment, to real-time control functions, e.g. Automatic Generation Control (AGC)

Definition of user calculation, both in the value dimension, e.g. power station sums, and in the time dimension like daily, weekly and monthly summations.

Play back of process data course using the normal process displays Presentation of trends and reports


VIKING Consortium


Data Mining, i.e. an easy of query process information. One example could be to use a query command to find out which 220 kV lines have an availability of less than 95% over the past five years.

Programming of advanced statistical applications that use historical real-time data for to find changes in process characteristics

Archiving functions must have the capacity to record all information coming from the process through RTUs, station computers, and other control centres, plus archiving of event and alarm and calculation results. This means that many thousands of data points have to be stored every second. The archive function uses many types of media such as DVD or tape robots for automatic long-term storage when disk space is no longer enough. In this way virtually unlimited storage can be achieved.

Information in the historical database of the SCADA servers is very often of interest to office user in order for them to create reports or make their own, dedicated applications, e.g. scheduled maintenance based on loading of primary equipment. In this case, like for the real-time database, a demilitarized zone (DMZ) is normally set up between the Office zone and the control center zone. To make the historical data available for office user in a secure way, a replica of the historical database is included in the DMZ. Such a replica is kept in a synchronized state with the master Historian on the control center zone.

3.1.9 Operator HMI client

HMI clients showing process state on man-machine workstations are the main service for the operator to visualize process state and send commands to the process. These workstations are usually based on PC computers equipped with multiple VDUs and are using modern, full graphic technologies to present process information in different views. The information on operator workstations is constantly and automatically updated by the SCADA servers as soon as any significant change is detected in the process. Modern presentation techniques, e.g. multiple windows, information zoom (declutter) and layers are employed to give ergonomic views of the process to the operators under various operational conditions.

Authority control is normally used to restrict operator presentation and intervention to process areas and functionalities allocated to specific operator.

The process and control system state is presented to the operators using the following types of displays:

Schematic and geographic single-line displays showing real-time network state Event and Alarm lists Trends and Reports Tabular displays Control system status and configuration displays Maintenance and diagnostic displays

As mentioned above under SCADA servers some vendors have moved parts of the SCADA functionality and the real-time database to the HMI Clients. The functionality in this alternative solution is the same but the distribution is different.


VIKING Consortium


3.1.10 HMI Server

Many SCADA systems include support for remote HIM clients, i.e. workstations not placed in the actual control room. These clients are typically used from remote district centres, demo centres, office networks, homes of operators, etc. Remote clients provide basically the same functionality as the control room operator clients but are limited by authority setup.

Two alternative technical solutions exist for remote clients. One is that the HMI client is installed on a remote workstation and the communication is the same as in the local control room. The other solution, supplied by this service, is that an HMI server is included in the SCADA configuration. This server can serve a number of workstations using standard technologies like VNC, Terminal servers or Citrix, with remote desktop presentation. One of the advantages with this service, as compared to the HMI client installed outside the control center, is that it does not require any local client installation.

One important aspect of remote client is to apply, at least, the same authority rules as in the control room. It is essential that the remote client users can be securely authenticated and that their usage of the SCADA system and possibilities for control of the process can be limited.

3.1.11 Remote client (e.g. Critix)

To utilize the HMI server or terminal server functionality running elsewhere in the SCADA system there must be a remote client capable of handling the remote graphical desktop protocol of the server. Common clients are Windows Remote Desktop or a web browser (for Citrix).

3.1.12 Application software for the office

In some cases connections to administrative applications, such as e-mail, text editors and calculation programs, are available on the same computers as the HMI Clients. These applications are used to create reports and document that are using real-time and historical information from the SCADA system. Some of these are allowing Internet connection.

3.1.13 Office applications software

It is quite common in modern SCADA system that office users require SCADA information to create ad-hoc reports and/or to write special applications. Normal office applications like MSOffice and/or programming languages are used to access data for these reports and applications. Some SCADA systems provide an Application Programming Interface (API) for this purpose that gives the office users an easy-to-use access to the information in the Historian and to the real-time database. These interfaces are normally built on standard database interfaces like SQL or ODBC. The APIs hide some of complexity of the SCADA data structures for the users, e.g. internal structures of the Historian. Also more advanced interfaces dedicated for process data like OPC and DAIS are commonly available for office users. It is necessary that the users know the SCADA models, Process models and Historian models of the respective SCADA system since these are not standardized.

In order to avoid direct access from office users to the SCADA system replicas of the Historian and the real-time database are placed in a DeMilitarized Zone (DMZ). As


VIKING Consortium


described in previous sections the master SCADA server and Historian will keep the replicas continuously updated.

3.1.14 Web Client

A Web client for the Historian is provided by some of SCADA vendors and vendors of standalone historical systems. These Web clients are used to access information in the Historian using web technologies. They give the possibility to build user defined reports and to make queries of the Historian database.

3.1.15 Office servers and power applications outside of the control room

Most modern SCADA systems have some type of connection to external applications that are placed in the office network. For smaller SCADA systems that do not have their power applications these applications could be externally developed and commercially available applications of the same type as those described in section 3.1.7. The data exchange between these application and SCADA is typically defined for each individual system and varies from implementation to implementation.

Also bigger types of SCADA systems frequently have connections to external applications placed in the office network. Connections to the SCADA system are normally created using some type of Enterprise System Bus, like BizTalk. The solution to interface the SCADA system will vary from implementation to implementation but the APIs provided by the SCADA system and described under section is 3.1.13 are normally used.

Examples of external applications that exchange data with the SCADA system are:

Enterprise Resource and Planning systems like SAP Work Order Management System Crew Dispatch Systems Call Centres Production Planning and Optimization system Enterprise Data Warehouses Market Systems Geographical Information Systems

The SCADA interfaces are normally quite complex in these cases but are still only parts of more multifaceted business processes. One of the major reasons for opening up the SCADA system to office networks during recent years is exactly this requirement in order to enable the automatic inclusion of SCADA information in such business processes.

3.1.16 Data engineering server

Defining the SCADA and process models for a utility requires a substantial effort because of the size of the process and the vast number of monitored objects. Normally this work is done on a dedicated server and special tools have been developed by the SCADA vendors to support the users in defining and maintaining these models efficiently. A Data Engineering server is normally based on a commercial relational database like Oracle and includes support for:

Definition of all SCADA objects (measurands and status information) Definition of all process objects like lines, breakers, isolators, buses,

transformers, generators, etc.


VIKING Consortium


Definition of connectivity, i.e. how the different process objects are statically connected

Definition of process pictures if not automatically created from objects and connectivity

Definition of control system pictures Definition of RTUs, protocols and Data Acquisition network Cross connection between SCADA objects and RTU I/O boards Definition of functional control parameters like alarm and event treatment Structures of the Historian and definition of data objects to be logged to the

Historian Definition of program control parameters Texts to enable translation between different languages

Various data input tool are provided for the Data Engineering users like:

Graphical Editors Forms or dialogue based input Import from legacy systems or from Geographical Information Systems (GIS)/

Network Information System (NIS)

The data is entered into the relational database using these tools and extensive plausibility checks are performed to check for data errors but also for consistency between objects. The reason for this is to avoid data errors after loading into the runtime SCADA systems which could cause program misbehaviour or power application non-convergence. Some vendors provide incremental loading of only new and changed objects from the Data Engineering server to the runtime system to avoid extensive loading times due to the big volumes of data. Versioning handling in the Data Engineering servers is normally provided to allow multiuser access and support for long and short term engineering sessions.

The new engineering data is entered either to the standby SCADA server where it can be further tested and verified with real process data or in secured database environment in the online system. Normally roll-back features are included of enable the user to go back to a previously functioning database.

Import from legacy systems are normally provided to secure that data, e.g. hardware connections in the RTUs, are transferred automatically to a new system. This secures that quality of data is not diminished by manual entries and thus avoids time consuming point testing in the new system. This type of import is only used during commissioning of a new system.

Many utilities prefer to maintain all its process data in a GIS system. In this case the data is maintained and verified in the GIS system and repeated import of new or changed data, e.g. on a daily basis, is required and must be supported from the SCADA system.

Some of the smaller SCADA systems do not have a Data Engineering server. In those cases the user will change the models and functional control parameters directly in the online, real-time database using the normal HMI Clients.

3.1.17 Data engineering client

Two alternative solutions exist for Data Engineering Clients. Either the normal HMI Clients includes a special mode for Data Engineering work or dedicated DE Clients are


VIKING Consortium


provided that interface the user to the relational database. In both cases special HMI features are implemented that ease the Data Engineering work.

3.1.18 Remote engineers’ workstation for substations

The functionality of RTUs is controlled by parameter settings. These parameters control a number of very important and critical local functions like protection settings and cross-referencing between protocol identity (object id) and channel number on the output boards of the RTUs.

RTU/SCS parameters can often be loaded from a central site to the local site using dial-in connections, e.g. modems or VPN connections over Internet. The benefit with remote parameter loading is, of course, that the remote sites do not have to be visited when reconfiguration of RTUs is required.

The workstations are normally placed on the office network with a dedicated connection to the RTU/SCS. Special applications are used to offline calculate, e.g. Protection settings. Basic data for RTU engineering can be often be imported from the Data Engineering server.

3.2 Data flows

This section identifies the data flows that are used between the different services as identified in Figure 7. The structure in the section follows the numbering of Figure 7 for ease of reading.

Some of the described data flows are essential and exist in all SCADA systems, in fact some flows, like Measurement from RTU to front-end and Commands to the process, are core functions of any SCADA system. Without these data flows SCADA has no meaning. Other flows are more infrequent and are dependent on the implemented functionality of a certain SCADA system, e.g. if no generation management functions exist in SCADA there will be no data flow from a Production Optimization system in the office environment.

Based on our vendor interviews we will for each separate data flow indicate how frequent it occurs in different types of SCADA systems to give a sense for its importance.

3.2.1 PMU Measurements

PMUs send process data, i.e. power flows and phase angels with a very accurate, GPS based time stamp, to the phasor data concentrator. The information volume transmitted is, due to very frequent measurements, substantially bigger than from normal RTUs and requires, therefore, more communication bandwidth. Applications using PMU data does not demand the same real-time performance as normal RTU protocols. Special PMU data protocols have been defined for this type communication, e.g. IEEE 1344-1995 or C37.118.


VIKING Consortium


Table 3 –Vendor’s estimates of how common the data flow “PMU Measurements” is in different environments.

(Merged cells mean that one estimate is given for the general case)

Environment Percent of installed systems

that has this data flow

Vendor Voltage level Region DMZ or

Single firewall

Isolated system

PSI >200 kV All 70%

PSI 110-200 kV All 40%

PSI <110 kV All 0%

Netcontrol <110 kV Europe 0% 0%

Siemens <110 kV All 0% 0%

ABB MicroScada <110 kV <110 kV 0%

ABB NM >200 kV US &

Australia 10% Not applicable3

ABB NM >200 kV Europe,

Middle east. Africa & Asia

5% 5%

3.2.2 Process measurements to control center

Measurements types coming from RTUs consist of Binary values (Indications), Analogue values and pulse counters. They are sent from the RTUs using specially designed RTU protocols. Previously such protocols were proprietary and unique for each SCADA and RTU vendor. The reason to use proprietary protocols was that the available communication bandwidths were very low, e.g. 200 bauds, and the requirement for safety very high which meant that many parity bits were required. The consequence of these circumstances is that the available bandwidths had to be used very efficiently (every bit counted) to reach reasonable response times and safety criteria. In many cases even bit synchronous protocols were used.

In the last decades more communication capacity, e.g. through the use of fibre technologies, has gradually been made available for power system communication. This fact combined with a strong requirement from the users for standards has resulted in standardization of RTU protocols. Today the standards IEC 60870-5-101 for point-to-point, byte oriented connections and IEC 60870-5-104 for TCP/IP based communication are common. Other standards, or de-facto standards, frequently used are DNP 3.0 and MODBUS.

Several types of polling schemes exist. The most common are:

Unbalanced, all data collected, i.e. polling initiated from front-end, all available data sent from RTU, cyclically scanned. Mostly used in old types of RTU protocols.

3 No Single firewall architectures exist in this environment.


VIKING Consortium


Unbalanced, report by exception, i.e. polling initiated from front-end, only changed data sent. Mostly used in point-to-point connections.

Balanced, report by exception, i.e. the RTU/SCS initiates sending when data is changed. Mostly used in TCP/IP protocols.

Response time requirements on this data flow are normally that indications (binary values) should be available at the front-end server in less than one second after the primary process object changed and analogue values should be available 2 to 10 seconds after a change more than its dead-band.

Table 4 –Vendor’s estimates of how common the data flow “Measurements” is in different environments.

Environment Percent of installed systems that has

this data flow

Vendor Voltage

level

Region DMZ or Single

firewall

Isolated system

All All All 100% 100%

3.2.3 SCADA sends commands to process

RTU protocols also support data to be sent from the SCADA system to the process. Supported data types in this case are binary outputs (commands) and analogue outputs (setpoints) which are received by the RTU, transmitted via the output boards to primary process objects. Some variants of these basic types are used for special purposes like raise/lower to step up generators or increase/decrease transformer tap changer. Binary commands are normally handled in two different modes, commands with check-back where a control signal is returned from the RTU before the command is executed and direct commands.

Commands and setpoints can be initiated from the operator via the HMI Clients, from predefined command sequences, form Inter-Centre Communication protocols or directly from applications. Some SCADA system supports commands via APIs from external sources.

Response time requirements are in the same range as Measurements but with higher priority. This means that a new command requests must be sent to the RTU or SCS in less than one second and will have priority on the communication line over the Measurement data.


VIKING Consortium


Table 5 –Vendor’s estimates of how common the data flow “SCADA to process” is in different environments.


this data flow

Vendor Voltage

level


firewall

Isolated system


3.2.4 Front-end data to real-time database

Measurements from the process are normally converted to a more convenient format and temporarily stored in the front-ends before sent to the SCADA server and stored in the real-time database. Several vendors use a proprietary protocol to send data from the front-ends to the SCADA server. The reason for using special protocols is that these protocols can support more services between SCADA server and front-ends than an RTU protocol. Examples of such services are database and parameter loading of the front-ends and control commands for reconfiguration of the front-end and data acquisition network. Other vendors use standardized RTU protocols like IEC 60870-5-104 which opens the possibility to mix front-ends and SCADA servers from different vendors.

The communication between front-ends and SCADA servers are normally done on a redundant Local Area Network (LAN) typically based on Ethernet, either on a dedicated LAN for process data or on the general SCADA LAN. The available bandwidth on the LAN must be bigger than the sum of all communication lines traffic to the RTUs.

If buffering is used between front-ends and SCADA servers to make the communication more efficient the maximum delay time is a few hundred milliseconds. When RTU protocols are used the received data is normally sent directly. In both cases the introduced delay times in this data flow in very small.

Table 6 –Vendor’s estimates of how common the data flow “front-end data to real-time database” is in different environments.


this data flow

Vendor Voltage

level


firewall

Isolated system


3.2.5 Phasor data sent to real-time database

PMU data from the phasor data concentrator is sent to the SCADA server in a similar way as RTUs send data to the SCADA server. A dedicated phasor application in the SCADA server receives this data without format or protocol conversions. Some vendors implement the phasor applications directly in the front-ends.


VIKING Consortium


Table 7 –Vendor’s estimates of how common the data flow “Phasor data sent to real-time database” is in different environments.


this data flow

Vendor Voltage

level


firewall

Isolated system


3.2.6 SCADA server sends command to process

If the operator has requested a command or a setpoint to be sent to the process using process displays and HMI dialogues this request will be sent to the SCADA server from the HMI Client, see section 3.1.9. The SCADA servers will convert this request to an RTU protocol format and send it to the front-ends which, in its turn, will send the request to the RTU. The SCADA server will use either the proprietary protocol described above or a standard RTU protocol to send the front-end dependent on the implementation. The communication media is, as for the previously described data flow, the process LAN or the SCADA LAN.


this data flow

Vendor Voltage

level


firewall

Isolated system


3.2.7 Data sent to emergency centre(s)

Many SCADA systems have an emergency centre on a geographically distant location. This emergency system can be a fully equipped SCADA system that is prepared to take over operation of the process if a catastrophic failure occurs at the main control centre, e.g. a comprehensive fire.

The emergency system must be kept as updated as possible with information of the latest process state to be able to be able to fulfil its tasks when operational responsibility is taken over. Process measurement can be acquired with a general interrogation to the RTU from the new site but a lot of process data and process related data like tags are only manually entered by the operators because of lacking process measurement. Also this type of data must be available at the emergency centre in order to correctly take over operation. Also historic and planning data in the Historian are important for the operation and must, therefore, also be synchronized with the emergency centre.


VIKING Consortium


The emergency centre is normally from the same vendor as the main centre. This means that proprietary, database-to-database protocols are used to keep the Emergency Centre synchronized. This will normally be the most efficient solution. If the historian is built on commercial relational database these database system often provides synchronization mechanism that are used to keep the emergency historian updated.

Basically two types of synchronization mechanisms are used, continuously updated and cyclically updated. Continuous updating means that all updates performed in the main system databases are sent to the emergency centre as soon as they occur and cyclic updates means that the two system are synchronized at regular intervals, e.g. once per 24 hours. The continuous update requires more communication capacity and this is normally the decision factor for which method is selected.

Sometimes the workstations in the emergency centre are used as remote HMI clients connected to the main system to have second control room, e.g. to supervise and control local parts of the network.

Normally dedicated communication lines are used to communicate between main centre and emergency centre. Communication speeds in the order of 10 Mbit/s is required for continuous updates and to support remote HMI Clients.

Table 8 –Vendor’s estimates of how common the data flow “Data sent to emergency centre” is in different environments.





Single firewall

Isolated system

PSI All Europe, Asia,

Africa 100%

Siemens Distribution All 0% (or close to)

Siemens Transmission and Sub-transmission

All In bigger utilities

ABB MicroScada Distribution Europe, the Middle East

and Asia <5%

ABB NM Transmission US and

Australia 30-40% Not applicable4

ABB NM Transmission

Europe, the Middle East, Africa and

Asia

40% 20%

ABB NM Sub-transmission


Asia

10% 10%



VIKING Consortium


ABB NM Distribution


Asia

10% 10%

Netcontrol Distribution Sweden and

Finland 5% 5%

3.2.8 Commands and status

In many cases the control centre will communicate with other control centres. Examples of such communication could be with regional control centres under a National Control Centre or with the control centre in a neighbouring country. Commonly used protocols for this type of communication are TASE.1 (ELCOM) and TASE.2 (ICCP). The actual communication with the other centre is handled in a dedicated Inter control centre communication server. This server needs access data from the real-time database that are converted to the used inter centre protocol.

The communication between SCADA server and inter control centre communication server is done within the control center zone and uses internal database protocols.

The usage of this data flow is coupled to the use of inter control centre communication, these two always coexists. See Table 10 in section 3.2.9 for figures on how common this data flow is.

3.2.9 Process data to/from other control centres

It is, as described in the previous section, common that control centre communicates with each other. Special standardized Inter centre communication protocols have been developed for this purpose like TASE.1 (ELCOM) and TASE.2 (ICCP). Both these protocols support more or less the same functionality so we can use ICCP as an example of included features.

ICCP functions are partitioned into a number of 'Conformance Blocks', which describe various capabilities that can be supported, see Table 9 below.

Table 9 –Conformance blocks in ICPP.

Block Function

1 Basic Services

Basic SCADA data exchange (periodic transfer and transfer on request). Accumulators, Indications, Measurements.

2 Extended Data Set Condition Monitoring

Complex SCADA data exchange (object change transfer, Report by Exception, integrity time-out transfer, etc.).

3 Blocked Transfers

'Blocked' (Compressed) data.

4 Information Message

Operator and other user-defined messages, such as alarms.

5 Device Control

Remote Device Control and Remote Control of Local Devices.


VIKING Consortium


7 Events

Event reporting, for completion of supervisory control requests.

8 Accounts

Interchange schedules and other account data.

Several types of communication media can be used for this communication but most common today is TCP/IP based Wide Area Networks. Response times are normally not as critical as for the RTU to front-end data flow but requirements still lie in the area of a few seconds.

Secure communication with encrypted traffic is often used for this communication. Several commercial products are available for this purpose but, in some cases, only on supported on certain operating system. For some vendors this requires the addition of an Inter Centre front-end.

Table 10 –Vendor’s estimates of how common the data flow “Process data to/from other control centers” is in different environments.





Single firewall

Isolated system

PSI Transmission Europe, Asia,

Africa 80%

(only 20% send commands)

PSI Distribution Europe, Asia,

Africa 60%

Siemens Distribution All 10%


All 90-100%


and Asia 10-15%



ABB NM Transmission


Asia

50% 50%

ABB NM Sub-transmission


Asia

30% 20%

ABB NM Distribution Europe, the 20% 10%



VIKING Consortium


Middle East, Africa and

Asia


Finland 5% 5%

3.2.10 Operator commands to process and manual changes and mark-ups of data

The operators can request a command or a setpoint to be sent to the process. They will select an object on a process display and invoke a dialogue for command or setpoints. In a two step command (check-back) first the binary output channel of the RTU will be selected and then the actual execution will be performed in the second, operator step. Single step commands are ´directly executed.

The following types of command outputs are normally included in all SCADA systems:

Check-back commands Direct commands Setpoints Raise/Lower (stepwise execution without reselection)

The data flow is normally an internal between HMI client and SCADA server using the redundant SCADA LAN and standardized protocols are not used for this data flow. The communication protocol is unique for each vendor.

The operator clients also enter and mark data in the SCADA server. Not all process values are measured and collected via RTUs. Many values have to be manually entered by the operators after receiving the information of the process state in some other way, e.g. over phone from maintenance crews. All SCADA systems, therefore, support manual entry of data directly into the real-time database from the operator HMI clients. These manually entered values are marked as manual but in other aspects treated in the same way as the automatically acquired process data.

With similar dialogues as manual entry the operators can block process objects for data acquisition or alarming if some error occurs or tag process objects to indicate, for example, that a maintenance crew is at work on a certain object.

These manual changes and mark-ups are entered into the real-time database using internal protocols between the operator HMI client and SCADA server. In configurations with the real-time database in the HMI workstation this is a purely internal inter-task communication. If a central real-time database is used the communication goes over the SCADA zone.


VIKING Consortium


Table 11 –Vendor’s estimates of how common the data flow “Commands to process and manual changes and mark-ups of data” is in different environments.


this data flow

Vendor Voltage

level


firewall

Isolated system


3.2.11 Operator views and alarms sent to HMI

In SCADA systems process information is presented to the operators in many different ways. The most important are the single line displays on network and station level and Event and Alarm lists.

It is important to understand that SCADA displays are “live” updated, or animated, whenever a change is detected in the process data by the SCADA server. To make the update efficient only the symbol or value of the changed objects is updated and redrawn to a new state, not the whole displays. Under process disturbances this will mean that several thousands of symbols in single line displays must be updated each second.

Also alarm and event lists are updated when process data change. These lists are collection of events in chronological order. Event lists register all events from the process, all operator actions and all control system events. Alarm lists displays and handles only those high priority events that require operation acknowledgement.

In SCADA systems with the real-time database implemented in the HMI workstations the real-time data to present in single line displays are fetched locally. In systems with a central database the real-time data is fetch over the control system LAN. The protocols used are internal and no standard is used for transferring real-time data to the HMI Clients.

This data flow is extremely performance sensitive. Especially during process disturbance situations vast amount data must transferred from the real-time database to the picture presentation programs in a very short time. It is vital that picture call-up times are short, normally less than one second is required, in order to enable for the operators to work efficiently and always have access to the latest process information. Process display can be very large including thousands of real-time process variables in a single display.


VIKING Consortium


Table 12 –Vendor’s estimates of how common the data flow “Commands Operator views and alarms” is in different environments.


this data flow

Vendor Voltage

level


firewall

Isolated system


3.2.12 A server hosts several HMIs

A common solution to support remote HMI clients is to implement a HMI server which includes one or several normal operator HMI clients that each will support one remote workstation. Several solutions for remote desktop presentation exist that do not require any special installation of special purpose software on the remote workstation, for example Windows Terminal Services or VNC solutions.

The communication between each HMI Client in the HMI server and the SCADA server is identical to the communication from the SCADA server to control room HMI Clients. This data flow symbolizes the data flow between the server hosting HMI clients and the clients running in it.

Table 13 –Vendor’s estimates of how common the data flow “Server hosts several HMIs” is in different environments.

3.2.13 Terminal services

Many SCADA systems support, as mentioned previously, remote operator HMI clients, i.e. HMI clients accessed from outside of the normal control room. One common solution to achieve this functionality is to use virtual desktop technologies such as Citrix, VNC or Windows Terminal Services. This data flow represents the data flowing in between the remote desktop client and the desktop server (in this document referred to as the HMI server).

Normally some degradation of HMI response times are accepted for remote HMI clients. The remote desktop will otherwise have the functionality and authority limitations as defined for the HMI Clients on the HMI server. In Table 14 it is noted how common it is that remote clients (i.e. those running in the HMI server or equivalent) are only-have read-only access. This is number only known for some vendors.


this data flow

Vendor Voltage

level Region

DMZ or Single

firewall

Isolated system

All All All 100% if the HMI server is used


VIKING Consortium


Table 14 – Vendor’s estimates of how common the data flow “Virtual, remote desktops” is in different environments.





Single firewall

Isolated system


Africa 80%



All 90-100%


and Asia

5% read only 10-15%


Australia 70-80% Not applicable6

ABB NM Transmission Europe, the Middle East,

Africa and Asia

40% Read only

60% Normal

20% Read only

50 % Normal

ABB NM Sub-transmission Europe, the Middle East,

Africa and Asia

30% Read only

60% Normal

10% Read only

50 % Normal

ABB NM Distribution Europe, the Middle East,

Africa and Asia

30% Read only

70% Normal

10% Read only

70 % Normal


Finland 100% 100%

3.2.14 Mail websites etc

In some SCADA system the users install Internet and E-mail functionality on the same workstations as the operator HMI clients. This enables the operators to use E-mail and surf on Internet directly from their work places in the control room. Standard protocols for these services are used. Automatic virus protection is normally applied in these situations. Utilities also normally have a protected internal office network where proxy servers are placed.



VIKING Consortium







Single firewall

Isolated system

PSI Transmission Distribution

Europe, Asia, Africa

0%

Siemens Distribution All Have not seen this


All Do exists


and Asia 30%



ABB NM Transmission

Sub-transmission Distribution

Europe, the Middle East,

Africa and Asia 0% 0%


Finland Probably on workstations outside the control center, e.g. portable

HMI clients

3.2.15 Read operator views and change parameters of power applications

The HMI Clients can also fetch data for presentation from the Power Application Server or they can change parameters in this server. The presented data are typically results from the applications that are mixed with real-time data from the process in single line displays or the presentation of power application control parameter. The HMI Clients can manually change these parameters and, in this way, control the execution of the applications, e.g. convergence criteria.

The communication is performed on the control system zone. The communication protocol for this data flow is normally an internal database format similar or identical to the communication between SCADA server and operator HMI clients. Performance requirement are similar to those for SCADA to HMI communication in order to meet requirements on picture call-up times but volumes of data is lower.



VIKING Consortium


Table 16 –Vendor’s estimates of how common the data flow “Read Operator views and change parameters” is in different environments.

Environment Percent of installed

systems that has this data flow


Single firewall

Isolated system

All but Netcontrol

All All

100% 100%


Finland 0% (do not have a own

Power Application server)

3.2.16 Process data to power applications

The power applications need real-time process data to calculate and update their dynamic models of the power network. This data is read from the SCADA Server real-time database using the available database access methods which could be proprietary calls or standardized APIs. The data flow goes over the control system zone if the Power Application server is not integrated in the SCADA server on one pair of redundant computers (such configurations are sometimes used for smaller networks to keep down hardware cost). In this case the data is read locally from the real-time database.

Some vendors write back the results of the power applications to the SCADA server and the access from the HMI Clients is then identical to that of the real-time data. The same protocol as for reading process data to the power application server is used for writing result back to the SCADA server.

Performance for this data flow is critical in order to meet required response times. It is a common requirement that a new State Estimator base case shall be ready in less than 5 seconds after a single process data change. The data volumes are also quite high since whole snaps shots of the process state is normally send to the power application server from the real-time database. It is, however, possible to group bigger volumes of data to make the communication on the control system LAN more efficient with low overheads.

Table 17 –Vendor’s estimates of how common the data flow “Process data to power applications” is in different environments.




Single firewall

Isolated system


3.2.17 Set points to generators

Automatic Generation Control (AGC) is one of the few applications that work as a close loop application in SCADA systems. Traditionally AGC uses deviations in electrical


VIKING Consortium


frequency compared to nominal value to calculate the required generation increase or decrease in order to return the frequency to the nominal value, e.g. 50 Hz. AGC controls generation units by automatically sending new setpoints to them. In modern SCADA systems that work on deregulated markets AGC is normally used to meet predefined schedules and keep power exchange over tie-line on scheduled values. These control schemes are also implemented by sending automatic setpoints to the generation units, i.e. from data flow point of view are these two operational modes identical.

AGC sends setpoint to the SCADA server in the same way as HMI Clients initiates sending of setpoints to the process. The SCADA servers forward the requested setpoints via the front-end and the RTUs to the actual generation units. The communication between AGC and SCADA servers uses the control system zone. Internal protocols and services are used. Performance requirements are high because of the close loop characteristic of AGC but the data volume is very small.

Table 18 –Vendor’s estimates of how common the data flow “Process data to power applications” is in different environments.




Single firewall

Isolated system

All All All When AGC is used, which is

seldom.

3.2.18 Historian used to update application server data

External applications can use the Historian to transfer data to the power applications. A typical example is that optimal generation schedules that are produced by a hydro-thermal optimization application working on the office zone. These schedules consist of time series of data that are stored in the Historian by the external application using Historian APIs and are used by the power applications, e.g. by AGC as input for generation control.

The power applications read the data from the Historian when needed using database access interfaces. The communication goes over the control system zone. For this type of access standard database interfaces like SQL or ODBC are commonly used provided that the Historian uses a relational database. No special performance requirements exist although the data volumes are medium to large.


VIKING Consortium







Single firewall

Isolated system


Africa 100%

Siemens Distribution All Sometimes


All 10-20%


and Asia 0%


Australia <15%








Finland In some installation maybe

3.2.19 Historical data sent to historian

One of the main purposes with the Historian is to archive all data and events that are handled by the SCADA system. The SCADA servers will log all incoming process data, manual entries, SCADA calculation results, and events and alarms and store this in dedicated buffers. The buffers are sent to the Historian on a frequent cycle, e.g. every 2 seconds, or when the sending buffers are full. This means that the data in the Historian is close to real-time.

The Historian receives the logged data and inserts the data into its data structures. Normally data is logged in the SCADA server with different time resolutions, e.g. every 10 seconds, every minute, every hour, etc., and old data is overwritten after a defined time. The archiving times are dependent on available disk space and are typically in the range of a few months up to a few years. Most Historians support long-term archiving on tapes or DVDs and in this way realizing virtually unlimited archiving times.

It is easy to understand that the data volumes from the SCADA server to the Historian are very large. The control system zone is used for the communication and performance


VIKING Consortium


optimized using internal protocols to secure that all data can be transferred in the times available between consecutive logging intervals.


this data flow

Vendor Voltage

level


firewall

Isolated system


3.2.20 External programs using API in the SCADA server

Some SCADA vendors provide the user with an extensive set of Application Programming Interfaces (APIs) to enable the users to write their own applications that uses real-time data and services of the SCADA systems.

These APIs can open the possibility for the users:

to read and write data into the real-time database including support for SQL and ODBC

to use OPC connections to use the inter-task communication system to send commands and setpoints to the process to use conversion routines and time services

The APIs are normally connected to the SCADA system authority system to limit the accessibility of certain critical SCADA functions and data.

The APIs are implemented to work in a heterogeneous environment, i.e. the applications can work in another computer architecture with different operating systems and formats that the SCADA system. The APIs are supported for many programming languages and environments like C, Visual Basic and .Net

The data flow between the external applications and the SCADA server can go directly from the office zone to the control system zone, especially when certain system services, e.g. sending of commands, are invoked. If a real-time database replica exists in the DMZ the APIs for data access will use this replica.

Neither data volumes nor performance are normally critical for this data flow because the nature of the external applications are seldom time critical.


VIKING Consortium


Table 20 – Vendor’s estimates of how common the data flow “External programs using API in the SCADA server” is in different environments.




Vendor Voltage level Region DMZ or Single

firewall Isolated system


Africa 0%

Siemens Distribution All Customers require APIs, but it is

unknown how they are used


All 0%


and Asia 5-10%


Australia 10%








Finland <60-70%

3.2.21 External programs using APIs in the historian

The APIs described in the previous section normally also support access to the Historian to enable for external applications to read and write historical and planning data. In this case the APIs are often extension of SQL and ODBC to ease the access of time series of data. In Historians built on proprietary data structures for archiving of data other types of APIs are provided.

In this case the access from the external applications often use a replica of the Historian placed in the DMZ.


VIKING Consortium









Africa 100%



All 100%


and Asia 100%


Australia A requirement

in 100% of procurements








Finland 100% if a DMS system is used,

otherwise 0%

3.2.22 Operator HMIs interface the historian

The normal operator HMI Clients must be able to present data from the historian in the process displays to the operators. Typical examples of such displays are curves and reports. The operator HMI Client will access this type of data directly from the historian using the database access methods that the Historian provides. The data flow between historian and operator HMI client will go over the control system zone and is time critical and includes high data volumes, e.g. when presentation a curve diagram with tens of curves each containing several hundreds of time values. The response time requirement for such curve displays is normally less than 2 seconds.


VIKING Consortium


Table 22 – Vendor’s estimates of how common the data flow “Operator HMIs interface the historian” is in different environments.


this data flow

Vendor Voltage

level


firewall

Isolated system


3.2.23 Web browsers used to access the historian

Some vendors a web Client dedicated for the historian. These clients enable the users, normally not control room operators, to query the Historian and to build reports. Many commercial database provides like Oracle includes such clients in the product suite and they simply adapted to the data structures of a specific Historian.

The web Client users are normally on the office network and use the historian replica in the DMZ to access data. Performance requirement are less than for control room operators and the data volumes are normally small.

Table 23 – Vendor’s estimates of how common the data flow Web client interface the historian” is in different environments.







Africa 40%



All 100%


and Asia 0%


Australia A requirement

in 100% of procurements






VIKING Consortium





Finland 100%

3.2.24 The historians API is used by office desktops’ applications

Some vendors also provide already prepared interfaces from office applications to the Historian. The most typical example is perhaps to interface MSExcel directly to the Historian by providing .dll scripts that is linked to MSExcel. This makes it possible for the users by using internal dialogues in MSExcel to configure permanent or ad-hoc reports in a convenient way using available object identities directly from the Historian.

The office application users are normally on the office network and use the Historian replica in the DMZ to access data. Performance requirement are less than for control room operators and the data volumes are normally small.

The web Client users are normally on the office network and use the Historian replica in the DMZ to access data. Performance requirement are less than for control room operators and the data volumes are normally small.

Table 24 – Vendor’s estimates of how common the data flow “The historians API is used by office desktops to extract data” is in different environments.







Africa Possible in 100% of installations

Siemens Distribution All <5%


All Not recognized


and Asia 0%


Australia 100%








VIKING Consortium



Finland 50%

3.2.25 External programs interface the application server directly

External applications in the office network are sometimes used to interface the power application servers directly. Examples of interfaces to such applications are:

Call centres exchange trouble calls and restoration information with Outage Management functions in DMS

Work Order Systems exchange information with Switch Order Management and Crew Management

SAP system exchange information with Crew Management Production Optimization applications exchange schedules with Generation

Control (often done over Historian) Market system giving schedules for production

Smaller SCADA systems often use commercial power applications executing on the office network. These applications will normally have some interfaces to a subset of power applications executing in the SCADA system.

These interfaces are normally solved individually on a case by case basis using available technologies in the SCADA system and in the external applications. If the SCADA system provides a set of APIs these are used to solve these interfaces.

The resulting data flow very often goes directly between the external office applications and the internal power applications. Performance requirements and data volumes vary dependent on the individual interface.








Africa Possible in 100% of installations



All Not recognized


and Asia 0%


Australia 100%




VIKING Consortium







Finland 50%

3.2.26 Viewing and changing engineering data

To keep the SCADA and process models updated to reflect the actual as-built status of the network requires substantial effort and good tools. Some vendors, especially those that provide bigger system, provide a special engineering environment for this purpose. Such an environment must be designed for multi-user and support version handling to be able to deal with short and long-term engineering sessions. After one session is finished and approved it is loaded into the operational SCADA system

Engineering work is commonly done from the office or back-office environment. To support multi-user the engineering environment is normally solved with a client-server technology, i.e. a single server supports many clients. Many of the engineering environments are built on commercial databases like Oracle and the underlying relational database technology is used to build the DE Clients and defines the communication and data flow. Other vendors use the normal control room HMI Clients to interface the engineering system.

In some case, especially for smaller SCADA vendors, the engineering work is done from the control room HMI Clients directly in the SCADA server.

Table 26 –Vendor’s estimates of how common the data flow “Viewing and changing engineering data” is in different environments.


this data flow

Vendor Voltage

level


firewall

Isolated system


3.2.27 NIS/GIS applications updates engineering data

Many utilities wish to maintain its process data in one place only. They will often already possess a Geographical Information System (GIS) or a Network Information System (NIS) that is used to document all their assets and is also used for planning of network extensions. It is natural to use this data to import to the SCADA system to avoid double and erroneous prone maintenance. Such an import will normally be done to the Data Engineering system and not to the operational system.

This type of import is a rather complicated exercise since the models in the GIS/NIS system often differs from those used by SCADA (SCADA uses simplified models). Also the import should preferably be incremental, i.e. only added or changed data should be imported, to keep down import times. Several engineering sessions are often, for


VIKING Consortium


handling reasons, grouped together and the import is done at regular intervals, e.g. once per 24 hours

The GIS/NIS system is normally placed in the office network and the Data Engineering server in the control system zone or in the DMZ. The data volumes can be huge but the imports are, as stated, above relatively infrequent.

Table 27 – Vendor’s estimates of how common the data flow “NIS/GIS applications update” is in different environments.


3.2.28 Data interchange with other DE systems

In some SCADA systems an engineering data exchange is implemented. This enables the engineering users to reuse already defined network parts, e.g. from a neighbouring utility. An XML file exchange built on the Common Information Model (CIM) for electrical networks is normally used for this exchange. The exchange takes place between the two engineering environments. The file sizes are huge but the exchange very infrequent.





PSI Transmission Europe, Asia, Africa

0%

PSI Sub-transmission Europe, Asia,

Africa 60%

PSI Distribution Europe, Asia,

Africa 80%

Siemens Distribution All Very seldom


All 50%


and Asia Files are transferred in some cases


Australia 100%








Finland 0%


VIKING Consortium


From the interview it seems that this exchange is up to now only practised on the US market.

Table 28 – Vendor’s estimates of how common the data flow “NIS/GIS applications update” is in different environments.


3.2.29 Define structure, views etc

When a session has been finished in the data engineering server it must be loaded to the operational SCADA system. Such a session might impact on:

the SCADA server to build the SCADA model the Power application server to build process models the Historian to build data structures the HMI Client to build static picture descriptions and, in the case of distributed

SCADA functionality, SCADA models

Specially prepared load files are created in the engineering environment and sent to update programs in the respective servers using the control system zone. The format of this load files are internal and adapted to give minimal loading times. The performance






Africa Unusual

Siemens Distribution All In larger utilities


All Common in the US, but not in Europe


and Asia Unknown


Australia 15-20% can,

<10% do








Finland 0%


VIKING Consortium


requirements are high and the data volumes varies from a few updated objects to total loading of the entire SCADA system databases.

Table 29 – Vendor’s estimates of how common the data flow “Define structure, views etc” is in different environments.


this data flow

Vendor Voltage

level


firewall

Isolated system


3.2.30 Disturbance records sent for analysis

Some RTUs and, and especially SCSes, can locally collect and store disturbance data that is recorded during power system disturbances. This data is a series of very accurately time stamped process values that is later used to analyse the disturbance.

Disturbance data is normally can be sent as files to the front-ends which will unpack this data and send it to special visualization and analyse applications. These applications are often used on dedicated workstations or PCs on the office network. Some vendors have integrated these applications in the SCADA server and, in these cases, the files are sent to the SCADA server.

The data volumes are moderate to big and performance requirements are low.

Table 30 – Vendor’s estimates of how common the data flow “Disturbance records sent for analysis” is in different environments.


Environment

Percent of installed systems that has this data flow



PSI All All Not asked

Siemens All All Not asked


and Asia Common

ABB NM All All Not asked


Finland 90%


VIKING Consortium


3.2.31 Settings and parameters for RTU and substation equipment

RTU/SCS parameters can often be loaded from a central site to the local site using dial-in connections, e.g. modems or VPN connections over Internet. The benefit with remote parameter loading is, of course, that the remote sites do not have to be visited when reconfiguration of RTUs/SCSes is required. The remotely changed parameters are loaded and used in the operational environment of the RTU/SCS without further testing.

Some vendors support the inclusion of RTU/SCS parameters as files in the normal RTU protocols and use the already existing communication lines to the RTU/SCS site. The RTU/SCS then includes programs for unpacking and loading of the received parameters.

Performance of this data flow is not critical. Remote loading will in any case be much faster than visiting all remote sites. Data volumes are moderate.

Table 31 – Vendor’s estimates of how common the data flow “Settings and parameters for RTU and substation equipment” is in different environments.







Africa 80% of the major stations



All Not common

ABB ABB MicroScada

Distribution Europe, the Middle East

and Asia 30%

ABB NM All All Not known Not known


Finland 40% if IP is used for substation

communication


VIKING Consortium


4 Control center architecture patterns

This chapter will outline the different variants identified through the interviews with SCADA system vendors. This chapter will first describe how services are placed in different zones through a number of basic descriptions representing the most common architecture. Together with these basic architectures a list of deviations are presented. These deviations fall under the two categories:

Services that are added or subtracted from the basic architecture. Services that are bundled together into one tightly coupled software module.

The first category includes the use special services to exchange data over network zones. The second does for example include the cases where the HMI software for both data engineering and operators is one component. These two categories of variation are described in section 4.3.

4.1 Zones and placement of services

This section will detail the variants that have been identified for zones and the placement of services. The in relationship to the architectural metamodel used in this study this section will instantiate the element “Zone” and relationship between “Service” and “Zone” called “Belong to”, cf. Figure 8.

Writes to

Service

Zone

Dataflow

1..*

1..1

1..22..2

Goes between

Belong to

0..*0..*

0..*

0..*

Can join

Figure 8 – Section 4.1 in relation to the architectural metamodel.

Table 32 describes the six different patterns identified throughout this study. The first three of these represent a division of services in a zone architecture. The three are: DMZ separating the control centre from other zone, single firewall separating the control centre from other zones, and isolated control centre. These three architecture patterns are mutually exclusive. Three variants can exist within these zone architectures. A


VIKING Consortium


separate front-end can be used, a back-office zone can be used and a quality assurance zone can be used. These six patterns and how services are placed in them are described in detail below in this chapter. Table 32 provides an overview the frequency of occurrence of the respective architectures and variants.

Table 32 –Zones and placement of services. (A dash denotes that this question was not raised during the interview)

Environment Zone architecture Variant

Vendor Voltage

level Region DMZ Single


front-end DMZ

Back-office zone

QA zone

PSI >200 kV Europe & ME 80 % 20% 0% In some cases

Often In some cases

PSI 110-200 kV Europe & ME 80% 20% 0% In some cases

Often In some cases

PSI 110-200 kV Africa 80% 20% 0% In some cases

Often In some cases

PSI <110 kV Europe & ME 80% 20% 0% In some cases

Often In some cases

PSI <110 kV Europe & ME 80% 20% 0% In some cases

Often In some cases

Netcontrol <110 kV Europe & ME 30% 60%8 10% 0% 0% 0%

Siemens >200 kV /

110-200 kV Europe & ME <5% 25-40% 60-70%

Maybe in some cases

Maybe in some cases

In some cases

Siemens >200 kV /

110-200 kV Africa 0% 10% 90%

Maybe in some cases

- No

Siemens <110 kV All Never seen one

45% 55% Maybe

in some cases

- In some cases

ABB Micro SCADA

<110 kV Europe & ME 33% 33% 33% In some cases

- No

ABB Micro SCADA

<110 kV Asia 10-15% 40-50% 40-50% In some cases

- No

ABB NM >200 kV America & Australia

Almost 100 %

~0% ~0% Almost 100 %

- Most

systems

ABB NM >200 kV Europe & ME 25% 50% 25% No - In some cases

ABB NM >200 kV Asia 10% 80% 10% No - In some cases

ABB NM 110-200 kV Europe & ME 10% 80% 10% No - In some cases

8 Of these 60 % roughly one third only has this architecture to allow the supplier to dial-in during debugging.


VIKING Consortium


ABB NM <110 kV America & Australia

50% 50% 0% Almost 100 %

- In some cases

ABB NM <110 kV Europe & ME 10% 80% 10% No - In some cases

ABB NM <110 kV Asia 10% 80% 10% No - In some cases

4.1.1 Demilitarized zone architecture

In the demilitarized zone architecture there is a one zone separating the control center zone form the office zone – the demilitarized zone. All vendors have some systems installed according to this architecture, but there are a number of variations that exists within these installations.

Figure 9 depict the most common placement of services in this zone architecture. As can be seen from this figure the DMZ contains a historian replica, servers for inter-control center communication, and servers for remote HMIs. The control center holds Front-end and Phasor data concentrators, the SCADA server, Power application server, the Historian, Data engineering server and the HMI client see section 3.1.


VIKING Consortium


Figure 9 – The demilitarized zone architecture.

Figure 9 does not depict the data flow between the services in this architecture since this would decrease readability of the figure. The majority of these data flows can however be deduced from the general description in section 3.2. This description does, for example, say that data engineering stations exchange data with the data engineering server. Hence, a data flow does in some cases go between the engineering stations in the office and the data engineering server in the control center. How often this is the case is in general is detailed in section 0. Some exceptions and amendments to the general data flows do however apply:

The historian replica synchronizes its data with the historian in the control center.

Access from the office and internet (i.e. the data flow “historian used to update power application server”) to the historian goes to the historian replica, and not directly into the control center.

When the historian is used to update the power application server the historian replica is used for this.


VIKING Consortium


The HMI server hosts operator HMI clients in the DMZ, not the ones in the control center.

It should be noted that the DMZ architecture does not always implement a DMZ in the strict sense. With a DMZ, all traffic should terminate in the DMZ and no traffic should go directly through it [3]. As can be deduced from Figure 9 this is not always the case for the DMZ between control centers. For example will engineering station s in the office access the engineering server in the control center, office users’ computers sometimes access control centers system’s API directly, and data sent from office servers to the data engineering server can in some cases pass directly through the DMZ.

The demilitarized zone architecture is the architecture that most new SCADA systems use. It is also the zone architecture in which most variation has been found. Table 34 list the deviations from Figure 9 that has been identified throughout the interview process. The deviations listed here are only those that related to the placement of services or how zones can join each other. This table also detail the section of this report where this deviation is further described.

Table 33 – Deviations within the demilitarized zone architecture.

# Vendor(s) Deviation Described in section

1 ABB NM Use a SCADA replica in the DMZ. 4.2.4

2 PSI HMI server gives access to Operator station in the control center zone.

4.2.2

3 Netcontrol, ABB MicroScada

No separate HMI server. 4.2.3

4 ABB Network Manager

Has a separate (thin) client for remote access. 4.2.1

5 ABB MicroScada, Netcontrol

An operator client is located outside the office and users “dial in” to the control center zone to use this one.

-

6 ABB NM Use a inter control center communication front end. 4.2.6

7 Netcontrol External power application server. 4.2.7

8 PSI An interface server manages all data exchange between office systems and the control center.

4.2.5

9 PSI No engineering stations in the office. They are instead in a back-office zone.

4.1.4.2

10 ABB NM, Siemens, PSI, Netcontrol

Front ends are placed in a special DMZ. 4.1.4.1

11 ABB NM, PSI, Siemens

A quality assurance zone exists. 4.1.4.3

12 All Concentrators are placed in substations. -

13 All Application software for the office (e.g. mail and web surfing) is executed in the control center zone.

-

14 ABB NM, ABB MicroScada, Siemens

Place an Engineering Station (HMI) in the DMZ. -


VIKING Consortium


15 PSI Data Engineering Server is placed in DMZ -

The last four deviations are not further described in this report as their impact in terms of data flows is apparent. Number 12 is partly described in section 3.1 and means that another node is added to the substation communication – a server which aggregates several RTUs into one node. Number 13 means that data is exchanged between the office application software running in the control center and internet/office services. Number 14 will mean that data engineering stations are placed in the DMZ and can be used to access the engineering server. Deviation 15 is sometimes used by PSI and means that the data engineering server is placed in the DMZ instead of the control center zone.

When it comes to other networks joining the control center zone this figure depict how suppliers can dial in to the control center. This appears to be common phenomenon. Siemens, PSI, Netcontrol and ABB NM in the American market all estimate that this is possible in more than 90 percent of their installed base. ABB NM for the rest of the world assesses this number to around 70-80 percent and ABB MicroScada assess it to about 40 percent. It should however be noted that dialling in often requires that the control center staff changes parameters in the firewalls or connect certain wires physically.

Most operational SCADA system provides a possibility for the original vendor to dial-in into the SCADA system for debugging purposes and to introduce patches. A dial-in connection will give the vendor access to all functions and data of the SCADA system even full access to the operating systems. It is a direct access path to the operational SCADA system. This type of connection is especially used when serious errors are detected in the SCADA system that prohibits the electrical network operation and needs immediate correction. If the utility has a Support and Maintenance contract with the SCADA vendor such a dial in function is normally part of the contract.

4.1.2 Single firewall architecture

In this architecture there are only two zones and the control center is not separated from the office with a DMZ. Instead these two zones interface each other directly with a single firewall. Most vendors state that this architecture is not longer used for new installations. However, it is still relatively common in the installed base of SCADA systems. The percent of system with this architecture ranging from a small percentage for ABB NM is the US market to about half of the installed base on the distribution level for ABB MicroSCADA, Siemens and Netcontrol.

Figure 10 illustrate the most common placement of services in this architecture. As can be understood by studying this figure this architecture involves a direct data exchange between the control center services and other services. For example, use of historical data mean that data is read directly from the control center zone. It should however be noted that some SCADA system installations only interface the office network to allow suppliers to connect when needed. For instance, Netcontrol states that about 20 percent of their system installations use this architecture although no data is exchanged with the office. The only reason is to allow vendors to access the SCADA system for debugging purposes.


VIKING Consortium


Office Control Center

Substation

Supplier



Office Servers and power applications located outside the control center

SCADA Server

Office user's computer (e.g.

spreadseet applications)

RTU


Phasor measurement unit

Inter control center communication server

Web browser

Front End

Operator HMI Client

Historian


Control center power application Server

DMS(Application

Server)

GMS (Application

Server)

EMS(Application

Server)

Remote client (e.g. Cirtix client)

HMI Server (e.g. VNC, Terminal

Services or web interface)

Figure 10 – The two zone architecture.

Just as for the DMZ architecture a number of deviations the basic architecture has been identified. These are listed in Table 34 and are a subset of those deviations that apply to the DMZ architecture.

Table 34 – Deviations within the zone architecture with two zones.


1 Netcontrol, ABB MicroScada

No separate HMI server 4.2.3

2 ABB NM Has a separate (thin) client for remote access. 4.2.1


4 Netcontrol External power applications server. 4.2.7


VIKING Consortium


5 PSI No engineering stations in the office. They are instead in a back-office zone.

4.1.4.2



7 ABB NM, PSI, Siemens

A quality assurance zone (and zone) exists. 4.1.4.3


9 All Application software for the office (e.g. mail and web surfing) is executed in the control center zone.

-

The last two deviations are briefly described in section 4.1.1. The other seven are described in the sections listed in Table 34. The estimates of how often suppliers can dial in is the same for this architecture as for the DMZ architecture.

4.1.3 Isolated zone architecture

The third zone architecture is the one where the control center is not at all connected to the office or the internet. Although the vast majority of new systems has some connectivity to the office and other external systems this architecture is still common in the installed base of SCADA system. In particular in Europe and in Africa (cf. Table 32). The general notion from respondents that have an installed base of with this architecture is that it is used in smaller utilities and in “low-tech-environments”.

Figure 11 does not include the office zone or the supplier’s zone. These are excluded since the services in the control center do not exchange any data with services in these zones. However, also the isolated architecture does sometimes exchanged data with other control center. In that case this data exchange comprise of inter control center communication and/or data exchanged with emergency control centers.


VIKING Consortium


Figure 11 – The isolated architecture.

Also Figure 11 is associated with a set of deviations. These are listed in Table 35 and are a subset of those applicable to the two zone architecture described above.

Table 35 – Deviations within the isolated architecture.



4 Netcontrol Does not have a Control center power application server.

4.2.7




4.1.4 Variants

Three additional zones presented below together with their impact on services placement and data flows. These three can coexist with any of the zones architectures described


VIKING Consortium


above. They are however used together with the DMZ architecture more often than with the other two.

4.1.4.1 Front-end zone

One possible deviation from the three zone architectures described above is to set up a separate zone for the front ends and phasor data concentrators. This zone is introduced to shield the control center zone from the SCADA wide area network, see Figure 12. All data flows are instantiated in the same way in this architecture.

Figure 12 – The separate front-end zone architecture.

All vendors has stated that this architecture is possible with their system and ABB NM, PSI and Siemens have all stated that they have a number of systems installed with this extra DMZ. It appears to be especially common in the US, where practically all ABB NM installations have this architecture.

4.1.4.2 Back-office zone

One variant presented through the respondent interviews introduces a new network zone called the back-office zone. PSI does for example have this variant in their installed base of systems. This zone typically holds operator clients, engineering stations, and any existing special purpose software for historians etc. The operator clients located in this zone, or the user’s accounts typically limit the functions that can be accessed from this zone.


VIKING Consortium


Figure 13 – The back-office zone.

The back office zone serves as something in-between the office and the control center. It is possible to use business software such as email clients, web browsers and ERP clients in the back-office zone. The security policy taken with the machines and users in this zone is however stricter than for the office zone. Special requirements are placed on physical and logical access, and there are limitations in the use of business software.

When this zone is in use it is to avoid use engineering stations from the office. Hence, if this zone is present there are no data engineering stations located in the office.

4.1.4.3 Quality assurance zone

In recent years some installations has been equipped with an extra zone to support quality assurance activities. This zone is typically a full clone of the control center and contains an equivalent to all services and machines running in the operational control center. This will make it possible to test updates of various kinds before they are deployed. Including, changes to data models, updates of the SCADA system’s components and updates to operating systems and other third party software.


VIKING Consortium


As depicted in Figure 14 the quality assurance zone receives updates from external systems. These may come in the form of portable media, but they may also come from be downloaded from the internet or internal configuration management systems (e.g. new engineering data). Moreover, some service to manage versions and configurations is typically present in this zone. This is not depicted in Figure 14.

Figure 14 – The quality assurance zone.

In order to test new changes with real process data some vendors support a “listening mode” in the front-end. This means that all received process data will be duplicated by the front-end server and sent in parallel to the to the QA system. The QA system will receive the parallel data flow in the same way as if it was an online system and all activations like alarms, events and power applications starts will occur as if it was the online system. Commands to the process are blocked from the QA system to avoid conflicts with the online system. Another to test with operational data is to have the quality assurance zone listen to the operational SCADA server.

When testing is finished the new engineering changes are loaded into the operational SCADA system using the control system zone using the same mechanism as when loading from the Data Engineering server.

The use of quality assurance is something recognized by ABB NM, Siemens and PSI. It appears to be a component introduced in recent years and the majority of systems lack this zone. In that case approach to updates is that they are avoided and if they are applied they are tested on the backup servers (i.e. the redundant SCADA server) prior to operational deployment. While updates are tested on the backup server the synchronization with the operational server is inactivated.


VIKING Consortium


4.2 Services added or subtracted

This section will describe identified patterns which involve creating introducing a new type of service into the architecture or removing an existing service. In other words, a set of service not described in section 3.1 will here be described. When these services are introduced they are placed in a zone and are involved in data flows. This section will also describe this.

Figure 15 – Added or removed services and the implication this.

Six patterns where has been identified where new services are introduced, or existing services are removed.

Table 36 – Services added or subtracted.

Applicable to zone architecture:

Vendor(s) Deviation

DMZ Two

zones Isolated

ABB NM A thin HMI-client for read-only access

exists Yes Yes No

Netcontrol, MicroSCADA

No separate server for remote HMIs Yes Yes No

ABB NM A SCADA Replica exists for read only

access Yes No No

PSI A special interface server is used as a

temporary storage for data to the control center.

Yes No No

ABB NM A separate Front-end for inter control

enter communication Yes Yes Yes

Netcontrol (+all) No DMS server exists in the control

center Yes Yes Yes


VIKING Consortium


4.2.1 Thin HMI client

Access control is by most vendors managed through access rights to the operating system running HMI clients and access rights defined within the SCADA system. It is common that certain types of users have a need to view the status of the power system, but does not require the capability to send commands to it. ABB NM have developed a web (Applet) based solution that only allow the user to do just this – monitor the state of the power system but not influence it (cf. Figure 16). The difference between this solution and the one where access rights are associated with user accounts is that the access restrictions are statically implemented in the software. This thin client is installed in approximately 10% of ABB NM’s installed base but is more frequent in newer installations.

Figure 16 – Thin client in the architecture.

As in the standard architecture the HMI server is placed in the DMZ if a DMZ exists. In the two zones architecture the HMI server is placed in the control center. In both cases the web browser is located either in the office or externally. It should be noted that since the thin client runs in a Java applet it practically executes in the web browser and not in the web server itself.

4.2.2 HMI server gives access to operator client in the office

The most common way to implement a HMI server is to have it host a number of operator clients in the same machine as the HMI server. This means that the operator stations will be located in the DMZ if this zone exists. PSI, however, have the HMI server host operator clients located in the control center zone.

Figure 17 – HMI server hosts operator clients located in the control center.


VIKING Consortium


As will be explained in section 4.3.4 PSI has an architecture that is different from other vendors when it comes to the relationship between operator clients and other services. This is one of the reasons for this placement architecture.

4.2.3 No separate server for remote HMIs

The architectures described in sections 4.1.1 and 0 includes a HMI server which hosts a number of operator clients. Netcontrol and ABB MicroSCADA do not always have a dedicated server for this. When no separate HMI server exists but remote access is made possible one of two alternative solutions are used:

To locate an operator client outside of the office, in an external machine. To turn use remote desktop features on a machine located in the control center.

Figure 18 – No separate server for remote HMIs.

The first variant means that operators have the client software installed on a computer located outside the control center zone, e.g. on a laptop. This operator client is then used by connecting to the control center servers via a virtual private network. Hence, operator clients are allowed to execute from the office zone or that remote users are allowed to join the control center via a VPN-tunnel or equivalent.

The second does not involve HMI client software installed externally. It does however require that connections can be initiated from external remote desktop clients to the remote server running on the control center workstation. As no special server is set up for this purpose this second alternative also means that the external remote desktop client occupies the control center machine while it is in use.

4.2.4 SCADA replica in the DMZ

When a DMZ is used ABB NM sometimes place an additional SCADA server in this zone. This SCADA replica synchronizes some of its data with the SCADA server in the control center. The default configuration is to update the replica with data that is needed for presentation, and to be strict about the data sent back into the control center. In essence this SCADA replica subscribed to data in the SCADA server located in the control center, but the replica cannot change anything in this operational server.


VIKING Consortium


As depicted in Figure 19 the HMI server is associated with the SCADA replica if the replica is in use. Also, in case the SCADA replica exists data flows that use the SCADA servers API to retrieve real-time data now do this from the replica.

Control Center

DMZ


SCADA Server

Historian (replica)


Inter control center i ti

Operator HMI Client

HMI Server

Historian

Control center powe

EMS(Application

Server)

SCADA Server replica

The replica subscribe to

real-time data


[13] A server hosts several HMIs

Figure 19 – SCADA replica in the DMZ.

Although it is less common it is also possible to place a replica of the power application server in the DMZ. This would in the same way allow external users to operate on the real-time data without influencing the operational servers.

4.2.5 An interface server in the DMZ

There are a number of data flows that could go in to the control center in the DMZ architecture. In particular is data from various office servers (e.g. ERP systems, GIS systems, Production Planning and Optimization systems) sometimes moved in to the control center. Several vendors offer the possibility to write and read data through APIs implemented in power application servers, SCADA servers and historians. However, not all system installations use these APIs to move data into the control center zone.

The historian is sometimes used to move data into the control center from the office, cf. 3.2.18. PSI uses a special, dedicated, interface server to move all data destined to the control center. Figure 20 illustrate how this interface server is used as an intermediary between the control center and the office.


VIKING Consortium


Figure 20 – Interface server in the DMZ.

The rationale behind the use of this intermediary is a policy stating that the services located in the control center should determine when their data should be updated. Data is therefore written to the interface server and from there pulled by the services located in the control center. As this policy is the reason for the interface server it is not combined with direct access to services in the control center using their APIs.

4.2.6 Separate inter control center communication front-end

ABB NM sometimes divide the inter control center communication into two separate services: a front-end part only managing protocol conversion is added to the inter control center communication server. See Figure 21.

Figure 21 – Inter control center front-end and its placement. Alternative a) is in the DMZ architecture, alternative b) is in the other architectures.

This separation is due to technical reasons. The Secure MMS implementation used by ABB NM for inter control center communication has run on a separate machine and it is


VIKING Consortium


therefore put in the front end. When ABB NM use Secure MMS in the DMZ architecture the front-end part is placed in the DMZ and the actual server is placed in the control enter zone (alternative a) in Figure 21). When it is used in the two zone architecture or in the isolated architecture they are both placed in the control center zone (alternative b) in Figure 21).

4.2.7 External DMS system

Most included in this study offer power applications (e.g. EMS and DMS applications) as a part of their product. Netcontrol, however, offer this through a partnership with another vendor – Tekla.

As illustrated in Figure 22 this means that none of the Netcontrol system has a power application server dedicated for the control center. Instead the SCADA server and historian are integrated with the power application server offered by another vendor Tekla, which is typically located in the office.

Office Control Center

SCADA Server

Operator HMI Client

External DMS server

DMSHMI Client

DMSHMI Client

[10-11] Commands to process, changes and markups of data


Commands to process

Commands to process using API

Commands to process

[new1] Read operator views and change parameters

[new1] Read operatorviews and change

parameters

Figure 22 – External DMS system.

The location of the DMS system means that real time data is read by the DMS server and that calculation are performed in the DMS server located in the office. Data can then be fed back to the SCADA server and viewed through the operator client, or viewed in the DMS systems own user client. The integration between the office DMS system and the SCADA system also make it possible to configure the system so that operator commands can be executed from the DMS system’s operator client. So, thorough this client the user can perform maneuvers on objects, with the same type of dialogs as in the operator HMI clients.

4.3 Bundling of services

This section will describe how services are bundled together by different vendors. Bundled in this case means that the software components supplying these services are tightly coupled to each other and that separating them (e.g. on different machines) is not easily done.


VIKING Consortium


Figure 23 – The focus of this section – how services are coupled to each other.

In Table 37 an overview of the variants is given. It should be noted that it is common that services share the same physical machine although their software is loosely coupled. For example, in ABB NM’s installations it is common that the SCADA server and the Power Application Server are installed in the same machine. And in Netcontrol’s system it is common that the historian and SCADA server is located in the same machine. Unlike the ones listed in Table 37 these are collocated to save physical machines, not because their software requires it.

Table 37 – Different ways services are bundled into one component.

Vendor Services bundled Described in

section

Siemens Netcontrol ABB Micro

SCADA

Operator HMI client Engineering station (HMI)

4.3.1

Netcontrol SCADA server

Data engineering server 4.3.2

Siemens

Phasor data concentrator Front end

4.3.3

PSI

Operator HMI client Engineering station (HMI)

Distributed SCADA Distributed power application server

Distributed historian

4.3.4


VIKING Consortium


4.3.1 One HMI for Data engineering and Operators stations

Netcontrol, Siemens and ABB MicroScada have implemented the operator HMI client and the data engineering station in the same software application. This means that if an engineering station is placed in one location, so is an operator HMI client, and vice versa.

Figure 24 – A single HMI application for data engineering and operators.

Access rights are used to separate the services of the operator from those of data engineers. Also, access rules defined over network interfaces may restrict the access to certain services from one location. For instance, requests to the SCADA server can be blocked from locations only intended to for data engineering.

4.3.2 Data engineering directly in the SCADA server

In the architectures of Netcontrol and ABB MicroSCADA the engineering server and the SCADA server are also implemented in one module. In other words are data engineering performed directly into the control center’s SCADA server.

Figure 25 – A single server for Data engineering and SCADA.

As Netcontrol and ABB MicroScada focus on the distribution market the utilities using their systems are comparatively small. As a consequence this architecture is rarely combined with data flows from office servers or other systems which update the data engineering server with new process models. The primary client to the data engineering part in this case is a client application with proper access rights.

4.3.3 Phasor data concentrator and front-end coupled

The phasor data concentrator collects data from phasor measurement devices and the front-end manages communication with RTUs. The role of the phasor data concentrator is more or less equivalent to the role of the front end, but with the restriction that it only works on a continuous stream of phasor data. Several vendors (e.g. PSI) sometimes place the front-end and phasor data concentrator in the same hardware. Siemens, however, have implemented the phasor data concentrator in the front end.

Figure 26 – Front-end implementing the phasor data concentrator.


VIKING Consortium


This solution is different from the one where the phasor data concentrator is implemented as a standalone service and placed on a separate hardware. However, as these two are always placed together anyway there is no impact to the architectures described above.

4.3.4 Distributed SCADA, power application and historian

As described above PSI have a different architecture for the distribution of services than the base figures presented above. In the PSI system the operator stations are equipped with the following:

The operator HMI client The data engineering station (HMI) A SCADA server with some limitations Parts of the power application server Parts of the historian database

As depicted in Figure 27 these components are distributed over the control center and each operator station hold an instance of them. To synchronize the content of these services (e.g. data corrections) a synchronization server is used. This server also has a supervisory role and make sure that no interlocking occur when commands are issued from the operator stations. Hence, not all functionality of the SCADA server is distributed to operator stations.

Figure 27 – Distributed server architecture.

The historian database located in the operator stations is also somewhat limited. It typically holds data for the most recent days and if older data is required this is gathered from the historian server. Furthermore, some calculations made by the power application server are made centrally in the synchronization server due to performance constraints. Also, some calculations involving contingency analysis require large amounts of historical data and are therefore made in the historian server.


VIKING Consortium


5 Identified cross-boundary data flows

From a security perspective are data flows that with between services in the control center zone and the less secure zones. According the classical Biba model [46] for data integrity an object should not read data from objects with lower integrity levels. Nor should objects with lower security levels be allowed to write data to objects with higher security levels. As can be seen from the descriptions given above data is in many cases moved from less secure zones in to the control center zone. This chapter will summarize the data flows where the control center’s or the substation’s boundary is traversed. This summary also included the cases where data is moved out from the substation or control center zone.

5.1 Substation communication

All SCADA systems within the domain of electrical power transmission and distribution needs to communicate with RTUs and other control equipment distributed in the field. In the description given in this document there are six data flows that involve substations:

Disturbance records read from substation equipment. Changes of configurations and parameters of substation equipment by remote

engineers. Suppliers dialing in to the substation. PMU measurements sent to the control center phasor data concentrator. Measurements sent from RTUs to the control center front ends. Commands sent from the control center to RTUs in substations.

The last two of these is exists to supply the basic functionality of the SCADA system and is always present. The fourth bullet is present in systems using PMU measurements, primarily systems on transmission levels. The first three exists to enhance and simplify support and maintenance. While they are not always present in today’s SCADA systems the tendency appears to be that these types of data flows are increasing in popularity.

5.2 Inter control center communication

Communication between control centers is common in SCADA systems. The data flows between control centers can include both direct operator commands from the external control center, or merely measurements and status updates. The peers in these data flows are located in the control center zone or in a DMZ just outside the control center.

5.3 Data sent to emergency control center

In addition to inter control center communication data is sometimes sent to emergency control centers. The protocols used here are often on synchronizing the real-time databases using some proprietary protocol. Since the other control center often is regarded as zone with equal security level this should perhaps not be regarded as a cross-boundary data flow. The data is however leaving the control center and is mediated by some network that could be regarded as less secure than the control center.


VIKING Consortium


5.4 Historical data going out from the control center

A common and natural data to move out from the control center is historical data. Most SCADA systems that interface other zones also include a historian that is accessible from the office zone. Historians are used to make historical data accessible in the following ways:

Web interfaces in implemented in the historian. Special purpose clients developed for the historian. Through the operator HMI clients. Programming interfaces (APIs) for custom made connections to other systems.

In the case of a DMZ architecture an historian replica is placed in the DMZ and is kept synchronized with the historian located in the control center. In case of a two zone architecture the historian is made accessible from the office zone.

5.5 Remote operator stations

Remote operator stations of various kinds appear to be commonly used feature in SCADA system. The ones identified in this study can be categorized into eight variants:

The operator client software is installed on a machine located outside the control center and is allowed (by network interfaces) to connect from an external zone, e.g. the office.

The operator client software is installed on a machine located outside the control center can be connected after this machine has joined the control center zone, e.g. through a VPN tunnel.

A remote desktop server (e.g. VNC or Windows Terminal Services) is activated and accessible on a host in the control center.

A HMI server runs in the control center zone and gives access to a number of “virtual” operator HMI clients.

A HMI server runs in the DMZ and gives access to a number of “virtual” operator HMI clients located in the DMZ.

A HMI server runs in the DMZ and gives access to a number of “virtual” operator HMI clients located in the control center zone.

The HMI server’s operator HMI clients only connect to a SCADA replica from which no commands is forwarded to the power process.

A special purpose client with read-only access is available for users outside of the control center.

In addition to the plethora of ways to offer remote access there are also different rule sets that can be applied for where the client must be. For instance, remote access might be limited to a number of dedicated machines in the office, or remote access can be enabled from all addresses (including external internet addresses). In the same way access controls can be implemented and enforced to various degrees.

5.6 Suppliers dialling in

The majority of systems with an interface to external network zones allow suppliers to dial in to the control center for maintenance and troubleshooting purposes, all but one


VIKING Consortium


The restrictions placed on this type of access appear to vary with the size of the utility. In many cases this type of access will require the vendor to ask for wires to be connected and/or firewall rules to be reconfigured.

5.7 Office services updating control center historian

The historian contains data which is valuable to power utilities. The historian can be accessed through operator HMI clients, web based interfaces, special purpose tools or APIs giving access to its data. Data is both read and written to the historian through these interfaces.

In the two zone architecture are data flows from all of these directly interfacing the historian service located in the control center. When a DMZ is used these services interface the historian placed in the DMZ instead.

5.8 Office services writing to and reading from SCADA server

In some installations services in the office interface the SCADA server through an API. This type of API of typically used by other services in the control center but can also be used to update the SCADA server’s data or to read data from the SCADA server. For instance, data missing in the SCADA server might be replaced with default data or real-time data may be recorded and stored in other (external systems). As exemplified by Netcontrol’s integrations with the DMS system, Tekla’s APIs can also be used to create custom operator clients and issue commands towards the power process.

5.9 Office services updating control center power application server

In the power application sever located a number of functions can be offered. Some of these benefit greatly from data retrieved from external systems. For instance, outage management can be enhanced if call centers can enter trouble calls into the system and if automated crew management systems can be integrated. The data is often exchanged through some enterprise services bus and/or APIs defined in the power application server. Sometimes an interface server is used as an intermediary and the power application server reads data from this interface server.

5.10 Data engineering updated by external systems

The data engineering server is sometimes subordinated some other system, e.g. a geographical information system located in the office. This phenomenon is particularly common in lager power distribution utilities where a large quantity of network data is updated each day. To avoid entering these updates multiple times a system located outside the control center is sometimes used as a master data. In this case an API or database interface is used to update the data engineering server.

In addition to the updates received from other application servers the use of engineering stations located in the office appears to be common. A variant is to place engineering station in a back-office zone and only allow updates from there. This does however appear to be less common than the scenario where engineering stations are used from office hosts. Moreover, most vendors seem to integrate the operator HMI client and the


VIKING Consortium


engineering station into one software application. In this case all remote operator clients will mean that it could be possible to perform data engineering remotely. If this can be done will depend on the access rights associated with the user accounts.

5.11 Updates from quality assurance zones

New SCADA systems sometimes include a special quality assurance zone where updates can be tested before deployment. As the whole purpose for this zone is to bring new software and data in to the control center it can be seen as cross-boundary data flow. Data can be brought in to the quality assurance zone in a number of ways, including: portable medias, downloads from suppliers and development efforts made in-house.

5.12 Office and internet applications used in the control center

Although it is not depicted in the figures of chapter 0 it appears as office software that access the internet is sometimes used in the control center. This could for example be Lotus Notes clients, email clients or internet browsers. Several vendors notice this as a practice that can be observed in some utilities, in particular smaller utilities where security awareness is on the lower end of the scale. When these applications are in use they are typically used from a machine that also holds the operator HMI client.


VIKING Consortium


6 Conclusions

The purpose of this report is to provide a descriptive account for state-of-the-practice architectures for SCADA systems. No analysis of the cyber security of these system architectures is given in this report and consequently no conclusions on cyber security can be drawn in this report either. Instead the different SCADA system architecture solutions described in the report is intended to serve as a base for further analyses of cyber security. Obviously, already at this stage, is that different architecture patterns will have an impact on cyber security. Exactly how and how much, is left for future work. In particular, Deliverable 3.1 of the VIKING project will use the result of this study as input to the architecture based vulnerability assessment.

Below some of the main findings with respect to SCADA system architectures as such are described.

6.1 Clear patterns exists in SCADA system architectures

Through the literature study and the interviews with vendors a number of patterns, or stereotypical architectures, has been identified. First it is interesting to note that it is possible to speak of SCADA systems as a relevant type of system. A majority of functions/services and data are found in all SCADA systems, and as a whole deviations are fairly small between systems. The software services constituting a SCADA system and the services interfacing these appears to be more or less standardized. Section 3.1 gives a list 18 services that exists in SCADA environments. In section 4.1 seven deviations from this set of services is described and in section 4.3 variants in how these services are composed into software packages is described. With these deviations accounted for the SCADA systems installed by vendors interviewed in this study can be described.

This is also the case for data flows between services. In total 31 services are described in section 3.2. Several of these are data flows that always exist in SCADA systems, e.g. the one where operators monitor the status of the power network. Other data flows exist in some SCADA system installations and not in other. Based on the interviews with vendors it does however appear as the set of 31 data flows accurately describes those that can exist.

The zone architectures used and the relationship between zones are also possible to describe in a number of patterns. Three zone architectures has been identified (e.g. the two zones-architecture), together with thee variations applicable to these (e.g. use of back-office zones). The descriptions given in this report do not go in to details of for example network perimeters access rules and how certificates are distributed in the architectures. These zone architectures, the placement of services, and the data flows between these services does however give a coarse description if the configurations applied in today’s SCADA system architectures.

6.2 The architecture is partly contingent on the type of utility

While clear patterns have been identified in this study, there are also a number of variations between architectures used. Data on how common different variants are is in this study associated to vendors, markets and the type of power grid the SCADA system operates. From the data collected in this study differences in architectures can be identified between these.


VIKING Consortium


The US market does for example appear to use the DMZ architecture more often, at least on the transmission level. The African market seems have a higher share of isolated systems than other markets and transmission utilities use the DMZ architecture more often than distribution utilities.

Also between vendors there is a difference in the zone architectures applied. PSI does for example not have a single system which is completely isolated, while other vendors do. Distribution utilities update the data engineering server from office servers more often than transmission utilities, and transmission utilities send data to other control centers more often than distribution utilities. In architectures delivered by vendors focused on smaller utilities (i.e. Netcontrol and ABB MicroScada) the remote access methods appears to be different, and HMI servers are used more seldom. These systems also have less data exchanged with external systems in general, although Netcontrol’s systems often exchange data with external DMS systems.

Even though the quantitative figures in the report are estimations provided by vendors answering for many installations over a long period of time, they clearly give approximate indications of differences that are likely to be reflecting the true state of affairs.

6.3 The share of legacy systems are substantial

SCADA systems have a longer lifecycle than most administrative systems used in office environments. This is also demonstrated by the estimates shown in this report. The vendors interviewed in this describe the typical architectures delivered today as DMZ architectures, with multiple data flows to the office. These architectures sometimes involve special servers to interface the office (cf. 4.2.5) or special solutions for remote access to operator clients (cf. 4.2.1 and 4.2.4). However, in the installed base architectures with the DMZ and these features is still uncommon. In fact, the DMZ architecture is less common than the other two in all environments except PSI customer base and ABB NM in US and Australia. The respondent for Siemens’ European distribution utilities did not know a single installation using this architecture.

6.4 It is common that several types of data are exchanged with SCADA systems

In chapter 0 a summary of the data flows that goes in and out from control centers. As can be seen from this summary these data flows are not limited to historical data going out and communication with field equipment. Data from the office zone is transferred over zone interfaces in to the control center. This includes the following interfaces:

engineering data received from external GIS systems, office servers using APIs in historians, SCADA servers and application servers, office applications (e.g. Spreadsheets) retrieving and manipulating data in

historians, SCADA servers and application servers, remote operator clients and remote engineering stations, software updates and debugging data sent to/from suppliers,

In some cases web traffic and emails are also targeting the control center zone. Also, substations are often interfacing some office server to send fault records, and substation RTUs can often be configured from the office zone.


VIKING Consortium


6.5 The DMZ is not always a “real” DMZ

The concept of a DMZ means that all traffic should terminate in the DMZ and no traffic should go directly through it [10]. Based on the interviews with respondents this is not always the case when the DMZ architecture is used. External services that sometimes can establish connections directly with services in the control center include:

office servers and desktop applications using APIs engineering stations and operator clients installed in the office web sites, email servers and other administrative office systems

The only vendor interviewed in this study who strictly maintained to the principle that no data should go directly into the control center in case of a DMZ was PSI.


VIKING Consortium


7 References

[1] IEEE-SA Standards Board, "IEEE Recommended Practice for Architectural Description of Software-Intensive Systems," October, 2000.

[2] NERC, "NERC CIP 002-009," 2007.

[3] K. Stouffer, J. Falco, and K. Kent, "Guide to Industrial Control Systems ( ICS ) Security Recommendations of the National Institute of Standards and Technology," Nist Special Publication, vol. 800, 2008.

[4] ANSI/ISA, "Integrating Electronic Security into the Manufacturing and Control Systems Environment," System, 2004.

[5] IEC, "TS62351-5 - Technical Specification Data and Communications Security Part 5 : Security for IEC 60870-5 and Derivatives," Security, 2008.

[6] IEC, "TS 62351-3: Power systems management and associated information exchange – Data and communications security – Part 3: Communication network and system security – Profiles including TCP/IP," vol. 2007, 2007.

[7] IEC, "TS 62351-1: Power systems management and associated information exchange Data and communications security, Part 1:Communication network and system security Introduction to security issues," vol. 2007, 2007.

[8] IEC, "TS 62351-4: Power systems management and associated information exchange Data and communications security Part 4: Profiles including MMS," vol. 2007, 2007.

[9] IEC, "TR62210: Power system control and associated communications – Data and communication security," vol. 2003, 2003.

[10] E. Byres, J. Karsch, and J. Carter, "NISCC good practice guide on firewall deployment for SCADA and process control networks," 2004.

[11] G. Finco and others, "Cyber Security Procurement Language for Control Systems," Idaho National Labs, 2007.

[12] J. Fernandez and A. Fernandez, "SCADA systems: vulnerabilities and remediation," Journal of Computing Sciences in Colleges, vol. 20, 2005, p. 160168.

[13] C. Bowen, T. Buennemeyer, and R. Thomas, "Next generation SCADA security: best practices and client puzzles," Proceedings of the 2005 IEEE.

[14] L. Nordstrom, "Assessment of Information Security Levels in Power Communication Systems Using Evidential Reasoning," IEEE Transactions on Power Delivery, vol. 23, 2008, pp. 1384-1391.

[15] D. Dzung, M. Naedele, T. Von Hoff, and M. Crevatin, "Security for Industrial Communication Systems," Proceedings of the IEEE, vol. 93, 2005, pp. 1152-1177.

[16] V. Igure, S. Laughter, and R. Williams, "Security issues in SCADA networks," Computers & Security, 2006.


VIKING Consortium


[17] R. Chandia, J. Gonzalez, T. Kilpatrick, M. Papa, and S. Shenoi, Critical Infrastructure Protection, Boston, MA: Springer US, 2007.

[18] T. Sommestad, M. Ekstedt, and L. Nordstrom, "Modeling Security of Power Communication Systems Using Defense Graphs and Influence Diagrams," IEEE Transactions on Power Delivery, vol. 24, 2009, pp. 1801-1808.

[19] S. Rinaldi, J. Peerenboom, and T. Kelly, "Identifying, Understanding, and Analyzing Critical Infrastructure Interdependencies," IEEE Control Systems Magazine, 2001, pp. 11-25.

[20] I. Nai Fovinoa, A. Carcanoa, M. Maseraa, and A. Trombettab, "An experimental investigation of malware attacks on SCADA systems," International Journal of Critical Infrastructure Protection, vol. 2, 2009, pp. 139-145.

[21] C.B. III, T. Buennemeyer, and R. Thomas, "Next generation SCADA security: best practices and client puzzles," Proceedings of the 2005 IEEE.

[22] P. Oman, E. Schweitzer, and J. Roberts, "Safeguarding IEDs, substations, and SCADA systems against electronic intrusions," Proceedings of the 2001 Western Power Delivery Automation Conference, 2001, p. 9–12.

[23] E. Johansson, T. Sommestad, and M. Ekstedt, "Security Isssues For SCADA Systems within Power Distribution," Nordic Distribution and Asset Management Conference (NORDAC), 2008.

[24] M. Naedele, "Addressing IT Security for Critical Control Systems," 2007 40th Annual Hawaii International Conference on System Sciences (HICSS'07), 2007, pp. 115-115.

[25] J. Slay and M. Miller, "A security architecture for SCADA networks," Proceeding of the Australasian Conference on Information Systems, Association for Information Systems, 2006.

[26] R.L. Krutz, Securing SCADA systems, Indianapolis, Indiana, USA: Wiley Publishing, 2006.

[27] J. Wiles, T. Claypoole, P. Henry, P. Drake, and L. Johnson, "Techno Security's Guide to Securing SCADA: A Comprehensive Handbook On Protecting The Critical Infrastructure," 2008.

[28] J. Stamp, M. Berg, and M. Baca, "Reference Model ofr Control and Automation Systems in Electrical Power," 2005.

[29] G. Clarke and D. Reynders, "Practical modern SCADA protocols: DNP3, 60870.5 and related systems," 2004.

[30] S. Mackay, J. Park, and E. Wright, "Practical data communications for instrumentation and control," 2003.

[31] S. Mackay and E. Wright, "Practical industrial data networks: design, installation and troubleshooting," 2004.

[32] J. Park and S. Mackay, "Practical data acquisition for instrumentation and control systems," 2003.

[33] C. Strauss, Practical electrical network automation and communication systems, Elsevier, 2003.


VIKING Consortium


[34] T. Cegrell, Power system control technology, Cambridge, Great britain: Prentice hall International, 1986.

[35] M. Qureshi, A. Raza, D. Kumar, M. Park, and B. Park, "A survey of communication network paradigms for substation automation," 2008 IEEE International Symposium on Power Line Communications and Its Applications, Jeju city, Jeju Island: IEEE, 2008, pp. 310-315.

[36] J. Toivonen, P. Trygg, A. Mäkinen, and P. Järventausta, "A survey of infromation systems in Finnish electricty distribution companies," Proceedings of NORDAC, 2006, pp. 1-8.

[37] G. Azevedo and A.O. Filho, "Control centers with open architectures," IEEE Computer Applications in Power, 2001.

[38] F. Wu, K. Moslehi, and A. Bose, "Power system control centers: past, present, and future," Proceedings of the IEEE, 2005.

[39] J. Andersson, E. Johansson, M. Haglind, and L. Johansson, "State-of-the-art study of commercial industrial control systems," 1997.

[40] IEEE Power Engineering Society, IEEE standard for SCADA and automation systems, IEEE Std C37.1-2007, New York, NY: IEEE Power Engineering Society, 2008.

[41] IEC, "IEC 61968-1, Application integration at electric utilities – System interfaces for distribution management – Part 1: Interface architecture and general requirements Reference num," vol. 2003, 2003.

[42] VLPGO, "EMS Architectures for the 21st Century - white paper," 2005.

[43] S. Owen, "Gaining momentum - The 2008 Energy& Resources Global Security Survey," Viral immunology, vol. 21, 2008.

[44] J. Sherwood, A. Clark, and D. Lynas, Enterprise Security Architecture: A Business-Driven Approach, USA: CMP Books, 2005.

[45] D. Bell, "Looking Back at the Bell-La Padula Model," 21st Annual Computer Security Applications Conference (ACSAC'05), Tucson, AZ: IEEE, 2005, pp. 337-351.

[46] K.J. Biba, "Integrity considerations for secure computer systems," 1977.

[47] M. Howard and D.C. LeBlanc, Writing Secure Code, Redmond, WA, USA: Microsoft Press, 2002.

[48] F. Swiderski and W. Snyder, Threat modeling, Redmond, WA, USA: Microsoft Press Redmond, WA, USA, 2004.

[49] G. Locke and P.D. Gallagher, "Smart Grid Cyber Security Strategy and Requirements," 2010.


VIKING Consortium


8 Appendix A –Substation ICT architectures

In the previous sections, the focus has been on the data flows within or between the control centers. However, another possible vulnerability regarding power system security is the remote communications happen between the control centers and geographically distributed substations. Though it is not tackled in the survey, the purpose for this chapter is to raise the awareness that this part is relevant for SCADA security analysis.

Substation Automation (SA) or Substation Control System (SCS) system usually delivers functions such as [A1]:

Monitoring/Control of substation electrical equipment from either central or bay level

Interface to remote SCADA system Status monitoring of all connected substation automation equipment System database management Condition monitoring of substation electrical equipment (switchgear,

transformer, relays. etc.)

To couple with these functional requirements, the major SA system components are:

Remote Terminal Unit (RTU) was developed with the aim of acting as an interface and communication unit between field instruments and a SCADA master. The pioneer generations did very little more than converting the information from the instruments to the SCADA language, send the information through or vice versa. RTUs are gradually improved, with strong communication capabilities in mind. [A2]

Intelligence Electronic Device (IED) is, to some extent, a loosely defined term. Concerning the protection and automation in the power industry, the term really came into existence to describe a device that has versatile electrical protection functions, advanced local control intelligence, monitoring capabilities and extensive capability to communicate with remote SCADA. [A2]

Human Machine Interface (HMI) is the principle user interface and normally would take the form of a computer which sometimes of the capability to communicate with remote SCADA [A1]

Communication bus or busses link various devices and enable the data exchange among them [A1].

To address the security issues within automation systems, it is necessary to indentify the SA system applications that publish and consume the information and the interfaces enabling the information exchange between them. SA system functions refer to tasks or functionality which are performed within substation domain. Two architectures with certain abstractions are presented in the following sections.

8.1 The Single RTU architecture

Prior to the wide utilization of Intelligence Electronic Devices (IED) as the monitoring and control interface to the power system equipment, RTUs dominate in conventional SA system by providing basic data acquisition, basic control and protection functions. Due to the limited processing power and communication techniques, the centralized solutions are the common for SA system configurations


VIKING Consortium


There are many possibilities to connect the RTUs and the master stations [IEEE standard for SCADA and Automation Systems] given scenarios vary in numbers of master stations and communication solutions. However, data flows in all the aforementioned SA system interconnection solutions remain the same which could be uniformly represented in Figure A1.

Figure A1 – Logical representation of simple single RTU SA system architecture.

In this simple layout, there are two major data flows: the data exchange between the control centers and substation RTUs and the data acquisitions and control executions happen between RTUs and the process equipments. The vulnerability for cyber attack in this architecture is the data flow between the control centers and the geographical scattered SA automation systems.

8.2 The field bus/process bus based architectures

Figure A2 presents a three-layer substation functional architecture (Process/Bay/Station) where the function allocations are illustrated together with the information exchange interfaces.


VIKING Consortium


IED

Remote process interface

Process Level

Sensors

Function B

Remote Control (NCC)

Technical Services /

remote engineers

Function A

Actuators

Protection

Bay Level

Station Level

Control[6] Prot/control data exchange Protection Control[6] Prot/control

data exchange

[4] Protection data [4] Protection data [5] Control data[5] Control data

[9] Control-data [10] CT and VT value

[1] Control data[2] Control data exchange between

substation and a remoteengineer’s workplace

[9] Control-data [10] CT and VT value

HV equipment

[3] Data exchange within station level

[7] Direct dataexchange

between baysfor fast functions

[8] Protection data Protection data [8]

Figure A2 – Logical representation of filed bus/process bus SA system architecture

Process level functions are all functions interfacing to the process. Specifically, binary and analogue I/O functions such as data acquisition (including sampling) and issuing of commands.

Bay level functions use mainly the data of one bay and act mainly on the primary equipment of one bay. The definition of bay level function considers some kind of a meaningful substructure in the primary substation configuration and, related to this substructure, some local functionality or autonomy in the secondary system (substation automation) for example, line protection and bay control.

Station level functions refer to substation as a whole. There are two classes of station level functions:

Process related substation functions use the data of more than one bay or of completed substation. Examples of such functions are station-wide interlocking, automatic sequencers or bus bar protection.


VIKING Consortium


Interface related station level functions represent the interface of SA systems to the local substation operator Human Machine Interface (HMI), to a remote control centre Tele Control Interface (TCI) or to the remote engineering workplace for monitoring and maintenance Tele Monitoring Interface (TMI).

8.3 Data flows and relevant communication protocols

In the following table the data exchange interfaces are summarized and the relevant protocols are presented.

Table A1 – data interfaces in substations.

No Data flow Data

transported Protocol

Correlation to image 2

[1] Control data exchange between substation devices and a remote

control center Control data

Can be DNP, IEC60870-5-101, IEC60870-5-104,

other vendor proprietary protocols

See (6)

[2] Data exchange between substation

(level) and a remote engineer’s workplace

Engineering data

Can be any type of protocol, for instance, FTP, HTTP, or vendor proprietary protocols

See (6)

[3] Data exchange within substation

level Any

MMS – see IEC 61850-8

GOOSE – see IEC 61850-8

See (4)

[4] Protection data exchange between

bay and station level Protection

data MMS – see IEC 61850-

8 See (1)

[5] Control data exchange between

bay and station level Control data

MMS – see IEC 61850-8

See (1)

[6] Data exchange within bay level Sample value,

interlocking


Sample Value – see IEC 61850-9.2

See (3)

[7] Direct data exchange between the

bays

Data for fast functions such as

interlocking



See (2), (3)

[8] Protection data exchange between bay level and remote protection,

normally goes to the RTU

Protection data

Non IEC 61850 protocol or RTU specific protocol

No correlation

[9] Control data exchange between

process and bay level (this can be a hard-wired cable to the devices)

Control data Can be any type of

protocol or proprietary protocol

No correlation

[10] Current Transformer (CT) and

Voltage Transformer (VT) instantaneous data exchange

CT and VT value


See (3)


VIKING Consortium


between process and bay level

To enable security analysis, an instance of physical implementation of a substation automation system is provided in Figure A3. Note that although there is a similarity between the logical and physical level representation, there is no unique way for mapping the logical function structure to the physical device structure. The mapping is dependent on availability and performance requirement, cost constraints and the state of the art in technology, etc. For instance, the use of GPS in providing precision timing as described in data flow 5 of Figure A3. Note that the control and protection data flow between the process and bay equipment are excluded from Figure A3 since in most cases they are realized by hard-wired cable, which means they are free of threat of cyber attack.

Pro

cess

bu

s

Figure A3 – Physical representation of filed bus/process bus substation architecture

In summary, the cross-boundary data flows for this SA architecture are

Data flows between IEDs and station level applications (1) Data flows between the IEDs (2) and (3) Data flows among station level applications (4)


VIKING Consortium


GPS data stream for IED time stamping (5)

8.4 References

[A1] AREVA, NPAG - Network Protection & Automation Guide, 2008.

[A2] C. Strauss, Practical electrical network automation and communication systems, Elsevier, 2003.


VIKING Consortium


9 Appendix B – Power System Communication Networks

This section provides a brief overview of common communication technologies used in the power system automation, it is not meant to be exhaustive. The use of communication system in Power System has enabled the development of large interconnected grids. This is because the reliance on communication made it possible to implement remote monitoring and control systems. The communication technology portfolio in any typical utility is offer a high degree of variety in terms of the media, protocols, and architectures used. This has been due to the many factors some of which are cost, technological advancement, and regulatory and environmental restrictions as well as for increased reliability.

9.1 Communication media

In terms of communication media, many options are implemented, the following illustrates taxonomy of the various media that are implemented.

Figure B.1 – Some Examples of Communication Media used in Power System Communication

There are several factors that influence the choice of media to be used, these are cost, bandwidth, media licensing in specifically of wireless technologies, right of way, remote station location and electromagnetic interference.

Cost is an important factor because some media are expensive and their maintenance and relating equipment are also expensive, this is the case for example for Fiber Optics, Power Line Carrier and Distribution Line Carrier, where the actual communication equipment to enable communication over these media is expensive.

Bandwidth refers to how much data the communication media can carry, for example fiber optic and microwave communication are very attractive due to the high bandwidth that these technologies can support. High bandwidth equipment is usually used for the backbone of the wide area network (see next section), but it is also becoming common practice to used them in substations. This refers specifically to optical fibers which also have other advantages such as low electromagnetic interference. Electromagnetic


VIKING Consortium


interference introduces errors into the communication stream which is not desirable especially in the case of critical communication for commands or measurements.

The right of way and the location of remote stations is an important factor that determines what media can be used. For example If the utility has existing pole infrastructure that it uses for the power lines than they can utilize wired media by placing the media on the poles as with the power lines, in the case of fiber optics or utilizing the power line itself as the media of communication, Power Line Carrier (PLC) in the case of transmission lines and Distribution Line Carrier (DLC) in the case of distribution lines. On the other hand if securing right of way permission is expensive or restricted due to environmental restrictions that the utility would opt for wireless media such as microwave radio or satellite. The use of wireless technology also applies if the substation is located in remote areas and laying wired media to such remote areas would not be feasible. On the other hand some radio frequencies may require the utility to purchase a license in order to utilize them.

Finally, it is typically the case that utilities utilize different media in many parts of their network, different media may be used for the same location in order to have redundancy and increase the overall availability of the link. Utilities may even purchase leased lines from telecom companies or used standard telephone lines to have a dial up connection to remote stations. On the other hand this typically is done when that option is available and the remote stations are not critical.

9.2 Communication Topologies and Network scale

Various topologies exist in power system communication, these topologies could be physical or logical based on the communication protocol used (see next section.). The main topologies that exist are Star, Ring, Bus and Meshed topologies. The following Figure B.28 illustrates these topologies

Figure B.28 – Common Physical Communication Topologies (a) Star, (b) Ring, (c) Bus (d) Meshed

A star topology has a central node where the communication passes through. The central node could be the SCADA master or a simple network switch. An advantage of the star topology is that it is easier to add and remove nodes without disrupting or reconfiguring the network. On the other hand, if the central node fails or breaks down then entire network is disrupted. The star topology is generally associated with Master-Slave polling protocols where the SCADA Master regulates communication by polling each station sequentially.


VIKING Consortium


The Ring Topology is another common topology and associated with token ring media access techniques. In general, In this topology the nodes are connected by a ring, the ring any media such as optical fibers or co-axial. A token signal propagated in an orderly fashion around the ring to each node, if a node has information to send, then it retains the token until it has finished sending, otherwise it will release the token back onto the ring for the next node in line to retain.

The bus topologies get its name traditionally from a single media line connecting all nodes. The advantage of both the bus and the ring are that it can decrease the complexity of wiring which greatly enhances maintenance.

The meshed topology is another common topology which is used mostly for redundancy in order to increase reliability of the network. In the Meshed topology all nodes in the network are connected to each other with a point to point link, the media in this case can vary it could be optical fibers, or microwaves or both.

Network topologies are usually mixed to increase flexibility and reliability and to mitigate limitations in have only one topology usually mixed, and in fact this mixture as well as the interconnection of local area networks it was makes up the power system network. The following Figure B29 illustrates a fictitious utility wide area network made up of various substations connected to the control center through a backbone or subnet. The backbone is a meshed network using high bandwidth media such as fiber optics and microwave radio. This is done to increase performance and reliability of the network. Notice from the figure the various combinations of topologies and media. For example substation 3 which has two bays each with a set of RTUs, the RTUs on each bay are connected together in a star topology and the bays are connected to the substation router in a bus topology configuration.

Also illustrated in Figure B3 is another common practice in modern utilities is to have back up communication links with a subset of their network (or in some cases the entire network). Substation 2 and substation 3 are part of the south west region in Figure B3 and since it is assumed that this region is critical a backup communication link using satellite is available from the control center.

So far we have discussed physical topologies, but nodes configured in physical topology do not necessary behave according to that constraint. It is possible for the system to be in a one physical topology and to behave in as if it was in another topology. This is related to the concept of logical topology is depended on media access techniques that are implemented on these systems. Examples of media access techniques are Carrier Sense Multiple Access with Collision Detection (CSMA/CD), Token Ring, and Time Division Multiplexing. Media Access Techniques are actually communication protocols that govern how nodes access shared media or how they communication in a local network, a general overview of protocols in power system automation is given in the next section.


VIKING Consortium


Figure B29 – An example network showing a control system, communication network.

9.3 Communication protocols

Communication Protocols define the rules and regulations that govern how nodes communicate to each other on the network. This includes how the media is accessed as well as how data is encoded and routed, as well as how sessions are established, maintained and when the communication has finished how sessions are closed.

Protocols, like media and network architecture are diverse in the modern electrical utility. This is due to many legacy and vender specific systems that are still being used in the industry. Older protocols used in the industry are based on the Enhanced Protocol Architecture (EPA). Such protocols typically have three layer, the physical, data link and application layer.

EPA based protocols minimized overhead which was an important requirement due to the limited bandwidth of older networking technologies. A good example of EPA based protocols that are used in the industry are MODBUS, IEC 607087-5-101 (IEC 101) and older version of the Distributed Network Protocol (DNP). On the other hand, the limited amount of layers limited the functionality and flexibility offered by these protocols especially in their use for inter network communication, where extra layers dealing with routing inter-network addressing and reliable session and transport management would be needed.


VIKING Consortium


The TCP/IP suite has recently become increasing deployed in utility networks due to its de fact status as the protocol of the internet and due to the availability of off the shelf components and is typically being used as the main protocol in utility Wide Area Networks (WAN). Due to the popularity of TCP/IP older protocols were extended to accommodate this demand. An example is IEC 60870-5-101 that was extend with IEC 60870-5-104 that allows the application layer data to be encapsulated in the TCP/IP packets. IEC 61850 is also gaining increasing attention. Generally, IEC 61850 is not a protocol per se as it utilizes the Ethernet protocol for substation communication and TCP/IP for communication to the control center.

Figure 30: An example network showing protocols.

The existence of many protocol some of them legacy has led to development of specialized nodes that act that facilitate protocol translation or encapsulation. Protocol Translation systems translate a protocol from one format to another protocol. These protocol translators can either be extra functionality on the front-end of the SCADA system or be separate nodes located at the control center and substations.


VIKING Consortium


9.4 Further reading

[B1] Daniel J. Marihart, “Communication Technology Guidelines for EMS/SCADA Systems”. IECC Transactions on Power Delivery, Vol. 16, No. 2, April 2001,

[B2] Vehbi C. Gungor , Frank C. Lambert, ”A Survey on communication networks for Electric system automation”. Computer Networks , Vol. 50, No. 7. May 2006.

[B3] Cobus Strauss, “Practical Network Automation and Communication Systems”. Newnes- Elsevier Press, 2003.

[B4] James Northcote-Green, Robert Wilson, “Control and Automation of Electrical Power Distribution Systems ” CRC Press, Taylor and Francis Group 2007.


VIKING Consortium


10 Appendix C – Questionnaire

The following pages shows the questionnaire used during the interviews with SCADA vendors.


VIKING Consortium

www.vikingproject.eu Page: I


VIKING Consortium

www.vikingproject.eu Page: II


VIKING Consortium

www.vikingproject.eu Page: III


VIKING Consortium

www.vikingproject.eu Page: IV


VIKING Consortium

www.vikingproject.eu Page: V


VIKING Consortium

www.vikingproject.eu Page: VI


VIKING Consortium

www.vikingproject.eu Page: VII


VIKING Consortium

www.vikingproject.eu Page: VIII


VIKING Consortium

www.vikingproject.eu Page: IX


VIKING Consortium

www.vikingproject.eu Page: X


VIKING Consortium

www.vikingproject.eu Page: XI


VIKING Consortium

www.vikingproject.eu Page: XII


VIKING Consortium

www.vikingproject.eu Page: XIII


VIKING Consortium

www.vikingproject.eu Page: XIV


VIKING Consortium

www.vikingproject.eu Page: XV


VIKING Consortium

www.vikingproject.eu Page: XVI


VIKING Consortium

www.vikingproject.eu Page: XVII


VIKING Consortium

www.vikingproject.eu Page: XVIII


VIKING Consortium

www.vikingproject.eu Page: XIX


VIKING Consortium

www.vikingproject.eu Page: XX


VIKING Consortium

www.vikingproject.eu Page: XXI


VIKING Consortium

www.vikingproject.eu Page: XXII


VIKING Consortium

www.vikingproject.eu Page: XXIII


VIKING Consortium

www.vikingproject.eu Page: XXIV

REPORT D2.3 SCADA system architectureskth.diva-portal.org/smash/get/diva2:495729/FULLTEXT01.pdf ·...

Documents

Transcript of REPORT D2.3 SCADA system architectureskth.diva-portal.org/smash/get/diva2:495729/FULLTEXT01.pdf ·...