5/21/2018 IEC 61850 Network Architectures
1/41
1
IEC 61850 Network Architectures
July, 2010
Maciej Goraj
5/21/2018 IEC 61850 Network Architectures
2/41
2Copyright RuggedCom Inc
Agenda
1. Requirements for substation communications network
2. Types of protocols and traffic patterns in IEC 61850 standard
3. Typical network architectures4. Problem of Multicast and Physical vs. Logical separation of
Process Bus and Station Bus
5/21/2018 IEC 61850 Network Architectures
3/41
3
Requirements for Substation
Hardened Networking Equipment
5/21/2018 IEC 61850 Network Architectures
4/41
4Copyright RuggedCom Inc
Substation Environment
Electric and Magnetic Fields
Electrostatic Discharge
Conducted High Frequency Electrical Transients
High Energy Power Surges Ground Potential Rise during ground faults
Climactic Variation: Temperature & Humidity
Seismic / Vibration
Pollution: Dust, Metallic Particles, Corrosive Chemical Particles,
Condensation, Solar Radiation, Salt, Bird Guano, etc.
EMI & Environmental Phenomena Typical of Substation Environments
Generation Plant HV/MV Substation Wind Farm
5/21/2018 IEC 61850 Network Architectures
5/41
5Copyright RuggedCom Inc
ContinuousPhenomena
Radiated RFI
Induced RFI
Power freq. Magnetic
Field
Slow Voltage Variations Harmonics,
Interharmonics
Ripple on d.c. power
supply
Power Frequency Voltage
Transient Phenomena
(High Occurrence)
Electrostatic Discharge
Voltage Dips
Lightning Ground
Potential Rise (GPR)
HV Switching by Isolators Reactive Load Switching
Transient Phenomena
(Low Occurrence)
Power Frequency
Variation
Power System Faults
Short Duration Power
Freq. Magnetic Fields
EMI Phenomenon
Devicesinsubstations mustdealwithacombinationofEMI
phenomenawhicharebothcontinuousandtransient.
5/21/2018 IEC 61850 Network Architectures
6/41
6Copyright RuggedCom Inc
Requirements for IEDs According to IEC 61850-3
Must operate properly under the influence of a variety of EMI
phenomena commonly found in the substation
IEC 61850-3 specifies a variety of type withstands tests designedto simulate EMI phenomena such as:
Inductive load switching
Lightening strikes
Electrostatic discharges from human contact Radio frequency interference due to personnel using portable radio
handsets
Ground potential rise resulting from high current fault conditions within
the substation
Ethernetswitches,routers,deviceservers,mediaconverters
shall
meet
EMI
requirements
to
the
same
extent
as
IEDs
5/21/2018 IEC 61850 Network Architectures
7/41
7Copyright RuggedCom Inc
Standard for Environmental and Testing Requirements for
Communications Networking Devices in Electric Power
Substations It goes one step further by defining Class 2 operation which
requires that, during the application of the type tests, the switch
must experiment:
No communications errors
No communications delays
No communication interruptions
RuggedizedEthernetswitchshallbeseenasyetanotherIED
Requirements for IEDs According to IEEE 1613
5/21/2018 IEC 61850 Network Architectures
8/41
8Copyright RuggedCom Inc
Fiber Optics Overview
Future proof
Theoretically infinite bandwidth
Up to 100 km distance possible
Immune to EMI
Supported by all current IEDs
Lightweight
Costs continue to drop
Multi-mode for short distances
Single-mode for long distances
5/21/2018 IEC 61850 Network Architectures
9/41
9Copyright RuggedCom Inc
Common Fiber Optic Connectors
ST Stick and Twist and SC Stick and Click historically popular
LC becoming prevalent especially for Gigabit because small form
factor (SFF) allows greater port density GBIC are pluggable SC transceivers using SC connectors
SFP are Small Form Factor Pluggable
SCST
LC MTRJ
5/21/2018 IEC 61850 Network Architectures
10/41
10
Types of Protocols and Traffic
Patterns in IEC 61850 Standard
5/21/2018 IEC 61850 Network Architectures
11/41
11Copyright RuggedCom Inc
Typically 2040IEDs per substation
Large substations mayhave 80
120
IEDs
Power Plants,Oil&Gasinstallation 150500IEDs
Large installations with LVIEDs 10001500IEDs
Large Wind Farms canhave +200IEDs
LargeSolargenerationsitescanhave600 1500IEDs
Number of Devices in Electrical Substations
5/21/2018 IEC 61850 Network Architectures
12/41
12Copyright RuggedCom Inc
IEC 61850 Ed. I Profiles and Protocols Stack
Will be moved to an Annex in Edition II of IEC 61850
5/21/2018 IEC 61850 Network Architectures
13/41
13Copyright RuggedCom Inc
IEC 61850 Ed. II Profiles and Protocols Stack
TimeSync
(SNTP)
TCP/IP
T-Profile
UDP/IP
GOOSESV MMS Protocol
Suite
ISO/IEC 8802-3
Core
ACSI
Services
Time
Sync
Generic
Object
Oriented
Substation
Event
Sampled
Values
(Multicast)
(Type 4) (Type 1, 1A) (Type 6) (Type 2, 3, 5)
ISO/IEC 8802-3 Ethertype
HSR (O)
SMV GOOSE
802.1Q 802.1Q 802.1Q (O)802.1Q (O)
IP (O)
5/21/2018 IEC 61850 Network Architectures
14/41
14Copyright RuggedCom Inc
Types of traffic
Client-server MMS services:
Polling
Reporting (Unsolicited and/or periodic)GOOSE
Asynchronous and unsolicited
Less often synchronous (for heartbeat and for analogue values)
Sampled Values (Process Bus) Synchronous unsolicited transmission
IEC
61850
network
is
a
combination
of
Raw
Ethernet,
MMS/TCP,
SNTP,
IEEE
1588,
TFTP,
FTP,
RSTP,
SNMP,
and
otherEthernetbasedprotocols
5/21/2018 IEC 61850 Network Architectures
15/41
15Copyright RuggedCom Inc
Encapsulated directly in Ethernet layer
High priority, critical, asynchronous and unsolicited
Less often synchronous (for heartbeat and for analogue values) MAC Multicast, uses VLAN for priority and traffic segregation
Frame size approx. 92 250 bytes
Periodic heartbeat messages of 1-60 seconds interval if no events occur
99% of time just the heartbeat message
In case of event an avalanche can occur as many IEDs detect state changes
Typically used for fast transmission of digital events
Less often for transmission of analogue data, e.g. sent every 250ms
Non-IP traffic in IEC 61850 - GOOSE
5/21/2018 IEC 61850 Network Architectures
16/41
16Copyright RuggedCom Inc
GOOSE is connectionless
No confirmation from receivers
Retransmission to increase the probablity of sucessful reception
A burst of 5-6 messages sent in case of event (critical information)
Example of implementation:
1st message: on event
2nd message: 4ms after event
3rd message: 16ms after event
4th message: 80ms after event
5th message: 500ms after event
Retransmission Scheme in GOOSE
Burst of GOOSEs sent on event occurence
Time
Heartbeat GOOSEs
Event occurs, GOOSE with incremented
stNum sent immediately
5/21/2018 IEC 61850 Network Architectures
17/41
17Copyright RuggedCom Inc
GOOSE and Network Performance
GOOSE messages shall be priority tagged
Configuration needed in IEDs and in Ethernet switches
GOOSE frames with the priority tag in VLAN field configured are
placed in the front of the store and forward queue
Frames already being sent are not interrupted
Delay of frames introduced by network is almost zero
Worst case of total network delay is
100 s at 100MBps links speeds
10 s at 1Gbps
5/21/2018 IEC 61850 Network Architectures
18/41
18Copyright RuggedCom Inc
GOOSE and Network Performance
IEC 61850-5 Type 1A Trip total transfer time defined at 4ms
Transfer time = Application to Application and includes:
GOOSE encoding at sender + network delays + GOOSE decoding at receiver
It is difficult to measure as defined in IEC 61850-5
Because the timestamp is added in IED after the internal function
execution time (one scan period)
Typical measured GOOSE total transfer time including functionexecution time in IED is in the range of 6-12ms
5/21/2018 IEC 61850 Network Architectures
19/41
19Copyright RuggedCom Inc
Encapsulated directly in Ethernet layer
High priority, critical, synchronous and unsolicited
MAC Multicast, uses VLAN for priority and traffic segregation
Currently dedicated wiring (IRIG-B or 1PPS) used for time synch of
devices, future will be IEEE 1588
A Merging Unit (sensor) sends 80 or 256 samples/power cycle. At 50Hz it
is 4000 and 12800 samples per second respectively.
A sample is a set of 8 analog values, 4 voltages + 4 currents
@80 samples 4000 packets/sec
A single Merging Unit uses approx. 4.4 5.2Mbit/s of bandwidth at 80 Smp
The bandwidth used depends of sampling rate and if Data Set is according
to IEC61850-9-2LE implementation or other Data Set
1Gbit Ethernet highly recommended for Process Bus in switched Ethernet
Non-IP traffic in IEC 61850 Sampled Values
5/21/2018 IEC 61850 Network Architectures
20/41
20Copyright RuggedCom Inc
Client-Server services
MMS protocol over TCP, port 102
measurements, events, status indications 100-500ms delay accepted Traffic generated by a single IED rarely exceeds 10kbps
Reports save bandwidth. Digitals via Buffered, Analogs via Unbuffred.
Time synchronization SNTP or IEEE 1588
For redundancy mutiple time masters used
File transfer MMS over TCP, FTP, TFTP, other protocols e.g. Modbus/TCP
Typically Oscillography, sequence of events, data logs. Ocassionallyconfiguration, settings, firmware upgrades, etc. File size typically 4 200 kbytes,
IP based traffic in IEC 61850
5/21/2018 IEC 61850 Network Architectures
21/41
21
Typical Network Architectures
5/21/2018 IEC 61850 Network Architectures
22/41
22Copyright RuggedCom Inc
HMI Gateway
Protection and Control IEDs
Star Topology
Not protected against single point of failure
Simplicity
5/21/2018 IEC 61850 Network Architectures
23/41
23Copyright RuggedCom Inc
HMI Gateway
Protection and Control IEDs
Redundant Star Topology
Blue LAN A
Red LAN B
The entire network is duplicated
Configuration and application complexity, cost issues
Each device has 2 IP addresses, 2 application instances
PRP will be the alternative
5/21/2018 IEC 61850 Network Architectures
24/41
24Copyright RuggedCom Inc
Fiber Optic Ethernet Ring
100/1000 Mbps
HMI Gateway
Dashed Redundant Connections
Protection and Control IEDs
Single Ring Topology
IEDs can be dual homed and connected via
redundant links
Redundancy with RSTP
PRP or HSR will be the alternative
Blue Electrical 100Mpbs
Red Fiber Optic 100Mbps
5/21/2018 IEC 61850 Network Architectures
25/41
25Copyright RuggedCom Inc
Secondary Ring 1
HMI Gateway
Protection and Control IEDs
Multiple Rings Topology
Secondary Ring n
Primary Ring
Limited number of switches in each ring
Minimize recovery time
Division criteria by voltage levels or by several bays
Redundancy with RSTP
5/21/2018 IEC 61850 Network Architectures
26/41
26Copyright RuggedCom Inc
Fiber Optic Ethernet
Ring 100 Mbps
Ring of IEDs
Protection and Control IEDs
Dashed Lines Redundant LAN
Connections
IEDs with Embedded Switch functionality
Multiple rings may be needed
Redundancy with RSTP
HSR will be the alternative
HMI Gateway
5/21/2018 IEC 61850 Network Architectures
27/41
27
Problem of Multicast and Physical vs.
Logical separation of Process Bus and
Station Bus
5/21/2018 IEC 61850 Network Architectures
28/41
28Copyright RuggedCom Inc
Problem of Multicast
Multicast is one-to-many communication scheme
Multicast MAC traffic is by default propagated
through the whole LAN
Consumes link bandwidth and increases latency at
switches
Introduces significant overhead at receiving IEDs ifmulticast addresses not allocated properly
5/21/2018 IEC 61850 Network Architectures
29/41
29Copyright RuggedCom Inc
Impact of Multicast Red MU (Merging Unit) multicasts Sampled Values to small group of IEDs
It is dictated by the protection application
In a large substation there can be dozens of IEDs sending multicast
GOOSE and dozens of Merging Units sending multicast Sampled Values
P
C
MU
C
NTP
P
MU
P
P P
C
MU
IEDIED
Primary Ring
Secondary Rings
P
C
MU
P
C
5/21/2018 IEC 61850 Network Architectures
30/41
30Copyright RuggedCom Inc
Impact of Multicast All nodes get the traffic red area
Repeat for every IED/MU in network
Critical messages delayed or maybe dropped
Steady state traffic load can exceed 100Mbps for many MUs Excessive MU traffic can cause IEDs and PCs can mis-operate or crash
P
C
MU
C
NTP
P
MU
P
P P
C
MU
IEDIED
Primary Ring
Secondary Rings
P
C
MU
P
C
Multicastmustbefiltered
5/21/2018 IEC 61850 Network Architectures
31/41
31Copyright RuggedCom Inc
Multicast Addresses and Traffic Management Efficient layer 2 multicast application
Proper allocation of multicast addresses
Filtering of multicast traffic Allocation of multicast addresses
improves processing times at receiving devices by discardingunwanted multicast traffic at hardware level
required for multicast filtering
Multicast filtering saves bandwidth and decreases latency at network switches by
limiting the traffic only to restricted areas of the network
Multicast filtering solves the primary problem of filtering unwanted
GOOSE and SV traffic Use VLAN or MAC address filtering ?
Static or dynamic filtering methods ?
5/21/2018 IEC 61850 Network Architectures
32/41
32Copyright RuggedCom Inc
Where we are today ?
In todays substations often no multicast
management used at all
Lack of knowledge at integrators and utilities Many users just tend to minimize configuration
efforts and rely on default settings
Until now the dominant method for restrictingmulticast traffic was the use of VLANs
Static configuration: manual process for all IEDs and
all network devices
5/21/2018 IEC 61850 Network Architectures
33/41
33Copyright RuggedCom Inc
Example of MisconfigurationCase Study 50 IEDs in the same network all sending GOOSE
No multicast filtering used Wrong!
All IEDs send multicast with the same destination multicast MAC address Wrong!
In case of event there is an avalanche of GOOSEs in the network and approx 20ms
additional processing delay observed at the receiver Improper functioning!
Implementation internals of an IED Network controller at IEDs has hash table that maps all possible multicast MACs to a
small group of addresses
Hash table permits discard unwanted multicast MACs at hardware level
If all IEDs send with the same multicast destination MAC then at receiving IED these are
mapped to the same hash and need to be discarded by software
In some IED implementations decoding of GOOSE message takes up to 1.5ms
Software decoding of 20 unwanted GOOSE messages can take up to 30ms!
5/21/2018 IEC 61850 Network Architectures
34/41
34Copyright RuggedCom Inc
VLAN (IEEE 802.1Q) Virtual LAN: an independent Ethernet network that shares
cabling infrastructure with other networks
Each VLAN has a separate broadcast domain VLANs permit:
Priority tagging
Logical separation of the network into various domains
Standard
FrameDest. Src. Length / Type Data
6 bytes 6 bytes 2 bytes Variable
Dest. Src. Length / Type Data
6 bytes 6 bytes 2 bytes Variable
TPID TCI
Priority CFI VID
2 bytes
3 bits 1 bit 12 bits
2 bytes
Tagged
Frame
5/21/2018 IEC 61850 Network Architectures
35/41
35Copyright RuggedCom Inc
Use of VLANs
VLAN is suitable mechanism for isolation of unrelated
traffic, eg. surveillance video from SCADA traffic
VLANs configuration can be: Static
Dynamic (GVRP)
Today static configuration is a manual process
Static configuration can be semi-automatic with future
enhanced configuration tools
Can use MAC address filtering instead of VLAN
VLANs for priority tagging in order to increasing
performance
5/21/2018 IEC 61850 Network Architectures
36/41
36Copyright RuggedCom Inc
Traffic Segregation with VLANs
Traffic separated with VLANs:
Substation LAN management
SCADA/Engineering Access
GOOSE Messages
Process Bus (Sampled Values)
Synchrophasors
Protection A vs. Protection B
Differenttrafficflowsinasubstationnetworkmerit
segregatingintoseparateVLANs
5/21/2018 IEC 61850 Network Architectures
37/41
37Copyright RuggedCom Inc
GMRP/MMRP for Dynamic Multicast Filtering Generic Multicast Registration
Protocol
Publisher / subscriber model like
IGMP Multicast filtered by default
must subscribe to get it
Adapts dynamically to anynetwork topology andaccommodates any application of
9-2 or GOOSE edge only pruning results in no
traffic delay after topology change
Allows process and station bus toco-exist on same physicalnetwork
P
C
MU
C
P
MU
P
P
Primary Ring
Secondary Rings
SV producer
simply multicasts no change
SV consumer sends a
subscribe message tonetwork periodically
Switches prune the trafficautomatically. Eitheroptimally or edge ony
FirstIEC61850110kVsubstationwithIEEE1588v2and
dynamicGMRPmulticastfilteringcommissionedin2010
5/21/2018 IEC 61850 Network Architectures
38/41
38Copyright RuggedCom Inc
Problem of Time Synchronization and Data Sharing
Process Bus requires that Sampled Values coming from different
sensors and received by an IED have to be synchronized
Synchronization islands are possible, each island spans a
protection zone
Problem of Line Differential protection with one line end using
Process Bus and the other line end using conventional wiring
5/21/2018 IEC 61850 Network Architectures
39/41
39Copyright RuggedCom Inc
Process Bus and Station Bus Separation
Process Bus and Station Bus are logically different
Multicast traffic from Merging Units flooding the network
A single Merging Unit consumes approx 5Mbps of bandwidth The problem of busbar protection based on Process Bus
In a topology with 60 feeders a process bus based busbar protection would
have multicast traffic of > 400Mbps!
5/21/2018 IEC 61850 Network Architectures
40/41
40Copyright RuggedCom Inc
Physically or Logically Separate Networks?
Physically separate LANs are more costly network switches are
duplicated
Physically separate LANs are perceived as more secure Logically separate LANs are more flexible as Merging Units can be
accessed from SCADA (remote maintenance, management, etc.)
Logically separate LANs require network engineering or more
sophisticated dynamic methods (GMRP, GVRP, etc.)
Station Bus could also be connected to Process Bus via router
5/21/2018 IEC 61850 Network Architectures
41/41
41
Questions?
Top Related