61850 Communication Networks
-
Upload
cyrus-kariuki -
Category
Documents
-
view
40 -
download
2
description
Transcript of 61850 Communication Networks
Communication networks for IEC
61850 systems
Dr. Alexander Apostolov
Questions
• What are we doing?
• Why are we doing it?
• How are we doing it?
Page 2
Technical Report IEC 61850-90-4
• IEC 61850-90-4 Technical report:
Network Engineering Guidelines
• Provides definitions, guidelines, and
specifications for the network
architecture of IEC 61850 based
systems
• Intended for an audience familiar with
network communication and/or
IEC 61850 based systems .
Technical Report IEC 61850-90-4
• Not a tutorial on networking or
IEC 61850
• It references and summarizes standards
and papers from other working groups
that may assist the engineers.
• Still provides significant amount of
useful information for people with basic
understanding of communications
technology
Technical Report IEC 61850-90-4
• Focuses on engineering a LAN/WAN
that is limited to the requirements of an
IEC 61850 based automation system.
• It outlines the advantages and
disadvantages of different approaches
to network topology, redundancy, clock
synchronization, etc. so that the network
designer can make educated decisions.
• Outlines possible improvements to both
IEDs and networking equipment.
Technical Report IEC 61850-90-4
• Addresses the most critical aspects of
IEC 61850 such as protection related
tripping via GOOSE and SVs.
• Addresses in particular the multicast
data transfer of large volumes of
sampled values (SV) from merging units
(MUs).
• Considers the high precision clock
synchronization and “seamless”
guaranteed transport of data across the
network under failure conditions that is
central to the Process Bus concept.
Technical Report IEC 61850-90-4
• Addresses the most critical aspects of
IEC 61850 such as protection related
tripping via GOOSE and SVs.
• Addresses in particular the multicast
data transfer of large volumes of
sampled values (SV) from merging units
(MUs).
Technical Report IEC 61850-90-4
• Considers the high precision clock
synchronization and “seamless”
guaranteed transport of data across the
network under failure conditions that is
central to the Process Bus concept.
• This report intentionally omits the
subject of network security, since it is
within the scope of IEC TC57 WG15.
Ethernet Technology for Substations
• Ethernet subset for substation
automation
• Topology
• Physical layer
• Link layer
• Network layer
Network Design Checklist
• Substation topology and physical
locations of IEDs
• Protection and control application
• Resiliency and redundancy
• Reliability, availability, maintainability
• Logical data flows and traffic patterns
• Performance
• Latency for different types of traffic
• Time synchronization and accuracy
• Network management
Network Design Checklist
• Environmental issues
• EMI immunity
• Form factor
• Physical media
• Cyber security
• Scalability, upgradeability and future-
proof
• Testing the design
• Cost
Network Topologies
• Single bridge
• Star
• Simple ring
• Multiple ring
• Process Bus
• Station and Process Bus connection
• Duplicate
Other Topics
• Dependability issues – requirements for
availability and reliability,
maintainability, dependability, risk
analysis
• Network configuration - assignment of
IP addresses
• Performance issues
• Quality of service
Other Topics
• Latency requirements
• Traffic control
• Clock synchronization
• Network security
• Network management
• Network testing
Network Symbols
IEC 61850 Engineering Process
Page 16
Allocate Logical Nodes (Protection & Control Functions) to the Single Line Diagram
Application
requirements &
design criteria
Select and assign IEDs to SLD covering all Logical Nodes,
perform proof of concept, interoperability testing before engineering
Network design including switch
configuration (RSTP, PRP, HSR,..)
Select data communications schemes and protocols (Client-Server services to
SCADA, GOOSE & SV messages, network-based time synchronization) and
other Ethernet-based applications such as video surveillance (SNMP, etc.)
Final communication network design
Select the network topology and redundancy technique used for SA
Perform convergence tests
Are reliability requirement
met ?
Review design network architecture
Yes
No
Decide on primary technology (AIS, GIS, Hybrid GIS, etc.).
Decide to use electronic transformers, intelligent switchgear and
Process Bus
Select physical locations of IEDs inside main building, in close proximity to
primary equipment e.g. mounted outdoor or in kiosk or containers.
Select switches, perform proof of concept, interoperability testing before
engineering, more than one vendor switches are used. Conduct network contingency
analysis to ensure the network topology will meet the SAS reliability requirement
Partially Redundant Network
Page 17
switched local
area network
(LAN)
node
node node
node
edge link
trunk link
switching nodes
node
nodetrunk link
trunk link
node
edge linkedge port
trunk port
switch
edge port
802.1Q tag
Page 18
Destination(6) Source(6)
XXXX
ET (2)
tag (4)Destination (6) Source ET (2) LPDU 46..1500 octest
0x8100 0xXXXX
VLAN Tag (2)
12-bit 802.1Q VLAN Identifier
Canonical - 1 bit
3-bit Priority Field (802.1p)
LPDU 46..1500 octets
Ethernet layer MAC header (layer 2) without 802.1Q tag
Ethernet MAC header (layer 2) with 802.1Q tag
FCS
FCS
Ethertype
Network Traffic
Page 19
Supervisory
Level
clock
SCADAHMI
gateway
firewall
Grid Control
bay
Station Bus
SCADA
IED
IED
IED
Process
Level
8-1 MMS
client-server traffic
SV
9-2 Sampled Valued
hard real-time traffic
pro
cess b
us
SCADAHMI
Engineering
8-1 GOOSE
soft real-time traffic
bay
IEDIED
IEDIED
IEDIED
bay
IEDIED
IEDIED
IEDIED
bay
IEDIED
IEDIED
IEDIED
bay
IEDIED
IEDIED
IEDIED
bay
IEDIED
IEDIED
IEDIED
horizontal communication
Vert
ical com
munic
ation
Electrical and Bay Topology
Page 20
bay 1 bay 1 bay i bay j bay k bay w
engineering
gateway
networkcontrol centre
loggerprinter
station bus
GPStime
operator workplace(SCADA)
bay 3 bay N
IED
switch
IED
IED
IED
switch
IED
IED
IED
switch
IED
IED
IED
switch
IED
IED
IED
switch
IED
IED
IED
switch
IED
IED
G
switch switch
switch
RSTP
• IEC 62439-1
• RSTP is a widely used redundancy
protocol standardized as
IEC/ISO 8802.1D.
• RSTP is flexible and can convert an
arbitrarily meshed network topology
to a logical tree, by eliminating loops
that redundant links would introduce
on the physical level.
Page 21
RSTP
• RSTP operates by peer-to-peer
messages, so-called BPDUs (Bridge
Protocol Data Units), between bridges.
• In normal operation, each node
indicates to its neighbors the costs to
reach the root node, which is at the top
of the hierarchy (independent of its
physical location).
Page 22
RSTP
• When a bridge can reach the root
through multiple ports, it considers only
the port with the cheapest costs and
blocks the other ports.
• If path costs are equal, the port number
is used to break the tie.
• The path costs considers the number of
intermediate bridges, communication
speed of the link between the bridges
and the identity of the bridges. Page 23
RSTP Principle
Page 24
inter-switch
link
edge portleaf
link
edge
links
alternate port
{blocked}
end
node
root bridge
end
node
end
node
end
node
end
node
end
node
end
node
end
node
end
node
end
node
end
nodeend
node
end
node
end
node
end
node
end
nodeend
node
end
node
end
node
end
nodeend
node
end
node
PORT STATE
learning
blocking
forwarding
PORT ROLE
Root (goes to root bridge)
Designated (goes away from root)
{ BackUp / Alternate}
Edge
root port
{forwarding}
designated port
{forwarding}
edge
ports
alternate port
{blocked}
inter-switch
link
edge portleaf
link
edge
links
alternate port
{blocked}
end
node
root bridge
end
node
end
node
end
node
end
node
end
node
end
node
end
node
end
node
end
node
end
nodeend
node
end
node
end
node
end
node
end
nodeend
node
end
node
end
node
end
nodeend
node
end
node
PORT STATE
learning
inter-switch
link
edge portleaf
link
edge
links
alternate port
{blocked}
end
node
root bridge
end
node
end
node
end
node
end
node
end
node
end
node
end
node
end
node
end
node
end
nodeend
node
end
node
end
node
end
node
end
nodeend
node
end
node
end
node
end
nodeend
node
end
node
PORT STATE
learning
blocking
forwarding
PORT ROLE
Root (goes to root bridge)
Designated (goes away from root)
{ BackUp / Alternate}
Edge
root port
{forwarding}
designated port
{forwarding}
edge
ports
alternate port
{blocked}
(Rapid Spanning Tree Protocol)
RPR
• PRP is specified in IEC 62439–3,
Clause 4 as a protocol that offers
seamless failover.
• RSTP achieves redundancy through
redundant network paths and a failover
protocol.
• PRP relies on complete duplication of
the LAN.
Page 25
RPR
• Both LANs operate in parallel with a
source node duplicating a frames to
send and the destination nodes
discarding the duplicates on the base of
their source and of a sequence number
appended to the frame’s payload.
• To achieve this, a PRP node is a doubly
attached node (DANP) with two ports,
one for each redundant LAN.
Page 26
RPR Principle
Page 27
switched local
area network
(tree) LAN_B
RB
switched local
area network
(ring) LAN_A
DANP
DANP DANP DANP
SAN
A2 SAN
B1
SAN
B2
SAN
A1
SAN
R1
SAN
R2
DANP
HSR
• HSR is specified in IEC 62439-3 Clause
5 and provides seamless failover.
• HSR uses the principle of frame
duplication of PRP, but achieves
redundancy through only a single
additional link.
• Nodes in HSR have (at least) two ports,
the nodes are daisy-chained, with each
one node connected to two neighbor
nodes Page 28
HSR
• The last node being connected to the
first node and closing the line to a
physical ring structure
• HSR uses nodes similar to PRP nodes
with two network interfaces (DANH).
• A node must be able to forward frames
from port to port at wire speed, which
requires a cut-through bridge in each
node and therefore a hardware
implementation. Page 29
HSR
Page 30
destinations
node node nodenodenode
nodenode
source
„A“-frame
(HSR tagged)
„B“-frame
(HSR tagged)
„C“-frame „D“-frame
AB
node
destinations
Single Switch Topology
Page 31
NC
P
C
MU
P
C
MU
P
C
MU
P
C
MU
bay bay bay bay
Star Topology
Page 32
NC
P
C
MU
P
C
MU
P
C
MU
P
C
MU
bay bay bay bay
Redundant Star Topology
Page 33
PA PBC
AB
NC
Station Bus (2 x RSTP)
PA PBC PA PBC
LAN A
LAN B
Simple Ring
Page 34
P
C
MU
NC
bay
P
C
MU
bay
P
C
MU
bay
Ring of Bridging IEDs
Page 35
PCM PCM PCM PCM PCM PCM PCM PCM
NTP
voltage level 1 voltage level 2
Redbox HRS-RSTP
DANH - IEDs
Separate Switches for Main 1 and 2
Page 36
P1
C
MU
bay
P2
MU
P1
C
MU
bay
P2
MU
P1
C
MU
NC
bay
P2
MU
Multiple Station Bus Ring
Page 37
PCM
PCM
PCM
PCM
PCM
PCM
PCM
PCM
PCM
PCM
PCM
PCM
NC
Ring-Ring Topology with RSTP
Page 38
PCM
PCM
PCM
PCM
PCM
PCM
PCM
PCM
PCM
PCM
PCM
PCM
PCM
PCM
PCM
PCM
PCM
PCM
PCM
PCM
PCM
PCM
PCM
PCM
NC
Rings and Sub-rings
Page 39
P
C
MU
P
C
MU
P
C
MU
Primary Ring
Secondary Ring (voltage level 1) Secondary Ring (voltage level 2)
bay
P
C
MU
P
C
MU
P
C
MU
BP IEDIEDs
NC
bay bay bay bay bay
bussbar
protection
Rings of Rings with HSR
Page 40
remove these switches
P
C
MU
P
MU
P
C
Primary Ring
Secondary Ring (voltage level 1) Secondary Ring (voltage level 2)
P
C
MU
P
MU
P
C
BP IEDIEDs
NC
bay bay bay bay bay
bussbar
protection
bay
Hierarchical Redundant Ring
Topology with HSR/PRP
Page 41
P1 C MU1 MU2
RB A RB B
P2
Process Bus
AB
P1 C P2 MU
RB A RB B
Process Bus
P1 C P2 MU
RB A RB B
NC
Station Bus with PRP (2 x RSTP)
Process Bus
Process Bus Point-to-Point
Page 42
IA1
IA2
UAL
IAL
UAS
IC2
IC1
UCL
ICL
UCS
IB1
IB2
IED
PMC1
U/I sensors
I sensors
switch control
actor
I sensors
U/I sensors
switch control
I sensors
I sensors
station bus with
8-1 and 9-2 traffic
PIPI
PIPI
PIPI
PIPI
PIPI
PIPI
PIPI
PIPI
PIPI
PIPI
PIPI
PMC2
Process Bus Ring
Page 43
IA1
IA2
UAL
IAL
UAS
IC2
IC1
UCL
ICL
UCS
IB1
IB2
U/I sensors
I sensors
switch control
actor
I sensors
U/I sensors
switch control
I sensors
I sensors
8-1 traffic
(HSR)
9-2 traffic
PMCB
PI
PI
PI
PI
PI
PI
PI
PI
PI
PI
PI
PMCA
RBdiagnostics
gateway
Station and Process Bus Rings
Page 44
HSR
RedBox A
A B
A
A B
B
PA C PB PM
Process Bus
A B
PA C PB PM
A B
PA C PB MUMU
B
NTP
PRP
Station Bus
HSR
RedBox B
Process Bus Process Bus
Separate Station and Process Bus
Page 45
Station Bus
Process Bus
MU MU MU
NTP
P C P C
MU MU MU MU MU
BC
Separate Station and Process Bus
Page 46
P C
Station Bus
Process Bus
MU MU MU
NTP
MU MU
P C P
Multicast Domain
Page 47
Station Bus (e.g. RSTP ring)
PI = Process Interface
MU = Merging Unit
PIP
PIPI
PIPI
PIMU
PIMU
PA
PI
PI
PI
MU
MU
HSR process bus
PBPA
PI
PI
PI
MU
MU
HSR process bus
PBPA
PI
PI
PI
MU
MU
HSR process bus
PB
PIP
PIPI
PIPI
PIMU
PIMU
NTP
multicast domains
that allow to limit
traffic on the trunk
Hierarchy of Clocks
Page 48
Test Setup
Page 49
Annexes
• Annex A: IEC 61850 bridge object
model
• Annex B: IEEE 1588 Clock model
• Annex C: Case study - Process Bus
configuration for busbar protection
system
• Annex D: Case study - An IEC 61850
Station Bus (Powerlink, Australia)
Annexes
• Annex E: Case study - Simple
Topologies (Transener/Transba,
Argentina)
• Annex F: Case study - Station Bus
configuration in a sophisticated
application with VLANs (Trans-Africa,
South Africa)
Annex F: Trans-Africa, South Africa