yadavpre_gi10

8/2/2019 yadavpre_gi10

http://slidepdf.com/reader/full/yadavpregi10 1/23

IP and Ethernet CommunicationTechnologies for Substations

Substation of the Future: Improve

Network Reliability and Protocol

© 2010 Cisco Systems, Inc. All rights reserved. -Ver 0.3 1

Navindra Yadav (Principal Engineer), [email protected]

Eruch Kapadia (Solution Architect), [email protected]

December 2, 2010

n erac on

Grid InterOp Dec 2010

Grid-Interop 2010



Outline

Reliability and Availability Basics Inter Substation Traffic

– Use Cases like Tele Protection, Primary Protection, etc

– Reliability and Design improvement

Intra Substation Traffic

© 2010 Cisco Systems, Inc. All rights reserved. 22

– Use Case local Protection

– Reliability and Design improvement

Intra Substation Designs to achieve 6+ Nines ofSystem Availability

Grid-Interop 2010



Reliability and Availability Basics…

© 2010 Cisco Systems, Inc. All rights reserved. 3Grid-Interop 2010



Measurement of Availability…

Availability = (MTBF)/(MTBF + MTTR)

– MTBF = Mean Time Between Failure

– MTTR = Mean Time To Repair

Unavailability = ( 1 – Availability )


va a y = – nava a y

• AvailabilityOfSerialParts = ∏(AvailabilityOfPart)

• AvailabilityOfParallelParts = 1 - [ ∏(1-AvailabilityOfPart)]

Grid-Interop 2010



Calculation of Network Availability Identify the Serial and Parallel components

1 3 5 7

2

9

4

2 64

1 3 5

6 8

9 11 13 15


• Failure of any node, link disrupts

• Network Availability = A = A1 * A2

* A3 * A4 * A5 * A6 * A7

• If all A* are 90%.

• A = 0.4782969 = 47%

• Assume a simple Active-Active Design

• Network Availability = 1 – (UnAvail_P1* UnAvail_P2 * UnAvail_P3) =

• = 1 – ((1-Avail_P1) * (1-Avail_P1) * (1-Avail_P1))

• If all A* are 90%

• A= 1 – ((1-0.59049)* (1-0.729) * (1-0.4782969))

• = 1 – (0.40951* .271 * . 0.5217031)

• = 0.942102845513649 = 94%

Grid-Interop 2010



IEC 61850 GOOSE and SV over the WAN


Challenges and Solutions

Grid-Interop 2010



IEC 61850-90-1 Solution to Carry GOOSE/SVover the WAN

Tunnel. Example

– Layer 2 Tunneling Protocol (L2TP / L2TPv3) – RFC 3931

– Generic Routing Encapsulation (GRE) Tunneling - RFC 2784

Gateway


– xamp e roxy a eways

– GWs Must Terminate Protocols

– GWs must Understand Applications and configuration changes inthe application

– Latency and Jitter addition, especially when GWs areimplemented in software

Tunneling or Encapsulation is the more realistic option

– MPLS, VPLS, PWs are examples of Encapsulation technologies

Grid-Interop 2010



Problem: IEC 61850 GOOSE/SV over WAN – Layer2Tunneling – Fault Domain Extension – Lower Availability

This approach creates Large Fault Domains – Substation network Faults spread

– Dramatically lower availability

• Faults in unrelated parts of the network propagate

• Calculation of Availability means factoring Availability of the entire L2domain – means UNRELATED networks too!!




Solution: IEC 61850 GOOSE/SV over IPv4/v6 –Higher Availability

Small and contained Fault Domains with IPv4/v6

– Layer 2 domains are small – Substation network Faults do NOT spread

– Higher availability

• Calculation of Availability involves only relevant networks




Problem: Layer 2 GOOSE / SV over the WAN – Implications on Scaling,Security, Replication, Flooding, etc


Issues:

Intra Substation Replication

Inter Substation Replication

Information Leakage – Security Implications

Wasted BandwidthLimited Scale

… Grid-Interop 2010



Solution: IEC 61850 with IPv4/v6 profile provides -Scalability, Security, etc

GOOSE/SV on IPv4/v6 routable protocol – Scalable

– Low (in usecs) Latency – All HW forwarding Path

– Low (in usecs) Jitter

– Cyber Security benefits

– Easy to trouble shoot and manage over WAN – proven model

– …




IEC 61850-90-5 for PMUs is working on a 61850 profileto carry GOOSE / SV over TCP/IP[v4v6] stack

IP profiles being developed above must Not only be

restricted to PMUs, but also to other all relays and


app ca ons.

Grid-Interop 2010



Comparison IEC 61850 SV/GOOSE over EthernetVs UDP/TCP/IP[v4/v6] Stack

Topic IEC 61850 overEthernet

IEC 61850 overUDP/TCP over IPv4/v6

Maturity Deployed in ProductionEnvironments

Standard Drafting Stage

(IEC 61850-90-5 draft has a IPv4/v6profile for GOOSE and SV)

IED/Relay Vendor Support Wide Spread ---

Scalability – Tunnels /Encapsulation

Low (10s of sites).

Overlay on top of the WAN topology.

Mgmt/Troubleshooting burden

Extremely High (Internet Scale)

Native Routing and Forwarding

Lower Availabilit . WAN tunnel failure Fast Conver ence techni ues can


is detected in order of seconds.

repair WAN faults in order of sub 50msecs

Inherent Network State Overlay Tunnels, etc createadditional state in the Network.Which reduces the overall availabilityof the solution.

Less state in the network

Fault Domain Size LARGE. Layer 2 faults spread acrosssubstation domains. Eg loops in onesubstation may disrupt other substations

SMALL. Restricted fault domains

Inter Substation TrafficReplication (Multicast)

Inefficient.

Replication at WAN edge boundary(overlay).

Efficient.

WAN Network replicates at mostoptimal points

Intra Substation TrafficReplication (Multicast)Inefficient.Flooding inside the substation LAN(Vlan)

Efficient. No flooding inside thesubstation – multicast delivered toonly interested hosts (IGMP/MLDsnooping)Grid-Interop 2010



Comparison Contd…Topic IEC 61850 over

EthernetIEC 61850 overUDP/TCP over IPv4/v6

Latency High.

Tunneling (typically in software/ucodefast path - msecs)

Protocol Translation (typically insoftware – 10 to 100 msecs)

Low.

Extremely low latency ( < 20 microsecs).

ASIC based forwarding

Jitter Higher Jitter (order of multiplemsecs)

Low (few usecs) Jitter. ASIC basedforwarding.

Cyber Security Weaker. Superior.


Large Flooding domain acrosssubstations create securitychallenges (like spoofing, replay,DOS, info leak domains)

No flooding. Traffic delivered toauthorized parties only.

State Scaling Domain (Eg. MacTable size)

Inferior. Limited all substations learnof everyone else… thanks to flooding

Superior. Only stations thatcommunicate with each other need toknow about each other

Layer 2 Media dependentsolution

Yes No

Grid-Interop 2010



Intra Substation Network Topology Choices…




Some Characteristics of Ring Topology




Some Characteristics of Redundant Tree Topology




Comparison notes between the two leading TopologiesTopic Redundant Trees Rings

Physical Redundancy Yes Yes

Connectivity/Topology More conductor.

If conductors are laid inside trenches and conduits with

limited capacity for extra cables its an issue for trees.

Simpler, Less conductor/fiber. In some cases trench and conduit size may

make ring the only viable option.

Predictable Latency Fixed and deterministic latency. Tree depth determines the

number of hops.

Latency varies. The number of hops between the source and the

destination depends on where the loop in the ring is broken. When the

blocking point changes the latency also changes.

Smaller Fault Domain Smaller Fault Domain – (fault limited to the triangle of

switches in the tree)

The whole ring is the fault domain.

Bandwidth / QoS

Predictability

All inter switch traffic contends at limited and few fixed

points in the tree topology.

All inter switch traffic contends for the ring bandwidth

Scalability Superior (only the the root switches need to have policies

and mac addresses for every device. The leaf switches ust

Inferior (all switches have to learn about all end points. Least capable

switch determines the ca acity of the rin


need to have capacities to support their downstream end

devices)

Maintenance and

serviceability

Superior (no downtime to the network to add a new leaf

switch)

Inferior (downtime seen by the ring when a switch is added or deleted

from the ring)

Fairness Fewer and equal number of contention points, through

which all traffic passes when going between two access

switches, yields a fairer system

Traffic sent by the edge switches has to compete with similar class of

traffic at every hop on the ring, the contention points and their number

can change over time. Also the contention points vary between twoaccess switches.

Fast Convergence Fast convergence can be achieved by using FlexLinks.

Zero down time with PRP (from IEC 62439).

Faster convergence (sub 50 msecs) can be achieved by using some

like REP. Zero down time with PRP (IEC 62439) protocols.

Availability Fewer and fixed number of switches in the switching

path results in a higher MTBF. As there are fewer

switches to switch through. Also aggregation switches

can be designed for higher availability.

Variable MTBF as the number of switches in the number of switches

vary depending on the topology.

Cyber Security /

Traffic containment

Superior (not all switches have to have all vlans, also the

flooding domains are smaller)

Authenticator Function (for protocols like 802.1x) only

on Switches

Inferior. (all switches have to have all vlans).

If the Ring nodes are end points then over all security suffers. End hosts

do not authenticate other end hostsAuthenticator Function (for protocols like 802.1x) has to be pushed to alldevices on the ring like end devicesGrid-Interop 2010



Substation Network Designs for 6+ Nines of




Designs to meet or exceed 6+ Nines of Availability(Proactive and Reactive Redundancy) with Redundant 61850 Actors

Green Network is ActiveRedundant Network. Smaller insize as not all devices will bedual attached

Blue Network is Active PrimaryNetwork. All devices are at least

attached to the Blue network

WANWAN

SUBSTATION


BIEDMUBIEDMU

Power Line

BPU BPU

Superior Availability Characteristics• Above can be built using a Tree or Ring topologies• Active-Active Design• No down time due to network convergence events• Zero Down time for any failure• N-1 Redundancy for any failure• Can be improved even further if Blue and Green end points can process each others

updates besides their own too. Get to N-2 redundancy

Other

Legend:

MU = Merging Unit

BPU=Bay Protection Unit

BIED= Breaker IED

Redundant Actors and sensors

Simplified view of the Power Network

Grid-Interop 2010



Designs to meet or exceed 6+ Nines of Availability(Proactive and Reactive Redundancy) without Redundant Actors

Green Network is ActiveRedundant Network. Smaller insize as not all devices will bedual attached

Using say IEC 62439 - PRP

Blue Network is Active PrimaryNetwork. All devices are at least

attached to the Blue network

WANWAN

SUBSTATION


BIEDMU

Power Line

BPU

Superior Availability Characteristics• Above can be built using a Ring topology too• Active-Active Design• No down time due to network convergence events

• Zero Down time for any network device failure

Other

Legend:

MU = Merging Unit

BPU=Bay Protection Unit

BIED= Breaker IED

NO Redundancy of Actors andsensors

Simplified view of the Power Network

Grid-Interop 2010

yadavpre_gi10

Documents

Transcript of yadavpre_gi10