8/2/2019 Lessons Learned Autumn08

1/6

C

E

F

D

The Cloud is a term used to depict

telecommunication network. Th

Cloud originated in AT&T sales presentations in the 1970s for data com

munications networks. The though

behind The Cloud simply was tha

the specific internal workings coul

be many and various, change from

time to time, and vary from plac

to place. The Cloud served to obscure those internal workings from view. The Clouwas a conceptual sale; data simply popped in on one end of the network and poppe

out on the other. Interestingly, the data network that gave rise to The Cloud neve

worked, but The Cloud lives on and now provides a perfect label for Ethernet network

Inside

the Cloud:Ethernet for

Teleprotection

A

3

4

1 4

3

5

7

6

2

21

B

lessonle

arned

28

by S. Ward, RFL Electronics, USA and A. Saciragic, Maritime Electric Company, Limited (A Fortis Company), Canada

PAC.AUTUMN.2008

1 System confguration and protection signaling

Solveig received

her M.S.E.E. from the

Royal Institute of

Technology, Sweden

n 1977. The same

year she joined ABB

Relays. She has held

many positions in

Marketing,

Application, and

Product Manage-

ment. After

transferring to ABB

n the US in 1992,

she was Product

Manager for ABBs

ine of current

differential and

phase comparison

relays. Solveig

has authored and

presented technical

papers at Protective

Relaying

Conferences. She

s a member of

EEE and holds one

patent.

n 2002, Solveig

oined RFL Electron-

cs Inc. as Director of

Product Marketing.

Substation

Substation

8/2/2019 Lessons Learned Autumn08

2/6

PAC.AUTUMN.2008

29

Ethernet communicationproducts and applications are be-coming more entrenched as the newstandard for data transport. Carri-

ers are cutting costs by eliminatingthe need for overlay networks thatrequire different equipment for dif-ferent service types.

The primary goal for carriers isto accommodate all services on asingle network and Ethernet IP (In-ternet Protocol) appears to be theclear winner. Ethernet networkscost less to build due to lower costswitches and they handle datatransfer more efficiently then thetraditional circuit-switched, time-division-multiplexed (TDM)alternatives.

Connecting traditional voice,video and data over Ethernetnetworks has become an attrac-tive alternative to running parallelvoice and data networks. It savesmoney on call and leased-line ser-vice charges, while consolidatingmanagement, cutting mainte-nance costs, and increasing userproductivity. This is achieved byconverging two important traffictypes onto one infrastructure, andtakes advantage of the simplicityand efficiency of IP routing andEthernet switching.

Voice over IP (VoIP) was firstintroduced in the mid-90s and pro-vides an acceptable QoS (Quality ofService) for telephone calls, but itslatency makes it unsuitable to usefor synchronous TDM data, suchas T1/E1. Consequently, a comple-mentary technology, TDM over IP(TDMoIP), was developed.

TDM over IP duplicates tradi-tional TDM services over an IP net-work. TDM services can be used totransport synchronous data, asyn-

chronous data, 4 wire telephonequality voice, POTS, high qual-ity audio, analog telemetry, or anyother low speed communicationsrequirement. T1/E1 data streamsgenerated from a multiplexer aretransported over a packet basednetwork, recovered and turned

back into synchronous data streams

at the receiving location.Ethernet Teleprotection

Case Study

In 2006, Maritime Electric

Company Limited (A Fortis Com-pany) decided to perform telepro-tection over Ethernet microwaveradios. For cost reasons, this wasthe only communication mediathat was readily available. The pro-curement of new communicationsequipment was not cost effective.The Ethernet radios were alreadyin existence for SCADA and tele-phone traffic. The communicationnetwork consists of backbonesystem with powerful 4T1 Eth-ernet radios, while T1 Ethernetradios were installed in a spursystem. Ethernet microwave ra-dios installed in the spur systemhave 768 kbps bandwidth, whichis shared between the protectionchannels, the SCADA channels andtelephone circuits. SCADA wastransported with DNP over TCP/IP, and phones had VoIP convertersso they were connected directly tothe router.

The system comprised four sub-stations and two additional micro-wave radio repeater stations. Therelatively simple protection signal-ing was as follows:

Uni-directional from Station1 to 4 (transfer trip, DTT)

Uni-directional from Station2 to 4 (DTT)

Bi-directional between Sta-tion 3 and 4 (distance relay permis-sive trip, PTT and DTT)

Teleprotection logic scheme atStation 4 had to satisfy certain sta-tus signals - breaker/switch-open/close at Stations 1and 2 as a resultof system configuration changes toachieve better system reliability.

TDM over EthernetOur first approach was to use

standard teleprotection equipmentwith TDM over Ethernet (TDMoE)converters to transport the syn-chronous teleprotection system(TPS) channels over the Ethernetnetwork. The converters are simi-lar in performance to TDMoIP but

use a proprietary method. (There isa TDMoIP standard, but the tech-nology is owned by RAD, whichis why some other manufacturersprefer to use proprietary methodsto avoid paying royalties).

The TDMoE Ethernet gate-ways have a T1 interface on theTDM side and each 64 kbps chan-nel needed one TDMoE gateway

The primary goalfor carriers is

to accommodate allservices ona single network,

and Ethernet IPappears to be the clear

winner

AlisaSaci

P.Eng, is a

Superinten

the Engine

DepartmenMaritime E

Company

(A Fortis Co

Her major

responsibil

include pow

system

protection

control app

SCADA, sys

tomation, c

cation, sup

of enginee

new capitalation and

of the exist

infrastructu

ated by Ma

Electric. Ali

received h

from the U

of Sarajevo

and Herzeg

1978. Alisa

registered

professiona

engineer in

province o

Edward Isla

a member

Canadian S

Association

by S. Ward, RFL Electronics, USA and A. Saciragic, Maritime Electric Company, Limited (A Fortis Company),Canada

2 Wind turbune& collector circuit

8/2/2019 Lessons Learned Autumn08

3/6

lessonle

arned

30

PAC.AUTUMN.2008

protection Systems was tested. Thapproach uses a new device that

based on IEC 61850 GOOSE mesaging. Note that these substation

did not use IEC 61850 and it waonly the teleprotection devices thused this protocol for communications over the Ethernet networAfter this upgrade, the systemlooked as in figure 4.

The TDMoE converters wertaken out and replaced with a direcIEC 61850 compliant, Ethernemodule in the Teleprotection Sytem. Station 4 required three Ethernet modules due to the fact ththe router did not support VLAconfiguration. The router providevirtual point-to-point connectionvia proprietary software and againas Station 4 needed to communcate with all three remote substations; three separate ports on throuter were required.

Using VLAN over a router thsupports multicast messages woulhave eliminated two of the threEthernet modules, as the Teleprotection System supports multicaas defined in IEC 61850.

Factory tests of the Etherneteleprotection system were peformed. The factory trip times fo500,000 trips at a rate of 5 per second through a fiber optic Etherneswitch were recorded.

Re-transmission times were ms, i.e. the GOOSE trip messagwas repeated 2 times, each 4 mapart.

Average trip time = ~3 msNumber of trips receive

within 5 ms = 99.7%Number of trips receive

within 10 ms = 99.99%Number of trips receive

within 17 ms = 100%

channel.The reason for this extra band-

width on the IP side is that theTDMoE gateway needs to convert

64 kbps synchronous data into IPpackets and during this process ad-ditional overhead such as IP framing

bits and bits for clock recovery areadded. Clearly this was a problem,especially for Station 4 where three64 kbps protection channels wereused. As the three 64 kbps chan-nels, when converted to IP requireda continuous bandwidth of 3 x 200kbps, this practically consumedall of the available bandwidth. Notunexpectedly, field testing showedthat a phone call would interruptprotection communications andthe scheme was determined to beunsuitable for the application.

In addition, the measured one-way trip t imes between Station 1and 4 with the TDMoE converters

jitter setting setup at 15 ms, werein an average of 38 ms and betweenStation 2 and 4, an average of 39ms.

Shorter jitter buffers, that wouldhave decreased the latency, resultedin an error message of data under-flow. Data underflow means thatthe converter did not receive a pack-et within the maximum allowedtime period for TDM conversion.

While the poor results in thisapplication were mainly due tothe limited bandwidth, it doespoint out the importance of takingthe network design into account.

While the theoretical bandwidthmight seem more than sufficient,converters needed for legacy devic-es may consume much more band-width than what would be obviousat first glance. This is certainly of

importance when tr ansitioninginto an IP-based communicationsnetwork that also needs to trans-port a large amount of data fromlegacy devices.

GOOSE over Ethernet

Having concluded that the TD-MoE gateways could not be used, anew approach with Ethernet Tele-

that converted the synchronous 64kbps channel for the teleprotectionsystem to Ethernet packets. On thereceiving end, the reverse processtook place. The Ethernet portionof the communications scheme isshown within the dashed lines infigure 3.

It seems logical that there couldhave been just one TDMoE T1 con-verter at Station 4, but unlike a T1multiplexer, the TDMoE devicecan not do grooming, i.e. separatethe Ethernet packets to differentdestinations. Station 4 needed tosend each 64 kbps channel to eachof the other three substations; con-sequently three converters wererequired.

Note: The TDMoE devices weretested in the factory, and the resultswere reassuring.

Typical end to end delay timeswere in the order of 5 9 ms whichmet the requirements for the ap-plication. However, these timesdid not include any delays intro-duced by a network, and field re-sults proved to be very different,for a number of reasons. One of themain problems was the fact that theEthernet microwave radios havevery limited bandwidth (768 kbps)

and this bandwidth is shared be-tween the protection channels, theSCADA channels and telephonecircuits.

During commissioning it wasnoted that the TDMoE converterswere consuming around 200 kbps ofIP network bandwidth per 64 kbps

TDM overEthernet

gateways couldnot meet theperformancerequirements.

Teleprotectiondevices use IEC 61850

GOOSE messaging

G

OOSEApplications

8/2/2019 Lessons Learned Autumn08

4/6

8/2/2019 Lessons Learned Autumn08

5/6

PAC.AUTUMN.2008

lessonle

arned

32

The test results indicated that99.7% of trips were based on thefirst GOOSE message, 0.29%tripped on the first re-transmis-

sion and 0.01% needed the secondre-transmission. The teleprotec-tion device sequentially, numbersand logs the GOOSE messagesmaking it easy to verify networkperformance.

Tests were repeatedat com-missioning, and the end-to-end triptimes over the network were 8 -10ms.

It was concluded that the com-munication system was providingthe performance required for theapplication and the available, eventhough limited bandwidth was suf-ficient for the devices.

A typical GOOSE message is300 bytes, and only requires band-width when trip messages are sent

(plus during the periodic broadcastevery 60 seconds), and the burdenon the network is minimal. EachGOOSE message occupies only

3.125 ms (300 x 8 bit / 768 kbps).However, the shared bandwidthwith SCADA and phones did affectreliability. With the alarm timerset at 90 seconds, alarms for non-received GOOSE messages wouldoccur several times per day.

Note that there is no re-trans-mit of the routine GOOSE messageso if it is dropped due to networkcongestion, an alarm will result ifno message is received within theset time.

This problem was resolved byincreasing the alarm timer to 300seconds.

However, reliability of the net-work was still considered sufficientfor the application as a trip GOOSE

5 Wind arm

mplementa-

tion of transfer

trip is essential

to avoid

slanding of

the wind farm

message would be re-transmittefor a total of 3 times (up to 16 timeare possible).

The lesson learnedfrom thproject is that:

When communicating ovan Ethernet network, native Ethernet devices are much more eficient than legacy devices witconverters.

The tests showedthat 99.7 %

of the trips werebased on the firsGOOSE

message

G

OOSEApplications

8/2/2019 Lessons Learned Autumn08

6/6

PAC.AUTUMN.2008

33

Microprocessor based protection does not use IEC 61850 inthis application. It was only the teleprotection devices that

used this protocol for communication overthe Ethernet network.

Meter is

installed

monitor

perform

based on

1953-20

Recomm

practice

measure

and limit

voltage

on AC P

Systems

6Protection and control panels

7 Wind arm breaker and metering tankNote: 138 kV Substation connected to the wind power generation facilities