IP/Ethernet in the TSO - Phoenix Forums Europe 2013 - Rui Franco... · 2013-07-03 · IP/Ethernet...

IP/Ethernet in the TSO Utilinet Europe 2013, Brussels, 3rd July 2013

Rui Franco, RENTELECOM, REN Group

1

Curriculum Vitae Rui Franco

Currently: RENTELECOM Director, REN Group

Rui Franco (1967) is graduated in Electrical Engineering and Computer Science by

the Technical University of Lisbon. Since May 2011 he is the Director of

RENTELECOM the telecommunications provider of REN, the Portuguese energy

transport group (electricity and gas). Previously he was the partner in charge for the launch of Nae,

an ICT consultancy, in the Portuguese market.

His previous job assignments included management positions during 7 years at Marconi (former

Portuguese international telecommunications carrier) and 10 years as head of Wholesale Business

Unit at OniCommunications the Portuguese utility based fixed line alternative telecommunications

operator. Later he was the head of T-systems Portugal (Deutsche Telekom group) in charge for

launch in the Portuguese market.

2

Agenda

1. REN Group and RENTELECOM

2. Current situation

3. Future trends and Evolution

4. Conclusions


REN Group

REN - Redes Energéticas Nacionais operates in

two major business areas:

Electricity - The transmission in very high

voltage electricity and overall technical

management of the National Electricity System;

Gas - The transport of high-pressure natural

gas and overall technical management of the

National Natural Gas System, guaranteeing the

reception, storage and regasification of LNG and

underground storage of natural gas.

4


RENTELECOM started in 2002, with providing internal services and with leveraging the existing

surplus of network capacity in the electricty and gas backbones, but is nowadays a true alternative

in the wholesale and corporate market, with the following business areas:

RENTELECOM, leveraging in the telecommunications market

Dark Fiber

• National

• International

Colocation

• Towers

• Technical areas

Housing

• Racks

• Rooms

Leased Lines

• SDH / DWDM

• Ethernet / IP-MPLS

ICT Services

• Special Projects

• Voice VPNs,…

5

Agenda




4. Conclusions


Technical Evolution

Telecom technologies at REN, a short history:

Before 1980 – only Power line communication was used to transmit information 1980’s – Setup of analog and digital Radio links 1990 – Setup of Optical Fibers (Radio links still used as main and redundant routes) 1992 – PDH on Optical fiber and radio links 2003 – SDH setup on Optical fiber 2006 – DWDM setup on Optical Fibers 2006 – IP/MPLS & Ethernet setup

7


Drivers for reaching current situation

Along the period of 2000-2010, REN made intensive

investments in its network and more than doubled its

electrical grid (gas grid grew slower):

Demand side – clients:

Dispatch and Automation services (SCADA, Command and Control, …), required scalable, flexible and reliable telecoms network to cope with grid growth. Telecoms are mission critical and outsourcing was not an option. RENTELECOM – A business opportunity to leverage on technical know how and physical assets.

Offer side – telecoms network: The intensive building of new lines allowed installation of OPGW cables at incremental cost. Nowadays REN manages a backbone network that is +90% covered with our own OF. OPGW cables are extremely reliable. Technical know how is an asset.

A perfect match!

8


Services carried by the telecoms network

Security network services:

RTU/SCADA ICCP (Inter Control Center Protocol) – connection to Spanish TSO REE and to local DSO EDP. Dispatch centers interconnection – REN manages 2 dispatch centers (primary / secondary) Telemetering Power quality monitoring Remote management and supervision: Protection, control and automation, low voltage power supply. Video surveillance and physical intrusion Voice in industrial facilities NTP – Network time protocol - time tags Wide Area Monitoring / Phasor Measurement Units Line differential protection Teleprotection

Other services:

Datacenters interconnection and data storage replication Corporate & LAN interconnections Telecommunications networks (RENTELECOM)

Almost all services use the IP/MPLS network except the Line differential

protection and the Teleprotection services

REN already has a very “smart” and strongly automated grid with

IP/MPLS as a significant enabler

9


Current telecommunications network architecture

DWDM SDH

IP / MPLS

SDH

SCADA Teleprotection Line Differential Protection TDM voice Command and Control Long distance high bitrate L 2

services Alarms Corporate network RENTELECOM services ...

SCADA Command and control VOIP TDM voice Telemetering Corporate network RENTELECOM services ...

SCADA interconnection

Corporate network

SCADA TDM Voice

PDH

PDH

PLC

Analog Voice Protection services

REN Gas

REN Group

REN Electricity

REN must manage IP/MPLS and legacy technologies back to PDH and even PLC

with significant O&M costs

Electric Grid protocols:

RTU/SCADA: IEC 60870-5-104 widely used with 101 residual and to be abandoned

Differential protection: G.703 64kbps (to be abandoned) E1 2Mbps and IEEE C37.94

Remote Mngmt & Spv: FTP, SSH, Telnet & others (proprietary)

10


Current telecommunications network dimension

Optical Fiber cables (mostly OPGW): 7.000km 1.100km radio links supported by 20 telecom towers still used for redundancy 2 datacenters managed by REN (Lisbon and Oporto) – also used by IT 285 Points of presence (at electric substations, gas stations and DCs):

PDH: 360 nodes / 1 supplier SDH: 390 nodes / 2 suppliers DWDM: 22 nodes IP/MPLS: 750 routing & switching nodes / 2 suppliers (not including Station Bus which is not managed by the telecommunications team) Protection Signalling Equipment: 450 nodes / 2 suppliers TDM Voice: 124 PABXs / 4 suppliers VOIP: 3 communication servers and multiple gateways / 2 suppliers

11


Current IP/MPLS network architecture (electric grid only)

The IP/MPLS network was designed to ensure full redundancy and optimized topology according to the traffic pattern (Control centers <=> Substations):

Substation

LAN

Substation

LANSubstation

LAN

Substation

LAN

Substation

LAN

IP/MPLS Core

IP/MPLS Edge

Ethernet Access

REN

Ele

ctr

ic

REN

Gro

up

Consists of 7 core nodes (5+2 main ones) all connected at 1Gbps (oversized trunks, hub & spoke)

Each substation is connected to 2 of the 7 core nodes

At each substation there are aggregators and access nodes (Ethernet) for each service (100Mbps)

The current architecture ensures: Minimum delay (best paths & minimum hops) Minimum jitter (oversized trunks)

The current architecture does not ensure: Guaranteed symmetric paths (hard to implement and not required by current client services)

12


IP/MPLS adoption by security network services (1/2)

Migration was smooth and nowadays, most of the security network services use IP/MPLS

with the exception of protection services (Teleprotection and Line Differential

Protection)

SCADA services were early adopters of IP due to the fact that TDM based interfaces could

no longer accommodate grid growth (new substations)

So adoption was fast but massive usage was slower due to IEDs* lifecycle - Other services

adoption of IP followed along 4 / 5 years and is still ongoing (e.g., remote access and

supervision 55%, video surveillance 78%, …)

* - Intelligent Electronic Devices 13


IP/MPLS adoption by security network services

Protection equipment is not IP native - so far, protection Services are based on equipment

which requirements are specified specifically for the usage of TDM networks

Protection equipment has requirements concerning latency, jitter, symmetry that are hard

to meet by IP/MPLS services

Protection services are supported by TDM / SDH – but even the usage of SDH requires a

fine tuning of the telecoms network (e.g., Line Differential Protection with redundant

paths but no self healing)

14


Current telecoms network management (electricity & gas)

REN has chosen to manage such diversified set of technologies and assets with its own

dedicated telecoms team

Telecoms team has a total of 20 professionals with Engineering (25%) and O&M (75%) skills

Most of the field services are increasingly outasked

The team manages the following O&M support facilities (24x7):

A first line Helpdesk for first interaction with internal and RENTELECOM customers

A Supervision Center with monitoring and remote control

capabilities for most of the telecoms network

Telecommunication costs (capex & opex) are recognized by the

energy regulator as part of system management costs

REN Supervision Center

15


Achievments (1/2)

Internal clients are enthusiastic about IP adoption

IP adoption allowed overall improvements in business processes:

REN has a Remote Access Center dedicated to SAS* and auxiliary power

supply systems, which allows the remote monitoring and control of most

recent technologies and equipment:

Activities like commissioning can be largely followed remotely

Real time monitoring of key SAS* equipment already in place

IP usage allowed new applications and a more efficient fulfillment of REN

obligations:

Support to onsite maintenance teams

Support to power system fault analysis (better data retrieval)

Implementing new settings on IEDs and corresponding backups

All electric substations and gas stations are not manned * - Substation Automation System

REN Remote Access Center

REN Dispatch at Sacavém

16


Achievments (2/2)

In 2012 REN Electric achieved, for the first time an

Average Interruption Time of zero seconds (16,2

seconds in 2011)

Telecommunications know how associated to utilities is

already leveraged for commercial telecoms

(RENTELECOM) and will be further leveraged in the

scope of the group activities diversification (coupled to

dispatch and automation & control services):

An example are the services currently provided by

RENTELECOM to EDP in accessing their substations

using our IP/MPLS core network

17


Issues (1/2)

Operating legacy technologies along with IP/MPLS is too costly in terms of O&M Legacy telecommunication technologies are at commercial end of life so its replacement must be faced as well as impact to telecom network customers Commercial off the shelf telecom technologies have shorter life cycles than utility equipment (e.g. SCADA/RTU IEDs):

Particularly true for younger technologies such as IP/MPLS Depreciation allowed by regulator is longer than commercial life cycle Utility security network equipment (IEDs) is still a “step behind” in comparison to commercial telecom technologies – both will have to converge

18


Issues (2/2)

Operating IP/MPLS is much more complex and demanding than operating TDM technologies. This requires workforce adaptation and criterious outasking both for telecommunications and Automation Teams Migration to IPv6 – not an issue since in a TSO like REN the number of sites and nodes fit in the IPv4 addressing space (few 1000’s) Security issues – has been a concern and has to be more and more assessed Due to the foreseen future need of real time inter-substation services, the current IP architecture will have to be assessed.

19

Agenda




4. Conclusions


Smart grid for a TSO (1/4)

Smart grid – a very broad definition… A smart grid is a digitally enabled electrical grid that gathers, distributes, and acts

on information about the behavior of all participants (suppliers and consumers) in

order to improve the efficiency, importance, reliability, economics, and

sustainability of electricity services (source: wikipedia)

TSOs are required to ensure the highest uptime in their backbones.

TSOs grids are already very “smart” (automated) but there are trends that will impose an even

“smarter” grid:

Renewable Energy Sources (RES) and Distributed Energy Resources (DER)

New businesses and Ancillary services

European SuperGrid

21

In the near future TSOs will have to ensure a wider monitoring and control scope as

energy sources are more dispersed and intermittent, energy flows in the grid change,

new businesses come into place and interconections will have a more relevant role



Renewable Energy Sources (RES) and Distributed Energy Resources (DER)

Climate Change concerns are imposing reduction of GHG (Green House Gas) emissions implying decabornisation and more RES which means:

Increasing intermittent sources (wind, sun) and distributed micro generation changing energy flows in the network (consumer becomes “prosumer”)

Significant increase in RES will create issues with power quality, voltage stability, harmonics, protection and control schemes and may affect grid functions such as frequency control. Currently RES are managed by TSOs in a limited manner (VHV and HV). As they increase and move into MV and LV, they will have to be manageable by TSOs – this implies more monitoring and control capabilities extending into DSOs networks and more information to analyze – a smarter grid!

European Union 2020 growth

strategy concerning climate / energy,

includes:

Reduce GHG emissions by at least

20% Energy from renewables at least

20% Energy efficiency to increase 20%

22



New businesses and Ancillary services

As RES and DER become significant, new services will have to become available for TSOs:

New businesses – it is foreseen that DER Aggregators will be a key player to be used by TSOs to manage the grid New businesses – it is foreseen that Demand Aggregators will aggregate local demand from consumers providing this service to TSOs as a one entity Balancing market – Already used by TSOs at national level and limited to classical production - will have to be expanded at European level and include also RES and DER Storage – besides hydro, it is still a subject of intensive R&D but may become a new service in the medium term

REN Dispatch at Vermoim

23

In the scope of the SmartGrid, new interfaces between the DSOs, TSOs,

Aggregators, Consumers and Generators will have to be developed



European SuperGrid

Geographic specialization in different production technologies calls for stronger and smarter Transmission Infrastructures

EU Energy roadmap 2050 foresees a European Super Smart Grid, integrating all European and neighbor grids (Desertec, Medgrid and Mediterranean Solar Plan initiatives) The European SuperGrid will be an overlay HVDC grid, centrally managed across different borders and TSOs, thus requiring “Smarter grid” technologies (Med-TSO)

Super Grid connecting Mediterranean ring

24


Drivers for telecommunication network evolution

Policy framework evolution will put more pressure on TSOs and push SmartGrid:

To manage more complex grids with more interconnections, more players and diverse flows of

energy. This means more points to control and more information to collect

To ensure quality and be more efficient – the regulatory framework will likely continue evolving

favoring quality & efficiency:

Ensuring quality in more complex grids, requires better knowledge of the grid which implies

massive data analysis.

More efficiency can be achieved by direct cost reduction (less technologies) and better O&M

End of life of legacy technologies (e.g.,SDH) will force evolution and full migration to IP/MPLS

with no SDH underlying is most likely

Technical standards are aligned with above trends and already provide support for both

suppliers of SAS equipment and telecoms equipment (e.g., IEC 61850)

Massive data collection and analysis will only be feasible using a full IP integrated network. 25


Intelligent substation – Incident analysis, an example of IP usage at REN

The project ongoing consists in the implementation of an Automatic Data Acquisition & Processing

System for data related to power system faults

Data acquired will be made available in near real time to key users such as:

System operator and fault analysis personnel

Protection and Control engineering teams

Substation apparatus maintenance teams

Line maintenance teams

Results to be obtained include relevant information for the operation and maintenance of the

bulk power system such as:

Fault location applied to power lines

Identification of fault type i.e. phase, magnitude, duration, cause, etc…

Information about the reaction of the SAS to the faults in the grid (relevant for the SAS

maintenance policy)

Automatic analysis of the performance of the protection system.

26


Near future evolution of telecoms the network architecture

SDH

IP / MPLS

SDH

Teleprotection

Differential Protection

TDM Voice

Long distance , high

bitrate L 2 services

Rentelecom

...

SCADA

Corporate network

VOIP

TDM Voice

Command & Control

Telemetering

Rentelecom

...

SCADA

Alarms

VoIP

Management &

remote monitoring

Corporate network

...

DWDM

REN Gas

REN Group

REN Electric

In the very near future all Gas security services will migrate to IP/MPLS (1 / 2 years)

In the very near future PDH will be discontinued (1 / 2 years)

SDH / DWM will be in place for a long time (4 / 6 years more)

27


Near future IP/MPLS network architecture (electric and gas)

IP/MPLS Core

IP/MPLS Edge

Ethernet Access

Substation

LAN

Substation

LANSubstation

LAN

Substation

LAN

Substation

LAN

Substation

LAN

Substation

LAN

Substation

LAN

Substation

LAN

Substation

LANSubstation

LAN

Substation

LAN

Substation

LAN

REN

Gas

REN

Ele

ctr

ic

REN

Gro

up

2 more main nodes will be added to the existing 7, all connected at 1Gbps; Each gas station will be connected on a regional ring;

The IP/MPLS network will keep the same architecture for the electric grid. Gas grid will be included

in the same network but will not have the same demanding requirements

28

Agenda




4. Conclusions

4. Conclusions Conclusions

REN has been an early adopter of IP/MPLS

REN has successfully conducted a smooth migration of security network services to IP/MPLS (still

ongoing)

REN expects to continue migrating services to IP/MPLS in the near future (full migration of gas

services in the short term)

Policy, climate change, regulatory and business trends will impose pressure on TSOs for running

smarter grids. IP/MPLs is a fundamental enabler to correspond to such demands

REN understands that a full IP network is the long term future, but in the short to medium term,

some services like Protection services will still require TDM / SDH technology (trials ongoing)

Despite standardization efforts, convergence to IP of telecoms network and every device in the

electric substation is still required

30

RENTELECOM Thank you

[email protected]

31

IP/Ethernet in the TSO - Phoenix Forums Europe 2013 - Rui Franco... · 2013-07-03 · IP/Ethernet...

Documents

Transcript of IP/Ethernet in the TSO - Phoenix Forums Europe 2013 - Rui Franco... · 2013-07-03 · IP/Ethernet...