Packet-Optical The Infinera Way eBook · be carried in digital OTN circuits, which in turn are...

I N F I N E R A 1

PACKET-OPTICAL THE INFINERA WAY

I N F I N E R A 3I N F I N E R A2 Footnote citations on page 111

OPTICAL FIBER PROVIDES almost lossless transmission of signals at an ultra-wide range of frequencies. Packet switching, implemented using the Ethernet family of protocols and interfaces, offers one of the most efficient ways to sort and direct streams of digital data. Packet-optical networking combines these two outstanding technologies, positioning them to dominate the next generation of transport networks.

Packet-Optical the Infinera Way was written to help Infinera’s customers, prospects and partners, and anyone else who needs to have a better understanding of the packet-optical world. This book focuses on the integration of higher Open Systems Interconnection (OSI) layer functionality into optical systems to create packet-optical transport systems (P-OTS), i.e. packet-optical networks. The term P-OTS is widely used within the industry and has been recycled from an earlier definition (plain old

telephone service) to cover a wide range of solutions and networks with varying degrees of capabilities and functionality.

Packet-optical integration has some great advantages in terms of cost and service differentiation. Infinera´s technologies take this one step further, with benefits including reduced equipment, lower operational costs and key capabilities such as low latency and excellent synchronization, outlined in Chapter 2. Chapter 3 describes how packet-optical networks are best managed and how to take advantage of current and future software-defined networking (SDN) developments. Chapter 4 takes the reader further into how the described technologies are leveraged by various applications, such as business Ethernet, mobile fronthaul/backhaul and cable TV backhaul.

For those looking for a better understanding of Layer 2 Ethernet technologies, Chapter 5 includes a more detailed description of how these technologies work and how they are leveraged in wide area networks.

We hope you find Packet-Optical the Infinera Way informative and useful, whether you use it to research a particular subject or read the complete volume from beginning to end.

The descriptions are kept independent of product releases as much as possible. Current details of the Infinera product portfolio are available at www.infinera.com.

Features of Infinera’s packet-optical solutions that we believe to be unique are highlighted with this marker throughout the text.

The information included is subject to change without further notice. All statements, information and recommendations are believed to be accurate but are presented without warranty of any kind.

P R E F A C E

https://www.infinera.com

I N F I N E R A 5I N F I N E R A4

1 INTRODUCTION ........................... 6

1.1 The Cloud-driven Transformation of the Network ..................................8

1.2 Packet-Optical Networking ..............9

1.3 The Infinera Intelligent Transport Network Portfolio ...........10

2 PACKET-OPTICAL NETWORKING ............................12

2.1 Chapter Summary .........................14

2.2 The Principles of Packet-Optical Integration .......................................14

2.2.1 Why Aggregate Traffic at Layer 2? ............................................14

2.2.2 Ethernet Transport at Layer 2 Versus at Layer 1 ..............................15

2.2.3 Native Ethernet and OTN Framing ............................................19

2.2.4 Packet-Optical Transport with the XTM Series ........................20

2.2.5 Packet-Optical Transport with the PXM ...................................22

2.2.6 Leveraging the XTM Series and PXM in the Same Network ......24

2.3 MPLS-TP for Traffic Engineering and Service Scalability ....................26

2.4 Traffic Management ........................30

2.4.1 Traffic Shaping and Policing ...........30

2.4.2 Implementing Traffic Shaping ........31

2.5 Migrating Legacy TDM Services to Ethernet .......................................32

2.5.1 One Common Infrastructure for Ethernet and Legacy TDM Services ...................................32

2.5.2 Using the iSFP to Convert TDM Services for Ethernet Transport ......33

2.6 The Main Elements of an Infinera Packet-Optical Network .................34

2.6.1 Ethernet Demarcation Units (EDU) and Network Interface Devices (NID) ...................................35

2.6.2 Packet-Optical Transport Switches (EMXP) and the PT-Fabric ...............35

2.6.3 Optical Add-Drop Multiplexers (OADM, ROADM) and Other Optical Elements .............................36

2.6.4 The DTN-X Packet-Switching Module (PXM) ..................................36

2.6.5 The Multi-layer Service Management System ......................37

2.7 Advantages of Packet-Optical Transport ..........................................38

2.7.1 Benefits of the Packet-Optical Approach .........................................38

2.7.2 Advantages when Using Infinera’s Portfolio for Packet-Optical Transport ..........................................40

3 OPERATING THE NETWORK ..... 42

3.1 Chapter Summary .........................44

3.2 Multi-layer Network Management ...................................44

3.2.1 Management Framework ...............45

3.2.2 Infinera’s Management Software Suite .................................47

3.2.3 Infinera Digital Network Administrator ...................................48

3.2.4 A Unified Information Model for Multi-layer Management ................49

3.2.5 Layer 2 Service Fulfillment ..............50

3.2.6 Layer 2 Service Assurance ..............50

3.2.7 Network Planning and Optimization ....................................51

3.3 Software-defined Networking and Network Virtualization .............51

3.3.1 Basic Concepts ................................51

3.3.2 Infinera’s Multi-layer SDN Vision ....56

3.3.3 Applications of Intelligent Packet Transport with SDN .........................57

4 APPLICATIONS OF PACKET-OPTICAL NETWORKING ............ 60

4.1 Chapter Summary .........................62

4.2 Ethernet Services for Enterprises – Business Ethernet ............................62

4.2.1 Serving Enterprise Customers .......62

4.2.2 A Metro Network for Business Ethernet ...........................................63

4.2.3 A Long-haul Network for Business Ethernet ............................64

4.3 Aggregation of IP Traffic – IP Backhaul ...........................................64

4.3.1 IP-based Services over a Common Infrastructure...................64

4.3.2 A Lean and Transport-centric Aggregation Network .....................65

4.3.3 The Flexible Optical Network Brings Scalability .............................67

4.4 Mobile Backhaul ..............................67

4.4.1 4G/LTE and 5G Place New Requirements on Mobile Backhaul ...........................................67

4.4.2 A Backhaul Network Optimized for 4G/LTE and 5G ..........................68

4.5 Switched Video Transport ..............69

4.5.1 Streaming 3D and HD Video to the Home .........................................69

4.5.2 Infinera’s Solution for Switched Video Transport ...............................70

5 ETHERNET AND LAYER 2 TECHNOLOGIES ..........................72

5.1 Chapter Summary .........................74

5.2 Ethernet Basics ................................74

5.2.1 Ethernet Mode of Operation .........74

5.2.2 Virtual LANs .....................................76

5.2.3 Ethernet Physical Media (PHY) .......78

5.3 Synchronization and Circuit Emulation Services over Ethernet ...........................................80

5.3.1 Synchronous and Asynchronous Transport ..........................................80

5.3.2 Synchronization Standards .............83

5.4 Ethernet Protection .........................85

5.4.1 Link Aggregation.............................865.4.2 Ethernet Ring Protection

Switching ..........................................87

5.5 Carrier Ethernet Architecture and Services .....................................88

5.5.1 Carrier Ethernet: Ethernet as a Transport Service .............................88

5.5.2 The Carrier Ethernet Architecture and Terminology ........91

5.5.3 Carrier Ethernet 2.0 Services ..........92

5.5.4 Carrier Ethernet Service Attributes .........................................94

5.6 Carrier Ethernet Traffic Management ...................................95

5.6.1 Bandwidth Profiles ..........................95

5.6.2 Class of Service and Service Level Agreements ..............96

5.7 Carrier Ethernet Operations, Administration and Maintenance (Ethernet OAM) ...............................97

5.7.1 The Management Framework ........97

5.7.2 Standards for Ethernet OAM .........98

5.7.3 The Service Lifecycle .......................98

5.7.4 Ethernet Service OAM – Performance and Fault Management .................................100

5.8 Metro Ethernet Forum “Third Network” Vision ................102

INDEX........................................104

C O N T E N T S


1 I N T R O D U C T I O N


1.1 THE CLOUD-DRIVEN TRANSFORMATION OF THE NETWORK

Over the past decade, IT infrastructure has seen a dramatic shift from local resources to those located in third-party facilities. Whether we receive the services we use, including storage of our personal files, streaming of personal and commercial video and, of course, photo albums and email, at home or on our mobile phones, they all reside somewhere “on the network,” or more fashionably, “in the cloud.” The acceleration of cloud services, such as

infrastructure, platform and software as a service (IaaS, PaaS and SaaS) offerings, is having a profound impact on both IT and network architectures among technology service providers.

As a result, the traditional layered network model of a hierarchical, functionally heavy data network comprising several layers of devices is evolving to a simplified model of Layer C, cloud services, and Layer T, intelligent transport (Figure 1).

The adoption of cloud services, consisting of the upper OSI layers of a network, is growing at a phenomenal rate. The

compute, storage and network functions once provided from a single data center are rapidly being expanded to multiple centers at geographically separate locations. As a result, operators need to scale, simplify and increase flexibility in their transport networks. Network functions virtualization (NFV) enables operators to migrate services like security, voice and broadband remote access server (BRAS) from dedicated devices to software services on standard x86 hardware within cloud data centers. NFV allows service providers to drive down the cost of network services, while delivering them in a more granular and responsive manner.

Meanwhile, transport functions at the lower layers are converging as operators encourage vendors to develop the most cost-effective network elements, giving rise to an intelligent transport layer, Layer T. Layer T combines Multiprotocol Label Switching (MPLS) and Ethernet packet services with the digital functionality of Optical Transport Network (OTN) switching, the capacity scaling of wavelength-division multiplexing (WDM) and the optical path flexibility of reconfigurable optical add-drop multiplexer (ROADM) functionality. The control plane of Layer T can use

established protocols, such as Generalized Multiprotocol Label Switching (GMPLS), but a common vision is to employ SDN technology that can abstract the functions of all Layer T technologies and present standardized application programming interfaces (APIs) such that control can be driven by Layer C. An integrated packet-optical approach to Layer T is manifested in Infinera’s Intelligent Transport Network architecture, which is a mix of packet, digital and optical functions that virtualize control at Layer C via open APIs, using both GMPLS and SDN.

1.2 PACKET-OPTICAL NETWORKING

Layer T traffic flows vary in many different ways, such as duration, size, time of day, burstiness and deterministic nature. As a consequence, flows can be handled by several technologies across the multi-layered Layer T network, including optical wavelengths, digital OTN circuits, MPLS

label-switched paths (LSPs) and Ethernet virtual connections (Figure 2). To optimize the cost-to-performance ratio of Layer T, it is of the utmost importance to match the transport and switching technology to the traffic flow characteristics, and ultimately to the end-user services. Also, these technologies are not necessarily exclusive. For example, Ethernet/MPLS packets may be carried in digital OTN circuits, which in turn are carried by coherent optical super-channels as Layer T services are aggregated. Meanwhile, switching capabilities are essential at each of the layers in order to maintain the efficient use of capacity and provide alternative routing for traffic engineering or protection events.

The choice of technologies to employ within Layer T also has a geographical aspect, splitting the industry into distinct sub-segments: metro/edge packet-optical systems and long-haul/core packet-optical systems.

Metro/edge packet-optical systems are aimed at applications toward the edge of an optical network. These systems typically take the form of WDM-based optical networking platforms with integrated Ethernet switching. They have varying degrees of support for Layer 1 technologies such as Synchronous Digital Hierarchy (SDH)/Synchronous Optical Networking (SONET)

and OTN, but must also offer excellent support for Layer 2 Ethernet functionality. This functionality includes the advanced provisioning, management, verification and protection of Layer 2 Ethernet services using a defined group of standardized protocols and procedures.

Systems at the core of a network are dealing with traffic from many applications, and as such require less service awareness. However, they do require the capacity to handle higher-capacity links, as well as a

Old Model New Model

SDN

Layer T: Intelligent Transport

Layer C: Cloud Services

Firewall

SBCB-RAS

MPLS PE

L2/3 Packet

Layer 1 OTNLayer 0 WDM

Scale Virtualization

L3–L7

L0–L3

FIGURE 1: A Simplified Network Model Comprising Cloud Services (Layer C) and Intelligent Transport (Layer T)

Client TrafficTransport

Technologies

5$

VoiceVideo

Email

DC Interconnect

Enterprise App

Match

Multiplex

Switch

Packet

Digital

Optical

FIGURE 2: Matching Network Technologies to Traffic Flows

I N T R O D U C T I O N


higher degree of OTN Layer 1 transport capabilities versus Layer 2 Ethernet. This geographical balance between Layer 1 and Layer 2 functionality is shown in Figure 3.

Long-haul/core packet-optical systems must therefore support not only high capacity but

also switching across the whole chassis or node, from any port to any other port, rather than just within ports on a single plug-in unit as is common in metro/edge platforms. Long-haul/core platforms typically are capable of packet aggregation, OTN switching, terabit-scale WDM transport and advanced ROADM functionality, all with a single control plane.

10G + Metro 100GColorless, Directionless

Gridless ROADM OverlayMixed Fiber Environment

100G + Multicarrier CoherentColorless, Directionless, Contentionless

Gridless ROADM DCM Free Fiber

METRO REGIONAL/CORE

PBB-TE CE 2.0 MPLS-TP

MPLS-TP MPLS

OTN SWITCHING (ODU-2)

OTN TRANSPORT, ODU-FLEX SWITCHING

Laye

r 0

Laye

r 1

Laye

r 2

Integration of OTN + Layer 2 from edge to core without technical or business compromise is the ultimate goal

FIGURE 3. Matching Network Technologies to Network Topology. Source: IHS Markit

1.3 THE INFINERA INTELLIGENT TRANSPORT NETWORK PORTFOLIO

Infinera’s end-to-end portfolio for packet-optical networks consists of the platforms shown in Figure 4.

In the metro/edge space, Infinera is a pioneer of metro WDM through the XTM Series, which today includes leading packet-optical metro access, aggregation and core platforms. This platform delivers services designed for applications such as triple play broadband, cable broadband, business Ethernet and mobile transport.

In the long-haul/core market, the DTN-X Family provides the world’s only commercial multi-terabit super-channel system based on large-scale photonic integrated circuit (PIC) technology. It includes the Packet Switching Module (PXM), which enables traffic over virtual local area networks (VLANs) or MPLS LSPs with assured quality of service (QoS). In long-haul packet-optical networks, the DTN-X family is a powerful companion to the XTM Series.

Triple Play

Cable Broadband

Business Ethernet

Data Center

Mobile Front/Backhaul

TM-102/IITM-301/II

TM-3000/II XTC-2XTC-2E

XTC-4

XTC-10

XT-500 XT/S-3300

EDUNID

CX CX2

METRO LONG-HAUL / SUBSEAACCESS AGGREGATION REGIONAL CORE

XTM Series

Cloud Xpress Family

DTN-X Family: XTC Series

XTG Series

DTN-X Family: XT Series

DC INTERCONNECT

INFINERA XCEED SOFTWARE SUITE AND DNA

INFINERA OPEN FLEXIBLE GRID LINE SYSTEM

XT/S-3600

FIGURE 4. Infinera’s Intelligent Transport Network Portfolio

I N T R O D U C T I O N


2 P A C K E T - O P T I C A L

N E T W O R K I N G


2.1 CHAPTER SUMMARYOptical fiber provides almost lossless transmission of signals at an ultra-wide range of frequencies. Packet switching, implemented according to the Ethernet family of protocols and interfaces, offers one of the most efficient ways ever to sort and direct streams of digital data. Packet-optical networking fundamentally leverages the outstanding characteristics of these two technologies to implement a new generation of telecommunication networks.

The scalability and cost-effectiveness of Ethernet has made it the unifying service protocol for modern local and wide area networking. Increasingly, consolidation of the optical and Ethernet/Internet Protocol (IP) transport infrastructures within the same network elements has become the means to drive down both network investment costs and the associated operational costs. The additional support for OTN1 and label-switching mechanisms (such as Multiprotocol Label Switching - Transport Profile, or MPLS-TP2) provides

an extra toolkit to complement Ethernet and enhance the transport capabilities and scalability of the network. Supervised by a multi-layer management system that integrates the handling of OSI3 Layer 1 optical channels and Layer 2 Ethernet services, a flexible, cost-efficient and future-proof telecommunications infrastructure is here and ready to be deployed.

This chapter deals with the principles of transporting Ethernet traffic over WDM networks and describes how these two key technologies can be integrated in a unifying architecture. It also highlights the advantages that arise from combining the two technologies, including how traffic is transported with minimal delay and without loss of synchronization.

2.2 THE PRINCIPLES OF PACKET-OPTICAL INTEGRATION

2.2.1 Why Aggregate Traffic at Layer 2?The introduction of aggregation at Layer 2, i.e. the Ethernet layer, brings several benefits to network operators. Traditionally, metro aggregation networks were implemented using dedicated WDM equipment that provided wavelengths connecting, for example, BRAS, radio

base stations and enterprise networks to more central locations. The traffic from the connected equipment was transported transparently to an IP core network at OSI Layer 1, via wavelengths on fibers in ring or star topologies. As the number of endpoints grew, the central IP core network had to be extended, requiring more IP routers at more sites and with more ports, as indicated on the left side of Figure 5.

In this type of network, there are several reasons to introduce Layer 2 aggregation:

• The IP core network can be reduced in size to just a few central routers instead of being spread out throughout a full metro network area. Layer 2 aggregation equipment is generally less costly,4 consumes less power, has lower latency and requires less expertise to configure than IP routers. Thus, centralization of the IP core network reduces the necessary investment and operational costs.

• Layer 2 aggregation can perform statistical multiplexing of data traffic, and the WDM channels of the underlying optical network can be used much more efficiently than if only Layer 1 aggregation is used. Statistical multiplexing allows the bandwidth to be divided arbitrarily among a variable number of users, in contrast to

Layer 1 aggregation (time or frequency multiplexing), where the number of users and their data rates are fixed. Statistical multiplexing makes use of the fact that the information rate from each source varies over time and that the optical path’s bandwidth only needs to be consumed when there is real information to send.

• Statistical multiplexing in a Layer 2 network also allows for the creation of new types of transport services, for example services with more granular data rates and with committed information rates that may be temporarily exceeded. Hence the operator may offer its customer a much more differentiated service portfolio than with only traditional Layer 1 transport services, as shown in Figure 6.

• Since data traffic is concentrated at Layer 2 in the aggregation network, it can be handed over to IP core routers via a few high-speed interfaces rather than over many lower-speed interfaces. This simplifies administration and contributes to a lower cost per handled bit.

• As an additional benefit, the aggregation network itself can be used to offer services within the metro/regional area. For example, point-to-point Ethernet connections can be provided between offices in a city center without loading any central router nodes. Such direct

connectivity provides more rational traffic handling and reduces forwarding delay compared to using IP routers.

2.2.2 Ethernet Transport at Layer 2 Versus at Layer 1Given the benefits of a Layer 2 aggregation network, it is important to understand how this type of network differs from a traditional network performing Layer 1 aggregation of Ethernet traffic over WDM.

P A C K E T - O P T I C A L N E T W O R K I N G

IP Core Network IP Core Network

Layer 1 Aggregation(Star, Ring)

Layer 2 Aggregation

Switch

Subscriber Aggregation Core

Switch

1GbE

1GbE

1GbE

1GbE

1GbE

10GbE

33% 100% 100%

L2

L2

FIGURE 5: Migrating from Layer 1 to Layer 2 Aggregation of Traffic

FIGURE 6: The Statistical Multiplexing of Ethernet Is Leveraged to “Fill the Pipes,” an Especially Useful Feature at the Edge of the Network Where Traffic Is Often More Variable


Transporting Ethernet traffic between two remote sites with WDM as the underlying bearer technology can be done in two fundamentally different ways:

• Transporting the Ethernet frames “as they are” (transparently) over a WDM channel, i.e. Layer 1 (optical) transport.

• Using an intermediate Carrier Ethernet network5 that in turn has its frames transported over one or more WDM channels, i.e. using Layer 2 (Carrier Ethernet) transport. This is the technology used in a Layer 2 aggregation network.

Both alternatives have advantages and disadvantages.

A basic Layer 1 Ethernet transport solution takes every incoming frame from the

sending subscriber Ethernet network and puts it into a digital wrapper adapted for transmission over the WDM channel. At the receiving end, the wrapper is removed and the original frame is handed over to the receiving Ethernet network. In this way, every single frame is forwarded without modification between the two subscriber networks.

The Layer 1 transport solution provides a transparent path between the two subscriber Ethernet networks, giving the highest possible QoS in terms of latency, latency variation and packet loss. A Layer 1 network is also fully deterministic and provides 100% throughput regardless of what services are carried by the Ethernet traffic. However, since a Layer 1 network is totally transparent to Ethernet traffic,

it is also unaware of any Ethernet service information and can only manipulate the traffic at Layer 1 (Figure 7).

In a Layer 2 transport network (Figure 8), subscriber Ethernet networks are interconnected via an intermediate Ethernet switching function called the Carrier Ethernet network. A service VLAN (SVLAN) or an MPLS-TP pseudowire in the Carrier Ethernet network is used to keep traffic from each set of subscriber Ethernet networks fully separated, i.e. the SVLAN/pseudowire establishes connectivity between the subscriber Ethernet networks belonging to the same subscriber. The frames of the Carrier Ethernet SVLAN/pseudowire are transported over channels of the WDM network, just as described previously.

A plain Layer 1 transport solution cannot concentrate the Ethernet traffic being aggregated, which may result in low utilization of WDM wavelengths. For example, a Layer 1 network collecting Gigabit Ethernet signals that is utilized to

a very low extent will still carry them at 1 gigabit per second (Gb/s) as if they were 100% loaded. This may lead to unnecessary investments in Layer 1 equipment to add wavelengths in the WDM network.

In a Layer 2 transport solution, the incoming subscriber Ethernet frames are analyzed and acted upon by the equipment located at the ingress point of the Carrier Ethernet network before being forwarded. It is possible to concentrate the incoming flow of Ethernet frames by statistical multiplexing and to apply shaping and policing to the Ethernet traffic. The Layer 2 solution can be made fully “Ethernet service-aware” and can analyze and act upon the Layer 2 traffic.

Multiple Ethernet signals are multiplexed using time-division multiplexing (TDM) in a Layer 1 solution. The aggregated signal is not a standard Ethernet signal, and consequently, demultiplexing must be done before the original signals are handed over to a switch/router. A Layer 2 solution performs aggregation to a new

native Ethernet signal that can be handed over without any need for demultiplexing. A Layer 2 solution will therefore normally require a lower number of ports on the receiving switch/router, which may have a beneficial impact on the cost of that equipment.

Since a Layer 2 network can use policing to separate the effective data rate offered to a subscriber from the actual line rate available on the access line, a Layer 2 network can also offer a more flexible and granular set of transport services than a Layer 1 network. While a Layer 1 network typically only provides services at standard Ethernet line rates, such as 1 Gb/s or 10 Gb/s, a Layer 2 network may offer much more flexibility,

such as 25 megabits per second (Mb/s), 200 Mb/s or 400 Mb/s transport services over a physical 1 Gb/s port.

Due to statistical multiplexing, a Layer 2 transport network solution is intrinsically less deterministic than a Layer 1 solution. The throughput of a Layer 2 network may suddenly change because a new service has been introduced or because the traffic situation has changed. However, the Layer 2 network can be made to behave in a deterministic way by the use of pre-defined capacity reservations, i.e. by the use of traffic engineering.

The table on the following page summarizes some of the main differences between Layer 1 and Layer 2 wide area Ethernet transport.

FIGURE 7: Layer 1 Ethernet Transport: Ethernet Traffic Is Carried Transparently at Layer 1 over a WDM Wavelength

FIGURE 8: Layer 2 Ethernet Transport: Ethernet Traffic Is Transported via a Service VLAN in a Carrier Ethernet Network Extending Between Operator Sites

Transparent transport of Ethernet frames

SubscriberEthernet 2


Transponder/Muxponder

Transponder/Muxponder

WDM channel (λ)MXP MXP

Carrier Ethernet Network

Service VLAN



Packet-Optical Switch Packet-Optical SwitchWDM channel (λ)

EMXP EMXP



The XTM Series provides the operator with three principal alternatives for transport of Ethernet traffic:

• Plain Layer 1 transport, i.e. transponders and muxponders that provide 100% transparent Ethernet transport at OSI Layer 1.

• Ethernet-aware Layer 1 transport, i.e. muxponders that provide 100% transparent transport but have Layer 2 features, such as providing information on to what extent a

Gigabit Ethernet connection is utilized. This enables the network operator to analyze the wavelength utilization and avoid unnecessary investment in transponders/muxponders to launch additional wavelengths. Another differentiating XTM Series feature is the ability to inject/extract management VLANs on a Gigabit Ethernet client port. This enables direct remote management of Layer 2 devices via the same data communications network (DCN6) solution as for the optical transport equipment.

• Full Layer 2 transport according to the Metro Ethernet Forum’s (MEF) Carrier Ethernet 2.0 (CE 2.0) specifications, i.e. equipment that provides aggregation and concentration of Ethernet and other traffic with a selected set of Layer 2 functions that support the transport task. Such functions include, for example, the Institute of Electrical and Electronics Engineers (IEEE) 802.3ad Ethernet Link Aggregation, traffic shaping, policing and bandwidth profiles with guaranteed bandwidth allocation. The XTM Series’ Layer 2 network elements – the EMXP packet-optical transport switches – are also Layer 1-aware, meaning that they can be connected directly to a WDM link and support features such as forward error correction (FEC) at the optical layer.

All the traffic units mentioned above are of the plug-in type and can be installed side by side with optical units such as ROADMs in the XTM Series platform’s different chassis options. As an example, Layer 2-capable EMXPs can be used at the edge of the network to collect and aggregate Ethernet traffic and hand it over to Layer 1 transponders or muxponders that provide continued transport with cost-efficiency and a high quality of service (Figure 10).

The DTN-X Family with the PXM provides a similar set of choices for the transport of Ethernet traffic:

Layer 1 Ethernet Transport Layer 2 Ethernet Transport

TDM multiplexing: Collects and delivers traffic without changing its format and data rate.

Statistical multiplexing: Collects traffic at one data rate and can deliver input from many sources over one single physical interface at a higher rate.

Fixed data rates: typically 100 Mb/s, 1 Gb/s, 10 Gb/s and 100 Gb/s.

Flexible selection of data rates. Allocation of bandwidth by policing and shaping of traffic.

Deterministic. “What goes in comes out.” Relies on traffic engineering to achieve deterministic behavior.

“Unaware” of Ethernet service information and can only manipulate traffic at Layer 1.

“Ethernet service-aware.” May analyze and act on Layer 2 control information, i.e. can be used to create Ethernet transport services with different characteristics and quality of service.

Lowest delay, jitter and packet loss. Statistical multiplexing implies a risk for delay, jitter and lost packets.

Layer 1 performance management based on bit errors (cyclic redundancy check or CRC).

Layer 2 performance management based on VLANs and Ethernet Virtual Connections, latency, jitter, frame loss, etc.

Embedded management channels via overhead bytes in line signal wrapper.

Embedded management channels via separate management VLAN at Layer 2.

FIGURE 9. Characteristics of Layer 1 and Layer 2 Wide Area Ethernet Transport

• Plain Layer 1 transport, i.e. transponders and muxponders that provide 100% transparent Ethernet transport at OSI Layer 1.

• Layer 2 transport. Using the PXM, a DTN-X network supports Ethernet aggregation, switching and CE 2.0 services over an OTN core, as described in section 2.2.5.

2.2.3 Native Ethernet and OTN FramingThe Carrier Ethernet network of a Layer 2 transport network is built on WDM optical channels, i.e. the frames of the Carrier Ethernet network are transported by the underlying WDM optical system. The Ethernet payload data can be packaged in the transport containers of the optical system in several different ways. Optical transport standards/systems such as SDH, SONET and OTN therefore incorporate various adaptation methods, e.g. Generic Framing Procedure (GFP), to allow for packet data transport. These methods allow the mapping of variable-length, higher-layer client signals, such as a flow of Ethernet frames, to the channels of the WDM network.

OTN is a more recent addition to the standards for public telecommunications networks and is sometimes referred to by its International Telecommunication Union Telecommunication Standardization Sector (ITU-T) name, G.709. The standard is designed as a “digital wrapper technology” to transport both packet-mode traffic, such as IP and Ethernet, and legacy SDH/ SONET traffic over fiber optics with WDM.


EMXP

TP

TP

TP

L2 services Packet-Optical System

Carrier EthernetSwitch

Transponders

Line SystemROADM,

Add-Drop,Amplifier

It supports FEC and management functions for monitoring a connection end-to-end over multiple optical transport segments.

OTN wraps any client signal in overhead information for operations, administration and management. The client signal to be transported is mapped into an optical channel payload unit (OPU). The OPU is then encapsulated into the basic unit of information transport by the protocol, the

FIGURE 10: The Primary Entities Forming a Node in an XTM Series Packet-Optical Network


optical channel data unit (ODU), which is carried within an optical channel transport unit (OTU), defining the line rate of the connection (Figure 11).

OTN includes OTU standards for a fixed number of line rates covering the range from 2.6 Gb/s up to 112 Gb/s. These bitrates have been selected based on the overall requirements of the optical transport system, but are not necessarily multiples of the bitrates of the signals to transport, e.g. Gigabit Ethernet (GbE). The consequence is

that optical channels may be underutilized if an efficient multiplexing scheme is not used at the ODU level. Three ODU multiplexing schemes important in a packet-optical network are ODU0, ODUflex (GFP) and ODU2e. ODU0 units are optimized to carry 1 GbE signals within an ODU1 unit, which in turn is carried by an OTU1 transport unit. ODUflex units can be multiplexed into any of the higher-rate ODUs and carried at the appropriate OTU channel line rate (Figure 12).

OTN systems are often divided into two

categories: transport systems and switches. OTN transport systems follow OTN standards when framing transported signals for a specific route. The transport systems are therefore compliant with the sections of the OTN standards relating to transmission, such as framing, FEC, performance monitoring, etc. Transport systems use transponders or muxponders to terminate the signal. OTN switches take this one stage further and add an OTN switching fabric that is capable of demultiplexing incoming OTN signals and switching the lower-speed signal components to other high-speed interfaces.

2.2.4 Packet-Optical Transport with the XTM SeriesThe XTM Series supports two types of encapsulation of Carrier Ethernet traffic for WDM transport:

• Native Ethernet: The frames of the Carrier Ethernet network are transported as is, i.e. using the same preamble and end-of-frame sequence as on an ordinary LAN7 when forwarded over the WDM wavelength. This is the standard encapsulation recommended by Infinera for metro/aggregation networks, since it allows each intermediate node to examine and manipulate the Ethernet

FIGURE 11: OTN Signal Structure and Terminology. The Carrier Ethernet Frame is Carried as the Payload of an Optical Channel Payload Unit (OPU)

FIGURE 12: Characteristics of Some of the Optical Channel Data Units

Payload

OPUOverhead

ODUOverhead

Carrier Ethernet frame

Header SVLANID

User data

Payload

Optical Channel Payload Unit (OPU)

Optical Channel Data Unit (ODU)

Optical Channel Transport Unit (OTU)

Carrier Ethernet Network frame

OTUOverhead

Signal Data Rate Application Examples

ODU0 1.24416 Gb/s Transport of a timing-transparent transcoded (compressed) 1000BASE-X signal or a stream of packets (such as Ethernet, MPLS or IP) using Generic Framing Procedure.

ODUflex (GFP)

Any configurable rate Transport of a stream of packets (such as Ethernet, MPLS or IP) using Generic Framing Procedure.

ODU2e 10.3995253164557 Gb/s Transport of a 10 Gigabit Ethernet signal or a timing-transparent transcoded (compressed) 10 Gigabit Fibre Channel (10GFC) signal.

control information, i.e. to perform statistical multiplexing and differentiation of services at the Ethernet level.

• Framing according to the OTN standard: The frames of the Carrier Ethernet network are carried over a WDM wavelength by ODUs according to the OTN standard. This encapsulation is especially favorable when the Carrier Ethernet network extends over longer distances, or the data is to be transported via an intermediate core OTN network. Using ODU2e framing between EMXPs allows the use of OTN’s inherent FEC mechanisms and optical path monitoring bits, which are important for long-reach links. Since the encapsulation is in ODU format, these data units can also easily traverse any intermediate OTN switches transparently before reaching their final destination (Figure 13).

Native Ethernet means that no additional framing is applied to the data payload at the edge of the network. By treating the Ethernet packets natively, it is possible to inspect them within the intermediate network nodes and to act upon the Ethernet headers so that the combined benefits of Layer 2 intelligence and efficient Layer 1 transport can be realized. This becomes especially important at the edge of the

FIGURE 13: Native Ethernet and OTN Framing with XTM Series EMXPs. The Two-colored Bars Symbolize Ethernet Frames, with the Control Information in Red





EMXP EMXP

Native Ethernet• Data is sent as ordinary

Ethernet frames.• Access to Ethernet

control information at intermediate nodes.

OTN framing• Data is packed into a

wrapper for transport via an OTN network.

• Ethernet control information is encapsulated.



network where decisions about traffic prioritization are made and where traffic is aggregated to “fill the pipes.” The wrapping of traffic into full OTN can then be done at the handover to the core network, after aggregated pipes of traffic that are correctly shaped have been created, avoiding wasted bandwidth.

Native Ethernet uses the VLAN tag or MPLS label to switch the frames to ports associated with either IP services or transport services. Each of these service domains is optimized and simplified for particular service types.

For example, frames containing data for high-value and high-quality IP services such as IP/MPLS, IP-virtual private network (IP-VPN) or virtual private LAN service (VPLS) can be switched to paths for transport to the desired IP devices. By contrast, frames that are destined for transport services (Ethernet, MPLS-TP or OTN) can be kept within the optical transport network, with minimal use of expensive Ethernet switching and IP routing resources (Figure 14).

The EMXPs have OTN framing capabilities for ODU2e, including optional G.709 FEC on all 10 Gb/s ports. The optional ODU2e framing on 10 Gb/s ports allows the native 10 Gb/s

Ethernet frame to be mapped into an ODU2e data unit ready for transport into the OTN core, including OTN switches. This is very useful once the traffic has been aggregated as much as possible to ensure the best utilization of the 10 Gb/s circuit. Figure 15 contrasts the differing characteristics of native Ethernet and OTN framing.

An OTN core network can also provide a unified transport layer where core nodes com-bine traffic from OTN muxponder-based Lay-er 1 services with EMXP-based, ODU-framed Ethernet services. End-to-end performance monitoring is achievable even over multi-ple operators’ networks through OTN with tandem connection monitoring and Carrier Ethernet’s inherent operations, administration and maintenance (OAM) capabilities.

2.2.5 Packet-Optical Transport with the PXM

The DTN-X Family of platforms performs integrated WDM transport and OTN switching, grooming and service protection at Layer 1, based on large-scale PIC technology. Using the PXM, the DTN-X Family can map Ethernet VLANs to ODUflex subnetwork connections, which enables ODUflex-based traffic engineering in long-haul networks.

With the introduction of the PXM, the DTN-X Family supports statistical multiplexing for

Ethernet services and Layer 2 QoS. Instead of requiring a dedicated port for every service, the PXM supports port consolidation, which can provide multiple services or flows on one port, overcoming the requirement of one physical port and fiber connection per service.

The PXM maps incoming streams of data packets, such as Ethernet VLANs, into flows that can be co-routed through the network with other flows that have the same destination. The packet flows with the same destination are groomed into a single ODUflex container for efficient routing. ODUflex containers are built as necessary

and can range from 1.25 Gb/s to 100 Gb/s, allowing for right-sized transport and maximum bandwidth efficiency (Figure 16).

Having the packet streams mapped to standard ODUflex containers allows for multiplexing with other types of ODU containers and for switching the flows as appropriate at the optical layer using the DTN-X OTN Switch Module (OXM). Hence, it becomes possible to apply standard Layer 1 traffic engineering and protection to flows of Ethernet packets as well.

FIGURE 14: Service-aware Transport Enables a Differentiated Service Offering with Multiple Classes of Services with Different Characteristics

High priority

Business apps

Low priority

Operator

Signaling Interactivevoice, video

Loss intolerant data

Loss tolerant data

Native Ethernet OTN Framing

Ethernet frames are transported natively over WDM channels with minimal extra overhead.

Ethernet frames are transported in OPUs within ODU and OTU containers according to the OTN standard.

The service information contained in the Ethernet frame can be accessed at every node of the network. This allows for Layer 2 service differentiation at intermediate network nodes.

The Ethernet frame is wrapped inside the OPU/ODU and cannot be read and acted upon without demultiplexing.

No FEC without an additional Layer 1 transponder.

Includes FEC and optical path monitoring mechanisms, which are important for long-distance links.

Traffic can be carried “transparently” over an OTN network.

Provides direct compatibility with intermediate core OTN networks – ODU containers can be multiplexed and switched like any other flow.

Suitable primarily for metro and regional networks where traffic is aggregated and service differentiation is applied.

Applicable in most situations but especially in core and long-distance networks where traffic has been aggregated into larger streams.

FIGURE 15: Some Characteristics of Native Ethernet Framing and OTN Framing

FIGURE 16: The DTN-X Family with Integrated PXM Consolidates Client Ports and Routes Packet Flows over ODUflex Subnetwork Connections

ODU�ex

ODU�ex

Consolidate 10G ports into 100G ports

100 GbE

N x 10 GbE

N x 10 GbE

N x 10 GbEODU�ex

Packet-awareOTN Core




FIGURE 17: Mapping of Ethernet Traffic into ODUflex Containers Allows for Multiplexing and Switching at Layer 1 Using Standard OTN Principles

Mapping the Ethernet traffic to the ODU can be done in various ways, for example by carrying the Ethernet VLANs in MPLS-TP pseudowires (PWs), creating additional options for traffic engineering and protection (Figures 17 and 18).8

2.2.6 Leveraging the XTM Series and PXM in the Same Network

The XTM Series, with its advanced handling of Ethernet services, can seamlessly interwork with the PXM and its super-channel-based transport of Ethernet traffic mapped to ODU containers. Deploying a high-capacity OTN core based on the DTN-X Family with PXM allows for the interconnection of multiple metro networks into one intelligent packet transport infrastructure, as depicted in Figure 19. In this infrastructure, end-to-end Ethernet

services, such as those defined by Metro Ethernet Forum for CE 2.0, can be offered to users in one or more geographical regions. As shown by the dotted lines in Figure 19, packet services may span one or more metro networks, have their endpoints in an EMXP or be delivered directly to, for example, a service router from a node in the core network via the PXM in the DTN-X.

The flexible architecture of the packet transport network allows for cost-efficient aggregation of all types of packet-based traffic. Service routers can be consolidated to a few central sites and equipped with high-speed interfaces, thereby reducing overall network investment and operational costs. The packet services offered adhere to recognized Metro Ethernet Forum standards and can be managed according to well-established OAM procedures defined for Carrier Ethernet. Layer 1 protection schemes can be applied in the core network segment, while additional Layer 2 protection schemes, such as MPLS-TP switching and G.8032v2 Ethernet Ring Protection Switching (ERPS), can be applied when services are delivered via EMXPs.

PXM DWDM-LMOXMOTN

OTN

OTN

OTN

OTN

PacketProcessing

ODUj(e.g. ODUFlex

or ODU2e)

ODUj(e.g. ODU2)

ODUj(e.g. ODU2)

N x ODUj(e.g. ODU2)

Super-channel(e.g. 5x ODU4)

Client Service Support• Untagged Ethernet• Priority Tagged Ethernet • 802.1Q (C-VLAN) Encapsulation • 802.1ad (C & S-VLAN) Encapsulation • MPLS-TP LSP*

OTN Network Interfaces• ODU�exi SNCs • ODU2e SNCs*

Network Service Support• MPLS-TP (Ethernet PWE)• 802.1Q (C-VLAN) Encapsulation*• 802.1ad (C & S-VLAN) Encapsulation*• MPLS-TP LSP*

Client (e.g: 100GbE)

FIGURE 18: Adaptation of Ethernet Services into ODU Containers (Example)

FIGURE 19: Multiple Metro Networks Interconnected by a High-capacity OTN Core. Each Dotted Line Represents an Ethernet Service Carried over the Packet-Optical Network

ODU2e to A

ODU�ex to B

ODU�ex Hitless resizing

Ethernet

MPLS-TP VLAN 1 to A

VLAN 2 to B PWE X

PWE Y

MPLS-TP

MPLS-TP VLAN 3 to B

Packet Processing Steps:• Incoming VLAN examined for

destination• VLANs mapped to

pseudowires (PWs) and then to ODU�ex or ODU2e* OTN containers

• Multiple VLANs per ODU2e/ODU�ex

ODU�ex is resized on demand (hitless resizing)

Aggregation

CCAPMSAN

Hub

CATV &Internet

BusinessInternet

Mobile Backhaul

Metro 1

.

SGW

NIDNID

PXM

PXM

PXM

Cell site router

Core

Metro 2 Metro N

XTC-2

XTC-4/10



In summary: In the access and aggregation part of the transport network, where service granularity is required, a service-aware packet-optical mechanism is beneficial to support different QoS. Also, access to Ethernet OAM bytes and service tags enables end-to-end management of Ethernet services. Using native Ethernet framing offers benefits from revenue-generation, investment and operational perspectives in the access and aggregation parts of the network. On the other hand, OTN has all the benefits of a long-haul optical transport network once the traffic has been sufficiently aggregated and traverses the core part of the network. Combining these two framing schemes within one intelligent packet-optical transport architecture makes it possible to exploit the benefits of both technologies in a common infrastructure for service creation.

2.3 MPLS-TP FOR TRAFFIC ENGINEERING AND SERVICE SCALABILITY

While Carrier Ethernet networks and the use of service VLANs bring great advantages to the packet-optical network, there are some limitations in terms of protection options, traffic engineering and service scalability.

These can be addressed by the use of MPLS-TP, which is the transport profile of multi-protocol label switching.

MPLS-TP is a way to simplify Carrier Ethernet services by pre-defining connection-oriented services over packet-based networking technologies in a way that gives support for traditional transport operational models. It takes the advantages of MPLS concepts by adding more flexibility and network manageability than the basic Ethernet SVLAN architecture (Figure 20).

Principles of MPLS-TPMPLS is a technique that forwards packets based on labels, as opposed to a standard Carrier Ethernet network where the frames

are switched based on their SVLAN tags and media access control (MAC) addresses.

A label-switched path is defined between nodes where traffic enters and leaves

the MPLS-TP network. Using MPLS-TP terminology, the entry and exit nodes are referred to as MPLS-TP provider edge (PE) nodes, and any intermediate nodes being passed by the LSP are referred to as MPLS-TP provider (P) nodes. Often the physical node performing the PE function is called a label edge router9 (LER) and the intermediate transit node is called a label switching router (LSR). EMXPs can act as both an LER and an LSR, and may also combine these roles (Figure 21).

An MPLS-TP tunnel is a pre-defined MPLS-TP transport path from the source LER to the destination LER. The MPLS-TP tunnel always has an active LSP that defines the primary and working paths. It may also have a protection LSP that defines a recovery path (Figure 22).

Both the tunnel and the LSP can be envisaged as predefined circuits for information to follow through the network, and consequently tunnels and LSPs are configured in advance from the network management system. A key feature of MPLS-TP that distinguishes it from classic IP/MPLS is that management and protection are designed to operate without a dynamic control plane, i.e. similar to a traditional SDH/SONET network, where circuits are set up by the management system.

The actual data traffic is carried by a pseudowire inside the LSP/tunnel. One MPLS-TP LSP may carry one or more pseudowires, i.e. the pseudowires offer a means for multiplexing of traffic (Figure 23).

A pseudowire is an emulation of a Layer 2, point-to-point, connection-oriented service over a packet-switching network (PSN), from one attachment circuit (AC) to another. The pseudowire used in MPLS-TP is a connection established between two MPLS-TP LERs across the MPLS-TP tunnel/LSP with the attachment circuit frames encapsulated as MPLS data (Figure 25, next page).

FIGURE 20: Using MPLS-TP to Define Label-switched Paths Within the Carrier Ethernet Network





EMXP EMXP

MPLS-TP label-switched path (LSP)

LER

Label Edge Router (LER)

LER + LSR

Label Switched Path 2 (LSP 2)

LSP 3

LSP 1

Label Switching Router (LSR)

D

BCA

FIGURE 21: MPLS-TP Framework

LER

LER

LSR

Protect LSP

LSR

Active LSP MPLS-TP Tunnel

A CD

B

FIGURE 22: MPLS-TP Tunnel

FIGURE 23: Data Is Carried by a Pseudowire Defined Within the MPLS-TP Tunnel and LSP

LER

LER

LSR

LSR

Pseudowire MPLS-TP Tunnel/LSP

Attachment Circuit (AC)Attachment

Circuit (AC)

UNI

DUNI

BCA

Ethernet frame from user

Forwarded Ethernet frame

Payload

Pseudowire payload

Ethernet header

Ethernet header

Local Ethernet header

MPLS labels used for switching

PW label

MPLS label

FIGURE 24: Ethernet over MPLS Encapsulation



A Layer 2 transport service is established between two attachment circuits and the service is carried by a pseudowire. The pseudowire travels through the network in an MPLS-TP tunnel. The MPLS-TP tunnel is in turn mapped to at least one LSP: the active LSP.

Figure 26 on the next page summarizes how transport in an MPLS-TP network relates to the OSI model. Note that Ethernet framing is present at both the link layer and the client service layer.

Both Ethernet SVLAN and MPLS-TP forwarding techniques have their own benefits, and it is often advantageous to be able to offer services based on both technologies. For example, multicast services can often be deployed directly over SVLANs on Ethernet, whereas point-to-point trunks requiring protection can benefit more from the MPLS-TP features. However, MPLS-TP pseudowires can also be used for multi-point communication using VPLS,10 which allows geographically dispersed sites to share an Ethernet broadcast domain by connecting sites through pseudowires.

MPLS-TP is fully supported by the EMXP range. Any physical port on an EMXP can support both native Ethernet and MPLS-TP, allowing operators to deploy MPLS-TP where and when it makes sense for them. It is possible to run MPLS selectively per port or to separate MPLS traffic based on MAC address and VLANs within the same port. This allows seamless migration and coexistence with the two protocols running independently side by side.

Services can be extended over DTN-X-based networks through the PXM, which supports MPLS-TP and VPLS over ODUflex in an OTN core network.

In order to deploy MPLS-TP in a production network, it can be introduced either as an overlay to the existing Ethernet or incrementally as an evolutionary build-out in parallel on the same networking hardware. Infinera believes in a smooth evolution, provided by a “ships in the night” capability, with Ethernet and MPLS-TP acting in parallel as independent non-interfering protocols in the same system.

MPLS-TP in ROADM-based Optical NetworksPacket-optical networks are often deployed today over ROADM-based optical networks,

which bring a mesh-based structure to the wavelength routing and paths available through the physical network for any services. One previous drawback with Ethernet was that it was not well-suited for protection and restoration over mesh-based networks, as the available protection schemes were largely based on point-to-point or ring architectures. MPLS-TP is highly suited for a mesh-based environment and allows network operators to design network resilience strategies that are closely aligned to the physical structure of the network, ensuring the best possible resilience and service uptime.

MPLS-TP—Easy Service CreationAnother advantage of MPLS-TP is that it breaks service creation into two steps. First, tunnels are created between endpoints within the network for service and protection paths for MPLS-TP-based services. Then the network administrator simply creates new services by adding them to the tunnel endpoints as pseudowires, safe in the knowledge that all routing aspects of the service have already been handled. This brings two distinct advantages. First, it makes the solution more scalable, as it is simpler to add a large number of services

to the network. Second, it brings a very familiar look and feel to service creation as it is similar to the processes involved in traditional transport networks. This helps operators migrate from traditional transport networks to packet-optical networks.

MPLS-TP Resolves the MAC Scalability ProblemIn an SVLAN-based Carrier Ethernet network, all MAC addresses in the attached subscriber Ethernet networks are visible to every switch within the Carrier Ethernet

network. Since each subscriber network may include an extensive number of devices and MAC addresses, this results in a need for large MAC address tables in each network node, creating various problems and extra equipment cost. Using MPLS-TP, the subscriber MAC addresses are encapsulated within the pseudowire payload and not seen by the intermediate switches of the Carrier Ethernet network.

FIGURE 25: Relation Between Pseudowire, Tunnel and LSP

FIGURE 26: MPLS-TP OSI Network Layers. The Two Variants of the Physical Layer Correspond to Native Ethernet Framing and OTN Framing Respectively

LERLER LSR

LSP

ACAC

PW Tunnel

LSP IDI/F I/F I/F I/F

in/out pw-label

in labelout label

out labelin label

in labelout label

out labelin labelActive ActiveEdge EdgeTransit

Tunnel ID

Applications

Pseudowire

Ethernet

ODU framing

Label Switched (LSP)/Tunnel

TDM

WDMWDM

Ethernet

Applications layer - Layer 7

Physical layers - Layer 0/1

Client service layer

MPLS-TP layers

Link layer


Classification by:• Protocol• Source interface• VLAN• Prio-bits

Queueingbuffer resources

Interface hardware• Ethernet

High

Incoming packets Medium

Normal

Length de�nedby queue limit

Transmit queue

Outgoing packets

Low

Classify

• The EMXP has a QoS Engine with 8 Queues where different CoS are mapped against

• Queues are emptied by a scheduler that can operate in different modes

• Strict priority• Round Robin• Weighted RR

• While a frame is waiting to be scheduled out on the line it is buffered

• If there’s no buffer left the frame is dropped• This is done in wirespeed


These switches do not have to be designed with the number of subscriber Ethernet MAC addresses in mind.

MPLS-TP Allows for a Virtually Unlimited Number of CustomersThe IEEE 802.1Q standard allows for a maximum of 4094 SVLANs in a Carrier Ethernet network, with one SVLAN typically required per subscribing customer. Since MPLS-TP uses tunnels and label-switched paths to define the connectivity within the

network, there is no such upper limit for the number of subscribers that can be handled by a network using MPLS-TP.

Of course, as the MPLS-TP services supported by Infinera’s XTM Series are delivered over the same hardware platform as native Ethernet services, they also benefit from the same transport-like performance with extremely low latency and almost zero jitter, and can be combined with synchronization schemes such as Synchronous Ethernet (SyncE) when required, in mobile backhaul networks for example.

2.4 TRAFFIC MANAGEMENTLayer 2 networks rely on packet switching and statistical multiplexing when forwarding information. Statistical multiplexing implies that the ports of the network can be allocated more “bandwidth” than is totally available, so a Layer 2 packet-optical network must have an inherent mechanism to control the amount of information entering the network and passing over network links. These procedures are collectively referred to as traffic management, and the two most important such tools are traffic shaping and policing.

2.4.1 Traffic Shaping and PolicingTraffic shaping is a traffic management technique that delays some frames in a packet-optical network node in order to bring them into compliance with a desired traffic profile. Traffic shaping is a form of rate limiting, as opposed to the policing of bandwidth profiles, where excess frames are simply dropped. Normally, traffic shaping is not part of any subscriber service level agreement (SLA), but rather an internal network mechanism used by the operator to “even out” traffic flows and create fairness between users of network resources. Traffic

FIGURE 27: The Difference Between Traffic Shaping and Traffic Policing

time

time

time

timeBefore shaping

Before policing

After shaping

After policing

∙ Requires buffering + scheduling∙ Delays some packets

Infrastructure level (per port)Advanced (hierarchical) is used for advanced and fair oversubscription of services (PE-router)

Service Level

* In real implementations traf�c is measured with leaky-bucket to allow controlled bursts

∙ Requires no buffering∙ Bandwidth is metered

shaping is typically done per port, interface or other physical entity, while policing can be applied more granularly and per Ethernet service, VLAN, etc.

As shown in Figure 27, policing plays an important role in traffic management of Carrier Ethernet services and their individual SLAs, as described in section 5.6.

Traffic shaping is done by imposing an additional delay on some packets such that the traffic conforms to a given bandwidth profile. Traffic shaping provides a means to control the volume of traffic being sent out on an interface in a specified period (bandwidth throttling), and the maximum rate at which the traffic is sent (rate limiting).

A drawback with traffic shaping is increased latency and jitter for the Ethernet Virtual Connection (EVC), but the gain can be better throughout since the overall flow of frames may be improved: instead of dropping traffic in a policer, it may be better to shape the traffic to make sure no frames are lost (or at least as few as possible), avoiding retransmissions at higher protocol layers.

2.4.2 Implementing Traffic Shaping

Traffic shaping is typically done by assigning incoming packets of different priority to separate queues in the network element. For example, the EMXP uses eight queues for different classes of services (CoS). The queues are emptied by a scheduler that can operate in different modes, e.g. strict priority or round robin. While a frame is waiting to be scheduled for transmission out on the line, it is buffered in the element, and if there is no buffer left the frame is dropped (Figure 28).

When a network element is congested, it starts to drop frames if there is no buffer left to allocate. Weighted random early detection (WRED) is a way for a network element to ask clients to require fewer resources from the network. However, WRED only acts on traffic that is transported by the TCP/IP protocol that ensures that data is re-sent. With WRED, different discard profiles can be set for different CoS, and TCP traffic can be dropped randomly so that the overall load on the network is

FIGURE 28: Traffic Shaping: Quality of Service Is Assured by Buffered Queues and Scheduled Forwarding of Packets



decreased. Dropping traffic at random prevents a “global TCP sync,” in which all TCP sessions try to increase their bandwidth at the same time, creating a “perfect storm.”

2.5 MIGRATING LEGACY TDM SERVICES TO ETHERNET

2.5.1 One Common Infrastructure for Ethernet and Legacy TDM ServicesTransmission systems based on Layer 1 TDM technology, such as SDH and SONET, have been the basis for many telecommunication services offered during the last few decades. However, from a

bandwidth perspective, Ethernet traffic has already surpassed the amount of legacy TDM traffic. Operators are faced with the challenging task of maintaining existing TDM services while upgrading networks for new Ethernet services, all in the most cost-efficient way. More capacity is required to cater to the growing amount of packet-mode traffic generated by Internet access, video on demand and cloud computing, while the shrinking amount of TDM traffic must be taken care of to sustain revenues from existing services. Building a separate new infrastructure for Ethernet traffic while maintaining a shrinking SDH/SONET network is one option, but an integrated

approach where the new packet-mode network also provides legacy TDM services is an attractive alternative (Figure 29).

Many legacy TDM services can be replaced by Ethernet equivalents and are well-suited for the packet-optical infrastructure described in the previous sections. However, there is a significant portion of traffic that cannot simply be migrated to Ethernet. This fact creates a dilemma for network operators as SDH/SONET systems start to reach end of life, or if a large SDH/SONET or even E11 network must be supported for a small number of services or subscribers.

Infinera’s packet-optical platforms offer several alternatives for the handling of legacy TDM-based services in a way that greatly facilitates a gradual shift toward one common, packet-mode, Ethernet infrastructure for all traffic.

• Existing Layer 1 services carried over SDH/SONET or E1 networks can be transported separately from Ethernet services using different wavelengths, while still using the same WDM platform and optical transmission network. Infinera’s platforms include powerful ROADMs for flexible handling of the optical paths and muxponders/transponders that adapt SDH/SONET traffic to optical transport.

• Layer 1 services such as E1, STM-1/OC3, channelized STM-1/OC3, STM-4/OC12 and STM-16/OC48 can easily be adapted for transport over Ethernet through the use of Infinera’s Intelligent SFP (iSFP) pluggable optics that provide circuit emulation over SyncE-capable Ethernet. The iSFP modules fit into any EMXP Gigabit Ethernet port, allowing a very flexible and tactical service migration.

2.5.2 Using the iSFP to Convert TDM Services for Ethernet Transport

The iSFP is used to deliver TDM services over a network built for Carrier Ethernet services. It provides circuit emulation for STM-1/OC3, STM-4/OC12 or STM-16/OC48 SDH/SONET services as well as E1 services or channelized STM-1 signals over a SyncE-capable Carrier Ethernet network.

The iSFP module performs packetizing of the TDM service, converting for example an STM1/OC-3 service into a 170 Mb/s Ethernet stream or an STM-4/OC-12 service into a 680 Mb/s stream, creating a transparent bit pipe between two locations in the packet-optical network. The existing TDM service, including payload structures, overhead bytes and protection protocols, is transparently migrated to the Carrier Ethernet network. Service adaptation is done at the edge of the network, and standard Ethernet frames are used between the ingress and egress points.

The TDM traffic is mapped into an Ethernet Virtual Connection that can be transported either as an Ethernet service VLAN or via an MPLS-TP service. The EMXPs can also be used to perform Layer 2 aggregation to ensure full utilization of higher-speed 10 Gb/s Ethernet connections. The adaptation of TDM services to Ethernet follows open Internet Engineering Task Force (IETF) standards (Transparent SDH/SONET over Packet and Structure-Agnostic Time Division Multiplexing over Packet) for TDM-over-packet transport (Figure 30).

SynchronizationNetwork synchronization is the cornerstone of an SDH/SONET network. Here the quality of the underlying Ethernet network is critical. The Infinera XTM Series provides excellent Synchronous Ethernet performance due to patented innovations in circuit design.

In Infinera’s iSFP solution, synchronization transport is provided by the differential clock recovery (DCR) mechanism, transferring the SDH/SONET clock to the transparent emulated service with SyncE as a reference at both ends. DCR transfers the synchronization clock to the emulated service and extracts it again at the far end of the service. (See also section 5.3.2.)

The two directions of the service can operate as two independent timing domains or one direction can be frequency-locked to the other direction.

In cooperation with an ecosystem partner, Infinera also supports synchronization

FIGURE 29: Migration Toward a Single, Packet-oriented Transport Infrastructure for All Services

SDH/SONET EthernetEthernet/MPLS-TP Transport

CE 2.0SyncE Capable

Layer 1Point-to-Point Services• E1• STM-1/OC-3• STM-4/OC-12• STM16/OC-48

Layer 1Point-to-Point Services• E1• STM-1/OC-3• STM-4/OC-12• STM16/OC-48

CE 2.0 Services• E-Line• E-LAN• E-Tree• E-Access• E-Transit

Existing Layer 2 Services• E-Line• Best effort transport

Ethernet Network

Steady stream of bits

Encapsulation headers

Ethernet headers

Steady stream of Ethernet frames

* RFC 4553 and draft-manhoudt-pwe3-tsop

SDH: 810 bytesE1: 256 bytes

• Provides TDM services at the edge of an Ethernet network• Transparent TDM over packet (TSOP and SaTOP) are outlines in open IETF drafts/standards*• TDM streams are “chopped up” and carried in Ethernet packets

Transparent transport of:• Payload structures• Overhead bytes and protection protocols• Synchronization• Line rates: E1, PC-3/STM-1 and OC-12/STM-4

TDM Network

FIGURE 30: Transparent SDH/SONET and E1 over Packet



management and monitoring. Probes can be deployed in the network to monitor SyncE quality, which is used as a reference for both ends of the transparent TDM service. The probes can also be used to monitor the SDH/SONET sync quality of the connected systems to provide confirmation of the sync in the TDM service that is carried over the Ethernet network (Figure 31).

ProtectionTo maintain SDH/SONET protection, the existing Automatically Switched Optical Network (ASON) and subnetwork connection protection (SNCP) protection

schemes are replaced by MPLS-TP and Ethernet protection options. Protection is supported by MPLS-TP for topologies such as ring, mesh and partial mesh. In addition, protection of the employed Ethernet Virtual Connections can be accomplished by the Ethernet Ring Protection Switching (ERPS) and Link Aggregation Group (LAG) options, as described in section 5.4.

2.6 THE MAIN ELEMENTS OF AN INFINERA PACKET-OPTICAL NETWORK

Having looked at the principles of packet-optical networking, it is time to discuss

how these functions are distributed in a complete network and between products.

A key objective in the development of the Infinera packet-optical portfolio has been to expand the number of services that can be provided by and over an optical infrastructure. More services over the same network mean more revenue and less cost for the operator. By integrating a selected set of Layer 2 functions with the optical layer, the network becomes much more potent in terms of service offerings and can be made more scalable. The tight integration between Layer 1 and Layer 2 also makes it possible to increase resilience and improve traffic management in ways not possible with less integrated approaches.

Another main objective of the chosen packet-optical architecture has been to make it agnostic to the underlying transport network technology. Traffic flows over a ROADM-based, photonic network that provides connectivity and can be used for transparent Layer 1 services. In addition, an Infinera packet-optical network can seamlessly interoperate with its transport network over MPLS-TP tunnels, Ethernet SVLANs or OTN switches and any combination of these.

A third central objective when developing the Infinera packet-optical portfolio has been to enable multi-layer traffic and service management. Depending on traffic load or link degradation, the packet-optical network must be able to switch between different transport alternatives, in order to deliver the highest possible quality of service for subscribers. The tight integration between Layer 2 and Layer 1 functionality in the architecture also enables software-defined networks and advanced traffic management.

2.6.1 Ethernet Demarcation Units and Network Interface Devices

Demarcation of a provided service is a key function of the packet-optical access network, as it enables the service provider to extend its control over the entire service path, starting from the customer handoff points. The customer’s equipment is connected to the Carrier Ethernet network via a provider-owned demarcation device (Ethernet demarcation unit [EDU] or network interface device [NID]) deployed at the customer location (Figure 32). These units enable a clear separation between the user and provider Ethernet networks.

Infinera’s Ethernet Demarcation Unit is an independent unit that supports service level agreement management capabilities, including sophisticated traffic management and hierarchical quality of service mechanisms; standard end-to-end operations, administration and maintenance and performance monitoring and extensive fault management and diagnostics, all to reduce service provider operating costs and capital expenses.

The same set of demarcation functionality is also available via Infinera’s Network Interface Device, which is a port device supported by its parent EMXP. The NID performs service OAM but leaves the service policing and tagging to be done by the EMXP, reducing the cost and complexity of the customer-located equipment.

FIGURE 31: Using Synchronous Ethernet (SyncE) as a Reference for SDH/SONET Differential Clock Recovery

E1STM-1/OC-3STM-4/OC-12STM16/OC-48

E1STM-1/OC-3STM-4/OC-12STM16/OC-48

Common timing reference from Synchronous EthernetDifferential clock recovery (DCR)

Primary Reference Clock

A Z

FIGURE 32: Infinera’s Network Interface Device (NID)

2.6.2 Packet-Optical Transport Switches (EMXP) and the PT-Fabric

In the metro aggregation network, traffic enters via an EMXP in the first node. The same physical node may also include muxponders/transponders using additional WDM channels for fully transparent Layer 1 transport, with Layer 2 transport used only where inspection and OAM information is required at intermediate points, and where traffic needs to be aggregated by statistical multiplexing.

The EMXPs are available in various configurations and mounting options, such as one- or two-slot chassis-mount units and a temperature-hardened access unit for deployment in non-controlled environments – all with full wire-speed forwarding on all ports for all traffic types.

FIGURE 33. Two Examples of Infinera’s Range of Packet-Optical Transport Switches (EMXP) in the XTM Series, a Chassis-based Unit and a 1RU Access Unit



A further development of the EMXP concept is the Packet Transport Fabric (PT-Fabric), where a high-capacity EMXP is interconnected to external 10 Gb/s and 100 Gb/s input/output (I/O) client units via a separate frontplane bus. The optical frontplane takes vertical-cavity surface-emitting laser (VCSEL) technology used in supercomputing, and brings it to optical transport equipment for the first time.

This frontplane enables the PT-Fabric to be deployed in any combination of available slots in single or multiple XTM Series chassis and without any backplane capacity limitations. Multiple PT-Fabrics can be deployed in a single chassis, giving a total of 4 terabits of switching within one XTM Series chassis.

The metro aggregation and metro core networks are interconnected via the XTM Series packet-optical platform, which can provide switching at OSI Layers 1 and 2. In the metro core, Ethernet SVLANs and MPLS-TP enable detailed handling of Layer 2 services, while WDM keeps legacy transport services at Layer 1. A ROADM-enabled Layer 1 provides flexible wavelength switching capabilities, while a unified control plane provides management

capabilities across the metro aggregation and core networks.

The metro core network usually connects to data centers, and also to IP core and mobile core networks, typically implemented by a set of IP routers using full IP/MPLS12 for traffic engineering purposes. Normally operators prefer to have a clear demarcation point toward the metro network at the edge of the IP/mobile core network, often referred to as a provider edge router. The routers of IP/mobile core networks have broad functionality and very high capacity; they are therefore normally interconnected via a strict Layer 1 WDM network, since the main objective is to provide “fat pipes” without any need for Ethernet aggregation.

2.6.3 Optical Add-Drop Multiplexers and Other Optical ElementsThe Layer 2-specific elements of metro aggregation and metro core networks use Layer 1 optical WDM channels for the transport of Ethernet frames between network nodes, as described in section 2.2.3. All the above Layer 2-specific network elements interwork seamlessly with the flexible optical networking elements at Layer 1 when present in a truly integrated packet-optical platform, such as Infinera’s XTM Series. An XTM Series node may

include Layer 1 transponders and muxponders, EMXPs, ROADMs and other optical network elements.

2.6.4 The DTN-X Packet-Switching Module (PXM)

The metro/regional packet-optical network described above can be further augmented, both in the metro core and regarding integration with long-haul optical networks, by Infinera’s DTN-X Family of converged digital photonic platforms. The DTN-X Family is a terabit-capable super-channel system leveraging large-scale photonic integrated circuit technology, integrating OTN switching and WDM transport. With the inclusion of the PXM, the DTN-X family is transformed into a fully packet-aware platform where packet flows can be aggregated and switched on individual ODUs to facilitate traffic engineering (Figure 34).

The PXM maps the Ethernet VLANs and MPLS services of the packet-optical network to the Layer 1 OTN transport services. This mapping is especially cost-saving when an OTN system is already in place, e.g.

for long-haul purposes or in a densely populated area, and where the packet-mode traffic now can be transported by individual flows fully managed according to OTN standards. Furthermore, the PXM is certified for and can terminate CE 2.0 services such as E-Line and E-LAN, making it possible to set up Ethernet services starting in an NID in the access network and terminating via a DTN-X PXM in the metro core or long-haul network. More about how the PXM maps Ethernet traffic to OTN can be found in section 2.2.5.

2.6.5 The Multi-layer Service Management SystemThe chosen network architecture has a profound influence on the degree of operational simplicity that it is possible to achieve when it comes to network

management. The real benefits of packet-optical networking can only be realized with a truly integrated Layer 1 and Layer 2 transport platform and a unified Layer 1 and Layer 2 management system.

A network for Carrier Ethernet services may be implemented as a separate Layer 1 optical network with Layer 2 Ethernet switches attached externally over standard interfaces, as depicted in Figure 35. Although conceptually simple, such a configuration results in a complex hierarchy of management systems. These systems must be carefully integrated in FIGURE 34: Block Diagram Showing the Main Functions of the Infinera DTN-X Family

Tributary Modules Line ModulesSwitch ModulesOTN

OTN

OTN

OTN

OTN

ODUk <-> ODUJ(e.g. ODU2 <-> ODU0)

ODUk(e.g. ODU2)

Client(e.g. 10GbE)

ODUj(e.g. ODU0)

ODUj(e.g. ODU0)

N x ODUj(e.g. ODU0)

Super-Channel(e.g. 5x ODU4)

OTN Mapping

OTN Multiplexing

OTN Switching

WDMSuper-Channels

FIGURE 35: Ethernet Services Provided by Separate Ethernet Switches Attached to an Optical Network Result in a Complex Hierarchy of Management Systems

Layer 2 (Packet)

Layer 1 (Optical)

Element Management System for the packet layer

Service Management

System

Element Management System for the optical layer

Integration


order to provide a useful Ethernet service provisioning and assurance environment.

Using a true packet-optical network and an integrated multi-layer management system, such as the Infinera DNA-M Digital Network Administrator for XTM Series, improves this situation.

A multi-layer management system has access to both Layer 1 and Layer 2 network

status information and can manage both optical- and packet-mode equipment. Since Layer 1 and Layer 2 functions are handled by a single system, provisioning of Ethernet services affecting both Layer 2 and Layer 1 can be done by simple point-and-click commands from the management system. Furthermore, Layer 2 services are monitored from end to end and adequate Layer 1 resources can be allocated directly, should optical paths be broken or changes in the traffic pattern occur.

This multi-layer management approach brings further benefits in terms of lower cost for management hardware, less training, less integration and simpler administration and maintenance of the entire network. Especially for network operators that do not already have an extensive operations support system (OSS) in place, the unified packet-optical management system offers significant advantages over the integration of multiple separate management systems.

2.7 ADVANTAGES OF PACKET-OPTICAL TRANSPORT

Now that we have a better understanding of the fundamental principles of packet-optical networking, we can summarize the advantages, both in general terms and more specifically as they are implemented with the Infinera product portfolio.

2.7.1 Benefits of the Packet-Optical ApproachPacket-optical networking in general helps the operator attain several types of valuable advantages:

Reduced Investment and Operational CostsThe network will have fewer physical units through the integration of Layer 1 and Layer 2 transport functions in the same hardware platform. This integration reduces packaging and cabling costs, as well as all types of inventory. This integrated approach is designed to reduce network complexity and lower operational expense (OpEx) significantly.

Efficient aggregation of Ethernet traffic can solve the problem of underutilized WDM channels at the edge of the network. By filling the pipes, packet-optical transport increases the utilization of available bandwidth and facilitates an efficient handover to the core network at a lowered cost.

If the network can distinguish between high-value IP services and legacy traffic, it is possible to offload legacy traffic to the most cost-effective transport at Layer 1, avoiding the consumption of equipment ports with higher complexity and cost, except where needed.

Potential for Additional Service RevenuesService awareness is critical if the differentiated QoS requirements of new multimedia applications and cloud services are to be met end-to-end throughout the network. To do that, it is important to retain service transparency where necessary, i.e. to leverage all existing information present in Ethernet VLAN tags and MPLS labels, for example. This should be done until such a point in the network where traffic with the same QoS requirements has been fully aggregated.

Full service awareness enables the operator to differentiate and market higher-value services with specific SLAs, increasing service revenues.

Better Customer Satisfaction and Thereby Reduced ChurnMassive scalability is easily achieved through the WDM functionality in an optical/Ethernet platform, where multiple services can be assigned to the same wavelength or to their own specific wavelengths. New wavelengths can be added on an as-needed

basis. Support for scaling Ethernet services is the most important requirement, since Ethernet traffic will constitute the great majority of future growth in bandwidth requirements. With an integrated packet-optical platform, it is easy to upgrade transport capacity as subscriber demand grows.

End-to-end OAM is important to ensure the reliability and resiliency of the transport underlying the services carried over the network. It is important to ensure that both the SLAs and service objectives that are internal to the service operator and those that might explicitly be offered to a subscriber are met. Efficient tools for operations and maintenance are vital to achieve customer satisfaction.

FIGURE 36: Ethernet Services Provided by an Integrated Packet-Optical Platform and Managed from a Multi-layer Management System such as DNA-M to Simplify Operations and Reduce Cost

DNA-M

Unified Packet-Optical Service Management and

Element Management (DNA/DNA-M)



2.7.2 Advantages to Using Infinera’s Portfolio for Packet-Optical Transport

Packet-optical transport provides many general advantages, as described above. But other characteristics more related to the actual implementation of the packet-optical nodes used in the network are also of significant importance. Infinera’s portfolio for packet-optical networking combines Infinera’s long and recognized optical networking experience with outstanding Ethernet/Layer 2 capabilities, including MEF Carrier Ethernet 2.0 services, MPLS-TP and OTN-compatible transport. Furthermore, the elements in this portfolio are truly optimized for the simplest and most cost-efficient implementation of packet-optical transport networks.

A Lean and Transport-centric Implementation of Ethernet/Layer 2 FunctionsThe packet-mode functionality of a packet-optical node may include anything from a simple Ethernet bridge to a full-fledged IP router and more. When implementing a transport-focused packet-optical node, it is of the utmost importance to select an optimal

level of functionality for the transport task. Including superfluous functions adds to both equipment cost and operational complexity, especially in the aggregation and metro/regional segments of the network, which have the majority of the network nodes. On the other hand, the node must include sufficient features to make service differentiation, OAM and necessary resilience possible.

The EMXPs and other Layer 2 equipment have all been designed with this balance in mind. Functionality of value in a metro aggregation or metro core network, such as the capability to handle MPLS-TP, is included, while most of the complex IP handling has been omitted.

Efficient Transport of SynchronizationAccurate synchronization is essential, especially in mobile networks. As an example, Long Term Evolution (LTE)/4G and 5G networks need an accurate time stamp as well as frequency synchronization. The EMXPs and PT-Fabric fully support ITU-T SyncE standards for the distribution of timing signals throughout the packet-optical network.

IEEE 1588v213 is a common standard to deliver both phase and frequency information over a packet network, but it is sensitive to packet delay variation (jitter). IEEE 1588v2 can be deployed over any legacy Ethernet network, but it may rapidly lead to quality problems if jitter becomes too high. With Infinera’s transport-centric, almost-zero-jitter implementation of network nodes, IEEE 1588v2 will converge fast and can deliver the required time stamps.

Low LatencyThe EMXPs and the PT-Fabric are designed for transport and capacity; they have no central switching fabric, no input queues and no network processors limiting performance. This results in minimal delay and no jitter within the EMXPs and the network. The ultra-low latency has significant value overall, and is crucial in certain applications, e.g. between data centers and in algorithmic trading applications for the financial world. Additionally, the stability and low latency of the EMXPs add minimal jitter and delay in mobile backhaul applications, enabling more and longer radio hops when a combined wireless and wired backhaul network is being deployed.

Low Power ConsumptionEnergy costs can be a significant item in the OpEx of any telecommunications network. The hardware elements of Infinera’s intelligent packet transport architecture have all been designed with this in mind, with typical power consumption as low as less than 7 watts (W) per 10 Gigabit Ethernet, for example.

Not only does low power consumption save on direct energy costs, but for every kilowatt saved in equipment power, an additional 0.5 kilowatt in reduced need for air conditioning is saved too.

High-density ImplementationThe EMXPs and PT-Fabric have all been designed to provide as high port density per unit as possible while still maintaining high capacity. This translates to smaller footprint and reduced volume requirements, leading to less costly installations than with legacy equipment.

Multi-layer ManagementInfinera’s multi-layer DNA-M management system enables unified Layer 1 and Layer 2 OAM of the packet-optical network. The management system supports the full range of operations, administration and maintenance functions defined by MEF for Carrier Ethernet 2.0. Furthermore, the system fulfills the requirements of ITU-T Recommendation Y.1731, which also addresses performance management.


3 O P E R A T I N G T H E

N E T W O R K


3.1 CHAPTER SUMMARYThe cost-effectiveness of a real network is highly dependent on how easy it is to operate and how well it can adapt to shifting service demands. While a well-designed multi-layer network management system is critical to efficient operations, network operators are also increasingly looking to software-defined networking technologies to enable more agile, dynamic service delivery and multi-layer optimization.

In Chapter 3, we therefore take a closer look at the network management principles applicable to packet-optical networks, and at how SDN complements network management for dynamic service delivery and network optimization.

The key point to emphasize is that multi-layer network management and multi-layer SDN are complementary. While SDN generally focuses on the automation and programmability of end user services,

multi-layer network management continues to support the full range of infrastructure management, from configuration and element management to monitoring, fault isolation and service assurance. Traffic engineering and service provisioning can be shared between traditional network management and SDN platforms, with hybrid operation that protects existing services while introducing new SDN-driven services on the same physical network. With this approach, SDN can be introduced step by step, for example, in only one domain of the network or applied only to a particular OSI layer.

3.2 MULTI-LAYER NETWORK MANAGEMENT

Many of the advantages of packet-optical networking originate from the ability to manage both the Layer 1 optical channels and Layer 2 Ethernet services of the network in a coordinated way (Figure 37). Layer 1 optical channels can be established at the same time as the Ethernet service attributes are assigned to the Ethernet Virtual Connections that are using the Layer 1 optical channels. This allows for an easy procedure for end-to-end service creation. Importantly, it is only then possible to efficiently monitor the performance of an

FIGURE 38: Service Provisioning via Network Management and SDN on the Same Physical Network

Network ManagementTraf�c engineering

Device and network management (FCAPS)Traditional service management

Software-defined NetworkingProgrammable services

Dynamic multi-layer service provisioningand optimization

App 1 App 2 App N

Network Manager

Multi-Layer SDN Platform

OSS SDN Applications

Physical Transport Network

Ethernet service and determine if a fault originated in the optical or packet switching elements of the network.

Coordinated handling of the optical and Ethernet layers of the network calls for a multi-layer management system and, when using SDN, a multi-layer SDN platform (Figure 38). These systems must be designed from the start to enable multi-layer visibility and management, while also keeping ease of use for network operators

in mind. Infinera’s suite of management systems, including the Infinera Digital Network Administrator (DNA), provides network operators with full control of their integrated packet-optical network and supports planning, deployment and operation of the network infrastructure in a cost-efficient and rational way.

3.2.1 Management FrameworkA great deal of the standardization of management procedures has taken place

among operators and their vendors in order to facilitate business and network operations. TM Forum (TMF, formerly TeleManagement Forum), an industry association for service providers and their suppliers in the telecommunications and entertainment industries, has played a central role in this work.

Frameworx, which was developed by TM Forum, is a suite of best practices and standards that enable a service-oriented, automated and efficient approach to business operations. Frameworx provides hundreds of standardized business metrics that allow for benchmarking, as well as a suite of interfaces and APIs that enable integration across systems and platforms. Frameworx also includes adoption best practices and examples.

The central business process model used in Frameworx is the Business Process Framework (eTOM),14 which also is maintained by TM Forum (Figure 39, next page). This model describes the required business processes, including network management and operations, of service providers, and defines their key elements and how they should interact. eTOM is a guidebook that defines the most widely used and accepted standard for business processes in the telecommunications industry today.

FIGURE 37: An Ethernet Service in a Packet-Optical Network Is Created over Multiple Layers of Underlying Connections that Need to Be Managed in a Coordinated Way

NID NID

Ethernet service no. 1

Ethernet service no. 2

The Ethernet Virtual Connection

The optical channel (λ)

The optical fiber and its amplifiers

O P E R A T I N G T H E N E T W O R K


Fulfillment Assurance Billing

Customer QoS/SLA Management

n/a

n/a

n/a

Service Problem Management

Service Configuration & Activation

Resource Provisioning Resource Trouble Management

Resource Performance Management

Cus

tom

erSe

rvic

eR

eso

urce

For the management of packet-optical networks, two areas (columns in the eTOM model) are of particular interest:

• Fulfillment, which involves supply chain activities for assembling and making services available to subscribers. These

activities create an infrastructure that matches the supply of services with subscriber demand in an economical way, and with consistently high levels of quality and reliability.

• Service assurance, which is the application of policies and processes to ensure that services offered over the network meet

a predefined service quality level for an optimal subscriber experience.

Infinera’s network management philosophy fully adheres to the relevant TM Forum standards. For example, for integration with back-office support systems, DNA comprises Frameworx-compliant web-services interfaces based on the TMF608 model. These interfaces hide the complexity of the underlying optical network and are designed to reduce the time, risk and costs associated with systems integration.

Infinera’s DNA multi-layer management functions (DNA-M) provide the following full suite of MTOSI 2.015 -compliant interfaces:

• Inventory• Alarms• Activation/provisioning• Performance statistics

Another method often used to classify the many diverse actions involved in the management of communications networks is the FCAPS suite:

• Fault management (F) encompasses functions for detecting failures and isolating the failed equipment, including the restoration of connectivity.

• Configuration management (C) refers to functions for making orderly and planned changes within the network. An important part of configuration management is keeping an inventory of equipment, software releases, etc. in the nodes.

• Accounting (or administration) management (A) deals with functions that make it possible to charge users for the network resources they use.

• Performance management (P) comprises functions for monitoring and fine-tuning the various parameters that measure the performance of the network and forms the basis for service level agreements with network users.

• Security management (S) refers to administrative functions for authenticating users and setting access rights and other permissions on a per-user basis.

3.2.2 Infinera’s Management Software SuiteInfinera offers a full suite of tools for the management of its optical and packet-optical transport networks. These tools help operators with tasks throughout the entire service and network life cycle when planning, deploying and operating the packet-optical

network and its services. The suite has been created based on the Frameworx standards and eTOM framework described above, but given the role of Infinera as an infrastructure vendor, Infinera’s management suite focuses on the subset of activities in eTOM shown in Figure 40.

The components included in Infinera’s suite of management software can broadly be grouped into five categories, as depicted in Figure 41 (see next page).

At the most fundamental level, each network element of an Infinera packet-optical network supports local configuration

FIGURE 39: The eTOM Business Process Framework. Source: TM Forum

FIGURE 40: The Main eTOM Business Processes Supported by Infinera’s Management Software Suite

Strategy, Infrastructure & Product

Enterprise Management

Strategy & Commit

OperationsSupport & Readiness

Fulfillment Assurance Billing & Revenue Management

InfrastructureLifecycleManagement

ProductLifecycleManagement

Marketing & Offer Management

Strategic & EnterprisePlanning

Financial & Asset Management

Stakeholder & External Relations Management

Human Resources Management

Enterprise Risk Management

Enterprise Effectiveness Management

Knowledge & Research Management

Service Development & Management

Resource Development & Management(Application, Computing and Network)

Supply Chain Development & Management

Customer Relationship Management

Service Management & Operations

Resource Management & Operations(Application, Computing and Network)

Supplier/Partner Relationship Management

Operations



and control by an internal element manager accessible through a local I/O device. Centralized, multi-layer management of all the elements in a network is done from a network manager – Digital Network Administrator – that provides an end-to-end view of the connections and services.

The network operator often has extensive other operations and support systems for administration, billing, etc. of the services provided. Hence, the network manager must interoperate smoothly with the operator’s OSS environment, a task accomplished by an OSS integration toolkit. It is also often desirable to give

network subscribers direct access to certain network management capabilities through a customer portal. Such a portal allows subscribers to monitor the performance of their services and adapt service parameters without a need for manual operator intervention. Finally, the suite includes planning and design tools, of great value when preparing network deployment and expansion.

3.2.3 Infinera Digital Network Administrator A central component in Infinera’s management suite is the Digital Network Administrator, a cost-effective and scalable carrier-class service and network management system. DNA provides a centralized system for operations, administration and management of the entire packet-optical network and hides the complexity of the underlying equipment to higher-order business and operations support systems. Its integrated management capabilities also provide the foundation for service management of individual end-to-end connections. DNA increases the visibility of the network and simplifies many repetitive tasks, with the goal of increasing the performance of the network while lowering operational expenses (Figure 42).

3.2.4 A Unified Information Model for Multi-layer ManagementAt the center of DNA’s multi-layer management architecture for packet-optical networks is the unified information model. There are several standards for modeling multi-layer networks: the ITU-T G.805 recommendation provides a description of circuit-switched network connections through a multi-layer network, while the G.809 recommendation provides the same but for connectionless networks. G.805 and G.809 provide generic methods for modeling and describing networks from a functional and structural architecture perspective.

The G.805/G.809 model can be mapped into management information models using the equipment management functions specified in the TMF608 (MTOSI 2.0) model. Management information models are specifically important because they formally define and describe the reference points that the operator’s OSS must interact with in order to manage a piece of transport equipment.

The unified information model allows the entire network to be modeled, all the way from the optical fibers run in the ducts up to the services created on top of them,

including all the OSI layers in between. At the bottom of the multi-layer information model used by Infinera is the TMF608 model for the Layer 1 network. The higher-order models for connection-oriented Ethernet (802.1Q tunneling, or Q-in-Q), MPLS-TP and multi-point Ethernet models are added onto the Layer 1 model, as indicated in Figure 43 (on the next page).

FIGURE 41: An Overview of Infinera’s Management Software Suite

End-user service management

Management of In�nera networksLocal management of nodes

Element Managers

OSS Integration Toolkit

Network Planning System

Customer Portals

Optical & service planning

Network Managers

Intagrate into OSS environment

Local Craft Terminal

• SNMP• TL1• XML

Netcool Open View OSS SDN

Digital Network Manager

OA

ROADM

EMXPEMXP

MDU MDU

OA

XTM SeriesMetro Access/Core

Inventory Provisioning

XTM Series

Digital Network Administrator

MTOSI 2.0

DTN-XMetro Core and

Long-haul

NMS Performance

FIGURE 42: Infinera Digital Network Administrator (DNA)

The fact that there is a single, unified information model from the fiber up to the Ethernet service is the enabler of unified management as well as all the operational benefits from using a truly integrated packet-optical network. In a network with separate Layer 1 and Layer



2 service delivery, a unified view must be achieved in a higher-order OSS through the integration of multiple systems. The unified information model, by contrast, enables all administrative and management actions, such as planning, provisioning and operations, to be performed across Layer 1 and Layer 2.

3.2.5 Layer 2 Service FulfillmentFor networks with Layer 2 equipment, DNA-M offer a provisioning web application that provides point-and-click provisioning for both Ethernet SVLANs and MPLS-TP paths. The provisioning app provides point-and-click creation of tunnels, label-switched paths and pseudowires, which automates the

configuration process and reduces the time and cost to provision resources and services.

To be able to deliver and provision a Layer 2 service, the optical channels must also be configured. One advantage with DNA multi-layer management is that the operator can provision an optical channel with the same management system before configuring a Layer 2 service, effectively speeding up the entire provisioning process.

Provisioning services via a multi-layer SDN control platform (as outlined in Section 3.3), such as the Infinera Xceed SDN controller, automates and simplifies this process even further, because the platform includes multi-layer path computation and constraint-based provisioning logic to enable the provisioning of all layers based on Layer 2 service requirements.

3.2.6 Layer 2 Service AssuranceA Carrier Ethernet network must provide the ability to monitor, diagnose and centrally manage the network, using standards-based vendor-independent implementations. Infinera’s DNA takes this one step further by offering management across the layers to further simplify management, and increase visibility of the network, with the goal of reducing operational costs.

The packet-optical assurance web app in DNA-M extends the management system’s operational model and well-known

management processes from Layer 1 into Layer 2. The app is plug-and-play on top of the Layer 1 assurance web app, and not only adds Layer 2 management capability but also provides integrated Layer 1 and Layer 2 management from one unified graphical interface.

Performance management of a Layer 2 network comprises the measuring of throughput, delay, jitter, and packet loss for a particular service. To define the endpoints between which such measurements are taken, Maintenance Entity Groups and Maintenance End Points are defined, as described in section 5.7.4.

The assurance web app provides a complement to established Ethernet standards for fault management, such as IEEE 802.1ag (Connectivity Fault Management or CFM). The goal of CFM is to monitor an Ethernet network and pinpoint where a problem occurs, while DNA-M provides a graphical multi-layer user interface to help in quickly finding the exact root cause of a networking problem, independent of whether it occurs on Layer 1 or Layer 2.

The web app discovers and tracks the operational state of the Layer 2 services and the underlying Layer 1 paths that support them. Fault information is indicated not only on the map but also graphically for individual paths, links and ports. If the origin of a networking problem resides in Layer 1, the user can turn to the features in the Layer 1 assurance app to quickly resolve the problem without having to change system or interface. The app also presents G.826 performance statistics for Layer 1 (Figure 44).

3.2.7 Network Planning and OptimizationIntelligent planning of network build-out is essential, both when new networks are established and when expanding an existing network. Infinera’s management suite therefore includes the Infinera Network Planning System (NPS), which facilitates planning tasks such as traffic engineering of optical links, planning of service routing and network optimization. The system supports extensive reporting of results in formats such as deployment packages, equipment bills of material, circuit traces and more.

3.3 SOFTWARE-DEFINED NETWORKING AND NETWORK VIRTUALIZATION

3.3.1 Basic ConceptsSoftware-defined networking is an approach to computer networking that allows the management of network services through an abstraction of the managed network and its functions. This network abstraction enables a physical decoupling of the system that makes decisions about where traffic is sent (the control plane) from the underlying system that forwards traffic to the selected destination (the data plane). Thus, instead of being executed in each network element,

FIGURE 43: The DNA Multi-layer Management Architecture is Based on a Unified Layer 1 and Layer 2 Information Model

Layer 1 TMF608 based information model

TMF608 based Ethernetmulti-point information model

TMF608 MPLS-TPinformation model

TMF608 SVLAN connectionless

information model

Point-to-point EVC(E-Line, E-Access)

Multipoint EVC(E-LAN, E-Tree)

MTOSI 2.0 compliant interfaces

L1/L2 Service Assurance

L1/L2 Service provisioning

L1/L2 Planning tool

XML-�le<ns2:name> <ns13:rdn> <ns13:type>MD</ns13:type> <ns13:value>TMPC-MACA</ns13:value> </ns13:rdn> <ns13:rdn>

OA

ROADM

EMXPEMXP

MDU MDU

OA

FIGURE 44: DNA-M Web App with a Multi-layer View of an Ethernet Virtual Connection



the functions of the control plane can be centralized to one or more SDN controllers.

The Infinera Xceed Software Suite is a portfolio of software solutions based on the OpenDaylight platform. In addition to this

open platform, the Xceed solution includes open application programming interfaces both “northbound” and “southbound” to and from the controller and a published open data model. This allows Infinera customers or third-party developers to integrate additional functionality beyond the range of Infinera-developed SDN applications, and to include non-Infinera network elements in the solution.

An SDN controller may control how optical paths are established, how Ethernet frames are switched and how IP packets are routed in a packet-optical network, or in the case of a multi-layer SDN controller, all of the above. It may even direct the handling of specific flows of application traffic, for example, a video stream that should receive the same forwarding treatment by all network elements. Thus, SDN allows network administrators and external applications to have programmable and centralized control of all types of traffic flows within a network (Figure 45).

Since packet-mode services such as IP, Ethernet. etc. are inherently connectionless (i.e. routing decisions are done “on the fly” packet by packet by the individual network elements), software-defined networking was originally used to centralize decisions

about packet routing and forwarding, and thus enable more flexibility in how services are controlled. One useful application of SDN control is to create packet-oriented services with some of the characteristics of connection-oriented16 services. Centralized routing control with SDN enables more efficient traffic engineering of connectionless data flows, of great value, for example, within data centers with a large number of nodes and where network topology seldom changes. Centralized control also allows for the programmability of routing decisions, thereby enabling the automation of tasks otherwise performed manually from the management system.

The concept of SDN is applicable not only to Layer 2 and Layer 3 packet networks, but also much more generally to all types of communications networks, including the photonic layer (Layer 0) and optical switching layer (Layer 1) of wide area transport networks. When SDN is applied to multiple layers in a wide area network, it is often referred to as multi-layer SDN or Transport SDN.17

Related to the centralization of routing control is the problem of finding the best paths for traffic flows in the SDN-controlled infrastructure. In a dynamic control plane architecture for connection-oriented networks, this calculation is done by some type of path computation function, which uses its knowledge of network status and topology, available resources and applicable constraints and policies to identify a route for a given request. As networks grow in complexity with several interdependent OSI layers and network domains, the task of path computation becomes increasingly challenging and resource-demanding.

A path computation element (PCE) is a software entity capable of computing a network path or route based on a network graph and applying computational constraints, i.e. capable of performing the path computation function. The main idea behind a separate PCE is to isolate the path computation function and place it into a dedicated high-performance system accessible via an open and well-defined interface and protocol. Path computation can be restricted to a single OSI layer or to a single network domain, but the real

benefits of a PCE are only attainable if it is capable of multi-layer and multi-domain path computations.

In theory, the PCE can be located in the network management system, in an SDN controller or in one or more network elements. However, given the computational resources required for multi-layer path computation, centralizing the PCE to the SDN controller is generally the most cost-efficient solution. When using a multi-layer SDN controller, the PCE must also be multi-layer, meaning that it can

correlate path computations between all the relevant layers. As an example, provisioning a Layer 2 Ethernet Virtual Connection could involve coordinated path computations at Layer 2, Layer 1 and Layer 0.

Closely related to SDN is the concept of network functions virtualization (Figure 46). The idea of NFV is to move network functionality such as intrusion detection, load balancing and network demarcation

Controller

Network equipmentNetwork equipment

Traditional SDN

Control

FIGURE 45: Software-defined Networking. Control of the Traffic Flows Within the Network Is Separated from the Network Equipment and Placed in an SDN Controller

Layer 2 Infrastructure Layer 1 Infrastructure Layer 3 Infrastructure

Virtualized Network

Bandwidth on Demand

NetworkPlanning

Virtual Networking

Data Plane

Control Plane

Application Plane

FIGURE 46: Network Virtualization Is a Central Feature of Software-defined Networks



from specialized devices to software run on common server platforms. Advances in technology have now made it possible to run applications that once required specialized central processing unit (CPU) and interface hardware on industry-standard server blades. Arguments in favor of the transition to NFV are both the lower cost of servers compared to specialized appliances and the rationalization of spares handling – instead of numerous specialized devices, only server blades need to be kept in stock, and the servers are given their functions by the installed software. In short, NFV allows for more dynamic administration of network resources, making it possible for the network operator to respond to new demands in a more agile way.

Software-defined networking increases the flexibility of how the physical network can be used, since network services are created dynamically by the controller and the image it has of the network, rather than by direct manipulation of the physical switches, routers, etc. The SDN controller thus enables network virtualization, i.e. it maps the status of the physical network

resources and network functionality into a single, software-based administrative entity, a virtualized network.

Network virtualization brings a higher degree of abstraction to the transport network, which is favorable when it comes to mobility of resources, service creation and management, especially in the context of cloud computing. The flexibility of

the virtual network matches the dynamic reconfiguration capabilities of computing and storage in cloud computing.

Network virtualization also enables the creation of logically isolated virtual networks over a shared physical network—multiple virtual networks can coexist simultaneously

FIGURE 47: Network Virtualization Allows Two Enterprises to “See” Different Virtual Networks, Although They Use the Same Physical Infrastructure

Physical Packet-Optical Network

Virtual Network 1

Virtual Network 2

Enterprise B

Enterprise A

over the same physical resources. Hence, different service providers using the transport network may “see” different virtual networks, depending on the resources they have been allocated (Figure 47). Furthermore, network virtualization offers the owner of the physical transport network the possibility to delegate more network control to the service providers or enterprises using its transport network.

Software-defined networking requires some method for the control plane to communicate with the data plane equipment making up the physical network infrastructure. OpenFlow, developed by the Open Networking Foundation (ONF),18 was the first such standard communications interface defined between the control and forwarding layers of an SDN architecture. OpenFlow allows direct access to and manipulation of the forwarding plane of network devices such as switches and routers, both physical and virtual (Figure 48).

OpenFlow enables SDN controllers to determine and set the paths of traffic flows across a network of switches. As an open standard, OpenFlow also allows switches from different vendors to be controlled from the same SDN controller. Generally, the OpenFlow protocol is carried within ordinary Transmission Control Protocol (TCP)/IP packets sent between the controller and the network elements.

The OpenFlow standard was developed by ONF, but ONF does not provide any implementation of either switch or controller software. There are, however, quite a few open-source components available, including the source code for OpenFlow itself, which will speed up both the standardization and rate of adaptation of SDN.

As SDN has matured and SDN principles have been applied more broadly to a wide range of networks, additional protocols for control plane-to-data plane communication have been developed to complement OpenFlow. Some examples of other “southbound” protocols from the control layer to the infrastructure are the Open vSwitch Database (OVSDB) management protocol, Representational State Transfer (REST) and NETCONF.

API API API

Business Applications

Network Services

Network Services

APPLICATION LAYER

CONTROL LAYER

INFRASTRUCTURE LAYER

FIGURE 48. OpenFlow Defines the Communication Interface and Protocol Between the Control Plane and the Data Plane. Source: Open Network Foundation



3.3.2 Infinera’s Multi-layer SDN VisionInfinera envisions that future network architectures will be simplified into just two main layers: Layer C, cloud services, and Layer T, intelligent transport. Layer C consists of all virtualized network functions, services and applications that run as software within a cloud environment, including virtualized routing control. Layer T consists of integrated packet-optical networks. SDN plays a central role in this architecture as the bridge between Layer C and Layer T (Figure 49).

SDN principles are applicable to all OSI levels of an Infinera Intelligent Transport Network; hence, Infinera’s SDN vision for the Xceed solution is also a multi-layer SDN vision. This integrated, multi-layer approach to SDN offers several benefits:

• Simplified service provisioning across multiple network layers

• A high level of automation when setting up end-to-end paths in transport networks

• Improved network resource utilization as a result of better topology information

• The programmability of new functions

• Open standards and vendor neutrality

Multi-layer SDN enables several important characteristics of a modern, flexible and cost-effective transport network, such as:

• Multi-layer virtual networks including OSI Layers 0 to 2 (photonic/WDM, OTN, Ethernet and MPLS-TP)

Virtual networks can be defined in a flexible way over physical network resources in one or more network domains. These virtual networks may span multiple network layers and multiple network types, and can be used by one or more customers with complete isolation as needed between customers, protocols employed and OSI layers.

• Automatic provisioning of MEF Carrier Ethernet 2.0 services

All Carrier Ethernet services as defined by MEF CE 2.0 can be provisioned automatically with SDN. Bandwidth can be allocated to services dynamically, with restoration and protection functions centralized.

• Virtual routing – distributed routing functionality

As described in section 3.3.3, an Intelligent Transport Network can employ SDN to implement virtualized IP control plane

instances that interoperate with the IP control plane, for example IP routers, in legacy networks. SDN-enabled Intelligent Transport Networks simplify the creation of software-defined virtual networks, improve geographical service scalability and can reduce routing costs.

Infinera’s multi-layer SDN architecture has standardized, open software interfaces, which make it possible for other higher-level management and orchestration systems, as well as other external SDN-compatible web apps, to interact with the multi-layer SDN control platform. The Xceed multi-layer SDN control platform is logically centralized, and may be physically located in a single central node or distributed to multiple sites in the network. Thus, Xceed enables seamless integration with other software-defined networks and is an excellent foundation for the virtualization of higher-order networks and services.

3.3.3 Applications of Intelligent Packet Transport with SDNWith multi-layer SDN as an element of intelligent transport, new and attractive options for service differentiation and network operations become available to

the network operator. The operator may, for example, adjust the bandwidth of a service dynamically based on customer requests (bandwidth on demand), or optimize the routing of an existing service as the network evolves without dropping a single packet (Figure 50).

Dynamic Layer 2 services with SDN provide a way for the network to reconfigure itself to meet bandwidth peaks and unpredictable

traffic patterns. Dynamic services will also automatically detect changes in network topology and optimize services accordingly.

An SDN application of specific interest from a cost and efficiency standpoint is the introduction of virtual routing, also referred to as distributed routing functionality (DRF),

FIGURE 49: The Infinera SDN Vision: Multi-layer SDN Control of a Multi-layer Packet-Optical Network19

Multi-layer Packet-Optical Network

Packets/Optical Transport

Optical Transport

White Box

Merchant Si

SDN Control

L0-3 Service Mediation

Layer C

Layer T

Multi-layer Control

Routers/Switches

EMXP

Video Source

Video Source

E-Line

EMXP EMXP

CE BCE A

FIGURE 50: Bandwidth on Demand



within the transport network. The objective of DRF is to let virtualized routing engines, implemented as software instances residing in the multi-layer SDN control platform, participate in Layer 3 routing decisions and control the physical network elements to make segments of the transport network behave as if they were IP routers when seen from the outside world of other IP routers. Note that a DRF solution is not intended as a direct replacement for conventional core routers, but can offer significant opportunities for capital expense (CapEx) reduction and a more seamless integration of Layer 3 out to the network edge, and can extend the operational life of existing routers in the metro environment.

Figure 51 shows how a DRF function might work. The DRF (shown in both logical and physical views in Figure 51) “listens” to Layer 3 control plane traffic (e.g. Open Shortest Path First [OSPF], Intermediate System to Intermediate System [IS-IS] and Border Gateway Protocol [BGP] packets) from its “real” router neighbors (peer routers) and passes the control packets to a routing

engine in the SDN controller. The routing engine makes decisions and sends updated routing control packets back toward the external peer routers. The routing engine for the DRF also creates its own routing tables, which are turned into forwarding rules, controlling the packet flows in the physical transport network.

DRF enables significant simplification and cost reduction, especially in aggregation networks, which are common in metro areas. As shown in Figure 52, without a DRF, all IP traffic is forwarded to the central core router, even traffic that could be sent directly between the metro nodes. Adding more base stations with site routers typically also involves manual configuration changes to the transport network. If DRF is enabled, local IP traffic between sites is no longer routed via the central core router. Since the DRF updates its routing tables dynamically, configuration changes at the IP level are detected automatically, without any manual intervention needed in the transport network.

FIGURE 51: A Physical Packet-Optical Network Acting as a Virtual Router

Logical

Physical

EMXP

EMXP

EMXP

CSRCSR

ASRASR

vRouterLink for

protection

FIGURE 52: Virtual Routing in a Mobile Backhaul Network



4 A P P L I C A T I O N S O F

P A C K E T - O P T I C A L

N E T W O R K I N G


4.1 CHAPTER SUMMARYPacket-optical networks that provide Carrier Ethernet services are rapidly becoming a primary infrastructure for telecommunications network operators. The versatility of packet-mode Ethernet services and the capacity and scalability of optical transport have proven to be a winning combination in a variety of applications. In this chapter, we take a brief look at some of the areas in which packet-optical technology has already proven its superiority.

4.2 ETHERNET SERVICES FOR ENTERPRISES – BUSINESS ETHERNET

4.2.1 Serving Enterprise CustomersThe enterprise information and communications technology (ICT) landscape is constantly changing. New IP-based voice and video services, cloud computing, and distributed data centers are adding new requirements to the enterprise wide area data network (WAN), especially in terms of increased bandwidth and enhanced performance. Legacy WAN connectivity technologies such as frame relay (FR) and asynchronous transfer mode (ATM)

are gradually being phased out by many network operators. Wide area connectivity based on TDM leased lines does not provide the operational flexibility expected by the modern enterprise (Figure 53).

As ICT managers look for solutions, Ethernet services such as MEF-defined Carrier Ethernet services have emerged as an

attractive alternative to provide businesses with a best-of-breed, cost-effective wide area networking solution for enterprise ICT applications. Ethernet standards from the MEF, ITU and IEEE have now added features and functionalities that make Ethernet a WAN-capable technology. The potential

of Ethernet services has already been discovered by industries such as finance, healthcare, education, government, IT, retail, real estate, legal, media and more.

An increasing number of network operators are offering MEF-compliant Carrier Ethernet services. In many cases, these services are replacing some of the operators’ legacy technologies (such as FR and ATM), while in other cases they coexist alongside other established wide area networking technologies, such as Layer 3 virtual private network services (also referred to as IP-VPN services).

For the network operator, Ethernet services offer an additional business opportunity, especially if Carrier Ethernet can be implemented in a cost-efficient way on the same equipment already used to provide Layer 1 transport services. Infinera and other packet-optical equipment vendors are actively enabling this important transition.

4.2.2 A Metro Network for Business EthernetFigure 53 shows the principal elements of a typical packet-optical network deployed by a network operator providing Carrier Ethernet services to enterprises in a metro area. Its aggregation and metro core

networks use WDM as the underlying transport technology and are enhanced with Ethernet switching functions to carry Ethernet traffic between the subscribers’ local area networks (LANs) and other data networks as needed.

The integrated packet-optical network enables both transparent Layer 1 services and more advanced Layer 2 (Carrier Ethernet) services. Having the optical network as the base makes it possible to support a broad range of traditional Layer 1 transport services. Also, thanks to the integrated Layer 2 functions, it is possible to offer the enterprise a wide range of Ethernet services with different bit rates, QoS and SLAs, using bandwidth profiles and various protection and monitoring functions. The packet-optical network is ideally suited to support a multitude of enterprise subscribers, each with their own individual requirements for the characteristics of the wide area service they want to buy.

Infinera’s portfolio for metro Ethernet includes both demarcation units (EDUs and NIDs), which can be placed at the customer site (as customer premise equipment or CPE), and packet-optical transport switches (EMXP)

with integrated Layer 2 functions, which are located in the network nodes. All units are fully MEF-certified for Carrier Ethernet 2.0 services and form an integral toolbox for the creation of a state-of-the-art packet-optical network. Furthermore, the EMXPs are seamlessly integrated with other traffic units and optical units in the Infinera XTM Series, forming a true packet-optical network.

The Ethernet Demarcation Unit is designed for minimal delay and jitter, providing unprecedented QoS and SLA fulfillment. The EDU includes highly accurate and precise OAM and performance monitoring through microsecond resolution and per-service visibility for all key OAM and SLA parameters, enabling individual SLA monitoring and service differentiation.

The EMXPs are scalable up to 440G and also designed to have ultra-low latency and jitter. They can be equipped with long-reach interfaces prepared for integration with the OTN transport core, have fast and flexible OAM processing and are fully integrated with the XTM Series optical transport platform.

All XTM Series equipment is managed by DNA-M, a multi-layer management system that includes advanced functions for the seamless provisioning and management of Carrier Ethernet 2.0-compliant services.

Metro Core

ROADM ROADM

EMXP

Aggregation

EMXPEMXP

EMXP EMXP

• Customer Premises Equipment• Ethernet Demarcation Units (EDU)• Network Interface Devices (NID)

• Packet-Optical Transport Switches (EMXP)

• All L2 units seamlessly integrated with L1 units (muxponders, ROADMs, etc.) forming one integrated packet-optical network

• Multi-layer management

Enlighten

EDU

FIGURE 53: A Typical Network Configuration when Providing Carrier Ethernet Services to Enterprises in a Metro Area

A P P L I C A T I O N S


4.2.3 A Long-haul Network for Business EthernetFor operators wanting to offer integrated long-haul and high-capacity Carrier Ethernet services, the Infinera DTN-X Family of super-channel systems presents a very attractive solution. Infinera’s DTN-X platforms perform integrated WDM and OTN switching, grooming and service protection at Layer 1, based on large-scale photonic integrated circuit technology designed for long-haul data transmission. Using the PXM, the DTN-X Family can map Ethernet VLANs to ODUflex subnetwork connections, which

enables ODUflex-based traffic engineering in the long-haul network (Figure 54).

Combining Infinera’s CE 2.0-certified XTM Series for metro Ethernet networks and Infinera’s ODUflex-capable PXM allows for both high-capacity, long-haul Ethernet transport and the termination of Ethernet services not only at regional metro networks, but directly at intermediate DTN-X nodes. The broad Infinera portfolio thus creates a strong basis for providing truly global Carrier Ethernet services with Layer 1 protection in both regional and core network segments.

4.3 AGGREGATION OF IP TRAFFIC – IP BACKHAUL

4.3.1 IP-based Services over a Common InfrastructureFixed and mobile networks provide telecommunications services to end users – services that are generated by the operators themselves and by external entities. The services can be Internet access, telephony with access to the public switched telephone network and consumer access to TV and media streams. Increasingly, all these services are built on the IP suite of protocols, with service provisioning to an end user becoming equivalent to enabling the flow of IP traffic to and from the subscriber.

Much of telecommunications operators’ efforts today are centered on creating seamless services working over both the fixed and mobile networks – fixed-mobile convergence (FMC). One important element of an operator’s FMC strategy is to use a common transport infrastructure of packet-optical equipment for all IP services, irrespective of whether access is fixed or mobile. This simplifies the implementation of common higher-layer service entities and reduces both CapEx and OpEx (Figure 55).

The infrastructure delivering the operator’s services is typically divided into an access network, an aggregation network and a core network.

• The access network must serve millions of endpoints, each requiring a finite amount of capacity, while leveraging the available access medium in an optimal way.

• The aggregation network collects traffic from a large number of end users via hundreds or thousands of access nodes in the access network, smoothing the individual peaks and troughs of traffic into a smoother average traffic flow. Packet-optical technologies can bring significant benefits to this part of the FMC common infrastructure.

• The core network routes and distributes traffic between the aggregation networks and a finite number of entities implementing the higher layers of the services, i.e. data centers, servers and other networks.

4.3.2 A Lean and Transport-centric Aggregation NetworkThe role of the aggregation network is to transport IP traffic from a large number

of access nodes to many fewer core nodes. Traditionally, this task has been performed by SDH/SONET links and WDM wavelengths, but a more capacity-efficient aggregation network is accomplished when Ethernet services are used to aggregate multiple traffic streams. Ethernet leverages the inherent possibilities for the statistical multiplexing of much of the data traffic and

fills the available bandwidth pipes more efficiently. That is why Infinera recommends the use of native Ethernet and MPLS-TP in the aggregation network.

FIGURE 54: Providing Long-haul Carrier Ethernet Services Using the Infinera XTM Series and DTN-X Series with PXM

Region 2

EMXP

EMXP

EMXP

EMXP

EMXP

EMXP

PXM

Region 1ODU�ex-based Long-haul

of Ethernet VLANs

End-to-end Ethernet Virtual Connection

Aggregation

Hub

EMXP

EMXP

EMXP

EMXPEDU

FIGURE 55: Fixed-Mobile Convergence Uses a Common Transport Infrastructure for IP Traffic from Mobile Users and Users in the Home and Enterprise.

Mobile network

Fixed network

Common transport infrastucture

The internet, cloud services

and more...

Access node

Mobile services

Fixed services

Home and business users

Mobile users



There is no need for advanced routing and filtering functions in the aggregation network (Figure 56). Instead, a carefully chosen combination of Ethernet switching and optical functions should make the aggregation network act as a bundle of

wires carrying the traffic. The handling of the IP protocol, including IP/MPLS, should be restricted to as few nodes as possible, i.e. to the core nodes and as needed in the access nodes. Such a consolidation of IP/MPLS reduces both cost and operational complexity. Furthermore, the simpler processing of Ethernet frames compared to IP packets in the nodes of the aggregation

network reduces delay and jitter, while making all traffic flows more predictable than in an IP network.

An aggregation network implemented with the Infinera XTM Series is optimized for transport and does not introduce unnecessary complexity or extra IP functions. It is designed, in particular, for Ethernet transport (Layer 2) according to the MEF CE 2.0 specification, providing:

• High bandwidth utilization

• The low latency necessary for IPTV, IP telephony, video on demand and mobile backhaul

• Support for both point-to-point and multicast applications

• Layer 2 performance management (utilization, latency, jitter, packet loss)

• Efficient synchronization (full SyncE support)

• Excellent protection and resilience LAG, Ethernet Ring Protection Switching Version 2 (ERPSv2), MPLS-TP linear protection

• Predictable performance, in order to be easy to maintain and troubleshoot

• Low power consumption

4.3.3 The Flexible Optical Network Brings Scalability

Given the rapid demand for more bandwidth from consumers and enterprises, the aggregation network requires flexibility to accommodate future growth. Infinera’s solution seamlessly integrates the capacity of an optical WDM network with the Layer 2 switching functions of a Carrier Ethernet network. It is easy to upgrade the capacity of the transmission links between the nodes and the switching capacity as needed according to a pay-as-you-grow model.

For example, assume that the demand for bandwidth has been growing especially fast at one particular remote node in the aggregation network. Thanks to the packet-optical integration of the XTM Series, an additional express wavelength can easily be opened up for the high-traffic node, leading its Ethernet traffic directly to the core node without loading any intermediate switches (Figure 57). This has several benefits:

• Only the high-traffic node and the core node need to be upgraded. There is no requirement for a forklift upgrade of the whole aggregation network—pay as you grow.

• A dedicated wavelength for the additional traffic means minimal latency.

• The express wavelength can be equipped with error correction (OTN-FEC) to cater to the longer distance.

• Protection schemes can be implemented in the optical layer to improve resilience.

Thanks to the integrated packet-optical functionality of the XTM Series, it is also possible to complement the EMXP with other types of muxponders, should there be

EMXP

Core IP/MPLS network

Aggregation networkAccess network

Thousands of locationsMillions of locations Tens of locations

Access node

TCP/P

Ethernet

Optical layer

Ethernet

Optical layer

TCP/P

Ethernet

Optical layer

FIGURE 56: Aggregation of IP Traffic over a Packet-Optical Infrastructure. The Aggregation Network Can Be Made Less Complex and Given Better Performance if the IP Functionality Is Restricted to the End Nodes, i.e. the Access Node Routers of the Core Network

EMXP

EMXP

EMXP

EMXP

10 GbEover one

10 GbEover another

IP Router

Mobile SGW

FIGURE 57: An Express Wavelength Is Added to the Aggregation Networks to Cater to Increased Bandwidth Demand from a Remote Node

a need to transport legacy TDM traffic over yet another wavelength toward the core node, for example.

4.4 MOBILE BACKHAUL4.4.1 4G/LTE and 5G Place New Requirements on Mobile BackhaulMobile networks have been undergoing a rapid evolution over the last few decades that shows no sign of abating anytime soon. Modern LTE/4G and LTE-Advanced



networks place tough demands on their mobile transport networks, and the step change in network performance that 5G brings will add to the demands on the underlying transport network.

The growth in data rates and higher number of cells place an ever-increasing demand for more capacity and reach on mobile backhaul networks, whether they are implemented by microwave links,

copper wires or fiber. A primary challenge for the mobile operator is to dimension the backhaul network to cope with upcoming traffic demand, avoiding a bottleneck for mobile services as well as end-user frustration over long response times and unpredictable performance.

4G/LTE introduced new requirements on synchronization and maintenance in the network, and mobile operators require transport services with more stringent service level agreements. For a regional utility carrier or local operator sitting on large fiber assets, this constitutes an important business opportunity if a network that allows the introduction of such services can be built in a cost-effective manner.

4.4.2 A Backhaul Network Optimized for 4G/LTE and 5GThe most attractive offering for a mobile operator running 4G/LTE and ultimately 5G services builds on established standards and is optimized for packet-mode transport (Figure 58). Using Ethernet as the packet switching technology makes it possible to benefit from the cost rationalization of Ethernet equipment, create tailor-made

backhaul offerings and make maximum use of installed fiber.

Implementing Ethernet services, rather than IP/MPLS-based services with similar characteristics, can make the necessary investments significantly lower, especially if the Ethernet services can be implemented on a previously deployed WDM platform, such as the XTM Series. The Ethernet products in Infinera’s XTM Series are fully MEF CE 2.0-certified and have functionality that makes it possible to implement all the CE 2.0 service types shown in Figure 58.

Furthermore, a backhaul network implemented with the XTM Series has the following characteristics of immediate importance for mobile operators:

• Very little latency and jitter. Latency and jitter have a cumulative effect as traffic passes through consecutive nodes in access rings, aggregation networks and the core. Latency must be low, predictable and stable, not varying with load, throughput or packet replication.

• Efficient mechanisms for synchronization. Synchronization is critical in mobile backhaul networks as users move from cell site to cell site and expect uninterrupted service. Proper performance of the packet-optical

network is ensured through technologies such as SyncE, IEEE 1588v2 and a low latency design philosophy.

• Resilience. The network must include protection mechanisms that guarantee carrier-class resilience against outages. An XTM Series network supports protection mechanisms such as link aggregation, ERPSv2 and MPLS-TP linear protection to ensure a high degree of reliability.

• Efficient fronthaul integration. The flexible structure of the XTM Series allows for efficient integration with the Layer 1 WDM elements necessary for a cloud-radio access network (C-RAN) architecture and supporting mobile fronthaul configurations.

• Efficient tools to manage operational complexity. To be successful in the long run, it is also necessary to control and operate the network in an efficient way. Infinera’s DNA multi-layer management suite has service-aware and integrated Layer 1 and Layer 2 management functions that make the network easy to operate. Service creation can otherwise be difficult for complex networks, but with DNA, setting up integrated Layer 1 services, MEF-based Layer 2 services and MPLS-TP services is easy. The DNA-M Portal even allows mobile operators to monitor the performance of subscriber services in real time.

4.5 SWITCHED VIDEO TRANSPORT4.5.1 Streaming 3D and HD Video to the HomeCable television (CATV) operators/multiple-system operators(MSO)20 offer a wide range of services over the transport infrastructure once built for TV distribution. These services encompass both entertainment and communications for residential users as well as connectivity and value-added services to enterprises. Some services, like LAN interconnect, may be created fully in-house by the CATV operator/MSO, while others rely on external networks, data centers and media hubs.

A major trend in CATV networks is the rapid growth of traffic per end user and in the overall network. The advent of streaming high-definition and 3D video services and cloud computing has made IP traffic grow some 30 to 50% per year. Network operators also feel the need to radically lower the cost of their transport networks to enable them to be profitable in the future, and that requires reducing costs without compromising on scalability, efficiency, manageability or the ability to offer differentiated services.

To meet these demands, network providers are going through a conversion from

Metro / Aggregation

Metro / Aggregation

Service GW/ MME

eNB eNB

eNB

UNI

UNI UNI UNI

UNI

UNINNI NNI

E-LAN / E-TreeE-Access E-Line

FIGURE 58: Carrier Ethernet 2.0 Services for Mobile Backhaul



analog to digital distribution technologies and packet-optical networks. The industry consensus is that packet-optical networks will meet future needs, with the

convergence of legacy and next-generation services onto Ethernet (Figure 59).

Essential for the transformation of CATV networks today is the introduction of DOCSIS 3.0- compatible21 equipment and the development of a new, super-dense,

power- and space-saving access node architecture. The new architecture, referred to as the Converged Cable Access Platform (CCAP), combines edge quadrature amplitude modulation (QAM) and cable modem termination system (CMTS) into one node. Such upcoming super nodes will require significant amounts of bandwidth in the aggregation/distribution network; whereas traditional CMTS and edge QAM (EQAM) nodes could be served with 1 Gb/s, capacities of 10 to 100 Gb/s per CCAP will be required.

4.5.2 Infinera’s Solution for Switched Video TransportInfinera’s Switched Video Transport solution integrates selected Layer 2 functionality into the Layer 1 WDM network, which enables cost-efficient capacity increases and the service differentiation capabilities typically found in Ethernet networks.

The EMXP range allows operators the use of unique features for the demarcation, aggregation and transport of Ethernet services. Enabling the cost-efficient increase of metro network capacity, the EMXPs use

Layer 2 Carrier Ethernet

Converged CableAccess Platform

Aggregation/distribution Core

DOCSIS: 64 Gb/sVideo on Demand: 16 Gb/sBroadcast Digital TV: 2.6 Gb/s

CCAP

PE router forbusiness services

PE router forresidential services

EMXPEMXP

FIGURE 59: New Access Nodes (CCAP) in CATV Networks Require Robust, High-capacity Aggregation Networks

packet-optical technology for the best transport network economics.

Infinera’s Switched Video Transport solution implements Internet Group Management Protocol, Version 3 (IGMPv3) features in the EXMP to listen to the IGMP network traffic between edge QAMs and multicast routers. By participating in the IGMP conversation and switching the individual channels, the Infinera solution is designed to reduce the cost of the network significantly. The IGMPv3 features, IGMP snooping and source-specific multicast, allow network resources to be utilized in an efficient way, as a destination only receives the traffic intended for it as opposed to a broadcast approach where the traffic is distributed to everyone in the network (Figure 60).

Layer 2 Carrier Ethernet

Aggregation/distribution Core

RF Gateway and EQAM

CMTS

PE router forbusiness services

PE router forresidential services

EMXPEMXP

Access

1 Gb/s

1 Gb/s

Hundreds of sites aggregated to 10 ports at 10 Gb/s

Thousands of business users aggregated to 10 ports at 10 Gb/sMulti-layer management

systemBusiness CPE

N x 1 Gb/s10 Gb/s

EnlightenCPE

FIGURE 60: An Overview of the Infinera Switched Video Transport Solution, Based on a Packet-Optical Aggregation Network



5 E T H E R N E T A N D L A Y E R 2

T E C H N O L O G I E S


5.1 CHAPTER SUMMARYPacket switching may come in many shapes, but in the context of packet-optical networks for access and metro/regional applications, packet switching is almost synonymous with Ethernet switching and the use of the Ethernet family of protocols. This chapter focuses on the characteristics of Ethernet, and especially on how the original connectionless LAN protocols for Ethernet have been augmented to make Ethernet a connection-oriented technology suitable for use in wide area transport networks, i.e. to make it into a Carrier Ethernet network. Attention is also paid to how to handle synchronization and how to create resilience in Ethernet networks.

This chapter is primarily intended as a tutorial and source reference for those interested in the general Ethernet and Layer 2 technologies used for wide area networking.

5.2 Ethernet BasicsEthernet is a family of protocols and networking technologies that was originally designed for local area networks in the 1980s but is now also widely used for other

topologies and distances. Standardized by IEEE in the IEEE 802.n family of standards, Ethernet has largely replaced competing wired LAN technologies and is today the dominating link layer protocol in data networks.

Ethernet can be used in bus, star and mesh topologies and over a variety of physical media, including coaxial cable, twisted pair copper cable, wireless media, and optical fiber. Typical Ethernet data rates today range from 100 Mb/s (Fast Ethernet), 1 Gb/s (Gigabit Ethernet or GbE), 10 Gb/s (10 GbE) and on up to 100 Gb/s, with standards for Terabit Ethernet being developed. The term Terabit Ethernet is used to collectively describe all rates above 100 Gb/s Ethernet. To date, this includes 200 Gb/s and 400 Gb/s standardization, with future plans for a full 1 Tb/s Ethernet capability.

5.2.1 Ethernet Mode of OperationSystems communicating over Ethernet divide a stream of data into individual packets called Ethernet frames. Each frame contains a source and destination address and an error-checking code so that damaged data can be detected and retransmitted (Figure 61).

A basic principle of the original Ethernet standard is that destination and source addresses refer to unique physical ports attached to the transmission medium, the MAC addresses, which are permanently written into the hardware of every Ethernet network interface card (NIC). The Ethernet protocol is concerned with the addressing, error checking and transport of frames across a physical transmission link and is referred to as a link layer or Layer 2 protocol.23

In shared Ethernet, the earliest mode of Ethernet operation, frames were broadcasted to every possible receiver in a broadcast domain, and a mechanism called carrier sense multiple access with collision detection (CSMA/CD) was used to avoid collisions on the transmission medium.24

In the 1990s, the IEEE 802.3x standards for Ethernet defined full-duplex operation between a pair of Ethernet stations, i.e. simultaneous transmission and reception of frames over a twisted copper pair or fiber pair. At the same time, a flow control mechanism called the MAC control protocol was introduced. If traffic gets too heavy, the control protocol can pause the flow of frames for a brief time period.

Today, practically all Ethernet LANs and every WAN are based on switched Ethernet. In switched Ethernet, all Ethernet stations have their own individual full-duplex connection to a central switch (sometimes called a multiport bridge). The switch has a forwarding table that matches Ethernet stations’ MAC addresses with a corresponding switch port and sends the frame to the correct destination.

Switched Ethernet removes the media contention and capacity problem inherent to shared Ethernet and reduces it to a

contention problem within the switch, which needs to buffer frames from multiple users trying to access the same destination simultaneously. Another advantage of switched Ethernet is that switches can be made non-blocking, allowing for simultaneous traffic between several ports. Furthermore, each switch port now provides the full bandwidth of the Ethernet medium to the connected station.

Basic Ethernet switches do not modify the Ethernet frames as they pass through and are generally much simpler than Layer 3 IP routers because they operate at the link layer and do not run complex routing protocols. Switches may also be both cheaper and faster than IP routers because

Destination

6 Bytes 6 Bytes 2 Bytes 4 Bytes

Source Data

46–1,500 Bytes

(Jumbo frame 1,500 Bytes – 10 kBytes)

Length/Type

FCS

FIGURE 61: The Basic Ethernet Frame (the Frame Check Sequence Is Used for Error Control)22

VPN1

VPN3

VPN3

VPN1VPN2

VPN2

Distributionnetwork

Core network

FIGURE 62: A Meshed and Switched Ethernet

E T H E R N E T A N D L A Y E R 2 T E C H N O L O G I E S


the switching function can be implemented entirely in hardware, rather than in software running on an expensive high-performance processor. Finally, switches are simpler to manage than IP routers, since configuration does not involve the same complexity as with routers.

In a meshed Ethernet, shown in Figure 62 on the previous page, there are several paths between nodes, and frames can

be forwarded in infinite loops within the network if no countermeasures are taken. There must be one and only one open route

between each node of the network, and all other interconnecting ports on the switches must be blocked. Such a “one-route” network topology is called a spanning tree. The Spanning Tree Protocol (STP) and the Rapid Spanning Tree Protocol (RSTP) in the Ethernet standards are distributed algorithms that can be run by the switches to form a spanning tree (Figure 63).

In wide area applications of Ethernet, e.g. Carrier Ethernet, other mechanisms, such as Ethernet Ring Protection Switching and manual configuration of virtual connections, are used to ensure that there is only one path between the ingress and egress nodes of the network. More information about this can be found in section 5.4.

5.2.2 Virtual LANsThe task of the Ethernet switch is to move the frame from one LAN segment to another based on the destination MAC

address. If the address is unknown, the frame is flooded to all the switch ports except the incoming one. This creates one single broadcast domain per switched network, which is a potential problem in larger networks since broadcast frames are propagated and replicated throughout the entire network. The problem of large broadcast domains as well as the security problem of having all traffic available at every Ethernet station is overcome by the introduction of virtual LANs.

A VLAN is a logical group of Ethernet stations that appear to one another as if they were on the same physical LAN segment, even though they may be spread across a large network. Each Ethernet

station on a particular VLAN will only hear broadcast traffic from the other members of the same VLAN. Using MAC address-based VLANs makes it possible to let the VLAN span multiple switches. Interconnection between VLANs may then be provided by Layer 3 devices such as IP routers.

All Ethernet frames in a VLAN have a distinct identifier, called the VLAN identifier (VID), located in a designated VLAN tag field (Figure 64), specified by the IEEE 802.1Q/p standard and inserted in the frame by the Ethernet switch. The full VLAN tag field is 4 bytes long and contains a tag protocol identifier (TPID) and a priority code point (PCP), which indicate the frame priority level. 12 bits of the VLAN tag are available for VLAN identification, but two values are reserved, making a maximum of 4094 VLANs possible in a single switched network using the basic standard.

VLANs can be used to implement virtual private networks, and VLAN frames include

FIGURE 63: Three Virtual LANs (Green, Blue and Red), Each with Its Own Individual Broadcast Domains, Interconnected by a Router

Destinationaddress

Sourceaddress

Length/type FCSDataVLAN

tag

6 bytes 6 bytes 4 bytes 4 bytes2 bytes

FIGURE 64: IEEE 802.1Q/p Encapsulation of the VLAN Tag

Service Provider

Multipoint

Point to point

SVID 1

MultipointSVID 4094

SVID 2

FIGURE 65: Customers Running Their Own VLANs Inside the Service VLANs of a Service Provider

Destinationaddress

Sourceaddress

CVLAN type FCSDataSVLAN

tag

6 bytes 6 bytes 4 bytes 4 bytes4 bytes

Length type

2 bytes

FIGURE 66: IEEE 802.1ad Encapsulation of the SVLAN Tag



priority fields that may be used to create services with different priorities, i.e. qualities. This feature is of particular interest in wide area applications where VLANs can be used to separate and classify traffic from different sources and users, and to direct it along different paths of the wide area network.

A further development along the same theme is the use of VLANs as “virtual trunks” in service provider networks, allowing customers to run their own customer VLANs inside the service VLANs of the service provider (Figure 65, previous page).

To allow for service VLANs, a 4-byte SVLAN tag is added to the header of the Ethernet frame in the IEEE 802.1ad standard. In the service provider domain, switching can then be based on the SVLAN tag and destination MAC address, and the customer VLAN (CVLAN) tag is used to create virtual LANs within the customer domain. This technology is also known as stacked VLANs, Q-in-Q25 or provider bridges (Figure 66, previous page).

Q-in-Q is primarily a way to overcome the limitations on the VLAN identifier space. It can also help in separating the customer and provider control domains when used with other features like control protocol

tunneling or Per-VLAN Spanning Tree. A further refinement of the concept of service VLAN is standardized in IEEE 802.1ah specifying Provider Backbone Bridges (PBB). The PBB standard supports complete isolation of customer VLAN addressing fields and service VLAN addressing fields.

5.2.3 Ethernet Physical Media (PHY)The Ethernet standard comprises a data link layer and an Ethernet physical media (PHY) part, the latter being specific for the transmission media and data rate employed. When Ethernet is transported over a WDM wide area network, the Gigabit Ethernet and 10 Gigabit Ethernet PHY standards are of the most interest since these are the types of Ethernet deployed in metropolitan and other wide area networks. The Infinera product portfolio includes transponders and muxponders that can be equipped with transceivers supporting Fast Ethernet, Gigabit Ethernet, 10 Gigabit Ethernet and 100 Gigabit Ethernet.

5.2.3.1 Fast Ethernet (FE) Physical LayerFast Ethernet, i.e. Ethernet at 100 Mb/s, has two predominant physical formats: 100BASE-TX, which runs over two unshielded twisted pairs (UTP) made of copper wires, and 100BASE-FX, which runs

over optical fiber. 100BASE-FX uses 1300 nanometer (nm) light transmitted via two strands of optical fiber, one for receive and the other for transmit. The maximum length is 2 kilometers (km) over multimode optical fiber.

The 100 in the media type designation refers to the transmission speed of 100 Mb/s. The “BASE” refers to baseband signaling, which means that only Ethernet signals are carried on the medium. The TX and FX refer to the physical medium that carries the signal.

XTM Series traffic units support both electrical 100BASE-TX and optical 100BASE-FX client interfaces.

5.2.3.2 Gigabit Ethernet Physical LayerGbE can be transmitted over shielded fiber cables or shielded copper cables. It can also be transmitted over unshielded twisted pairs of copper. This transmission is set up to operate in full-duplex (most common) or half-duplex mode. The standard defines a physical media dependent (PMD) sublayer that specifies the transceiver for the physical medium in use. There are three types of PMDs for GbE: short-range, long-range and shielded copper. The short-range PMD

uses 850 nm light with a reach of 220 to 250 meters (m) on multi-mode fiber. The long-range PMD uses 1310 nm light with a reach of 550 m on multi-mode fiber and 5 km on single-mode fiber. The PMD for shielded copper reaches only 25 m. For unshielded copper, which is common in many office installations, multiple twisted pairs are used to send multi-level signals in a way that extends the reach to 100 m. Ethernet First Mile (EFM), an initiative for Ethernet in broadband access networks, later added 1000BASE-LX10 and -BX10 (Figure 67).

The XTM Series traffic units support both the electrical (1000BASE-T) and optical (1000BASE-L) variants for single- and multi-mode fiber in the GbE physical layer when interfacing to client systems.

5.2.3.3 10 Gigabit Ethernet Physical Layer10GbE can be transmitted over fiber optic and copper cables, but copper cables are only used over short distances such as interconnections within a chassis.

For fiber optic cables, the physical layer can be implemented in two main variants: LAN PHY and WAN PHY, optimized for use in local and wide area networks respectively.

FIGURE 67: Gigabit Ethernet Physical Media

Both LAN PHY and WAN PHY operate over a short-range, long-range, extended-range or long-reach PMD. Short-range PMD uses 850 nm light over multi-mode fibers up to 300 m; long-range uses 1310 nm light and reaches 260 m on multi-mode and 10 km on single-mode fiber. The extended-reach PMD uses 1550 nm light and has a maximum reach of 40 km on single-mode fiber. 10GbE is supported across the Infinera packet-optical portfolio.

5.2.3.4 40 and 100 Gigabit Ethernet Physical LayerThe 40 and 100 Gigabit Ethernet standards encompass a number of different Ethernet physical layer specifications. A networking device may support different PHY types by means of pluggable optical modules. The modules are not standardized by any official standards body but are defined by multi-source agreements (MSAs). One such

Name Medium Specific Distance

1000BASE-CX Twinaxial cabling 25 meters

1000BASE-SX Multi-mode fiber 220 to 550 meters dependent on fiber diameter and bandwidth

1000BASE-LX Multi-mode fiber 550 meters

1000BASE-LX Single-mode fiber 5 km

1000BASE-LX10 Single-mode fiber using 1310 nm wavelength 10 km

1000BASE-ZX Single-mode fiber at 1,550 nm wavelength ~70 km

1000BASE-BX10 Single-mode fiber, over single-strand fiber: 1,490 nm downstream/1,310 nm upstream

10 km

1000BASE-T Twisted-pair cabling (Cat-5, Cat-5e, Cat-6 or Cat-7) 100 meters

1000BASE-TX Twisted-pair cabling (Cat-6 or Cat-7) 100 meters



agreement that supports 40 and 100GbE is the C form-factor pluggable (CFP) agreement, which was adopted for distances of 100 meters or more. Quad small form-factor pluggable (QSFP) and CXP connector modules support shorter distances. 100GbE is supported across the Infinera packet-optical portfolio.

The 40/100GbE standard supports only full-duplex operation (Figure 68).

5.3 SYNCHRONIZATION AND CIRCUIT EMULATION SERVICES OVER ETHERNET

5.3.1 Synchronous and Asynchronous TransportPacket switching technologies, including Ethernet, are inherently asynchronous, i.e. incoming frames are received at one data

rate (one rate of bits per second), buffered and multiplexed with other frames over intermediate links with higher data rates, and delivered at yet another rate to the receiver. There is no fixed relationship between the timing, phase or frequency of the incoming bit stream and the outgoing bit stream from the network.

This is quite different from the principles of time-division multiplexed transmission technologies, such as PDH, SDH and SONET, traditionally used in wide area transport networks. In TDM, each stream of information to be transferred over the network is allocated a specific timeslot in the transmission system, a procedure that requires careful frequency and phase synchronization of all intermediate network nodes handling the flow of passing bits.

Today, services such as circuit-switched telephony and storage area networks in data centers are still based on TDM technologies, but increasingly this TDM traffic needs to be transported over packet-optical networks. It becomes necessary to emulate a traditional wireline circuit over an Ethernet network and to maintain synchronization between the ingress and egress ports of the Ethernet wide area network.

Somewhere in every TDM network there is an extremely accurate frequency source, a primary reference clock (PRC) from which all other TDM clocks in the network directly or indirectly derive their timing, i.e. frequency. Clocks derived in this manner are said to be traceable to a PRC. The primary clock signal is distributed “downward” through the network in order to synchronize all other devices, which are normally grouped into separate stratum clock levels depending on how “far” from the original PRC the device is located (Figure 69).

The migration from such a synchronous TDM network to Ethernet-based asynchronous transport introduces new challenges. When a packet network is to support TDM-based services, it must provide correct timing at the traffic interfaces. The transport of TDM signals through Ethernet requires that the signals at the output of the packet network comply with the TDM timing requirements in order for the attached TDM equipment to interwork correctly. Such an adaptation of TDM signals to be transported by a packet network is called a circuit emulation, and the entity performing the adaptation is referred to as an interworking function (IWF) (Figure 70, next page).

Circuit emulation is closely related to the concept of pseudowires. A pseudowire is an emulation of a point-to-point connection over a packet-switching network. The pseudowire emulates the operation of a “transparent wire” carrying the service, but it must be realized that this emulation rarely

will be perfect. The service being carried by the pseudowire may be SDH, SONET, ATM or frame relay, while the underlying packet network may be a Layer 2 network such as Ethernet, an IP network or an MPLS network.

FIGURE 68: 40 and 100 GbE Physical Media

Stratum 2 Stratum 2

4 4

~

Stratum 3

Stratum 3 Stratum 3

Stratum 3 Stratum 3

Stratum 1: International gateway

Stratum 2: Transit exchanges

Stratum 3: Local exchanges

Stratum 4: PABXs

Primary Reference Clock (PRC)

FIGURE 69: Stratum Clock Levels in a TDM Network for Circuit-switched Telephony. Clock Signals May Be Distributed over Several Paths to Ensure Redundancy

Physical Layer 40 Gigabit Ethernet 100 Gigabit Ethernet

Backplane 100GBASE-KP4

Improved Backplane 40GBASE-KP4 100GBASE-KP4

7 m over twinax copper cable 40GBASE-CR4 100GBASE-CR10

30 m over “Cat-8” twisted pair 40GBASE-T 100GBASE-CR4

100 m over OM3 MMF 100GBASE-SR10

125 m over OM4 MMF 10GBASE-SR4 100GBASE-SR4

2 km over SMF, serial 40GBASE-FR 100GBASE-CWDM4

10 km over SMF 40GBASE-LR4 100GBASE-LR4

40 km over SMF 40GBASE-ER4 100GBASE-ER4



A pseudowire encapsulates incoming cells, bit streams and protocol data units (PDUs) and transports them through tunnels set up in the packet network.

In addition to the synchronization of frequencies required by TDM networks, other types of network are dependent on having the exact same time in every node. Frequency is a relative entity, measured relative to a frequency standard, e.g. the PRC, with respect to jitter, wander and slip. Time represents an absolute, monotonically increasing value, generally traceable back to the rotation of the earth (day, hour, minute, second). Mechanisms to distribute absolute time in a network are significantly different

from what is used to distribute frequency, and they normally rely on time stamps that are sent between nodes. Time stamps may also in some network applications be used

for the indirect generation of frequency (differential timing).

The most advanced requirements on synchronization in today’s metro networks are generated by mobile backhaul traffic, which is dependent on both frequency, phase and time synchronization. For example, for 3GPP2 base stations, including those for LTE, the following requirements are typical:

• A frequency accuracy of 0.05 parts per million (ppm) at the air interface

• 2.5 microseconds (µs) time accuracy between neighboring base stations, i.e. ± 1.25 µs difference to coordinated universal time (UTC)

5.3.2 Synchronization StandardsTwo main principles and related standards are available for providing synchronization across an Ethernet network, as shown in Figure 71:

• Synchronous Ethernet. An ITU-T standard using Ethernet PHY media to distribute timing (frequency). SyncE uses PHY clock transmissions and regenerates the clock signal from the incoming bit stream in a way similar to traditional TDM systems, by use of phase-locked loop (PLL) circuitry. This approach requires SyncE support in every network node traversed by the Ethernet signal and only provides frequency and phase synchronization, not time of day synchronization.

• Precision Time Protocol (PTPv2) – IEEE 1588v2. An IEEE standard using Layer 2 embedded OAM packets with the highest priority to ship clock/phase and time of day information across a packet network. Special hardware is often used to process these packets for higher accuracy, but the packets remain in standard Ethernet frames.

5.3.2.1 Synchronous EthernetWith Synchronous Ethernet, a master/slave

architecture at the physical level is used to provide timing distribution from the backbone to the access nodes. A reference timing signal traceable to a PRC is injected into a backbone Ethernet switch, and this signal is then distributed to the next node, which extracts timing from the incoming bit stream and synchronizes the outgoing bit stream to this rate. Timing for TDM circuits transported over the packet network can be recovered at the appropriate interworking functions for the circuit emulation service.

The EMXPs used as nodes in an Infinera packet-optical network fully support the ITU-T Synchronous Ethernet recommendations (G.8262/Y.1362 and others) for jitter and wander tolerances, supported frequencies, and clock specifications for Synchronous Ethernet equipment clocks (EEC). The EMXPs are also compatible with ITU-T Recommendation G.823 regarding clock selection logic, possible clock quality levels, noise tolerances, noise generation and transfer limits, holdover performance, etc.

The EMXPs can do automatic selection of the synchronization source to improve synchronization resilience. Network-level Synchronization Status Messages (SSM) complying with ITU-T Recommendation G.781 are sent between nodes, identifying

the primary synchronization source. If this primary clock fails, the messages can identify a secondary clock that then becomes the primary. The SSM messages are sent over the Ethernet Synchronization Message Channel (ESMC), as described in ITU-T Recommendation G.8264.

In an Infinera packet-optical network, the Synchronous Ethernet function is entirely based on the Ethernet PHY media and its circuitry in the EMXPs. Using either of the Layer 2 traffic forwarding mechanisms (see Chapter 2), MPLS-TP label-switched paths or Ethernet service VLANs, has no effect on timing or the ESMC. A physical Ethernet interface using MPLS-TP will forward Synchronous Ethernet in the same way as a physical Ethernet interface using an Ethernet user network interface (UNI) or network to network interface (NNI) (Figure 72).

TDM circuit TDM circuit

Circuit emulation service

Interworking Function(IWF)

Interworking Function(IWF)

Ethernet

FIGURE 70: Emulation of a TDM Circuit over a Packet-mode Network. The Interworking Functions Provide Traffic Interfaces for the TDM Circuits

~

Time stamp

Time stamp

Packet networke.g. Ethernet

Primary Reference Clock (PRC)Traceable reference frequency

Slave clock

Slave clock

Master clock

IEEE 1588V2

SyncE

FIGURE 71: Standards for Synchronization over Ethernet. SyncE Uses Ethernet PHY to Transfer a Reference Frequency to Every Node. IEEE 1588v2 Uses Ethernet Frames Carrying Time Stamps to Send Current “Time of Day” to the Nodes

Primary clock

Secondaryclock

Secondary Clock

EMXP

EMXP

EMXP

EMXP

ESMC EMXP

EMXP

EDU

FIGURE 72: In the Case of a Primary Clock Failure, Messages in the ESMC Ensure that a Secondary Clock Can Take Over



5.3.2.2 Precision Time Protocol (PTP) – IEEE 1588v2IEEE 1588 (PTP) enables sub-microsecond synchronization of clocks by having a master clock send multicast synchronization packets containing time stamps. All IEEE 1588 receivers correct their local time based on the received time stamp and an estimate of the one-way delay from transmitter to receiver.

PTP is based on IP multicasting and can be used on any network that supports multicasting. Precision is typically in the range of 100 ns to 100 µs, depending on the real-time capabilities of the end systems. The PTP standard can distribute time/phase, frequency or both. It is resilient because a failed network node can be routed around and the time can be taken from more than one master clock. IEEE 1588v2 packets fully comply with Ethernet and IP standards and are backward compatible with all existing Ethernet switching and IP routing equipment. There is no requirement for intermediate Ethernet switches traversed by the emulated circuit to be IEEE 1588v2-aware, as they see the timing frames as normal data.

The convergence time of the PTP protocol, i.e. the time it takes for the protocol to achieve the desired level of synchronization, is dependent on the quality of the underlying packet network. Although the EMXPs are not directly involved in PTP/1588v2 protocol handling, the very low wander, jitter and delay within an EMXP-based packet-optical network makes the PTP protocol converge extremely rapidly.

IEEE 1588 specifies two different mechanisms to maintain synchronization: boundary clock (BC) and transparent clock

(TC). The BC mode of operation relies primarily on the master/slave relationship between network elements. In the TC mode of operation, the network elements add an arrival and departure time stamp to outgoing messages, indicating the delay that has been added by internal processing (Figure 73).

The TC mode of operation, which EMXPs support, has several advantages since it does not require any preconfiguration, is easily interoperable between network elements and handles various types of asymmetries in the network. IEEE 1588 TC can be used for Ethernet VLANs, Ethernet service VLANs and MPLS-TP pseudowires.

The BC mode of operation allows the network to realign the clock and compensate for the wander that builds up in the network.

5.3.2.3 Differential TimingTiming recovery for a TDM circuit emulation service requires that the timing of a signal is similar on both ends of the packet network, i.e. at the “outside” of the IWFs. The clock of the TDM service must be preserved in

such a way that the incoming service clock frequency is replicated as the outgoing service clock frequency. In network-synchronous operation, the packet network operates in a fully synchronized manner using a PRC-traceable clock, but this does not necessarily preserve the timing of the external TDM service. Using differential timing, the difference between the external TDM service clock and the network reference clock is encoded and transmitted across the packet network. Differential timing makes it possible to recover the external TDM service clock at the far end of the packet network.

More information on timing recovery and transport of legacy TDM services such as SDH and SONET over an Infinera packet-optical network can be found in section 2.5 of this book.

5.4 ETHERNET PROTECTIONWide area services, such as telephony, internet access and video on demand, require a high level of availability; unavailability is typically only tolerated for a few minutes per operating year. When failures occur in the network, they are not

supposed to be noticed by the subscriber. The main purpose of an automatic protection switching (APS) mechanism is to guarantee the availability of backup resources and ensure that switchover is achieved within milliseconds.

Protection switching can be implemented at various OSI network layers. An optical transmission network may include alternative fiber routes and protection switching at Layer 1. Ethernet/Layer 2 can perform protection switching as well, and there may also exist protection mechanisms at higher OSI layers. This section deals

with the Ethernet/Layer 2 protection mechanisms.

Ethernet Spanning Tree Protocol and Rapid Spanning Tree Protocol can prevent loops and assure backup paths. However, both protocols are too slow to respond to network failures and are not used in packet-optical networks.26 Instead, other mechanisms, such as Link Aggregation Group (LAG) and Ethernet Ring Protection Switching are more suitable. For Layer 2 networks employing MPLS-TP label-

Output buffer

Output buffer

Corr: + 1231.662356

Corr: + 861.452344

FIGURE 73: IEEE 1588 Transparent Clock (TC) Mode of Operation

MAC Client

MAC

Distributor Collector

MAC

MAC Client

MAC

Distributor Collector

MAC

FIGURE 74: Link Aggregation Functional Diagram


I N F I N E R A86 I N F I N E R A 87

switched paths (see section 2.3), even more advanced protection schemes such as MPLS-TP linear protection are available.

5.4.1 Link Aggregation5.4.1.1 Principles Link aggregation, as defined by IEEE 802.3ad, is a method for aggregating two or more parallel physical transmission links to a LAG, such that a

MAC client can treat the group as if it were a single link. Link aggregation is capable of increasing both the capacity and the availability of the communication channel between devices interconnected by Ethernet. Link aggregation can also provide load balancing, in which traffic is spread across several physical links in order to avoid the overload of any single link (Figure 74, previous page).

Most implementations of LAG now conform to what used to be clause 43 of the IEEE 802.3-2008 Ethernet standard, informally referred to as “802.3ad.” This includes Infinera’s EMXPs, which have been interoperability-tested against several other vendor solutions. With the EMXPs, it is possible to define a link aggregate group consisting of up to eight ports. All ports in the group must have the same port speed.

A dedicated protocol, the Link Aggregation Control Protocol (LACP), is used to authenticate and coordinate the network elements participating in the LAG (Figure 75).

It is important to understand that a Layer 2 network may not change the order of frames

sent between two endpoints (hosts). The LACP protocol will therefore assign each “conversation” between two hosts to one specific physical link and send all frames belonging to that conversation over this link. Dedicated hash algorithms, analyzing the fields of the frame header (VLAN ID, MAC address, MPLS-TP label, etc.), are used to spread the conversations evenly across the available physical links.

5.4.1.2 Multi-chassis LAGMulti-chassis LAG (MC-LAG) refers to the ability to set up LAGs in which one or both endpoints have physical links that end in two or more different chassis, or in the case of the XTM Series, EMXPs. MC-LAG is supported by many vendors, but is not fully standardized, and its implementation varies by vendor. Multi-chassis LAG uses normal LACP signaling toward the connected peer and is “seen” by the connected peer as an ordinary LAG (Figure 76).

State and protection are coordinated between the EMXPs involved, using a special signaling protocol (Inter-Control Center Protocol or ICCP). Protection LAG 1+1 and N+N are supported by the Infinera packet-optical network.

The XTM Series’ multi-chassis LAG brings a number of benefits, such as:

• Increased network availability

• Fast protection switching (< 50 ms) that allows for demanding customer SLAs (compare to IP-level protection < 5 s)

• Network redundancy without complex signaling between nodes

• Few interoperability issues, since each side can see the other just as in standard protection LAG

• Supported by almost all client equipment

5.4.2 Ethernet Ring Protection Switching Ring-based networks are often attractive since they can offer a simple form of redundancy in a network consisting of many nodes. A redundant path for each node is provided by just one additional link that closes two adjacent branches to form a loop.

EMXPEMXP

FIGURE 75: LAG Assigns Different Conversations Between Hosts to Separate Physical Links

CarrierEthernet ICCP

Peer System 2

Peer System 1

LAG

EMXP

EMXP

FIGURE 76: Multi-chassis LAG

However, Ethernet as such does not allow for loops since frames would circulate forever, so the loop has to be blocked at some point in the network. This can be accomplished

by the use of ERPS, as specified in ITU-T Recommendation G.8032 (Figure 77).

An Ethernet ring is made up of two or more nodes interconnected via transmission links.

FIGURE 77: How Ethernet Ring Protection Switching Works. The Solid Line Between Nodes C and D Represents a Ring Protection Link (RPL)

Normal operation

BlockedBlocked

OpenSignaling

Failure Failure

A failure is detected Traf�c �ow is restored

A B

E

F

D

C

A B

E

F

D

C

A B

E

F

D

C



Each node has two links connected to its adjacent nodes. To avoid loops, one of the links in the full ring is always blocked. This link, which is blocked under normal conditions, is called the Ring Protection Link (RPL). One of the nodes connected to the RPL is designated the RPL owner and is responsible for controlling the status of the link. In the example above, the link between node C and node D is the RPL and node C is the RPL owner.

When a failure is detected, the RPL owner is responsible for unblocking the RPL and opening it for traffic. A link failure can be detected by link-down events or OAM frames, for example, a loss of continuity. When a failure has been detected, the ring is said to be in a “signal failure” state. The node detecting the failure condition will block the traffic on the failed link and inform the other nodes in the ring.

When the nodes are informed about a ring failure, the learned MAC addresses are flushed. In the example above, the nodes in the ring will relearn the destination MACs of the stations participating in the blue traffic, which will now flow over the unblocked RPL.

The Ethernet Ring Protection Switching available with EMXP-based nodes provides sub-50 ms protection for Ethernet traffic using ring topology, and ensures that there are no loops formed at the Ethernet layer.

EMXP-based nodes can also support multiple rings, so that several Ethernet protection switching rings can be joined at one physical location. This makes the Infinera packet-optical solution extremely useful for the aggregation of traffic from multiple rings covering many sites, for example, in a mobile backhaul network.

Ethernet Ring Protection Switching Version 2 adds some very useful features. Most important is that it introduces more advanced Ethernet ring interconnection architectures (multi-ring/ladder network), with the concept of sub-rings. This creates the ability to have different rings interconnected at two or more points, avoiding a single point of failure.

ERPSv2 can support multiple ERP instances on a single ring. In addition to the possibility to allocate a specific set of VLANs to a ring instance, there are also a number of new deployment possibilities:

• MPLS-TP backbone VLANs

• Common spans with other rings

• Mixing protected and unprotected VLANs

ERPSv2 supports both revertive and non-revertive operation after the condition that caused the switch has been cleared to minimize unplanned traffic hits. ERPSv2 also includes “manual switch” and “force switch” operator administrative commands.

The XTM Series includes support for both ERPSv1 and ERPSv2.

5.5 CARRIER ETHERNET ARCHITECTURE AND SERVICES

5.5.1 Carrier Ethernet: Ethernet as a Transport ServiceLegacy carrier networks provide transport services with very high availability and predictable performance, using, for example, SDH or SONET multiplexing schemes. Packet-switched transport networks are different, as they offer new services based on features such as asynchronous transport, statistical multiplexing and full connectivity between multiple endpoints. The classical “carrier-grade” characteristics, such as quality of service, security and high availability, must be realized by new mechanisms in a packet network.

Packet-switched wide area networks, however, especially those based on Ethernet, offer other advantages to network operators than those typically found in legacy carrier networks:

• Most potential users of a WAN or metropolitan area network (MAN) service have an Ethernet-based LAN and want to extend it to multiple sites. It makes sense for a carrier to offer Ethernet-type transport services, since customers are familiar with the protocol, and their equipment already has Ethernet interfaces.

• Ethernet is the all-dominant Layer 2 data networking protocol, which means that economies of scale have made Ethernet switching equipment very attractive from a cost perspective. Naturally, carriers want to take advantage of the continued performance and cost evolution of the Ethernet technology.

• Many non-Ethernet wide area technologies such as TDM, ATM, frame relay, etc. can be replaced or emulated by Ethernet alternatives, making a transport network based on Ethernet more uniform and less complex to operate than a network using multiple other technologies.

To transport Ethernet traffic efficiently in metro and regional networks, the carrier/network operator/service provider must establish an Ethernet of its own – a Carrier Ethernet network27– that forwards traffic between the customer LANs “in the Ethernet way.” Using Ethernet as the transport mechanism requires the addition of functions that transform the connectionless and broadcast-oriented

Ethernet for LAN use, into a more predictable and “circuit-like” channel suitable for wide area networking.

The need for such a Carrier Ethernet has been recognized for some time. The Metro Ethernet Forum was formed in 2001 by vendors and service providers to develop standards for ubiquitous LAN interconnect

FIGURE 78: Carrier Ethernet: A Connection-oriented Version of Ethernet Provided as a Service to Subscribing Customers


Subscriber Ethernet

Subscriber Ethernet

Subscriber Ethernet

Ethernet Virtual Connection



services over metropolitan optical networks. The principal concept was to bring the simplicity and cost model of Ethernet to wide area networks, while adding stability, predictability and manageability. Since then, MEF has issued a wide range of technical specifications for Carrier Ethernet equipment, specifications that are adhered to by Infinera (Figure 78, previous page).

Metro Ethernet Forum defines a Carrier Ethernet network as a set of certified network elements that are interconnected and provide Carrier Ethernet services, locally and worldwide.

In this context, it is important to remember that the term “Ethernet” is ambiguous and is standardized in various ways by multiple interest groups.

• Ethernet regarded as a “point-to-point” transmission link, i.e. the physical characteristics of Ethernet framing and transmission. This is the scope and view of IEEE 802.3.

• Ethernet regarded as a packet-switched network infrastructure. This is the 802.1 (bridging) view as well as the managed Ethernet network view by ITU-T SG15/SG13.

• Ethernet regarded as a service. This is the Metro Ethernet Forum’s scope and view. Carrier Ethernet services concern the user-to-user transfer of Ethernet 802.3 frames over any available physical transport layer.

A Carrier Ethernet network is by definition a two-layer structure, consisting of a physical transport layer, which can be WDM, SDH/SONET, Ethernet PHY or any other physical transport technology, and a pure Ethernet frame handling layer, the Ethernet MAC (ETH) layer.28 The Ethernet services offered are created “on top of” transmission technologies such as WDM optical networks. The following discussion focuses on Carrier Ethernet services and the Ethernet MAC layer. Details of how the Ethernet MAC layer is carried by the optical WDM layer of the Infinera XTM Series are described in Chapter 2.2.4, “Packet-Optical Transport with the XTM Series.”

Metro Ethernet Forum has defined five main attributes of Carrier Ethernet that distinguish it from the familiar LAN-oriented Ethernet, and make it suitable as a transport service offered by carriers. The five attributes are:

• Standardized services. Carrier Ethernet provides five standardized service types (E-Line, E-LAN, E-Tree, E-Access and

E-Transit) that enable transparent, private line, virtual private line and multipoint-to-multipoint connectivity over the wide area network. The services are provided independent of the underlying transport protocols and media used, with a high level of choice and granularity of bandwidth as well as quality of service options. The services require no changes to customer LAN equipment or networks and accommodate existing network connectivity, such as time-sensitive TDM traffic and signaling, while being delivered over a single Ethernet connection between network and customer.

• Scalability. The scale of a customer LAN and that of a service provider network are fundamentally different in terms of geographical reach, number of users (endpoints) and bandwidth. Carrier Ethernet is scalable in all those dimensions, while allowing the service provider to use various underlying transport technologies to achieve the best total economy.

• Reliability. Carrier Ethernet is resilient and reliable. Protection mechanisms are available to provide end-to-end and individual link protection. The speed of

recovery from failures is comparable to that of SDH/SONET networks or better.

• Quality of service. Carrier Ethernet supports the delivery of critical applications that are expected to meet high performance levels. The performance parameters of Carrier Ethernet are quantifiable and measurable so that they can be included in an SLA for voice, video and data over converged business and residential networks.

• Service management. Carrier Ethernet service providers are expected to manage large numbers of customers and their multiple services, spanning wide geographical areas. Carrier Ethernet includes advanced capabilities for provisioning, maintaining and upgrading Ethernet services (Figure 79).

5.5.2 Carrier Ethernet Architecture and TerminologyA service provided by a Carrier Ethernet network starts at one user network interface (UNI) and ends at another UNI. The UNI is the point where the service provider

accepts and delivers Ethernet frames, i.e. a dedicated, physical demarcation point between the responsibility of the service provider and that of the subscriber. The attached customer equipment (CE) can be a router, switch or computer system, and the physical medium of the UNI can be copper, coax or fiber operating at 10 Mb/s, 100 Mb/s, 1 Gb/s, 10 Gb/s or 100 Gb/s, according to the IEEE 802.3 Ethernet PHY/MAC protocol.

The UNI functions are divided between the CE and provider edge equipment as the function sets UNI-C and UNI-N, respectively. Sometimes the CE does not support all the UNI-C functions; in such cases a network interface device or an Ethernet demarcation unit belonging to the Carrier Ethernet network is located at the customer site and acts as the actual physical demarcation point of the service. Carrier Ethernet demarcation is a key element in Carrier Ethernet networks as it enables service providers to extend their control over the entire service path, starting and ending at the customer handoff points.

The association between two or more UNIs via the Carrier Ethernet network is referred to as an Ethernet Virtual Connection. In the Carrier Ethernet world, this association is

the equivalent of a “circuit,” and it is the Ethernet Virtual Connection that is assigned the various characteristics – attributes – to which a customer subscribes.

Quality of

Service

Standardized Services

Reliability

Scalability

ServiceManagement

CARRIER ETHERNET

Source: Metro Ethernet Forum

FIGURE 79: Carrier Ethernet Attributes as Defined by Metro Ethernet Forum



Service Provider 1

Carrier Ethernet Network 1

Service Provider 2

Carrier Ethernet Network 2

Customer Equipment

(CE)

Customer Equipment

(CE)

User Network Interface(UNI)

User Network Interface(UNI)

Ethernet Virtual Connection (EVC)

External Network to Network Interface

(ENNI)

Sometimes an Ethernet Virtual Connection passes through the networks of more than one service provider. The interface between two such service providers is referred to as an external network to network interface (ENNI) (Figure 80).

When an Ethernet Virtual Connection spans multiple Carrier Ethernet networks, it is composed of segments in each CE network that are concatenated together to form the EVC. These segments are called Operator Virtual Connections (OVCs). An OVC is the

association of UNIs and ENNIs within a single CE network in which at least one of these external interfaces is an ENNI.

The EVC, i.e. the association between two or more UNIs, performs two basic functions:

• Connects two or more subscriber sites (UNIs), enabling the transfer of Ethernet service frames between them.

• Prevents data transfer between subscriber sites that are not part of the same EVC. This capability enables an EVC to provide data privacy and security similar to frame

relay or ATM permanent virtual circuits (PVC).

The Carrier Ethernet specifications allow for three types of Ethernet Virtual Connections: point-to-point EVC, multipoint-to-multipoint EVC and rooted multipoint (point-to-multipoint) EVC, which are used to create Carrier Ethernet services (Figure 81).

5.5.3 Carrier Ethernet 2.0 ServicesIn the Carrier Ethernet network, data is transported across point-to-point, point-to-multipoint and multipoint-to-multipoint EVCs and OVCs according to the attributes and definitions of a set of well-defined Ethernet service types. Each service type provides transparent data transport between the UNIs and/or ENNIs of subscribers and operators. Three of the service types are based on EVCs (i.e. are for services between UNIs) and two are based on OVCs (i.e. at least one endpoint is an ENNI). EVC-based service types are E-Line, E-LAN and E-Tree, while E-Access and E-Transit are based on OVCs.

A complete MEF Carrier Ethernet service consists of an Ethernet service type associated with one or more bandwidth profiles and supporting one or more

classes of service. A service also defines the transparency of Layer 2 control protocols and how they should be handled.

For Ethernet services based on EVCs, two variants are defined, differentiated by the method of service identification used at the UNIs. Services using port-based UNIs, i.e. where there is only one UNI and EVC per physical port of the provider edge device, are referred to as “private,” while services using UNIs that are VLAN-based and multiplexed over the same physical interface are referred to as “virtual private.” For example, an E-Line service that is port-based is referred to as Ethernet Private Line.

E-Access and E-Transit services, which are based on Operator Virtual Connections and primarily defined for use within a service provider’s network, have been restructured to eliminate the port/VLAN distinction (Figure 82, next page).

The five Carrier Ethernet 2.0 service types target different applications. E-Line services are typically used to replace private TDM lines and frame relay VPNs, as well as for internet access. The other three

service types are used in more specialized applications, such as mobile backhaul and applications in which several Carrier Ethernet service providers cooperate. The E-Transit service addresses the multi-operator wholesale of Carrier Ethernet connectivity.

The Infinera packet-optical portfolio is MEF CE 2.0-certified, including up to 100 Gb/s where relevant, and has the functionality needed to implement all the service types above in a packet-optical network.

Implementing CE 2.0 services gives the network operator a whole new range of services to offer “on top” of the legacy Layer 1 transport service, thereby adding new revenue streams to an existing network investment. Furthermore, implementing Ethernet services, rather than IP/MPLS-based services with similar characteristics,

FIGURE 80: Carrier Ethernet Basic Concepts and Terminology

Root

Leaf

Leaf

Point-to-point EVC: Each EVC associates exactly two UNIs with each other.

Multipoint-to-multipoint EVC:Each EVC associates ≥ two UNIs. In this diagram, three sites joint-share a multipoint EVC and can forward Ethernet frames to each other.

Rooted Multipoint EVC: EachEVC associates ≥ two UNIs with one or more UNIs as roots—the roots can forward to the leaves, while each leaf can only forward to the roots.

FIGURE 81: Ethernet Virtual Connection Types



can make the necessary investments significantly lower, especially if the Ethernet services can be implemented on the previously deployed WDM platform.

5.5.4 Carrier Ethernet Service AttributesA user of Carrier Ethernet subscribes to an Ethernet service type (E-Line, E-LAN, E-Tree, E-Access, E-Transit) with either a port-based

or VLAN-based UNI, and with more detailed characteristics as specified by its Ethernet service attributes, which are listed in MEF specifications 6.1 and 10.2.

The service attribute represents a service characteristic, which in turn is further defined by a set of Ethernet service attribute parameters. The attributes and parameters customize the overall performance and quality of service for each individual service and subscriber (Figure 83).

The Ethernet service attributes are of three main types:

1 Per-EVC service attributes defining the characteristics of the Ethernet Virtual Connection as such, such as:• EVC ID and type; point-to-point or

multipoint• List of connected UNIs• Customer VLAN ID and class of service

preservation• EVC performance: frame delay (latency),

inter-frame delay variation, frame loss ratio and availability

2 EVC-per-UNI service attributes, such as:• UNI/EVC ID• Customer VLAN ID/EVC mapping• Ingress/egress bandwidth profiles per

class of service

Service Type Port-based Service VLAN-aware Service

EV

C S

ervi

ces

Ethernet Private Line

Ethernet Private LAN

Ethernet Private Tree

Ethernet Virtual Private Line

Ethernet Virtual Private LAN

Ethernet Virtual Private Tree

Access EPL Access EVPL

Access E-Line

Access E-LAN

Transit E-Line

Transit E-LAN

OV

C S

ervi

ces

E-LineP2P EVC

E-LANMP2MP EVC

(EPL) (EVPL)

(EP-LAN) (EVP-LAN)

(EP-Tree) (EVP-Tree)

P2P OVC with 1 UNI and 1 ENNI

P2P OVC with 2 ENNIs

MP2MP OVC with ENNIs

MP2MP OVC with UNI(s) and 1 ENNI

E-TreeRMP EVC

E-Access

E-Transit

FIGURE 82: Ethernet Service Types Defined in MEF Carrier Ethernet 2.0 and Their Relation to Ethernet Virtual Connections and Operator Virtual Connections. Source: Metro Ethernet Forum

3 Per-UNI service attributes, such as• UNI ID and physical interface

capabilities: data rate, frame format• Ingress and egress bandwidth profiles• Service multiplexing capability• Layer 2 control protocol processing

5.6 CARRIER ETHERNET TRAFFIC MANAGEMENT

The different applications, users and data flows in a Carrier Ethernet network require different priorities and performance guarantees. The process of differentiating

Ethernet Service

AttributeParameters

Ethernet Service

Attribute

Ethernet Service

Type

FIGURE 83: Each Subscribed-to Ethernet Service Type Has an Associated Set of Attributes and Parameters Defining Its Behavior in Detail

traffic in this way is referred to as traffic management, and involves mechanisms such as queuing, scheduling and policing of Ethernet frames. With traffic management in place, it is possible to guarantee a certain quality of service for a given service with respect to, for example, data rate, delay, jitter and packet dropping probability.

Quality of service guarantees are important if network capacity is insufficient, especially for real-time streaming multimedia applications such as voice over IP and IPTV, since these often require a fixed bit rate and are delay- and loss-sensitive. In the absence of network congestion, QoS mechanisms are in principle not required. However, temporary changes in traffic patterns and reconfigurations, for example

when caused by protection switching, make QoS mechanisms necessary in virtually all networks.

Carrier Ethernet defines several traffic management mechanisms, which are described below.

5.6.1 Bandwidth ProfilesA bandwidth profile is a set of traffic parameters that defines the maximum average bandwidth available for the customer’s traffic. An ingress bandwidth profile limits traffic transmitted into the network and an egress bandwidth profile can be applied anywhere to control overload problems from multiple UNIs sending data to an egress UNI simultaneously. Frames that meet the profile are forwarded; frames that do not meet the profile are dropped.

Bandwidth profiles allow service providers to offer services to users in increments lower than what is set by the physical interface speed. Also, they provide the possibility to engineer the network and make sure that certain parts of the network are not overloaded.



MEF 10.2 specifies three levels of bandwidth profile compliance for each individual service frame:29

• Green: the service frame is subject to service level agreement performance guarantees.

• Yellow: the service frame is not subject to SLA performance guarantees, but will be forwarded on a “best-effort” basis. These frames have lower priority and are discard-eligible in the event of network congestion.

• Red: the service frame is to be discarded at the UNI by the traffic policer.

Bandwidth profiles can be defined per Ethernet Virtual Connection and per class of service and are governed by a set of parameters, the most important being:

• Committed information rate (CIR), which defines the assured bandwidth, expressed as bits per second.

• Excess information rate (EIR), which defines “extra” bandwidth that may be temporarily used, expressed as bits per second.

• Committed burst size (CBS) and excess burst size (EBS), which define temporary bursts of information that can be handled.

The EMXP also supports a simpler bandwidth profile that only uses the traffic parameters:• Rate, expressed as bits per second.• Burst size, expressed as bytes (Figure 84).

Ingress bandwidth profiles can be applied per UNI (to all traffic regardless of VLAN tag or EVC ID), more granularly on an EVC basis, or even based on a class of service marking such as a customer-applied VLAN priority tag (Figure 85).

Compliance to the bandwidth profile is determined through two “leaky-bucket” algorithms, using a principle referred to as two-rate three-color marker (TrTCM).

5.6.2 Class of Service and Service Level Agreements The integration of real-time and non-real-time traffic over Ethernet requires differentiating packets from different applications and providing differentiated performance according to the needs of each application. When a network experiences congestion and delay, some packets must be dropped or delayed. This differentiation is referred to in Carrier Ethernet as class of service.

CoS can be applied at the EVC level (with the same CoS applied to all frames transmitted over the EVC), or applied within the EVC by customer-defined priority values

Total Bandwidth at UNI

EIR traf�c is marked orange—will be dropped at congestion points

EVC1

EVC3

EVC2

EIR CIR

EIR

CIR

EIR CIR

FIGURE 84: Conceptual Example with Three EVCs Sharing the Same UNI. The Three CIRs Can Always Be Met; the Three EIRs Cannot Always Be Met Simultaneously

PORT-BASED

Ingress BandwidthPro�le Per Ingress UNIUNI

UNI

EVC1

EVC2

EVC3

PORT/VLAN-BASED

Ingress BandwidthPro�le Per EVC1



UNI

EVC1

EVC2

EVC3

EVC2

EVC1

CE-VLAN CoS 6

CE-VLAN CoS 4

CE-VLAN CoS 2

Ingress Bandwidth Pro�le Per CoS ID 6



PORT/VLAN/CoS-BASED

FIGURE 85: Three Types of Bandwidth Profiles Are Defined in MEF 10.1

in the data, such as customer VLAN IEEE 802.3 “q” or “p” tag markings.

The class of service settings, together with bandwidth profiles, are used to make service level agreements between the service

provider and its customers. In addition to the various parameters of the bandwidth profiles (CIR, EIR, etc.), the SLA typically also specifies maximum values for various types of frame delay and frame delay variation (jitter), and values for the availability of the subscribed-to service (Figure 86, next page).

5.7 CARRIER ETHERNET OPERATIONS, ADMINISTRATION AND MAINTENANCE (ETHERNET OAM)

Using Ethernet as an end-to-end wide area network service rather than as a link layer protocol creates a need for a new set of operations, administration and maintenance mechanisms and protocols. Service providers must be able to provision and maintain large volumes of Ethernet services and subscribers in a rational and cost-efficient way.

Furthermore, an end-to-end wide area Ethernet service, i.e. an Ethernet Virtual Connection, often involves one or more carriers/network operators providing the underlying transmission capacity in addition to the Ethernet service provider. Carrier Ethernet OAM requires the coordination of OAM performed by a number of administrative entities and by different technical systems.

5.7.1 The Management FrameworkEthernet OAM builds on an established management framework and terminology



FIGURE 86: Examples of Service Level Agreements for Different Applications. Source: Metro Ethernet Forum

Priority

Committed Information

Rate

Excess Information

RateFrame Delay

Frame Loss Ratio

0 10 Mb/sVoIP calls

Interactive business and consumer video programming

Telepresence

Streamed HD live content

Content distributed.Development and non-real time delivery.

0 5ms 0.1%

1 100 Mb/s 0 5ms 0.01%

2 50 Mb/s 0 25ms 0.1%

3 40 Mb/s 0 N/A 0.01%

4 0 500 Mb/s N/A 1%

Application EVPL Profiles, Sample CoS Objectives

Carrier Ethernet Service Provider

Metro Fiber Ethernet Virtual Private Line Services

COMPANYHQ

UNI

using the concept of a data model, a management information base (MIB), describing the status of the individual elements in the managed network (Figure 87).

A management information base is a database representation of the managed objects in a telecommunications network. This database, normally located in the central network management system, keeps an updated view of network element status by sending queries to the elements, and is also used for configuration and provisioning activities. An MIB, together with an associated management protocol

such as Simple Network Management Protocol Version2 (SNMPv2), defines a standard network management interface for the administration and maintenance of a particular network element (Figure 88).

5.7.2 Standards for Ethernet OAMFrom an OAM perspective, there are several standards that work together in a layered fashion to provide Carrier Ethernet OAM. IEEE 802.3ah defines OAM at the link level. With more of an end-to-end focus, 802.1ag defines connectivity fault management for identifying network-level faults, while ITU Y.1731 adds performance management, which enables SLAs to be monitored. The functions of these OAM layers are implemented either in a standalone network demarcation device (i.e. the NID or EDU) or integrated into the node equipment (i.e. into the EMXP, in Infinera’s case) (Figure 89).

5.7.3 The Service LifecycleThe life of an EVC starts with a service order initiated by a customer. The order contains various types of EVC information, such as UNI locations, bandwidth profiles, class of service, etc. After provisioning the EVC, the service provider and involved network

operators conduct initial turn-up testing to verify that the EVC is operational and fulfills the subscribed-to characteristics. While the EVC is in use, all parties involved—the subscriber, the network operators/carriers, and the service provider—want to monitor the same EVC to ensure that it adheres to the specified SLA regarding delay, jitter, loss, throughput, availability, etc. Finally, when the EVC is not needed any longer, the assigned network resources should be freed up and made available to other EVCs.

The described life cycle of an Ethernet service is depicted in Figure 90, next page. The involved processes fall into three main categories: provisioning, performance

management and fault management. The IEEE, ITU-T and MEF OAM standards provide the means to monitor and execute the required actions on the Carrier Ethernet.

Ethernet service provisioning comprises the processes of setting up the required EVCs for a customer and assigning them attributes such as bandwidth profiles and class of service. The provisioning process also includes procedures for testing the service once it is set up, but before it is turned over to the customer. The configuration of the service is checked for correctness and verified against its service acceptance criteria (SAC).

The Carrier Ethernet provisioning process specified by MEF is based on the ITU-T Specification Y.1564.

MIB for

NE1

NetworkManagement

System

Alarms, status messages, etc.

Configuration commandsNetwork element NE1

FIGURE 87: The Network Management System Uses a Data Model – an MIB – to Keep Track of the Status of the Individual Network Elements

MIB

NetworkManagementSystem


Customer Equipment

(CE)

Demarcationunit

Demarcationunit

Customer Equipment

(CE)

Network 1 Network 2

UNI UNIENNI

SNMPV2

FIGURE 88: The Ethernet OAM Framework and Terminology

Customer Equipment

(CE) Networkinterfacedevice

Link OAMIEEE 802.3ah

Link OAMIEEE 802.3ah

Networkinterfacedevice

Customer Equipment

(CE)


Network and connectivity OAMIEEE 802.1ag and ITU Y.1731

Ethernet Service OAMMEF Specifications, IEEE 802.1ag and ITU Y.1731

FIGURE 89: Standards for Carrier Ethernet OAM



5.7.4 Ethernet Service OAM – Performance and Fault ManagementPerformance management and fault management comprise the processes of monitoring EVCs for proper operations, discovering any problems and correcting faults that have occurred.

• Link level performance and fault management as defined by IEEE 802.3ah provides mechanisms to monitor link operation and health, and for elementary fault isolation.

• The more sophisticated end-to-end service level performance and fault management of the Ethernet Virtual Connection is often referred to as Ethernet service OAM (SOAM) and is addressed by MEF Specifications 17, 30 and 31, and the ITU-T Y.1731 and IEEE 802.1ag standards.

In a real wide area network, Ethernet Virtual Connections may span several networks, each with their own management needs. Since managing functionality “end-to-end” means different things to the end customer, the network operator and the Carrier Ethernet service provider, Ethernet SOAM must handle and interact with performance and fault management over several Ethernet OAM domains. An OAM domain is simply a network or sub-network of elements belonging to the same administrative entity managing them (Figure 91).

Recognizing the fact that Ethernet networks often encompass multiple administrative domains, IEEE 802.1, ITU-T SG 13 and MEF have adopted a common, multi-domain SOAM reference model. The Carrier Ethernet is portioned into customer, service provider, and operator maintenance levels. Service providers have end-to-end service responsibility; operators provide service transport across a sub-network.

In this model, an entity that requires management is called a maintenance entity (ME). An ME is essentially an association between two maintenance endpoints within an OAM domain, where each endpoint corresponds to a provisioned reference point. For example, in the previous figure, the green arrow between the two CEs represents a subscriber ME.

A maintenance entity group (MEG) consists of the MEs that belong to the same service inside a common OAM domain. The MEs exist within the same administrative boundary and belong to the same point-to-point or multipoint Ethernet Virtual Connection. For a point-to-point EVC, the MEG contains a single ME.

An MEG endpoint (MEP) is a provisioned OAM reference point that can initiate and terminate proactive OAM frames. It can also initiate and react to diagnostic OAM frames. The MEPs are indicated by triangles in the figures.

An MEG intermediate point (MIP) is any intermediate point in an MEG that can react to some OAM frames. An MIP does not initiate OAM frames; neither does it take

action on the transit Ethernet traffic flows. The MIPs are indicated by circles in the figures.

The concepts of the SOAM reference model are summarized by Figure 92 (on the next page), indicating the six default MEG levels considered by MEF.

Given the above SOAM reference model, MEF specifications 30.1 and 35 define

a wide set of performance and fault management activities for the EVC and its sub-components, such as:

Performance Management• Frame delay. Measurement of one-way and

two-way (round-trip) delay from MEP to MEP.

New customer order

1. Provisioning of the circuit• Pre-test (turn-up)• Verify compliance with SLA

2. Performance management• Does the EVC meet the SLA?• Is the customer happy?

3. Fault management• Sectionalize the problem• Identify and correct

Tear down circuitReturn to inventory

MEF specifications + ITU-T Y.1564

MEF specifications + ITU-T Y.1564

Pass

Fail

End of lifeYes NoMEF specifications + IEEE 802.1ag andIEEE 802.3ah

Source: Metro Ethernet Forum

Customer Equipment

(CE)

Customer Equipment

(CE)Demarcation Demarcation

Operator 1 Operator 2

Customer Domain

Provider Domain

Operator 1 Domain

Operator 2 Domain


FIGURE 91: Hierarchical OAM Domains Define the OAM Responsibilities and Flows of OAM DataFIGURE 90: The Life Cycle of an Ethernet Service According to Metro Ethernet Forum. Some of the Involved Standards Have Been Indicated



FIGURE 92: Example of Ethernet SOAM Maintenance Entities. Source: Metro Ethernet Forum

1 82 3 4 5 6 7

654321

SubscriberEquipment

SubscriberEquipmentService Provider

Subscriber ME

Test ME

EVC ME

SP ME

E-NNI ME

Operator A NEs Operator B NEs

Operator A ME Operator B ME

Default MEG Level

MEP (Up orientation)MEP (down orientation)

MIP Logical path of SOAM PDUs

• Inter-frame delay variation. Differences between consecutive frame delay measurements.

• Frame loss ratio. The number of frames delivered at an egress UNI compared to the number of transmitted frames over a specified time, e.g. a month.

• Availability. Downtime is measured over a period of time, e.g. a year, and used to calculate the availability of the service.

Fault Management• Continuity check. “Heartbeat” messages

are issued periodically by the MEPs and

used to proactively detect the loss of connection between endpoints. Continuity check is also used to detect unintended connectivity between MEGs, as well as to verify basic service connectivity and health.

• Remote defect indication signal. When a downstream MEP detects a fault, it will signal the condition to its upstream MEP(s). This behavior is similar to the remote defect indicator function in SDH/SONET networks.

• Alarm indication signal. An MEP can send an alarm signal to its higher-level MEs, thereby informing them of the disruption, immediately following the detection of a fault.

• Linktrace is an on-demand OAM function initiated in an MEP to track the path to a destination MEP. It allows the transmitting node to discover connectivity data about the path.

• Loopback is an on-demand OAM function used to verify the connectivity of an MEP with its peers.

5.8 METRO ETHERNET FORUM “THIRD NETWORK” VISION

Current trends in data communications imply a scenario in which network operators can deliver self-service, on-demand communication services over multiple, interconnected networks. Metro Ethernet Forum refers to this evolution as the emergence of a “Third Network” (Figure 93). A central element of MEF’s Third Network vision is the ability to enable services not only between physical service endpoints such as Ethernet ports (UNIs), but also between virtual service endpoints running on a blade server in the cloud, connecting to virtual machines (VMs) or virtual network functions (VNFs).

MEF aims to achieve this vision by defining Lifecycle Service Orchestration (LSO) capabilities and supporting application program interfaces. MEF will define

functional requirements and APIs for LSO that support capabilities for fulfillment, control, performance, assurance, usage, security, analytics, and policy across multi-operator networks. This approach overcomes some of the existing complexity by defining service abstractions that hide the complexity of underlying technologies and network layers from the applications and users of the services, thus facilitating, for example, customer self-service of network capacity.

SUMMARYOptical fiber provides almost lossless transmission of signals at an ultra-wide range of frequencies. Packet switching, implemented according to the Ethernet family of protocols, offers one of the most efficient ways ever to sort and direct streams of digital data. With packet-optical networking, these two outstanding technologies are positioned to dominate the next generation of transport networks. In addition, the continuing evolution driven by industry groups ensures dependable and open standards for future needs.

Infinera’s packet-optical technology, realized by the XTM Series platform, the DNA multi-layer management suite and the Xceed SDN

platform, represents a powerful toolbox for the deployment of these new networks. The architecture encompasses elements that are fully certified for Carrier Ethernet 2.0 services, capable of both native Ethernet and OTN-adapted transport and with additional scalability and resilience enabled by MPLS-TP. Special attention has been given to efficient synchronization, ultra-low latency and minimal jitter in small as well as large networks. Configuring and managing an Infinera packet-optical network is simple and straightforward thanks to the carefully

designed suite of multi-layer management and control tools.

As the adoption of cloud services accelerates, it is imperative that the transport network becomes more intelligent for the emerging Layer C to thrive. The simplified Layer T network is best built using a combination of scalable technologies like super-channels and intelligent capabilities delivered using packet-optical technologies combined with rich software management and control applications.

Self-serviceWeb Portal

Lifecycle ServiceOrchestration

Lifecycle ServiceOrchestration

OperatorService

Endpoint

UserService

Endpoint

End-to-endLifecycle Service Orchestration

End-to-End Network as a Service

APIsAPIs

APIs

APIs

APIs

APIs

Service Provider OSS/BSSBusiness Applications

Operator Virtual ConnectionOperator Virtual Connection

Network Operator(WAN Service Provider)

Network Operator(Data Center)

VM or VNF

Subscriber

FIGURE 93: The Metro Ethernet Forum’s “Third Network” Vision. Source: MEF 2015.



6 I N D E X


110 GbE ........................................................... 79

Aactive LSP ...................................................... 27

aggregation network ............................. 14, 65

ATM ............................................................... 62

attachment circuit (AC) ................................ 27

automatic protection switching .................. 85

Bbandwidth profile ......................................... 95

boundary clock ............................................. 84

broadcast domain ........................................ 75

burst size ....................................................... 96

Business Ethernet ......................................... 62

CCarrier Ethernet 2.0 ...................................... 88

network .............................................. 16, 88

service ................................................ 62, 90

CFP ................................................................ 80

circuit emulation ........................................... 80

class of service (CoS) .................................... 96

cloud services ................................................. 8

committed

burst size (CBS) ....................................... 96

information rate (CIR) ............................. 96

connection-oriented service ....................... 26

connectionless service ................................. 52

control plane ................................................. 51

Converged Cable Access Platform (CCAP) ........................................................... 70

convergence time ........................................ 84

CSMA/CD ..................................................... 75

customer

equipment (CE) ...................................... 91

portal ....................................................... 48

VLAN ....................................................... 78

Ddata plane ..................................................... 55

differential clock recovery (DCR) ................ 33

differential timing ................................... 82, 84

Digital Network Administrator (DNA) ........ 48

distributed routing functionality (DRF)....... 56

DOCSIS ......................................................... 70

DTN-X ...................................................... 36, 64

EE-

Access ............................................................ 90

E-LAN ............................................................ 90

E-Line ............................................................. 90

E-Transit ......................................................... 90

E-Tree ............................................................ 90

element manager ......................................... 48

EMXP ....................................................... 21, 35

enterprise network ....................................... 62

ERPSv2 ........................................................... 88

Ethernet ......................................................... 74

frame........................................................ 74

meshed .................................................... 76

OAM ........................................................ 97

physical media ........................................ 78

service attribute ...................................... 94

service attribute parameter ................... 94

service provisioning ............................... 99

service type ............................................. 94

shared ...................................................... 75

station ...................................................... 75

switch ....................................................... 75

switched .................................................. 75

Synchronization Message Channel ...... 83

synchronization over .............................. 82

traffic management ................................ 95

Virtual Connection (EVC) ....................... 91

Ethernet demarcation unit (EDU) ......... 35, 63

Ethernet Ring Protection Switching (ERPS) ............................................................ 87

Ethernet-aware Layer 1 transport ............... 15

eTOM Business Process Framework .......... 45

excess

burst size (EBS) ....................................... 96

information rate (EIR) ............................. 96

express wavelength ..................................... 67

external network to network interface (ENNI) ............................................................ 92

FFast Ethernet .......................................... 74, 78

FCAPS suite .................................................. 46

fixed mobile convergence (FMC) ............... 64

forward error correction (FEC) .................... 18

frame check sequence (FCS) ...................... 74

frame relay (FR) ............................................. 62

Frameworx .................................................... 45

frontplane ...................................................... 36

GG.709 ............................................................. 19

G.781 ............................................................. 83

G.8032 ........................................................... 87

G.805/G809 ................................................... 49

G.8264 ........................................................... 83

Generic Framing Procedure (GFP) ............. 19

Gigabit Ethernet ........................................... 74

IIEEE 1588v2................................................... 83

IEEE 802.1ag ................................................. 98

IEEE 802.1Q/p .............................................. 77

IEEE 802.3ah ................................................. 98

IEEE 802.n ..................................................... 74

IGMPv3 .......................................................... 71

Infinera portfolio ..................................... 10, 40

Infinera’s management software suite ....... 47

Intelligent

SFP (iSFP) ................................................ 33

transport .................................................... 8

Transport Network architecture .............. 9

interworking function (IWF) ......................... 81

IP backhaul .................................................... 64

IP-VPN ..................................................... 22, 62

Llabel

edge router (LER) ................................... 27

switched path (LSP) ................................ 26

switching router (LSR) ............................ 24


Layer

C ........................................................... 8, 56

T ........................................................... 8, 56

Layer 1

aggregation ............................................ 15

Ethernet transport .................................. 16

Layer 2

aggregation ............................................ 14

Ethernet transport .................................. 17

leased line ..................................................... 62

Lifecycle Service Orchestration (LSO) ...... 102

Link Aggregation.......................................... 86

Control Protocol (LACP) ........................ 86

Group (LAG) ............................................ 85

link layer ........................................................ 74

MMAC address .......................................... 26, 74

maintenance entity group (MEG) ............. 101

management

framework ......................................... 45, 97

information base (MIB) .......................... 98

MEF “Third Network” ................................ 102

MEG

endpoint ................................................ 101

intermediate point ............................... 101

level........................................................ 101

Metro Ethernet Forum (MEF) ...................... 90

mobile

backhaul .................................................. 67

fronthaul .................................................. 69

MPLS-TP ........................................................ 26

tunnel ....................................................... 27

MTOSI 2.0 ..................................................... 46

multi-chassis LAG ......................................... 86

multi-layer

management .......................................... 38

management system ............................. 45

network management ........................... 44

SDN ......................................................... 50

multipoint-to-multipoint EVC ..................... 92

Multiprotocol Label Switching (MPLS) ......... 8

muxponder ............................................. 18, 36

Nnative Ethernet ....................................... 19, 20

network

functions virtualization (NFV) ................ 53

interface card (NIC) ................................ 74

interface device (NID) ...................... 35, 62

manager .................................................. 48

planning .................................................. 51

segments ................................................. 64

virtualization ............................................ 51

non-blocking switches ................................. 75

OOAM domain .............................................. 100

ODU2e .......................................................... 20

ODUflex (GFP) ........................................ 20, 64

OpenFlow ..................................................... 55

operations, administration and maintenance ......................................... 97

operations and support system (OSS) ................................................. 48

operator virtual connection (OVC) ............. 92

optical channel

data unit (ODU) ...................................... 20

payload unit (OPU) ................................. 19

transport unit (OTU) ............................... 20

Optical Transport Network (OTN) ................ 8

OSS integration toolkit ................................ 48

OTN

framing .................................................... 19

switch ....................................................... 20

transport system ..................................... 20

PPacket Switching Module (PXM)..... 22, 36, 64

Packet Transport Fabric (PT-Fabric) ............ 36

Packet-Optical

Transport Switch (EMXP) ....................... 35

transport system ....................................... 3

networking, applications ....................... 62

transport, advantages of ................. 38, 44

path computation

element (PCE) ......................................... 53

function ................................................... 53

point-to-point EVC ....................................... 92

policing .......................................................... 30

port device .................................................... 35

power consumption ..................................... 41

Precision Time Protocol (PTPv2) ........... 82, 84

primary reference clock (PRC) ..................... 81

Provider

Backbone Bridges .................................. 78

bridges .................................................... 78

edge (PE) ................................................. 27

pseudowire ....................................... 16, 24, 80

PT-Fabric........................................................ 36

PXM ................................................... 22, 36, 64

QQ-in-Q ..................................................... 49, 78

quality of service (QoS) .......................... 35, 88

RRapid Spanning Tree Protocol (RSTP) ........ 76

ring protection link ....................................... 87

ROADM ................................................... 28, 36

Sscalability ..................................... 26, 39, 67, 90

SDH ...................................................... 9, 19, 32

SDN ............................................................... 51

applications............................................. 57

controller ................................................. 52


service

assurance .......................................... 46, 50

awareness ................................................ 39

frame........................................................ 96

fulfillment ................................................ 50

lifecycle .................................................... 98

VLAN (SVLAN) .................................. 26, 78

acceptance criteria (SAC) ...................... 99

level agreement (SLA) ............................ 96

SNMPv2 ......................................................... 98

software-defined networking (SDN) ........... 51

SONET ................................................ 9, 19, 32

Spanning Tree Protocol (STP) ..................... 76

statistical multiplexing ................................. 15

switched video transport ............................. 69

SyncE ....................................................... 30, 83

synchronization ....................................... 33, 80

probe ....................................................... 34

Synchronization Status Message (SSM) ............................................. 83

Synchronous Ethernet (SyncE) .............. 30, 83

TTDM dervices, migrating to Ethernet ........ 32

time-division multiplexing (TDM) ............... 80

TM Forum ..................................................... 45

TMF608 ................................................... 46, 49

traffic

engineering....................................... 22, 26

management .......................................... 30

shaping .................................................... 30

transparent clock .......................................... 84

Transport SDN .............................................. 52

Uunified information model .......................... 49

user network interface (UNI) ................. 83, 91

Vvideo services ............................................... 69

virtual

LAN (VLAN) ............................................. 76

private LAN service (VPLS) .............. 22, 28

private network (VPN) ............................ 77

routing ..................................................... 56

virtualized network ....................................... 54

VLAN

identifier (VID) ......................................... 77

tag ............................................................ 77

Wweighted random early detection (WRED)......................................... 31

XXTM Series .................................. 18, 20, 24, 36

YY.1731....................................................... 41, 98

1 Optical Transport Network is a standard for multiplexing and switching signals at OSI Layer 1.

2 Multiprotocol Label Switching—Transport Profile is a mechanism to create virtual connections.

3 The Open Systems Interconnection (OSI) model is a conceptual model that characterizes and standardizes the communication functions of a telecommunication or computing system.

4 Analysts estimate that Layer 2 equipment is only 30 to 50% of the cost of Layer 3 equipment.

5 See section 5.5 for more information about Carrier Ethernet as defined by Metro Ethernet Forum.

6 A data communications network is used for the management and control of network equipment.

7 Known as e.g. 10G BASE-X.

8 See section 2.3 for more on MPLS-TP.

9 Note that the use of the term “router” is historic and neither requires nor precludes the ability to perform IP forwarding. It is sometimes used instead of “node” in an MPLS context.

10 VPLS is defined in IETF RFC 4761 and RFC 4762.

11 E1 is a 2 Mb/s signal in the older Plesiochronous Digital Hierarchy (PDH) standard.

12 MPLS used in conjunction with IP and its routing protocols.

13 The IEEE 1588 standards describe a hierarchical master-slave architecture for clock distribution in computer networks, originally known as the Precision Time Protocol (PTP).

14 Enhanced Telecom Operations Map.

15 Multi-Technology Operation Systems Interface (MTOSI) is a TM Forum standard for implementing interfaces between operations and support systems (OSS).

16 Connection-oriented service: A semi-permanent connection is established before any useful data can be transferred, and a stream of data is delivered in the same order as it was sent.

17 Transport SDN is sometimes used to specifically designate integrated Layer 0/Layer 1 networks operating under the SDN scheme.

18 ONF is a trade organization standardizing the OpenFlow protocol and related technologies.

19 ”White box” is a generic term for minimalistic packet switches built from merchant silicon switching chips and commodity CPU and memory.

20 North American operators of cable and direct-broadcast television systems are often referred to as MSOs. In other parts of the world, the abbreviation CATV operator is used.

21 Data Over Cable Service Interface Specification. DOCSIS 3.0 has been ratified as ITU-T Recommendation J.222.

22 For Gigabit Ethernet, some vendors provide equipment that supports a jumbo frame option,

where frames can have a data payload of up to 10,000 bytes.

23 The number 2 refers to the second layer in the standardized OSI reference model for data communications.

24 In CSMA/CD, the devices (called stations) can broadcast data over the medium whenever it is idle. If more than one station transmits at the same time and signals collide, the transmission is stopped by the involved stations, which will then wait for some random time and then restart transmission.

25 The “Q” refers to the name of the IEEE 802.1Q standard for VLANs.

26 Service restoration times of 30 seconds or more are typical for STP.

27 In many Metro Ethernet Forum technical specifications and some other literature, the Carrier Ethernet network is referred to as a metro Ethernet network (MEN), which is the same thing.

28 The ETH layer is sometimes referred to as the path layer.

29 A service frame is a subscriber Ethernet frame to be forwarded by a service in a Carrier Ethernet.

F O O T N O T E S

I N F I N E R A112 Footnote citations on page 111

w w w . i n f i n e r a . c o m

https://www.infinera.com

Packet-Optical The Infinera Way eBook · be carried in digital OTN circuits, which in turn are...

Documents

Transcript of Packet-Optical The Infinera Way eBook · be carried in digital OTN circuits, which in turn are...