The intersection of optical transport and routing in next-

generation networksPhil Bedard – TME@Cisco

NANOG7710/30/2019

Agenda• Current multi-layer network architecture

• Design • Issues

• Next-generation architecture• Design• Benefits • Enabling technologies• Benefits

• Modeling example

Topology Evolution for SP Networks

Traffic Demand

• Ring based networks based on physical fiber

• Optical transport was not very advanced, so either P2P IP links or, channelized over TDM, or ATM overlay

• 4 interfaces to upgrade, 10 for redundancy

• Fiber still ring in many cases, but L1 circuit topology dictated by traffic demands

• Why did this happen? • Router interfaces were very expen$ive• Optical transport is less expensive? • Pace of optical channel innovation allowed better

utilization of fiber resources

A

Z

A

Z

“Hop by Hop” Bypass

Traffic Demand

Current Multi-Layer Networks

• Active transport elements with ROADMs and muxponders

• Transport and packet networks isolated using gray optics

• Most common deployment in high speed regional and backbone networks

• 1G/10G CWDM/DWDM optics in routers

• 100G/200G using integrated optics, ACO, or DCO optics, typically CFP2 or shorter reach with SFP+, QSFP28

• Most popular in DCI and access networks

Router + Muxponder IPoWDM

Connectivity Types – Linear View

• Standard optics based on distance

• Single network to manage

• BW per fiber limited to single optic BW

• Grey optics between router and muxponder, coherent output from muxponder

• Clients 1:1 mapping to wavelength or multiplexed onto 200G,400G,600G, 800G, 1Tbps channels

• 100-150W power dissipation per combined ”port”

• Minimum 5 optics per circuit

• No transponder required

• Requires channel mux

• 15-21W Power dissipation per port

• Interop with ROADM, maximize use of existing infrastructures

• 2 optics per circuit

IP+DWDMDark Fiber IPoWDM

Issues with current IP+Optical deployments • Multiple opaque networks

• CapEx, OpEx cost of multiple networks • Difficult to share information such as SRLG, GMPLS was not successful• Transport control-plane is proprietary • Circuit deployment complexity across networks

• Different protection schemes at both layers of the network• Difficult to coordinate between layers• Network state difficult to sync after optical protection/restoration

• Organizational silos• Planning • Engineering • Operations

”Router” Group

“Transport” Group

Existing Multi-Layer Topology

High speed IP network – No OTN services/switching

New Single-Layer Single-Hop Fabric

Single-Layer Transport Elements

• C-band and L-band passive multiplexers • Local amplifiers and ILA

No complex CDC add/drop, IP is any to anyNo multi-degree ROADMs, every wavelength terminates at every IP node No regen with OEO at every junction (distancedependent)

Haven’t we been here before? • “Multilayer Convergence” has been talked about for many years

• CRS with 40G IPoDWDM line cards in 2007, many other vendor initiatives, TIP Voyager, etc.

• Becoming more popular in access and aggregation with CFP2-DCO 100G/200G

• Why hasn’t this previously become mainstream for most networks? • Integrated optics were expensive and locked ports into being only DWDM • Optical channel tech has outpaced router electronics and standards • Density, density, density (and cost)

Benefits

• IP layer FRR is used for all traffic protection driven by optical performance (pre-FEC)

• Overall IP network utilization• Wavelength efficiency • Simplifies network operation• Similar or lower CapEx spend • Faster time to turn up

Does not solve

• Still need wavelength planning

• Still need to manage some photonic equipment

• TDM, OTN, and wavelength services need emulation for transparency

Wavelength Utilization

• ROADM bypass requires longer optical links

• Longer links supports lower bit-rates, reducing wavelength (and consequently L0 infra) utilization

• Hop-by-Hop uses more L3 I/Fs, but optimizes fiber utilization

ROADM by-passOptical Length

1,600 km

Hop-by-HopOptical Length

400 km

L0 Forwarding(ROADM By-pass)

L3 Forwarding

400 km

200G

400G

50% Util.

100% Util.

Innovations enabling the architecture

100G/200G CFP2-DCO now for access and aggregation

400G ZR+ using CFP2-DCO and QSFP56-DD pluggable transceiversQSFP56-DD = No density tradeoff

Router NPU bandwidth capacity

Continued reduction in IP router interface costs

Complexity in optics will drive minimization of transceiver counts

Coherent router optics evolution

5x7 inches

3x6 inchesCFP2-ACO

2011 2014 2016 2019 2020

QSFP-DD DCO

CFP2-DCO

More integration

400G Standards Reference

3 m

400G-CR88x 50G-CR

100 m

400G-SR8400G-SR4.2400G-AOC

0.5-2 km

400G-DR4400G-FR4

10 km

400G-LR4400G-LR8

20+ km

OIF 400ZROpen 400ZR+

Copper Cables

MMF / AOC

SMFDuplex

SMFDuplex

SMFDuplex

Dist

ance

Opt

ics

Med

ia

ZR/ZR+ are coherent DWDM optics, ZR+ >15W

400G ZR OIF and Open ZR+FORM FACTOR QSFP56-DD – No Density Penalty CFP2-DCO

TECHNOLOGY ZR OpenZR+ ZR Open ZR+

BW SUPPORT400G

(Single Fixed Wavelength)

400G/300G/200G/100G(Tunable)

400G (Single Fixed Wavelength)

400G/300G/200G/100G (Tunable)

Tx POWER -10 -10/-9/-6/-6 -10 +3

UNAMPLIFIED REACH 400G: 20km

400G: 40km300G: 40km200G: 80km100G: 80km

400G: 40km

400G: 100km300G: 112km200G: 124km100G: 124km

AMPLIFIED REACH 400G: 80km

400G: 1300km300G: 2500km200G: 2500km100G: 2500km

400G: 80km

400G: 1300km300G: 2500km200G: 2500km100G: 4000km

POWER DISSIPATION 15 15-21 15 15-21

FEC C-FEC oFEC, SC-FEC for 100G C-FEC oFEC, SC-FEC for 100G

2004 2005 2006 2007 2008 2009 2010 2011 2012 2013 2014 2015 2015 2016 2017 2018 2019 2020 2021

Router capacity and economics

14M

12M

10M

8M

6M

4M

2M

Gig

abits

NPU

(Mbps)

300B

250B

200B

150B

100B

50B

Gigabits NPU Bandwidth

Internet Gigabits per Month vs. NPU BW

Platform HWOptics

Optics vs. Host Interface

WHY? Host port costs are decreasing FASTER than optics technologies. Optics complexity increases with speed.

Pluggable Optics

Router Interface

100%

0%Speed

Total Cost

10G 100G 400G

Example Regional Network Topology

• 55 Sites• ~100 km across• Two Cache Sites• Two Internet Gateways• Cache and Gateway sites also support local subscribers

Traffic Model

• Edge site traffic scaled to site weight• Site weight dynamic range 60:1

• 75% of traffic to cache sites, 25% to gateways• Each edge site connected diversely to gateways and to cache sites

Hop-by-Hop Approach

• Single-layer one-hop end to end • Single hop transmission negates need for regeneration• Some spans require large LAGs

• Largest LAG 29 x 400G for illustrated example of 50 Tbps load

1330 x 400G Interfaces

Hollow Core Approach

• One-to-one mapping of trafficdemands to end-to-end wavelengths

• Longer reach wavelengths feature B2B regen• Fewer, smaller LAGs (Largest 3 x 400G)

794 x 400G Interfaces(one third as regen)

Optimized Bypass Approach

• Bypasses kept below reach limitto avoid regeneration

• Largest LAG 11 x 400G506 x 400G Interfaces

Comparing The Models

Hop-by-Hop Hollow Core Bypass

ROADM Layer No Yes Yes

Reach Sensitivity

None High Low

Interfaces 1330 794 506

Utilization 95% 45% 82%

Largest LAG 29 3 11

Max Waves/Span

29 58 30

Adjacencies 79 216 132

Load:50 Tbit/s

Reach:5 Hops

Contrasting Approaches As Traffic Scales

0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

0

250

500

750

1000

1250

1500

1750

2000

2250

2500

5000

10000

15000

20000

25000

30000

35000

40000

45000

50000

55000

60000

65000

70000

75000

80000

85000

90000

95000

100000

U"lisa"

on

Interfaces

Load(Gbit/s)

H-by-H Bypass HollowCore HHU<l BPU<l HollowU<l

400G

Q & A

400G DCO Pluggable - Options

• Supported Trunk Rate:• 200G, 300G and 400G (AT&T OpenFEC)• OTU4 with Staircase FEC

• Channel Spacing:• 75GHz (Min)• OTU4: 50GHz (Min)

• TX Power Range: • -10 to +1dBm (SW Configurable)

• RX Sensitivity:• -19dBm• OTU4: -22dBm

• CD Compensation:• OTU4: +/-20,000ps/nm• 200G: +/-50,000ps/nm• 300G: +/-30,000ps/nm• 400G: +/-16,000ps/nm

• DGD Compensation:• 200G, 300G, 400G: 60ps• OTU4: 90ps

• Supported Trunk Rate:• 200G, 300G and 400G (AT&T OpenFEC)• OTU4 with Staircase FEC

• Channel Spacing:• 75GHz (Min)• OTU4: 50GHz (Min)

• TX Power Range: • -10dBm

• RX Sensitivity:• -19dBm• OTU4: -22dBm

• CD Compensation:• OTU4: +/-20,000ps/nm• 200G: +/-50,000ps/nm• 300G: +/-30,000ps/nm• 400G: +/-16,000ps/nm

• DGD Compensation:• 200G, 300G, 400G: 60ps• OTU4: 90ps

400G CFP2 DCO

400G DD-QSFP56 ZR/ZR+ DCO