© 2013 Ethernet Alliance1 Moderator: Scott Kipp, President of Ethernet Alliance, Principle...

© 2013 Ethernet Alliance 1

Moderator: Scott Kipp, President of Ethernet Alliance, Principle Engineer, BrocadePanelist #1: Alan Weckel, Vice President, Dell’Oro groupPanelist #2: Dr. Jeffery J. Maki, Distinguished Engineer, JuniperPanelist #3: Dr. Gordon Brebner, Distinguished Engineer, Xilinx

Need for Speed: Beyond 100GbE

2© 2013 Ethernet Alliance© 2012 Ethernet Alliance

Agenda

Introductions: Scott Kipp, Moderator Panelist #1: Alan Weckel,

10, 40 and 100GbE Deployments in the Data Center

Panelist #2: Dr. Jeffery J. Maki, Stepping Stones to Terabit-Class Ethernet

Panelist #3: Dr. Gordon Brebner, Technology Advances in 400GbE Components

Q&A 2:40 – Live Broadcast from IEEE 802.3 Meeting

in Orlando from John D’Ambrosia Update on 400GbE Call For Interest

3© 2013 Ethernet Alliance

Disclaimer

The views WE ARE expressing in this presentation are our own personal views and should not be considered the views or positions of the Ethernet Alliance.


Bandwidth Growth

Increased # of

Users

Increased Access

Rates and Methods

Increased Services++ =Bandwidth

ExplosionEverywher

e

Source: nowell_01_0911.pdf citing Cisco Visual Networking Index (VNI) Global IP Traffic Forecast, 2010–2015, http://www.ieee802.org/3/ad_hoc/bwa/public/sep11/nowell_01_0911.pdf

More Devices

More Internet Users

More Rich Media Content

Key Growth Factors

Speed Increasing

Broadband2010- 7Mbps

2015 – 28 Mbps

15B DevicesIn 2015

2010- 1 Minute video2015 – 2 hour HDTV

Movie

3B UsersIn 2015

http://www.ieee802.org/3/ad_hoc/bwa/public/sep11/nowell_01_0911.pdf


Bandwidth Growth Vs Ethernet Speeds

IP Traffic is growing ~ 30%/year If 400GbE is released in 2016, Ethernet speeds

will grow at about 26%/year

2010

2011

2012

2013

2014

2015

2016

2017

2018

2019

2020

0

200

400

600

800

1000

1200

1400

1600

Ethernet SpeedInternet Traffic

Eth

ern

et

Speed (

Gb

/s)

Inte

rnet

traffi

c norm

aliz

ed

to

100 in 2

010

Internet traffic would grow ~10X by 2019 at 30%/year

Ethernet speeds to grow 4X by 2016 at 26%/year


Ethernet Optical Modules

XENPAKXPAKX2

300 Pin MSA100G

10G

1G

1995 2000 2005 2010 2015

Standard Completed

40G

100GbE

40GbE

Data

Rate

an

d L

ine R

ate

(b

/s)

Key:Ethernet Standard ReleasedModule Form Factor Released

GbE

CFP

QSFP+

SFPGBIC

10GbE

SFP+

XFP

CFP2

QSFP28

CFP4

CXP


Ethernet Speeds 2010-2025Key:EthernetSpeeds

Ethernet ElectricalInterfaces

Hollow Symbols = predictions

Stretched Symbols = Time Tolerance

1T

100G

10G

400G

40G4x10G

10X10G

2010 2015 2020 2025

Standard Completed

100GbE10X10

G

40GbE4X10G

Data

Rate

an

d L

ine R

ate

(b

/s)

16x25G

400GbE16X25

G

4x25G

100GbE4X25G

8X50G

400GbE8X50G 400GbE

4X100G

100GbE1X100G

TbE10X100G

nX100G

1.6TbE16X100G

If Ethernet line rates doubles the line rate every 3 years at 26% CAGR, then 400GbE would come out in 2016 and TbE would come out in 2020. Something will have to change.


Ethernet Success

Ethernet has been extremely successful at lowering the price/bit of bandwidth

If the cost of a new speed/technology is too high, then it is not widely deployed

Technology needs to be ripe for picking 400GbE is ripe with 100GbE technology TbE isn’t ripe and a revolutionary

breakthrough would be needed to get it before 2020

This panel will look at how high speeds of Ethernet are being deployed and the technology that is leading to the next generation of Ethernet


10, 40 and 100GbE Deployments in the Data Center

Alan WeckelVice President, Data Center ResearchDell’Oro Group

10© 2013 Ethernet Alliance 10© 2013 Ethernet Alliance

INTRODUCTION

Progress on server migration from 1 GbE to 10 GbE

10G Base-T update

Data center networking market update

40 GbE and 100 GbE market forecasts


OVERVIEW

Dell’Oro Group is a market research firm that has been tracking the Ethernet Switch and Routing markets on a quarterly basis since 1996

We also track the SAN market, Optical market, and most Telecom equipment markets

We produce quarterly market share reports that include port shipments as well as market forecasts


0

350

700

2009

2010

2011

2012

2013

2014

2015

2016 P

eta

byte

s p

er

Secon

d

Sh

ipp

ed

per

Year

DATA CENTER BANDWIDTH SHIPPING – ETHERNET SWITCHING


0%

25%

50%

75%

100%

Perc

en

t of

Serv

er

Sh

ipm

en

ts

SWITCH ATTACH RATE ON SERVERS

10 GbE1 GbE 40 GbE


0

90

180

1Q11

2Q11

3Q11

4Q11

1Q12

2Q12

3Q12

4Q12

Port

Sh

ipm

en

ts in

Th

ou

san

ds

DATA CENTER PORT SHIPMENTS –10 G BASE-T PORT SHIPMENTS

10G Base-T controller and adapter ports

10G Base-T switch ports


0

25

50

2009

2010

2011

2012

2013

2014

2015

2016

1 GbE 10 GbE 40 GbE 100 GbE

Port

Sh

ipm

en

ts in

Million

sDATA CENTER PORT SHIPMENTS –ETHERNET SWITCHING


0

3

6

2009

2010

2011

2012

2013

2014

2015

2016

40 GbE 100 GbE

Port

Sh

ipm

en

ts in

Million

sDATA CENTER PORT SHIPMENTS –ETHERNET SWITCHING


SUMMARY

Ethernet Switches will be responsible for the majority of 40 GbE and 100 GbE port shipments over the next five years

Form-factor and cost driving 40 GbE over 100 GbE

10 GbE server access transition is key to higher speed adoption


Stepping Stones to Terabit-Class Ethernet:

Electrical Interface Rates andOptics Technology Reuse

Jeffery J. MakiDistinguished Engineer, OpticalJuniper Networks, Inc.


100G


CFP, CFP2 and CFP4 forSMF or MMF Applications

CFP(L

C)

CFP

4(L

C)

CF

P CFP

2 CFP

4 CFP2(LC

)

CFP MSA Form Factors:http://www.cfp-msa.org/

Optical Connector• LC Duplex (depicted)• MPO

Courtesy ofTE

Connectivity


CF

P CFP

2 CFP

4

Module Electrical Lane Capability

12x10Gelectrical

lanes

10x10G or 8x25Gelectrical

lanes

4x25Gelectrical

lanes

CAUI-4 for 4x25GCPPI & CAUI for 10x10GCAUI-4 for 4x25G

CAUI for 10x10G


CFP, CFP2, and CFP4 for 100G Ethernet SMF PMD

Transmit side only depicted.

Current Options• Up to 10 km: 100GBASE-LR4

• Up to 40 km: 100GBASE-ER4

Gea

r B

ox 1295.56 nm

1300.05 nm1304.58 nm1309.14 nm

Gea

r B

ox 1295.56 nm

1300.05 nm1304.58 nm1309.14 nm

CFP

CFP2

CFP44 λ on LAN WDM

LAN WDM

LAN WDM


400G


Projection of Form Factor Evolution to 400G

CD

-CFP

CFP

4C

FP

4C

FP

4C

FP

4

400G

CD

-CFP

2

16x25Gelectrical

lanes

8x50Gelectrical

lanesspeculation

defe

nsi

ble C

D-C

FP

4

4x100Gelectrical

lanes

CFP

CFP

2

CFP

4

100G

Roman NumeralsXL = 40C = 100CD = 400


Likely MSA Activity CFP MSA http://www.cfp-msa.org/

CD-CFP: Current CFP needs revamping to support 16 x 25G CD-CFP2: Current CFP2 is ready for 8 x 50G CD-CFP4: Unclear

New CDFP MSA http://www.cdfp-msa.org/ High-density form factor supporting 16 x 25G From slide 26 of

http://www.ieee802.org/3/cfi/0313_1/CFI_01_0313.pdf

http://www.cfp-msa.org/

http://www.cdfp-msa.org/

http://www.ieee802.org/3/cfi/0313_1/CFI_01_0313.pdf


400G Optics Requirements

First-generation transceivers have to be implementable that meet and eventually do better than these requirements Size (Width): 82 mm (CFP width, ~4 x

CFP4) Cost: 4 x CFP4 Power: 24 W (4 x 6 W power profile of CFP4)

Improved bandwidth density transceivers will need higher rate electrical-lane technology 50G 100G


How 400G Ethernet Can Leverage 100G Ethernet

CFP4-LR4

CFP4-LR4

CFP4-LR4

CFP4-LR4

CFP4-LR4

CFP4-LR4

CFP4-LR4

CFP4-LR4

CFP4-LR4 CFP4-LR4Duplex Single-Mode Fiber Infrastructure

100G Ethernet up to 10 km

400G Ethernet up to 10 kmParallel Single-Mode Fiber Infrastructure

Only 8 Fibers Used


Possible SMF Ethernet Road Map: 100G, 400G, 1.6T

4 x 100GBASE-LR4or

“400GBASE-PSM4”

CD-CFP4(LC)

CFP4(LC)CFP4(LC)CFP4(LC)

CD-CFP(MPO)

400GBASE-???

CD-CFP2(LC)

CFP4(LC)

4 x 400GBASE-???or

“1600GBASE-PSM4”

CD-CFP4(LC)

(High-Density 100GE)

Early Adopter 400G Mature 400G Early Adopter 1.6T

Parallel Single Mode, 4 Lanes (PSM4)4, Tx Fibers and 4, Rx Fibers1x12 MPO Connector

CD-CFP2(MPO)

CD-CFP4(LC)CD-CFP4(LC)CD-CFP4(LC)


Early Adopter 400G using SMF Structured Cabling

Technology Reuse:4 x 100GBASE-LR4

Parallel SMF:“400GBASE-PSM4”

Courtesy of Commscope


Early Adopter 400G using MMF Structured Cabling

Technology Reuse:4 x 100GBASE-SR4

Parallel MMF:“400GBASE-SR16”

Parallel Multi-Mode• 100GBASE-SR4, 4 x 25G optical lanes:

4, Tx Fibers and 4, Rx Fibers using 1x12 MPO

• “400GBASE-SR16”, 16 x 25G optical lanes:16, TX Fibers and 16, Rx Fibers using 2x16 MPO

Courtesy of Commscope


2 x 16 MPO

MMF Breakout Cables—Enabling 400G Adoption

1 x 12 (8 used) MPO

1 x 12 (8 used) MPO

1 x 12 (8 used) MPO

1 x 12 (8 used) MPO

Courtesy of USConec

2 x 16 MMF MT ferrule


100G Can Build 400G atthe Cost of 4 x 100G

Technology Reuse:4 x 100GBASE-SR4

Parallel MMF:“400GBASE-SR16”

Technology Reuse:4 x 100GBASE-LR4

Parallel SMF:“400GBASE-PSM4”


Early Adopter PMD Parallel Fiber, SMF or MMF Leverage of mature PMD from previous speed of

Ethernet Planned obsolescence Implementation (with MPO connector) persists as

high-density support of previous speed of Ethernet (e.g., 4 x 100G)

Mature PMD SMF: Duplex SMF cabling (e.g., with LC duplex

connector) MMF: Lower fiber count MMF cabling

Ethernet PMD Maturity & Possible Obsolescence


SMF Density Road Map

Front-PanelBandwidthDensity(Relative)

100G 400G 1.6T

CFP(LC)CFP2(LC)

CFP4(LC) CFP4(LC)4 x or

CD-CFP(MPO)

CD-CFP2(LC)

CD-CFP4(LC) CD-CFP4(LC)4 x

CD-CFP2(MPO) CD-CFP2(MPO)

12

4

8

16

Port Bandwidth

(mature) (early adopter) (mature)

(mature) (earlyadopter)

(early adopter)


Summary

Form-factor road map for bandwidth evolution

Early adopter 400G Ethernet by reusing 100G module and parallel cabling, SMF or MMF

Need for a new, 2 x 16 MMF MT ferrule Possible common module for 400G

Ethernet and high-density (4-port) 100G Ethernet

Need for new electrical interface definitions supporting lane rates at 50G 100G


Gordon BrebnerDistinguished EngineerXilinx, Inc.

Technology Advances in 400GbE Components


400GbE PCS/MAC

Expect first: 16 PCS lanes, each at 25.78125 Gbps Glueless interface to optics Possible re-use of the 802.3ba PCS Other options possible for PCS, maybe native FEC

Later: 8 lanes, each at 51.56Gbps Or 4 lanes with 2 bits/symbol at 56Gbaud (e.g. PAM4)

Packet size 64 bytes to 9600 bytes

Use 100GbE building blocks where possible


Silicon technology

Technology nodes (silicon feature size) 130nm, 65nm, 40nm, 28/32nm, 20/22nm,

14/16nm

Application-Specific Integrated Circuit (ASIC) Fixed chip Increasingly expensive: need high volumes Best suited to post-standardization Ethernet

Field Programmable Gate Array (FPGA) Programmable logic chip Suitable for prototyping and medium volumes Best choice for pre-standardization Ethernet


400GbE line/system bridge

500G

Interlaken

40 x 12.5Gor

48 x 10GSERDESBridge

logic

400GbE

PMA/PCS

CDFPor

4xCFP4

Optical

16 x 25GSERDES 400GbE

MAC

Wide parallel data path between blocks

ASIC or FPGA chip

Line side System side


MAC rate = Width x Clock

400 Gbps and 1 Tbps Ethernet MAC options

MAC rate

Silicon node

Technology

Data path width

Clock frequency

100 Gbps 45, 40nm ASIC 160 bits 644 MHz

100 Gbps 45, 40nm FPGA 512 bits 195 MHz

400 Gbps 28, 20nm ASIC 400 bits 1 GHz

400 Gbps 28, 20nm FPGA 1024 bits1536 bits

400 MHz267 MHz

1 Tbps 20, 14nm ASIC 1024 bits 1 GHz

1 Tbps 20, 14nm FPGA 2048 bits2560 bits

488 MHz400 MHz


Multiple Packets/Word

Up to 512-bit, only one packet completed Just need to deal with EOP then SOP in word

Beyond 512-bit, multiple packets completed Need to add parallel packet processing Must deal with varying EOP and SOP positions

Bus width Max packets Max EOPs

512 2 1

1024 3 2

1536 4 3

512 * n n+1 n


400GbE CRC Example

All Ethernet packets carry Cyclic Redundancy Code (CRC) for error detection Computed using CRC-32 polynomial Critical function within Ethernet MAC

Requirements Computed at line rate Deal with multiple packets in wide data path Economical with silicon resources


400GbE CRC Prototype

Xilinx Labs research project Modular: built out of 512-bit 100G units Computes multiple CRCs per data path word Targeting 28nm FPGA (Xilinx Virtex-7 FPGAs)

N-bit data path partitioned into 512-bit sections

512-bit unit CRC results combined to get final CRC results


400GbE CRC Prototype

Results:

1024-bit width is feasible for 400GbE Other widths:

Less challenging clock frequencies Demonstrate scalability beyond 400GbE

Data bus word size 1024-bit 1536-bit 2048-bit

Max clock frequency (MHz) 400 381 326

Maximum line rate (Gbps) 409 585 668

Latency (ns) 17.5 18.4 21.5

FPGA resources (slices) 2,888 4,410 5,719


Conclusions

Can anticipate 400GbE PCS/MAC standard

Ever-increasing rates mean ever-wider internal data path width in electronics Leading to multiple packets per data word

Possible to prototype pre-standard PCS/MAC using today’s FPGA technology

Demonstrated modular Ethernet CRC block based on 100GbE units Silicon resource scales linearly with line rate

© 2013 Ethernet Alliance1 Moderator: Scott Kipp, President of Ethernet Alliance, Principle...

Documents

Transcript of © 2013 Ethernet Alliance1 Moderator: Scott Kipp, President of Ethernet Alliance, Principle...