2006 03 Healey · 2006. 3. 10. · Title: Microsoft PowerPoint - 2006_03_Healey.ppt Author: lokea...

Exceeding the Speed Limit

Challenges and solutions to the support of serial 10 Gigabit Ethernet over backplane interconnects

Adam Healey

Chair, IEEE P802.3ap Task Force

Consulting Member of Technical Staff, Agere Systems

March 10, 2006

2 March 10, 2006 (r1.0)

Before we begin…

� Per IEEE-SA Standards Board Operations Manual, January 2005: • At lectures, symposia, seminars, or educational courses, an individual

presenting information on IEEE standards shall make it clear that his or her views should be considered the personal views of that individual rather than the formal position, explanation, or interpretation of the IEEE.

� Note:• Currently, IEEE P802.3ap is under review per the IEEE 802.3 Working

Group ballot process and is subject to change.

• The work in this presentation is per IEEE P802.3ap Draft 2.1.

3 March 10, 2006 (r1.0)

Agenda

� Introduction to Backplane Ethernet

� Backplane Architecture and Reference Model

� 10GBASE-KR

� Forward Error Correction for 10GBASE-KR

� Closing Remarks

4 March 10, 2006 (r1.0)

Backplane Ethernet Profile

� “Ethernet in a box”

� Target Applications• Wireline and wireless access equipment

• Blade servers

• Enterprise switching

� Leverages field proven Ethernet controller and switching IP

� Reduces solution cost and complexity• Leverages Ethernet economies of scale

• Flattens the backplane fabric eliminating encapsulation protocols and associated processing

• Standard enables multi-vendor interoperability and facilitates use of COTS devices

5 March 10, 2006 (r1.0)

Specification Methodology

� Supply a device specification and not a backplane specification

� Be application agnostic (do no specify the backplane connector)

� Supply informative recommendations to assist backplane designers in identifying backplane channels that are interoperable with “Backplane Ethernet compliant” devices

6 March 10, 2006 (r1.0)

Agenda



� 10GBASE-KR


� Closing Remarks

7 March 10, 2006 (r1.0)

Backplane Architecture and Topology

B

N1

N2

H

“stub”

Blade may be a generic µP and memory complex with an I/O-specific mezzanine card

AC-coupling capacitor (at

receiver)

via

Backplane connector

A B BG C D DG E F FG G H HG

TX +/- RX +/- TX +/- RX +/-

Port 2

Hub Slot XChannel N

Hub Slot XChannel (N+1)

Port 0 Port 1

Port 3

Pin-out example

8 March 10, 2006 (r1.0)

Reference Model� The transmitter and receiver blocks include all off-chip components

associated with the respective block• Example, external AC-coupling capacitors, if required, are included in the

receiver block

� Channel consists of line card and backplane traces plus associated backplane and mezzanine connectors

package package

TP1 TP4

Transmitter ReceiverChannel

9 March 10, 2006 (r1.0)

Backplane Ethernet Objectives

AC or DC278722943212705 to 8 chassis/rack4AC or DC2101630555915202 to 3 chassis/rack4

Switch / Router

AC383812753310276Proposed worst-case3Blade Server

AC27871275331270Full mesh2AC24471022441020Example (dual-star)1

AdvancedTCA

AC / DC Coupling

No. Connectors

Total (mm)

H (mm)

B (mm)

N2 (mm)

N1 (mm)Description

� Support up to 1 m differential traces, including two connectors, on improved FR-4 printed circuit boards

� Support a BER of 10-12 or better

1 kundu_01_0504, 2 PICMG 3.0 R2.0, March 18, 2005, 3 koenen_01_0504, 4 goergen_01_0304

10 March 10, 2006 (r1.0)

Fitted Attenuation

� Defined to be the least-mean squares line fit to the insertion loss data within the frequency range f1 and f2

• The values of f1 and f2 are dependent on the PHY type of interest

� Fitted attenuation is compared to a limit based on a 1 m differential trace (W = 6 mil) on an improved FR-4 printed circuit board

� In effect, constrains the dielectric loss-length product of the channel

1000 1500 2000 2500 3000 3500 4000 4500 5000 5500 6000

0

5

10

15

20

25

30

Inse

rtio

n Lo

ss [

dB]

Frequency [MHz]

1000 1500 2000 2500 3000 3500 4000 4500 5000 5500 6000

0

5

10

15

20

25

30

Inse

rtio

n Lo

ss [d

B]

Frequency [MHz]1000 1500 2000 2500 3000 3500 4000 4500 5000 5500 6000

5

10

15

20

25

30

Inse

rtio

n Lo

ss [d

B]

Frequency [MHz]

Example #1 Example #2 Example #3

11 March 10, 2006 (r1.0)

Insertion Loss Limits

� Discourages encroachment of stub-related resonances into critical frequencies for the PHY type of interest

� Based on the fitted attenuation limit with allowances for passband ripple and stub-related resonances

0 5000 10000 15000

0

10

20

30

40

50

60

70

80

Inse

rtio

n Lo

ss [d

B]

Frequency [MHz]0 5000 10000 15000

0

10

20

30

40

50

60

70

80

Inse

rtio

n Lo

ss [d

B]

Frequency [MHz]0 5000 10000 15000

0

10

20

30

40

50

60

70

80

Inse

rtio

n Lo

ss [d

B]

Frequency [MHz]


12 March 10, 2006 (r1.0)

Insertion Loss Deviation

� Defined to the be the difference between the channel insertion loss and the fitted attenuation in the frequency range f1 to f2

� In effect, constrains passband ripple due to impedance mismatches in the transmission path

1000 1500 2000 2500 3000 3500 4000 4500 5000 5500 6000-4

-3

-2

-1

0

1

2

3

4

Inse

rtio

n Lo

ss D

evia

tion

[dB

]

Frequency [MHz]

1000 1500 2000 2500 3000 3500 4000 4500 5000 5500 6000-4

-3

-2

-1

0

1

2

3

4

Inse

rtion

Los

s D

evia

tion

[dB

]

Frequency [MHz]1000 1500 2000 2500 3000 3500 4000 4500 5000 5500 6000-4

-3

-2

-1

0

1

2

3

4

Inse

rtion

Los

s D

evia

tion

[dB

]

Frequency [MHz]


13 March 10, 2006 (r1.0)

Crosstalk

� Total crosstalk is defined to the be the power sum of the individual aggressors as measured at TP4

� Includes near-end crosstalk (NEXT) and far-end crosstalk (FEXT)

� Assumes the aggressors and victim are asynchronous and (or) uncorrelated

� Assumes the aggressors and victim are driven by identical sources

0 5000 10000 15000

0

10

20

30

40

50

60

70

80

Cro

ssta

lk L

oss

[dB

]

Frequency [MHz]0 5000 10000 15000

0

10

20

30

40

50

60

70

80

Cro

ssta

lk L

oss

[dB

]

Frequency [MHz]0 5000 10000 15000

0

10

20

30

40

50

60

70

80

Cro

ssta

lk L

oss

[dB

]

Frequency [MHz]


14 March 10, 2006 (r1.0)

Insertion Loss to Crosstalk Ratio (ICR)

� Defined to be the least-mean squares line fit to the difference of the total crosstalk loss and the insertion loss (both in dB) in the frequency range fa to fb

� Limits the total crosstalk seen at TP4

� Lower loss links are allowed higher crosstalk and vice versa

100 1000 100000

5

10

15

20

25

30

35

40

45

50

Inse

rtio

n Lo

ss t

o C

ross

talk

Rat

io [d

B]

Frequency [MHz]100 1000 100000

5

10

15

20

25

30

35

40

45

50

Inse

rtio

n Lo

ss t

o C

ross

talk

Rat

io [d

B]

Frequency [MHz]100 1000 100000

5

10

15

20

25

30

35

40

45

50

Inse

rtio

n Lo

ss t

o C

ross

talk

Rat

io [d

B]

Frequency [MHz]


15 March 10, 2006 (r1.0)

0 0.25 0.5 0.75 10

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1

Frequency x Differential Skew

Gai

n (W

/W)

Differential Skew

� To isolate the impact of skew, render the

and sides of the differential pair as delay T1 and T2 respectively

� The skew is the magnitude of the difference between the two delays, |T1−T2|

� The differential skew from TP1 to TP4 is recommended to be less than 24 ps (10GBASE-KR)

1T

2T

2iv

2iv−

ov

( )( )( )122

cos121

)()(

TTVV

i

o −+= ωωω

16 March 10, 2006 (r1.0)

Environmental Variation

� Channel recommendations are to be satisfied over the full range of system operating conditions

0 1000 2000 3000 4000 5000 6000

0

5

10

15

20

25

30

35

40

Inse

rtion

loss

(dB

)

Frequency (MHz)

20oC, 20% RH40oC, 20% RH60oC, 20% RH20oC, 85% RH40oC, 85% RH60oC, 85% RH

Insertion loss variation due totemperature and humidity

17 March 10, 2006 (r1.0)

Impact of Device Terminations

( )( )��

��

�

ΓΓ+ΓΓ−Γ−Γ−Γ−Γ+=

221121122211

21

111

2 SSSSSSS

vv

LSLSLS

SL

i

o

1 11

2

SZ

LZ

��

�

�

2221

1211

SS

SS

SΓ LΓoviv

2 2

Transmitter Receiver

0 1000 2000 3000 4000 5000 6000 7000 8000 9000 10000

0

5

10

15

20

25

30

Inse

rtion

loss

(dB

)

Frequency (MHz)

i

o

vv

2

21S

18 March 10, 2006 (r1.0)

Agenda



� 10GBASE-KR


� Closing Remarks

19 March 10, 2006 (r1.0)

10GBASE-KR Architecture

� 10 Gb/s connection between two media access controllers over an electrical backplane

� Low-overhead 64B/66B encoding

� Statistical transition density and DC balance (scrambled)

� Signaling speed is 10.3125 Gbaud

� Optional Forward Error Correction

� Transmitter equalizer programmed by the link partner during start-up

� Adaptive receiver equalization

MAC

MAC CONTROL (OPTIONAL)

LLC

HIGHER LAYERS

10GBASE-KRPHY

RECONCILIATION

64B/66B PCS

XGMII *

PMA

AN

MEDIUM

PMD

FEC*

* Optional

20 March 10, 2006 (r1.0)

Transmit Equalizer Signal Shaping

pkV

pkV−

0

0c1c 1−c

0c1c 1−c

0c1c 1−c

0c1c 1−c

preV

ssV

pstV

T T

101 −+−−= cccVpre

101 −++−= cccVpst

101 −++= cccVss

101 −++= cccVpk

Assumes that the implementation is a three-tap transversal filter with unit delay.Assumes that the implementation is a three-tap transversal filter with unit delay.

21 March 10, 2006 (r1.0)

Receive Equalizer

� Requirements developed under the assumption of a 5-tap decision feedback equalizer

� However, the implementation of this architecture is not necessarily required for compliance

� Instead, the receiver in question must exhibit an expected level of performance as established via the interference tolerance test

� The interference tolerance test examines the ability of the receiver to equalize a high-loss channel in the presence of horizontal (jitter) and vertical (noise) interference

22 March 10, 2006 (r1.0)

Interference Tolerance Testing

Pattern generator

Clock source

Frequency synthesizer

Frequency-dependent attenuator

Interference injection

Device under test

Interference generator

Test channel

Transmitter control

Modulation input

Equalizer

Introduces static timing offset and jitter

Additive sinusoid swept in frequency and amplitudeInsertion loss emulates

worst-case attenuation

Transmitter signal shaping controlled by adaptation algorithm in the DUT

23 March 10, 2006 (r1.0)

10GBASE-KR Link Model

-0.5 0 0.5 1 1.5-0.5

-0.4

-0.3

-0.2

-0.1

0

0.1

0.2

0.3

0.4

0.5

Am

plitu

de [a

u]

Time Offset [UI]

Am

plitu

de (a

u)

Time Offset (baud)-0.5 -0.4 -0.3 -0.2 -0.1 0 0.1 0.2 0.3 0.4 0.5

-2.5

-2

-1.5

-1

-0.5

0

0.5

1

1.5

2

2.5

Transmitter output eye

Slicer input eye

��

��

� −ΠTkTt

tA

)( fHtkx

)(zF

)( fHr

)(zD

)(zCkx̂

M↓MT

kts +

)( fGr

kε

)(0 fH

10GBASE-KR Performance Simulations

01

01

σσ +−= AASNR

1A

0A

1σ

0σ

Slicer Input Eye: Test Case D

Nor

mal

ized

Am

plitu

de

Time Offset [UI]-0.5 -0.4 -0.3 -0.2 -0.1 0 0.1 0.2 0.3 0.4 0.5

-2

-1.5

-1

-0.5

0

0.5

1

1.5

2

Required SNR for 1E-12 Bit Error Ratio (BER)

Loss-limited case

Stub-limited case

Crosstalk-limited cases

25 March 10, 2006 (r1.0)

10GBASE-KR Start-Up Protocol

� Optimizes transmitter FIR

� Automatic power control• Receiver may steer the transmitter output voltage to the minimum

level required for acceptable performance

• May also mitigate crosstalk

� Optimize receiver equalizer• Joint adaptation of transmitter and receiver yields superior solution

to independent adaptation

� Accomplished through the exchange of fixed-length training frames

26 March 10, 2006 (r1.0)

Training Frame Structure

Frame marker

Coefficient update

Status report

Training pattern

4

16

16

512

Octets

c(1) c(0) c(−1)c(1) c(0) c(−1) RRUG Reserved Reserved

Update gain11 = 1X10 = 2X01 = 4X00 = 8X

Coefficient update11 = reserved10 = increment01 = decrement00 = hold

Receiver Ready1 = Training complete0 = Training in progress

Coefficient status11 = maximum10 = minimum01 = updated00 = not_updated

15 0 15 0

Control channel is differential Manchester encoded at one-quarter of the 10GBASE-KR signaling speed (one octet per control bit)

Two cycles of a PRBS-11 pattern (seeded with all ones) padded with two zeros

0xFFFF0000 (does not appear in control channel or training pattern)

27 March 10, 2006 (r1.0)

Timing Diagram

Device B

ReceiverRdy = 0

Device A

Auto-Negotiation

ReceiverRdy = 0 ReceiverRdy = 1

ReceiverRdy = 1

Equalizer Training Period

Equalizer Training Period

wait_timer

wait_timer

IDLE and DATA

IDLE and DATATraining Frames Training Frames

Training Frames Training Frames

28 March 10, 2006 (r1.0)

Agenda



� 10GBASE-KR


� Closing Remarks

29 March 10, 2006 (r1.0)

Forward Error Correction for 10GBASE-KR

� FEC improves 10GBASE-KR link performance at the expense of added complexity and latency

� Not required to meet objectives

� Burst error correction especially suitable for DFE-based receiver architecture

• Counter-acts error propagation

� Signaling speed is not altered by FEC

MAC

MAC CONTROL (OPTIONAL)

LLC

HIGHER LAYERS

10GBASE-KRPHY

RECONCILIATION

64B/66B PCS

XGMII *

PMA

AN

MEDIUM

PMD

FEC*

30 March 10, 2006 (r1.0)

FEC Frame Structure

� Transcode 66B blocks into FEC frame (32 blocks/frame)• T-bit is the second bit of the 66B sync header (0 = control, 1 =

data)

� Append 32 parity bits (total frame length is 2112 bits)

� Transmission order is left-to-right and top-to-bottomT0 64-bit payload word 0 T1 64-bit payload word 1 T2 64-bit payload word 2 T3 64-bit payload word 3T4 64-bit payload word 4 T5 64-bit payload word 5 T6 64-bit payload word 6 T7 64-bit payload word 7

T8 64-bit payload word 8 T9 64-bit payload word 9 T10 64-bit payload word 10 T11 64-bit payload word 11






32 parity bits

31 March 10, 2006 (r1.0)

FEC Encoder and Decoder

� Parity bits computed using a shortened cyclic code (2112, 2080)

� Frame is then scrambled

� Approximately 2 dB coding gain

� Guaranteed error correction for bursts up to 11 bits in length

� Complexity estimated to be 14K gates plus a 64 x 33 dual port RAM at 312.5 MHz

� Latency estimated to be 200 ns

32 March 10, 2006 (r1.0)

Agenda



� 10GBASE-KR


� Closing Remarks

33 March 10, 2006 (r1.0)

Observations

� Backplane channels are not as predictable or easily classified as cabling channels

� Backplane channel length is a misleading metric• Significant passband ripple, stub-related resonances, or crosstalk

coupling may make short links unworkable

• With proper material selection and design practices, 1 m links are feasible

� Backplane behavior may differ significantly from what is measured in the lab

• Ideal terminations are replaced with actual devices

• Temperature, humidity, and resin tolerances can cause variability in backplanes in the field – margin must be allocated

34 March 10, 2006 (r1.0)

Conclusions

� IEEE P802.3ap defines a comprehensive Physical Layer suite to support Ethernet as a backplane fabric

• Addresses the absence of an industry standard for Backplane Ethernet

• Provides the industry a roadmap to 10 Gb/s serial transmission over an electrical backplane

� Still IEEE P802.3ap is only a piece of an Ethernet fabric solution

• Improvements to congestion management are necessary to make Ethernet a viable storage or IPC solution

• Improvements to congestion management enhance networking and transport performance

Thank you!

2006 03 Healey · 2006. 3. 10. · Title: Microsoft PowerPoint - 2006_03_Healey.ppt Author: lokea...

Documents

Transcript of 2006 03 Healey · 2006. 3. 10. · Title: Microsoft PowerPoint - 2006_03_Healey.ppt Author: lokea...