Post on 09-Feb-2021
Exceeding the Speed Limit
Challenges and solutions to the support of serial 10 Gigabit Ethernet over backplane interconnects
Adam Healey
Chair, IEEE P802.3ap Task Force
Consulting Member of Technical Staff, Agere Systems
March 10, 2006
2 March 10, 2006 (r1.0)
Before we begin…
� Per IEEE-SA Standards Board Operations Manual, January 2005: • At lectures, symposia, seminars, or educational courses, an individual
presenting information on IEEE standards shall make it clear that his or her views should be considered the personal views of that individual rather than the formal position, explanation, or interpretation of the IEEE.
� Note:• Currently, IEEE P802.3ap is under review per the IEEE 802.3 Working
Group ballot process and is subject to change.
• The work in this presentation is per IEEE P802.3ap Draft 2.1.
3 March 10, 2006 (r1.0)
Agenda
� Introduction to Backplane Ethernet
� Backplane Architecture and Reference Model
� 10GBASE-KR
� Forward Error Correction for 10GBASE-KR
� Closing Remarks
4 March 10, 2006 (r1.0)
Backplane Ethernet Profile
� “Ethernet in a box”
� Target Applications• Wireline and wireless access equipment
• Blade servers
• Enterprise switching
� Leverages field proven Ethernet controller and switching IP
� Reduces solution cost and complexity• Leverages Ethernet economies of scale
• Flattens the backplane fabric eliminating encapsulation protocols and associated processing
• Standard enables multi-vendor interoperability and facilitates use of COTS devices
5 March 10, 2006 (r1.0)
Specification Methodology
� Supply a device specification and not a backplane specification
� Be application agnostic (do no specify the backplane connector)
� Supply informative recommendations to assist backplane designers in identifying backplane channels that are interoperable with “Backplane Ethernet compliant” devices
6 March 10, 2006 (r1.0)
Agenda
� Introduction to Backplane Ethernet
� Backplane Architecture and Reference Model
� 10GBASE-KR
� Forward Error Correction for 10GBASE-KR
� Closing Remarks
7 March 10, 2006 (r1.0)
Backplane Architecture and Topology
B
N1
N2
H
“stub”
Blade may be a generic µP and memory complex with an I/O-specific mezzanine card
AC-coupling capacitor (at
receiver)
via
Backplane connector
A B BG C D DG E F FG G H HG
TX +/- RX +/- TX +/- RX +/-
Port 2
Hub Slot XChannel N
Hub Slot XChannel (N+1)
Port 0 Port 1
Port 3
Pin-out example
8 March 10, 2006 (r1.0)
Reference Model� The transmitter and receiver blocks include all off-chip components
associated with the respective block• Example, external AC-coupling capacitors, if required, are included in the
receiver block
� Channel consists of line card and backplane traces plus associated backplane and mezzanine connectors
package package
TP1 TP4
Transmitter ReceiverChannel
9 March 10, 2006 (r1.0)
Backplane Ethernet Objectives
AC or DC278722943212705 to 8 chassis/rack4AC or DC2101630555915202 to 3 chassis/rack4
Switch / Router
AC383812753310276Proposed worst-case3Blade Server
AC27871275331270Full mesh2AC24471022441020Example (dual-star)1
AdvancedTCA
AC / DC Coupling
No. Connectors
Total (mm)
H (mm)
B (mm)
N2 (mm)
N1 (mm)Description
� Support up to 1 m differential traces, including two connectors, on improved FR-4 printed circuit boards
� Support a BER of 10-12 or better
1 kundu_01_0504, 2 PICMG 3.0 R2.0, March 18, 2005, 3 koenen_01_0504, 4 goergen_01_0304
10 March 10, 2006 (r1.0)
Fitted Attenuation
� Defined to be the least-mean squares line fit to the insertion loss data within the frequency range f1 and f2
• The values of f1 and f2 are dependent on the PHY type of interest
� Fitted attenuation is compared to a limit based on a 1 m differential trace (W = 6 mil) on an improved FR-4 printed circuit board
� In effect, constrains the dielectric loss-length product of the channel
1000 1500 2000 2500 3000 3500 4000 4500 5000 5500 6000
0
5
10
15
20
25
30
Inse
rtio
n Lo
ss [
dB]
Frequency [MHz]
1000 1500 2000 2500 3000 3500 4000 4500 5000 5500 6000
0
5
10
15
20
25
30
Inse
rtio
n Lo
ss [d
B]
Frequency [MHz]1000 1500 2000 2500 3000 3500 4000 4500 5000 5500 6000
5
10
15
20
25
30
Inse
rtio
n Lo
ss [d
B]
Frequency [MHz]
Example #1 Example #2 Example #3
11 March 10, 2006 (r1.0)
Insertion Loss Limits
� Discourages encroachment of stub-related resonances into critical frequencies for the PHY type of interest
� Based on the fitted attenuation limit with allowances for passband ripple and stub-related resonances
0 5000 10000 15000
0
10
20
30
40
50
60
70
80
Inse
rtio
n Lo
ss [d
B]
Frequency [MHz]0 5000 10000 15000
0
10
20
30
40
50
60
70
80
Inse
rtio
n Lo
ss [d
B]
Frequency [MHz]0 5000 10000 15000
0
10
20
30
40
50
60
70
80
Inse
rtio
n Lo
ss [d
B]
Frequency [MHz]
Example #1 Example #2 Example #3
12 March 10, 2006 (r1.0)
Insertion Loss Deviation
� Defined to the be the difference between the channel insertion loss and the fitted attenuation in the frequency range f1 to f2
� In effect, constrains passband ripple due to impedance mismatches in the transmission path
1000 1500 2000 2500 3000 3500 4000 4500 5000 5500 6000-4
-3
-2
-1
0
1
2
3
4
Inse
rtio
n Lo
ss D
evia
tion
[dB
]
Frequency [MHz]
1000 1500 2000 2500 3000 3500 4000 4500 5000 5500 6000-4
-3
-2
-1
0
1
2
3
4
Inse
rtion
Los
s D
evia
tion
[dB
]
Frequency [MHz]1000 1500 2000 2500 3000 3500 4000 4500 5000 5500 6000-4
-3
-2
-1
0
1
2
3
4
Inse
rtion
Los
s D
evia
tion
[dB
]
Frequency [MHz]
Example #1 Example #2 Example #3
13 March 10, 2006 (r1.0)
Crosstalk
� Total crosstalk is defined to the be the power sum of the individual aggressors as measured at TP4
� Includes near-end crosstalk (NEXT) and far-end crosstalk (FEXT)
� Assumes the aggressors and victim are asynchronous and (or) uncorrelated
� Assumes the aggressors and victim are driven by identical sources
0 5000 10000 15000
0
10
20
30
40
50
60
70
80
Cro
ssta
lk L
oss
[dB
]
Frequency [MHz]0 5000 10000 15000
0
10
20
30
40
50
60
70
80
Cro
ssta
lk L
oss
[dB
]
Frequency [MHz]0 5000 10000 15000
0
10
20
30
40
50
60
70
80
Cro
ssta
lk L
oss
[dB
]
Frequency [MHz]
Example #1 Example #2 Example #3
14 March 10, 2006 (r1.0)
Insertion Loss to Crosstalk Ratio (ICR)
� Defined to be the least-mean squares line fit to the difference of the total crosstalk loss and the insertion loss (both in dB) in the frequency range fa to fb
� Limits the total crosstalk seen at TP4
� Lower loss links are allowed higher crosstalk and vice versa
100 1000 100000
5
10
15
20
25
30
35
40
45
50
Inse
rtio
n Lo
ss t
o C
ross
talk
Rat
io [d
B]
Frequency [MHz]100 1000 100000
5
10
15
20
25
30
35
40
45
50
Inse
rtio
n Lo
ss t
o C
ross
talk
Rat
io [d
B]
Frequency [MHz]100 1000 100000
5
10
15
20
25
30
35
40
45
50
Inse
rtio
n Lo
ss t
o C
ross
talk
Rat
io [d
B]
Frequency [MHz]
Example #1 Example #2 Example #3
15 March 10, 2006 (r1.0)
0 0.25 0.5 0.75 10
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1
Frequency x Differential Skew
Gai
n (W
/W)
Differential Skew
� To isolate the impact of skew, render the
and sides of the differential pair as delay T1 and T2 respectively
� The skew is the magnitude of the difference between the two delays, |T1−T2|
� The differential skew from TP1 to TP4 is recommended to be less than 24 ps (10GBASE-KR)
1T
2T
2iv
2iv−
ov
( )( )( )122
cos121
)()(
TTVV
i
o −+= ωωω
16 March 10, 2006 (r1.0)
Environmental Variation
� Channel recommendations are to be satisfied over the full range of system operating conditions
0 1000 2000 3000 4000 5000 6000
0
5
10
15
20
25
30
35
40
Inse
rtion
loss
(dB
)
Frequency (MHz)
20oC, 20% RH40oC, 20% RH60oC, 20% RH20oC, 85% RH40oC, 85% RH60oC, 85% RH
Insertion loss variation due totemperature and humidity
17 March 10, 2006 (r1.0)
Impact of Device Terminations
( )( )���
����
�
ΓΓ+ΓΓ−Γ−Γ−Γ−Γ+=
221121122211
21
111
2 SSSSSSS
vv
LSLSLS
SL
i
o
1 11
2
SZ
LZ
��
�
�
2221
1211
SS
SS
SΓ LΓoviv
2 2
Transmitter Receiver
0 1000 2000 3000 4000 5000 6000 7000 8000 9000 10000
0
5
10
15
20
25
30
Inse
rtion
loss
(dB
)
Frequency (MHz)
i
o
vv
2
21S
18 March 10, 2006 (r1.0)
Agenda
� Introduction to Backplane Ethernet
� Backplane Architecture and Reference Model
� 10GBASE-KR
� Forward Error Correction for 10GBASE-KR
� Closing Remarks
19 March 10, 2006 (r1.0)
10GBASE-KR Architecture
� 10 Gb/s connection between two media access controllers over an electrical backplane
� Low-overhead 64B/66B encoding
� Statistical transition density and DC balance (scrambled)
� Signaling speed is 10.3125 Gbaud
� Optional Forward Error Correction
� Transmitter equalizer programmed by the link partner during start-up
� Adaptive receiver equalization
MAC
MAC CONTROL (OPTIONAL)
LLC
HIGHER LAYERS
10GBASE-KRPHY
RECONCILIATION
64B/66B PCS
XGMII *
PMA
AN
MEDIUM
PMD
FEC*
* Optional
20 March 10, 2006 (r1.0)
Transmit Equalizer Signal Shaping
pkV
pkV−
0
0c1c 1−c
0c1c 1−c
0c1c 1−c
0c1c 1−c
preV
ssV
pstV
T T
101 −+−−= cccVpre
101 −++−= cccVpst
101 −++= cccVss
101 −++= cccVpk
Assumes that the implementation is a three-tap transversal filter with unit delay.Assumes that the implementation is a three-tap transversal filter with unit delay.
21 March 10, 2006 (r1.0)
Receive Equalizer
� Requirements developed under the assumption of a 5-tap decision feedback equalizer
� However, the implementation of this architecture is not necessarily required for compliance
� Instead, the receiver in question must exhibit an expected level of performance as established via the interference tolerance test
� The interference tolerance test examines the ability of the receiver to equalize a high-loss channel in the presence of horizontal (jitter) and vertical (noise) interference
22 March 10, 2006 (r1.0)
Interference Tolerance Testing
Pattern generator
Clock source
Frequency synthesizer
Frequency-dependent attenuator
Interference injection
Device under test
Interference generator
Test channel
Transmitter control
Modulation input
Equalizer
Introduces static timing offset and jitter
Additive sinusoid swept in frequency and amplitudeInsertion loss emulates
worst-case attenuation
Transmitter signal shaping controlled by adaptation algorithm in the DUT
23 March 10, 2006 (r1.0)
10GBASE-KR Link Model
-0.5 0 0.5 1 1.5-0.5
-0.4
-0.3
-0.2
-0.1
0
0.1
0.2
0.3
0.4
0.5
Am
plitu
de [a
u]
Time Offset [UI]
Am
plitu
de (a
u)
Time Offset (baud)-0.5 -0.4 -0.3 -0.2 -0.1 0 0.1 0.2 0.3 0.4 0.5
-2.5
-2
-1.5
-1
-0.5
0
0.5
1
1.5
2
2.5
Transmitter output eye
Slicer input eye
��
���
� −ΠTkTt
tA
)( fHtkx
)(zF
)( fHr
)(zD
)(zCkx̂
M↓MT
kts +
)( fGr
kε
)(0 fH
10GBASE-KR Performance Simulations
01
01
σσ +−= AASNR
1A
0A
1σ
0σ
Slicer Input Eye: Test Case D
Nor
mal
ized
Am
plitu
de
Time Offset [UI]-0.5 -0.4 -0.3 -0.2 -0.1 0 0.1 0.2 0.3 0.4 0.5
-2
-1.5
-1
-0.5
0
0.5
1
1.5
2
Required SNR for 1E-12 Bit Error Ratio (BER)
Loss-limited case
Stub-limited case
Crosstalk-limited cases
25 March 10, 2006 (r1.0)
10GBASE-KR Start-Up Protocol
� Optimizes transmitter FIR
� Automatic power control• Receiver may steer the transmitter output voltage to the minimum
level required for acceptable performance
• May also mitigate crosstalk
� Optimize receiver equalizer• Joint adaptation of transmitter and receiver yields superior solution
to independent adaptation
� Accomplished through the exchange of fixed-length training frames
26 March 10, 2006 (r1.0)
Training Frame Structure
Frame marker
Coefficient update
Status report
Training pattern
4
16
16
512
Octets
c(1) c(0) c(−1)c(1) c(0) c(−1) RRUG Reserved Reserved
Update gain11 = 1X10 = 2X01 = 4X00 = 8X
Coefficient update11 = reserved10 = increment01 = decrement00 = hold
Receiver Ready1 = Training complete0 = Training in progress
Coefficient status11 = maximum10 = minimum01 = updated00 = not_updated
15 0 15 0
Control channel is differential Manchester encoded at one-quarter of the 10GBASE-KR signaling speed (one octet per control bit)
Two cycles of a PRBS-11 pattern (seeded with all ones) padded with two zeros
0xFFFF0000 (does not appear in control channel or training pattern)
27 March 10, 2006 (r1.0)
Timing Diagram
Device B
ReceiverRdy = 0
Device A
Auto-Negotiation
ReceiverRdy = 0 ReceiverRdy = 1
ReceiverRdy = 1
Equalizer Training Period
Equalizer Training Period
wait_timer
wait_timer
IDLE and DATA
IDLE and DATATraining Frames Training Frames
Training Frames Training Frames
28 March 10, 2006 (r1.0)
Agenda
� Introduction to Backplane Ethernet
� Backplane Architecture and Reference Model
� 10GBASE-KR
� Forward Error Correction for 10GBASE-KR
� Closing Remarks
29 March 10, 2006 (r1.0)
Forward Error Correction for 10GBASE-KR
� FEC improves 10GBASE-KR link performance at the expense of added complexity and latency
� Not required to meet objectives
� Burst error correction especially suitable for DFE-based receiver architecture
• Counter-acts error propagation
� Signaling speed is not altered by FEC
MAC
MAC CONTROL (OPTIONAL)
LLC
HIGHER LAYERS
10GBASE-KRPHY
RECONCILIATION
64B/66B PCS
XGMII *
PMA
AN
MEDIUM
PMD
FEC*
30 March 10, 2006 (r1.0)
FEC Frame Structure
� Transcode 66B blocks into FEC frame (32 blocks/frame)• T-bit is the second bit of the 66B sync header (0 = control, 1 =
data)
� Append 32 parity bits (total frame length is 2112 bits)
� Transmission order is left-to-right and top-to-bottomT0 64-bit payload word 0 T1 64-bit payload word 1 T2 64-bit payload word 2 T3 64-bit payload word 3T4 64-bit payload word 4 T5 64-bit payload word 5 T6 64-bit payload word 6 T7 64-bit payload word 7
T8 64-bit payload word 8 T9 64-bit payload word 9 T10 64-bit payload word 10 T11 64-bit payload word 11
T12 64-bit payload word 12 T13 64-bit payload word 13 T14 64-bit payload word 14 T15 64-bit payload word 15
T16 64-bit payload word 16 T17 64-bit payload word 17 T18 64-bit payload word 18 T19 64-bit payload word 19
T20 64-bit payload word 20 T21 64-bit payload word 21 T22 64-bit payload word 22 T23 64-bit payload word 23
T24 64-bit payload word 24 T25 64-bit payload word 25 T26 64-bit payload word 26 T27 64-bit payload word 27
T28 64-bit payload word 28 T29 64-bit payload word 29 T30 64-bit payload word 30 T31 64-bit payload word 31
32 parity bits
31 March 10, 2006 (r1.0)
FEC Encoder and Decoder
� Parity bits computed using a shortened cyclic code (2112, 2080)
� Frame is then scrambled
� Approximately 2 dB coding gain
� Guaranteed error correction for bursts up to 11 bits in length
� Complexity estimated to be 14K gates plus a 64 x 33 dual port RAM at 312.5 MHz
� Latency estimated to be 200 ns
32 March 10, 2006 (r1.0)
Agenda
� Introduction to Backplane Ethernet
� Backplane Architecture and Reference Model
� 10GBASE-KR
� Forward Error Correction for 10GBASE-KR
� Closing Remarks
33 March 10, 2006 (r1.0)
Observations
� Backplane channels are not as predictable or easily classified as cabling channels
� Backplane channel length is a misleading metric• Significant passband ripple, stub-related resonances, or crosstalk
coupling may make short links unworkable
• With proper material selection and design practices, 1 m links are feasible
� Backplane behavior may differ significantly from what is measured in the lab
• Ideal terminations are replaced with actual devices
• Temperature, humidity, and resin tolerances can cause variability in backplanes in the field – margin must be allocated
34 March 10, 2006 (r1.0)
Conclusions
� IEEE P802.3ap defines a comprehensive Physical Layer suite to support Ethernet as a backplane fabric
• Addresses the absence of an industry standard for Backplane Ethernet
• Provides the industry a roadmap to 10 Gb/s serial transmission over an electrical backplane
� Still IEEE P802.3ap is only a piece of an Ethernet fabric solution
• Improvements to congestion management are necessary to make Ethernet a viable storage or IPC solution
• Improvements to congestion management enhance networking and transport performance
Thank you!