Fast Readout Links - indico.cern.ch

23 June 201123 June 2011 [email protected]@cern.ch 00

Fast Readout Links Fast Readout Links

Optical interconnects by network applicationOptical interconnects by network application

Comparison with developments for LHC and HLComparison with developments for LHC and HL-- LHCLHC

What to expect for future High Energy Physics What to expect for future High Energy Physics experimentsexperiments

F. Vasey, CERN, PH-ESE


Optical Link MetricsOptical Link Metrics

BandwidthBandwidth

Energy (Energy (pJpJ/bit, /bit, mW/GbpsmW/Gbps))

1mW/Gbps = 1pJ/bit1mW/Gbps = 1pJ/bit

1 1 googlegoogle search = 1kJ search = 1kJ

1 1 PbitsPbits (@1pJ/bit)(@1pJ/bit) !!!!!!

Density (mmDensity (mm22/Gbps)/Gbps)

Cost ($/Cost ($/GbpsGbps))

Environmental ResistanceEnvironmental Resistance


Use and Misuse of Scaled MetricsUse and Misuse of Scaled Metrics

mW/GbpsmW/Gbps, mm, mm22/Gbps, $//Gbps, $/GbpsGbps

Systematically favour very high bandwidth Systematically favour very high bandwidth systemssystems

Cannot be used to extrapolate likely figures for Cannot be used to extrapolate likely figures for bandwidthbandwidth--distributed systemsdistributed systems

Per device (Per device (TRxTRx) scaling more appropriate) scaling more appropriate

mW/TRxmW/TRx, mm, mm22/TRx, $//TRx, $/TRxTRx

Nevertheless very useful to compare systemsNevertheless very useful to compare systems


Existing Optical Links in CMS (1)Existing Optical Links in CMS (1)

CMS Tracker AOHCMS Tracker AOH

39k links39k links

40MS/s x 8 bits = 320 Mbps40MS/s x 8 bits = 320 Mbps

100 100 mWmW, 3 channels , 3 channels (dependent on laser bias)(dependent on laser bias)

30mm x 23mm, 3 channels30mm x 23mm, 3 channels

300 CHF / channel300 CHF / channel

CMS ECAL GOHCMS ECAL GOH

10k links10k links

800 Mbps800 Mbps

GOL operates up to 1600 GOL operates up to 1600 MbpsMbps

Aggregate bandwidth ~ Tk: 12.5Tbps

ECAL: 8Tbps


Existing Optical Links in CMS (2)Existing Optical Links in CMS (2)

AOHAOH19971997

Bandwidth Bandwidth GbpsGbps 0.320.32

Reach mReach m 100100

Energy Energy pJpJ/bit, /bit, mW/GbpsmW/Gbps 150 150 * extrapolating to * extrapolating to bidibidi linklink

Density mmDensity mm22/Gbps/Gbps 10001000* extrapolating to * extrapolating to bidibidi linklink

Cost $/Cost $/GbpsGbps 900900


Interconnects by Network ApplicationInterconnects by Network Application

Chip to ChipChip to Chip 1 1 –– 50 cm50 cm

Data CenterData Center(Rack to Rack)(Rack to Rack) 11--100 Meters100 Meters

EthernetEthernetInfiniBandInfiniBand

FibreFibre Channel / SASChannel / SASProprietaryProprietary

Board to BoardBoard to Board ~0.5 ~0.5 --1.5M1.5M

MemoryMemoryMemoryMemory MemoryMemoryMemoryMemory

MemoryMemory

TeraTera--scale CPUscale CPU

MemoryMemory

CPU 2CPU 2

Metro &Metro &Long HaulLong Haul(0.1(0.1--80km)80km)

SONETSONETWDMWDM

EthernetEthernet

IntraIntra--RackRackEthernetEthernetInfiniBandInfiniBandFibreFibre Channel / SASChannel / SASPCIePCIeProprietaryProprietary

BackplaneBackplane

100s of100s ofThousandsThousands

10s of10s ofMillionsMillions

10s of10s ofBillionsBillions

Volume100s of100s ofMillionsMillions


Wide Area Network (WAN)Wide Area Network (WAN)







MemoryMemory


MemoryMemory

CPU 2CPU 2


SONETSONETWDMWDM

EthernetEthernet


BackplaneBackplane






Bandwidth and CapacityBandwidth and Capacity

TDM

WDM

Bits/Symbol

Bandwidth Bandwidth

#Symbols/s x bits/symbol x ##Symbols/s x bits/symbol x #

Only one fibre considered hereOnly one fibre considered here

Capacity = Reach x BandwidthCapacity = Reach x Bandwidth

Reach not an issue in typical Reach not an issue in typical detector applications (~100m)detector applications (~100m)

Only bandwidth optimization Only bandwidth optimization considered hereconsidered here

QPSK 8-PSK

16-QAM

c


WAN Network featuresWAN Network features

Ultimate CapacityUltimate Capacity

Design for minimum Design for minimum received energyreceived energy

Exploit TDM, WDM, Exploit TDM, WDM, Advanced modulation Advanced modulation formatsformats

Single ModeSingle Mode

Electronic Mitigation Electronic Mitigation Techniques Techniques

Ultra fast electronics Ultra fast electronics operating at line rate (operating at line rate (f.if.i. . 50GS/s ADC)50GS/s ADC)

FEC, equalizers, FEC, equalizers, ……

Metrics (very coarse guess)Metrics (very coarse guess)

Bandwidth: 10TbpsBandwidth: 10Tbps

Reach: 100kmReach: 100km

Energy: 10Energy: 10’’000 000 pJpJ/bit/bit

Density: 10Density: 10’’000 mm000 mm22/Gbps/Gbps

Cost: 100$/GbpsCost: 100$/Gbps

Not in our ballpark,

but watch out for emerging technologies


Access NetworkAccess Network







MemoryMemory


MemoryMemory

CPU 2CPU 2


SONETSONETWDMWDM

EthernetEthernet


BackplaneBackplane






Access over Passive Optical NetworkAccess over Passive Optical NetworkFTTH

FTTC

FTTB

GPON, EPON 1.25Gbps GPON, EPON 1.25Gbps -- 2Gbps2Gbps

Consumes 18x Consumes 18x lessless energyenergy per user per user thanthan VDSL2VDSL2

Sharing of Sharing of thisthis capacitycapacity amongamong multiple multiple usersusers (time(time--division division multiplexingmultiplexing))

Distance to centralDistance to central--office office cancan go up to 60km go up to 60km whenwhen amplifieramplifier--assistedassisted, , not a few 100s not a few 100s metersmeters fromfrom setset--top box to DSLAM.top box to DSLAM.

10GPON solutions 10GPON solutions recentlyrecently proposedproposed by system by system vendorsvendors

WDM dimension WDM dimension cancan alsoalso bebe exploitedexploited to to increaseincrease capacitycapacity,,

And And provideprovide Point to Point connections, Point to Point connections, capacitycapacity on on demanddemand……whilewhile preservingpreserving scalabilityscalability


PON Access Network featuresPON Access Network features

Moderate speed, reachModerate speed, reach

ScalabilityScalability

Shared resourceShared resource

High dynamic rangeHigh dynamic range

High split ratioHigh split ratio

High Volume, Low CostHigh Volume, Low Cost

Single Mode, bidirectionalSingle Mode, bidirectional

10G PON coming up, WDM PON 10G PON coming up, WDM PON nextnext

Metrics (coarse Metrics (coarse guestimatesguestimates))

Bandwidth: 2.5GbpsBandwidth: 2.5Gbps

Reach: 10kmReach: 10km

Energy: 400pJ/bitEnergy: 400pJ/bit

Density: 200mmDensity: 200mm22/Gbps/Gbps


Not in our ballpark,

Traffic flows in wrong direction

But high volume and low cost model in right bandwidth range


Local Area Network (LAN)Local Area Network (LAN)







MemoryMemory


MemoryMemory

CPU 2CPU 2


SONETSONETWDMWDM

EthernetEthernet


BackplaneBackplane






40/100GbE based on 10G 40/100GbE based on 10G PHYsPHYs

10G: directly modulated EE lasers or 10G: directly modulated EE lasers or VCSELsVCSELs

25G: externally modulated lasers25G: externally modulated lasers

40G40G--100G: aggregation100G: aggregation


LAN Network FeaturesLAN Network Features

converging with MANconverging with MAN

TDM limits reached TDM limits reached

Aggregation via parallelismAggregation via parallelism

PhysicalPhysical

Wavelength Wavelength

Inheriting WAN featuresInheriting WAN features

High DensityHigh Density

Low PowerLow Power

High Volume Low CostHigh Volume Low Cost

Multimode OM3 & OM4 fibreMultimode OM3 & OM4 fibre

10G over 100m & 300m10G over 100m & 300m

Single Mode nextSingle Mode next

Metrics (Metrics (guestimatesguestimates))

Bandwidth: 100GbpsBandwidth: 100Gbps

Reach: 100mReach: 100m

Energy: 10pJ/bitEnergy: 10pJ/bit

Density: 2mmDensity: 2mm22/Gbps/Gbps



From modules to chips: packaging issuesFrom modules to chips: packaging issues


Link Type Power Efficiency (pJ/bit)

Distance

Electrical 3 < 1 mEOE 7 < 100 m

Yearm

W/G

b/s

Electrical Link Power Efficiency Progress

20002002

20042006

20082010

05

1015202530354045

Electrical Interconnects are still alive Electrical Interconnects are still alive

For short distances, EOE For short distances, EOE cannot beat EEcannot beat EE

Short range low power Short range low power electrical links are the electrical links are the subject of intense subject of intense development effortsdevelopment efforts

Moving targetMoving target


Optical network metricsOptical network metrics A risky comparison* (1)A risky comparison* (1)

AOHAOH19971997

WANWAN19901990

AccessAccess20002000

LANLAN20102010

Bandwidth Bandwidth GbpsGbps

0.320.32 1010’’000000 2.52.5 100100

Reach Reach mm

100100 100100’’000000 1010’’000000 100100

Energy Energy pJpJ/bit/bit

150 150 1010’’000000 400400 1010

Density Density mmmm22/Gbps/Gbps

10001000 1010’’000000 200200 22

Cost Cost $/$/GbpsGbps

900900 100100’’ss 1010 11

*This comparison expresses the views of the author at presentation time and do not necessarily represent an accurate vision of the current and future situation Arrays


Next for HEP: Front End VersatileNext for HEP: Front End Versatile--TRxTRx for HLfor HL--LHCLHC

Low Mass & VolumeLow Mass & Volume

Minimize material, avoid metalsMinimize material, avoid metals

Capable of operating in a magnetic Capable of operating in a magnetic fieldfield

Requires replacement of ferrite bead used in Requires replacement of ferrite bead used in laser bias networklaser bias network

Radiation TolerantRadiation Tolerant

BitrateBitrate determined by determined by ASICsASICs: 5 : 5 –– 10 10 GbpsGbps

850nm and 1310nm 850nm and 1310nm flavoursflavours

TOSA

ROSA

LDD

GBTIA

VTRx


Front End VersatileFront End Versatile--TRxTRx Results at 5GbpsResults at 5Gbps

TxTxSM@1310nmSM@1310nm MM@850nmMM@850nm

RxRx



AOHAOH19971997

WANWAN19901990


LAN R2RLAN R2R20102010

VTRxVTRx20082008


0.320.32 1010’’000000 2.52.5 100100 55

Reach Reach mm

100100 100100’’000000 1010’’000000 100100 100100


150 150 1010’’000000 400400 1010 100100


10001000 1010’’000000 200200 22 100100


900900 100100’’ss 1010 11 60 ?60 ?

*This comparison expresses the views of the author at presentation time and do not necessarily represent an accurate vision of the current and future situation

Arrays


A A TRxTRx for an allfor an all--inin--one moduleone module

50% scaling required50% scaling required

SizeSize

powerpower



AOHAOH19971997

WANWAN19901990


LAN R2RLAN R2R20102010

VTRxVTRx2008 2008 1212


0.320.32 1010’’000000 2.52.5 100100 55

Reach Reach mm

100100 100100’’000000 1010’’000000 100100 100100


150 150 1010’’000000 400400 1010 100100

5050


10001000 1010’’000000 200200 22 100100

5050


900900 100100’’ss 1010 11

*This comparison expresses the views of the author at presentation time and do not necessarily represent an accurate vision of the current and future situation

Arrays


LetLet’’s not forget radiation tolerances not forget radiation tolerance

Our daily breadOur daily bread


Radiation Tolerance: Is there a limit?Radiation Tolerance: Is there a limit?

Active Active optoopto devices tested up devices tested up to now do not to now do not resist resist fluencesfluences beyond 10beyond 1016 16 cmcm--22

Still OK for SLHC Still OK for SLHC TkTk

Tighter marginsTighter margins

Are there Are there alternatives for alternatives for fluencesfluences beyond beyond 101016 16 cmcm--2 2 ? ?

(Re)(Re)--consider consider Passives ?Passives ?

modulatorsmodulators


Extrapolating I/O Bandwidth RequirementsExtrapolating I/O Bandwidth Requirements

Need to go optical, on chip, with WDM · Si – CMOS Photonics


Evolution of high speed interconnects Evolution of high speed interconnects 20102010--20152015

Base circuit board withoptical waveguides & turning mirrors

Lens array

CPU CMOS TRX and SLC

OE50mm Organic Module

CPU

Base circuit boardPower and slow signals

HSI

Electrical or Optical engines + Opto PCB MMPIC TRx

20152015--20202020CPU

Base circuit boardPower and slow signals

Monolithic integration of TRx and Processing chip SM

OR

Will this paradigm shift occur ?

Will it bring us benefits

(density, power, radition tolerance) ?


ConclusionsConclusions

Broadband optical communication is a commodity and is ubiquitousBroadband optical communication is a commodity and is ubiquitous

Also in HEPAlso in HEP

Scaling bandwidth and connectivity remains a major issue at all Scaling bandwidth and connectivity remains a major issue at all levels of the levels of the network hierarchynetwork hierarchy

HEP has a distributed bandwidth need. It cannot simply aggregatHEP has a distributed bandwidth need. It cannot simply aggregate.e.

Current optical communication technologies will not allow to meeCurrent optical communication technologies will not allow to meet the challenges t the challenges ahead without major breakthroughs:ahead without major breakthroughs:

Advanced modulation, coding and electronic mitigation schemesAdvanced modulation, coding and electronic mitigation schemes

High density parallel and wavelength multiplexed optics (opticalHigh density parallel and wavelength multiplexed optics (optical engines)engines)

Photonic integration with CMOSPhotonic integration with CMOS

Pragmatically speaking, all incremental electrical chip to chip Pragmatically speaking, all incremental electrical chip to chip interconnection interconnection approaches will be exhausted before fast interconnects go opticaapproaches will be exhausted before fast interconnects go opticall

Will this kill photonics integration?Will this kill photonics integration?

In the short/medium term: laserIn the short/medium term: laser--based EOE conversion such as proposed by the based EOE conversion such as proposed by the Versatile Link common project is the way forwardVersatile Link common project is the way forward

10Gbps, 1010Gbps, 10’’s s pJpJ/bit, 10/bit, 10’’s mms mm22/Gbps/Gbps

In the long term:In the long term:

Worth watching photonics integrationWorth watching photonics integration

Embracing a new technology is resource intensiveEmbracing a new technology is resource intensive

Fast Readout Links - indico.cern.ch

Documents

Transcript of Fast Readout Links - indico.cern.ch