READOUT LINK
description
Transcript of READOUT LINK
READOUT LINK
Filippo Costa
Filippo Costa - readout link 2
DDL
RORC
SLINK64CMC
FED
FRL
ROB
ROBINROS
ROL
DIUSLINK SIU
TELL1
Filippo Costa - readout link 3
DDL
RORC
SLINK64CMC
FED
FRL
ROB
ROBINROS
ROL
DIUSLINK SIU
TELL1
READOUTLINKS
Filippo Costa - readout link 4
SIU Source Interface Unit
CONVERTER CARDCMC Common Mezzanine CardHOLA High-speed Optical Link for AtlasTELL1 Trigger ELectronics Level 1DDL Detector Data Link
READOUT LINKROL Readout LinkSLINK64 Simple Link InterfaceGbE Gigabit EthernetRORC Readout Receiver Card
DAQ RECEIVER CARDROBIN Readout Buffer INput moduleFRL Front end Readout LinkNIC Network Interface Card
LEGEND
Filippo Costa - readout link 5
FEEConverter
cardDAQ receiver
cardreadout link
PC or crate(VME, Compact PCI)
General picture
Filippo Costa - readout link 6
Filippo Costa - readout link 7
Filippo Costa - readout link 8
Filippo Costa - readout link 9
Filippo Costa - readout link 10
IT’s
Filippo Costa - readout link 11
THE
Filippo Costa - readout link 12
SAME
Filippo Costa - readout link 13
THING(almost)
Filippo Costa - readout link 14
Filippo Costa - readout link 15
~2500 CHF
(~1300 per link)
Filippo Costa - readout link 16
~2500 CHF
(~1300 per link)
~3300 CHF
(~1100 per link)
Filippo Costa - readout link 17
~2500 CHF
(~1300 per link)
~3300 CHF
(~1100 per link)
1550 CHF
(~800 per link)
Filippo Costa - readout link 18
~2500 CHF
(~1300 per link)
~3300 CHF
(~1100 per link)
1550 CHF
(~800 per link)
> 6000 CHF
(~300 per link)
ALICE ATLAS CMS LHCb
Filippo Costa - readout link 19
# links protocol Throughput Flow control
Main buffering
ATLAS 1600 Slink optical 160/200 MB/s
Yes On board.3 x 64 MB
ALICE 500 DDL optical 200 MB/s Yes On the PC memory 8 GB
CMS 600 Slink64 copper 400 MB/s Yes On board.64 KB
LHCb 400 Ethernet 4 x 1Gb/s NO On board.128 KB
Filippo Costa - readout link 20
No general software tools to debug dataflow.
Filippo Costa - readout link 21
• S-Link is a that can be implemented in an FPGA (not resources demanding)
Filippo Costa - readout link 22
• Very robust link.• Cost is pretty low.
Filippo Costa - readout link 23
GOL
Filippo Costa - readout link 24
IN GENERAL
Filippo Costa - readout link 25
RUN 2
Filippo Costa - readout link 26
We are busy updating the experiment for you and will be back shortly
• Faster readout and higher data throughput for TPC and TRD.• Higher data throughput between HLT to DAQ.
Filippo Costa - readout link 27
• PROTOCOL UPGRADE:• faster DDL protocol from 2 Gb/s to 6 Gb/s.
• HARDWARE UPGRADE:• INPUT: from 2 ch @ 2Gb/s to 12 ch @ 6 Gb/s.• OUTPUT: from PCIe gen.1x4 to PCIe gen.2x8.
• Current RORC firmware implemented on new hardware.• Same software framework, only driver upgrade to handle the new
hardware.
Filippo Costa - readout link 28
We are busy updating the experiment for you and will be back shortly
• 64-bits PCI technology , obsolete and causes problems for upgrading DAQ PCs.
• Number of readout links will increase.• Reduce amount of RACK space.• Increase data throughput from ROS to HLT after LS1.
Filippo Costa - readout link 29
• Use of the ALICE CRORC, but is called ROBINNP.• HARDWARE UPGRADE:
• INPUT: from 3 ch @200 MB/s to 12 ch @ 200 MB/s (6Gb/s)• OUTPUT: from PCI 64 bit 66MHz to PCIe gen.2x8.• Bigger buffering capacity (up to 4 GB of DDR2/3).
• New firmware (the PPC present in the ROBIN will be moved out and implemented in software).
30
ALICE ATLAS
12 ch @ 6 Gb/s 12 ch @ 2 Gb/s (in average, if needed 6Gb/s)
No use of internal memory Up to 4 GB of DDR2/3 used to buffer events in the board
Same hardware – different features
Filippo Costa - readout link
Filippo Costa - readout link 31
PCI busPC memory, managed by PHYSMEM
Receiver DMA controller
Receive Address FIFO
DMA report #1
DMA report #2
DMA report #3
DMA report #4
DMA report #5
Buffer Descriptor #1
Buffer Descriptor #2
Buffer Descriptor #3
Buffer Descriptor #4
Buffer Descriptor #5
Receive Report FIFOFirmware
Software
D-RORC
DDL
Push
ALICE CRORC FIRMWARE
Filippo Costa - readout link 32
ATLAS ROBINNP firmware
Filippo Costa - readout link 33
We are busy updating the experiment for you and will be back shortly
• Hardware is aging (Myrinet) … needs for new components and replacements.
• New components are not compatible with the current hardware.• Faster readout and higher data throughput, needed by trigger and
HCAL.
Filippo Costa - readout link 34
Filippo Costa - readout link 35
• The existing DAQ receiver card (FRL) stays in place to support legacy FEDs.
• New FEROL card has 2 x 10 Gb/s and 2 x 6 Gb/s channels.• No CMC , but IPCORE, same interface to the FEE.• The new 6 Gb/s link has re-transmit feature.• 500 MB internal buffer.• 2 input links are treated independently and can share the
same 10 Gb/s.• The second link 10 Gb/s will be used to receive 10 Gb/s in
future (1 Input 1 Output)• 10 Gb/s reduced TCP/IP implemented in FPGA.
Filippo Costa - readout link 36
Filippo Costa - readout link 37
ABORT/RST
Filippo Costa - readout link 38
CONGESTION CONTROL
Filippo Costa - readout link 39
Filippo Costa - readout link 40
The don’t have special plan for the upgrade during LS1 …however …
Filippo Costa - readout link 41
Look (again) what LHCb has done
Filippo Costa - readout link 42
They have achieved goals that other experiments are planning in the future:
• Common readout unit (TELL1) used by all the detector groups.
• Use of Ethernet protocol.
Filippo Costa - readout link 43
RUN 3
Filippo Costa - readout link 44
LS2 Online Upgrade
FLP
DAQ and HLT
FLP
FLP
ITS
TRD
Muon
FTP
L0L1
FLPEMCal
FLPTPC
FLP
FLPTOF
FLPPHOS
Trigger Detectors
~ 2500 DDL3s 10 Gb/s (~1 TB/s in input)
L0
RORC3
RORC3
RORC3
RORC3
RORC3
RORC3
RORC3
RORC3
∞CLK
EVB farm
Filippo Costa - readout link 45
ALICE readout linkSIU3 RORC3
TBD
XILINX / ALTERA / ACTELFPGA
Custom DDL 10 Gb/sEthernet @ 10 Gb/s
PCIe over fibreGBT
Det. Read-Out FPGA
SIU IP CORE
Filippo Costa - readout link 46
DDL 3 (UDP Ethernet)
10 Gb/s embedded in the motherboard
XILINX VIRTEX 6
DDL optical fibre
SFP+ 10 Gb/s
Filippo Costa - readout link 47
DetectorReadoutElectronics
DCS
FPGAreadout
Physics DATA Physics DATA Physics DATA
Our proposal is to share the same physical link between the 2 data traffics: 99% PHYSICS data 1% or less, DCS monitoring data
DAQ+DCS proposal of sharing DDL3
data taking in progress…
DCSDATA
DCSDATA
DCSdata
Filippo Costa - readout link 48
Detector link Readout electronic
GBT TELL40
10 GbE (production and early tests)40 GbE, Infiniband or PCIe
Reduced TCP/IP or advanced UDP
Filippo Costa - readout link 49
CONGRATULATIONS TO ALL OF US !!!!
50
• ALICE, ATLAS, CMS and LHCb have different physics goals, but from the readout link point of view they try to achieve the same thing, move data from A to B. Why things are so different? Requirements and data throughput are not the same, but features could be adapted, instead writing them from scratch.
• Common effort to standardize the link and the electronics, ultimate goal:
• common readout unit for detector groups,• use of commercial hardware and industrial-standard protocols.
• How to increase the knowledge sharing between all the experiments groups? It is always a joy to share good experience with a protocol or a link, but maybe we could learn more from things that went wrong (and usually are not showed at conferences)
Some remarks …
Filippo Costa - readout link
Filippo Costa - readout link 51
Philippe Farthouat Dominique GigiBenedetto GoriniMarkus JoosNiko NeufeldChristoph SchwickPetr Zejdl