Common meeting of CERN DAQ teams
CERN May 3rd 2006Niko Neufeld PH/LBC for the LHCb Online team
Niko NeufeldCERN, PH
LHCb Trigger & DAQ
• Single arm spectrometer
• 1st level trigger in hardware (high pt)
• 2nd level trigger - full detector readout. (Search for secondary vertices)
Niko NeufeldCERN, PH
Readout Network
Detector
L0 FE L0 FE L0 FE L0 FE
VELO
L0 FE L0 FE L0 FE
ST OT RICH ECAL HCAL MUON
TELL1 TELL1 TELL1 UKL1 TELL1 TELL1 TELL1
SWITCH SWITCHSWITCH SWITCH SWITCH SWITCH SWITCH
CPU
CPU
CPU
CPU
CPU
CPU
CPU
CPU
CPU
CPU
CPU
CPU
CPU
CPU
CPU
CPU
CPU
CPU
CPU
CPU
CPU
CPU
CPU
CPU
TFC SYSTEM
L0 TRIGGER
L0 triggerLHC clock
Fron
t-End
CPU farm
MEP Requests
DAQ Architecture
• Installed in UX85 (everything but the storage)
• Based on Ethernet and TTC
• Readout Boards (TELL1/UKL1) send IP over Ethernet
Niko NeufeldCERN, PH
LHCb DAQ: fact-sheetMean total event-size 30 kB (zero-suppressed from MC)
Number of data sources approx. 310 with up to 4 x 1000 MBit/s outputs
Number of data sinks up to 2200 event-filter-farm nodes
DAQ network protocol Raw IP - sources pack several triggers into one packet. IP routing of packets (Layer-3)
Nominal readout rate 1 MHz Trigger rate. Typical packet rate per source: 100 kHz
Network topology Single core router with edge switches to farm nodes (2 hops between source and destination, second router can be added if required)
Network architecture Physically separate Ethernet LANs for Data and Controls. Separate VLANs for data from detector / to storage
Destination assignment Centrally by TFC system using TTC broadcasts (sources announce availability to TFC via Ethernet)
Dataflow regulation Push-protocol with central throttle mechanism
Niko NeufeldCERN, PH
Data-source: Common readout board TELL1
• Common board used by most sub-detectors in LHCb
• Up to 48 optical / copper analogue input links from the detector @ 1.6 GBit/s
• Preprocessing, zero-suppression and data-formatting in large FPGAs
• Data are sent directly to the Event Filter Farm via a 4-channel Gigabit Ethernet plugin-card (developed by PH/ED)
• A functionally equivalent board is used by the RICH (UKL1)
PP-FPGA
A-RxCard
L1B
SyncLink-FPGA
PP-FPGA
L1B
PP-FPGA
L1B
PP-FPGA
L1B
GigEcardTTCrx FEMECS
DAQ
TTCECS
FE FE FE FE
A-RxCard O-RxCard
Throttle
Niko NeufeldCERN, PH
Components
• Network:– DAQ core Force10 E1200
router (bought via IT frame contract)
– DAQ edge HP3400/3500 (bought via IT frame contract)
– Control (installed & operated by us too!). We use a lot of HP 2400 (Fast Ethernet with Gigabit Uplink) in the edge of the control network: save 2000 CHF compared to HP3400 - for a total 80 to 100 pieces!
• Timing and Fast Control (TFC) - dedicated, custom system – on top of TTC – central module:
Readout Supervisor (“Odin”)
• Farm– PC servers– Custom cooling– Rack powering (simple &
cheap custom distribution box co-developed with TS/EL)
Niko NeufeldCERN, PH
Main tasks this year
• Installation: network infrastructure (cabling done by IT/CS), racks, control-room
• Commissioning of the full readout network• Support for sub-detector installation and
commissioning• Pre-series farm installation• Testbeam (until November)
– *lots* of test-beam activity– test-beam is part of commissioning for some sub-detectors
Niko NeufeldCERN, PH
Installation / Purchase Planning
• By Q3/06: Finish all infrastructure installation (in particular for control system)
• By Q3/06: Install core DAQ network: Force10 E1200 router – 3 line-cards (= 270 1 Gig ports) – commission the readout system
• July/06: Market Survey for event-filter farm farm-nodes– October/06: Buy pre-series of ~ 150 nodes (dual-core AMD/Xeon
quad-core if available?)– December/06: Start installation of pre-series in Point 8
• Q4/06: Prepare Tender for main farm purchase in 2007– Q2/07: Ideally a blanket-contract with 2 - 3 companies (?)
• Q4/06: Ramp up E1200 router for full connectivity: approx 450 ports
• Q2/06 to Q4/06: specify, buy and install the storage system
Niko NeufeldCERN, PH
Event-filter-farm Node• LHCb requirements
– < 700 mm deep (due to old DELPHI racks) / 1 U– + a lot of obvious things (Linux supported, dual full-speed Gig Ethernet,
“proper” mechanics)• LHCb DAQ does not want to pay for
– redundancy (PS, disk)– rails– hard-disk (under discussion with LHCb Offline group who want local hard-
disks because of Tier-1 use during shutdown)• Open Questions:
– Which CPU (AMD / Intel, Dual-/Quad-core) - want to be open!– First criterion: MIPS/CHF– Second criterion: MIPS/Watt (only when we hit the power/cooling-limit of our
farm)– How much memory / core (we think 512 MB)– How to estimate the performance? Ideally we would like to tender a farm for
“1 MHz of LHCb triggers”– We are working on an “LHCb-live”-DVD, which allows manufacturers to do a
self-contained run of the LHCb trigger code - IT has recently done something similar (using SpecINT)
Niko NeufeldCERN, PH
Farm Installation
• Event filter farm in LHCb is installed in UX85– physically very close to the readout electronics
(maximal cable distance 36 m)– 50 racks with 44 x “1 U” slots available– 11 kW power and cooling available / rack
• Servers are mounted by being put on angles spacing between angles is 1.05 U: this makes us independent of vendor-specific rails, but required modification of the racks (see next slide)
• Installation of networking and computers done by DAQ team.
• Mechanics done by LHCb experimental area team under our supervision
Radiation P
rotection Wall
Floor D1: Event Filter
Farm
Floor D2: DAQ Core Network
Floor D3: ReadoutBoards
Detector
Niko NeufeldCERN, PH
Mounting of 1 U servers for Event Filter Farm
• Create a new Unit = 1U + 2mm (“LHCb-U”)• Make place for corner angles• 4 supporting bars / rack• Uses 23 real U to fit 22 new units
Niko NeufeldCERN, PH
• Outcome of a common Rack-Cooling project
• ~ 80 Computing racks (D1, D2 and SX8 server room) equipped with CIAT cooling doors.
• Each cooling door uses 3 big fans
• Used in slight variations by all expts
• In the LHCb setup can cool max. 12 kW
Rack cooling doors
Rack
PCs
Heat-exchanger
Fans
20-25 °C35-40 °C
20-25 °C
Niko NeufeldCERN, PH
Rack cooling in LHCb
• Most doors in Point 8 already installed
• Testing (water tightness) to start soon
• Bad surprise: MTBF for fans is 20000 hrs == 21/3 years 24/7 usage – LEP experience suggests life-
time of 4 years– preemptive change every 2 / 3 / 4
years?– look for higher quality
replacement?– buy more spares!
• Fan control - dedicated control board being developed (based on the ATLAS ELMB)
Niko NeufeldCERN, PH
DAQ Commissioning
• Commissioning of low-level hardware (racks) until June/06
• Installation & Commissioning of basic controls network in Point 8 from June/06 on
• In parallel (from July/06 on)– Installation and commissioning of TFC system (fibers, modules)– Installation, cabling up and commissioning of Readout boards:
validation of all central paths (DAQ, TTC, Control) done by central installation team
– Commissioning of data-path to the detector done by sub-detector teams (depends on long-distance cabling)
• Organization:– Followed up by weekly meetings of the whole Online team– Special regular meetings for installation issues
Niko NeufeldCERN, PH
LHCb’s wish-list for further information exchange
• Online data-base infrastructure:– centrally managed by IT? If so, where: in 513 at the Pit– locally managed? Which resources (hardware, software, configuration,
man-power) are foreseen
• Event-filter farm purchasing– interest in common technical specifications (with variants) or a common
MS?– maybe even a common blanket contract? (even though we are very different
in size, our farms are quite comparable, so we could get a better price!)
• Online computing management:– how to integrate with General Purpose Network (GPN) and Technical Network
(TN)– how to handle security – how to manage / monitor the network equipment?– how to boot, configure, monitor the servers (farm and others)? How and to which
extent are the CNIC tools used (CMF, Quattor)
Top Related