Architecture and implementation of the ALICE Data-Acquisition system

Architecture and implementation

of the ALICE Data-Acquisition system

T. Anticic, F. Carena, W. Carena, S. Chapeland, O. Cobanoglu, E. Dénes, R. Divià, U. Fuchs, I. Makhlyueva, J.C. Marin,

F. Ozok, K. Schossmaier, C. Soós, P. Vande Vyvre, A. Vascotto, S. Vergara

for the ALICE collaboration

CHEP 06 – 13/17 Feb. 2006

CHEP 06 – Architecture and Implementation of the ALICE Data-Acquisition system

ALICE DAQ

– 17 detectors

– Pb-Pb

– pp and pA

• Big events (86.5 MB)• Large bandwidth (1.25 GB/s to tape)• Low interaction rate (10 KHz)• Complex triggers• Different sets of detectors

• Small events (2.5 MB)• High interaction rate (200 KHz)• Less bandwidth• Simpler triggers

• Synchronized / standalone tuning

25 GB/s raw from the detectors => 1.25 GB/s data files

Flexibility


ALICE DAQ Architecture

GDC GDCGDCGDC

CTP

LTU

TTC

FERO FERO

LTU

TTC

FERO FERO

LDCLDC

BUSY BUSY

Rare/All

Storage Network

TDS

343 DDL

EDM

LDCLoad Bal.

LDC LDC

HLT Farm

FEPFEP

DDL

H-RORC

10 DDL

10 D-RORC

10 HLT LDC

163 DDL

TDS

DS DS

Event-Building Network

425 D-RORC

50 GDC 10 DS

PDS

25 TDS

200 Detector LDC

EventFragment

Sub event

Event

File


ALICE DAQ Architecture

GDC GDCGDCGDC

CTP

LTU

TTC

FERO FERO

LTU

TTC

FERO FERO

LDCLDC

Storage Network

TDS

EDM

LDC LDC LDC

HLT Farm

FEPFEP

TDS

DS DS


PDS

Trigger High-Level Trigger

Detectors

Data Acquisition


Trigger

GDC GDCGDCGDC

CTP

LTU

TTC

FERO FERO

LTU

TTC

FERO FERO

LDCLDC

BUSY BUSY

Rare/All

Storage Network

TDS

EDM

LDC LDC LDC

HLT Farm

FEPFEP

TDS

DS DS


PDS

Initiate detectors readout of selected events


Trigger

GDC GDCGDCGDC

CTP

LTU

TTC

FERO FERO

LTU

TTC

FERO FERO

LDCLDC

BUSY BUSY

Rare/All

Storage Network

TDS

EDM

LDC LDC LDC

HLT Farm

FEPFEP

TDS

DS DS


PDS

Central Trigger Processor: decisions


Trigger

GDC GDCGDCGDC

LTU LTU

CTP

TTC

FERO FERO

TTC

FERO FERO

LDCLDC

BUSY BUSY

Rare/All

Storage Network

TDS

EDM

LDC LDC LDC

HLT Farm

FEPFEP

TDS

DS DS


PDS

Local Trigger Unit: receives decisions for each detector


Trigger

GDC GDCGDCGDC

TTC TTC

CTP

LTU

FERO FERO

LTU

FERO FERO

LDCLDC

BUSY BUSY

Rare/All

Storage Network

TDS

EDM

LDC LDC LDC

HLT Farm

FEPFEP

TDS

DS DS


PDS

Timing, Trigger and Control system: broadcast to electronics


Trigger

GDC GDCGDCGDC

CTP

LTU

TTC

LTU

TTC

FERO FERO FERO FERO

LDCLDC

BUSY BUSY

Rare/All

Storage Network

TDS

EDM

LDC LDC LDC

HLT Farm

FEPFEP

TDS

DS DS


PDS

Front-End Readout Electronics: use triggers, signal BUSY


Trigger

GDC GDCGDCGDC

CTP

LTU

TTC

FERO FERO

LTU

TTC

FERO FERO

LDCLDC

BUSY BUSY

Rare/All

Storage Network

TDS

EDM

LDC LDC LDC

HLT Farm

FEPFEP

TDS

DS DS


PDS

Some DAQ components can feed back information


Data Acquisition

CTP

LTU

TTC

FERO FERO

LTU

TTC

FERO FERO 343 DDL

Load Bal.

HLT Farm

FEPFEP

163 DDL

GDC GDCGDCGDC

LDCLDC

Storage Network

TDS

EDM

LDC LDC LDC

TDS

DS DS


425 D-RORC

50 GDC 10 DS

PDS

25 TDS

200 Detector LDC

EventFragment

Sub event

Event

File

Data-flow from electronics to permanent storage


Data Acquisition

CTP

LTU

TTC

FERO FERO

LTU

TTC

FERO FERO 343 DDL

HLT Farm

FEPFEP

163 DDL

GDC GDCGDCGDC

LDCLDC

Storage Network

TDS

EDM

LDC LDC LDC

TDS

DS DS


PDS

EventFragment

Detector Data Links: optical transmission


Data Acquisition

CTP

LTU

TTC

FERO FERO

LTU

TTC

FERO FERO HLT Farm

FEPFEP

GDC GDCGDCGDC

LDCLDC

Storage Network

TDS

EDM

LDC LDC LDC

TDS

DS DS


425 D-RORC

PDS

EventFragment

DAQ Read-Out Receiver Card: PCI interface


Data Acquisition

CTP

LTU

TTC

FERO FERO

LTU

TTC

FERO FERO HLT Farm

FEPFEP

LDCLDCLDC

GDC GDCGDCGDC

Storage Network

TDS

EDM

LDC LDC

TDS

DS DS


PDS

200 Detector LDCSub event

Local Data Concentrators: ship sub-events

EventFragment


Data Acquisition

CTP

LTU

TTC

FERO FERO

LTU

TTC

FERO FERO HLT Farm

FEPFEP

GDC GDCGDCGDC

LDCLDC

Storage Network

TDS

EDM

LDC LDC LDC

TDS

DS DS


PDS

Sub event

Event

Global Data Collectors: build events

50 GDC


Data Acquisition

CTP

LTU

TTC

FERO FERO

LTU

TTC

FERO FERO

Load Bal.

HLT Farm

FEPFEP

EDM

GDC GDCGDCGDC

LDCLDC

Storage Network

TDS

LDC LDC LDC

TDS

DS DS


PDS

Event Distribution Manager: load balancing


Data Acquisition

CTP

LTU

TTC

FERO FERO

LTU

TTC

FERO FERO HLT Farm

FEPFEP

TDS TDS

GDC GDCGDCGDC

LDCLDC

Storage Network

EDM

LDC LDC LDC

DS DS


PDS

25 TDS

File

Transient Data Storage: local storage before migration


Data Acquisition

CTP

LTU

TTC

FERO FERO

LTU

TTC

FERO FERO HLT Farm

FEPFEP

GDC GDCGDCGDC

LDCLDC

Storage Network

TDS

EDM

LDC LDC LDC

TDS

DS DS


PDS

Permanent Data Storage: file archive


Data Acquisition

CTP

LTU

TTC

FERO FERO

LTU

TTC

FERO FERO HLT Farm

FEPFEP

DS DSGDC GDCGDCGDC

LDCLDC

Storage Network

TDS

EDM

LDC LDC LDC

TDS


10 DS

PDS

DAQ Services: database and central facilities


High-Level Trigger

GDC GDCGDCGDC

CTP

LTU

TTC

FERO FERO

LTU

TTC

FERO FERO

LDCLDC

Storage Network

TDS

EDM

LDC LDC LDC

HLT Farm

FEPFEP

DDL

H-RORC

10 DDL

10 D-RORC

10 HLT LDC

TDS

DS DS


PDS

Filter events to optimize amount of valuable data

343 DDL


High-Level Trigger

GDC GDCGDCGDC

CTP

LTU

TTC

FERO FERO

LTU

TTC

FERO FERO

LDCLDC

Storage Network

TDS

EDM

LDC LDC LDC

HLT Farm

FEPFEP

DDL

TDS

DS DS


PDS

Detector Data-Links: transfer of event fragments

343 DDL


High-Level Trigger

GDC GDCGDCGDC

CTP

LTU

TTC

FERO FERO

LTU

TTC

FERO FERO

LDCLDC

Storage Network

TDS

EDM

LDC LDC LDC

HLT Farm

FEPFEPH-RORC

TDS

DS DS


PDS

HLT Read-Out Receiver Card: data reception


LDC LDC

HLT Farm

High-Level Trigger

GDC GDCGDCGDC

CTP

LTU

TTC

FERO FERO

LTU

TTC

FERO FERO

LDCLDC

Storage Network

TDS

EDM

LDC

FEPFEP

TDS

DS DS


PDS

Front-End Processors: data analyzing and decision


High-Level Trigger

GDC GDCGDCGDC

CTP

LTU

TTC

FERO FERO

LTU

TTC

FERO FERO

LDCLDC

Storage Network

TDS

EDM

LDC LDC LDC

HLT Farm

FEPFEP

10 DDL

10 D-RORC

10 HLT LDC

TDS

DS DS


PDS

DDLs, D-RORCs & LDCs: injection of results in DAQ

HLT decision


ALICE DAQ

GDC GDCGDCGDC

CTP

LTU

TTC

FERO FERO

LTU

TTC

FERO FERO

LDCLDC

BUSY BUSY

Rare/All

Storage Network

TDS

343 DDL

EDM

LDCLoad Bal.

LDC LDC

HLT Farm

FEPFEP

DDL

H-RORC

10 DDL

10 D-RORC

10 HLT LDC

163 DDL

TDS

DS DS


425 D-RORC

50 GDC 10 DS

PDS

25 TDS

200 Detector LDC

EventFragment

Sub event

Event

File


Implementation

• Transfer of data to the DAQ

• Computers

• Network

• Infrastructure

• Software


Detector Data Link

• Radiation tolerant SIU

• D-RORC

• DDG: DDL Data Generator (for tests)

Transfer of data to the DAQ


D-RORC data throughput

0

200

400

600

800

1000

1200

0 50000 100000 150000 200000 250000 300000

Event size, bytes

Th

rou

gh

pu

t, M

B/s

1 D-RORC4 D-RORCs

PCI-64 bandwidth fully used – 1GB/s with 4 D-RORCS


TRG2

TRG1

Trigger crate

DDG1

DDG2

DDL1

DRORC

LDC

LTU

Test setup

DDL, DDG, Trigger

DDL2

D-RORC

DDGs


DDL, DDG, Trigger

LTU TTC

DDG

DRORC

PCmemory

PCmemory

DDG

PCmemory

Trigger LDC

Electronics emulation

DDL

DDL

Test setup


Computer choices

• Intel 32 (LDC,GDC)• AMD 64 (DS)

• Scientific Linux CERN – SLC3

• GDC: CPU/memory performance• LDC: number / architecture of PCI slots• DS: database, multithreading performance

SLC4 when certified


CPU performance

2.4

Xeon 2.0

Pentium III 0.7

Xeon 2.4

Xeon 3.2Pentium IV 3.0

Xeon 2.8

Opteron 275 2.2

Opteron 244 1.8

0

20000

40000

60000

80000

100000

120000

140000

0.0 0.5 1.0 1.5 2.0 2.5 3.0 3.5

32-bit CPU

64-bit CPU

Ub

en

ch

/#C

PU

GHz

64-bit promising


Memory performance

Sysbench MEM

0

500

1000

1500

2000

2500

1 2 3 10 30

Threads

MB

/s

pcald45

pcald28

pcald44

aldaqcon02

pcald47

aldaqds002

aldaqpc010

aldaqpc001

pcald18 SDRAM 133

DDR 266

DDR 400 - Opteron

DDR2 400

Speed and architecture important


DB performance tests

0 10 20 30 40

1

2

3

4

5

6

Con

nect

ions

to s

erve

r

Thousands inserts per second

2 x Opteron Dual-Core 2.2 GHz DDR 400SLC4 64bit exec



2 x Xeon 3.2 GHz DDR2 400

1 x Pentium 4 2.8 GHz DDR 400

2 x Xeon 2.4 GHz DDR 266

2 x Xeon 2.8 GHz DDR 333

(same order on plot in each group)


DB performance tests

0

50

100

150

200

250

300

15:00 15:05 15:11 15:17 15:23

Time

Number ofconnections

New (per second) Active Maximum

Operator console MySQLserver

DAQ nodes

0

50

100

150

200

250

300

15:11:31 15:11:40 15:11:48 15:11:57 15:12:06 15:12:14

New connections Active connections

Max. number of active select queries

insert queries (logs) / 10

Configuration DB:concurrent access65 LDC, 65 GDC


Experimental area

CR1: DAQCR2: HLTCR3: DCS/HLTCR4: MISC

Counting roomsin the pit

50 meters


DAQ Counting Room

• Tight space: 70m2

• 33 racks

• ~300 computers


Racks

cooling doors 100 kW total37 kW on UPS

56U per rack


Racks

• Remote control– Rack control unit (T°, smoke)

– Power Distribution Units– KVM switch

• Optical patch panel• Computers


ALICE Control Room

• Multiscreen PCs

Nvidia NVS 280Linux Xinerama

• 30 workstations• separate room for PCs

• console access to machines in all counting rooms


DAQ Network

ComputingCenter

ALICE DAQCR1

2x2x10GbEthernet

Trigger HLT DCS

gw gw gwGeneralPurposeNetwork

DHCPCentralDataRecording

gw

ALICEControlRoom

Eventbuilding

•Private network•Gateways: IPtables

TDS

DNS NTP

DHCP

DNS

DAQservices 1Gb Ethernet

DAQcontrol

ECS


Network performanceThroughput by # of GDCs

0

500

1000

1500

2000

2500

3000

3500

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33

# of LDCs

MB

/s t

ota

l

27 28 29 30 31

GDCs @ wire speed

Force10 E-1200


Software

• DATE: DAQ software handling the data flow

• ECS: Experiment Control System

• AFFAIR: fabric monitoring


ECSUnified view of experiment

ECS

TRG DAQ HLT DCS

DIM/SMI: Communication/State machines


AFFAIRDAQ performance monitoring


AFFAIRDAQ performance monitoring

Event Building

DDL


Conclusion

• ALICE DAQ built with distributed components• General purpose and custom

hardware/software• Synchronization by state machines• Architecture tested extensively• Deployment progressing• DAQ/ECS for detector commissioning

installed at Point 2

http://cern.ch/alice-daq

Architecture and implementation of the ALICE Data-Acquisition system

Documents

Transcript of Architecture and implementation of the ALICE Data-Acquisition system