2003 Conference for Computing in High Energy and Nuclear Physics La Jolla, California Giovanna...

2003 Conference for Computing in High Energy and Nuclear PhysicsLa Jolla, California

Giovanna Lehmann - CERN EP/ATD

The DataFlow of the ATLAS Trigger and Data Acquisition

System

Giovanna LehmannOn Behalf of the ATLAS

Trigger/DAQ DataFlow Subsystem



Outline

ATLAS Interaction rates and event sizesThe Trigger/DAQ architecture

The DataFlowROS

• Design & Performance

LVL2 dataflow• Design & Performance

Event Builder• Design & Performance

Conclusions & Outlook



CERN Accelerators’ Complex

•Colliding particles: protons•Center of mass Energy: 14 TeV•Bunch crossing rate: 40 MHz•Interaction rate: 109 Hz•Event size: 1-2 Mbytes



DATAFLOW

EB

120 GB/s

H

L

T

L1

DET ROD

LVL2

Trigger DAQARCHITECTURE

2.5

s

~ 10 ms

Calo MuTrChOther detectors

SFI

SFO

EFN

FE Pipelines

ROIB

L2P

L2SV

L2N

Event FilterEFP

EFPEFP

EFP

RoI

RoI data = 2%

RoI requests

Lvl2 acc= ~2 kHz

~ sec

Lvl1 acc = 75 kHz

40 MHz

~3 GB/s

EFacc= ~0.2 kHz

EB req/clears

EBN

ROSROB

IOManager

data

DFM

RRM

40 MHz

75 kHz

~2 kHz

~ 200 Hz

120 GB/s

~ 300 MB/s

~3+3 GB/s

O(100) L2P

O(1000) EFP

1628 ROLs

O(100) ROS

O(100) SFI



The ROS

Receive & buffer event fragments from the 1628 detector ROLs Up to 160 MB/s per ROL

Send selected event fragments on requestROI requests : high rate, low data volume

• Rate: LVL1 rate (75 kHz), volume: ~2% of ROLs

EB requests : low rate, high data volume• Rate: ~3% of LVL1 rate (~2 kHz), volume:

complete event data

Provide fragment sampling for data monitoring



ROS High Level Components

Data requests

Event Fragments

Monitoring data

Control/Configuration

RODsRODsROS subsystemL2 & EBL2 & EB

Online SWOnline SW

I/O Manager(SW process)

RobIn(custom module)

Local Controller(SW process)

500x

150x

150x



Test Setup: ROS performance

ROS implemented on a 2 GHz PC, with 4 PCI busses (64 bit/66 MHz)

3 RobIn emulators on PCI On-board “local” bus limited to 266MB/s Each simulates 4 input channels 12 ROLs per

ROS

I/O to/from L2 & EB emulator Connected to the ROS through a GE switch Sends ROI/EB requests and clears to the ROS Receives data fragments back Uses TCP as communication protocol

(maximum possible overhead for message passing)



0

20

40

60

80

100

120

140

0 1 2 3 4 5 6 7

ROI Volume [% of full event]

Max

. LV

L1

Rat

e [k

Hz]

2 GHz PC; 3% EB requests

ROS Performance

ATLAS baseline conditions

(from paper model: contains safety factor 4 with respect to physics simulation )



The LVL2 Dataflow

Receive RoI information from LVL18 ROLs @ LVL1 rate (75 kHz)

Form a LVL1 result recordBuild 1 record out of 8 @ LVL1 rate

Retrieve RoI data from ROSs2 % of full Event (~30 kB)

Forward the LVL2 decision to the EB@ LVL2 accept rate (rejects are

grouped) Forward the LVL2 decision record to the

EB@ LVL2 accept rate (~2 kHz)



RoI req./data

Decisions


EBEBLVL2 subsystemLVL1LVL1

Online SWOnline SW

RoIBuilder(custom module)

L2SV(SW process)

L2PU(SW process)

pROS(SW process)

L2 recordROSROS

RoI information

1x 10x

200-500x

1x

LVL2 High Level Components

DC Controller(SW process)



Performance of RoI Builder, L2SV and pROS

Performance of RoI BuilderCustom built 12U VME prototype has

achieved required performance Performance of each LVL2 supervisor

Measured to be ~30 kHz on a 2.4 GHz dual CPU PC

Is insensitive to the number of L2PUs Performance of pROS

Not a demanding applicationRequirement to receive <10 kB at LVL2

accept rate (~3 kHz) and forward them to the EB is largely satisfied.



Test Setup: Performance of L2PU

ROS emulators used to send data over Gbit Ethernet.

RoI data collection takes always a small fraction of the time requested by the LVL2 event processing (~10 ms).

From a dataflow point of view << 100 L2PUs could sustain already the LVL1 rate.



The Event Builder

Receive LVL2 decisions @ LVL2 accept rate (~2 kHz; rejects are

grouped) Request data from ROS and pROS Build complete events

Depending on ROS implementation merge 140-1600 fragments into one.

~70 MB/s at every SFI Distribute clears to ROS and pROS

@ rate < LvL2 accept rate Forward complete events to EF

~70 MB/s at every SFI Provide fragment sampling for data monitoring



EB req./data

LVL2 Decisions


L2L2EB subsystempROSpROS

Online SWOnline SW

DFM(SW process)

SFI(SW process)

Clears

ROSROS

1x

50x

EB High Level Components

EFEF

Monitoring

DC Controller(SW process)

10x

Complete Event



Test Setup: EB Performance

DFM

16x

ROS em

... Switch

SFI

SFI

... 8x

ROS em

• SFI applications were run on 2.4 GHz dual CPU PCs

Many ROSs to many SFIs

ROS Emulators: ALTEON programmable GE NICs Raw ethernet communication protocol Simulating n sources



EB Performance

0

50

100

150

200

250

300

350

400

0 2 4 6 8 10

# of SFIs

EB

ra

te H

z

Limit of 16 ROS emulators for single frame messages

EB rate with 8 SFIs ~ 350Hz (17% of ATLAS EB rate)

8 ROLs/ROS

Flow Control

1 ROL/ROS

No Flow Control



Conclusions & Outlook

All elements of the DataFlow system have shown that they can satisfy the ATLAS requirements already with the present implementations and with today’s technology.

From now on emphasis will be put on the performance of the integrated DataFlow system.

Testbeds are being setup to measure its behaviour and the first results are encouraging.



Spares



0

50

100

150

200

250

300

350

400

0 2 4 6 8 10 12

ROI Volume [% of full event]

Max

. L

VL

1 R

ate

[kH

z]

EB rate =1% of LVL1 rate

EB rate = 2% of LVL1 rate



ATLAS baseline conditions (from paper model)

Results of Test 1 (no I/O to LVL2 & EB)



Test Setup 2: Scaling of LVL2 Network

From a dataflow point of view a few L2PUs sustain already a large fraction of the LVL1 rate.



Test Setup 1: Performance of the DFM

Tester:L2SV + n SFIs

DFM

LVL2 decision(group)

SFI EOE

DFM_Decision

Clears (group 300)

Tester Application emulates L2SV and many SFIs

DFM handling full I/O as for real ATLAS

DFM exposed to full input message rate from tester

DFM sending to non existing destinations (Connectionless protocol used)



DFM Performance

ATLAS event building rate

raw ethernet frames

udp

Test on a 2.2 GHz dual CPU PC:

Rate= function of CPU clock ->



Test Setup 1: SFI Performance

DFM 1 Gbit/s Ethernet

16x

ROS em

... EFSwitch

ROS em• SFI application was run on

a 2.4 GHz dual CPU PC

Many ROSs to 1 SFI

ROS Emulators: ALTEON programmable GE NICs Raw ethernet communication protocol Simulating n sources

SFI



SFI Performance

0

10

20

30

40

50

60

0 2 4 6 8 10#ROLs/ROS

EB

ra

te H

z95 MB/s – IO limited

#ROLs/ROS

EB only

With output to EFCPU limited (2.4 GHz

CPU)

Reaching I/O limit at 95 MB/s otherwise CPU limited

35% performance gain with at least 8 ROLs/ROS

Will approach I/O limit for 1 ROL/ROS with faster CPU

2003 Conference for Computing in High Energy and Nuclear Physics La Jolla, California Giovanna...

Documents

Transcript of 2003 Conference for Computing in High Energy and Nuclear Physics La Jolla, California Giovanna...