1060204 %d0%bb%d0%b8%d1%81%d1%82%d0%be%d0%b2%d0%ba%d0%b0 %d0%bf%d0%be %d1%82%d0%b5%d0%bf%d0%bb%d0%be
The D0 DAQ and its Operational Control
description
Transcript of The D0 DAQ and its Operational Control
CHEP 2000, February 7 - February 11, 2000, Padova (Italy), Abstract 379.
The D0 DAQ and its Operational Control
G. BriskinD. Cutts
A. KarachintsevS. Mattingly
G. WattsR. Zeller
Gennady Briskin, Brown University
2Gennady Briskin (Brown University)
The DØ Trigger System
Tape Output
Level 1
Level 2
Level 3
Trigger Information
Full Readout
C/R
Detector
DAQ
RIP
~50-70 Hz~50-70 Hz
~1 kHz~1 kHz
~10 kHz~10 kHz
~1 kHz~1 kHz
~7 MHz~7 MHz
3Gennady Briskin (Brown University)
Run II DAQ Numbers• Readout channels:
– Will be ~800,000 in Run 2 <250 kBytes>/event
• Data rates:– 60-80 readout crates– Many to one:
• All GS to one of 64 L3 farm nodes– Initial design capacity:
• ~1000 Hz• 250 MBytes/sec into the DAQ/l3-farm• Staged upgrades to more than 3 GB/sec
4Gennady Briskin (Brown University)
VBDVBD
VRCVRC
SBSB
MPMMPM
{ FCIETIDCI
ETGETG
VME Buffer/Driver
VBD Readout Concentrator
Segment BridgeFibre Channel InterfaceEvent Tag InterfaceData Cable Interface
Event Tag Generator
Multiport Memory
Digitizing crate
Level-3 filter node
NewNew
NewNew
NewNew
L3/DAQ Hardware Glossary
5Gennady Briskin (Brown University)
MPM MPM
SB
o o o
MPM MPM
SB
o o o
MPM
SB
o o o
MPM MPM
SB
o o o
MPM
VRC (1 of 8)
VB
D
VB
D
VB
D
o o oV
BD
VB
D
VB
Do o o
ETG
data collection pathdata collection path
primary FC Path
L3 data distribution to Level 3 farmLevels 1,2Levels 1,2trigger datatrigger data
event tagevent tagpathpath
(stage C)
• direct flow from VBD to MPM
• each data block flows freely and independently
• blocks for different events flow simultaneously
and asynchronously
• recirculation loops allow maximum use of data
path bandwidth
• Segment Bridges use Event Tag data to perform
complex realtime routing decisions
L3/DAQ Data Path
6Gennady Briskin (Brown University)
Segment Data
Cables
Segment Data
Cables
)
VRC1
)
Front EndCrate
Front EndCrate
Front EndCrate
Front EndCrate
Front EndCrate
Front EndCrate
Front EndCrate
Front EndCrate
VRC8
S(4 DATA CES
L3 Node(1 of 16)
L3 Node(1 of 16)
L3 Node(1 of 16)
SB1
SB4
ETG
Event Tag Loop
Primary Fiber Channel Loop #1
Primary Fiber Channel Loop #8
Front End TokenReadout Loop
Front End TokenReadout Loop
TriggerFramework )
L3 Node(1 of 16)
L3 Node(1 of 16)
L3 Node(1 of 16)
To Collector
Router
To Collector
Router
Ethe
rnet
Ethe
rnet
7Gennady Briskin (Brown University)
DAQ Architecture• Data flow is unidirectional
– No complex setup communication is required
• Each packet is sent only once– No retransmission if packet error is detected
• Routing is driven by the contents of the VBD header– No communication with front end crates
• Sender does not know or care where data is going
• Designed for expansion beyond the run 2 era
8Gennady Briskin (Brown University)
VME Buffer Driver Module
• Digitized data enters DAQ system via a VBD module– Typically 4-6 Kbytes per VBD/event
Dig
itiz
ers
Dig
itiz
ers
Dig
itiz
ers
VBD
VBD
Front End Crate
Token Control
Data Pathway48 MBytes/sec
From last VBD or VRC
To next VBDor VRC
•Components:–VME interface with list processing DMA–Dual SRAM buffers–External data cable interface with token arbitration logic.
•Performance:–VME: BLK mode DMA at 25-30 MB/sec–Data cable output: 48 MB/sec.–Token arbitration time: <10 usec.
9Gennady Briskin (Brown University)
VBD Readout Concentrator
• Bridges VBD readout loops to Primary Fibre Channel loop
• Provides readout token generation and general front end loop management: e.g. initialization, resets, and diagnostics
• Primary loop interface via full-duplex Fibre channel port– transmits data to first Segment Bridges (SB’s), receives
recirculated packets from last SB;
VRC
VBD
Data Cable
Token CLK
VBD
VBD
VBD
VBD
VBD
Token CLK
100 MB/s
FCI to 1st SB
100 MB/s
FCI from last SB
Data Cable
Loop #1
Loop #2
10Gennady Briskin (Brown University)
VBD Readout Concentrator
PCI/66Pentium
XXX128Mb
Ethernet
PCI
PCI
DPMem i960 Buffer
Mem
HD C A B L E
HD C A B L E
Term & FBTrans
64/26
64/26
VRC 1
FC P O R T
HD C A B L E
Term & FBTrans
64/26
64/26
FC out
FC in
DPMem i960 Buffer
Mem
HD C A B L E
HD C A B L E
Term & FBTrans
64/26
64/26
VRC 2
FC P O R T
HD C A B L E
Term & FBTrans
64/26
64/26
FC out
FC in
11Gennady Briskin (Brown University)
The Event Tag Generator• Receives L2 Accept Trigger Information
• Correlates Trigger Framework information with preprogrammed parameters for Level-3 node configuration, and then classifies event as:– generic: any segment, any node;– targeted: specific segment/node;– special function: diagnostic, broadcast, etc.
• Creates corresponding ‘event tag’ that instructs SB’s on how to handle data from that event.
• Places event tag in primary tag queue and transmits it when tag queue is available.
• Queue fill status is used to provide rate limiting feedback to Level 1 triggers– L1 disable lines
12Gennady Briskin (Brown University)
ETG System– Links L3/DAQ with the L1/L2 trigger systems– Uses trigger information for a given event to create an
“event tag” which enables complex data block routing in real time
•16 bit L3 Transfer Number•Trigger Bits
ET Creator(CAM Lookup)
SC SC SC SC
Return ETGMonitor
From Trigger FrameWork
ET Queue
L1 Disable
MonitorCPU
Ethernet
Event Tag Format:• Header
• Event Number• Flags (diagnostics or regular)
• Segment Controller 1 Block• Event Type• FEC Readout Bit Masks
• Segment Controller 2 Block• Event Type• FEC Readout Bit Masks
• etc.
}}
L3 Farm
13Gennady Briskin (Brown University)
Segment Bridge• SB routes data from the VRCs to a L3 Node(s)
– Bridge Primary Fibre Channel loop to Level-3 data cables
– Each Level-3 Segment has a Segment Bridge– ETG circulates event tag information to each SB.– ETG/SB is capable of complex node assignments
• Node assignment based on event type (calibration, etc…)• Shadow Nodes
– A second nodes receives the same event; allows testing of new L3 exes– Allows partitioning into several simultaneous data
taking runs
• SBs are serially linked together– One can expand the number of farm segments
14Gennady Briskin (Brown University)
Event-Node Assignment
ETGSB
L2 TriggerBits
Event Type
Trigger-bitevent type
lookup tables
Node Assignment
• Node Readies• Node-Event type
lookup table
Event Tag
• Regular• CAL_CALIB• Regular + Shadow #2• COSMIC
15Gennady Briskin (Brown University)
Segment Bridge
Router
DataCable Drivers
Fiber Out to next SC
or VRC
Fiber In from SC or VRC
•Receives Event Tag from ETG• accepts/declines the event based on characteristics of available L3 nodes
16Gennady Briskin (Brown University)
Dedicated 100 Mbits/sEthernet to
Online Collector/Router
L3 Farm Nodes
• Input rate starts at 1000 Hz• Output rate is ~50-70 Hz• Intel Based SMP System• Windows NT• As many L3 Filters as possible• Prototype hardware in hand
100-BaseT
Ethernet
L3FilterL3
FilterL3 FilterProcess
DMA capableVME-PCI Bridge
L3 Node FrameworkEach 48 MB/s
Control,Monitoringand ErrorModule
L3 Filter InterfaceModule
Node-VME I/O Module
Shared Memory Buffers
VME Crate
MPM
MPM
CollectorRouterModule
17Gennady Briskin (Brown University)
Validation Queue
Event Validation
• FECs presence validation• Checksum validation
L3 Filter Input Interface
Process Interface
Pool Queue
Collector/Router Network Interface•Determine where this event should be sent•Sent event to collector/router node
Filter Queue
Data to Online Host System
Data
Control
OutputEventsQueue
•Get a pointer to an event buffer •Configures MPMs for receiving new event•Wait till complete event arrives into MPM•Load event data into shared memory buffer•Insert event pointer into the next queue
L3 Supervisor Interface
L3 Monitor Interface
L3 Error Interface
Command/Monitor/ErrorShared Memory
Event BufferShared Memory
L3 Filter Process
L3 Filter Output Interface
Process Interface
OutputPool
Queue
MPM Reader
18Gennady Briskin (Brown University)
The L3 DAQ Control System Goals
•High speed configuration changes–Run start/stop–Physics Run/Calibration Run Modes
•Diagnose problems quickly
19Gennady Briskin (Brown University)
Environment•All nodes will use Commodity Hardware & Windows OS
•C++–MS Visual Studio 6.0
•Works well, IDE is a blessing.–Visual Source Safe for source control
•Very tightly integrated into the MS Visual Studio IDE
•Communication–ACE as the cross platform wrapper around TCP/IP stack–itc package to manage messages and event transport with D0 Online system (Fermi CD)–DCOM for internal control of DAQ components–XML message formatting for internal control/configuration of DAQ components
20Gennady Briskin (Brown University)
SC
The Level 3 DAQ/Trigger system
L3 FarmNode
L3 Supervisor
ETGVRCVRCVRCVRC
SCSCSB
OnlineSystem
CollectorRouter
L3 Monitor
21Gennady Briskin (Brown University)
DAQ Components States
Boot
IdleFinishRun
Start Run
Shut-down
PauseRun
•Every DAQ component can be in 1 of the 6 states
22Gennady Briskin (Brown University)
Auto Start SystemAuto Start Server
Web Server
Configuration Database
Client Machine
Auto StartService
Get Package List
Install Packages
Package Database
Configuration
Change Packages,Get Status,Reboot, etc.
Package
Running Packages
(COM Object)
23Gennady Briskin (Brown University)
The Level 3 DAQ/Supervisor System
•Translate D0 online system commands into L3 specific commands
– Begin Run– End Run– Run Script ele_high on L2 Trigger bit 2 on 6 nodes of type REGULAR.
•Report configuration errors (synchronous)– Must track a single online system command as it is distributed to DAQ components
–Maintain DAQ System state
24Gennady Briskin (Brown University)
L3 Supervisor Interface
COOR
COORCommandInterface
Current Configuration
DBResourceAllocator
CommandGeneratorSequencer
L3 NodeL3 NodeL3 NodeL3 NodeL3 Node
Supervisor Online System
CommandsConfiguration
Request
DesiredConfiguration
Data Base
DirectCommands
Commands
25Gennady Briskin (Brown University)
Monitor System
CollatorProcess
L3Monitor
Monitor Client
Process1
Process2
Process3
TransmitterProcess
Any L3 Computer
SharedMemory
SharedMemory
SharedMemory
Process1
Process2
Process3
Any L3 Computer
SharedMemory
SharedMemory
SharedMemory
TransmitterProcess
Monitor Client
Monitor Client
26Gennady Briskin (Brown University)
VRC Interface
FCI from last SB
VRCPrograms
ControlDisk
L3 SupervisorL3 Monitor
VRC Node
EmbeddedSystems
EmbeddedSystems
EmbeddedSystems
VBD Data Cables50 MB/s50 MB/s
100 MB/sFCI to 1st SB
100 MB/s
DCOMDCOM
27Gennady Briskin (Brown University)
ETG Interface
ETGPrograms
ControlDisk
L3 SupervisorL3 Monitor
ETG Node
EmbeddedSystems
EmbeddedSystems
EmbeddedSystems
TriggerFramework
Triggers TriggersDisable
DCOMDCOM
28Gennady Briskin (Brown University)
Segment Bridge Interface
SBPrograms
ControlDisk
L3 SupervisorL3 Monitor
SB Node
EmbeddedSystems
EmbeddedSystems
EmbeddedSystems
MPM Data Cables
100 MB/s FCI from VRC or SB
FCI to next SB
100 MB/s
DCOMDCOM
29Gennady Briskin (Brown University)
Conclusion• Natural upgrade of the Run I DAQ
– Allowed us to put together the framework for the L3 farm node with modest amounts of hardware effort
• New DAQ hardware permits > 10 increase in bandwidth
• DAQ control system is designed to be modular– makes use of industry standard software
30Gennady Briskin (Brown University)
The valiant warriors of D0 are strong thanks to a magic potion prepared by druid Getafix and this recipe is secret (all we know is that it has lobster and beer)