David Abbott CODA3 - DAQ and Electronics Development for the 12 GeV Upgrade.
-
Upload
sheila-walsh -
Category
Documents
-
view
218 -
download
0
Transcript of David Abbott CODA3 - DAQ and Electronics Development for the 12 GeV Upgrade.
David AbbottDavid Abbott
CODA3 - DAQ and Electronics CODA3 - DAQ and Electronics Development for the 12 GeV UpgradeDevelopment for the 12 GeV Upgrade
OutlineOutline
What is CODAWhat is CODA DAQ Issues / Aging TechnologiesDAQ Issues / Aging Technologies Requirements for 12 GeVRequirements for 12 GeV CODA3 DevelopmentsCODA3 Developments
SoftwareSoftware HardwareHardware
SummarySummary
The CODA missionThe CODA mission
CODA is a software toolkit with specialized CODA is a software toolkit with specialized hardware supporthardware support
Modular software components use the Modular software components use the networknetwork for for inter-process communication and event transport.inter-process communication and event transport.
Use Use open standardsopen standards and minimize the use of and minimize the use of commercial software while maximizing use of commercial software while maximizing use of commercial hardware.commercial hardware.
DAQ systems for each experimental Hall can be DAQ systems for each experimental Hall can be ““built-upbuilt-up” from common components to fit their ” from common components to fit their needs.needs.
What is CODAWhat is CODACODA is a software toolkit from which data acquisition systems with varying degrees of complexity can be built.
A typical system might look
Single Board Computer
FASTBUSVMECAMAC
ROC rcServer
EB
ROC
Network: Ethernet FDDI ATM
UNIX/LINUX
DISK/TAPEET
User Proc.
RunControlGUI
ER
cdev
dp_tcl
mSQL Database
msqld cmlogServer
cmlogDatabase
Data Acquisition StatusData Acquisition Status The Data Acquisition Group is charged with the task of The Data Acquisition Group is charged with the task of
providing a functional, unified acquisition system providing a functional, unified acquisition system appropriate for appropriate for ALLALL experimental programs at JLAB. experimental programs at JLAB.
Recent experiments have begun to test the limits of the Recent experiments have begun to test the limits of the current CODA system.current CODA system.
Development in both hardware and software technologies Development in both hardware and software technologies will be necessary to provide continued support of the future will be necessary to provide continued support of the future 6GeV program.6GeV program.
Current DAQ projects reflect the philosophy that we can Current DAQ projects reflect the philosophy that we can progress to support the physics of the 12 GeV program progress to support the physics of the 12 GeV program through an “through an “upgradeupgrade” of the existing proven system.” of the existing proven system.
Front-End Issues…Front-End Issues…
The CODA ROC is a customizable software object that The CODA ROC is a customizable software object that provides a standard for the User to access front-end provides a standard for the User to access front-end hardware - whatever that may be. It must evolve.hardware - whatever that may be. It must evolve.
Obsolete FASTBUS digitizing electronics needs a Obsolete FASTBUS digitizing electronics needs a replacement alternative (in VME). There are far fewer replacement alternative (in VME). There are far fewer appropriate commercial solutions today.appropriate commercial solutions today.
Real-Time readout on a per event basis limits the maximum Real-Time readout on a per event basis limits the maximum accepted L1 trigger rate (~10 KHz).accepted L1 trigger rate (~10 KHz).
32 crate limit on the trigger distribution system is nearly 32 crate limit on the trigger distribution system is nearly reached in Hall B.reached in Hall B.
DAQ - Trigger - Electronics - All must be designed to work DAQ - Trigger - Electronics - All must be designed to work together, efficiently, “customized” !! together, efficiently, “customized” !!
Other DAQ LimitationsOther DAQ Limitations
Event transport limitations in the Event transport limitations in the current CODA architecture are being current CODA architecture are being seen for moderately complex seen for moderately complex systems.systems.
Aging software technologies and Aging software technologies and reliance on third party packages are reliance on third party packages are making code portability and upkeep making code portability and upkeep difficult.difficult.
Monitoring and control of large Monitoring and control of large numbers of distributed objects are numbers of distributed objects are not handled in a consistent way (too not handled in a consistent way (too many protocols).many protocols).
Slow controls only minimally Slow controls only minimally supportedsupported
~30 ---> 1
EB
ROC ROC ROC
50-70 MB/sec
3 - 9 KHz Trigger
Hall B “bottlenecks”
A General PlanA General Plan Replace aging technologiesReplace aging technologies
Run ControlRun Control Tcl/Tk based DAQ ComponentsTcl/Tk based DAQ Components mSQLmSQL FASTBUS/CAMACFASTBUS/CAMAC
Hall D (GlueX) requirements drive development directionHall D (GlueX) requirements drive development direction Maintain Cross platform supportMaintain Cross platform support
Linux - Solaris - vxWorks - OS X - 64 bit…Linux - Solaris - vxWorks - OS X - 64 bit… Support and use new commercial advances where Support and use new commercial advances where
possible.possible. More custom hardware design will be required and an More custom hardware design will be required and an
electronics support group to maintain.electronics support group to maintain.
CODA3 - Requirements/GoalsCODA3 - Requirements/Goals Pipelined Electronics (FADC, TDC)Pipelined Electronics (FADC, TDC)
Dead-timeless systemDead-timeless system Replacement for obsolete electronicsReplacement for obsolete electronics Eliminate large numbers of delay cablesEliminate large numbers of delay cables
Integrated L1/L2 Trigger and Trigger Distribution SystemIntegrated L1/L2 Trigger and Trigger Distribution System Support up to 200 KHz L1 TriggerSupport up to 200 KHz L1 Trigger Use FADC for L1 trigger inputUse FADC for L1 trigger input Support 100+ crates Support 100+ crates
Parallel/Staged Event BuildingParallel/Staged Event Building Handle ~100 of input data streamsHandle ~100 of input data streams Scalable (>1 GByte/s) aggregate data throughputScalable (>1 GByte/s) aggregate data throughput
L3 Online FarmL3 Online Farm Online (up to x10) reduction in data to permanent storageOnline (up to x10) reduction in data to permanent storage
Integrated Experiment ControlIntegrated Experiment Control DAQ RunControl + “Slow” control/monitoringDAQ RunControl + “Slow” control/monitoring Distributed, scalable, and “intelligent” Distributed, scalable, and “intelligent”
Anatomy of a CODA3 DAQ System
Example - Hall D DAQExample - Hall D DAQ
ExistingHalls
Front-End GoalsFront-End Goals
L1 Trigger rate - 200 KHzL1 Trigger rate - 200 KHz Block up Events (200 event block -> 2kHz)Block up Events (200 event block -> 2kHz) Move some ROL runtime code to modules (FPGAs)Move some ROL runtime code to modules (FPGAs) ADCs provide L1 trigger data ( hence we need a ADCs provide L1 trigger data ( hence we need a
distributed high speed clock - 250 MHzdistributed high speed clock - 250 MHz
New Trigger SupervisorNew Trigger Supervisor Perhaps 100+ cratesPerhaps 100+ crates Support pipeline, event blockingSupport pipeline, event blocking Manage flow control into DAQ system backendManage flow control into DAQ system backend
Front-End SystemsFront-End Systems
VME CPUVME CPU -(MV6100) PowerPC, GigEvxWorks or LinuxCODA ROCReadout ~160+ MB/s
Trigger InterfaceTrigger Interface - (V3)
Pipeline TriggerEvent BlockingClock distributionEvent ID Bank Info
F1 TDCF1 TDCFlash ADCFlash ADC
R&D to support fully pipelined crates capable of 200 KHz trigger rates
Level 1 TriggerLevel 1 Trigger Distributed high speed clockDistributed high speed clock
• derived from Accel clock 499 MHzderived from Accel clock 499 MHz• stepped down at crate - inter crate jitter <50ps stepped down at crate - inter crate jitter <50ps
Subset of ROCs collect summed ADC dataSubset of ROCs collect summed ADC data and send it to L1 Trigger - in syncand send it to L1 Trigger - in sync 12 bit sums/crate x 250MHz --> 3 Gbit/s links12 bit sums/crate x 250MHz --> 3 Gbit/s links Trigger decision goes to Trigger SupervisorTrigger decision goes to Trigger Supervisor for pipeline distribution to all crates for pipeline distribution to all crates
ROC ROC ROC ROC ROC ROC ROC ROC ROCROC ROC ROC
L1 Trigger
Trigger Supervisor
200 KHz
Staged/Parallel Event BuildingStaged/Parallel Event Building• Divide total throughput into N streams (1GB/sec -> N*xMB/sec).Divide total throughput into N streams (1GB/sec -> N*xMB/sec).• Two stages - Data Concentration -> Event Two stages - Data Concentration -> Event BuildingBuilding..• Each EMU is a software component running on a separate host.Each EMU is a software component running on a separate host.
2N hosts are interconnected though one BIG switch
Event Management UnitEvent Management Unit
• EMU built around the ETEMU built around the ET system for customizablesystem for customizable processing/distribution ofprocessing/distribution of event streams.event streams.
Examples:Examples:• Data Concentrator for ROCsData Concentrator for ROCs• Sub-Event builderSub-Event builder• Farm distribution pointFarm distribution point• Event RecorderEvent Recorder• User Processes can attach toUser Processes can attach to any EMU in the systemany EMU in the system
Level 3 FarmLevel 3 Farm
Can be used for analysis Can be used for analysis or filteringor filtering
Support 100s of nodesSupport 100s of nodes Nodes can come and go Nodes can come and go
during event takingduring event taking Filtered events will be Filtered events will be
time ordered on the time ordered on the back-end. back-end.
1 GB/s
100 MB/s
Current DAQ Projects - OverviewCurrent DAQ Projects - Overview
Components:Components: Run Control Coda ROC Coda EB/ER (EMU)
Software Tools:Software Tools: cMsg ET evio
Hardware:Hardware: FADC/F1TDC Trigger Interface (VME/PCI) Trigger/Clock Distribution
R&D:R&D: Embedded Linux Experiment Control Staged/Parallel Event Building VXS - High speed serial 200KHz Trigger/readout Low jitter clock distribution
Some Current ProjectsSome Current Projects
cMsgcMsg - homegrown publish-subscribe messaging - homegrown publish-subscribe messaging which will become the basis for DAQ online which will become the basis for DAQ online communicationcommunication
JAVA Agent-based Run ControlJAVA Agent-based Run Control EMUEMU - C/C++ based, modular, threaded - C/C++ based, modular, threaded Event BlockingEvent Blocking
Hardware - New TI to support existing Trigger Sup.Hardware - New TI to support existing Trigger Sup. Software - new ROC, parallel readout lists, EVIOSoftware - new ROC, parallel readout lists, EVIO
Embedded LinuxEmbedded Linux - establish a stable maintainable - establish a stable maintainable Linux distribution for SBCs with access to VME and Linux distribution for SBCs with access to VME and PCI hardware.PCI hardware.
Software ToolsSoftware Tools
cMsgcMsg - Homegrown publish-subscribe - Homegrown publish-subscribe messaging - the basis for all DAQ online messaging - the basis for all DAQ online communication.communication.
ET ET - Event Transport - High speed, shared - Event Transport - High speed, shared memory based event distribution system. memory based event distribution system. Also supports network transport.Also supports network transport.
EVIOEVIO - Library/Tools for reading, writing, - Library/Tools for reading, writing, building, analyzing and displaying events in building, analyzing and displaying events in the CODA framework. the CODA framework.
CODA is a software toolkit from which data acquisition systems with varying degrees of complexity can be built.
A typical system might look
Single Board Computer
FASTBUSVMECAMAC
ROC rcServer
EB
ROC
Network: Ethernet FDDI ATM
UNIX/LINUX
DISK/TAPEET
User Proc.
RunControlGUI
ER
cdev
dp_tcl
mSQL Database
msqld cmlogServer
cmlogDatabase
cMsg - CODA Messaging SystemcMsg - CODA Messaging System
cMsg
Publish-subscribe messaging system JAVA,C++,C implementation and APIs Replaces older separate packages with one maintainable system
Experiment ControlExperiment Control
JVM
Host 1
A
ROC
AHV
RC
CODA Java-Based (v 1.5) “Intelligent” agents (AFECS)CODA Java-Based (v 1.5) “Intelligent” agents (AFECS) JADE extensions provide a runtime “distributed” JVM.JADE extensions provide a runtime “distributed” JVM. Agents provide a customizable intelligence and communication with Agents provide a customizable intelligence and communication with
external processes.external processes.
Host 1 Host 2
Host 3
A
A
ROC
HV
RC
CODA ROC
CODA EMU
EPICS IOC 1
EPICS CAG
Trigger soft
Trigger hard
Online ANA
A
A
A
A
A
A
A
S
S
S
S
S
NR NA NC
Physical ComponentsNormative Agents
Supervisor agent
Grand supervisor
Front-End
ACC
AFECS Platform
IPC
GUI/user
IPC
WEB
IPC
Hierarchy of Control
IPC
Run Control GUIRun Control GUI
Experiment Monitoring and Experiment Monitoring and ControlControl
JLAB Pipeline TDCJLAB Pipeline TDC
010002000300040005000600070008000
0800016000240003200040000480005600064000
Normal Resolution
Delay (ns)
Count
slope = 118.94 ps/cnt
High Resolutionslope = 59.47 ps/cntTDC ASIC (8 channels)
Hits
Trigger
Time -->
JLAB Flash ADCJLAB Flash ADCfADC250 @ 250 MSPS
Pulse on CH 1: 30 MHz, tr & tf = 5 ns, Width = 8 ns
0
200
400
600
800
1000
0 10 20 30 40 50
Sample # (4 ns/sample)
ADC Value
FPGAs
Hits
TriggerTime -->
8µs
VME64X - VXS InterconnectVME64X - VXS Interconnect
J0 - 200 total pins 45 differential pairs 6 GHz Bandwidth
18 VME Payload Slots
2 Switching slots
Electronics DevelopmentElectronics Development
VXS - L1 Trigger VXS - L1 Trigger
VME CPUVME CPU -(V7865) Intel, GigELinux CODA ROCVME Readout of Event Data
Switch Sum and Trigger Switch Sum and Trigger Distribution Modules (VXS)Distribution Modules (VXS)
Collect Sums/HitsPass Data to Master L1Clock distributionTrigger Distribution
Flash ADCFlash ADC
Use VXS High speed serial backplane (P0) to collect Energy sum and hit data from FADCs
Flash ADCFlash ADC
P0P0
ATCA as a StandardATCA as a Standard
Becoming a widely adoptedBecoming a widely adoptedCommercial standardCommercial standard
8U Blade Form-factor 8U Blade Form-factor
High-Power/High AvailabilityHigh-Power/High Availability
Few applications in Physics Few applications in Physics yet and not particularly cheapyet and not particularly cheap
Possible use for Master Possible use for Master Trigger processor and TriggerTrigger processor and TriggerSupervisorSupervisor
SummarySummary
CODA version 3 is now being molded - and CODA version 3 is now being molded - and can be adapted to future requirements!! can be adapted to future requirements!!
Our plan is to phase in new tools to provide Our plan is to phase in new tools to provide a smooth transition from CODA2 --> a smooth transition from CODA2 --> CODA 3 (CODA 2.6 Release)CODA 3 (CODA 2.6 Release)
New custom electronics will be available for New custom electronics will be available for use soon.use soon.