Post on 26-Jan-2016
description
LUX Trigger Control Software. Wojtek Skulski University of Rochester, November 2007
The architecture of
LUX Trigger Control Software
November/06 2007
Wojtek Skulski
LUX Collaboration University of Rochester
skulski _at_ pas . rochester . edu
LUX Trigger Control Software. Wojtek Skulski University of Rochester, November 2007
The goals of this presentation
• I will present the architecture and software components which I used
to develop several DAQ programs under BlackBox Component
Builder.
• The DAQ programs ran for thousands of hours in mission-critical
experiments without either crashing or leaking a single byte of
memory.
• The architecture can be used to develop the LUX Trigger Control
software running under BlackBox Component Builder.
• In order to make this presentation self-contained, I am also
explaining the hardware architecture of the LUX trigger boards.
LUX Trigger Control Software. Wojtek Skulski University of Rochester, November 2007
• Development of BlackBox DAQ & control software: 2001-2007.
• Duties of the LUX trigger DAQ and control system.
• Data display and monitoring.
• Board control.
• Reconfiguration of FPGAs.
• Architecture of the LUX trigger system.
• Architecture of the BlackBox DAQ & control system.
• Parts of the BlackBox DAQ & control system:
• Object management POM.
• Waveform graphics Gr.
• Scientific Library Lib (graphics, matrices, special functions, digital filters, etc.)
• DAQ script DataLogger.
• Board control GUI panels : see my “without a sting” presentation
• Low-level hardware interface libraries: see my “without a sting” presentation.
Outline
LUX Trigger Control Software. Wojtek Skulski University of Rochester, November 2007
NIM pulser
Analog signal reconstruction: digital FIR filter output
Control & waveform display:BlackBox and Gr
DDC-8
DDC-x development system under BlackBox
LUX Trigger Control Software. Wojtek Skulski University of Rochester, November 2007
Development of BlackBox DAQ & control software: 2001-2007
• I have been developing and refining the BlackBox-based DAQ and control since
2001 with my DDC-x waveforms digitizers.
• Several students used and extended the DDC software.
• I use the same DAQ software package at LLE on a daily basis with the following
hardware: capacitance sensors (interfaced via the same USB controller as the DDC
cards), VME-based distance-measuring interferometers, National Instruments A/D
cards, sliding-scale sensors (interfaced via a serial port), humidity and temperature
sensors (interfaced via a serial port).
• All my experiments work smoothly. The BlackBox software never caused any
significant problems or delays in my experimental work.
• Software components described in this presentation were tested over thousands of
hours in mission-critical experiments, some involving hundreds of thousand $
worth of equipment.
• BlackBox is the best and the most robust development software I have ever used.
LUX Trigger Control Software. Wojtek Skulski University of Rochester, November 2007
HW architecture of the trigger
The boards are described to give the context.
Block diagrams of the trigger board connections.
Block diagrams of the boards (the DDC and the EB).
LUX Trigger Control Software. Wojtek Skulski University of Rochester, November 2007
Hardware architecture of the LUX trigger boards
• LUX trigger is composed of two kinds of boards.
• The front-end DDC-8 LUX is a waveform digitizer and signal processor.
• The back-end Trigger Event Builder is a communication board.
• Both boards use the same programmable chips:
• Xilinx FPGAs perform real-time signal processing.
• Cypress microcontroller manages the FPGA and provides the USB-2 data
link with the PC.
• PC is running BlackBox control software, which monitors and
controls the trigger boards.
LUX Trigger Control Software. Wojtek Skulski University of Rochester, November 2007
Trigger consists of several DDC boards and one Event Builder board
Trigger system architecture
NIM pulses to DAQ system(event yes/no)
Up to 8 front-enddigital pulse processors
A single Level-2 decision board (Trigger Event Builder)
Fast data linksUp to 200 MB/s per link
Data readoutUSB 2.050 MB/s to the PC
LUX Trigger Control Software. Wojtek Skulski University of Rochester, November 2007
Architecture of the front-end DDC boards
Xilinx
Cypress
The new DDC-LUX board will also have a fast data link, 200 MB/s
LUX Trigger Control Software. Wojtek Skulski University of Rochester, November 2007
Architecture of the Event Builder boards
Xilinx FPGA
Cypress controller
The Trigger Event Builder is a new board designed for the LUX trigger
Eight fast data linksconnected to DDC
Up to 200 MB/s per link
USB 2.050 MB/s to the PC
Clock IN * 1NIM IN * 5NIM OUT * 4
LUX Trigger Control Software. Wojtek Skulski University of Rochester, November 2007
Xilinx ISE/WebPack
Three development systems for LUX board programming
PC
Cypress EZ-USBdevelopment SW
DAQ and control SW Component Pascal
running under BlackBox Component Builder
Trigger boards
Xilinx FPGA
Cypresscontroller
FPGA configuration
(VHDL)
controllerprogram
(C)
Board controldatagrams
Cypress controlleris running board control
firmware
1
2
3
LUX Trigger Control Software. Wojtek Skulski University of Rochester, November 2007
On-board firmware development for trigger boards
• FPGA development software.
• The FPGA is a massively parallel signal processor with 20 parallel multiply-
accumulate units. It implements real-time signal processing and decision logic.
• The original DDC-8 firmware was developed in 2004 and extensively tested. The
current LUX firmware is built upon this original firmware.
• Xilinx FPGAs type Spartan 3E-500 are programmed under Xilinx ISE WebPack.
• Currently we use high-level language VHDL to implement the firmware. Other
approaches (e.g., graphical System Generator) are possible, but not used yet.
• Microcontroller development software.
• Cypress controller manages the FPGA and provides the USB-2 data link with
the PC. Its processing power is relatively low (about 4 MIPS).
• The controller is programmed in C.
LUX Trigger Control Software. Wojtek Skulski University of Rochester, November 2007
LUX Trigger Control
software architecture
Running on a PC under BlackBox
Data display and monitoring
Board control
Remote FPGA reconfiguration
LUX Trigger Control Software. Wojtek Skulski University of Rochester, November 2007
Duties of the LUX Trigger Control software
• Data acquisition and writing to disk (optionally).
• Data display and monitoring. • One-dimensional waveforms (raw data from the ADC).
• Transformed waveforms (e.g., smoothed or filtered with some sort of “matched filter”).
• Two-dimensional correlations (e.g. time versus amplitude).
• Scalers (e.g., numbers of rejected and accepted events).
• Board control.• Thresholds, digital filter parameters, input DC offsets, etc.
• Several boards have to be controlled.
• Reconfiguration of the trigger system.
• The trigger operation is almost entirely defined by the FPGA. The FPGA has to be “loaded”
before the DDC boards can operate.
• The FPGA configuration has to be performed remotely, without unplugging the DDC boards.
LUX Trigger Control Software. Wojtek Skulski University of Rochester, November 2007
Components of LUX Trigger Control Software
• Graphics.• Integer-valued 1D waveform graphics Gr (raw ADC data).
• Real-valued 1D vector graphics Lib (transformed data).
• Real-valued 2D matrix graphics Lib (correlations, e.g., amplitude vs time).
• Object management.• A package named Persistent Object Manager (POM) manages the above-
mentioned graphical objects.
• Numerical values such as scalers.• Numbers can be entered or shown in graphical panels.
•Data-acquisition script.• A “script” for performing DAQ is compiled and runs at full speed.
• USB-2 communication with trigger boards.• The component for USB communication is very well tested.
LUX Trigger Control Software. Wojtek Skulski University of Rochester, November 2007
Waveform graphics GrShows raw ADC data.
Real-valued graphics LibMath and graphics library.Shows transformed data
and 2D matrices.
Persistent Object Manager POMManages graphical objects on screen.
Saves histograms and matrices to disk.
Simplified structure of DAQ software running under BlackBox.
USB communicationReceives data from the trigger boards.Sends control commands to the boards.
Data Loggeracquisition “script”
Compiled, runs at full speed.
Diagram of LUX Trigger Control Software
Management
Histogramming
Readout
LUX Trigger Control Software. Wojtek Skulski University of Rochester, November 2007
Presentation of the software
components
which can be used in the
Trigger Software
Gr, POM, Lib, and the DAQ “script”
LUX Trigger Control Software. Wojtek Skulski University of Rochester, November 2007
Waveform graphics Gr*
Easy to use histogram and waveform display package.
Designed for data acquisition (DAQ).
Extensively used for DAQ display:
DDC-1 and DDC-8 waveform digitizers, capacitance sensors,
temperature and humidity monitoring, CAMAC, and student projects.
*Developed by Wojtek Skulski.
LUX Trigger Control Software. Wojtek Skulski University of Rochester, November 2007
Why Gr was needed?
Why needed: A typical DAQ program uses either histograms, waveforms, or time
“histories” to record integer-valued ADC data. I found BlackBox Component Builder a
tremendous tool to develop DAQ software. However, the waveform graphics was
missing when I started using BlackBox circa 1999. Therefore, I decided to develop the
histograming and display package for BlackBox. As a basis for development I used my
previous package developed for Oberon System-3, because the two languages (Oberon
and Component Pascal) are very similar to each other. I finished the development and
released Gr to public domain in September 2001.
To my own surprise, I have been using Gr in its almost original 2001 form ever since
its release. I made only very few additions to the original version. It turned out that the
design principle “make it as simple as possible, but not simpler” has yielded a software
package which is useful, robust, and maintainable. The Gr package became my “secret
weapon” which enabled me to develop robust DAQ software for my many projects.
Students, who also used Gr, have found it useful for their own projects as well.
LUX Trigger Control Software. Wojtek Skulski University of Rochester, November 2007
When to use Gr
When to use Gr:
• Integer-valued data, e.g., acquired from a digitizer.• One-dimensional data. E.g. temperature recordings, etc.• The data is not an image (which would require 2D display).• The display has to be “live” and interactive.• Convenient zooming and scaling is implemented.
When to use Lib rather than Gr:
• Real-valued data (64-bit floating point numbers).• Two-dimensional display (e.g., time vs amplitude).• Lots of math, matrix algebra, or vector operations on data.• Special functions and/or digital filtering.
LUX Trigger Control Software. Wojtek Skulski University of Rochester, November 2007
Gr has a reasonable GUI and is very easy to use.
LUX Trigger Control Software. Wojtek Skulski University of Rochester, November 2007
Several DAQ examples
and a tutorial are provided
with the Gr package.
Example data acquisition panel.
Detailed instructions how to use the Gr package.
Gr User Guide
LUX Trigger Control Software. Wojtek Skulski University of Rochester, November 2007
Persistent Object Manager POM*
It manages and organizes any objects.
Designed for organizing data acquisition display.
Extensively used in mission-critical experiments:
Tiled-grating assembly, capacitance sensors, temperature and humidity monitoring.
*Developed by Daniel Miner (with some W.S. help).
LUX Trigger Control Software. Wojtek Skulski University of Rochester, November 2007
Why POM was needed?
Why needed: A typical DAQ program may use a large number of histograms,
waveforms, matrices, or other data objects. Typically, the histograms are initialized and
incremented under program control. Users want to display the content of these objects
while the DAQ program is running. (The “post-mortem” examination is inconvenient,
though often used.) This poses a problem of “dual ownership” of the data objects,
which are “owned” by the DAQ program, but need to be displayed by the GUI.
The Persistent Object Manager (POM) was developed to implement this feature. The
DAQ program will be running, processing data, and incrementing the histograms, and
yet you can display and save the histograms without stopping the DAQ. The histograms
are managed as a “tree” (similar to a Windows Explorer). You can double-click on any
histogram’s name, and you can examine its content. You can also save the entire tree.
Not just histograms: any object whatsoever can be managed by POM when the wrapper
for this object is available. We plan to write wrappers for several types of objects, such
as vectors, matrices, etc., as needed by the LUX Trigger project.
LUX Trigger Control Software. Wojtek Skulski University of Rochester, November 2007
POM organizes Gr histograms into a tree structure.
We will implement management of other arbitrary objects, as needed.
Graphical object management of Gr histograms
POMwindows
HistosApplication-specific
panel
LUX Trigger Control Software. Wojtek Skulski University of Rochester, November 2007
Another POM screenshot
Show another POM screenshot from the LLE experiment
Point out that POM was used for thousands of hours and never
crashed
LUX Trigger Control Software. Wojtek Skulski University of Rochester, November 2007
POM “to do” list
Currently, POM can only manage Gr objects.
• The “Gr wrapper” is implemented.
Other objects (e.g., Lib objects) need to be managed as well.
• We need to develop the wrappers similar to the Gr wrapper, using the Gr wrapper
as the template for development.
• It is very straightforward to develop such wrappers.
• Any object can be managed by POM when the wrapper is available.
• The most immediate need is for Lib objects (real-valued vectors and matrices).
• A text object (implementing a formatted BlackBox text) will be a very attractive
addition. Such text objects can hold run-time comments and remarks, including
graphics. Using the text objects, the experimenter’s comments will never be lost.
LUX Trigger Control Software. Wojtek Skulski University of Rochester, November 2007
DAQ DataLogger “script”*
It receives and processes experimental data.
Designed for organizing data acquisition process.
Extensively used in mission-critical experiments:
Tiled-grating assembly, capacitance sensors, temperature and humidity monitoring.
*Developed by Wojtek Skulski.
LUX Trigger Control Software. Wojtek Skulski University of Rochester, November 2007
Why the “script” was needed?
Why needed: Writing a DAQ program may seem straightforward. Just write a piece of
code with whatever actions you want... However, the DAQ development will benefit
from having a template program divided into typical sections, such as initialization of
histograms, the DAQ “subroutine”, and possibly others.
The Data Logger “script” was developed to help writing typical DAQ programs. There
are sections to initialize histograms, a hook to perform the DAQ actions, and the
interface to the BlackBox multitasker, which makes the DAQ run.
And of course, the Data Logger uses POM to manage histograms (and other data
objects in the near future).
NB: The “script” is not a script. It is a compiled “module”, which means two things: (1)
It runs at full speed. (2) The compiler performs full consistency checking with the rest
of the DAQ system. I use the name “script” because functionally it is like a script, but
its implementation is vastly superior to any kind of a scripting language (such as cint,
I beg your pardon). The LUX Trigger software will be another example of this “script”.
LUX Trigger Control Software. Wojtek Skulski University of Rochester, November 2007
Example experiment using the Data Logger + POM
Interferometer panel
(application-specific)
Example data: interferometer history recordings (popped up by clicking POM)
POM panel
This experiment was performed using POM and the Data Logger “script”
Data Logger panel
(start/stop, etc.)
LUX Trigger Control Software. Wojtek Skulski University of Rochester, November 2007
Scientific and engineering library Lib*
1D and 2D display package.
Matrix and vector calculus, digital filters, and special functions.
Designed for data analysis, theoretical calculations, and modeling.
Extensively used for mission-critical applications.
Airborne antenna systems (BAE Systems), adaptive optics (UofR Laser Lab).
*Developed by Robert Campbell, BAE Systems.
LUX Trigger Control Software. Wojtek Skulski University of Rochester, November 2007
Why Lib is needed?
Why needed: Writing a numerical program may seem straightforward... Until you
actually start doing it. Then you realize you need a library. Lib is a tremendous
computational library with over 6000 items. It covers vector and matrix calculus,
Fourier transforms, special functions, digital filtering, and many other areas.
In addition, Lib provides a very decent plotting utility which can render both 1D and
2D data. The latter is of course of interest to us (e.g., time-amplitude correlations).
Lib graphics and Gr graphics are complementary. I mostly use my own Gr graphics to
display raw data. However, when it comes to showing processed data, I often use Lib.
And of course, when I need to plot 2D data, I always use Lib because Gr only provides
1D display.
NB: You may ask why there are both Gr and Lib graphics? Part of this is history. The
two packages were developed independently. However, as I often use both, I appreciate
that Gr is great for displaying raw data, while Lib has a more “official look”.
LUX Trigger Control Software. Wojtek Skulski University of Rochester, November 2007
When to use Lib libraries
When to use:• Real-valued data (64-bit floating point numbers).• Two-dimensional display (e.g., camera images, or time vs amplitude plots).• Lots of math, matrix algebra, or vector operations on data.
The list of subsystems:
Lib - Engineering & Scientific library, including 1D, 2D, and 3D plotting.
Algebra - Computational Algebra library.
Multi - Arbitrary ultra high precision arithmetic.
Filter - Digital and other filter design and analysis.
Nav - Navigation & Geodesic Coordinate transform modules and tools.
Wands - A collection of general purpose tools (Wand: a tool used by a magician).
Cpc and Ctls- several general-purpose tools and utilities.
LUX Trigger Control Software. Wojtek Skulski University of Rochester, November 2007
Lib: math and plotting facilities
Vectors & Matrices, including 0-length cases.
Polynomial forms of vectors, matrix determinant, inverse, singular-value
decomposition, etc…
Complex numbers and complex functions.
Special Functions & curve fitting.
Random numbers: several r.n. generators.
Numerical methods.
Numerical integration, interpolation, root finding, and minimisation.
1D, 2D, and 3D plotting.
Includes interface to OpenGL.
LUX Trigger Control Software. Wojtek Skulski University of Rochester, November 2007
Scientific plotting and computation library Lib
LUX Trigger Control Software. Wojtek Skulski University of Rochester, November 2007
Scientific plotting and computation library Lib
LUX Trigger Control Software. Wojtek Skulski University of Rochester, November 2007
A mission-critical application using Lib
Adaptive optics control package, including hardware control.
LUX Trigger Control Software. Wojtek Skulski University of Rochester, November 2007
Conclusion:
Tools to develop
LUX Trigger Control Software
The tools are ready: Gr, Lib, POM, Data Logger.
Other tools are available from BlackBox community (e.g. web server).
Several experiments were performed using these software tools.
The experiments ran successfully for thousands of hours.
Some development will be required (as is always the case).