1

Update on the BNL SRS readout and analysis system

• First presented at the CERN Coll Meeting

• Is in production at @BNL

• Presentation about our data on Thursday

Martin Purschke

Brookhaven National Laboratory

2

Smaller DAQ Systems in PHENIX

• PHENIX has its main, 2 mill channels, 1400MB/s DAQ system that fills a room the size of a squash court

• There has always been the need for “smaller”, more lightweight DAQ systems for tests, R&D, calibrations and external beam tests (at FNAL, AGS, CERN, …) - o(100K channels)

• I personally also support another project at BNL & UPenn, a Medical Imaging group which develops various detectors for PET / MRI

• These small DAQ systems are format-compatible with the main PHENIX DAQ.

• Monitoring, debugging, and analysis code developed in the R&D phase can be re-used and built upon later

• Existing software can be used in calibration beams, etc

• Need to debug the readout of new electronics only once

3

Our GEM Stack Setup

GEM Stack

x-y step motor

Collimated CO60 SourceHybrids

There will be a presentation on Thursday morning (Bob Azmoun) about the data from this setup.

4

RCDAQ – a lightweight and powerful DAQ

Client-server based DAQ based on “Remote Procedure Call” network protocol. Arbitrary number of clients, clients can be remote

RCDAQ server

Network USB PCIe

HardwareHardware Hardware

RCDAQ Client RCDAQ ClientCommand line

RCDAQ ClientScript

Writes to stdout (logfile) but doesn't have a UI component itself

5

Devices implemented in rcdaq

PET Scanner for Mammography / Rodents (3072 LYSO crystals)

RCDAQ

“TSPM”

Prototype

Read out with 4 “Timing and Signal Processing Modules” (TSPM)

DRS4 Eval board

“USB Oscilloscope”

Nice for triggering, say, for cosmics

The SRS System

(that's why I'm here of course)

Not shown: PHENIX's “Data Collection Module 2”

This is our largest client currently, data rate saturates 4 gigabit lines

6

RDAQ Features

• Very modest investment in setup, maintenance, manpower

• You can be up in one hour starting from scratch:

• Download, configure, make && make install

• Comes with working example scripts and pseudo devices (random numbers) to read out to start playing w/o hardware

• Runs on virtually any modern Linux distro (rule of thumb: if root is available on your OS, it will work) – looking into MacOS

• Have ran it on the “Redhat family”: { RH5/6, Scientifix Linux, Fedora}, the Ubuntu family, Gentoo (Gentoo is the prime development platform)

• All it takes for you is to “buy into” the raw data format and a root interface library

7

(Almost) Everything is a Plugin

• The devices in rcdaq have very diverse and sometimes proprietary drivers / libraries (e.g. the DCM2 PCIe board needs the Jungo / windrvr6 kernel module + libs / utils)

• An initially monolithic rcdaq soon became untenable

• Switched to a plugin concept, where support for a device gets loaded on demand

• Locality of support libraries, ease of distribution, etc

• rcdaq by itself knows only very few generic and pseudo-devices, everything else is loaded as plugins

8

No configuration files

• … only config scripts. Much more powerful.

• Scripts in both in the rcdaq setup as well as for the SRS

• Each interaction with the system is a shell command

• You can have conditions, loops, error checking, you name it

# Setting up the Hybrids

for sa in 0x801 0x401 0x201 0x101 0x8001 0x4001 0x2001 0x1001 0x802 0x402 0x202 0x102 0x8002 0x4002 0x2002 0x1002 ; do

srs_control SETVALUE 6263 0x01 $sa 0x19 srs_control SETVALUE 6263 0x02 $sa 0x80 srs_control SETVALUE 6263 0x03 $sa 0x4 # ...stuff deleted...done

You may recognize this as the configuration sequence for all master and slave hybrids – port, address, subaddress, value

9

Network Transparency

All client-server interaction based on RPC - Remote Procedure Call

• Widely established standard (RFC 1831) for remote execution of code controlled by a client

• Makes it look like a local function call, but the function executes on a server

• The ubiquitous NFS is based on RPC, it is available virtually everywhere

• It is a network protocol, so client and server don't have to be on the same machine

• Can have DAQ and control machine in different rooms (or as far apart as you like as long as the connection traverses the firewalls)

• Very robust and easy, and an open standard built into virtually any OS

Your rcdaq server machine can be in a hot area but all your controls, including yourself, can be outside

10

A minimalistic session ( could be typed in)

rcdaq_client load /home/rcdaq/install/lib/librcdaqplugin_srs.so

rcdaq_client create_device device_srs 1 1010 10.0.0.2

rcdaq_client daq_openrcdaq_client daq_begin 201

..... run 201 is in progress ....

rcdaq_client daq_endrcdaq_client daq_begin

...... run 202 is in progress .....

rcdaq_client daq_endrcdaq_client daq_close

Start logging

Define readout

Start a run (201)

End the run

Stop logging

11

… and you get

$ printf "rcdaq-%08d-%04d.evt\n" 221 0rcdaq-00000221-0000.evt

The raw FEC sampled waveform

The decoded time sample vs channel plot

12

RCDAQ configuration

The “rcdaq_client <command>” construct can be made more convenient via aliases for interactive use

#! /bin/sh# load the srs pluginrcdaq_client load /home/rcdaq/install/lib/librcdaqplugin_srs.so# create the read listrcdaq_client create_device device_srs 1 1010 10.0.0.2

As I said, everything is a script:

% rcdaq_client daq_status -llStoppedFilerule: rcdaq-%08d-%04d.evtBuffer Sizes: 65536 KBList of loaded Plugins: - SRS Plugin, provides - - device_srs (evttype, subid, IP addr) - readout an SRS crate - device_srspsrams (evttype, subid, IP addr) – read back the SRS configuration% rcdaq_client daq_list_readlistSRS Device Event Type: 1 Subevent id: 1010 IP: 10.0.0.2%

13

The Quest for Speed

• You may have seen many DAQs which format the raw data into a root tree, or some other generic format

• We don't do that

• Most modern devices give their data in a nicely compact format already, the SRS is no exception

• We encapsulate the device data unaltered into a “Packet” with envelope information what is inside – no wasting cycles on reformatting

• The envelope information identifies a decoding algorithm how to deal with the data

• A typical event from PHENIX has about 1200 packets or so, one for each “device” which we read

• Here we tend to have just one, same deal

rcdaq_client create_device device_srs 1 1010 10.0.0.2

Event type packetid

SRS coodinates

14

Raw Data and Analysis

• Flat files with a structure to get at events and the packets therein

• Needs a root library (or standalone one for root-less ops) to get at the data

• Comes with a framework called “pmonitor” for no-frills monitoring or analysis

• Pmonitor supports true monitoring – looking at histograms while they are being filled and updated

• Pmonitor is PHENIX's both online monitoring and offline framework

• Access to files, rcdaq online streams, list of files

• Another cool features: if file1 and file2 are valid raw data files, the concatenation is again a valid file – cat file1.evt file2.evt > file_combined.evt

• Wealth of tools to quickly inspect data without writing any software

15

Example of such a tool: ddump

$ ddump -O -i srs-00000083-0000.evt -- Event 2 Run: 83 length: 4850 type: 1 (Data Event) 1329511626Packet 1010 4842 -1 (ONCS Packet) 70 (IDSRSV01) - -- ----------------------4Number of Hybrids: 4Framecounter: 0HDMI Channel: 0Description: ADCWords: 2410Rows : 16 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 123 122 126 127 125 124 116 117 119 118 114 115 113 112 80 81 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 ---------------------------------------------------------------------------------------- 0 | 2643 2634 2662 2714 2719 2680 2647 2650 2659 2614 2626 2710 2682 2645 2671 2640 1 | 2634 2605 2614 2637 2630 2644 2623 2623 2641 2645 2661 2636 2594 2601 2622 2590 2 | 2622 2604 2606 2627 2634 2639 2636 2625 2614 2616 2635 2663 2620 2604 2625 2611 3 | 2612 2596 2605 2619 2599 2601 2587 2595 2609 2609 2614 2633 2598 2583 2609 2583 4 | 2618 2588 2594 2626 2635 2620 2602 2567 2564 2588 2625 2650 2611 2615 2633 2584 5 | 2644 2622 2638 2641 2648 2655 2650 2634 2623 2624 2640 2666 2643 2629 2626 2608 … 126 | 2497 2505 2518 2499 2477 2477 2487 2514 2500 2505 2507 2486 2473 2435 2456 2479 127 | 2481 2463 2480 2481 2470 2467 2466 2473 2493 2479 2480 2452 2462 2474 2460 2466Framecounter: 1HDMI Channel: 2Description: ADCWords: 2410Rows : 16 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 173 175 174 170 171 169 168 184 185 187 186 190 191 189 188 180 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 ---------------------------------------------------------------------------------------- 0 | 2649 2654 2581 2604 2740 2785 2703 2686 2696 2705 2659 2644 2758 2704 2615 2632 1 | 2539 2556 2533 2518 2571 2588 2621 2620 2604 2596 2589 2575 2606 2581 2575 2579 2 | 2522 2551 2558 2575 2604 2607 2594 2594 2600 2582 2583 2578 2617 2612 2595 2562 3 | 2569 2563 2559 2557 2582 2574 2593 2634 2620 2616 2612 2596 2618 2589 2611 2616 4 | 2510 2505 2507 2535 2588 2597 2604 2627 2590 2585 2585 2568 2593 2562 2565 2555

Quick data inspection

16

Data APIs

• SRS Crate data are conceptually 3-dimensional

• {hybrid, channel, time sample} ---- NO need to decode the hybrid data on your own

• You can query the size of any dimension

• You can get a hybrid's raw samples if you need that

• Expert access to anything you can think of

• Yet simple non-expert access for rapid results

• Any future raw data format changes are carried by the access libraries and hidden from your code

17

SRS interfaces• Getting the actual ADC samples

for ( j = 0; j< 128; j++) { for (i = 0; i< p->iValue(0, “NSAMPLES”); i++) { int c = p->iValue ( j, i, 0); if (c > 10) h2->Fill ( i, j, pedestal[j] - c ); }

How many actual time samples / slots did hybrid 0 have ? (28 here)

Give me the ADC value for channel “j”, time slot “i”, hybrid 0

18

Special Events

• Your DAQ is only half as good if it doesn't allow you to capture “external” parameters about your data

• A description, numeric parameters such as HV, device configuration parameters, position of a collimator, temperatures, and so on - “Slow controls” in general

rcdaq_client create_device device_srsparams 9 905 10.0.0.2rcdaq_client create_device device_srs 1 1010 10.0.0.2

• The “device_srsparams” doesn't read any SRS data, but reads back all configuration parameters from the system and saves them in a packet. They are in the data stream, and cannot get separated from the data. The “9” refers to a begin-run event, which is generated when we start data taking

• “devices” to read (small) files, such as the config script itself (text)

• Another one to extract numbers from a file and store them as actual numbers

• You can capture virtually all setup parameters automatically

19

ELOG Support

rcdaq_client daq_clear_readlistrcdaq_client load librcdaqplugin_srs.so

rcdaq_client create_device device_srs 1 1010 10.0.0.2

rcdaq_client elog localhost 666 RCDAQLog

Stefan Ritt's elog is the workhorse electronic logbook RCDAQ can make automated entries in the logWe just need to tell it the whereabouts of the log (host, port, name)

20

GUIs

• MLP's rule: GUIs must not be stateful

• Statelessness allows to have multiple copies of the same GUIs at the same time

• Allows to click start on one instance, click stop on another

• And allows to mix GUIs with commands and scripts

• (also makes GUI design so much easier)

21

Our setup

SRS Crate

GEM Stack

Hybrid

HDMI Cable

DAQ machine (we have something more powerful, yes)

22

X-Ray scan over 10x10 positions

The x-ray gun has an aperture of 0.14x4mm

The whole data set was acquired and analyzed “on autopilot”

This was mostly a showcase for my co-workers how this works

All information (x-ray gun position, everything) is included in the data

23

How did we do the “autopilot” aspect?

We set up two “devices” reading a file “position.txt”

rcdaq_client create_device device_file 9 900 $0rcdaq_client create_device device_file 9 905 position.txtrcdaq_client create_device device_filenumbers 9 906 position.txt

rcdaq_client create_device device_srs 1 1010 10.0.0.2

For a long list of x-y positions of the x-ray gun, the main script “drives” the step motor there, and writes the current x-y positions to “positions.txt”

Start taking data, wait for a given number of events, stop

Move to next position

$ ddump -O -p 905 -t 9 /data/purschke/srsdata/xrayscan-00002000-0000.evtX-ray gun position 15748 7874 in run 2000 157480000787400002000$$ ddump -O -p 906 -t 9 /data/purschke/srsdata/xrayscan-00002000-0000.evtPacket 906 8 -1 (ONCS Packet) 6 (ID4EVT) 0 | 157480000 78740000 2000

This saves the file as text

This extracts the (3) numbers

24

Question:

At the last meeting at CERN we had a live demonstration of the system

I could bring in the BNL system on Thursday, sit with interested folks somewhere in a quiet place

Show of hands here, or if you are shy, email me later, and I'll try to set something up in a quiet corner

25

Summary

The RCDAQ + assorted analysis software is a viable alternative to the ALICE system

“full” DAQ system, device data, auxiliary data, elog support

Lightweight and easy

In production at BNL

System is available to RD51 members (for now, email me)

Fast readout (little overhead, reached Gbit line speed)

Largely Linux-flavor agnostic, you don't have to devote your system to it

Mac implementation might come – should work on any Unix flavor

Has all analysis support, setup examples, GUIs, you name it.

26

Backups

27

Interface to our SRS data...• Getting the raw samples for hybrid 0:

for (i = 0; i< p->iValue(0, “NRAWSAMPLES”); i++) { h1->Fill( i, p->iValue(i, 0, "RAWSAMPLES")); }

How many raw samples did the FEC card give for hybrid 0?

Give me FEC raw sample “i” for hybrid 0

28

“HV Scan”

#! /bin/sh

INITIAL_RUN=$1

[ -n "$INITIAL_RUN" ] || INITIAL_RUN=1000

RUN=$INITIAL_RUN

for highvoltage in $( seq 1500 10 1600 ) ; do

# here you would do something to actually set the HV

cat >> description.txt <<EOFThis is Run $RUN with the High Voltage set to $highvoltageThis uses sample number5of the calorimeter.EOF

echo "starting run $RUN with High Voltage set to $highvoltage" rcdaq_client daq_begin $RUN

# engineer some way for the run to end somehow, automatic end after # a given number of events, or some such. Here we do

sleep 30 rcdaq_client daq_enddone

$ ddump -O -t 9 -p 910 rcdaq-00001001-0000.evtThis is Run 1001 with the High Voltage set to 1510This uses sample number5of the calorimeter.$ ddump -O -t 9 -p 911 rcdaq-00001001-0000.evtPacket 911 6 -1 (ONCS Packet) 6 (ID4EVT)

0 | 5e6 5$ ddump -O -t 9 -g -d -p 911 rcdaq-00001001-0000.evtPacket 911 6 -1 (ONCS Packet) 6 (ID4EVT)

0 | 1510 5$

0x5e6 = 1510

29

A neat trick

#! /bin/sh

rcdaq_client daq_clear_readlistrcdaq_client load librcdaqplugin_srs.so

rcdaq_client create_device device_file 9 910 description.txtrcdaq_client create_device device_filenumbers 9 911 description.txtrcdaq_client create_device device_srs 1 1010 10.0.0.2

ELOG=`which elog`[ -n "$ELOG" ] && rcdaq_client elog localhost 666 RCDAQLog

• Say you have a rcdaq setup script such as

rcdaq_client create_device device_file 9 900 $0

This will cause this script itself to to be captured so you always know how rcdaq was configured in this run

30

pmonitor

Evolved as a PHENIX standard monitoring framework

Was soon recognized as a valuable tool for offline processing (just leave the online capabilities alone)

Supports online/interactive and batch-style operation

Still has all the online power under the hood

Relieves the user of the mundane and repetitive tasks (opening a stream, event loop, inspection points, flow control)

Shields the user from the complexities of root linking

Much, much easier yet much more powerful

31

Monitoring

What's the big deal???

Normally when you process data, you run through a dataset. When done, you save histograms, then look at distributions, fit, whatever

In a monitoring process, you want to look at histograms etc while the processing of data is in progress. You want instantaneous feedback how you detector is doing, etc

Manipulate histograms/objects on the fly, plot, fit, reset, rescale, draw,...

Control the processing (update gains, scale factors, what have you)

32

Repetitive tasks

How does a standard analysis process look like?

Initialize (book histograms, set up constants, read database, whatever)

Choose what data stream to open

Open it

Loop through the events in the stream

Call analysis routine for each event

Close stream

Save output

Initialize (book histograms, set up constants, read database, whatever)

Choose what data stream to open

Open it

Loop through the events in the stream

Call analysis routine for each event

Close stream

Save output

Only those are really project-specific

Pmonitor relieves you of all repetitive (white) tasks in the abovelist (and then some)