Current and Future YARR in a Nutshell · FPGA cards for FE-I4 (IBL) • It tried to perform as much...

UNIVERSITY OF CALIFORNIA

Current and Future YARR in a Nutshell

ITk DAQ Workshop - 27.05.19

�1

Timon Heim - LBNL

Timon Heim �2 ITk DAQ Workshop

Outline

• Motivation and Goals

• Conceptual overview and design philosophy

• Hardware controller

• Front-End chip

• Scans

• Data processing: decoding, histogramming, & analysis

• Next development steps


Motivation

• Example IBL: lab testing → USBPix, stave testing → RCE, operation → ROD/BOC

• Each system had a different code base (sw + fw)

• SW entangled with fw, not easy to simply migrate

• Expert knowledge and maturity of sw lost or not available at next stage

• Had a huge impact on DAQ for operation and how many developers were needed to get it running

• Want to maintain a mature sw and have experts be able to apply their knowledge over a broad spectrum of test scales


Goals

• For ITk we should strive to have software base which can be used for small scale lab tests as well full detector operation

• SW should be common to Pixels and Strips

• This results in certain requirements:

• Agnostic to hardware/firmware

• Somewhat agnostic to Front-End chip type

• Scalable in both senses (small and large)


Where does YARR come from?

• YARR was originally designed as readout system using PCIe FPGA cards for FE-I4 (IBL)

• It tried to perform as much processing as possible in SW

• This resulted in minimum entanglement with FW, the PCIe card simply acted as a FIFO

• After some abstraction of the hardware and chip interface YARR seemed useable as a basis to be expanded and used as ITk SW

• However i.e. there are some remnant from the old days:

• Certain things are general but still in the FE-I4 class

• The hardware interface has certain functions which are driven by the features of the YARR-FW for PCIe (primarily naming)


Conceptual OverviewYARR SW core:• Common sw core used in small lab systems up to full detector readout• Improvements to core transfer to all DAQ systems• Well defined interfaces required

YARR SW Core

ATLAS TDAQInterface

CalibrationConsole

TestbeamMonitor

ROD/BOC

BDAQ

RCE

YARR PCIe

Config DB Result DB

...

User interface HardwareInterface

Config/ResultInterface

ReadoutHardware

✓FELIX ✓

✓

✓

✓

✓

✓

...

FE-I3

FE-I4

RD53

HCC/ABC130

ReadoutASIC

✓✓

✓

FE Interface

HCC/ABCStar


Design Philosophies

• Simple firmware, smart software: the more we do in software, the less we are bound to specific features in hw/fw → hardware agnostic

• Keep it modular: the more often code and structures can be re-used, the better → Front-End chips are more alike than you might think

• Simplicity can be key: carefully balance performance and simplicity, we are bad at writing documentation and the code has to live for the next 10+ years, still don’t want it to be slow of course

• Pipeline it: wherever possible data should only travel in one direction, avoid process interdependency → eases scaling


Hardware Controller

Assumptions:• Interaction with chip can be broken down into sending and receiving of data

• Represented by the Tx and RxCore in the interface• Sending of commands:

• Small to medium bandwidth• Primarily configuration data• Broadcast wherever possible• Some support from firmware, but not necessary for everything

• Receiving of data:• Max. bandwidth• One data stream per front-end object• Will enter processing chain• Only rudimentary decoding done in hardware

• Sending either to one or all chips*• Receiving from all chips (demultiplexing in sw)*

*see future section


TxCoreFiFo style sending of commands

Trigger interface

Interface to a buffer which can be send with a programmed frequency.Can do this from either sw or fw (but better timing via fw).

writeFifo() can be buffered in SW to increase package size, until releaseFifo() is called

https://gitlab.cern.ch/YARR/YARR/blob/master/src/libYarr/include/TxCore.h

https://gitlab.cern.ch/YARR/YARR/blob/master/src/libYarr/include/TxCore.h


RxCoreReading data FiFo style

Raw data object

SW will read until FiFo is empty

If data give as pointer, does not copy data.

https://gitlab.cern.ch/YARR/YARR/blob/master/src/libYarr/include/RawData.hhttps://gitlab.cern.ch/YARR/YARR/blob/master/src/libYarr/include/RxCore.h

https://gitlab.cern.ch/YARR/YARR/blob/master/src/libYarr/include/RawData.h

https://gitlab.cern.ch/YARR/YARR/blob/master/src/libYarr/include/RxCore.h


Front-End Chip Implementation

• Chip only needs to implement a config file interface and basic configuration routines

• Advanced functions determined by scan needs which are not generic

• There will be one object for each chip

• A virtual copy of the chip config is saved within the object

• Wherever possible register should be referred to by object and not by string (can’t avoid this fully)

https://gitlab.cern.ch/YARR/YARR/blob/master/src/libYarr/include/FrontEnd.h

https://gitlab.cern.ch/YARR/YARR/blob/master/src/libYarr/include/FrontEnd.h


Scans• Scans typically do the following:

• Configure all activated chips

• Run a loop actions as nested structure:

• Loop over parameter

• Activate portion of pixels

• Inject & Trigger O(100) times

• Read data

• This is facilitated by the scan engine

• Most loop actions are custom for each chip, there are some more general though


Scans II• All data from one innermost loop iteration is packaged and meta

data describing the current loop state is added

• These data packages are then run through the processing chain

• Most scans are fully described a-priori, except tunings:

• Tunings require a parameter change which depends on the analysis outcome

• FeedbackLoops facilitate the interface to the analysis and allow the analysis to change parameters

• Usually use a “hot or cold” scheme, where the analysis only determines the direction and the LoopAction applies the correct parameter change (LoopAction is in charge of tuning Algorithm)


An Example ScanN

estin

g di

rect

ion

oute

rmos

tin

nerm

ost

LoopAction name

LoopAction config

Fei4MaskLoop:Enables an n-th of all pixel, where n=max.

Fei4DcLoop:Enables an n double columns, where n is defined by max/mode.

Fei4TriggerLoop:Injects and triggers at specified frequency for specified number of times.

StdDataLoop:Collect data during triggering.

Timon Heim 15 ITk DAQ Workshop

Data Processing

Bookkeeper

Raw Data

Scan Engine Data Processor Histogrammer Analyser

Configure

Scan Loop

Read Data

Pick up Event

Histogrammer

Publish Histogram

Pick upHistogram

Analysis

PublishResult

Pick upRaw Data

Process and build Event

Raw DataRaw Data

Raw DataEvents

Raw DataEvents

Raw DataEvents

Histogram

Raw DataHistogram

Raw DataHistogram

Raw DataResult

Raw DataResult

Raw DataResult

Tuning Feedback

Datacontainer

Thread

Algorithm

One thread per FE

ParallelisedMonolithic


Data Pipeline


scanConsole• scanConsole is the main tool which will be used for calibration

Important command line arguments:• -c : the connectivity tell scanConsole which chip is connected where and also points to

the right chip config• -r : the controller config tell scanConsole which controller to use and how to configure it• -s : the scan config contains all necessary information to construct the scan


configs


configs II


Work In Progress


Scaling It Up

• How to scale this up?

• Break pipeline into pieces distributed over multiple machines

• Have multiple scan engines delivering data to a central or multiple central data processing servers

• Requires:

• Orchestration of scan engines and data processors, distribution of configuration to all sub-processors (RPC)

• Serialisation of data in between processes (IPC)

https://indico.cern.ch/event/609081/contributions/2636091/attachments/1483038/2300644/ItkWeek_SW_20170626.pdf

https://indico.cern.ch/event/609081/contributions/2636091/attachments/1483038/2300644/ItkWeek_SW_20170626.pdf

Timon Heim �22 Meeting Name

Title Text

Hideyuki Oide 2017-06-26

The current scan operation model being assumed

For performing a scan, for each hardware, there is a software process which exclusively governs the control of configuration and trigger (the one that TxCore() and RxCore() are equipped). This part of the software module is referred to as “Scan Engine”.

Each scan engine is agnostic to the presence of the other hardwares.

Each hardware board works in parallel between the start and the end of the scan, but they do the same scan task.

6

FE FE FE …

Hardware

FE FE FE …

Hardware

FE FE FE …

Hardware

ScanEngine ScanEngine ScanEngine

Tx Rx Tx Rx Tx Rx

MasterThe organization of multiple scan engines is synchronization of the configuration and states (in terms of slow control) via high-level messaging.

The data processing (histogramming and analysis) may be performed locally, or delegated to the specialized computing farm allocated in the downstream. (Arbitrarieness of the arrangement should be ensured.)

sync.states sync.

states

sync.states

these work independently (asynchronously)

Expected to be applicableto most use-cases in the lab.


Title Text

Hideyuki Oide 2017-06-26

Data flow designWish to have flexibility of grouping of the function modules within a process.

Object data need to support serialization.

14

TxCore Data In

Hardware

RxCore Data OutBitStream

Config/Trigger

ScanEngine

DataProcessor Histogrammer

OutEventDataIn

Data Out

In

Analysis

Out

RawDataContainer

Wrapping

Decoding Fill Accum

TxRx

FE

TxRx

FE

TxRx

FE

InOut

Histogrammer

Out In

Analysis

OutFill AccumIn

Histogrammer

Out In

Analysis

OutFill AccumIn

ScanCrewAnaCrew

Data needs serialization!

(instance)

(instance)


Generic Data Processor

• Already heavily rely on nlohmann::json, convenient format also for serialisation

• Use msgpack to serialise json object

• Where json is too costly in terms of memory or bandwidth, usually already have handy RawData format

• By performing some optimisation to nlohmann::json could even be used as a histogram container (see recent work from Matthias)

https://msgpack.org/index.html


FELIX Integration

• FELIX is in many ways similar to YARR-PCIe as it is trying to stay agnostic and just shuffle data from and to the chip

• However primarily difference is that one does not interact directly with FELIX, but rather NetIO

• NetIO is an IPC package and enables subscription to single data channels

• NetIO has been successfully implemented as a hardware controller, however the interface is somewhat unoptimised towards it


NetIO optimisatione-link0

e-link1

e-link2

...

RxCore DataProcessor

Histogrammer

Histogrammer

Histogrammer

...

Currently assuming data comes through

single interface, hence demultiplexing

in DataProcessor.For NetIO we have to aggregate again

because of this.

e-link0 DataProcessor Histogrammer

RxCoree-link1 DataProcessor Histogrammer

If data is available already demultiplexed, should just pass it on

Otherwise hw specific RxCore takes care of

demux

Timon Heim

YARR

�27 ITk DAQ Workshop

Database

local DB Production DB

Viewer

YARRQC Analysis

Retriever

Configs Scans

local DB:• Possibly gitDB based (prototype exists

but needs some scrutinisation)• Via git features can be used to sync

configs over multiple machines or even to other institutes (aka remotes)

• All stored files are json based, plane file editing still possible

Production DB:• Should only store good and interpreted

QC data (result based on input from multiple scans)

• Can retrieve configs from last step (or before) to local DB

• QC Analysis can also take other source of data into account (e.g. pictures)

Multiple scans/configs

QC result

Config fromlast step

Viewer\ Editor

Other sourcesof data


Further Outlook• Target supporting larger system tests:

• Distributed processing

• O(100) chip operation

• FELIX

• Develop and document routines for QC

• Interaction with database

• Also interesting for Strips as we will run surface tests with FELIX and have to compare to previous QC

• Test and benchmark detector-level operation of the code

• Pulling/Pushing configurations from DB

• Crashing sub-processes

• Develop and implement SW ROD


Backup


References

• Gitlab: https://gitlab.cern.ch/YARR/YARR

•

https://gitlab.cern.ch/YARR/YARR


An example

Computer

Kintex7FMC Adapter Card

PCIe Card

PCIe

CPU

Memory

YARR SWModuleModuleModuleModules

RD53A, FE-I4, FE65P2

YARR FW

HW: https://gitlab.cern.ch/YARR/YARR-FW

DisplayPort Cable

RD53A

Ohio AdapterPCIe FPGA

https://gitlab.cern.ch/YARR/YARR-FW


Loop Actions

init()

execPart1()

execPart2()

end()

init()

execPart1()

init()

execPart1()

execPart2()

end()

execPart2()

end()

Loop 1 Loop 2 Loop 3

has inner

has inner

done

done

done

!(has inner)!done

!done!done

Current and Future YARR in a Nutshell · FPGA cards for FE-I4 (IBL) • It tried to perform as much...

Documents

Transcript of Current and Future YARR in a Nutshell · FPGA cards for FE-I4 (IBL) • It tried to perform as much...