Status and future perspectives of the

analysis language (eDSL) for HWC and LHC Operation

Kajetan FuchsbergerTE-MPE TM, 2014-02-27

On behalf of the TE-MPE-MS Software Team:D. Anderson, M. Audrain, M. Dragu, J.-C. Garnier,

A. Gorzawski, M. Koza, K.H. Krol, A. Moscatelli, K.Stamos,P.C. Turcu

2Kajetan Fuchsberger TE-MPE TM, 2014-01-20, Analysis Language (eDSL) for HWC and Operation

Content

Motivation

Status & Usage

Behind the Scenes

Outlook & Visions

3Kajetan Fuchsberger TE-MPE TM, 2014-01-20, Analysis Language (eDSL) for HWC and Operation

Warning!Vi

siona

rity

Tem

pera

ture

100°

50°

0°

Visionarity Temperature might be High!

(But not unrealistic ;-)

4Kajetan Fuchsberger TE-MPE TM, 2014-01-20, Analysis Language (eDSL) for HWC and Operation

Content

Motivation

Status & Usage

Behind the Scenes

Outlook & Visions

5Kajetan Fuchsberger TE-MPE TM, 2014-01-20, Analysis Language (eDSL) for HWC and Operation

eDSL ?

eDSLDomain

Specific Language

embedded

Hwc; Systems; Accelerators; Signals;

6Kajetan Fuchsberger TE-MPE TM, 2014-01-20, Analysis Language (eDSL) for HWC and Operation

Why a DSL at all? Currently:

E.g. writing a PM module (or an analysis for a test) is pure coding (Java/LV). A lot of knowledge required, which distracts from the real logic: Where to get the data from? What tools to use to display it? What algorithms to apply?

A lot of code duplication. E.g.: E.g. ArrayUtils in PM modules Every application again connects in its own way to the Logging DB,

PM, reads files....

DSL focuses on the logic of the domain:

7Kajetan Fuchsberger TE-MPE TM, 2014-01-20, Analysis Language (eDSL) for HWC and Operation

Why (Java) embedded? Integration within the Accelerator SW Architecture Java

We are not language designers, but software developers.

Reuse: All the native language features of Java, while providing the

required readability for non-programmers. Tools for writing and verification of analysis scripts (eclipse). Strongly typed; with the help of an external library, it was

easy to enforce the correct usage of physical units.

It provides the necessary possibility to constrain the user input to the available functionality.

8Kajetan Fuchsberger TE-MPE TM, 2014-01-20, Analysis Language (eDSL) for HWC and Operation

Where could it be used? Test Analysis within AccTesting (origin)

PM modules

Pre/Post operational checks

periodic calls + reports: E.g. Splice measurements LHC statistics

…?

Any Java Application!

Visio

narit

y Te

mpe

ratu

re

100°

50°

0°

9Kajetan Fuchsberger TE-MPE TM, 2014-01-20, Analysis Language (eDSL) for HWC and Operation

What about a better name?

ZeldaZuper Easy Language

for Data Analysis

Brainstorming….

GENIAL

Generic Integrated Analysis Language

Analysis Language for Operation and Hardware

Commissioning

10Kajetan Fuchsberger TE-MPE TM, 2014-01-20, Analysis Language (eDSL) for HWC and Operation

Content

Motivation

Status & Usage

Behind the Scenes

Outlook & Visions

11Kajetan Fuchsberger TE-MPE TM, 2014-01-20, Analysis Language (eDSL) for HWC and Operation

Actual Status

Courtesy: M. Audrain

Visio

narit

y Te

mpe

ratu

re

100°

50°

0°

12Kajetan Fuchsberger TE-MPE TM, 2014-01-20, Analysis Language (eDSL) for HWC and Operation

Full Automation of PNO.d1 and PCC.1

See Scott & Zinur‘s presentation

Executed ~2000 times within an HWC campaign

Visio

narit

y Te

mpe

ratu

re

100°

50°

0°

13Kajetan Fuchsberger TE-MPE TM, 2014-01-20, Analysis Language (eDSL) for HWC and Operation

What are we missing? More signals from post mortem

(E.g. QPS buffers)

Circuit parameters (deferred scalars)

Calculations:

All logging variables (code generation)

Visio

narit

y Te

mpe

ratu

re

100°

50°

0°

14Kajetan Fuchsberger TE-MPE TM, 2014-01-20, Analysis Language (eDSL) for HWC and Operation

How to test? - Framework On our (SW) side:

http://sonar.cern.ch See also Cristina‘s recent Presentation:https://indico.cern.ch/event/293574/

15Kajetan Fuchsberger TE-MPE TM, 2014-01-20, Analysis Language (eDSL) for HWC and Operation

How to test? – Module Developer

Use a set of known cases for which the module passes or fails.

To be automated (bamboo)!

16Kajetan Fuchsberger TE-MPE TM, 2014-01-20, Analysis Language (eDSL) for HWC and Operation

Systematic Checks against old data

17Kajetan Fuchsberger TE-MPE TM, 2014-01-20, Analysis Language (eDSL) for HWC and Operation

Integration in AccTestingTo come:Add possibility to display arbitrary signals

18Kajetan Fuchsberger TE-MPE TM, 2014-01-20, Analysis Language (eDSL) for HWC and Operation

(mis ;-) use for data extraction Do not care, where data comes from!

Visio

narit

y Te

mpe

ratu

re

100°

50°

0°

19Kajetan Fuchsberger TE-MPE TM, 2014-01-20, Analysis Language (eDSL) for HWC and Operation

Content

Motivation

Status & Usage

Behind the Scenes

Outlook & Visions

20Kajetan Fuchsberger TE-MPE TM, 2014-01-20, Analysis Language (eDSL) for HWC and Operation

Behind the scenes

request

„compile“

Data

Result

Only extract necessary data!

21Kajetan Fuchsberger TE-MPE TM, 2014-01-20, Analysis Language (eDSL) for HWC and Operation

The Tree …

Courtesy: Kamil Krol

22Kajetan Fuchsberger TE-MPE TM, 2014-01-20, Analysis Language (eDSL) for HWC and Operation

Resolving….

Main Principle: Ask the Modules, to resolve the biggest subtree possible in one go.

23Kajetan Fuchsberger TE-MPE TM, 2014-01-20, Analysis Language (eDSL) for HWC and Operation

Content

Motivation

Status & Usage

Behind the Scenes

Outlook & Visions

24Kajetan Fuchsberger TE-MPE TM, 2014-01-20, Analysis Language (eDSL) for HWC and Operation

A more abstract view Requirements for data Analysis:

Data incompleteness (e.g. a beam positions along the ring: added/removed BPMs)

Data invalidity: (e.g BPMs might be misbehaving) Mathematical operations: (e.g.

average/addition/scaling) Physical units: (Different units in storage ->

conversion left to the user) Error propagation: left to the user

Visio

narit

y Te

mpe

ratu

re

100°

50°

0°

25Kajetan Fuchsberger TE-MPE TM, 2014-01-20, Analysis Language (eDSL) for HWC and Operation

Anything out there?

Courtesy:K. Stamos, D. Anderson

Visio

narit

y Te

mpe

ratu

re

100°

50°

0°

26Kajetan Fuchsberger TE-MPE TM, 2014-01-20, Analysis Language (eDSL) for HWC and Operation

Tensorics® Everything is a tensor.

Common Operations.

Direct and deferred Usage.

Choosable strategies(with useful defaults): Error propagation Data incompleteness Units

To be released as Open Source soon.

Courtesy:A. GorzawskiVi

siona

rity

Tem

pera

ture

100°

50°

0°

27Kajetan Fuchsberger TE-MPE TM, 2014-01-20, Analysis Language (eDSL) for HWC and Operation

The dispatcher – possible Improvements

Caching

All operations are Stateless (basically mathematical functions) easy to parallelize.

E. Montesnitsalis: UFO search as an example Use Case

Visio

narit

y Te

mpe

ratu

re

100°

50°

0°

28Kajetan Fuchsberger TE-MPE TM, 2014-01-20, Analysis Language (eDSL) for HWC and Operation

Blockers & Wishes ;-) Java 8: Closures!

Would allow us to make the language more dynamic and easily extensible.

Will be production released on 18th March. Unfortunately blocked by CO. Will not come

before end of Run 2! Cluster!

To evaluate different options (parallelization, Scalability)

Useful also for different other R&D: PM storage, distributed file systems, test environments, R&D for future protection systems.

Visio

narit

y Te

mpe

ratu

re

100°

50°

0°

29Kajetan Fuchsberger TE-MPE TM, 2014-01-20, Analysis Language (eDSL) for HWC and Operation

Only measured data? Circuit Models?

Accelerator Models (MadX)?

Acquisitions (live data)

…

Beam Commissioning?Visio

narit

y Te

mpe

ratu

re

100°

50°

0°

30Kajetan Fuchsberger TE-MPE TM, 2014-01-20, Analysis Language (eDSL) for HWC and Operation

Summary & Outlook Ready (from language point of view) to analyse PNO.d1 and

PCC.1

Next features: More Signals, Circuit parameters (scalars), Calculations.

Switch to Tensorics® when ready.

Optimization (Caching, Parallelization…) to come?… as soon as we can get some cluster!

Can be use in other fields (Models? Big Data analysis? Supervision? Statistics…)

Wishes/dreams: Java8 and a Cluster ;-)

Visio

narit

y Te

mpe

ratu

re

100°

50°

0°Visio

narit

y Te

mpe

ratu

re

100°

50°

0°Visio

narit

y Te

mpe

ratu

re

100°

50°

0°Visio

narit

y Te

mpe

ratu

re

100°

50°

0°Visio

narit

y Te

mpe

ratu

re

100°

50°

0°

31Kajetan Fuchsberger TE-MPE TM, 2014-01-20, Analysis Language (eDSL) for HWC and Operation

Growing a Language http://www.youtube.com/watch?v=_ahvzDzKdB0

Guy Steele ‚Growing a Language‘(Keynote OOPSALA Conference, 1998)

„If one lets the users help do the work, growth can be quick!“

„if many persons work side by side and the best work is added with care and good taste, a great deal can be added in short time.“

32Kajetan Fuchsberger TE-MPE TM, 2014-01-20, Analysis Language (eDSL) for HWC and Operation

Thanks a lot for your Attention!

33Kajetan Fuchsberger TE-MPE TM, 2014-01-20, Analysis Language (eDSL) for HWC and Operation

Resolving…. Ask each resolver: can resolve ?

If a resolver can resolve the given node remember.

If none of the resolvers can resolve the node, then the same procedure is recursively invoked with all the children of the node (e.g. node B in the example).

Else (at least one potential candidate resolver), the dispatcher stops the iteration through the tree in this branch and does not try to resolve the child nodes anymore (in the initial state of the example, this would happen for node C).

As soon as the iterations through all branches are the dispatcher selects one of the candidate resolvers per node, invokes it and rebuilds the tree with the resolved node.

After all resolvers return, the whole algorithm is looped starting at item (1), until all nodes (including the root node) are resolved.

34Kajetan Fuchsberger TE-MPE TM, 2014-01-20, Analysis Language (eDSL) for HWC and Operation

The Tree …

Courtesy: Kamil Krol