Crest project description · Andrea Formica: Svom dev meeting CPPM Conditions data Outline •...

Crest project description

Andrea Formica IRFU

Conditions dataAndrea Formica: Svom dev meeting CPPM

Outline

• Crest project‣ Which use cases are addressed

‣ It belong to the set of commons service we are starting to develop for SVOM community

• Data model‣ Description and functionalities overview

• Architecture‣ Toward a full REST access for conditions

2


Conditions data in Crest

3

• Attempting a definition‣ Conditions/Calibration data are non-event data accessed

pipelines

• Concrete examples‣ The above definition may cover:

- Calibrations data from instrument center

- An additional step using CALDB is needed in order to deliver these data to pipelines if they use existing software

- Part of the configuration data needed for pipelines


Crest data model

4

ATLAS_PHYS_COND.GLOBAL_TAG_MAP*PF GLOBAL_TAG_NAME VARCHAR2 (100 CHAR)*P LABEL VARCHAR2 (100 CHAR)*P RECORD VARCHAR2 (100 CHAR)*F TAG_NAME VARCHAR2 (100 CHAR)

GLOBAL_TAG_MAP_PK (GLOBAL_TAG_NAME, LABEL, RECORD)

FK_2LB1J0I8MW5YFR7OCPYSVRV56 (GLOBAL_TAG_NAME)FK_76CJJ2HVL9AKLLLG6E3YDNT91 (TAG_NAME)

ATLAS_PHYS_COND.IOV*P INSERTION_TIME TIMESTAMP*P SINCE NUMBER (38)*PF TAG_NAME VARCHAR2 (256 CHAR)*F PAYLOAD_HASH VARCHAR2 (64 CHAR)

IOV_PK (TAG_NAME, SINCE, INSERTION_TIME)

FK_EMC1JLDXMYST41O982A1SPF6 (TAG_NAME)FK_IOV_PAYLOAD (PAYLOAD_HASH)


ATLAS_PHYS_COND.PAYLOAD*P HASH VARCHAR2 (64 CHAR)* DATA BLOB* INSERTION_TIME TIMESTAMP* OBJECT_TYPE VARCHAR2 (100 CHAR)* STREAMER_INFO BLOB* VERSION VARCHAR2 (20 CHAR)

PAYLOAD_PK (HASH)

ATLAS_PHYS_COND.TAG*P NAME VARCHAR2 (256 CHAR)* DESCRIPTION VARCHAR2 (4000 CHAR)* END_OF_VALIDITY NUMBER (38)* INSERTION_TIME TIMESTAMP* LAST_VALIDATED_TIME NUMBER (38)* MODIFICATION_TIME TIMESTAMP* OBJECT_TYPE VARCHAR2 (256 CHAR)* SYNCHRONIZATION VARCHAR2 (20 CHAR)* TIME_TYPE VARCHAR2 (16 CHAR)

TAG_PK (NAME)

ATLAS_PHYS_COND.GLOBAL_TAG*P NAME VARCHAR2 (100 CHAR)* DESCRIPTION VARCHAR2 (4000 CHAR)* INSERTION_TIME TIMESTAMP* RELEASE VARCHAR2 (100 CHAR)* SCENARIO VARCHAR2 (100 CHAR)* SNAPSHOT_TIME TIMESTAMP* TYPE CHAR (1 CHAR)* VALIDITY NUMBER (38)* WORKFLOW VARCHAR2 (100 CHAR)

GLOBAL_TAG_PK (NAME)

5 tables: global tags, tags, mappings, iovs, payloadsAll conditions stored in the same schema

Simple ManyToMany mapping between tags and global tags


Crest data model

5









PAYLOAD_PK (HASH)


TAG_PK (NAME)



IOVs are metadata as wellThey can be queried separately from payload

Simple OneToMany mapping between tags and iovs


Crest data model

6









PAYLOAD_PK (HASH)


TAG_PK (NAME)



IOVs request for global tag with snapshot timeSnapshot time can be used to say that a global tag needs to access IOVs inserted “before” that time. Useful for reprocessing campaign.

Can access older version of IOVs


Crest data model

7









PAYLOAD_PK (HASH)


TAG_PK (NAME)



PAYLOADs are Blobs (files)They can be accessed by individual queries

Simple OneToMany mapping between iovs and payloads PAYLOADs hide conditions complexity (channels,…)


Data model in words

8

• Conditions data‣ Payload : values are consumed as an aggregated set,

typically an header and some parameters container(s)

• Conditions metadata‣ IOV: time information, based on 1 time column (time, run

number, …), valid until the next entry in timean IOV point to 1 payload (via an sha256 hash)

‣ Tag: label to identify a specific set of IOVs

‣ Global Tag: consistent set of tags, involved in a given data-flow. A tag can be associated to many global tags.


Crest architecture

• Designing the global architecture of the system:‣ Take into account the caching layer

‣ Keep the different components well decoupled

‣ Reuse tools and libraries in Open Source community

• The proposed architecture is a multi-tier model, with the middleware as a simple REST web application‣ cacheable, language independent

9


Crest architecture

• Components‣ Database, Backend services , Frontend controllers, clients

10

Crest DB

JPA JDBC

HTTPJAX-RS

DataPkg

DataRepositories

WebControllers

Oracle

Sqlite

Postgres

h2

EXPERT(Python, Java,

C++, Javascript)


Crest DB server• As Frontier, the CrestDB server exposes functionalities via REST

‣ But SQL is not involved in the dialogue between client and server, instead the internal resources are accessible via URLs

‣ All HTTP verbs can be used: POST/PUT (to create/update resources), GET and DELETE… - GET <server>/tags/A_TAG_NAME => retrieve resource with id A_TAG_NAME - POST <server>/tags {request body} => create new resource (parameters are taken from body ) - PUT <server>/tags/A_TAG_NAME {request body} => update resource A_TAG_NAME - DELETE <server>/tags/A_TAG_NAME => delete resource A_TAG_NAME

‣ Request and Response bodies are formatted in JSON

‣ Header of the requests can be used (e.g. for formatting the output, deal with caching related parameter etc.)

• Prototype Implementation‣ Based on standard Java technologies (JEE, Spring) and specifications (JAX-RS, JPA)

‣ Can be deployed in the same Tomcat server as Frontier

11


Lexicon• JEE

‣ Java Enterprise Edition: is a set of specifications, extending Java SE[1] with specifications for enterprise features such as distributed computing and web services.[2] Java EE applications are run on referencing runtimes, that can be microservices or application servers, which handle transactions, security, scalability, concurrency and management of the components it is deploying.

• Spring‣ is an application framework and inversion of control container for the Java platform. The framework's core

features can be used by any Java application, but there are extensions for building web applications on top of the Java EE (Enterprise Edition) platform.

• JAX-RS (reference implementation: Oracle Jersey)‣ Java API for Restful Web Services : provides support in creating web services according to

the Representational State Transfer (REST) architectural pattern.[1] JAX-RS uses annotations, introduced in Java SE 5, to simplify the development and deployment of web service clients and endpoints.

• JPA (reference implementation: RedHat Hibernate)‣ Java Persistence API: specification that describes the management of relational data in

applications using Java Platform, Standard Edition and Java Platform, Enterprise Edition.

12

https://en.wikipedia.org/wiki/Java_SE

https://en.wikipedia.org/wiki/Distributed_computing

https://en.wikipedia.org/wiki/Web_service

https://en.wikipedia.org/wiki/Application_server

https://en.wikipedia.org/wiki/Concurrency_(computer_science)

https://en.wikipedia.org/wiki/Application_framework

https://en.wikipedia.org/wiki/Inversion_of_control

https://en.wikipedia.org/wiki/Servlet_container

https://en.wikipedia.org/wiki/Java_platform

https://en.wikipedia.org/wiki/Java_EE

https://en.wikipedia.org/wiki/Web_service

https://en.wikipedia.org/wiki/Representational_State_Transfer

https://en.wikipedia.org/wiki/Java_annotation

https://en.wikipedia.org/wiki/Java_Platform,_Standard_Edition

https://en.wikipedia.org/wiki/Relational_data_model

https://en.wikipedia.org/wiki/Java_Platform,_Standard_Edition

https://en.wikipedia.org/wiki/Java_Platform,_Enterprise_Edition


Spring, Hibernate, JAX-RS examples

• Spring repositories

• Hibernate entities

• JAX-RS Jersey endpoints

13


Crest API• The Crest API is a set of REST methods, allowing

external clients to interact with the resources represented in the data model.‣ Set of methods are today focused on storage of file into the

system using a TAG name and an IOV, and also on retrieval capabilities (search by IOV range etc.)

• Crest API is written using OpenAPI specifications‣ A JSON file describe the URLs (a.k.a. endpoints) and their

parameters, and the Request and Response bodies

‣ Using this file(s) we can generate clients and server stubs in several frameworks, via the swagger-codegen library

14

http://crest-undertow.web.cern.ch/crestapi/swagger.json


Swagger and OpenAPI

• Simplified view of the swagger generation‣ Endpoint definition, component definition

‣ Client and server generation

15

Components(objets)

Endpoints(urls)

JSON - OpenAPI

Server stubs(Python)

Server stubs(Java)

Client(Python)

Client (Java)


Building the prototype• Focus on business code‣ Reduce the amount of boiler plate by using existing

frameworks

16

HTTP Jerseyservlet

/tags

/iovs

/globaltags

Tags controller

Iovs controller

Globaltags controller

Functions to handle tags

Functions to handle iovs

Functions to handle gtags

Request / Responsebody is JSON

DB


Building the prototype

• Generate what you can…

17

HTTP Jerseyservlet

/tags

/iovs

/globaltags

Tags controller

Iovs controller

Globaltags controller

Functions to handle tags

Functions to handle iovs

Functions to handle gtags

Request / Responsebody is JSON

DB


Some URL examples• Retrieve resources

‣ GET method, retrieve tags filter by name: <host>/tags?by=name:ECL&size=1000&sort=objectType:DESC

‣ GET method, select iovs in a range for a given tag:<host>/iovs/selectIovs?tagname=[some tag]&since=0&until=INF

‣ GET method, retrieve one payload:<host>/payloads/[some hash]

• Insert a tag‣ POST method, body is a TAG data transfer object in JSON

<host>/tags. Req body: { ‘name’: ‘ECL_MY_TAG’, ‘objectType’: ‘some type’…}

• Full API documentation‣ To be stored in software.svom.fr (in particular in swagger-ui)

18

http://software.svom.fr


Prototype implementation• Prototype Implementation‣ Based on standard Java technologies (JEE, Spring) and

specifications (JAX-RS, JPA)

‣ Can be deployed in the same Tomcat server as Frontier

‣ Can run as a micro-service (embedded Tomcat, Wildfly, Undertow, Jetty, …all these are servlet containers)

‣ Can build on any system with Java8 (no other dependencies needed, all downloaded automatically)

‣ Multi-DB platform support (thanks to JPA): tested on Oracle, SQLite, H2, Postgresql

19


Continuous Integration

• Crest DB server repository in git‣ https://gitlab.cern.ch/formica/swagger_crestdb

- Exists also in drf-gitlab, but there I cannot yet run the pipeline because it requires Java for compiling

‣ use gitlab-ci to trigger a pipeline when code is committed - Perform unit tests - Create a docker container - Run the container in svomtest.svom.fr

- use an embedded undertow web server (RedHat Wildfly)

20

http://gitlab.cern.ch/formica/swagger_crestdb

Conditions databaseA.Formica: Svom dev meeting CPPM

On going tests

• Use Crest server to test uploads and downloads‣ The server deployed on svomtest.svom.fr has been used

by Chinese colleagues to test network performances for upload and download - Tests are still on going - Chinese colleagues want to deploy as well the server in

China for completeness

http://svomtest.svom.fr

Crest project description · Andrea Formica: Svom dev meeting CPPM Conditions data Outline •...

Documents

Transcript of Crest project description · Andrea Formica: Svom dev meeting CPPM Conditions data Outline •...