ALMA Integrated Computing Team

Coordination & Planning Meeting #2 Santiago, 28-29 January 2014

ASDM relational database

Rafael Hiriart / Jorge Avarias

ICT-L1 10-12 December 2013

Original Goals

Our main motivation to create this database was to improve DataCapturer and QuickLook. DataCapture was running out of memory, observations were reaching a limit of ~40

minutes with 32 antennas.

The original DataCapture “incremental writes” architecture – based on incremental writes at the level of the ASDM and XML Store – was dropped because of lack of resources in HLA/ASDM and Archive. It was suggested at this time to explore a relational DB solution.

We implemented a workaround for the Pointing table, the biggest ASDM table. By saving this table incrementally to a file and streaming it at the end of the observation to the binary store, we now support long observations (several hours).

We continued working on DC relational database aiming to improve QuickLook. This subsystem, which currently relies in a complicated collaboration between its own components, DataCapture an Quicklook, can be refactored to be a simple application that queries the database, performs a little processing and plot results.

ICT-L1 10-12 December 2013

Status

First version of the relational database has been delivered (ICT-150) as part of release 10.4. It is working in the OSF.

A new version of QuickLook is being delivered with 10.6.This has become quite urgent because of frequent

“freezes” of QuickLook running in the OMC.Besides, QuickLook plots don't scale well with the

number of antennas. A more interactive application is required, able to support filtering on antennas, scans, baselines, etc.

ICT-L1 10-12 December 2013

Demo

ICT-L1 10-12 December 2013

Implementation

Based on Hibernate The Hibernate mapping file is generated using Java reflection to

know what are the columns and the keys of every table. ASDM Java row classes cannot be used directly as mapping

classes for Hibernate. New wrapper classes were created to circunvent problems:

• Define the no-param constructor.

• Setters need to be public.

• Some getters need to be fixed (they throw runtime exceptions).

Database is auto-generated by Hibernate, currently deployed in Scheduling database.

Custom Hibernate types were implemented to handle simple types like angle, temperature, speed, etc.; and complex types like timeInterval, arrays, IDL enumerations.

ICT-L1 10-12 December 2013

Problems/Improvements

Wrapper classes were manually (and painfully) produced. Mappings are created statically.The system is fragile over changes in the ASDM.The ideal way to create these artefacts would be to

generate them as part of the HLA/ASDM build.

Relationships between tables (FK) are not supported.

Arrays are stored in the database as strings following the ASDM array serialization (<num-dim> <dim1> <dim2> … <value-1> <value-2> ...).

Only Java is supported.

ICT-L1 10-12 December 2013

Our plan We stop here!

The mappings, as they are, fullfill our requirements to refactor QuickLook.

We don't have plans (or resources) to continue dedicating to this task.

• Jorge is in transition to 100% Scheduling.• Other resources in Control are fully allocated.

We are well aware of the interest to use this database in a broader context, but• The database could need significant work/modifications to be useful

for other applications (e.g., QA).

• DB is temporary. If a permanent DB is needed, APO needs to be involved. Maintainance, migrations, data duplication, etc., will need to be addressed.

• These developments are well outside Control group's scope and expertise.

ICT-L1 10-12 December 2013

Decision from Last Leads Meeting

ICT-2048, action for Alisdair Manning:

Provide a list of options to ICT management about possible delays to "fund" making the ASDM relational database permanent, and integrated into the code generation system.

To be discussed in Alisdair's presentation.