REMI Database: A Data Management System for Nuclear ...REMI Facts about REMI Developed for the...

REMI Database: A Data Management System forNuclear Medicine Studies

Antall Johann Fernandes

Florida Institute of Technology

afernandes2010@my.fit.edu

30th November, 2011

Antall Johann Fernandes (Florida Tech.) REMI Database 30th November, 2011 1 / 43

Facts about REMI

Developed for the Nuclear Imaging Group of the Life SciencesDivision at Lawrence Berkeley National Laboratory, California.

Supported by a subcontract on NIH grant 5-R01-EB007219-04,”Molecular Imaging of Cardiac Hypertrophy Using microPET andPinhole SPECT”.

Currently is in production and houses 180 experiments, and 8105 datafiles.

Motivation

Typical Work Process by the Nuclear Imaging Group

Perform a SPECT, DT-MRI or Micro-PET experiment.

Copy experimental data to individual researcher’s workstation.

Perform data analysis, image reconstruction and various otherprocessing tasks on experimental data.

Extract results from processed data.

Publish results and findings based on results.

What happens to the experiment raw data or processed data?

Data resides on individual researcher’s workstation under a researchergenerated directory structure.

Data is moved or copied into a central file server into a researchergenerated location and directory structure.

Data about the experimental data may or may not be captured.

Motivation

Knowledge about Data

What experimental data has been collected?

Where does all of the experimental data reside?

Data Sharing

Can we share the experimental data with others?

Motivation

The Need for a System

Aware of all the experimental data captured over the years.

Knowledge of the location of the experimental data.

Provide the capabilities to share the experimental data.

Sharing Experimental Data

Data should be stored in a standardized manner.

Data should be query-able.

Meta-data needs to be captured in a timely manner. (Meta-data isexplained later).

Users should not need to deal with file systems.

Design Considerations

Typical Scientific Data Management Systems should have...

Creation of logical collections - physical data to logical collections.

Physical data handling - storage of the physical data files.

Security support - data access authorization and change verification.

Data ownership - who is responsible for data quality and meaning.

Persistence - data lifetime.

Knowledge and information discovery - identify useful informationinside the data collection.

Meta-data collection, management and access.

Design Considerations

REMI’s Design Considerations

Open access to downloading of experimental data.

Users should be able to access the database from anywhere.

Authorized researchers should be able to upload experimental data.

REMI should handle the physical storage of data files (explainedlater).

Understanding Experiments and its Data

SPECT Experiment

Data and Meta-Data

Patient Coordinate System

Detector Coordinate System

Meta-data

What is Meta-data?

Information about data.

Describes how data is measured, acquired, and computed.

Enables data browsing, data transfer, and data documentation

Makes it possible to build data independent automated tools.

Entities and Relationships

Entity

A ”thing” or ”object” in the real world that is distinguishable from allother objects. Eg. Person or Vehicle.

An entity has a set of properties and values that identify an entity.

Relationship

An association among several entities. Eg. A Person owns a Vehicle.

Entities and Relationships

Entity-Relationship Data Model

Based on the perception of the real world that consists of a set of basicobjects called entities, and of relationships among these objects.

Mapping Cardinalities

Represents constraints on the relationship.

Expresses the number of entities to which another entity can beassociated via a relationship.

REMI Meta-data Database Development

Logical Entity Relationship Diagram

Physical Table Design Diagram

Physical Table Design Diagram: Machine

Physical Table Design Diagram: Tracer

Physical Table Design Diagram: File Tag

Physical Table Design Diagram: File Association

Owner ID maps to the entity which owns the File.

Physical Table Design Diagram: User

REMI File Storage Development

REMI’s Design Consideration (mentioned earlier)

REMI should handle the physical storage of data files.

File Storage Schemes

(c) Single Directory (d) Directory Hierarchy

All Files under a single directory.

Individual directory for each experiment.

REMI File Storage

Concatenate experiment ID and filename and generate a 64 lengthSHA256 string.

Create directory structure based on SHA256 string, directory levels,and directory name size

Parameter directory name size determines how many characters touse of the hash value

Parameter directory levels determines the depth of the directorystructure

Example 1

SHA256 string: fb80f1735b3153e7a41d38390de5d9773c35259965..directory levels : 2directory name size : 2=>ROOT DIR/fb/80/<experiment id> <filename>.<extension>

REMI File Storage: Example 2

SHA256 string: fb80f1735b3153e7a41d38390de5d9773c35259965..directory levels : 2directory name size : 3=>ROOT DIR/fb8/0f1/<experiment id> <filename>.<extension>

REMI File Storage: Example 3

SHA256 string: fb80f1735b3153e7a41d38390de5d9773c35259965..directory levels : 3directory name size : 2=>ROOT DIR/fb/80/f1/<experiment id> <filename>.<extension>

File Storage

directory levels : 2directory name size : 2

REMI Application Development

Rails Model-View-Controller (MVC) Architecture

System Design

System Component Design

Reasons to upload Data Files in Parts

Web Browsers can upload a maximum of 4 GB of data per request.

Certain data files are in excess of 50 GB in size.

File Upload Process

File Upload Process: Save the chunk files

File Upload Process: Concatenate the chunk files

REMI User Interface Development

JSON (JavaScript Object Notation)

A lightweight data-interchange format.

Easy for humans to read and write.

Built on two structures.

A collection of name/value pairs.Realized as an object, record, struct, dictionary, hash table, keyed list,or associative array.An ordered list of values.Realized as an array, list, or sequence.

JSON Syntax

(e) Object Syntax

(f) Array Syntax

(g) Value Syntax

REMI Menu

id: Translates to the element ID on the HTML page.

value: Display name shown on the user interface.

link: URL to be called.

submenu: Array to contain sub-menus.Antall Johann Fernandes (Florida Tech.) REMI Database 30th November, 2011 33 / 43

Download Menu Options

Searching for Experiment based on Modality

Creating a New SPECT Experiment

Various Sections within the SPECT Experiment

Weaknesses

Weaknesses within REMI

Lack of full support for Semi-Structured Data.

Semi-Structured Data does not conform with the formal structure oftables and data models associated with relational databases.Meta-data is all semi-structured data.

Resume functionality for uploading and downloading data files.

Server side file compression on large files.

Meta-data Templates and User Personalization.

Future Enhancements

Provide support for Semi-Structured Data.

Relational databases with XML support.

Move from a relational model to Entity-Attribute-Value model.

Document based databases.

User Personalization.

Provide user specific views of owned experiments.

Support saving of personalized search queries.

Meta-data

Best approach to modeling meta-data.

Make the meta-data capture process less cumbersome.

Reading meta-data from header files.

Future Enhancements

Resume functionality for Data File Uploads.

Files uploaded in chunks.

Query the server for existing file size on the server.

Transfer file chunks greater than the size on the server.

Resume functionality for Data File Downloads.

Look at HTTP Byte Range Retrieval Extensions.

See how the resume functionality affects the compression process onthe server.

Questions

Demonstration

REMI Website

Thank You

REMI Database: A Data Management System for Nuclear ...REMI Facts about REMI Developed for the...

Documents

Transcript of REMI Database: A Data Management System for Nuclear ...REMI Facts about REMI Developed for the...

National Nuclear Security Report (PER) · National Nuclear Security Administration Lawrence Livermore National Security, LLC Performance Evaluation ... radioisotope thermoelectric

San Onofre Nuclear Waste Problems - The Samuel Lawrence ...

Matrimonio Gabi y Remi

Jutta Escher Nuclear Theory & Modeling Lawrence Livermore National Lab Jutta Escher Nuclear Theory & Modeling Lawrence Livermore National Lab UCRL pending.

Remi/Rough book

Nuclear Science Merit Badge Howard Matis - hsmatis@lbl.gov 1 Nuclear Science Merit Badge Howard Matis Lawrence Berkeley National Laboratory.

March 2007 - Lawrence Livermore National Laboratory · March 2007 National Nuclear Security Administration’s Lawrence Livermore National Laboratory ... center’s director, Ralph

Visual Resume Remi Ramiano

Remi Oyeniran Actor - Singing CV

Remi Project Specsjvnjernjernjvnrjivnmirevnirv;;

Lawrence Livermore National Laboratory Noble liquid and gas detectors for nuclear security Adam Bernstein Advanced Detector Group Leader Lawrence Livermore.

User’s Guide - REMI

Siete maravillas remi

Sizing Remi Plant

Relativistic Nuclear Collisions (RNC) Group Nuclear Science Division (NSD), Lawrence Berkeley National Lab The Relativistic Nuclear Collisions (RNC) group.

Portfolio Remi Buscot EN

Decommissioning of Nuclear Facilities Decommissioning ......Decommissioning of Nuclear Facilities Decommissioning Planning Lawrence E. Boing IAEA Regional Workshop Manila, The Philippines

Remi Pyrdol for Goop Label

REMI TRAINING SERIES SESSION 3: POLICY ANALYSIS WITH …...REMI TRAINING SERIES SESSION 3: POLICY ANALYSIS WITH REMI PI+ REMI WEBINAR ... • If no survey data –use case studies

Remi Elektrotechnik Ltd (ONF)...REMI ELEKTROTECHNIK LIMITED Regd. Office : Remi House. Plot No.ll. Cama Industrial Estate. Goregaon (East), Mumbai - 400 063 CIN : L51900MH1988PLC047157