Organizing Data: A Challenge or just a Bureaucratic...

Organizing Data:A Challenge or just a Bureaucratic Task?

João Luiz Kohl Moreira

Observatório Nacional - MCT

Oct-2012

João Luiz Kohl Moreira Organizing Data

Area of Interest

Software for astronomy

Astronomical Data Analysis Software and Systems(ADASS) - Conferences since 1991.

Is it science?


What is science?

Popper:1 Collect data (observing)

2 Taxonomy

1 Organize data2 Associate pieces of data with each other

3 Model & Theory


What is science?

Popper:1 Collect data (observing)2 Taxonomy


3 Model & Theory


What is science?


1 Organize data

2 Associate pieces of data with each other

3 Model & Theory


What is science?



3 Model & Theory


Myself

Doing science is to find out paradoxes


Where is it being done?

USANOAO: National Optical Astronomy Observatory -Tucson - Az

Manage KPO and CTIO andDevelop an integrated analysis facility: IRAF - ImageReduction and Analysis Facility

STScI: Space Telescope Science Institute

Operate Scientific Data from Hubble Telescope andDevelop an integrated analysis facility: STSDAS - SpaceTelescope Science Data Analysis System

Illinois, Princeton, Caltech, SAO ...

CANADA: CDCEUROPE

StrasbourgMunichUK


Virtual Observatory: Who does it?

IVOA: International Virtual Observatory Alliance: 19members

Brasil: BRAVO


Data Handle

Data:

Every day generated data in astronomy is measuredthrough units of terabytesMoore’s law for astronomy: data volume and datarate doubles each 1.5 yr


Astronomical Moore’s Law


Data Analysis

Who does it?What for?

Data mining

V.O.


V.O. Operation Schema

Raw data

RegistryCenters

CustomApps

V.O.


CosmoBook

CosmoBook


Some Facilities (to be) available on CosmoBook

1 2DPHOT (La Barbera etal, 2008)2 Solar V.O. (A. Valio, U. Mackenzie, SP)3 IRAF Work-flow (CEFET, Rio)4 GMA V.O. (F. Ferrari, FURG)


Social Network


2DPHOT

La Barbera, F. et al (2008) PASP 120, 681-702


2DPHOT


Data administration

“Astronomer” data −→ DATABASE −→ Custom

Astronomers organize data according to their specificrules (conceptual rules)Database is organized according to specific rules(structured data rules)


Data administration

“Astronomer” data −→ DATABASE −→ CustomAstronomers organize data according to their specificrules (conceptual rules)

Database is organized according to specific rules(structured data rules)


Data administration

“Astronomer” data −→ DATABASE −→ CustomAstronomers organize data according to their specificrules (conceptual rules)Database is organized according to specific rules(structured data rules)


Two problems

�� DATABASE −→ Custom

�� Astronomical data −→ DATABASE


Two problems

�� DATABASE −→ Custom�� Astronomical data −→ DATABASE


Why is it‘ problematic?

Astronomers are concerned about science, thecomprehension of the universe

Database designers are concerned about theintegrity of data


Why is it‘ problematic?

Astronomers are concerned about science, thecomprehension of the universeDatabase designers are concerned about theintegrity of data


Data −→ Custom

Databasestructured tables

Database views


Views

idBands Value

1 V

2 B

3 r

4 i

BandsidObj idBand Value

17711 1 12.7

17728 1 13.9

17711 3 16.5

17711 4 19.2

17728 3 17.1

17711 2 16.9

Mags

Database

idObj mV

mB

mr

mi

17711 12.7 16.9 16.5 19.2

17728 13.9 17.1

... ... ... ... ...

Astronomical


Querying database

Tell me, oracle...


A query language

SQL (Structured Query Language) - trying to beintuitive

SNINFE: SQL is Not IRAF, Not FORTRAN Even...


A query language

SQL (Structured Query Language) - trying to beintuitiveSNINFE: SQL is Not IRAF, Not FORTRAN Even...


NVO Advanced SkyQuery

Challenge: Create an effective way for ’real’astronomers to easily query V.O.


Dilemma

Easier −→ less options −→ less enlightening


Data −→ DATABASE

Database load process

Databasestructured tables

Astronomicaltables

?


To take into account

1 Data in astronomical tables are not static;

astronomers use to change data in their tables

2 Load process is then frequent3 Each astronomical dataset needs a specific load

program4 Lives would be wasted in writing load programs





2 Load process is then frequent

3 Each astronomical dataset needs a specific loadprogram

4 Lives would be wasted in writing load programs






program

4 Lives would be wasted in writing load programs






program4 Lives would be wasted in writing load programs


Semiology: Schema E.R.C.

“Rélation Expression - Contenu”

Roland Barthes (1964), Elements of Semiology, Ed. Hilland Wang


Semiology: Schema E.R.C.

Expression Content

Relation

Result: Meta-language


A Control File (YAML)


Data Structure (YAML)


Data file (Tabular)


Scenario (ideal) for CosmoBook

User:

creates his own database (based on some availablestandards);generates the syntax for load program to read datafiles and store data in a structured way;retrieve his data from database



User:

creates his own database (based on some availablestandards);

generates the syntax for load program to read datafiles and store data in a structured way;retrieve his data from database



User:

creates his own database (based on some availablestandards);generates the syntax for load program to read datafiles and store data in a structured way;

retrieve his data from database



User:

creates his own database (based on some availablestandards);generates the syntax for load program to read datafiles and store data in a structured way;retrieve his data from database


The End

“This is Apollo 13, signing off”James Lovell - Cmt


Organizing Data: A Challenge or just a Bureaucratic...

Documents

Transcript of Organizing Data: A Challenge or just a Bureaucratic...