NWO Meeting 24 May 04

a Virtual Survey System

AstroWise- OmegaCAM – Lofar – Euro-VO

a Virtual Survey System

AstroWise- OmegaCAM – Lofar – Euro-VO

Edwin A. ValentijnEdwin A. Valentijn

NWO Meeting 24 May 04

statementstatement

• Big steps with small groups of people– serve individual research groups– don’t set out to heal the world– sociology

• Astronomers and very large db– 1989 ESO-LV 30.000 images– Work station– Sloan Digital Sky Survey– MicroSoft Research– AstroWISE – Euro VO

• Big steps with small groups of people– serve individual research groups– don’t set out to heal the world– sociology

• Astronomers and very large db– 1989 ESO-LV 30.000 images– Work station– Sloan Digital Sky Survey– MicroSoft Research– AstroWISE – Euro VO

NWO Meeting 24 May 04

VSTVST

NWO Meeting 24 May 04

OmegaCAM DetectorsOmegaCAM Detectors

• Science array 1 x 1 degree, 32 CCDs– 15 m pixels – 0.21 arcsec/pixel– Marconi (former EEV) 2k x 4k= 8 Mpix– 16k x 16k pixels = 256 Mpix

• Auxiliary CCD’s – 4 CCDs– For guiding– Image analysis

• Science array 1 x 1 degree, 32 CCDs– 15 m pixels – 0.21 arcsec/pixel– Marconi (former EEV) 2k x 4k= 8 Mpix– 16k x 16k pixels = 256 Mpix

• Auxiliary CCD’s – 4 CCDs– For guiding– Image analysis

NWO Meeting 24 May 04

Paranal Paranal

Mid 2005Mid 2005

+ 3 year+ 3 year

NWO Meeting 24 May 04

LofarLofar

IBM- Blue Gene/LIBM- Blue Gene/L

NWO Meeting 24 May 04

Large Data VolumeLarge Data Volume

• Wide-field imaging = vast amounts of data– VST maps equivalent of Southern sky in 3 years 30 min exp, 300 nights/y. 100 Tbyte image data Tbytes source list data

• Wide-field imaging = vast amounts of data– VST maps equivalent of Southern sky in 3 years 30 min exp, 300 nights/y. 100 Tbyte image data Tbytes source list data

• Handling of the data is non-trivial– Pipeline data reduction– Calibration – Image comparisons/combinations– source lists / catalogues– Visualization

• Science can only be archive based

• Handling of the data is non-trivial– Pipeline data reduction– Calibration – Image comparisons/combinations– source lists / catalogues– Visualization

• Science can only be archive based

NWO Meeting 24 May 04

ASTRO-WISE Mission Virtual Survey SystemASTRO-WISE Mission Virtual Survey System

• Environment that provides systematic and controlled– Access to all raw and calibration data– Execution and modification image/calibration pipelines– Execution of source extraction algorithms- catalogues– Archiving or regenerate dynamically– federated to link different data centers

• Environment that provides systematic and controlled– Access to all raw and calibration data– Execution and modification image/calibration pipelines– Execution of source extraction algorithms- catalogues– Archiving or regenerate dynamically– federated to link different data centers

• Dynamical archive continuously grows-- small or large science projects– generating and checking calibration data– sharing methods, scripts

• Dynamical archive continuously grows-- small or large science projects– generating and checking calibration data– sharing methods, scripts

raw pixel data pipelines/cal files catalogues raw pixel data pipelines/cal files catalogues

NWO Meeting 24 May 04

New technologies-paradigmNew technologies-paradigm

• raw data centralized – on the fly reprocessing

• sociology- project management• ++ • Python • parallel – IP - seti@home• user tunable pipelines• ++ db• 5 LS – 5 lines script

• raw data centralized – on the fly reprocessing

• sociology- project management• ++ • Python • parallel – IP - seti@home• user tunable pipelines• ++ db• 5 LS – 5 lines script

NWO Meeting 24 May 04

development issuesdevelopment issues

• Indexing Terabytes on the sky• Associating entries • World wide linking entries

(referencing) - registries• Filtering• Visualisation - imaging • Visualisation -multidimensional

data• VO world

• Indexing Terabytes on the sky• Associating entries • World wide linking entries

(referencing) - registries• Filtering• Visualisation - imaging • Visualisation -multidimensional

data• VO world

NWO Meeting 24 May 04

NWO Meeting 24 May 04

VST - Virtual Survey TelescopeVST - Virtual Survey Telescope

SourceList

RA0, Dec0

t0

V filterSource

ListV filter

t2

RA0?, Dec0?

SourceList

V filterRA0?, Dec0?

t1

SS

C

I

TES

A

A

O

B filterRA0, Dec0

SourceList

SourceList

V filterRA0, Dec0

SS

C

I

TES

A

A

OS

S

C

I

TES

A

A

O

SourceList

B filterRA0, Dec0

SourceList

B filterRA1, Dec1

SourceList

V filterRA0, Dec0

SourceList

B filterRA1, Dec1

SourceList

B filterRA0, Dec0

SourceListB - VRA0, Dec0

SourceList

V filterRA0, Dec0

t0

SourceList

V filterRA0?, Dec0?

t1

SourceList

V filter

t2

RA0?, Dec0?

SourceList

V filterRA0?, Dec0?

tn

Proper motion

Red quasars

Supernovae

Tbyte source lists brains make the associations

Tbyte source lists brains make the associations

Link -lists

as fast as possible

NWO Meeting 24 May 04