EURO-VO workshop 1 July 2005

A Virtual Survey SYSTEMA Virtual Survey SYSTEM

Astro-Wise

NOVA/Kapteyn – OA CapodimonteESO – Terapix – US Munich/MPE

National WFI datacenters NL-I-D-Fr/ESO EU – FP5 RTD programme parallel to AVO

5 year programme -> dec 2006

Astro-Wise

NOVA/Kapteyn – OA CapodimonteESO – Terapix – US Munich/MPE

National WFI datacenters NL-I-D-Fr/ESO EU – FP5 RTD programme parallel to AVO

5 year programme -> dec 2006

Edwin A. ValentijnEdwin A. Valentijn

EURO-VO workshop 1 July 2005

Basic objectivesWide Field imaging EU

Basic objectivesWide Field imaging EU

Facilitate: handling, calibration, quality control, pipelining, user tuned research, archiving, disseminating results

100’s Tbyte of image data and 10’s Tbyte of catalogue dataWith production spread over EU

What-ever –> object model / scalability Where-ever -> federations, GRIDS Who-ever -> Python as glue (+GUIs)

Facilitate: handling, calibration, quality control, pipelining, user tuned research, archiving, disseminating results

100’s Tbyte of image data and 10’s Tbyte of catalogue dataWith production spread over EU

What-ever –> object model / scalability Where-ever -> federations, GRIDS Who-ever -> Python as glue (+GUIs)

(O)MegaCAM(O)MegaCAM

EURO-VO workshop 1 July 2005

statusstatus

• VO conference 2002 - design• Build information system - working• Implemented, qualified WFI@2.2m, WFC@INT, MDM,

OmegaCAM@ILT• >>Qualify with OmegaCAM@VST-2006• >>tune to run Public Surveys• >>quality control• >>Optimize federation/replication

• VO conference 2002 - design• Build information system - working• Implemented, qualified WFI@2.2m, WFC@INT, MDM,

OmegaCAM@ILT• >>Qualify with OmegaCAM@VST-2006• >>tune to run Public Surveys• >>quality control• >>Optimize federation/replication

EURO-VO workshop 1 July 2005

new paradigmtarget processing

new paradigmtarget processing

from:-waterfall/ multi-tier -data pushing-raw data processing-raw data delete-result ->archive -”releases”

from:-waterfall/ multi-tier -data pushing-raw data processing-raw data delete-result ->archive -”releases”

to:-hunting /full linking-data pulling-target processing-raw data -archive -all in archive -request driven

to:-hunting /full linking-data pulling-target processing-raw data -archive -all in archive -request driven

raw pixel data pipelines/cal files cataloguesall integrated in one information system

• distributed services Virtual Survey Telescope• processing GRID • Storage GRID • Methods/services GRID

raw pixel data pipelines/cal files cataloguesall integrated in one information system

• distributed services Virtual Survey Telescope• processing GRID • Storage GRID • Methods/services GRID

EURO-VO workshop 1 July 2005

Astro-Wise VO Properties Benefits integrated dynamic db

Astro-Wise VO Properties Benefits integrated dynamic db

• on-the fly re-processing• 5LS: 5 Lines Script• All bits are traced• Administration for parallel processing compute GRID SETI@home• Global solutions –astrometry/photometry• Build–in workflow• Fully user tunable – own provided script• Context: projects/surveys, instruments, mydb• Publish directly in EURO-VO

• on-the fly re-processing• 5LS: 5 Lines Script• All bits are traced• Administration for parallel processing compute GRID SETI@home• Global solutions –astrometry/photometry• Build–in workflow• Fully user tunable – own provided script• Context: projects/surveys, instruments, mydb• Publish directly in EURO-VO

EURO-VO workshop 1 July 2005

componentscomponents

• Procedures + Cal plan at telescope• Data model -> object model ++ ->++db• Central db ; server/clients

– All I/O except images– Meta data– Source lists = catalogues + associate lists– Links = references = joints

• Fileserver – distributed- via db• Python clients• CVS distributed code base - opipe

• Procedures + Cal plan at telescope• Data model -> object model ++ ->++db• Central db ; server/clients

– All I/O except images– Meta data– Source lists = catalogues + associate lists– Links = references = joints

• Fileserver – distributed- via db• Python clients• CVS distributed code base - opipe

EURO-VO workshop 1 July 2005

Astro-Wise PipelinesAstro-Wise Pipelines

Photometric pipelinePhotometric pipeline

Bias pipeline

Flatfield pipeline

Image pipeline

Source pipeline

EURO-VO workshop 1 July 2005

Target processing:++ the make metaphor

Target processing:++ the make metaphor

awe> targethot=HotPixelMap.get(date='2003-02-14', chip='A5382')

The processing chain is

ReadNoise <-- Bias <-- HotPixels

> class HotPixelMap(ProcesTarget): > > def self.make()

> class ProcessTarget(): > > def get(date, chip) # if not exist/up-to-date then make() > > def exist() # does the target exist? > > def uptodate() # is each dependency up to date?

Fully recursive

awe> targethot=HotPixelMap.get(date='2003-02-14', chip='A5382')

The processing chain is

ReadNoise <-- Bias <-- HotPixels

> class HotPixelMap(ProcesTarget): > > def self.make()

> class ProcessTarget(): > > def get(date, chip) # if not exist/up-to-date then make() > > def exist() # does the target exist? > > def uptodate() # is each dependency up to date?

Fully recursive

EURO-VO workshop 1 July 2005

Intra-operability peer to peerIntra-operability peer to peer

• code base + docs : CVS

• Db: “Advanced Replication” evolving to streaming

• code base + docs : CVS

• Db: “Advanced Replication” evolving to streaming

WRITE

–READ-ONLY

–READ-ONLY

–RE

AD

-ON

LY

–RE

AD

-ON

LY

WRITE

–REPLICATIO

N

–REPLICATION

–RE

PLIC

AT

ION

–RE

PLI

CA

TIO

N

EURO-VO workshop 1 July 2005

Contents of federationContents of federation

• Raw data– Observed images– Ancillary information

• Calibration results– Calibration files time stamped

• Reduced images– Single observation– Co added images

• Software– Methods (pipelines) for processing calibration– Configuration files

• Source lists – catalogues– Extracted source information– Associated among different data objects

• Raw data– Observed images– Ancillary information

• Calibration results– Calibration files time stamped

• Reduced images– Single observation– Co added images

• Software– Methods (pipelines) for processing calibration– Configuration files

• Source lists – catalogues– Extracted source information– Associated among different data objects

EURO-VO workshop 1 July 2005

Example 5LSExample 5LS

#Find ScienceFrames for a ccd named ccd53 and filter

Awe> q = (ReducedScienceFrame.chip.name == 'ccd‘) and (ReducedScienceFrame.filter == ‘841’)

# From the query result, get the rms of the sky in image Awe> x = [k.imstat.stdev for k in q]

# get the rms of the used MasterflatAwe> y = [k.flat.imstat.stdev for k in q]

# Make a plot Awe> pylab.scatter(x,y)

#Find ScienceFrames for a ccd named ccd53 and filter

Awe> q = (ReducedScienceFrame.chip.name == 'ccd‘) and (ReducedScienceFrame.filter == ‘841’)

# From the query result, get the rms of the sky in image Awe> x = [k.imstat.stdev for k in q]

# get the rms of the used MasterflatAwe> y = [k.flat.imstat.stdev for k in q]

# Make a plot Awe> pylab.scatter(x,y)

EURO-VO workshop 1 July 2005

Astro-Wise PORTALAstro-Wise PORTAL

EURO-VO workshop 1 July 2005

Web services- object viewerWeb services- object viewer

EURO-VO workshop 1 July 2005

QC - calibration scientist monitoring

QC - calibration scientist monitoring

EURO-VO workshop 1 July 2005

QC - calibration scientist monitoring

QC - calibration scientist monitoring

EURO-VO workshop 1 July 2005

Web services- object makerWeb services- object maker

EURO-VO workshop 1 July 2005

VST - Virtual Survey TelescopeVST - Virtual Survey Telescope

EURO-VO workshop 1 July 2005

LofarLofar

IBM- Blue Gene/LIBM- Blue Gene/L

EURO-VO workshop 1 July 2005

Thanks!Thanks!

Welcome at next Astro-Wise tutorial!October 2005 Netherlands

www.astro-wise.org

Welcome at next Astro-Wise tutorial!October 2005 Netherlands

www.astro-wise.org