Virtual Observatories

Wolfgang Voges 1

Virtual Observatories

Wolfgang Voges

Max-Planck-Institut für extraterrestrische Physik

Garching

Workshop ‘‘Astronomie mit Großgeräten‘‘Am 17.Oktober 2003 in Potsdam

Wolfgang Voges 2


Workshop ‘‘Astronomie mit Großgeräten‘‘Am 17.Oktober 2003 in Potsdam

Overview:

Historical roots *

What’s happening in the world: IVOA

European VO-activities

German VO-activities (GAVO) Federation of local data-sets

Next-generation search engine

Grid

Theory in GAVO

Outlook* Viewgraphs partly copied from other presentations

Wolfgang Voges 3


1. VO meeting at Caltec in Pasadena (June 2000)

Astronomers mostly from the US

Very enthusiastic talks

Big vision of the future

Foundation of the NVO (US)

Since then similar meeting in Europe (Garching)

Foundation of national European and later

Other VOs

Historical remarks

Wolfgang Voges 4

The exponential growth of data volume (and complexity, quality)

driven by the exponential growth in information

technology …

… But our understanding of the universe increases much more slowly -- Why?

Methodological bottleneck VO is the answer Human wetware limitations … AI-assisted discovery NGVO?

Data Knowledge ?

Wolfgang Voges 5

How and Where are Discoveries Made?

• Conceptual Discoveries: e.g., Relativity, QM, Brane World, Inflation … Theoretical, may be inspired by observations

• Phenomenological Discoveries: e.g., Dark Matter, QSOs, GRBs, CMBR, Extrasolar Planets, Obscured Universe …

Empirical, inspire theories, can be motivated by them

New TechnicalCapabilities

ObservationalDiscoveries

TheoryIT/VO (VO)

Phenomenological Discoveries:

Pushing along some parameter space axis VO useful

Making new connections (e.g., multi-) VO critical!Understanding of complex astrophysical phenomena requires

complex, information-rich data (and simulations?)

Wolfgang Voges 6

Why is VO a Good Scientific Prospect?• Technological revolutions as the drivers/enablers

of the bursts of scientific growth

• Historical examples in astronomy:– 1960’s: the advent of electronics and access to space

Quasars, CMBR, x-ray astronomy, pulsars, GRBs, …

– 1980’s - 1990’s: computers, digital detectors (CCDs etc.)

Galaxy formation and evolution, extrasolar planets, CMBR fluctuations, dark matter and energy, GRBs, …

– 2000’s and beyond: information technology

The next golden age of discovery in astronomy?

VO is the mechanism to effect this process

Wolfgang Voges 7

SurveysObservatories

Missions

Surveyand

MissionArchives Follow-Up

Telescopesand

Missions

Results

Data Services---------------Data Miningand Analysis,

Target Selection

Digital libraries

Primary Data Providers

VOSecondary

DataProviders

A Schematic Illustration of the VO-Based Science

VO as an integral partof the whole system …

Wolfgang Voges 8

The Changing Style of Observational Astronomy

The Old Way: Now: Future:

Pointed, heterogeneous

observations (~ MB - GB)

Large, homogeneous sky surveys

(multi-TB, ~ 106 - 109 sources)

Multiple, federated sky surveys and archives (~ PB)

Small samples of objects (~ 100 - 103)

Archives of pointed observations (~ TB) Virtual

Observatory

Wolfgang Voges 9

This quantitative change in the information volume and complexity will enable the

Science of a Qualitatively Different Nature:

• Statistical astronomy done right – Precision cosmology, Galactic structure, stellar astrophysics …– Discovery of significant patterns and multivariate correlations– Poissonian errors unimportant

• Systematic exploration of the observable parameter spaces (NB: Energy content = Information content)

– Searches for rare or unknown types of objects and phenomena

– Low surface brightness universe, the time domain …

• Confronting massive numerical simulations with massive data sets

Wolfgang Voges 10

Panchromatic Views of the Universe:A More Complete, Less Biased Picture

Radio Far-Infrared Visible

Visible + X-ray

Dust Map

Galaxy Density Map

Wolfgang Voges 11

Examples of Possible VO Projects:• A Panchromatic View of AGN and Their Evolution

– Cross-matching of surveys, radio to x-ray– Understanding of the selection effects– Obscuration, Type-2 AGN, a complete census Evolution and net energetics, diffuse backgrounds

• A Phase-Space Portrait of Our Galaxy– Matching surveys: visible to NIR (stars), FIR to radio (ISM)– A 3-D picture of stars, gas, and dust, SFR …– Proper motions and gas velocities: a 6-D phase-space picture Structure, dynamics, and formation of the Galaxy

• Galaxy Clusters as Probes of the LSS and its Evolution– Cluster selection using a variety of methods: galaxy overdensity,

x-rays, S-Z effect …– Understanding of the selection effects Probing the evolution of the LSS, cosmology

Wolfgang Voges 12

Exploration of new domains of the observable parameter space: the Time Domain

Faint, Fast Transients (Tyson et al.)

Megaflares on normalMS stars (DPOSS)

Existing and Forthcoming surveys: Microlensing experiments (OGLE, MACHO …)

Solar System patrols, GRB patrols …

DPOSS plate overlaps (Mahabal et al.)

QUEST-2 and NEAT at Palomar… and many, many others …

The future: LSST ?

Wolfgang Voges 13

Data Mining in the Image Domain: Can We Discover New Types of Phenomena Using Automated Pattern

Recognition?(Every object detection algorithm has its biases and limitations)

– Effective parametrization of source morphologies and environments– Multiscale analysis (Also: in the time/lightcurve domain)

Wolfgang Voges 14

Exploration of observable parameter spaces and searches for rare or new types of objects

Wolfgang Voges 15

new, more, better, faster, and easier science comparative analysis of multi-instrument data,

permit new approaches to research and multi-wavelength exploration,

opening discovery capabilities not otherwise possible

This is clearly the primary mandate of all VO efforts

minimise redundancy:data collected by a single telescope / instrument can be re-used multiple times by different teams and for different scientific purposes

data integrity: data are archived and documented in a controlled and uniform fashion, ensuring long-term scientific usage

improving calibrations and creating more higher-level data products to make data more science-ready

Advantages of a Virtual Observatory

Wolfgang Voges 16

interoperability of archives:- strengthening connections to other archives, catalogues and abstract

services for broader research parameter space and links to the literature

advancing technologies for computers, networks, data compression, and storage media:- to retrieve and analyse more information more readily at lower cost

efficient serving of data to the public:

- there will be different levels of end-user from professional astronomers to interested (high-school) students and enthusiastic amateurs – many of whom may undertake projects which are simply unrealisable by large

institutes

data-mining with new software tools and new catalogues of object properties:

- to enable higher-order research based on questions posed in scientific terms


Wolfgang Voges 17

improving the preparation, development, building of new ground-based and space-based projects

improving new observation proposals comparison of real data with simulated data – to provide feedback

to new insights, new models, new physics


Wolfgang Voges 18

Korea, Japan, China, Australia, India, Russia, Hungary, Italy, France, Germany, Europe (ESO++), Canada, USA

International Virtual Observatory Alliance

Wolfgang Voges 19

Virtual ObservatoriesWhat’s happening in the world: IVOA

International STANDARDS are needed

Registry

Data-Models

VO-Table

VO-Query

Uniform Content Descriptors (UCD)

Simple Image Access (SID)

GRID-standards

Tools e.g. data-mining

Wolfgang Voges 20


In the AVO (euro-vo.org) under the leadership

of ESO/ESA the following institutes/groups

are collaborating: ESO

ESA/STECF

University of Edinburgh

CDS Strasbourg

University Louis Pasteur

Centre National de la Reserche Scientifique Delegation Paris

The Victoria University of Manchester

GAVO (RDS:MPE,AIP,HS,MPA)

European VO-activities

Wolfgang Voges 21

German Astrophysical Virtual Observatory

GAVO-Team:Wolfgang Voges (PI)

Hans-Martin Adorf, Gerard Lemson, Achim Bohnet, Joachim Paul

Max-Planck-Institut für extraterrestrische Physik, Garching

Matthias Steinmetz (Co-I)Harry Enke, Detlef Elstner

Astrophysikalisches Institut Potsdam

Dieter Reimers, (Co-I)Dieter Engels, Peter Hauschildt

Hamburger Sternwarte

Simon White, Anthony Banday, Volker SpringelMax-Planck-Institut für Astrophysik, Garching

Other institutes are most welcome to join

>>>www.g-vo.org<<<

Wolfgang Voges 22

to remain internationally competitive (proposals, data utilisation, quality of science output)

to make available VO services to everyone and provide support for the science community and public in Germany

to prepare and maintain datasets obtained from German facilities for GAVO and IVO

to establish a network, within which the needs of the German science community and public are coordinated

to obtain financial support from German agencies for such a national task

Why do we need a German AVO?

Wolfgang Voges 23

Activities and responsibilities of partners

Wolfgang Voges 24

Main goal is science driven, but it will drive science, too

- fast access to all kinds of astronomical and related data

- capability to use highly sophisticated software tools for new studies

- GAVO will provide interoperability of distributed archives over a high speed network through a set of interface/infrastructure tools

- GAVO ultimately will be incorporated into larger IVO federation

- Astronomical institutes will require expert data centres of different local character e.g. for providing key data archives, documentations, “simple” analysis-, correlation- and visualisation tools

- computer science groups will develop data handling and novel analysis tools and are responsible for their maintenance


Wolfgang Voges 25

- university institutes will be able to use GAVO for teaching and will provide a simple gateway to the public and to schools

- the “service community” will be responsible for designing and developing the interface/infrastructure tools necessary for communication between the users


Wolfgang Voges 26

Archive publication through GAVO

• ROSAT source catalogs published in IVOA compliant manner: – simple cone search– webservices

• RASS Photons stored in PostgreSQL database– Spatial index using HEALPix – Cone search, webservices

• Federation: fast match between ROSAT source catalogues and RASS photons.• Published first proposal for unified datamodel to serve as an ontology for the

IVOA.• Plans:

– Extend query capabilities– Publish ROSAT fields and pointed observations– Federate with SDSS mirror at MPA– Federate ROSAT catalogues with external catalogues for classification of

X-ray sources (in collaboration with ClassX team).

Wolfgang Voges 27

Top priority during initial stage of development federation of local key datasets and provision of key applications

ROSAT, SDSS, Planck, RAVE development of meta-data standards, especially for simulations development of common query tools for the local archives

need ability to query/compare both real sky and simulated data post-processing tools - must be platform-independent installation of visualisation packages

existing software provides a strong foundation to allow

extension to different types of data and archives

The local GAVO activities

Wolfgang Voges 28

Technical challenges and requirements

archive standards: rules for ingestion, data quality, associated meta-data schema, data attributes

archive maintenance/evolution: migration of data with new technology and enhancements in data attributes

meta-data requirements/standards for different data-sets (observations, simulation, calibration)

federation of archives, interoperability high-speed networking, streaming formats for data distributed processing power – GRID concept seeking active cooperation with industry in many of these areas

The local GAVO activities

Wolfgang Voges 29

Next Generation Search Engine

• Download Manager– Retrieves data from multiple distributed databases

• Matcher– Matches sources based on sky-position (astronomical

sources have no unique identifier)

• Classifier– Uses multi-wavelength data for identification purposes

Wolfgang Voges 30

NextGen Search Engine (cntd.)

Multi-Catalogue Multi-Cone Search

"Download Manager"Probabilistic MatcherVOTable Processor

Simple ConeSearch Service #1

ServiceRegistry

Table onLocal Disk


VOTables

VOTables

VOTable

BaseURLs

BaseURLs


VOTable

MatcherDataSets

Local Disk

VOTable

VOTablesTable

One or moreSCS Queries

Local Disk

InternetTable

Wolfgang Voges 31

Download Manager

• Features– Tool …

• … accesses registry at JHU

– User …• … selects distributed catalogues• … specifies one or more sky-locations

– Tool …• … queries remote catalogues• … retrieves datasets for further processing

Wolfgang Voges 32

Download Manager (cntd.)

Wolfgang Voges 33

Matcher

• Essential for data mining• Prototype features

• Performs “fuzzy” match between pairs of source lists from different catalogues

• Computes probability of real match

• Moving matcher into production use– Collaboration with Canadian Virtual

Observatory (CVO)– Feeding ROSAT source matches to classifier

Wolfgang Voges 34

Matcher (cntd.)

Wolfgang Voges 35

Classification of ROSAT X-ray sources

• ClassX: in collaboration with US-VO

• Requires data from several large sky-surveys

• X-ray: ROSAT (BSC + FSC)

• Optical: SDSS, USNO B1.0

• Infrared: 2MASS

• Radio: FIRST, NVSS, SUMSS

• True multi-wavelength VO-application

Wolfgang Voges 36

Probing the large-scale structure of the universe

with clusters of galaxies Project outline:

(ideally on single photon/galaxy basis) (Schuecker, Boehringer,Voges)

- identify a sample of galaxy clusters using X-ray/optical correlation

>>>>>> see next 3 viewgraphs

- utilise optical multi-colour images (u,g,r,i,z) to derive photometric redshifts

- quantify completeness and selection limits by comparison to simulated cluster data

- search for IR correlation and quantify galaxy evolution in clusters

- determine correlation with radio surveys to identify the frequency of radio galaxies and AGN in clusters, search for radio halos

- optical correlation to identify AGN in clusters

- identify correlations with microwave/sub-mm data to search for the Sunyaev-Zeldovich (SZ) effect (distance measurements, velocity determination)

Correlation of ROSAT and SDSS data

Wolfgang Voges 37

Maximum likelihood contours based on RASS-3 X-ray photons (upper panel, 1, 2 .. contours), SDSS galaxies (middle panel, >10), and the combined maximum likelihood contours of RASS-3 and SDSS data (lower panel, >10). Crosses mark the position of the deepest X-ray clustersamples available sofar (REFLEX-2, X-ray flux limit 1.8 .10-12 erg s-1cm-2). Squares mark the position of the X-ray clusters of the final sample.

Search for clusters of galaxies

Wolfgang Voges 38


Cumulative X-ray cluster number counts of the RASS/SDSSclusters (histograms) for a log-likelihood minimum of 15 applied to SDSS data (continuous line), for 25 (lower dashed line), and for 5 (upper dashed line). The RASS/SDSS cluster counts are compared with results obtained with other surveys (squares: RDCS, REFLEX,REFLEX-2). No corrections for variations of the angular survey-sensitivity (effective survey area) are applied to the RASS/SDSS and REFLEX-2 data. The figure shows that with the combination of RASS and SDSS data a 10 times deeper X-ray flux limit can be obtained compared to traditional X-ray cluster surveys like REFLEX.

Wolfgang Voges 39

General remarks: Our first results are quite important as a guideline for future X-ray missions like ROSITA and DUO. For the latter, about 8,000 X-ray clusters are expected to be detectable with standard methods. The application of the matched-filter technique allows the extraction of about 30,000 X-ray clusters with DUO. Such large numbers of X-ray clusters are needed for precise tests of the dark energy and alternative gravitational theories.


Wolfgang Voges 40

VO functionality required:- federation of relevant datasets including interchange/merging of meta-data

- identification of candidate cluster members by appropriate query applications to optical and/or X-ray catalogues

- acquire multi-colour information to determine photometric redshifts

- identification of candidate radio galaxy cluster members by querying radio catalogues with search criteria (e.g. location) tailored to the derived cluster sample

- identification of associated SZ by specific queries in existing catalogues; if no candidate SZ cluster can be identified apply suitable search algorithms to Cosmic Microwave Background (CMB) sky surveys to determine effect or limits thereon

- visualisation of multi-wavelength cluster data

- deprojection algorithms to allow study of morphology in survey data

- conversion of simulation data to the space of observable parameters

- 3D-interface for visualisation (to schools)

Correlation of ROSAT and SDSS data

Wolfgang Voges 41

Correlation between radio, IR, optical and X-ray sources

Search for SDSS QSO´s with 1 < z < 2, which are variable in one of

the 4 wavelength-bands

search-engine:

- which datasets do exist and in which archive?

- multiple availability? parallel handling on different servers

- data available for different epochs? comparison of fluxes, light curves, period-search

- Source catalogues available? - radio: FIRST, NVSS, …

- IR: IRAS, 2MASS, …

- Optical: SDSS, Tycho-2, HST-GSC, USNO-2…

- X-ray: ROSAT, ASCA, XMM-Newton, Chandra, …

Query example

Wolfgang Voges 42

- if no catalogue entry exists

postage-stamp (pixel-image with/without contour-lines) creation of light curves, fluxes, spectra, etc.

by using original-data

- high demand of CPU?

GRID implementation

- search for publications on derived variable SDSS QSO objects

Query example

Wolfgang Voges 43

Grid Technology for GAVO

GAVO grid :

integration of all GAVO-workstations

at MPE and AIP into a cluster

Basic services on GAVO-grid:

CertificationAuthority provides

single-sign-on/access-all facility via proxy-ca

Resource discovery and runtime information,

network-weather for the grid

Running distributed applications

Running MPI-based applications on the GAVO-cluster

Wolfgang Voges 44


Simulations

Comparison of Simulations and observations

Theory and GAVO

Wolfgang Voges 45

Merging of the Milky Way with the Andromeda galaxy (M31) (3 Mio particles, cluster of 16 CPU’s, 1 week of CPU time) (30 k particles would need 25 minutes of cluster-CPU time)

Simulations in the Virtual Observatory

Wolfgang Voges 46

COMA type cluster of galaxies (>1000 galaxies, 10^15Msolar, 7 Mio particles)

8 CPUs, runtime:2 days; Gravitation, Hydrodynamics, not included: cooling, star formation

(Volker Springel, MPA)

Simulations in the Virtual Observatory

C:\Programme\Windows Media Player\mplayer2.exe D:\Wolfgang\Gavo\S2_960x640.avi

Wolfgang Voges 47

The Role of Datasets fromTheoretical Astrophysics

• Direct Comparisons with Observations– Verification (or not) of Models

• Data Mining for Both Observations and Theory– New Applications– Buried Physics

• Resource for Education and Outreach

Wolfgang Voges 48

Theory and the VirtualObservatory

• Size of Datasets Appropriate to VO– Large Scale Simulations, Parameter Space

Libraries Imply 10GB – 10 TB Datasets

• Rich Complement to Observational Side

• Same/Similar Tools as for Obs. Datasets

• Use of VO Infrastructure– Grid Technology, Portals, etc.

Wolfgang Voges 49

Conclusions

• Theoretical Astrophysics is an Essential Part of the Virtual Observatory Concept

– Provides Benefits to Theorists– Provides Benefits to Observers– Provides Benefits to Education/Outreach

• Drives New Science

Wolfgang Voges 50

GAVO efforts on the TVO

• Published IVOA whitepaper on “Theory in the VO”• Leading theory subgroup in IVOA data modeling effort.• Chair in IVOA special interest group on theory• Plans:

– Publish simulation archives at AIP, LMU-Obs., andMPA – Collaborate with UPitt on publishing services on theoretical

datasets (NSF grant proposal)– Collaboration with Technion Haifa to publish observed and

simulated Ly- forest spectra (GIF proposal)– Collaboration in RTN proposal for comparison of simulated and

observed X-Ray clusters (Boehringer et al@MPE)

Wolfgang Voges 51


Great enthusiasm among astronomical community

that VO will work and will make life easier

BUT still a lot to be done

IVOA is combining all available forces

to attack the manyfold problems

There is still the need to incorporate other fields like mathematics, informatics,computer science, networks

etc. since there is parallel work in progress

GRID paradigm, fast data-links, super-computer access, etc.

MORE MANPOWER NEEDED

Outlook

Wolfgang Voges 52

The end

Virtual Observatories

Documents

Transcript of Virtual Observatories