Bob Hanisch Gretchen Greene Wil O’Mullane

2 Dec 2004

Technology Open House 2004

1

The National Virtual Observatory

THE US NATIONAL VIRTUAL OBSERVATORY

Bob HanischGretchen Greene

Wil O’Mullane

2 Dec 2004


2

Abstract

The National Virtual Observatory has been under

development for three years, and is now at the point of making its first release of applications software and tools to the astronomy community. The team will describe the main technology development areas of the VO and demonstrate some of the tools that will be released for general community use in conjunction with the January AAS meeting. We will also briefly describe the software development toolkit that we assembled for the NVO Summer School, which includes software libraries and sample code for VO-compatible data publication and retrieval.

2 Dec 2004


3

Astronomy is facing a data avalanche

Multi-Terabyte (soon: multi-Petabyte) sky surveys and archives over a broad range of wavelengths

Billions of sources, hundreds of attributes per source

1 nanoSky (HDF-S)

1 microSky (DPOSS)

2 Dec 2004


4

The changing face of observational astronomy

• Large digital sky surveys are becoming dominant source of data in astronomy: > 100 TB, growing rapidly– SDSS, 2MASS, DPOSS, GSC, FIRST, NVSS, RASS, IRAS,

QUEST, GALEX, SST; CMBR experiments; Microlensing experiments; NEAT, LONEOS, and other searches for Solar system objects

– Digital libraries: ADS, astro-ph, NED, CDS, NSSDC– Observatory archives: HST, CXO, space and ground-based– Future: PanSTARRS, LSST, and other synoptic surveys;

astrometric missions, GW detectors

• Data sets orders of magnitude larger, more complex, more homogeneous than in the past

• Roughly 1 TB/Sky/band/epoch– Human Genome is < 1 GB, Library of Congress ~ 20 TB

2 Dec 2004


5

Toward a “new astronomy”

• Past: Observations of small, carefully selected samples (often with a priori prejudices) of objects in one or a few wavelength bands

• Future: Multi-wavelength data for millions of objects, allowing us to– Discover significant patterns from the analysis of

statistically rich and unbiased image/catalog databases (e.g., Gunn-Peterson effect in high-z quasars)

– Understand complex astrophysical systems via confrontation between data and sophisticated numerical simulation

• Discovering new phenomena and patterns in these datasets will require simultaneous access to multi-wavelength archives, advanced visualization and statistical analysis tools

2 Dec 2004


6

Motivation

• National Academy of Sciences Decadal Survey recommended NVO as highest priority small (<$100M) project

“ Several small initiatives recommended by the committee span both ground and space. The first among them—the National Virtual Observatory (NVO)—is the committee’s top priority among the small initiatives. The NVO will provide a “virtual sky” based on the enormous data sets being created now and the even larger ones proposed for the future. It will enable a new mode of research for professional astronomers and will provide to the public an unparalleled opportunity for education and discovery.”

—Astronomy and Astrophysics in the New Millennium, p. 14

2 Dec 2004


7

History• 1990s: NASA establishes wavelength-oriented science archive centers;

multiple large ground-based digital sky survey projects initiated (following 1988 Astrophysics Data System report)

• April 1999, Decadal Survey Panel on Theory, Computation, and Data Discovery met in Los Alamos– Szalay, Prince, and Alcock coin the name “National Virtual Observatory”

• November 1999, NVO organizational workshop at JHU• February 2000, 2nd NVO workshop at NOAO-Tucson• June 2000, conference held at Caltech, “Towards a Virtual

Observatory”• June 2000, ad hoc steering committee formed• February 2001, AASC/NAS report “Astronomy and Astrophysics in the

New Millennium” released• April 2001, proposal submitted to NSF ITR program, 17 collaborating

organizations, led by A. Szalay (JHU)• September 2001, NSF announces proposal selection• January 2003, first NVO science prototypes shown at Seattle AAS

2 Dec 2004


8

What is the Virtual Observatory… and what it is not…

The VO is:• A set of international standards to share complex data• A modular set of tools to work with distributed data• A simple environment to publish data to• An essential part of the research astronomer’s toolkit• A catalyst for world-wide access to astronomical archives• A vehicle for education and public outreach

The VO is not:• A replacement for building new telescopes and

instruments• A centralized repository for data• A data quality enforcement organization

2 Dec 2004


9

Who is the National Virtual Observatory?

• US NVO development project, funded by NSF Information Technology Program and managed by NSF Astronomy Division, is entering 4th year of 5-year project

• Funding is $10M+ over the 5 years• 17 organizations (astro, CS, IT) involved

– JHU (PI Alex Szalay), STScI, Caltech (Astronomy, IPAC, CACR), HEASARC, SAO, NRAO, NOAO, NCSA, SDSC, FNAL, USNO, et al.

• Collaboration being extended to Gemini Science Archive, LSST, Keck

2 Dec 2004


10

International collaboration

• NVO is co-founder of the International Virtual Observatory Alliance

• IVOA now has 15 member projects• Adopted a standards process based on

W3C• Forum for technical development,

working groups, discussion and sharing of experience http://ivoa.net

http://ivoa.net/

http://ivoa.net/

http://ivoa.net/

2 Dec 2004


11

Exposure

• NVO Summer School (Sept. 2004, Aspen) trained 40 students and software developers in VO tools and technology

• Special Session scheduled for January 2005 AAS meeting: Astronomical Research with the Virtual Observatory; includes several papers based on Summer School projects (environments of radio galaxies, starburst galaxies)

• IAU Joint Discussion on Future Large Telescopes and the Virtual Observatory (July 2003, Sydney)

• IAU Symposium being proposed for 2006 (Prague)

http://us-vo.org/summer-school/proceedings/index.cfm

http://www.aas.org/publications/baas/v36n5/aas205/S1300.htm

http://www.aas.org/publications/baas/v36n5/aas205/S1300.htm

http://cdsweb.u-strasbg.fr/misc/jd_08.htm

http://cdsweb.u-strasbg.fr/misc/jd_08.htm

2 Dec 2004


12

Science prototypes

• Science demonstrations show capabilities of new infrastructure, motivate and guide technical developments. For example:– Data discovery, multi-λ

comparisons– Search for brown dwarfs– Galaxy morphologies in clusters– Globular cluster simulations

2 Dec 2004


13

Science results

• Padovani et al. (2004)) demonstrates that VO tools are mature enough to produce cutting-edge science results by exploiting astronomical data beyond classical identification limits (R 25)

http://www.edpsciences.org/articles/aa/pdf/2004/35/aa1153.pdf

http://www.edpsciences.org/articles/aa/pdf/2004/35/aa1153.pdf

2 Dec 2004


14

Science results

• McGlynn et al. (2004) classified all unidentified ROSAT WGACAT objects using VO data access methods to cross-correlate multi-wavelength catalogs– Technique applied to

find candidate X-ray binaries and now to SDSS photometric catalog

• More than 400 papers related to “virtual observatory” in ADS

2 Dec 2004


15

Technology development

• Resource registries• Data models• Content description (UCDs)• Data access layer (SIAP, SSAP, cone search)• VO Query Language (ADQL, OpenSkyQuery)• VOTable• Grid and Web ServicesActivities can be followed on IVOA web

(http://ivoa.net)

2 Dec 2004


16

Software toolkit

• Assembled software development toolkit for NVO Summer School– AXIS (Java web services)– ANT (Java-based software build tool)– Mirage, Topcat, VOPlot applications– Sample data files

• STScI Web Services course (http://chart.stsci.edu/twiki/bin/view/Main/ STWebServicesCource)

• See http://us-vo.org/summer-school/proceedings/software/index.html

• Summer School proceedings available at http://us-vo.org/summer-school/proceedings/index.cfm

2 Dec 2004


17

Capabilities

• Tools and applications available at http://us-vo.org/• Data location and access

– Resource registry (yellow pages) has thousands of catalogs and archives and is easily extended

– DataScope uses registry to locate data and allow user to display or download data of interest; can cache results on popular objects or for transient events

• Spectrum and filter database and analysis tools• HST/MAST VO services• Catalog cross-correlation (test version)• SDSS web services• Dynamic source-list generation and cross-correlation• Visualization of tabular, spectral, and imaging data• Source classification• Try these yourself at

http://chart.stsci.edu/twiki/bin/view/Main/AASJanuary2005

http://us-vo.org/

http://nvo.stsci.edu/voregistry/index.aspx

http://heasarc.gsfc.nasa.gov/cgi-bin/vo/datascope/init.pl

http://voservices.net/spectrum/

http://voservices.net/filter/

http://archive.stsci.edu/

http://www.openskyquery.net/sky/skysite//Browse.aspx

http://test.openskyquery.net/Sky/SkySite/default.aspx

http://skyserver.sdss.org/dr2/en/tools/explore/

http://frank.phyast.pitt.edu:8080/wsext

http://vo.iucaa.ernet.in/~voi/voplot.htm

http://www.stsci.edu/resources/software_hardware/specview

http://aladin.u-strasbg.fr/aladin.gml

http://heasarc.gsfc.nasa.gov/classx/web_classify/core.html

2 Dec 2004


18

Future VO-enabled science

• January 2005: software release in conjunction with AAS, international data access through standard registries– Applications you have just seen

• January 2005 AVO demos focusing on dynamic generation of SEDs for galaxies and comparisons with stellar spectral evolution models, and search for stars in transition from AGB to PN (unidentified IR sources); see

• TeraGrid-based analyses (CPU time already allocated): standard “sky atlas” spatial sampling and data federation, galaxy SED fitting, image mosaicing, quasar spectral modeling, N-point correlation function for SDSS galaxies, CMB modeling

2 Dec 2004


19

Future VO-enabled science

• NVO applications for 2005-2006 TBD in consultation with Science Steering Committee– Dynamic time series analysis, period fitting– “VO-Google”– Fast data inventory service– Flux-recovery service– Image registration and subtraction services– VO integration with legacy software systems (web

service interfaces, data access)– Datamining and data federation on increasingly

large, distributed databases

2 Dec 2004


20

STScI/JHU activities

• NVO project management• Technical development

– Resource metadata and registry services– OpenSkyNodes (GALEX, HDF, UDF, GOODS)– HST/MAST catalog and image services with VOTable

support and direct links to VOPlot– Links to/from the literature, ADS

• Collaboration with GOODS team on catalog and image cutout services

• Collaboration with JHU/SDSS personnel; joint technical discussions, augmented by co-location in Bloomberg building

• Subscribe to [email protected] (majordomo)

2 Dec 2004


21

Ear to the ground

• NSF and NASA working to create joint program; draft RFP could be available as soon as next spring– Agency support is firm, but not unwavering

• Continuing to build community support from the ground up– Demonstrations– Software releases– Summer School– EPO partnerships

2 Dec 2004


22

Next steps

• Submitted white paper “The National Virtual Observatory: From Framework to Facility” to NSF and NASA this past summer– Suggests responsibilities and scope for eventual NVO

operational (distributed) facility– Suggests smaller, tighter collaboration– Describes several possible management models,

advocates management by consortium– Urges creation of joint NSF/NASA/other program to

provide single point of contact for funding

2 Dec 2004


23

Summary

• $10M committed in US, >$40M worldwide, to VO development

• Active international community is working and meeting regularly to establish the VO

• Major archives and catalogs available through VO and more coming

• Refereed research papers utilizing VO now beginning to appear

2 Dec 2004


24

Bob Hanisch Gretchen Greene Wil O’Mullane

Documents

Transcript of Bob Hanisch Gretchen Greene Wil O’Mullane