The World Wide Telescope – a Digital Library Prototype

Jim Gray, Microsoft Research

Alex Szalay, Johns Hopkins University

Talk at OCLC @ Dublin, OH, 17 May 2004http://research.microsoft.com/~gray/talks/OCLC_WWT.ppt

Jim’s Model of Library Science

• Alexandria

• Gutenberg

• (Melvil) Dewey Decimal

• MARC (Henriette Avram)

• Dublin Core

Yes, I know there have been other things.

Dublin CoreElements

– Title– Creator– Subject– Description– Publisher– Contributor– Date– Type– Format– Identifier– Source– Language– Coverage– Rights

Elements+– Audience– Alternative– TableOfContents– Abstract– Created– Valid– Available– Issued– Modified– Extent– Medium– IsVersionOf– HasVersion– IsReplacedBy– Replaces– IsRequiredBy– Requires– IsPartOf– HasPart– IsReferencedBy– References– IsFormatOf– HasFormat– ConformsTo– Spatial– Temporal– Mediator– DateAccepted– DateCopyrighted– DateSubmitted– EducationalLevel– AccessRights– BibliographicCitation

Encoding– LCSH (Lb. Congress Subject Head)– MESH (Medical Subject Head)– DDC (Dewey Decimal Classification)– LCC (Lb. Congress Classification)– UDC (Universal Decimal Classification)– DCMItype (Dublin Core Meta Type)– IMT (Internet Media Type)– ISO639-2 (ISO language names)– RFC1766 (Internet Language tags)– URI (Uniform Resource Locator)– Point (DCMI spatial point)– ISO3166 (ISO country codes)– Box (DCMI rectangular area)– TGN (Getty Thesaurus of Geo Names)– Period (DCMI time interval)– W3CDTF (W3C date/time)– RFC3066 (Language dialects)

Types– Collection– Dataset– Event– Image– InteractiveResouce– Service– Software– Sound– Text– PhysicalObject– StillImage– MovingImage

What’s Happening?• We are drowning in information• Single fixed hierarchy is hopeless

– Can’t organize/find things in a simple tree• HOPE: “schematized storage”

– Objects have “Dublin-like” facets– Most facets acquired automatically (email, photo, doc,…)– Users add annotations and relationships

Librarians call this accession• Automate accession as much as possible • Folders/directories are standing queries

– Organization is “search based” demo sis.

• Interesting (public) research projects – Stuff I’ve Seen: http://research.microsoft.com/adapt/sis/

– MyLifebits: http://research.microsoft.com/barc/mediapresence/MyLifeBits.aspx

• Longhorn product embraces & extends these ideas.

The World Wide Telescope – a Digital Library Prototype

Jim Gray, Microsoft Research

Alex Szalay, Johns Hopkins University

Talk at OCLC @ Dublin, OH, 17 May 2004http://research.microsoft.com/~gray/talks/OCLC_WWT.ppt

But, what about the talk I promised you?

The Talk• Libraries morphing to integrated text + data (you know that)

• Dublin Core is bedrock, but many issues remain. (you know that)  

• WWT: All Astronomy data and literature online and integrated

• Problems Librarians have grappled with for centuries: curation, preservation, indexing, access, summarization.  

1. Overview of the World-Wide Telescope as a digital library

2. Focus on metadata, schema, curation, and preservation..

• Candidly, we have more problems than solutions, but the data is arriving and we are doing the best we can.

New Science Paradigms

• Thousand years ago: science was empirical describing natural phenomena

• Last few hundred years: theoretical branch using models, generalizations

• Last few decades: a computational branch simulating complex phenomena

• Today: data exploration (eScience) synthesizing theory, experiment and

computation with advanced data management and statistics

2

22.

3

4

a

cG

a

a

2

22.

3

4

a

cG

a

a

The Big Picture

Experiments &Instruments

Simulationsfacts

facts

answers

questions

• Data ingest

• Managing a petabyte

• Common schema

• How to organize it?

• How to reorganize it

• How to coexist with others

• Data Query and Visualization tools • Support/training• Performance

– Execute queries in a minute – Batch (big) query scheduling

?The Big Problems

Literature

Other Archives facts

facts

The Virtual Observatory• Premise: most data is (or could be online)• The Internet is the world’s best telescope:

– It has data on every part of the sky– In every measured spectral band:

optical, x-ray, radio..– As deep as the best instruments (2 years ago).

– It is up when you are up– The “seeing” is always great– It’s a smart telescope:

links objects and data to literature

• Software is the capital expense– Share, standardize, reuse..

Why Is Astronomy Special?• Almost all literature online and public ADS: http://adswww.harvard.edu/ CDS: http://cdsweb.u-strasbg.fr/

• Data has no commercial value– No privacy concerns, freely share results with others– Great for experimenting with algorithms

• It is real and well documented– High-dimensional (with confidence intervals)

– Spatial, temporal• Diverse and distributed

– Many different instruments from many different places and many different times

• The community wants to share the data• There is a lot of it (soon petabytes)

IRAS 100

ROSAT ~keV

DSS Optical

2MASS 2

IRAS 25

NVSS 20cm

WENSS 92cm

GB 6cm

Like all sciences, Astronomy Faces an Information Avalanche• Astronomers have a few hundred TB now

– 1 pixel (byte) / sq arc second ~ 4TB– Multi-spectral, temporal, … → 1PB

• They mine it looking for new (kinds of) objects or more of interesting ones (quasars), density variations in 400-D space correlations in 400-D space

• Data doubles every year• Data is public after 1 year• So, 50% of the data is public• Same access for everyone

Publishing Data

• Exponential growth:– Projects last at least 3-5 years– Data sent upwards only at the end of the project– Data will never be centralized

• More responsibility on projects– Becoming Publishers and Curators

• Data will reside with projects– Analyses must be close to the data

Roles

Authors

Publishers

Curators

Consumers

Traditional

Scientists

Journals

Libraries

Scientists

Emerging

Collaborations

Project www site

Bigger Archives

Scientists

How to Publish Data: Web Services• Web SERVER:

– Given a url + parameters – Returns a web page (often dynamic)

• Web SERVICE:– Given a XML document (soap msg)– Returns an XML document (with schema)– Tools make this look like an RPC.

• F(x,y,z) returns (u, v, w)

– Distributed objects for the web.– + naming, discovery, security,..

• Internet-scale distributed computing

Yourprogram

DataIn your address

space

Web Service

soap

object

in

xml

Yourprogram Web

Server

http

Web

page

The Core Problem: No Economic Model• The archive user has not yet been born.

How can he pay you to curate the data?

• Q: The Scientist gathered data for his own purpose.Why should he pay (invest time) for your needs?A: that’s the scientific method

• Curating data (documenting the design, the acquisition, and the processing)

is very hard and there is no reward for doing it.Results are rewarded, not the process of getting them.

• Storage/archive NOT the problem (it’s almost free)

• Curating/Publishing is expensive.• Better standards & tools lower costs

Data Inflation – Data Pyramid• Level 1A

Grows 5TB pixels/year growing to 25TB~ 2 TB/y compressed growing to 13TB~ 4 TB today (level 1A in NASA terms)

• Level 2Derived data products ~10x smaller But there are many catalogs.

• Publish new edition each year – Fixes bugs in data.– Must preserve old editions– Creates data pyramid

• Store each edition – 1, 2, 3, 4… N ~ N2 bytes

• Net: Data Inflation: L2 ≥ L1

E1

E2

E3E4

4 editions oflevel 1A data(source data)

4 editions of level 2 derived data products. Note that each derived product is small, but they are numerous. This proliferation combined with the data pyramid implies that level2 data more than doubles the total storage volume.

time

Level 1A 4 editions of Level 2 products

What SDSS is Doing: Capture the Bits

• Best-effort documenting data and process.

• Publishing data: often by UPS(~ 5TB today and so ~5k$ for a copy)

• Replicating data on 3 continents.• EVERYTHING online (tape data is dead data)

• Archiving all email, discussions, ….

• Keeping all web-logs.

• Now we need to figure out how to organize/search all this metadata.

Making Discoveries• Where are discoveries made?

– At the edges and boundaries– Going deeper, collecting more data, using more colors….

• Metcalfe’s law: quadratic benefit– Utility of computer networks grows as the

number of possible connections: O(N2)

• Data Federation: quadratic benefit– Federation of N archives has utility O(N2) – Possibilities for new discoveries grow as O(N2)

• Current sky surveys have proven this– Very early discoveries from SDSS, 2MASS, DPOSS

Federation

Global Federations• Massive datasets live near their owners:

– Near the instrument’s software pipeline– Near the applications– Near data knowledge and curation

• Each Archive publishes a (web) service– Schema: documents the data– Methods on objects (queries)

• Scientists get “personalized” extracts• Uniform access to multiple Archives

– A common global schema

Schema (aka metadata)• Everyone starts with the same schema

<stuff/>Then the start arguing about semantics.

• Virtual Observatory: http://www.ivoa.net/

• Metadata based on Dublin Core:http://www.ivoa.net/Documents/latest/RM.html

• Universal Content Descriptors (UCD): http://vizier.u-strasbg.fr/doc/UCD.htxCaptures quantitative concepts and their unitsReduced from ~100,000 tables in literature to ~1,000 terms

• VOtable – a schema for answers to questionshttp://www.us-vo.org/VOTable/

• Common Queries:Cone Search and Simple Image Access Protocol, SQL

• Registry: http://www.ivoa.net/Documents/latest/RMExp.htmlstill a work in progress.

Data Access is Hitting a Wall

•You can GREP 1 MB in a second•You can GREP 1 GB in a minute •You can GREP 1 TB in 2 days•You can GREP 1 PB in 3 years

•Oh!, and 1PB ~4,000 disks

•At some point you need indices to limit searchparallel data search and analysis

•This is where databases can help

• You can FTP 1 MB in 1 sec• You can FTP 1 GB / min (= 1 $/GB)• You can FTP 1 TB in 2 days and 1K$• You can FTP 1 PB in 3 years and 1M$

Current practice of data download (FTP/GREP)

will not scale to petabyte datasets

Smart Data• Better Data Schemas• There is too much data to move around

Do data manipulations at database– Build custom procedures and functions into DB– Unify data Access & Analysis– Examples

• Temporal and spatial indexing

• Pixel processing

• Automatic parallelism • Auto (re)organize • Scalable to Petabyte datasets

Move Mohamed to the mountain, not the mountain to Mohamed.

Next-Generation Data Analysis• Looking for

– Needles in haystacks – the Higgs particle– Haystacks: dark matter, dark energy,

turbulence, ecosystem dynamics• Needles are easier than haystacks• Global statistics have poor scaling

– Correlation functions are N2, likelihood techniques N3

• As data and computers grow at Moore’s Law, we can only keep up with N logN

• A way out? – Relax optimal notion (data is fuzzy, answers are approximate)– Don’t assume infinite computational resources or memory

• Requires combination of statistics & computer science

The Sloan Digital Sky Survey• Goal

– Create the most detailed map of the Northern Sky to-date

• 2.5m telescope– 3 degree field of view

• Two surveys in one– 5-color images of ¼ of the sky– Spectroscopic survey of a million

galaxies and quasars

• Very high data volume– 40 Terabytes of raw data– 10 Terabytes processed– All data public

The University of Chicago Princeton University The Johns Hopkins University The University of Washington New Mexico State University University of Pittsburgh Fermi National Accelerator Laboratory US Naval Observatory The Japanese Participation Group The Institute for Advanced Study Max Planck Inst, Heidelberg

Sloan Foundation, NSF, DOE, NASA

The University of Chicago Princeton University The Johns Hopkins University The University of Washington New Mexico State University University of Pittsburgh Fermi National Accelerator Laboratory US Naval Observatory The Japanese Participation Group The Institute for Advanced Study Max Planck Inst, Heidelberg

Sloan Foundation, NSF, DOE, NASA

SkyServer• A multi-terabyte database

• An educational website– More than 50 hours of educational exercises– Background on astronomy– Tutorials and documentation– Searchable web pages

• Easy astronomer access to SDSS data.

• Prototype eScience lab

• Interactive visual tools for data exploration

http://skyserver.sdss.org/

Demo SkyServer

• atlas• education project• Mouse in pixel space• Explore an object

(record space)• Explore literature• Explore a set• Pose a new question

SkyQuery (http://skyquery.net/)

• Distributed Query tool using a set of web services• Many astronomy archives from

Pasadena, Chicago, Baltimore, Cambridge (England)• Has grown from 4 to 15 archives,

now becoming international standard

• Allows queries like:SELECT o.objId, o.r, o.type, t.objId FROM SDSS:PhotoPrimary o,

TWOMASS:PhotoPrimary t WHERE XMATCH(o,t)<3.5

AND AREA(181.3,-0.76,6.5) AND o.type=3 and (o.I - t.m_j)>2

2MASS

INT

SDSS

FIRST

SkyQueryPortal

ImageCutout

Demo SkyQuery Structure• Each SkyNode publishes

– Schema Web Service– Database Web Service

• Portal is – Plans Query (2 phase) – Integrates answers– Is itself a web service

MyDB: eScience Workbench• Prototype of bringing analysis to the data

• Everybody gets a workspace (database)– Executes analysis at the data– Store intermediate results there– Long queries run in batch– Results shared within groups

• Only fetch the final results

• Extremely successful – matches work patterns

National Center Biotechnology Information (NCBI) A Better Example• Pubmed:

– Abstracts and books and..

• Genbank:– All Gene sequences deposited– BLAST and other searches– Website to explore data and literature

• Entrez: – unifies many databases

with literature (books, journals,..)– Organizes the data

The Big Picture

Experiments &Instruments

Simulationsfacts

facts

answers

questions

• Data ingest

• Managing a petabyte

• Common schema

• How to organize it?

• How to reorganize it

• How to coexist with others

• Query and Vis tools • Support/training• Performance

– Execute queries in a minute – Batch query scheduling

?The Big Problems

Literature

Other Archives facts

facts

The Talk• Libraries morphing to integrated text + data (you know that)

• Dublin Core is bedrock, but many issues remain. (you know that)  

• WWT: All Astronomy data and literature online and integrated

• Problems Librarians have grappled with for centuries: curation, preservation, indexing, access, summarization.  

1. Overview of the World-Wide Telescope as a digital library

2. Focus on metadata, schema, curation, and preservation..

• Candidly, we have more problems than solutions, but the data is arriving and we are doing the best we can.

Education

• Educational Projects, aimed at advanced high school students, but covering middle school

• Teach how to analyze data, discover patterns,not just astronomy

• 3.7 million project hits, 1.25 million page views of educational content

• More than 4000 textbooks• On the whole web site: 44 million web hits• Largely a volunteer effort by many individuals• Matches the 2020 curriculum

SkyServer project page views

010,00020,00030,00040,00050,00060,00070,00080,000

Date

Pag

e v

iew

s