1

Information At Your Fingertips

Web ServicesJim Gray & Tom Barclay

Microsoft Research

Alex SzalayJohns Hopkins University

2

Communications Excitement!!

Point-to-Point Broadcast

Immediate

TimeShifted

conversationmoney

lectureconcert

mail booknewspaper

NetNetWorkWork+ DB+ DB

DataDataBaseBase

Its ALL going electronicImmediate is being stored for analysis (so ALL database)Analysis & Automatic Processing are being added

Slide borrowed from Craig Mundie

3

Information Excitement! • All information will be online (somewhere)

text, speech, sound, vision, graphics, spatial, time…

• You might record everything – read: 10MB/day, 400 GB/lifetime (5 disks today)– hear: 400MB/day, 16 TB/lifetime (2 disks/year today)– see: 1MB/s, 40GB/day, 1.6 PB/lifetime (150 disks/year maybe

someday)

• Information at Your Fingertips–Make it easy to capture & present –Make it easy to store & organize & access

–Make it easy to analyze & summarize

4

How much information is there?• Soon everything can be

recorded and indexed• Most bytes will never be

seen by humans.• Data summarization,

trend detection, anomaly detection are key technologies

See Mike Lesk: How much information is there: http://www.lesk.com/mlesk/ksg97/ksg.html

See Lyman & Varian:

How much informationhttp://www.sims.berkeley.edu/research/projects/how-much-info/

Yotta

Zetta

Exa

Peta

Tera

Giga

Mega

KiloA BookA Book

.Movie

All LoC books(words)

All Books MultiMedia

Everything!

Recorded

A PhotoA Photo

24 Yecto, 21 zepto, 18 atto, 15 femto, 12 pico, 9 nano, 6 micro, 3 milli

5

How do we get information today.

• Human searches web (with an index)

• Human browses pages

6

How do we get information tomorrow?

• Agents gather and digest it for us.

• Q: How?

• A Microsoft: Dot Net

– Discovery: UDDI,

WSDL– Explore: SOAP

My Agents

Digital Dashboard

Web Services

SOAPWSDL

7

How do you publish information?• Get the data.

• Conceptualize the data schema • Provide methods that return data subsets.

– Challenge: how much processing on your server?

• Publish the schema and methods.

• We are exploring these issues.

f, g, x, y…

8

TerraServer Example• What is TerraServer?

– 3TB Internet Map DB available since June 1998– USGS photo and topo maps of the US– Integrated with Home Advisor– Shows off SQL Server availability & scalability– Designed for basic computer systems and low

speed communications

• What is TerraService? – A .NET web service– Makes TerraServer data available to other apps

10

Application Goals

• Available – Always, 24x7x52 99.99% of the time

• Programmable -- .NET applications can integrate TerraServer data into their apps

• BIG — 1 TB of data including catalog, temporary space, etc.

• PUBLIC — available on the world wide web

• INTERESTING — to a wide audience

• ACCESSIBLE — using standard browsers (IE, Netscape)

• REAL — a LOB application (users can buy imagery)

• FREE — cannot require NDA or money to a user to access

• FAST — usable on low-speed (56kbps) and high speeds(T-1+)

• EASY — we do not want a large group to develop, deploy, or maintain the application

3 TB

11

Demo http://terraserver.microsoft.com

Show

photo

topo

gazetteer

demographics

12

Hardware

SQL\Inst1SQL\Inst1

SQL\Inst2SQL\Inst2

SQL\Inst3SQL\Inst3

SpareSpare

F GLKP Q

E EJ JO O

IHM NR S

22002200 22002200 22002200

220022002200220022002200

22002200 22002200 22002200

One SQL database per rackOne SQL database per rackEach rack contains 4.5 tbEach rack contains 4.5 tb261 total drives / 13.7 TB total261 total drives / 13.7 TB total

Meta DataMeta DataStored on 101 GBStored on 101 GB““Fast, Small Disks”Fast, Small Disks”(18 x 18.2 GB)(18 x 18.2 GB)

Imagery DataImagery DataStored on 4 339 GBStored on 4 339 GB““Slow, Big Disks”Slow, Big Disks”(15 x 73.8 GB)(15 x 73.8 GB)

To Add 90 72.8 GBTo Add 90 72.8 GBDisks in Feb 2001Disks in Feb 2001to create 18 TB SANto create 18 TB SAN

8 Compaq DL360 “Photon” Web Servers8 Compaq DL360 “Photon” Web Servers

Fiber SANFiber SANSwitchesSwitches

4 Compaq ProLiant 8500 Db Servers4 Compaq ProLiant 8500 Db Servers

13

TerraServer Experience• Successful Web Site

– Met all 8 goals – interesting, big, real, public, fast, easy, accessible, and free

– High Availability – Windows Data Center & Compaq SAN Technology

– Top 1000 Web Site – continues to be popular• New Feature Requests

– Programmable access to meta-data– User selectable image sizes, i.e. “a map server”– Permission to use TerraServer data within

server applications

14

What is a Web Service?

SOAPSOAPSOAPSOAP Web Service consumers can send and receive messages using XML

SOAPSOAPContract LanguageContract Language

SOAPSOAPContract LanguageContract Language

Web Services are defined in terms of the formats and ordering of messages

SOAPSOAPDiscoveryDiscoverySOAPSOAP

DiscoveryDiscovery You can ask a site for a description of the

Web Services it offers

All these capabilities are built using open Internet protocols XML & HTTPXML & HTTP

Open Internet Protocols

Web Web ServiceService

A programmable application component accessible via standard Web protocols

UDDIUDDIUniversal Description, Design, and IntegrationUniversal Description, Design, and Integration

UDDIUDDIUniversal Description, Design, and IntegrationUniversal Description, Design, and Integration

Provide a Directory of Services on the Internet

15

.NET TerraService Architecture

ExistingDB Server

SQL 2000

1.0 TB Db

SQL 2000

1.0 TB Db

SQL 2000

1.0 TB Db

705 m Rows705 m Rows

ADO.NETADO.NETADO.NETADO.NET

TerraServer TerraServer Web ServiceWeb Service

OLEDB

Map Server Map Server Http HandlerHttp Handler

Map UI Map UI Web FormsWeb FormsStandardStandard

BrowsersBrowsersStandardStandardBrowsersBrowsers

SmartSmartClientsClientsSmartSmartClientsClients

WindowsWindowsFormsForms

WindowsWindowsFormsForms

.NET.NETFrameworkFramework

.NET.NETFrameworkFramework SOAP/XML

HTMLImage/jpeg

Image/jpeg

16

TerraServer Web Services

• Query Gazetteer• Retrieve imagery

meta-data • Retrieve imagery • Simple Projection

conversions

• Geo-coded places, e.g. Schools, Golf Courses, Hospitals, etc.

• Place Polygons e.g. Zip Codes, Cities, etc.

Terra-Tile-Service Landmark-Service

allows “overlay” information for Terra-Tile-Service applications

Clients can present TerraServer imageryin new ways.

17

Web Service Methods• Place Search

– GetPlaceFacts– GetPlaceList– GetPlaceListInRect– CountPlacesInRect

• Projection– ConvertLonLatPtToUtmPt– ConvertUtmPtToLonLatPt– ConvertLonLatTo NearestPlace – GetTheme– GetLatLonMetrics

• Tile– GetAreaFromPt– GetAreaFromRect– GetAreaFromTileId– GetTileMetaFromLonLatPt– GetTileMetaFromTileId– GetTile (Image)

• Landmark– GetLandmarkTypes– CountOfLandmarkPointsByRect– GetLandmarkPointsByRect– CountOfLandmarkShapesByRect– GetLandmarkShapesByRect

http://terraservice.net

18

Soil Viewer Uses TerraService

19

Custom End ProductWeb Soil Data Viewer XML Soil ReportSoil Interpretation Map

20

What Tom Showed You• Converted a Web Server

–HTML get post–Server returns pictures

to people

• to a Web Service–SOAP service–returns XML self-describing data–Application integrates data

(Agriculture and Geo data)

21

Rosetta Stone• Distributed computing

+ basic services• Yellow Pages

• ?• RPC – remote procedure

call, CORBA, DCOM, RMI• IDL – interface definition

language• XDR - eXternal Data

Representation

• Dot Net

• UDDI – Universal description, discovery, and integration

• Schema, XLANG• SOAP – simple object

access protocol• WSDL – web services

definition language• XML- eXtended Markup

Language

22

Sky Server– Like TerraServer pictures of the sky.

– But also LOTS of data on each object

So a data mining web service• Luminosity (multi-spectra), morphology, spectrum• So, it is a data mining application• Cross-correlation is challenging because

–Multi-resolution–Data is dirty/fuzzy (error bars, cosmic rays, airplanes…)

–Time varying

+

•50 K Spectro Objects •~ 100 attributes + 30 lines

•15M Photo Objects ~ 400 attributes

23

Astronomy Data• In the “old days” astronomers took photos.

• Starting in the 1960’s they began to digitize.• New instruments are digital (100s of GB/nite)

• Detectors are following Moore’s law.

• Data avalanche: double every year

Total area of 3m+ telescopes in the world in m2, total number of CCD pixels in megapixel, as a function of time. Growth over 25 years is a factor of 30 in glass, 3000 in pixels.

Courtesy of

Alex Szalay

24

Astronomy Data• Astronomers have a few Petabytes now.

– 1 pixel (byte) / sq arc second ~ 4TB– Multi-spectral, temporal, … → 1PB

• They mine it looking for new (kinds of) objects or more of interesting ones(quasars), density variations in 400-D space correlations in 400D space

• Data doubles every year.• Data is public after a year.• So, 50% of the data is public.• Some have private access to 5% more data.• So: 50% vs 55% access for everyone

25

Astronomy Data• But…..• How do I get at that 50% of the data?• Astronomers have culture of publishing.

– FITS files and many tools.http://fits.gsfc.nasa.gov/fits_home.html

– Encouraged by NASA.• Publishing data “details” is difficult.

Astronomers want to do it but it is VERY hard.(What programs where used? what were the processing steps? How were errors treated?…)

26

Virtual Observatoryhttp://www.astro.caltech.edu/nvoconf/

http://www.voforum.org/

• Premise: Most data is (or could be online)

• So, the Internet is the world’s best telescope:– It has data on every part of the sky– In every measured spectral band: optical, x-ray, radio..

– As deep as the best instruments (1 year ago).– It is up when you are up.

The “seeing” is always great (no working at night, no clouds no moons no..).

– It’s a smart telescope: links objects and data to literature on them.

27

Virtual Observatory The Age of Mega-Surveys

• Large number of new surveys– multi-TB in size, 100 million objects or more– individual archives planned, or under way– Data publication an integral part of the survey– Software bill a major cost in the survey

• Multi-wavelength view of the sky– more than 13 wavelength coverage in 5 years

• Impressive early discoveries– finding exotic objects by unusual colors

• L,T dwarfs, high-z quasars

– finding objects by time variability• gravitational micro-lensing

MACHO2MASSDENISSDSSPRIMEDPOSSGSC-IICOBE MAPNVSSFIRSTGALEXROSATOGLE ...

MACHO2MASSDENISSDSSPRIMEDPOSSGSC-IICOBE MAPNVSSFIRSTGALEXROSATOGLE ...

Slide courtesy of Alex Szalay, modified by jim

28

Virtual Observatory Federating the Archives

• The next generation mega-surveys are different– top-down design– large sky coverage– sound statistical plans– well controlled/documented data processing

• Each survey has a publication plan• Data mining will lead to stunning new discoveries

• Federating these archives

Virtual Observatory

Slide courtesy of Alex Szalay

29

The Multiwavelength Crab Nebula

Nova first sighted 1054 A.D. by

Chinese Astronomers

Now: Crab Nebula X-ray, optical,

infrared, and radio

Slide courtesy of Robert Brunner @ CalTech.

Crab star 1053 AD

30

Exploring Parameter SpaceGiven an arbitrary

parameter space:• Data Clusters• Points between Data

Clusters• Isolated Data Clusters• Isolated Data Groups• Holes in Data Clusters• Isolated Points

Nichol et al. 2001

Slide courtesy of Robert Brunner @ CalTech.

31

Virtual Observatory and Education• In the beginning science was empirical.

• Then theoretical branches evolved.

• Now, we have a computational branches.– The computational branch has been simulation– It is becoming data analysis/visualization

• The Virtual Observatory can be used to – Teach astronomy:

make it interactive, demonstrate ideas and phenomena

– Teach computational science skillsand the process of scientific discovery

32

Sloan Digital Sky Survey http://sdss.org/

• A group of astronomers has been building a telescope (with 90M$ from Sloan Foundation, NSF, and a dozen universities). for the last 12 years!

• Now data is arriving: – 250GB/nite (20 nights per year).– 100 M stars, 100 M galaxies, 1 M spectra.

• Public data at http://sdss.org/ – 5% of the survey, 600 sq degrees, 15 M objects 60GB.– This data includes most of the known high z quasars.– It has a lot of science left in it but… that is just the start.

33

Demo of Sky ServerAlex built SkyServer (based on TerraServer design).

http://skyserver.sdss.org/

Demo: famous places navigator data shopping cart spectrum SQL? ?

34

Virtual Observatory Challenges• Size : multi-Petabyte

40,000 square degrees is 2 Trillion pixels

– One band (at 1 sq arcsec) 4 Terabytes– Multi-wavelength 10-100 Terabytes– Time dimension >> 10 Petabytes

– Need auto parallelism tools

• Unsolved Meta-Data problem– Hard to publish data & programs– Hard to find/understand data & programs

• Current tools inadequate– new analysis & visualization tools

• Transition to the new astronomy– Sociological issues

35

3-steps to Virtual Observatory • Get SDSS and Palomar online

– Alex Szalay, Jan Vandenberg, Ani Thakar….– Roy Williams, Robert Brunner, Julian Bunn

• Do queries and crossID matches with CalTech and SDSS to expose – Schema, Units,…– Dataset problems– the typical use scenarios.

• Implement WebServices at CalTech and SDSS

36

The Challenges• How to federate the Archives to make a VO?• The hope: XML is the answer.• The reality: XML is syntax and tools:

FITS on XML will be good but….. Explaining the data will still be very difficult.

• Define Astronomy Objects and Methods.– Based on UDDI, WSDL, SOAP.– Each archive is a service

• http://TerraService.net/ shows the idea.– Working with Caltech (Brunner, Williams, Djorgovski, Bunn)

– But, how does data mining work?

37

SkyServer as a WebServiceWSDL+SOAP

just add details

Archive ss = new VOService(SkyServer);

Attributes A[] = ss.GetObjects(ra,dec,radius)

…

?? What are the objects (attributes…)?

?? What are the methods (GetObjects()...)?

?? What query language? SQL, Xquery…?

38

Summary• All information at your fingertips.

• How do we publish information so that our agents can digest it?

• Example: TerraServer -> TerraService

• The Virtual Observatory Concept

– The Internet is worlds best telescope• For astronomy• For teaching astronomy and • For teaching computational science

39