Alex SzalayDepartment of Physics and Astronomy

The Johns Hopkins University

The Sloan Digital Sky Survey

The Sloan Digital Sky Survey

Alex Szalay, JHU

A project run by the Astrophysical Research Consortium (ARC)A project run by the Astrophysical Research Consortium (ARC)

Goal: To create a detailed multicolor map of the Northern Skyover 5 years, with a budget of approximately $80M

Data Size: 40 TB raw, 2 TB processed

Goal: To create a detailed multicolor map of the Northern Skyover 5 years, with a budget of approximately $80M

Data Size: 40 TB raw, 2 TB processed

The University of Chicago Princeton University The Johns Hopkins University The University of Washington Fermi National Accelerator Laboratory US Naval Observatory The Japanese Participation Group The Institute for Advanced Study Max Planck Inst, Heidelberg

SLOAN Foundation, NSF, DOE, NASA

The University of Chicago Princeton University The Johns Hopkins University The University of Washington Fermi National Accelerator Laboratory US Naval Observatory The Japanese Participation Group The Institute for Advanced Study Max Planck Inst, Heidelberg

SLOAN Foundation, NSF, DOE, NASA

The Sloan Digital Sky SurveyThe Sloan Digital Sky SurveyThe Sloan Digital Sky SurveyThe Sloan Digital Sky Survey

Alex Szalay, JHU

Scientific MotivationScientific MotivationScientific MotivationScientific Motivation

Create the ultimate map of the Universe: The Cosmic Genome Project!

Study the distribution of galaxies: What is the origin of fluctuations?

What is the topology of the distribution?

Measure the global properties of the Universe: How much dark matter is there?

Local census of the galaxy population: How did galaxies form?

Find the most distant objects in the Universe: What are the highest quasar redshifts?

Alex Szalay, JHU

Cosmology PrimerCosmology PrimerCosmology PrimerCosmology Primer

The spatial distribution of galaxies is correlated, due to small ripples in the early Universe

P(k):P(k): power spectrumP(k):P(k): power spectrum

v = Hv = Hoo r rHubble’s law

v = Hv = Hoo r rHubble’s law

The Universe is expanding: the galaxies move away from us spectral lines are redshifted

= = density/criticaldensity/criticalif <1, expand forever

= = density/criticaldensity/criticalif <1, expand forever

The fate of the universe depends on the balance between gravity and the expansion velocity

dd> > **dd> > **

Most of the mass in the Universe is dark matter, and it may be cold (CDM)

Alex Szalay, JHU

The ‘Naught’ ProblemThe ‘Naught’ ProblemThe ‘Naught’ ProblemThe ‘Naught’ Problem

What are the global parameters of the Universe?

H0 the Hubble constant 55-75 km/s/Mpc0 the density parameter 0.25-10 the cosmological constant 0 - 0.7

Their values are still quite uncertain today...

Goal: measure these parameters with an accuracy of a few percent

What are the global parameters of the Universe?

H0 the Hubble constant 55-75 km/s/Mpc0 the density parameter 0.25-10 the cosmological constant 0 - 0.7

Their values are still quite uncertain today...

Goal: measure these parameters with an accuracy of a few percent

High Precision Cosmology!High Precision Cosmology!High Precision Cosmology!High Precision Cosmology!

Alex Szalay, JHU

The Cosmic Genome ProjectThe Cosmic Genome ProjectThe Cosmic Genome ProjectThe Cosmic Genome Project

The SDSS will create the ultimate mapof the Universe, with much more detailthan any other measurement before

Gregory and Thompson 1978

deLapparent, Geller and Huchra 1986daCosta etal 1995

SDSS Collaboration 2002

Alex Szalay, JHU

Area and Size of Redshift SurveysArea and Size of Redshift SurveysArea and Size of Redshift SurveysArea and Size of Redshift Surveys

1.00E+03

1.00E+04

1.00E+05

1.00E+06

1.00E+07

1.00E+08

1.00E+09

1.00E+04 1.00E+05 1.00E+06 1.00E+07 1.00E+08 1.00E+09 1.00E+10 1.00E+11

Volume in Mpc 3

No

of

ob

jec

ts

LCRS

SDSSmain

SDSSred

SDSSabs line

SDSSphoto-z

2dFRCfA+SSRS

SAPMQDOT

2dF

1.00E+03

1.00E+04

1.00E+05

1.00E+06

1.00E+07

1.00E+08

1.00E+09

1.00E+04 1.00E+05 1.00E+06 1.00E+07 1.00E+08 1.00E+09 1.00E+10 1.00E+11

Volume in Mpc 3

No

of

ob

jec

ts

LCRS

SDSSmain

SDSSred

SDSSabs line

SDSSphoto-z

2dFRCfA+SSRS

SAPMQDOT

2dF

Alex Szalay, JHU

Clustering of GalaxiesClustering of GalaxiesClustering of GalaxiesClustering of Galaxies

We will measure the spectrum of the density fluctuations to high precision even on very large scales

The error in the amplitude of the fluctuation spectrum

1970 x1001990 x21995 ±0.41998 ±0.21999 ±0.12002 ±0.05

The error in the amplitude of the fluctuation spectrum

1970 x1001990 x21995 ±0.41998 ±0.21999 ±0.12002 ±0.05

Alex Szalay, JHU

Relevant ScalesRelevant ScalesRelevant ScalesRelevant Scales

Distances measured in Mpc [megaparsec] 1 Mpc = 3 x 1024 cm 5 Mpc = distance between galaxies3000 Mpc = scale of the Universe

if >200 Mpcfluctuations have a PRIMORDIAL shape

if <100 Mpcgravity creates sharp features, like walls,filaments and voids

Biasingconversion of mass into light is nonlinearlight is much more clumpy than the mass

Alex Szalay, JHU

The Topology of Local UniverseThe Topology of Local UniverseThe Topology of Local UniverseThe Topology of Local Universe

Measure the Topology of the Universe Does it consist of walls and voids or is it randomly distributed?

Alex Szalay, JHU

Finding the Most Distant ObjectsFinding the Most Distant ObjectsFinding the Most Distant ObjectsFinding the Most Distant Objects

Intermediate and high redshift QSOs Multicolor selection function. Luminosity functions and spatial clustering. High redshift QSO’s (z>5).

Intermediate and high redshift QSOs Multicolor selection function. Luminosity functions and spatial clustering. High redshift QSO’s (z>5).

Alex Szalay, JHU

Special 2.5m telescope, located at Apache Point, NM3 degree field of view.Zero distortion focal plane.

Two surveys in one:Photometric survey in 5 bands.Spectroscopic redshift survey.

Huge CCD Mosaic30 CCDs 2K x 2K (imaging)22 CCDs 2K x 400 (astrometry)

Two high resolution spectrographs2 x 320 fibers, with 3 arcsec diameter.R=2000 resolution with 4096 pixels.Spectral coverage from 3900Å to 9200Å.

Automated data reductionOver 100 man-years of development effort.(Fermilab + collaboration scientists)

Very high data volumeExpect over 40 TB of raw data.About 2 TB processed productsData made available to the public

Features of the SDSSFeatures of the SDSSFeatures of the SDSSFeatures of the SDSS

Alex Szalay, JHU

Apache Point ObservatoryApache Point ObservatoryApache Point ObservatoryApache Point Observatory

Located in New Mexico,near White Sands National Monument

Located in New Mexico,near White Sands National Monument

Alex Szalay, JHU

The TelescopeThe TelescopeThe TelescopeThe Telescope

Special 2.5m telescope 3 degree field of view Zero distortion focal plane Wind screen moved separately

Special 2.5m telescope 3 degree field of view Zero distortion focal plane Wind screen moved separately

Alex Szalay, JHU

Northern Galactic Cap 5 broad-band filters ( u', g', r', i', z’ ) limiting magnitudes (22.3, 23.3, 23.1, 22.3, 20.8) drift scan of 10,000 square degrees 55 sec exposure time 40 TB raw imaging data -> pipeline ->

100,000,000 galaxies 50,000,000 stars

calibration to 2% at r'=19.8 only done in the best seeing (20 nights/yr) pixel size is 0.4 arcsec, astrometric precision is 60 milliarcsec

Southern Galactic Cap multiple scans (> 30 times) of the same stripe

Continuous data rate of 8 Mbytes/sec

The Photometric SurveyThe Photometric SurveyThe Photometric SurveyThe Photometric Survey

Alex Szalay, JHU

Survey StrategySurvey StrategySurvey StrategySurvey Strategy

Overlapping 2.5 degree wide stripes

Avoiding the Galactic Plane (dust)

Multiple exposures on the three Southern stripes

Overlapping 2.5 degree wide stripes

Avoiding the Galactic Plane (dust)

Multiple exposures on the three Southern stripes

Alex Szalay, JHU

Measure redshifts of objects distance

SDSS Redshift Survey:1 million galaxies100,000 quasars100,000 stars

Two high throughput spectrographsspectral range 3900-9200 Å.640 spectra simultaneously.R=2000 resolution.

Automated reduction of spectraVery high sampling density and completenessObjects in other catalogs also targeted

The Spectroscopic SurveyThe Spectroscopic SurveyThe Spectroscopic SurveyThe Spectroscopic Survey

Alex Szalay, JHU

Optimal TilingOptimal TilingOptimal TilingOptimal Tiling

Fields have 3 degree diameterCenters determined by an optimization procedureA total of 2200 pointings640 fibers assigned simultaneously

Fields have 3 degree diameterCenters determined by an optimization procedureA total of 2200 pointings640 fibers assigned simultaneously

Alex Szalay, JHU

The Mosaic CameraThe Mosaic CameraThe Mosaic CameraThe Mosaic Camera

Alex Szalay, JHU

Photometric CalibrationsPhotometric CalibrationsPhotometric CalibrationsPhotometric Calibrations

The SDSS will create a new photometric system:

u' g' r' i' z'

Primary standards: observed with the USNO 40-inch telescope in Flagstaff

Secondary standards: observed with the SDSS 20-inch telescope at Apache Point – calibrating the SDSS imaging data

The SDSS will create a new photometric system:

u' g' r' i' z'

Primary standards: observed with the USNO 40-inch telescope in Flagstaff

Secondary standards: observed with the SDSS 20-inch telescope at Apache Point – calibrating the SDSS imaging data

Alex Szalay, JHU

The SpectrographsThe SpectrographsThe SpectrographsThe Spectrographs

Two double spectrographs very high throughput two 2048x2048 CCD detectors mounted on the telescope light fed through slithead

Two double spectrographs very high throughput two 2048x2048 CCD detectors mounted on the telescope light fed through slithead

Alex Szalay, JHU

The Fiber Feed SystemThe Fiber Feed SystemThe Fiber Feed SystemThe Fiber Feed System

Galaxy images are captured by optical fibers lined up on the spectrograph slitManually plugged during the day into Al plugboards640 fibers in each bundleThe largest fiber system today

Galaxy images are captured by optical fibers lined up on the spectrograph slitManually plugged during the day into Al plugboards640 fibers in each bundleThe largest fiber system today

Alex Szalay, JHU

First Light ImagesFirst Light ImagesFirst Light ImagesFirst Light Images

Telescope: First light May 9th 1998 Equatorial scans

Telescope: First light May 9th 1998 Equatorial scans

Alex Szalay, JHU

The First StripesThe First StripesThe First StripesThe First Stripes

Camera: 5 color imaging of >100 square degrees Multiple scans across the same fields Photometric limits as expected

Camera: 5 color imaging of >100 square degrees Multiple scans across the same fields Photometric limits as expected

Alex Szalay, JHU

NGC 2068NGC 2068NGC 2068NGC 2068

Alex Szalay, JHU

UGC 3214UGC 3214UGC 3214UGC 3214

Alex Szalay, JHU

NGC 6070NGC 6070NGC 6070NGC 6070

Alex Szalay, JHU

The First QuasarsThe First QuasarsThe First QuasarsThe First Quasars

The four highest redshift The four highest redshift quasars have been found in the quasars have been found in the

first SDSS test data !first SDSS test data !

The four highest redshift The four highest redshift quasars have been found in the quasars have been found in the

first SDSS test data !first SDSS test data !

Alex Szalay, JHU

Methane/T DwarfMethane/T DwarfMethane/T DwarfMethane/T Dwarf

Discovery of several newobjects by SDSS & 2MASS

SDSS T-dwarf (June 1999)

Alex Szalay, JHU

Detection of Gravitational LensingDetection of Gravitational LensingDetection of Gravitational LensingDetection of Gravitational Lensing

28,000 foreground galaxies and 2,045,000 background galaxies in test data(McKay etal 1999)

Alex Szalay, JHU

SDSS Data FlowSDSS Data FlowSDSS Data FlowSDSS Data Flow

Alex Szalay, JHU

Distributed CollaborationDistributed CollaborationDistributed CollaborationDistributed Collaboration

JapanJapan

FermilabFermilab

U.WashingtonU.WashingtonU.ChicagoU.Chicago

USNOUSNO

JHUJHU

VBNS

NMSUNMSUApache PointObservatory

Apache PointObservatory

I. AdvancedStudy

I. AdvancedStudy

Princeton U.Princeton U.

ESNETESNET

Alex Szalay, JHU

Data Processing PipelinesData Processing PipelinesData Processing PipelinesData Processing Pipelines

Alex Szalay, JHU

Concept of the SDSS ArchiveConcept of the SDSS ArchiveConcept of the SDSS ArchiveConcept of the SDSS Archive

OperationalArchive

(raw + processed data)

Science Archive(products accessible to users)

Other ArchivesOther ArchivesOther Archives

Alex Szalay, JHU

All raw data saved in a tape vault at Fermilab

Object catalog 400 GB parameters of >108 objects

Redshift Catalog 1 GB parameters of 106 objects

Atlas Images 1.5 TB 5 color cutouts of >108 objects

Spectra 60 GB in a one-dimensional form

Derived Catalogs 20 GB - clusters - QSO absorption lines

4x4 Pixel All-Sky Map 60 GB heavily compressed

Object catalog 400 GB parameters of >108 objects

Redshift Catalog 1 GB parameters of 106 objects

Atlas Images 1.5 TB 5 color cutouts of >108 objects

Spectra 60 GB in a one-dimensional form

Derived Catalogs 20 GB - clusters - QSO absorption lines

4x4 Pixel All-Sky Map 60 GB heavily compressed

SDSS Data ProductsSDSS Data ProductsSDSS Data ProductsSDSS Data Products

Alex Szalay, JHU

Who will be using the archive?Who will be using the archive?Who will be using the archive?Who will be using the archive?

Power Userssophisticated, with lots of resourcesresearch is centered around the archive data

moderate number of very intensive queriesmostly statistical, large output sizes

General Astronomy Publicfrequent, but casual lookup of objects/regionsthe archives help their research, but not central to it

large number of small queriesa lot of cross-identification requests

Wide Publicbrowsing a ‘Virtual Telescope’can have large public appealneed special packaging

could be a very large number of requests

Power Userssophisticated, with lots of resourcesresearch is centered around the archive data

moderate number of very intensive queriesmostly statistical, large output sizes

General Astronomy Publicfrequent, but casual lookup of objects/regionsthe archives help their research, but not central to it

large number of small queriesa lot of cross-identification requests

Wide Publicbrowsing a ‘Virtual Telescope’can have large public appealneed special packaging

could be a very large number of requests

Alex Szalay, JHU

How will the data be analyzed?How will the data be analyzed?How will the data be analyzed?How will the data be analyzed?

The data are inherently multidimensional=> positions, colors, size, redshift

Improved classifications result in complex N-dimensional volumes=> complex constraints, not ranges

Spatial relations will be investigated=> nearest neighbors=> other objects within a radius

Data Mining: finding the ‘needle in the haystack’=> separate typical from rare=> recognize patterns in the data

Output size can be prohibitively large for intermediate files=> import output directly into analysis tools

The data are inherently multidimensional=> positions, colors, size, redshift

Improved classifications result in complex N-dimensional volumes=> complex constraints, not ranges

Spatial relations will be investigated=> nearest neighbors=> other objects within a radius

Data Mining: finding the ‘needle in the haystack’=> separate typical from rare=> recognize patterns in the data

Output size can be prohibitively large for intermediate files=> import output directly into analysis tools

Alex Szalay, JHU

Geometric ApproachGeometric ApproachGeometric ApproachGeometric Approach

The Main Problem:•fast, indexed, complex searches of Terabytes in k-dim space•searches are not necessary parallel to the axes

=> traditional indexing (b-tree) does not work

The Main Problem:•fast, indexed, complex searches of Terabytes in k-dim space•searches are not necessary parallel to the axes

=> traditional indexing (b-tree) does not work

Geometric Approach:•Use the geometric nature of the k-dimensional data•Quantize data into containers of ‘friends’:

objects of similar colorsclose on the skystored together=> efficient cache performance

•Containers represent a coarse grained density map of the datamultidimensional index tree: k-d tree + r-tree

Geometric Approach:•Use the geometric nature of the k-dimensional data•Quantize data into containers of ‘friends’:

objects of similar colorsclose on the skystored together=> efficient cache performance

•Containers represent a coarse grained density map of the datamultidimensional index tree: k-d tree + r-tree

Alex Szalay, JHU

Geometric IndexingGeometric IndexingGeometric IndexingGeometric Indexing

“Divide and Conquer”“Divide and Conquer” PartitioningPartitioning

3 N M3 N M

HierarchicalTriangular

Mesh

HierarchicalTriangular

Mesh

Split as k-d treeStored as r-tree

of bounding boxes

Split as k-d treeStored as r-tree

of bounding boxes

Using regularindexing

techniques

Using regularindexing

techniques

Attributes Number

Sky Position 3Multiband Fluxes N = 5+Other M= 100+

Attributes Number

Sky Position 3Multiband Fluxes N = 5+Other M= 100+

Alex Szalay, JHU

Sky coordinatesSky coordinatesSky coordinatesSky coordinates

Stored as Cartesian coordinates:projected onto a unit sphere

Longitude and Latitude lines:intersections of planes and the sphere

Boolean combinations:query polyhedron

Stored as Cartesian coordinates:projected onto a unit sphere

Longitude and Latitude lines:intersections of planes and the sphere

Boolean combinations:query polyhedron

Alex Szalay, JHU

Sky PartitioningSky PartitioningSky PartitioningSky Partitioning

Hierarchical Triangular Mesh - based on octahedron

Alex Szalay, JHU

Hierarchical SubdivisionHierarchical SubdivisionHierarchical SubdivisionHierarchical Subdivision

Hierarchical subdivision of spherical trianglesrepresented as a quadtree

In SDSS the tree is 5 levels deep - 8192 triangles

Hierarchical subdivision of spherical trianglesrepresented as a quadtree

In SDSS the tree is 5 levels deep - 8192 triangles

Alex Szalay, JHU

Result of the QueryResult of the QueryResult of the QueryResult of the Query

Alex Szalay, JHU

Magnitudes and Multicolor SearchesMagnitudes and Multicolor SearchesMagnitudes and Multicolor SearchesMagnitudes and Multicolor Searches

Galaxy fluxes• large dynamic range• errors

divergent as x 0 ! 2

22

22

10010 log5.2)/(log5.2

x

xx

x

mm

xffm

2

22

22

10010 log5.2)/(log5.2

x

xx

x

mm

xffm

But: this is an artifact of the logarithm at zero flux,

in flux space the object is well localized

But: this is an artifact of the logarithm at zero flux,

in flux space the object is well localized

For multicolor magnitudes the error contours can be very anisotropic and skewed,

extremely poor localization!

Alex Szalay, JHU

Novel Magnitude ScaleNovel Magnitude ScaleNovel Magnitude ScaleNovel Magnitude Scale

cb

f

1sinh

10ln

5.2

cb

f

1sinh

10ln

5.2

b: softnessc: set to match normal magnitudes

Advantages: monotonic degrades gracefully objects have small error ellipse unified handling of detections

and upper limits!

Disadvantages: unusual

(Lupton, Gunn and Szalay, AJ 99)

Alex Szalay, JHU

Flux IndexingFlux IndexingFlux IndexingFlux Indexing

Split along alternating flux directionsCreate balanced partitionsStore bounding boxes at each stepBuild a 10-12 level tree in each triangle

Split along alternating flux directionsCreate balanced partitionsStore bounding boxes at each stepBuild a 10-12 level tree in each triangle

Alex Szalay, JHU

Therefore: first create a local density and split on its value (Csabai etal 96)

typical (98%) rare (2%)

Therefore: first create a local density and split on its value (Csabai etal 96)

typical (98%) rare (2%)

The SDSS will measure fluxes in 5 bands => asinh magnitudes

Axis-parallel splits in median flux, in 8 separate zones in Galactic latitude

=> 5 dimensional bounding boxes

The SDSS will measure fluxes in 5 bands => asinh magnitudes

Axis-parallel splits in median flux, in 8 separate zones in Galactic latitude

=> 5 dimensional bounding boxes

How to build compact cells?How to build compact cells?How to build compact cells?How to build compact cells?

The fluxes are strongly correlated=> 2 + dimensional distribution of typical objects=> widely scattered rare objects

=> large density contrasts

The fluxes are strongly correlated=> 2 + dimensional distribution of typical objects=> widely scattered rare objects

=> large density contrasts

Alex Szalay, JHU

Analysis Engine

Query Support

Data Warehouse

User Interface

Archive

Coarse Grained DesignCoarse Grained DesignCoarse Grained DesignCoarse Grained Design

Alex Szalay, JHU

User InterfaceUser Interface Analysis EngineAnalysis Engine

Master

Objectivity

ObjectivityObjectivity

RAIDRAID

Slave

ObjectivityObjectivity

RAIDRAID

Slave

ObjectivityObjectivity

RAIDRAID

Slave

ObjectivityObjectivity

RAIDRAID

Slave

SX Engine Objectivity Federation

Distributed ImplementationDistributed ImplementationDistributed ImplementationDistributed Implementation

Alex Szalay, JHU

JHU ContributionsJHU ContributionsJHU ContributionsJHU Contributions

Fiber spectrographs

P. FeldmanA. UomotoS. FriedmanS. Smee

Fiber spectrographs

P. FeldmanA. UomotoS. FriedmanS. Smee

Science Archive

A. SzalayA. ThakarP. Kunszt

I. CsabaiGy. SzokolyA. ConnollyA. Chaudhaury

A lot of help from

Jim Gray, Microsoft

Science Archive

A. SzalayA. ThakarP. Kunszt

I. CsabaiGy. SzokolyA. ConnollyA. Chaudhaury

A lot of help from

Jim Gray, Microsoft

Management

T. HeckmanT. PoehlerA. DavidsenA. UomotoA. Szalay

Management

T. HeckmanT. PoehlerA. DavidsenA. UomotoA. Szalay

Alex Szalay, JHU

Processing PlatformsProcessing PlatformsProcessing PlatformsProcessing Platforms

At Fermilab:

2 AlphaServer 8200 data processing

1 SGI Origin 2000 data bases

Archive at JHU:

1 AlphaServer 1000A (development)

10 Intel based servers w. LVD RAID

software verified on

Digital Unix, IRIX, Solaris, Linux

Alex Szalay, JHU

Exploring new methodsExploring new methodsExploring new methodsExploring new methods

New spectral classification techniquesgalaxy spectra can be expressed as a superpositionof a few (<5) principal components

=> objective classification of 1 million spectra!

New spectral classification techniquesgalaxy spectra can be expressed as a superpositionof a few (<5) principal components

=> objective classification of 1 million spectra!

Photometric redshiftsgalaxy colors systematically change with redshift,the SDSS photometry works like a 5-pixel spectrograph

=> z=0.05, but with 100 million objects!

Photometric redshiftsgalaxy colors systematically change with redshift,the SDSS photometry works like a 5-pixel spectrograph

=> z=0.05, but with 100 million objects!

Measuring cosmological parametersbefore: data analysis was limited by small number statisticsafter: dominant errors are systematic (extinction)

=> new analysis methods are required!

Measuring cosmological parametersbefore: data analysis was limited by small number statisticsafter: dominant errors are systematic (extinction)

=> new analysis methods are required!

Alex Szalay, JHU

Photometric redshiftsPhotometric redshiftsPhotometric redshiftsPhotometric redshifts

Multicolor photometry maps physical parametersluminosity Lredshift z spectral type T

Inversion: u’,g’,r’,I’,z’ => z, L, T

Multicolor photometry maps physical parametersluminosity Lredshift z spectral type T

Inversion: u’,g’,r’,I’,z’ => z, L, T

Redshifts are statistical, with large errors: z0.05

The data set is huge, more than 100 million galaxies

Easy to subdivide into coarse z bins, and by type=> study evolution=> enormous volume - 1 Gpc3

Redshifts are statistical, with large errors: z0.05

The data set is huge, more than 100 million galaxies

Easy to subdivide into coarse z bins, and by type=> study evolution=> enormous volume - 1 Gpc3

observed fluxes

Alex Szalay, JHU

Measuring P(k) Measuring P(k) Measuring P(k) Measuring P(k)

Karhunen-Loeve transform:Signal-to-noise eigenmodes of the redshift surveyOptimal extraction of clustering signalMaximal rejection of systematic errors

(Vogeley and Szalay 96, Matsubara, Szalay and Landy 99)

Karhunen-Loeve transform:Signal-to-noise eigenmodes of the redshift surveyOptimal extraction of clustering signalMaximal rejection of systematic errors

(Vogeley and Szalay 96, Matsubara, Szalay and Landy 99)

Pilot project using the Las Campanas Redshift Survey with 22,000 galaxies

Pilot project using the Las Campanas Redshift Survey with 22,000 galaxies

03.003.0

04.004.0

15.014.0

05.005.0

05.005.0

22.019.0

05.005.0

06.006.0

22.020.0

8

14.078.040.0

14.075.031.0

15.082.048.0

Combined

South

North

We simultaneously measure the values of the redshift-distortion parameter (=0.6/b), the normalization (8 ) and the CDM shape parameter ( = h).

We simultaneously measure the values of the redshift-distortion parameter (=0.6/b), the normalization (8 ) and the CDM shape parameter ( = h).

Alex Szalay, JHU

TrendsTrendsTrendsTrends

• Future dominated by detector improvements

Total area of 3m+ telescopes in the world in m2, total number of CCD pixels in Megapix, as a function of time. Growth over 25 years is a factor of 30 in glass, 3000 in pixels.

• Moore’s Law growth in CCD capabilities

• Gigapixel arrays on the horizon

• Improvements in computing and storage will track growth in data volume

• Investment in software is critical, and growing

Alex Szalay, JHU

The next generation of astronomical archives with Terabyte catalogs will dramatically change astronomy

top-down designlarge sky coveragebuilt on sound statistical plansuniform, homogeneous, well calibratedwell controlled and documented systematics

The technology to acquire, store and index the data is herewe are riding Moore’s Law

Data mining in such vast archives will be a challenge, but possibilities are quite unimaginable

Integrating these archives into a single entity is a project for the whole community

=> National Virtual Observatory

The Age of Mega-SurveysThe Age of Mega-SurveysThe Age of Mega-SurveysThe Age of Mega-Surveys

Alex Szalay, JHU

New Astronomy – Different!New Astronomy – Different!New Astronomy – Different!New Astronomy – Different!

Systematic Data Explorationwill have a central role in the New Astronomy

Digital Archives of the Skywill be the main access to data

Data “Avalanche”the flood of Terabytes of data is already happening, whether we like it or not!

Transition to the newmay be organized or chaotic

Alex Szalay, JHU

NVO: The ChallengesNVO: The ChallengesNVO: The ChallengesNVO: The Challenges

Size of the archived data• 40,000 square degrees is 2 trillion pixels

• One band: 4 Terabytes

• Multi-wavelength: 10-100 Terabytes

• Time dimension: few Petabytes

The development of• new archival methods

• new analysis tools

• new standards (metadata, interchange formats)

Hardware/networking requirements

Training the next generation!

Alex Szalay, JHU

SummarySummarySummarySummary

The SDSS project combines astronomy, physics, and computer scienceThe SDSS project combines astronomy, physics, and computer science

It promises to fundamentally change our view of the universeIt promises to fundamentally change our view of the universe

It will determine how the largest structures in the universe were formedIt will determine how the largest structures in the universe were formed

Its ‘virtual universe’ can be explored by both scientists and the publicIts ‘virtual universe’ can be explored by both scientists and the public

It will serve as the standard astronomy reference for several decadesIt will serve as the standard astronomy reference for several decades

Through its archive it will create a new paradigm in astronomyThrough its archive it will create a new paradigm in astronomy

Alex Szalay, JHU

www.sdss.org

www.sdss.jhu.edu