HEASARC and Related Data Center Activities

HEASARC and Related Data Center Activities

Frank Marshall

Acting HEASARC Director

NASA’s archive for X-ray and Gamma ray data

• Established in Nov 1990 – First wavelength specific “active” archive

• Partnership between GSFC and SAO (since 1999)• Contains data from 26 missions • All data in FITS format, along with associated software and calibrations• Provides the necessary scientific and technical expertise for the use and

interpretation of the data • Develop/maintain multi-mission analysis/support tools such as XSPEC,

PIMMS, and Proposal Submission• Online access to all data, catalogs of observations and sources and

browse data products• Defines and coordinates data, software, and media standards

HEASARC OverviewHEASARC Overview

Laboratory for High Energy Astrophysics OrganizationLaboratory for High Energy Astrophysics Organization

LHEAFrank Jones (acting)

Lois Workman660

X-ray Astrophysics Branch

Robert Petre662

Gamma Ray, Cosmic Ray & Gravitational Wave Astrophysics Branch

Neil Gehrels661

Instrument Development Office

Lois Workman (acting)660.2

Office of Guest Investigator Programs

& Data ManagementFrank Marshall

660.1

HEASARC Overall OrganizationHEASARC Overall Organization

HEASARC provides the multi-mission infrastructure that is used by the GOFs and science support centers:

Archive, database, web services, FITS standards, multi-mission software, & expertise

Science staff are co-located with LHEA science branches

HEASARC provides the multi-mission infrastructure that is used by dedicated mission data centers (RXTE, Chandra, XMM-Newton, INTEGRAL, Swift, Astro-E2, and GLAST)

Archive infrastructure, database support, web services, proposal software, FITS standards, multi-mission software, & science expertise

Prevents duplication of effort and promotes reuse of software which result in cost savings to missions

At the end of mission life the HEASARC maintains the archive, science expertise and software (e.g. CGRO, ASCA, ROSAT)

Astro-E2

INTEGRAL

Swift

GLAST

Rossi XTE

XMM-Newton

HEASARC Mission SupportHEASARC Mission Support

Chandra

HEASARC Budget & StaffingHEASARC Budget & Staffing

• Budget of $1.5M that mostly pays for people

• LHEA Science Staff (4 USRA, 1 UMD, 3 GSFC): – Angelini (Restoration, BeppoSAX, Swift, XIMAGE, XRONOS)– Arnaud (XSPEC, XSELECT)– Corcoran (ROSAT, Caldb, HETE-2)– Drake (User support, EUVE, Archive population)– Lochner (E/PO)– McGlynn (Archive Scientist, VO, SkyView, ClassX)– Pence (FTOOLS, FITSIO, FITS standards, HERA, FV)– Frank Marshall (Acting Director while N. White is at HQ)

• Two addition FTE at SAO (separately funded)

• Science staff have 30% time for research– Successfully compete for observing time e.g. XMM-Newton and Chandra

• Programming Staff (5 Software contractor, 1 GSFC)– Database, archive access, data ingest, data restoration, web services,

XANADU analysis software, education & outreach, system support, and user support.

• Additional funding competed for via e.g. AISRP

HEASARC Co-located Data CentersHEASARC Co-located Data Centers

• Provide dedicated support to specific missions utilizing HEASARC infrastructure– Develop mission specific software

– Interface between community, instrument teams and/or foreign data center

– Data processing and archive creation

– Proposal support and grants administration

– Science expertise in the specific mission

• Currently there are 7 active GOFs/SSCs:– XMM-Newton, RXTE, BeppoSAX (ASI), INTEGRAL, Swift, Astro-E2, GLAST

– 1 to 5 scientists per mission (typically 3), plus comparable number of programmers and support staff

– Co-located staff at foreign data centers (XMM-Newton, INTEGRAL, Astro-E2)

– Current total cost ~ $3M (depending on mission phase)

SAO ConnectionSAO Connection

• Coordinates SAO Chandra and HEASARC activities– Chandra calibration data & archive interface– Transparent access to Chandra archive from

HEASARC Browse– DS9 Software– Remote Proposal Submission Software– Two FTE staff at SAO– Steve Murray directs the activity

Actions from Previous HUGActions from Previous HUG

• Offsite archive backup. Backup in second building is planned.• Consistent statistics for outreach web-sites. Done.• Browse suggestions/concerns: plotting, cross-correlation radii, data

products selection, exposure times, results formatting. Some implemented, others in process.

• Need HETE-2 Data in FITS. Limited FITS support, see HETE-2 talk.

• nH issues, limits. Link to MAST-supported alternative

• Integral simulator and calibration concerns. See Integral Talk.• NVO impact on HEASARC. ADEC White Paper.

HighlightsHighlights

• INTEGRAL launch• New load-balancing Web and database servers• 2500 Chandra and 1100 XMM datasets in archive by end of

2002.• Browse full integration of remote (Chandra) data products.• Bibliography/data links to and from ADS for XMM, RXTE,

ASCA, ROSAT• XTE table and data product simplification.• GRBCAT and “Is it an X-ray source?” tools.• Outreach CD rated “outstanding” in NASA review• SkyView featured in Scientific American and Wired.• GRB follow-up VO prototype shown at AAS.

Highlights: HEADCC MeetingHighlights: HEADCC Meeting

• Coordinate activities of all high-energy astrophysics missions and centers.

• Held Nov. 13, 2002 at CfA• Agreements

– Work to ensure interoperability of Chandra and XMM data sets.– Coordinate development of POW, FV and DS9 viewing tools– Look at upgrading CALDB to meet requirements of Chandra and

other newer missions.– Continue to work towards a single parameter interface layer.– Explore building standard model for coded aperture mask

instruments.

The Physical ArchiveThe Physical Archive

Active Missions

RXTE (1995- )Chandra (1999- ) [data at CXC]HETE-2 (2000- ) Integral (2001-)XMM-Newton (1999- )

Past Missions

Ariel 5 EXOSATASCA GingaBBXRT BeppoSAX CGRO HEAO 1Copernicus HEAO 3COS B OSO 8DXS ROSAT Einstein SAS 2EUVE SAS 3 Vela 5B

• Data from 26 missions currently in the archive

• 344 astronomical catalogs & mission tables (up from 307 at last HUG meeting)

• The archive volume was2700 Gigabytes as of the end of 2002

Upcoming Missions

Swift (late 2003)Astro-E2 (2005)GLAST (2006)

Usage & Data StatisticsUsage & Data Statistics

Gigabytes retrieved per quarter

Gigabytes transferred per year1998 1999 2000 2001 2002

FTP 842 1391 1846 2251 2481WWW (http) 424 603 1083 1428 2931

Browse tar file directory

Data Transfer Rates (ftp & http)Data Transfer Rates (ftp & http)

The ftp transfer rate in 2002 averaged 6.8 GB per day, up from 6.2 GB per day in 2001. Data transferred was predominantly RXTE (56%) with other missions of note, ASCA and ROSAT (3.7% each), CGRO (3.2%) and XMM-Newton (2.7%). Significant data transfers were also made from non-mission directories such as Software (17%) and Retrieve (13%).

Data Attractiveness (DA, defined for each mission as data ftped in a year divided by the archive size):

RXTE DA=129% ROSAT DA=63%CGRO DA=37% XMM DA=29%

The http transfer rate of Web pages, images, etc., in 2002 was 8 GB per day (244 GB per month), compared to 3.9 GB per day (119 GB per month) in 2001. The http transfer rate in 2002 was 118% of the ftp rate - the first time the http transfers exceeded the ftp transfers.

Monthly transfers from the HEASARC’s EPO areas “Imagine the Universe” and “StarChild” averaged ~500,000 requests and ~1,100,000 requests during the school years, respectively. They get fewer hits during the summer.

HEASARC Data ArchiveHEASARC Data Archive

As of November 2002, the HEASARC’s public data holdings (the anonymous ftp area) were about 2688 GB, including some still-proprietary datasets. (Duplicate ROSAT and ASCA datasets totaling ~160 GB have been removed from these statistics.) The biggest mission archives are RXTE (40% of the total) and ASCA (20%), with XMM-Newton and CGRO each ~8%, ROSAT and BeppoSAX each ~5%, and EXOSAT and HETE-2 just under 4% each.

In the last 12 months, the HEASARC archive growth was about 1.4 GB per day (~500 GB over the year), to which the rapid growth of the XMM-Newton archive and the opening of the HETE-2 archive were major contributors. The growth rate in 2003 is expected to be similar.

Future Archive GrowthFuture Archive Growth

In the next 3 years, there are several new missions that plan to archive data at the HEASARC:

Swift (late 2003 launch)Astro-E2 (early 2005 launch)GLAST (fall 2006 launch)

These new missions are expected to each generate 400-1000 GB of data per year so the HEASARC archive doubling time is predicted to be about 2.5 years.

Very rough HEASARC archive size projections:Date Archive size2004.0 41202005.0 60552006.0 7990

Summary and ProspectsSummary and Prospects• The HEASARC is a mature system with diverse and growing

holdings.• Archive and software systems continue to evolve to meet the

needs of the community• Over the next several years, many of the newer missions will

operate in the hard X-ray and gamma-ray regimes.– Science goals, operations and other requirements somewhat different from

many earlier missions.

• The diversity of the HEASARC holdings makes it a smaller scale ‘Virtual Observatory’– Links to other NASA and ground data enhances value of HEASARC holdings– HEASARC experience can be valuable in guiding VO efforts– Conversely, more generic VO approaches may be useful within the

HEASARC.