Earth Science Data and Information System (ESDIS) Project Update

Karen.Michael@nasa.gov, EOSDIS System ManagerAMSR Science Team Meeting Sept 23-24, 2014

Huntsville, AL

ESDIS Project

EOSDIS Facilities

3

Data centers, collocated with centers of science discipline expertise, archive and distribute

standard data products produced by Science Investigator-led Processing Systems (SIPSs)

ASF DAACSAR ProductsSea Ice, Polar

Processes

SEDACHuman Interactions

in Global Change

LP DAACLand Processes

& Features

PO.DAACOcean Circulation

Air-Sea Interactions

ASDCRadiation Budget,Clouds, Aerosols,Tropo Chemistry

ORNL DAACBiogeochemical

Dynamics, EOS Land Validation

GES DISCAtmos Composition &

Dynamics, GlobalModeling, Hydrology,

Radiance

LAADS/MODAPS

Atmosphere

OBPGOcean Biology &Biogeochemistry

GHRCHydrological Cycle &

Severe Weather

CDDISCrustal Dynamics

Solid Earth

NCAR, U of Col.HIRDLS, MOPITT,

SORCE

GSFCGLAS, MODIS,

OMI, OBPG

LaRCCERES, SAGE III

GHRCAMSR-E, LIS,

AMSR2

JPLMLS, TES

San DiegoACRIM

NSIDC DAACCryosphere, Polar

Processes

SIPSs

Key

DataCenter

ECSSites

0

100

200

300

400

500

600

700

800

900

FY00 FY01 FY02 FY03 FY04 FY05 FY06 FY07 FY08 FY09 FY10 FY11 FY12 FY13

EOSDIS Product Distribution Trend Millions

0.00

2.00

4.00

6.00

8.00

10.00

12.00

FY00 FY01 FY02 FY03 FY04 FY05 FY06 FY07 FY08 FY09 FY10 FY11 FY12 FY13

Volu

me

(PBs

)

Multi-year Total Archive Volume (PBs) Trend

EOSDIS Key Metrics

4

EOSDIS Science Data Products Distribution

FY2000 through FY2013

EOSDIS Science Data Products Distribution

FY2000 through FY2013

EOSDIS Science Data Volume Growth

EOSDIS Science Data Volume Growth

EOSDIS Metrics FY2013 (Oct 1, 2012 to Sept 30, 2013)

Unique Data Products 6,861

Distinct Users of EOSDIS Data and Services

1.7 M

Average Daily Archive Growth8.5

TB/day

Total Archive Volume 9.8 PB

End User Distribution Products 839 M

End User Average Daily Distribution Volume

22 TB/day

Atmosphere, 224.2

Geodesy/Solid Earth, 121.7

Land, 109.4

Ocean, 93.0

Hydrology, 83.6

Others, 83.4

Terrestrial Ecology, 51.4

Radiance/Geolocation, 50.8

Raw Data, 21.9

Distribution by Discipline Product Counts (M)

(Oct 2012- Sep 2013)

REVERB - Data and Service Access Client

• Primary web-based client for discovering and ordering cross-discipline data from all of EOS Clearing House (ECHO) metadata holdings.

• Allows users, including those without specific knowledge of the data, to search science data holdings, retrieve high-level descriptions of data sets, view browse images, and submit orders via ECHO to appropriate data providers.

• Users can submit queries using spatial and temporal criteria and examine search results for relevance using built-in tools.

https://earthdata.nasa.gov/reverb

GCMDScienceKeywords

FreeTextSearch

The GCMD and ECHO repositories will eventually merge to form the Common Metadata Repository (CMR), a single, shared metadata repository for EOSDIS collection and granule level metadata.

REVERB - Data and Service Access Client

LANCE – A Component of EOSDIS

The Land Atmosphere Near-real time Capability for EOS (LANCE) is a component of EOSDIS that generates and distributes near real-time (NRT) products from the following instruments:

• Terra: MODIS

• Aqua: MODIS, AIRS

• Aura: OMI, MLS

• AMSR-E (Aqua) – ended 2011

• AMSR2 (GCOM-W1) – future

• MISR (Terra) – being added

LANCE Objectives: • To leverage science processing expertise to create high quality NRT products• To provide data to applications community with a maximum latency of 3 hours

from observation (The standard, science-quality products are typically available with a latency of 20-48 hours)

• To provide data products with high reliability using redundant systems• To provide an umbrella environment with uniform high level requirements to

better serve user needs (Consistency, Collaboration, Coordination)7

LANCE Metrics 2014

8

ESDIS Data Discovery Capabilities

Worldview (https://earthdata.nasa.gov/labs/worldview)• Supports time-critical application areas such as wildfire management, air quality

measurements, and weather forecasting.• Allows users to interactively browse satellite imagery in near real-time, add

layers, and also provides a time series for comparison. GIBS (Global Imagery Browse Services)• GIBS are a set of standard services to deliver global, full-resolution browse

satellite imagery in a highly responsive manner• Full Resolution mosaicked layers that are chopped into imagery tiles stored at

predefined zoom levelsWorldview/GIBS currently consists of:

• Routine NRT production of: MODIS (C5), AIRS, MLS, OMI, Ocean Color• AMSR-E historical data in process of being added

– Just added 5 Cryospheric Layers in Arctic, Antarctic and Geographic projections from 2002-2011» : Sea Ice Brightness Temperatures (89H GHz, 89V GHz), Sea Ice

Concentration (12km,25km), Snow Depth over Ice (12km)• GIBS/Worldview upcoming plans consist of adding:

• Routine NRT production of GPM from PPS• Standard SMAP products• MODIS Collection 6 (Forward + Reprocessed)

9

Continuing Browse Enhancements

GIBS and Worldview• Data granules, used to create the imagery are also searchable and can be

downloaded through the Worldview interface. • Delivery via Standard Services OGC WMTS / TWMS / WMS / KML

Clients can be built to use and display images in GIBS – WorldView is an example

10

All GIBS Product Downloads vs LANCE NRT

File Transfer Protocol Testing

Compared file transfer protocol performance across different network environments in ESDIS Network lab and to Alaska• FTP, FTPS, HTTP, SCP, SFTP, HTTPS, bbFTP, GridFTP• Varied file size, packet loss rate, round-trip time to emulate

different network environmentsTest findings:• Multiple stream file transfer protocols perform best sending

large files over networks with high packet loss and long round-trip times (WAN)• bbFTP, GridFTP

• Single stream file transfer protocols perform best sending small files over networks with short round-trip times (LAN)• FTP, FTPS, HTTP

• Protocols that encrypt the data are consistently the slowest. For best performance, data encryption should be avoided unless it is a requirement. • SCP, SFTP, HTTPS

Future File Transfer Protocol Work

Continue working with DAACs and EOSDIS partners choose the best performing protocols for file transfers• Some DAACs have started incorporating multi-stream

protocols, bbFTP and GridFTP, for sending large files

Test additional protocols, including Aspera and MMC ftp including tests with JAXA

Recent and Upcoming Missions

14

Missions Launch Date DAAC

SNPP Oct 28, 2011 Multiple

GPM Feb 27, 2014 GES DISC

OCO-2 Jul 2, 2014 GES DISC

SMAPDec 15, 2014 or

Jan 29, 2015 ASF / NSIDC

DSCOVR Jan 2015 ASDC

SAGE-III on ISS Jun 30, 2015 ASDC

OCO-3 on ISS Aug 1, 2016 GES DISC

CYGNSS (EV-M) Oct 28, 2016 PO.DAAC

GRACE FO Jul 8, 2017 PO.DAAC

ICESat-2 Jul 27, 2017 NSIDC

Future Suomi National Polar-orbiting Partnership Plans for NASA SDS

• Current architecture–SNPP data is processed by NOAA IDPS

–ESDIS manages the Science Data Segment (SDS) of SNPP and was responsible for product evaluation for VIIRS, CrIS, ATMS, OMPS Nadir as well as the production and distribution of CERES, and OMPS Limb

–Archive and distribution of SNPP is at NOAA CLASS

• Future Architecture–NASA will be producing and distributing it’s own products from

all SNPP instruments and potentially future JPSS missions including NRT

–Science Investigator-led Processing Systems (SIPS) and Science Teams have recently been selected for the following disciplines: Land, Ocean, Atmosphere, Ozone, Sounder

15

Soil Moisture Active Passive

• SMAP Soil Moisture Active Profile• ORR Nov 4-5, 2014• Launch Dec 15, 2014, or January 29, 2015• Downlink to Alaska, McMurdo, Svalbard, Wallops (backup)• 15 contacts per day; 135 GB/day• 3 year mission timeline• SMAP data will be processed to L0 by EDOS• SMAP science processing will take place at JPL and

GSFC/GMAO• Ingest, archive and distribution of SMAP data will be at ASF

for Radar data (L1) and NSIDC for Radiometer data (L1-L4)

16

Preservation

NASA is not a “permanent archive” agency• Must maintain “research archive” for as long as data are used for scientific

research and/or transition responsibility to permanent archives• Research archive responsibilities persist well beyond lives of missions • NASA works with USGS and NARA for long-term preservation• NASA has to ensure data and other critical items are preserved and made

available to permanent archival agenciesNASA has developed Preservation Content Specifications for Earth Science DataCategories of Data to be preserved include:• Preflight/Pre-Operations• Science Data Products• Mission Data Calibration• Science Data Product Software• Science Data Product Algorithm Input• Science Data Product Validation• Science Data Software Tools

17

Acronyms

AIRS Atmospheric Infrared SounderAMSR Advanced Microwave Scanning Radiometer ASDC Atmospheric Science Data Center ASF Alaska Satellite FacilityATMS Advanced Technology Microwave SounderbbFTP BaBar File Transfer ProtocolCERES Clouds and the Earth’s Radiant Energy SystemCMR Common Metadata RepositoryCrIS Cross-track Infrared SounderCTGNSS Cyclone Global Navigation Satellite SystemDAAC Distributed Active Archive CenterDSCOVR Deep Space Climate ObservatoryECHO EOS Clearing HouseEDOS EOS Data and Operations SystemEOS Earth Observing CenterEOSDIS Earth Observation System Data and Information SystemESDIS Earth Science Data and Information SystemESIP Earth Science Information Partners FTP File Transfer ProtocolFTPS File Transfer Protocol SecureGB GigabyteGCMD Global Change Master Directory

GCOMGlobal Change Observation Mission – 1st Water Satellite in Series

GES DISCGSFC Earth Sciences Data and Information Services Center

GHRC Global Hydrology Resource CenterGIBS Global Imagery Browse Services GMAO Global Modeling and Assimilation OfficeGPM Global Precipitation MeasurementGRACE-FO Gravity Recovery And Climate Experiment – Follow-onGridFTP Grid File Transfer ProtocolHTTP Hypertext Transfer ProtocolICESat Ice, Cloud, and Land Elevation Satellite-2IDPS Interface Data Processing Segment

ISS International Space StationJAXA Japan Aerospace Exploration AgencyJPL Jet Propulsion LaboratoryJPSS Joint Polar Satellite SystemKML Keyhole Markup LanguageLAN Local Area NetworkLANCE Land Atmosphere Near real-time Capability for EOSMISR Multi-angle Imaging SpectroRadiometerMLS Microwave Limb SounderMODIS Moderate Resolution Imaging SpectrometerMSFC Marshall Space Flight CenterNARA National Archives and Records AdministrationNASA National Aeronautics and Space AdministrationNOAA National Oceanic and Atmospheric AdministrationNRT Near Real-Time

NSIDCNational Snow and Ice Data Center Distributed Active Archive Center

OCO Global Modeling and Assimilation OfficeOGC Open Geospatial ConsortiumOMI Ozone Monitoring InstrumentOMPS Ozone Mapper/Profiler SuiteORR Operational Readiness ReviewPPS Precipitation Processing SystemSAGE Stratospheric Aerosol Gas ExperimentSCP Secure CopySFTP Secure Shell File Transfer ProtocolSIPS Science Investigator-led Processing SystemSMAP Soil Moisture Active PassiveSNPP Suomi National Polar-orbiting PartnershipTWMS Tile Web Map ServiceUSGS United States Geological SurveyVIIRS Visible Infrared Imager Radiometer SuiteWAN Wide Area NetworkWMS Web Map ServiceWMTS Web Map Tile Service