AMSR-E and AMSR-E Validation Status at NSIDC Amanda Leon, NSIDC AMSR-E Lead Joint AMSR-E Science...
-
Upload
paulina-flynn -
Category
Documents
-
view
225 -
download
0
Transcript of AMSR-E and AMSR-E Validation Status at NSIDC Amanda Leon, NSIDC AMSR-E Lead Joint AMSR-E Science...
AMSR-E and AMSR-E ValidationStatus at NSIDC
Amanda Leon, NSIDC AMSR-E LeadJoint AMSR-E Science Team MeetingAsheville, NC28-29 June 2011
Joint AMSR-E Science Team Meeting
Amanda Leon - NSIDC DAAC 28-29 June 20112
Outline
• Accomplishments• Data Support • Activities at NSIDC• Metrics
AMSR-E AMSR-E Validation
Sea ice derived from the AMSR-E 89 GHz Tbs. Image courtesy
of NASA/GSFC Visualization Studio
Joint AMSR-E Science Team Meeting
Amanda Leon - NSIDC DAAC 28-29 June 20113
Accomplishments
• Release of Sea Ice Drift Version 1 product (24 June 2011) Documentation & metadata creation End-to-end ingest testing with the SIPS Comprehensive distribution testing
o Enabled spatial and parameter subsetting via WIST/Reverb
• Improved product latency Level-2 product latency reduced from 5 days to 20 hours!!! Level-3 daily product latency reduced from 5 days to 4 days
AMSR-E Data Versions: http://nsidc.org/data/amsre/versions.html
Accomplishments
• LANCE: SIPS-NSIDC collaboration Close coordination to ensure Preliminary data from the SIPS phased out in lieu of LANCE Joint poster presented at A-Train meeting and AGU
• Google Earth Animations using LANCE data 12 km Sea Ice & Daily Soil Moisture nsidc.org/data/virtual_globes/index.html
• Upgrade to EOSDIS Core System (ECS)
Joint AMSR-E Science Team Meeting
Amanda Leon - NSIDC DAAC 28-29 June 20114
Joint AMSR-E Science Team Meeting
Amanda Leon - NSIDC DAAC 28-29 June 20115
Data Support: Data Access
Data Pool• On-line access to entire archive nsidc.org/data/data_pool/• Subsetting, reformatting, reprojection
Warehouse Inventory Search Tool (WIST)• Search/order from ECHO wist.echo.nasa.gov• Subsetting
Subscriptions• Automatic delivery of new data nsidc.org/opendap/
through FTP push or pull
http://nsidc.org/data/amsre/order_data.html
Joint AMSR-E Science Team Meeting
Amanda Leon - NSIDC DAAC 28-29 June 20116
Data Support: New Data Access
REVERB• WIST replacement reverb.echo.nasa.gov/reverb/• Improved user interface and search/order workflow
OpenDAP• Provides simple access to remote nsidc.org/opendap/
data• Currently limited use and not publicized
Data Casting (RSS Feed)• Automatic delivery of new data nsidc.org/feeds/data/AE_Ocean.xml
through an RSS feed• Currently available for L2 Ocean only
http://nsidc.org/data/amsre/order_data.html
Joint AMSR-E Science Team Meeting
Amanda Leon - NSIDC DAAC 28-29 June 20117
Data Support
User Services [email protected]
• Provide support of data products, in-house developed tools, and proprietary software
Documentation nsidc.org/data/amsre/data_summaries/
• Comprehensive product documentation• Algorithm version information nsidc.org/data/amsre/versions.html
Tools nsidc.org/data/amsre/data_summaries/
• General HDF and specific data set tools• New Data Viewer - interactive tool for viewing satellite passive
microwave data sets
Published Research nsidc.org/data/amsre/research.html
NSIDC Activities
Joint AMSR-E Science Team Meeting
Amanda Leon - NSIDC DAAC 28-29 June 20118
Digital Object Identifiers (DOIs) & Versioning• Develop process for implementation of DOIs
•Recommendations for NSIDC•Major.Minor version schema (AMSR-E Version 2, V10 = 2.10)•Each major version of a data set gets a separate DOI
•Recommendation for AMSR-E data managementIncrement the ESDT (major) version number when a change to an
algorithm warrants reprocessing the entire time seriesCurrently archiving both Version 1 & 2. Will need to discuss version
retention policy moving forward.
HDF4 content maps• Collaborative study between NASA Earth Science Data Centers & The HDF
Group focused on ensuring long term readability of HDF4 data• XML file containing a “map” to the contents of the HDF4 file
Enables reading of the data without HDF4 libraries• ECS 8.1 will generate, archive, and distribute the HDF4 content maps
NSIDC Activities
Joint AMSR-E Science Team Meeting
Amanda Leon - NSIDC DAAC 28-29 June 20119
Metadata Project• Provide overall improvements to NSIDC’s metadata acquisition,
creation, management, and dissemination • Replace or enhance existing metadata infrastructure
NOAA Sea Ice Concentration CDR – W. Meier• Funded by the NOAA Climate Data Record grant• Integrates current SSM/I-SSMIS algorithms (NASA Team and
Bootstrap) distributed by NSIDC into one combined product• NetCDF4 file will include data quality fields, metadata, & provenance
Green Data Center• $525,000 NSF research instrumentation grant• Data center will transition power consumption from 100kw (42kw load for
data center equipment, 58kw for cooling) to ~45-50kw (cooling reduced by 95%)
NSIDC Activities
Joint AMSR-E Science Team Meeting
Amanda Leon - NSIDC DAAC 28-29 June 201110
• Four Phase project: 1. Machine room
consolidation2. Server Virtualization
(60% reduction in servers)
3. High efficiency cooling system
4. Solar photo-voltaic power supplementation system with battery array
Coolerado Units
http://news.nationalgeographic.com/news/news/energy/2010/11/101227-national-snow-and-ice-data-center-makeoverNational Geographic Online Article
Joint AMSR-E Science Team Meeting
Amanda Leon - NSIDC DAAC 28-29 June 201111
Archive Metrics: Volume1 June 2002 – 1 June 2011
Stats from EMS
Version 2 Archive Volume
Total Archive: 20 TBs
Total Version 2 Archive: 11 TB
ProductVolume (GB)
AE_5DSNO 1.28
AE_DYOCN 44.04
AE_DYSNO 6.49
AE_L2A 4795.08
AE_LAND 65.48
AE_LAND3 195.56
AE_MOOCN 1.46
AE_MOSNO 0.22
AE_OCEAN 1119.35
AE_RAIN 1001.46
AE_RNGD 0.02
AE_SI12 187.50
AE_SI25 61.79
AE_SI6 139.93
AE_WKOCN 6.32
AMSREL1A 3456.872002 2003 2004 2005 2006 2007 2008 2009 2010 2011
0.00
1000.00
2000.00
3000.00
4000.00
5000.00
6000.00
Yearly Total Archive(excludes deleted data)
Joint AMSR-E Science Team Meeting
Amanda Leon - NSIDC DAAC 28-29 June 201112
Distribution Metrics: Total
1 June 2002 – 1 June 2011
Stats from EMS
Total Distribution:454 TBs
13.8 million files
Data Type Files Volume (TB)
AE_5DSNO 54331 0.09
AE_DYOCN 72224 0.80
AE_DYSNO 360128 0.63
AE_L2A 6444515 330.61
AE_LAND 1246548 0.75
AE_LAND3 897300 43.06
AE_MOOCN 2717 0.03
AE_MOSNO 11517 0.02
AE_OCEAN 1330601 14.39
AE_RAIN 2188690 18.56
AE_RNGD 3980 0.00
AE_SI12 435767 20.62
AE_SI25 179977 2.95
AE_SI6 168152 6.25
AE_WKOCN 7086 0.08
AMSREL1A 448112 15.12
Total 13851645 453.97
Joint AMSR-E Science Team Meeting
Amanda Leon - NSIDC DAAC 28-29 June 201113
Distribution Metrics: Files2003 - 2011 File Distribution
Stats from EMS
2003 2004 2005 2006 2007 2008 2009 2010 20110
500000
1000000
1500000
2000000
2500000
3000000
3500000
332369
874314
561729
704166
13370341555637
752830
457444
307185
222507
537633349628
434835
397739 468197
AMSREL1A
AE_WKOCN
AE_SI6
AE_SI25
AE_SI12
AE_RNGD
AE_RAIN
AE_OCEAN
AE_MOSNO
AE_MOOCN
AE_LAND3
AE_LAND
AE_L2A
AE_DYSNO
AE_DYOCN
AE_5DSNO
3 million files
2 million files
3 million files
Joint AMSR-E Science Team Meeting
Amanda Leon - NSIDC DAAC 28-29 June 201114
Distribution Metrics: Volume2003 – 2011 Volume Distribution
Stats from EMS
2003 2004 2005 2006 2007 2008 2009 2010 20110.00
20000.00
40000.00
60000.00
80000.00
100000.00
120000.00
15177.22
37136.91 34356.60
55704.22
76351.78
72975.47
34232.06
6859.83
11013.6211448.51
AMSREL1A
AE_WKOCN
AE_SI6
AE_SI25
AE_SI12
AE_RNGD
AE_RAIN
AE_OCEAN
AE_MOSNO
AE_MOOCN
AE_LAND3
AE_LAND
AE_L2A
AE_DYSNO
AE_DYOCN
AE_5DSNO
100 TB
50 TB
Joint AMSR-E Science Team Meeting
Amanda Leon - NSIDC DAAC 28-29 June 201115
Distribution Metrics: Data Access Method
Stats from EMS
2003 – 2011 Data access method based on data volume
Overall Average: Past 3 Year Average:
ECHO/EDG orders = 52 % ECHO = 28%DataPool = 43% DataPool = 70%Subscriptions = 5% Subscriptions = 2%
2003 2004 2005 2006 2007 2008 2009 2010 20110%
10%
20%
30%
40%
50%
60%
70%
80%
90%
100%
81%
19%
48%
36%
77%
61%
73%
37%
50%
46%
35%
34%24%
18%
21% 18%
23%
18%
ECHO - SUBSETTED
ECHO - FTP PULL
ECHO - FTP PUSH
SUBSCRIPTION
FTP PUSH
FTP PULL
DVD
DLT
CD
DATAPOOL
Joint AMSR-E Science Team Meeting
Amanda Leon - NSIDC DAAC 28-29 June 201116
Distribution Metrics: Users1 June 2002 – 1 June 2011
Stats from EMS & USO TRAK Database
2192 Distinct Users
Product Distinct Users
AE_5DSNO 167
AE_DYOCN 102
AE_DYSNO 305
AE_L2A 712
AE_LAND 305
AE_LAND3 719
AE_MOOCN 47
AE_MOSNO 124
AE_OCEAN 166
AE_RAIN 189
AE_RNGD 71
AE_SI12 339
AE_SI25 165
AE_SI6 120
AE_WKOCN 25
AMSREL1A 97
User Type Distinct Users
FOREIGN COMMERCIAL 28
FOREIGN GOVERNMENT 117
FOREIGN OTHER 647
FOREIGN UNIVERSITY 461
ECS OTHER 128
NASA 140
NOAA 29
US COMMERCIAL 35US FEDERAL GOVERNMENT 27
US GENERAL PUBLIC 8
US GOVERNMENT 14
US OTHER 89
US STATE/LOCAL GOVERNMENT 11
US UNIVERSITY 455
WDC 2
U.S. Users - 43%
Foreign Uses - 57%
Joint AMSR-E Science Team Meeting
Amanda Leon - NSIDC DAAC 28-29 June 201117
Distribution Metrics: Users
Stats from USO TRAK Database
2003 – 2011 Total Active Distinct Users per Year
2003 2004 2005 2006 2007 2008 2009 2010 20110
100
200
300
400
500
600
52
198
336
429
493
510
466
414
228
Joint AMSR-E Science Team Meeting
Amanda Leon - NSIDC DAAC 28-29 June 201118
AMSR-E Validation: Distribution Metrics
• No new AMSR-E Validation data sets delivered to NSIDC
• Published last data set in March 2010
Joint AMSR-E Science Team Meeting
Amanda Leon - NSIDC DAAC 28-29 June 201119
Distribution Metrics: Archive volume
Rainfall Cryosphere Soil Moisture0
50
100
150
200
250
300
350
400
450
500
5.1
476
94
Archive volume (GB) through 1 June 2011Total Archive Volume = 575 GB
Joint AMSR-E Science Team Meeting
Amanda Leon - NSIDC DAAC 28-29 June 201120
Distribution Metrics: UsersDistinct users through 1 June 2011.
Total Distinct Users = 4,95233% increase from last year
Statistics from our University FTP logs
Rainfall Cryosphere Soil Moisture0
500
1000
1500
2000
2500
3000
435
1780
2737
Joint AMSR-E Science Team Meeting
Amanda Leon - NSIDC DAAC 28-29 June 201121
AMSR-E Validation: MetricsFiles distributed through 1 June 2011
Total Files Distributed = 772,36423% increase from last year
Statistics from our University FTP logs
Rainfall Cryosphere Soil Moisture0
50000
100000
150000
200000
250000
300000
350000
400000
450000
500000
23870
273349
475145
Joint AMSR-E Science Team Meeting
Amanda Leon - NSIDC DAAC 28-29 June 201122
Metrics: VolumeVolume (GB) distributed through 1 June 2011
Total Volume Distributed = 2 TB25% increase from last year
Statistics from FTP logs
Rainfall Cryosphere Soil Moisture0.00
200.00
400.00
600.00
800.00
1000.00
1200.00
121.88
846.16
1093.41
Joint AMSR-E Science Team Meeting
Amanda Leon - NSIDC DAAC 28-29 June 201123
AMSR-E Team at NSIDC
Thanks to the AMSR-E Teams at NSIDC!
AMSR-E
Molly Thomas – Operations
Karla LeFevre – Technical Writer
Lisa Booker – User Services
Sean Mallory – Programmer
Siri Jodha Khalsa – ECS Science Coordinator
Walt Meier – Scientist
Doug Fowler – Test Engineer
Cathy Fowler – Database Administrator
AMSR-E Validation
Molly Thomas – Operations
Karla LaFevre – Technical Writer
Dan Webster – Technical Writer
Lisa Booker – User Services
Sean Mallory – Programmer
Joint AMSR-E Science Team Meeting
Amanda Leon - NSIDC DAAC 2-3 June 201024