IDN Metadata Overarching goal: A dynamic component for energizing CEOS Cyber-infrastructure....
68
IDN Metadata Overarching goal: A dynamic component for energizing CEOS Cyber-infrastructure. n Microwave Anisotropy Probe | http://map.gsfc.nasa.gov/inde
-
Upload
reynold-holt -
Category
Documents
-
view
213 -
download
0
Transcript of IDN Metadata Overarching goal: A dynamic component for energizing CEOS Cyber-infrastructure....
IDN Metadata Overarching goal: A dynamic component for energizing CEOS Cyber-
infrastructure.
IDN Metadata Overarching goal: A dynamic component for energizing CEOS Cyber-
infrastructure.
Wilkinson Microwave Anisotropy Probe | http://map.gsfc.nasa.gov/index.html
WGISS23- IDN TT Hanoi , Vietnam
presented by Arturo Restrepo - Task leader: Lola Olsen
WGISS23- IDN TT Hanoi , Vietnam
presented by Arturo Restrepo - Task leader: Lola Olsen
• Introduction• Review of Minutes on Action Items from Annapolis
New Portals added since September 2006
NASA Atmospheric Science Data Center
National Snow and Ice Data Center
AMD: China AMD: Korea
AMD Malaysia
AMD: The Netherlands
SCAR-Marine Biodiversity Information Network
Currently being reviewed *
Ocean Climate
Global Climate Observing System (GCOS)
(2) PODAAC Data Center
GEOSS Data
GEOSS Data Services
In Response to Proposed GEOSS Action Items…
List of GEOSS demonstration portals: Tsunami Data Tsunami Data Services Forest Ecosystems Water Management Data in Latin America
Portals created in responseto CEOS/GEOSS requests *
IDN Newsletter and Plans for Germany
IDN Newsletter and Plans for Germany
IDN Newsletter | April 2007 IDN Newsletter | April 2007
Volunteers for Articles to be published at next WGISS
Germany
IDN Usage Statistics
CEOS IDN Task Team
MetricsTools Used by the IDN
MetricsTools Used by the IDN Analalog Webalizer (AW) Stats
Free, open source, real-time log analyzer Basic web usage: unique hosts, pages viewed, hits Includes graphs Runs daily Provides GCMD and IDN metrics for hits on web site
Nettracker Commercial, Goddard Space Flight Center-licensed product; currently being
upgraded by NASA Highly customizable Includes graphs Runs daily Provides GCMD search and retrieval and portal usage metrics
Custom In-house log analysis Highly customizable Metrics can be imported into spreadsheets Runs monthly Provides GCMD and IDN metrics on population , search and retrieval, portal
usage. Provides custom AMD and IDN portal metrics
Analalog Webalizer (AW) Stats Free, open source, real-time log analyzer Basic web usage: unique hosts, pages viewed, hits Includes graphs Runs daily Provides GCMD and IDN metrics for hits on web site
Nettracker Commercial, Goddard Space Flight Center-licensed product; currently being
upgraded by NASA Highly customizable Includes graphs Runs daily Provides GCMD search and retrieval and portal usage metrics
Custom In-house log analysis Highly customizable Metrics can be imported into spreadsheets Runs monthly Provides GCMD and IDN metrics on population , search and retrieval, portal
usage. Provides custom AMD and IDN portal metrics
6000
8000
10000
12000
14000
16000
18000
20000
Jan-
01
Apr-0
1
Jul-0
1
Oct-0
1
Jan-
02
Apr-0
2
Jul-0
2
Oct-0
2
Jan-
03
Apr-0
3
Jul-0
3
Oct-0
3
Jan-
04
Apr-0
4
Jul-0
4
Oct-0
4
Jan-
05
Apr-0
5
Jul-0
5
Oct-0
5
Jan-
06
Apr-0
6
Jul-0
6
Oct-0
6
Jan-
07
Apr-0
7
Date
#
DIF
s
Total DIFs January 2001 - Apr 2007
New Antarctic DIFs
Split of Aqua MODIS DIFs from Terra MODIS DIFs
Removal of 400+ EOSWEBSTER ESIP Child DIFs
Replacement ofAntarctic DIFs
New, Revised and Deleted DIFs 2003-2007
0
500
1000
1500
2000
2500
3000
3500
4000
# DIFs
New Deletes Revised
Actions
DIF Population by Earth Science Topic
1649
5393
2552
4784
255
1646
3126
2510
3994 4072
1099
1885
278
2278
0
1000
2000
3000
4000
5000
6000
AGRICULTURE
ATMOSPHERE
BIOSPHERE
BIOLOGICAL CLASSIFICATION
CLIMATE INDICATORS
CRYOSPHERE
HUMAN DIMENSIONS
TERRESTRIAL HYDROSPHERE
LAND SURFACE
OCEANS
PALEOCLIMATE
SPECTRAL/ENGINEERING
SUN-EARTH INTERACTIONS
SOLID EARTH
“Searches” as a function of science “Topic” population in 2006.
“Searches” as a function of science “Topic” population in 2006.
Topic
DIF Population by Internal Directory Node*Total Through April 2007
7331
2744
664
205
1177
54 25
738
203 65 19293
4276
1361
192 126
0
1000
2000
3000
4000
5000
6000
7000
8000
NASANOAA
USDACIESIN
USGSCONAE
INPECCRS
ESA/ESRIN
DLRCNES
JAXAAMD
UN/UNEP
OBISGOMMP
Web Usage Metrics
Total Unique Users 1999 - 2007
0
100000
200000
300000
400000
500000
600000
un
iqu
e u
sers
num
erica
l.n
et
inte
rnat
iona
l
.com .e
du.g
ov .us
.org .m
il
0
1000000
2000000
3000000
4000000
5000000
6000000
7000000
8000000
Jan-
03
Apr-0
3
Jul-0
3
Oct-0
3
Jan-
04
Apr-0
4
Jul-0
4
Oct-0
4
Jan-
05
Apr-0
5
Jul-0
5
Oct-0
5
Jan-
06
Apr-0
6
Jul-0
6
Oct-0
6
Jan-
07
month
#h
its
Number of Web Page Hits Since January 2003
Cache opened to InternetSearch robots
Introduction of the new web page
Summer Declines
Problem with Google Cache
Search Metrics
Searches by Controlled Keyword
Agriculture9%
Atmosphere14%
Biosphere7%
Climate Indicators7%
Cryosphere4%
Human Dimensions7%
Hydrosphere6%
Land Surface13%
Oceans15%
Paleoclimate4%
Spectral/Engineering4%
Sun-Earth Interactions3%
Solid Earth7%
Agriculture Atmosphere Biosphere Climate Indicators Cryosphere
Human Dimensions Hydrosphere Land Surface Oceans Paleoclimate
Spectral/Engineering Sun-Earth Interactions Solid Earth
User Activity (Service Keywords Hits)August 2004 – Mar 2007
Several GIS conferences
Index by Google
User Support and Metadata
Authoring Tool Usage
User Support Questions Answered
0
5
10
15
20
25
30
35
40
45
Jan Feb Mar Apr May Jun Jul Aug Sep Oct Nov Dec
Months
Qu
es
tio
ns A
ns
we
red 2005
2006
2007
NASA PAO 2005
NASA PAO 2006
NASA PAO 2007
Global Change Calendar Entries
0
5
10
15
20
25
30
35
40
45
50
55
Jan-0
4
Apr-0
4Ju
l-04
Oct-04
Jan-0
5
Apr-0
5Ju
l-05
Oct-05
Jan-0
6
Apr-0
6Ju
l-06
Oct-06
Jan-0
7
Apr-0
7
Month
Nu
mb
er o
f E
ntr
ies
Deleted New Revised
HTML Metadata Authoring Tool Usage
0
200
400
600
800
1000
1200
Year
Num
ber
of
Entr
ies
GCMD Authors
Non GCMD Authors
Usage of the IDN Web Page (idn.ceos.org)
0
10000
20000
30000
40000
50000
60000
70000
80000
Sep-06 Oct-06 Nov-06 Dec-06 Jan-07 Feb-07 Mar-07 Apr-07
# H
its
New Features in MD 9.7 New Features in MD 9.7
DemoDemo
Highlights of new 9.7 web page webHighlights of new 9.7 web page web
New Topic Keyword
New graphical titleand color scheme
Two main focus areas “Find Data”and
“Find Data Services”easier to locate and distinguish on page
Full text search offered within Find Data search box
Full text services search(allows users to only view one search box on home page by default)
New area for latestGCMD featuresand highlights
Geographicrelated search
features groupedusing map icon
Map/Date Search; Combined Spatial, Temporal & Full-Text Search
Query Refinement Example
1. Query by Science Keyword: Biological Classification > Vertebrates > Mammals > Cetaceans
Query Refinement Example (cont’d)
2. Refine by Spatial Search: 3. Results
Instrument Description
Platform Description
Data Set Description
Data Service Description
New DIF Display
Direct Access to Data & Services (FTP, HTTP, OpenDAP, THREDDS,
WMS, WFS, WCS..etc)
New DIF Display
Direct access to data or images through multimedia sample
Multiple bounding coordinates
and/or points on Google map
Horizontal/vertical and temporal data resolution ranges in Brief Display
New DIF Display
FGDC Display
Note: The DIF has also been mapped to metadata standards such as ISO19115, Dublin Core, and ANZLIC.
Expanded database table structures for Instrument and Platform hierarchical keywords.
10 Platform Categories66 Platform Series/Entities
3 Instrument Categories25 Instrument_Classes29 Instrument_Types 22 Instrument_Subtypes
Instrument Ancillary Descriptions (AD-I)Instrument Ancillary Descriptions (AD-I)
Instrument_Identification:Instrument_CategoryInstrument_ClassInstrument_TypeInstrument_SubtypeShort_NameLong_Name
Instrument-Associated_Sensors:Short_Name
Associated_Platforms:Short_Name
Spectral/Frequency_Information:Wavelength_KeywordNumber_ChannelsSpectral/Frequency_Coverage/RangeSpectral/Frequency_Resolution
Description
Creation_Date
Revision_Date
Online_Resource
Sample_Image
IDN_Node
Instrument_Logistics:Data_RateInstrument_Start_DateInstrument_Stop_DateInstrument_Owner
Controlled keywords
Extended Ancillary descriptions to include expanded Instrument information.
Platform_Identification:Platform_CategoryPlatform_Series/EntityShort_NameLong_Name
Synonymous_Platform_Names:Short_Name
Platform-Associated_Instruments:Short_Name
Orbit:Orbit_AltitudeOrbit_InclinationEquator_Crossing PeriodRepeat_CyclePerigeeApogee Orbit_Type
Description
Creation_Date
Revision_Date
Online_Resource
Sample_Image
IDN_Node
Platform_Logistics:Launch_DateLaunch_SiteDesign_LifePrimary_Sponsor
Controlled keywords
Platform Ancillary Description (AD-P)
Extended Ancillary descriptions to include expanded Platform information.
Enhanced Platform & Instrument Descriptions
New/Modified Earth Science & Services Keywords Since September
2006
New/Modified Earth Science & Services Keywords Since September
2006
NEW Earth Science Keywords New Topic: Biological Classifications ( User-oriented + non-
authori taxonomy) Takes advantage of full range of new 5-level controlled
keyword hierarchy. Allows for full biological taxonomic classification using
ITIS, OBIS, GBIF, Species2000 and nomenclature.
Modified Topic name “Hydrosphere” to “Terrestrial Hydrosphere” and changed all underlying Terms and Variables.
Other science keyword changes pending for expanded 5-level hierarchy.
New daily RSS feed offered for new data set and data service descriptions
NEW Earth Science Keywords New Topic: Biological Classifications ( User-oriented + non-
authori taxonomy) Takes advantage of full range of new 5-level controlled
keyword hierarchy. Allows for full biological taxonomic classification using
ITIS, OBIS, GBIF, Species2000 and nomenclature.
Modified Topic name “Hydrosphere” to “Terrestrial Hydrosphere” and changed all underlying Terms and Variables.
Other science keyword changes pending for expanded 5-level hierarchy.
New daily RSS feed offered for new data set and data service descriptions
Node Reports and Feedback
Node Reports and Feedback
For Joint Committee on Antarctic Data Management (JCADM) /
Antarctic Master Directory (AMD) Node Report
Antarctic Master Directory (AMD)
Records: > 4300 DIFs(45% with direct data links)
Countries: 25
Hits: >14,000 in 2006(35% increase from 2005)
Retrievals: > 6,500 in 2006(20% increase from 2005)
Portals: 19 National Antarctic Data Center (NADC) portals
AMD Content Growth
Over 4,300 records as of April 2007
AMD Content Distribution by Country
Contributions from 25 countries (mainly Australia, United States, Argentina, New Zealand, Italy and Spain)
OAI - PMH Harvesting
IDN is harvesting new/updated records from the Australian Antarctic Data Center (AADC) since Feb. 07.
Highly customized HTML docBUILDER & portal. Records are sent to AADC for review/editing and harvested by IDN through OAI-PMH on a weekly basis.http://gcmd.nasa.gov/portals/amd_au/
AMD Hits vs. Retrievals - 2006
14, 000 Hits in 20066,500 Retrievals in 2006
AMD Total Hits since 2004
35 % increase in 2006 from 2005
AMD Total Retrievals since 2004
XXIX SCAR/JCADM Meeting
Launching of IPY 2007-2008(over 500 unique IPs)
New NADC Portals
http://gcmd.nasa.gov/portals/amd_kr/
Chinese Antarctic and Arctic Data Centre - Apr. 06Korea Polar Research Institute - Jun. 06
Australian Antarctic Data Centre - Nov. 06 The Netherlands Polar Programme - Feb. 07
IPY 2007/09Projects: > 200 from over 60 nations
The IDN is working with the IPY Data and Information Service (DIS) to host an IPY data and services portal. A prototype is currently underway.
The portal will provide links to data, services, projects and other links related to IPY. HTML docBUILDER will be customized to reflect the IPY profile.
Re-emergence of Interoperability The
Onslaught of “Standards”
Re-emergence of Interoperability The
Onslaught of “Standards”DIF evolution
Standards Proliferation vs Adopting DIF
Darwinism “seems” to have an effect on Metadata Standards
Inclusive adoption of metadata
Standards>Formats>Profiles>extensions>etc or Integrated Metadata Mining Technologies ?
DIF evolution
Standards Proliferation vs Adopting DIF
Darwinism “seems” to have an effect on Metadata Standards
Inclusive adoption of metadata
Standards>Formats>Profiles>extensions>etc or Integrated Metadata Mining Technologies ?
1987 1988 1989 1990 1991 1994 2004 2006+
BDP-CSDGM Interop
ISO 19115-Compliant
FGDC-CSDGM node
Global Change & Earth Sci. Appl
DIF Start-up
IDN-DIF | evolution and trends
DIF…
1987 → Over 100 DIF entries were available in the prototype NMD database.
1988 → After several demonstrations, workshops, and feedback from the scientific community, the Directory Interchange Format (DIF) was formally approved and adopted by a CI
science advisory group at a CI workshop in 1988.
1989 → The Committee on Earth Observation Satellites (CEOS) Data Working Group (DWG) began attending the CI Workshop meetings and provided valuable feedback on the DIF structure.
1990 → The Interagency Working Group on Data Management for Global Change (IWGDMGC) adopted the directory as a prototype to facilitate global change research - in response to the challenge by the Earth System Science Committee (ESSC).
1990 → The NMD was renamed the Global Change Master Directory (GCMD) for its Earth sciences applications.
1991 → The first release of the IDN was named the Prototype International Directory (PID) in 1990. [Actual DIF exchange procedures were agreed on by February 1991.
1994 → The GCMD serves as NASA's FGDC Clearinghouse node for geospatial metadata. Elements of the Content Standard for Digital Geospatial Metadata (CSDGM) were incorporated in the DIF in 1994.
2004 → The ISO 19115/TC211 geospatial metadata standard was adopted in GCMD.
2006+ → Planning guidelines to include BDP (taxonomic trees and geo-referencing) within GCMD .
DIF <evolution_adaptation_inclusiveness/>
Ecological Network (LTER) highlights the IDN functional development
“Advances in database and web-based technologies enabled traditional published data catalogs to be replaced by electronic and searchable data catalogs and data directories. Presently, many such databases that support data discovery are available through the web and provide numerous searching points (e.g. keyword, data, and location). Examples from U.S. include NASA’s Global Change Master Directory (IDN) and USGS-BRD. These databases provide either a controlled vocabulary that facilitates standardization of keywords and subsequent discovery.”
William K. Michener. 2006. Meta-information concepts for ecological data management. Ecological Informatics (1):3-7
Ecological Network (LTER) highlights the IDN functional development
“Advances in database and web-based technologies enabled traditional published data catalogs to be replaced by electronic and searchable data catalogs and data directories. Presently, many such databases that support data discovery are available through the web and provide numerous searching points (e.g. keyword, data, and location). Examples from U.S. include NASA’s Global Change Master Directory (IDN) and USGS-BRD. These databases provide either a controlled vocabulary that facilitates standardization of keywords and subsequent discovery.”
William K. Michener. 2006. Meta-information concepts for ecological data management. Ecological Informatics (1):3-7
ROSCOP/CSR
MARC
ANZLIC
DIF
EDMED
DCMI
DIDG
EML
FGDC/CSDGM
EDDF
ISO19115/NAP
1960 1970 1986 1988 1991 1994 1995 1997 1998 2001 2003 2007+
Some World Standards History and Usage/Interoperability : A “Bottleneck effect” for knowledge-based communities
1960: ROSCOP/CSR (Cruise Summary Report) 1970: MARC (Machine-Readable Cataloging) 1986: Australia New Zealand Land Information Council (ANZLIC) 1988: Directory Interchange Format (DIF) 1991: European Directory of Marine Environmental Datasets (EDMED) 1994: Dublin Core Metadata Initiative (DCMI) 1994: US Federal Geographic Data Committee (FGDC) Content Standard for Digital Geospatial Metadata
(CSDGM) 1995: Directory Information Describing Geo-referenced Datasets (DIDG) 2001: NOAA/NODC Electronic Data Description Format (EDDF) 2003: ISO-19115 Geographic Information Metadata International Standard > ISO North American Profile
(NAP)
Functionality / Complexity
Cost of adoption
Adapted from: Arms, et al., 2002
Adoption std. cost (Interop.) vs. Functionality/ Complexity• Adoption of a common standard
1- low cost of adoption with low functionality 2 - higher functionality but with a greater cost of adoption
(1 and 2) No best point on the curve – every point is optimal for some purpose
3 - high cost of adoption (no datasets completeness) with low functionality (creation + dissemination + usage)4- low cost of adoption with higher functionality
1
2 DIF3
4
Evidence of Darwinian effects on Metadata Standards• Evolution as is. Metadata std. are not constant, recently created
nor perpetually cycling, but rather is steadily changing, and that standards are transformed in time.
• Multiplication of standards. This explains the origin of enormous standards diversity. It postulates that standards multiply, either by splitting into profiles or by “extending“ in order to evolve into new standards.
• Common XML descent . This is the theory that every group of Metadata descended from a common XML ancestor, and that all groups of standards, including profiles, formats, and extensions, ultimately go back to a plain single origin due to quick and broad usage.
• Natural selection. evolutionary change comes about through the abundant production of metadata records in every generation. The relatively few standards who survive, owing to a particularly well-adapted combination of inheritable characters, adaptability, data discovery, ingestion technology, comprehension, processing and analysis coupled with scientific workflow systems will give rise to the next generation “The use-level metadata”
Inclusive adoption of endless metadata Standards>Formats>Profiles>extensions
>etc or Integrated Metadata Mining Technologies ?
Inclusive adoption of endless metadata Standards>Formats>Profiles>extensions
>etc or Integrated Metadata Mining Technologies ?
Andes Architecture, Jussi Myllymaki, http://www10.org/cdrom/papers/102/index.html
What about Content ?... IDN makes a difference
Map Server highlighting Vietnam
Map Server highlighting Vietnam
GCMD MapserverProvides access and visualization to selected NASA geospatial data sets.
Created using ArcIMS 9.1.
Direct access to data using the OGC Web Map Service (WMS) standard.
The Future IDNThe Future IDN
MD9.8 and MD10MD9.8 and MD10
Version 9.8Version 9.8Migrate to MySQLOffer Ancillary Use Level Metadata with
the DIF.Implement next stage of interoperability
with ECHO’s Event Notification Service.Support 5-level service taxonomy.Create a usability taxonomy for Data
Handling.Preserve small data sets “at risk” and
offer when possible through Google Earth.
Migrate to MySQLOffer Ancillary Use Level Metadata with
the DIF.Implement next stage of interoperability
with ECHO’s Event Notification Service.Support 5-level service taxonomy.Create a usability taxonomy for Data
Handling.Preserve small data sets “at risk” and
offer when possible through Google Earth.
Version 10Version 10
Represent the GCMD vocabulary in SKOS, including multilingual capabilities.
Empower data providers to manage their metadata (load, query, share, etc.) through a distributed capability at the IDN.
Re-engineer server to support rising levels of data sets and increasing demands for IDN services.
Represent the GCMD vocabulary in SKOS, including multilingual capabilities.
Empower data providers to manage their metadata (load, query, share, etc.) through a distributed capability at the IDN.
Re-engineer server to support rising levels of data sets and increasing demands for IDN services.
Innovations of Interoperability through the
IDN
Innovations of Interoperability through the
IDN
IDN-NOAA Meta-Harvesting ←IDN and MIRADOR GSFC-DAAC ←
IDN - ECHO ←
JCADM -AMD ←
IDN-NOAA Meta-Harvesting ←IDN and MIRADOR GSFC-DAAC ←
IDN - ECHO ←
JCADM -AMD ←
IDN-NOAA Metadata Harvesting
E-mail & SGML
(2000-2004)
Rsync & XML
(2004-2007)
OAI-PMH & XML
(2008 - Beyond)
IDN and MIRADOR
1. IDN 3. MIRADOR
2. GET DATA
Discover Data using the IDN and Order Granules via MIRADORDiscover Data using the IDN and Order Granules via MIRADOR
4. Get Granules
ID is automatically passed into Mirador
via MIRADOR link
ECHO Customized Portal
gcmd.nasa.gov/Data/portal_index.html
Select Metadata
IDNECHO
WIST
Select Granule
ECHO Portal at IDN
Search ECHO Portal
Send Query
Search ECHO
IDNIDN
ECHO
Met
adat
a re
cord
to
EC
HO
Send metadata to
GCMD
Translate
Metadata
Using XSLT
Ret
urn
tran
slat
ed r
ecor
d (G
ES
DIS
C D
AA
C)
Current InteroperabilityC
Write metadata record
Future Interoperability
ECHO
Data to ECHO
IDN / ECHO
Portal
Insert Delete Update
Event Notification Service Triggered
Add Entry to Portal Delete Entry from Portal
Thank You
[email protected] TT Task Leader
[email protected] Informatics
Coordinator
Thank You
[email protected] TT Task Leader
[email protected] Informatics
Coordinator
