Data access to distributed databases A case study from oceanography Taco de Bruin
description
Transcript of Data access to distributed databases A case study from oceanography Taco de Bruin
![Page 1: Data access to distributed databases A case study from oceanography Taco de Bruin](https://reader034.fdocuments.us/reader034/viewer/2022051401/568143fc550346895db08f3c/html5/thumbnails/1.jpg)
1NIOZ is part of the Netherlands Organisation for Scientific Research (NWO)
Koninklijk Nederlands Instituut voor ZeeonderzoekRoyal Netherlands Institute for Sea Research
Data access to distributed databases
A case study from oceanography
Taco de Bruin
SCADM/SCAGI meetings
Amsterdam, NL, Sep. 7-9, 2009
![Page 2: Data access to distributed databases A case study from oceanography Taco de Bruin](https://reader034.fdocuments.us/reader034/viewer/2022051401/568143fc550346895db08f3c/html5/thumbnails/2.jpg)
2NIOZ is part of the Netherlands Organisation for Scientific Research (NWO)
Koninklijk Nederlands Instituut voor ZeeonderzoekRoyal Netherlands Institute for Sea Research
Goal of all our data management activities is to provide the user access to the data holdings
“Collect once, use many times”
![Page 3: Data access to distributed databases A case study from oceanography Taco de Bruin](https://reader034.fdocuments.us/reader034/viewer/2022051401/568143fc550346895db08f3c/html5/thumbnails/3.jpg)
3NIOZ is part of the Netherlands Organisation for Scientific Research (NWO)
Koninklijk Nederlands Instituut voor ZeeonderzoekRoyal Netherlands Institute for Sea Research
Situation of data exchange within NL-NODC a couple of years ago:
Online accessibility and availability of data
• Waterbase (RIKZ)• CODIS (NIOZ)• KODAC (KNMI)• Watermarkt (RIKZ, many other institutes)• HYDRA (KNMI)• COREBASE (NIOZ)• DINO (TNO- B&O)• Getij.nl (RIKZ)• Waterstat.nl (RIKZ)
![Page 4: Data access to distributed databases A case study from oceanography Taco de Bruin](https://reader034.fdocuments.us/reader034/viewer/2022051401/568143fc550346895db08f3c/html5/thumbnails/4.jpg)
4NIOZ is part of the Netherlands Organisation for Scientific Research (NWO)
Koninklijk Nederlands Instituut voor ZeeonderzoekRoyal Netherlands Institute for Sea Research
Situation of data exchange within NL-NODC a couple of years ago:
Online accessibility and availability of data
• Waterbase (RIKZ)• CODIS (NIOZ)• KODAC (KNMI)• Watermarkt (RIKZ, many other institutes)• HYDRA (KNMI)• COREBASE (NIOZ)• DINO (TNO- B&O)• Getij.nl (RIKZ)• Waterstat.nl (RIKZ)
MEDIN Meeting 20090630
![Page 5: Data access to distributed databases A case study from oceanography Taco de Bruin](https://reader034.fdocuments.us/reader034/viewer/2022051401/568143fc550346895db08f3c/html5/thumbnails/5.jpg)
5NIOZ is part of the Netherlands Organisation for Scientific Research (NWO)
Koninklijk Nederlands Instituut voor ZeeonderzoekRoyal Netherlands Institute for Sea Research
Situation of data exchange within NL-NODC a couple of years ago:
Online accessibility and availability of data
• Waterbase (RIKZ)• CODIS (NIOZ)• KODAC (KNMI)• Watermarkt (RIKZ + many partner institutes)• many more …..
- user had no overview of data availability
- user had to do all format conversions and syntheses
- user had to spend at least 50% of time on finding data and data conversion, instead of doing science
![Page 6: Data access to distributed databases A case study from oceanography Taco de Bruin](https://reader034.fdocuments.us/reader034/viewer/2022051401/568143fc550346895db08f3c/html5/thumbnails/6.jpg)
6NIOZ is part of the Netherlands Organisation for Scientific Research (NWO)
Koninklijk Nederlands Instituut voor ZeeonderzoekRoyal Netherlands Institute for Sea Research
SeaDataNet:
A Pan-European distributed data access infrastructure
http://www.seadatanet.org/data_access
NODC-i:
A national distributed data access infrastructure, based on SeaDataNet technology
http://www.nodc.nl – Data Access Services
![Page 7: Data access to distributed databases A case study from oceanography Taco de Bruin](https://reader034.fdocuments.us/reader034/viewer/2022051401/568143fc550346895db08f3c/html5/thumbnails/7.jpg)
7NIOZ is part of the Netherlands Organisation for Scientific Research (NWO)
Koninklijk Nederlands Instituut voor ZeeonderzoekRoyal Netherlands Institute for Sea Research
Total 49 partners, links with JCOMM, WDC, DMAC, MMI, EDIOS, EU-MarBEF (NoE), EU-Seased, ESEAS, Esonet, EuroGeoSurveys, POGO, etc.
SeaDataNet, consortium of national oceanographic data centres:
35 countries
IOC/IODE
ICES
JRC
![Page 8: Data access to distributed databases A case study from oceanography Taco de Bruin](https://reader034.fdocuments.us/reader034/viewer/2022051401/568143fc550346895db08f3c/html5/thumbnails/8.jpg)
8NIOZ is part of the Netherlands Organisation for Scientific Research (NWO)
Koninklijk Nederlands Instituut voor ZeeonderzoekRoyal Netherlands Institute for Sea Research
SeaDataNet System
SeaDataNetPortal
Data centreData centreData centreData centre
Search&
shop
Metadata + transaction data
![Page 9: Data access to distributed databases A case study from oceanography Taco de Bruin](https://reader034.fdocuments.us/reader034/viewer/2022051401/568143fc550346895db08f3c/html5/thumbnails/9.jpg)
9NIOZ is part of the Netherlands Organisation for Scientific Research (NWO)
Koninklijk Nederlands Instituut voor ZeeonderzoekRoyal Netherlands Institute for Sea Research
• CDI is a fine-grained index to individual data measurements (such as a CTD cast or moored instrument record).
• CDI: ‘common’ metadata of individual measurements (location, time, parameter, availability, owner, etc.)
Common Data Index (CDI)
CDI records from >29 countries
![Page 10: Data access to distributed databases A case study from oceanography Taco de Bruin](https://reader034.fdocuments.us/reader034/viewer/2022051401/568143fc550346895db08f3c/html5/thumbnails/10.jpg)
10NIOZ is part of the Netherlands Organisation for Scientific Research (NWO)
Koninklijk Nederlands Instituut voor ZeeonderzoekRoyal Netherlands Institute for Sea Research
SeaDataNet system
CDI
User interface
DownloadManager
datadata
data
Shopping basket
Request Status Manager
DownloadManager
DownloadManager
Central User
Register
![Page 11: Data access to distributed databases A case study from oceanography Taco de Bruin](https://reader034.fdocuments.us/reader034/viewer/2022051401/568143fc550346895db08f3c/html5/thumbnails/11.jpg)
11NIOZ is part of the Netherlands Organisation for Scientific Research (NWO)
Koninklijk Nederlands Instituut voor ZeeonderzoekRoyal Netherlands Institute for Sea Research
CDI Search – home page
![Page 12: Data access to distributed databases A case study from oceanography Taco de Bruin](https://reader034.fdocuments.us/reader034/viewer/2022051401/568143fc550346895db08f3c/html5/thumbnails/12.jpg)
12NIOZ is part of the Netherlands Organisation for Scientific Research (NWO)
Koninklijk Nederlands Instituut voor ZeeonderzoekRoyal Netherlands Institute for Sea Research
Search – records selected for shopping basket
![Page 13: Data access to distributed databases A case study from oceanography Taco de Bruin](https://reader034.fdocuments.us/reader034/viewer/2022051401/568143fc550346895db08f3c/html5/thumbnails/13.jpg)
13NIOZ is part of the Netherlands Organisation for Scientific Research (NWO)
Koninklijk Nederlands Instituut voor ZeeonderzoekRoyal Netherlands Institute for Sea Research
Contents of shopping basket
![Page 14: Data access to distributed databases A case study from oceanography Taco de Bruin](https://reader034.fdocuments.us/reader034/viewer/2022051401/568143fc550346895db08f3c/html5/thumbnails/14.jpg)
14NIOZ is part of the Netherlands Organisation for Scientific Research (NWO)
Koninklijk Nederlands Instituut voor ZeeonderzoekRoyal Netherlands Institute for Sea Research
Submit Shopping list => User Authentication
![Page 15: Data access to distributed databases A case study from oceanography Taco de Bruin](https://reader034.fdocuments.us/reader034/viewer/2022051401/568143fc550346895db08f3c/html5/thumbnails/15.jpg)
15NIOZ is part of the Netherlands Organisation for Scientific Research (NWO)
Koninklijk Nederlands Instituut voor ZeeonderzoekRoyal Netherlands Institute for Sea Research
Authentication, Authorization and Administration
- Single Sign On system set up for access to distributed system
- User’s authentication information based on personal login / password
- User must register to get one login, thereby agreeing with the “SeaDataNet User Licence”
- Central User Register, but management is decentralized
- Authorisation based on “User Roles”
![Page 16: Data access to distributed databases A case study from oceanography Taco de Bruin](https://reader034.fdocuments.us/reader034/viewer/2022051401/568143fc550346895db08f3c/html5/thumbnails/16.jpg)
16NIOZ is part of the Netherlands Organisation for Scientific Research (NWO)
Koninklijk Nederlands Instituut voor ZeeonderzoekRoyal Netherlands Institute for Sea Research
Data access restrictions in CDI V1
CC Collection cost charge
FE Commercial charge
LI Licence
MO Moratorium
NA No access
NC Distribution cost
OG Organisation
RS Restricted
SR Academic
UK Unknown
UN Unrestricted
See Vocabulary L081 for definitions Multiple restrictions may apply
![Page 17: Data access to distributed databases A case study from oceanography Taco de Bruin](https://reader034.fdocuments.us/reader034/viewer/2022051401/568143fc550346895db08f3c/html5/thumbnails/17.jpg)
17NIOZ is part of the Netherlands Organisation for Scientific Research (NWO)
Koninklijk Nederlands Instituut voor ZeeonderzoekRoyal Netherlands Institute for Sea Research
User Roles in AAA Register, of relevance for data access
SDRN02 Public
SDRN03 Academic
SDRN04 Commercial
SDRN05 National and local government
SDRN06 Pan-National government
See Vocabulary C866 for definitions
Multiple roles may apply
![Page 18: Data access to distributed databases A case study from oceanography Taco de Bruin](https://reader034.fdocuments.us/reader034/viewer/2022051401/568143fc550346895db08f3c/html5/thumbnails/18.jpg)
18NIOZ is part of the Netherlands Organisation for Scientific Research (NWO)
Koninklijk Nederlands Instituut voor ZeeonderzoekRoyal Netherlands Institute for Sea Research
Business Matrix
W
V
U
DCBA Restriction
Role
OK
OK
OK
TBDOK
OKTBD
OK
OK
NO
TBDNO
![Page 19: Data access to distributed databases A case study from oceanography Taco de Bruin](https://reader034.fdocuments.us/reader034/viewer/2022051401/568143fc550346895db08f3c/html5/thumbnails/19.jpg)
19NIOZ is part of the Netherlands Organisation for Scientific Research (NWO)
Koninklijk Nederlands Instituut voor ZeeonderzoekRoyal Netherlands Institute for Sea Research
Request confirmed and link to RSM (also by e-mail)
![Page 20: Data access to distributed databases A case study from oceanography Taco de Bruin](https://reader034.fdocuments.us/reader034/viewer/2022051401/568143fc550346895db08f3c/html5/thumbnails/20.jpg)
20NIOZ is part of the Netherlands Organisation for Scientific Research (NWO)
Koninklijk Nederlands Instituut voor ZeeonderzoekRoyal Netherlands Institute for Sea Research
User RSM – Overview of request per Data Centre and Status
![Page 21: Data access to distributed databases A case study from oceanography Taco de Bruin](https://reader034.fdocuments.us/reader034/viewer/2022051401/568143fc550346895db08f3c/html5/thumbnails/21.jpg)
21NIOZ is part of the Netherlands Organisation for Scientific Research (NWO)
Koninklijk Nederlands Instituut voor ZeeonderzoekRoyal Netherlands Institute for Sea Research
Downloading data from SeaDataNet infrastructure:
Standard formats (depending on data type):
- ODV ASCII
- NetCDF
- MedAtlas
Agreed QC flags
![Page 22: Data access to distributed databases A case study from oceanography Taco de Bruin](https://reader034.fdocuments.us/reader034/viewer/2022051401/568143fc550346895db08f3c/html5/thumbnails/22.jpg)
22NIOZ is part of the Netherlands Organisation for Scientific Research (NWO)
Koninklijk Nederlands Instituut voor ZeeonderzoekRoyal Netherlands Institute for Sea Research
SeaDataNet is a working, distributed data delivery
infrastructure, featuring all elements required by SCADM,
SCAR or the Southern Ocean Observing System (SOOS).
The big challenge will be to couple the European SeaDataNet
infrastructure to the emerging Australian and American
systems and prevent these developments from diverging.
The International Oceanographic Data and Information
Exchange Committee (IODE) is working on achieving this
with its Ocean Data Portal project.