EGI DataHub Documentation

EGI DataHub Documentation

Baptiste Grenier <[email protected]>

Oct 15, 2020

Contents:

1 Introduction 31.1 Motivation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 31.2 Components and concepts . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 31.3 Highlighted features . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4

2 Clients 92.1 Using the web interface . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 92.2 Generating tokens for using Oneclient or APIs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 92.3 Installing and testing Oneclient in a docker container . . . . . . . . . . . . . . . . . . . . . . . . . . 92.4 Testing Oneclient in a Oneclient docker container with NFS or samba . . . . . . . . . . . . . . . . . 152.5 Testing Oneclient in a Oneclient docker container with local file access . . . . . . . . . . . . . . . . 162.6 Testing Oneclient in a Virtual Machine . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 162.7 Testing Oneclient in a Vagrant box . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17

3 Providers 193.1 Transparent data access . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 203.2 Federation of service providers . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 213.3 Smart caching . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 223.4 Publication of datasets . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 233.5 Integrating DataHub and EGI Notebooks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 243.6 Steps to use DataHub and Onedata . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 243.7 Deploying Onedata . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 253.8 Requirements for production . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 25

4 API 274.1 Getting an API access token . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 274.2 Testing the API with the REST client . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 284.3 Printing the raw REST calls of a wrapped command . . . . . . . . . . . . . . . . . . . . . . . . . . 284.4 Working with PID / Handle . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 28

5 Links 31

i


Note: This documentation is obsolete and was replaced by https://docs.egi.eu/users/datahub/ and https://docs.egi.eu/providers/datahub/.

Contents: 1

https://docs.egi.eu/users/datahub/

https://docs.egi.eu/providers/datahub/

https://docs.egi.eu/providers/datahub/


2 Contents:

CHAPTER 1

Introduction

1.1 Motivation

• Putting up a (scalable) distributed data infrastructure needs specific expertise, resources and knowledge

• No easy way to discover and transfer data

• No easy way of making data (publicly) accessible without transferring it a sharing service

• No easy way of combining multiple datasets from different data providers

• Users need to access data locally and from compute resources

1.2 Components and concepts

Space a virtual volume where users will organize their data. A space is supported by one or multipleOneproviders providing actual storage resources

EGI DataHub a Onedata Onezone, the federation and authentication service. Single Sign On (SSO)with all the connected storage providers (Oneprovider) through EGI Check-in

Onezone a central component federating providers, it will take care of Authentication and Authorizationand other management tasks (like space creation). EGI DataHub is a Onezone instance.

Oneprovider a data management component deployed in the data centres, provisioning data and man-aging transfers. A Oneprovider is typically deployed at a site near the local storage resources, andcan access local storage resources over multiple connectors (CEPH, POSIX,. . . ). A default one isoperated for EGI by CYFRONET.

Oneclient a client application providing access to the spaces through a FUSE mount point (local POSIXaccess). Spaces are accessible as if they were part of the local file system. Oneclient can be usedfrom VM, containers, desktop,. . .

Web interfaces and APIs are also available

3


1.3 Highlighted features

Fig. 1: Viewing a data space using the EGI DataHub web interfaceUsing the EGI DataHub web interface it’s possible to manage the space.

4 Chapter 1. Introduction


Fig. 2: Viewing a data space in a console locally mounted using OneclientUsing Oneclient it’s possible to mount a space locally, and access it over a POSIX interface, using files as they were stored locally.

The file’s blocks are downloaded on demand.

1.3. Highlighted features 5


Fig. 3: Viewing file distribution over the OneprovidersIn Onedata the file distribution is done on a block basis, blocks will be replicated on the fly, and it’s possible to instrument the

replication.

Fig. 4: Management of metadata using the web interfaceThree different formats of metadata can be attached to files: basic (key/value), JSON and RDF. The metadata can be managed

using the Web interface and the APIs. It’s also possible to create indexes and query them.



Fig. 5: Viewing file popularity for smart cachingIt’s possible to view the popularity of a file and manage smart caching.

1.3. Highlighted features 7

CHAPTER 2

Clients

The Oneclient code and basic documentation are available on GitHub.

The official documentation is hosted on the Onedata homepage.

2.1 Using the web interface

2.2 Generating tokens for using Oneclient or APIs

Important: In order to be able to access your spaces using Oneclient or APIs, it’s required to generate an accesstoken.

Tokens have to be generated from the EGI DataHub (Onezone) interface.

2.3 Installing and testing Oneclient in a docker container

A quick and simple solution for testing is to install the client on demand in a container for a supported OperatingSystem flavor (mainly various CentOS and Ubuntu releases).

The following variables have to be exported in the container:

• ONECLIENT_ACCESS_TOKEN: access token allowing to access all the spaces.

• ONECLIENT_PROVIDER_HOST: name or IP of the Oneprovider the client should connect to.

Important: In order to be able to use FUSE, the container should run in privileged mode.

9

https://github.com/onedata/oneclient

https://onedata.org/#/home/documentation/doc/using_onedata/oneclient.html


Fig. 1: Selecting EGI to connect using EGI Check-inUsing EGI Check-in it’s possible to connect with your institute credentials.

10 Chapter 2. Clients


Fig. 2: EGI DataHub landing pageOn this page it’s possible to have an overview of all the spaces and their supporting providers.

2.3. Installing and testing Oneclient in a docker container 11


Fig. 3: Information about spaces supported by a OneproviderOn this capture, the information about the spaces supported by a specific provider is displayed.



Fig. 4: Information about spaces supported by a OneproviderThe data space can be managed (ie. uploading/downloading/managing files and metadata, managing space access) using the web

browser.

2.3. Installing and testing Oneclient in a docker container 13


Fig. 5: Managing access tokens from EGI DataHubThe access tokens can be created and managed using the EGI DataHub web interface.



docker run -it --privileged centos:7 /bin/bashroot@81dbd7e84438 /]# curl -sS http://get.onedata.org/oneclient-1902.sh | bash# (...)Complete!Installation has been completed successfully.Run 'oneclient --help' for usage info.root@81dbd7e84438 /]# export ONECLIENT_ACCESS_TOKEN=<ACCESS_TOKEN_FROM_ONEZONE>root@81dbd7e84438 /]# export ONECLIENT_PROVIDER_HOST=plg-cyfronet-01.datahub.egi.euroot@81dbd7e84438 /]# mkdir /tmp/spaceroot@81dbd7e84438 /]# oneclient /tmp/spaceroot@81dbd7e84438 /]# ls /tmp/space

Here the data is mounted in /tmp/space, creating a file into it will push it to the Oneprovider and it will be accessiblein the web interface and from other providers supporting the space.

For a real production usage it’s preferable to use the Oneclient container as a source for a volume mounted into anothercontainer.

2.4 Testing Oneclient in a Oneclient docker container with NFS orsamba

Docker containers for the Oneclient are available, the existing versions can be seen on the Oneclient docker hub.

It’s possible to use the most recent version by specifying the latest tag. We also recommend using the same versionas shown on the Onezone and Oneprovider pages.

The following variables have to be exported to be used in the container:




export ONECLIENT_ACCESS_TOKEN=<ACCESS_TOKEN_FROM_ONEZONE>export ONECLIENT_PROVIDER_HOST=plg-cyfronet-01.datahub.egi.eudocker run -it --privileged -e ONECLIENT_ACCESS_TOKEN=$ONECLIENT_ACCESS_TOKEN -e→˓ONECLIENT_PROVIDER_HOST=$ONECLIENT_PROVIDER_HOST onedata/oneclient:19.02.0-rc2Connecting to provider 'plg-cyfronet-01.datahub.egi.eu:443' using session ID:→˓'4138963898952098752'...Getting configuration...Oneclient has been successfully mounted in '/mnt/oneclient'

Now the client will run in the background and the data will be available through samba/CIFS or nfs protocols:

# Identifying the IP of the containerdocker inspect --format "{{ .NetworkSettings.IPAddress }}" $(docker ps -ql)172.17.0.2

So the data can be accessed at

• smb://172.17.0.2/onedata

• nfs://172.17.0.2/onedata

2.4. Testing Oneclient in a Oneclient docker container with NFS or samba 15

https://hub.docker.com/r/onedata/oneclient/tags


2.5 Testing Oneclient in a Oneclient docker container with local fileaccess

Another solution is to mount a local directory as a volume in the container, allowing to access both the workingdirectory as well as the Onedata spaces, thus allowing to easily exchange files between a local directory and a Onedataspace.

In order to do this we will open a bash shell in the container then we will mount manually the Onedata spaces.




export ONECLIENT_ACCESS_TOKEN=<ACCESS_TOKEN_FROM_ONEZONE>export ONECLIENT_PROVIDER_HOST=plg-cyfronet-01.datahub.egi.eudocker run -it --privileged -e ONECLIENT_ACCESS_TOKEN=$ONECLIENT_ACCESS_TOKEN -e→˓ONECLIENT_PROVIDER_HOST=$ONECLIENT_PROVIDER_HOST -v $PWD:/mnt/src --entrypoint bash→˓onedata/oneclient:19.02.0-rc2root@aca612a84fb4:/tmp# oneclient /mnt/oneclientConnecting to provider 'plg-cyfronet-01.datahub.egi.eu:443' using session ID:→˓'1641165171427694510'...Getting configuration...Oneclient has been successfully mounted in '/mnt/oneclient'.root@aca612a84fb4:/tmp# ls /mnt/oneclient(...)root@aca612a84fb4:/tmp# ls /mnt/src(...)

Now it’s possible to use the following mount points:

• /mnt/oneclient: the Onedata spaces

• /mnt/src: the local directory (any absolute path could have been used instead of $PWD that points to theworking directory)

2.6 Testing Oneclient in a Virtual Machine

The following variables have to be exported:



curl -sS http://get.onedata.org/oneclient-1902.sh | bashexport ONECLIENT_ACCESS_TOKEN=<ACCESS_TOKEN_FROM_ONEZONE>export ONECLIENT_PROVIDER_HOST=plg-cyfronet-01.datahub.egi.eumkdir /tmp/spaceoneclient /tmp/space



2.7 Testing Oneclient in a Vagrant box

It’s possible to quickly test Oneclient using Vagrant.

The following variables have to be exported:



vagrant init ubuntu/xenial64vagrant upvagrant sshcurl -sS http://get.onedata.org/oneclient-1902.sh | bashexport ONECLIENT_ACCESS_TOKEN=<ACCESS_TOKEN_FROM_ONEZONE>export ONECLIENT_PROVIDER_HOST=plg-cyfronet-01.datahub.egi.eumkdir /tmp/spaceoneclient /tmp/space

2.7. Testing Oneclient in a Vagrant box 17

https://www.vagrantup.com/

CHAPTER 3

Providers

Usage and deployment may vary depending on the use case.

19


3.1 Transparent data access

• Clients use one ore more providers to access data

• Data can be accessed over multiple protocols

20 Chapter 3. Providers


3.2 Federation of service providers

• Heterogeneous backend storage

• Common interfaces (Web, REST, POSIX, CDMI)

• Common AAI with Check-in

• Discovery of Datasets in the EGI DataHub

3.2. Federation of service providers 21


3.3 Smart caching

• Site A hosts data and computing resources

• Site B hosts only data

• Site X uses data from A and B without pre-staging

• Pre-staging can also be done using APIs

• Data is accessed locally “à la” POSIX with FUSE



3.4 Publication of datasets

• PID minting

• Publishing, discovery and access to datasets

3.4. Publication of datasets 23


3.5 Integrating DataHub and EGI Notebooks

3.6 Steps to use DataHub and Onedata

• Collecting and analysing dataset specificities

– Number of files

– Size of files

• Preparing a pilot

– Designing and validating usage model

– Integrating Onedata with existing resources

• Validating the pilot

• Deploying a production setup

– Ensuring hardware requirements are sufficient

* RAM, CPU, Disk, Network,. . .

* Storage backend



3.7 Deploying Onedata

Preferred model: using docker containers, with docker-compose.

Packages for Ubuntu 16.04 and CentOS 7 are also available.

3.8 Requirements for production

• Oneprovider

– RAM: 32GB

– CPU: 8 vCPU

– Disk: 50GB SSD

– To be adjusted for the dataset and usage scenario

• For high IOPS

– High performance backend storage (CEPH)

– Low latency network

• POSIX mounting

3.7. Deploying Onedata 25


– Oneprovider close to the Oneclient


CHAPTER 4

API

Most if not all operations can be performed using the Onedata API.

The official documentation is at https://onedata.org/#/home/api.

Important: In order to be able to access the Onedata APIs, an access token is required. See below for instructionson how to generate one.

4.1 Getting an API access token

Tokens have to be generated from the EGI DataHub (Onezone) interface as documented in Generating tokens forusing Oneclient or APIs or using a command line call as documented hereafter.

Bear in mind that a single API token can be used with both Onezone, Oneprovider and other Onedata APIs.

It’s possible to retrieve the CLIENT_ID, CLIENT_SECRET and REFRESH_TOKEN using a special OIDC clientconnected to Check-in. See Check-in documentation on EGI Wiki for more information.

CLIENT_ID=<CLIENT_ID>CLIENT_SECRET=<CLIENT_SECRET>REFRESH_TOKEN=<REFRESH_TOKEN># Retrieving an OIDC token from Check-incurl -X POST -u "$CLIENT_ID":"$CLIENT_SECRET" \

-d "client_id=$CLIENT_ID&$CLIENT_SECRET&grant_type=refresh_token&refresh_token=→˓$REDRESH_TOKEN&scope=openid%20email%20profile" \

'https://aai.egi.eu/oidc/token' | python -m json.tool;# Token is in the `access_token` field of the response

The following variables should be set:

• OIDC_TOKEN: OpenID Connect Access token.

• ONEZONE_HOST: name or IP of the Onezone host (to use Onezone API).

27

https://onedata.org/#/home/api

https://aai.egi.eu/fedcloud/

https://aai.egi.eu/fedcloud/

https://wiki.egi.eu/wiki/AAI


• ONEPROVIDER_HOST: name or IP of the Oneprovider host (to use Oneprovider API).

ONEZONE_HOST=https://datahub.egi.euOIDC_TOKEN=<OIDC_ACCESS_TOKEN>curl -H "X-Auth-Token: egi:$OIDC_TOKEN" -X POST \

-H 'Content-type: application/json' -d '{}' \"$ONEZONE_HOST/api/v3/onezone/user/client_tokens"

4.2 Testing the API with the REST client

A docker container with clients acting as wrappers around the API calls is available: onedata/rest-cli. It’s veryconvenient for discovering and testing the Onezone and Oneprovider API.

docker run -it onedata/rest-cli# Exporting env for Onezone APIexport ONEZONE_HOST=https://datahub.egi.euexport ONEZONE_API_KEY=<ACCESS_TOKEN># Checking current useronezone-rest-cli getCurrentUSer | jq '.'# Listing all accessible spacesonezone-rest-cli listEffectiveUserSpaces | jq '.'

docker run -it onedata/rest-cli# Exporting env for Oneprovider APIexport ONEPROVIDER_HOST=https://plg-cyfronet-01.datahub.egi.euexport ONEPROVIDER_API_KEY=<ACCESS_TOKEN># Listing all spaces supported by the Oneprovideroneprovider-rest-cli getAllSpaces | jq '.'# Listing content of a spaceoneprovider-rest-cli listFiles path='EGI Foundation/'oneprovider-rest-cli listFiles path='EGI Foundation/CS3_dataset'

4.3 Printing the raw REST calls of a wrapped command

Raw REST calls (used with curl) can be printed using the --dry-run switch.

docker run -it onedata/rest-cliexport ONEZONE_HOST=https://datahub.egi.euexport ONEZONE_API_KEY=<ACCESS_TOKEN># Listing all accessible spacesonezone-rest-cli listEffectiveUserSpaces | jq '.'# Printing the curl command without running itonezone-rest-cli listEffectiveUserSpaces --dry-run

4.4 Working with PID / Handle

It’s possible to mint a Permanent Identifier (PID) for a space or a subdirectory of a space using a handle service (likeHandle.net) that is registered in the Onezone (EGI DataHub).

Once done, accessing the PID using its URL will redirect to the Onedata share allowing to retrieve the files.

28 Chapter 4. API


Prerequisites: access to a Handle service registered in the Onezone. See the Handle Service API documentation fordocumentation on registering a new Handle service or ask a Onezone administrator to authorize you to use an existingHandle service already registered in the Onezone.

The following variables should be set:

• API_ACCESS_TOKEN: Onedata API access token

• ONEZONE_HOST: name or IP of the Onezone host (to use Onezone API).

• ONEPROVIDER_HOST: name or IP of the Oneprovider host (to use Oneprovider API)

# Getting the IDs of the available Handle Servicescurl -sS --tlsv1.2 -H "X-Auth-Token: $API_ACCESS_TOKEN" \

"$ONEZONE_HOST/api/v3/onezone/user/handle_services"HANDLE_SERVICE=<HANDLE_SERVICE_ID>

# Getting details about a specific Handle servicecurl -sS --tlsv1.2 -H "X-Auth-Token: $API_ACCESS_TOKEN" \

"$ONEZONE_HOST/api/v3/onezone/user/handle_services/$HANDLE_SERVICE"

# Listing all spacescurl -sS --tlsv1.2 -H "X-Auth-Token: $API_ACCESS_TOKEN" \

"$ONEZONE_HOST/api/v3/onezone/user/effective_spaces/" | jq '.'

# Displaying details of a spacecurl -sS --tlsv1.2 -H "X-Auth-Token: $API_ACCESS_TOKEN" \

"$ONEZONE_HOST/api/v3/onezone/spaces/$SPACE_ID" | jq '.'

# Listing content of a spacecurl -sS --tlsv1.2 -H "X-Auth-Token: $API_ACCESS_TOKEN" \

"$ONEPROVIDER_HOST/api/v3/oneprovider/files/EGI%20Foundation/" | jq '.'

# Creating a share of a subdirectory of a spaceDIR_ID_TO_SHARE=<DIR_ID>curl -sS --tlsv1.2 -H "X-Auth-Token: $API_ACCESS_TOKEN" \

-X POST -H 'Content-Type: application/json' \-d '{"name": "input"}'"$ONEPROVIDER_HOST/api/v3/oneprovider/shares-id/$DIR_ID_TO_SHARE" | jq '.'

# Displaying the shareSHARE_ID=<SHARED_ID>curl -sS --tlsv1.2 -H "X-Auth-Token: $API_ACCESS_TOKEN" \

"$ONEZONE_HOST/api/v3/onezone/shares/$SHARE_ID" | jq '.'

# Registering a handle# Proper Dublin Core metadata is required# It can be created using https://nsteffel.github.io/dublin_core_generator/generator_→˓nq.htmlcat metadata.xml# Escape double quotes and drop line returnMETADATA=$(cat metadata.xml | sed 's/"/\\"/g' | tr '\n' ' ')# On handle creation the created handles is provided in the Location headercurl -D - --tlsv1.2 -H "X-Auth-Token: $API_ACCESS_TOKEN" \

-H "Content-type: application/json" -X POST \-d '{"handleServiceId": "'"$HANDLE_SERVICE_ID"'", "resourceType": "Share",

→˓"resourceId": "'"$SHARE_ID"'", "metadata": "'"$METADATA"'"}' \"$ONEZONE_HOST/api/v3/onezone/user/handles"

(continues on next page)

4.4. Working with PID / Handle 29

https://onedata.org/#/home/documentation/doc/using_onedata/handle_services.html

https://onedata.org/docs/doc/using_onedata/using_onedata_from_cli.html#authentication


(continued from previous page)

# Listing handlescurl --tlsv1.2 -H "X-Auth-Token: $API_ACCESS_TOKEN" \

"$ONEZONE_HOST/api/v3/onezone/user/handles"

# Displaying a handleHANDLE_ID=<HANDLE_ID>curl --tlsv1.2 -H "X-Auth-Token: $API_ACCESS_TOKEN" \

"$ONEZONE_HOST/api/v3/onezone/user/handles/$HANDLE_ID"

30 Chapter 4. API

CHAPTER 5

Links

• EGI DataHub

– https://datahub.egi.eu/

– Presentation: http://go.egi.eu/datahub-intro

– Community Forum: https://community.egi.eu/c/egi-services/datahub

– https://wiki.egi.eu/wiki/EGI_Federated_Data

• System requirements

– https://onedata.org/docs/doc/system_requirements.html

• Official Onedata documentation

– https://onedata.org

– https://onedata.org/#/home/documentation

– Getting started: https://github.com/onedata/getting-started

– Source code: https://github.com/onedata

31

https://datahub.egi.eu/

http://go.egi.eu/datahub-intro

https://community.egi.eu/c/egi-services/datahub

https://wiki.egi.eu/wiki/EGI_Federated_Data

https://onedata.org/docs/doc/system_requirements.html

https://onedata.org

https://onedata.org/#/home/documentation

https://github.com/onedata/getting-started

https://github.com/onedata

EGI DataHub Documentation

Documents

Transcript of EGI DataHub Documentation