30-31 Jan 2003J G Jensen, RAL/WP5 Storage Elephant Grid Access to Mass Storage.

30-31 Jan 2003 J G Jensen, RAL/WP5

Storage Elephant

Grid Access to Mass Storage


Overview

Control Info Data Xfer

Storage Element

MSS2MSS1

Grid clientsGrid clientsGrid clients


Interfaces

• Control• Pin/unpin files (reserve for reading)

• Create new file (reserve for writing)

• Modify metadata

• ...

• Information• Get amount of free space

• Protocols supported by the SE

• ...

• Data Transfer


• HTTPS– Reliable, well known protocol– Looking at extensions for Grid security

• Web services– Rapid development– Language independent (but some languages are more

independent than others…)

• GridFTP– Extension of well known FTP protocol, complicated– Achieves good transfer performance

• LDAP• WP3 information producer

– Currently provides only status of daemons• Worker nodes can access their local SE via NFS

– We will investigate using SlashGrid instead – RFIO is trivial to add

Protocols


Portability

• RPMs have been or can be built for RedHat 6.2, 6.3, 7.2, 7.3– Support for RedHat 8.x soon

• SE builds from source on both Debian x86 stable and testing and on other Linux.

• TODO: Test on Solaris - ?


Client App

API

SE HTTPlibrary

SSL socketlibrary

ApacheTomcat

Axis

RMANMAN

SE core

C Client

SE

Web ServicesClient

Axislibraries

JavaClients

The design of the SE follows a layered model with a central core handling all paths between client and MSS.

Core is flexible and extensible making

it easy to support new protocols, features and MSS


RManager

SE CORE

Handler1

Handler2

Handler3

Handler4

Responsereturnedto user

MSS

• Each handler performs a simple task.

• The sequence of handlers depends on type of request, the type of storage system, local configurations, etc.

• Rmanager (request manager) is responsible for running handlers

This architecture is extremely flexible and allows SE to easily meet diverse

requirements for both clients and backend MSS.


Interfaces to SE

Control Information Data Transfer

HTTPS

GridFTP

Web S.

Done Done

Nearly done

Planned

Planned

DoneDone Not planned

Not planned

Note: diagram shows existing interfaces, not implemented functionality


SE functionality

• For MSS with many features, the SE will be a thin layer

• For MSS with few features (or disk), SE will provide “missing” functionality F

unct

iona

lity

CASTOR ADS Disk

= SE


Security• Access control:

– Currently uses Andrew McNab’s GACL– Will also use VoMS (and CAS?)– Fine grained access control – files and dirs all

have ACLs but ACLs can also be shared between files/dirs

• GSI– Will use G-HTTPS– Currently uses Mike Jones’ patch to mod_ssl to

accept proxies


SRM Collaboration

• SRM – storage resource manager – collaboration with LBNL, Fermi, Jefferson, CERN

• Particle Physics Mass Storage System interface– Mostly control interface; data transfer done through

GridFTP

• “Old” SRM specification version 1.0 – IDL, web service..?– Some naughtiness: server callbacks

• New specification: interoperability through WSDL– Is WSDL sufficient for interoperability?– Complicated space reservation semantics


Non-PP Grids• There exist people who will use Grids for non-PP,

non-HEP projects!• Examples from DataGrid:

– Earth observations (WP9): will write handlers for their own MSS.

• EO applications have many (106) small (2-30 kB) files

– Medical Imaging (WP10): will do the same but WP10 have more challenging requirements

• Confidentiality – data must be enciphered when it goes out to the Grid

• Key will be held in a Grid database on trusted site

• CrossGrid is / will be based on DataGrid software


Deployment

• SEs are deployed and working at– CERN, talking to CASTOR using RFIO– RAL, talking to ADS and disks

• SE soon to be deployed for testing and development at INSA Lyon– INSA SE will talk to DICOM (medical images)

• Integration with testbed in March, tested by applications in May


Concluding Remarks

• Current interfaces are flexible andcan be expanded to support new commandsand features, or add support for new protocols

• Core is very flexible and it is easy to add new functionality

• We’re making the SE easier to install and maintain – will be installable by sysadmins rather than developers!

30-31 Jan 2003J G Jensen, RAL/WP5 Storage Elephant Grid Access to Mass Storage.

Documents

Transcript of 30-31 Jan 2003J G Jensen, RAL/WP5 Storage Elephant Grid Access to Mass Storage.