Grid Interoperability Through Standards Dr. Alistair Dunlop Project Manager.

Grid Interoperability Through Standards Dr. Alistair Dunlop

Project Manager

EU project: RIO31844-OMII-EUROPE

The OMII-Europe Standardisation process

Globus

OMII-UK

CROWN

Components

Components

IN

OUT

Re-engineering

Compliance testing and QA

Benchmarking

Component Integration

Repository

Evaluation Infrastructure

Identifying new components

Standards Implementation


Session Outline

• Overall architecture and components in OMII-Europe (Morris Riedel)

• Standards and implementations from OMII-Europe– SAML (Valerio Venturi) – BES/JSDL (Stephen Crouch)– RUS and UR (Gilbert Netzer)– GLUE II (Sergio Andreozzi)– DAIS, DMI and ByteIO (Neil Chue Hong)

• Future services


Future Work: 2008-2010

• Common implementations of specification and services for:– Data storage services

• Common data service layer building on DAIS, ByteIO, DMI and GSM

– Service discovery – Grid monitoring and service mangement– Grid billing and pricing– Grid visualisation and steering– Grid Authorisation

Architecture and SecurityOpen Standards used in real interoperability scenarios and use cases

Morris Riedel, Forschungszentrum Juelich (FZJ), Jülich Supercomputing Centre (JSC), Germany

Leader Infrastructure Integration (Interoperability) ActivityOpen Grid Forum 21, Seattle, 17th October 2007

http://www.ogf.org/index.php


Outline

• Motivation: e-Infrastructure Islands in Europe• OMII-Europe in Context• Common Vision: Interoperability Highway• Interoperability scenarios / use cases• Infrastructure to test interoperability scenarios• Interoperability scenario: WISDOM• Conclusion


• DEISA Grid (Supercomputing/HPC community)– Non WS-based UNICORE 5: Proprietary jobs (AJO/UPL)– No Virtual Organization Membership Service (VOMS), Full X.509– Suitable for massively parallel scientific jobs (MPI, …)

• EGEE Grid (mainly HEP community + others)– Non WS-based gLite: Proprietary jobs (JDL)– Proxy-based X.509 security, but proprietary VOMS support– Suitable for embarrassingly parallel scientific jobs

• Both Grids are currently not technical interoperable– Scientists cannot use one middleware to access both– Both Middleware’s had so far less adoption of open standards

e-Infrastructure Islands in EuropeDEISA [2]

EGEE [3]


OMII – Europe in ContextEC e-Infrastructures [4]


GridMiddlewares

EmergingOpen

Standards

others

End-usersvia clients& portals

„Interoperability highway“based on open standards

GOAL: Transparencyof Grids for end-users

GridResources

others

Common Vision: Interoperability Highway

UNICORE [5] gLite [6] GT [7] CROWN [8]


Infrastructure to test interoperability scenarios

Next steps in RED/ORANGE SAML- based Attribute Authority (AA) VOMS

gets central role & middleware independent

Venturi et al. [9]


• OMII – Europe implements open standards…– OGSA-BES, OGSA-RUS, WS-DAIS (OGSA-DAI), SAML (VOMS)…

• E.g: OGSA - Basic Execution Services (OGSA-BES)– In real deployments is not a ‘vanilla OGSA-BES interface’ available– Same exact “client” works not directly with gLite & UNICORE

• E.g. different security models: X.509 Proxies vs. full X.509 certificates• Solution in OMII-Europe: Agree on a Common Security Profile

• Grid interoperability: use components together…– e.g. OGSA-BES job submission secured via VOMS AuthZ

• Interoperability scenarios / use cases: e-Infrastructure use cases requiring resources in more than one Grid!

Interoperability Scenarios / Use cases


• The following interoperability scenarios have been discussed with members of the respective projects

• These scenarios demonstrate the required technical interoperability between e-Infrastructures in future that can be provided by OMII-Europe

• However, the negotiation for getting computational time on resources within these e-Infrastructures is still required by the respective projects

Disclaimer


• WISDOM (Wide In Silicio Docking on Malaria)– WISDOM aims at developing new drugs for Malaria– WISDOM uses EGEE for large scale in silicio docking

• Comp. method for prediction of whether one molecule will bind to another

– Using AutoDock and FlexX software provided via gLite in EGEE– Output is a list of best chemical compounds (potential drugs)

• That is not the final solution, only a potential list of drugs

• Refine best compound list via molecular dynamics(MD)– Fast MD computations use highly scalable AMBER in DEISA

• AMBER (Assisted Model Building with Energy Refinement) , version 9

• Goal: Accelerate drug discovery using EGEE + DEISA

Interoperability Scenario: WISDOM (1)

WISDOM [1]


Interoperability Scenario: WISDOM (2)


Conclusion

• Solutions for European e-Infrastructure Islands– Adopt components from OMII-Europe to gain interoperability

• Why OMII – Europe components…– Components going through quality assurance steps– Components adopt standards whereever possible– Components tested in real interoperability scenarios (e.g. WISDOM)

• Not only interesting for European e-Infrastructures– Beneficial for all Grids that adopt Globus, gLite, and UNICORE

• E.g. National German Grid Initiative (D-Grid)

• Continuing work in the open standards working groups!– „Interoperability highway…“ realize the „true global Grid“


References

• [1] WISDOM Project, http://wisdom.eu-egee.fr/

• [2] DEISA Project, http://www.deisa.org

• [3] EGEE Project, http://public.eu-egee.org/

• [4] European e-Infrastructures, http://www.beliefproject.org/cookbook/cookbook-intro/view

• [5] European UNICORE Grid middleware, http://www.unicore.eu

• [6] European gLite Grid middleware, http://glite.web.cern.ch/glite/

• [7] US Globus Toolkit, http://www.globus.org/

• [8] CROWN, http://www.crown.org.cn/en/

• [9] Using SAML-based VOMS for Authorization within Web Services-based UNICORE Grids, Venturi et al., UNICORE Summit 2007 @ EuroPar 2007

• [10] Improving Quantum Computing Simulations, http://www.deisa.org/applications/projects2005-2006/iqcs.php

http://wisdom.eu-egee.fr/

http://wisdom.eu-egee.fr/

http://www.deisa.org/

http://www.deisa.org/

http://public.eu-egee.org/

http://public.eu-egee.org/

http://www.beliefproject.org/cookbook/cookbook-intro/view



http://www.unicore.eu/

http://www.unicore.eu/

http://glite.web.cern.ch/glite/



http://www.globus.org/



http://www.crown.org.cn/en/



http://www.deisa.org/applications/projects2005-2006/iqcs.php


Acknowledgements

• Open Middleware Infrastructure Institute for Europe

– OMII – Europe project under EC grant RIO31844-OMII-EUROPE, duration May 2006 - April 2008

• Jülich Supercomputing Centre (JSC)of Forschungszentrum Jülich (FZJ) in the HELMHOLTZ association

Forschungszentrum Jülichin der Helmholtz-Gesellschaft


IGIIW @ e-Science 2007

International Grid Interoperability & Interoperation

Workshop

in conjunction with

e-Science 2007, Bangalore, India

http://www.omii-europe.org/OMII-Europe/igiiw2007.html

http://www.omii-europe.org/OMII-Europe/igiiw2007.html


• IQCS (Improving Quantum Computater Simulation)– One project of DECI (DEISA Extreme Computing Initiative)

• Evaluation period/test runs of IQCS code in EGEE– IQCS scientists use currently their own desktop machines or some

spare resources within institutes to develop/evaluate/test their code– That‘s slow and the management doing it personally is cumbersome

• This can be significantly improved using EGEE clusters!

• Final IQCS code production runs within DEISA– Leverage massive amount of CPUs/memory available on HPC

resources

• Goal: Accelerate code development with EGEE + DEISA

Optional: Interoperability Scenario: DECI IQCSDEISA DECI IQCS [10]


Optional: Interoperability appraoch in OMII-Europe

e-InfrastructureInteroperability by using more

than one technology!

Venturi et al. [9]


Questions for JRA3 – Task 2

Morris Riedel

[email protected]

JRA 3 Team

[email protected]

mailto:[email protected]

mailto:[email protected]


Attribute Exchange Profile

• Problem statement– Interoperable, pluggable, replaceable attribute services– Easier aggregation of attributes coming from difference sources

• OGSA Architecture envision the use of authorization policies coming from different stakeholders

• Solution– Standard interface for requesting attribute assertions


Attribute Exchange Profile

• Problem statement– Attributes assertions understood across middleware boundaries

• Solution– Agreed format for attribute assertions


Status of the specification

• The specification is being worked on inside the OGF OGSA Authorization Working Group– Basically profiles existing specification (OASIS, WS-I) for use

with Grid Services• Bases are solid

– OASIS SAML V2.0 set of specifications• SAML V2.0 Deployment Profiles for X.509 Subjects

• Profile presented on Monday• Those interested should join the OGS Authorization WG

http://www.ogf.org/gf/group_info/view.php?group=ogsa-authz-wg


Specification Implementers

• VOMS– gLite and OMII-Europe developed, used in EGEE,

OSG, Naregi– Implementing the service

• UNICORE– Implementing the client part of the profile in OMII-

Europe• Part of the 6.2 release planned for early 2008

– Implementing the service in the chemomentum project• GridShib


Implementation status

• OMII-Europe is re-engineering the Virtual Organization Membership Service (VOMS) to meet the specification

• VOMS is developed also in EGEE• VOMS is an Attribute Authority focused on VO management

– It allows users to register to a VO and being assigned attributes such as groups and roles

– The attributes are used by Grid Services to drive authorization decisions

• VOMS is already used in EGEE, OSG and Naregi– Plugin for the globus Authorization framework

• OMII-Europe is also adding VOMS support to UNICORE– UNICORE is implementing the client part of the profile


Implementation Status

• Alpha version of the VOMS SAML Service available since May 2007 for integration testing purposes– Demonstrated at OGF 20 in Manchester in conjunction with

UNICORE OGSA-BES• Beta version available early November

– Then into the QA process of OMII-Europe, that comprises standard compliance testing

• VOMS SAML Fed back to gLite early 2008• OMII-Europe support for VOMS

– Available from version 6.2 early 2008


Implementation Plans

• Undergoing integration with BES services re-engineered by OMII-Europe– UNICORE OGSA BES already done, demonstrated at OGF 20– gLite CREAM BES nearly done– May be demonstrated at SC07

• Planned integration with other OMII-Europe re-engineered components– RUS services: SGAS, gLite DGAS, UNICORE OGSA RUS– OGSA DAI

• Integration with BES services is a proof of concept of the first addressed issue– Availability of attributes across middleware boundaries


Further plans

• Shibboleth 2 on its way out– GridShib will implement the same interface

• We plan to test integration scenarios– Shibboleth attributes concern Home Institution

Affiliation– VOMS attributes concern Virtual Organization

• This would be proof of concept of the second issue addressed– Easy aggregation of attributes coming from different

sources

Job Submission: OGSA-BES & JSDLMoreno Marzolla ([email protected]), Stephen Crouch SOTON

[email protected]


Contents

• The problem of incompatible job submission systems

• The solution: interoperability through standards– OGSA-BES– JSDL

• Standards status and limitations• Implementations of standards within OMII-Europe

– Computation endpoints– Meta-scheduler

• Future interactions with other OMII-Europe activities


Setting the Scene: Grid Job Submission

• User problem:– User (e.g. researcher) has a problem they need to solve computationally– They do not have local access to appropriate resources in terms of

computation and/or applications• A non-local provider:

– Has the required computation/application resources– Wishes to allow them to be shared by appropriate users

• Grid job submission provides the technological ‘glue’ to allow users to utilise the provider’s computational/application resources

UserJob

Comp/AppResources

Job Submission


The Problem

• Different job submission systems often incompatible in terms of– Interface– Job description– Security (more general problem)

• Overhead in terms of– Use – learning curve– Infrastructure support – multiple client installations– Application development – knowledge of multiple APIs

• Interoperation solutions expensive in terms of development & maintenance

ClientApp

Comp/AppResources

Job Submission A

Client (A)

Client (B)

Job Submission B

Comp/AppResources

…

… …Desc A

Desc B


Solution: Interoperability through Standards

• Two key standards for two key elements:– The Basic Execution Service web service interface (OGSA-BES)

• 1 instance of job container within OGSA-EMS (Execution Mngt Service)• Handles basic job lifecycle management• Defines simple (but extendable) job state model

– Pending, running, cancelled, failed or finished– The Job Submission Description Language (JSDL)

• Specify job executable, data staging and resource requirements

ClientApp

Comp/AppResources

Job Sub. A

JobSub. B

Comp/AppResources

BESClient

JSDL BESI/F

BESI/F

JSDL

… …

…


Summary of OGSA-BES

BES-Management Port-type

StopAcceptingNewActivities Request that the BES stop accepting new activities

StartAcceptingNewActivities Request that the BES start accepting new activities

BES-Factory Port-type

CreateActivity Request the creation of a new activity

GetActivityStatuses Request the status of a set of activities

TerminateActivities Request that a set of activities be terminated

GetActivityDocuments Request the JSDL documents for a set of activities

GetFactoryAttributesDocument Request XML document containing BES properties

BES-Activity Port-type (optional)

GetStatus Request the status of an activity

Terminate Request that an activity be terminated

GetDocument Request the JSDL document for an activity

GetActivityAttributesDocument Request XML document containing activity properties


JSDL Example I<JobDefinition xmlns="http://schemas.ggf.org/jsdl/2005/11/jsdl"> <JobDescription> <JobIdentification> <JobName>cat job</JobName> <Description>cat job description</Description> <JobAnnotation>no annotation</JobAnnotation> <JobProject>an example project</JobProject> </JobIdentification> <Application> <POSIXApplication xmlns="http://schemas.ggf.org/jsdl/2005/11/ jsdl-posix"> <Executable>/bin/cat</Executable> <Argument>dir1/file1.txt</Argument> <Output>stdout.txt</Output> <Error>stderr.txt</Error> </POSIXApplication> </Application>


JSDL Example II <DataStaging> <FileName>dir1/file1.txt</FileName> <CreationFlag>overwrite</CreationFlag> <Source> <URI>http://server.domain/helloworld.txt</URI> </Source> </DataStaging> <DataStaging> <FileName>stdout.txt</FileName> <CreationFlag>overwrite</CreationFlag> <DeleteOnTermination>true</DeleteOnTermination> <Target> <URI>ftp://anonymous:[email protected]:45521/public/output/stdout.txt</URI> </Target> </DataStaging> </JobDescription></JobDefinition>


Standards Status & Limitations

• Both JSDL & OGSA-BES have reached version 1.0 (recommendation)

• Future support being considered– Parametric jobs – being considered as JSDL

extension– Collections of jobs with dependencies

• Neither addresses security– Fundamental requirement for interoperability in real

deployments– Mutually compatible security model (JRA3/Security,

JRA1/VOMS)


Implementations: Endpoint Supportfor BES & JSDL

• BES 1.0/JSDL 1.0 support developed in OMII-Europe for– UNICORE 6.1 (Jan/Feb 2008):

• BES/JSDL support in vendor middleware

– gLite 3.1:• CREAM-BES component based on CREAM-CE

• Available through OMII-Europe repository in Dec 2007

– Globus 4.2:• Some BES/JSDL support exists within vendor middleware

• Work progressing – target release date TBD

• Support for BES within other job submission implementations:– OMII-UK, Beihang, Microsoft, University of Virginia, Platform

Computing


Implementations: Meta-schedulingsupport for BES & JSDL

• BES/JSDL 1.0 support in CROWN– Progressing towards support for BES 1.0

• CROWN Meta-scheduler hosted within Globus 4.2:– Accept BES submissions, delegate to BES endpoints based on

policy– Release targeted March 2007, available from OMII Europe

repository

Res.Job

Sub. ABESI/F

Res.Job

Sub. BBESI/F

ClientApp

BESClient

JSDL

JSDL

MetaScheduler

BES I/FJSDL


Future Interactions with OMII-Europe Activities

• Security (JRA3)– Support common security profile

• Compliance testing (SA2) w.r.t. OGSA-BES/JSDL– Test endpoints– Meta-scheduler ‘plugs in’ to compliance test

• Benchmarking (JRA4)• Training (NA3)

Accounting in OMII-EuropeGilbert Netzer (KTH)


Outline

• Accounting Standards– Overview– Usage Record Format– Resource Usage Service

• OMII-Europe Accounting Activity– Goals– Used Standards– Schedule– OMII-Europe Accounting Scenarios

• Questions


Benefits of Standardization

• Many (batch) systems are alike– All batch systems collect similar metrics– However units, data format and semantics differ

• Large systems are heterogeneous– Standards help in building these systems– Off-the-shelf components can be plugged together

• Standards give guidance for implementations– Define parts of the requirements– Can help defining the scope of an application

• Promote sharing of knowledge


OGF Standards for Accounting

• Usage Record Format (URF, GFD-R-P. 98)– Recommends a document format for usage information– Does not address transport of information– Proposed Recommendation status– Interoperability workshop scheduled for OGF22

• Resource Usage Service (RUS, GWD)– Specifies means to transport usage information– Uses the URF as data format– Working Draft stage– Meetings were held at OGF19, 20 and 21


Accounting Standards Usage Scenario

Usage Records

Pro

du

ce

s

URStorage

RUS

Resource

SensorMetering Component

`

Grid User

Publish

Submit Job

Submission Endpoint

Queue

Accounting System

Ch

ec

k

Qu

ota

Usage Record Format (UR)

Resource Usage Service

(RUS)


OMII-Europe Goals

• Make existing accounting infrastructure interoperable• Use OGF standards as vehicle:

– Usage Record Format (UR)– Resource Usage Service (RUS)

• Address the three middlewares of the project:– Distributed Grid Accounting System (DGAS)– SweGird Accounting System (SGAS)– UNICORE-RUS

• First step, focus on the existing basics only• Augment existing systems to allow easy transition


OMII-Europe Schedule

• Alpha versions demonstrated at OGF20 (Manchester)– DGAS client uploads/downloads from SGAS server– UNICORE demonstrated using RUS for monitoring

• Beta versions November-December 2007– Will have both server and client components for all

three middlewares– Will be available for download from repository

• Final versions for general public release– Will be ready in March 2008

• Software will be released via OMII-Europe Repository– We also try to feed SW back to the MW distributions


Alpha Demonstration - OGF20

OMII-Europe added value (RUS)

UNICORE gLite DGAS Globus SGAS

End User Applications

Services

Resources

LLView DGAS Clients SGAS Clients

UNICORE RUS LUTS

UNICORE UR Generator DGAS Sensor

UNICORE Alpha at German eScience 2007


OMII-Europe Accounting Scenario

OMII-Europe added value (RUS)

UNICORE gLite DGAS Globus SGAS

End User Applications

Services

Resources

LLView DGAS Clients SGAS Clients

UNICORE RUS

HLR RUS Service LUTS

UNICORE UR Generator DGAS Sensor JARM

LegacyInteraction


Standards in OMII-Europe Accounting

• Phase I (Current Project)– Uses RUS draft version 17

• Is available• Has passed public comment phase

– Uses late URF draft• This is used by RUS draft 17• Different namespace than GWD-R.P. 98• Minor differences (e.g. new GlobalUserName)

– RUS is work in progress!• Phase II (Next Project, 2008-2010)

– Upgrade to final RUS&URF recommendation


Usage Record Format

• XML Document Format for Usage Information• Approach

– Defines many elements for different metrics– Most elements optional, implementation picks– Extension framework for further properties

• Limitations– Only computational jobs considered in version 1– One record per job, no summarization– VO membership information was not considered– Limited extension framework, only string content


Anatomy of a UsageRecord <JobUsageRecord xmlns="http://schema.ogf.org/urf/2003/09/urf" xmlns:urf="http://schema.ogf.org/urf/2003/09/urf"> <RecordIdentity urf:recordId="http://www.emsl.pnl.gov/mscf/colony/PBS.1234.0" urf:createTime="2003-08-13T18:56:56Z" /> <JobIdentity> <LocalJobId>PBS.1234.0</LocalJobId> </JobIdentity> <UserIdentity> <LocalUserId>scottmo</LocalUserId> </UserIdentity> <Charge>2870</Charge> <Status>completed</Status> <Memory urf:storageUnit="MB">1234</Memory> <ServiceLevel urf:type="QOS">BottomFeeder</ServiceLevel> <Processors>4</Processors> <ProjectName>mscfops</ProjectName> <MachineName>Colony</MachineName> <WallDuration>PT1S</WallDuration> <StartTime>2003-08-13T17:34:50Z</StartTime> <EndTime>2003-08-13T18:37:38Z</EndTime> <NodeCount>2</NodeCount> <Queue>batch</Queue> <Resource urf:description="quoteId">1435</Resource> <Resource urf:description="application">NWChem</Resource> <Resource urf:description="executable">nwchem_linux</Resource></JobUsageRecord>

Unique ID &TimestampReference to Job

Reference to User

Usage Information

Extensions


Anatomy of the Resource Usage Service

• Defines a web-service interface (draft 17):– Store (insert) Usage Records– Extract information using XPath queries

• Whole UsageRecords wrapped with audit information• Only the ID assigned by the RUS (not RecordId!)

– Modify stored Usage Records• Replace whole UR with new UR• Modify UR using XUpdate expression• Atomically increment numeric property of a UR

– Delete stored Usage Records by ID or XPath query– Get local constraints (mandatory elements)


Limitations of RUS

• Extract always returns results in one message– Problems with large queries (memory, timeouts)– Planned to support enumerations

• No summarization possible– Cannot return e.g. total jobs this month– Planned for Advanced RUS specification

• Not aware of VOs in standard way– Can use UR extensions, standardization in URF v. 2

• No federation support– Need to always query whole set, no diffs.


OMII-Europe's Contribution in Accounting

• Usage Record Working Group– Plan to supply UR-data to interop WS at OGF22

• Resource Usage Service Working Group– 2 of 3 co-chairs come from OMII-Europe– Will provide three independent implementations– Will provide implementation experience


References

• OMII-Europe Website:http://www.omii-europe.org

• UR-WG GridForgehttp://forge.ogf.org/sf/projects/ur-wg

• RUS-WG GridForgehttp://forge.ogf.org/sf/projects/rus-wg

GLUE 2: Common Information Model for Grid resources

Sergio Andreozzi, INFN

17 Oct 2007, Seattle, WA, USA


Problem Statement

• How do we describe resources shared in Grid systems in order to enable:

– Resource awareness

– Resource discoverability

– Resource requirements expression

– Resource basic monitoring


Use Case 1

• I want to run my job on an execution environment characterized by:

– OS• Linux, Distribution X, version Y

– CPU Archicture • IA64

– Available software packages: • S1, S2


Use Case 2

• I want to know

– how many job slots are used by members of the VO A

– what is the global available storage space for the users of VO B


Focus on Computing Resources

• OGSA-BES and JSDL are already considered by OMII-Europe

• They lack a common description of Grid resources suitable for discovery, monitoring and scheduling

• Many descriptions exist– e.g.: GLUE Schema, NorduGrid

Schema

• Barrier to interoperable Grid systems


How Are We Contributing

• Co-leading the OGF GLUE WG which purpose is to define the next-generation information model for Grid resoruces as a community standard

• OGF GLUE WG Goals:– define a use case document collecting use cases from different

Grid projects/infrastructures– define a conceptual model defining the abstract schema GLUE

2.0 satisfying the collected use cases. – develop reference implementations

• Timeline: go public comments by OGF 22 (Jan 2008)


Relationship to other OGF WGs

JSDL

GLUE

Reference Model

SAGA

GSMOGSA Res.Mgt.

Used to express requirements inCommon service descripton

for discovery API

Used to describe exposed resources

Should fit into the picture

Should fit into the picture

BES

Used to describe exposed resources


How Are We Contributing

• Rendering of the GLUE conceptual model as XML Schema and inclusion into the OMII-Europe repository– Other renderings like LDIF are expected

• implementation of information providers – for BES implementations developed in the context of OMII-

Europe– leveraging DMTF standards

• Inclusion of this capability into the OMII-Europe Repository


Activity Status

• OGF Documents– Specification and Use Cases drafts available– http://forge.ogf.org/sf/docman/do/listDocuments/projects.glue-wg/docman.root.drafts

• Rendering as XML Schema– First version to be issued in Nov 2007 based on OGF21 draft

• Information providers– Prototype framework based on OpenPegasus to be released in

Nov 2007

• Production Quality full implementation by Apr 2008

http://forge.ogf.org/sf/docman/do/listDocuments/projects.glue-wg/docman.root.drafts


Adoption and Related Efforts

• The output of this activity will be adopted by the OMII-Europe middleware providers

• The software will be publicly accessible from the OMII-Europe repository and will be advertised for larger adoption

• Other projects will make their own implementation of the GLUE spec (e.g. ARC/NorduGrid)

Data Specs: DAIS, OGSA ByteIO, OGSA-DMINeil Chue Hong

OMII-UK


Overview

• Data specifications and OMII-Europe– Database Access and Integration Service (DAIS)– OGSA ByteIO (ByteIO)– OGSA Data Movement Interface (OGSA-DMI)

• What is the problem• What is the status of the specification• What is the status of the implementations


Uniform access to structured data

• Each database vendor typically supports different methods of access and different data models– difficult to federate

• DAIS provides a uniform interface for data access that allows– location independence– heterogeneity– composability

• It also presents interfaces that allow virtualisation of database resources, and result sets


DAIS Specifications

WS-DAI

WS-DAIR WS-DAIX WS-DAIO

Sets general pattern for DAIS realisations

Currently defined

Draft at GGF14

RDF

In discussion


Data Service Composition

GDSClient

GDSClient

Client

1 Operation

GDSClient

2

DB

Operation

Operation

OperationDB

4

Operation

Operation

Operation

DB

GDS

GDS

GDS

3

Operation

Operation

Operation

DB

GDS

GDS

GDS

Client5

Operation

Operation

Operation

DB

GDS

GDS

GDS


Complex Scenario

GFactory G Q ES F

GFactory G Q ES F

GFactory G Q ES F

N 2

N 1

N3

GC lie n tGG D S

GG D S

G D Q

G D T

G D Q S

N 0G D S

GFactory G Q ES F

N4

p erform (Q u ery)1

cre a te S e rv ice

cre a te S e rv ice2

cre ate S e rvi ce

2

2

GG D S G Q ES 2

G D T

GG D S G Q ES 3

G D T

GG D S G Q ES 1

G D T

GG D S G Q ES 1

G D T

p erform (Q u ery S u b p la n )

p erform (Q u ery S u b p la n )

perform(Q

uer ySu bpl an)

3

s eq u en t ial_ s can

red u ce (p r o tein ID ,s eq u en ce )

s eq u en t ial_ s can ( ter m = 8 3 7 2 )

red u ce (p r o tein ID )

h as h _ jo in(p .p r o tein ID = t.p r o tein ID )

3

o p erat io n _ callb la s t(p .s eq u en ce)

red u ce (p .p r o tein ID , b la s t)

o p erat io n _ callb la s t(p .s eq u en ce)

red u ce (p .p r o tein ID , b la s t)

3

W e b S e rvi ce s (B L A S T)

resu lts

resu lts

resu lts

4

1144


DAIS

• The OGSA-DAI project has been heavily involved in standards process– implementation of DAIS proposed recommendation in

progress• Work in OMII-Europe has concentrated on

– interoperability across Globus, UNICORE and gLite– integration with VOMS security

• Other implementations of DAIS from– Ohio State University– EGEE (DAIS interface to AMGA)– IBM


DMI Scenario


User moves data from A B

• Does not care how it is done• Does not want to:

– Deal with different protocols– Deal with different security models– Learn new interface for different data types

• Handle structured & un-structured data– Agnostic about the data semantics

77


ByteIO: POSIX like

• A concise, standard way of interacting with grid data resources as POSIX-like files– interfaces defined for random access and streamable

• With other specs, allows a file and directory-like view of a Grid and many other simplified views of data– shell, file explorer, database result sets, pipes

• ByteIO Spec out of public comment with four implementations – UNICORE implementation by FZJ– OGSA-DAI implementation by EPCC– Clean room implementation by FLE– Genesis II implementation by UVa


Staging files in Job Submission

• HPC Profile enables interoperable job submission systems

• OGSA-DMI provides standard way of staging files in and out

• OGSA ByteIO allows part of a large remote file to be staged in without transferring the whole file


Summary• OMII-Europe is a 24 Month EU funded project with 16 partners to establish grid

infrastructure interoperability through implementing a set of agreed open standards on all middleware platforms

• OMII-Europe is implementing a common Job submission system, Accounting service, Database service, Virtual Organisation service, Information Service and Security model for major middleware platforms. This will allow identically specified jobs to be run, managed and migrated to different middleware platforms thus enabling “the Global Grid”

• Initial versions of BES, VOMS and security service have already enabled UNICORE and EGEE resources to be used by the same job

• A complete set of fully interoperable services will be available in spring 2008 with early versions of some services available already

• Users can try interoperability on the OMII-Europe evaluation infrastructure, or obtain services for installation on their own resources from the OMII-Europe repository

• We anticipate OMII-Europe services to be integrated into standard middleware distributions as well as deployed on large scale e-infrastructures such as EGEE and DEISA

• OMII-Europe will request continuing funding in the September EU call to support the existing services and provide further services in the areas of data and grid management


Further Information

• http://www.omii-europe.org

Q & A

Grid Interoperability Through Standards Dr. Alistair Dunlop Project Manager.

Documents

Transcript of Grid Interoperability Through Standards Dr. Alistair Dunlop Project Manager.