A glance to the future

EGEE-II INFSO-RI-031688

Enabling Grids for E-sciencE

www.eu-egee.org

www.glite.org

A glance to the future

Mike Mineter

TOE-NeSC

Application Developers CourseMay 12-13, 2007Manchester, UK

2



Overview

• WMProxy

• GridWay – RESPECT – EGEE Initiative to collect useful tools that work

with gLite See EGEE application portal: http://egeena4.lal.in2p3.fr/index.php Under construction

– GridWay is one of the soon-to-be RESPECTed tools

• Projects to watch include– ETICS– OMII-Europe

3



New Functionality: WMProxy

• WMProxy server– Will replace the old C++ based socket connection service– Implements an interoperable interface

Web Service based WS-I compliant

• WMProxy client– Provides C++ based WMS command-line User Interface (UI),

which executes all the needed operation automatically– Provides multi language (C++, Java and Python) provided APIs

4



WMS Architecture overview

Job Controller CondorG

gLite WMS

Workload Manager

LB Proxy

WMProxyUserInterface

LB Server

gLite

CE

LCG

CE

Job Controller

CondorC

Log Monitor

5



JDL: Single Types

• Single Jobs– Normal: single and simple batch job with no peculiar

requirements– MPICH: a parallel application to be run on the nodes of a

cluster using the MPICH implementation of the message passing interface new MPI flavours support planned

– Interactive: a job whose standard streams are forwarded to the submitting client, that can actually interact and steer the job execution by providing real-time input information

7



JDL: Compound Jobs

• Definition– Aggregation of Single/Normal Jobs

• Benefits– One Shot submission for (up to thousands of) jobs

Single call to WMProxy server Single AuthN and AuthZ process Submission time reduction

– Single Identification to manage all jobs (father Job) Not an actual Job, used to monitor the whole bunch

– Sharing of files between jobs

8



Middleware structure

• Applications have access both to Higher-level Grid Services and to Foundation Grid Middleware

• Higher-Level Grid Services are supposed to help the users building their computing infrastructure but should not be mandatory

• Foundation Grid Middleware will be deployed on the EGEE infrastructure– Must be complete and robust– Should allow interoperation

with other major grid infrastructures

– Should not assume the use of Higher-Level Grid Services

Foundation Grid Middleware

Security model and infrastructure

Computing (CE) and Storage Elements (SE)

Accounting

Information and Monitoring

Higher-Level Grid Services

Workload Management

Replica Management

Visualization

Workflow

Grid Economies

...

Applications

9



GridWay

• GridWay – one of the tools recognised by EGEE’s RESPECT– Alternative to WMS

• Examples of use:– Many similar jobs– Short jobs– Resources outside EGEE also to be used– User-site-specific policies are required (priorities of users’ jobs)– …

Contents

What is GridWay?

Which are the benefits of using GridWay?

How do I use GridWay in EGEE?

Who is using GridWay in EGEE?

Where can I get GridWay?

What is GridWay?

Ap

plic

atio

n-I

nfr

astr

uct

ure

dec

ou

plin

g

PBS

GridWay

SGE

$>CLI

Results

.C, .java

DRMAA

.C, .java

Infrastructure

Gri

d M

idd

lew

are

Ap

plic

atio

ns

Globus, gLite, …

Gri

d M

eta-

Sch

edu

ler

• standard API (OGF DRMAA)• Command Line Interface

• open source• job execution management • resource brokering

• Basic Grid services• Standard interfaces• end-to-end (e.g. TCP/IP)

• highly dynamic & heterogeneous• high fault rate

Which are the benefits of using GridWay?

FEATURES BENEFITS

Support for DRMAA standard (C and JAVA bindings) & JSDL standard

Compatibility of applications with DRM systems that implements the standard, such as SGE, Torque,...

DRM Command Line Interface (allow users to submit, kill, migrate, monitor and synchronize jobs)

CLI similar to that provided by local resource managers

Lightweight middleware Higher efficiencies for given application profiles

Site-level accounting and scheduling policies

Analysis of resource utilization, determining trends in usage and monitoring user behavior

Installation: minimal requirements –

also portable (Mac..), ..

Easy and fast deployment and maintenance

Interoperability with resources

outside EGEE

Simultaneous access to different infrastructures (LCG, gLite…)

GridWay Complements gLite Providing the Following Benefits

Who is using GridWay in EGEE?

SGE Cluster

Users

PBS Cluster PBS Cluster

gLite

GridWay

gLite gLite

• Services: BDII, GRAM, GridFTP• EGEE Resource Broker

• DRMAA interface• VO Schedulers

GridWay

Users

BiomedFusion

EGEE RB

Massive Ray Tracing CD-HIT workflow

Where can I get GridWay?

• From the Gridway webpage: www.gridway.org

• From the ETICS repository

• From the Globus CVS repository (cvs.globus.org)

Download the software

• Gridway webpage: www.gridway.org

• Application porting with GridWay

http://www.gridway.org/successstories/applicationporting.php

• Infrastructures using GridWay

http://www.gridway.org/successstories/projectsinfrastructures.php

More Information

21



Two projects adding value to EGEE

Significant for future application developers….

• ETICS– Build/test for grid services– Spin-off from gLite development & certification– Used in OMII-Europe software repository (among others)

• OMII-Europe– Creating / re-engineering services that use standards– Effect will include bridge-building across grids

The vision: a VO will be able to use services across gLite / UNICORE/ Globus/ CROWN / OMII-UK grids

22EU project: RIO31844-OMII-EUROPE

What will OMII-Europe do?

• Initial focus on providing common interfaces and integration of major Grid software infrastructures

• Common interoperable services:– Data Access, Virtual Organisation Management,

Portal, Accounting, Job Submission and Job Monitoring

– Capability to add additional services• Infrastructure integration

– Initial EGEE/UNICORE/Globus/CROWN interoperability

– Interoperable security framework

23EU project: RIO31844-OMII-EUROPE

OMII-Europe Repository and ETICS

ProjectRepository

P PSoftware Repository

(CVS, Subversion, tar.gz, zip)

R R R

Build Artefact Repository(rpm, deb, tar.gz, zip)

B B B

PUBLIC VIEW

ETICS(Build & Test)

NMI Scripts

Condor Pools

NMI Build

Config

Created Artefact

NMI Build

Config

NMI Test

Config

S S

24



Summary

• Application developers will benefit from up-coming functionality from:

– gLite with WMProxy

– RESPECT – prominent is GridWay

– Related projects

ETICS: build and test of grid services

OMII-Europe: components that will permit a VO’s resources to span grids

A glance to the future

Documents

Transcript of A glance to the future