Conference name Company name INFSOM-RI-1234567 Speaker name The ETICS Job management architecture...

Conference nameCompany name

INFSOM-RI-1234567

Speaker name

The ETICS Job management architecture

EGEE ‘08Istanbul, September 25th 2008

Valerio Venturi

INFN - ETICS

[email protected]

The ETICS Job Management Architecture

INFSO-RI-223782

2Istanbul, September 2008

Outline

• Deploying to the Grid infrastructures

• Identification of deployment issues

• Re-engineering the ETICS services

• Submitting ETICS jobs to the EGEE infrastructure• Technical issues

• Organizational issues

• Submitting ETICS jobs using UNICORE

• Submitting ETICS jobs to EC2


INFSO-RI-223782


Integrating ETICS in Grid InfrastructuresProblem statement

Software projects maintain a variable-sized pool of machines for the various tasks of development and maintenance

• Building software releases

• Running continuous builds of the development release

• Testing the software

For a number of reasons (needs of high-availability for release-rush, needs of possibly a lot of different platforms) this machines are likely to have high idle times.

When a software projects happens to have other computing needs, it would great benefit to be able the idle time of their resources for their other computing needs.


INFSO-RI-223782


Integrating ETICS in Grid InfrastructuresProblem statement

Providing resources for software development tasks is what ETICS does for its customers.

So far the machines are owned and managed by the ETICS project, thus eliminating the problem of allocating resources.

Not sustainable. Where is gLite going to be build in a couple of years from now?

We propose integrating the ETICS services in infrastructures (such as EGEE) to solve the sustainability problem and the more general problem of wisely using the resources projects allocate to software development tasks.


INFSO-RI-223782


Integrating the ETICS servicesAnalysis of deployment issues

1

Build/TestArtefacts

ConfigurationWeb Service

Report/MetricsDB

ConfigurationDB

ExecutionEngine

Command Line User Interface

RepositoryWeb Service

ETICS Infrastructure

Web Portal


INFSO-RI-223782


Build/Test Jobs Management in ETICSArchitecture

ExecutionEngine

Physical

Worker

Nodes

Virtual

OS Images

ETICS Submission Engine


INFSO-RI-223782


Build/Test Jobs Management in ETICSRe-engineering the ETICS services

• This wouldn't be necessary if the various infrastructures had interoperable interfaces and security mechanisms

• Could have used BES and VOMS to submit to both gLite and UNICORE

• We will separate the part of the ETICS service that handle the jobs submission so that it can be deployed separately

• A submitting interface implemented using Metronome over a Condor pool, gLite over the EGEE infrastructure, and UNICORE over DEISA

• Operations to submit, cancel, getStatus

• Why not a standard interface like BES?

• ETICS jobs are pretty simple, doesn't really fit the bill


INFSO-RI-223782

1 8Istanbul, September 2008

Submitting ETICS Jobs to EGEEOverview

ConfigurationWeb Service

Submitter

RepositoryWeb Service

ETICS Infrastructure EGEE Infrastructure

VOMS

JDL

VOs


INFSO-RI-223782


Submitting ETICS Jobs to EGEETechnical issues

We don't envision big problems from a technical point of view, as gLite is thought to be easily integrated

• The WMS has a WS interface that makes it pretty easy to use

• Jobs are described using an established language as JDL

• The GLUE Schema allows for a (quite) standard description of resource requirements

• Use information published by sites through the GLUE Schema elements to land on nodes with the required architectures, operating system and compiler

• Future works on virtualizations will eliminate this problem

• Unfortunately VOMS doesn't a WS interface• Hopefully gLite can get to it in the future


INFSO-RI-223782


Submitting ETICS Jobs to EGEEOrganizational issues

In order to have access to the EGEE infrastructure, you have to be in one of the EGEE VOs.

Two solutions

• Apply for an ETICS VO, moving some resources to the infrastructures and gaining share quotas

• Use the current EGEE Vos

With either solutions, we have to decide whether

• Use delegation to submit the ETICS jobs with the user credentials

• Have the ETICS service registered to the VO and knows which credentials to use for submitting jobs


INFSO-RI-223782


Submitting ETICS Jobs to EGEEAn ETICS Virtual Organization?

Apply for an ETICS VO

• Moving part of the nodes that are currently under exclusive ETICS management to the EGEE infrastructure

• Getting a quota with which to run build and test jobs

In this case ETICS will manage its internals resource sharing, assigning priority and quotas to its projects.


INFSO-RI-223782


Submitting ETICS Jobs to EGEEAn ETICS Virtual Organization?

PROS

• Simple solution

• ETICS may provide resources for building projects that are not in a VO and so don't have resources on the infrastructure

CONS

• Not complete integration, ETICS would “buy” resources on the infrastructure and act as a broker to projects using it

• Does ETICS fit the profile for a VO• Typically scientific collaborations


INFSO-RI-223782


Submitting ETICS Jobs to EGEEIntegrating with the current EGEE VOs

ETICS uses an EGEE VO credentials for submitting build jobs, thus consuming resources assigned to that VO

• If a VO decide to use ETICS for building its software, it can use its resources shares for doing that

• For example, the gLite middleware builds could be accounted to DTEAM

Two ways of doing that

• Using delegation• The VO is able to grant single users the permission to build and

test, and have fine grained control

• Having the ETICS service credentials registered in the VO, and the ETICS services maintaining a mapping to know which VO credentials to use for each builds


INFSO-RI-223782



Real-life example, building the gLite middleware

• The DTEAM VO in EGEE is used by gLite developers for testing the developments

• The DTEAM VO may have developer, integrator, tester, roles (or groups)

• The DTEAM VO assigns shares to those roles, so that importance of the work is reflected

• This can be done dynamically, so to handle possible bug-fix-release-rush situations

• Developer, integrator and tester submits their jobs and these are executed on the infrastructure according to the policy decided by the VO

• Can be handled both by using delegation or the ETICS user in the VO


INFSO-RI-223782



PROS

• Complete integration, resource shares are arranged within EGEE and not between EGEE and ETICS

CONS

• Projects that don't have EGEE shares won't build

• Registering the ETICS service credentials to the VO may scare someone

• Delegation would be the way round this

• The VO would assign to the ETICS service the same level of trust it assigns to its user

• The system provides controls to stop misuses


INFSO-RI-223782


Submitting ETICS Jobs to EGEEMixed approach

We will go both ways, and will use the ETICS VOs

• We'll be able to provide resources for those projects that don't have shares on the infrastructures

• Will compete for resources, and likely have low priority

• We'll let VOs use their shares to build and test their software• According to the priority that the infrastructures assign them, and

the VO choices


INFSO-RI-223782


Submitting ETICS jobs to UNICORE

The ETICS project is going to implement a submitter able to send jobs to UNICORE, so to be able to integrate into infrastructures using it (such as DEISA)

• Submits to UNICORE OGSA-BES

• Describe the job using JSDL


INFSO-RI-223782


Conclusions

• The ETICS System provides a grid-based remote build and test functionality

• The current implementation is based on Condor, but generalization and extension to additional infrastructures is required

• The ETICS submission mechanism has been generalized to be able to plugin different middleware implementations

• We are adding support for EGEE and gLite and investigating the possible authorization and usage models based on the VO concept

• Further extensions to UNICORE and other implementations are foreseen

Conference name Company name INFSOM-RI-1234567 Speaker name The ETICS Job management architecture...

Documents

Transcript of Conference name Company name INFSOM-RI-1234567 Speaker name The ETICS Job management architecture...