SCI-BUS is supported by the FP7 Capacities Programme under contract nr RI-283481

CloudBroker Platform integration into WS-PGRADE/gUSE

Zoltán FarkasMTA SZTAKI LPDS

zoltan.farkas@sztaki.mta.hu

Outline

• Motivation• WS-PGRADE/gUSE• CloudBroker Platform• Integration:

– Architecture, features– DCI Bridge modifications– WS-PGRADE modifications– WFI modifications

• SaaS vs IaaS execution modes

Motivation• There are many user communities who would like to access

compute resources in a transparent way through science gateways

• Cloud resources are getting more and more popular• Clouds are more reliable than grids• Additionally, cloud resource can provide a vast amount of

resources (flexible resource allocation)• A customizable science gateway with cloud support is capable

of satisfying compute needs of complex scientific computations

• Aim: provide access to cloud resources in a transparent way

WS-PGRADE/gUSE in SCI-BUS

The CloudBroker Platform• Easy, scalable, secure, integrable and pay-per-use access to scientific and

technical applications in the cloud• High performance computing application store and marketplace with

accounting and billing• Deployment and execution of applications in the cloud with the same

parameters and files as for local execution• Accessible through the internet via a web browser UI• Integration possibilities into third party tools via web service-based APIs• Public version available under https://platform.cloudbroker.com

• Using infrastructure as a service (IaaS) from cloud providers• Offering platform as a service (PaaS) for software vendors• Providing software as a service (SaaS) to end users• Available compute resources: Amazon, IBM, OpenStack, OpenNebula• Available storage resources: Amazon, Walrus, Rados S3

CloudBroker Platform Architecture

CloudBroker PlatformCloudBroker Platform

AmazonCloud

IBMCloud

…Cloud

ChemistryAppli-

cations

BiologyAppli-

cations

MedicineAppli-

cations

WebBrowser

UI

Engineering Appli-

cations

Web Service API

GenericWorkbenches CloudBroker Integration

CloudBroker Integration

Domain-Specific Gateways

R&D End Users and Software Vendors

CLI

…Appli-

cations

Integrated architecture

Integration features I.

• Slight modification in the fully featured UI• Also available in every customization level,

completely hiding the fact that cloud is being used:– ASM– Remote API– End-user view– DCI Bridge

WS-PGRADEWF

DeveloperUI

gUSE DCI Bridge

DCI 1

DCI 2

DCI n

ApplicationSpecific

User Interface

ExistingApplicationSpecific UI

WS-PGRADEEnd-User

UI

Remote API

BES interface

ASM API

A

B

C

D

E

BES interface

Integration features II.

• Support for commercial clouds with costs (prices configured in CloudBroker Platform):– Estimated job cost before submission– Actual job and workflow cost after execution

Integration features III.IaaS model SaaS model

Enables users to run their own executables Yes No

What has to be pre-deployed in the virtual machine image

A single wrapper application Every application that users would like to use

What has to be configured in the CloudBroker Platform

Only the wrapper application Executables for the different application

Level of security Low: user can run anything High: users can run only pre-registered, tested apps

Ease of use (user’s perspective) Very easy: very similar to existing WS-PGRADE/gUSE interface

Very easy: user simply selects from the pre-defined Software

Ease of use (portal administrator’s perspective)

Easy: only two additional properties have to be set in the DCI Bridge configuration (see previous slide)

Very easy: no need to set additional properties

Ease of use (CloudBroker administrator’s perspective)

Easy: only one application has to be configured for each cloud resource

Hard: a number of applications have to be configured

Easy of use (cloud administrator’s perspective)

Easy: only one VM image has to be deployed Hard: either a number of VM images have to be deployed or one VM image must be updated a number of times.

DCI Bridge modifications I.

• New Configuration tab for CloudBroker

DCI Bridge modifications II.• New CloudBroker plugin to manage CloudBroker jobs• Using slightly extended JSDL

<JobDefinition xmlns="http://schemas.ggf.org/jsdl/2005/11/jsdl" xmlns:ns2="http://schemas.ggf.org/jsdl/2005/11/jsdl-posix" xmlns:ns3="uri:MBSchedulingDescriptionLanguage" xmlns:ns4="extension.dci" xmlns:ns5="http://cloud.extension.jsdl" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://schemas.ggf.org/jsdl/2005/11/jsdl"> <JobDescription> <JobIdentification> <JobName>Workflow Test Applications 1.0</JobName> <JobAnnotation>1f803ac0-8cfd-47de-9b9a-718f2a287f20</JobAnnotation> </JobIdentification> <Application> <ApplicationName>Workflow Test Applications 1.0</ApplicationName> <ApplicationVersion>Workflow Test Applications 1.0 ARInew.sh</ApplicationVersion> <ns2:POSIXApplication_Type> <ns2:Executable>Workflow Test Applications 1.0 ARInew.sh</ns2:Executable> <ns2:Argument>A</ns2:Argument> <ns2:Output>job.out</ns2:Output> <ns2:Error>job.err</ns2:Error> <ns2:UserName>10196</ns2:UserName> <ns2:GroupName>http://192.168.152.73:8080/wspgrade</ns2:GroupName> </ns2:POSIXApplication_Type> </Application> <Resources> <ns5:CloudResources_Type> <ns5:CloudResource> <ns5:ResourceName>OpenNebula MTA SZTAKI SZTAKI</ns5:ResourceName> <ns5:RegionName>OpenNebula MTA SZTAKI SZTAKI SZTAKI OpenNebula</ns5:RegionName> <ns5:InstanceTypeName>OpenNebula MTA SZTAKI SZTAKI SZTAKI OpenNebula x86_64</ns5:InstanceTypeName> </ns5:CloudResource> </ns5:CloudResources_Type> </Resources>

WS-PGRADE modifications I.

• CloudBroker Authentication Portlet

WS-PGRADE modifications II.

• IaaS and SaaS-based configuration

WFI modifications

• Produce JSDL document for DCI Bridge based on workflow XML containing CloudBroker-specific settings

Accessible Cloud Resources

• Access provided by the CloudBroker Platform• Commercial:

– Amazon EC2– IBM

• OpenSource/Free:– OpenStack– OpenNebula

• Currently, within SCI-BUS accessible:– MTA SZTAKI OpenNebula (400 cores)– BIFI OpenStack (50 cores)

Example applications - IaaS

• University-level cloud resource access• Run any application

Example applications - SaaS

• Predeployed application execution• Docking portal: AutoGrid, AutoDock,

AutoDock Vina (should be deployed)

Summary

• WS-PGRADE/gUSE offers production-level cloud access through the CloudBroker Platform

• Mainstream cloud resources can be used (both IaaS and SaaS model is supported)

• Slight UI changes (can be completely hidden for end-users)

• Commercial exploitation available

Questions?

?