www.bsc.es

Orchestration of Applications on MultipleClouds with COMPSs

CLOUD COMPUTING 2013 - Valencia, May 28 2013

Clouds with COMPSs

Daniele LezziJavier Alvarez, Rosa M. BadiaJorge Ejarque, Francesc

Lordan, Roger Rafanell, Raul Sirvent, Enric Tejedor

Outline

Overview of COMPSs/ServiceSs

Overview of interoperability approaches withCOMPSs/ServiceSs

Interoperability to cloud middleware through connectors

Use cases & projects

• StarSs

StarSsCellSs

SMPSs

GPUSs

GridSs

ClearSpeedSs

ClusterSs

OmpSs

ClusterSs

COMPSs

• Programmability/Portability– “Same” source code runs on “any” machine

The StarSs programming model

• StarSs

– Sequential C/Fortran/Java +annotations

– Task based

– Simple linear address space

– Support for SMP, GPUs, Cluster, Gridsand Clouds

– “Same” source code runs on “any” machine

– Incremental parallelization/restructure

– Focus in the problem, not in the hardware

• Performance– Intelligent Runtime

– Automatically extracts and exploitsparallelism

– Locality awareness

– Matches computations to specificresources on each type of targetplatform

Open Sourcehttp://compss.sourceforge.net

initialize(f1);

for (int i = 0; i < 2; i++) {

genRandom(f2);

add(f1, f2);

}

print(f2);

Annotatedinterface

T1 T3

Grids

COMPSs Infrastructure

User code

Custom Loader

Javassist

print(f2);T2 T4

GridsClustersClouds

Files

Overview of interoperability approaches withCOMPSs/ServiceSs

Interoperability through web services: ServiceSs– “Tasks” in ServiceSs can be WS

– Whole applications can be exposed as WS

Interoperability throughhigh-level standards– Venus C execution platform

implements OGSA-BEimplements OGSA-BEstandard

– Enables transparent executionof applications

Interoperability to cloud middleware through connectors

COMPSs'Run+me'

Cloud'

Azure'

Job' Storage' Accoun+ng'Adaptors'CDMI' S3'

OpenNebula'

Introduction to COMPSs

OCCI' EC2'

Azure'

Usage'Records'

GAT' GRAM' SSH'

Azure'gLite'

FTP'

gridFTP'

Interoperability to cloud middleware through connectors

The runtime communicates with the Cloud by means of Cloudconnectors

The connectors implement the interaction of the runtime with a givenCloud provider

Connectors abstract theruntime from the particular APIof each provider

This design facilitates theThis design facilitates theaddition of new connectors forother providers.

Middleware interoperability in COMPSs

Task Scheduler– Assigns tasks to VMs or physical resources

– Each VM or resource has its own task queue

Scheduling Optimizer– Checks status of workers

– Can decide• To perform load balancing

• Create/destroy new VMs

Resource ManagerResource Manager– Manages all cloud middleware related features

– Holds information about all workers and about cloud providers

– Scheduler Optimizer sends to the RM requirements about new VM characteristics• i.e., VM that can run 3 tasks of type T1 and 2 tasks of type T2

– Resource Manager, evaluates the cloud providers and chooses the best option• More economic

• The decision can be to open a new private or public VM

– For each Cloud provider, a data structure stores the different available instances(with its features) and the connector that should be used

Middleware interoperability in COMPSs

Cloud Connector– Interface that enables

• Create VM

• Destroy VM

• Cost?

• Time to create?

– When we want to add a new Cloud Provider, we just need toimplement this interfaceimplement this interface

• A special case is an implementation that supports the OCCI standard

– Two type of threads in the Connector• Creation thread

– To create and contextualize a VM

• Deletion thread– To destroy a machine

– Before destroying the VM, TS waits until all tasks assigned finish and FTMmoves remaining files to the master

Federatedclouds

ServiceProvider

InfrastructureProvider

Internalinfra-

structure

Service Provider

Burstedinternalclouds

OPTIMISdeployment

scenarios

InfrastructureInfrastructure

InfrastructureProvider

ServiceProvider

Multi-clouds

ProviderInfrastructure

ProviderInfrastructure

Provider

InfrastructureProvider

InfrastructureProvider

InfrastructureProvider

Broker

The VENUS-C Platform

Goal: “Create a sustainableinfrastructure that enables userapplications to leverage cloudcomputing principles”

e-Science as a Service– 7 Scenarios

– 15 Open-Call Pilots

– 5 Open-Call Experimentscomputing principles”

Funded by EuropeanCommission as FP7 ResearchInfrastructures Projects

Interactive web and trainingchannel: http://www.venus-c.eu/

– 5 Open-Call Experiments

June ‘10 - May ‘12(support until May ‘13)

Free of charge access to Azure

ExecutionEnvironments EMIC Generic Worker BSC COMPSsVENUS-

Arc

hit

rave

Scenario /Algorithm C

OLB

CN

R

AEG

EA

N

UP

V.B

io

Co

SB

I

UN

EW

C++Programming

LanguageC++ Java .NET C++ .NET Java

Parametersweep

Type ofworkload

Batch HTC Data flow WorkflowMap /

ReduceCEP

Users

The VENUS-C Platform

MSFT ENG KTH BSCCloud

Provider Customer

Environments EMIC Generic Worker BSC COMPSsVENUS-C

WindowsAzure

OpenNebula EMOTIVE

Windows LinuxOperating

System

CloudTechnology

IaaSPaaSCloud

Paradigm

…

Windows

BSC

Su

per

Co

mp

ute

r(n

ot

inth

ecl

ou

d)

On

Pre

mis

es

(no

tin

the

clo

ud

)

Infra-structure

EU-BrazilOpenBio

EU-Brazil OpenBio

Combining Biodiversity Science and the Open AccessMovement to deploy a joint European and Brazilian e-Infrastructure of open access resources supporting the

needs of the

biodiversity scientific community.

EU-Brazil Open Data and Cloud Computing e-Infrastructure for

Biodiversity

Further EU-Brazilcollaboration in support

of the biodiversity area &infrastructures

Computingresources& SW platforms EU & Brazilian biodiversity

scientific communitiesData and resource managers &Open Access communityEuropean & Brazilian policy andfunding bodies

Who will benefit fromEUBrazilOpenBio?

biodiversity scientific community.

Two biodiversity usecases

EU-BrazilOpenBio

Biodiversity VRE

BiodiversityUse Case GUI

Use Case API Instances Legacy Clients

InformationSystem

Workspace andStorage Service

ResultVisualisation

Service

SpeciesDiscovery

Service

Use CaseEnactment

Service

Experiment Orchestrator Service

VENUS-CHTCondor OMWS

VENUS-C Cloud

COMPSsWorkflow

Orchestrator

InternalInternalStorage

Condor

InternalInternalStorage

WorkflowOrchestrator

Public clouds(EC2, Azure)

Private clouds(OpenNebula, EMOTIVE)

OMWS Cluster

OMWSServer

Interoperable execution of workflows in EGI Cloud

• EGI Federated Cloud:interoperable integration ofvirtualised resources fromdifferent resource providers toprovide an integrated federatedvirtualised resourcesinfrastructure for exploitation byEGI’s user community.

• Interoperability based on• Interoperability based onstandards

• Different communities samearchitecture.

• COMPSs enables the executionof Taverna workflows thanks tointeroperability features

Evaluation: Elasticity and Bursting

Evaluation: Performance and Scalability

Evaluation: Performance

Hybrid configuration:workload unbalance.When the number ofresources allows a good loadbalancing, the speedup curverecovers (see 32+12)

Execution time (a) and speedup (b) values depending on the number of processors

The speedup keeps aquasi-linear gain.

private private+public(outsourcing)

Conclusions

COMPSs/ServiceSs abstract application developers from theunderlying infrastructure

Provides a PaaS interoperable with different Cloud providers

ServiceSs applications can be offered as SaaS

Interoperability offered at different levels

Thanks for your attention

www.bsc.es/compsswww.bsc.es/compss