EUBra-BIGSEA:cloud serviceswith QoS guarantees for Big

DataAnalytics

IgnacioBlanqueron Behalf oftheEUBRA-BIGSEAConsortium

www.eubra-bigsea.eu|@bigsea_eubr

EUBra-BIGSEA

www.eubra-bigsea.eu |@bigsea_eubr

• AEuropean-BrazilianConsortiumaimingat― ThedevelopmentofQoS andsecurecloudservicestosupportBig

Data.― ThedevelopmentofBigDataservicesforcapturing,federatingand

annotatinglargevolumesofdata.― Theuseofefficient technologies forguaranteeingthefulfilmentof

thesecurity andprivacy policies.― Thetransfer ofthistechnologytoareal user scenario withhigh

socialandbusinessimpact,andofhighinterestforbothEUandBR.

BIGSEAinaNutShell

www.eubra-bigsea.eu |@bigsea_eubr

• BIGSEAis structured along 7(+Projectcoordination)activities defined asWPs.

• User Scenario will make useofthe DataAnalytics APIandthe Programming modelinterface.

• The Programming model interfacewill usethedervices deployed using TOSCAstandard

― Service specifications will beinstantiatedon acloud IaaS,automatically managingQuality ofService.

― The BigData Analytics services will runonthe cloud infrastructure.

• Asecurity framework will bedefined toprovide security andprivacy.

• Globally,dissemination andexploitationactivities will pursue the outreach andtransferoftechnology.

QoS IaaS

BigDataAnalytics

Program.model

interface

Security

UserScenario

Dissemin.&Outreach

InnovationManag.

Accesstokens

Assurance

BigDataPrivacy

Assurance

Accesstokens

TOSCAspecs

Requests

DevelopmentIDE

BIGSEAArchitecture

www.eubra-bigsea.eu |@bigsea_eubr

Mesos

Scheduler

Resources

CloudManag.Framew.

EC3/IM

CLOUDVAMP

Monitor

ProactivePolicies

Program.Model

(COMPSs)

IDE(Lemonad

e)

Creates & executesruntime

Triggers Vertical elasticity

Request and configures VMs

ProvidesAgents

PhysicalResources

LogicalComponents

ManagementofPhysicalResources

Physical Resources

www.eubra-bigsea.eu |@bigsea_eubr

• Resources areprovisioned by aMesos Cluster totheexecution frameworks- COMPSs,Marathon,Spark,Chronos.

• Mesos Cluster is composed ofasetofVMs provisionedfrom aCMF- Automated Deployment- Platform-agnosticism- Vanilla VMIs

• HorizontalElasticity atthe level ofthe resources- The registration ofnewframeworks will check for the availability ofthe needed resources.

- Newrequests will trigger booting upnewresources if needed.- Idlepure computing resources will bepowered off.

• VerticalElasticity atthe level ofthe VMs- Higher flexibility inthe allocation ofphysical resources.

VMRC

InfrastructureManager

CloudSelectorVMRC

CloudConnector

Conf.Manager

FicherosConf&CTX

XML-RPCAPI RESTAPI

WebInterface CLIInterface

MV

MV

MV

...

RADL

VMMasterCntxt.Agent

Ansible

OpenNebula OpenStack EC2 ...

Ansible

VMRC

...

Infrastructure Manager(IM)www.grycap.upv.es/im

www.eubra-bigsea.eu |@bigsea_eubr

• TwoapproachestoVMImanagement- Deployexistingvanillaimages(plainOS)andconfigureinstancesusingtoolstore-createthedesiredconf.• Example:InstantiateaplainUbuntu12.04AMIon

AmazonEC2anduseAnsible toautomaticallyinstallafullLAMP(Apache,MySQL,PHP)stack.

- CreatespecificVMIsfordifferentCloudsfromtemplates.• Deploymentsbasedonrecipes,configurationand

contextualizationservices.

- Enablesplatform-agnosticorhybriddeployments.

VMI catalogue

Platform-agonstic Deployment

www.eubra-bigsea.eu |@bigsea_eubr

LocalComputer

IMserver

EC3client

RADL

credentials

$ docker pull eubrabigsea/ec3client$ docker pull grycap/im

Mesosmaster

HadoopDN

HadoopDN

HadoopDN

Mesosslave

Mesosslave

Mesosslave

Mesosslave

$ ./ec3 launch myMesosHadClus ubuntu-one mesos docker spark nfs hadoop-a /root/auth.dat -u http://localhost:8899

Bound to ONE, but easy to extend by:• Changing base image• Providing credentials

Address of the IM server

Credentials

Platform-agnostic Deployment

www.eubra-bigsea.eu|@bigsea_eubr

• Threeparts:• Cloud-backend(ubuntu-one.radl)

• Threeobjects:front,wn &wnmesos• Fourattributesperobject:name,imageurl,usernameand

password.• ResourceManagementFramework(mesos.radl)

• Definesnetwork:ports,DNSnames,interfaces.• Definessystemvirtualhardware:Memory,CPUs.• Configuration:MasterandSlaves.

• Otherdependencies• docker.radl:On every node.• spark.radl:On every node.• nfs.radl:On every node.• hadoop.radl:on master&wn instances.

Elastic ComputeClusters on the Cloud(EC3)

• EC3(Elastic CloudComputingCluster)leverages CLUEStocreateelastic virtualclusters inthe Cloud.

- Noupfront investment.- Customizable.- Usage-dependent cost.- Automaticelasticity.

Resource-level elasticity

• When anewMesos framework is requested,EC3will check ifthere areenough resources tofulfill the request

- Additionally,Marathon andChronos plugins will capturethe details oftherequest.

- EC3will request the CloudManagementFrameworktodeploy newVMs toallocate the necessary resources for the newframework• EC3usesInfrastructure Manager

toinstall andre-configurethe software.

- When resources become idlefor awhile,they areundeployed.

- Dataresources arenotundeployed.

Mesosmaster

Mesos MasterDocker

ChronosMarathon

SparkHadoop NN

Hadoop2ndNN

NFS ServerEC3

Vnode1Hadoop DN

Spark

Vnode2Hadoop DN

Spark

Vnode3Hadoop DN

Spark

Vnode4Mesos Slave

DockerSpark

mounted home

Vnode7Mesos Slave

DockerSpark

mounted home

158.42.104.229

10.0.0.6710.0.0.70 10.0.0.74 10.0.0.81

10.0.0.XX10.0.0.XX

Verticalelasticity attheresource level

•VMs areallocated from aCMFthat manages ageneral-purpose on-premise cloud• Physical limitations arenormally atthe memory sizerather than atthe CPUshare.

• Aphysical machinecanbeshared among differentdeployments• Both BIGSEAstack or other one.• Or due toincreased isolation.

• Oversubscribing memory canenable reaching highernumber ofVMs perphysical node,not bounding toaspecific partitioning.• AVMfrom one deployment can”release”the freememory toother VMs inthe node.

Infrastructure management

• Interaction with the CLUESclient toget information about theresources powered on /offandtoforce them tobepowered on /off.

• Mainly intended for administration,it canbeused toanticipatepowering on aresource .

/v1/resource/slaves GET method, lists all the resources registered in the master orcluster of masters.

/v1/resource/slaves/slaveid GET method, provides the status information of a specific slave.

/v1/resource/up PUT method, boots up a new resource.

/v1/resource/slaves/slaveid DELETE method, powers down a specific resource.

/v1/resource/slaves/slaveid/enable POST method, enables a disabled resource.

/v1/resource/slaves/slaveid/disable POST method, disables an active resource – it is not considered anylonger to be powered off by the system.

Automated Oversubscription forCloudPlatforms (CLOUDVAMP)http://www.grycap.upv.es/cloudvamp

•Asystem todynamically adjust the physical memoryallocated toVMs tothe realusage.

• It canbeused toallocatemoreVMs than inafixedmemory allocation model.

• It resizes memory inthe VMasneed andeven migratesVMs tonewhosts.

Logical Components

User-level interaction

User’sComputer

github

Docker Hub BIGSEA end-point

BIGSEA LemonadeIDE

Resources

app

Spark / COMPSsGenerated

code

Docker commit& push (3)

Dockerfilecommit & push (3) Automated

build (3)Docker pull

(5) Framework (5)

Submit(incl QoS)

(4)

BIGSEA QoSproactive

Resourcerequest (4)

Docker pull (1)

Characterization (4)

User-level interactionUser’sComputer

github

Docker Hub

BIGSEA end-point

BIGSEA LemonadeIDE

Resourcesap

p

COMPSs / Spark UserCode (2)

GUI (2)

Spark / COMPSs Generatedcode (3)

Docker commit& push (3)

Dockerfile commit & push (3)

Automatedbuild (3)

Docker run (5)

Framework (5)

Submit(incl QoS)

(4)

BIGSEA QoSproactive

Resourcerequest (4)

Docker pull (1)

Characterization (4)

ProgrammWriting

QoS characterization

CustomizingDepend.

ProactiveResourceCharacter.

Submission

• Combining Spark and general code in an IDE.• Using COMPSs for DAG parallelilsation.• Use of pre-characterized building-blocks and algorithms.

• Container images with BIGSEA dependencies (preferred), alsosupporting direct execution (e.g. Spark).

• Automated build or container push.

• Run-time policies optimize resource allocation at initial deployment. • Performance models recalculate QoS metrics (execution time) and

trigger optimization module (if necessary).• API provides system parameter reconfiguration.

• Submission from a single end-point.• Frameworks interacting with the Mesos cluster.• Adding QoS constraints (deadlines, execution time,…) and

previous proactive resources demand characterization.

SingleSubmission end-point

• JSONdocument with the information on theexecution container,framework resources andQoS.

/v1/scheduler POST method, submits a jobrequest in JSON.

/v1/scheduler/jobs GET method, lists all the jobs inthe scheduler (a JSON with all thejob). It can accept a JSON with adeadline that defines the datelimit.

/v1/scheduler/job/jobid GET method, gets all theinformation from a specific job;DELETE method, kills the specificjob; POST method, reallocatesresources.

/v1/scheduler/job/jobid/scale PUT method, changes theresource allocation of aframework.

{ "type": "CMD","name": "my_job_name","deadline": "2016-06-10T17:22:00Z+2","periodic": "R24P60M","expectedduration": "10M""container" : [

"type": "DOCKER","image": "eubrabigsea/ubuntu","forcePullImage": true"volumes": [{ "containerPath": "/var/log/","hostPath": "/logs/","mode": "RW"}],

"portMappings": [{ "containerPort": 8080,"hostPort": 0,"protocol": "tcp" }]],

"environmentVariables": [{ "name": "value" } ],"cpu" : "1.5", "mem" : "512M", "disk" : "1G","command" : "python -m SimpleHTTPServer 8000"

}

QoS Proactive Policies• Policies arebased on historical data(lookuptable)and

current system status(QoS monitoring)• Analytical performancemodels provide newestimation of

quality metrics tothe optimization module(atinitialdeployment andexecution times)

• Optimization modulerecalculates resource allocation,ifnecessary (newapplicationarrival,changeson the systemconfiguration).

Monitoring•UseOpenStackMONASCAasmonitoringframework

•Definemetrics atthecontainer level.

• Triggers newresource allocationrequest ifnecessary.

VerticalElasticityattheleveloftheframework

•Directly managed by the submission service• It hides the different framework features,interactingwith the scheduler and/or Mesos.

•Asexample• InChronos it gets the current configuration ofthe job andresubmits the job with the newresource allocation andthesame configuration.

• InMarathon it updates the json andusesthe specific APIcall$ curl -i -L -H 'Content-Type: application/json' -X PUT -d@”temporary.json" marathonserver:port/v2/apps/appname?force=true

• InSpark andCOMPSs directly interacting with the MesosFramework.

Elasticity scenario

• Two multithreaded processing benchmarksrunningconcurrently inthe same node• Change the allocation from 50%-50%ofCPUto25%-

75%asthe deadline was approaching.

Conclusions

•EUBra-BIGSEAaims atproviding asetofcloudservices to• Facilitate the deployment ofcomplex multi-frameworkBigDatainfrastructures.

• Provide elasticity atthe level ofthe physical resourcesandframeworks.

• Include QoS Constraints.• Facilitate the useofcustomized environments.

•Those cloud services aredirectly used by higher-level programming models hiding the platformparticularities.