D4.4 QoS extensions for the big and fast data eco-system 1 · The EUBra-BIGSEA project is aimed at...

www.eubra-bigsea.eu|[email protected]|@bigsea_eubr 1

D4.4:QoSextensionsforthebigandfastdataeco-system

Author(s)

SandroFiore (CMCC),DonatelloElia (CMCC),CosimoPalazzo(CMCC),WalterdosSantosFilho(UFMG),TulioBraga(UFMG),Carlos Eduardo Pires (UFCG), Demetrio Mestre (UFCG),GiovanniAloisio(CMCC)

Status Draft/Review/Approval/Final

Version v1.0

Date 31/01/2018

DisseminationLevelX PU:Public PP:Restrictedtootherprogrammeparticipants(includingtheCommission) RE:Restrictedtoagroupspecifiedbytheconsortium(includingtheCommission) CO:Confidential,onlyformembersoftheconsortium(includingtheCommission)

Abstract: Europe - Brazil Collaboration of BIG Data Scientific Research through Cloud-CentricApplications (EUBra-BIGSEA) is a medium-scale research project funded by the EuropeanCommissionundertheCooperationProgramme,andtheMinistryofScienceandTechnology(MCT)of Brazil in the frame of the third European-Brazilian coordinated call. The document has beenproducedwiththeco-fundingoftheEuropeanCommissionandtheMCT.The purpose of this document is to provide an overview of the extensions and adaptationsdevelopedforthemaintechnologiesincludedinthebigandfastdataeco-systeminordertoaddressQoS-based elastic scenarios over cloud infrastructures and properly support QoS-constrainedapplications,suchasthosedevelopedinthecontextoftheEUBra-BIGSEAproject.

EUBra-BIGSEAisfundedbytheEuropeanCommissionunderthe CooperationProgramme,Horizon2020grantagreementNo690116.

Esteprojetoéresultanteda3aChamadaCoordenadaBR-UEemTecnologiasdaInformação eComunicação(TIC),anunciadapeloMinistériodeCiência,TecnologiaeInovação(MCTI)


Documentidentifier:EUBRABIGSEA-WP4-D4.4

Deliverablelead CMCC

Relatedworkpackage WP4

Author(s) SandroFiore(CMCC),DonatelloElia(CMCC),CosimoPalazzo(CMCC),WalterdosSantosFilho(UFMG),TulioBraga(UFMG),CarlosEduardoPires(UFCG),DemetrioMestre(UFCG),GiovanniAloisio(CMCC)

Contributor(s) Seeversioningandcontributionhistory

Duedate 31/12/2017

Actualsubmissiondate 31/01/2018

Reviewedby IgnacioBlanquer(UPV),NádiaPuchalski(UFTPR)

Approvedby PMB

StartdateofProject 01/01/2016

Duration 24months

Keywords QoS,cloudelasticity,bigdata

Versioningandcontributionhistory

Version Date Authors Notes

0.1 27/04/2017 SandroFiore,DonatelloElia(CMCC) DraftToC

0.2 28/11/2017 SandroFiore,DonatelloElia(CMCC) RefinedToC

0.3 07/12/2017 Donatello Elia, Cosimo Palazzo(CMCC)

SectionsaboutOphidiaimplementationsforQoS

0.4 07/12/2017 CarlosEduardoPires,DemetrioMestre(UFCG)

EvaluationofEMaaSperformance

0.5 11/12/2017 WalterdosSantosFilho(UFMG) SectionsaboutSparkimplementationsforQoS

0.6 14/12/2017 TulioBraga(UFMG) Section about performance modelintegrationwithSpark

0.7 19/12/2017 DonatelloElia,SandroFiore(CMCC) Executive summary, introduction,backgroundandconclusions

0.8 21/12/2017 DonatelloElia(CMCC) EvaluationofOphidia

0.9 12/01/2018 SandroFiore,DonatelloElia,GiovanniAloisio(CMCC)

Revisionofthedocument

1.0 25/01/2018 Allcontributors Addressed internal reviewerscomments


Copyright notice: This work is licensed under the Creative Commons CC-BY 4.0 license. To view a copy of thislicense,visithttps://creativecommons.org/licenses/by/4.0. Disclaimer: The content of the document herein is the sole responsibility of the publishers and it does notnecessarilyrepresenttheviewsexpressedbytheEuropeanCommissionoritsservices. Whiletheinformationcontainedinthedocumentisbelievedtobeaccurate,theauthor(s)oranyotherparticipantintheEUBra-BIGSEAConsortiummakenowarrantyofanykindwithregardtothismaterialincluding,butnotlimitedtotheimpliedwarrantiesofmerchantabilityandfitnessforaparticularpurpose. Neither the EUBra-BIGSEA Consortium nor any of its members, their officers, employees or agents shall beresponsibleorliableinnegligenceorotherwisehowsoeverinrespectofanyinaccuracyoromissionherein. Without derogating from the generality of the foregoing neither the EUBra-BIGSEA Consortium nor any of itsmembers, their officers, employees or agents shall be liable for any direct or indirect or consequential loss or


TABLEOFCONTENT

EXECUTIVESUMMARY................................................................................................................6

1. INTRODUCTION....................................................................................................................71.1. ScopeoftheDocument...................................................................................................................71.2. TargetAudience..............................................................................................................................71.3. Structure.........................................................................................................................................7

2. BIGANDFASTDATAECO-SYSTEMQoSBACKGROUNDANDMOTIVATIONS..........................8

3. DESIGNOFTHEQoSCLOUD-BASEDEXTENSIONS................................................................103.1. QoS-basedelasticityinOphidia.....................................................................................................10

3.1.1. QoSExtensionsforscalability..................................................................................................................................113.1.2. Automateddeployment.............................................................................................................................................143.1.3. Brokerplugin...................................................................................................................................................................163.1.4. IntegrationwiththeOptimizationService.........................................................................................................19

3.2. QoS-basedelasticityinSpark........................................................................................................203.2.1. QoSExtensionsforscalability..................................................................................................................................203.2.2. Automateddeployment.............................................................................................................................................213.2.3. Brokerplugin...................................................................................................................................................................213.2.4. Monitorplugin................................................................................................................................................................223.2.5. Controllerplugin............................................................................................................................................................223.2.6. Performancemodels....................................................................................................................................................22

4. BENCHMARKING................................................................................................................244.1. Usecase1ondescriptiveanalytics................................................................................................244.2. Usecase2onEntityMatching......................................................................................................26

5. CONCLUSIONS....................................................................................................................29

6. APPENDIX...........................................................................................................................30

7. REFERENCES.......................................................................................................................32

GLOSSARY.................................................................................................................................33 LISTOFTABLESTable1.ExecutiontimeandspeeduprelatedtoOphidiaoperatorsusedinthedescriptiveanalytics..........25Table2.ExecutiontimeandspeeduprelatedtoSpark-basedBULMA..........................................................27


LISTOFFIGURES Figure1.EUBra-BIGSEAplatformarchitecture[R02].......................................................................................8Figure2.OphidiaoveralladaptationsforQoSscenarios................................................................................11Figure3.OverallOphidiaarchitecture[R04]..................................................................................................12Figure4.Outputprovidedbyoph_resumeoperatorshowingtheworkflowprogressrate...........................13Figure5.Outputprovidedbyget_progressPyOphidiamethodshowingtheworkflowprogressrate..........14Figure6.OverviewofanOphidiaclusterdeploymentthroughtheIM..........................................................16Figure7.DiagramofactionsperformedbytheBrokerpluginforOphidia.....................................................17Figure8.ExampleofprocesseddatalogsinCSVformatforSparkjobs.........................................................20Figure9.ExecutiontimeandspeedupofasingleparallelCOMPSstaskbasedonOphidiaoperators..........25Figure10.ExecutiontimeandspeedupforSpark-basedBULMA(using2npartitions).................................27Figure11.ExecutiontimeandspeedupforSpark-basedBULMA(using4npartitions).................................28


EXECUTIVESUMMARYTheEUBra-BIGSEAprojectisaimedatdevelopingasetofcloudservicesempoweringBigDataanalyticstoeasethedevelopmentofmassivedataprocessingapplications.EUBra-BIGSEAdevelopedmodels,predictiveand reactivecloud infrastructureQoS techniques,efficientandscalableBigDataoperatorsandaprivacyand quality analysis framework, integrating several programming environments. EUBra-BIGSEA aims tocover general requirements ofmultiple application areas, although it will showcase in the treatment ofmassiveconnectedsocietyinformation,andparticularlyintrafficrecommendation.

Thisdeliverabledescribesthedesignandthe implementationdetailsregardingthebigandfastdataeco-system extensions and adaptations for Quality of Service (QoS) guarantees. It focuses on the maintechnologiesinvolvedintheeco-system(i.e.,OphidiaandSpark)andtheextensionsandimplementationstosupportautomateddeployment, jobmonitoring,elasticity,aswellastheintegrationactivitiesofthesetechnologies with the QoS infrastructure developed at the level of WP3. The extensions support dataprocessing applications with QoS deadlines by dynamically adjusting the resources available for the jobexecution.

Thesolutionspresentedinthedocumentcapitalizeontheplugin-basedarchitectureimplementedbyWP3toprovideareactiveanddynamicPaaSenvironmentovercloud-basedresources,capableofhandlingBigDataapplicationswithQoSconstraints,minimizingresourceusageatthesametime.

Moreover, the document provides some usage examples and relevant code snippets related to theseextensions,alongwithabriefevaluationofacoupleofWP4implementationsbuiltontopoftheadaptedbigdata technologies tohighlighthowtheresourceavailabilitycan impactontheexecutiontimeand, inturn,affectQoSconstraints.


1. INTRODUCTION

1.1. ScopeoftheDocumentThis document provides an overview of the extensions and adaptations developed for the maintechnologiesincludedinthebigandfastdataeco-systemtosupportdynamicandelasticQualityofService(QoS)-basedscenarioson IaaS. Inparticular, itprovidesadetaileddescriptionof theapproachesdefinedandimplemented,alongwiththeintegrationstrategyofthedataprocessingtechnologieswiththemodulesand services developed byWP3 to support elasticity of the infrastructure for achievingQoS guarantees,while also presenting some relevant code blocks and usage examples. Moreover, it presents somebenchmark results from the evaluation ofWP4-level applications in order to highlight how the resourceavailabilitycanimpactontheguaranteeofQoSdeadlines.

1.2. TargetAudienceThis report ismainly intended for internaluse, although it ispublicly released. Thedocument targets, inparticular, the technical experts ofWP4 andWP3who developed these features, as well as, of course,thosefromWP5andWP7,whobenefitfromthedescriptionoftheextensionsandtheexamplesshowedinthisdocumenttobuildandsubmit,intheproperformat,theactualjobrequests.

1.3. StructureAfterthisintroduction,Section2ofthedocumentprovidesabriefbackgroundoftheQoSinfrastructureoftheEUBra-BIGSEAprojectanditsimportanceatthePaaSlevel;Section3andSection4respectivelyprovidea detailed report of the implementation aspects to support QoS-based scenarios for themain Big Datatechnologies and an evaluation of the performance of someWP4-level implementations built on top ofthesetechnologies.Finally,Section5drawsthemainconclusionsoftheactivity.


2. BIGANDFASTDATAECO-SYSTEMQOSBACKGROUNDANDMOTIVATIONSThe big and fast data eco-system integrates different technologies and tools to support and run dataanalytics & mining applications and services used by the EUBra-BIGSEA platform users and developers.These systems may not provide an out-of-the-box support for tailored QoS-based scenarios and,furthermore,applicationsmayintegratevariouscomponentsthatneedaspecificmanagementtoaddressthe QoS deadlines. Hence, the systems included in the eco-system, i.e. at the PaaS level, require someadaptationsandextensionsinordertointegratethecomponentsintotheWP3QoSinfrastructureservicesandtoproperlyaddressvariousQoS-basedelasticanddynamicscenarios.

Fromthepointofviewofformalrequirements,asdepictedinD4.1[R01],theexecutionrequirementsaremainlyaddressedbyWP3,attheIaaSlevel,whilethedataplatformatthePaaSlevel,whichreliesontheQoSinfrastructuredefinedanddevelopedbyWP3,providesthenecessarycomponentstoperformBigDataprocessingalongwiththeadaptationsrequiredtobetterintegratethemintotheQoScloudinfrastructuremodulesandcomplywiththerequestedQoSdeadline.

The WP3 architecture provides the services and mechanisms to support the execution of Big Dataapplicationsbyprovisioningtheunderlyingresources, reducingtheexecutioncostsanddealingwithQoSguarantees. The proposed modules support monitoring, horizontal and vertical elasticity, as well ascomplexinfrastructuremanagementinordertominimizetheexecutioncost,whilemeetingdeadlines,bydynamicallyallocatingandscalingtheresources.

In particular, among these modules it provides: a plugin-based broker API for job submission, anoptimizationservicetoestimatetherequiredinitialconfigurationintermsofresourcesforthedeployment,aperformancepredictionservicetoestimatetheexecutiontimebasedonthecurrentconfiguration,apro-active policies module which adapts the running configuration to reduce the resource usage cost andguarantee QoS deadlines, and a configuration and contextualization service to actually deploy and re-configuretheinfrastructure.Figure1showstheWP3platformarchitecturediagramfromdeliverableD3.5[R02].

Figure1.EUBra-BIGSEAplatformarchitecture[R02]


TheextensionsandimplementationsrequiredbytheBigDatatechnologiesmainlyincludethoserelatedto:(i) monitoring of the job execution status and progress, (ii) elasticity of the systems and I/O, and (iii)automationof thedeployment.The following sectionprovidesa summaryof theadaptationsperformedfortheprincipalcomponentsincludedintheeco-systemtotargetQoS-basedscenarios.

A detailed description of the final QoS reference architecture, including the infrastructure serviceimplementationandvalidation, isprovided inD3.5[R02],whilethesupportedservicesandtheexecutionworkflowaredepictedinD3.4[R03].


3. DESIGNOFTHEQOSCLOUD-BASEDEXTENSIONSThis section reports about the implementation aspects regarding the developments, extensions andadaptations of the main technologies included in the big and fast data eco-system in order to addresselasticQoS-basedscenarios. Italsoprovidessomecodesnippetsandexamplesofusagerelated to theseextensions.

3.1. QoS-basedelasticityinOphidiaOphidiaisoneofthemaintechnologiesincludedinthebigandfastdataeco-systemanditisexploitedbythe applications built on top of the EUBra-BIGSEA project platform for data processing to build, forexample,descriptivemodelsforstatisticalanalysis.

Ophidia representsa frameworkthatprovidesa fullenvironment forefficientscientificdataanalysisandmanagementofmulti-dimensionalheterogeneousdatasets.Someoftheframeworkkeyaspectsinclude:

● it provides an array-based storage model along with a hierarchical and distributed storageorganization;

● itimplementsthedatacubeabstractionfromOLAPsystems;● it leverages parallel computing and data distribution, jointly with a native in-memory engine to

performparallelI/Ooperations.

AmoredetaileddescriptionoftheOphidiaFrameworkarchitecture, itsmaincomponentsandthedesignchoicesisprovidedinsomebackgroundwork[R04][R05].

ToaddressQoS-basedscenarios,theframeworkhasbeenextendedinorderto(i)supportthedynamicandautomated deployment of a cluster over the IaaS, (ii) improve the job status monitoring and instancelogging and (iii) allow an easier scaling of the resources at the framework level. Ophidia has also beenintegratedintotheWP3infrastructuretofullysupportQoS-baseddynamicscenarios.

Figure 2 shows a high-level view of how theOphidia framework interactswith the services provided byWP3andhowtheframeworkrunsinthisenvironment.AnAnsiblerolehasbeendefinedtoautomatethedynamic deployment of anOphidia cluster,which ismanaged through the services included in theWP3infrastructure; i.e.theElasticCloudComputingCluster (EC3)[R06]and, inparticular,bythe InfrastructureManager (IM) [R07]. The initial deployment of the cluster is performed through a request issued to theWP3 QoS architecture services (i.e., the broker API); the optimization service provides the initialconfiguration size. Monitoring and horizontal scalability have also been addressed through specificextensionsofOphidia.Dependingontheusecaseanddatarequirements,horizontalelasticitymechanismscan be triggered at the programmingmodel level. After the cluster is up and running, requests can besubmittedtotheclusterfront-end(i.e.,theOphidiaserver),whichinturnsubmitsthejobexecution(singletasks or workflows of tasks) to the rest of the nodes bymeans of the Slurm workloadmanager [R08].Verticalelasticity,whichhasbeenportrayed inpreviousversionofthisdiagram, isnotdirectlypresentedhere for the sake of clarity, since it is not being exploited by the use case considered. However, thesemechanismsare transparent to theOphidia framework thatcan takeadvantage (inotherusescasesandapplications) of the increase of resources, e.g. in terms of memory, without any further change in itsconfiguration. The following subsections provide a better insight into the main extensions related toOphidiaforQoSsupportwithinthescopeoftheproject.


Figure2.OphidiaoveralladaptationsforQoSscenarios

3.1.1. QoSExtensionsforscalability

Beforeanalysing themainadaptations implemented for theOphidia framework,a shortoverviewof themainarchitecturalcomponentsoftheframeworkandtheirgoalisprovidedtobetterunderstandhowtheQoSmechanismshavebeenintegrated.

AnOphidiainstanceconsistsofmultiplecomponents:

● theOphidiaServer,a front-endserverwithmultiple interfaces toaccess the instanceandsubmittheexecutionofanalyticstasksorworkflows.Italsomanagesjobsschedulingandmonitoring;

● the Analytics Framework, providing a wide set of MPI-based operators executed over thecomputationalresources(i.e.,multiplecomputenodes);

● asetof I/Oserversperformingoperationsoverthedatapartitionedonthestoragelayer.Varioustypes of I/O server, also including a native in-memory kind, can run overmultiple I/O nodes fordistributedaccessandprocessing.

ThelatestOphidiaarchitectureisportrayedinFigure3.Moredetailsregardingtheoverallarchitectureandthesinglecomponentsareprovidedin[R04].


Figure3.OverallOphidiaarchitecture[R04]

IntermsofextensionstosupportQoS-basedelasticscenarios,somemodificationshavebeenperformedinthesystemcatalog(calledOphidiaDB)managementsystemanditsschematosupporttheOphidiaclusterhorizontal elasticity. In particular, the extension allows to easily identify the candidate nodes forundeployment by keeping track of the number of data cubes managed by each I/O server - i.e. thosewithout data can be safely removed. The resource manager has also been improved by exploiting thepreviousextension, inorder tobetterbalance the jobexecutionbasedon the I/Oserverworkload.Withthis new schema, jobs are actually scheduled in a more dynamic and balanced fashion, since they areassigned based both on the availability of computational resources and on the number of data cubesalready loaded in the I/O servers (disks or memory), allowing new nodes to be quickly exploited forcomputation.

Moreover, the Ophidia I/O server, the native array-based server for in-memory processing, has beenstronglyimprovedtoprovidebetterdecouplingamongtheI/Ofeaturesfromthestoragelayer.

Jobexecutionmonitoring&logginginformationhavebeenveryvaluableregardingthelinkwiththeWP3services, with special regard to the initial infrastructure setup for the big data applications (e.g. CityAdministrationDashboard).Tothisend,theOphidiaServer,namelythefront-endofaclusterinstance,hasbeenextendedtobothprovideadditionalinformationaboutthesubmittedjobs(aswellasworkflows)andimprovethelogging&monitoringfeatures.


Themonitoringinformationincludedattheleveloftheserverlogare:

● numberofactiveworkflows(aworkflowisactiveifithasbeenacceptedanditsexecutionhasnotended);

● numberofpendingworkflows(withnorunningtask);● numberofrunningworkflows(withatleastonerunningtask);● numberofworkflowswaitingforuserinteraction;● numberofincomingworkflowsduringthelastmonitoringperiod;● numberofactivetasks;● numberofpendingtasks;● numberofrunningtasks;● numberofmassivetasks;● numberofsubmittedtaskssinceservicestarttime;● numberofactiveusers(auserisactiveifthereexistsatleastoneactiveworkflowsubmittedbythe

user);● numberofbusycores.

Inaddition,foreachactiveworkflowtheOphidiaserverkeepstrackofthe

● progressratio,i.e.thepercentageofcompletedworkflowjobs,and● numberofcompletedtasks(successfullyornot)

Themetrics are loggedby theOphidia server periodically after amonitoring periodwhose duration is aconfigurableparameter,setto1secondinthecurrentimplementation.

Toallowaneasyaccesstothemonitoringinformationbyclient-sideapplications,theoph_resumeoperatorhasbeenextendedtoprovideinformationregardingthejob(orworkflow)statusanditsprogress,suchas:

● status,i.e.iftheworkflowisrunning,completedorfinishedwitherror;● progressratio(asdefinedabove);● creationdate,i.e.thedateandtimewhentheworkflowhasbeensubmitted.

Anexampleoftheoutputprovidedbythisoperator(fromtheOphidiaTerminal)isshowninFigure4.

Figure4.Outputprovidedbyoph_resumeoperatorshowingtheworkflowprogressrate


Additionally,themethodget_progress(jobid)hasbeenincludedinPyOphidia,theOphidiaPythonbindings,todirectlymonitorthejobprogressinPython-basedapplications.AnexampleofexecutionofthismethodisshowninFigure5.

Figure5.Outputprovidedbyget_progressPyOphidiamethodshowingtheworkflowprogressrate

The source code of the extensions is included in various Ophidia components -Ophidia Server, OphidiaAnalyticsFramework,Ophidia I/OServerandPyOphidia -allavailableon theBIGSEAorganizationGithubrepositories(https://github.com/eubr-bigsea/).

3.1.2. Automateddeployment Fromaninfrastructuralpointofview,theOphidiacomponentsintroducedintheprevioussectionscanbedeployedindifferentfashions,suchas:

1. One-tier:asinglenodeisusedforallthecomponents;2. Two-tiercluster:aservernodeisdedicatedforthefront-endandasetofnodesareusedtohost

boththeframeworkandtheI/Oservers(aconfigurationcalledsuper-nodewherecomputeandI/Ocomponentsaredeployedonthesamephysicalnode);

3. Three-tier cluster: a servernode is dedicated for the front-end,oneormore computenodesareusedfortheframework,whereasoneormoreI/OnodesareusedfortheI/Oservers.

Inthetwolattercases,thestorageresourcesshouldbesharedamongthevariousI/Onodesfordistributedaccessandprocessing.

InthecontextoftheEUBra-BIGSEAproject,thedeploymentisperformedaccordingtothesecondschemapresented, which guarantees a good balance between the scalability of the instance and deploymentsimplicity.Inparticular,aspreliminaryreportedinmilestone“MS9:Secondreleaseofthebigandfastdatacloud platform with preliminary security, privacy and QoS-based extensions”, an Ansible role has beenimplemented to automate the deployment of an Ophidia cluster. The role includes the tasks toautomaticallysetupandcontextualizethenodesandall therequiredservices,aswellastoconfigurethesecurityrulesforthecommunicationbetweenthenodes.

TheroleisavailableonGithubathttps://github.com/OphidiaBigData/ansible-role-ophidia-clusterandalsoonAnsibleGalaxyathttps://galaxy.ansible.com/OphidiaBigData/ophidia-cluster/.

Basedontheschemadescribedbefore,theroleallowsthecreationoftwotypesofnodes:

● Server node: it contains the Ophidia Server, the Slurm scheduler controller, the NFS server, thesystemcatalogandalltheotherless-elasticcomponentsofanOphidiacluster.ForagivenOphidiacluster,asingleservernodeisrequired,servingasthefront-endtoaccesstherestofthesystem.Hence,thisnoderepresentsastaticcomponentinthecontextofacluster;

● I/O-computenode(s): it runsall theother components required for theexecutionof theOphidiaoperators:i.e.theOphidiaAnalyticsFramework,theSlurmdeamon,theNFSclient,theI/OserversandtheOphidiaPrimitives.Multiplenodesofthistypecanbedeployedbasedontheworkloadofthecluster.


Tospeedupthedeployment,sincetherolecouldbeexecutedmultiple timesto increasethenodesofacluster,ithasbeensplitintotwocomplementaryphasestoseparate:

● Installationstage:theslowerstageincludingallthestepsrelatedtosoftwarebuilding,installationandmore static configurations,aswell asOSandpackagesupdate.This stepcanbeexecuted inadvancetocreatea“goldenimage”,whichcanbethenreusedforsubsequentdeploymentsoftheinfrastructure;

● Configurationstage: fasterstageconsistingofall thestepsrequiredtoconfigureandstart-uptheservicesonapre-installedVMI, also including the contextualizationand the checks toadjust thelocal node setup based on the number of nodes included in the cluster and the user-specificparameters,suchaspasswords,certificates,etc.

Thesetwophasescanalsobeexecutedtogetherinasingledeploymentrequest.

To deploy a cluster, anAnsible playbook [R09], specifying the infrastructure and configuration variables,must be created. The playbook is required by Ansible to orchestrate the deployment of the cluster. AnexampleofplaybooktodeploytheserverandI/O-computenodeisprovidedinthesnippetbelow:

- hosts: oph-server pre_tasks: - name: Creates NFS shared directory file: path=/data state=directory owner=root group=root roles: - { role: 'OphidiaBigData.ophidia-cluster', node_type: 'server', deploy_type: 'complete', server_hostname: "{{ansible_hostname}}", io_hostnames: "{{ groups['oph-io']|map('extract', hostvars, 'ansible_hostname')|list }}", io_ips: "{{ groups['oph-io']|map('extract', hostvars, 'ansible_default_ipv4')|list }}", server_ip: "{{ ansible_default_ipv4.address }}" , nfs_subnet: 'oph-*', mysql_subnet: 'oph-%'} - hosts: oph-io roles: - { role: 'OphidiaBigData.ophidia-cluster', node_type: 'io', deploy_type: 'complete', server_hostname: "{{ hostvars['oph-server']['ansible_hostname'] }}", io_hostnames: "{{ groups['oph-io']|map('extract', hostvars, 'ansible_hostname')|list }}", io_ips: "{{ groups['oph-io']|map('extract', hostvars, ['ansible_default_ipv4', 'address'])|list }}", server_ip: "{{ hostvars['oph-server']['ansible_default_ipv4']['address'] }}", nfs_subnet: "oph-*", mysql_subnet: "oph-%" }

The Ansible role has also been designed to provide operations to automatically adjust the nodesconfigurationandsupporthorizontalelasticityatdeploymenttime;i.e.scalingupordowntheclustersizebyaddingor removing I/O-computenodes.This reconfiguration isperformedatdeploymenttimeontheVMsthatarestillupandrunning.Inparticular,asetofprocedures(i.e.,Ansibletasksorbashscripts)havebeenimplementedtocheckthenodestatuseverytimetheroleisre-executed,sothat:

1. the server node deletes from the central configuration (in the OphidiaDB) all the IP addressesrelatedtoI/O-computenodethatarenolongeravailableinthecluster;

2. eachrunningI/O-computenoderegistersitsIPaddressontheservernode(ontheOphidiaDB).


Besides this, the Slurm resource manager is also restarted during the re-deployment of the cluster toupdatetheavailabilitystatusoftheprocessingnodes(i.e.,I/O-computenodes).

Additional information regarding the Ansible role, the dependencies and its usage can be found in theGithubrepository.

In the context of the EUBra-BIGSEA project, the Ansible role is used for the deployment, setup andundeploymentof aVMI-basedvariable-sizedOphidia cluster. The InfrastructureManager (IM) isused toperform the deployment, configuration, contextualization and monitoring of the VMs composing thecluster. IM is one of the physical resource management components of the EUBra-BIGSEA platformidentifiedatthelevelofWP3anditrepresentsatoolthateasestheaccessandtheusabilityofIaaScloudsbyautomatingtheVMIdeploymentandmanagement.Figure6describestheOphidiaclusterdeploymentschemathroughtheIMonacloudinfrastructure.ThecontextualizationofthetwonodetypesisperformedbyexploitingtheAnsiblerolefortheOphidiacluster.

Figure6.OverviewofanOphidiaclusterdeploymentthroughtheIM

IMprovidesseveralmeanstodeployaninfrastructure:aweb-basedGUI,acommandlineclient,oranXML-RPCorRESTAPI.The followingexampleshowshowtodeployanOphidiacluster through the IMPythoncommandline(basedonXML-RPCAPI):

./im_client.py -u https://localhost:8888 -a auth.dat create ophidia-cluster

After thedeployment is completed, theOphidia Terminal or other applications exploiting PyOphidia caninteractwiththenewlycreatedclusterfront-endnode.

3.1.3. Brokerplugin TheBroker is oneof themain components of theWP3 architecture and it is used as the entry point tosubmitjobrequestsandtoprovideinformationregardingtheapplicationconfigurationandQoSdeadlines.It canbeeasilyextended throughplugins inorder tomeet the specific requirementsofapplicationsandinfrastructurecontextsasreportedinD3.4[R03].

A specific plugin for Ophidia service has been developed to allow the execution of Ophidia-basedapplicationsoveramulti-nodeclusterdeployedontopoftheIaaS.ThedeploymentoftheOphidiamulti-nodeclusterisautomated,asdescribedinSection3.1.2,bymeansoftheAnsiblerole,whereastheactual


deployment, configuration and contextualization are performed with the IM. The code of the plugin isavailableonthegithubrepository:https://github.com/bigsea-ufcg/bigsea-manager/.

Thepluginhasbeen implemented followinga loosely coupledapproach inorder toabstract theOphidiaexecutionfromthespecificapplicationandprovideamorereusablemechanismforQoSdeadlines.Tothisend,whenthe request foranapplication is submitted, thepluginonly takes intoconsideration the tasksrelatedtoOphidiaoperatorsfortheQoSguarantees.Thestepsperformedbythepluginaresummarisedinthefollowingpicture.

Figure7.DiagramofactionsperformedbytheBrokerpluginforOphidia

Figure 7 shows the procedure implemented by the plugin in order to deploy,monitor and undeploy anOphidia cluster. The plugin is written in Python language and performs the following sequence ofoperations.

1. LetusassumethattheusersubmitsarequesttotheBrokerAPIbysendingaJSONfileinthisway:{ "bigsea_username": "user", "bigsea_password": "pass", "opportunistic": "True", "cluster_size": "1", "auth_file": "...auth_string...", "xmlrpc_url": "http://imserver:8899", "xmlrpc_ssl_ca_certs": "/tmp/pki/ca-chain.pem" }

This request includes: the user credentials to access the Broker (bigsea_username andbigsea_password);aproposalforthenumberofI/Onodestobedeployed(cluster_size),validonlyincaseopportunistic is set to“True”; theauthorizationstring (auth_file) toaccesscloudservices;the entry point of IM (xmlrpc_url). The authentication string is a list of credentials: each item is


relatedtoaspecificservice:e.g.OpenNebula(thecloudplatform),IM,etc.AnexampleofthestringusedtodeploytheOphidiaclusterhasthefollowinglines:

id = one; type = OpenNebula; host = osenserver:2633; username = user; password = pass id = im; type = InfrastructureManager; username = user; password = pass

Therequest(e.g.thefilerequest.json)canbesenttotheBrokerwithinaPOSTHTTPmessageasinthenextexample:

curl -H "Content-Type: application/json" \ -X POST --data @request.json \ http://broker:1514/manager/execute

TheBroker replieswitha request identifier$APP_ID like“ophidiaX”,whereX isanumber,whichtheusercanusetorefertothatrequest(seestep7).

2. The Broker processes incoming request and contacts the Optimization Service to retrieve theoptimal cluster size (i.e., thenumberof I/O-computenodes tobedeployed): this represents thenumberNof I/Onodestobedeployed.Section3.1.4providesmore informationaboutthisstep.Note that, in case of opportunistic setting, the effective cluster size N is the maximum valuesbetweenthevalueproposedbytheuserandtheoptimalvalue.

3. TheBrokerformatstheApplicationDescriptionfiletobesubmittedtoIMbyreplacingthenumberNofI/Onodestobesetupandotheruser-specificinformationinatemplatefile:e.gthepasswordassigned to the superuser (root) of the virtualmachines (VMs) tobedeployed, theOphidiauseraccount, etc. These user-specific parameters are configurable arguments of the plugin, retrievedfrom the user request or from the Broker configuration accordingly. The template for theApplication Description file is defined according to the description provided in Section 3.1.2. AnexampleofafullApplicationDescriptionfileisprovidedintheAppendix(Section6).

4. TheBrokersubmitsthisApplicationDescriptionfiletoIMbyusingaRemoteProcedureCall(RPC)namedCreateInfrastructure.ThisprocedureisimplementedasamethodofServerProxyclass:thismodule,thattheplugincreatesaninstanceof, implementsthecommunicationprotocolbetweenBroker and IM,exploitingXMLmessageexchanges. TheServerProxy class ispartof the IMclientmodule, a python tool available at https://github.com/grycap/im-client. TheCreateInfrastructureprocedure, which ends before the deployment begins, returns an infrastructure identifier inf_idthatcanbeused(bytheBroker)torefertotheclustertobedeployedinnextcalls.Then,IMstartsthedeploymentofthecluster.

5. TheIMstartsthedeploymentofN+1virtualmachines,onefortheOphidiaServernodeandNforOphidiaI/Onodes,basedontheruleslistedintheApplicationDescriptionfile.

6. Aftersubmittingtherequestforclusterdeployment,theBrokerwaitsfortheclustertobecreated.During this period the Broker polls IM periodically, by executing the RPC calledGetInfrastructureStateandusingtheidentifierinf_idobtainedatstep4,tochecktheclusterstate.Theprocedurealso returns a listwith the stateof eachVM.Thepollingperiod is a configurableparameterofthepluginandithasbeensetto30secondsinthecurrentimplementation.

7. When the cluster configuration ends, the Broker retrieves the IP address of the VMwhere theOphidiaserver isrunning(this is theOphidiaentrypoint)andwrites itontheapplication log, forexample:Success 192.168.0.1


TheRPCGetVMPropertyisusedtorequesttheaddressasfollows

GetVMProperty(inf_id, 0, "net_interface.1.ip", auth_data)

where inf_id is the infrastructure identifierobtainedatstep4andauth_data is theauthorizationstring set by the user, whereas net_interface.1.ip is the property name identifying the publicnetworkinterface(asdefinedintheApplicationDescriptionfileinSection6).TheuserisabletoretrievetheOphidiaentrypoint(IPaddressoftheServernode)bysendingaGETHTTPmessageasinthenextexample:

curl -X GET http://broker:1514/manager/logs/execution/$APP_ID

where$APP_IDisthereferencetorequestsentatstep1. 8. TheBrokerperiodicallycheckstheOphidiaServerstatebyexecutingtheOphidiacommandnamed

oph_service: it is assumed that the server state is “up” during the working session, whereas itbecomes“down”whentheuserclosestheworkingsession.ThepluginexploitsPyOphidialibrarytosubmittheoph_servicecommand.

9. WhentheOphidiaServerstateis“down”,theBrokerstartstheclusterundeploymentbycallingtheRPCDestroyInfrastructure.

Incaseoferrors,theprocedureisclearlystoppedand,ifnecessary,theclusterisautomaticallyundeployed(usingtheRPCDestroyInfrastructure).AnerrormessageisalsosavedinthelogfileinsteadoftheOphidiaServernodeIPaddress(seestep7)toinformtheuserabouttheerror.

It is important to note that monitoring and dynamic scaling at cluster run-time is not handled by theproposed approach at the Ophidia level, but rather at the programmingmodel level. Indeed, once theinitial sizeof thecluster isdefinedaccording to theestimationprovidedby theoptimization service, theperformancetuningtoreachfasterthedeadlineincaseofdelay,iscarriedoutbyscalingupthenumberofnodesattheprogrammingmodel level(e.g.COMPSs) inordertotriggermoreparallelismontheOphidiaI/Oservices.

3.1.4. IntegrationwiththeOptimizationService Asstatedintheprevioussection,theOphidiaclustersetupisnotupdatedafterthefirstdeployment,butreliesontheprogrammingmodelleveltoincreasetheparallelismonOphidiaandapplicationspeedup.Asthe example, this approach has been implemented in the case of the City Administration Dashboardapplication.

Undertheseconditionsitiscrucialtodefineanadequatestartingsetupofthecluster,whichcanguaranteeQoS deadlines andminimize the resource usage cost at the time of the initial deployment. To this end,several runsof theCityAdministrationDashboardapplication, asa test case,havebeenperformedwithvariousconfigurationsof I/Onodesandcoresandtheexecution timeof theOphidiaoperatorshasbeenisolatedandextracted.Thecodeoftheapplicationhasbeenextendedtointegrateloggingcapabilitiestomeasure the execution time of themost significant operations. This information has been then used tobuildaperformancemodeltopredict,atinitialdeploymenttime,themostsuitablenumberofI/Oserversto include in the cluster. The Optimization Service, according to this performance model, provides theresource specification to theOphidiaBroker’splugin,which sets thenumberofnodes in theapplicationdocumentfile(Section6).TheevaluationoftheexecutiontimebasedontheclusterresourcesavailabilityisexplainedinSection4.1.


3.2. QoS-basedelasticityinSparkApache Spark1 is an external tool, maintained by Apache Software Foundation; according to itsdocumentation, it “is a fast andgeneral-purpose cluster computing system. Itprovideshigh-levelAPIs inJava,Scala,PythonandR,andanoptimizedenginethatsupportsgeneralexecutiongraphs.Italsosupportsa rich set of higher-level tools including Spark SQL for SQL and structured data processing, MLlib formachinelearning,GraphXforgraphprocessing,andSparkStreaming”.Duringtheanalysisoftoolsforthebig and fast eco-system, Apache Spark was chosen thanks to its vast number of implementedmachinelearningalgorithms, the support to streamingprocessingand its integrationwith job schedulers, suchasApache Yarn and Apache Mesos; the later is the job scheduler defined in the deliverable D3.5, QoSinfrastructureservices[R02].Figure1,inSection2,showstheEUBra-BIGSEAplatformarchitecture,whereSparkisintegratedwiththedifferentinfrastructurecomponents.

3.2.1. QoSExtensionsforscalability

Thissectionmainly relates to theHybridMachineLearningModule (HML) implementation.TheHML isatool used for the performance prediction of Big Data cloud applications. Themachine learningmodel isused by the optimization components [R10] in order to define the minimum cost infrastructureconfigurationwhich fulfils the deadline constraints. In particular, this component is basedon the hybridmachine learning approach initially published in Ataie et al. [R11] and described in EUBra-BIGSEADeliverableD3.2,Section5.5[R12],whichcombinesanalyticandmachinelearningmodelswhilebeingabletoidentifyaccuratemodels,thusreducingthenumberofexperimentsforbigdataapplicationprofiling.

ThefirststepforusingHMLinacontextofSparkdeploymentistocollectandprocesstheSparklogsandgenerate theperformancevalues fromananalyticalmodel (e.g.,approximated formulaordagSim2), thatarenecessaryasinputforthetrainingoftheHybridalgorithm,whichisthecoreoftheimplementedtool.InordertoparseSparklogs,atool,calledSparkLogParser3,wascreated.

Figure8.ExampleofprocesseddatalogsinCSVformatforSparkjobs

AnexamplefromtheprocessedlogsgeneratedbytheLogSparkParserisshowninFigure8.Inparticular,thegeneratedsummaryCSVcontainsasetoffeatureswhichcancharacterizeSparkjobsobtainedduringtheexecutionofaquery.Thosefeaturesincludejobcompletiontime,thenumberofstagesandtasksper

1https://spark.apache.org2https://github.com/eubr-bigsea/dagSim3https://github.com/deib-polimi/Spark-Log-Parser


stage,themax/minandaveragetimeoftasks,Shuffletimeandbytessent.Moreover,thenumberofusers,theapplicationdatasetsizeandthenumberofcontainersare included.MoredetailsaboutHMLcanbefoundindeliverableD3.5[R02].

3.2.2. Automateddeployment

AutomateddeploymentofSparkcanbedoneindifferentways.Inascenariowheretheuserjustwantstodeploy a instance of Spark, it is possible to do so by using a Docker hub container image as the oneavailableathttps://hub.docker.com/r/p7hb/docker-spark/,whichinstallsApacheSpark2.2.0.

In a scenario requiring a large number of Spark instances to be deployed in a cluster environment, it ispossible to useApacheMesoswith a pre-defined SparkDocker image.During the creation of the Sparksession, the property spark.mesos.executor.docker.image must be defined, referencing thenameofavaliddocker image.Such imagecanalreadybepresent intheMesosagentor itcanbepulledfromDockerHub.TheDockerimageusedmusthaveanappropriateversionofSparkasapartoftheimage.Iftheimagealreadyexistsintheagent,itisnotpulledagain.

Still,inaMesoscluster,itispossibletohaveMesosdownloadSparkviatheusualmethods,bydefiningthepropertyspark.executor.uri for a Spark package hosted at anyHadoop-accessibleURL, includingHTTPviahttp://,AmazonSimpleStorageServicevias3n://,orHDFSviahdfs://.

In the second case, when a custom installation of Spark has to be built, Ansible scripts can be used toautomate the process. Ansible playbooks with roles defined in Ansible Galaxy(https://galaxy.ansible.com/azavea/spark/)havebeenused.

Ascanbeseen,ApacheSparkalreadysupportsautomateddeploymentanditsfunctionalitiesaresufficienttomeettheprojectrequirementswithoutanyfurtheradaptationoradditionalsoftwaredevelopment.

3.2.3. Brokerplugin

Anotherimprovementinthepro-activecomponentwastheimplementationofplugins4forrunningSparkoverMesosinOpenNebulacloudsfortheBroker5,thecomponentresponsibleforexecutingallthestepsintherightsequenceinordertoexecutetheapplicationpreservingtheQoSrequirements.

TheSpark-MesospluginusesaclusterpreconfiguredwithMesosphere[R13]toexecuteandmanageSparkjobs. This plugin obtains the hosts where the application runs (similarly as the Spark-Sahara pluginimplemented in the first year for OpenStack), but with a different cluster provider (OpenNebula in thiscase).Agreat advantagehere is thepossibility touse the currentapproaches for vertical andhorizontalscaling (EC3 [R06], IM [R07], and CLUES [R14]) combined. The execution flow for the Spark-Mesos-OpenNebulascenariois:accessviaSSHtheMesosmasternode;downloadthejobbinaryandinputfiles(ifitisnotstoredinalocalHDFS);executethespark-submitcommandwiththecorrectexecutionparameters;xgetthemasterandslavenodesfromMesosAPIto letthecontrollerserviceknowinwhichVMs itmustact;retrievetheapplicationidfromtheSparkAPI,informthecontrollerandmonitorservicessothattheycanconsulttheprogressoftheapplication'stasksandactintheinfrastructurebasedonthatinformation;finally, close all the services (broker,monitor and controller) after complete execution or in case of anyerror.

4https://github.com/bigsea-ufcg/bigsea-manager/tree/master/application_manager/plugins/spark_mesos5https://github.com/bigsea-ufcg/bigsea-manager/


ToexecuteaSparkjobusingthistemplate6,itwillbenecessarytoprovidesomespecificinformationaboutthe application, deadline (in seconds), job binary location, output, and infrastructure, besides theinformationregardingcontrollerandmonitor. It is importanttohighlightthat inthiscasetheapplicationmustorganizeitsparametersinthisexactorder:[inputfiles][othersarguments].

3.2.4. Monitorplugin

Themonitor plugin for Spark-Mesos is pretty similar to the Spark-Sahara. The only difference betweenthemisthatinSpark-Mesos,theAPIconsultismadeviaSSHtotheMesosmasternode,toavoidlimitationswiththenetworktopologyandbeabletodeploythebrokerserviceinadifferentplaceoftheinfrastructureused to deploy the application. This plugin uses Monasca (the OpenStack’s monitoring-as-a-service).Basically, the monitor plugin executes the steps in the following order: call the Spark API and get thenumber of total and complete tasks until that exact moment and calculate the progress proportion(complete_tasks/total_taks); calculate the proportion of the time spent from the beginning of theapplication based on the deadline described in the submission (time_spent/deadline); calculate theprogress error (propotional_progress/proportional_time); parse the timestamp to a Monasca friendlyformat;andfinallypublishthemetrics inMonasca.Thisflowisrepeatedatatimeintervaldefinedintheserviceconfiguration.

3.2.5. Controllerplugin

TheControllerplugin ispartof a servicealso called controller7 inWP3deliverables [R02,R10]. TogetherwiththeActuatorandMetricSource,theControllerplugincomposesthecontrollerservice.TheActuatoractsathypervisor layer,providing theabilityofchangingCPUand I/Omaximumamountofutilization. Ithas a direct effect over Spark, but it is not a direct integration. TheMetric Source is used by the otherplugins to retrievemetrics fromtheMonitor service.Again, it isnotadirect integration:SparkpublishesmetricstotheMonitorservice(Monasca)andMetricSourcereadsthem.

Actually,thereare3versionsoftheControllerplugin:● Thereferencecontrollerplugin (Progresserror) isbasedon fixed-stepactuationswhenthere isa

progresserrorasdetailedindeliverableD3.4[R03];● Proportional Controller: changes the number of allocated resources based on the error values it

receives from themetric source and a proportional factor (also known as the proportional gain)receivedasaparameter.Theactuationsizeisproportionaltotheerrorvaluereceivedandtotheproportional factor. The higher the error absolute value, the higher the number of added orremovedresources[R02];

● Derivative-proportional Controller: changes the number of allocated resources based on thecurrent error value, the difference between the current and the last received value and twoparameters:theproportionalandderivativefactors.Theactuationsizeisproportionaltotheerrorvalueandthereceivedfactors[R02].

3.2.6. Performancemodels

WefocusontwoperformancemodelsforpredictingtheresponsetimesofBigDatacloudapplications: i)the Task Precedence [R15], and ii) the dagSim simulation tool. Both approaches consider the parallel

6https://github.com/bigsea-ufcg/bigsea-manager/tree/master/client/mesos7https://github.com/bigsea-ufcg/bigsea-controller


applicationsasadirectacyclicgraph(DAG)representingthetask'sprecedencerelationship.Inadditiontoit, thefirstapproachmodelsthesystemresources(e.g.,processingcores)asservicecenters inaqueuingnetwork.TasksareallocatedtotheservicecentersbasedonaFCFSpolicy.Tocomputethemodel inputsmatrices, the implemented tool8 parses previous execution logs to obtain the resource demands, taskresponses,and taskoverlapprobabilities.The tooloutput is thepredictedresponse time inmilliseconds.The second approach is a tool9 developed as a part of EUBra-BIGSEA, based on a classic discrete eventsimulationalgorithmanddesignedforhighperformance.ItspecificallytargetsDAGmodelsandconsistsofthreemain components: i) a parser, ii) a simulation component, and iii) an outputmodule. It also usespreviousexecutionlogstoserveas inputdataforthesimulation.BothmodelsaredetailedinDeliverableD3.2[R02],Section5.3.

8https://github.com/eubr-bigsea/lundstrom9https://github.com/eubr-bigsea/dagSim


4. BENCHMARKINGThis section presents some results from the execution of twoWP4 implementations (e.g. services andalgorithms)exploitingBigDatatechnologies, likeOphidiaandSpark.Thepurposeof thisbenchmark is toshowhowtheresourceavailability(i.e.numberofnodesorexecutors)canimpactontheexecutiontime.This information is critical for the assignment of the resources in order to properly address the QoSdeadlines.

4.1. Usecase1ondescriptiveanalyticsTheCity AdministrationDashboard application exploits severalOphidia operators to performdescriptiveanalyticsmainlyonbuscarddata,asdescribedindetailinD4.2[R16],andrepresentsagoodcandidatefortheevaluationofsomeresultsconcerningtheanalyticsrunwithOphidia.TheevaluationaimsatidentifyingtheexecutiontimeobtainedwithdifferentconfigurationsoftheOphidiacluster(i.e.,numberofI/Oserversandcomputecoresavailable)inordertoprovidetheinformationrequiredforthebasisoftheperformanceprediction model. The model will then provide the resources required to be allocated for the clusterdeploymentwithrespecttotheQoSdeadlines.

To isolateandextract theexecutiontimefromtheapplication, thecodehasbeenextendedtoallowthesequentialexecutionoftheoperators(i.e.,withoutCOMPSs)andtologtheexecutiontimeoftheOphidiatasks.Tomodeltheexecutiontimeforahighworkloadscenario,an inputfilederivedfromtherawdatarelatedtoCuritibaandcoveringarelativelylongtimerangehasbeenconsidered(around3monthsofinputdata–65dailyfiles,from1/5/17to18/7/17-NetCDFfileimportedinOphidiaofabout3.1GB).

TheevaluationhasbeenperformedontheCMCCtestbedinfrastructure(5nodeswith20coresand256GBofmemoryeach),reportedinD4.2[R16],usingavariablenumberofnodesandcores.Allcomputationhasbeen performed using the in-memory I/O server, while scaling the number of I/O nodes (and computecoresatthesametime)-from1to5.Multiplerunsforeachconfigurationhavebeenperformedandtheaverage execution time in seconds and the speedup have been computed, as reported in the followingtableandlinechart(Figure9). Inparticular,oneof themostarticulatedaggregationstatistic, thecomputationof thehourlyaggregatedstatisticsforeachbuslineandweekday(e.g.allMondays,Tuesdays,etc.),hasbeenevaluatedforthetest.Inthiscase,theoperatorsthatcanbeexecutedinparallelandcanhenceslowdown/speedupaccordingtotheavailableresources,are:(i)afirstimportoftheinputfile(a3.1GBNetCDFfilederivedfromtherawbuscarddata),(ii)apreliminaryaggregationtogrouptogetherallbussesrelatedtothesamebusline,(iii)asetof seven subsets, one for eachday of theweek, (iv) two custom reduction operations for each day andstatistictype(e.g.,min,max,avg,sum)executedwiththeOPH_APPLYoperator,andfinally(v)amergeofthetworeductionscomputedintheprevioussteptocreatethecompletetimeserieswiththefinaloutput.The“Ophidia-COMPSsTask”inTable1representsthesetofOphidiaoperators(i.e,thetworeductionandthemerge)thatcouldbeexecutedasasingleCOMPSstask.


Table1.ExecutiontimeandspeeduprelatedtoOphidiaoperatorsusedinthedescriptiveanalytics

I/Onodes-cores 1 2 3 4 5 1 2 3 4 5

OperatorName Meanexecutiontime[s] Speedup

IMPORTNC 26.80 14.21 10.32 8.42 6.48 1.00 1.89 2.60 3.18 4.14

AGGREGATE 5.58 2.98 2.45 1.91 1.58 1.00 1.87 2.28 2.92 3.52

SUBSET 1.24 0.82 0.68 0.62 0.57 1.00 1.51 1.80 2.00 2.16

APPLY(REDUCE) 1.42 0.89 0.74 0.66 0.60 1.00 1.59 1.90 2.15 2.35

MERGECUBES 1.04 0.73 0.63 0.57 0.53 1.00 1.44 1.66 1.83 1.95

OPHIDIA-COMPSsTASK

5.99 4.62 4.24 3.99 3.81 1.00 1.30 1.41 1.50 1.57

Figure9.ExecutiontimeandspeedupofasingleparallelCOMPSstaskbasedonOphidiaoperators

Figure9focusesontheexecutiontime(y-axisontheleft)andthespeedup(y-axisontheright)ofthesetofOphidiaoperators associated to a single COMPSs task. It isworthmentioning that the selecteduse caseapplicationconsistsofmultipletasksandeachparalleltasktimeexecutiondependsonthesizeoftheinputand thenumberof cores/nodesallocated. As shown in the figure,when thenumberofnodes increases,alongwiththelevelofparallelismexploitedforthecomputationinOphidia,theexecutiontimedecreases.Since thereare severalpiecesof code inherently sequential, this reduction isnotexactlyproportional tothe resources available, as highlighted from the speedup curve, but that’s an application-specific aspectwhich does not depend on the proposed big data services. Nevertheless, an important reduction in the


overall execution time can be achieved by incrementing the resources available in the Ophidia cluster.TheseresultsmustbetakenintoaccountinthedefinitionoftheperformancemodelfortheOphidia-basedapplications.

4.2. Usecase2onEntityMatchingEntity Matching as a Service (EMaaS) is a WP4 resource composed by three Entity Matching (EM)approaches.TheEMapproachesare:MATCHUP,BULMA,andBUSTE.TheMATCHUPapproachiscapableofdealingwithentities that are representedby spatial geometries, e.g. polygonsandpoints. Thus, this EMapproachperformspairwise comparisonsbetweengeometries availableat thedata sourceand classifieseachgeometrypairaccordingtothesimilaritybetweentheinvolvedgeometries.

ThesecondEMapproach,BULMA(BUsLineMAtching), isanEMapproachthat identifiesbustrajectoriesfromthesequencesofnoisygeospatial-temporaldatasources.ItinvolvesperformingthelinkagebetweenthebusGPStrajectoriesandtheircorrespondingroadsegments(i.e.,predefinedtrajectoryorshapes)onadigitalmap. Inthissense,BULMAisanovelunsupervisedtechniquecapableofmatchingabustrajectorywith the "correct" shape, considering the cases in which there are multiple shapes for the same route(commoncases inmanyBrazilcities,e.g.,CuritibaandSãoPaulo).Furthermore,BULMAisabletodetectbustrajectorydeviationsandmarktheminitsoutput.

While trying to enrich the BULMA output, the third EM approach, named BUSTE (BUs Stop TicketingEstimation),wasdesignedtoperformatimeinterpolationovertheshapes(basedontheBULMAoutput).Basically,BUSTEpositionsthebusstopsoverthe interpolatedshapeandgroupsthepassengersboardingbyeachbus stop. The idea is toprovideanestimateof thenumberofpassengersboardingat eachbusstop.MoredetailsabouttheEMaaSapproachescanbefoundindeliverableD4.5[R17].

EM approaches are known for being heavy to perform. The need to applymatching techniques on theCartesian product of all input entities (naive) leads its execution to a quadratic complexity. For largedatasets,theapplicationofsuchanapproachcanbeveryineffective.Therefore,EMisanidealproblemtobetreatedwithadistributedsolution.Theexecutionofblocking-basedEMcanbedoneinparallelwiththeSparkmodelbyusingseveralmapperandreducertransformationfunctions.Morespecifically,themapperscan read the inputentities inparallel anddistribute themamong the reducersaccording to theblockingkey.Entitiessharing thesameblockingkeyareassignedtoacommonreducerandseveralblockscanbematchedinparallel.

In this section,weevaluateBULMAreferring tooneperformancecritical factor: thenumberofavailableexecutors(cores)ninthecloudenvironment.Inthisexperiment,weevaluateBULMAaimingtoinvestigatehowitscaleswiththeincreasingnumberofavailableexecutors.Toachievethis,weranourexperimentsonaclustercomposedof8virtualmachines.Eachonehas fourcores,8GBofRAMand500GBofharddisk.Amongthesoftwaresinstalledonthemachine,Ubuntu64-bitandJAVA1.8wereutilized.ForeachroundofBULMAexecution,anewinstanceofJVMwascreatedtodisableinformationreuse(inmemory).Wealsoutilizedtworeal-worlddatasources.Thefirstdataset(DS-GPS)containsfivedaysofGPSinformation(from2017-05-22 to 2017-05-27) collected using the API available from the public transportation agency ofCuritiba10(Brazil).Eachfile(day)containsabout250MBofGPSdata(about1.2GBofGPSdataintotal).Thesecondone(DS-shapes)containsalltheshapes(i.e.,thepre-definedtrajectoriesthatabusshouldfollow)

10http://transporteservico.urbs.curitiba.pr.gov.br


extracted from theGTFS, alsoprovidedby thepublic transportationagencyofCuritiba. The shape file issmallandhasabout14.73MBoftextdata.

Table2.ExecutiontimeandspeeduprelatedtoSpark-basedBULMA

I/Onodes-cores

1 3 5 10 20 30 1 3 5 10 20 30

SparkApproach

Meanexecutiontime[min] Speedup

BULMA(2npartitions)

151.41 56.82 44.07 21.50 16.10 13.40 1.00 2.66 3.43 7.04 9.40 11.29

BULMA(4npartitions)

161.38 59.21 38.21 22.66 15.70 12.75 1.00 2.72 4.22 7.12 10.27 12.64

Figure10.ExecutiontimeandspeedupforSpark-basedBULMA(using2npartitions).


Figure11.ExecutiontimeandspeedupforSpark-basedBULMA(using4npartitions).

As shown in Table 2 and Figures 10 and 11, we can note that BULMA scales almost linearly up to fiveexecutorsshowingitsabilitytoevenlydistributetheworkloadacrosstheworkers.Theloadbalancingwasproperlytreatedwhentheblockingkeysrouteandbus identifierwereutilizedasapartitionsplitter.Theusage of the route and bus identifier promotes a significant reduction in the number of shapes to becompared(atmostsixshapes).Suchstrategyincreasesthegranularityofparalleltasks,thusfavoringloadbalancing.However,withaboutsevenworkers,theparallelismwascompromisedduetothepresenceofamuch larger amount of executors needed to process all workload generated (because the input data issmallrelatedtothedistributedcomputationalresourcesavailable).Notethat,theexecutiontimestabilizeswhenmorethansixworkersareutilized.Also,notethattheexecutiontimedecreasesfromalmost150min(foroneworker)toabout15min(fortwentyworkers).


5. CONCLUSIONSThisdocumentreportsaboutthemainextensionsandadaptationsimplementedforthebigandfastdataeco-systemtechnologiesinordertoaddressQoSdeadlinesinthecloudenvironmentwithdynamicelasticsolutions.Afterprovidingabriefoverviewof theWP3QoSarchitecture frompreviousWP3deliverables,introducingthemajormotivationsandissuesrelatedtoQoSguaranteesindataprocessingapplications,itdescribesindetailtheimplementationrequiredtoaddresstheseissuesatthePaaSlevel.

In particular, the focus is placed on the adaptation activities regarding two of the main technologiesincludedinthebigandfastdataeco-system(i.e.,OphidiaandSpark)andhowthesehavebeenintegratedwith the services and modules provided by the QoS infrastructure to support horizontal and verticalelasticityforQoS-basedscenarios.Relevantcodesnippets,aswellasexecutionexamplesareprovidedtodemonstrate theeffectivenessof the implementationandtosupport thedevelopersanduserswilling toachieveQoS-awaredataprocessingwithintheEUBra-BIGSEAinfrastructure.

AbenchmarkofsomekeyWP4applications,developedontopofthetechnologies integrated intheQoSarchitecture,demonstratinghowthenumberofresourcescanimpactontheprocessingexecutiontime,ispresentedattheendofthisreport.


6. APPENDIXEXAMPLEOFAPPLICATIONDESCRIPTIONFILEFOROPHIDIADEPLOYMENT

network public (outbound = 'yes' and outports='80,11732') network private () system oph-server ( cpu.count>=2 and memory.size>=2g and net_interface.1.connection = 'public' and net_interface.0.connection = 'private' and net_interface.0.dns_name = 'oph-server' and disk.0.os.name='linux' and disk.0.os.flavour='centos' and disk.0.os.version>='7' and disk.0.os.credentials.new.password = 'PASSWORD' and disk.0.applications contains (name='ansible.modules.OphidiaBigData.ophidia-cluster') and disk.1.size=25g and disk.1.fstype='ext4' and disk.1.mount_path='/data' ) system oph-io ( cpu.count>=4 and memory.size>=4g and net_interface.0.connection = 'private' and net_interface.0.dns_name = 'oph-io#N#' and disk.0.os.name='linux' and disk.0.os.flavour='centos' and disk.0.os.version>='7' and disk.0.os.credentials.new.password = 'PASSWORD' ) configure oph-server ( @begin --- - roles: - role: 'OphidiaBigData.ophidia-cluster' node_type: 'server' deploy_type: 'complete' server_hostname: "{{ansible_hostname}}" io_hostnames: "{{ groups['oph-io']|map('extract', hostvars, 'ansible_hostname')|list }}" io_ips: "{{ groups['oph-io']|map('extract', hostvars, ['ansible_default_ipv4', 'address'])|list }}" private_server_ip: "{{ IM_NODE_PRIVATE_IP }}" public_server_ip: "{{ IM_NODE_PUBLIC_IP }}" nfs_subnet: "oph-*.localdomain" mysql_subnet: "oph-%.localdomain" io_node_number: "10" oph_account_user: "OPH_USER" oph_account_passwd: "OPH_PASSWD" pre_tasks: - name: stop service firewalld systemd: name=firewalld state=stopped - name: Creates NFS shared directory file: path=/data state=directory owner=root group=root


@end ) configure oph-io ( @begin --- - roles: - role: 'OphidiaBigData.ophidia-cluster' node_type: 'io' deploy_type: 'complete' server_hostname: "{{ hostvars[groups['oph-server'][0]]['ansible_hostname'] }}" io_hostnames: "{{ groups['oph-io']|map('extract', hostvars, 'ansible_hostname')|list }}" io_ips: "{{ groups['oph-io']|map('extract', hostvars, ['ansible_default_ipv4', 'address'])|list }}" private_server_ip: "{{ hostvars[groups['oph-server'][0]]['IM_NODE_PRIVATE_IP'] }}" public_server_ip: "{{ hostvars[groups['oph-server'][0]]['IM_NODE_PUBLIC_IP'] }}" nfs_subnet: "oph-*.localdomain" mysql_subnet: "oph-%.localdomain" io_node_number: "10" pre_tasks: - name: stop service firewalld systemd: name=firewalld state=stopped @end ) deploy oph-server 1 deploy oph-io 2


7. REFERENCES[R01]S.Fiore,D.Elia,W.dosSantos,C.E.Pires,"D4.1:Designoftheintegratedbigandfastdataeco-system",EUBRABIGSEADeliverableD4.1,EUBRABIGSEA-WP4-D4.1.Online:http://www.eubra-bigsea.eu/deliverable-41-design-integrated-big-and-fast-data-eco-system

[R02]J.Almeida,D.Ardagna,I.Ataíde,E.Barbierato,I.Blanquer,S.López,T.Braga,A.Brito,A.P.Couto,A.Evangelinou,I.Ferreira,A.Goes,M.Gribaudo,R.Mirandola,F.Morais,"D3.5:EUBra-BIGSEAQoSinfrastructureservicesfinalversion",EUBRABIGSEADeliverableD3.5,EUBRABIGSEA-WP3-D3.5.

[R03]A.Brito,D.Ardagna,I.Blanquer,A.Evangelinou,E.Barbierato,M.Gribaudo,J.Almeida,A.P.Couto,T.Braga,“D3.4:EUBra-BIGSEAQoSinfrastructureservicesintermediateversion”,EUBRABIGSEADeliverableD3.4,EUBRABIGSEA-WP3-D3.4.Online:http://www.eubra-bigsea.eu/sites/default/files/D3.4%20EUBra-BIGSEA%20QoS%20infrastructure%20services.pdf

[R04]D.Elia,S.Fiore,A.D'Anca,C.Palazzo,I.T.Foster,D.N.Williams:Anin-memorybasedframeworkforscientificdataanalytics.Conf.ComputingFrontiers2016:424-429.

[R05]S.Fiore,A.DAnca,C.Palazzo,I.Foster,D.N.Williams,andG.Aloisio,“Ophidia:Towardbigdataanalyticsforescience,”ProcediaComputerScience,vol.18,pp.2376–2385,2013.

[R06]MiguelCaballer,CarlosdeAlfonso,FernandoAlvarruiz,GermánMoltó,"EC3:ElasticCloudComputingCluster".JournalofComputerandSystemSciences,Volume78, Issue8,December2013,Pages1341-1351, ISSN0022-0000,10.1016/j.jcss.2013.06.005.

[R07]MiguelCaballer,IgnacioBlanquer,GermánMoltó,andCarlosdeAlfonso."Dynamicmanagementofvirtualinfrastructures".JournalofGridComputing,Volume13,Issue1,Pages53-70,2015,ISSN1570-7873,DOI:10.1007/s10723-014-9296-5.

[R08]SlurmWorkloadManager-Online:https://slurm.schedmd.com.VisitedonDecember,22nd2017.

[R09]AnsiblePlaybooks-Online:http://docs.ansible.com/ansible/latest/playbooks.html.VisitedonDecember,22nd2017.

[R10]I.Blanquer,G.Moltó,"D3.3:BIGSEAQoSinfrastructureservicesinitialversion",EUBRABIGSEADeliverableD3.3,EUBRABIGSEA-WP3-D3.3.Online:http://www.eubra-bigsea.eu/sites/default/files/D3_3-BIGSEAQoSInfrastructureservicesinitialversionv4.0.pdf

[R11]E.Ataie,E.Gianniti,D.Ardagna,andA.Movaghar,“AcombinedanalyticalmodelingmachinelearningapproachforperformancepredictionofmapreducejobsinHadoopclusters,”inProc.ofSYNASC,2016.

[R12]D.Ardagna,E.Barbierato,J.M.Almeida,A.P.CoutoSilva,"D3.2:BigDataApplicationPerformancemodelsandrun-timeOptimizationPolicies",EUBRABIGSEADeliverableD3.2,EUBRABIGSEA-WP3-D3.2.Online:http://www.eubra-bigsea.eu/sites/default/files/D32-Big%20Data%20Application%20Performance%20models.pdf

[R13]Mesosphere-Online:https://mesosphere.com.VisitedonDecember,22nd2017.

[R14]CarlosdeAlfonso,MiguelCaballer,FernandoAlvarruiz,VicenteHernández,"AnEnergyManagementSystemforClusterInfrastructures",Computers&ElectricalEngineering,Volume39,Issue8,2013,Pages2579-2590,ISSN0045-7906,10.1016/j.compeleceng.2013.05.004.

[R15]V.W.Mak,S.F.Lundstrom,PredictingPerformanceofParallelComputations,IEEETransactionsonParallelandDistributedSystems,v.1n.3,p.257-270,July1990[doi>10.1109/71.80155].

[R16]S.Fiore,etal., "D4.2:Reportabout the final implementationof the integratedbigand fastdataeco-system",EUBRABIGSEADeliverableD4.2,EUBRABIGSEA-WP4-D4.2.

[R17]C.Cappiello,C.E.Pires,D.G.Mestre,"D4.5:Designoftheintegratedbigandfastdataeco-system",EUBRABIGSEADeliverableD4.5,EUBRABIGSEA-WP4-D4.5.


GLOSSARY

Term Explanation

API ApplicationProgrammingInterface

CSV CommaSeparatedValue

DAG DirectedAcyclicGraph

EC3 ElasticCloudComputingCluster

EM EntityMatching

EMaaS EntityMatchingasaService

FCFS FirstCome,FirstServed

GPS GlobalPositioningSystem

GTFS GeneralTransitFeedSpecification

GUI GraphicalUserInterface

HML HybridMachineLearning

HTTP HyperTextTransferProtocol

IaaS InfrastructureasaService

IM InfrastructureManager

IP InternetProtocol

JSON JavaScriptObjectNotation

JVM JavaVirtualMachine

MPI MessagePassingInterface

NetCDF NetworkCommonDataForm

NFS NetworkFileSystem

OLAP On-lineAnalyticalProcessing

PaaS PlatformasaService

QoS QualityofService

REST REpresentationalStateTransfer

RPC RemoteProcedureCall

SQL StructuredQueryLanguage

SSL SecureSocketsLayer

SSH SecureShell


VM VirtualMachine

VMI VirtualMachineImage

WP Workpackage

D4.4 QoS extensions for the big and fast data eco-system 1 · The EUBra-BIGSEA project is aimed at...

Documents

Transcript of D4.4 QoS extensions for the big and fast data eco-system 1 · The EUBra-BIGSEA project is aimed at...