Prototype, Cloud&Heat SafeCloud-based cloud storage platform

D5.3

Project reference no. 653884

August 2017

DocumentinformationScheduleddelivery 31.08.2017Actualdelivery 31.08.2017Version 1.0ResponsiblePartner Cloud&HeatTechnologiesGmbH

DisseminationlevelPublic

RevisionhistoryDate Editor Status Version Changes 22.08.2017 S.Schmerler Draft 0.1 Initialversion22.08.2017 H.Mercier Draft 0.2 UniNErevision31.08.2017 S.H.Totakura Draft 0.2 TUMrevision31.08.2017 S.Schmerler Final 1.0 Incorporateallreviewsuggestions

ContributorS.Schmerler(C&H)G.Miegel(C&H)S.H.Totakura(TUM)D.R.Matos(IN-ID)

InternalreviewersH.Mercier(UniNE)S.H.Totakura(TUM)

AcknowledgementsThis project is partially funded by the European Commission Horizon 2020 workprogrammeundergrantagreementno.653884.

MoreinformationAdditional information and public deliverables of SafeCloud can be found athttp://www.safecloud-project.eu

D5.3–Cloud&Heatstorageplatform 3

GlossaryofacronymsAcronym DefinitionDC DatacenterIaaS InfrastructureasaServiceMON CephMonitorOSD ObjectStorageDaemonRBD RADOSBlockDeviceSaaS SoftwareasaServiceSCFS SafeCloudFileSystemVM VirtualmachineWAN WideAreaNetwork

TableofcontentsDocumentinformation.................................................................................................................................2Disseminationlevel.......................................................................................................................................2Revisionhistory..............................................................................................................................................2Contributor.......................................................................................................................................................2Internalreviewers.........................................................................................................................................2Acknowledgements........................................................................................................................................2Moreinformation...........................................................................................................................................2Glossaryofacronyms....................................................................................................................................3Tableofcontents............................................................................................................................................4ExecutiveSummary.......................................................................................................................................51 Introduction............................................................................................................................................62 SafeCloudBox..........................................................................................................................................71.1 Productdesigngoals..........................................................................................................................................71.2 Cloudbackendstoragetechnology..............................................................................................................81.3 Prototypeimplementation...............................................................................................................................91.4 Nextsteps.............................................................................................................................................................11

2 CloudBlockStorage............................................................................................................................122.1 Productdesigngoals.......................................................................................................................................122.2 Prototypeimplementation............................................................................................................................122.3 Nextsteps.............................................................................................................................................................14

3 Conlcusion............................................................................................................................................154 References............................................................................................................................................16

D5.3–Cloud&Heatstorageplatform 5

ExecutiveSummaryThe prototypes for the SafeCloudBox and CloudBlockStorage solutions based onSafeCloudtechnologyanddevelopedbyC&Haredescribedindetail.BothprototypesarestoragesolutionsbackedbytheCephdistributedstoragesoftware.Beingprototypes,bothsolutionsarerealizedusingVMsandinonecaseDockercontainers.Theproductswillbedeveloped further during the remaining project time with the goal of being fullyintegratedintoC&H'sinfrastructure.Inthisdeliverable,wedescribeworkingsetupsandshowthatSafeCloudtechnologycanbefullyintegratedintoC&Hproducts.Specifically,theSafeCloudBoxproductisanentirelynewprototype,whichreliesontheSafeCloudFileSystemtoprovidecustomerswithasecureandfault-tolerantdatastoragesolution.TheCloudBlockStorageproductextendsC&H'sblockstoragecloudofferwithinter-datacentersecuritybyusingtheSafeCloudPrivateCommunicationMiddleware.

1 IntroductionThecurrentdeliverableoutlinesthetechnicaldevelopmentanddeploymentofprototypesetups,showingthefunctionalityoftheSafeCloudBoxandCloudBlockStoragesolutions,whichareindevelopmentatC&HandbasedontechnologycreatedwithintheSafeCloudproject.ThedeliverabledescribestwoprototypesdevelopedatC&HincollaborationwithIN-IDandTUM,whichshowthatalltechnologiesfromwithinSafeCloudandusedbyC&HareabletobefullyintegratedintoC&H'sproductioninfrastructure,whichisthemaingoalofthisdeliverable.

D5.3–Cloud&Heatstorageplatform 7

2 SafeCloudBoxThis chapter describes the implementation of the SafeCloudBox prototype at C&H,utilizingtheSafeCloudFileSystem.

1.1 ProductdesigngoalsA summary of the design goals described in D5.1 is given here for convenience. TheSafeCloudBoxproduct aims toprovideuserswith enhanceddata storage security anddisasterrecoverycapabilitiescomparedtoservicessuchasDropBox.Toaccomplishthatgoal,C&HusestheSafeCloudFileSystem(SCFS)developedwithintheSafeCloudproject.DetailsaboutthearchitectureofSCFSareshowninFig.1andareoutlinedin[SCFS14].

Figure1:SCFSarchitecture(imageprovidedbyIN-ID)The SCFS client applicationprovides a FUSE-basedmount point towhich data can bewritten.Thedataisthentransferredtocloudstorage.Theclientwillberunningonthecustomers infrastructure, such as a datacenter (DC) or a local office machine. Oneadvantage of SCFS is that it writes data encrypted to the cloud storage backendsautomatically.ThesecondandevenmoreimportantfeatureisthatSCFSwritesdatatomultiple cloud backends, thus even allowing data recovery in case a cloud provider'sdatacentergoesdown.Thesepointsarevery important tocustomersandas such,areequallyimportanttoC&H.C&HoperatesanumberofDCsacrossGermanyandcanuseSCFSinaquitenaturalwaybyusingseveralphysicallydistinctDCsascloudstoragebackends.

1.2 CloudbackendstoragetechnologyAscanbeseeninFig.1,SCFShasbackenddriversforanumberofmajorcloudproviderssuch as Amazon AWS S3, Google Cloud etc. C&H uses OpenStack1as cloud operatingsystem on all DCs. Until recently, C&H used OpenStack Swift 2 as object storageimplementation(aswellasGlusterFS3forblockstorage).SwifthasverysimilarsemanticsintermsofdatastoragecomparedtoAWSS3(buckets,objects).However,ithasitsownAPI,whichisnotsupportedbySCFS.ThereisaprojectimplementingaS3APIlayerforSwift(swift34)andC&Hspentsometimetodeploythatlayerinproductionsystems.ItturnedoutthattheSwiftS3APIisnotasmatureandstableasisneededforproductionsystems.Atthesametime,C&HwasplanningseveralnewDCsandcoincidentallydecidedtomovetoanotherstoragetechnologyentirely,namelyCeph5.TheCephsystemisamodernandscalablesolutionwhichoffersobjectstorageandblockstorageusingoneunderlyingstoragetechnologycalledRADOS,asshowninFig.2.

Figure2:Cepharchitecture(fromhttp://docs.ceph.com)

1 https://www.openstack.org2 https://wiki.openstack.org/wiki/Swift3 https://www.gluster.org4 https://github.com/openstack/swift35 http://ceph.com

D5.3–Cloud&Heatstorageplatform 9

Inparticular,Ceph'sS3API isprovidedbyasoftwarelayercalledtheRADOSGateway(RADOSGW).WefoundittobemuchmorerobustandstablethantheSwiftS3APIlayer.However, as the newDCs are not fully operational and the Swift storage systems aredeprecatedinoldDCsatthetimeofwriting,therewasnopossibilitytodeploySCFSusingCeph backends running on geographically distinct locations. Instead, we focused onbuilding theprototypeSCFSdeploymentsetupusingVMsandDockercontainers.Thisapproachallowsustouseallfunctionalityinacontrolledtestbedandisarchitecturallyequivalenttoafullydeployedsystem.OnceCephisfullyoperationalinallofourDCs,itistrivialtoconfigureactualDCsrunningCephasstoragebackend.

1.3 PrototypeimplementationBeforewecoulddeployourprototype,asmallchangeintheSCFScodewasrequired.C&HneededtobeabletoconfigureanalternativeendpointURLtothedefaultAmazonS3APIendpoint.IN-IDperformedthechangequickly,suchthatC&Hcouldproceedtousethecode.TheSCFS+CephprototypesetupisshowninFig.3.

Figure3:SafeCloudBoxSCFS+CephprototypesetupWe setup aVM running twoDocker6containers, one running aCeph cluster andonerunningallofSCFS(usually,theDepSkyclientaswellastheDepSpacereplicasrunondifferentmachines,seeFig.1).SCFSneedsatleast4cloudstoragebackendstoruninafail-safemode. Inourprototype, theCephcontainer is configured4x inSCFSas cloudstoragebackendandtherefore,thesamedataisstoredineachofthe4bucketsinCeph.Insteadofrunning4DepSpacecoordinationservicereplicas,C&HandIN-IDoptedforasimplersetupusingonlyonereplicafortheprototype.OncetheSCFSsystemisstarted,aFUSEdeviceisprovided,whichcanbemountedandusedtoreadandwritedata.ThedataflowisalsoshowninFig.3.BelowweshowanexamplesessionusingSCFS.The4CephS3bucketsarenamed"depskyXXXXXXXXXXX-cocY".

6 https://www.docker.com

First,wecheckthecontentoftheSCFSmountfromwithintheSCFSdockercontainer: root@scfs:~# ls -la /mnt/scfs -rw-r--r-- 1 root root 797 Aug 11 11:42 .statistics.txt Thereisonlyonedefaultfilehere(.statistics.txt).ThusthereshouldbealreadydatapresentinSCFSwhichrepresentsthatfile.WecheckwithanS3APIclient,whichtalkstotheS3APIendpointof theCephdockercontainer running inside theprototypeVM(ascriptwhichusestheboto27library): user@vm:~$ ./s3test.py la depskysbolvkzymdd-coc1 415024517702370metadata 415024517702370value1004 depskysbolvkzymdd-coc2 415024517702370metadata 415024517702370value1004 depskysbolvkzymdd-coc3 415024517702370metadata 415024517702370value1004 depskysbolvkzymdd-coc4 415024517702370metadata 415024517702370value1004 Ascanbeseen,thereisadataandmetadataentryforthefileineachbucket.Notethateach bucket contains the same data, since in a multi-DC deployment case, data isreplicatedover4DCs,eachholdingoneofthebuckets.Now,againintheSCFScontainer,wewriteafiletoSCFS: root@scfs:~# echo test > /mnt/scfs/file ThebucketcontentsintheCephcontainernowlooklikethis:: user@vm:~$ ./s3test.py la depskysbolvkzymdd-coc1 415024517702370metadata 415024517702370value1004 415033170075570metadata 415033170075570value1004 depskysbolvkzymdd-coc2 415024517702370metadata 415024517702370value1004 415033170075570metadata 415033170075570value1004 depskysbolvkzymdd-coc3 415024517702370metadata 415024517702370value1004 415033170075570metadata 415033170075570value1004 depskysbolvkzymdd-coc4 415024517702370metadata 415024517702370value1004

7 http://boto.cloudhackers.com/en/latest/

D5.3–Cloud&Heatstorageplatform 11

415033170075570metadata 415033170075570value1004 Therearenewentriesineachbucket,holdingthedataofthewrittenfile.With this prototype, C&H and IN-ID have verified that SCFS is a solution that can bedeployedandtestedtogetherwithreal-worldstoragetechnologyrunningonC&H'sDCs.Thisisanimportantsteptowardsafullstorageproductofferingforendusers.

1.4 NextstepsAsoutlinedinD5.1,C&HplanstoreleasetheSafeCloudBoxproductasSaaS-typesolution.For example, one possible product is a Docker container with SCFS installed andconfiguredtouseC&H'scloudstorage.Inthiscase,thebusinessmodelissimplychargingforusedstorage,whiletheSafeCloudBoxclientcontaineritselfcanbedistributedasopensource,whichispossiblesincetheSCFScodehostedatgithub.comspecifiesnolicenseandisthuspublicdomain.Such a storage client can then be deployed by private end users or by systemadministrators in companies and organizations, thereby extending their storageinfrastructurewithasecureandfault-tolerantcloudbackup.AnextensionofthestorageclientisaSaaSproductshippinganopensourceprivatecloudsolutionsuchasNextcloud8,alsoalreadypre-configuredtouseSCFS,at leastpartly,assecureandfault-tolerantcloudbackup.

8 https://nextcloud.com

2 CloudBlockStorageThischapterdescribestheimplementationoftheCloudBlockStorageprototypeatC&H,utilizingtheAdvancedPort-KnockingtechniquedevelopedbyTUMinD1.2aspartoftheSafeCloudPrivateCommunicationMiddleware.

2.1 ProductdesigngoalsAsdescribedinD5.1,C&HoperatesanumberofDCsacrossGermanyandoffersIaaSlevelproducts,inparticularVMs.BlockstoragewasuntilnowprovidedbyGlusterFSusinga3-fold replication schemewithin a DC.With thementioned switch to Ceph at C&H, thesituationisbasicallyunchangedinthatC&Hoffersblockstorageusing3-foldreplicationwithinaDC.TheblockstorageinterfacetoRADOSinCephisprovidedbytheRBDlayer(RADOSBlockDevice),asshowninFig.2.There are two related use cases of block storage. First, users are able to attach blockstoragevolumestoVMs(likeaddingaharddisktoacomputer),whichallowspersistentdatastorage,evenifVMsareshutdownorthecomputenoderunningtheVMdies.ThisdoesnotprotectthecompleteVMdata,butonlythedataauserchoosestostoreontheblockstoragevolume,whereasVMdata(e.g.theoperatingsystem,userhomedirectory,...)arestoredonthecomputenode'sharddiskbydefault.Toaddressthisissue,usershavethe option of placing thewholeVMon a block storage volume. This has performanceimplications but adds data redundancy and recovery properties independently fromcomputenodes.In the SafeCloud context, C&H evaluated the possibilities to go beyond intra-DC dataredundancyusing inter-DCdatareplication,whereDCsareconnectedviaaWAN.Thishas,ofcourse,securityimplicationswhicharemetbyemployingSafeCloudtechnology.Inter-DC replication gives users additional fault-tolerant data storage options andenhancesC&Hoffersofsafeandsecuredatastorage.EspeciallyasC&HoperatesmanyDCs, there is a great demand for distributing data acrossDCs to leverage the presentinfrastructureforincreasedfault-tolerantdatastorage,evenincaseofDCunavailability(e.g.localpoweroutage,localnetworkfailure).ItisthereforevitalforC&H'sproductstohaveaccesstosecuredatareplicationchannelsbetweenDCs.

2.2 PrototypeimplementationInaproductionsetting,wewouldhavetwophysicallydistinctDCs.EachwouldrunitsownCephclusterandthegoalissecuredatareplicationovertheWANconnectingthetwoDCs.Forthesamereasonsexplainedabove(lackofproductionCephinstallations),webuiltaprototypeusing twoVMs.Wedeploy twoCephclusters,each inaseparateVMwhichrepresentaDCinthereal-worldproductionsetting.Oneoftheclusters(primary)is the block storage backend used by VMs running in that DC. The second cluster(secondary)isthefallbackcluster,towhichdataissynchronized.ThisisshowninFig.4.

D5.3–Cloud&Heatstorageplatform 13

Figure4:CloudBlockStorageprototypesetupIntermsofdatareplicationtechnology,weemployaCephfeaturecalled"RBDmirroring".RBD mirroring allows to synchronize data between Ceph clusters. Quoting the CephdocumentationonRBDmirroring9:"RBDimagescanbeasynchronouslymirroredbetweentwoCephclusters.Thiscapabilityuses theRBD journaling image feature toensure crash-consistent replicationbetweenclusters. Mirroring is configured on a per-pool basis within peer clusters and can beconfiguredtoautomaticallymirrorall imageswithinapooloronlyaspecificsubsetofimages. Mirroring is configured using the rbd command. The rbd-mirror daemon isresponsibleforpullingimageupdatesfromtheremote,peerclusterandapplyingthemtotheimagewithinthelocalcluster".Beforewe explain the SafeCloud-based security enhancements developedbyC&H,weneedtoestablishsomeCephterminology.AnOSD(objectstoragedaemon)isadaemonthatstoresdataonalocalfilesystem(usuallyaharddisk)andprovidesaccesstothedataover the network in a Ceph cluster. Each Ceph cluster requires at least one monitordaemon(MON)tomonitortheclusterstatusandtocoordinatereplicationofdataamongOSDs.Notethatsincetheprimaryandsecondaryclusterare,ingeneral,connectedbyaWANandtheRBDmirrorprotocolhasnosecuritylayer,oneneedstosecurethatchannelbyothermeans.ThisiswhereC&HemploysSafeCloudtechnology.IntheC&Hprototype,theWANcommunicationbetweenprimaryandsecondaryclusterissecuredbeacombinationof sshuttle10andsKnock. sshuttle isasimpleVPN-like technologybasedonSSH,whilesKnock is the port-knocking technology developed by TUM as part of SafeCloud (seeD1.2).Thereasonwhyweusesshuttleinadditiontoport-knockingisbecauseitaddsanextralayerofsecuritybyencryptingalltraffic.AsecondreasonisthatRBDmirroringrequiresanadditionaldaemon (rbd-mirrordaemon).The rbd-mirrordaemonof the secondarycluster needs to have full layer-3 connectivity to the OSD and MON daemons of theprimaryclusterinordertoreplicatedatafromtheprimarycluster.Theportnumbersof

9 http://docs.ceph.com/docs/master/rbd/rbd-mirroring10 https://github.com/apenwarr/sshuttle

theOSDandMONdaemonsontheprimaryclusterarenotpredictableinascalablefashionbecauseinrealworldDCscenariosOSDandMONdaemonarespreadoverahugenumberofhostsandarenotreachabledirectlyatonepublichostaddress.Thesecurity-enhancedversionofRBDmirroringintheprototypeworksasfollows.ThesKnockserverisrunningonahostontheprimaryclusterandgetsport-knockedfromthesKnockclientonthesecondaryclustertoopenaspecificport.AfterthesKnockserverhasopenedtherequestedport,sshuttleonahostonthesecondaryclusterisusedtoestablishatransparentandencryptedroutingofTCPflowsintothelayer-3networkoftheprimarycluster.Therbd-mirrordaemonrunninginthesecondaryclusterthencontactstheMONdaemonontheprimaryclusteroverthistunneltoreceivetheclustermapandthejournalof the primary cluster. The changes in the dataset of the primary cluster are thenreplicated("replayed")bytheOSDsofthesecondaryclusterformtheircounterpartOSDsintheprimarycluster.

2.3 NextstepsC&H plans to deploy the described setup between two DCs, once they run Ceph inproduction.C&HmayreplacesshuttlewithafullVPNsolutionifneeded.C&Hwillalsoevaluatetheperformanceimplicationsofinter-DCdatareplicationandmayrestrictdatasynchronizationinwayscompatiblewiththeuser'sdataredundancyrequirementsandavailablebandwidth.

D5.3–Cloud&Heatstorageplatform 15

3 ConlcusionThis deliverable presented the developed prototypes for the C&H SafeCloudBox andCloudBlockStorage solutions, both of which are storage products based on the Cephdistributedstoragesoftware.ItwasshownthatSafeCloudtechnologycanbeusedtoaddlevelsofdatasecurityandfault-tolerancetothesolutionsandthatthistechnologycanbeintegratedinprototypicalC&Hsystems.Itwasconcludedthatitispossibletoscaletheprototypestoproductionlevelsystems.

4 References[SCFS14] Alysson Bessani, Ricardo Mendes, Tiago Oliveira, and Nuno

Neves, "SCFS:ASharedCloud-backedFileSystem",2014USENIXAnnualTechnicalConference(USENIXATC14),p169-180