Joao Fernandes - 6th International LSDMA Symposium, 29th August 2017
-
Upload
helix-nebula-the-science-cloud -
Category
Science
-
view
72 -
download
1
Transcript of Joao Fernandes - 6th International LSDMA Symposium, 29th August 2017
HelixNebulaScienceCloud6th LSDMASymposium
KIT
29th August2017João Fernandes
CERNJoao.Fernandes <at>cern.ch
HelixNebula– TheScienceCloudwithGrantAgreement687614isaPre-CommercialProcurementActionfundedbyH2020FrameworkProgramme
Outline
Pre-CommercialProcurementHybridCloudChallengesFunctionalRequirementsContractorsDesignsTestingFutureAdoptersNextSteps
9/4/17
HNSciCloud JointPre-CommercialProcurementProcurers:CERN,CNRS,DESY,EMBL-EBI,ESRF,IFAE,INFN,KIT,STFC,SURFSaraExperts:Trust-IT&EGI.eu
Thegroupofprocurershavecommitted• Procurementfunds• Manpowerfortesting/evaluation• Use-caseswithapplications&data• In-houseITresources
Resultingserviceswillbemadeavailabletoend-usersfrommanyresearchcommunities
Co-fundedviaH2020GrantAgreement687614
Totalprocurementbudget>5M€
9/4/17
HNSciCloud projectphases
Preparation
•Analysisofrequirements,currentmarketoffersandrelevantstandards
•Buildstakeholdergroup•Developtendermaterial
Implementation andsharing
Jan’16 Dec’18
Eachstepiscompetitive - onlycontractorsthatsuccessfullycompletethepreviousstepcanbidinthenext
5
4Designs 3Prototypes 2Pilots
Call-offFeb’17
Call-offDec’17
TenderJul’16
PhasesofthetenderaredefinedbytheHorizon2020Pre-CommercialProcurementfinancialinstrument
9/4/17
WhatwillbeprocuredAhybridcloudplatformfortheEuropeanresearchcommunity
6
HNSciCloudPCP
Source:C
loud
Compu
tingforG
ovies,DL
TSolutio
ns,
DavidBlankenh
orn,VanRistau
andCa
ronBe
esley
CombiningservicesattheIaaSleveltosupportscienceworkflows
TheR&DservicestobedevelopedaretobeintegratedwithResourcesindatacentresoperatedbythebuyersgroupGEANTnetwork
9/4/17
ChallengesInnovativeIaaSlevelservicesintegratedwithprocurersin-houseresourcesandpublice-infrastructureaspartofahybridcloudtosupportarangeofscientificworkloads
ComputeandStoragesupportarangeofvirtualmachineandcontainerconfigurationsincludingHPCworkingwithdatasetsinthepetabyterange
NetworkConnectivityandFederatedIdentityManagementprovidehigh-endnetworkcapacityviaGEANTforthewholeplatformwithcommonidentityandaccessmanagement
ServicePaymentModelsexplorearangeofpurchasingoptionstodeterminethosemostappropriateforthescientificapplicationworkloadstobedeployed
79/4/17
BaselineofServicesRequiredIaaS
HTCVMconfigurations4-32vCPUs,2-8GBRAM
HPCaaS (lessthan5%ofthetotal)64vCPUsandmore,64GBRAM
DataManagementGlobalnamespaceregardlessofthelocationPOSIXsupportAutomateddatareplication
Network(Contractor’sDCstoGEANTVRF)Prototype:10Gbps,Pilot:40Gbps
IAMSupportforeduGAIN &ELIXIRFederatedAAI
APISupportTerraform,ObjectStores,COEs(Kubernetes,Docker),Monitoring,etc.04/09/17
HNSciCloudPrototypeActivityReport Unclassified 14
OVERVIEWOFPROTOTYPECLOUDPLATFORM
Integrated
Realistic
MinimallyIntrusive
Scalable
Managed
Sustainable
Functionalcomponentstodelivercompletedataanalysisplatform…
HNSciCloudPrototypeActivityReport Unclassified 15
OVERVIEWOFPROTOTYPESOLUTIONFunctionalcomponentstodelivercompletedataanalysisplatform…
HNSciCloudPrototypeActivityReport Unclassified 16
OVERVIEWOFPROTOTYPESOLUTIONFunctionalcomponentstodelivercompletedataanalysisplatform…
HNSciCloudPrototypeActivityReport Unclassified 17
OVERVIEWOFPROTOTYPESOLUTIONFunctionalcomponentstodelivercompletedataanalysisplatform…
HNSciCloudPrototypeActivityReport Unclassified 18
OVERVIEWOFPROTOTYPESOLUTIONFunctionalcomponentstodelivercompletedataanalysisplatform…
Martin Gasthuber | Next Steps | 03/04/2017 | Page 19
Test Phases & Resource Ramp-Up
TP1 TP2 TP3 TP42000C200TB
100C 10TB
3500C350TB
TestSuiteCERN• Edugain AAI• CERNBenchmarkSuite• S3Endpoints• PerfSONAR Network• VMProvisioningandPersonalisation• GridJobsinDocker• Dockerized deploymentofEOS+CERNBOX+SWANCNRS• IAASaccessviaEduGain andlocalaccountsDESY• HDF5_IOEMBL• CLOUD_BENCHMARK• ELIXIR_AAI• data_transparencyESRF• FDMNES
IFAE• monitoring-accounting• imagecreationandcontextualizationINFN• BelleII_SIM• COMPASS/TGEANT_IO• INFN/GPFS_AFM• DynfarmKIT• static-batch-extension• dynamic-batch-extension• AFM-deployment-testSTFC• batch-system-extension• jobs-on-kubernetesSURFsara• BBMRi_data• dCache_io• kubertest
Moredetails:https://cernbox.cern.ch/index.php/s/zCd8KRtBUfH9yjF
9/4/17
Specsvs.Tests
Criteria 3.2 - Standard IaaS Service CERN,KIT,EMBL
Criteria 3.3 - Storage CERN, SURFsara,INFN,DESY
Criteria 3.4 - Transparent Data Access DESY
Criteria 3.5 - Data Ingress/Egress DESY
Criteria 3.6 - Data Location EMBL
Criteria 3.7 - Container Support DESY
Criteria 3.8 - Networking ALL
Criteria 3.9 - Identity and Access Management CERN,EMBL,KIT
Criteria 3.10 - Service Capacity KIT
Criteria 3.11 - Service Availability ALL
Criteria 3.12 - Service Elasticity KIT
Criteria 3.14 -Service Reporting & Usage Monitoring IFAE
Criteria 3.15 - Cloud Service Support ALL
Criteria 3.16 - HPCaaS ESRF
ActivitiesinTestPhase1(TP1):April-June2017
EstablishingNetworkConnectivityviaGEANT/NRENsFederatedIdentityviaeduGAIN &ELIXIR
FineTuningTenancydefinitionandisolationrequirementsTokenandaccountterminationpolicies
Creatingtenantsandaccesses04/09/17
ActivitiesinTestPhase2(TP2):June-September2017Benchmarking
KV,Dirac,WhetstonePhoronix
ValidationscriptsofVMflavours
NetworktestsCommissioning
10GbpstesttoabaremetalperfSONAR serveratContractorsDCw/10GbpsNICRegulartestasthelinkgetsused
Throughput,Latency,PacketLoss
FederatedAAIAuthC attemptsinseveralscenariosCompliantattributereleaseAuthZ testsatTenancy,sub-Tenancylevels
ObjectStoresfunctionaltestsS3operationsPUT,GET(AuthC,Single/Multiplechunks,prefixmatching)
NextSteps20172nd progressreview4th-5th September
Scale-upIaaSresourcesMoretesting
3rd progressreview9th-10th October(DESY)LivewebcastdemonstrationsFeaturefreezeScale-upIaaSresources(totalling3.5kcores350TBstorage)Finalizethetests
Endofprototypephaseassessment11th -13th December
Selectionof2/3prototypesforpilotphase
04/09/17
PilotPhase2018Kick-offevent6th-8th Feb2018(INFNBologna)
Progressivelyscale-upIaaSresourcestotalling20kcores2PBstorage
FocusonPerformance,Scalability,Portability,Security
Deploymentofapplications
Assessprocurement/paymentmodels
04/09/17
Growingthebuyersgroup
Privatesector
Gov.agencies
Additionalresearchorgs.
Procuringbeneficiariesof
ECproject
Theinitialgroupofbuyersisthesetofresearchorgs.thatcommittedtheirresourcesatthestartoftheprojectandbecamebeneficiariesoftheECproject
Goalistoprogressivelygrowthebuyersgroupbyincludingmorepubliclyfundedresearchorganisations:Early adoptergroup
Draftdocumentdescribesbenefitsforthestakeholders,procurementprocess,stepsfor implementation
CERN,DESY,EMBL-EBI,ESRF,KIT,CNRS,INFN,PIC-IFAE,SURFsara,STFC
adoptergroup
Summary3prototypes(IBM,T-SystemsandRHEA)arebeingdeployedandtestedbytheBuyersGroup
Testingandscalingwillcontinueuntiltheendof2017
2/3prototypeswillbeselectedforlarge-scalepilotsin2018
TheEarlyadoptergroupwillallowmoreresearchorganisationstoprocuretheresultingservices
04/09/17