Supporting the scientific data lifecycle · 2016-01-07 · • Horizon 2020 project starting April...

SupportingtheScientificDataLifecycle|ISGC2015,Taipei|PatrickFuhrmann|17March2015|1

PatrickFuhrmann

Onbehaveoftheprojectteam

Supportingthescientificdatalifecycle


Content• Howaresoftwarefeaturesselected.• Howaresoftwarefeaturesfunded.• Hardeningnewfeatures.• Exploringnewcommunities.• RespondingonnewtechnologiesHWandSW• SomethingaboutINDIGO-DataCloud• EssentiallyarandomwalkfocusingonthingsIthoughtmightbeinteresting.


Somewordsonwhyandwhen

dCachedoeswhatitdoes.


Howaresoftwarefeaturesselected?

• Scientificcommunitiesbelieve, thatOpenSourceSoftwareisgrowingontrees.

• Consequently theyarenotwillingtocontributetothedevelopment andsoftwaremanagementatall.

• Theyassumethatcomplainsareveryvaluablecontributions.• Nextconsequence isthatOpenSourceteamsmainly

implementsoftwarefeatures,whicharerequiredbythelabs,wherethecoreteammembersarehosted.

• Inordertoexplorenewcommunitiesandsatisfytheirsoftwarerequirements,OpenSourceProjectsneedexternalmoney.


Howarenewfeaturesfunded?

• Thisiswhere“National”and“European”projectscomeintoplay.

• FordCache,this:– wasEMI– istheGermannationalLSDMAproject– andwillbeINDIGO-DataCloud

• Thedrawback:Theytellyouwhattheywanttoseeinyourcode.


Fundedfeaturesarenotnecessarythoseyouneed?

• However,dCachehassomeinvariantobjectives:– Themasterplan(lastslideofthispresentation)– Beuptodateonnewtechnologies,eithersoftwareorhardware.

– Attractnewcommunitiesastheirspecificrequirements, iftheycanbefulfilled,makedCacheevenbetter.

• Itcanbeabittrickytotunethefundingprojectsexactlyintothedirectionofourobjectives.

• So,let’sseehowdCachemanaged/es that…….


FundinginfluencesdCachedevelopmenttopics

2010 2013

Standardization

NFS4.1/pNFS

HTTP/WebDAV

ContributingtotheDynamicFederation

INDIGO DataCloud

2015 2018

DataLifeCycleMultiTierStorage

QualityofService

MigrationArchivingAAI

Deployingnewtechnologies intoProductionandexploringnewcommunities


From2013tonow,wesloweddowndevelopmentbetweentwoverydemandingdevelopmentprojects,EMIandINDIGO-DataCloud,to:

• Deploynewlyimplementedtechnologiesintoproduction.

• Explorenewcommunitiesandlearnabouttheirneeds.


DeployingNFSintoproduction

• CMSGridInfrastructure@DESY• TheDesy-Cloud• FERMIlab(variousIntensityDrontier)• Andtheissues


NewProductionSystemsbasedondCacheNFS.

NFS4.1/pNFS

DirectlowlatencyaccessWorkernodesHPC

dCacheBackendStorageLayer

WideAreaFTSGLOBUS(ONLINE)

Sync&ShareLaptopsMobileDevices

SeePaul’spresentationonThursday


CMSTierII@DESY

• SlowlymigratingCMSGridworkernodestoNFS4.1dataaccess.

• Goodexperienceaslongasthenetworkisstable.


NFS4.1pNFS dCap

ExecutionTime(hours)

JobEfficiency(CPU

/W

allTim

e)

JobEfficiency(NFS– dCap)


Aswithallnewsspec’s,thereareissues

• Networkproblemscausethesystemtobebehaveunpredictable.

• DataServerbehindfirewalls• WeakclientsonVM’s• SpecificationViolation– infinitestaterecoverywithLinuxkernel


Exploringnewcommunities.


Exploringnewcommunities.

• Jülich – AachenResearchAssociation,JADE– "SupercomputingandmodelingfortheHumanBrain(SMHB)”,associatedtotheEuropeanHumanBrainProject(PlenarybyKHMeier)

• MoSGrid– ScientificGatewayformolecularsimulation.

• VAVID– DataGatewayforanalyzingwindenergyinfrastructures


JADE

Aachen

Jülich


ProjectsinHPC

HPCjobsonsupercomputer

HPCjobsgetaccesstodCachestorage.


WiththestartofINDIGO-DataCloud,itsmoneyandalargerteam(8+3)wecancontinueto

explorenewhorizons.(Backtodevelopmentmode)


• NewDiskTechnologies– OpenEthernetDisks(HGST)

• NewObject-StoreBack-ends– CEPH

• NewEuropeanProjects(INDIGODC)– FocusingonDataQualityofServiceand– DataLifecycleManagement

Respondingtonewtechnologies


HGSTOpenEthernetDisks

• SmallARMCPUwithEthernetpiggybackedonregularDisk.

• Spec:– AnyLinux(Debian ondemo)– CPU32-bitARM,512Level2– 2GBDRAMDDR-3Memory

• 1792MBavailable

– BlockstoragedriverasSCSIsda– Ethernetnetworkdriveraseth0


HGSTOpenEthernetDisks(cont)

• AdditionalCPUisnotusedbydiskitselfandcanrunarbitrarycustomerOS.

• Diskisseenasregularblockdevice.

• Notyetonthemarket.• dCachegot5disksandweareevaluatingtorunpoolnodesonthediskitself.

• SeetalkonThursday.


ResponsetoCEPH

• CEPHcomplementsdCacheperfectly.– SimplifiesoperatingdCachedisks.– dCacheaccessesdataasobject-storeanywayalready.

• dCacheisevaluatinga‘twostepapproach’.– Eachpoolssees itownobjectspaceinCEPH– Allpoolshaveaccess totheentirespace,whichisaslightchangeofdCache

poolsemantics.• WouldmergeCEPHanddCacheadvantages

– MultiTier(Tape,Disk,SSD)– Multiprotocolsupportforacommonnamespace.

• Allprotocolsseethesamenamespace– AllthedCacheAAIfeatures

• SupportforX509,Kerberos,username/password


INDIGO-DataCloudCheat-Sheet

• Horizon2020projectstartingAprilorMay• Budget11.1MillionEuros(800.000fordCache)• 26Partners• Duration30months• TheprojectaimsforanOpenSourceDataandComputingplatformtargetedatscientificcommunities,deployableonmultiplehardware,andprovisionedoverprivateandpublice-infrastructures.

SeeLudek’s presentationonWednesday


INDIGOinanutshell

1. Self-service,on-demand2. Accessthroughthenetwork3. Resourcepooling4. Elasticity(withinfinite resources)5. Payasyougo

Intheend,ApplicationsRule.

StolenfromDavide Salomoni (ProjectDirector)


dCacheinvolvementinINDIGO

• dCacheismostlyinvolvedinWP4,whichisaboutVirtualInfrastructures.(IaaS)

• Forstoragesystems,likedCache,thisessentiallymeansSDS(SoftwareDefinedStorage),whichaccordingtoWikipedia is:– Software-definedstorage(SDS) isanevolvingconceptforcomputerdatastoragesoftwaretomanagepolicy-basedprovisioningandmanagementofdatastorageindependentofhardware.


SDSaccordingtodCache

• User/PaaS defined“QualityofService”management– User/PaaS defined“AccessLatency”

• SSDorTapedependingfromapplicationrequirements.

– User/PaaS Defined“DataProtection”• Ononedisk,twodisksortreetapesdependingonhowpreciousyourdatais.

– User/PaaS Defined“DataMigrationPolicies”• LikeAmazonGlaciervers.S3

• AutomaticStorage-Tiermigration– Basedonaccessprofile

• Allthiswouldn’tbeneededifSSD’s wouldbecheapand100%reliable.


dCacheiswellprepared

HistoricallydCachesupportsmulti-tierstorageandthecorrespondingtransition.

SSDs

SpinningDisks

Tape, BlueRay…

Virtual File-systemLayer

NFS/pNFS gridFTPhttpWebDAV xRootd/dCapAutomatic

andManualMedia

transitions


Recentlyadded

Weoptimizedthe‘smallfile’problemwithdisk<->tapetransitions.

TapeSystem

Containers


What’smissing

• Mainlyacommonagreement(standard)onhowtotriggertransitions.(Protocol,API??)

• WehavesomeexperiencewithSRM,howeveritseemsnottobesuitableforthispurpose.

• AnothercandidateisCMDI(SNIA),whichisanindustrystandard.

• MigrationPoliciesarealreadydiscussed,documentedandimplementedwithinRDA(PracticalPolicyWorkingGroup).

• DetailswillonlybeavailableaftertheINDIGOkickoffmeetingendofApril‘15.


Summary

Magically,uptonow,attherightmoment,therewasalwaysanEUorNationalProject,fundingdCacheexactlyforthosefeaturesoractivites,dCachewasplanningtodoanywayandwiththattheyhelpedusfollowingourmasterplan:

ThesupportoftheCompleteScientificBigDataLifeCycleManagement.


ScientificDataLifecycle

HighSpeedDataIngest

FastAnalysisNFS4.1/pNFS

WideAreaTransfers(Globus Online,FTS)byGridFTP

Visualization&SharingbyWebDAV,OwnCloud


Don’tforget

UpcomingdCacheWorkshop


TheEND

furtherreadingwww.dCache.org

Supporting the scientific data lifecycle · 2016-01-07 · • Horizon 2020 project starting April...

Documents

Transcript of Supporting the scientific data lifecycle · 2016-01-07 · • Horizon 2020 project starting April...