Science Data and the NDN paradigm · 2016. 9. 13. · Program-mable network Extreme Data Science...
Transcript of Science Data and the NDN paradigm · 2016. 9. 13. · Program-mable network Extreme Data Science...
![Page 1: Science Data and the NDN paradigm · 2016. 9. 13. · Program-mable network Extreme Data Science Facility (XDSF) MS-DESI producing ALS LHC JGI APS LCLS ... 5 × 1015 1 × 1016 1.5](https://reader033.fdocuments.us/reader033/viewer/2022052022/6036dbda55f6ef2e980a002e/html5/thumbnails/1.jpg)
ScienceDataandtheNDNparadigmInderMongaCTO,ESnetDivisionDeputyofTechnology,Scien?ficNetworkingDivisionLawrenceBerkeleyNa?onalLab NDNComm2015
![Page 2: Science Data and the NDN paradigm · 2016. 9. 13. · Program-mable network Extreme Data Science Facility (XDSF) MS-DESI producing ALS LHC JGI APS LCLS ... 5 × 1015 1 × 1016 1.5](https://reader033.fdocuments.us/reader033/viewer/2022052022/6036dbda55f6ef2e980a002e/html5/thumbnails/2.jpg)
Experimental and observational science deals with big and small
instruments, and a lot of data!
2
● DatavolumesareincreasingfasterthanMoore’sLaw
● Newalgorithmsandmethodsforanalyzingdata
● Infeasibletoputasupercompu>ngcenterateveryexperimentalfacility
Compu?ngSciencesArea
![Page 3: Science Data and the NDN paradigm · 2016. 9. 13. · Program-mable network Extreme Data Science Facility (XDSF) MS-DESI producing ALS LHC JGI APS LCLS ... 5 × 1015 1 × 1016 1.5](https://reader033.fdocuments.us/reader033/viewer/2022052022/6036dbda55f6ef2e980a002e/html5/thumbnails/3.jpg)
Alltoocommonprocessofdiscovery
3
![Page 4: Science Data and the NDN paradigm · 2016. 9. 13. · Program-mable network Extreme Data Science Facility (XDSF) MS-DESI producing ALS LHC JGI APS LCLS ... 5 × 1015 1 × 1016 1.5](https://reader033.fdocuments.us/reader033/viewer/2022052022/6036dbda55f6ef2e980a002e/html5/thumbnails/4.jpg)
ESnet
NewMath
Real->meanalysis
HighperformanceSoFware
Novelcompute/dataplaHorms
Datamgmt.andsharing
Program-mablenetwork
Extreme Data Science Facility
(XDSF)
MS-DESI
ALS
LHC
JGI
APS
LCLS
Other data-producing sources
-4-
‘Superfacility’Vision:Anetworkofconnectedfacili>es,soFwareandexper>setoenablenewmodesofdiscovery
![Page 5: Science Data and the NDN paradigm · 2016. 9. 13. · Program-mable network Extreme Data Science Facility (XDSF) MS-DESI producing ALS LHC JGI APS LCLS ... 5 × 1015 1 × 1016 1.5](https://reader033.fdocuments.us/reader033/viewer/2022052022/6036dbda55f6ef2e980a002e/html5/thumbnails/5.jpg)
ESnetisadedicatedmissionnetworkengineeredtoaccelerateabroadrangeofscienceoutcomes.
Wedothisbyofferinguniquecapabili?es,andop?mizingthenetworkfordataacquisi?on,data
placement,datasharing,datamobility.
![Page 6: Science Data and the NDN paradigm · 2016. 9. 13. · Program-mable network Extreme Data Science Facility (XDSF) MS-DESI producing ALS LHC JGI APS LCLS ... 5 × 1015 1 × 1016 1.5](https://reader033.fdocuments.us/reader033/viewer/2022052022/6036dbda55f6ef2e980a002e/html5/thumbnails/6.jpg)
ESnetisdesignedfordifferentgoalsthan
generalInternet.
![Page 7: Science Data and the NDN paradigm · 2016. 9. 13. · Program-mable network Extreme Data Science Facility (XDSF) MS-DESI producing ALS LHC JGI APS LCLS ... 5 × 1015 1 × 1016 1.5](https://reader033.fdocuments.us/reader033/viewer/2022052022/6036dbda55f6ef2e980a002e/html5/thumbnails/7.jpg)
Feb-94 Apr-98 Jun-02 Aug-06 Oct-10 Dec-14
1017
1010
1011
1012
1013
1014
1015
1016
Month
Tota
l
Bytes of Science Data Transferred Each Monthby the Energy Sciences Network
August2015:29.13PB
Lotsofdatatomovearound
![Page 8: Science Data and the NDN paradigm · 2016. 9. 13. · Program-mable network Extreme Data Science Facility (XDSF) MS-DESI producing ALS LHC JGI APS LCLS ... 5 × 1015 1 × 1016 1.5](https://reader033.fdocuments.us/reader033/viewer/2022052022/6036dbda55f6ef2e980a002e/html5/thumbnails/8.jpg)
Feb-94 Apr-98 Jun-02 Aug-06 Oct-10 Dec-14
3 × 1016
0
5 × 1015
1 × 1016
1.5 × 1016
2 × 1016
2.5 × 1016
Month
Tota
l
Bytes of Science Data Transferred Each Month
by the Energy Sciences Network
August2015:29.13PB
Lotsofdatatomovearound(contd.)
![Page 9: Science Data and the NDN paradigm · 2016. 9. 13. · Program-mable network Extreme Data Science Facility (XDSF) MS-DESI producing ALS LHC JGI APS LCLS ... 5 × 1015 1 × 1016 1.5](https://reader033.fdocuments.us/reader033/viewer/2022052022/6036dbda55f6ef2e980a002e/html5/thumbnails/9.jpg)
High-levelobjec>vesforscien>ficdata:alignmentwithNDNapproach
10/25/159
• Radicallysimplifyhowscien?ficusersmanage,moveandmanipulatelarge,distributed,sciencedatarepositories,butwithhigh-throughputend2end
• Abstractthestorageandnetworkcapabilityandloca?ondependencefromtheuser-datainterac?on
• Enabletheabilityforuserstospecifyandretrievepor?onsofdatatheworkflowneeds
• Createasecure,scalableframeworkbasedonintegrateddatamanagementandnetworktransport
![Page 10: Science Data and the NDN paradigm · 2016. 9. 13. · Program-mable network Extreme Data Science Facility (XDSF) MS-DESI producing ALS LHC JGI APS LCLS ... 5 × 1015 1 × 1016 1.5](https://reader033.fdocuments.us/reader033/viewer/2022052022/6036dbda55f6ef2e980a002e/html5/thumbnails/10.jpg)
UseCase#1
10/25/1510
ResearchersfromBerkeleyLabandSLACconductedproteincrystallographyexperimentsatLCLStoinves>gatephotoexcited
statesofPSII,withnear-real->mecomputa>onalanalysisatNERSC.
“Takingsnapshotsofphotosynthe?cwateroxida?onusingfemtosecondX-raydiffrac?onandspectroscopy,”NatureCommunica.ons5,4371(9July2014)
50TBmovedanight
![Page 11: Science Data and the NDN paradigm · 2016. 9. 13. · Program-mable network Extreme Data Science Facility (XDSF) MS-DESI producing ALS LHC JGI APS LCLS ... 5 × 1015 1 × 1016 1.5](https://reader033.fdocuments.us/reader033/viewer/2022052022/6036dbda55f6ef2e980a002e/html5/thumbnails/11.jpg)
UseCase#2:LHCONEdata–mul>plereplicas,globalreach
![Page 12: Science Data and the NDN paradigm · 2016. 9. 13. · Program-mable network Extreme Data Science Facility (XDSF) MS-DESI producing ALS LHC JGI APS LCLS ... 5 × 1015 1 × 1016 1.5](https://reader033.fdocuments.us/reader033/viewer/2022052022/6036dbda55f6ef2e980a002e/html5/thumbnails/12.jpg)
UseCase#3:Interna>onalClimateData
10/25/1512
![Page 13: Science Data and the NDN paradigm · 2016. 9. 13. · Program-mable network Extreme Data Science Facility (XDSF) MS-DESI producing ALS LHC JGI APS LCLS ... 5 × 1015 1 × 1016 1.5](https://reader033.fdocuments.us/reader033/viewer/2022052022/6036dbda55f6ef2e980a002e/html5/thumbnails/13.jpg)
Percep>onoflimita>onsofNDNmo>va>ngresearchques>ons
1. IfIammoving50TBofdatathroughasinglepath,fromanexperimenttoastoragefacility,IreallydonotwanttocacheitateveryintermediateNDNnode
– Whatistherightstrategyforalloca?ngdiskresourcestocaching?Whatifonedatatransferconsumesallcacheresourcesorthereisnotenoughspace?
2. Whatistheperformanceoftheend-to-enddatatransfer?HowcanIgetlineratethroughput?
3. HowdoIleveragetheknowledgeofnetworkcapabilityinchoosingthetransferpath?HowdoIbuildintheknowledgeofunderlayintotheNDNoverlay?
4. HowdoIleveragenetworkprogrammabilitytodotheabove?
5. Andmanyotherques?ons….
10/25/1513
![Page 14: Science Data and the NDN paradigm · 2016. 9. 13. · Program-mable network Extreme Data Science Facility (XDSF) MS-DESI producing ALS LHC JGI APS LCLS ... 5 × 1015 1 × 1016 1.5](https://reader033.fdocuments.us/reader033/viewer/2022052022/6036dbda55f6ef2e980a002e/html5/thumbnails/14.jpg)
Whereareweat?
• Collabora?onwithChristosandColoradoState–high-poweredNDNdevicesbetweenthreerepresenta?veclimatesitesasatestbed– Susmitworkingonansweringsomeofthehigh-levelobjec?vesasdescribed
• HEPandASCRinterestinNDNfromaresearchperspec?ve–paperearlierthisyear@CHEP,andPhilwilltalkaboutnext-stepsrightaher
• Interestinexpandingafedera?onofhigh-poweredNDNdeviceswiththerightstrategyforcachinganddatamanagement
• CombiningNDNwithSDN–wehaveanext-genSDNtestbedacrossUSandEurope–canwecombinethattoprovidetherightprimi?vesforhigh-performanceNDN?– Letsdoitera?veexperimenta?onandimprovement!!!!!!!
10/25/1514
![Page 15: Science Data and the NDN paradigm · 2016. 9. 13. · Program-mable network Extreme Data Science Facility (XDSF) MS-DESI producing ALS LHC JGI APS LCLS ... 5 × 1015 1 × 1016 1.5](https://reader033.fdocuments.us/reader033/viewer/2022052022/6036dbda55f6ef2e980a002e/html5/thumbnails/15.jpg)
ALBQ
AMST
ANL
AOFA
ATLA
BNL
BOIS
BOST
CERN
CHICDENV
ELPA
FNAL
HOUS
KANS
LANL
LBL LLNL
LOND
NASHNERSC
NEWY
ORNL
PNNLPNWG
SACR
SAND
SLAC
STAR
SUNN
WASH
ESnet PE Router
(2+)x10GE
(n)x10GE
Testbed Host
Deployed SDN Testbed node locations Deployed SDN Testbed connectivity overlay (using OSCARS circuits)
ESnetSDNTestbed
AMST
CERN
AOFA
WASH
STAR
ATLA
DENV
LBL
August2015 iDiscovery2020InderMonga15
StatusUpdate:• Testbeddeployedatallloca?ons• QoSsupportverified,pressrelease
nextweek• ENOSdemoonTestbed@SC
![Page 16: Science Data and the NDN paradigm · 2016. 9. 13. · Program-mable network Extreme Data Science Facility (XDSF) MS-DESI producing ALS LHC JGI APS LCLS ... 5 × 1015 1 × 1016 1.5](https://reader033.fdocuments.us/reader033/viewer/2022052022/6036dbda55f6ef2e980a002e/html5/thumbnails/16.jpg)
Thankyou!
• Pleasefeelfreetoemailmewithques?ons,commentsorarrowsat
imongaatesdotnet
10/25/1516
![Page 17: Science Data and the NDN paradigm · 2016. 9. 13. · Program-mable network Extreme Data Science Facility (XDSF) MS-DESI producing ALS LHC JGI APS LCLS ... 5 × 1015 1 × 1016 1.5](https://reader033.fdocuments.us/reader033/viewer/2022052022/6036dbda55f6ef2e980a002e/html5/thumbnails/17.jpg)
10/25/1517