Kantor LSST Networks and Sitestical2019.redclara.net/media/attachments/2019/08/26/kantor-lsst... ·...
Transcript of Kantor LSST Networks and Sitestical2019.redclara.net/media/attachments/2019/08/26/kantor-lsst... ·...
1TICAL2019- September3,2019
LSSTDistributedComputingandNetworks
JeffreyKantorSr.ManagerTICAL2019
September3,2019
2TICAL2019- September3,2019 Photo:GianlucaLombardi
Isan8.4metertelescopebeingbuiltinChile
Willconducta10yearsurveyofthesky
Beginsoperationsinlate2022
Photo:Gianluca Lombardi
LargeSynopticSurveyTelescope(LSST)
3TICAL2019- September3,2019
Eachimageis:• 9.6squaredegrees• 3.2gigapixels• 6.4GBofdata
Observingis:• 2x15sec
exposure/pointing• ~2000images/night• 15TB/night• 7PB/yr
LSSTObservingCadenceandDataRate
4TICAL2019- September3,2019
SummitandBaseSites
TelescopeandCameraDataAcquisition
CrosstalkCorrectionLong-termstorage(copy1)ChileanDataAccessCenter
DedicatedLongHaulNetworks
Tworedundant100Gbit linksfromSantiagotoFlorida(existing
fiber)Additional100Gbit link
(spectrumonnewfiber)fromSantiago– Florida
(ChileandUSnationallinksnotshown)
SatelliteProcessingCenter
CC-IN2P3,Lyon,France
DataReleaseProduction(50%)FrenchDAC
ArchiveSiteArchiveCenter
AlertProductionDataReleaseProduction(50%)CalibrationProductsProduction
EPOInfrastructureLong-termStorage(copy2)
DataAccessCenterDataAccessandUserServices
HQSiteScienceOperationsObservatoryManagementEducationandPublicOutreach
CameraSiteCameraSupport
LSSTSitesandLong-HaulNetworks
5TICAL2019- September3,2019
NightlyDataFlows
Archive Site
Base Site
VOEventBroker Serverin La Serena
(Non-DM)
10M Alertsper night(average)
60secondrequirementstopshere
VOEvent Broker Server(Non-DM)
OCS/TCSData(upto1Gbps)
EngineeringFacilityDBDataQualityMetricsAtmosphericTelescopeImagesandSpectra
DMData(upto39Gbps)Northbound SouthboundCrosstalkImages CalibrationImagesRawImages DIAObject CatalogWavefront Images DIASource CatalogRawCalibrationImages
RawandCrosstalkCorrectedImages24- 30TB/night6.4GBperimage
18-bituncompressedcompressedatBaseSite
6TICAL2019- September3,2019
Non-NightlyDataFlowsandDistributedComputing
Archive Site
Base Site
US DAC
CalibrationproductsL2DataProducts
Physical ShipmentsArchive Site to Base SiteAnnually
• Hardware• Data Release Products
(secondary method)
Chilean DAC
EPO Center
CalibrationproductsL2DataProducts
L2DataProducts
DataRelease:CoaddsObjectcatalogSourcecatalogForcedSourcecatalog
Eastbound WestboundRawimages 1/2ofL2DataProductsCalibrationproductsEngineeringFacilityDatabase1/2ofL2DataProducts(coadds,catalogs,SDQA)
French Site
7TICAL2019- September3,2019
LSSTDataProductsandDistributedComputing
− Nightlydataproducts– Alerts– Differenceimagesandcatalogs– 60slatencyfromtimeofreadoutfromcamera
− Annualdataproducts– Processallaccumulateddatafromstartofsurvey(distributedUS,France)– Producesallnightlydataproductsplus– Catalogsofdeep,faintobjects– “Forced”photometricmeasurements
− Supportingcommunity-developeddataproducts• “Nearby”computingandstorageatDataAccessCenters• Software(middleware,pipelines,algorithms,tools)
8TICAL2019- September3,2019
CurrentLSSTSchedule– 3.5MonthsContingency
Camera
FullIntegration&Verification
OperationalReadinessReview
EarlyIntegration&Testing
DataManagement
Telescope&Site
+3.5
ConstructionStart
FullSystemIntegration“Start”
ScienceOperationsStart
FinalDRPRelease
PreCommissioningPreparations
Facility Support
FullOperations
CD-4
ComCamonSummit
Pre-
NSFMREFC
DOEMIE
Commissioning
Operations
ScheduleContingency
17July2019
MIEtoOps
CriticalPath
EngineeringFirstLight
SystemFirstLight
CriticalPathisDome/TMA/AIV
Now
9TICAL2019- September3,2019
LSSTLongHaulNetworkLinks(BaselineFY21)
Tucson
LaSerena
CerroPachon
Santiago
SaoPaolo
Chicago40x10GbAURA(ScienceData)
20x10GbAURA(OCS/TCSControl)
Panama
1x100Gb1x40– 100GbREUNA
2x100GbAmlight/ANSP/RedCLARA
CC-IN2P3Lyon
Miami
100Gb(spectrum)Amlight/AngolaCable
Champaign
2*100GbNCSA/ICCN
Santiago
BocaRaton
10GbAmlight/internet2
10GbIN2P3
10x10GbNon-LSSTAURA(Fail-over) 100Gb
(spectrum)AmLight/RNP/RedCLARA(viaBuenosAires)
SecondaryPath
PrimaryPath
SLAC,OtherDOE
10GbESnet(institutional)
2x100GbEsnet/FLR/i2
10TICAL2019- September3,2019
LSSTInternationalNetworks(Chile– US)
Dark Fiber
LSSTBorderRouter
LSSTCerro Pachon/
La Serena
Spectrum
Spectrum
SaoPaolo
Miami FL Boca RatonFL
LSST Dedicated Primary Path
LSST Dedicated Backup Path
Central LinkSantiago
Panama
REUNASantiago
Dark Fiber
11TICAL2019- September3,2019
LSSTUSNetworks(Florida– Chicago)
Internet2AL2S
AtlanticWave100G
LSST(NCSA)
LSST
Spectrum
Spectrum
Spectrum
ESNET
ESN
ET
ChicagoIL Site A
AtlantaGE
JAXFL
Dark Fiber
Dark FiberBoca RatonFL
MiamiFL
FLRWaves
LSST Dedicated Primary Path
AmLight shared
FLR SharedLSST + AmLight
LSST Dedicated Backup Path
I2 AL2S If needed
ChicagoIL Site B
LSST Network – US Networks
I2 AL2
S
10/1/19 2x20G10/1/20 2x100G
Plan A: 10/1/20 1x100GPlan B: 10/1/21 1x100G
AmLig
ht P
rote
cted
FLRnet
10/1/2019
*
* Backhaul being addressed
12TICAL2019- September3,2019
LSSTNorthernNetworks(Champaign– Chicago– Lyon)
NCSA
600 W.Chicago
710 NLSDChicago
Internet2
100G Shared
MREN
100G Shared
IN2P3
ESnet
ESnet
100G Shared
100G Shared
GEANT
AL2S
MANLAN
RENATER
2x10G
Atlantic WaveAtlanta
ESnet
ESnet
13TICAL2019- September3,2019
ESNet forLSST
WASH
PHILNEWY32AOANEWY1118TH
BOSTALBA
BUFF
CLEV
JACK
CHAR
RALE
CHAT
ASHB
PITT
ATLA
DETR
INDI
CINC
LOUI
NASH
MEMP
JCSN
BATO
DALL
KANS
STLO
TULS
SANA
ALBQ
ELPA
DENV
PUEB
PHOE
TUCS
SALT
LASV
SAND
LOSA
SANL
RENO
BOIS
SEATOLYM
PORT
EUGE
SACR
SUNN
STAR
CHIC EQCH
HOUS
HOUH
BALT
Ciena_Map v12pld 9/30/2016
ESnet6 Optical Footprint: Add/DropsWave options Atla-Chic
atla-cr5rtr
LSSTDWDM
ESnetIP/MPLS core
star-cr5rtr
chic-cr5rtr
to Florida
MREN 8700
StarLight
NCSArtr1
ATLA HUB180 PeachTree(Site A)
Pre-FY21
100G100G
100G
100G
100G
100G
Before FY21 but after 2018 there will be a switch between the DWDM and alta-cr5 rtr. The handoff will be 1x100G.
atla-cr5rtr
LSSTDWDM
star-cr5rtr
chic-cr5rtr
to Florida
MREN 8700
Starlight
NCSArtr1
Post-FY21
100G 100G
100G
100G
100G
ESnetDWDM
ESnetDWDM
ESnetDWDM
LSST SW
LSST SW
LSST SW
100G
100G
100G
100G
100G
100G
PS
PS
PS
Last update 1/18/2018
600 W. Chicago(Site B)
710 N. Lake Shore Drive(Site C)
LSST SW
PS
NCSArtr2
NCSArtr2
710 N. Lake S
hore Drive
(Site C
)
600 W. C
hicago
(Site B)
100G
ATLA HUB180 PeachTree(Site A)
100G
FY20(2x20Gbps) FY21+(2x100Gbps)
14TICAL2019- September3,2019
End-to-EndNetworkBandwidthEvolution
Note:LSSTinternet,web,voice,videogooverAURAcircuits,whichareshared,andarecurrentlylimitedby1GFirewallsinLSand10Gbps internet2linksintheUS.ThiswillbeimprovedbytheendofFY19bythemoveto10GfirewallsinLS.
15TICAL2019- September3,2019
FirstOpticLightDemonstration– December2017
− TheLSSTNetworkEngineeringTeam(NET)completedthefirstsuccessfultransferofdigitaldataoverLSST/AURAfiberopticnetworksfromtheSummitSiteonCerroPachon,ChiletotheBaseSiteinLaSerena,ChileandontotheArchiveSiteatNCSAinChampaign.Asetof6x10Gbps NetworkInterfacecardsonDataTransferNodes(DTN)configuredwithiPerf3generatedasustaineddatarateofapproximately44gigabitspersecond,overaperiodof24hours.Thisexceededthetesttargetof40gigabitspersecond.
DTN 1 NIC1NIC2
NIC3
DTN 2 NIC1NIC2
NIC3
DTN 3 NIC1NIC2
NIC3
LSST
DWDM
Transponder
10x10G
12
34
56
La Serena
ManagementRouter
REUNA
LSST
DWDM
Transponder
10x10G
12
34
56
AmLight
AndesLight1
Brocade
MLXe
1/11/2
1/31/4
1/51/6
3/2
3/1
10G SR+
1G UTP
DWDM
AmLight
Pacific
Atlantic
100G Leased
AmLight
AMPATH01
Brocade
MLXe
7/2
AmLight
AMPATH02
Brocade
MLXe
7/2
6/2
FLR +
Internet2
Internet2
Rtsw-chic
0/0/5
100G
VID 3890
VID 3891
VID 3892
VID 3893
VID 3894
VID 3895
VID 3890
VID 3891
VID 3892
VID 3893
VID 3894
VID 3895
VID 3890
VID 3891
VID 3892
VID 3893
VID 3894
VID 3895
Level3 SCL AMPATH6000W
ChicagoICCN NCSA/NPCF
VID 3890
VID 3891
VID 3892
VID 3893
VID 3894
VID 3895
7/1
7/1
DTN 3
NCSAEX9214
NIC
NCSAMX960
DTN 1NIC
DTN 2NIC
DTN 4NIC
DTN 5NIC
DTN 6NIC
NCSAnet
NIC
9/1/0
VID 3890
VID 3891
VID 3892
VID 3893
VID 3894
VID 3895
VID 3890
VID 3891
VID 3892
VID 3893
VID 3894
VID 3895
NIC
NIC
NIC
NIC
NIC
16TICAL2019- September3,2019
Base– LDF100Gbps NetworkDemonstration– November2018
− FollowinguponthefirstsuccessfultransferofdigitaldataoverLSST/AURAfiberopticnetworksfromtheSummitSiteonCerroPachon,ChiletotheBaseSiteinLaSerena,ChileandontotheArchiveSiteatNCSAinChampaign inDecember2017,thatachieved44Gbps sustaineddatarate.
− 100Gbps demonstrationfromLaSerena– NCSANovember14,duringSupercomputing2018.Datasetis10TBofDECam publicdata.UsingSCInet connectionsfromMiami– Dallas– Chicago.DM-provided“watch”JupyterHubapplicationwillbeusedtomonitortransferfromDataTransferNode(DTN)toGPFSfile-system/disk.
Via Atlantic
DWDM
LSSTDWDM10x10G
1
2
3
4
5
6
La Serena REUNA
LSSTDWDM10x10G
1
2
3
4
5
AmLightAndes1
1/1
1/2
1/3
1/4
1/5
3/2
3/110G SR
1G UTP
DWDM
AmLight
Via Pacific
100G Leased AmLightAMPATH01
7/2
AmLightAMPATH02
7/2
8/1
Cent
ury
Link
100G LR4
CLARA/SCL AMPATH ICCN NCSA/NPCF
5x100G
7/1
7/1
NCSAEX9214
NCSAMX960
NIC
9/1/0
100G DTNeth0
6/1
Dell Z9100
7
8
9
10
en
p6
s0
100G DAC
6
7
8
9
10
1/6
1/7
1/8
2/6
2/7
AmLightAndes2
3/2
3/1
6/1
SC18SCINet
StarLight & MREN
100GDTN
Version: SC18.0929.01
Date: 09.29.2018
Author: LSST-NET
Ready
Pending
OOB
NVMeNVMeNVMe
NVMe
100G
DTN
NVMeNVMe
NVMeNVMeNVMe
NVMeNVMe
NVMe
Co
rsa
De
ll Z
91
00
-ON
AMIS
Sens
or
8700HD
SC18
AMIS
Sens
or
8/2
AMISSensor
Mirror
Mirror
Mirror
1
14
3321
OOB
SC15
SC14
SL-10
100G DTNs
17TICAL2019- September3,2019
VirtualNetworkOperationsCenter(VNOC)
− TheLSSTLong-HaulNetworkisasingleoperationalnetworkimplementedwithdiverseprovidersandlinksthatmustworkseamlesslytogether
− TheVNOCprovidesasingle,integratedoperationalcapabilityforend-to-endengineering,performancemonitoring,security,maintenance,andallotheroperations
− Thegoaliscompletevisibilityinalllinksandsitestoallparticipants,withasingleentrypointforinformationandassistance
18TICAL2019- September3,2019
SummitSite(RemoteDisplayonly)
VirtualNetworkOperationsCenter(VNOC)Sites
HQSite(RemoteDisplayOnly)
FrenchSite(???)
SantiagoSite(FullVNOCSite)
ArchiveSite(FullVNOCsite)
BaseSite(FullVNOCsite)
MiamiSite(FullVNOCsite)