BigData Express: Toward Predictable, Schedulable, and High...
16
BigData Express: Toward Predictable, Schedulable, and High-Performance Data Transfer BigData Express Research Team September 25, 2019
Transcript of BigData Express: Toward Predictable, Schedulable, and High...
BigData Express: Toward Predictable, Schedulable, and High-Performance Data Transfer
BigData Express Research Team September 25, 2019
Acknowledgments
This work was supported by the U.S. DOE Office of Science ASCR network research program
BigData Express
• BigData Express: a schedulable, predictable, and high-performance data transfer service– A peer-to-peer, scalable, and extensible data transfer model– A visually appealing, easy-to-use web portal– A high-performance data transfer engine– A time-constraint-based scheduler– On-demand provisioning of end-to-end network paths with guaranteed QoS– Robust and flexible error handling– CILogon-based security (certificate version)
BigData Express Major Components
• BigData Express Web Portal– Access to BigData Express services
• BigData Express Scheduler– Time-constraint-based scheduler– Co-scheduling DTN, storage, & network
• AmoebaNet– Network as a service– Rate control
• mdtmFTP– High-performance data transfer engine– http://mdtm.fnal.gov
• DTN Agent– Manage and configure DTNs– Collect & report DTN configuration and
status
• Storage Agent– Manage and configure storage systems– I/O estimation
• Data Transfer Launching Agent– Launch data transfer jobs– Support different data transfer
protocols
BigData Express -- Distributed
A Peer-to-Peer model
DTNs
Networks
DTNs
DTNs
DTNs
DTNs
…
Data Transfer Federation 1
Data Transfer Federation 3
…
…
Data Transfer Federation 2
BigData Express -- Flexible
• Flexible to set up data transfer federations
• Providing inherent support for incremental deployment
BDE SchedulerSDN Agent
Storage Agent
DTN AgentDTN Agent
SDN Agent
BDE Web Portal
Data Transfer Launching AgentData Transfer
Launching AgentData Transfer Launching Agent
Message Queue
SDN Agent(AmoebaNet)
DTN Agent
Storage AgentStorage Agent
BDE Scheduler
BigData Express -- Scalable
• BigData Express scheduler manages site resources through agents• Use MQTT as message bus
BDE SchedulerSDN Agent
Storage Agent
DTN AgentDTN Agent
SDN Agent
BDE Web Portal
Data Transfer Launching AgentData Transfer Launching Agent
Data Transfer Launching Agent
mdtmFTPPlugin
GridFTPPlugin
XrootDPlugin
…
Message Queue
SDN Agent(AmoebaNet)
DTN Agent
Storage AgentStorage Agent
BDE Scheduler
BigData Express -- Extensible
• Extensible Plugin framework to support various data transfer protocols• mdtmFTP, GridFTP, XrootD, …
BigData Express -- End-to-End Data Transfer Model
• Application-aware network serviceo On-demand programming
• Fast-provisioning of end-to-end network paths with guaranteed QoS
• Distributed resource negotiation & brokeringLAN WAN LAN
mdtmFTP
AmoebaNet AmoebaNetSENSE
Edge DTNStorage Edge DTN Storage
DTN Agent
Storage Agent
mdtmFTP
DTN Agent
Storage Agent
Web Portal
Scheduler
Data Transfer Launching Agent
Web Portal
Scheduler
Data Transfer Launching Agent
Site A - Smart E2EData Transfer Orchestrator
Site B - Smart E2EData Transfer Orchestrator
A End-to-End Data Transfer Loop with Guaranteed QoS
Resource negotiation & brokering
Resource
negotiation & brokering
Resource
negotiation & brokering
mdtmFTP A high-performance data transfer tool
• Pipelined I/O-centric design to streamline data transfer• Multicore-aware data transfer middleware (MDTM) optimizes use of
underlying multicore system• Extremely efficient in transferring of Lots Of Small Files• Various optimization mechanisms– Zero copy – Asynchronous I/O– Batch processing
A DOE/SC/ASCR-sponsored research projectSoftware is available at: http://mdtm.fnal.gov
MDTM Middleware Services
OS Services
mdtmFTP
Hardware
Access services
Access services
Access services
BigData Express SC18 DEMO
4/3
4/6
FNAL Border router
ESNET
40GE
vlan 3619
49
50
BDE4
Pica8 P3930
40GE
FNAL
SENSE Service AmoebaNet
bde-hp2.fnal.gov
yosemite.fnal.gov
BDE Web Protal
BDE Scheduler
47
48
65
66
Testbed Switch
100GE
Border Router
vlan 3619
DTN
UMD
SENSE Service
180-147.research.maxgigapop.net
BDE Web Protal
BDE Scheduler
STP
10.36.19.15
vlan 3619SENSE Path
180-149.research.maxgigapop.net
DTN
180-148.research.maxgigapop.net
BDE3
ProductionSwitch
Production Switch
40GE
HP Z91000
KISTI
AmoebaNet
BDE Web Protal
BDE Scheduler
DTN3DTN2
134.75.125.77
134.75.125.78 134.75.125.79
192.2.2.8 192.2.2.910GE 10GE
StarLight
vlan 1662STP
4/1
4/2
4/3
4/4
4/5
4/6
BDE1 BDE2
Pica8 P5101
40GE 40GE40GE
7374
192.2.2.1 192.2.2.2
77
KREONET
4/1
vlan 1662
STP
STP
StarLight
165.124.33.157
BDE Web Protal
BDE Scheduler
DTN 165.124.33.142
DTN
CENI162.244.229.52Ottawa
DTN
162.244.229.116Hanover
100GE 100GE
UVA
145.100.132.188
BDE Web Protal
BDE Scheduler
DTN 145.100.132.187
vlan
203
8
40GE
KSTAR
BDE Web Protal
BDE Scheduler
DTN3DTN2
203.230.120.130
203.230.120.127
100GE 100GE
10.36.19.11
203.230.120.128
203.230.120.227 203.230.120.228
10.250.38.107
10.250.38.53
• Asia– KISTI, South Korea– KSTAR
• Europe– University of Amsterdam, Netherlands
• North America– Fermilab– StarLight, Northwestern University– UMD/MAX, University of Maryland, College Park– Ciena (Canada)
• US East• CA East
• Australia & Pacific areas– National Computational Infrastructure (NCI)
BigData Express – Deployment
• License– Apache 2.0
• Docker release– Repository
• publicregistry.fnal.gov
– Docker images• bdeagent:1.5-xenial• bdeserver:1.5-xenial• bdeportal:1.5-xenial
– Pull instruction• https://bigdataexpress.fnal.gov/Releases.html
• Source code package release– Available
BigData Express Release
• BigData Express Admin Manual–https://bigdataexpress.fnal.gov/admin_manual/index.html
• BigData Express Design and Architecture–https://bigdataexpress.fnal.gov/design_manual/index.html
• BigData Express Web Portal User Manual–https://bigdataexpress.fnal.gov/portal_manual/index.html
BigData Express Documents
• REST APIs support for scientific workflows
• Support Kubernetes–Automating docker-based deployment
• Support CILogon (Token version)
BigData Express Roadmap
More information about BigData Express
http://bigdataexpress.fnal.gov
Contact: [email protected]
This document was prepared by BigData Express using the resources of the Fermi National Accelerator Laboratory (Fermilab), a U.S. Department of Energy, Office of Science, HEP User Facility. Fermilab is managed by Fermi Research Alliance, LLC (FRA), acting under Contract No. DE-AC02-07CH11359.
