Data Flow Prioritization for Scientific Workflows Using A ... · PDF fileData Flow...
1
Data Flow Prioritization for Scientific Workflows Using A Virtual SDX on ExoGENI The 25th GENI Engineering Conference March 14-15, 2017 Florida International University, Miami, FL DoE ASCR Award (DE-SC0012390) Pegasus WMS 1. Pegasus determines “Express flow” requirements for workflow data transfers 2. Pegasus sends “modifyNetwork” QoS requests to messaging space Modify requests Request Manager (RM), AHAB, VirtualSDX API Mobius ++ 3. RM uses AHAB and VirtualSDX API to send SDN QoS requests to SDX Tree Network 4. SDX Tree Network actuates QoS actions using REST API of SDN controller 6. Slice modified with required bandwidth QoS NIaaS/Infrastructure VirtualSDX + ExoGENI Slice ready with QoS 7. RM sends ack. to Pegasus when Express flow QoS is set 5. SDN controller contacts SDX switches for each site Workflow Models (Aspen) + Simulation (CODES) Resource Provisioner Demonstration Virtualized SDX OpenFlow Controller Data Node HTCondor Pool 1 HTCondor Pool 2 (StarLight ExoGENI Rack) (UMass ExoGENI Rack) (PSC ExoGENI Rack) ORCA/ExoGENI provisioning Resource requests Data transfer requests Data transfer QoS requirements Pegasus Workflow Management System AHAB Library (libndl + libtransport) Request Manager AMQP space Mobius++ Workflow Models (Aspen) + Simulation (CODES) Resource Provisioner virtual SDX API (SDXTreeNetwork, …) End-to-End Architecture • We present a novel, dynamically adaptable networked cloud infrastructure driven by the demand of a data-driven scientific workflow running on dynamically provisioned 'slices' spanning multiple ExoGENI racks. • We show how a virtual Software Defined Exchange (SDX) platform, instantiated on ExoGENI, provides additional functionality for management of scientific workflows. • We demonstrate how tools developed in the DoE Panorama project can enable the Pegasus Workflow Management System to monitor and manipulate network connectivity and performance. Abstract Virtual SDX Site 1 OpenFlow Controller Site 4 Site 3 Site 2 ExoGENI Slice(s) Dynamic Connectivity ExoGENI Virtual SDX for Workflow Data Flows Virtualized SDX Site 1 OpenFlow Controller Site 3 Site 2 Data Site ExoGENI Slice with Virtual SDX + HTCondor Pools + Data Site Dynamic Connectivity HTCondor Pool HTCondor Pool HTCondor Pool Virtual Overlay acting as SDX SDX Tree Network with Express Flows RYU OpenFlow Controller OpenFlow v1.3 Site0 Site1 Express Flows (Bandwidth Provisioned) Site5 Site6 Site4 Site3 Site2 Data Flow Prioritization Virtual SDX transparently arbitrates workflow data flows communicated by Pegasus Anirban Mandal, Paul Ruth, Ilya Baldin (RENCI, UNC – Chapel Hill) Rafael Ferreira da Silva, Ewa Deelman (USC)
Transcript of Data Flow Prioritization for Scientific Workflows Using A ... · PDF fileData Flow...
Data Flow Prioritization for Scientific Workflows Using A Virtual SDX on ExoGENI
The 25th GENI Engineering Conference March 14-15, 2017
Florida International University, Miami, FL DoEASCRAward(DE-SC0012390)
Pegasus WMS
1. Pegasus determines “Express flow” requirements for workflow data transfers
2. Pegasus sends “modifyNetwork” QoS requests to messaging space
Mod
ify
requ
ests
Request Manager (RM), AHAB, VirtualSDX API
Mobius ++
3. RM uses AHAB and VirtualSDX API to send SDN QoS requests to SDX Tree Network
4. SDX Tree Network actuates QoS actions using REST API of SDN controller
6. Slice modified with required bandwidth QoS
NIaaS/Infrastructure
VirtualSDX + ExoGENI
Slic
e re
ady
with
Q
oS 7. RM sends ack. to Pegasus when Express flow QoS is set
5. SDN controller contacts SDX switches for each site
Workflow Models (Aspen) +
Simulation (CODES)Resource
Provisioner
Demonstration
Virtualized SDX
OpenFlowController
Data Node
HTCondorPool 1
HTCondorPool 2
(StarLight ExoGENI Rack)
(UMass ExoGENI Rack)
(PSC ExoGENI Rack)
ORCA/ExoGENI provisioning
Resource requestsData transfer requests
Data transfer QoS requirements
Pegasus Workflow Management System
AHAB Library(libndl +
libtransport)
Request Manager
AMQP space
Mobius++
Workflow Models (Aspen) + Simulation (CODES) Resource Provisioner
virtual SDX API (SDXTreeNetwork, …)
End-to-End Architecture
• We present a novel, dynamically adaptable networked cloud infrastructure driven by the demand of a data-driven scientific workflow running on dynamically provisioned 'slices' spanning multiple ExoGENI racks.• We show how a virtual Software Defined Exchange (SDX) platform, instantiated on ExoGENI, provides additional functionality for management of scientific workflows. • We demonstrate how tools developed in the DoE Panorama project can enable the Pegasus Workflow Management System to monitor and manipulate network connectivity and performance.
Abstract
Virtual SDX
Site 1
OpenFlowController
Site 4Site 3
Site 2
ExoGENI Slice(s)
DynamicConnectivity
ExoGENI Virtual SDX for Workflow Data Flows
Virtualized SDX
Site 1
OpenFlowController Site 3Site 2
Data Site
ExoGENI Slice with Virtual SDX + HTCondor Pools + Data Site
DynamicConnectivity
HTCondorPool
HTCondorPool
HTCondorPool
Virtual Overlay acting as SDX
SDX Tree Network with Express Flows
RYU OpenFlowController
OpenFlow v1.3
Site0
Site1
Express Flows (Bandwidth Provisioned)
Site5
Site6
Site4
Site3
Site2
Data Flow Prioritization
Virtual SDX transparently arbitrates workflow data flows communicated by Pegasus
AnirbanMandal,PaulRuth,IlyaBaldin (RENCI,UNC– ChapelHill)RafaelFerreiradaSilva,Ewa Deelman (USC)
