Data Flow Prioritization for Scientific Workflows Using A Virtual SDX on ExoGENI

The 25th GENI Engineering Conference March 14-15, 2017

Florida International University, Miami, FL DoEASCRAward(DE-SC0012390)

Pegasus WMS

1. Pegasus determines “Express flow” requirements for workflow data transfers

2. Pegasus sends “modifyNetwork” QoS requests to messaging space

Mod

ify

requ

ests

Request Manager (RM), AHAB, VirtualSDX API

Mobius ++

3. RM uses AHAB and VirtualSDX API to send SDN QoS requests to SDX Tree Network

4. SDX Tree Network actuates QoS actions using REST API of SDN controller

6. Slice modified with required bandwidth QoS

NIaaS/Infrastructure

VirtualSDX + ExoGENI

Slic

e re

ady

with

Q

oS 7. RM sends ack. to Pegasus when Express flow QoS is set

5. SDN controller contacts SDX switches for each site

Workflow Models (Aspen) +

Simulation (CODES)Resource

Provisioner

Demonstration

Virtualized SDX

OpenFlowController

Data Node

HTCondorPool 1

HTCondorPool 2

(StarLight ExoGENI Rack)

(UMass ExoGENI Rack)

(PSC ExoGENI Rack)

ORCA/ExoGENI provisioning

Resource requestsData transfer requests

Data transfer QoS requirements

Pegasus Workflow Management System

AHAB Library(libndl +

libtransport)

Request Manager

AMQP space

Mobius++

Workflow Models (Aspen) + Simulation (CODES) Resource Provisioner

virtual SDX API (SDXTreeNetwork, …)

End-to-End Architecture

• We present a novel, dynamically adaptable networked cloud infrastructure driven by the demand of a data-driven scientific workflow running on dynamically provisioned 'slices' spanning multiple ExoGENI racks.• We show how a virtual Software Defined Exchange (SDX) platform, instantiated on ExoGENI, provides additional functionality for management of scientific workflows. • We demonstrate how tools developed in the DoE Panorama project can enable the Pegasus Workflow Management System to monitor and manipulate network connectivity and performance.

Abstract

Virtual SDX

Site 1

OpenFlowController

Site 4Site 3

Site 2

ExoGENI Slice(s)

DynamicConnectivity

ExoGENI Virtual SDX for Workflow Data Flows

Virtualized SDX

Site 1

OpenFlowController Site 3Site 2

Data Site

ExoGENI Slice with Virtual SDX + HTCondor Pools + Data Site

DynamicConnectivity

HTCondorPool

HTCondorPool

HTCondorPool

Virtual Overlay acting as SDX

SDX Tree Network with Express Flows

RYU OpenFlowController

OpenFlow v1.3

Site0

Site1

Express Flows (Bandwidth Provisioned)

Site5

Site6

Site4

Site3

Site2

Data Flow Prioritization

Virtual SDX transparently arbitrates workflow data flows communicated by Pegasus

AnirbanMandal,PaulRuth,IlyaBaldin (RENCI,UNC– ChapelHill)RafaelFerreiradaSilva,Ewa Deelman (USC)