GEC17 Using ExoGENI

Ilya Baldin ibaldin@renci.org

2

ExoGENI Overview

3

14 GPO-funded racks built by IBM◦ Partnership between RENCI, Duke and IBM◦ IBM x3650 M4 servers (X-series 2U)

1x146GB 10K SAS hard drive +1x500GB secondary drive 48G RAM 1333Mhz Dual-socket 8-core CPU Dual 1Gbps adapter (management network) 10G dual-port Chelseo adapter (dataplane)

◦ BNT 8264 10G/40G OpenFlow switch◦ DS3512 6TB sliverable storage

iSCSI interface for head node image storage as well as experimenter slivering

◦ Cisco(UCS-B) and Dell configuration also exist Each rack is a small networked cloud

◦ OpenStack-based with NEuca extensions◦ xCAT for baremetal node provisioning

http://wiki.exogeni.net

Testbed

4

ExoGENI is a collection of off-the shelf institutional clouds◦ With a GENI federation on top◦ xCAT – IBM product◦ OpenStack- RedHat product

Operators decide how much capacity to delegate to GENI and how much to retain for yourself

Familiar industry-standard interfaces (EC2) GENI Interface

◦ Mostly does what GENI experimenters expect

ExoGENI

ExoGENI at a glance

6

Rack Software Stack

Deployment structure

An ExoGENI cloud “rack site”

11

ExoGENI racks are separate aggregates but also act as a single aggregate◦ Transparent stitching of resources from multiple

racks ExoGENI is designed to bridge distributed

experimentation, computational sciences and Big Data◦ Already running HPC workflows linked to OSG

and national supercomputers◦ Newly introduced support for storage slivering◦ Strong performance isolation is one of key goals

ExoGENI unique features

12

GENI tools: Flack, GENI Portal, omni◦ Give access to common GENI capabilities◦ Also mostly compatible with

ExoGENI native stitching ExoGENI automated resource binding

ExoGENI-specific tools: Flukes◦ Accepts GENI credentials◦ Access to ExoGENI-specific features

Elastic Cluster slices Storage provisioning Stitching to campus infrastructure

ExoGENI experimenter tools

Presentation title goes here 13

ExoGENI activities

14

Compute nodes◦ Up to 100 VMs in each full rack◦ A few (2) bare-metal nodes◦ BYOI (Bring Your Own Image)

True Layer 2 slice topologies can be created ◦ Within individual racks ◦ Between racks◦ With automatic and user-specified resource binding and

slice topology embedding◦ Stitching across I2, ESnet, NLR, regional providers.

Dynamic wherever possible OpenFlow experimentation

◦ Within racks◦ Between racks◦ Include OpenFlow overlays in NLR (and I2)◦ On-ramp to campus OpenFlow network (if available)

Experimenters are allowed and encouraged to use their own virtual appliance images

Since Dec 2012◦ 2500+ slices

Experimentation

Virtual network exchange

Virtual colocampus net to circuit fabric

Multi-homedcloud hosts

with network control

Topology embedding and stitching

Computed embedding

Workflows, services,

etc.

Slice half-way around the world ExoGENI rack in Sydney, Australia Multiple VLAN tags on a pinned path from Sydney to

LA Internet2/OSCARS ORCA-provisioned dynamic circuit

◦ LA, Chicago NSI statically pinned segment with multiple VLAN

tags◦ Chicago, NY, Amsterdam◦ Planning to add dynamic NSI interface

ExoGENI rack in Amsterdam◦ ~14000 miles◦ 120ms delay

17

Strong isolation is the goal Compute instances are

KVM based and get a dedicated number of cores

VLANs are the basis of connectivity◦ VLANs can be best effort or

bandwidth-provisioned (within and between racks)

ExoGENI slice isolation

Scientific Workflows

Workflow Management Systems◦ Pegasus, Custom scripts, etc.

Lack of tools to integrate with dynamic infrastructures◦ Orchestrate the infrastructure in response to application◦ Integrate data movement with workflows for optimized

performance◦ Manage application in response to infrastructure

Scenarios◦ Computational with varying demands◦ Data-driven with large static data-set(s)◦ Data-driven with large amount of input/output data

Scientific Workflows

Dynamic Workflow Steps (Computational)

21

Workflow slices• 462,969 condor jobs since using the on-ramp to engage-submit3 (OSG)

Dynamic Workflow Steps (On-Ramp)

Dynamic Workflow Steps (Data-driven)

Hardware-in-the loop slices•Hardware-in-the-Loop Facility Using RTDS & PMUs (FREEDM Center, NCSU)

RENCI Data Center

Experimental PMU Data from RTDS

0 500 1000 1500 2000 2500 3000 35000.98

0.99

1

1.01

1.02

1.03

1.04

Time in seconds (sec.) with origin at: 12:38:00 EST

Voltage M

agnitude (

pu)

PMU Voltage Magnitudes in AEP - Richview Event

JF KR MF RK OR OS

60 65 70 75 80 85 90

-0.02

-0.015

-0.01

-0.005

0

0.005

0.01

0.015

0.02

Time (sec)

Fa

st O

scill

atio

ns (

pu

)

Bus 1Bus 2Midpoint

800 810 820 830 840 850 860 8709.1

9.2

9.3

9.4

9.5

9.6

9.7

9.8

Angle

diffe

rence (

deg)

Time in seconds starting at:04-Aug-2007 16:30:00

tva mac

PMU Measurements

1 #

112

11

ModeInterarea

aa

aa

ss

s

2 #

222

22

ModeInterarea

aa

aa

ss

s

3 #

332

33

ModeInterarea

aa

aa

ss

s

Zero/First Order Hold

2 #

222

22

ModeInterarea

i iaia

iaia

ss

s

Intra-clusterVirtualization

PoP at UNC Chapel Hill

PoP at Duke University

Cluster 1

Cluster 2Cluster 3

NC State PoP

Distributed Execution of Time-critical Synchrophasor Applications

Fiber optic network for PMU data communication using IEEE C37.118

New GENI-WAMS Testbed

Latency & Processing Delays

Packet Loss

Network Jitters

Cyber-security Man-in-middle attacks

Aranya Chakrabortty, Aaron Hurz (NCSU)Yufeng Xin (RENCI/UNC)

28

http://www.exogeni.net◦ ExoBlog http://www.exogeni.net/category/exoblog/

http://wiki.exogeni.net

Thank you!