GENI Experimenters Workshop (1)

8/4/2019 GENI Experimenters Workshop (1)

1/82

An Experimenters Guide to

OpenFlowGENI Engineering Workshop June 2010

Rob Sherwood(with help from many others)


2/82

Talk Overview

What is OpenFlow

How OpenFlow Works

OpenFlow for GENI Experimenters Deployments

Next Session: OpenFlow Office Hours

Overview of available software, hardware

Getting started with NOX


3/82

What is OpenFlow?


4/82

Short Story: OpenFlow is an API

Control how packets are forwarded

Implementable on COTS hardware

Make deployed networks programmable not just configurable

Makes innovation easier

Goal(experimenters perspective): No more special purpose test-beds

Validate your experiments on deployedhardware with real traffic at full line speed


5/82

How Does

OpenFlow Work?


6/82

Ethernet Switch


7/82

Data Path (Hardware)

Control PathControl Path (Software)


8/82

Data Path (Hardware)

Control Path OpenFlow

OpenFlow Controller

OpenFlow Protocol (SSL/TCP)


9/82

Controller

PC

Hardware

Layer

Software

Layer

Flow Table

MAC

src

MAC

dst

IP

Src

IP

Dst

TCP

sport

TCP

dport Action

OpenFlow Firmware

**5.6.7.8*** port 1

port 4port 3port 2port 1

1.2.3.45.6.7.8

OpenFlow Flow Table Abstraction


10/82

OpenFlow BasicsFlow Table Entries

Switch

PortMAC

src

MAC

dst

Eth

typeVLAN

ID

IP

Src

IP

Dst

IP

Prot

TCP

sport

TCP

dport

Rule Action Stats

1. Forward packet to port(s)

2. Encapsulate and forward to controller

3. Drop packet

4. Send to normal processing pipeline

5. Modify Fields

+ mask what fields to match

Packet + byte counters


11/82

ExamplesSwitching

*

Switch

Port

MAC

src

MAC

dst

Eth

type

VLAN

ID

IP

Src

IP

Dst

IP

Prot

TCP

sport

TCP

dportAction

* 00:1f:.. * * * * * * * port6

Flow Switching

port3

Switch

Port

MAC

src

MAC

dst

Eth

type

VLAN

ID

IP

Src

IP

Dst

IP

Prot

TCP

sport

TCP

dportAction

00:20.. 00:1f.. 0800 vlan1 1.2.3.4 5.6.7.8 4 17264 80 port6

Firewall

*

Switch

Port

MAC

src

MAC

dst

Eth

type

VLAN

ID

IP

Src

IP

Dst

IP

Prot

TCP

sport

TCP

dportForward

* * * * * * * * 22 drop


12/82

ExamplesRouting

*

Switch

Port

MAC

src

MAC

dst

Eth

type

VLAN

ID

IP

Src

IP

Dst

IP

Prot

TCP

sport

TCP

dportAction

* * * * * 5.6.7.8 * * * port6

VLAN Switching

*

Switch

Port

MAC

src

MAC

dst

Eth

type

VLAN

ID

IP

Src

IP

Dst

IP

Prot

TCP

sport

TCP

dportAction

* * vlan1 * * * * *

port6,

port7,

port9

00:1f..


13/82OpenFlowSwitch.org

Controller

OpenFlowSwitch

PC

OpenFlow UsageDedicated OpenFlow Network

OpenFlowSwitch

OpenFlowSwitch

OpenFlowProtocol

Aarons code

Rule Action Statistics

Rule Action Statistics Rule Action Statistics


14/82

OpenFlow Road Map

OF v1.0 (current)

bandwidth slicing

match on Vlan PCP, IP ToS

OF v1.1: Extensions for WAN, late 2010

multiple tables: leverage additional tables

tags, tunnels, interface bonding

OF v2+ : 2011?

generalized matching and actions: aninstruction set for networking


15/82

What OpenFlow Cant Do (1)

Non-flow-based (per-packet) networking

ex: sample 1% of packets

yes, this is a fundamental limitation

BUT OpenFlow can provide the plumbing toconnect these systems

Use all tables on switch chips

yes, a major limitation (cross-product issue)

BUT an upcoming OF version will exposethese


16/82

What OpenFlow Cant Do (2)

New forwarding primitives BUT provides a nice way to integrate them

New packet formats/field definitions

BUT plans to generalize in OpenFlow (2.0) Setup new flows quickly

~10ms delay in our deployment

BUT can push down flows proactively to avoiddelays

Only a fundamental issue when delays are largeor new flow-rate is high


17/82

OpenFlow forExperimenters

Experiment Setup

Design considerations

OpenFlow GENI architecture

Limitations


18/82

Why Use OpenFlow in GENI?

Fine-grained flow-level forwarding control

e.g., between PL, ProtoGENI nodes

Not restricted to IP routes or Spanning tree

Control real user traffic with Opt-In

Deploy network services to actual people

Realistic validations

by definition: runs on real production network

performance, fan out, topologies


19/82

Experiment Setup Overview

Step 1:Write/Configure/Deploy

OpenFlow controller

Step 2:Create Slice and

register experiment

Step 3:Control the traffic ofUsers that opt-in to

Your experiment

Each controller implements per-experimentcustom forwarding logic

Write your own or download pre-existing

Configure per-experiment topology, queuing

restricted to subset of real topology

Specify desired user traffic: e.g., tcp.port=80

Users opt-in via the Opt-In Manager website

Reserving a compute node makes theexperimenter a user on the network


20/82

Experiment Design Decisions

Forwarding logic (of course)

Centralized vs. distributed control

Fine vs. coarse grained rules Reactive vs. Proactive rule creation

Likely more: open research area


21/82

Centralized vs DistributedControl

Centralized Control

OpenFlowSwitch

OpenFlowSwitch

OpenFlowSwitch

Controller

Distributed Control

OpenFlowSwitch

OpenFlowSwitch

OpenFlowSwitch

Controller

Controller

Controller


22/82

Flow Routing vs. AggregationBoth models are possible with OpenFlow

Flow-Based

Every flow is individually

set up by controller Exact-match flow entries Flow table contains one

entry per flow Good for fine grain

control, e.g. campusnetworks

Aggregated

One flow entry covers

large groups of flows Wildcard flow entries Flow table contains one

entry per category offlows

Good for large number offlows, e.g. backbone


23/82

Reactive vs. ProactiveBoth models are possible with OpenFlow

Reactive

First packet of flow

triggers controller toinsert flow entries

Efficient use of flowtable

Every flow incurs smalladditional flow setuptime

If control connectionlost, switch has limitedutility

Proactive

Controller pre-populates

flow table in switch Zero additional flow setuptime

Loss of controlconnection does not

disrupt traffic Essentially requires

aggregated (wildcard)rules


24/82

Examples of OpenFlow inAction

VM migration across subnets energy-efficient data center network WAN aggregation network slicing default-off network scalable Ethernet scalable data center network load balancing formal model solver verification

distributing FPGA processing

Summary of demos in next session


25/82


26/82

Opt-In Manager

User-facing website + List of experiments

Users login and opt-in to experiments Use local existing auth, e.g., ldap

Can opt-in to multiple experiments subsets of traffic: Rob & port 80 == Robs port 80

Use priorities to manage conflicts

Only after opt-in does experimenter controlany traffic


27/82

Deployments


28/82

OpenFlow Deployment at Stanford

34

Switches (23)

APs (50)

WiMax (1)


29/82

Live Stanford

Deployment Statistics

http://yuba.stanford.edu/ofhallway/wide-right.htmlhttp://yuba.stanford.edu/ofhallway/wide-left.html
http://yuba.stanford.edu/ofhallway/wide-right.htmlhttp://yuba.stanford.edu/ofhallway/wide-left.htmlhttp://yuba.stanford.edu/ofhallway/wide-left.htmlhttp://yuba.stanford.edu/ofhallway/wide-left.htmlhttp://yuba.stanford.edu/ofhallway/wide-left.htmlhttp://yuba.stanford.edu/ofhallway/wide-right.htmlhttp://yuba.stanford.edu/ofhallway/wide-right.htmlhttp://yuba.stanford.edu/ofhallway/wide-right.html


30/82

GENI OpenFlow deployment (2010)

8 Universities and 2 National Research Backbones

Th EU P j t i il t GENI


31/82

Three EU Projects similar to GENI:Ophelia, SPARC, CHANGE

37

L2 Packet

Wireless

Routing

Pan-European experimental facility

L2 Packet

Optics

Content delivery

L2 Packet

Shadow networks

L2 L3Packet

Optics

Content delivery

L2 Packet

Emulation

Wireless

Contentdelivery


32/82

Other OpenFlow deployments

Japan

- 3-4 Universities interconnected by JGN2plus

Interest in Korea, China, Canada,

An Experiment of OpenFlow enabled Network


33/82

KOREA OpenFlow Network

Seoul

Daejeon

Deagu

Busan

Gwangju

Suwon

Controller

VLAN on KOREN

OpenFlow Switch (Linux PC)

NOX OpenFlow Controller

TJB

TJB Broadcasting Company

Japan OpenFlowNetwork

Sapporo

Studio

Asahi Broadcasting Cooperation (ABC) at Osaka, Japan

Sapporo Japan

Server

Data Transmission

An Experiment of OpenFlow-enabled Network(Feb. 2009 - Sapporo Snow Festival Video Transmission)

A video clip of Sapporo snow festival is transmitted to

TJB (Daejeon, KOREA) via ABC server (Osaka, JAPAN).


34/82

Highlights of Deployments

Stanford deployment McKeown group for a year: production and experiments

To scale later this year to entire building (~500 users)

Nation-wide trials and deployments

7 other universities and BBN deploying now

GEC9 in Nov, 2010 will showcase nation-wide OF

Internet 2 and NLR to deploy before GEC9

Global trials Over 60 organizations experimenting

2010 likely to be a big year for OpenFlow


35/82

Slide Credits

Guido Appenzeller

Nick McKeown

Guru Parulkar Brandon Heller

Lots of others

(this slide was also stolen)


36/82

Conclusion

OpenFlow is an API for controlling packetforwarding

OpenFlow+GENI allows more realisticevaluation of network experiments

Glossed over many technical details

What does the API look like?

Stay for the next session


37/82

An Experimenters Guide to

OpenFlow: Office HoursGENI Engineering Workshop June 2010

Rob Sherwood(with help from many others)


38/82

Office Hours Overview

Controllers

Tools

Slicing OpenFlow OpenFlow switches

Demo survey

Ask questions!


39/82

Controllers


40/82

Controller is King

Principle job of experimenter: customize acontroller for your OpenFlow experiment

Many ways to do this:

Download, configure existing controller

e.g., if you just need shortest path

Read raw OpenFlow spec: write your own

handle ~20 OpenFlow messages

Recommended: extend existing controller

Write a module for NOX www.noxrepo.org


41/82

Starting with NOX

Grab and build `git clone git://noxrepo.org/nox`

`git checkout -b openflow-1.0 origin/openflow-1.0`

`sh boot.sh; ./configure; make`

Build nox first: non-trivial dependencies

API is documented inline

`cd doc/doxygen; make html`

Still very UTSL


42/82

Writing a NOX Module

Modules live in ./src/nox/{core,net,web}apps/*

Modules are event based

Register listeners using APIs

C++ and Python bindings Dynamic dependencies

e.g., many modules (transitively) use discovery.py

Currently have to update build manually

Automated with ./src/scripts/nox-new-c-app.py

Most up to date docs are at noxrepo.org


43/82

Useful NOX Events

Datapath_{join,leave} New switch and switch leaving

Packet_in/Flow_in

New Datagram, stream; respectively Cue to insert a new rule/flow_mod

Flow_removed

Expired rule (includes stats) Shutdown

Tear down module; clean up state


44/82

Tools OpenFlow Wireshark plugin MiniNet

oftrace

many more


45/82

OpenFlow WireShark Plugin

Ships with OpenFlow reference controller


46/82

MiniNet

Machine-local virtual network

great dev/testing tool

Uses linux virtual network features

Cheaper than VMs

Arbitrary topologies, nodes

Scriptable Plans to move FV testing to MiniNet http://www.openflow.org/foswiki/bin/view/OpenFlow/Mininet


47/82

OFtrace

API for analyzing OF Control traffic

Calculate:

OF Message distribution

Flow Setup time

% of dropped LLDP messages

extensible

http://www.openflow.org/wk/index.php/Liboftrace


48/82

Slicing OpenFlow

Vlan vs. FlowVisor slicing

Use cases


49/82

Switch Based VirtualizationExists for NEC, HP switches but not flexible enough for GENI

Normal L2/L3 Processing

Flow Table

Production VLANs

Research VLAN 1

Controller

Research VLAN 2

Flow Table

Controller


50/82

OpenFlowSwitch

OpenFlowProtocol

OpenFlow FlowVisor& Policy Control

Craigs

Controller

Heidis

Controller

Aarons

Controller

OpenFlowProtocol

FLOWVISOR BASED VIRTUALIZATION

OpenFlowSwitch

OpenFlowSwitch

St f d I f t t U B th


51/82

The individual controllers and the FlowVisor are applications on commodity PCs (not

shown)

Stanford Infrastructure Uses Both

Flows

OpenFlow switches

WiMax

Packet processors

WiFi APs

Use Case VLAN Based


52/82

Use Case: VLAN BasedPartitioning

Basic Idea: Partition Flows based on Ports andVLAN Tags

Traffic entering system (e.g. from end hosts) is tagged VLAN tags consistent throughout substrate

SwitchPort

MACsrc

MACdst

Ethtype

VLANID

IPSrc

IPDst

IPProt

TCPsport

TCPdport

* * * * 1,2,3 * * * * *

* * * * 7,8,9 * * * * *

* * * * 4,5,6 * * * * *


53/82

OpenFlowProtocol

OpenFlowFlowVisor & Policy Control

Broadcast Multicast

OpenFlowProtocol

http

Load-balancer

FLOWVISOR BASED VIRTUALIZATIONSeparation not only by VLANs, but any L1-L4 pattern

OpenFlowSwitch

OpenFlowSwitch

OpenFlowSwitch

U C N CDN T b C l


54/82

Use Case: New CDN - Turbo Coral++

Basic Idea: Build a CDN where you control the entire network

All traffic to or from Coral IP space controlled by Experimenter

All other traffic controlled by default routing

Topology is entire network

End hosts are automatically added (no opt-in)

Switch

Port

MAC

src

MAC

dst

Eth

type

VLAN

ID

IP

Src

IP

Dst

IP

Prot

TCP

sport

TCP

dport

* * * * * 84.65.* * * * *

* * * * * * 84.65.* * * *

* * * * * * * * * *


55/82

Use Case: Aarons IP A new layer 3 protocol

Replaces IP

Defined by a new Ether Type

Switch

Port

MAC

src

MAC

dst

Eth

type

VLAN

ID

IP

Src

IP

Dst

IP

Prot

TCP

sport

TCP

dport

* * * AaIP * * * * * *

* * * !AaIP * * * * * *


56/82

Switches


57/82

Linux based Software Switch

Release concurrently with specification

Kernel and User Space implementations

Note: no v1.0 kernel-space implementation Limited by host PC, typically 4x 1Gb/s

Not targeted for real-world deployments

Useful for development, testing

Starting point for other implementations Available under the OpenFlow License (BSD Style) at

http://www.openflowswitch.org

Stanford ReferenceImplementation


58/82

Wireless Access Points

Two Flavors:

OpenWRT based (BusyboxLinux)

v0.8.9 only Vanilla Software (Full Linux)

Only runs on PC EnginesHardware

Debian disk image Available from Stanford

Both implementations aresoftware only.

G


59/82

NetFPGA

NetFPGA-based implementation

Requires PC and NetFPGA card

Hardware accelerated 4 x 1 Gb/s throughput

Maintained by Stanford University $500 for academics

$1000 for industry Available at http://www.netfpga.org

O S i h


60/82

Linux-based Software Switch Released after specification (v1.0 support 1 week old!)

Not just an OpenFlow switch; also supports VLANtrunks, GRE tunnels, etc

Kernel and User Space implementations Limited by host PC, typically 4x 1Gb/s

Available under the Apache License (BSD Style) athttp://www.openvswitch.org

Open vSwitch

OpenFlow Vendor Hardware


61/82

OpenFlow Vendor Hardware

more to follow...

NEC IP8800

HP ProCurve 5400

and others

Juniper MX-series

(prototype)Cisco Catalyst 6k(prototype)

CoreRouter

EnterpriseCampus

Data Center

CircuitSwitch

Wireless

Pronto

Prototype Product

Ciena CoreDirector

WiMAX (NEC)

Cisco Catalyst 3750

(prototype) Arista 7100 series(Q4 2010)

67

HP ProCurve 5400 Series (+


62/82

HP ProCurve 5400 Series (+others)

Praveen

Yalagandula

Jean

Tourrilhes

Sujata

Banerjee

Rick

McGeer

Charles

Clark

Chassis switch with up to 288 ports of 1G or 48x10G(+ other interfaces available)

Line-rate support for OpenFlow

Deployed in 23 wiring closets at Stanford

Limited availability for Campus Trials Contact HP for support details

NEC IP8800


63/82

NEC IP8800

24x/48x 1GE + 2x 10 GE

Line-rate support for OpenFlow

Deployed at Stanford

Available for Campus Trials

Supported as a product

Contact NEC for details:

Don Clark ([email protected])

Atsushi Iwata ([email protected])

Hideyuki

Shimonishi

Jun

Suzuki

Masanori

Takashima

Nobuyuki

Enomoto

Philavong

Minaxay

Shuichi

Saito

Tatsuya

Yabe

Yoshihiko

Kanaumi(NEC/NICT)

Atsushi

Iwata(NEC/NICT)

P t S it h
mailto:[email protected]:[email protected]:[email protected]:[email protected]:[email protected]:[email protected]


64/82

Pronto Switch

Broadcom based 48x1Gb/s + 4x10Gb/s Bare switch you add the software

Supports Stanford Indigo and Toroki releases

See openflowswitch.org blog post for more details

Stanford Indigo Firmware for


65/82

Sta o d d go a e oPronto

Source available under OpenFlow License to partiesthat have NDA with BRCM in place

Targeted for research use and as a baseline for vendorimplementations (but not direct deployment)

No standard Ethernet switching OpenFlow only!

Hardware accelerated

Supports v1.0

Contact Dan Talayco ([email protected])

T ki Fi f P t


66/82

Toroki Firmware for Pronto

Fastpath-based OpenFlow Implementation Full L2/L3 management capabilities on switch

Hardware accelerated

Availability TBD

Ciena CoreDirector


67/82

Ciena CoreDirector

Circuit switch with experimental OpenFlow support Prototype only

Demonstrated at Super Computing 2009

Juniper MX Series


68/82

Umesh

Krishnaswamy

Michaela

Mezo

Parag

Bajaria

James

Kelly

Bobby

Vandalore

Juniper MX Series

Up to 24-ports 10GE or 240-ports 1GE OpenFlow added via Junos SDK

Hardware forwarding

Deployed in Internet2 in NY and at Stanford

Prototype Availability TBD

Cisco 6500 Series


69/82

Flavio

Bonomi

Sailesh

Kumar

Pere

Monclus

Various configurations available Software forwarding only

Limited deployment as part of demos

Availability TBD

Work on other Cisco models in progress

Cisco 6500 Series

Stanford Reference Controller


70/82

Comes with reference distribution Monolithic C code not designed for extensibility

Ethernet flow switch or hub

Stanford Reference Controller

NOX Controller


71/82

Available at http://NOXrepo.org Open Source (GPL)

Modular design, programmable in C++ or Python

High-performance (usually switches are the limit)

Deployed as main controller in Stanford

NOX Controller

Martin

Casado

Scott

Shenker

Teemu

Koponen

Natasha

Gude

Justin

Pettit

Simple Network Access Control (SNAC)
http://noxrepo.org/http://noxrepo.org/


72/82

Available at http://NOXrepo.org Policy + Nice GUI

Branched from NOX long ago

Available as a binary

Part of Stanford deployment

Simple Network Access Control (SNAC)
http://noxrepo.org/http://noxrepo.org/


73/82

Demo Previews

FlowVisor

Plug-n-Serve

Aggregation

OpenPipes

OpenFlow Wireless

MobileVMs ElasticTree

Demo Infrastructure with Slicing


74/82

The individual controllers and the FlowVisor are applications on commodity PCs (notshown)

Demo Infrastructure with Slicing

Flows

OpenFlow switches

WiMax

Packet processors

WiFi APs

Be sure to check out the demos during the break!!


75/82

OpenFlow Demonstration Overview

NetworkVirtualization

FlowVisor

Hardware

Prototyping OpenPipesLoad Balancing PlugNServe

Energy Savings ElasticTree

Mobility MobileVMs

Traffic Engineering Aggregation

Wireless Video OpenRoads

Topic Demo

FlowVisor Creates Virtual Networks


76/82

FlowVisor Creates Virtual Networks

OpenFlowSwitch

OpenFlowSwitch

OpenFlowSwitch

OpenFlowProtocol

FlowVisor

OpenPipes

Demo

OpenRoads

Demo

OpenFlow

Protocol

PlugNServe

Load-balancer

OpenPipesPolicy

FlowVisor slicesOpenFlow networks,

creating multiple isolatedand programmable

logical networks on thesame physical topology.

Each demo presentedhere runs in an isolatedslice of Stanfords

production network.

Plumbing with OpenFlow

O Pi


77/82

Plumbing with OpenFlowto build hardware systemsOpenPipes

Partition hardware designs

TestMixresources

Plug-n-Serve:


78/82

Goal: Load-balancing requests in unstructured networks

Plug n Serve:Load-Balancing Web Traffic using OpenFlow

OpenFlow means

Complete control over traffic within thenetwork

Visibility into network conditions

Ability to use existing commodity hardware

What we are showing

OpenFlow-based distributed load-balancerSmart load-balancing based on network and server

loadAllows incremental deployment of additional

resources

This demo runs on top of the FlowVisor, sharing the same physical network with other experiments and production traffic.

Dynamic Flow Aggregation on an OpenFlow Network


79/82

ScopeDifferent Networks want different flow granularity (ISP, Backbone,)

Switch resources are limited (flow entries, memory) Network management is hard

Current Solutions : MPLS, IP aggregation

How OpenFlow Helps?Dynamically define flow granularity by wildcarding arbitrary header fields

Granularity is on the switch flow entries, no packet rewrite or encapsulation

Create meaningful bundles and manage them using your own software (reroute, monitor)Higher Flexibility, Better Control, Easier Management, Experimentation


80/82

Intercontinental VM Migration


81/82

Intercontinental VM Migration

Moved a VM from Stanford to Japan without changing its IP.

VM hosted a video game server with active network connections.

ElasticTree:


82/82

ElasticTree:Reducing Energy in Data Center Networks

The demo: Hardware-based 16-

node Fat Tree

Your choice of traffic

pattern, bandwidth,optimization strategy

Graph shows livepower and latency

Shuts off links and switches to reduce data center power Choice of optimizers to balance power, fault tolerance, and

BW

OpenFlow provides network routes and port statistics

GENI Experimenters Workshop (1)

Documents

Transcript of GENI Experimenters Workshop (1)