Dynamic Circuit Network

Hands-On Workshop

University of Nebraska-Lincoln Nebraska Student Union

Lincoln, NE July 19th and 20th, 2008

Welcome!

• Wireless – cannot access workshop system from

Joint Techs Wireless

• Wired connections also available

Welcome!

• This is the 7th DCN Workshop– Nysernet– MAX– NASA Ames– University of Houston– University of Hawaii (double header)– University of Nebraska - Lincoln

• Introductions

Welcome!• Key objectives of this workshop are:

– Disseminate information to the R&E community regarding the emerging class of Hybrid Network and the associated techniques for Dynamic provisioning and configuration

– Review in detail and provide instruction on how to use the control plane software currently in service on the Internet2 Dynamic Circuit Network (DCN), ESnet Science Data Network (SDN), and several regional networks.

– Obtain feedback directly from the community on how to improve the technologies…Hopefully, to help guide future development and deployment priorities and speed adoption

– Review the state of implementation and deployment of these types of dynamic networks throughout the R&E community.

Instructors

• Tom Lehman (USC/ISI)• Chris Tracy (MAX)• Andy Lake (Internet2)

• These people are involved in numerous projects related to deploying dynamic control planes:– Internet2 Dynamic Circuit Network – ESnet OSCARS Project – NSF DRAGON– Internet2 HOPI Testbed– DICE (Dante, Internet2, Canarie, Esnet) – International

development activities

Why do a workshop?• Dynamic Hybrid Networks are new…

– The service concepts are still unfamiliar to many networker experts and users… What does one gain with DCN?

– The software and hardware implementations are still evolving…

– Even the standards are still evolving…– The networks that support these capabilities are few but

growing.– The user base is small [for now]…. But will grow as the

capabilities mature and become more ubiquitous, persistent, robust, and the utility of both connection oriented services and dynamic provisioning becomes more widely recognized and accepted.

• Providing hands-on experience to design and deploy these architectures is one way to broaden and promote adoption.

Agenda• Day 1

– 9:00 am Overview of GMPLS and DRAGON– 10:00 am Exercise #1: Designing a GMPLS Control Plane

for Ethernet Data Planes– 10:15-10:45 am Break– 12noon Lunch– 1:00pm Continue working on Exercise #1– 2:00pm Overview of Web Services and OSCARS– 2:30-3:00pm Break– 3:00pm Exercise #2: IntraDomain provisioning with OSCARS– 5:00pm AdjournDay 2

– 9:00am Overview of Inter-Domain implementation in OSCARS– 10:00amExercise #3: Inter-domain Provisioning with OSCARS– 10:15-10:30am Break– 12noon Lunch– 1:00pm Continue working with Exercise #3– 2:30-3:00pm Break– 3:00pm Use of Internet2 DCN and peering dynamic networks– 4pm Adjourn

Workshop Perspective• In this workshop we focus on implementation

– We will design and build a multi-domain GMPLS controlled ethernet network– We have a mobile GMPLS test and evaluation lab consisting of 24 PCs and 12

switches

• We will be focused on the GMPLS intra-domain control plane issues

– Specifically, OSPF and RSVP protocols and Path Computation– We will do a very brief and cursory review of RSVP and OSPF.

• For detailed information on the protocols themselves see the IETF RFCs. • We will not deal with ISIS or CR/LDP or LMP

• We will focus on the “DICE” Inter-domain architecture – Web Services based topology distribution and provisioning

• We use open source software developed by the NSF DRAGON Project, the DOE OSCARS Project

– Intra-domain: Adapted versions of KOM-RSVP and Zebra OSPF plus the NARB for path computing

– This software is the only GMPLS software available to support dynamic ethernet services

– Uses OSCARS (Dept of Energy) for book-ahead scheduling and AAA– Additional software and interfaces have been developed under auspices of the

DICE effort (DANTE, Internet2, Canarie, ESnet)– The code has been adapted to support a wide variety of vendor equipment

(e.g. Force10, Extreme, Dell, Ciena, Cisco, Raptor)

OSCARSDRAGON DOE

Office of Science

Internet2 CoreDynamic Circuit Network

Green PodASN4

Red PodASN1

Yellow PodASN3

Blue PodASN2

DCN Workshop Architecture

Data Plane

Control Plane

Pod Network Elements Control and Data Planes

VLSR1-PC

VLSR1-SW

ES1

Virtual Label Switching Router- “VLSR”

VLSR1

ES2

VLSR3-PC

VLSR3-SW

VLSR3

VLSR2

NARB / IDC

Network Aware Resource Broker- “NARB”Inter-Domain Controller – “IDC”

gre2 gre4

gre3

gre6

Control Plane PC (VLSR#-PC, NARB, IDC)

Data Plane Ethernet Switch (VLSR#-SW)

End System (ES#)

D2 D4

D3D1 D5

Dynamic NetworksOverview and Status

• Objectives and of Dynamic Hybrid Networks

• Hybrid Networking and the Global R&E Community

• Standardization Efforts • Internet2 Dynamic Circuit Network

(DCN)– Control Plane Software– Network Architecture

Hybrid Networking

• There has been interest from many communities for the development of network architectures and mechanisms that utilize lower layers of the protocol stack along with IP at layer 3

• This has become known as “hybrid networking”

• It is motivated by applications from the research and education community that require greater capabilities– High bandwidth flows (for example, flows that

come close to saturating links in the shared IP backbone)

– Flows with special requirements related to quality of service, for example jitter requirements

– Network and Application Virtualization

Hybrid Networks - Motivating Factors

• Hybrid networks are intended to provide a flexible mix of IP routed service and “lower layer services” – “flexible” means the network can respond quickly to

user/application/connector requirements and requests to access both the IP Routed and/or lower layer services

– “lower layer services” means access to layer 2 and below paths which can be utilized in a multitude of ways by creative users.

• Typical user requirements for these lower layer services are based on: – critical, large bandwidth flows which may require one of

more of the following: deterministic network performance, dedicated network resources, guaranteed network capacity, freedom to use protocols other than (congestion control friendly) TCP, privacy/security requirements, scheduled services

– User/application communities which desire to build entire topologies which integrate domain specific resources along with dedicated network resources (which have one or more of the above mentioned characteristics)

Hybrid NetworksHeterogeneous By Nature

• Hybrid networks are extremely heterogeneous at several levels

• DataPlane can be constructed from– router based Multiprotocol Label Switching (MPLS)

tunnels– Ethernet VLAN based Circuits– Synchronous Optical Network / Synchronous

Digital Hierarchy (SONET/SDH) circuits– Wavelength Division Multiplexing (WDM)

connections– Combinations of the above

Hybrid NetworksHeterogeneous By Nature

• Control Planes can be based on– Multiprotocol Label Switching (MPLS)– Generalized Multiprotocol Label Switching

(GMPLS)– Web Services– Management Systems– Combinations of the above

• Client (user) services or attachment points could be– Ethernet– SONET– IP Router– InfiniBand

Multi-Domain, Multi-Layer Control Planes Key Requirements

• The “Multi-Layer” is meant to identify several items regarding how hybrid networks may be built. In this context it includes the following:– Multi-Technology - MPLS, Ethernet, Ethernet

PBB-TE, SONET, NG-SONET, T-MPLS, WDM – Multi-Level - domains or network regions may

operate in different routing areas/regions, and maybe be presented in an abstracted manner across area/region boundaries

• Multi-Domain indicates that we want to allow hybrid network service instantiation across multiple domains

• And of course all this implies that this will be a Multi-Vendor environment.

• Multi-Control – mpls, gmpls, management, vendor proprietary

Dynamic Network Services

IntraDomain•Source Address•Destination Address•Bandwidth•VLAN TAG (untagged | any | tagged | tunnel)

•User Identification (certificate)•Schedule

Client A

Client B

Circuit Request

Ethernet Mapped SONET or

SONET Circuits

Dynamically Provisioned Dedicated Resource Path (“Circuit”)

Internet2 DCN Service

Internet2 IDC

•api can run on the client, or in a separate machine, or from a web browser

XMLUSER API

Actual Network Path

DRAGON Enabled Control Plane

Dynamic Network Services

InterDomain• No difference from a client (user) perspective

for InterDomain vs IntraDomain

RON Dynamic Infrastructure Ethernet VLAN

RON Dynamic Infrastructure Ethernet VLAN

Internet2 DCNEthernet Mapped SONET

1. Client Service Request2. Resource Scheduling 5. Service Instantiation (as a result of Signaling)

A. Abstracted topology exchange

AA

22

1

USER API

XML

Multi-Domain Dynamically Provisioned Circuit

DCN Control Plane

Domain Controller

Network 1

IDC

Domain Controller

Network 2

IDC

User Request/IDC Response

IDC to IDC communication

Domain Controller

Network 3

IDC

IDC to IDC communication

DCN Control Plane Software

• OSCARS (Web Service)– Started by ESnet, merged with Internet2’s

BRUW project in 2006– Web service architecture, interfaces to lower

level network specific provisioning systems– Vendor based MPLS L2VPN (Martini Draft)

• Internet2 DCS/HOPI– DRAGON (NSF funded project in development

by USC/ISI EAST and MAX)– Uses GMPLS protocols to build layer 2

circuits

I2 DCN Software Suite

• OSCARS (IDC)– Web service layer, InterDomain messaging, AAA,

Scheduling

• DRAGON (DC)– Control of domain network elements (Core

Directors and/or Ethernet Switches)– Intra and Inter Domain Path Computation– RSVP based signaling

• Version 0.3.1 of DCNSS released April, 2008– https://wiki.internet2.edu/confluence/display/DCNSS

OSCARS-DRAGON Integration

DRAGON

• Virtual Label Switched Router(VLSR)– PC based control plane software– Manages and provisions various network

equipment such as ethernet switches, SDH/SONET

– Signaling with RSVP packets

• Network Aware Resource Broker (NARB)– Stores topology in OSPF-TE database– Performs inter/intradomain path calculation– Exchanges interdomain topology

IDC - Web Service Based Definition

• Four Primary Web Services Areas: • Topology Exchange, Resource Scheduling, Signaling, User Request

Other AAA Models Possible

• Meta-Scheduler Approach• Same set of Web Services used for linear instantiation model can be

used by a high level process to build services:• Topology Exchange, Resource Scheduling, Signaling, User Request

• A key issue is that this requires a trust relationship between the “meta-scheduler” and all the domains with which it needs to talk

Domain Routing and Path

Computation Element

WS-schedule

Client Client

User Client

Domain Routing and Path

Computation Element

Domain Routing and Path

Computation Element

Domain 2 Domain 3

User ClientProvisioning and

Edge Stitching

Provisioning and Edge Stitching

Provisioning and Edge Stitching

EthernetSONET Router

Domain 1

InternalDomainDesign

WS-top

WS-sigWS-user_request

IDC

WS-schedule

WS-sig

WS-top

WS-schedule

WS-sig

WS-top

WS-user_request

IDCIDC

MetaScheduler

Topolo

gy

Signali

ng

Sched

uling

Topology

Scheduling

Signaling

InterDomain Controller (IDC) Protocol (IDCP)

• Developed via collaboration with multiple organizations– Internet2, ESnet, GEANT2, Nortel, University of Amsterdam, others

• The following organizations have implemented/deployed systems which are compatible with this IDCP

– Internet2 Dynamic Circuit Network (DCN)– ESNet Science Data Network (SDN)– GÉANT2 AutoBahn System– Nortel (via a wrapper on top of their commercial DRAC System)– Surfnet (via use of above Nortel solution)– LHCNet (use of I2 DCN Software Suite)– Nysernet (use of I2 DCN Software Suite)– University of Amsterdam (use of I2 DCN Software Suite)– DRAGON Network

• The following "higher level service applications" have adapted their existing systems to communicate via the user request side of the IDCP:

– LambdaStation (FermiLab)– TeraPaths (Brookhaven)– Phoebus

DCN – Global NetworkInteroperation via IDCP

InterDomain Controller Protocol Standardization Activities

• Standardization process and increasing community involvement continues

• Optical Grid Forum (OGF)– Network Markup Language (NML) Working Group

• Standardizing topology schemas (perfsonar and control plane)

– Network Services Interface (NIS-WG)– Grid High Performance Networking (GHPN) Research

Group– Network Measurement (NM-WG)– Network Measurement Control (NMC-WG)– Information Services (IS-WG)

• GLIF– Control Plane Subgroup working on normalizing

between various interdomain protocols (IDCP, G-Lambda GNS-WSI, Phosphorus API)

– Also other GLIF subgroups in this and related space (global id format, PerfSonar)

Internet2 DCN Working Group• DCN WG has been formed under NTAC

– Chair: Linda Winkler (Argonne National Laboratory)

• DCN WG will drive directions and set agenda in this area

• Mailing list and Wiki available– dcn-wg@internet2.edu– https://spaces.internet2.edu/display/DCN/Home

• DCN WG BOF on Monday, July 21, 12:30 PM 1:50 PM

Internet2 DCN Infrastructure

Internet2 DCN Services

1-A-5-1-11-A-6-1-1

1-A-6-1-1

DCN Services - circuits• Physical Connection:

– 1 or 10 Gigabit Ethernet– SONET (Future)

• Circuit Service:– Point to Point Ethernet (VLAN) Framed SONET Circuit– Point to Point SONET Circuit (future)– Bandwidth provisioning in 100 Mbps increments

• How do Clients Request?– Client must specify [VLAN ID | ANY ID | Untagged |

Tunnel], SRC Address, DST Address, Bandwidth– Request mechanism options are Web Service API, Web

Page, phone call, email• What is the definition of a Client?

– Anyone who connects to an ethernet or SONET port on an Ciena Core Director; could be RON, other wide area networks, domain specific applications

DCN Services - topologies

• Individual circuits are the “atomic” service provided by the DCN and control plane

• These circuits could be intra or inter domain

• It is envisioned that higher level “services” may be developed which coordinate the instantiation of multiple individual circuits to develop entire “topologies”– co-scheduling/allocation of other resources

(compute, data storage) may also be desired– Probably a task for individual

science/application domains or someone developing middleware on their behalf

Workshop Details

Internet2 CoreDynamic Circuit Network

Green PodASN4

Red PodASN1

Yellow PodASN3

Blue PodASN2

DCN Workshop Architecture

Data Plane

Control Plane

Pod Network Elements

VLSR1-PC

VLSR1-SW

ES1

Virtual Label Switching Router- “VLSR”

VLSR1

ES2

VLSR3-PC

VLSR3-SW

VLSR3

VLSR2

NARB / IDC

Inter-Domain Controller – “IDC”Network Aware Resource Broker- “NARB”

Control Plane PC (VLSR#-PC, NARB, IDC)

Data Plane Ethernet Switch (VLSR#-SW)

End System (ES#)

Basic Pod Data Plane

ES1 ES2

End System

Ethernet Switch

Data Plane via Cat5 Patch Cable

D2D4

D3D1 D5

VLSR1-SW

VLSR2-SW

VLSR3-SW

Data Plane Ethernet Switch (VLSR#-SW)

End System (ES#)

Data Plane (D#)

VLSR1-PC

Virtual Label Switching Router- “VLSR”

VLSR3-PC

NARB / IDC

Network Aware Resource Broker- “NARB”Inter-Domain Controller – “IDC”

gre2 gre4

gre3

gre6

Control Plane PC (VLSR#-PC, NARB, IDC) End System (ES#)

Basic Pod Control Plane

Pod Network Elements Control and Data Planes

VLSR1-PC

VLSR1-SW

ES1

Virtual Label Switching Router- “VLSR”

VLSR1

ES2

VLSR3-PC

VLSR3-SW

VLSR3

VLSR2

NARB / IDC

Network Aware Resource Broker- “NARB”Inter-Domain Controller – “IDC”

gre2 gre4

gre3

gre6

Control Plane PC (VLSR#-PC, NARB, IDC)

Data Plane Ethernet Switch (VLSR#-SW)

End System (ES#)

D2 D4

D3D1 D5

Pod Management Addressing

Workshop Gateway Router

Management VLAN 192.168.<asn>.n/16

“Red” pod: ASN=1“Blue” pod: ASN=2“Yellow” pod: ASN=3“Green” pod: ASN=4 192.168.1.1

eth0 .2

.3

eth0 .4

.5

.6

.8

.10

.9 eth0

.7

VLSR1

ES1 ES2

VLSR3

NARB / IDC

VLSR2

eth1 eth1

eth0 - Management Plane Interface and Control Channel (PCs)

eth1 - Data Plane Interfaces (PCs)

Rack Layout

VLSR1-PCVLSR1-SW

GW2SW2

NARB

VLSR2-PCVLSR2-SW

VLSR3-SWVLSR3-PC

ES1ES2

VLSR1-PCVLSR1-SW

NARB

VLSR2-PCVLSR2-SW

VLSR3-SWVLSR3-PC

ES1ES2

.

VLSR1-SWVLSR1-PC

GW1SW1

NARB

VLSR2-SWVLSR2-PC

VLSR3-PCVLSR3-SW

ES1ES2

VLSR1-SWVLSR1-PC

NARB

VLSR2-SWVLSR2-PC

VLSR3-PCVLSR3-SW

ES1ES2

123456789012345678901234567890

123456789012345678901234567890

Rack 1 Rack 2

Workshop Pods

Red Pod

Green Pod

Yellow Pod

Blue Pod

Exercise #1 Intra-Domain Detail(Answer Sheet)

Workshop Gateway Router

Management VLAN 192.168.<asn>.n/16

“Red” pod: ASN=1“Blue” pod: ASN=2“Yellow” pod: ASN=3“Green” pod: ASN=4

Management VLAN 192.168.<asn>.n/16

GRE<x> = 10.<asn>.<x>.n / 30 GRE7= 10.1.7.0 / 30

192.168.1.1

eth0 .2

.3

eth0 .4

.5

.6

.8

.10

.9 eth0

.7GRE3

GRE2

GRE4

GRE6

TEaddr = 11.<asn>.<x>.n / 30

D2

D4

D3

D1D5

VLSR1-PC

VLSR1-SW

ES1 ES2

VLSR3-PC

VLSR3-SW

NARB / IDC

VLSR2-PC

VLSR2-SW

3 5

4

1

10.a.2.1

10.a.3.1

10.a.3.2

10.a.2.2

13

4

10.a.6.1

10.a.6.2

10.a.4.1

10.a.4.2

13

4

5

eth1 eth1

11.a.4.1

11.a.4.211.a.2.1

11.a.2.2

11.a.3.211.a.3.1

Dynamic Data plane port group = g3-g24Dynamic VLAN range = 100…200

Exercise #1 Data and Control links

“Red” pod: N=1“Blue” pod: ASN=2“Yellow” pod: ASN=3“Green” pod: ASN=4

GRE3

GRE2GRE4

GRE6

D2D4

D3D1

VLSR1

ES1 ES2

VLSR3

NARB / IDC

VLSR2

3 5

4

4

3

4

5

eth1 eth1

D5

3

Login information• Wireless Network:

– SSID: DCNworkshop– WPA Personal Key: Workshop!

• Login to all VLSR, ES and NARB– ssh port 22– username: user[1-16]; password: Workshop!– username: root; password: rootme

• Login to all switches– telnet port 23– username: admin; password: admin

• OSCARS configuration; login to the NARB/IDC machine– ssh port 22– username: tomcat55; password: dragon

• OSCARS axis2 login– https://idc.<color>.pod.lan:8443/axis2/axis2-admin/– username: admin; password: axis2

• OSCARS web user interface;– https://idc.<color>.pod.lan:8443/OSCARS/– username: oscars-admin; password: oscars

• Command Line Interface ports–dragond 2611–ospfd 2604 (intra-domain)–narb 2626–rce 2688

> telnet localhost 2611> password: dragon

Login information

Workshop Laboratory

• Four “Pods”: Red, Blue, Yellow, Green• Each Pod represents an independent network

domain• Each Pod has two End Systems: ES1 and ES2• Each Pod has three Virtual LSRs (VLSRs)

– Each VLSR has a PC (for ctrl plane) and a Ethernet switch (for data plane)

• Each Pod has one PC for interdomain routing support of the NARB and OSCARS

• The PCs are running Debian Linux– We have installed it and all the software required to

download, build, and run the control plane software, and to perform the workshop labs

• We installed the DRAGON software and OSCARS software– /usr/local/dragon/{bin,etc}– /usr/local/tomcat, /home/tomcat55

Workshop Exercises

• Exercise 1: Designing a GMPLS Control Plane for Ethernet Data Planes

• Exercise 2: Intra-Domain Provisioning with OSCARS

• Exercise 3: Inter-Domain Provisioning with OSCARS

Exercise #1 Designing a GMPLS Control Plane For Ethernet Data Planes• Diagram a control plane for each pod• Construct an addressing scheme for

the control plane• Configure the network elements’

data plane• Configure the control plane software• Set up an LSP

• …and if that fails…read the instructions.

GMPLS Snapshot• Generalized Multi-Protocol Label Switching – GMPLS

– Evolved from MPLS concepts, and experiences gained from deployments within the IP packet world

• GMPLS extends Traffic Engineering (TE) concepts to the multiple layers:– Packet Switching Capable (PSC) – standard MPLS LSPs– Layer2 switch capable (L2SC) – Ethernet and VLANs– TDM switch capable (TDM) – SONET/SDH – Lambda switching (LSC) – Wavelength – Fiber Switch capable (FSC) - Automated Patch Panel

• In the GMPLS, any network element that supports one of the above switching capabilities and participates in the GMPLS control plane protocols is referred to as a “Label Switching Router” or LSR.

• GMPLS Protocols: – Routing: GMPLS-OSPF-TE – Signaling: GMPLS-RSVP-TE – Link layer: LMP (not widely implemented)– ISIS and CR/LDP are also considered part of the GMPLS protocols– In this workshop we will focus only on OSPF and RSVP

What is the Control Plane?• The Control Plane is the network facilities and

associated protocols that select, allocate/deallocate, and provision network resources to fulfill a user service request.– Typically this includes routing protocols that distribute

topology and reachability information among interconnected networks and network elements

– It also includes other functions that allocate appropriate resources and put those resources into service (Path computing and signaling)

• With GMPLS, routing and signaling messages between LSRs do not travel along the same [physical] path as the circuit being established.– The set of facilities between LSRs that carry the data

circuits themselves is called the “Data Plane”– The set of facilities between LSRs that carry the routing

and signaling protocols is called the “Control Plane”

• It is good practice to design the control plane so as to be highly robust and impervious to effects of other network traffic or malicious activity

• In this workshop, our control plane and data plane will be separate as is typically the case for GMPLS networks.

Control Plane and Data Plane

CP CP CP

LabelSwitched

Paths

GMPLSProtocols

GMPLSProtocols

LabelSwitched

Paths

ControlPlane

Data Plane

A [Typical] Label Switching Router – “LSR”

• What is an “LSR”– In the MPLS world, it is any router capable of recognizing

and processing the MPLS shim header in the IP packet• In the GMPLS world, an LSR is any network element that

is able to establish “label switched paths” (LSPs) under control of the GMPLS protocol suite:– This now includes fiber switches, wave division

multiplexors, sonet (tdm) switches, ethernet switches, and traditional packet switches (MPLS routers)

Label Switching

Fabric

Data Interfaces

Switching Fabric Interface Link

Control Processor

Management Interface

Key Control Plane Features

• Routing– distribution of "data" between networks. The data that

needs to be distributed includes reachability information, resource usages, etc

• Path computation– the processing of information received via routing data

to determining how to provision an end-to-end path. This is typically a Constrained Shortest Path First (CSPF) type algorithm for the GMPLS control planes. Web services based exchanges might employ a modified version of this technique or something entirely different.

• Signaling– the exchange of messages to instantiate specific

provisioning requests based upon the above routing and path computation functions. This is typically a RVSP-TE exchange for the GMPLS control planes. Web services based exchanges might employ a modified version of this technique or something entirely different.

OSPF – “Open Shortest Path First”

• OSPF is a “Link State” Routing Protocol– OSPF routers discover each other thru a HELLO protocol

exchanged over OSPF interfaces – Routers identify themselves with a “router id” (typically the

loopback IP address or another unique IP address is used)– OSPF routers flood Link State Announcements (LSAs) to each

other that describe their connections to each other and that specify the current link state of these connections

• In the GMPLS and TE extensions to OSPF, the LSA contains information about the available bandwidth, routing metrics, switching capabilities, encoding types, etc.

• LSAs are not flooded in the direction from which they are heard

– Link State flooding does not scale well • OSPF routing is often divided into “areas” to reduce or

limit LSA flooding in large networks• Other routing protocols are used between routing

“domains” that distribute reachability information but not link state info

– Each OSPF router in an area has a full topological view of its area

– SPF identifies the next-hop for each known destination prefix

CSPF• Constrained Shortest Path First

– In OSPF TE, reachability is no longer the only criteria for deciding next-hop

• E.g. Bandwidth available on each intemediate link could be a constraint used to identify or select a path

• In GMPLS, with multiple switching capabilities, there are many constraints to be considered

– Path Computation is used differently for selecting circuit layout than for selecting the next-hop for shortest path packet forwarding

• Two identical path requests may generate two completely separate paths (unlike traditional routed IP which would select only the single “best” path for forwarding packets)

• Paths are not computed until or unless a path is needed.– Some GMPLS service models do propose precomputing paths

(or at least next-hops) based on certain apriori assumptions about the LSP – the tradeoff is generally one of scheduled “book ahead” reservations vs fast “on-demand” provisioning.

RSVP – ReSerVation Protocol• GMPLS-RSVP-TE is the signaling (provisioning)

protocol used to instantiate a Label Switched Path (LSP) thru the network

• Five basic RSVP messages we will reference:– PATH = First message issued by the source towards

the destination requesting a connection be established– RESV = Response from the destination towards the

source accepting the connection– PATH_TEAR = Message sent to tear down an LSP– PATH_ERR = Error message sent when a PATH request

is denied or encounters a problem – REFRESH = Message sent between LSRs indicating a

connection is still active (prevent timeout and deletion)

Path Computation Element• In GMPLS, the Path Computation Element (PCE) is

separated from the routing protocol. – The routing protocol distributes topology information

and builds the topology database that contains all the [visible] resources and their state – the Traffic Engineering Data Base (TEDB)

– PCE is responsible for processing the TEDB to select a path through the network that meets the constraints specified in the service request (e.g. BW, encoding, Src/Dst, Policy, etc.)

• In GMPLS, the path computed is expressed as an “Explicit Route Object” (ERO). – An ERO is simply a data structure that contains a

sequentially ordered list of routers (LSRs) that the path will travels from Source to Destination

– A “Loose Hop” ERO specifies a partial set of transit nodes – the path may contain other nodes as long as it passes through the specified nodes in the order specified.

– A “Strict Hop” ERO specifies a complete list of transit nodes – no other intervening nodes are allowed.

– RSVP includes the ERO in the PATH message to pin the path through specific nodes

DRAGON Control Plane - Key Elements

• Virtual Label Switching Router – VLSR– Open source protocols running on PC act as

GMPLS network element (OSPF-TE, RSVP-TE)– Control PCs participate in protocol exchanges and

provisions covered switch according to protocol events (PATH setup, PATH tear down, state query, etc)

• Network Aware Resource Broker – NARB– Intradomain listener, Path Computation,

Interdomain Routing and Path Computation

• More information: – dragon.east.isi.edu– dragon.maxgigapop.net

The Virtual Label Switching Router “VLSR”• The DRAGON Project developed a control

plane "proxy" element to cover non-GMPLS capable devices like standard ethernet switches.

EthernetSwitch

SNMP

Linux Control PC

Mgmt Interface

GMPLS Control Plane

Data Plane

VLSR - conceptual

CoreEthernetSwitch

OperationsAccessSwitch

Linux Control PC

Control LinksVia

GRE tunnels

VLSR – physical

Data Plane

VLSR(Virtual Label Switching Router)

• RSVP Signaling module– Originated from Martin Karsten’s C++ KOM-RSVP– Extended to support RSVP-TE (RFC 3209)– Extended to support GMPLS (RFC 3473)– Extended to support Q-Bridge MIB (RFC 2674)– For manipulation of VLANs via SNMP (cross-connect)– Extended to support VLAN control through CLI

• OSPF Routing module– Originated from GNU Zebra– Extended to support OSPF-TE (RFC 3630)– Extended to support GMPLS (RFC 4203)

• Ethernet switches tested to date– Dell PowerConnect, Extreme, Intel, Raptor, Force10

NARB(Network Aware Resource Broker)

• NARB is an agent that represents a domain• Intra-domain Listener

– Listens to OSPF-TE to acquire intra-domain topology– Builds an abstracted view of internal domain topology

• Inter-domain routing– Peers with NARBs in adjacent domains– Exchanges (abstracted) topology information– Maintains an inter-domain link state database

• Path Computation– Performs intra-domain (strict hop) TE path computation – Performs inter-domain (loose hop) TE path computation– Expands loose hop specified paths as requested by domain

boundary (V)LSRs.• Hooks for incorporation of AAA and scheduling into path

computation via a “3 Dimensional Resource Computation Engine (3D RCE)”– The Traffic Engineering DataBase (TEDB) and Constrained

Shortest Path Computation (CSPF) are extended to include dimensions of GMPLS TE parameters, AAA constraints, and Scheduling constraints.

– 3D RCE is the combination of 3D TEDB and 3D CSPF

Heterogeneous Network Environmentmulti-technology, multi-level, multi-domain,

multi-vendor, multi-provision system network environments

GMPLS MPLSManagement Plane

IDCDC

IDCDC

IDCDC

DRAGON

DRAGON GMPLS Control Plane

CoreDirector

Ciena Region

uni, tl1

CD_a CD_z

uni, tl1

CoreDirector

subnet signaling flow

IDC

• DRAGON is used as the DOMAIN Controller for I2 DCN Ciena Core Directors

GMPLS to other domains

GMPLS to other domains

to other domain IDCs

to other domain IDCs

• DRAGON allows for incorporation of non-GMPLS equipment and vendor proprietary provisioning methods into the overall GMPLS environment

Exercise #2: Intra-domain Provisioning with OSCARS• In this exercise we will bring up the OSCARS

software, configure the network topology and candidate paths, and provision LSPs across a single administrative network domain

• OSCARS:– “On-demand Secure Circuits and Advanced Reservation

System”– Provides Authentication and Authorization for LSP

requests– Provides book-ahead scheduling for network path

resources– Interim: implements the static topology distribution

function and provides precomputed static EROs for provisioning

• OSCARS is a Java based application. OSCARS runs on top of Tomcat, uses MySQL and AXIS2.

Exercise #3: Inter-domain Provisioning with OSCARS• In this exercise we will configure and use

OSCARS to accomplish InterDomain provisioning. – Design (and implement) the inter-domain Data

plane– Layout the inter-domain control plane– Configure OSCARS for inter-domain– Test

IDC - Web Service Based Definition

• Four Primary Web Services Areas: • Topology Exchange, Resource Scheduling, Signaling, User Request

DCN Web Services• Web Service Definitions• wsdl - web service definition of message

types and formats• xsd – definition of schemas used for

network topology descriptions and path definitions

• Ongoing work with OGF Working Group(s), PerfSonar, and GLIF with the goal to achieve interoperability amongst all groups.

InterDomain SpecificationWeb Services

• https://wiki.internet2.edu/confluence/display/CPD/OSCARS+Web+Service+Definition

• Specification is defined by a Web Service Desciption Language (WSDL) document and XML Schema files containing associated data types.

• OSCARS.wsdl - web service definition of OSCARS messages

• OSCARS.xsd - data types used by OSCARS.wsdl

• nmtopo-ctrlp.xsd - NMWG control plane topology schema used by OSCARS.xsd for topology-related data types

AAA and Security

• OSCARS AAA• SSL Encryption• Authentication

– X.509 Certificates• User to Domain• Domain to Domain

– Web Service Security by OASIS– SAML assertions about end-user

(future)

• Authorization– OSCARS attribute based system

DCN Control Plane uses OGF Topology Schema

Information Services Topology Service and LookUp Service

• Control Plane uses Information Services Topology Service and LookUp Service

• LookUp Service– Provides a mapping from circuit end points to user

friendly names

• Topology Service– Provides an infrastructure from which to retrieve

topologies from other domains– Will be utilized for global path computation

Information Services Topology Service and LookUp Service

DCN Information Service - Lookup Service

DCN ProvisioningWeb Page or API

Web Page Based Provisioning

Internet2 IDC

USER APIjava createReservation https://dcn.internet2.edu:axis2/services/dcn

reservation.properties

Web Service

DCN – Circuit Status Description

DCN – Circuit Status Description

Requesting a circuit - Interfaces

• Web User Interface (WBUI)– Java servlet interface used by OSCARS web

page– Not intended for use by other applications

• Web Service API– XML-based API intended for use by

applications• e.g. Phoebus, LambdaStation, TeraPaths

Requesting a circuit – WS API

• Used by applications to contact IDC• Authenticate using an X.509

certificate– Generate with command-line tools– Have CA sign (Internet2 has test CA)

• Message format defined in DICE Control Plane group

• Custom applications should use this interface

Additional Information

• DCN Software Suite– https://wiki.internet2.edu/

confluence/display/DCNSS/Home

• Java Client API– https://wiki.internet2.edu/

confluence/display/CPD/OSCARS+Client+Java+API

Workshop Details - end

DCN Control Plane Possible Future Features and Work Areas

• Improved user documentation and software installation procedures

• Improved reliability and redundancy of dynamic provisioning operations. (better automated logging and failure reporting, redundant control plane elements, automated interaction between control plane and monitoring systems and NOC operations)

• Support for VLAN Translation across a multi-domain circuits• Support for SONET Client Access ports and Interdomain Links• Design for automated multi-domain topology exchange• Enhanced user request options (additional parameters and

ability to ask questions without actually making a reservation)

• Enabling other signaling methods, e.g. RSVP (as opposed to only Web Service method)

• Continue work with international groups, standards bodies to formalize the IDC InterDomain Protocol to further increase interconnected global community for these services

Use of Internet2 DCN and peering dynamic networks

1. Physical connection2. Access to control plane software

How do I connect? – Physical Connection• Internet2 Connectors

– Connect to Internet2 DCN

• Universities and campuses– Contact Internet2 Connector

How do I connect? – Software Configuration

• Option 1: No local IDC• Option 2: Install local IDC

How do I connect? – Software Configuration

• Option 1: No local IDC– Statically configure your local network – Applications/Users can dynamically

request circuits from the nearest IDC

How do I connect? – Software Configuration

Network 1

Domain Controller

Network 2

IDCUser Request/IDC Response

Network 3

Statically Configured Dynamically Configured Statically Configured

• Option 1: No local IDC

Domain Controller

Network 1

IDC

Domain Controller

Network 2

IDC

User Request/IDC Response

IDC to IDC communication

Domain Controller

Network 3

IDC

IDC to IDC communication

How do I connect? – Software Configuration• Option 2: Install local IDC

How do I request a circuit? - Clients

• User-initiated– OSCARS Web Page– Simple command-line tools

• Program-initiated– Phoebus

• Transparently request circuit upon data transfer initiation

– Custom applications you build!

How do I request a circuit? - Interfaces• Web User Interface (WBUI)

– Java servlet interface used by OSCARS web page

– Not intended for use by other applications

• Web Service API– XML-based API intended for use by

applications• E.g. Phoebus, LambdaStation, TeraPaths

How do I write my own DCN application?• Java library for making DCN calls• Can call simple command-line client

directly from application• Google Summer of Code students

will be developing PERL, C, and Python libraries

backup

VLSR(Virtual Label Switching Router)

• GMPLS Proxy– (OSPF-TE, RSVP-TE)

• Local control channel– CLI,TL1, SNMP, others

• Used primarily for ethernet switches

Web page

XML Interface User API

CLI Interface One NARB per Domain

• Provisioning requests via CLI, XML, or ASTB

DRAGON Virtual Label Switching Router (VLSR)

– Control channels could also be provisioned out-of-band via GRE tunnels over an IP network

IPsec is one of several

mechanisms recommended for securing out-of-band

control channels provisioned

over IP networks(RFC3945)

DCN – Circuit Status Description

Laying Out the Control Plane

S

DR1

R2

R3

R4

R5

R6

D1

D2

D3

D4

D5

D6

D7

• Lay out the data plane between NEs first. – For now, we are going to ignore intervening static NEs.– Make sure all Nes and links are uniquely labeled

• Then, control links connect the dynamic network elements• If you are including end systems in the dynamic network,

you should add them where appropriate

C1

C2C3

C4

C5

C6

C7

D8

D9

C9

C8

Control Plane

R1 R2 R3

Data plane

• Often, the dynamic network elements are not directly adjacent to one another – but the control structure expects them to be (at least logically adjacent)

• We employ Generic Routing Encapsulation (GRE) tunnels for the control links in order to create logical adjacencies

– GRE Tunnels are set up between two IP hosts over the conventional internet interface. (these are the “tunnel endpoints”)

– They present a pseudo interface to the end host that appears to be directly linked to the remote endpoint, thus allowing a single common IP subnet to be allocated on this GRE (pseudo) interface.

GRE TunnelEndpoints

Control LinkEndpoints

D2

C2

D3

C1

D4

C3

D1

C4

Generic Network ElementConsider all of the components in a network element:

Case Study: Control ChannelsDRAGON Virtual Label Switching Router (VLSR)

– Linux PC implements GMPLS control plane protocols– Control channels may be provisioned in-band or

out-of-band

One goal of

DRAGON’s VLSR software is to

provide GMPLS protocol support for devices which

do not support GMPLS

Case Study: Control ChannelsDRAGON Virtual Label Switching Router (VLSR)

– Assuming underlying network uses Ethernet VLANs, control channels may be provisioned in-band with static control VLANs

In-band control channels are

considered somewhat less

vulnerable than out-of-band

(RFC3945)

Case Study: Control ChannelsDRAGON Virtual Label Switching Router (VLSR)

– Control channels could also be provisioned out-of-band via GRE tunnels over an IP network

IPsec is one of several

mechanisms recommended for securing out-of-band

control channels provisioned

over IP networks(RFC3945)

Case Study: Control Channels

Data plane

GRE TunnelEndpoints

Control LinkEndpoints

Hybrid NetworksWeb Service Control Plane Interfaces

• Web Services provides a mechanism to deal with heterogeneous control planes• inspired by the standards bodies work on control plane protocols,

but not just recreating that work at the web service level • Better described as using control plane techniques to develop a

“service plane”

Ethernet/L2SC(Dataplane)

SONET/TDM(Dataplane)

Router(MPLS)/PSC(Dataplane)

GMPLS(I-NNI)

MPLS(I-NNI)

Management System(I-NNI)

Inter-Domain Controller (IDC)

WS E-NNIWS E-NNI

WS UNI WS UNI

IDC

IDCWS I-NNI IF

WS I-NNI IF WS I-NNI IF

Hybrid NetworksControl Plane Architecture

• The benefits offered by Web Services include• standardized mechanisms for user authentication and policy

management• flexible features for interfacing with a diverse set of I-NNI

mechanisms• Allows focus on several issues that current control plane work

has not addressed in a robust manner:• scalability, stability, security, flexible application of policy, AAA,

scheduling

• Will still allow for peering domains with compatible non web service E-NNI (i.e. GMPLS based) to utilize that as desired• a domain might peer with one domain at GMPLS level, and

another at the Web Service level

Web Service based E-NNIThree Main Components

• Routing– Topology Exchange– Domain Abstraction– Varying levels of dynamic information

• Resource Scheduling– Multi-Domain path computation techniques– Resource identification, reservation, confirmation

• Signaling– path setup, service instantiation

Key Control Plane Key Capabilities

• Domain Summarization– Ability to generate abstract representations of your domain for

making available to others– The type and amount of information (constraints) needed to be

included in this abstraction requires discussion. – Ability to quickly update this representation based on

provisioning actions and other changes • Multi-layer “Techniques”

– Stitching: some network elements will need to map one layer into others, i.e., multi-layer adaptation

– In this context the layers are: PSC, L2SC, TDM, LSC, FSC– Hierarchical techniques. Provision a circuit at one layer, then

treat it as a resource at another layer. (i.e., Forward Adjacency concept)

• Multi-Layer, Multi-Domain Path Computation Algorithms– Algorithms which allow processing on network graphs with

multiple constraints– Coordination between per domain Path Computation Elements

OSCARS Architecture

End-HostApplication

User

Topology

LinkReservations

Policy

Web-UserInterface

Path Setup(MPLS)

BandwidthScheduler

AuthenticationAuthorization

Path Setup(GMPLS)

OSCARSResourceManager

ResourceManager

Customer Site External Peer

Web-Services Interface(Signed SOAP Messages)

I-NNI

Integration Core Director Domain into the End-to-End Signaling

VLSR uni-subnet

• Signaling is performed in contiguous mode.• Single RSVP signaling session (main session) for end-to-end circuit.• Subnet path is created via a separate RSVP-UNI session (subnet session),

similar to using SNMP/CLI to create VLAN on an Ethernet switch.

• The simplest case: one VLSR covers the whole UNI subnet.• VLSR is both the source and destination UNI clients.• This VLSR is control-plane ‘home VLSR’ for both CD_a and CD_z.• UNI client is implemented as embedded module using KOM-RSVP API.

CoreDirector

Ciena Region

LSRdownstream

LSRupstream

data flow signaling flow

subnet signaling flow

uni, tl1

CD_a CD_z

uni, tl1CoreDirector

DRAGON enables integration of the Core Director Domain into Multi-Domain, Multi-Layer, Multi-Service, Multi-Vendor Provisioning Environment

• Goal is to utilize Ciena Domain control plane and advanced features to maximum extent possible

• advanced provisioning, management, monitoring, restoration and protection features

• applicable to single domain, single vendor

• Integrate these capabilities into the Multi-X environment

VLSR uni-subnet1

CoreDirector

Ciena Region

LSRdownstream

LSRupstream

data flow signaling flow

subnet signaling flow

VLSR uni-subnet2

VLSR

CD_a CD_z

uni style control

CoreDirector

uni style control

Domain Boundary

Domain Boundary