Network Infrastructure Virtualization Case Study


#CNSF2011

Dave Zacks CCIE 8887 Technical Solu7ons Architect

© 2011 Cisco and/or its affiliates. All rights reserved. 3 CNSF - May, 2011 Cisco Public

This session focuses on a customer case study in which Network Virtualization has been deployed. The focus of this session is to cover the actual business requirements of the customer involved, how Network Virtualization met those requirements, the network design that was employed, and the benefits that were derived. Introducing the session will be a brief outline of Cisco's approach to Network Virtualization design methodology. The customer case study itself will focus on a Campus Network Virtualization deployment. Presenting this case study will be Dave Zacks, a Technical Solution Architect with Cisco Systems. Attendees at this session will learn about virtualized network deployments, and how these can be used to provide unique and compelling architectural solutions, addressing both business and technical requirements.

© 2011 Cisco and/or its affiliates. All rights reserved. Cisco Public CNSF - May, 2011 4

#CNSF2011


  Groups and services are logically separated … Guest / partner access

Departmental separation

Regulatory compliance (HIPPA, PCI …)

Building controls, video surveillance Mergers, acquisitions … and many others

  Closed User Groups … Private

Secure

Independent policies

  Service differentiation is configured per group / service …

The same application can be unique per group / service

Resources

Dept A Partner Guest

Internet

Dept B

Non-Virtualized Network

Virtualized Network


Virtual

Function B

Actual Physical Network Infrastructure

Department A

Virtual Virtual

Guests / Partners

 One physical network supports many virtual networks


Access Control Path Isolation Services Edge

Functions   Authenticate client (user, device, application) attempting to gain network access

  Authorize client into a partition (VLAN)

  Deny access to unauthenticated clients

  Maintain traffic partitions over Layer 3 infrastructure

  Transport traffic over isolated Layer 3 partitions

  Map Layer 3 isolated path to VLANs in access and services edge

  Provide access to services –

Shared

Dedicated

  Apply policy per partition

  Isolate application environments if necessary

Service

Data Center – Internet Edge – Campus

Internet

WAN – MAN – Campus

VRFs

GRE MPLS

Branch – Campus


  Device virtualization – Control plane virtualization Data plane virtualization Services virtualization

VRF VRF

Global

IP

802.1q

VRF – Virtual Routing and Forwarding

Per VRF:

Virtual Routing Table

Virtual Forwarding Table

  Data path virtualization –

Hop-by-Hop – (VRF-Lite End-to-End) Multi-Hop – (VRF-Lite+GRE, MPLS VPN)


1.  Create L2 VLANs and trunk them to the first L3 device

2.  Define VRFs at the first L3 devices (PE) and map the L2 VLANs to the proper VRF

3.  Enable MPLS on all Layer 3 interfaces in the network

4.  Enable MP-BGP on the PE devices to exchange VPN routes

PEs become iBGP neighbors

5.  VPN traffic is now carried end-to-end across the network, maintaining logical isolation between the defined groups Each frame is double-tagged (IGP label + VPN label)

Enable MPLS

Enable MPLS

PE

P P Label Switch Router (LSR)

PE


S3 S2 R1 R4 S1 S4

172.16.5.0/24

172.17.6.0/24

172.18.7.0/24

172.16.8.0/24

172.17.9.0/24

172.18.10.0/24

N * (N-1) / 2 = 8 * 7 / 2 = 28 iBGP requires a full mesh of neighbors

For Your Reference


S3 S2 R1 R4 S1 S4

172.16.5.0/24

172.17.6.0/24

172.18.7.0/24

172.16.8.0/24

172.17.9.0/24

172.18.10.0/24

Ro ute Reflector Route Reflector

For Your Reference


  Firewall contexts in transparent mode act as L2 bridges

Red VPN Blue VPN

GreenVPN

Campus Core

Shared Services E-mail Storage Web …

L2 L2 L2

EIGRP, OSPF, eBGP, Static

(no ISIS)

  Fusion router establishes routing peering with the various VRFs

The fusion router has complete knowledge of all the routes existing in the VRFs

  The peering protocol may vary depending on the path isolation strategy

Use IGP (EIGRP or OSPF) for VRF-Lite deployments Use eBGP for MPLS VPN scenarios

  The fusion router could typically advertise only a default route into the various VRFs

  A dedicated “Fusion” VRF may be used in place of an external fusion router device


  Firewall contexts in routed mode act as an L3 hop routing traffic between interfaces

No routing protocol support on firewall deployed in multi-context mode

Red VPN Blue VPN

GreenVPN

Campus Core

Shared Services E-mail Storage Web …

L3 L3 L3

eBGP

  The only recommended peering protocol is eBGP, independently from the Path Isolation technique adopted in the Campus

Configuring static routing is also possible

  The fusion router could typically advertise only a default route into the various VRFs

  A dedicated “Fusion” VRF may be used in place of an external fusion router device


The global table is considered as another VPN (in fact can be usually considered the “default VPN”) and it is front-ended by its own security device

Red VPN Blue VPN

Global Table

Shared Services

The global table is treated as a shared service – access to the global table from each VPN is subject to the policy enforcement provided by the Services Edge

Red VPN Blue VPN

GreenVPN

Global Table


#CNSF2011


 UBC is one of the largest Universities in Canada.

 The main UBC Campus is located in Vancouver, BC (UBCV).

 UBCV Campus is 1.55 square miles in extent, covers several hundred buildings in total.

 Three additional (smaller) Campuses located around BC UBC

is here You are here


Clock Tower, adjacent to the I.K Barber

Learning Centre, University of BC

On a beautiful summer day …

June 25th, 2010

I.K Barber Learning Centre

University of BC


  50,332 students – 4,669 faculty – 8,953 staff Source: www.ubc.ca (as of November, 2008)

 Number of Ethernet switches installed = 2,709

 Number of wired Ethernet ports = approx. 60,000

 Number of VLANs allocated = 3,394  All core network links are 10 Gigabit Ethernet

 Core network links are Jumbo Frame-enabled

 Number of Wireless APs deployed = approx. 2,000

 Total Internet bandwidth used = approx. 1750Mbps


 UBC has adopted a standardized Campus network architecture.

 Due to the size of UBC’s Vancouver Campus, a four-layer network architecture has been used –

Access Layer – Stackable Layer 2 switches

Distribution Layer – Mixture of stackable and chassis switches, mostly Layer 2 but with some Layer 3 termination

Outer Core Layer – All remaining Layer 3 terminations

Inner Core Layer – Layer 3 routing between Outer Core platforms


No deliberate Layer 2 loops in the network

Fully resilient core network, based on

OSPF routing

No VLAN bridging across the core – routed links only

Layer 3 FHRP provided by HSRP at Outer Core layer

Typical Large

Building

Distribution

Outer Core

Access

Inner Core

Additional Buildings

Data Center

Internet Edge


 UBC departments and faculties use VLANs to segregate functions within buildings –

servers, students, faculty, staff, admin, labs, etc.

 Any port within a building at UBC can be on any VLAN within that building (or building complex).

 Some large UBC buildings, holding multiple departments, have over 100 VLANs.

 Don’t forget – VLANs are a form of virtualization (at Layer 2)!


 Statement of the Problem –

Nineteen faculties – and over 200 departments. Several thousand subnets, Campus-wide.

Faculties / departments have space in many buildings. Example – Faculty of Arts has space in 31 buildings. Many UBC faculties / departments require secure connections within, and between, their areas. To this end, many have implemented their own firewall solutions.


Proliferation of different firewall

types

Performance issues as the Campus scales to 10GE

Difficult to support and troubleshoot

Many UBC faculties / departments want a single, fully redundant firewall system for their subnets

and VLANs, Campus-wide.

How to accommodate this?


 Technically, it would be possible to bridge VLANs across the core UBC network –

This has often been requested historically.

 However, in practice this creates many difficulties… It does not scale well. The proliferation of Layer 2 loops in a fully-resilient network design leads to many blocking / forwarding ports – this is considered to be undesired in UBC’s network design.

 As a consequence, bridging of VLANs across the core is not supported by UBC.


 To meet their needs, UBC looked towards a Network Virtualization solution.

 VRFs (Virtual Routing and Forwarding instances) – provide for a separate IP routing table per department or faculty – even when distributed over multiple buildings.

 VRFs provide the required segmentation – to provide a “virtual network” for the department or faculty involved.


 Within a VRF High-performance routing for the department / faculty.

 Between VRFs Traffic flows across Virtual Firewalls, hosted on UBC’s Catalyst 6500-based Firewall Services Modules (FWSMs). Each Virtual Firewall can be managed separately by the department / faculty involved (aids in scaling manageability).

 No need for VLAN bridging The UBC core network remains entirely routed – promotes stability, maintains core network policies.


 UBC needed a solution that could scale to handle the number of departments / faculties on Campus.

 The options for a Network Virtualization deployment include …

- VRF-Lite with GRE Tunnels,

- VRF-Lite End-to-End, or

- MPLS VPNs

 For the scale requirements of UBC, an MPLS VPN deployment model was chosen.


 When designing, planning, and building their Network Virtualization deployment, UBC used Cisco’s Network Virtualization Design Guides extensively …

NV Access Control Design Guide

NV Path Isolation Design Guide

NV Services Edge Design Guide

www.cisco.com/go/designzone


 MPLS VPN uses Multi-Protocol BGP (MP-BGP) for exchanging MPLS tags (for creating isolated domains).

The first step was implementing the Route Reflectors for MP-BGP.

UBC used a pair of Cisco 7200 routers for this purpose.


 The next step was determining where to place the P (Provider) and PE (Provider Edge) routers.

UBC uses Catalyst 6509 switches, which support MPLS.

The Inner Core 6509s are the P routers.

The Outer Core 6509s are the PE routers.


 After determining the P and PE router locations, it was necessary to set up the appropriate routing.

All of the Outer Core 6509s peer (MP-BGP) with the two Route Reflectors –

for MPLS VPN (VRF) route exchange.

The Inner Core routers are not aware of VRFs –

(and do not run MP-BGP – OSPF only, with MPLS tagging).


 Providing for Layer 3 VLAN termination, mapping into VRFs, and First Hop redundancy were the next tasks.

VLANs terminate at the Outer Core layer, and are mapped into VRFs using SVIs.

HSRP is used across the PE router pair for first-hop redundancy.


 VLAN and VRF definitions at the PE routers …

Note – VRFs are only defined on the PE routers where the VRFs need to terminate … not all VRFs are defined on all PE routers …

Outer Core (PE), VLANs and VRFs vlan 627 name VLAN-627-MED-ADMIN ip vrf MED-ADMIN rd 65111:521 route-target export 65111:521 route-target import 65111:521

VLAN definition – Layer 2

VRF definition – Layer 3

The RD is a 64-bit value (unique per VRF). When added to the 32-bit IP address, this forms a unique 96-bit VPNv4 address.

Routes are imported and exported within the VRF, to populate the VRF’s IP routing table.



 SVI definition on the PE router,

showing SVIs with VRF mapping, and HSRP and DHCP Relay functionality enabled …

Outer Core (PE), SVI configuration OUTER-2# show run int Vlan 627 ! interface Vlan627 description VLAN-627-MED-ADMIN mtu 8500 ip vrf forwarding MED-ADMIN ip address 172.16.10.253 255.255.255.0 ip helper-address 10.10.5.1 ip helper-address 10.10.6.1 standby ip 172.16.10.254 standby priority 105 standby preempt standby authentication md5 key-string 7 <secret> For DHCP forwarding

(within the VRF) For HSRP (within the VRF)


 VLANs are mapped into VRFs via definitions on SVIs.

“show ip vrf” shows all VLAN-to-VRF mappings …


OUTER-2# show ip vrf Name Default RD Interfaces MED-ADMIN 65111:521 Vl185 Vl431 Vl627 Vl2199

VLANs mapped into the VRF (in this example, 4 VLANs total in this VRF, on this PE)

OUTER-2# show run int Vlan 627 ! interface Vlan627 description VLAN-627-MED-ADMIN mtu 8500 ip vrf forwarding MED-ADMIN ip address 172.16.10.253 255.255.255.0 ip helper-address 10.10.5.1 ip helper-address 10.10.6.1 standby ip 172.16.10.254 standby priority 105 standby preempt standby authentication md5 key-string 7 <secret>

Outer Core (PE), SVI configuration


 Connected routes and static routes within the VRF are redistributed into MP-BGP …

making these routes available on other PEs that also host this VRF …

Outer Core (PE), Connected / Static interface Vlan185 ip vrf forwarding MED-ADMIN ip address 172.16.2.253 255.255.255.248 interface Vlan431 ip vrf forwarding MED-ADMIN ip address 172.16.5.253 255.255.255.128 interface Vlan627 ip vrf forwarding MED-ADMIN ip address 172.16.10.253 255.255.255.0 interface Vlan2199 ip vrf forwarding MED-ADMIN ip address 172.16.33.253 255.255.255.192 ip route vrf MED-ADMIN 0.0.0.0 0.0.0.0 172.16.2.249 name DEFAULT-ROUTE=MED-ADMIN

Static route to virtual firewall (in this example, co-located on this PE router, via VLAN 185)

Connected routes within the VRF (local SVIs on this PE router)

Named Static

Routes


router bgp 65111 ! address-family ipv4 vrf MED-ADMIN redistribute connected redistribute static no synchronization network 0.0.0.0 exit-address-family

Outer Core (PE), Connected / Static Outer Core (PE), Redistribution interface Vlan185 ip vrf forwarding MED-ADMIN ip address 172.16.2.253 255.255.255.248 interface Vlan431 ip vrf forwarding MED-ADMIN ip address 172.16.5.253 255.255.255.128 interface Vlan627 ip vrf forwarding MED-ADMIN ip address 172.16.10.253 255.255.255.0 interface Vlan2199 ip vrf forwarding MED-ADMIN ip address 172.16.33.253 255.255.255.192 ip route vrf MED-ADMIN 0.0.0.0 0.0.0.0 172.16.2.249 name DEFAULT-ROUTE=MED-ADMIN

Redistribution of routes to other PEs (via MP-BGP)

 Connected routes and static routes within the VRF are redistributed into MP-BGP …

making these routes available on other PEs that also host this VRF …


OUTER-2# show ip route vrf MED-ADMIN Routing Table: MED-ADMIN Codes: C - connected, S - static, R - RIP, M - mobile, B – BGP . . . Gateway of last resort is 172.16.2.249 to network 0.0.0.0 172.16.0.0/16 is variably subnetted, 3 subnets, 3 masks C 172.16.2.248/29 is directly connected, Vlan185 B 172.16.139.192/27 [200/0] via 192.168.1.3, 5w3d C 172.16.5.128/25 is directly connected, Vlan431 C 172.16.33.192/26 is directly connected, Vlan2199 B 172.16.155.128/26 [200/0] via 192.168.1.5, 5w3d B 172.16.211.0/26 [200/0] via 192.168.1.5, 5w3d C 172.16.10.0/24 is directly connected, Vlan627 S* 0.0.0.0/0 [1/0] via 172.16.2.249

Outer Core (PE), IP Routing Table

 Now that we have done all of our VLAN and VRF definitions, SVI configuration, and route redistribution …

… let’s see the results …

Route via PE – Outer-3

Route via PE – Outer-5


OUTER-2# show ip route vrf MED-ADMIN Routing Table: MED-ADMIN Codes: C - connected, S - static, R - RIP, M - mobile, B – BGP . . . Gateway of last resort is 172.16.2.249 to network 0.0.0.0 172.16.0.0/16 is variably subnetted, 3 subnets, 3 masks C 172.16.2.248/29 is directly connected, Vlan185 B 172.16.139.192/27 [200/0] via 192.168.1.3, 5w3d C 172.16.5.128/25 is directly connected, Vlan431 C 172.16.33.192/26 is directly connected, Vlan2199 B 172.16.155.128/26 [200/0] via 192.168.1.5, 5w3d B 172.16.211.0/26 [200/0] via 192.168.1.5, 5w3d C 172.16.10.0/24 is directly connected, Vlan627 S* 0.0.0.0/0 [1/0] via 172.16.2.249

Outer Core (PE), IP Routing Table

 Now we have Campus-wide “virtual networks” for departments and faculties, on demand!

  Simplified departmental routing table

  Secure network segmentation

  Scalable


From this …

To this …

Faculty Virtual Firewall


OUTER-2# ping vrf MED-ADMIN 172.16.155.188 Type escape sequence to abort. Sending 5, 100-byte ICMP Echos to 172.16.155.188, timeout is 2 seconds: !!!!! Success rate is 100 percent (5/5), round-trip min/avg/max = 1/1/1 ms OUTER-2#

OUTER-2# traceroute vrf MED-ADMIN 172.16.155.188 Type escape sequence to abort. Tracing the route to 172.16.155.188 1 10.1.1.1 [MPLS: Labels 165/27 Exp 0] 4 msec 0 msec 0 msec 2 172.16.155.188 0 msec 0 msec 0 msec OUTER-2#

“ping” within a VRF “traceroute” within a VRF

 Network maintenance and troubleshooting tasks are readily adapted into a multi-VRF environment …


OUTER-2# show ip arp vrf MED-ADMIN Protocol Address Age (min) Hardware Addr Type Interface Internet 172.16.2.250 204 0034.a984.4c80 ARPA Vlan185 Internet 172.16.2.249 169 0086.c873.d200 ARPA Vlan185 Internet 172.16.2.254 - 0000.0c07.ac00 ARPA Vlan185 Internet 172.16.2.253 - 0018.a4e4.d700 ARPA Vlan185 Internet 172.16.5.253 - 0018.a4e4.d700 ARPA Vlan431 Internet 172.16.33.202 10 0019.e8db.7da3 ARPA Vlan2199 Internet 172.16.5.177 3 001c.f257.4581 ARPA Vlan431 Internet 172.16.10.14 28 001e.a599.b9a7 ARPA Vlan627 Internet 172.16.10.25 1 001a.02ec.3af2 ARPA Vlan627 . . .

“show ip arp” within a VRF

 Network maintenance and troubleshooting tasks are readily adapted into a multi-VRF environment …


IP Header IP payload IGP Tag VPN Tag

IP Header IP payload VPN Tag

IP Header IP payload

IP Header IP payload

Logical Network Flow

OUTER-2# show mls cef vrf MED-ADMIN 172.16.139.92 Codes: + - Push Label Prefix Adjacency 172.16.139.92/27 Te1/1 19(+), 1193(+)

PE

P

PE

Tag push

RED = traffic within the Red VRF

GOLD = MPLS-encapsulated traffic

Tag pop (IGP) Tag pop (VPN),

IP lookup


 Traffic between VRFs will traverse a Virtual Firewall –

… these are hosted on UBC’s FWSMs …

UBC operates multiple FWSMs, distributed around Campus.

The default route in each departmental VRF points to that VRF’s redundant Virtual Firewall instance …

… hosted on an FWSM pair (on the PEs).

OUTER-2# show ip route vrf MED-ADMIN Routing Table: MED-ADMIN Codes: C - connected, S - static, R - RIP, M - mobile, B – BGP . . .

Gateway of last resort is 172.16.2.249 to network 0.0.0.0

...

S* 0.0.0.0/0 [1/0] via 172.16.2.249



PE

Virtual FW

PE

PE

P

PE


PURPLE = IP routed traffic (via OSPF)

GREEN = traffic within the Green VRF

Virtual FW

Policy

Policy



Border

P

PE

PE


PURPLE = IP routed traffic (via OSPF)

Virtual FW

Policy


 Previously, UBC did not provide Campus-wide multicast –

… inability to securely limit multicast use.

 Now with Network Virtualization, UBC can provision multicast support in some VRFs, and not in others.

 UBC employs both intra-VRF (MVPN) and inter-VRF (Extranet MVPN) in their Campus multicast deployment.

 Benefit – UBC can now securely enable multicast on-demand, within and between VRFs, Campus-wide.

This opens up many new possibilities for educational data delivery.


 UBC is virtualizing their Data Center –

using industry-leading virtualized load-balancing, Virtual Machine, and virtualized storage technologies.

 Virtual Load Balancing –

UBC is leveraging the capabilities of their ACE platforms to create per-department / faculty virtual load balancer instances, associated with the department / faculty VRFs and Virtual Firewalls.

 Benefit – allows for individualized, high-performance load-balancing, with separated management

(aids in scaling, manageability, and solution customization).


 Virtual Machines – can be mapped into VLANs … which are mapped into VRFs …

Direct VM access, at high speed, from anywhere on Campus

VMs within the DC can be protected behind departmental / faculty Virtual Firewalls as needed.

 New virtualized services can be spun up on demand in the DC, and mapped out to departments and faculties –

Fully secured by the UBC virtualized network infrastructure.

 Benefit – allows rapid adoption of VM services –

Hosted in the UBC Data Center, securely integrated Campus-wide.


 Virtual Storage – UBC is using Nexus 5000 switches and IP-based NAS storage to front-end their SAN.

This NAS storage can be virtualized, mapped into VLANs / VRFs, provided to Virtual Machines, and located behind virtual firewalls.

 This allows UBC to deploy virtualized, IP-based storage –

Mapped into their virtualized Campus and Data Center network topology, for secure, flexible access.

 Benefit – allows high-performance, virtualized, IP-accessible storage

Securely integrated into, and made available seamlessly across, the UBC virtualized network architecture.


 UBC is examining the use of IPv6 on Campus – Small deployments underway, with a larger Campus deployment under consideration / in planning.

Catalyst 6500 core switches offer support for 6VPE capability.

Cisco Catalyst 6500: Building IPv6-Ready Campus Networks -

http://www.cisco.com/en/US/prod/collateral/switches/ps5718/ps708/solution_overview_c22-531339.html


 Response from UBC Faculties and Departments –

Demand for UBC’s virtualized network deployment has been strong since inception in January, 2009.

Departments and faculties have seen the advantages of virtualized networking, and are embracing the technical and business benefits which UBC’s virtualized network infrastructure provides.

UBC is also employing virtualization for additional Campus services and functions.


 UBC has derived many technical benefits from their Network Virtualization deployment …

Scalable deployment for departmental / faculty segmentation.

End-to-end high performance –

High-performance IP / MPLS routing within a VRF.

Ability to consolidate and scale virtualized firewalls, for traffic moving between VRFs.

Ability to securely deploy multicast (MVPN / Extranet MVPN).

Ability to securely roll out a suite of virtualized DC services, for high-performance, secure data access.


 UBC has derived many business benefits from their Network Virtualization deployment …

The network design reflects UBC organizational boundaries.

For the first time, security is an integral part of network deployment.

The network design can scale to 10Gbps+, low-performance firewalls are no longer a constraint.

UBC departments and faculties can locate services anywhere on Campus (including within the UBC DC), and enjoy the same secure, high-performance, seamless access.

Economies of scale, “greener” – using centralized services.


www.cisco.com/go/networkvirtualization


#CNSF2011

Thank you.

#CNSF2011

Network Infrastructure Virtualization Case Study

Technology

Transcript of Network Infrastructure Virtualization Case Study