Data Center Networking Introduction · ***BGP EVPN, NSX, DCN and DCB are Aruba roadmap items -...
Transcript of Data Center Networking Introduction · ***BGP EVPN, NSX, DCN and DCB are Aruba roadmap items -...
1
Data Center Networking Introduction
2
What makes a data center network?
– Local Area Network (LAN) / Campus Networks
– Same geographical location, building, campus etc.
– Wired and wireless network connects users, IP phones and wireless APs
– Typical features required: POE, 802.1X etc
– Data Center Networks
– Same geographical location
– Connects Servers/VMs/Containers, applications, storage, firewalls/ load
balancers, etc. – wired connectivity
– Stable, low latency fabrics with high availability / high performance and
throughput / density and scale
– Build revenue for business (E-Commerce)!
– Typical features required: VXLAN/EVPN, BGP, OSPF, DCB, etc..
– Focus on improving East - West traffic between racks
Spines
Leafs
…
3
What is a network fabric?
Marketing term
– Optimally interconnect 1,000, 10,000, 100,000 or more end points (servers, storage)
– Provide redundancy when any node or any link fails
– Failure will happen – it’s just a question of time
– Minimize # hops to reach any other peer in the fabric
– Latency impact
– East/West (E/W) traffic vs North/South (N/S) traffic
– E/W traffic = Servers to servers inside the DC
– N/S traffic = Clients to servers entering / servers to clients leaving the DC
4
Data Center Networking Architectures
5
Scalability, Agility, Orchestration
Enterprise Datacenter Network Architecture Evolution
* VXLAN connections are created automatically, on demand, between leaf Switches/vSwitches. ** Network Virtualization – VMware NSX, OpenStack, HPE Distributed Cloud Networking/Nuage
L3
L2
LACP
Classic / Underlay VxLAN Overlay
SW VTEPs
SW & HW VTEPs
HW VTEPs
Optimized L2/L3 FabricIRF/VSX MLAG
L3 Fabric
Spine&Leaf L3 ECMPVXLAN* ,EVPN & Network Virtualization**
VMs VMs
vSwitch vSwitch
…
…
Spine&Leaf L2 ECMPTRILL/SPB
L2 Fabric
…
…
Traditional 3 layer STP
6
Does Every DCN Solution = Spine/Leaf?
L2
Spines
Leafs
L3
1-Tier Data Center
Core
L2
L3
L2
Agg
Access
…
L3
…Core
Multi-Tier Data Center
– Spine = Multiple individual backbone devices that provide redundant connectivity for each leaf
– Leaf = Switch which connects to every spine switch (can be VTEP but not mandatory), provides entry into equidistant networks with no constraints on workload placement
– Core = A single device (logical or physical) that provides centralized connectivity to other devices (servers/switches)
– Aggregation = Aggregates multiple access switches – usually performs L2/L3 services
– Access = Typically connects into a Core or Aggregation device – usually running L2 services
– ToR = Umbrella term, referring to a switch located at the Top-of-Rack
7
EoR / MoR (End of Row / Middle of Row)
– EoR/MoR refers to physical location of switches where switches are placed in one rack
– Server-to-switch cables stretch from rack to rack, usually requires less equipment than ToR deployment
– Usually lower latency for intra-row traffic because of less hops
– Less problem isolation, less scalability
– EoR/MoR could be spine switches which connect to ToRs within the same Row
– Can be considered as 1 POD, replicate design to scale up multiple PODs
EoR MoR
Data center spine/core/WAN edge
EoR MoR
Data center spine/core/WAN edge
ToR ToR ToR ToR ToR ToR ToR ToR
9
Leaf 64Leaf 33
40/100G 40/100G
ISL MLAGISL
Data Center Fabric (Spine-Leaf)Scaling up the leafs
40/100G
32 x 100G
Leaf 1 Leaf 32
40/100G
Question: What determines the number of leafs supported in a spine leaf topology?
Answer: The number of ports supported in the spine switch.Answer: The number of physical ports supported in a single spine switch.
– Every leaf needs to connect to every spine
– Recommendation is not to use VSX / IRF on spines
10
Data Center Fabric (Spine-Leaf)Understanding oversubscription
Spine-1 Spine-2 Spine-3 Spine-4
40/100G40/100G
40/100G
40/100G160G / 400G
(4x40G / 4x100G)
Leaf Leaf
• Scale of the fabric defined by the density of the spine switch
• Fabric bandwidth can be increased by adding more spine switches
48 x 10G ports = 480G
– 40G Uplinks = 3:1 Oversubscription
(480G/160G = 3)
– 100G Uplinks = 1.2:1 Oversubscription (480G/400G = 1.2)
48 x 25G ports = 1,200
– 40G Uplinks = 7.5:1 Oversubscription
(1,200G/160G = 7.5)
– 100G Uplinks = 3:1 Oversubscription (1,200G/400G = 3)
40/100G
12
Aruba Data Center NetworksBenefits or modern DCs
– A stable, low latency fabric with high availability/ performance/ density/ scalability
– N/S campus/client traffic connectivity achieved via border switches (service leafs) /routers
– L2 extension between racks: Essentially driven by VM mobility
– VXLAN as de-facto solution by many overlay vendors
– Scalable, up to 16M Virtual Network Identifier (VNIs) to support multi-tenancy
– Oversubscription
– Spine and leaf for fewer layers and reduced hop count / latency / oversubscription levels
– Designed for E/W application traffic performance (80% of traffic is EW)
– Mac Address Explosion
– DC fabric becomes a big L3 domain (no STP) with L2 processing (encapsulation / de-capsulation) at the edge
13
Data Center Networking Portfolio
14
Addressable Market for Aruba Switching will Double from CY18
8.9 9.0 9.3 9.5 9.6 9.7
4.0 4.1 4.2 4.3 4.4
6.1 5.7 5.2 4.5
3.4 4.0 4.7
1.6
20222018 2019 20202017
13.0
Telco
2021
DC for Tier 2 Cloud
23.1
DC in the Enterprise
Campus Core & Agg
Campus Access
19.4
22.8
24.9
8.9
Note: Excludes hyper-scale data center TAMSource: Dell’Oro (Worldwide Datacenter Ethernet Switching Revenue 2016-22); HPE Market Model
Aru
ba
po
rtfo
lio b
rea
dth
and s
tre
ngth
2015+
2017+
2019+
2020+
TAM ($B)
Today
16
High Level Selection Considerations
Consistency with campusAnalytics, automation, and simplicity
Interest in CX Innovations
Traditional requirementsSoftware feature depth
Unique requirements & integrations
ArubaFlexFabric
18
FY19 DC Portfolio: FlexFabric Options
Sp
ine
Leaf
12900E Series: 4, 8, 16 slots
12901E Series
5710 Series 1/10GbE
– 40GbE up ToR / server iLO
594x fixed/modular 10/40GbE
5950 fixed/modular 10/25/50/100GbE
1/10GbE ToR, price/perf 1-100GbE fixed and modular ToR flexibility
Compact, cost effective, 100GbE (small core/spine) Highest density, 25/100GbE flexibility and features
5980 advanced 10/100GbE Storage/HPC ToR
5980 advanced 10/100GbE5950 32 * 100G
12902E Series
FlexFabric Portfolio
19
FlexFabric Leaf Options
HPE FlexFabric
5710 series
HPE FlexFabric
5940 and 5945 series
– 1/10 GbE downlinks x
40/100G uplinks
– Low-latency, high-availability
connectivity
– Perfect for out-of-band
management (iLO)
connectivity
– 10 or 25 GbE downlinks x
40/100G uplinks
– VXLAN support for network
virtualization
– Low-latency, high-availability
connectivity
– Enhanced support for
telemetry
HPE FlexFabric
5980 series
– Full data path error detection
– 1/10 GbE downlinks x 100G
uplinks
– VXLAN support for network
virtualization
– Deep buffers to ensure
network connectivity
– Flexible port configurations
20
FlexFabric 12900E FamilyChassis Sizes to Match Capacity Needs
Up to 120 TbpsSwitching Capacity
Up to 768Concurrent EVPN Sessions
Up to 307225 GbE ports
– Mix and match switches to meet local needs
– Ideal for east-west traffic and reducing the number of tiers
– Leaf-spine increases resiliency and reduces complexity for traffic policies
– Upgrade to 10/25GbE at the edge
– Future-proof with 100GbE in the core today and 400GbE in the future
12904E 12908E 12916E12902E12901E
21
Aruba 8320
Aruba 8400
Mo
bile
Fir
st
Ca
mp
us
Ne
two
rkin
gC
ore
Op
po
Rtu
nis
tic
ally
Sp
ine
/ L
ea
f
Future proof wired infrastructure, WLAN and IoT enabling
Highly scalable, programmable automated Data Center solution
User, device, server aware – ZTP ease of deployment
Aruba Core & Datacenter Switching: Powered by CX Innovations
Aruba 8325
22
Modernizing Campus Core, Aggregation and Data Center
Aruba 8000 Series
with ArubaOS-CX
Aruba 8400
• Highest reliability, flexibility, performance, port density
• 19 Tbps system, 8-slot chassis
• Redundancy everywhere: Mgmt.
Module, Fabric, Power, Fans
• Up to 512 10GbE, 128 40GbE, 96 100GbE in a 2-chassis pair
Aruba 8320
• Workhorse for mid-size core/aggregation use cases
• 2.5 Tbps system, 1RU
• N+1 redundant hot swappable
power supplies, fans
• Three models: 48 x 10GbE, 48 x 10GBASE-T, 32 x 40GbE
Aruba 8325
• Mid-size core/aggregation use cases and DC ToR or EoR
• 6.4 Tbps system, 1RU
• N+1 redundant hot swappable power supplies, fans
• 32 ports of 40/100 GbE or
• 48 ports of 10/25 GbE and 8 ports of 40//100 GbE
• Front to back or back to front airflow
23
Typical Customer Deployments – Products and features
Collapsed 1-TierIRF/VSX/ MLAG
#Servers: 50-100 ~ 100 - 500 ~500 – 2000 ~ 2000+VM Scale: 5000+ 25,000 – 50,000+ 100,000 – 500,000 + 750,000 +
Customer Small Server Rooms Small to Medium Data Centers Medium to Large DC Large and Complex Data Centers Persona K-12 School Districts Education, Local Gov., Retail Enterprises, Universities Financial Services, Large Enterprises
Features: L2, MCLAG, VSX, DCB, L2, VSX (MCLAG + Config Sync), ECMP, L3 Routing, IPv6, VSX, VXLAN with MP-BGP EVPN, ECMP, API Integration DCB, API Integration DCB, NSX, API Integration L3, VSX, DCB, NSX, API Integration***
Optimized L2 FabricIRF/VSX MLAG
Optimized L3 FabricIRF/VSX MLAG
L3 Fabric
Spine&Leaf L3 ECMPVXLAN* & EVPN
VMs VMs
vSwitch vSwitch
…
…
Products: Aruba 8400* Core: Aruba 8400/FF 5950/12900E Spine: Aruba 8400/FF 5950/12900E Spine: Aruba 8400/FF 5950/12900E Access: Aruba 83xx/FF 57XX/59XX. Leaf: Aruba 83xx/FF 57XX/59XX Leaf: Aruba 8325**/FF 57XX/59XX
*Aruba 8400 will cover majority of the use-cases. **Aruba 8325 supports Static VXLAN. ***BGP EVPN, NSX, DCN and DCB are Aruba roadmap items
- Position 12901/2 for Small Spine/Core and 12904/8/16 for high density Spine/Core.- Position 5940/5950/12900E for FC requirements. - Position 59XX/12900E for FCoE requirements
24
Scalability, Agility, Mobility
Enterprise Data Center Aruba Network Architecture Evolution*
Phase 1: 1H FY19
Phase 2:2H FY19
*This is an Aruba portfolio view
Collapsed 1-TierIRF/VSX/ MLAG
Optimized L2 FabricIRF/VSX MLAG
Optimized L3 FabricIRF/VSX MLAG
L3 Fabric
Spine&Leaf L3 ECMPVXLAN* & EVPN
VMs VMs
vSwitch vSwitch
…
…
25
Management and Orchestration
AirWave
Unified multi-vendor
wired + wireless network management
Core, Aggregation and Data Center
NAE
Flexible troubleshooting and automated root
cause analytics simplify and enhance visibility
and control
NetEdit
Scalable, Simple
CLI-based Orchestration
IMC
Advanced wired management
26
Data Center Networking Technologies
27
Connecting Servers and Switches
28
Connecting Servers/Endpoints
Single Attached
– No HW networking redundancy
– Fewer cables/NICs
– Example: partitioned replica set solutions like Hadoop or MongoDB
– i.e. – if single server looses connectivity the application and data is still accessible to clients
Dual attached
– Link redundancy
– Higher bandwidth
– Multiple NICs can provide NIC redundancy
– More cables/NICs
– LACP recommended to detect required links
– Example: environments that require more bandwidth
– Switch/Link/NIC redundancy
– Higher bandwidth/performance with no downtime via upgrades
– SW and HW redundancy with active-active L2/L3
– Simplified management
– LACP recommended to detect required links
– Example: VMware environments that require hosts always online
Switch
Server
Dual switch attached
29
– Active / Passive
– Servers normally don‘t care about exit
– Suboptimal traffic flow
– High Inter-Switch-Link load
– Only one exit is usable(MAC flapping)
– LACP
– Servers can use both links
– Optimal traffic flow
– No Inter-Switch-Link load
Topology ExamplesSingle Server Connectivity – Active/Passive vs. LACP
Active / Passive LACP
Active
Backup
30
VSXMCLAG
vPCMLAG
Switch Virtualization Comparison
Chassis 1 Chassis 2
Management
Control
Routing
Chassis 1 Chassis 2
Management
Control
Routing
Ethernet Links
Shared
Management
Control
Routing
Ethernet Links
Shared
SYNC?SYNC
IRF VSFVSS
Virtual Chassis
IRF / VSX
ISL
VSX is ideal solution providing minimal scope of outage, each box operates separately in concert
Single control plane Dual control plane
31
Comparison of Virtualization Solutions
Features Aruba 8400(1)/8320
(with VSX)
FlexFabric
129xx/59xx/57xx
(with IRF)
Cisco Nexus
3/5/7/9xxx
(with vPC)
Arista 7xxx
(with MLAG)
IP’s for Switch Management 2 1 2 2
Control Planes 2 1 2 2
HA during upgrades Built in by designISSU within major
code branchesBuilt in by design No
Active-Active Unicast &
Multicast(2)Supported Supported Supported No
Config Simplicity and
TroubleshootingExtensive Support Single config Limited No support
MC Port-channel/LAG L2 and L3 L2 and L3L2 Only
(except 5K)L2
First Hop RedundancyEliminates need for
VRRP – less configuration
Eliminates need for
VRRP – less configuration
Needs VRRP/HSRPNeeds proprietary
virtual ARP feature
(1) Requires dual supervisor in each chassis. (2) Intended for a future software release
32
Aruba CX Major Release versus Minor ReleaseUpgrade Scenarios
Upgrade between Minor
Releases
Upgrade between Major
Releases (ISLP comp.)
Versions Path Example from 10.1.0008 to 10.1.0015 from 10.1.0018 to 10.3.0009
ISLP Version Compatibility Yes Yes
When one switch is upgraded and reboots, it
re-joins the current VSX pair.Yes Yes
VSX pair operates with different SW release
for a transition periodYes Yes
Maximum Impact during upgrade steps ~ 300ms (unicast) ~ 300ms (unicast)
Goal cumulated Impact for complete upgrade <1s (unicast) <1s (unicast)
33
Aruba CX Upgrade Process Details
Start with VSX Secondary node:
• copy tftp://.. primary
• boot system primary
– Routing protocols graceful shutdown
– While SW2 reboots, SW1 is forwarding
Step 0
Step 1
– As soon as SW2 is back from reboot, VSX is In-Sync state
– SW1 is forwarding
– SW2 is learning (LACP, MAC, ARP, routing) and linkup-delay
– SW2 is forwarding. VSX-sync is stopped.
Step 2
Step 3
Step 4
ISL
VSX
primary secondary1 2
VSX LAG
VSX LAG
VSX
primary secondary1 2
VSX LAG
VSX LAG
ISL
VSX
primary secondary1 2
VSX LAG
VSX LAG
VSX
primary secondary1 2
VSX LAG
VSX LAG
ISL
VSX
primary secondary1 2
VSX LAG
VSX LAG
Finish with VSX Primary node:
• copy tftp://.. primary
• boot system primary
– Routing protocols graceful shutdown
– While SW1 reboots, SW2 is forwarding
– As soon as SW1 is back from reboot, VSX is In-Sync state.
– SW2 is forwarding
– SW1 is learning (LACP, MAC, ARP, routing) and linkup-delay
– SW1 is forwarding. VSX-sync is running.
– Routing protocols nominal metrics restore
UP-to-DOWN < 667 ms
DOWN-to-UP < 1775 ms
UP-to-DOWN < 18 ms
DOWN-to-UP < 1551 ms
– SW1 and SW2 are forwarding.
– VSX-sync is running
34
FlexFabric Upgrade Scenarios
Compatible Incompatible
4 steps:– issu load file … slot <slave>– issu run switchover
– issu accept– issu commit slot <x>
4 steps:– boot-loader … slot 1– reboot slot 1
– boot-loader … slot 2– reboot slot 2
3 reboots:– Original IRF master: 2 reboots– Original IRF slave: 1 reboot
2 reboots:– Original IRF master: 1 reboot– Original IRF slave: 1 reboot
The ISSU run switchover command reboots the current
master with the old software version, causing the
upgraded subordinate member to be elected as the new
master.
Brute force method.
IRF will not reform so there is no switchover option
35
FlexFabric Compatible upgradeProcess details
IRF1 2master slaveSW1 and SW2 are forwarding.
Initial state: Unit 1 is master.
Note: Unit 1 can also be slave and Unit 2 master.
LACP
LACP
IRF1 2
LACP
LACP
Start ISSU with member 2:
– issu load file … slot <slave>
While 2 reboots, 1 is forwarding
Step 0
Step 1
IRF1 2
LACP
Switch master/slave:
– issu run switchover
While 1 reboots, 2 is forwarding and master
Step 2
LACP
IRF1 2
LACP
LACP
Step 3
IRF1 2
LACP
LACP
Step 4 Initial state: version R24xx (new code). Unit 2 is master
master
master
master
slave
slaveComplete ISSU on member 1:
– issu commit slot 1<new slave>
While 1 reboots, 2 is forwarding and master
masterslave
UP-to-DOWN < 5 ms
DOWN-to-UP < 11 ms
UP-to-DOWN < 14 ms
DOWN-to-UP < 6 ms
36
FlexFabric Incompatible upgradeProcess details
Initial state: version R23xx. Unit 1 is master.
Note: Unit 1 can also be slave and Unit 2 master.
– upload new firmware on flash of IRF members
Start with IRF member 1:
– boot-loader … slot 1
– reboot slot 1
While 1 reboots, 2 is master and forwarding
Step 0
Step 1
As soon as unit 1 is back from reboot, MAD occurs
As unit 1 is the lower member ID, it becomes new master and forwarding
Unit 2 enters MAD recovery state and shutdown all its ports
except IRF ones
Control plane switchover leads to reset of all routing peerings
Step 2master
MAD faulty state
Step 3– boot-loader … slot 2
– reboot slot 2
Step 4 Initial state: version R24xx. Unit 1 is master
IRF1 2master slave
LACP
LACP
IRF1 2
LACP
LACP
IRF1 2
LACP
LACP
IRF1 2
LACP
LACP
IRF1 2
LACP
LACP
master
masterMAD recovery state
master
master slave
UP-to-DOWN < 5 ms
DOWN-to-UP < 254 ms
UP-to-DOWN < 14 ms
DOWN-to-UP <5ms
37
Connectivitry Options
38
Connectivity OptionsForm
FactorSpeed Connectivity Options Type of Fibers/Conductors
Optical
Connector.
BaseT10/100/1000M
RJ-45Cat 5 and up n/a
10GbE Cat 6a (55M) / Cat 6a and up (100m) n/a
SFP 1GbESX 2-strand MMF (multi-mode fiber) LC
LX, LH 2-strand SMF (single-mode fiber) LC
SFP+ 10GbE
SR, LRM 2-strand MMF LC
LR, ER 2-strand SMF LC
Copper Direct Attach Cable (DAC) 4-conductor twinax copper n/a
Active Optical Cable (AOC) Fixed MMF cable n/a
SFP28 25GbE
SR 2-strand MMF LC
Copper DAC 4-conductor twinax copper n/a
AOC Fixed MMF cable n/a
QSFP+ 40GbE
SR4, eSR4 8-strand MMF MPO
LR4, ER4 2-strand SMF LC
BiDi 2-strand MMF LC
Copper DAC 16-conductor twinax copper n/a
AOC Fixed MMF cable n/a
QSFP28 100GbE
SR4 8-strand MMF MPO
LR4, ER4 2-strand SMF LC
BiDi 2-strand MMF LC
Copper DAC 16-conductor twinax copper n/a
AOC Fixed MMF cable n/a
39
How do we achieve 40GbE/100GbE?Multiple lanes bundled into single link
10Gb
10Gb
10Gb
10Gb
Se
rve
r Sw
itch
Today’s 4 lane 10GbE offeringsefficiently scale to 40GbE
10 Gb
10 Gb
10 Gb
10 Gb
Serv
er S
witc
h10 Gb
10 Gb
10 Gb
10 Gb
10 Gb
10 Gb
A 10GbE lane solutionDOES NOT efficiently scale to 100GbE
25 Gb
25 Gb
25 Gb
25 Gb
Serv
er S
witc
h
25GbE provides a seamless and far more efficient migration to 100GbE
40
Splitting Ports
– 40GbE/100GbE ports can be split to leverage individual 10/25GbE lanes
– “Splitter cable” DACs available:
– QSFP+ (40GbE) to 4 x SFP+ (10GbE) DAC = “using tengige”
– QSFP28 (100GbE) to 4 x SFP28 (25GbE/10GbE) DAC = “using twenty-fivegige”
– QSFP+ (40GbE) to 4 x SFP+ (10GbE) optical splitter options use:
– QSFP+ SR4 (MPO) > 40GbE MPO x 4 10GbE LC Cable (K2Q46A)
– QSFP+ 40GbE SR4 MPO Optic
– SFP+ 10GbE SR LC Optic
– QSFP28 (100GbE) to 4 x SFP28 (25/10GbE) optical splitter options use:
– QSFP28 SR4 (MPO) > 40GbE MPO x 4 10GbE LC Cable (K2Q46A)
– QSFP28 40GbE SR4 MPO Optic
– SFP28 25/10GbE SR LC Optic
41
Which ports can split?
– The ability to split ports is dependent on PHY used in port
– Not all PHYs are equal - not all 40/100GbE ports can be split
– See FlexFabric Splitting Ports doc on Arubapedia, Iris, and Configuration Guides
42
Emerging MSAs & Consortiums
– Focused on optimizations for 25G & 50G per-lane based physical layers
– 25, 50, 100, 200, and 400 GbE
– RCx MSA
– microQSFP MSA
– QSFP-DD Consortium
RCx microQSFP QSFP-DD
43
Take the express lane with 25/100GbESolutions across HPE server, firmware, NICs, 25/100 GbE fabric
HPE 25GbE
Network Adapters
HPE ProLiant DL/ML, Blade servers HPE access/leaf switches
HPE FlexFabric 5950/5945
HPE Aruba 8325
HPE core/spine switches
HPE FlexFabric 129xx
HPE Aruba 8400
Notes:
– 25-GbE interfaces can work at 25, 10, or 1 Gbps.
– 25-GE interfaces do not support speed or duplex mode autonegotiation.
– Must manually configure speed / duplex to ensure interfaces at both ends have the same speed and duplex mode settings.
44
Media (Channels) for IEEE 25GbE Spec
– Backplane: 25GBase-KR (<30”)
– Autoneg between 10Gb & 40Gb
– Next Gen Blade Servers will be plumbed for 25Gb-KR
– Passive DAC (Twinax) Cable: 25GBase-CR (<= 3m & 5m)
– <2m may not need any FEC. Lowest Latency
– 3m requires Base-R FEC (clause 74)
– >3m requires RS-FEC (clause 108)
– RCx Copper Cables (consortium MSA)
– Low-cost high-density 25G electrical cable & connector set
– Optic MMF (OM4): 25GBase-SR (<=100m)
– Optic SMF: 25GBase-LR (<=10km) – draft
– Optic SMF: 25GBase-ER (<=40km) – draft
QSFP28 to
x4 SFP28
45
Switch upgrade (Incremental)Consistent transceiver form factor 10G/40G > 25G/100G allows seamless transition
40G TOR Switch
(10G IO)
1
4
2
4
8
10G
Servers
10G
10G
1 96
10GE
40GE
To Spine
4
100G TOR Switch
(10/25G IO)
1
4
2
4
8
10G
Servers
10G
10G
1 96
10GE
40GE
To Spine
100G TOR Switch
(10/25G IO)
1
4
2
4
8
10G
Servers
10G
25G
1 96
25GE
100GE
To Spine
40GE
10GE
100G TOR Switch
(10/25G IO)
1
4
2
4
8
25GG
Servers
25GG
25G
1 96
25GE
100GE
To Spine
4
Day 1 (starting point)
– 8 x 40GE up
– 96 x 10GE down
Install new TOR !
– 8 x 40GE up
– 96 x 10GE down
– No network capacity
loss !
Incrementally upgrade
uplinks to 100GE and
downlinks to 25GE
– Hybrid configuration
Upgrade complete!
– 8 x 100GE up
– 96 x 25 GE down
46
Buffers?
47
BuffersDo they matter? And where?
2.4:1 oversubscription
100G
32 x 100G
Leaf 1 Leaf 32
200G (2 x 100G)
480G (48 x 10G)
100G
1:1 oversubscription
100G
32 x 100G
Leaf 1 Leaf 32
200G (2 x 100G)
200G (20 x 10G)
100G
– Congestion will exist in TCP/Ethernet Networks. Buffers are used to help address congestion
– Congestion Collapse prevents or limits useful communication / Bufferbloat can cause excess packets in switch buffers
– Spine experiences same traffic load in both scenarios – uneven flows can cause congestion at spine
– Spine switches experience more consistent traffic flows with possible congestion
– Use techniques like WRED and RED end-to-end to address congestion situations
48
BuffersDeep Buffer vs Standard Buffer Switches
To profile an application answer these 3 questions:
– Does the application utilize close to 100% of line rate for sustained periods?
Consider Deep Buffer:
– Where loss sensitive applications are mixed with bursty applications.
– Large amount of Elephant flows which can fill out buffers and starve mice flows.
– Usually greater than 1GB
Use Standard Buffer for:
– Low latency
– A network where there is a lower oversubscription rates.
– Streaming applications where bandwidth is relatively constant.
– Usually smaller than 30MB
74
ArubaOS-CX & HPE FlexFabric Interop
Can customers integrate ArubaOS-CX switches with HPE FlexFabric?
Refer to ArubaOS-CX & HPE FlexFabric Interop guide on Arubapedia for tested features
75
Interop Test CasesInterop test cases listed are focused on features used in Data Center Networks.
LLDP
• LLDP Test #1 – Neighbor Detection
STP
• STP Test #1 – Loop Prevention
LACP
• LACP Test #1 – L2 Dynamic link aggregation
• LACP Test #2 – L3 Dynamic link aggregation
• LACP Test #3 – L2 Dynamic link aggregation (VSX & IRF)
OSPF
• OSPF Test #1 – L3 Network advertisement and reachability
• OSPF Test #2 – BFD Interop
BGP
• BGP Test #1 – IBGP network advertisement and reachability
• BGP Test #2 – EBGP network advertisement and reachability
• BGP Test #3 – BFD Interop
ArubaOS-CX Leafs & HPE FlexFabric Spines in an L3 fabric
• Test #1 – ArubaOS-CX Leafs with VSX
Additional test cases will be added in future as required
76
ArubaOS-CX VSX Leafs with 12900E Spines
• Expected test result: EBGP neighbors should form, networks should be advertised and received. L3 network
connectivity between servers in different racks should work.
• Final test result: Works as expected
• Note: In a production deployment, it is recommend that each physical leaf switch utilize multiple uplinks to
different spines
Rack 12 Rack 14
12904-212904-1
8320-1 8320-2
AS#65001 AS#65002
AS#65003 AS#65004AS#650051/1/15
Server 2 -
192.168.12.10/24
VSX active gateways -
192.168.12.1/24
IRF 5940-3
Server 4 -
192.168.14.10/24
Default gateway -
192.168.14.1/24
/31 links between leafs/spines
Leaf
switches
Spine
switches
77
Questions?
78
Thank You!