Dallas, TX June, 2005

Cindy K YoungIBM Advanced Technical Support, Dallas

youngck@us.ibm.com

IP Multipath Routing IP Multipath Routing and and

Dead Gateway Detection Dead Gateway Detection in AIX V 5.3in AIX V 5.3

en1 tr0

10.5.6.0 Router10.5.2.1

10.5.2.0

.6 .6.6

scenario #1

Router10.5.6.111

ent1

ent2

10.9

0.5.

0

.67

.68

scenario #2

©IBM 2005 - 1

IP Multipath Routing / Dead Gateway DetectionIP Multipath Routing / Dead Gateway Detection

en1 tr0

10.5.6.0 Router10.5.2.1

10.5.2.0

.6 .6.6

scenario #1

Router10.5.6.111

significant IP routing subsystem change

allows multiple routes to the same destination, including multiple defaults

uses route cost value to manipulate local host's IP routing table to load balance and/or react to changing network conditions (failover)

best suited for multi-homed hosts; may be combined with other AIX networking solutions

bandwidth potential due to multiple adapters

AIX v5 minimum, key enhancements in v5.3

function transparent to the network (but assumes IP routing protocols)

ent1

ent2

10.9

0.5.

0.67

.68

scenario #2

©IBM 2005 - 2

IP Multipath RoutingIP Multipath Routing and Route CostRoute Cost

IP Multipath Routingallows more than one route to the same destination - including multiple defaults for all IP interfaces - regardless of network typeworks with or without AIX routing protocolsenabled when a new route to the same destination is added - previously, flagged as a duplicate route but only one route used)for load balancing - or -for failover to backup routes (in conjunction with Dead Gateway Detection)implemented first in AIX V5.0, enhanced in AIX V 5.3

Route Cost attribute used for route selection - lowest value always selected; round robin selection for duplicate routes with lowest cost (equivalent routes); higher value routes never selectedassign to individual routes - via mktcpip -C command, -hopcount flag of route command, or in SMIT menu. Flag existed previously, but didn't impact route selectionview route costs via netstat -Cn commandvalue adjusted by system when a gateway is determined to be down

©IBM 2005 - 3

SMIT Configuration:SMIT Configuration:

Note: The Network Mask field should be left blank for host and default routes. Use only for non-default, network routes.

Add a Static Route

Type or select values in entry fields.Press Enter AFTER making all desired changes. [Entry Fields] Destination TYPE net * DESTINATION Address [default] (dotted decimal or symbolic name) * GATEWAY Address [10.5.2.1] (dotted decimal or symbolic name) COST [0] Network MASK (hex or dotted decimal) [] Network Interface [tr0] (interface to associate route with) Enable Active Dead Gateway Detection? no Is this a Local (Interface) Route? no

fastpath smitty mkroute

New AIX V5 SMIT

fields

©IBM 2005 - 4

IP Multipath Routing for load balancingIP Multipath Routing for load balancing

Note: Path MTU discovery was changed in AIX V 5.3 -- cloned host routes can no longer interfere with round-robin selection!

# netstat -CnRouting tablesDestination Gateway Flags Refs Use If Cost Config_CostRoute tree for Protocol Family 2 (Internet):default 10.5.6.1 UG 0 752 en1 0 0=>default 10.5.2.1 UG 0 0 tr0 0 010.5.2/24 10.5.2.6 U 6 1820 tr0 0 010.5.6/24 10.5.6.6 U 3 2115 en1 0 010.90.5/24 10.5.2.1 UG 0 0 tr0 0 0=>10.90.5/24 10.5.6.1 UG 0 0 en1 0 010.5.2.6 127.0.0.1 UGHS 0 58 lo0 0 0 10.5.6.6 127.0.0.1 UGHS 0 137 lo0 0 0 127/8 127.0.0.1 U 0 542 lo0 0 0

10.5.6.0

Router10.5.6.1

Router10.5.2.1

10.5.2.0

.610

.90.

5.0

en1 tr0

.6 .6

scenario #1

ent1

ent2 67

68

scenario #2

equivalent routes permitted (same destination, netmask, group id, cost)

applies to two or more IP interfaces on different or same network

cost determines route to use -- lowest cost always selected; round robin between equivalent cost; higher cost never selected

©IBM 2005 - 5

AIX V5.3 IP Multipath Routing EnhancementsAIX V5.3 IP Multipath Routing Enhancements

Additional route selection choices within the "multipath routing set" in IP routing table

WRRWRR weighted round-robin -- Default!RNDRND randomWRNDWRND weighted randomLUTLUT lowest utilization (lowest reference count)HSHHSH hash-based (destination IP address)

Default behavior (WRR, weights of 1) matches previous AIX v5 round robin

Avoid HSH if set is comprised of host routes -or- if significant traffic to same IP addr

Terminology change: Multipath routing set replaces equivalent routes. Set of routes with same destination, cost, subnetmask and group values

policy is stored with the first route in set so a single change affects entire set

the mpr_policy (no option) sets system-wide behavior for all multipath routing sets - individual route values override

©IBM 2005 - 6

# netstat -CnRouting tablesDestination Gateway Flags Wt Policy If Cost Config_Cost

Route Tree for Protocol Family 2 (Internet):default 10.3.1.161 UG 3 WRR en0 0 0 =>default 10.10.10.2 UG 2 -"- en1 0 010.3.1.160 10.3.1.172 UHSb 1 - en0 0 0 =>10.3.1.160/27 10.3.1.172 U 1 - en0 0 010.3.1.172 127.0.0.1 UGHS 1 - lo0 0 010.3.1.191 10.3.1.172 UHSb 1 - en0 0 010.10.10.0 10.10.10.4 UHSb 1 - en1 0 0 =>10.10.10/24 10.10.10.4 U 1 - en1 0 010.10.10.4 127.0.0.1 UGHS 1 - lo0 0 010.10.10.255 10.10.10.4 UHSb 1 - en1 0 0127/8 127.0.0.1 U 1 - lo0 0 0

fastpath smitty mkroute Add Static Route

Type or select values in entry fields. [Entry Fields] Destination TYPE net DESTINATION Address [default] Default GATEWAY Address [10.10.10.2] COST [0] Network MASK (hexadecimal or dotted decimal) [] Network Interface [] (interface to associate route with) Enable Active Dead Gateway Detection? no Is this a Local (Interface) Route? no Policy (for Multipath Routing Only) Weighted RR Weight (for Weighted Multipath Routing Policy) [3] Apply change to DATABASE only no

Configurable IP Multipath Routing ExampleConfigurable IP Multipath Routing Example

first route in the set will be selected three times, then the second route will be used twice before cycling back around

two default routes are a multipath routing set

verify route usage via "Ref" and "Use" columns in netstat -rn

©IBM 2005 - 7

Dead Gateway Detection Dead Gateway Detection (DGD)Functional Overview

allows a host to detect and adjust its IP routing tables when a local (first hop) gateway goes down - previously, the host continually tried the dead gatewayassumes network gateways (IP routers) run routing protocols and can react to other router failuresdo not run routing protocols (e.g., routed or gated) and DGD on the same hosttwo discovery alternatives - both active and passive manipulate "route cost" value but differ in the amount of overhead incurred

Passive mode Dead Gateway Detection (Recommended!)"best effort" service with little overheadfor those needing simple detection and failover - but don't want much overheadenabled or disabled for the whole system - passive_dgd option of the no command

Active mode Dead Gateway Detectionquick detection - within secondsfor those requiring very high availability and no disruption in service visible to users - ping traffic from several hosts could degrade network performance enabled or disabled on a route-by-route basis in SMIT or commands - "A" flag

©IBM 2005 - 8

IP Multipath Routing w/DGD for failover to backup routesIP Multipath Routing w/DGD for failover to backup routes

Default passive mode behaviorenabled for the entire system (disabled by default)ARP and TCP/IP recognize lack of response from a gateway

Failure/recovery behavior (both modes)

ARP entry is deleted and route cost set to "max" (2147483647) on all routes to dead gatewayif available, backup routes are usedroutes rechecked every five minutes and, when the gateway is restored, the route cost value is returned to user configured values

enabled on a route-by-route basispings every 5 seconds to all gateways in routing tablewith "A" flaggateway considered deadif three ARP, TCP, or ping responses are missed

Default active mode behavior

en1 tr0

Router10.5.6.1

Router10.5.2.1

10.5.2.0

.6.6

10.5.6.0

©IBM 2005 - 9

AIX V5.3 Routing TableAIX V5.3 Routing Table# netstat -inName Mtu Network Address Ipkts Ierrs Opkts Oerrs Collen0 1500 link#2 0.d.60.a.56.82 394952 0 68600 0 0en0 1500 9.19.126 9.19.126.84 394952 0 68600 0 0lo0 16896 link#1 33917 0 33946 0 0lo0 16896 127 127.0.0.1 33917 0 33946 0 0lo0 16896 ::1 33917 0 33946 0 0

# pmtu display dst gw If pmtu refcnt redisc_t exp -------------------------------------------------------------------9.19.98.112 9.19.126.1 en0 1500 0 3 29.19.126.84 127.0.0.1 lo0 16896 6 29 09.19.126.114 9.19.126.84 en0 1500 1 2 0127.0.0.1 127.0.0.1 lo0 16896 2 29 0

# netstat -rnRouting tablesDestination Gateway Flags Refs Use If PMTU Exp Groups

Route tree for Protocol Family 2 (Internet):default 9.19.126.1 UG 0 2495 en0 - -9.19.126.0 9.19.126.84 UHSb 0 0 en0 - - =>9.19.126/24 9.19.126.84 U 3 50467 en0 - -9.19.126.84 127.0.0.1 UGHS 6 113 lo0 - -9.19.126.255 9.19.126.84 UHSb 0 1 en0 - -127/8 127.0.0.1 U 5 33806 lo0 - -

New pmtu command manages PMTU table, "C" flag is gone!PMTU and Exp columns unused and may be deleted in the future

©IBM 2005 - 10

ReferencesReferences

AIX V5.3 Product documentation (Information Center)http://publib.boulder.ibm.com/infocenter/pseries/index.jsp

AIX V5.3 Sys Mgmt Guide: Communications and Networkshttp://publib.boulder.ibm.com/infocenter/pseries/ topic/com.ibm.aix.doc/aixbman/commadmn/commadmn.pdf

AIX pSeries Advanced Tech Support website (IBM internal only)http://w3.ibm.com/support/americas/pseries/network.html

©IBM 2005 - 11