1 Saino Hashing On Broken Assumptions - NANOG Archive...Link aggregation switch switch...
Transcript of 1 Saino Hashing On Broken Assumptions - NANOG Archive...Link aggregation switch switch...
![Page 1: 1 Saino Hashing On Broken Assumptions - NANOG Archive...Link aggregation switch switch physical*links logical*link Combine multiple physical links between network devices into one](https://reader034.fdocuments.us/reader034/viewer/2022052616/60a03a2754395c6948290543/html5/thumbnails/1.jpg)
Name of Presentation
Hashing on broken assumptionsLorenzo Saino (@lorenzosaino)Fastly
![Page 2: 1 Saino Hashing On Broken Assumptions - NANOG Archive...Link aggregation switch switch physical*links logical*link Combine multiple physical links between network devices into one](https://reader034.fdocuments.us/reader034/viewer/2022052616/60a03a2754395c6948290543/html5/thumbnails/2.jpg)
Problem: Spreading traffic across multiple links, paths, hosts
Solutions: • Link Aggregation • Equal Cost Multipath (ECMP)
![Page 3: 1 Saino Hashing On Broken Assumptions - NANOG Archive...Link aggregation switch switch physical*links logical*link Combine multiple physical links between network devices into one](https://reader034.fdocuments.us/reader034/viewer/2022052616/60a03a2754395c6948290543/html5/thumbnails/3.jpg)
Link aggregation
switch switch
physical links logical link
Combine multiple physical links between network devices into one logical link
![Page 4: 1 Saino Hashing On Broken Assumptions - NANOG Archive...Link aggregation switch switch physical*links logical*link Combine multiple physical links between network devices into one](https://reader034.fdocuments.us/reader034/viewer/2022052616/60a03a2754395c6948290543/html5/thumbnails/4.jpg)
Equal Cost Multipath (ECMP)
switch
host host host
Balance traffic across paths Balance traffic across hosts
switch
switch
switch switch
![Page 5: 1 Saino Hashing On Broken Assumptions - NANOG Archive...Link aggregation switch switch physical*links logical*link Combine multiple physical links between network devices into one](https://reader034.fdocuments.us/reader034/viewer/2022052616/60a03a2754395c6948290543/html5/thumbnails/5.jpg)
Requirements
Load balanceTraffic must be uniformly spread across next-‐hops
Stateless-‐but-‐sticky path pinning All packets of a flow must take the same path
![Page 6: 1 Saino Hashing On Broken Assumptions - NANOG Archive...Link aggregation switch switch physical*links logical*link Combine multiple physical links between network devices into one](https://reader034.fdocuments.us/reader034/viewer/2022052616/60a03a2754395c6948290543/html5/thumbnails/6.jpg)
Load imbalance
Load imbalance reduces system capacity
![Page 7: 1 Saino Hashing On Broken Assumptions - NANOG Archive...Link aggregation switch switch physical*links logical*link Combine multiple physical links between network devices into one](https://reader034.fdocuments.us/reader034/viewer/2022052616/60a03a2754395c6948290543/html5/thumbnails/7.jpg)
Load imbalance
Load imbalance reduces system capacity
Perfect load balance
![Page 8: 1 Saino Hashing On Broken Assumptions - NANOG Archive...Link aggregation switch switch physical*links logical*link Combine multiple physical links between network devices into one](https://reader034.fdocuments.us/reader034/viewer/2022052616/60a03a2754395c6948290543/html5/thumbnails/8.jpg)
Load imbalance
Load imbalance reduces system capacity
All resources fully utilized
![Page 9: 1 Saino Hashing On Broken Assumptions - NANOG Archive...Link aggregation switch switch physical*links logical*link Combine multiple physical links between network devices into one](https://reader034.fdocuments.us/reader034/viewer/2022052616/60a03a2754395c6948290543/html5/thumbnails/9.jpg)
Load imbalance
Load imbalance reduces system capacity
Load imbalance
![Page 10: 1 Saino Hashing On Broken Assumptions - NANOG Archive...Link aggregation switch switch physical*links logical*link Combine multiple physical links between network devices into one](https://reader034.fdocuments.us/reader034/viewer/2022052616/60a03a2754395c6948290543/html5/thumbnails/10.jpg)
Load imbalance
Load imbalance reduces system capacityUnused capacity
Cannot take any additional load
![Page 11: 1 Saino Hashing On Broken Assumptions - NANOG Archive...Link aggregation switch switch physical*links logical*link Combine multiple physical links between network devices into one](https://reader034.fdocuments.us/reader034/viewer/2022052616/60a03a2754395c6948290543/html5/thumbnails/11.jpg)
Quantifying impact of load imbalance
Lmax
Lavg
Umax
=
✓Lmax
Lavg
◆�1
=Lavg
Lmax
Umax
2 (0, 1]
Lmax
Lavg
= [1,+1)
load of most loaded resource average load
max attainable utilization
Load imbalance: Max attainable utilization:
![Page 12: 1 Saino Hashing On Broken Assumptions - NANOG Archive...Link aggregation switch switch physical*links logical*link Combine multiple physical links between network devices into one](https://reader034.fdocuments.us/reader034/viewer/2022052616/60a03a2754395c6948290543/html5/thumbnails/12.jpg)
Quantifying impact of load imbalance
1.0 1.5 2.0 2.5 3.0Lmax/Lavg
0.0
0.2
0.4
0.6
0.8
1.0
1.2U
max
![Page 13: 1 Saino Hashing On Broken Assumptions - NANOG Archive...Link aggregation switch switch physical*links logical*link Combine multiple physical links between network devices into one](https://reader034.fdocuments.us/reader034/viewer/2022052616/60a03a2754395c6948290543/html5/thumbnails/13.jpg)
Quantifying impact of load imbalance
1.0 1.5 2.0 2.5 3.0Lmax/Lavg
0.0
0.2
0.4
0.6
0.8
1.0
1.2U
max • Perfect balance
• Full utilization
![Page 14: 1 Saino Hashing On Broken Assumptions - NANOG Archive...Link aggregation switch switch physical*links logical*link Combine multiple physical links between network devices into one](https://reader034.fdocuments.us/reader034/viewer/2022052616/60a03a2754395c6948290543/html5/thumbnails/14.jpg)
Quantifying impact of load imbalance
1.0 1.5 2.0 2.5 3.0Lmax/Lavg
0.0
0.2
0.4
0.6
0.8
1.0
1.2U
max
• Most loaded resource 1.5x average
X
• 33.3% reduction of capacity
![Page 15: 1 Saino Hashing On Broken Assumptions - NANOG Archive...Link aggregation switch switch physical*links logical*link Combine multiple physical links between network devices into one](https://reader034.fdocuments.us/reader034/viewer/2022052616/60a03a2754395c6948290543/html5/thumbnails/15.jpg)
What happens without path pinning?
Same endpoints, different paths: • Out-‐of-‐order packets • Frequent drops of TCP congestion window (CWND) • Poor throughput performance
Different endpoints: • TCP resets
![Page 16: 1 Saino Hashing On Broken Assumptions - NANOG Archive...Link aggregation switch switch physical*links logical*link Combine multiple physical links between network devices into one](https://reader034.fdocuments.us/reader034/viewer/2022052616/60a03a2754395c6948290543/html5/thumbnails/16.jpg)
TCP resets
hostSYN
SYN/ACK
ACK
RST
router
host
host
![Page 17: 1 Saino Hashing On Broken Assumptions - NANOG Archive...Link aggregation switch switch physical*links logical*link Combine multiple physical links between network devices into one](https://reader034.fdocuments.us/reader034/viewer/2022052616/60a03a2754395c6948290543/html5/thumbnails/17.jpg)
Solution: Flow-‐level hashing
Requirements: • Load balance • Path pinning
![Page 18: 1 Saino Hashing On Broken Assumptions - NANOG Archive...Link aggregation switch switch physical*links logical*link Combine multiple physical links between network devices into one](https://reader034.fdocuments.us/reader034/viewer/2022052616/60a03a2754395c6948290543/html5/thumbnails/18.jpg)
Flow-level hashing
• src IP addr• dst IP addr• protocol• src port• dst port
next-hop
hash function
packet
readfive tuple
![Page 19: 1 Saino Hashing On Broken Assumptions - NANOG Archive...Link aggregation switch switch physical*links logical*link Combine multiple physical links between network devices into one](https://reader034.fdocuments.us/reader034/viewer/2022052616/60a03a2754395c6948290543/html5/thumbnails/19.jpg)
Assumptions
Load balance Hashing uniformly spread traffic across next-‐hops
Path pinning Hashing pins packets of a flow to the same path
![Page 20: 1 Saino Hashing On Broken Assumptions - NANOG Archive...Link aggregation switch switch physical*links logical*link Combine multiple physical links between network devices into one](https://reader034.fdocuments.us/reader034/viewer/2022052616/60a03a2754395c6948290543/html5/thumbnails/20.jpg)
Do these assumptions hold?
![Page 21: 1 Saino Hashing On Broken Assumptions - NANOG Archive...Link aggregation switch switch physical*links logical*link Combine multiple physical links between network devices into one](https://reader034.fdocuments.us/reader034/viewer/2022052616/60a03a2754395c6948290543/html5/thumbnails/21.jpg)
Assumptions
Load balance Hashing uniformly spread traffic across next-‐hops
Path pinning Hashing pins packets of a flow to the same path
![Page 22: 1 Saino Hashing On Broken Assumptions - NANOG Archive...Link aggregation switch switch physical*links logical*link Combine multiple physical links between network devices into one](https://reader034.fdocuments.us/reader034/viewer/2022052616/60a03a2754395c6948290543/html5/thumbnails/22.jpg)
Hashing quality
switch
Two switch models: • Switch A • Switch B
.
.
.
256 nexthops2^16 five-‐tuple combinations
![Page 23: 1 Saino Hashing On Broken Assumptions - NANOG Archive...Link aggregation switch switch physical*links logical*link Combine multiple physical links between network devices into one](https://reader034.fdocuments.us/reader034/viewer/2022052616/60a03a2754395c6948290543/html5/thumbnails/23.jpg)
0 50 100 150 200 250Nexthop rank
0.0
0.5
1.0
1.5
2.0
L/L
avg
0 50 100 150 200 250Nexthop rank
0.0
0.5
1.0
1.5
2.0
L/L
avg
0 50 100 150 200 250Nexthop rank
0.0
0.5
1.0
1.5
2.0
L/L
avg
Switch A
MeasuredPerfect hashing
![Page 24: 1 Saino Hashing On Broken Assumptions - NANOG Archive...Link aggregation switch switch physical*links logical*link Combine multiple physical links between network devices into one](https://reader034.fdocuments.us/reader034/viewer/2022052616/60a03a2754395c6948290543/html5/thumbnails/24.jpg)
0 50 100 150 200 250Nexthop index
0.0
0.5
1.0
1.5
2.0
L/L
avg
0 50 100 150 200 250Nexthop index
0.0
0.5
1.0
1.5
2.0
L/L
avg
Switch B
Perfect hashing
Measured
1.5x
6x
![Page 25: 1 Saino Hashing On Broken Assumptions - NANOG Archive...Link aggregation switch switch physical*links logical*link Combine multiple physical links between network devices into one](https://reader034.fdocuments.us/reader034/viewer/2022052616/60a03a2754395c6948290543/html5/thumbnails/25.jpg)
Switch B
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 1617 18 19 20 21 22 23 24 25 26 27 28 29 30 31 3233 34 35 36 37 38 49 40 41 42 43 44 45 46 47 4849 50 51 52 53 54 55 56 57 58 59 60 61 62 63 6465 66 67 68 69 70 71 72 73 74 75 76 77 78 79 8081 82 83 84 85 86 87 88 89 90 91 92 93 94 95 9697 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112
113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128… … … … … … … … … … … … … … … …
Vendor claims supporting an arbitrary number of next-‐hops [1, 256]
![Page 26: 1 Saino Hashing On Broken Assumptions - NANOG Archive...Link aggregation switch switch physical*links logical*link Combine multiple physical links between network devices into one](https://reader034.fdocuments.us/reader034/viewer/2022052616/60a03a2754395c6948290543/html5/thumbnails/26.jpg)
Switch B
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 1618 20 22 24 26 28 30 3234 36 38 40 44 48
52 56 60 6472 8088 96104 112120 128
…
Only a subset of next-‐hops are actually supported
![Page 27: 1 Saino Hashing On Broken Assumptions - NANOG Archive...Link aggregation switch switch physical*links logical*link Combine multiple physical links between network devices into one](https://reader034.fdocuments.us/reader034/viewer/2022052616/60a03a2754395c6948290543/html5/thumbnails/27.jpg)
Switch B
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 1618 20 22 24 26 28 30 3234 36 38 40 44 48
52 56 60 6472 8088 96104 112120 126 128
…
Only a subset of next-‐hops are actually supported
![Page 28: 1 Saino Hashing On Broken Assumptions - NANOG Archive...Link aggregation switch switch physical*links logical*link Combine multiple physical links between network devices into one](https://reader034.fdocuments.us/reader034/viewer/2022052616/60a03a2754395c6948290543/html5/thumbnails/28.jpg)
Switch B
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 1618 20 22 24 26 28 30 3234 36 38 40 44 48
52 56 60 6472 8088 96104 112120 X X X X X X 128
…
Only a subset of next-‐hops are actually supported
6 next-hops don’t get any traffic
![Page 29: 1 Saino Hashing On Broken Assumptions - NANOG Archive...Link aggregation switch switch physical*links logical*link Combine multiple physical links between network devices into one](https://reader034.fdocuments.us/reader034/viewer/2022052616/60a03a2754395c6948290543/html5/thumbnails/29.jpg)
Assumptions
Load balance Hashing uniformly spread traffic across next-‐hops
Path pinning Hashing pins packets of a flow to the same path
![Page 30: 1 Saino Hashing On Broken Assumptions - NANOG Archive...Link aggregation switch switch physical*links logical*link Combine multiple physical links between network devices into one](https://reader034.fdocuments.us/reader034/viewer/2022052616/60a03a2754395c6948290543/html5/thumbnails/30.jpg)
Hashing on IPv4 TOS field 0 1 2 3 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ |Version| IHL |Type of Service| Total Length | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | Identification |Flags| Fragment Offset | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | Time to Live | Protocol | Header Checksum | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | Source Address | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | Destination Address | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | Options | Padding | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
![Page 31: 1 Saino Hashing On Broken Assumptions - NANOG Archive...Link aggregation switch switch physical*links logical*link Combine multiple physical links between network devices into one](https://reader034.fdocuments.us/reader034/viewer/2022052616/60a03a2754395c6948290543/html5/thumbnails/31.jpg)
Hashing on IPv4 TOS field 0 1 2 3 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ |Version| IHL |Type of Service| Total Length | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | Identification |Flags| Fragment Offset | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | Time to Live | Protocol | Header Checksum | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | Source Address | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | Destination Address | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | Options | Padding | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
![Page 32: 1 Saino Hashing On Broken Assumptions - NANOG Archive...Link aggregation switch switch physical*links logical*link Combine multiple physical links between network devices into one](https://reader034.fdocuments.us/reader034/viewer/2022052616/60a03a2754395c6948290543/html5/thumbnails/32.jpg)
Hashing on IPv4 TOS field 0 1 2 3 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ |Version| IHL |Type of Service| Total Length | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | Identification |Flags| Fragment Offset | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | Time to Live | Protocol | Header Checksum | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | Source Address | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | Destination Address | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | Options | Padding | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
![Page 33: 1 Saino Hashing On Broken Assumptions - NANOG Archive...Link aggregation switch switch physical*links logical*link Combine multiple physical links between network devices into one](https://reader034.fdocuments.us/reader034/viewer/2022052616/60a03a2754395c6948290543/html5/thumbnails/33.jpg)
Hashing on IPv4 TOS field
RFC 1812 -‐ Requirements for IP Version 4 Routers explicitly permits to involve the second-‐to-‐last bit of the TOS/DS octet in routing decisions
RFC 2474 -‐ Definition of the Differentiated Services Field deprecates the IPv4 Type of Service field redefines it as the Differentiated Services field
RFC 3168 -‐ The Addition of Explicit Congestion Notification (ECN) to IP reserves the last two bits of the DS octet for ECN
![Page 34: 1 Saino Hashing On Broken Assumptions - NANOG Archive...Link aggregation switch switch physical*links logical*link Combine multiple physical links between network devices into one](https://reader034.fdocuments.us/reader034/viewer/2022052616/60a03a2754395c6948290543/html5/thumbnails/34.jpg)
Hashing on IPv4 TOS field
host
host
host
Scenario • Hosts are ECN capable • Router uses IPv4 TOS for hash computation (RFC 1812)
TCP handshake: • Hosts negotiate ECN support • ECN-‐capable bits unset
Flow data: • ECN-‐capable bits set
router
TCP handshake flow data
![Page 35: 1 Saino Hashing On Broken Assumptions - NANOG Archive...Link aggregation switch switch physical*links logical*link Combine multiple physical links between network devices into one](https://reader034.fdocuments.us/reader034/viewer/2022052616/60a03a2754395c6948290543/html5/thumbnails/35.jpg)
IPv6 flow label rewrite
middlebox
host
host
host switch
uses IPv6 flow label for hash computation
x, x!= 0 y
y
zx, x!= 0
zif flow_label != 0: flow_label = rand()
forbidden by RFC 6437 allowed by RFC 6437
![Page 36: 1 Saino Hashing On Broken Assumptions - NANOG Archive...Link aggregation switch switch physical*links logical*link Combine multiple physical links between network devices into one](https://reader034.fdocuments.us/reader034/viewer/2022052616/60a03a2754395c6948290543/html5/thumbnails/36.jpg)
SYN proxies
SYN proxy switch
switch
host
host
Switches: • use ingress interface for hash computation, or • use different hash function seeds
TCP handshake flow data
host
![Page 37: 1 Saino Hashing On Broken Assumptions - NANOG Archive...Link aggregation switch switch physical*links logical*link Combine multiple physical links between network devices into one](https://reader034.fdocuments.us/reader034/viewer/2022052616/60a03a2754395c6948290543/html5/thumbnails/37.jpg)
Conclusions
Path pinning Hashing on fields other than five tuples breaks ECMP • Ingress port • IPv4 TOS • IPv6 flow label
Load balancing There are devices that do not hash traffic uniformly
![Page 38: 1 Saino Hashing On Broken Assumptions - NANOG Archive...Link aggregation switch switch physical*links logical*link Combine multiple physical links between network devices into one](https://reader034.fdocuments.us/reader034/viewer/2022052616/60a03a2754395c6948290543/html5/thumbnails/38.jpg)
Recommendations
Vendors: • Disable hashing inputs other than five-‐tuple by default • Make hash input fields configurable • Make hash seed configurable
Operators: • Ensure that your network devices hash flows uniformlyor that could cost you money
• Disable additional inputs if you do not need extra entropy
![Page 39: 1 Saino Hashing On Broken Assumptions - NANOG Archive...Link aggregation switch switch physical*links logical*link Combine multiple physical links between network devices into one](https://reader034.fdocuments.us/reader034/viewer/2022052616/60a03a2754395c6948290543/html5/thumbnails/39.jpg)
FIN