IBM Research GmbH, Zürich Research Laboratory R 3 C 2 : Reactive Route & Rate Control for CEE Mitch...

IBM Research GmbH, Zürich Research Laboratory

R3C2: Reactive Route & Rate Control for CEE

Mitch Gusat, Daniel Crisan, Cyriel Minkenberg, and Casimer DeCusatis

IBM Research GmbH, ZRL

2

Outline

• Introduction• Background• R3C2

• Evaluation• Conclusions


3

Introduction & Motivation• Context: Congestion management in L2

datacenter networks

• Existing congestion mgmt schemes for CEE– Rate-only: Quantized Congestion Notification

• Rate/window reduction don’t benefit trading, HPC, BA apps

– Route-only: Switch Adaptive Routing• Based on QCN’s load sensor• Exploits path diversity

• Proposal: R3C2

– Dual Route & Rate control– Source-driven


4

Outline




5

QCN 802.1Qau

switc

h

NIC RL

switc

h

switc

h

switc

h

NIC

NICendnode

endnode

CNM

CNM

endnode


6

R3C2 Concept

Take advantage of CNMs at the source for

adaptive load-balancing

• Congestion Point issues CNMs– Where is the hotspot?– How severe is the hotspot?

• Source receives the CNMs– Identifies the most severe hotspots– Reroutes traffic around the hotspots– Splits flows and rate-limits subflows


7

Source Routing in CEE: VLAN

• Plain Ethernet is not source-routed

• Solution: VLAN – One tree per VLAN

• Source– Set VLAN# at

injection path selection

P0 P1 P2 P3 P4 P5 P6 P7

S1_0 S1_1 S1_2 S1_3

S2_0 S2_1 S2_2 S2_3

S3_0 S3_1 S3_2 S3_3

VLAN0

VLAN1

VLAN2

VLAN3


8

R3C2 Algorithm

P0 P7S1_0 S1_3

S2_0

S2_1

S2_2

S2_3

S3_0

S3_1

S3_2

S3_3

• No overload: Deterministic single path• Congestion: Activate additional paths• Path activation: avoid hotspots• Use RL along each path


9

R3C2 Reaction Point

• Packet assigned the VLAN# of the 1st eligible Rate Limiter

RLflow 1

RLflow 2

RLflow 3

Packets from upper layers for a given destination D

TX queue

Reaction Point for D

To network

Assign VLAN1

Assign VLAN2

Assign VLAN3

MU

X


10

Outline




11

Evaluation Methodology• Venus + Dimemas simulator

• Traffic– Synthetic: permutations + hotspot– HPC Traces:

• NAS: BT, CG, FT, IS, MG• WRF, NAMD, Liso, Airbus

• Model parameters– 10Gbps CEE with MTU = 1500B– QCN and PFC: 802 DCB settings

• Topology: 2-ary n-tree


12

Permutation Traffic

0.2

0.4

0.6

0.8

1

Switch AR

Switch AR w

ith RL

Random

Random with

RL

Deterministi

c

Deterministi

c with

RLR3C2

Rela

tive

tota

l Tpu

t

Bit reverse Bit rotation Bit shuffle Bit transpose


13

Hotspot Traffic Scenario

P0 P7S1_0 S1_3

S2_0

S2_1

S2_2

S2_3

S3_0

S3_1

S3_2

S3_3

C=25%

C=10%

C=10%

C=50%

Single 95% flow


14

Hotspot Traffic

0

200

400

600

800

1000

1200

1400

0 0.02 0.04 0.06 0.08 0.1Time [s]

Tput

[MB/

s]

Random Switch AR R3C2


15

HPC Traces: Hotspot

00.5

11.5

22.5

33.5

44.5

R3C2

Switch AR

Switch AR w

ith RL

Deterministi

c

Random

Random with

RL

Deterministi

c with

RL

Hashed

Hashed w

ith RL

Rela

tive

slow

dow

n

Min Average Max


16

Conclusions

• Introduced R3C2

– Source-driven adaptive routing scheme– QCN and VLAN are key

• Dual Route & Rate control– Improved stability and performance

• Performance benefits– 80% over Deterministic– 40% over Random


17

Backup


18

Routing SchemesDeterministic

0

0.25

0.5

0.75

1

1 2 3 4Path

Prob

abili

ty

Random

0

0.25

0.5

0.75

1

1 2 3 4Path

Prob

abili

ty

Switch AR

0

0.25

0.5

0.75

1

1 2 3 4Path

Prob

abili

ty


19


0

0.25

0.5

0.75

1

1 2 3 4Path

Prob

abili

ty

Random

0

0.25

0.5

0.75

1

1 2 3 4Path

Prob

abili

ty

Switch AR

0

0.25

0.5

0.75

1

1 2 3 4Path

Prob

abili

ty


20


0

0.25

0.5

0.75

1

1 2 3 4Path

Prob

abili

ty

Random

0

0.25

0.5

0.75

1

1 2 3 4Path

Prob

abili

ty

Switch AR

0

0.25

0.5

0.75

1

1 2 3 4Path

Prob

abili

ty


21


0

0.25

0.5

0.75

1

1 2 3 4Path

Prob

abili

ty

Random

0

0.25

0.5

0.75

1

1 2 3 4Path

Prob

abili

ty

Switch AR

0

0.25

0.5

0.75

1

1 2 3 4Path

Prob

abili

ty

R3C2

0

0.25

0.5

0.75

1

1 2 3 4Path

Prob

abili

ty


22


0

0.25

0.5

0.75

1

1 2 3 4Path

Prob

abili

ty

Random

0

0.25

0.5

0.75

1

1 2 3 4Path

Prob

abili

ty

Switch AR

0

0.25

0.5

0.75

1

1 2 3 4Path

Prob

abili

ty

R3C2

0

0.25

0.5

0.75

1

1 2 3 4Path

Prob

abili

ty


23


0

0.25

0.5

0.75

1

1 2 3 4Path

Prob

abili

ty

Random

0

0.25

0.5

0.75

1

1 2 3 4Path

Prob

abili

ty

Switch AR

0

0.25

0.5

0.75

1

1 2 3 4Path

Prob

abili

ty

R3C2

0

0.25

0.5

0.75

1

1 2 3 4Path

Prob

abili

ty


24


0

0.25

0.5

0.75

1

1 2 3 4Path

Prob

abili

ty

Random

0

0.25

0.5

0.75

1

1 2 3 4Path

Prob

abili

ty

Switch AR

0

0.25

0.5

0.75

1

1 2 3 4Path

Prob

abili

ty

R3C2

0

0.25

0.5

0.75

1

1 2 3 4Path

Prob

abili

ty


25

HPC Traces: No Hotspot

00.5

11.5

22.5

33.5

44.5

55.5

66.5

Random

Random with

RLR3C2

Switch AR

Switch AR w

ith RL

Deterministi

c

Hashed

Deterministi

c with

RL

Hashed w

ith RL

Rela

tive

slow

dow

n

Min Average Max


26

Hotspot Traffic (1)

0

200

400

600

800

1000

1200

1400

0 0.02 0.04 0.06 0.08 0.1

Time [s]

Tput

[MB/

s]

Random Random with RL


27

Hotspot Traffic (2)

0

200

400

600

800

1000

1200

1400

0 0.02 0.04 0.06 0.08 0.1

Time [s]

Tput

[MB/

s]

Switch AR Switch AR with RL


28

Hotspot Traffic (3)

0

200

400

600

800

1000

1200

1400

0 0.02 0.04 0.06 0.08 0.1

Time [s]

Tput

[MB/

s]

Deterministic R3C2

IBM Research GmbH, Zürich Research Laboratory R 3 C 2 : Reactive Route & Rate Control for CEE Mitch...

Documents

Transcript of IBM Research GmbH, Zürich Research Laboratory R 3 C 2 : Reactive Route & Rate Control for CEE Mitch...