Not All Microseconds are Equal:Fine-Grained Per-Flow Measure-
ments with Reference Latency Inter-polation
Myungjin Lee†, Nick Duffield‡, Ramana Rao Kompella†
†Purdue University, ‡AT&T Labs–Research
2
Low-latency applications
Several new types of applications require ex-tremely low end-to-end latency Algorithmic trading applications in financial data
center networks High performance computing applications in data
center networks Storage applications
Low latency cut-through switches Arista 7100 series Woven EFX 1000 series
3
… … … …
ToR S/W
Edge Router
Core Router
…Need for high-fidelity measurements
At every router, high-fidelity measurements are critical to localize root causes
Once root cause localized, operators can fix by rerouting traffic, upgrade links or perform detailed diagnosis
Which router causes the prob-
lem??
1ms
Router
Measurement within a router is
necessary
4
Measurement solutions today
SNMP and NetFlow No latency measurements
Active probes Typically end-to-end, do not localize the root cause
Expensive high-fidelity measurement box Corvil boxes (£ 90,000): used by London Stock Exchange Cannot place these boxes ubiquitously
Lossy Difference Aggregator (LDA) [Kompella, SIG-COMM’09] Provides average latency and variance at high-fidelity
within a switch Provides a good start but may not be sufficient to diag-
nose flow-specific problems
5
Motivation for per-flow measurements
Key observation: Significant amount of difference in average latencies across flows at a router
Dela
y
Time
S/W
…
Queue
Average la-tency
Measurement period
Large de-lay
Small de-lay
6
Outline of the rest of talk
Measurement model
Alternative approaches
Intuition behind our approach: Delay locality
Our architecture: Reference Latency Interpola-tion (RLI)
Evaluation
7
Measurement model
Assumption: Time synchronization between router interfaces Constraint: Cannot modify regular packets to carry time-
stamps Intrusive changes to the routing forwarding path Extra bandwidth consumption up to 10% capacity
Router
Ingress I
Egress E
8
Naïve approach
For each flow key, Store timestamps for each packet at I and E After a flow stops sending, I sends the packet timestamps
to E E computes individual packet delays E aggregates average latency, variance, etc for each flow
Problem: High communication costs At 10Gbps, few million packets per second Sampling reduces communication, but also reduces accu-
racy
Ingress I Egress E
10
− =2023
2730
+151
318
−=
2232Avg. delay = 22/2
= 11Avg. delay = 32/2 = 16
−+ −
9
A (naïve) extension of LDA
Maintaining LDAs with many counters for flows of interest
Problem: (Potentially) high communication costs Proportional to the number of flows
Ingress I Egress E
LDA
LDA
LDA
LDA
LDA
LDA
28
15
2
1
Packet
count
Sum of time-
stamps
…
Coordina-tion
Per-flow la-tency
10
Key observation: Delay locality
LocaTrue mean delay = W(D1 + WD2 + WD3) / 3
Localized mean delay = (WD1 + WD2 + WD3) / 3
WD1 WD3WD2
How close is localized mean delay to
true mean delay as window size varies?
Dela
y
Time
D1
D2D3
11
Key observation: Delay locality
True Mean delay per key / ms
Loca
l m
ean
dela
y p
er
key /
ms
Global Mean
0.1ms: RMSRE=0.054
10ms: RMSRE=0.16
1s: RMSRE=1.72
Data sets from real router and synthetic queueing model
12
Exploiting delay locality
Reference packets are injected regularly at the ingress I Special packets carrying ingress timestamp Provide some reference delay samples Used to approximate the latencies of regular packets
Dela
y
Time
ReferencePacket
IngressTimestamp
13
RLI architecture
Component 1: Reference Packet generator Injects reference packets regularly
Component 2: Latency Estimator Estimates packet latencies and updates per-flow statis-
tics Estimates directly at the egress with no extra state main-
tained at ingress side (reduces storage and communica-tion overheads)
Egress E
Ingress I
1) ReferencePacket
Generator
2) LatencyEstimator
123 123
RL
IngressTimestamp
14
Component 1: Reference packet generator
Question: When to inject a reference packet ?
Idea 1: 1-in-n: Inject one reference packet every n packets Problem: low accuracy under low utilization
Idea 2: 1-in-τ: Inject one reference packet every τ seconds Problem: bad in case where short-term delay variance is
high Our approach: Dynamic injection based on utilization
High utilization low injection rate Low utilization high injection rate Adaptive scheme works better than fixed rate schemes
15
Component 2: Latency estimator
Question 1: How to estimate latencies using reference packets
Solution: Different estimators possible Use only the delay of a left reference packet (RLI-L) Use linear interpolation of left and right reference packets (RLI) Other non-linear estimators possible (e.g., shrinkage)
LInterpolated
delay
Dela
y
Time
Error indelay estimate
RegularPacket
ReferencePacket
Linear interpolationline
Arrival time is known
Arrival time and delay are
known
Estimateddelay
Error indelay estimate
R
16
Component 2: Latency estimatorFlowkey C1 C2 C3
8 11 39
2 3 6
Interpolation buffer
Estimate
10
20
80
3 4 7
Avg. latency = C2 / C1
RL
Right Reference Packet arrived
When a flow is ex-
ported
Question 2: How to compute per-flow latency statistics Solution: Maintain 3 counters per flow at the egress side
C1: Number of packets C2: Sum of packet delays C3: Sum of squares of packet delays (for estimating variance) To minimize state, can use any flow selection strategy to main-
tain counters for only a subset of flows
Flow Key4 51Delay
Square of de-
lay
16
25
1
Update
Any flow se-lection
strategy
Up-date
Selection
17
Experimental environment
Data sets No public data center traces with timestamps Real router traces with synthetic workloads: WISC Real backbone traces with synthetic queueing: CHIC
and SANJ
Simulation tool: Open source NetFlow software – YAF Supports reference packet injection mechanism Simulates a queueing model with RED active queue
management policy
Experiments with different link utilizations
18
Accuracy of RLI under high link utilization
Relative error
CD
F
Median relative erroris 10-12%
19
Comparison with other solutions
Utilization
Avera
ge r
ela
tive e
rror
Packet sampling rate = 0.1%
1-2 orders of magnitude dif-
ference
20
Overhead of RLI
Bandwidth overhead is low less than 0.2% of link capacity
Impact to packet loss is small Packet loss difference with and without RLI is at
most 0.001% at around 80% utilization
21
Summary
A scalable architecture to obtain high-fidelity per-flow latency measurements between router interfaces
Achieves a median relative error of 10-12%
Shows 1-2 orders of magnitude lower relative error compared to existing solutions
Measurements are obtained directly at the egress side
Future work: Per-packet diagnosis
22
Thank you! Questions?
23
Backup
24
Comparison with other solutions
Relative error
CD
F
25
Bandwidth overhead
Utilization
Band
wid
th c
onsu
mpti
on
26
Interference with regular traffic
Per-flow delay interference (seconds)
Cum
ula
tive f
ract
ion
27
Impact to packet lossesLo
ss r
ate
diff
ere
nce
Utilization
Top Related