Capturing Complexity in Networked Systems Design: The Case for Improved Metrics
description
Transcript of Capturing Complexity in Networked Systems Design: The Case for Improved Metrics
Capturing Complexity in Networked
Systems Design:
The Case for Improved Metrics
Sylvia Ratnasamy
Intel Research
Motivation
Our community values “simple”, “clean” system designs
“The advantage of Chord is that it is less complicated…” – Chord, Sigcomm’01
“The complexity of the architectural design…” – Sprint, IEEE Network 2000
“A design for multicast that is simple…” – Perlman. Simple Multicast, IETF Draft.
“Complex routing algorithms may have difficulty scaling… simple floods are
sufficient” -- TinyOS, NSDI’04
“This paper proposed a simple and effective approach …” – SOSR, OSDI’04
“Thus, simplicity is our primary design requirement …” – BVR, NSDI’05
Motivation
Our community values “simple”, “clean” system designs
• But lacks metrics that rigorously capture this aesthetic
• Best practice: metrics borrowed from the theory community• e.g., total_messages, total_state
• Problem: at times incongruent with our notion of “simplicity”• e.g., flooding: inefficient but simple• e.g., dijkstra’s: O(n) state, but simple• e.g., leaderID vs. neighborID
This Paper
• Can we quantify design “simplicity”?– more rigorously compare-and-contrast design options– align design goals of algorithms, systems communities
• First stab: complexity metrics for the algorithmic component of a networked system
• Note– just one aspect to system complexity– intent: complement, not replace, existing metrics
Outline
• Strawman
• Simple evaluation scenarios
• Limitations and Future Directions
Strawman
Two observations:– much of system design centers around state
• what state is required?• how is it constructed/maintained?• how is it used?
– what distinguishes wide-area state:• derived from remote nodes dependencies are distributed • relayed via intermediate nodes more dependencies
Current metrics mostly treat all state as equal
Proposal
For a given piece of state s, measure the ensemble of distributed dependencies that yield s
• First cut: measure counting – akin to existing metrics: #msgs, #state– amenable to evaluation by simple examination, simulation
Value vs. Transport Dependencies
s is value dependent on x, s is transport dependent on next(B) state at A, R1, R2, R3.
vs = #pieces of state on which s is value dependent
tsx = #pieces of state relied on to transport x to s
A R1 R2 R3 B
s=x
next(B)
x=30°
next(B) next(B)next(B)
Counting Dependencies
A R1 R2 R3 B
s=x
next(B)
x=30°
next(B) next(B)
vx = txx = 0 vs = 1 ; tsx = 4
next(B)
Counting Dependencies
A R1 R2 R3 B
s=x x=30+y°
C R4
y=20°
vy = tyy = 0 vs = 2 tsx = 4
vx = 1 txy = 2
Note: value dependencies accumulate
Dependencies Complexity
single input: state s, derived from x
– complexity of s, cs = (vx + 1) tsx + cx
A R1 R2 R3 B
x=y+30°
C R4
y=20°
vy = tyy = 0 vs = 2 , tsx = 4vx = 1 txy = 2
s=x
cy = 0 cx = 1.2+0 = 2 cs = 2.4+2 = 10
single input: state s, derived from x
– complexity of s, cs = (vx + 1) tsx + cx
BA
x=0vs = d , tss-1 = 1vx = i tii-1 = 1
c = 0 cs = O(d2)
s=1 s=2 s=ds=i
…
cs = 1 cs = 3
Dependencies Complexity
multiple inputs: state s, derived from x1, x2, …, xm
two basic variants:
1. s = ALL (xi) cs = Σ ((vxi + 1) tsxi + cxi)
– if vxi = cxi = 0, tsxi = 1, then cs = m
2. s = ANY (xi) cs = 1/m Σ ((vxi + 1/m) tsxi + cxi)
– if vxi = cxi = 0, tsxi = 1, then cs = 1/m
Dependencies Complexity
multiple paths: state s derived from x, multiple paths from x to s
simple case: m disjoint paths, each incurring d transport dependencies
tsx = d/m, cs = vs tsx + cx
• if vx = cx = 0, then cs = d/m
Dependencies Complexity
Complexity: toy scenarios
scenario vs cs
s=x; s,x are 1 hop apart 1 O(1)
s=x; s,x are k hops apart 1 O(k)
s=distance(x); s,x k hops apart k O(k2)
s=ALL(x1, x2, …, xm) s, xi are 1 hop apart
m O(m)
s=ANY(x1, x2, …, xm)
s, xi are 1 hop apart1/m O(1/m)
s=x m distinct 1-hop paths from x to s
1 O(1/m)
Preliminary Evaluation (summary)
system vs cs croute
distance-vector O(d) O(d2) O(d3)
link-state O(1) O(d) O(d3)
compact [SPAA04]?? O(n) > O(nd) > O(nd2 )
tree-based O(n1/2) O(n) O(n3/2)
geo-routing O(1) O() O(n)virtual-geo [mobicom03] O(n) O(n3/2) O(n5/2)
Limitations, Future Directions
• scope: no validation of assumptions, correctness, quality, performance, etc.
• correlated inputs• discriminating transport dependencies• capturing absent dependencies• capturing robustness
– count “reverse” dependencies?
• role of time• evaluating the metric: future studies
Summary
• design simplicity: valued, but poorly measured
• question: is design complexity measurable?
• conjecture: capturing dependencies in state is key (w.r.t. algorithmic component of a system)
Thanks! Questions?
complexity of operations
• an operation acts on different pieces of state
• each piece of state has a complexity
• compute complexity of an operation by (appropriately) combining complexity of state it acts on
cs = 0 1 ... O((d-1)2) O(d2)
BA
x=0 s=1 s=2 s=ds=i
…
complexity of forwarding operation: cfwd = O(d3)
complexity of operations
• an operation acts on different pieces of state
• each piece of state has a complexity
• compute complexity of an operation by (appropriately) combining complexity of state it acts on
cs = k
cs = k
cs = k
cs = km
12
3
…fetch(obj)
complexity of fetch: cfetch = O(k/m)
Future Studies
• centralized solutions simpler?– e.g., RCP vs. DV
• designing for low dependency– e.g., soft state-based approaches
• layered vs. customized– e.g., PHTs [sigcomm’05] vs. Mercury [sigcomm’04]
• different routing options– mesh (RON, ESM), DHT-inspired (VRR/ROFL),
centralized (RCP), compact routing-based, landmark routing, coordinate-based, …