Communication Complexity Lower Bound Techniques for...
Transcript of Communication Complexity Lower Bound Techniques for...
1-1
Lower Bound Techniques for MultipartyCommunication Complexity
Qin Zhang
Indiana University Bloomington
Based on works with Jeff Phillips, Elad Verbin and David Woodruff
2-1
The multiparty number-in-hand (NIH) comm.
x1 = 010011 x2 = 111011
x3 = 111111
xk = 100011
They want to jointly compute f(x1, x2, . . . , xk)
Goal: minimize total bits of communication
– A model for proving lower bounds
2-2
The multiparty number-in-hand (NIH) comm.
x1 = 010011 x2 = 111011
x3 = 111111
xk = 100011
They want to jointly compute f(x1, x2, . . . , xk)
Message-Passing (MP):If x1 talks to x2, otherscannot hear.
Blackboard (BB): Onespeaks, everyone hears.
Goal: minimize total bits of communication
– A model for proving lower bounds
2-3
The multiparty number-in-hand (NIH) comm.
x1 = 010011 x2 = 111011
x3 = 111111
xk = 100011
They want to jointly compute f(x1, x2, . . . , xk)
Message-Passing (MP):If x1 talks to x2, otherscannot hear.
Blackboard (BB): Onespeaks, everyone hears.
· · ·S1 S2 S3 Sk
C=Coordinator
Goal: minimize total bits of communication
– A model for proving lower bounds
3-1
Previously, for multiparty NIH comm. model
Well studied? YES and NO.
3-2
Previously, for multiparty NIH comm. model
Well studied? YES and NO.
Quite a few papers in BB model:
[Alon et al. ’96], [Bar-Yossef et al. ’04], . . .
3-3
Previously, for multiparty NIH comm. model
Well studied? YES and NO.
Quite a few papers in BB model:
But, Almost nothing in MP model before 2012.
[Alon et al. ’96], [Bar-Yossef et al. ’04], . . .
3-4
Previously, for multiparty NIH comm. model
Well studied? YES and NO.
Quite a few papers in BB model:
But, Almost nothing in MP model before 2012.
• Too easy?
[Alon et al. ’96], [Bar-Yossef et al. ’04], . . .
3-5
Previously, for multiparty NIH comm. model
Well studied? YES and NO.
Quite a few papers in BB model:
But, Almost nothing in MP model before 2012.
• Too easy?
• Lack of motivations?
[Alon et al. ’96], [Bar-Yossef et al. ’04], . . .
4-1
Too easy?
Could be more difficult than BB, since in MP each playerobserves only a piece of protocol transcript.
4-2
Too easy?
Could be more difficult than BB, since in MP each playerobserves only a piece of protocol transcript.
We do not want to analyze a k-dimensional matrix directly.
0 1
0
1
0 0
01
AB A communication matrix of the
2-party AND function
4-3
Too easy?
Could be more difficult than BB, since in MP each playerobserves only a piece of protocol transcript.
We do not want to analyze a k-dimensional matrix directly.
0 1
0
1
0 0
01
AB A communication matrix of the
2-party AND function
Even for the 2-party Gap-Hamming problem, where Alicehas a n-bit vector and Bob has a n-bit vector, and they wantto compute if the hamming distance of these two vectorsis bigger than n/2 +
√n or less than n/2 −
√n, it tooks
researchers in this area about 10 years and dozens of papers(IW03, Woodruff04, . . ., Sherstov12) to fully understand its2-dim communication matrix!
5-1
Lack of motivations?
(Continous) Distributed Monitoringcommunication → energy, bandwidth, . . .
Data in the cloud
Sensor networksNetwork routers
6-1
Lack of motivations? (cont.)
Data Streamscommunication → space
Memory
CPU
6-2
Lack of motivations? (cont.)
Data Streamscommunication → space
Memory
CPUP1 P2 Pk
7-1
The BSP model.E.g., Apache Giraph, Pregel.
Input
MapShuffle
Reduce
Output
The MapReduce model.E.g., Hadoop, Google MapReduce.
Distributed Computation for Big Datacommunication → bandwidth, . . .
Lack of motivations? (cont.)
8-1
Interesting problems
• Frequency moments Fp
F0: #distinct elements
F2: size of self-join
• Heavy hitters
• Quantile
• Entropy
• . . .
Statisticalproblems
8-2
Interesting problems
• Frequency moments Fp
F0: #distinct elements
F2: size of self-join
• Heavy hitters
• Quantile
• Entropy
• . . .
Statisticalproblems Graph problems
• Connectivity
• Bipartiteness
• Counting triangles
• Matching
• Minimum spanning tree
• . . .
8-3
Interesting problems
• Frequency moments Fp
F0: #distinct elements
F2: size of self-join
• Heavy hitters
• Quantile
• Entropy
• . . .
Statisticalproblems Graph problems
• Connectivity
• Bipartiteness
• Counting triangles
• Matching
• Minimum spanning tree
• . . .
Numericallinear algebra
• Lp regression
• Low-rank
approximation
• . . .
8-4
Interesting problems
• Frequency moments Fp
F0: #distinct elements
F2: size of self-join
• Heavy hitters
• Quantile
• Entropy
• . . .
Statisticalproblems Graph problems
• Connectivity
• Bipartiteness
• Counting triangles
• Matching
• Minimum spanning tree
• . . .
Numericallinear algebra
• Lp regression
• Low-rank
approximation
• . . .
DB queries
• Conjuntive
queries
Distributedlearning
• Classifiers
. . .
Geometryproblems
9-1
This talk
Introduce two general techniques thatwork in both BB model and MP modelfor proving lower bounds, with examples.
Conclude with several future directions.
We will talk lower bounds.Most lower bounds match the upper bounds.
10-1
Our new ideas and techniques
Ideas
1. Reduce a k-player problem to a 2-player problem.
2. Decompose a complicated problem to several simpler problems
10-2
Our new ideas and techniques
Ideas
1. Reduce a k-player problem to a 2-player problem.
2. Decompose a complicated problem to several simpler problems
In fact, we are
exploring the underlying patterns of communication matrices
10-3
Our new ideas and techniques
Ideas
1. Reduce a k-player problem to a 2-player problem.
2. Decompose a complicated problem to several simpler problems
In fact, we are
exploring the underlying patterns of communication matrices
SymmetrizationWith Phillips and Verbin, 2012
CompositionWith Woodruff, 2012
1.
2.
Concrete techniques
10-4
Our new ideas and techniques
Ideas
1. Reduce a k-player problem to a 2-player problem.
2. Decompose a complicated problem to several simpler problems
In fact, we are
exploring the underlying patterns of communication matrices
SymmetrizationWith Phillips and Verbin, 2012
CompositionWith Woodruff, 2012
1.
2.
Concrete techniques
Example: Graph Connecitivy
Example: Distinct Elements
11-1
Communication complexity
I choose not to introduce formal definitions/concepts ofcommunication complexity. Just want to mentionYao’s lemma:
We will prove lower bounds for randomized protocols. Todo that we will usually pick some input distribution, andthen prove lower bounds for any deterministic protocol thatsucceeds w.pr 0.99 under that input distribution.
12-1
1st Technique: Symmetrization
13-1
Example: graph connectivity
k sites each holds a set of edges of a graph
Goal: compute whether the graph is connected.
A trivial UB: O(nk log n) bits.
We prove that this is tight up to a log factor.
14-1
Set disjointness (2-DISJ)
X ⊆ 1, . . . , n Y ⊆ 1, . . . , n
X ∩ Y = ∅?
14-2
Set disjointness (2-DISJ)
X ⊆ 1, . . . , n Y ⊆ 1, . . . , n
X ∩ Y = ∅?
Exists a hard distribution µβ , under which
|X ∩ Y | = 1 (YES instance) w.p. β and
|X ∩ Y | = 0 (NO instance) w.p. 1− β.
Lemma: (Generalization of [Razborov ’90, BJKS ’04])
E[CCβ/100µβ(2-DISJ)] = Ω(n)
15-1
2-DISJ ⇒ k-CONNx1
x2
x3
x4
x5Alice and Bob want to solve the 2-DISJ(X,Y ) ∼ µβ . (set β = 1/k2)⇒ running a protocol for k-CONN on:
Alice
Bob
Alice plays a random site SI with input XI = X.Bob plays other k − 1 sites, and constructs inputs for them using Y : itchoose Xi ∼ µβ |Y (i 6= I).Note: X1, . . . , Xk are i.i.d. conditioned on Y .
Vertices u1, . . . , uk, v1, . . . , vn. An edge (ui, vj) exists iff Xi,j = 1.Let ν be the distribution of the graph.
16-1
2-DISJ ⇒ k-CONN (cont.)
vj |j ∈ [r]\Y vj |j ∈ Y
vj|Y |+1vj|Y |+2
vj|Y |+3vjn vj1 vj|Y |vj2
u1 u3u2 uk
(ui, vj) exists if and only if Xi,j = 1
XI ∩ Y = ∅?
17-1
2-DISJ ⇒ k-CONNx1
x2
x3
x4
x5Alice and Bob want to solve the 2-DISJ(X,Y ) ∼ µβ . (set β = 1/k2)⇒ running a protocol for k-CONN on:
Alice
Bob
Alice plays a random site SI with input XI = X.Bob plays other k − 1 sites, and constructs inputs for them using Y : itchoose Xi ∼ µβ |Y (i 6= I).Note: X1, . . . , Xk are i.i.d. conditioned on Y .
Correctness: Solving k-CONN w.pr. 1 − β1.5 under ν also solves2-DISJ w.pr. 1− β/100 under µβ .
Vertices u1, . . . , uk, v1, . . . , vn. An edge (ui, vj) exists iff Xi,j = 1.Let ν be the distribution of the graph.
17-2
2-DISJ ⇒ k-CONNx1
x2
x3
x4
x5Alice and Bob want to solve the 2-DISJ(X,Y ) ∼ µβ . (set β = 1/k2)⇒ running a protocol for k-CONN on:
Alice
Bob
Alice plays a random site SI with input XI = X.Bob plays other k − 1 sites, and constructs inputs for them using Y : itchoose Xi ∼ µβ |Y (i 6= I).Note: X1, . . . , Xk are i.i.d. conditioned on Y .
Correctness: Solving k-CONN w.pr. 1 − β1.5 under ν also solves2-DISJ w.pr. 1− β/100 under µβ .
Vertices u1, . . . , uk, v1, . . . , vn. An edge (ui, vj) exists iff Xi,j = 1.Let ν be the distribution of the graph.
E[CCβ/100µβ (2-DISJ)] = O
(1k
)· CCβ
1.5
ν (k-CONN)
17-3
2-DISJ ⇒ k-CONNx1
x2
x3
x4
x5Alice and Bob want to solve the 2-DISJ(X,Y ) ∼ µβ . (set β = 1/k2)⇒ running a protocol for k-CONN on:
Alice
Bob
Alice plays a random site SI with input XI = X.Bob plays other k − 1 sites, and constructs inputs for them using Y : itchoose Xi ∼ µβ |Y (i 6= I).Note: X1, . . . , Xk are i.i.d. conditioned on Y .
Correctness: Solving k-CONN w.pr. 1 − β1.5 under ν also solves2-DISJ w.pr. 1− β/100 under µβ .
Vertices u1, . . . , uk, v1, . . . , vn. An edge (ui, vj) exists iff Xi,j = 1.Let ν be the distribution of the graph.
E[CCβ/100µβ (2-DISJ)] = O
(1k
)· CCβ
1.5
ν (k-CONN)
Ω(n)
17-4
2-DISJ ⇒ k-CONNx1
x2
x3
x4
x5Alice and Bob want to solve the 2-DISJ(X,Y ) ∼ µβ . (set β = 1/k2)⇒ running a protocol for k-CONN on:
Alice
Bob
Alice plays a random site SI with input XI = X.Bob plays other k − 1 sites, and constructs inputs for them using Y : itchoose Xi ∼ µβ |Y (i 6= I).Note: X1, . . . , Xk are i.i.d. conditioned on Y .
Correctness: Solving k-CONN w.pr. 1 − β1.5 under ν also solves2-DISJ w.pr. 1− β/100 under µβ .
Vertices u1, . . . , uk, v1, . . . , vn. An edge (ui, vj) exists iff Xi,j = 1.Let ν be the distribution of the graph.
E[CCβ/100µβ (2-DISJ)] = O
(1k
)· CCβ
1.5
ν (k-CONN)
Ω(n) Ω(nk)
18-1
A useful message
We can show trivial UBs are tight for many statisticaland graph problems in the MP model (e.g., #distinctelements, bipartiteness, cycle/triangle-freeness, . . .), whenexact answers are wanted. (Woodruff, Z, 2013a)
Approximation is often necessary if we want comm.efficient protocols.
18-2
A useful message
We can show trivial UBs are tight for many statisticaland graph problems in the MP model (e.g., #distinctelements, bipartiteness, cycle/triangle-freeness, . . .), whenexact answers are wanted. (Woodruff, Z, 2013a)
Approximation is often necessary if we want comm.efficient protocols.
Example:
Computing the diameter of a graph exactly needs Ω(kn2) bits(when m = n2) in the MP model.
However, there is a protocol that computes the diameter up to
an additive error of 2 using O(kn3/2) bits.
19-1
Other problems using symmetrization
• k-bitwise-MAJ
• k-bitwise-OR/AND/XOR
With applications in
– Distinct elements reporting
– ε-kernels (geometry)
– Heavy-hitters
• Testing bipartiteness,
Testing cycle
Testing triangle-freeness,
. . .
19-2
Other problems using symmetrization
• k-bitwise-MAJ
• k-bitwise-OR/AND/XOR
With applications in
– Distinct elements reporting
– ε-kernels (geometry)
– Heavy-hitters
• Testing bipartiteness,
Testing cycle
Testing triangle-freeness,
. . .
Key:
Find the right 2-playerproblem for reduction
20-1
2nd Technique: Composition
21-1
Example – the F0 (distinct elements) problem
k sites each holds a set Xi (i ∈ 1, 2, . . . , k).
Goal: compute #distinct elements(∪ki=1Xi) up to a (1 + ε)-approx.
21-2
Example – the F0 (distinct elements) problem
k sites each holds a set Xi (i ∈ 1, 2, . . . , k).
Goal: compute #distinct elements(∪ki=1Xi) up to a (1 + ε)-approx.
Current best UB: O(k/ε2)[Cormode, Muth, Yi. 2008]
Holds in MP model
21-3
Example – the F0 (distinct elements) problem
k sites each holds a set Xi (i ∈ 1, 2, . . . , k).
Goal: compute #distinct elements(∪ki=1Xi) up to a (1 + ε)-approx.
Current best UB: O(k/ε2)[Cormode, Muth, Yi. 2008]
Holds in MP model
Previous LB: Ω(k)
And Ω(1/ε2)Direct reduction from Gap-Hamming
21-4
Example – the F0 (distinct elements) problem
k sites each holds a set Xi (i ∈ 1, 2, . . . , k).
Goal: compute #distinct elements(∪ki=1Xi) up to a (1 + ε)-approx.
Current best UB: O(k/ε2)[Cormode, Muth, Yi. 2008]
Holds in MP model
Previous LB: Ω(k)
And Ω(1/ε2)Direct reduction from Gap-Hamming
New LB: Ω(k/ε2). [Woodruff, Z. 2012, 2013b]. Holdsin MP model. Better UB in the BB model
22-1
The proof framework
Step 1: Find two modular problems k-GAP-MAJ and 2-DISJof simpler structures s.t.
F0 can be thought as a composition of them.
Step 2: Analyze the complexities of k-GAP-MAJ and 2-DISJ.
Step 3: Compose two modular problems so that:
Complexity(F0) = Complexity(k-GAP-MAJ)
× Complexity(2-DISJ).
22-2
The proof framework
Step 1: Find two modular problems k-GAP-MAJ and 2-DISJof simpler structures s.t.
F0 can be thought as a composition of them.
Step 2: Analyze the complexities of k-GAP-MAJ and 2-DISJ.
Step 3: Compose two modular problems so that:
Complexity(F0) = Complexity(k-GAP-MAJ)
× Complexity(2-DISJ).
The proof presented here is for a special case when k = 2/ε2.
23-1
k-GAP-MAJ
k sites each holds a bit Zi chosen uniform at random from 0, 1
Goal: compute
k-GAP-MAJ(Z1, Z2, . . . , Zk) =
0, if
∑i∈[k] Zi ≤ k/2−
√k,
1, if∑i∈[k] Zi ≥ k/2 +
√k,
don’t care, otherwise,
23-2
k-GAP-MAJ
k sites each holds a bit Zi chosen uniform at random from 0, 1
Goal: compute
k-GAP-MAJ(Z1, Z2, . . . , Zk) =
0, if
∑i∈[k] Zi ≤ k/2−
√k,
1, if∑i∈[k] Zi ≥ k/2 +
√k,
don’t care, otherwise,
Lemma: Any protocol Π that computes k-GAP-MAJ correctlyw.p. 0.9999 has to learn Ω(k) Zi’s well, that is,
H(Zi | Π) ≤ Hb(1/100) for Ω(k) of i.
In other words: I(Π;Z1, . . . , Zk) = Ω(k).
24-1
Set disjointness (2-DISJ)
X ⊆ 1, . . . , n Y ⊆ 1, . . . , n
X ∩ Y = ∅?
24-2
Set disjointness (2-DISJ)
X ⊆ 1, . . . , n Y ⊆ 1, . . . , n
X ∩ Y = ∅?
Exists a hard distribution µβ , under which
|X ∩ Y | = 1 (YES instance) w.p. β and
|X ∩ Y | = 0 (NO instance) w.p. 1− β.
Lemma: (Generalization of [Razborov ’90, BJKS ’04])
E[CCβ/100µβ(2-DISJ)] = Ω(n)
25-1
The proof sketch
· · ·S1 S2 S3 Sk
Ccoordinator
sites
Y
X1 X2 X3 Xk
2-DISJ
(Xi, Y ) ∼ µ1/2 (hard input dist. for DISJ) for each i ∈ [k]
25-2
The proof sketch
· · ·S1 S2 S3 Sk
Ccoordinator
sites
Y
X1 X2 X3 Xk
2-DISJ
(Xi, Y ) ∼ µ1/2 (hard input dist. for DISJ) for each i ∈ [k]
Zi = |Xi ∩ Y |
1 w.p. 1/20 w.p. 1/2
25-3
The proof sketch
· · ·S1 S2 S3 Sk
Ccoordinator
sites
Y
X1 X2 X3 Xk
2-DISJ
F0(X1, X2, . . . , Xk) ⇐⇒ k-GAP-MAJ(Z1, Z2, . . . , Zk)
(Zi = |Xi ∩ Y |)=⇒ learn Ω(k) Zi’s well
=⇒ need Ω(nk) bits
(using an embedding argument)
(Xi, Y ) ∼ µ1/2 (hard input dist. for DISJ) for each i ∈ [k]
Zi = |Xi ∩ Y |
1 w.p. 1/20 w.p. 1/2
25-4
The proof sketch
· · ·S1 S2 S3 Sk
Ccoordinator
sites
Y
X1 X2 X3 Xk
Q.E.D.
For the reduction setuniverse n = Θ(1/ε2)
we get Ω(k/ε2) LB.
2-DISJ
F0(X1, X2, . . . , Xk) ⇐⇒ k-GAP-MAJ(Z1, Z2, . . . , Zk)
(Zi = |Xi ∩ Y |)=⇒ learn Ω(k) Zi’s well
=⇒ need Ω(nk) bits
(using an embedding argument)
(Xi, Y ) ∼ µ1/2 (hard input dist. for DISJ) for each i ∈ [k]
Zi = |Xi ∩ Y |
1 w.p. 1/20 w.p. 1/2
26-1
Other problems using composition
• Frequency moments Fp (p > 1)
• Entropy
• `p
• Heavy-hitters
• Quantile
26-2
Other problems using composition
• Frequency moments Fp (p > 1)
• Entropy
• `p
• Heavy-hitters
• Quantile
Key:
Find the right modular problems
27-1
Furture directions
28-1
Future directions – Problem space
The current understandings
Exactcomputation
Approximation
statistics graph linearalgebra
DBqueries
well
well
medium
limited
limited
limitedlimited
28-2
Future directions – Problem space
Concretely, e.g.,
What is the CC of computing a const approx of the size ofthe matching?
The current understandings
Exactcomputation
Approximation
statistics graph linearalgebra
DBqueries
well
well
medium
limited
limited
limitedlimited
29-1
Future directions – Problem space (cont.)
Communication complexity of distributed property testing.
Communication complexity of distributed property testing canalso be used for proving LBs for traditional property testing,by extending a framework by Blais et al. 2011.
29-2
Future directions – Problem space (cont.)
Communication complexity of distributed property testing.
Communication complexity of distributed property testing canalso be used for proving LBs for traditional property testing,by extending a framework by Blais et al. 2011.
Communication complexity of relations(instead of functions, useful for database queries)
29-3
Future directions – Problem space (cont.)
Communication complexity of distributed property testing.
Communication complexity of distributed property testing canalso be used for proving LBs for traditional property testing,by extending a framework by Blais et al. 2011.
Communication complexity of relations(instead of functions, useful for database queries)
Need to understand more primitive problems (building blocks),e.g., breadth-first search (BFS), pointer-chasing, etc.
So far building blocks are quite limited: mainly 2-DISJ, 2-GAP-HAMMING, k-GAP-MAJ, k-OR and their variants.
30-1
Future directions – Model/Technique space
Can we relax or generalize the symmetrization technique?
E.g., do not require “totally symmetric distributions”
30-2
Future directions – Model/Technique space
Can we relax or generalize the symmetrization technique?
E.g., do not require “totally symmetric distributions”
Consider round complexity (practical needs).
Consider players with limited space (practical needs).
31-1
The end
T HANK YOU
Questions?