Efficient Algorithms via Precision Sampling

Robert Krauthgamer (Weizmann Institute)

joint work with:

Alexandr Andoni (Microsoft Research)Krzysztof Onak (CMU)

Goal

Compute the fraction of Dacians in the empire

Estimate S=a1+a2+…an where ai[0,1]

Sampling Send accountants to a subset J of provinces, |

J|=m Estimator: S =∑jJ aj * n / m

Chebyshev bound: with 90% success probability

0.5*S – O(n/m) < S < 2*S + O(n/m) For constant additive error, need m~n

Send accountants to each province, but require only approximate counts Estimate ai up to pre-selected precision ui i.e. |ai–

ai|<ui

Challenge: achieve good tradeoff between quality of approximation to S total cost of computing each a i (within precision ui)

Precision Sampling Framework

Formalization

Estimator (Alg) Adversary

1. fix (hidden) a1,a2,…an1. fix precisions ui

2. fix a1,a2,…an s.t. |ai–ai|<ui

3. report S s.t. |∑iai–S| < 1

What is our cost model? Here, average cost = 1/n * ∑i 1/ui Achieving precision ui requires 1/ui “resources”: e.g., if ai is

itself a sum ai=∑jaij computed by subsampling, then one needs Θ(1/ui) samples

For example, can choose all ui=1/n Average cost ≈ n This is best possible, if estimator S = ∑i a i

Precision Sampling Lemma Goal: estimate ∑ai from ai satisfying |ai-ai|

<ui. Precision Sampling Lemma: can get, with 90%

success: O(1) additive error and 1.5 multiplicative error:

S – O(1) < SM < 1.5*S + O(1) with average cost O(log n)

Example: distinguish Σai=3 vs Σai=1 Consider two extreme cases:

if three ai=1: estimate all ai with crude approx (ui=0.1) if all ai=3/n: estimate few with good approx ui=1/n, the

rest with ui=1

ε 1+εS – ε < S < (1+ ε)S + ε

O(ε-3 log n)

Precision Sampling Algorithm Precision Sampling Lemma: can get, with 90%

success: O(1) additive error and 1.5 multiplicative error:

S – O(1) < SM < 1.5*S + O(1) with average cost equal to O(log n)

Algorithm: Choose each ui[0,1] i.i.d. Estimator: S = count number of i‘s s.t. ai/ui > 6

(and normalize) Outline of analysis:

E[S ] = ∑i Pr[ai/ui > 6] = ∑i Pr[ai > (6±1)ui] ≈ ∑ ai/6. Actually, ai may have also 1.5-multiplicative error

w.r.t. ai

E[1/ui] = O(log n) w.h.p. (after truncation)

function of [ai/ui - 4/ε]+ and ui’sconcrete distrib. = minimum of O(ε-3) uniform r.v.

O(ε-3 log n)

ε 1+εS – ε < S < (1+ ε)S + ε

Why? Save time:

Problem: computing edit distance between two strings [FOCS’10]

new algorithm that obtains (log n)1/ε approximation in n1+O(ε) time

via property-testing-like algorithm using Precision Sampling (recursively)

Save space: Problem: compute norms/moments of frequencies

in a data-stream [FOCS’11]

a simple and unified approach to compute all lp-norms/moments, and related problems

Streaming/sketching

IP Frequency

131.107.65.14

3

18.0.1.12 2

80.97.56.20 2

131.107.65.14

131.107.65.14

131.107.65.14

18.0.1.12

18.0.1.12

80.97.56.20

80.97.56.20

IP Frequency

131.107.65.14

3

18.0.1.12 2

80.97.56.20 2

127.0.0.1 9

192.168.0.1 8

257.2.5.7 0

16.09.20.11 1

Challenge: log statistics of the data, using small space

Streaming moments Setup:

1+ε estimate frequencies in small space Let xi = frequency of IP i pth moment: Σixi

p

p=1: keep one counter! p[0,2]: space O(ε-2 ¢log n)

[AMS’96, I’00, GC’07, Li’08, NW’10, KNW’10, KNPW’11] p>2: space Oε(n1-2/p)

[AMS’96, SS’02, BJKS’02, CKS’03, IW’05, BGKS’06, BO’11]

Generally, xRn (updates: to coordinate i with ±1) Sketch = embedding into a “space” of small

dimension Usually, linear L:RnRm for m¿n, thus L(x±ei)=Lx±Lei

IP Frequency

131.107.65.14

3

18.0.1.12 2

80.97.56.20 2

lp moments

Theorem: linear sketch for lp with O(1) approximation, and O(n1-2/p log n) space (90% succ. prob.). =weak embedding of lpn

into l∞m of dim m=O(n1-2/p log n)

Sketch: pick random ui[0,1], ri±1

and let yi = ri∙xi/ui1/p

throw yi‘s into hash table H

with m=O(n1-2/p log n) cells Estimator:

via PSL or just Maxj[m] |H[j]|p

Randomness: O(1) independence suffices

x1 x2 x3 x4 x5 x6

y1

+y3

y4 y2

+y5+y6

x=

H=

1 … m

Under the Hood: Using PSL Idea: Use PSL to compute the sum ||x||p

p=∑i |xi|p

Assume ||x||2=1 by scaling Set PSL additive error ε small compared to

||x||2p/np/2-1·||x||p

p

Outline: 1. Pick ui’s according to PSL and let yi=xi/ui

1/p

2. Compute every yip=xi

p/ui within additive approximation 1 done via heavy hitters of the vector y

3. Use PSL on |yipui|=|xi|p

to compute the sum ∑i |xi|p

Space bound is controlled by the norm ||y|| 22

.

Since heavy hitters under l2 is the best we can do

Notice E||y||22 = ||x||2

2 ¢ E[1/u2/p] · (1/ε)2/p=(np/2-1)2/p.

More Streaming Algorithms Other streaming algorithms:

Same algorithm for all p-moments, including p≤2 For p>2, improves existing space bounds [AMS96, IW05,

BGKS06, BO10] For p≤2, worse space bounds [AMS96, I00, GC07, Li08, NW10,

KNW10, KNPW11]

Algorithms for mixed norms (lp of lq) [CM05, GBD08, JW09] Space bounded by (Rademacher) p-type constant

Algorithm for lp-sampling problem [MW’10] This work extended to give tight bounds by [JST’10]

Connections: Inspired by the streaming algorithm of [IW05], but

simpler Turns out to be distant relative of Priority Sampling

[DLT’07]

Finale Other applications for Precision Sampling framework? Better algorithms for precision sampling?

For average cost (for 1+ε approximation) Upper bound: O(ε-3 log n) (tight for our algorithm) Lower bound: Ω(ε-2 log n)

Bounds for other cost models? E.g., for 1/square root of precision, the bound is O(ε-3/2)

Other forms of “access” to ai’s?