Administrivia, Lecture 4

• HW #2 assigned this weekend, due Thurs Week 4• HWs will be due Thurs of Weeks 2, 4, 6, 7, 9, 10• HW #1 solutions should be posted tonight (TA’s

have compiled them)• Reading for next week: Chapter 9.2-3 (linear-time

selection), Chapter 33.3-4 (convex hull, closest-pair), Chapter 23 (minimum spanning trees)

Divide and Conquer for Sorting (2.3/1.3)

• Divide (into two equal parts)• Conquer (solve for each part separately)• Combine separate solutions• Mergesort

– Divide into two equal parts– Sort each part using Mergesort (recursion!!!)– Merge two sorted subsequences

Merging Two Subsequences

x[1]-x[2]- … - x[K]

y[1]-y[2]- … - y[L]

if y[i] > x[j] y[i+1] > x[j]

<< K+L-1 edges = # (comparisons) = linear time

Merge Sort Execution Example

285 179 652 351 423310 861 254 450 520

285 179 652 351

423

310

861 254 450 520

285

179 652 351

423

310

861 254 450 520

285

179 652 351

423

310

861 254 450 520

310

179 652 351

423

285

861 254 450 520

310 179 652 351

423

285

861 254 450 520

310 179

652 351

423

285

861 254 450 520

310 179 351 652

423

285

861 254 450 520

310 179 351 652

423

285

861 254 450 520

285 310 351 652

423

179

861 254 450 520285 310 351 652

423

179

861 254 450 520

285 310 351 652

423

179

861

254 450 520

285 310 351 652

423

179

861 254 450 520

285 310 351 652

423

179

861

254 450 520

285 310 351 652

423

179

861

254

450 520

285 310 351 652

423

179

861

254 450 520

285 310 351 652

423

179

861 254 450 520

285 310 351 652

254

179

423 450 520 861

254 285 310 351 423179 450 520 652 861

Recursion Tree

1 2 3 4 5 6 7 8

1 3 5 8 2 4 6 7

1 5 3 8

5 1 3 8

4 7 2 6

7 4 6 2

log n

• n comparisons per level• log n levels• total runtime = n log n

Master Method (4. 3)

Recurrence T(n) = aT(n/b) + f(n)

1) If for some > 0 then

2) If then

3) If for some > 0 and

a f(n/b) c f(n) for some c < 1 then

)()( log abnOnf

)()( log abnnT

)()( log abnnf

)()( log abnnf

)log()( log nnnT ab

))(()( nfnT

Master Method Examples

• Mergesort T(n) = 2T(n/2) + (n)

• Strassen (28.2) T(n) = 7T(n/2) + (n2)

2)(2log2 Casennn

)log()( nnnT

1)( 281.081.27log2 CasenOnn

)()()( 807.27log2 nnnT

Quicksort (7.1-7.2/8.1-8.2)

• Sorts in place like insertion sort, unlike merge sort• Divide into two parts such that

– elements of left part < elements of right part• Conquer: recursively solve for each part separately• Combine: trivial - do not do anything

Quicksort(A,p,r) if p < r then q Partition (A,p,r) Quicksort (A,p,q) Quicksort (A,q+1,r)

//divide//conquer left//conquer right

Divide = Partition

PARTITION(A,p,r)

//Partition array from A[p] to A[r] with pivot A[p]

//Result: All elements original A[p] have index ix = A[p]i = p - 1j = r + 1repeat forever

repeat j = j - 1 until A[j] xrepeat i = i +1 until A[i] xif i < j then exchange A[i] A[j] else return j

How It Works

9 7 6 15 16 5 10 11

i j

9 7 6 15 16 5 10 11

i j

9 7 6 15 16 5 10

i j

119 7 6 15 16 5 10

i j

11109 7 6 15 16 5

i j

11109 7 6 15 16 5

i j

11109 7 6 15 16 5

i j

11109 7 6 15 16 5

i j

11109 7 6 15 16 5

i j

11105* 7 6 15 16 9*i j

11105 7 6 15 16 9i j

11105 7 6 15 16 9i j

11105 7 6 15 16 9i j

11105 7 6 15 16 9i j

11105 7 6 15 16 9i j

11105 7 6 15 16 9i j

11105 7 6 15 16 9i j

11105 7 6 15 16 9i j

11105 7 6 15 16 9i j

11105 7 6 15 16 9

i

j

11105 7 6 15 16 9

i

j

11105 7 6 15 16 9ij

11105 7 6 15 16 9ij

11105 7 6 15 16 9 11

left right

9 7 6 15 16 5 10 11

i j

105 7 6 15 16 9 11

left right

105 7 6 15 16 9 11

left right

105 7 6 15 16 9 11

left right

105 7 6 15 16 9 11

left right

ji

105 7 6 15 16 9 11

left right

ji

105 7 6 15 16 9 11

left right

ji

105 7 6 15 16 9 11

left right

j

i

105 7 6 15 16 9 11

left right

ji

105 7 6 15 16 9 11

left right

105 6 7 15 16 9 11

left right

105 6 7 15 16 9 11

left right

105 6 7 15 16 9 11

left right

105 6 7 15 16 9 11

left right

jj

105 6 7 15 16 9 11

left right

ji

105 6 7 11* 16 9 15*

left right

ji

105 6 7 11 16 9 15

left right

ji

105 6 7 11 16 9 15

left right

ji

165 6 7 11 10 9 15

left right

ji

165 6 7 11 10 9 15

left right

j i

165 6 7 11 10 9 15

left right

165 6 7 9 10 11 15

left right

155 6 7 9 10 11 16

left right

155 6 7 9 10 11 16

Runtime of Quicksort

• Worst case: – every time nothing to move– pivot = left (right) end of subarray– O(n2)

0123456789

123456789

8

0

9

89

n

Recursion Tree of QSort

Runtime of Quicksort

• Best case: – every time partition in (almost) equal parts – no worse than in given proportion– O(n log n)

• Average case = ?

What is the DQ Recurrence for QSort?

• t(n) = (n) + 1/n j = 1 to n (t(j-1) + t(n-j)) pivots pivots equiprobable lengths of subproblems

= (n) + 2/n k = 0 to n-1 t(k)

• Guess t(n) O(n log n)

• t(n) (n) + 2/n [ i=2 to n-1 c i log i] c n log n – cn/2

4

2^log

2

2^...

^24

loglog

2

2^loglog

log

1

1

1

1

0

00

nn

n

xe

xx

xdxxii

kkBound

en

n

e

n

ni

e

n

k

Another QSort Analysis (Shift / Cancel)

• (1) T(n) = n-1 + 2/n j = 1 to n-1T(i), n 2, T(1) = 0

• (2) T(n+1) = n+1 - 1 + 2/(n+1) j = 1 to nT(i)

(1) (3) nT(n) = n(n-1) + 2 j = 1 to n-1T(i)

(2) (4) (n+1)T(n+1) = (n+1)n + 2 j = 1 to nT(i)

(4) - (3) (n+1)T(n+1) – nT(n) = 2n + 2T(n)

T(n+1) = (n+2)/(n+1) T(n) + 2n/(n+1)

T(n+1) (n+2)/(n+1) T(n) + 2

Shift / Cancel (cont.)

Unroll:

Hn = 1 + ½ + 1/3 + …+ 1/n = ln n + + O(1/n)

= Euler’s constant = 0.577…

T(n) 2(n+1) (ln n + – 3/2) + O(1)

T(n) O(n log n)

)2

3)(1(2

)3

1...

1

11

1

1()1(2

3

1...

2

1

1

111(2

)3

4..

1...

2

1

1

1

1

111(2

)))3

4(...

2

12(

12(

12)(

1

nHn

nnnn

n

n

n

n

n

n

nn

n

n

n

n

n

n

n

n

n

n

n

n

nn

n

n

n

n

nnT

Randomized Analysis of QSort• Probability space

– Set of elementary events experiment outcomes– Family F of subsets of , called events– Probability measure Pr, real-valued function on

members of F– where A , A F Ac F

F closed under union, intersection

For all A F, 0 Pr(A) 1

Pr() = 1

For disjoint events A1, A2, ... Pr(U Ai) = Pr(Ai)

• Random variable is a function from elements of into – e.g., event (X = x) set of elements of for which X assumes the

fixed value x

• For integer-valued r.v. X, expectation E[X] = i Pr(X=i)

Randomized Analysis of QSort (cont.)

• At each level, we compare each element to the splitter in its partition

• Def. A successful pivot p satisfies n/8 < p < 7n/8• Three facts

– (1) E[ Xi] = E[Xi] linearity of expectation– (2) If Pr(Heads) is q, expected # tosses up to and

including first Head is 1/q // family size puzzle

– (3) Pr(Ui Ai) i Pr(Ai) prob of union sum of probs

• Given that Pr(successful pivot) = 3/4 , a single element ei participates in at most how many successful partitioning steps?– Ans: log8/7 n

Randomized Analysis of QSort (cont.)

• What is number of partition steps expected between kth , (k+1)st successful pivots?– (2) 4/3 steps

– (1) 4/3 log8/7 n

– Each element expects to participate in O(log n) pivots (comparisons)

• Define r.v.’s which give # comparisons for each element – (1) get O(n log n) expected comparisons

• So, we have another analysis that gives us the O(n log n) expected complexity of QSort

How Badly Can We Deviate From Expectation?

• E.g., what is Pr (ith element sees 20 log n pivots)?– How many successes possible?

– Know log8/7 n

• Pivots are independent (Bernoulli trials)– Pr(success) = 3/4, Pr(failure) = 1/4– To see 20 log n pivots, need 20 log n - log n failures in

20 log n tries

• Can use (3) to bound probability of so many “bad” events, since in general events may not be independent– Can get: Pr( 20 n log n) is 1 - O(n-6)

Selection

• Best pivot: median– exercise: analyze complexity when bad pivots chosen– too expensive random splitter

• This leads to SELECTION:– select (L, k) returns kth - smallest element of L– e.g., k = |L|/2 median

• What is an efficient algorithm?– O(n log n)?– sorting

• D/Q Recursion: – N.B.: This if RSelect, below– recall pivot from QSort– if i<k, look in right part for (k-i)th smallest– if i>k, look in left part for kth smallest

Randomized Selection

• Worst case:– (n2) as in QSort analysis

• Suppose can guarantee “good” pivot– e.g., n/4 i 3n/4– subproblem size 3n/4– Let s(n) time to find good pivot– t(n) s(n) + cn + t([3n/4])

• find pivot

• pivot, make subproblem

• solve subproblem

Randomized Selection

• Suppose further: S(n) dn for some d;

t(n) (c+d)n + t([3n/4])

• Claim: t(n) kn for some k would follow– Constructive induction or “substitution”– Ind. Hyp.: t(m) km for m n-1– Ind. Step: t(n) (c+d)n + k(3n/4) = (c+d+3k/4)n kn

which we want to be equivalent to t(n) kn– But this is true if k 4(c+d)

Break

• Celebrity Problem: Given n people and a “knows” relation, is there a celebrity?– Notation: Directed graph (and, let’s say that there can be 0, 1 or 2

directed edges between any two vertices (== people))

• What is obvious algorithm?– Test each person’s celebrityhood

• Induction– Hyp: Can tell whether there is a celebrity among the first n-1 people– Induction Step:

• Have a celebrity among first n-1 people two queries needed to verify whether known by nth person

• No celebrity among first n-1 people check whether nth person is a celebrity (2(n-1) queries needed)

• (Else no celebrity) (n2) queries

Break (cont.)

• Celebrity Problem: Can you do better?– Hint: (n-1) + 2(n-1) queries suffice

• Why are we focusing on identifying the celebrity?• Key idea: eliminate a non-celebrity with each query

– Kij = 0 j not celebrity– Kij = 1 i not celebrity– (We’ve seen this “complement” idea in DQ-MAXMIN)

• Another question: Given the adjacency matrix of an undirected graph, describe a fast algorithm that finds all triangles in the graph.– Q: Why am I asking this now?

D/Q for Arithmetic• Multiplying Large Integers

– A = a0 + r1a1 + ... + rn-1an-1, r radix– “classic” approach (n2) work

• Can we apply D/Q?– Let n = 2s, r = 10 radix– AB = xz + 10s(wz + xy) + 102swy– T(n) = 4T(n/2) + (n) a = 4, b = 2 in Master Method– T(n) (n2)– Need to reduce # subproblems, i.e., want a < 4

• Observation: r’ = (w+x)(y+z) = wy + (wz+xy) + xz– r’ (w+x)(y+z)– p wy– q xz– return 102sp + 10s(r’-p-q) + q– T(n) O(n log

23) = O(n 1.59)

Matrix Multiplication

• A = […] , B = […] are n x n matrices• a11, a12, etc are n/2 x n/2 submatrices• M = AB = […]

– where m11 = a11b11 + a12b21 etc.

– Evaluation requires 8 multiplies, 4 adds• T(n) = 8T(n/2) + O(n) (n3)• Strassen:

– p1 = (a21 + a22 - a11)(b22 - b12 + b11)– p2 = a11b11

– p3 = a12b21

– p4 = (a11-a21)(b22-b12)– p5 = (a21+a22)(b12 - b11)– p6 = (a12-a21+a11-a22)b22

– p7 = a22(b11+b22-b12-b21)

Strassen’s Matrix Multiplicationp1 = (a21 + a22 - a11)(b22 - b12 + b11)

p2 = a11b11

p3 = a12b21

p4 = (a11-a21)(b22-b12)

p5 = (a21+a22)(b12 - b11)

p6 = (a12-a21+a11-a22)b22

p7 = a22(b11+b22-b12-b21)

AB11 = p2 + p3

AB12 = p1 + p2 + p5 + p6

AB21 = p1 + p2 + p4 + p7

AB22 = p1 + p2 + p4 + p5

• T(n) (n2.81) // 7 multiplies, 24 adds– Can get to 15 adds