CSE 101€¦ · COOK-TOOM ALGORITHM In fact, if you split up your number into k equally sized...

Algorithm Design and Analysis

Miles Jones

[email protected]

Office 4208 CSE Building

Lecture 14: Divide & Conquer

CSE 101

mailto:[email protected]

DIVIDE AND CONQUER

Break a problem into similar subproblems

Solve each subproblem recursively

Combine

MULTIPLYING BINOMIALS

if you want to multiply two binomials

𝑎𝑥 + 𝑏 𝑐𝑥 + 𝑑 = 𝑎𝑐𝑥2 + 𝑎𝑑𝑥 + 𝑏𝑐𝑥 + 𝑏𝑑

It requires 4 multiplications. 𝑎𝑐, 𝑎𝑑, 𝑏𝑐, 𝑏𝑑


if you want to multiply two binomials

𝑎𝑥 + 𝑏 𝑐𝑥 + 𝑑 = 𝑎𝑐𝑥2 + (𝑎𝑑 + 𝑏𝑐)𝑥 + 𝑏𝑑

It requires 4 multiplications. 𝑎𝑐, 𝑎𝑑, 𝑏𝑐, 𝑏𝑑

If we assume that addition is cheap (has short runtime.) Then we can

improve this by only doing 3 multiplications:

𝑎𝑐, 𝑏𝑑, (𝑎 + 𝑏)(𝑐 + 𝑑)


Reducing the number of multiplications from 4 to 3 may not seem very

impressive when calculating asymptotics.

If this was only a part of a bigger algorithm, it may be an

improvement.

MULTIPLYING BINARY NUMBERS

MULTIPLYING BINARY NUMBERS (DC)

Suppose we want to multiply two n-bit numbers together where n is a

power of 2.

One way we can do this is by splitting each number into their left and

right halves which are each n/2 bits long

x=

y=

xL xR

yL yR


Suppose we want to multiply two n-bit numbers together where n is a

power of 2.

One way we can do this is by splitting each number into their left and

right halves which are each n/2 bits long

𝑥 = 2𝑛/2𝑥𝐿 + 𝑥𝑅

𝑦 = 2𝑛/2𝑦𝐿 + 𝑦𝑅


𝑥 = 2𝑛/2𝑥𝐿 + 𝑥𝑅𝑦 = 2𝑛/2𝑦𝐿 + 𝑦𝑅

𝑥𝑦 = 2𝑛

2𝑥𝐿 + 𝑥𝑅 2𝑛

2𝑦𝐿 + 𝑦𝑅

𝑥𝑦 = 2𝑛𝑥𝐿𝑦𝐿 + 2𝑛

2 𝑥𝐿𝑦𝑅 + 𝑥𝑅𝑦𝐿 + 𝑥𝑅𝑦𝑅

ALGORITHM MULTIPLY

function multiply (x,y)

Input: n-bit integers x and y

Output: the product xy

If n=1: return xy

𝑥𝐿 , 𝑥𝑅 and 𝑦𝐿 , 𝑦𝑅 are the left-most and right-most n/2 bits of x and y, respectively.

𝑃1 = 𝑚𝑢𝑙𝑡𝑖𝑝𝑙𝑦 𝑥𝐿 , 𝑦𝐿 𝑃2 = 𝑚𝑢𝑙𝑡𝑖𝑝𝑙𝑦 𝑥𝐿 , 𝑦𝑅 𝑃3 = 𝑚𝑢𝑙𝑡𝑖𝑝𝑙𝑦 𝑥𝑅 , 𝑦𝐿 𝑃4 = 𝑚𝑢𝑙𝑡𝑖𝑝𝑙𝑦 𝑥𝑅 , 𝑦𝑅

return 𝑃1 ∗ 2𝑛 + 𝑃2 + 𝑃3 ∗ 2

𝑛

2 + 𝑃4

ALGORITHM

Runtime

Let T(n) be the runtime of the multiply algorithm.

Then

𝑇 𝑛 = 4𝑇𝑛

2+ 𝑂(𝑛)

Multiplication

Insight: replace

one (of the 4)

multiplications by

(linear time)

subtraction

ALGORITHM MULTIPLY KS

function multiplyKS (x,y)

Input: n-bit integers x and y

Output: the product xy

If n=1: return xy

𝑥𝐿 , 𝑥𝑅 and 𝑦𝐿 , 𝑦𝑅 are the left-most and right-most n/2 bits of x and y, respectively.

𝑅1 = 𝑚𝑢𝑙𝑡𝑖𝑝𝑙𝑦 𝑥𝐿 , 𝑦𝐿 𝑅2 = 𝑚𝑢𝑙𝑡𝑖𝑝𝑙𝑦 𝑥𝑅 , 𝑦𝑅

𝑅3 = 𝑚𝑢𝑙𝑡𝑖𝑝𝑙𝑦 (𝑥𝐿+𝑥𝑅) 𝑦𝐿 + 𝑦𝑅

return 𝑅1 ∗ 2𝑛 + 𝑅3 − 𝑅1 − 𝑅2 ∗ 2

𝑛

2 + 𝑅2

ALGORITHM MULTIPLYKS

Runtime

Let T(n) be the runtime of the multiply algorithm.

Then


2+ 𝑂(𝑛)

MASTER THEOREM

How do you solve a recurrence of the form

𝑇 𝑛 = 𝑎𝑇𝑛

𝑏+ 𝑂 𝑛𝑑

We will use the master theorem.

Summation Lemma

Consider the summation

𝑘=0

𝑛

𝑟𝑘

It behaves differently for different values of 𝑟.

Summation Lemma


𝑘=0

𝑛

𝑟𝑘


If 𝑟 < 1 then this sum converges. This means that the sum is bounded above by some

constant 𝑐. Therefore

𝑖𝑓 𝑟 < 1 𝑡ℎ𝑒𝑛

𝑘=0

𝑛

𝑟𝑘 < 𝑐 𝑓𝑜𝑟 𝑎𝑙𝑙 𝑛 𝑠𝑜

𝑘=0

𝑛

𝑟𝑘 ϵ 𝑂(1)

Summation Lemma


𝑘=0

𝑛

𝑟𝑘


If 𝑟 = 1 then this sum is just summing 1 over and over n times. Therefore

𝑖𝑓 𝑟 = 1

𝑘=0

𝑛

𝑟𝑘 =

𝑘=0

𝑛

1 = 𝑛 + 1 ϵ 𝑂(𝑛)

Summation Lemma


𝑘=0

𝑛

𝑟𝑘


If 𝑟 > 1 then this sum is exponential with base 𝑟.

𝑖𝑓 𝑟 > 1

𝑘=0

𝑛

𝑟𝑘 < 𝑐𝑟𝑛 𝑓𝑜𝑟 𝑎𝑙𝑙 𝑛, 𝑠𝑜

𝑘=0

𝑛

𝑟𝑘 ϵ 𝑂 𝑟𝑛 𝑐 >𝑟

𝑟 − 1

Summation Lemma

𝑖𝑓 𝑟 > 1

𝑘=0

𝑛

𝑟𝑘 < 𝑐𝑟𝑛 𝑓𝑜𝑟 𝑎𝑙𝑙 𝑛: 𝑐 >𝑟

𝑟 − 1… . . 𝑤ℎ𝑦? ? ? ? ?

Summation Lemma


𝑘=0

𝑛

𝑟𝑘


𝑘=0

𝑛

𝑟𝑘 ϵ ቐ

𝑂 1 𝑖𝑓 𝑟 < 1

𝑂 𝑛 𝑖𝑓 𝑟 = 1

𝑂 𝑟𝑛 𝑖𝑓 𝑟 > 1

Master Theorem

Theorem: If 𝑇(𝑛) = 𝑎𝑇(𝑛/𝑏) + 𝑂(𝑛𝑑) for some constants

𝑎 > 0, 𝑏 > 1, 𝑑 ≥ 0 ,

Then

𝑇 𝑛 ϵ

𝑂 𝑛𝑑 𝑖𝑓 𝑎 < 𝑏𝑑

𝑂 𝑛𝑑 log 𝑛 𝑖𝑓 𝑎 = 𝑏𝑑

𝑂 𝑛log𝑏 𝑎 𝑖𝑓 𝑎 > 𝑏𝑑

Master Theorem: Solving the recurrence

𝑇(𝑛) = 𝑎𝑇(𝑛/𝑏) + 𝑂(𝑛𝑑)

Size 𝑛

Size 𝑛/𝑏

Size 𝑛/𝑏2

Size 1

Depth log𝑏 𝑛

1 subproblem

𝑎 subproblems

𝑎2 subproblems

𝑎log𝑏 𝑛 subproblems


After 𝑘 levels, there are 𝑎𝑘 subproblems, each of size 𝑛/𝑏𝑘.

So, during the 𝑘th level of recursion, the time complexity is 𝑂𝑛

𝑏𝑘

𝑑𝑎𝑘 =

𝑂 𝑎𝑘𝑛

𝑏𝑘

𝑑

= 𝑂 𝑛𝑑𝑎

𝑏𝑑

𝑘


After 𝑘 levels, there are 𝑎𝑘 subproblems, each of size 𝑛/𝑏𝑘.

So, during the 𝑘th level, the time complexity is 𝑂𝑛

𝑏𝑘

𝑑𝑎𝑘 = 𝑂 𝑎𝑘

𝑛

𝑏𝑘

𝑑

= 𝑂 𝑛𝑑𝑎

𝑏𝑑

𝑘

After log𝑏 𝑛 levels, the subproblem size is reduced to 1, which usually is the size

of the base case.

So the entire algorithm is a sum of each level.

𝑇 𝑛 = 𝑂 𝑛𝑑

𝑘=1

log𝑏 𝑛𝑎

𝑏𝑑

𝑘

Master Theorem: Proof


𝑘=1

log𝑏 𝑛𝑎

𝑏𝑑

𝑘

Case 1: 𝑎 < 𝑏𝑑

Then we have that 𝑎

𝑏𝑑< 1 and the series converges to a constant so




𝑘=1

log𝑏 𝑛𝑎

𝑏𝑑

𝑘

Case 2: 𝑎 = 𝑏𝑑

Then we have that 𝑎

𝑏𝑑= 1 and so each term is equal to 1

𝑇 𝑛 = 𝑂 𝑛𝑑 log𝑏 𝑛



𝑘=1

log𝑏 𝑛𝑎

𝑏𝑑

𝑘

Case 2: 𝑎 > 𝑏𝑑

Then the summation is exponential and grows proportional to its last term

𝑎

𝑏𝑑

log𝑏 𝑛so

𝑇 𝑛 = 𝑂 𝑛𝑑𝑎

𝑏𝑑

log𝑏 𝑛

= 𝑂 𝑛log𝑏 𝑎

Master Theorem Applied to Multiply

The recursion for the runtime of Multiply is

T(n) = 4T(n/2) + cn

So we have that a=4, b=2, and d=1. In this case, 𝑎 > 𝑏𝑑 so

𝑇 𝑛 ϵ𝑂 𝑛log2 4 = 𝑂 𝑛2

Not any improvement of grade-school method.

𝑇 𝑛 ϵ




Master Theorem Applied to MultiplyKS

The recursion for the runtime of Multiply is

T(n) = 3T(n/2) + cn

So we have that a=3, b=2, and d=1. In this case, 𝑎 > 𝑏𝑑 so

𝑇 𝑛 ϵ𝑂 𝑛log2 3 = 𝑂 𝑛1.58

An improvement on grade-school method!!!!!!

𝑇 𝑛 ϵ




Multiply vs MultiplyKS

In terms of worst-case performance.

n n2 𝒏𝟏.𝟓𝟖

100 10 000 ~1 500

1 000 1 000 000 ~60 000

1 000 000 1 000 000 000 000 ~3 000 000 000

Karatsuba wins big!

CAN WE DO BETTER THAN 𝑛1.58?

What if instead of splitting the number in half, we split it into thirds.

x=

y=xL xM

yL yM

xR

yR

CAN WE DO BETTER THAN 𝑛1.58?

What if instead of splitting the number in half, we split it into thirds.

𝑥 = 22𝑛/3𝑥𝐿 + 2𝑛/3𝑥𝑀 + 𝑥𝑅

𝑦 = 22𝑛/3𝑦𝐿 + 2𝑛/3𝑦𝑀 + 𝑦𝑅

MULTIPLYING TRINOMIALS

𝑎𝑥2 + 𝑏𝑥 + 𝑐 𝑑𝑥2 + 𝑒𝑥 + 𝑓



= 𝑎𝑑𝑥4 + 𝑎𝑒 + 𝑏𝑑 𝑥3 + 𝑎𝑓 + 𝑏𝑒 + 𝑐𝑑 𝑥2 + 𝑏𝑓 + 𝑐𝑒 𝑥 + 𝑐𝑓

9 multiplications means 9 recursive calls.

Each multiplication is 1/3 the size of the original.




9 multiplications means 9 recursive calls.

Each multiplication is 1/3 the size of the original.


3+ 𝑂(𝑛)





3+ 𝑂(𝑛)

𝑇 𝑛 ϵ




a=9 9 > 31

b=3 𝑇 𝑛 = 𝑂 𝑛log3 9

d=1 𝑇 𝑛 = 𝑂 𝑛2




There is a way to reduce from 9 multiplications down to just 5!!!

Then the recursion becomes

𝑇 𝑛 = 5𝑇(𝑛/3) + O(n)

So by the master theorem




There is a way to reduce from 9 multiplications down to just 5!!!

Then the recursion becomes

𝑇 𝑛 = 5𝑇(𝑛/3) + O(n)

So by the master theorem T(n)=O(𝑛 log3 5) = 𝑂 𝑛1.43

DIVIDING INTO K SUBPROBLEMS

What happens if we divide into k subproblems each of size n/k.

(𝑎𝑘−1𝑥𝑘−1 + 𝑎𝑘−2𝑥

𝑘−2 + ⋯ 𝑎1𝑥 + 𝑎0)(𝑏𝑘−1𝑥𝑘−1 + 𝑏𝑘−2𝑥

𝑘−2 + ⋯ 𝑏1𝑥 + 𝑏0)

How many terms are there? (multiplications.)

DIVIDING INTO K SUBPROBLEMS

What happens if we divide into k subproblems each of size n/k.

(𝑎𝑘−1𝑥𝑘−1 + 𝑎𝑘−2𝑥

𝑘−2 + ⋯ 𝑎1𝑥 + 𝑎0)(𝑏𝑘−1𝑥𝑘−1 + 𝑏𝑘−2𝑥

𝑘−2 + ⋯ 𝑏1𝑥 + 𝑏0)

How many terms are there? (multiplications.)

There are 𝑘2 multiplications. The recursion is

𝑇 𝑛 = 𝑘2𝑇𝑛

𝑘+ 𝑂 𝑛 … … … 𝑎 = 𝑘2 , 𝑏 = 𝑘, 𝑑 = 1

𝑇 𝑛 = 𝑂(𝑛 log𝑘 𝑘2) = 𝑂 𝑛2

COOK-TOOM ALGORITHM

In fact, if you split up your number into k equally sized parts,

then you can combine them with 2k-1 multiplications instead of

the 𝑘2 individual multiplications.

This means that you can get an algorithm that runs in

𝑇 𝑛 = (2𝑘 − 1)𝑇(𝑛/𝑘) + O(n)

COOK-TOOM ALGORITHM

In fact, if you split up your number into k equally sized parts,

then you can combine them with 2k-1 multiplications instead of

the 𝑘2 individual multiplications.

This means that you can get an algorithm that runs in

𝑇 𝑛 = (2𝑘 − 1)𝑇(𝑛/𝑘) + O(n)

𝑇 𝑛 = 𝑂 𝑛log(2𝑘−1)

log 𝑘 time!!!!

COOK-TOOM ALGORITHM

𝑇 𝑛 = (2𝑘 − 1)𝑇(𝑛/𝑘) + O(n)

𝑇 𝑛 = 𝑂 𝑛log 2𝑘−1

log 𝑘 time.

So we can have a near-linear time algorithm if we take k to be

sufficiently large. The O(n) term in the recursion takes a lot of

time the bigger k gets. So is it worth it to make k very large?

What if you have an algorithm that has a recursion such as:

𝑇 𝑛 = 𝑎𝑇 𝑛 − 𝑏 + 𝑂 𝑛𝑑

This is sometimes referred to as a reduce and conquer. Be careful of

this type of recursion because if a is greater than 1 then the algorithm

may take exponential time.

WARNING

Given a sorted list of integers and a target integer, determine what is

the index of the target if it is in the list.

What is the fastest runtime that we could hope for?

SEARCH

Given a sorted list of integers and a target integer, determine what is

the index of the target if it is in the list.

What is the fastest runtime that we could hope for?

We have two options for each number we compare to the target.

(Binary Tree.)

SEARCH

4, 29, 31, 52, 79, 82, 83, 99, 100, 111, 142, 153, 160, 169, 181

SEARCH

The binary tree will have n nodes. What is the shortest height

possible for a binary tree with n nodes?

SEARCH

The binary tree will have n nodes. What is the shortest height

possible for a binary tree with n nodes?

Any search algorithm takes Ω log 𝑛

SEARCH

DIVIDE AND CONQUER SEARCH

Starting with a sorted list of integers and a target, the goal is to

output the index of the target if it is in the list.

Break a problem into similar subproblems

Split the list into two sublists each of half the size

Solve each subproblem recursively

recursively search one of the lists

Combine

add the index if necessary

BINARY SEARCH

Given a sorted list of integers and an integer, determine what position the integer is in the list or say that it is not in the list.

procedure BinSearch(𝑎1 , … , 𝑎𝑛 ,k)

if n=1 and a_1=k then return 1

if n=1 and a_1≠ 𝑘 then return “not in list”

L_1≔𝑎1 , … 𝑎 𝑛

2

L_2≔𝑎 𝑛

2+1, … , 𝑎𝑛

if 𝑎 𝑛

2

<k then

return 𝑛/2 +BinSearch(L_2,k)

else return BinSearch(L_1,k)

BINARY SEARCH

We split the problem into subproblems of size n/2

Then we only recursively call once.

To combine, we simply compare again and add the index if necessary

• Break a problem into similar subproblems: 1

• Solve each subproblem recursively: n/2

• Combine: O(1)

BINARY SEARCH

𝑇 𝑛 = 𝑇𝑛

2+ 𝑂 1

𝑎 = 1, 𝑏 = 2, 𝑑 = 0 1 = 20

𝑇 𝑛 = 𝑂 log 𝑛

• Break a problem into similar subproblems: 1

• Solve each subproblem recursively: n/2

• Combine: O(1)

SORTING

How long should it take to sort things?

Bubble sort

Insertion sort

Bogo sort

Selection sort

Quicksort

Mergesort

SORTING

How long should it take to sort things?

If we must sort things based on comparisons we must travel down a

path in this tree.

SORTING

How many leaves does the tree have?

What is the height?

SORTING LOWER BOUND RUNTIME

Any sorting algorithm that relies on comparisons between elements

runs in Ω(log 𝑛! ) time.

Is there a simpler expression than log 𝑛! ?


Any sorting algorithm that relies on comparisons between elements

runs in Ω(log 𝑛! ) time.

Is there a simpler expression than log 𝑛! ?

log 𝑛! = log 𝑛 + log 𝑛 − 1 + ⋯+ log 2 + log 1 < 𝑛 log 𝑛

𝑛

2log

𝑛

2< log 𝑛!

log 𝑛! = Θ 𝑛 log 𝑛


any sorting algorithm that relies on comparisons between elements

takes Ω 𝑛 log 𝑛 .

There are sorting algorithms out there that run faster but they rely on

some prior knowledge about the elements. (for example, if you know

that the values are only allowed to come from a small range of values.)

CSE 101€¦ · COOK-TOOM ALGORITHM In fact, if you split up your number into k equally sized...

Documents

Transcript of CSE 101€¦ · COOK-TOOM ALGORITHM In fact, if you split up your number into k equally sized...