THE UNIVERSITY OFNEW SOUTH WALES

Algorithms:COMP3121/3821/9101/9801

Aleks Ignjatovic, ignjat@cse.unsw.edu.auoffice: 504 (CSE building); phone: 5-6659

Course Admin: Amin Malekpour, a.malekpour@unsw.edu.au

School of Computer Science and EngineeringUniversity of New South Wales Sydney

INTRODUCTION

COMP3121/3821/9101/9801 1 / 35

Introduction

What is this course about?

It is about designing algorithms for solving problems using acomputer.

Our goal:

To learn techniques which can be used to solve new, unfamiliarproblems that arise in a rapidly changing field.

Course content:

a survey of algorithms design techniques

particular algorithms will be mostly used to illustrate designtechniques

emphasis on development of algorithm design skills

COMP3121/3821/9101/9801 2 / 35

Textbooks

Textbook:

Kleinberg and Tardos: Algorithm Designpaperback edition available at the UNSW book storegood: very readable (and very pleasant to read!); an excellent textbook;not so good: as a reference manual for later use.

An alternative textbook:

Cormen, Leiserson, Rivest and Stein: Introduction to Algorithmspreferably the third edition, should also be available at the bookstoreexcellent: to be used later as a reference manual;not so good: quite formalistic and written in a rather dry style.

Why should you study algorithms design?

Every algorithm you might need can be found using Google?

COMP3121/3821/9101/9801 3 / 35

The role of proofs in algorithm design

When do we need to prove that an algorithm we have just designedterminates and returns a solution to the problem at hand?

Only when this is not clear by common sense.

COMP3121/3821/9101/9801 4 / 35

Example: MergeSort

Merge-Sort(A,p,r) *sorting A[p..r]*

1 if p < r

2 then q ← bp+r2 c

3 Merge-Sort(A, p, q)

4 Merge-Sort(A, q + 1, r)

5 Merge(A, p, q, r)

1 The depth of recursion in Merge-Sort is log2 n.

2 On each level of recursion merging intermediate arrays takes O(n)steps.

3 Thus, MergeSort always terminates and, in fact, it terminatesin O(n log2 n) many steps.

4 Merging two sorted arrays always produces a sorted array, thus,the output of MergeSort will be a sorted array.

5 The above is essentially a proof by induction, but we will neverbother formalizing proofs of (essentially) obvious facts.

COMP3121/3821/9101/9801 5 / 35

The role of proofs in algorithm design

However, sometimes it is NOT clear from a description of analgorithm that such an algorithm will not enter an infinite loopand fail to terminate;

Sometimes it is not clear that an algorithm will not run inexponentially many steps (in the size of the input), which isessentially as bad as never terminating;

Sometimes it is not clear from a description of an algorithm whysuch an algorithm, after it terminates, produces a desired solution.

Proofs are needed for such circumstances; thus, proofs are NOTacademic embellishments - in lots of cases they are the only wayto know that the algorithm will always work and that no disasteris waiting to happen!

For that reason we will NEVER prove the obvious (your CLRStextbook sometimes does, being too pedantic!) and will prove onlywhat is nontrivial.

COMP3121/3821/9101/9801 6 / 35

Role of proofs - example

Stable Matching Problem

Assume that you are running a dating agency and have n men and nwomen as customers;

They all attend a dinner party; after the party• every man gives you a list with his ranking of all women present, and• every woman gives you a list with her ranking of all men present;

Design an algorithm which produces a stable matching:a set of n pairs p = (m,w) of a man m and a woman w so thatthe following situation never happens:

for two pairs p = (m,w) and p′ = (m′, w′):• man m prefers woman w′ to woman w, and• woman w′ prefers man m to man m′.

COMP3121/3821/9101/9801 7 / 35

Stable Matching Problem

m1# w1#

m2# w2#

m1# w1#

m2# w2#

m1# w1#

m2# w2#

m1# w1#

m2# w2#

Case1:#Preferences:#########################stable#matching############################not#stable#

Case2:#Preferences:#########################stable#matching############################also#stable!#

m1# w1#

m2# w2#

m1# w1#

m2# w2#

−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−

m1# w1#

m2# w2#

m1# w1#

m2# w2#

m1# w1#

m2# w2#

m1# w1#

m2# w2#

Case1:#Preferences:#########################stable#matching############################not#stable#

Case2:#Preferences:#########################stable#matching############################also#stable!#

m1# w1#

m2# w2#

m1# w1#

m2# w2#

COMP3121/3821/9101/9801 8 / 35

Stable Matching Problem: Gale - Shapley algorithm(invented to pair newly graduated physicians with UShospitals for residency training)

Question 1: Is it true that for every possible collection of n lists ofpreferences provided by all men, and n lists of preferences provided byall women, a stable matching exists?

Answer: YES, but this is NOT obvious!

Question 2: Given n men and n women, how many ways are there tomatch them, i.e., just to form n couples?

Answer: n! ≈ (n/e)n - more than exponentially many in n (e ≈ 2.71);

Can we find a stable matching in a reasonable amount of time??

Answer: YES, using the Gale - Shapley algorithm.

COMP3121/3821/9101/9801 9 / 35

Stable Matching Problem: Gale - Shapley algorithm

• Produces pairs in stages, with possible revisions;• A man who has not been paired with a woman will be called free.• Men will be proposing to women.Women will decide if they accept aproposal or not.• Start with all men free;While there exists a free man who has not proposed to all women

pick such a free man m and have him propose to the highestranking woman w on his list to whom he has not proposed yet;If no one has proposed to w yet

she always accepts and a pair p = (m,w) is formed;Else she is already in a pair p′ = (m′, w);

If m is higher on her preference list than m′

the pair p′ = (m′, w) is deleted;m′ becomes a free man;a new pair p = (m,w) is formed;

Else m is lower on her preference list than m′;the proposal is rejected and m remains free.

COMP3121/3821/9101/9801 10 / 35

Stable Matching Problem: Gale - Shapley algorithm

Claim 1: Algorithm terminates after ≤ n2 rounds of the While loopProof:• In every round of the While loop one man proposes to one woman;• every man can propose to a woman at most once;• thus, every man can make at most n proposals;• there are n men, so in total they can make ≤ n2 proposalsThus the While loop can be executed no more than n2 many times.

Claim 2: Algorithm produces a matching, i.e., every man is eventuallypaired with a woman (and thus also every woman is paired to a man)Proof:• Assume that the while While loop has terminated, but m is still free.• This means that m has already proposed to every woman.• Thus, every woman is paired with a man, because a woman is not

paired with anyone only if no one has made a proposal to her.• But this would mean that n women are paired with all of n men so

m cannot be free. Contradiction!COMP3121/3821/9101/9801 11 / 35

Stable Matching Problem: Gale - Shapley algorithm

Claim 3: The matching produced by the algorithm is stable.Proof: Note that during the While loop:• a woman is paired with men of increasing ranks on her list;• a man is paired with women of decreasing ranks on his list.

Assume now the opposite, that the matching is not stable;thus, there are two pairs p = (m,w) and p′ = (m′, w′) such that:

m prefers w′ over w;w′ prefers m over m′.

Since m prefers w′ over w, he must have proposed to w′ beforeproposing to w;Since he is paired with w, woman w′ must have either:

rejected him because she was already with someone whom sheprefers, ordropped him later after a proposal from someone whom she prefers;

In both cases she would now be with m′ whom she prefers over m.Contradiction!

COMP3121/3821/9101/9801 12 / 35

Puzzles!!!

Why puzzles? It is a fun way to practice problem solving!

Problem : Tom and his wife Mary went to a party where nine morecouples were present.

Not every one knew everyone else, so people who did not knoweach other introduced themselves and shook hands.

People who knew each other from before did not shake hands.

Later that evening Tom got bored, so he walked around and askedall other guests (including his wife) how many hands they hadshaken that evening, and got 19 different answers.

How many hands did Mary shake?

How many hands did Tom shake?

COMP3121/3821/9101/9801 13 / 35

Puzzles!!!

Problem : We are given 27 coins of the same denomination; we knowthat one of them is counterfeit and that it is lighter than the others.Find the counterfeit coin by weighing coins on a pan balance only threetimes.

Note: this method is called “divide-and-conquer”.

COMP3121/3821/9101/9801 14 / 35

Puzzles!!!

Problem : We have nine coins and three of them are heavier than theremaining six. Can you find the heavier coins by weighing coins on apan balance only four times?

How many outcomes of 4 weighings are there in total?

34 = 81

How many ways are there to hide three heavier coins among sixgood coins? (

9

3

)=

9!

3!6!=

7× 8× 9

3!= 84

More ways to hide than the number of all possible weightingoutcomes! Thus, it is impossible to do it!

COMP3121/3821/9101/9801 15 / 35

Puzzles!!!

Problem: Consider a block of 7 X 7 houses:

The inhabitant of each house thinks that all four houses around him(to the left, right, top and bottom) are nicer than his house and wouldlike to move to any of the four. Can you move the inhabitants aroundto make them all happier?

COMP3121/3821/9101/9801 16 / 35

Puzzles!!!

Hint:

COMP3121/3821/9101/9801 17 / 35

Puzzles!!!

Problem: Consider an 8× 8 board with two diagonal squares missing,and an 1× 2 domino:

Can you cover the entire board with 31 such dominoes?

COMP3121/3821/9101/9801 18 / 35

Puzzles!!!

Hint:

COMP3121/3821/9101/9801 19 / 35

Puzzles!!!

Problem: In Elbonia all cities have a circular one-way highwayaround the city; see the map. All streets in the cities are one-way, andthey all start and end on the circular highway (see the map).

• A block is a part of the city that is not intersected by any street.• Design an algorithm that, given a map of a city, finds a block (justone such block, not all such blocks) that can be circumnavigated whilerespecting all one-way signs.(for example, the green block has such property, but not the red one)

COMP3121/3821/9101/9801 20 / 35

Basics revisited: how do we add two numbers?

C C C C C carry

X X X X X first integer

+ X X X X X second integer

-----------

X X X X X X result

adding 3 bits can be done in constant time;

the whole algorithm runs in linear time i.e., O(n) many steps.

can we do it faster than in linear time?

no, because we have to read every bit of the input

no asymptotically faster algorithm

COMP3121/3821/9101/9801 21 / 35

Basics revisited: how do we multiply two numbers?

X X X X <- first input integer

* X X X X <- second input integer

-------

X X X X \

X X X X \ O(n^2) intermediate operations:

X X X X / O(n^2) elementary multiplications

X X X X / + O(n^2) elementary additions

---------------

X X X X X X X X <- result of length 2n

We assume that two X’s can be multiplied in O(1). time (each X

could be a bit or a digit in some other base).Thus the above procedure runs in time O(n2).Can we do it in LINEAR time, like addition?No one knows!“Simple” problems can actually turn out to be difficult!

COMP3121/3821/9101/9801 22 / 35

Can we do multiplication faster than O(n2)?

Let us try a divide-and-conquer algorithm:take our two input numbers A and B, and split them into two halves:

A = A12n2 + A0 XX . . .X︸ ︷︷ ︸XX . . .X︸ ︷︷ ︸

B = B12n2 + B0

n

2

n

2

• A0, B0 - the least significant bits; A1, B1 the most significant bits.• AB can now be calculated as follows:

AB = A1B12n + (A1B0 + B1A0)2

n2 + A0B0 (1)

What we mean is that the product AB can be calculated recursively bythe following program:

COMP3121/3821/9101/9801 23 / 35

1: function Mult(A,B)2: if |A| = |B| = 1 then return AB3: else4: A1 ← MoreSignificantPart(A);5: A0 ← LessSignificantPart(A);6: B1 ← MoreSignificantPart(B);7: B0 ← LessSignificantPart(B);8: X ←Mult(A0,B0);9: Y ←Mult(A0,B1);

10: Z ←Mult(A1,B0);11: W ←Mult(A1,B1);12: return W 2n + (Y + Z) 2n/2 + X13: end if14: end function

COMP3121/3821/9101/9801 24 / 35

How many steps does this algorithm take?

Each multiplication of two n digit numbers is replaced by fourmultiplications of n/2 digit numbers: A1B1, A1B0, B1A0, A0B0,plus we have a linear overhead to shift and add:

T (n) = 4T(n

2

)+ c n (2)

COMP3121/3821/9101/9801 25 / 35

Can we do multiplication faster than O(n2)?

Claim: if T (n) satisfies

T (n) = 4T(n

2

)+ c n (3)

then

T (n) = n2(c + 1)− c n

Proof: By “fast” induction. We assume it is true for bn/2c:

T(n

2

)=(n

2

)2(c + 1)− c

n

2

and prove that it is also true for n:

T (n) = 4T(n2

)+ c n = 4

((n2

)2(c + 1)− n

2 c)

+ c n

= n2(c + 1)− 2c n + c n = n2(c + 1)− c n

COMP3121/3821/9101/9801 26 / 35

Can we do multiplication faster than O(n2)?

Thus, if T (n) satisfies

T (n) = 4T(n

2

)+ c n

then

T (n) = n2(c + 1)− c n = O(n2)

i.e., we gained nothing with our divide-and-conquer!

Some history: In 1952, one of the most famous mathematicians ofthe 20th century, Andrey Kolmogorov, conjectured that you cannotmultiply in less than Ω(n2) elementary operations. In 1960, Karatsuba,then a 23-year-old student, found an algorithm (later it was called“divide and conquer”) that multiplies two n-digit numbers inΘ(nlog2 3) ≈ Θ(n1.58...) elementary steps, thus disproving theconjecture!! Kolmogorov was shocked!

COMP3121/3821/9101/9801 27 / 35

The Karatsuba trick

How did Karatsuba do it??

Take again our two input numbers A and B, and split them into twohalves:

A = A12n2 + A0 XX . . .X︸ ︷︷ ︸XX . . .X︸ ︷︷ ︸

B = B12n2 + B0

n

2

n

2

• AB can now be calculated as follows:

AB = A1B12n + (A1B0 + A0B1)2

n2 + A0B0

= A1B12n + ((A1 + A0)(B1 + B0)−A1B1 −A0B0)2

n2 + A0B0

Thus, the algorithm will look like this:

COMP3121/3821/9101/9801 28 / 35

1: function Mult(A,B)2: if |A| = |B| = 1 then return AB3: else4: A1 ← MoreSignificantPart(A);5: A0 ← LessSignificantPart(A);6: B1 ← MoreSignificantPart(B);7: B0 ← LessSignificantPart(B);8: U ← A0 + A1;9: V ← B0 + B1;

10: X ←Mult(A0,B0);11: W ←Mult(A1,B1);12: Y ←Mult(U,V);13: return W 2n + (Y −X −W ) 2n/2 + X14: end if15: end function

COMP3121/3821/9101/9801 29 / 35

The Karatsuba trick

Since

T (n) = 3T

(n

2

)+ c n

implies

T

(n

2

)= 3T

(n

22

)+ c

n

2

and

T

(n

22

)= 3T

(n

23

)+ c

n

22

. . .

we get

T (n) = 3T

(n

2

)︸ ︷︷ ︸+c n = 3

3T

(n

22

)+ c

n

2︸ ︷︷ ︸+ c n

= 32T

(n

22

)︸ ︷︷ ︸+c

3n

2+ c n = 3

2

3T

(n

23

)+ c

n

22︸ ︷︷ ︸+ c

3n

2+ c n

= 33T

(n

23

)︸ ︷︷ ︸+c

32n

22+ c

3n

2+ c n = 3

3

3T

(n

24

)+ c

n

23︸ ︷︷ ︸+ c

32n

22+ c

3n

2+ c n = . . .

COMP3121/3821/9101/9801 31 / 35

The Karatsuba trick

T (n) = 3T(n2

)+ c n = 3

(3T(

n22

)+ cn

2

)+ c n = 32 T

(n

22

)︸ ︷︷ ︸+c 3n

2 + c n

= 32

3T

(n

23

)+ c

n

22︸ ︷︷ ︸+ c 3n

2 + c n = 33T(

n23

)+ c 32n

22+ c 3n

2 + c n

= 33 T

(n

23

)︸ ︷︷ ︸+c n

(32

22+ 3

2 + 1)

= 33

3T

(n

24

)+ c

n

23︸ ︷︷ ︸+ c n

(32

22+ 3

2 + 1)

= 34T(

n24

)+ c n

(33

23+ 32

22+ 3

2 + 1)

. . .

= 3blog2 ncT

(n

b2log2 nc

)+ c n

((32

)blog2 nc−1 + . . . + 32

22+ 3

2 + 1)

≈ 3log2 nT (1) + c n

(32

)log2 n−1

32−1

= 3log2 nT (1) + 2c n((

32

)log2 n − 1)

COMP3121/3821/9101/9801 32 / 35

The Karatsuba trick

So we got

T (n) ≈ 3log2 nT (1) + 2c n

((3

2

)log2 n

− 1

)

We now use alogb n = nlogb a to get:

T (n) ≈ nlog2 3T (1)+2c n(nlog2

32 − 1

)= nlog2 3T (1)+2c n

(nlog2 3−1 − 1

)= nlog2 3T (1) + 2c nlog2 3 − 2c n

= O(nlog2 3) = O(n1.58...) << n2

Please review the basic properties of logarithms and the asymptoticnotation from the review material (the first item at the class webpageunder “class resources”.)

COMP3121/3821/9101/9801 33 / 35

Next time:

1 Can we multiply large integers faster than O(nlog2 3

)??

2 Can we avoid messy computations like:

T (n) = 3T

(n

2

)+ cn = 3

(3T

( n

22

)+ c

n

2

)+ cn = 3

2T

( n

22

)+ c

3n

2+ cn

= 32(3T

( n

23

)+ c

n

22

)+ c

3n

2+ cn = 3

3T

( n

23

)+ c

32n

22+ c

3n

2+ cn

= 33T

( n

23

)+ cn

32

22+

3

2+ 1

=

= 33(3T

( n

24

)+ c

n

23

)+ cn

32

22+

3

2+ 1

=

= 34T

( n

24

)+ cn

33

23+

32

22+

3

2+ 1

=

. . .

= 3blog2 nc

T

(n

b2log2 nc

)+ cn

( 3

2

)blog2 nc−1+ . . . +

32

22+

3

2+ 1

≈ 3log2 n

T (1) + cn

(32

)log2 n − 1

32− 1

= 3log2 n

T (1) + 2cn

(( 3

2

)log2 n− 1

)

COMP3121/3821/9101/9801 34 / 35

That’s All, Folks!!

COMP3121/3821/9101/9801 35 / 35