BASICS OF ALGORITHMSECE 650 - REZA BABAEE

GREAT NEWS!

▸ Our recently assigned TA for the course:

▸ Manasa Sandhya G

▸ msgunda@uwaterloo.ca

▸ MEng in AI and ML

▸ 2 years software development experience in India

▸ One of the top students at the last offering of the course (Fall 2019)

LEARNING OBJECTIVES

▸ Illustrate the steps of writing an algorithm to solve a problem and how to analyze its correctness and complexity

▸ Model the efficiency of an algorithm using asymptotic notations

▸ Develop an algorithm using

divide-and-conquer, dynamic programming,

or greedy approaches

▸ Express what is a hard and undecidable

problem Identify the process

of building software

Today’s class

REFERENCES

▸ Chapter 2,3,4, 15, 16 of the 3rd Edition of the CLRS book

▸ Chapter 4 of the 3rd Edition of Introduction to the Theory of Com putation by Michael Spiser

▸ https://www.cs.virginia.edu/~robins/Sipser_2006_Second_Edition_Problems.pdf

FOUNDATIONS

https://www.coulterfamilycounseling.com/building-a-strong-relationship-foundation/

WHAT’S AN ALGORITHM?

▸ “A sequence of computational steps that transform the input into the output”

▸ Example: Sorting Problem

▸ Instance of a problem

▸ Many sorting algorithms

▸ The goal is to develop an algorithm that gives the best correct solution A software build system is the usage of a set of tools for building software applications

EXAMPLE: INSERTION SORT

▸ Idea: exactly the way that you sort a hand of playing cards

1. Start with an empty left hand

2. Remove one card at a time and insert it into the correct position in the left hand

3. To find the correct position for a card: compare it with each of the cards already in the left hand, from right to left

INSERTION SORT DEMO

52

46

13

52

46

13

INSERTION SORT DEMO

52

46

13

24 5 6

13

INSERTION SORT DEMO

24 5 6

13

1 24 5 6

3

Sorted!

INSERTION SORT PSEUDOCODE

ALGORITHM CORRECTNESS

▸ Correctness:

▸ For any input instance, it halts with the correct (expected) output

▸ One way to show the correctness !Loop Invariants:

▸ A variant that is:

1. True prior to the first iteration of the loop (initialization)

2. If it’s true before an iteration of the loop, it remains true before the next iteration (maintenance)

3. When the loop terminates the invariant gives us a useful property that helps show that the algorithm is correct (termination).

IS OUR ALGORITHM, INSERTION SORT, CORRECT?

▸ Loop Invariant:

▸ At the start of each iteration of the for loop of lines 1–8, the subarray A[1..j-1] consists of the elements originally in A[1..j-1], but in sorted order.

▸ On whiteboard:

Initialization

Maintenance

Termination

ALGORITHM EFFICIENCY

▸ Defining efficiency:

▸ Predicting the resources algorithms require

▸ Memory

▸ Communication bandwidth

▸ Computer hardware (e.g. disk)

▸ Most important one: computational time

▸ Identifying the best one…

COMPUTATIONAL TIME

▸ The computational model: RAM (Random-Access Memory)

▸ Instructions executed one by one, no concurrency

▸ The running time is a function of the input size to the algorithm

▸ Size of the input heavily depends on the problem

▸ Calculating the number of steps executed

▸ As machine-independent as possible (line of the pseudocode takes )

ith

ci

IS OUR ALGORITHM, INSERTION SORT, EFFICIENT?

▸ Let’s see…

CHALLENGES

▸ Detailed analysis requires:

▸ Define instructions in RAM model

▸ Calculate cost for each instruction

▸ Add up the costs of the instructions used in an algorithm

▸ Not to abuse the RAM model (sort instruction?!)

▸ Data types in RAM model limits the word size

▸ Grey areas: exponentiation, memory hierarchy

▸ A simple way indicating algorithm’s resource requirements, without getting into details

ALGORITHM ANALYSIS

https://fr.coursera.org/learn/advanced-manufacturing-process-analysis

TECHNIQUES

▸ In analyzing algorithms we use two effective techniques:

▸Worst-case analysis

▸ Asymptotic notations

WORST-CASE ANALYSIS (WCA)

▸ Worst-case running time: longest running time for any input size of n

▸ Why WCA?

▸ Upper-bound for any input

▸ For some algorithms, occurs fairly often (e.g. searching in a database)

▸ Average case is often roughly as bad as the worst case (e.g. in insertion sort tj = j/2)

ASYMPTOTIC NOTATIONS

▸ A concise way to show a set of functions (running time, space, etc.)

▸ Why? To describe the running time T(n) (or other characteristics) of the algorithms expressed as functions of n

▸ Which running time:

▸ Sometimes worst-case

▸ Often no matter what the input (asymptotic notation is useful)

▸ For all notations, every member of f(n) is asymptotically non-negative (f(n) is non-negative for sufficiently large n)

ASYMPTOTIC NOTATIONS - O (BIG O)

▸ Definition…

▸ An asymptotic upper-bound

▸ f(n) = O(g(n)) ⟹ f(n) ∈ O(g(n))

ASYMPTOTIC NOTATIONS - Ω

▸ Definition…

▸ An asymptotic lower-bound

▸ f(n) = Ω(g(n)) ⟹ f(n) ∈ Ω(g(n))

▸ Definition…

▸ f(n) belongs to (g(n)), for sufficiently large n, it is sandwiched between c1g(n) and c2g(n)

▸

▸ Asymptotically tight bound

Θ

f(n) = Θ(g(n)) ⟹ f(n) ∈ Θ(g(n))

ASYMPTOTIC NOTATIONS - Θ

ASYMPTOTIC NOTATIONS - ANALYSIS OF INSERTION SORT

▸ On white-board

ASYMPTOTIC NOTATIONS - RELATIONS

▸ Theorem: for any two functions f(n) and g(n), we have if and only if and (i.e., )

▸ Analogy with real numbers:

▸ is like

▸ is like

▸ is like

f(n) ∈ Θ(g(n)) f(n) ∈ O(g(n))f(n) ∈ Ω(g(n)) O/Ω( f(n)) ⊆ Θ( f(n))

f(n) = O(g(n)) a ≤ b

f(n) = Ω(g(n)) a ≥ b

f(n) = Θ(g(n)) a = b

ASYMPTOTIC NOTATIONS - NOTES

▸ Be careful when interpreting “=“ using asymptotic notations:

▸

▸

▸

▸

▸

▸ and (not tight notations)

n = O(n2)

2n2 + 3n + 1 = 2n2 + Θ(n)

T(n) = 2T(n /2) + Θ(n)n

∑i=1

O(i) = O(1) + O(2) + … + O(n)

2n2 + Θ(n) = Θ(n2)

o ω

08-ANALYSIS-ALGORITHMS

DESIGNING ALGORITHMS

Divide-And-Conquer

https://lizkislik.com/when-leaders-divide-conquer/

DIVIDE-AND-CONQUER

▸ Divide-and-conquer approach:

▸ Divide: Break the problem to similar subproblems (smaller in size)

▸ Conquer: straightforwardly solve the subproblems in a recursive manner

▸ Combine: Combine the solutions to find the solution for the whole problem

▸ Typically recursive algorithms

DIVIDE-AND-CONQUER - EXAMPLE

▸ Take a divide-and-conquer approach for sorting:

▸Divide: Divide the n-element sequence to be sorted into two subsequences of n/2 elements each

▸ Conquer: Sort the two subsequences recursively using merge sort

▸ Combine: Merge the two sorted subsequences to produce the sorted answer

DIVIDE-AND-CONQUER - EXAMPLE DEMO - DIVIDE

52

47

13 2

6

DIVIDE-AND-CONQUER - EXAMPLE DEMO - DIVIDE

52

47

13 2

6

DIVIDE-AND-CONQUER - EXAMPLE DEMO - DIVIDE

52

47

13 2

6

DIVIDE-AND-CONQUER - EXAMPLE DEMO - DIVIDE

52

47

13 2

6

DIVIDE-AND-CONQUER - EXAMPLE DEMO - CONQUER

52

47

13 2

6

DIVIDE-AND-CONQUER - EXAMPLE DEMO - COMBINE

52

47

13 2

6

DIVIDE-AND-CONQUER - EXAMPLE DEMO - COMBINE

24 5

7

1 2 36

DIVIDE-AND-CONQUER - EXAMPLE DEMO - COMBINE

1 2 2 3 4 5 6 7Sorted!

MERGE SORT PSEUDOCODE

MERGE SORT ANALYSIS

▸ Analyzing recursive algorithms:

▸ Recurrence equation: overall running time on a problem of size n in terms of the running time on smaller inputs

▸ Mathematical tools to solve the recurrence and find bounds on algorithm’s performance

T(n) = {Θ(1) if n = 12T( n

2 ) + Θ(n) otherwise.

MASTER THEOREM

▸ A mathematical tool to solve recurrences of the form:

▸

▸ , and is asymptotically positive

▸ Theorem:

▸ Let be constants, let be a function, and let be defined on the nonnegative integers by the recurrence

▸ Then has the following asymptotic bounds

1. If for some constant , then

2. If then

3. If for some constant , and if for some constant and all sufficiently large n,

T(n) = aT(n /b) + f(n)

a ≥ 1, b > 1 f(n)

a ≥ 1, b > 1 f(n) T(n)T(n) = aT(n /b) + f(n)

T(n)

f(n) = O(nlogb a−ϵ) ϵ > 0 T(n) = Θ(nlogb a)

f(n) = Θ(nlogb a) T(n) = Θ(nlogb algn)

f(n) = Ω(nlogb a+ϵ) ϵ > 0 af(n /b) ≤ cf(n)c < 1 T(n) = Θ( f(n))

https://toronto-employmentlawyer.com/break-workplaces/

DESIGNING ALGORITHMSDynamic Programming

https://www.udemy.com/course/master-rubiks-cube-in-just-4-days/

DYNAMIC PROGRAMMING

▸ How the divide-and-conquer method solves a problem?

▸ What if sub-problems overlap (share sub-subproblems)?

▸ Dynamic programming (DP) saves the repeated solutions!

▸ Dynamic programming is typically applied to optimization problems:

▸ Has many solutions, each has a value, find the optimal solution (maximum or minimum)

DYNAMIC PROGRAMMING - EXAMPLE 1

▸ Consider a program that generates the Fibonacci sequence (Fib):

▸ 1,1,2,3,5,8,13,21,34,55,89,144,233,377,…

DYNAMIC PROGRAMMING - EXAMPLE 1

▸ Write a program that given n computes Fib(n).

https://en.wikibooks.org/wiki/Algorithms/Dynamic_Programming

DYNAMIC PROGRAMMING - ELEMENTS

▸ When use dynamic programming?

▸ Optimal substructure

- Optimal solutions to a problem incorporate optimal solutions to related subproblems, which we may solve independently.

‣ Overlapping subproblems

‣ Steps:

1. Show that a solution to the problem consists of making a choice leaving one or more subproblems to be solved

2. Assume that for a given problem, you are given the choice that leads to an optimal solution

3. Given this choice, determine which subproblems ensue and how to best characterize the resulting space of subproblems

4. Show that the solutions to the subproblems used within an optimal solution to the problem must themselves be optimal by using a “cut-and-paste” technique.

DYNAMIC PROGRAMMING - EXAMPLE 2

▸ Activity-Selection Problem (ASP):

▸ A set of activities

▸ Each activity has a start time and a finish time , , takes place (suppose monotonically

ordered finish time)

▸ and are mutually compatible if their intervals do not overlap (?)

▸ Select a maximum-size subset of mutually compatible activities

S = {a1, a2, …, an} n

ai si fi0 ≤ si < fi < ∞ ai [si, fi)

ai aj

DYNAMIC PROGRAMMING - EXAMPLE 2 - DEMO

i 1 2 3 4 5 6 7 8 9 10 11si 1 3 0 5 3 5 6 8 8 2 12

fi 4 5 6 7 9 9 10 11 12 14 16

* Mutually compatible? * Maximum?

a1

a2

a3

a4

a5

a6

a7

a8

a9

a10

a11

a3

a9

a11

a1

a4

a8

a11

a9

Time

DYNAMIC PROGRAMMING - EXAMPLE 2 - APPROACH

▸ : activities start after and finish before

▸ : max set of mutually compatible activities in

▸ Some activity ! two subproblems: and

▸ and

▸

▸

▸ Does it have optimal substructure?

Sij ai aj

Aij Sij

ak Sik Skj

Aik = Aij ∩ Sik Akj = Aij ∩ Skj

Aij = Aik ∪ {ak} ∪ Akj

|Aij | = ?

DYNAMIC PROGRAMMING - EXAMPLE 2 - APPROACH - CONT.

▸

▸ To find :

▸ Bottom-up DP approach

▸

c[i, j] = c[i, k] + 1 + c[k, j]

k

c[i, j] =0 if Sij = ∅maxak∈Sij

(c[i, k] + c[k, j] + 1) if Sij ≠ ∅

A MORE STRAIGHTFORWARD SOLUTION FOR ASP?

▸ What if we do not solve all subproblems, only select the best solution at the time:

▸ Choose the activity in with the earliest finish time, ! why?

▸ Then, solve the only one remaining subproblem ! finding activities that start after the first selection, , finishes ! why?

▸ Optimal substructure: if is in the optimal solution for a subproblem, then an optimal solution to the original problem consists (same as the remaining subproblem)

S a1

a1

a1

a1

DESIGNING ALGORITHMSGreedy Approach

https://www.which.co.uk/news/2018/03/with-the-future-of-1p-and-2p-coins-under-scrutiny-is-your-small-change-worth-a-fortune/

GREEDY APPROACH

▸ Obtains an optimal solution to a problem by making a sequence of locally optimum choices

▸ Does it produce always an optimum solution (like ASP)?

▸ Two requirements:

- Greedy-choice property

- Optimal substructure

GREEDY APPROACH - GREEDY-CHOICE PROPERTY

▸ Greedy-Choice Property (GCP):

- we can assemble a globally optimal solution by making locally optimal (greedy) choices

‣ Differing from dynamic-programming:

- Choice depends on solutions to subproblems (bottom-up)

‣ May depend on choices so far, but not on any future choices ! making choice before solving subproblems! (top-down)

‣ GCP should be demonstrated for a problem (via a theorem and its proof)

‣ If input is preprocessed properly, the greedy approach is efficient!

GREEDY APPROACH - GCP FOR ASP

▸ Theorem: Consider any nonempty subproblem , and let be an activity in with the earliest finish time. Then is included in some maximum-size subset of mutually compatible activities of

▸ Proof on white-board

Sk amSk am

Sk

GREEDY APPROACH - OPTIMAL SUBSTRUCTURE

▸ An optimal solution to the subproblem, combined with the greedy choice already made, yields an optimal solution to the original problem

▸ Like using induction on the subproblems to prove that making the greedy choice at every step produces an optimal solution

GREEDY APPROACH - ELEMENTS

▸ Steps:

1. Cast the optimization problem as one in which we make a choice and are left with one subproblem to solve

2. Prove that there is always an optimal solution to the original problem that makes the greedy choice, so that the greedy choice is always safe

3. Demonstrate optimal substructure by showing that, having made the greedy choice, what remains is a subproblem with the property that if we combine an optimal solution to the subproblem with the greedy choice we have made, we arrive at an optimal solution to the original problem

GREEDY APPROACH - EXAMPLE

▸ Write the pseudocode for a greedy approach for ASP

- Supposing activities are increasingly sorted by their finish time (why?) ! if not?

- Analysis?

‣ We do not need to solve all subproblems to find the best (in ASP?)

‣ We do not need any bottom-up solution ! greedy strategy is top-down: make a choice and then solve a subproblem (in ASP?)

GREEDY APPROACH - EXAMPLE - DEMO

i 1 2 3 4 5 6 7 8 9 10 11si 1 3 0 5 3 5 6 8 8 2 12

fi 4 5 6 7 9 9 10 11 12 14 16

m sm fm

a0

a0

a1

0 - 0

1 1 4m = 1

(s, f, 0, 11)

2 3 5 a2

a1

(s, f, 1, 11)

GREEDY APPROACH - EXAMPLE - DEMO

m sm fm

2 3 5

a2

a1

(s, f, 1, 11)3 0 6 a1

a3

4 5 7 a1

a4

m = 4

5 3 9 (s, f, 4, 11)a1 a4

a5

GREEDY APPROACH - EXAMPLE - DEMO

m sm fm

5 3 9 (s, f, 4, 11)a1 a4

a5

6 5 9 a1 a4

a6

7 6 10 a1 a4

a7

8 8 11 a1 a4

a8

m = 8

GREEDY APPROACH - EXAMPLE - DEMO

m sm fm

9 8 12 (s, f, 8, 11)

m = 11

a1 a4 a8

a9

10 2 14 a1 a4 a8

a10

11 12 16 a1 a4 a8

a11

a1 a4 a8 a11(s, f, 11, 11)

08-DESIGN-ALGORITHMS

HARD PROBLEMS

https://www.freepik.com/free-photos-vectors/chess-creativity

COMPLEXITY CLASSES

▸ : solvable in polynomial time, i.e., of for some constant and is the size of input (example: sorting, ASP, shortest-path)

▸ : verifiable in polynomial time, i.e., given a solution we can say in polynomial time that the solution to the problem is correct (example: vertex-cover)

▸ Observation:

▸ : it is and as “hard” as any other problem, i.e., solving any problem in polynomial time means every problem can be solved in polynomial time!

P O(nk) kn

NP

P ⊂ NP

NP − Complete NP NPNP − Complete

NP − Complete

NP − Completeness

▸ Showing a problem is :

▸ Reduction

▸ Example: vertex-cover problem and 3-CNF

▸ Showing using reduction

NP − Complete

NP − Completeness

THE MYSTERY OF P ?= NP

▸ One of the deepest most perplexing problems in CS, why?

▸ A polynomial-time solution to an problem is a solution to all the problems!

▸ Two problems similar on the surface, one is in the other is in :

- Shortest vs longest simple paths

- Euler tour vs Hamiltonian cycle

- 2-CNF vs 3-CNF

NP − CompleteNP

PNP

UNDECIDABLE PROBLEMS

http://charlessledge.com/use-incubation-solve-hard-problems/

DECIDABILITY

▸ Question: are there “problems” that regardless of the power of the computation, memory, etc. can not be solved by any algorithm?

▸ Why answering this question?

▸ What do we mean by a computation or an algorithm?

- RAM model

- Turing machine

‣ Decidability:

‣ A problem is said to be decidable if there exists a Turing machine that solves the problem

TURING MACHINE

▸ A mathematical model of computation

UNDECIDABILITY

▸ Back to our question:

▸ Are there “problems” that regardless of the power of the computation, memory, etc. can not be solved by any computer?

▸ Halting problem:

▸ Design a Turing machine that given an “arbitrary” program and an input, tells if the program will run forever or will halt (i.e., terminate)?

▸ Application: we can not have a general-purpose program that verifies all Java programs!

08-COMPLEXITY-ALGORITHMS