Introduction to AlgorithmsAlgorithms •Algorithms are addressed to some audience. Consider: A set...

83
Introduction to Algorithms Introduction Proof By Induction Asymptotic notation Asymptotic Performance

Transcript of Introduction to AlgorithmsAlgorithms •Algorithms are addressed to some audience. Consider: A set...

Page 1: Introduction to AlgorithmsAlgorithms •Algorithms are addressed to some audience. Consider: A set of instructions for building a child’s bicycle. A program for class scheduling.

Introduction to Algorithms

IntroductionProof By InductionAsymptotic notation

Asymptotic Performance

Page 2: Introduction to AlgorithmsAlgorithms •Algorithms are addressed to some audience. Consider: A set of instructions for building a child’s bicycle. A program for class scheduling.

The Course

• Purpose: an introduction to the design and analysis of algorithms§ Not a math course§ Implementation the algorithms!

• Textbook: Introduction to Algorithms, Cormen, Leiserson, Rivest, Stein§ An excellent reference you should own

Page 3: Introduction to AlgorithmsAlgorithms •Algorithms are addressed to some audience. Consider: A set of instructions for building a child’s bicycle. A program for class scheduling.

Textbook

Page 4: Introduction to AlgorithmsAlgorithms •Algorithms are addressed to some audience. Consider: A set of instructions for building a child’s bicycle. A program for class scheduling.

C++ / Data Structure

• C++ Primer• Stack Overflow: https://stackoverflow.com/• https://www.zhihu.com/topic/19591797/top-

answers

Page 5: Introduction to AlgorithmsAlgorithms •Algorithms are addressed to some audience. Consider: A set of instructions for building a child’s bicycle. A program for class scheduling.

Online Judge System

Page 6: Introduction to AlgorithmsAlgorithms •Algorithms are addressed to some audience. Consider: A set of instructions for building a child’s bicycle. A program for class scheduling.

The Course

• Grading policy:§ Homework: 20%§ Programming assignment: 20%§ Final: 60%

§ Course Webpage:§ https://algustc.com/

§ Online Judge:§ https://oj.algustc.com/

Page 7: Introduction to AlgorithmsAlgorithms •Algorithms are addressed to some audience. Consider: A set of instructions for building a child’s bicycle. A program for class scheduling.

Why study algorithms?

• Internet. Web search, packet routing, distributed file sharing, ...• Biology. Human genome project, protein folding, …• Computers. Circuit layout, file system, compilers, …• Computer graphics. Movies, video games, virtual reality, …• Security. Cell phones, e-commerce, voting machines, …• Multimedia. MP3, JPG, DivX, HDTV, face recognition, …• Social networks. Recommendations, news feeds, advertisements, …• Physics. N-body simulation, particle collision simulation, …• ……

Their impact is broad and far-reaching.

Page 8: Introduction to AlgorithmsAlgorithms •Algorithms are addressed to some audience. Consider: A set of instructions for building a child’s bicycle. A program for class scheduling.

Why study algorithms?

“ I will, in fact, claim that the difference between a bad programmer and a good one is whether he considers his code or his data structures more important. Bad programmers worry about the code. Good programmers worry about data structures and their relationships. ”

— Linus Torvalds (creator of Linux)

To become a proficient programmer.

“ Algorithms + Data Structures = Programs. ” — Niklaus Wirth

Page 9: Introduction to AlgorithmsAlgorithms •Algorithms are addressed to some audience. Consider: A set of instructions for building a child’s bicycle. A program for class scheduling.

Why study algorithms?For fun and profit.

Page 10: Introduction to AlgorithmsAlgorithms •Algorithms are addressed to some audience. Consider: A set of instructions for building a child’s bicycle. A program for class scheduling.

Algorithm Definition

• A process or set of rules to be followed in calculations or other problem-solving operations, especially by a computer.

o From Google

• Shackelford, Russell L. in Introduction to Computing and Algorithms –

o“An algorithm is a specification of a behavioral process. It consists of a finite set of instructions that govern behavior step-by-step.”

Page 11: Introduction to AlgorithmsAlgorithms •Algorithms are addressed to some audience. Consider: A set of instructions for building a child’s bicycle. A program for class scheduling.

Notice

• Notice the term finite. Algorithms should lead to an eventual solution.

• Step by step process. Each step should do one logical action.

Page 12: Introduction to AlgorithmsAlgorithms •Algorithms are addressed to some audience. Consider: A set of instructions for building a child’s bicycle. A program for class scheduling.

Algorithms

• Algorithms are addressed to some audience. Consider:§ A set of instructions for building a child’s bicycle.§ A program for class scheduling.§ An algorithm that will run on a computer system to

calculate student GPA’s.

Page 13: Introduction to AlgorithmsAlgorithms •Algorithms are addressed to some audience. Consider: A set of instructions for building a child’s bicycle. A program for class scheduling.

Good vs. Bad Algorithms

• All algorithms will have input, perform a process, and produce output.

• A good algorithm should be:§ Simple - relative§ Complete – account for all inputs & cases§ Correct (Right)§ should have appropriate levels of Abstraction. –

grouping steps into a single module§ Precise§ Mnemonic - SCRAP

Page 14: Introduction to AlgorithmsAlgorithms •Algorithms are addressed to some audience. Consider: A set of instructions for building a child’s bicycle. A program for class scheduling.

Precision

• Precision means that there is only one way to interpret the instruction. Unambiguous

• Words like “maybe”, “sometimes” and “occasionally” have no business in a well developed algorithm.

• Instead of “maybe”, we can specify the exact circumstances in which an action will be carried out.

Page 15: Introduction to AlgorithmsAlgorithms •Algorithms are addressed to some audience. Consider: A set of instructions for building a child’s bicycle. A program for class scheduling.

Simplicity

• Simple can be defined as having no unnecessary steps and no unnecessarycomplexity. (You may lose points if your algorithm contains unnecessary steps)

• Each step of a well developed algorithm should carry out one logical step of the process.§ Avoid something like: “Take 2nd right after you

exit at King Street”

Page 16: Introduction to AlgorithmsAlgorithms •Algorithms are addressed to some audience. Consider: A set of instructions for building a child’s bicycle. A program for class scheduling.

Algorithm Design Process

• Analyze the problem and develop the specification.• Design the solution

§ Test the solution as part of the design steps.• Implement the program (code the program)• Test the program• Validate the program (further extensive testing) to

insure it works under all circumstances.

Page 17: Introduction to AlgorithmsAlgorithms •Algorithms are addressed to some audience. Consider: A set of instructions for building a child’s bicycle. A program for class scheduling.

Induction

• Suppose § S(k) is true for fixed constant k

o Often k = 0§ S(n) ! S(n+1) for all n >= k

• Then S(n) is true for all n >= k

Page 18: Introduction to AlgorithmsAlgorithms •Algorithms are addressed to some audience. Consider: A set of instructions for building a child’s bicycle. A program for class scheduling.

Proof By Induction

• Claim:S(n) is true for all n >= k• Basis:

§ Show formula is true when n = k• Inductive hypothesis:

§ Assume formula is true for an arbitrary n• Step:

§ Show that formula is then true for n+1

Page 19: Introduction to AlgorithmsAlgorithms •Algorithms are addressed to some audience. Consider: A set of instructions for building a child’s bicycle. A program for class scheduling.

Induction Example: Gaussian Closed Form

• Prove 1 + 2 + 3 + … + n = n(n+1)/2§ Basis:

o If n = 0, then 0 = 0(0+1)/2§ Inductive hypothesis:

o Assume 1 + 2 + 3 + … + n = n(n+1)/2§ Step (show true for n+1):

1 + 2 + … + n + n+1 = (1 + 2 + …+ n) + (n+1)= n(n+1)/2 + n+1 = [n(n+1) + 2(n+1)]/2 = (n+1)(n+2)/2 = (n+1)(n+1 + 1)/2

Page 20: Introduction to AlgorithmsAlgorithms •Algorithms are addressed to some audience. Consider: A set of instructions for building a child’s bicycle. A program for class scheduling.

Induction Example:Geometric Closed Form

• Prove a0 + a1 + … + an = (an+1 - 1)/(a - 1) for all a ≠ 1§ Basis: show that a0 = (a0+1 - 1)/(a - 1)

a0 = 1 = (a1 - 1)/(a - 1)§ Inductive hypothesis:

o Assume a0 + a1 + … + an = (an+1 - 1)/(a - 1) § Step (show true for n+1):

a0 + a1 + … + an+1 = a0 + a1 + … + an + an+1

= (an+1 - 1)/(a - 1) + an+1 = (an+1+1 - 1)/(a - 1)

Page 21: Introduction to AlgorithmsAlgorithms •Algorithms are addressed to some audience. Consider: A set of instructions for building a child’s bicycle. A program for class scheduling.

Induction

• We’ve been using weak induction• Strong induction also holds

§ Basis: show S(0)§ Hypothesis: assume S(k) holds for arbitrary k <= n§ Step: Show S(n+1) follows

• Another variation:§ Basis: show S(0), S(1)§ Hypothesis: assume S(n) and S(n+1) are true§ Step: show S(n+2) follows

Page 22: Introduction to AlgorithmsAlgorithms •Algorithms are addressed to some audience. Consider: A set of instructions for building a child’s bicycle. A program for class scheduling.

Asymptotic Performance

• In this course, we care most about asymptotic performance§ How does the algorithm behave as the problem

size gets very large?o Running timeo Memory/storage requirementso Bandwidth/power requirements.

Page 23: Introduction to AlgorithmsAlgorithms •Algorithms are addressed to some audience. Consider: A set of instructions for building a child’s bicycle. A program for class scheduling.

Analysis of Algorithms

• Analysis is performed with respect to a computational model

• We will usually use a generic uniprocessor random-access machine (RAM)§ All memory equally expensive to access§ No concurrent operations§ All reasonable instructions take unit time

o Except, of course, function calls

Page 24: Introduction to AlgorithmsAlgorithms •Algorithms are addressed to some audience. Consider: A set of instructions for building a child’s bicycle. A program for class scheduling.

Reasons to analyze algorithms

• Predict performance• Compare algorithms• Provide guarantees• Understand theoretical basis

Their impact is broad and far-reaching: avoid performance bugs.

Page 25: Introduction to AlgorithmsAlgorithms •Algorithms are addressed to some audience. Consider: A set of instructions for building a child’s bicycle. A program for class scheduling.

Input Size

• Time and space complexity§ This is generally a function of the input size

o E.g., sorting, multiplication§ How we characterize input size depends:

o Sorting: number of input itemso Multiplication: total number of bitso Graph algorithms: number of nodes & edgeso Etc

Page 26: Introduction to AlgorithmsAlgorithms •Algorithms are addressed to some audience. Consider: A set of instructions for building a child’s bicycle. A program for class scheduling.

Running Time

• Number of primitive steps that are executed§ Except for time of executing a function call most

statements roughly require the same amount of time

o y = m * x + bo c = 5 / 9 * (t - 32 )o z = f(x) + g(y)

• We can be more exact if need be

Page 27: Introduction to AlgorithmsAlgorithms •Algorithms are addressed to some audience. Consider: A set of instructions for building a child’s bicycle. A program for class scheduling.

Analysis

• Worst case§ Provides an upper bound on running time§ An absolute guarantee

• Average case§ Provides the expected running time§ Very useful, but treat with care: what is “average”?

o Random (equally likely) inputso Real-life inputs

Page 28: Introduction to AlgorithmsAlgorithms •Algorithms are addressed to some audience. Consider: A set of instructions for building a child’s bicycle. A program for class scheduling.

Some algorithmic successes

• Break down waveform of N samples into components.• Applications: DVD, JPEG, MRI, astrophysics, …• Brute force: 𝑁" steps.• FFT algorithm: Nlog𝑁 steps, enables new technology.

Discrete Fourier transform.

Page 29: Introduction to AlgorithmsAlgorithms •Algorithms are addressed to some audience. Consider: A set of instructions for building a child’s bicycle. A program for class scheduling.

Mathematical models for running time

• Need to analyze program to determine set of operations.• Cost depends on machine, compiler.• Frequency depends on algorithm, input data.

Total running time: sum of cost × frequency for all operations.

In principle, accurate mathematical models are available.

http://product.dangdang.com/1190486604.html

Page 30: Introduction to AlgorithmsAlgorithms •Algorithms are addressed to some audience. Consider: A set of instructions for building a child’s bicycle. A program for class scheduling.

An Example: Insertion Sort

InsertionSort(A, n) {for i = 2 to n {

key = A[i]j = i - 1;while (j > 0) and (A[j] > key) {

A[j+1] = A[j]j = j - 1

}A[j+1] = key

}}

Page 31: Introduction to AlgorithmsAlgorithms •Algorithms are addressed to some audience. Consider: A set of instructions for building a child’s bicycle. A program for class scheduling.

An Example: Insertion Sort

InsertionSort(A, n) {for i = 2 to n {

key = A[i]j = i - 1;while (j > 0) and (A[j] > key) {

A[j+1] = A[j]j = j - 1

}A[j+1] = key

}}

30 10 40 20

1 2 3 4

i = ∅ j = ∅ key = ∅A[j] = ∅ A[j+1] = ∅

Page 32: Introduction to AlgorithmsAlgorithms •Algorithms are addressed to some audience. Consider: A set of instructions for building a child’s bicycle. A program for class scheduling.

An Example: Insertion Sort

InsertionSort(A, n) {for i = 2 to n {

key = A[i]j = i - 1;while (j > 0) and (A[j] > key) {

A[j+1] = A[j]j = j - 1

}A[j+1] = key

}}

30 10 40 20

1 2 3 4

i = 2 j = 1 key = 10A[j] = 30 A[j+1] = 10

Page 33: Introduction to AlgorithmsAlgorithms •Algorithms are addressed to some audience. Consider: A set of instructions for building a child’s bicycle. A program for class scheduling.

An Example: Insertion Sort

InsertionSort(A, n) {for i = 2 to n {

key = A[i]j = i - 1;while (j > 0) and (A[j] > key) {

A[j+1] = A[j]j = j - 1

}A[j+1] = key

}}

30 30 40 20

1 2 3 4

i = 2 j = 1 key = 10A[j] = 30 A[j+1] = 30

Page 34: Introduction to AlgorithmsAlgorithms •Algorithms are addressed to some audience. Consider: A set of instructions for building a child’s bicycle. A program for class scheduling.

An Example: Insertion Sort

InsertionSort(A, n) {for i = 2 to n {

key = A[i]j = i - 1;while (j > 0) and (A[j] > key) {

A[j+1] = A[j]j = j - 1

}A[j+1] = key

}}

30 30 40 20

1 2 3 4

i = 2 j = 1 key = 10A[j] = 30 A[j+1] = 30

Page 35: Introduction to AlgorithmsAlgorithms •Algorithms are addressed to some audience. Consider: A set of instructions for building a child’s bicycle. A program for class scheduling.

An Example: Insertion Sort

InsertionSort(A, n) {for i = 2 to n {

key = A[i]j = i - 1;while (j > 0) and (A[j] > key) {

A[j+1] = A[j]j = j - 1

}A[j+1] = key

}}

30 30 40 20

1 2 3 4

i = 2 j = 0 key = 10A[j] = ∅ A[j+1] = 30

Page 36: Introduction to AlgorithmsAlgorithms •Algorithms are addressed to some audience. Consider: A set of instructions for building a child’s bicycle. A program for class scheduling.

An Example: Insertion Sort

InsertionSort(A, n) {for i = 2 to n {

key = A[i]j = i - 1;while (j > 0) and (A[j] > key) {

A[j+1] = A[j]j = j - 1

}A[j+1] = key

}}

30 30 40 20

1 2 3 4

i = 2 j = 0 key = 10A[j] = ∅ A[j+1] = 30

Page 37: Introduction to AlgorithmsAlgorithms •Algorithms are addressed to some audience. Consider: A set of instructions for building a child’s bicycle. A program for class scheduling.

An Example: Insertion Sort

InsertionSort(A, n) {for i = 2 to n {

key = A[i]j = i - 1;while (j > 0) and (A[j] > key) {

A[j+1] = A[j]j = j - 1

}A[j+1] = key

}}

10 30 40 20

1 2 3 4

i = 2 j = 0 key = 10A[j] = ∅ A[j+1] = 10

Page 38: Introduction to AlgorithmsAlgorithms •Algorithms are addressed to some audience. Consider: A set of instructions for building a child’s bicycle. A program for class scheduling.

An Example: Insertion Sort

InsertionSort(A, n) {for i = 2 to n {

key = A[i]j = i - 1;while (j > 0) and (A[j] > key) {

A[j+1] = A[j]j = j - 1

}A[j+1] = key

}}

10 30 40 20

1 2 3 4

i = 3 j = 0 key = 10A[j] = ∅ A[j+1] = 10

Page 39: Introduction to AlgorithmsAlgorithms •Algorithms are addressed to some audience. Consider: A set of instructions for building a child’s bicycle. A program for class scheduling.

An Example: Insertion Sort

InsertionSort(A, n) {for i = 2 to n {

key = A[i]j = i - 1;while (j > 0) and (A[j] > key) {

A[j+1] = A[j]j = j - 1

}A[j+1] = key

}}

10 30 40 20

1 2 3 4

i = 3 j = 0 key = 40A[j] = ∅ A[j+1] = 10

Page 40: Introduction to AlgorithmsAlgorithms •Algorithms are addressed to some audience. Consider: A set of instructions for building a child’s bicycle. A program for class scheduling.

An Example: Insertion Sort

InsertionSort(A, n) {for i = 2 to n {

key = A[i]j = i - 1;while (j > 0) and (A[j] > key) {

A[j+1] = A[j]j = j - 1

}A[j+1] = key

}}

10 30 40 20

1 2 3 4

i = 3 j = 0 key = 40A[j] = ∅ A[j+1] = 10

Page 41: Introduction to AlgorithmsAlgorithms •Algorithms are addressed to some audience. Consider: A set of instructions for building a child’s bicycle. A program for class scheduling.

An Example: Insertion Sort

InsertionSort(A, n) {for i = 2 to n {

key = A[i]j = i - 1;while (j > 0) and (A[j] > key) {

A[j+1] = A[j]j = j - 1

}A[j+1] = key

}}

10 30 40 20

1 2 3 4

i = 3 j = 2 key = 40A[j] = 30 A[j+1] = 40

Page 42: Introduction to AlgorithmsAlgorithms •Algorithms are addressed to some audience. Consider: A set of instructions for building a child’s bicycle. A program for class scheduling.

An Example: Insertion Sort

InsertionSort(A, n) {for i = 2 to n {

key = A[i]j = i - 1;while (j > 0) and (A[j] > key) {

A[j+1] = A[j]j = j - 1

}A[j+1] = key

}}

10 30 40 20

1 2 3 4

i = 3 j = 2 key = 40A[j] = 30 A[j+1] = 40

Page 43: Introduction to AlgorithmsAlgorithms •Algorithms are addressed to some audience. Consider: A set of instructions for building a child’s bicycle. A program for class scheduling.

An Example: Insertion Sort

InsertionSort(A, n) {for i = 2 to n {

key = A[i]j = i - 1;while (j > 0) and (A[j] > key) {

A[j+1] = A[j]j = j - 1

}A[j+1] = key

}}

10 30 40 20

1 2 3 4

i = 3 j = 2 key = 40A[j] = 30 A[j+1] = 40

Page 44: Introduction to AlgorithmsAlgorithms •Algorithms are addressed to some audience. Consider: A set of instructions for building a child’s bicycle. A program for class scheduling.

An Example: Insertion Sort

InsertionSort(A, n) {for i = 2 to n {

key = A[i]j = i - 1;while (j > 0) and (A[j] > key) {

A[j+1] = A[j]j = j - 1

}A[j+1] = key

}}

10 30 40 20

1 2 3 4

i = 4 j = 2 key = 40A[j] = 30 A[j+1] = 40

Page 45: Introduction to AlgorithmsAlgorithms •Algorithms are addressed to some audience. Consider: A set of instructions for building a child’s bicycle. A program for class scheduling.

An Example: Insertion Sort

InsertionSort(A, n) {for i = 2 to n {

key = A[i]j = i - 1;while (j > 0) and (A[j] > key) {

A[j+1] = A[j]j = j - 1

}A[j+1] = key

}}

10 30 40 20

1 2 3 4

i = 4 j = 2 key = 20A[j] = 30 A[j+1] = 40

Page 46: Introduction to AlgorithmsAlgorithms •Algorithms are addressed to some audience. Consider: A set of instructions for building a child’s bicycle. A program for class scheduling.

An Example: Insertion Sort

InsertionSort(A, n) {for i = 2 to n {

key = A[i]j = i - 1;while (j > 0) and (A[j] > key) {

A[j+1] = A[j]j = j - 1

}A[j+1] = key

}}

10 30 40 20

1 2 3 4

i = 4 j = 2 key = 20A[j] = 30 A[j+1] = 40

Page 47: Introduction to AlgorithmsAlgorithms •Algorithms are addressed to some audience. Consider: A set of instructions for building a child’s bicycle. A program for class scheduling.

An Example: Insertion Sort

InsertionSort(A, n) {for i = 2 to n {

key = A[i]j = i - 1;while (j > 0) and (A[j] > key) {

A[j+1] = A[j]j = j - 1

}A[j+1] = key

}}

10 30 40 20

1 2 3 4

i = 4 j = 3 key = 20A[j] = 40 A[j+1] = 20

Page 48: Introduction to AlgorithmsAlgorithms •Algorithms are addressed to some audience. Consider: A set of instructions for building a child’s bicycle. A program for class scheduling.

An Example: Insertion Sort

InsertionSort(A, n) {for i = 2 to n {

key = A[i]j = i - 1;while (j > 0) and (A[j] > key) {

A[j+1] = A[j]j = j - 1

}A[j+1] = key

}}

10 30 40 20

1 2 3 4

i = 4 j = 3 key = 20A[j] = 40 A[j+1] = 20

Page 49: Introduction to AlgorithmsAlgorithms •Algorithms are addressed to some audience. Consider: A set of instructions for building a child’s bicycle. A program for class scheduling.

An Example: Insertion Sort

InsertionSort(A, n) {for i = 2 to n {

key = A[i]j = i - 1;while (j > 0) and (A[j] > key) {

A[j+1] = A[j]j = j - 1

}A[j+1] = key

}}

10 30 40 40

1 2 3 4

i = 4 j = 3 key = 20A[j] = 40 A[j+1] = 40

Page 50: Introduction to AlgorithmsAlgorithms •Algorithms are addressed to some audience. Consider: A set of instructions for building a child’s bicycle. A program for class scheduling.

An Example: Insertion Sort

InsertionSort(A, n) {for i = 2 to n {

key = A[i]j = i - 1;while (j > 0) and (A[j] > key) {

A[j+1] = A[j]j = j - 1

}A[j+1] = key

}}

10 30 40 40

1 2 3 4

i = 4 j = 3 key = 20A[j] = 40 A[j+1] = 40

Page 51: Introduction to AlgorithmsAlgorithms •Algorithms are addressed to some audience. Consider: A set of instructions for building a child’s bicycle. A program for class scheduling.

An Example: Insertion Sort

InsertionSort(A, n) {for i = 2 to n {

key = A[i]j = i - 1;while (j > 0) and (A[j] > key) {

A[j+1] = A[j]j = j - 1

}A[j+1] = key

}}

10 30 40 40

1 2 3 4

i = 4 j = 3 key = 20A[j] = 40 A[j+1] = 40

Page 52: Introduction to AlgorithmsAlgorithms •Algorithms are addressed to some audience. Consider: A set of instructions for building a child’s bicycle. A program for class scheduling.

An Example: Insertion Sort

InsertionSort(A, n) {for i = 2 to n {

key = A[i]j = i - 1;while (j > 0) and (A[j] > key) {

A[j+1] = A[j]j = j - 1

}A[j+1] = key

}}

10 30 40 40

1 2 3 4

i = 4 j = 2 key = 20A[j] = 30 A[j+1] = 40

Page 53: Introduction to AlgorithmsAlgorithms •Algorithms are addressed to some audience. Consider: A set of instructions for building a child’s bicycle. A program for class scheduling.

An Example: Insertion Sort

InsertionSort(A, n) {for i = 2 to n {

key = A[i]j = i - 1;while (j > 0) and (A[j] > key) {

A[j+1] = A[j]j = j - 1

}A[j+1] = key

}}

10 30 40 40

1 2 3 4

i = 4 j = 2 key = 20A[j] = 30 A[j+1] = 40

Page 54: Introduction to AlgorithmsAlgorithms •Algorithms are addressed to some audience. Consider: A set of instructions for building a child’s bicycle. A program for class scheduling.

An Example: Insertion Sort

InsertionSort(A, n) {for i = 2 to n {

key = A[i]j = i - 1;while (j > 0) and (A[j] > key) {

A[j+1] = A[j]j = j - 1

}A[j+1] = key

}}

10 30 30 40

1 2 3 4

i = 4 j = 2 key = 20A[j] = 30 A[j+1] = 30

Page 55: Introduction to AlgorithmsAlgorithms •Algorithms are addressed to some audience. Consider: A set of instructions for building a child’s bicycle. A program for class scheduling.

An Example: Insertion Sort

InsertionSort(A, n) {for i = 2 to n {

key = A[i]j = i - 1;while (j > 0) and (A[j] > key) {

A[j+1] = A[j]j = j - 1

}A[j+1] = key

}}

10 30 30 40

1 2 3 4

i = 4 j = 2 key = 20A[j] = 30 A[j+1] = 30

Page 56: Introduction to AlgorithmsAlgorithms •Algorithms are addressed to some audience. Consider: A set of instructions for building a child’s bicycle. A program for class scheduling.

An Example: Insertion Sort

InsertionSort(A, n) {for i = 2 to n {

key = A[i]j = i - 1;while (j > 0) and (A[j] > key) {

A[j+1] = A[j]j = j - 1

}A[j+1] = key

}}

10 30 30 40

1 2 3 4

i = 4 j = 1 key = 20A[j] = 10 A[j+1] = 30

Page 57: Introduction to AlgorithmsAlgorithms •Algorithms are addressed to some audience. Consider: A set of instructions for building a child’s bicycle. A program for class scheduling.

An Example: Insertion Sort

InsertionSort(A, n) {for i = 2 to n {

key = A[i]j = i - 1;while (j > 0) and (A[j] > key) {

A[j+1] = A[j]j = j - 1

}A[j+1] = key

}}

10 30 30 40

1 2 3 4

i = 4 j = 1 key = 20A[j] = 10 A[j+1] = 30

Page 58: Introduction to AlgorithmsAlgorithms •Algorithms are addressed to some audience. Consider: A set of instructions for building a child’s bicycle. A program for class scheduling.

An Example: Insertion Sort

InsertionSort(A, n) {for i = 2 to n {

key = A[i]j = i - 1;while (j > 0) and (A[j] > key) {

A[j+1] = A[j]j = j - 1

}A[j+1] = key

}}

10 20 30 40

1 2 3 4

i = 4 j = 1 key = 20A[j] = 10 A[j+1] = 20

Page 59: Introduction to AlgorithmsAlgorithms •Algorithms are addressed to some audience. Consider: A set of instructions for building a child’s bicycle. A program for class scheduling.

An Example: Insertion Sort

InsertionSort(A, n) {for i = 2 to n {

key = A[i]j = i - 1;while (j > 0) and (A[j] > key) {

A[j+1] = A[j]j = j - 1

}A[j+1] = key

}}

10 20 30 40

1 2 3 4

i = 4 j = 1 key = 20A[j] = 10 A[j+1] = 20

Done!

Page 60: Introduction to AlgorithmsAlgorithms •Algorithms are addressed to some audience. Consider: A set of instructions for building a child’s bicycle. A program for class scheduling.

Animating Insertion Sort

• Check out the Animator, a java applet at:

• https://courses.cs.vt.edu/csonline/Algorithms/Lessons/InsertionCardSort

Page 61: Introduction to AlgorithmsAlgorithms •Algorithms are addressed to some audience. Consider: A set of instructions for building a child’s bicycle. A program for class scheduling.

Insertion Sort

InsertionSort(A, n) {for i = 2 to n {

key = A[i]j = i - 1;while (j > 0) and (A[j] > key) {

A[j+1] = A[j]j = j - 1

}A[j+1] = key

}}

What is the preconditionfor this loop?

Page 62: Introduction to AlgorithmsAlgorithms •Algorithms are addressed to some audience. Consider: A set of instructions for building a child’s bicycle. A program for class scheduling.

Insertion Sort

InsertionSort(A, n) {for i = 2 to n {

key = A[i]j = i - 1;while (j > 0) and (A[j] > key) {

A[j+1] = A[j]j = j - 1

}A[j+1] = key

}}

How many times will this loop execute?

Page 63: Introduction to AlgorithmsAlgorithms •Algorithms are addressed to some audience. Consider: A set of instructions for building a child’s bicycle. A program for class scheduling.

Insertion Sort

Statement EffortInsertionSort(A, n) {

for i = 2 to n { c1nkey = A[i] c2(n-1)j = i - 1; c3(n-1)while (j > 0) and (A[j] > key) { c4T

A[j+1] = A[j] c5(T-(n-1))j = j - 1 c6(T-(n-1))

} 0A[j+1] = key c7(n-1)

} 0}

T = t2 + t3 + … + tn where ti is number of while expression evaluations for the ith

for loop iteration

Page 64: Introduction to AlgorithmsAlgorithms •Algorithms are addressed to some audience. Consider: A set of instructions for building a child’s bicycle. A program for class scheduling.

Analyzing Insertion Sort

• T(n) = c1n + c2(n-1) + c3(n-1) + c4T + c5(T - (n-1)) + c6(T - (n-1)) + c7(n-1)

= c8T + c9n + c10

• What can T be?§ Best case -- inner loop body never executed

o ti = 1 è T(n) is a linear function§ Worst case -- inner loop body executed for all

previous elementso ti = i è T(n) is a quadratic function

§ Average caseo ???

Page 65: Introduction to AlgorithmsAlgorithms •Algorithms are addressed to some audience. Consider: A set of instructions for building a child’s bicycle. A program for class scheduling.

Common order-of-growth classifications

Page 66: Introduction to AlgorithmsAlgorithms •Algorithms are addressed to some audience. Consider: A set of instructions for building a child’s bicycle. A program for class scheduling.

Analysis

• Simplifications§ Ignore actual and abstract statement costs§ Order of growth is the interesting measure:

o Highest-order term is what counts" Remember, we are doing asymptotic analysis" As the input size grows larger it is the high order term that

dominates

Page 67: Introduction to AlgorithmsAlgorithms •Algorithms are addressed to some audience. Consider: A set of instructions for building a child’s bicycle. A program for class scheduling.

Upper Bound Notation

• We say InsertionSort’s run time is O(n2)§ Properly we should say run time is in O(n2)§ Read O as “Big-O” (you’ll also hear it as “order”)

• In general a function§ f(n) is O(g(n)) if there exist positive constants c

and n0 such that f(n) ≤ c×g(n) for all n ≥ n0

• Formally§ O(g(n)) = { f(n): positive constants c and n0

such that f(n) ≤ c×g(n) n ≥ n0 }∃

Page 68: Introduction to AlgorithmsAlgorithms •Algorithms are addressed to some audience. Consider: A set of instructions for building a child’s bicycle. A program for class scheduling.

• Question§ Is InsertionSort O(n3)?§ Is InsertionSort O(n)?

Page 69: Introduction to AlgorithmsAlgorithms •Algorithms are addressed to some audience. Consider: A set of instructions for building a child’s bicycle. A program for class scheduling.

Big O Fact

• A polynomial of degree k is O(nk)• Proof:

§ Suppose f(n) = bknk + bk-1nk-1 + … + b1n + b0o Let ai = | bi |

§ f(n) ≤ aknk + ak-1nk-1 + … + a1n + a0

ki

kk

i

ik cnan

nnan £££ åå

Page 70: Introduction to AlgorithmsAlgorithms •Algorithms are addressed to some audience. Consider: A set of instructions for building a child’s bicycle. A program for class scheduling.

Lower Bound Notation

• We say InsertionSort’s run time is (n)• In general a function

§ f(n) is (g(n)) if positive constants c and n0such that 0 ≤ c×g(n) ≤ f(n) n ≥ n0

• Proof:§ Suppose run time is an + b

o Assume a and b are positive (what if b is negative?)§ an ≤ an + b

Ω

Ω ∃∀

Page 71: Introduction to AlgorithmsAlgorithms •Algorithms are addressed to some audience. Consider: A set of instructions for building a child’s bicycle. A program for class scheduling.

Asymptotic Tight Bound

• A function f(n) is (g(n)) if positive constants c1, c2, and n0 such that

c1 g(n) ≤ f(n) ≤ c2 g(n) n ≥ n0

• Theorem§ f(n) is (g(n)) iff f(n) is both O(g(n)) and (g(n))

Θ ∃

Θ Ω

Page 72: Introduction to AlgorithmsAlgorithms •Algorithms are addressed to some audience. Consider: A set of instructions for building a child’s bicycle. A program for class scheduling.

Commonly-used notations in the theory of algorithms

Page 73: Introduction to AlgorithmsAlgorithms •Algorithms are addressed to some audience. Consider: A set of instructions for building a child’s bicycle. A program for class scheduling.

Practical Complexity

0

250

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20

f(n) = nf(n) = log(n) f(n) = n log(n)f(n) = n 2̂f(n) = n 3̂f(n) = 2 n̂

Page 74: Introduction to AlgorithmsAlgorithms •Algorithms are addressed to some audience. Consider: A set of instructions for building a child’s bicycle. A program for class scheduling.

Practical Complexity

0

500

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20

f(n) = nf(n) = log(n) f(n) = n log(n)f(n) = n 2̂f(n) = n 3̂f(n) = 2 n̂

Page 75: Introduction to AlgorithmsAlgorithms •Algorithms are addressed to some audience. Consider: A set of instructions for building a child’s bicycle. A program for class scheduling.

Practical Complexity

0

1000

1 3 5 7 9 11 13 15 17 19

f(n) = nf(n) = log(n) f(n) = n log(n)f(n) = n 2̂f(n) = n 3̂f(n) = 2 n̂

Page 76: Introduction to AlgorithmsAlgorithms •Algorithms are addressed to some audience. Consider: A set of instructions for building a child’s bicycle. A program for class scheduling.

Practical Complexity

0

1000

2000

3000

4000

5000

1 3 5 7 9 11 13 15 17 19

f(n) = nf(n) = log(n) f(n) = n log(n)f(n) = n 2̂f(n) = n 3̂f(n) = 2 n̂

Page 77: Introduction to AlgorithmsAlgorithms •Algorithms are addressed to some audience. Consider: A set of instructions for building a child’s bicycle. A program for class scheduling.

Practical Complexity

1

10

100

1000

10000

100000

1000000

10000000

1 4 16 64 256 1024 4096 16384 65536

Page 78: Introduction to AlgorithmsAlgorithms •Algorithms are addressed to some audience. Consider: A set of instructions for building a child’s bicycle. A program for class scheduling.

Other Asymptotic Notations

• A function f(n) is o(g(n)) if positive constants c and n0 such that

f(n) < c g(n) n ≥ n0

• A function f(n) is (g(n)) if positive constants c and n0 such that

c g(n) < f(n) n ≥ n0

• Intuitively,§ o() is like < § O() is like ≤

§ () is like > § () is like ≥

§ () is like =

ω ∃

ω ΘΩ

Page 79: Introduction to AlgorithmsAlgorithms •Algorithms are addressed to some audience. Consider: A set of instructions for building a child’s bicycle. A program for class scheduling.

Types of analyses

• Determined by “easiest” input.• Provides a goal for all inputs.

Best case. Lower bound on cost.

• Determined by “most difficult” input.• Provides a guarantee for all inputs.

Worst case. Upper bound on cost.

• Need a model for “random” input.• Provides a way to predict performance.

Average case. Expected cost for random input.

Page 80: Introduction to AlgorithmsAlgorithms •Algorithms are addressed to some audience. Consider: A set of instructions for building a child’s bicycle. A program for class scheduling.

Memory: Basics

Page 81: Introduction to AlgorithmsAlgorithms •Algorithms are addressed to some audience. Consider: A set of instructions for building a child’s bicycle. A program for class scheduling.

Typical memory usage for primitive types and arrays

Page 82: Introduction to AlgorithmsAlgorithms •Algorithms are addressed to some audience. Consider: A set of instructions for building a child’s bicycle. A program for class scheduling.

Typical memory usage summary

Page 83: Introduction to AlgorithmsAlgorithms •Algorithms are addressed to some audience. Consider: A set of instructions for building a child’s bicycle. A program for class scheduling.

Up Next

• Solving recurrences§ Substitution method§ Master theorem