Computer Science 212 Title: Data Structures and Algorithms Instructor: Harry Plantinga Text:...

Computer Science 212 Title: Data Structures and Algorithms Instructor: Harry Plantinga Text: Algorithm Design: Foundations,

Analysis, and Internet Examples by Goodrich and Tamassia

Computer Science 212

Data Structures and Algorithms

The Heart of Computer Science Data structures Study of important algorithms Algorithm analysis Algorithm design techniques

Plus Intelligent systems

Course Information: http://cs.calvin.edu/curriculum/cs/212/Grades on moodle

Why study DS & Algorithms? Some problems are difficult to solve

and good solutions are known Some “solutions” don’t always work Some simple algorithms don’t scale well Data structures and algorithms make

good tools for addressing new problems Fun! Joy! Esthetic beauty!

Place in the curriculum The study of interesting areas of

computer science: upper-level electives 108, 112: basic tools (like algebra to

mathematics) 212: More advanced tools. Introduction

to the science of computing. (A little more mathematical than other core CS courses, but not as rigorous as a real math course.)

Example: Search and Replace Goal: replace one string with another Which version is better?

#!/usr/bin/perlmy $a = shift;my $b = shift;my $input = "";

while (<STDIN>) { $input .= $_; }

while ($input =~ m/$a/) { $input =~ s/$a/$b/; }

#!/usr/bin/perlmy $a = shift;my $b = shift;my $input = "";

while (<STDIN>) { $input .= $_; }

$input =~ s/$a/$b/g;

Empirical Analysis Implement Gather runtime data for various inputs Analyze runtime vs. size

Example: suppose we test swap1 Input sizes: 1, 2, 4, 8, 16, 32, 64, 128, 256 Runtimes: 70, 110, 250, 560, 810, 1750,

6510, 25680, 102050

swap1 runtime analysis

sizeSwap1

(ms)

1 70

2 110

4 250

8 560

16 810

32 1750

64 6510

128 25680

256 102050

sw ap1 runtime

y = 1.555x2 - 0.3518x + 232.72

1

10

100

1000

10000

100000

1000000

1 10 100 1000

Problem s ize (1000s)

Runt

ime

(ms)

swap2 runtime analysissize swap2

4 70

8 50

16 70

32 80

64 100

128 120

256 230

512 380

1024 580

2048 910

4096 1680

8192 3130

16384 6160

swap2 runtimes

0

1000

2000

3000

4000

5000

6000

7000

0 5000 10000 15000 20000

Problem s ize (1000s)

Ru

nti

me

(ms)

Runtime vs. Problem Size

0

20000

40000

60000

80000

100000

120000

0 2000 4000 6000 8000 10000 12000 14000 16000 18000

swap1

swap2

Note that slope gives growth rate of runtimeNote that first few points are inflated by startup costs

Runtime vs. Size, lg-lg plot

y = 0.9175x - 0.2822

y = 1.9577x + 0.957

0

2

4

6

8

10

12

14

16

18

0 2 4 6 8 10 12 14 16

Lg(problem size)

Lg

(ru

nti

me)

Empirical runtime analysis What can we say about the runtimes of

these two algorithms? Could give runtime as a function of input

size, for a particular implementation But would that apply on different hardware? What about a different implementation in

C++ instead of Perl? Is there anything we can say that is true in

general? Is version 2 better than version 1 in

general? Why?

Empirical analysis: conclusions Empirical analyses are implementation-

dependent You have to use representative sample

data You can learn something about the

growth rate from the slope or curve of the runtime graph

Bad algorithms can’t be fixed with faster computers

Big companies (e.g. Microsoft) are not immune to bad algorithms

What’s the runtime?

for i 0 to n 1 dofor j 0 to n 1 do

if A[i] A[j] then A[j] A[i]

Theoretical Analysis

Uses a high-level description of the algorithm instead of an implementation

Characterizes running time as a function of the input size, n.

Takes into account all possible inputs, often analyzing the worst case

Allows us to evaluate the speed of an algorithm independent of the hardware/software environment

Algorithm arrayMax(A, n)Input array A of n integersOutput maximum element of A

currentMax A[0]for i 1 to n 1 do

if A[i] currentMax thencurrentMax A[i]

return currentMax

Example: find max element of an array

Pseudocode (§1.1)

High-level description of an algorithm

More structured than English prose

Less detailed than a program

Preferred notation for describing algorithms

Hides program design issues

Pseudocode Details Control flow

if … then … [else …] while … do … repeat … until … for … do … Indentation replaces

braces Method declaration

Algorithm method (arg [, arg…])

Input …

Output …

Method callvar.method (arg [, arg…])

Return valuereturn expression

Expressions Assignment

(like in Java) Equality testing

(like in Java)n2 Superscripts and

other mathematical formatting allowed

Questions…

Can a program be asymptotically faster on one type of CPU vs another?

Do all CPU instructions take equally long?

The Random Access Machine (RAM) Model

A CPU

An potentially unbounded bank of memory cells, each of which can hold an arbitrary number or character

01

2

Memory cells are numbered and accessing any cell in memory takes unit time.

Primitive Operations

Basic computations performed by an algorithm

Identifiable in pseudocode Largely independent from

any programming language Exact definition not

important (constant number of machine cycles per statement)

Assumed to take a constant amount of time in the RAM model

Examples: Evaluating an

expression Assigning a

value to a variable

Indexing into an array

Calling a method Returning from

a method

Counting Primitive Operations (§1.1)

By inspecting the pseudocode, we can determine the maximum number of primitive operations executed by an algorithm, as a function of the input size

Algorithm arrayMax(A, n)

# operations

currentMax A[0] 2for i 1 to n 1 do 2 n

if A[i] currentMax then 2(n 1)currentMax A[i] 2(n 1)

{ increment counter i } 2(n 1)return currentMax 1

Total 7n 1

Estimating Running Time

Algorithm arrayMax executes 7n 1 primitive operations in the worst case. Define:a = Time taken by the fastest primitive operationb = Time taken by the slowest primitive operation

Let T(n) be worst-case time of arrayMax. Thena (7n 1) T(n) b (7n 1)

Hence, the running time T(n) is bounded by two linear functions

Theoretical analysis: conclusion We can count the number of RAM-

equivalent statements executed as a function of input size

All remaining implementation dependencies amount to multiplicative constants, which we will ignore (But watch out for statements that take

more than constant time, such as $input =~ s/$a/$b/g)

Linear, quadratic, etc. runtime is an intrinsic property of an algorithm

What’s the runtime?int n;

cin >> n;

for (int i=0; i<n; i++)

for (int j=0; j<n; j++)

for (int k=0; k<n; k++) {

cout << "Hello, ";

cout << "greetings, ";

cout << "bonjour, ";

cout << "guten tag, ";

cout << "Здравствуйте, ";

cout << "你好 ";

cout << " world!";

}


cin >> n;

if (n<1000)



for (int k=0; k<n; k++)

cout << "Hello\n";

else



cout << "world!\n";

Function Growth Rates

1E+01E+21E+41E+61E+8

1E+101E+121E+141E+161E+181E+201E+221E+241E+261E+281E+30

1E+0 1E+2 1E+4 1E+6 1E+8 1E+10n

T(n

)

Cubic

Quadratic

Linear

Growth rates of functions:

Linear n Quadratic n2

Cubic n3

In a log-log chart, the slope of the line corresponds to the growth rate of the function

Constant factors

1E+01E+21E+41E+61E+8

1E+101E+121E+141E+161E+181E+201E+221E+241E+26

1E+0 1E+2 1E+4 1E+6 1E+8 1E+10n

T(n

)

Quadratic

Quadratic

Linear

Linear

The growth rate is not affected by

constant factors or lower-order terms

Examples 102n 105 is a linear

function 105n2 108n is a

quadratic function

Asymptotic (big-O) Notation (§1.2)

1

10

100

1,000

10,000

1 10 100 1,000n

3n

2n+10

n

Given functions f(n) and g(n), we say that f(n) is O(g(n)) if there are positive constantsc and n0 such that

f(n) cg(n) for n n0

Example: 2n 10 is O(n) 2n 10 cn (c 2) n 10 n 10(c 2) Pick c 3 and n0 10

Example

1

10

100

1,000

10,000

100,000

1,000,000

1 10 100 1,000n

n^2

100n

10n

n

Example: the function n2 is not O(n)

n2 cn n c The above

inequality cannot be satisfied since c must be a constant

More Big-O Examples

7n-27n-2 is O(n)need c > 0 and n0 1 such that 7n-2 c•n for n n0

this is true for c = 7 and n0 = 1

3n3 + 20n2 + 53n3 + 20n2 + 5 is O(n3)need c > 0 and n0 1 such that 3n3 + 20n2 + 5 c•n3 for n

n0

this is true for c = 4 and n0 = 21 3 log n + log log n3 log n + log log n is O(log n)need c > 0 and n0 1 such that 3 log n + log log n c•log n

for n n0

this is true for c = 4 and n0 = 2

Asymptotic analysis of functions Asymptotic analysis is equivalent to

ignoring multiplicative constants ignoring lower-order terms “for large enough inputs”

Big-O and growth rate Big-O gives an upper bound on the growth

rate of a function Think of it as <= [asymptotically speaking]

Big-O Rules

If is f(n) a polynomial of degree d, then f(n) is O(nd), i.e.,

Drop lower-order terms Drop constant factors

Use the smallest possible class of functions, if possible

Say “2n is O(n)” instead of “2n is O(n2)” (The former is a stronger statement)

Use the simplest expression of the class Say “3n 5 is O(n)” instead of “3n 5 is O(3n)”

Asymptotic Algorithm Analysis

Asymptotic analysis: determine the runtime in big-O notation

To perform the asymptotic analysis Find the worst-case number of primitive operations

executed as a function of the input size Express this function with big-O notation

Example: We determine that algorithm arrayMax executes at

most 7n 1 primitive operations We say that algorithm arrayMax “runs in O(n) time”

Since constant factors and lower-order terms are eventually dropped anyway, we can disregard them when counting primitive operations

Computing Prefix Averages

Runtime analysis example: Two algorithms for prefix averages

The i-th prefix average of an array X is average of the first (i 1) elements of X:

A[i] X[0] X[1] … X[i])/(i+1)

Computing the array A of prefix averages of another array X has applications to financial analysis

0

5

10

15

20

25

30

35

1 2 3 4 5 6 7

X

A

Prefix Averages (Quadratic)

The following algorithm computes prefix averages in quadratic time by applying the definition

Algorithm prefixAverages1(X, n)Input array X of n integersOutput array A of prefix averages of X #operations A new array of n integers nfor i 0 to n 1 do n

s X[0] nfor j 1 to i do 1 2 … (n 1)

s s X[j] 1 2 … (n 1)A[i] s (i 1) n

return A 1

Prefix Averages (Linear) The following algorithm computes prefix averages in

linear time by keeping a running sum

Algorithm prefixAverages2(X, n)Input array X of n integersOutput array A of prefix averages of X #operationsA new array of n integers ns 0 1for i 0 to n 1 do n

s s X[i] nA[i] s (i 1) n

return A 1 Algorithm prefixAverages2 runs in O(n) time

Arithmetic Progression

The running time of prefixAverages1 isO(1 2 …n)

The sum of the first n integers is n(n 1) 2

There is a simple visual proof of this fact

Thus, algorithm prefixAverages1 runs in O(n2) time 0

1

2

3

4

5

6

7

1 2 3 4 5 6


cin >> n;




cout << “Hello world!\n”;

What if the last line is replaced by:

string *s=new string(“Hello world!\n”);

O(n3) runtime

O(n3) time and space

2n3+n2+n+2?


cin >> n;









(n3) + (n3) = (n3)Statements or blocks in sequence: add


cin >> n;


for (int j=n; j>1; j/=2)


Loops: add up cost of each iteration(multiply loop cost by number of iterations

if they all take the same time)

log n iterations of n steps (n log n)


cin >> n;

for (int i=1; i<=n; i++)

for (int j=1; j<=i; j++)


Loops: add up cost of each iteration

1 + 2 + 3 + … + n = n(n+1)/2 = (n2)

What’s the runtime?template <class Item>

void insert(Item a[], int l, int r)

{ int i;

for (i=r; i>l; i--) compexch(a[i-1],a[i]);

for (i=l+2; i<=r; i++)

{ int j=i; Item v=a[i];

while (v<a[j-1])

{ a[j] = a[j-1]; j--; }

a[j] = v;

}

}

properties of logarithms:logb(xy) = logbx + logby

logb (x/y) = logbx - logby

Logb xa = a logb x

logba = logxa/logxb properties of exponentials:

a(b+c) = aba c

abc = (ab)c

ab /ac = a(b-c)

b = a logab

bc = a c*logab

Summations (Sec. 1.3.1) Logarithms and Exponents (Sec. 1.3.2)

Proof techniques (Sec. 1.3.3) Basic probability (Sec. 1.3.4)

Math you need to Review

Relatives of Big-Oh

big-Omega f(n) is (g(n)) if there is a constant c > 0

and an integer constant n0 1 such that

f(n) c•g(n) for n n0

big-Theta f(n) is (g(n)) if there are constants c’ > 0 and c’’ > 0 and

an integer constant n0 1 such that c’•g(n) f(n) c’’•g(n) for n n0

little-o f(n) is o(g(n)) if, for any constant c > 0, there is an integer

constant n0 0 such that f(n) c•g(n) for n n0

little-omega f(n) is (g(n)) if, for any constant c > 0, there is an integer

constant n0 0 such that f(n) c•g(n) for n n0

Intuition for Asymptotic Notation

Big-Oh f(n) is O(g(n)) if f(n) is asymptotically less than or equal to

g(n)big-Omega

f(n) is (g(n)) if f(n) is asymptotically greater than or equal to g(n)

big-Theta f(n) is (g(n)) if f(n) is asymptotically equal to g(n)

little-oh f(n) is o(g(n)) if f(n) is asymptotically strictly less than g(n)

little-omega f(n) is (g(n)) if is asymptotically strictly greater than g(n)

Example Uses of the Relatives of Big-Oh

f(n) is (g(n)) if, for any constant c > 0, there is an integer constant n0 0 such that f(n) c•g(n) for n n0

need 5n02 c•n0 given c, the n0 that satisfies this is n0

c/5 0

5n2 is (n)

f(n) is (g(n)) if there is a constant c > 0 and an integer constant n0 1 such that f(n) c•g(n) for n n0

let c = 1 and n0 = 1

5n2 is (n)

f(n) is (g(n)) if there is a constant c > 0 and an integer constant n0 1 such that f(n) c•g(n) for n n0

let c = 5 and n0 = 1

5n2 is (n2)

Asymptotic Analysis: Review What does it mean to say that an

algorithm has runtime O(n log n)? n: Problem size

Big-O: upper bound over all inputs of size n

“Ignore constant factor” (why?) “as n grows large”

O: like <= for functions (asymptotically speaking): like >=: like =

Asymptotic notation: examples Asymptotic runtime, in terms of O, ,

? Suppose the runtime for a function is

n2 + 2n log n + 40 0.0000001 n2+ 1000000n1.999

n3 + n2 log n n2.0001 + n2 log n 2n+ 100 n2

1.00001n+ 100 n97

Asymptotic comparisons 0.0000001 n2 = O(1000000n1.999 )?

n1.000001 = O(n log n)?

1.0001n = O(n943)?

lg n = (ln n)?

(Evaluate the limit of the quotient of the functions)

No – a polynomial with a higher power dominates one with a lower power

No – all polynomials (n.000001) dominate any polylog (log n)

No – all exponentials dominate any polynomial

Yes – different bases are just a constant factor difference

Estimate the runtime Suppose an algorithm has runtime (n3)

suppose solving a problem of size 1000 takes 10 seconds. How long to solve a problem of size 10000?

Suppose an algorithm has runtime (n log n)

suppose solving a problem of size 1000 takes 10 seconds. How long to solve a problem of size 10000?

runtime 10-8 n3; if n=10000, runtime 10000s = 2.7hr

runtime 10-3 n lg n; if n=10000, runtime 133 secs

Worst vs. average case You might be interested in worst, best, or

average case analysis of an algorithm You can have upper, lower, or tight bounds on

each of those functions. Eg. For each n, some problem instances of

size n have runtime n and some have runtime n2.

Worst case: Best case: Average case:

(n2), (n), (log n), O(n2), O(n3)

(n), (log n), O(n2), (n)

(n), (log n), O(n2), O(n3)

Average case: need to know distribution of inputs

Problem: Extensible Arrays A data structure for a stack that is…

As fast and easy as an array (usually) Not limited in size How?

Extensible Array: Usually works like a normal array When it fills, copy the entire contents into a

new array of twice the size.

Runtime?

Analysis of Extensible Arrays Worst-case runtime for a push() operation? Is that the most useful way of describing the

result?

Amortized analysis: averaging out the cost over many operations.

Amortized runtime of a push() operation? Total runtime of m push operations:

m steps when array doesn’t grow Worst-case cost of grow operations:

1+2+4+8+…+m < 2m Total: O(m) time for m operations.

O(1) amortized runtime per operation

Theoretical and empirical analysis Why do we ignore constants in analyzing

algorithms theoretically? Wouldn’t it be better if we knew the

actual constants? Theoretical and empirical analysis gives

us Theoretical knowledge of asymptotic runtime Empirical knowledge of actual constants for

some implementation

Best of both worlds!

Analyze this…function fun (int n)

if (n<1000) for (i=1; i<=n; i++)

for (j=1; j<=n; j++) for (k=1; k<=n; k++) cout << “Hello world!\n”;

else { for (i=1; i<=n; i++)

j=i; while (j >= 1) j = j / 2; cout << “Hello world!\n”;

if (n%2)for (i=1; i<=n; i++) for (j=1; j<=n; j++) cout << “Hello world!\n”;

}cout << “Have a nice day.\n”;

Computer Science 212 Title: Data Structures and Algorithms Instructor: Harry Plantinga Text:...

Documents

Transcript of Computer Science 212 Title: Data Structures and Algorithms Instructor: Harry Plantinga Text:...