Post on 29-Dec-2015
Computer Science 212 Title: Data Structures and Algorithms Instructor: Harry Plantinga Text: Algorithm Design: Foundations,
Analysis, and Internet Examples by Goodrich and Tamassia
Computer Science 212
Data Structures and Algorithms
The Heart of Computer Science Data structures Study of important algorithms Algorithm analysis Algorithm design techniques
Plus Intelligent systems
Course Information: http://cs.calvin.edu/curriculum/cs/212/Grades on moodle
Why study DS & Algorithms? Some problems are difficult to solve
and good solutions are known Some “solutions” don’t always work Some simple algorithms don’t scale well Data structures and algorithms make
good tools for addressing new problems Fun! Joy! Esthetic beauty!
Place in the curriculum The study of interesting areas of
computer science: upper-level electives 108, 112: basic tools (like algebra to
mathematics) 212: More advanced tools. Introduction
to the science of computing. (A little more mathematical than other core CS courses, but not as rigorous as a real math course.)
Example: Search and Replace Goal: replace one string with another Which version is better?
#!/usr/bin/perlmy $a = shift;my $b = shift;my $input = "";
while (<STDIN>) { $input .= $_; }
while ($input =~ m/$a/) { $input =~ s/$a/$b/; }
#!/usr/bin/perlmy $a = shift;my $b = shift;my $input = "";
while (<STDIN>) { $input .= $_; }
$input =~ s/$a/$b/g;
Empirical Analysis Implement Gather runtime data for various inputs Analyze runtime vs. size
Example: suppose we test swap1 Input sizes: 1, 2, 4, 8, 16, 32, 64, 128, 256 Runtimes: 70, 110, 250, 560, 810, 1750,
6510, 25680, 102050
swap1 runtime analysis
sizeSwap1
(ms)
1 70
2 110
4 250
8 560
16 810
32 1750
64 6510
128 25680
256 102050
sw ap1 runtime
y = 1.555x2 - 0.3518x + 232.72
1
10
100
1000
10000
100000
1000000
1 10 100 1000
Problem s ize (1000s)
Runt
ime
(ms)
swap2 runtime analysissize swap2
4 70
8 50
16 70
32 80
64 100
128 120
256 230
512 380
1024 580
2048 910
4096 1680
8192 3130
16384 6160
swap2 runtimes
0
1000
2000
3000
4000
5000
6000
7000
0 5000 10000 15000 20000
Problem s ize (1000s)
Ru
nti
me
(ms)
Runtime vs. Problem Size
0
20000
40000
60000
80000
100000
120000
0 2000 4000 6000 8000 10000 12000 14000 16000 18000
swap1
swap2
Note that slope gives growth rate of runtimeNote that first few points are inflated by startup costs
Runtime vs. Size, lg-lg plot
y = 0.9175x - 0.2822
y = 1.9577x + 0.957
0
2
4
6
8
10
12
14
16
18
0 2 4 6 8 10 12 14 16
Lg(problem size)
Lg
(ru
nti
me)
Empirical runtime analysis What can we say about the runtimes of
these two algorithms? Could give runtime as a function of input
size, for a particular implementation But would that apply on different hardware? What about a different implementation in
C++ instead of Perl? Is there anything we can say that is true in
general? Is version 2 better than version 1 in
general? Why?
Empirical analysis: conclusions Empirical analyses are implementation-
dependent You have to use representative sample
data You can learn something about the
growth rate from the slope or curve of the runtime graph
Bad algorithms can’t be fixed with faster computers
Big companies (e.g. Microsoft) are not immune to bad algorithms
What’s the runtime?
for i 0 to n 1 dofor j 0 to n 1 do
if A[i] A[j] then A[j] A[i]
Theoretical Analysis
Uses a high-level description of the algorithm instead of an implementation
Characterizes running time as a function of the input size, n.
Takes into account all possible inputs, often analyzing the worst case
Allows us to evaluate the speed of an algorithm independent of the hardware/software environment
Algorithm arrayMax(A, n)Input array A of n integersOutput maximum element of A
currentMax A[0]for i 1 to n 1 do
if A[i] currentMax thencurrentMax A[i]
return currentMax
Example: find max element of an array
Pseudocode (§1.1)
High-level description of an algorithm
More structured than English prose
Less detailed than a program
Preferred notation for describing algorithms
Hides program design issues
Pseudocode Details Control flow
if … then … [else …] while … do … repeat … until … for … do … Indentation replaces
braces Method declaration
Algorithm method (arg [, arg…])
Input …
Output …
Method callvar.method (arg [, arg…])
Return valuereturn expression
Expressions Assignment
(like in Java) Equality testing
(like in Java)n2 Superscripts and
other mathematical formatting allowed
Questions…
Can a program be asymptotically faster on one type of CPU vs another?
Do all CPU instructions take equally long?
The Random Access Machine (RAM) Model
A CPU
An potentially unbounded bank of memory cells, each of which can hold an arbitrary number or character
01
2
Memory cells are numbered and accessing any cell in memory takes unit time.
Primitive Operations
Basic computations performed by an algorithm
Identifiable in pseudocode Largely independent from
any programming language Exact definition not
important (constant number of machine cycles per statement)
Assumed to take a constant amount of time in the RAM model
Examples: Evaluating an
expression Assigning a
value to a variable
Indexing into an array
Calling a method Returning from
a method
Counting Primitive Operations (§1.1)
By inspecting the pseudocode, we can determine the maximum number of primitive operations executed by an algorithm, as a function of the input size
Algorithm arrayMax(A, n)
# operations
currentMax A[0] 2for i 1 to n 1 do 2 n
if A[i] currentMax then 2(n 1)currentMax A[i] 2(n 1)
{ increment counter i } 2(n 1)return currentMax 1
Total 7n 1
Estimating Running Time
Algorithm arrayMax executes 7n 1 primitive operations in the worst case. Define:a = Time taken by the fastest primitive operationb = Time taken by the slowest primitive operation
Let T(n) be worst-case time of arrayMax. Thena (7n 1) T(n) b (7n 1)
Hence, the running time T(n) is bounded by two linear functions
Theoretical analysis: conclusion We can count the number of RAM-
equivalent statements executed as a function of input size
All remaining implementation dependencies amount to multiplicative constants, which we will ignore (But watch out for statements that take
more than constant time, such as $input =~ s/$a/$b/g)
Linear, quadratic, etc. runtime is an intrinsic property of an algorithm
What’s the runtime?int n;
cin >> n;
for (int i=0; i<n; i++)
for (int j=0; j<n; j++)
for (int k=0; k<n; k++) {
cout << "Hello, ";
cout << "greetings, ";
cout << "bonjour, ";
cout << "guten tag, ";
cout << "Здравствуйте, ";
cout << "你好 ";
cout << " world!";
}
What’s the runtime?int n;
cin >> n;
if (n<1000)
for (int i=0; i<n; i++)
for (int j=0; j<n; j++)
for (int k=0; k<n; k++)
cout << "Hello\n";
else
for (int j=0; j<n; j++)
for (int k=0; k<n; k++)
cout << "world!\n";
Function Growth Rates
1E+01E+21E+41E+61E+8
1E+101E+121E+141E+161E+181E+201E+221E+241E+261E+281E+30
1E+0 1E+2 1E+4 1E+6 1E+8 1E+10n
T(n
)
Cubic
Quadratic
Linear
Growth rates of functions:
Linear n Quadratic n2
Cubic n3
In a log-log chart, the slope of the line corresponds to the growth rate of the function
Constant factors
1E+01E+21E+41E+61E+8
1E+101E+121E+141E+161E+181E+201E+221E+241E+26
1E+0 1E+2 1E+4 1E+6 1E+8 1E+10n
T(n
)
Quadratic
Quadratic
Linear
Linear
The growth rate is not affected by
constant factors or lower-order terms
Examples 102n 105 is a linear
function 105n2 108n is a
quadratic function
Asymptotic (big-O) Notation (§1.2)
1
10
100
1,000
10,000
1 10 100 1,000n
3n
2n+10
n
Given functions f(n) and g(n), we say that f(n) is O(g(n)) if there are positive constantsc and n0 such that
f(n) cg(n) for n n0
Example: 2n 10 is O(n) 2n 10 cn (c 2) n 10 n 10(c 2) Pick c 3 and n0 10
Example
1
10
100
1,000
10,000
100,000
1,000,000
1 10 100 1,000n
n^2
100n
10n
n
Example: the function n2 is not O(n)
n2 cn n c The above
inequality cannot be satisfied since c must be a constant
More Big-O Examples
7n-27n-2 is O(n)need c > 0 and n0 1 such that 7n-2 c•n for n n0
this is true for c = 7 and n0 = 1
3n3 + 20n2 + 53n3 + 20n2 + 5 is O(n3)need c > 0 and n0 1 such that 3n3 + 20n2 + 5 c•n3 for n
n0
this is true for c = 4 and n0 = 21 3 log n + log log n3 log n + log log n is O(log n)need c > 0 and n0 1 such that 3 log n + log log n c•log n
for n n0
this is true for c = 4 and n0 = 2
Asymptotic analysis of functions Asymptotic analysis is equivalent to
ignoring multiplicative constants ignoring lower-order terms “for large enough inputs”
Big-O and growth rate Big-O gives an upper bound on the growth
rate of a function Think of it as <= [asymptotically speaking]
Big-O Rules
If is f(n) a polynomial of degree d, then f(n) is O(nd), i.e.,
Drop lower-order terms Drop constant factors
Use the smallest possible class of functions, if possible
Say “2n is O(n)” instead of “2n is O(n2)” (The former is a stronger statement)
Use the simplest expression of the class Say “3n 5 is O(n)” instead of “3n 5 is O(3n)”
Asymptotic Algorithm Analysis
Asymptotic analysis: determine the runtime in big-O notation
To perform the asymptotic analysis Find the worst-case number of primitive operations
executed as a function of the input size Express this function with big-O notation
Example: We determine that algorithm arrayMax executes at
most 7n 1 primitive operations We say that algorithm arrayMax “runs in O(n) time”
Since constant factors and lower-order terms are eventually dropped anyway, we can disregard them when counting primitive operations
Computing Prefix Averages
Runtime analysis example: Two algorithms for prefix averages
The i-th prefix average of an array X is average of the first (i 1) elements of X:
A[i] X[0] X[1] … X[i])/(i+1)
Computing the array A of prefix averages of another array X has applications to financial analysis
0
5
10
15
20
25
30
35
1 2 3 4 5 6 7
X
A
Prefix Averages (Quadratic)
The following algorithm computes prefix averages in quadratic time by applying the definition
Algorithm prefixAverages1(X, n)Input array X of n integersOutput array A of prefix averages of X #operations A new array of n integers nfor i 0 to n 1 do n
s X[0] nfor j 1 to i do 1 2 … (n 1)
s s X[j] 1 2 … (n 1)A[i] s (i 1) n
return A 1
Prefix Averages (Linear) The following algorithm computes prefix averages in
linear time by keeping a running sum
Algorithm prefixAverages2(X, n)Input array X of n integersOutput array A of prefix averages of X #operationsA new array of n integers ns 0 1for i 0 to n 1 do n
s s X[i] nA[i] s (i 1) n
return A 1 Algorithm prefixAverages2 runs in O(n) time
Arithmetic Progression
The running time of prefixAverages1 isO(1 2 …n)
The sum of the first n integers is n(n 1) 2
There is a simple visual proof of this fact
Thus, algorithm prefixAverages1 runs in O(n2) time 0
1
2
3
4
5
6
7
1 2 3 4 5 6
What’s the runtime?int n;
cin >> n;
for (int i=0; i<n; i++)
for (int j=0; j<n; j++)
for (int k=0; k<n; k++)
cout << “Hello world!\n”;
What if the last line is replaced by:
string *s=new string(“Hello world!\n”);
O(n3) runtime
O(n3) time and space
2n3+n2+n+2?
What’s the runtime?int n;
cin >> n;
for (int i=0; i<n; i++)
for (int j=0; j<n; j++)
for (int k=0; k<n; k++)
cout << “Hello world!\n”;
for (int i=0; i<n; i++)
for (int j=0; j<n; j++)
for (int k=0; k<n; k++)
cout << “Hello world!\n”;
(n3) + (n3) = (n3)Statements or blocks in sequence: add
What’s the runtime?int n;
cin >> n;
for (int i=0; i<n; i++)
for (int j=n; j>1; j/=2)
cout << “Hello world!\n”;
Loops: add up cost of each iteration(multiply loop cost by number of iterations
if they all take the same time)
log n iterations of n steps (n log n)
What’s the runtime?int n;
cin >> n;
for (int i=1; i<=n; i++)
for (int j=1; j<=i; j++)
cout << “Hello world!\n”;
Loops: add up cost of each iteration
1 + 2 + 3 + … + n = n(n+1)/2 = (n2)
What’s the runtime?template <class Item>
void insert(Item a[], int l, int r)
{ int i;
for (i=r; i>l; i--) compexch(a[i-1],a[i]);
for (i=l+2; i<=r; i++)
{ int j=i; Item v=a[i];
while (v<a[j-1])
{ a[j] = a[j-1]; j--; }
a[j] = v;
}
}
properties of logarithms:logb(xy) = logbx + logby
logb (x/y) = logbx - logby
Logb xa = a logb x
logba = logxa/logxb properties of exponentials:
a(b+c) = aba c
abc = (ab)c
ab /ac = a(b-c)
b = a logab
bc = a c*logab
Summations (Sec. 1.3.1) Logarithms and Exponents (Sec. 1.3.2)
Proof techniques (Sec. 1.3.3) Basic probability (Sec. 1.3.4)
Math you need to Review
Relatives of Big-Oh
big-Omega f(n) is (g(n)) if there is a constant c > 0
and an integer constant n0 1 such that
f(n) c•g(n) for n n0
big-Theta f(n) is (g(n)) if there are constants c’ > 0 and c’’ > 0 and
an integer constant n0 1 such that c’•g(n) f(n) c’’•g(n) for n n0
little-o f(n) is o(g(n)) if, for any constant c > 0, there is an integer
constant n0 0 such that f(n) c•g(n) for n n0
little-omega f(n) is (g(n)) if, for any constant c > 0, there is an integer
constant n0 0 such that f(n) c•g(n) for n n0
Intuition for Asymptotic Notation
Big-Oh f(n) is O(g(n)) if f(n) is asymptotically less than or equal to
g(n)big-Omega
f(n) is (g(n)) if f(n) is asymptotically greater than or equal to g(n)
big-Theta f(n) is (g(n)) if f(n) is asymptotically equal to g(n)
little-oh f(n) is o(g(n)) if f(n) is asymptotically strictly less than g(n)
little-omega f(n) is (g(n)) if is asymptotically strictly greater than g(n)
Example Uses of the Relatives of Big-Oh
f(n) is (g(n)) if, for any constant c > 0, there is an integer constant n0 0 such that f(n) c•g(n) for n n0
need 5n02 c•n0 given c, the n0 that satisfies this is n0
c/5 0
5n2 is (n)
f(n) is (g(n)) if there is a constant c > 0 and an integer constant n0 1 such that f(n) c•g(n) for n n0
let c = 1 and n0 = 1
5n2 is (n)
f(n) is (g(n)) if there is a constant c > 0 and an integer constant n0 1 such that f(n) c•g(n) for n n0
let c = 5 and n0 = 1
5n2 is (n2)
Asymptotic Analysis: Review What does it mean to say that an
algorithm has runtime O(n log n)? n: Problem size
Big-O: upper bound over all inputs of size n
“Ignore constant factor” (why?) “as n grows large”
O: like <= for functions (asymptotically speaking): like >=: like =
Asymptotic notation: examples Asymptotic runtime, in terms of O, ,
? Suppose the runtime for a function is
n2 + 2n log n + 40 0.0000001 n2+ 1000000n1.999
n3 + n2 log n n2.0001 + n2 log n 2n+ 100 n2
1.00001n+ 100 n97
Asymptotic comparisons 0.0000001 n2 = O(1000000n1.999 )?
n1.000001 = O(n log n)?
1.0001n = O(n943)?
lg n = (ln n)?
(Evaluate the limit of the quotient of the functions)
No – a polynomial with a higher power dominates one with a lower power
No – all polynomials (n.000001) dominate any polylog (log n)
No – all exponentials dominate any polynomial
Yes – different bases are just a constant factor difference
Estimate the runtime Suppose an algorithm has runtime (n3)
suppose solving a problem of size 1000 takes 10 seconds. How long to solve a problem of size 10000?
Suppose an algorithm has runtime (n log n)
suppose solving a problem of size 1000 takes 10 seconds. How long to solve a problem of size 10000?
runtime 10-8 n3; if n=10000, runtime 10000s = 2.7hr
runtime 10-3 n lg n; if n=10000, runtime 133 secs
Worst vs. average case You might be interested in worst, best, or
average case analysis of an algorithm You can have upper, lower, or tight bounds on
each of those functions. Eg. For each n, some problem instances of
size n have runtime n and some have runtime n2.
Worst case: Best case: Average case:
(n2), (n), (log n), O(n2), O(n3)
(n), (log n), O(n2), (n)
(n), (log n), O(n2), O(n3)
Average case: need to know distribution of inputs
Problem: Extensible Arrays A data structure for a stack that is…
As fast and easy as an array (usually) Not limited in size How?
Extensible Array: Usually works like a normal array When it fills, copy the entire contents into a
new array of twice the size.
Runtime?
Analysis of Extensible Arrays Worst-case runtime for a push() operation? Is that the most useful way of describing the
result?
Amortized analysis: averaging out the cost over many operations.
Amortized runtime of a push() operation? Total runtime of m push operations:
m steps when array doesn’t grow Worst-case cost of grow operations:
1+2+4+8+…+m < 2m Total: O(m) time for m operations.
O(1) amortized runtime per operation
Theoretical and empirical analysis Why do we ignore constants in analyzing
algorithms theoretically? Wouldn’t it be better if we knew the
actual constants? Theoretical and empirical analysis gives
us Theoretical knowledge of asymptotic runtime Empirical knowledge of actual constants for
some implementation
Best of both worlds!
Analyze this…function fun (int n)
if (n<1000) for (i=1; i<=n; i++)
for (j=1; j<=n; j++) for (k=1; k<=n; k++) cout << “Hello world!\n”;
else { for (i=1; i<=n; i++)
j=i; while (j >= 1) j = j / 2; cout << “Hello world!\n”;
if (n%2)for (i=1; i<=n; i++) for (j=1; j<=n; j++) cout << “Hello world!\n”;
}cout << “Have a nice day.\n”;