CSE 326: Data Structures Introduction & Part One: Complexity Henry Kautz Autumn Quarter 2002.
-
Upload
andrea-miles -
Category
Documents
-
view
220 -
download
0
Transcript of CSE 326: Data Structures Introduction & Part One: Complexity Henry Kautz Autumn Quarter 2002.
![Page 1: CSE 326: Data Structures Introduction & Part One: Complexity Henry Kautz Autumn Quarter 2002.](https://reader035.fdocuments.us/reader035/viewer/2022062304/56649da85503460f94a949d0/html5/thumbnails/1.jpg)
CSE 326: Data Structures Introduction &
Part One: Complexity
Henry Kautz
Autumn Quarter 2002
![Page 2: CSE 326: Data Structures Introduction & Part One: Complexity Henry Kautz Autumn Quarter 2002.](https://reader035.fdocuments.us/reader035/viewer/2022062304/56649da85503460f94a949d0/html5/thumbnails/2.jpg)
Overview of the Quarter• Part One: Complexity
– inductive proofs of program correctness– empirical and asymptotic complexity– order of magnitude notation; logs & series– analyzing recursive programs
• Part Two: List-like data structures• Part Three: Sorting• Part Four: Search Trees• Part Five: Hash Tables• Part Six: Heaps and Union/Find• Part Seven: Graph Algorithms• Part Eight: Advanced Topics
![Page 3: CSE 326: Data Structures Introduction & Part One: Complexity Henry Kautz Autumn Quarter 2002.](https://reader035.fdocuments.us/reader035/viewer/2022062304/56649da85503460f94a949d0/html5/thumbnails/3.jpg)
Material for Part One
• Weiss Chapters 1 and 2• Additional material
– Graphical analysis
– Amortized analysis
– Stretchy arrays
• Any questions on course organization?
![Page 4: CSE 326: Data Structures Introduction & Part One: Complexity Henry Kautz Autumn Quarter 2002.](https://reader035.fdocuments.us/reader035/viewer/2022062304/56649da85503460f94a949d0/html5/thumbnails/4.jpg)
Program Analysis
• Correctness– Testing
– Proofs of correctness
• Efficiency– How to define?
– Asymptotic complexity - how running times scales as function of size of input
![Page 5: CSE 326: Data Structures Introduction & Part One: Complexity Henry Kautz Autumn Quarter 2002.](https://reader035.fdocuments.us/reader035/viewer/2022062304/56649da85503460f94a949d0/html5/thumbnails/5.jpg)
Proving Programs Correct• Often takes the form of an inductive proof• Example: summing an array
int sum(int v[], int n)
{
if (n==0) return 0;
else return v[n-1]+sum(v,n-1);
}
What are the parts of an inductive proof?
![Page 6: CSE 326: Data Structures Introduction & Part One: Complexity Henry Kautz Autumn Quarter 2002.](https://reader035.fdocuments.us/reader035/viewer/2022062304/56649da85503460f94a949d0/html5/thumbnails/6.jpg)
Inductive Proof of Correctnessint sum(int v[], int n)
{
if (n==0) return 0;
else return v[n-1]+sum(v,n-1);
}
Theorem: sum(v,n) correctly returns sum of 1st n elements of array v for any n.
Basis Step: Program is correct for n=0; returns 0.
Inductive Hypothesis (n=k): Assume sum(v,k) returns sum of first k elements of v.
Inductive Step (n=k+1): sum(v,k+1) returns v[k]+sum(v,k), which is the same of the first k+1 elements of v.
![Page 7: CSE 326: Data Structures Introduction & Part One: Complexity Henry Kautz Autumn Quarter 2002.](https://reader035.fdocuments.us/reader035/viewer/2022062304/56649da85503460f94a949d0/html5/thumbnails/7.jpg)
Proof by Contradiction
• Assume negation of goal, show this leads to a contradiction
• Example: there is no program that solves the “halting problem”– Determines if any other program runs forever or not
Alan Turing, 1937
![Page 8: CSE 326: Data Structures Introduction & Part One: Complexity Henry Kautz Autumn Quarter 2002.](https://reader035.fdocuments.us/reader035/viewer/2022062304/56649da85503460f94a949d0/html5/thumbnails/8.jpg)
• Does NonConformist(NonConformist) halt?
• Yes? That means HALT(NonConformist) = “never
halts”
• No? That means HALT(NonConformist) = “halts”
Program NonConformist (Program P)If ( HALT(P) = “never halts” ) Then
Halt Else
Do While (1 > 0)Print “Hello!”
End WhileEnd If
End Program
Contradiction!
![Page 9: CSE 326: Data Structures Introduction & Part One: Complexity Henry Kautz Autumn Quarter 2002.](https://reader035.fdocuments.us/reader035/viewer/2022062304/56649da85503460f94a949d0/html5/thumbnails/9.jpg)
Defining Efficiency
• Asymptotic Complexity - how running time scales as function of size of input
• Why is this a reasonable definition?
![Page 10: CSE 326: Data Structures Introduction & Part One: Complexity Henry Kautz Autumn Quarter 2002.](https://reader035.fdocuments.us/reader035/viewer/2022062304/56649da85503460f94a949d0/html5/thumbnails/10.jpg)
Defining Efficiency
• Asymptotic Complexity - how running time scales as function of size of input
• Why is this a reasonable definition?– Many kinds of small problems can be solved in practice
by almost any approach• E.g., exhaustive enumeration of possible solutions
• Want to focus efficiency concerns on larger problems
– Definition is independent of any possible advances in computer technology
![Page 11: CSE 326: Data Structures Introduction & Part One: Complexity Henry Kautz Autumn Quarter 2002.](https://reader035.fdocuments.us/reader035/viewer/2022062304/56649da85503460f94a949d0/html5/thumbnails/11.jpg)
Technology-Depended Efficiency
• Drum Computers: Popular technology from early 1960’s
• Transistors too costly to use for RAM, so memory was kept on a revolving magnetic drum
• An efficient program scattered instructions on the drum so that next instruction to execute was under read head just when it was needed– Minimized number of full revolutions
of drum during execution
![Page 12: CSE 326: Data Structures Introduction & Part One: Complexity Henry Kautz Autumn Quarter 2002.](https://reader035.fdocuments.us/reader035/viewer/2022062304/56649da85503460f94a949d0/html5/thumbnails/12.jpg)
The Apocalyptic Laptop
Seth Lloyd, SCIENCE, 31 Aug 2000
Speed Energy Consumption
E = m c 2
25 million megawatt-hours
Quantum mechanics:Switching speed = h / (2 * Energy)
h is Planck’s constant
5.4 x 10 50 operations per second
![Page 13: CSE 326: Data Structures Introduction & Part One: Complexity Henry Kautz Autumn Quarter 2002.](https://reader035.fdocuments.us/reader035/viewer/2022062304/56649da85503460f94a949d0/html5/thumbnails/13.jpg)
1
100000
1E+10
1E+15
1E+20
1E+25
1E+30
1E+35
1E+40
1E+45
1E+50
1E+55
1E+60
1 10 100 1000
2^N
1.2^N
N 5̂
N 3̂
5N
Big BangUltimate Laptop,
1 year1 second
1000 MIPS,
since Big Bang
1000 MIPS,
1 day
![Page 14: CSE 326: Data Structures Introduction & Part One: Complexity Henry Kautz Autumn Quarter 2002.](https://reader035.fdocuments.us/reader035/viewer/2022062304/56649da85503460f94a949d0/html5/thumbnails/14.jpg)
Defining Efficiency
• Asymptotic Complexity - how running time scales as function of size of input
• What is “size”?– Often: length (in characters) of input
– Sometimes: value of input (if input is a number)
• Which inputs?– Worst case
• Advantages / disadvantages ?
– Best case• Why?
![Page 15: CSE 326: Data Structures Introduction & Part One: Complexity Henry Kautz Autumn Quarter 2002.](https://reader035.fdocuments.us/reader035/viewer/2022062304/56649da85503460f94a949d0/html5/thumbnails/15.jpg)
Average Case Analysis
• More realistic analysis, first attempt:– Assume inputs are randomly distributed according to
some “realistic” distribution – Compute expected running time
– Drawbacks• Often hard to define realistic random distributions
• Usually hard to perform math
Inputs( )
( , ) Prob ( ) RunTime( )x n
E T n x x
![Page 16: CSE 326: Data Structures Introduction & Part One: Complexity Henry Kautz Autumn Quarter 2002.](https://reader035.fdocuments.us/reader035/viewer/2022062304/56649da85503460f94a949d0/html5/thumbnails/16.jpg)
Amortized Analysis
• Instead of a single input, consider a sequence of inputs
• Choose worst possible sequence• Determine average running time on this sequence• Advantages
– Often less pessimistic than simple worst-case analysis
– Guaranteed results - no assumed distribution
– Usually mathematically easier than average case analysis
![Page 17: CSE 326: Data Structures Introduction & Part One: Complexity Henry Kautz Autumn Quarter 2002.](https://reader035.fdocuments.us/reader035/viewer/2022062304/56649da85503460f94a949d0/html5/thumbnails/17.jpg)
Comparing Runtimes
• Program A is asymptotically less efficient than program B iffthe runtime of A dominates the runtime of B, as the size
of the input goes to infinity
• Note: RunTime can be “worst case”, “best case”, “average case”, “amortized case”
RunTime( , ) as
RunTime( , )
A nn
B n
![Page 18: CSE 326: Data Structures Introduction & Part One: Complexity Henry Kautz Autumn Quarter 2002.](https://reader035.fdocuments.us/reader035/viewer/2022062304/56649da85503460f94a949d0/html5/thumbnails/18.jpg)
Which Function Dominates?
n3 + 2n2
n0.1
n + 100n0.1
5n5
n-152n/100
82log n
100n2 + 1000
log n
2n + 10 log n
n!
1000n15
3n7 + 7n
![Page 19: CSE 326: Data Structures Introduction & Part One: Complexity Henry Kautz Autumn Quarter 2002.](https://reader035.fdocuments.us/reader035/viewer/2022062304/56649da85503460f94a949d0/html5/thumbnails/19.jpg)
Race In3 + 2n2 100n2 + 1000vs.
![Page 20: CSE 326: Data Structures Introduction & Part One: Complexity Henry Kautz Autumn Quarter 2002.](https://reader035.fdocuments.us/reader035/viewer/2022062304/56649da85503460f94a949d0/html5/thumbnails/20.jpg)
Race IIn0.1 log nvs.
![Page 21: CSE 326: Data Structures Introduction & Part One: Complexity Henry Kautz Autumn Quarter 2002.](https://reader035.fdocuments.us/reader035/viewer/2022062304/56649da85503460f94a949d0/html5/thumbnails/21.jpg)
Race IIIn + 100n0.1 2n + 10 log nvs.
![Page 22: CSE 326: Data Structures Introduction & Part One: Complexity Henry Kautz Autumn Quarter 2002.](https://reader035.fdocuments.us/reader035/viewer/2022062304/56649da85503460f94a949d0/html5/thumbnails/22.jpg)
Race IV5n5 n!vs.
![Page 23: CSE 326: Data Structures Introduction & Part One: Complexity Henry Kautz Autumn Quarter 2002.](https://reader035.fdocuments.us/reader035/viewer/2022062304/56649da85503460f94a949d0/html5/thumbnails/23.jpg)
Race Vn-152n/100 1000n15vs.
![Page 24: CSE 326: Data Structures Introduction & Part One: Complexity Henry Kautz Autumn Quarter 2002.](https://reader035.fdocuments.us/reader035/viewer/2022062304/56649da85503460f94a949d0/html5/thumbnails/24.jpg)
Race VI82log(n) 3n7 + 7nvs.
![Page 25: CSE 326: Data Structures Introduction & Part One: Complexity Henry Kautz Autumn Quarter 2002.](https://reader035.fdocuments.us/reader035/viewer/2022062304/56649da85503460f94a949d0/html5/thumbnails/25.jpg)
Order of Magnitude Notation (big O)
• Asymptotic Complexity - how running time scales as function of size of input– We usually only care about order of magnitude of
scaling
• Why?
![Page 26: CSE 326: Data Structures Introduction & Part One: Complexity Henry Kautz Autumn Quarter 2002.](https://reader035.fdocuments.us/reader035/viewer/2022062304/56649da85503460f94a949d0/html5/thumbnails/26.jpg)
Order of Magnitude Notation (big O)
• Asymptotic Complexity - how running time scales as function of size of input– We usually only care about order of magnitude of
scaling
• Why?– As we saw, some functions overwhelm other functions
• So if running time is a sum of terms, can drop dominated terms
– “True” constant factors depend on details of compiler and hardware
• Might as well make constant factor 1
![Page 27: CSE 326: Data Structures Introduction & Part One: Complexity Henry Kautz Autumn Quarter 2002.](https://reader035.fdocuments.us/reader035/viewer/2022062304/56649da85503460f94a949d0/html5/thumbnails/27.jpg)
• Eliminate low order terms
• Eliminate constant coefficients
3 2 28
3 28
3 28
3 28 8
3 3 28 8
3 28
38
38
38
3
16 log (10 ) 100
16 log (10 )
log (10 )
log (10) log ( )
log (10) log ( )
log ( )
2 log ( )
log ( )
log (2) log( )
log( )
n n n
n n
n n
n n
n n n
n n
n n
n n
n n
n n
3 2 2 3816 log (10 ) 100 ( log( ))n n n O n n
![Page 28: CSE 326: Data Structures Introduction & Part One: Complexity Henry Kautz Autumn Quarter 2002.](https://reader035.fdocuments.us/reader035/viewer/2022062304/56649da85503460f94a949d0/html5/thumbnails/28.jpg)
Common NamesSlowest Growth
constant: O(1)
logarithmic: O(log n)
linear: O(n)
log-linear: O(n log n)
quadratic: O(n2)
exponential: O(cn) (c is a constant > 1)
Fastest Growth
superlinear: O(nc) (c is a constant > 1)
polynomial: O(nc) (c is a constant > 0)
![Page 29: CSE 326: Data Structures Introduction & Part One: Complexity Henry Kautz Autumn Quarter 2002.](https://reader035.fdocuments.us/reader035/viewer/2022062304/56649da85503460f94a949d0/html5/thumbnails/29.jpg)
Summary• Proofs by induction and contradiction• Asymptotic complexity• Worst case, best case, average case, and amortized
asymptotic complexity• Dominance of functions• Order of magnitude notation• Next:
– Part One: Complexity, continued– Read Chapters 1 and 2
![Page 30: CSE 326: Data Structures Introduction & Part One: Complexity Henry Kautz Autumn Quarter 2002.](https://reader035.fdocuments.us/reader035/viewer/2022062304/56649da85503460f94a949d0/html5/thumbnails/30.jpg)
Part One: Complexity, continued
Friday, October 4th, 2002
![Page 31: CSE 326: Data Structures Introduction & Part One: Complexity Henry Kautz Autumn Quarter 2002.](https://reader035.fdocuments.us/reader035/viewer/2022062304/56649da85503460f94a949d0/html5/thumbnails/31.jpg)
Determining the Complexity of an Algorithm
• Empirical measurement• Formal analysis (i.e. proofs)• Question: what are likely advantages and
drawbacks of each approach?
![Page 32: CSE 326: Data Structures Introduction & Part One: Complexity Henry Kautz Autumn Quarter 2002.](https://reader035.fdocuments.us/reader035/viewer/2022062304/56649da85503460f94a949d0/html5/thumbnails/32.jpg)
Determining the Complexity of an Algorithm
• Empirical measurement• Formal analysis (i.e. proofs)• Question: what are likely advantages and
drawbacks of each approach?– Empirical:
• pro: discover if constant factors are significant• con: may be running on “wrong” inputs
– Formal:• pro: no interference from implementation/hardware details• con: can make mistake in a proof!
In theory, theory is the same as practice, but not in practice.
![Page 33: CSE 326: Data Structures Introduction & Part One: Complexity Henry Kautz Autumn Quarter 2002.](https://reader035.fdocuments.us/reader035/viewer/2022062304/56649da85503460f94a949d0/html5/thumbnails/33.jpg)
Measuring Empirical Complexity:Linear vs. Binary Search
Linear Search Binary Search
Time to find one item:
Time to find N items:
• Find a item in a sorted array of length N• Binary search algorithm:
![Page 34: CSE 326: Data Structures Introduction & Part One: Complexity Henry Kautz Autumn Quarter 2002.](https://reader035.fdocuments.us/reader035/viewer/2022062304/56649da85503460f94a949d0/html5/thumbnails/34.jpg)
My C Code
void bfind(int x, int a[], int n)
{ m = n / 2;
if (x == a[m]) return;
if (x < a[m])
bfind(x, a, m);
else
bfind(x, &a[m+1], n-m-1); }
for (i=0; i<n; i++) a[i] = i;
for (i=0; i<n; i++) lfind(i,a,n);
void lfind(int x, int a[], int n)
{ for (i=0; i<n; i++)
if (a[i] == x)
return; }
or bfind
![Page 35: CSE 326: Data Structures Introduction & Part One: Complexity Henry Kautz Autumn Quarter 2002.](https://reader035.fdocuments.us/reader035/viewer/2022062304/56649da85503460f94a949d0/html5/thumbnails/35.jpg)
Graphical Analysis
![Page 36: CSE 326: Data Structures Introduction & Part One: Complexity Henry Kautz Autumn Quarter 2002.](https://reader035.fdocuments.us/reader035/viewer/2022062304/56649da85503460f94a949d0/html5/thumbnails/36.jpg)
Graphical Analysis
![Page 37: CSE 326: Data Structures Introduction & Part One: Complexity Henry Kautz Autumn Quarter 2002.](https://reader035.fdocuments.us/reader035/viewer/2022062304/56649da85503460f94a949d0/html5/thumbnails/37.jpg)
![Page 38: CSE 326: Data Structures Introduction & Part One: Complexity Henry Kautz Autumn Quarter 2002.](https://reader035.fdocuments.us/reader035/viewer/2022062304/56649da85503460f94a949d0/html5/thumbnails/38.jpg)
![Page 39: CSE 326: Data Structures Introduction & Part One: Complexity Henry Kautz Autumn Quarter 2002.](https://reader035.fdocuments.us/reader035/viewer/2022062304/56649da85503460f94a949d0/html5/thumbnails/39.jpg)
slope 2
slope 1
![Page 40: CSE 326: Data Structures Introduction & Part One: Complexity Henry Kautz Autumn Quarter 2002.](https://reader035.fdocuments.us/reader035/viewer/2022062304/56649da85503460f94a949d0/html5/thumbnails/40.jpg)
Property of Log/Log Plots
• On a linear plot, a linear function is a straight line• On a log/log plot, any polynomial function is a
straight line!– The slope y/ x is the same as the exponent
Proof: Suppose
Then log log( )
log log log
log log log
k
k
k
y cx
y cx
y c x
y c k x
horizontal axis
vertical axis slope
![Page 41: CSE 326: Data Structures Introduction & Part One: Complexity Henry Kautz Autumn Quarter 2002.](https://reader035.fdocuments.us/reader035/viewer/2022062304/56649da85503460f94a949d0/html5/thumbnails/41.jpg)
slope 1
Why does O(n log n) look like a straight line?
![Page 42: CSE 326: Data Structures Introduction & Part One: Complexity Henry Kautz Autumn Quarter 2002.](https://reader035.fdocuments.us/reader035/viewer/2022062304/56649da85503460f94a949d0/html5/thumbnails/42.jpg)
Summary
• Empirical and formal analyses of runtime scaling are both important techniques in algorithm development
• Large data sets may be required to gain an accurate empirical picture
• Log/log plots provide a fast and simple visual tool for estimating the exponent of a polynomial function
![Page 43: CSE 326: Data Structures Introduction & Part One: Complexity Henry Kautz Autumn Quarter 2002.](https://reader035.fdocuments.us/reader035/viewer/2022062304/56649da85503460f94a949d0/html5/thumbnails/43.jpg)
Formal Asymptotic Analysis
• In order to prove complexity results, we must make the notion of “order of magnitude” more precise
• Asymptotic bounds on runtime– Upper bound
– Lower bound
![Page 44: CSE 326: Data Structures Introduction & Part One: Complexity Henry Kautz Autumn Quarter 2002.](https://reader035.fdocuments.us/reader035/viewer/2022062304/56649da85503460f94a949d0/html5/thumbnails/44.jpg)
Definition of Order Notation• Upper bound: T(n) = O(f(n)) Big-O
Exist constants c and n’ such that
T(n) c f(n) for all n n’
• Lower bound: T(n) = (g(n)) OmegaExist constants c and n’ such that
T(n) c g(n) for all n n’
• Tight bound: T(n) = θ(f(n)) ThetaWhen both hold:
T(n) = O(f(n))
T(n) = (f(n))
![Page 45: CSE 326: Data Structures Introduction & Part One: Complexity Henry Kautz Autumn Quarter 2002.](https://reader035.fdocuments.us/reader035/viewer/2022062304/56649da85503460f94a949d0/html5/thumbnails/45.jpg)
Example: Upper Bound
2
2
2 2
2
2
2
Proof: Must find , such that for all ,
100
Let's try setting 2. Then
100 2
100
100
So we
Claim
can
: 10
set 100 and reverse the steps abov .
(
e
0 )
c n n n
n n cn
c
n n n
n
n n O
n
n
n
n
![Page 46: CSE 326: Data Structures Introduction & Part One: Complexity Henry Kautz Autumn Quarter 2002.](https://reader035.fdocuments.us/reader035/viewer/2022062304/56649da85503460f94a949d0/html5/thumbnails/46.jpg)
Using a Different Pair of Constants2
2 2
2 2
2
Proof: Must find , such that for all ,
100
Let's try setting 101. Then
100 100
100 101
Claim: 100 ( )
(divide both sides by
100 100
1
So we can set 1 and reve
n)
rs
c n n n
n n cn
c
n n n
n n
n
n
n
n n O n
e the steps above.
![Page 47: CSE 326: Data Structures Introduction & Part One: Complexity Henry Kautz Autumn Quarter 2002.](https://reader035.fdocuments.us/reader035/viewer/2022062304/56649da85503460f94a949d0/html5/thumbnails/47.jpg)
Example: Lower Bound2
2
2 2
2 2
2
Proof: Must find , such that for all ,
100
Let's try setting 1. Then
100
0
So we can set 0 and reverse the steps above.
Thus we can also conc
Claim: 100 ( )
100lude
c n n n
n n cn
c
n n n
n
n
n n n
n n
2( )n
![Page 48: CSE 326: Data Structures Introduction & Part One: Complexity Henry Kautz Autumn Quarter 2002.](https://reader035.fdocuments.us/reader035/viewer/2022062304/56649da85503460f94a949d0/html5/thumbnails/48.jpg)
Conventions of Order Notation2 2
2 2
Order notation is not symmetric: write
but never
The expression ( ( )) ( ( )) is equivalent to
( ) ( ( ))
The expression ( ( )) ( ( )) is equivalent to
( ) ( (
( )
2
))
T
(
he right
2
)
-h
O f n O g n
f n O g n
f n g n
f n
n n
g n
O n n n
O n
2
2 2
2 318
and side is a "cruder" version of the l
( ) ( )
18 ( ) (
(2
log ) ( )
f
)
e t:nn O n O n
n n n
O
n n
![Page 49: CSE 326: Data Structures Introduction & Part One: Complexity Henry Kautz Autumn Quarter 2002.](https://reader035.fdocuments.us/reader035/viewer/2022062304/56649da85503460f94a949d0/html5/thumbnails/49.jpg)
Upper/Lower vs. Worst/Best
• Worst case upper bound is f(n)– Guarantee that run time is no more than c f(n)
• Best case upper bound is f(n)– If you are lucky, run time is no more than c f(n)
• Worst case lower bound is g(n)– If you are unlikely, run time is at least c g(n)
• Best case lower bound is g(n)– Guarantee that run time is at least c g(n)
![Page 50: CSE 326: Data Structures Introduction & Part One: Complexity Henry Kautz Autumn Quarter 2002.](https://reader035.fdocuments.us/reader035/viewer/2022062304/56649da85503460f94a949d0/html5/thumbnails/50.jpg)
Analyzing Code
• primitive operations • consecutive statements• function calls• conditionals• loops• recursive functions
![Page 51: CSE 326: Data Structures Introduction & Part One: Complexity Henry Kautz Autumn Quarter 2002.](https://reader035.fdocuments.us/reader035/viewer/2022062304/56649da85503460f94a949d0/html5/thumbnails/51.jpg)
Conditionals
• Conditionalif C then S1 else S2
• Suppose you are doing a O( ) analysis?
• Suppose you are doing a ( ) analysis?
![Page 52: CSE 326: Data Structures Introduction & Part One: Complexity Henry Kautz Autumn Quarter 2002.](https://reader035.fdocuments.us/reader035/viewer/2022062304/56649da85503460f94a949d0/html5/thumbnails/52.jpg)
Conditionals• Conditional
if C then S1 else S2
• Suppose you are doing a O( ) analysis?Time(C) + Max(Time(S1),Time(S2))or Time(C)+Time(S1)+Time(S2)or …
• Suppose you are doing a ( ) analysis?Time(C) + Min(Time(S1),Time(S2))or Time(C)or …
![Page 53: CSE 326: Data Structures Introduction & Part One: Complexity Henry Kautz Autumn Quarter 2002.](https://reader035.fdocuments.us/reader035/viewer/2022062304/56649da85503460f94a949d0/html5/thumbnails/53.jpg)
Nested Loops
for i = 1 to n do
for j = 1 to n do
sum = sum + 1
![Page 54: CSE 326: Data Structures Introduction & Part One: Complexity Henry Kautz Autumn Quarter 2002.](https://reader035.fdocuments.us/reader035/viewer/2022062304/56649da85503460f94a949d0/html5/thumbnails/54.jpg)
Nested Loops
for i = 1 to n do
for j = 1 to n do
sum = sum + 1
2
1 1 1
1n n n
i j i
n n
![Page 55: CSE 326: Data Structures Introduction & Part One: Complexity Henry Kautz Autumn Quarter 2002.](https://reader035.fdocuments.us/reader035/viewer/2022062304/56649da85503460f94a949d0/html5/thumbnails/55.jpg)
Nested Dependent Loops
for i = 1 to n do
for j = i to n do
sum = sum + 1
![Page 56: CSE 326: Data Structures Introduction & Part One: Complexity Henry Kautz Autumn Quarter 2002.](https://reader035.fdocuments.us/reader035/viewer/2022062304/56649da85503460f94a949d0/html5/thumbnails/56.jpg)
Nested Dependent Loops
for i = 1 to n do
for j = i to n do
sum = sum + 1
1
1 ?n n
i j i
![Page 57: CSE 326: Data Structures Introduction & Part One: Complexity Henry Kautz Autumn Quarter 2002.](https://reader035.fdocuments.us/reader035/viewer/2022062304/56649da85503460f94a949d0/html5/thumbnails/57.jpg)
Summary
• Formal definition of order of magnitude notation• Proving upper and lower asymptotic bounds on a
function• Formal analysis of conditionals and simple loops• Next:
– Analyzing complex loops
– Mathematical series
– Analyzing recursive functions
![Page 58: CSE 326: Data Structures Introduction & Part One: Complexity Henry Kautz Autumn Quarter 2002.](https://reader035.fdocuments.us/reader035/viewer/2022062304/56649da85503460f94a949d0/html5/thumbnails/58.jpg)
Part One: Complexity,Continued
Monday October 7, 2002
![Page 59: CSE 326: Data Structures Introduction & Part One: Complexity Henry Kautz Autumn Quarter 2002.](https://reader035.fdocuments.us/reader035/viewer/2022062304/56649da85503460f94a949d0/html5/thumbnails/59.jpg)
Today’s Material
• Running time of nested dependent loops• Mathematical series• Formal analysis of linear search• Formal analysis of binary search • Solving recursive equations• Stretchy arrays and the Stack ADT• Amortized analysis
![Page 60: CSE 326: Data Structures Introduction & Part One: Complexity Henry Kautz Autumn Quarter 2002.](https://reader035.fdocuments.us/reader035/viewer/2022062304/56649da85503460f94a949d0/html5/thumbnails/60.jpg)
Nested Dependent Loops
for i = 1 to n do
for j = i to n do
sum = sum + 1
1
1 ?n n
i j i
![Page 61: CSE 326: Data Structures Introduction & Part One: Complexity Henry Kautz Autumn Quarter 2002.](https://reader035.fdocuments.us/reader035/viewer/2022062304/56649da85503460f94a949d0/html5/thumbnails/61.jpg)
Nested Dependent Loops
for i = 1 to n do
for j = i to n do
sum = sum + 1
1 1
2
1 1 11
1 ( 1)
1
n n n
i j i i
n n n
i i
n
ii
n
i
i
n i n n
![Page 62: CSE 326: Data Structures Introduction & Part One: Complexity Henry Kautz Autumn Quarter 2002.](https://reader035.fdocuments.us/reader035/viewer/2022062304/56649da85503460f94a949d0/html5/thumbnails/62.jpg)
Arithmetic Series
• Note that: S(1) = 1, S(2) = 3, S(3) = 6, S(4) = 10, …• Hypothesis: S(N) = N(N+1)/2
Prove by induction
– Base case: for N = 1, S(N) = 1(2)/2 = 1 – Assume true for N = k– Suppose N = k+1.
– S(k+1) = S(k) + (k+1)= k(k+1)/2 + (k+1) = (k+1)(k/2 + 1)
= (k+1)(k+2)/2.
N
i
iNNS1
?21)(
![Page 63: CSE 326: Data Structures Introduction & Part One: Complexity Henry Kautz Autumn Quarter 2002.](https://reader035.fdocuments.us/reader035/viewer/2022062304/56649da85503460f94a949d0/html5/thumbnails/63.jpg)
Other Important Series
• Sum of squares:
• Sum of exponents:
• Geometric series:
• Novel series: – Reduce to known series, or prove inductively
N largefor 36
)12)(1( 3
1
2 NNNNi
N
i
-1k and N largefor |1|
1
1
k
Ni
kN
i
k
1
11
0
A
AA
NN
i
i
![Page 64: CSE 326: Data Structures Introduction & Part One: Complexity Henry Kautz Autumn Quarter 2002.](https://reader035.fdocuments.us/reader035/viewer/2022062304/56649da85503460f94a949d0/html5/thumbnails/64.jpg)
Nested Dependent Loopsfor i = 1 to n do
for j = i to n do
sum = sum + 1
2 2
2 2
1
( 1)
2
( 1) ( 1)( 1)
2 2
/ 2 / 2 ( )
n
i
n n n n
n n n nn n
in n
n n n
![Page 65: CSE 326: Data Structures Introduction & Part One: Complexity Henry Kautz Autumn Quarter 2002.](https://reader035.fdocuments.us/reader035/viewer/2022062304/56649da85503460f94a949d0/html5/thumbnails/65.jpg)
Linear Search Analysis
• Best case, tight analysis:
• Worst case, tight analysis:
void lfind(int x, int a[], int n)
{ for (i=0; i<n; i++)
if (a[i] == x)
return; }
![Page 66: CSE 326: Data Structures Introduction & Part One: Complexity Henry Kautz Autumn Quarter 2002.](https://reader035.fdocuments.us/reader035/viewer/2022062304/56649da85503460f94a949d0/html5/thumbnails/66.jpg)
Iterated Linear Search Analysis
• Easy worst-case upper-bound:
• Worst-case tight analysis:
for (i=0; i<n; i++) a[i] = i;
for (i=0; i<n; i++) lfind(i,a,n);
![Page 67: CSE 326: Data Structures Introduction & Part One: Complexity Henry Kautz Autumn Quarter 2002.](https://reader035.fdocuments.us/reader035/viewer/2022062304/56649da85503460f94a949d0/html5/thumbnails/67.jpg)
Iterated Linear Search Analysis
• Easy worst-case upper-bound:• Worst-case tight analysis:
– Just multiplying worst case by n does not justify answer, since each time lfind is called i is specified
for (i=0; i<n; i++) a[i] = i;
for (i=0; i<n; i++) lfind(i,a,n);
2
1 1 1
( 1)1 ( )
2
n i n
i j i
n ni n
2( ) ( )nO n O n
![Page 68: CSE 326: Data Structures Introduction & Part One: Complexity Henry Kautz Autumn Quarter 2002.](https://reader035.fdocuments.us/reader035/viewer/2022062304/56649da85503460f94a949d0/html5/thumbnails/68.jpg)
Analyzing Recursive Programs
1. Express the running time T(n) as a recursive equation
2. Solve the recursive equation• For an upper-bound analysis, you can optionally
simplify the equation to something larger
• For a lower-bound analysis, you can optionally simplify the equation to something smaller
![Page 69: CSE 326: Data Structures Introduction & Part One: Complexity Henry Kautz Autumn Quarter 2002.](https://reader035.fdocuments.us/reader035/viewer/2022062304/56649da85503460f94a949d0/html5/thumbnails/69.jpg)
Binary Search
void bfind(int x, int a[], int n)
{ m = n / 2;
if (x == a[m]) return;
if (x < a[m])
bfind(x, a, m);
else
bfind(x, &a[m+1], n-m-1); }
What is the worst-case upper bound?
![Page 70: CSE 326: Data Structures Introduction & Part One: Complexity Henry Kautz Autumn Quarter 2002.](https://reader035.fdocuments.us/reader035/viewer/2022062304/56649da85503460f94a949d0/html5/thumbnails/70.jpg)
Binary Search
void bfind(int x, int a[], int n)
{ m = n / 2;
if (x == a[m]) return;
if (x < a[m])
bfind(x, a, m);
else
bfind(x, &a[m+1], n-m-1); }
What is the worst-case upper bound?
Trick question:
![Page 71: CSE 326: Data Structures Introduction & Part One: Complexity Henry Kautz Autumn Quarter 2002.](https://reader035.fdocuments.us/reader035/viewer/2022062304/56649da85503460f94a949d0/html5/thumbnails/71.jpg)
Binary Searchvoid bfind(int x, int a[], int n)
{ m = n / 2;
if (n <= 1) return;
if (x == a[m]) return;
if (x < a[m])
bfind(x, a, m);
else
bfind(x, &a[m+1], n-m-1); }
Okay, let’s prove it is (log n)…
![Page 72: CSE 326: Data Structures Introduction & Part One: Complexity Henry Kautz Autumn Quarter 2002.](https://reader035.fdocuments.us/reader035/viewer/2022062304/56649da85503460f94a949d0/html5/thumbnails/72.jpg)
Binary Search
Introduce some constants…
b = time needed for base case
c = time needed to get ready to do a recursive call
Running time is thus:
void bfind(int x, int a[], int n)
{ m = n / 2;
if (n <= 1) return;
if (x == a[m]) return;
if (x < a[m])
bfind(x, a, m);
else
bfind(x, &a[m+1], n-m-1); }
(1)
( ) ( / 2)
T b
T n T n c
![Page 73: CSE 326: Data Structures Introduction & Part One: Complexity Henry Kautz Autumn Quarter 2002.](https://reader035.fdocuments.us/reader035/viewer/2022062304/56649da85503460f94a949d0/html5/thumbnails/73.jpg)
Binary Search Analysis
One sub-problem, half as large
Equation: T(1) b
T(n) T(n/2) + c for n>1
Solution:
T(n) T(n/2) + c write equation T(n/4) + c + c expand T(n/8) + c + c + c T(n/2k) + kc inductive leap T(1) + c log n where k = log n select value for k b + c log n = O(log n) simplify
![Page 74: CSE 326: Data Structures Introduction & Part One: Complexity Henry Kautz Autumn Quarter 2002.](https://reader035.fdocuments.us/reader035/viewer/2022062304/56649da85503460f94a949d0/html5/thumbnails/74.jpg)
Solving Recursive Equations by Repeated Substitution
• Somewhat “informal”, but intuitively clear and straightforward
log
substitute for T(n/2)
substitute for T(
( )
( )
( )
( )
( )
( ) log
( ) log
(1) log
n/4)
"inductive leap"
choose k=lo
( / 2)
( /
( / 4)
(
g
4)
( / 2
( / 2 )
( /
)
/ 8)
n
)
k
n
T n
T n
T n c
T n c
T n c
T n
T n c
T n
c
T n c c
T n
T n c n
T n c n
T c n
c
T
T n
kc
n
n
log (log )b c n n
![Page 75: CSE 326: Data Structures Introduction & Part One: Complexity Henry Kautz Autumn Quarter 2002.](https://reader035.fdocuments.us/reader035/viewer/2022062304/56649da85503460f94a949d0/html5/thumbnails/75.jpg)
Solving Recursive Equations by Telescoping
• Create a set of equations, take their sum
( ) ( / 2)
( / 2) ( / 4)
( / 4) ( / 8)
( / 8) ( /1
initial equation
so this holds...
and this...
and this...6)
...
(2) (
and eventually...
sum equations, cancelling
terms that appear on both s
1)
T n T n c
T n T n c
T n T n c
T n T n c
T T c
ides
look famili( ) (1) log
( ) (log )
ar?T n T c n
T n n
![Page 76: CSE 326: Data Structures Introduction & Part One: Complexity Henry Kautz Autumn Quarter 2002.](https://reader035.fdocuments.us/reader035/viewer/2022062304/56649da85503460f94a949d0/html5/thumbnails/76.jpg)
Solving Recursive Equations by Induction
• Repeated substitution and telescoping construct the solution
• If you know the closed form solution, you can validate it by ordinary induction
• For the induction, may want to increase n by a multiple (2n) rather than by n+1
![Page 77: CSE 326: Data Structures Introduction & Part One: Complexity Henry Kautz Autumn Quarter 2002.](https://reader035.fdocuments.us/reader035/viewer/2022062304/56649da85503460f94a949d0/html5/thumbnails/77.jpg)
base case
Assume hypothesis
definition of T(n)
by in
(1) log1
( ) log
(2 ) ( )
(2 ) ( log )
(2 ) ((log ) 1)
(2 ) ((
duction hypothesis
Q.E
log ) (log 2))
(2 ) log(2 ) .D.
Thus: (
T b c b
T n b c n
T n T n c
T n b c n c
T n b c n
T n b c n
T n b c n
T n
) (log )n
Inductive Proof
![Page 78: CSE 326: Data Structures Introduction & Part One: Complexity Henry Kautz Autumn Quarter 2002.](https://reader035.fdocuments.us/reader035/viewer/2022062304/56649da85503460f94a949d0/html5/thumbnails/78.jpg)
Example: Sum of Integer Queue
sum_queue(Q){
if (Q.length() == 0 ) return 0;
else return Q.dequeue() +
sum_queue(Q); }– One subproblem
– Linear reduction in size (decrease by 1)
Equation: T(0) = b
T(n) = c + T(n – 1) for n>0
![Page 79: CSE 326: Data Structures Introduction & Part One: Complexity Henry Kautz Autumn Quarter 2002.](https://reader035.fdocuments.us/reader035/viewer/2022062304/56649da85503460f94a949d0/html5/thumbnails/79.jpg)
Lower Bound Analysis: Recursive Fibonacci
int Fib(n){ if (n == 0 or n == 1) return 1 ;
else return Fib(n - 1) + Fib(n - 2); }
• Lower bound analysis (n)• Instead of =, equations will use
T(n) Some expression
• Will simplify math by throwing out terms on the right-hand side
![Page 80: CSE 326: Data Structures Introduction & Part One: Complexity Henry Kautz Autumn Quarter 2002.](https://reader035.fdocuments.us/reader035/viewer/2022062304/56649da85503460f94a949d0/html5/thumbnails/80.jpg)
Analysis by Repeated Subsitution
( ) 2 ( 2) simplify to smaller quantit
(0) (1)
( ) 2( 2 ( 2 2)) substitute
( ) 3 4 ( 4)) simplify
( ) 3 4( 2 ( 4 2)) substitute
( ) 7
base case
( ) ( 1) ( 2) recursive ca
y
se
8 (
T n b b T n
T n b T n
T n b b T n
T n b T n
T T a
T n b T n T
n b
n
T T
/ 2 / 2
/
2
2
/
( ) (2 1) 2 ( 2 ) inductive
6)) simplify
( ) 7 8( 2 ( 6
( ) (2
2)) substitu
Note: this
leap
1) 2 ( 2( / 2)) choose k=(n
te
( ) 15 16 ( 8) simp
( ) (
/2)
( ) 2
lify
( ) s
2
i p
)
m lify
n
n
k k
n
n
T n b T n
T n b T n n
T n b a
n
T n b b T n
T n
b
n
k
b T n
T
is not the same as (2 )!!!n
![Page 81: CSE 326: Data Structures Introduction & Part One: Complexity Henry Kautz Autumn Quarter 2002.](https://reader035.fdocuments.us/reader035/viewer/2022062304/56649da85503460f94a949d0/html5/thumbnails/81.jpg)
Learning from Analysis
• To avoid recursive calls– store all basis values in a table
– each time you calculate an answer, store it in the table
– before performing any calculation for a value n • check if a valid answer for n is in the table
• if so, return it
• Memoization– a form of dynamic programming
• How much time does memoized version take?
![Page 82: CSE 326: Data Structures Introduction & Part One: Complexity Henry Kautz Autumn Quarter 2002.](https://reader035.fdocuments.us/reader035/viewer/2022062304/56649da85503460f94a949d0/html5/thumbnails/82.jpg)
Amortized Analysis
• Consider any sequence of operations applied to a data structure
• Some operations may be fast, others slow• Goal: show that the average time per operation is
still good
n
operationsn for timetotal
![Page 83: CSE 326: Data Structures Introduction & Part One: Complexity Henry Kautz Autumn Quarter 2002.](https://reader035.fdocuments.us/reader035/viewer/2022062304/56649da85503460f94a949d0/html5/thumbnails/83.jpg)
Stack Abstract Data Type
• Stack operations– push
– pop
– is_empty
• Stack property: if x is on the stack before y is pushed, then x will be popped after y is popped
What is biggest problem with an array implementation?
A
BCDEF
E D C B A
F
![Page 84: CSE 326: Data Structures Introduction & Part One: Complexity Henry Kautz Autumn Quarter 2002.](https://reader035.fdocuments.us/reader035/viewer/2022062304/56649da85503460f94a949d0/html5/thumbnails/84.jpg)
Stretchy Stack Implementation
int[] data;int maxsize;int top;
Push(e){if (top == maxsize){
temp = new int[2*maxsize];for (i=0;i<maxsize;i++) temp[i]=data[i]; ;data = temp; maxsize = 2*maxsize; }
else { data[++top] = e; }
Best case Push = O( )
Worst case Push = O( )
![Page 85: CSE 326: Data Structures Introduction & Part One: Complexity Henry Kautz Autumn Quarter 2002.](https://reader035.fdocuments.us/reader035/viewer/2022062304/56649da85503460f94a949d0/html5/thumbnails/85.jpg)
Stretchy Stack Amortized Analysis
• Consider sequence of n operationspush(3); push(19); push(2); …
• What is the max number of stretches?• What is the total time?
– let’s say a regular push takes time a, and stretching an array contain k elements takes time bk.
• Amortized time =
![Page 86: CSE 326: Data Structures Introduction & Part One: Complexity Henry Kautz Autumn Quarter 2002.](https://reader035.fdocuments.us/reader035/viewer/2022062304/56649da85503460f94a949d0/html5/thumbnails/86.jpg)
Stretchy Stack Amortized Analysis
• Consider sequence of n operationspush(3); push(19); push(2); …
• What is the max number of stretches?• What is the total time?
– let’s say a regular push takes time a, and stretching an array contain k elements takes time bk.
• Amortized time =
log n
log
(1 2 4 8 ... ) 2n
i
i o
an b n an b
![Page 87: CSE 326: Data Structures Introduction & Part One: Complexity Henry Kautz Autumn Quarter 2002.](https://reader035.fdocuments.us/reader035/viewer/2022062304/56649da85503460f94a949d0/html5/thumbnails/87.jpg)
Geometric Series
1
11
0
A
AA
NN
i
i
11
0
2 12 2 1
2 1
nni n
i
loglog 1 log 1
0
2 2 1 (2 )2 1 2 1n
i n n
i
n
![Page 88: CSE 326: Data Structures Introduction & Part One: Complexity Henry Kautz Autumn Quarter 2002.](https://reader035.fdocuments.us/reader035/viewer/2022062304/56649da85503460f94a949d0/html5/thumbnails/88.jpg)
Stretchy Stack Amortized Analysis
• Consider sequence of n operationspush(3); push(19); push(2); …
• What is the max number of stretches?• What is the total time?
– let’s say a regular push takes time a, and stretching an array contain k elements takes time bk.
• Amortized time =
log n
log
(1 2 4 8 ... ) 2
(2 1)
ni
i o
an b n an b
an b n
(2 1)( )
an b n
n
![Page 89: CSE 326: Data Structures Introduction & Part One: Complexity Henry Kautz Autumn Quarter 2002.](https://reader035.fdocuments.us/reader035/viewer/2022062304/56649da85503460f94a949d0/html5/thumbnails/89.jpg)
Surprise
• In an asymptotic sense, there is no overhead in using stretchy arrays rather than regular arrays!