2IL50 Data Structures
Fall 2015
Lecture 2: Analysis of Algorithms
Analysis of algorithms
the formal way …
Analysis of algorithms
Can we say something about the running time of an algorithm without implementing and testing it?
InsertionSort(A)1. initialize: sort A[1]2. for j = 2 to A.length3. do key = A[j] 4. i = j -15. while i > 0 and A[i] > key6. do A[i+1] = A[i] 7. i = i -18. A[i +1] = key
Analysis of algorithms
Analyze the running time as a function of n (# of input elements)
best case average case worst case
elementary operationsadd, subtract, multiply, divide, load, store, copy, conditional and unconditional branch, return …
An algorithm has worst case running time T(n) if for any input of size n the maximal number of elementary operations executed is T(n).
Linear Search
Input: increasing sequence of n numbers A = ‹a1, a2, …, an› and value v
Output: an index i such that A[i] = v or NIL if v not in A
LinearSearch(A, v)1. for i = 1 to n2. do if A[i] = v3. then return i 4. return NIL
Running time
best case: 1 average case: n/2 (if successful) worst case: n
1 3 4 7 8 14 17 21 28 35
v = 7
Binary Search
Input: increasing sequence of n numbers A = ‹a1, a2, …, an› and value v
Output: an index i such that A[i] = v or NIL if v not in A
BinarySearch(A, v)1. x = 02. y = n + 13. while x + 1 < y4. do h = floor((x + y)/2)5. do if A[h] ≤ v then x = h6. else y = h 7. if A[x] = v then return x else return NIL
Running time best case: log n average case: log n worst case: log n
1 3 4 7 8 14 17 21 28 35
v = 7
Analysis of algorithms: example
InsertionSort: 15 n2 + 7n – 2
MergeSort: 300 n lg n + 50 n
n=10 n=100 n=1000
1568 150698 1.5 x 107
10466 204316 3.0 x 106
InsertionSort6 x faster
InsertionSort1.35 x faster
MergeSort5 x faster
n = 1,000,000 InsertionSort 1.5 x 1013
MergeSort 6 x 109 2500 x faster !
The rate of growth of the running time as a function of the input is essential!
Θ-notation
Intuition: concentrate on the leading term, ignore constants
19 n3 + 17 n2 - 3n becomes Θ(n3)
2 n lg n + 5 n1.1 - 5 becomes
n - ¾ n √n becomes
Θ(n1.1)
---
Θ-notation
Let g(n) : N ↦ N be a function. Then we have
Θ(g(n)) = { f(n) : there exist positive constants c1, c2, and n0 such that 0 ≤ c1g(n) ≤ f(n) ≤ c2g(n) for all n ≥
n0 }
“Θ(g(n)) is the set of functions that grow as fast as g(n)”
Θ-notation
Let g(n) : N ↦ N be a function. Then we have
Θ(g(n)) = { f(n) : there exist positive constants c1, c2, and n0 such that 0 ≤ c1g(n) ≤ f(n) ≤ c2g(n) for all n ≥ n0 }
nn0
c1g(n)
c2g(n)
f(n)
0
Notation: f(n) = Θ(g(n))
Θ-notation
Let g(n) : N ↦ N be a function. Then we have
Θ(g(n)) = { f(n) : there exist positive constants c1, c2, and n0 such that 0 ≤ c1g(n) ≤ f(n) ≤ c2g(n) for all n ≥ n0 }
Claim: 19n3 + 17n2 - 3n = Θ(n3)
Proof: Choose c1 = 19, c2 = 36 and n0 = 1.
Then we have for all n ≥ n0:
0 ≤ c1n3 = 19n3 (trivial)
≤ 19n3 + 17n2 - 3n (since 17n2 > 3n for n ≥ 1)
≤ 19n3 + 17n3 (since 17n2 ≤ 17n3 for n ≥1) = c2n3
■
Θ-notation
Let g(n) : N ↦ N be a function. Then we have
Θ(g(n)) = { f(n) : there exist positive constants c1, c2, and n0 such that 0 ≤ c1g(n) ≤ f(n) ≤ c2g(n) for all n ≥ n0 }
Claim: 19n3 + 17n2 - 3n ≠ Θ(n2)
Proof: Assume that there are positive constants c1, c2, and n0 such that for all n ≥ n0
0 ≤ c1n2 ≤ 19n3 + 17n2 - 3n ≤ c2n2
Since 19n3 + 17n2 - 3n ≤ c2n2 implies
19n3 ≤ c2n2 + 3n – 17n2 ≤ c2n2 (3n – 17n2 ≤ 0)
we would have for all n ≥ n0
19n ≤ c2.
O-notation
Let g(n) : N ↦ N be a function. Then we have
O(g(n)) = { f(n) : there exist positive constants c and n0 such that 0 ≤ f(n) ≤ cg(n) for all n ≥ n0 }
“O(g(n)) is the set of functions that grow at most as fast as g(n)”
O-notation
Let g(n) : N ↦ N be a function. Then we have
O(g(n)) = { f(n) : there exist positive constants c and n0 such that 0 ≤ f(n) ≤ cg(n) for all n ≥ n0 }
nn0
cg(n)
f(n)
0
Notation: f(n) = O(g(n))
Ω-notation
Let g(n) : N ↦ N be a function. Then we have
Ω(g(n)) = { f(n) : there exist positive constants c and n0 such that 0 ≤ cg(n) ≤ f(n) for all n ≥ n0 }
“Ω(g(n)) is the set of functions that grow at least as fast as g(n)”
Ω-notation
Let g(n) : N ↦ N be a function. Then we have
Ω(g(n)) = { f(n) : there exist positive constants c and n0 such that 0 ≤ cg(n) ≤ f(n) for all n ≥ n0 }
nn0
cg(n)
f(n)
0
Notation: f(n) = Ω(g(n))
Asymptotic notation
Θ(…) is an asymptotically tight bound
O(…) is an asymptotic upper bound
Ω(…) is an asymptotic lower bound
other asymptotic notation
o(…) → “grows strictly slower than”ω(…) → “grows strictly faster than”
“asymptotically equal”
“asymptotically smaller or equal”
“asymptotically greater or equal”
More notation …
f(n) = n3 + Θ(n2) means
f(n) = means
O(1) or Θ(1) means
2n2 + O(n) = Θ(n2) means
there is a function g(n) such that
f(n) = n3 + g(n) and g(n) = Θ(n2)
there is one function g(i) such that
f(n) = and g(i) = O(i)
a constant
for each function g(n) with g(n)=O(n)
we have 2n2 + g(n) = Θ(n2)
n
1iO(i)
n
1ig(i)
Quiz
1. O(1) + O(1) = O(1)
2. O(1) + … + O(1) = O(1)
3. =
4. O(n2) ⊆ O(n3)
5. O(n3) ⊆ O(n2)
6. Θ(n2) ⊆ O(n3)
7. An algorithm with worst case running time O(n log n) is always slower than an algorithm with worst case running time O(n) if n is sufficiently large.
true
false
true
true
false
true
false
n
1iO(i)
n
1ii)O(
Quiz
8. n log2 n = Θ(n log n)
9. n log2 n = Ω(n log n)
10. n log2 n = O(n4/3)
11. O(2n) ⊆ O(3n)
12. O(2n) ⊆ Θ(3n)
false
true
true
true
false
Analysis of algorithms
Analysis of InsertionSort
InsertionSort(A)1. initialize: sort A[1]2. for j = 2 to A.length3. do key = A[j] 4. i = j -15. while i > 0 and A[i] > key6. do A[i+1] = A[i] 7. i = i -18. A[i +1] = key
Get as tight a bound as possible on the worst case running time.
➨ lower and upper bound for worst case running time
Upper bound: Analyze worst case number of elementary operations
Lower bound: Give “bad” input example
Analysis of InsertionSort
InsertionSort(A)1. initialize: sort A[1]2. for j = 2 to A.length3. do key = A[j] 4. i = j -15. while i > 0 and A[i] > key6. do A[i+1] = A[i] 7. i = i -18. A[i +1] = key
Upper bound: Let T(n) be the worst case running time of InsertionSort on an array of length n. We have
T(n) =
Lower bound:
n
2j
O(1)
O(1)
O(1)
worst case:(j-1) ∙ O(1)
O(1) + { O(1) + (j-1)∙O(1) + O(1) }
= O(j) = O(n2)
n
2j
Array sorted in de-creasing order ➨ Ω(n2)
The worst case running time of InsertionSort is Θ(n2).
MergeSort(A)
// divide-and-conquer algorithm that sorts array A[1..n]
1. if A.length = 1
2. then skip
3. else
4. n = A.length ; n1 = floor(n/2); n2 = ceil(n/2);
5. copy A[1.. n1] to auxiliary array A1[1.. n1]
6. copy A[n1+1..n] to auxiliary array A2[1.. n2]
7. MergeSort(A1); MergeSort(A2)
8. Merge(A, A1, A2)
Analysis of MergeSort
O(1)
O(1)
O(n)
O(n)
O(n)??
T( n/2 ) + T( n/2 )
MergeSort is a recursive algorithm ➨ running time analysis leads to recursion
Analysis of MergeSort
Let T(n) be the worst case running time of MergeSort on an array of length n. We have
O(1) if n = 1 T(n) = T( n/2 ) + T( n/2 ) + Θ(n) if n > 1
frequently omitted since it (nearly) always holds
often written as 2T(n/2)
Solving recurrences
Solving recurrences
Easiest: Master theoremcaveat: not always applicable
Alternatively: Guess the solution and use the substitution method to prove that your guess it is correct.
How to guess:
1. expand the recursion
2. draw a recursion tree
Let a and b be constants, let f(n) be a function, and let T(n)be defined on the nonnegative integers by the recurrence
T(n) = aT(n/b) + f(n)
Then we have:
1. If f(n) = O(nlog a – ε) for some constant ε > 0, then T(n) = Θ(nlog a).
2. If f(n) = Θ(nlog a), then T(n) = Θ(nlog a log n)
3. If f(n) = Ω(nlog a + ε) for some constant ε > 0, and if af(n/b) ≤ cf(n) for some constant c < 1 and all sufficiently large n, then T(n) = Θ(f(n))
The master theorem
b
b
b
b
can be rounded up or down
note: logba - ε
b
The master theorem: Example
T(n) = 4T(n/2) + Θ(n3)
Master theorem with a = 4, b = 2, and f(n) = n3
logba = log24 = 2
➨ n3 = f(n) = Ω(nlog a + ε) = Ω(n2 + ε) with, for example, ε = 1
Case 3 of the master theorem gives T(n) = Θ(n3), if the regularity condition holds.
choose c = ½ and n0 = 1
➨ af(n/b) = 4(n/2)3 = n3/2 ≤ cf(n) for n ≥ n0
➨ T(n) = Θ(n3)
b
The substitution method
The Master theorem does not always apply
In those cases, use the substitution method:
1. Guess the form of the solution.
2. Use induction to find the constants and show that the solution works
Use expansion or a recursion-tree to guess a good solution.
Recursion-trees
T(n) = 2T(n/2) + n
n
n/2 n/2
n/4 n/4 n/4 n/4
n/2i n/2i … n/2i
Θ(1) Θ(1) … Θ(1)
Recursion-trees
T(n) = 2T(n/2) + n
n
2 ∙ (n/2) = n
4 ∙ (n/4) = n
2i ∙ (n/2i) = n
n ∙ Θ(1) = Θ(n)
log n
+
Θ(n log n)
n
n/2 n/2
n/4 n/4 n/4 n/4
n/2i n/2i … n/2i
Θ(1) Θ(1) … Θ(1)
Recursion-trees
T(n) = 2T(n/2) + n2
n2
(n/2)2 (n/2)2
(n/4)2 (n/4)2 (n/4)2 (n/4)2
(n/2i)2 (n/2i)2 … (n/2i)2
Θ(1) Θ(1) … Θ(1)
Recursion-trees
T(n) = 2T(n/2) + n2
n2
(n/2)2 (n/2)2
(n/4)2 (n/4)2 (n/4)2 (n/4)2
(n/2i)2 (n/2i)2 … (n/2i)2
Θ(1) Θ(1) … Θ(1)
n2
2 ∙ (n/2)2 = n2/2
4 ∙ (n/4)2 = n2/4
2i ∙ (n/2i) 2 = n2/2i
n ∙ Θ(1) = Θ(n) +
Θ(n2)
Recursion-trees
T(n) = 4T(n/2) + n
n
n/2 n/2 n/2 n/2
… n/4 n/4 n/4 n/4 …
Θ(1) Θ(1) … Θ(1)
Recursion-trees
T(n) = 4T(n/2) + n
n
n/2 n/2 n/2 n/2
… n/4 n/4 n/4 n/4 …
Θ(1) Θ(1) … Θ(1)
n
4 ∙ (n/2) = 2n
16 ∙ (n/4) = 4n
n2 ∙ Θ(1) = Θ(n2) +
Θ(n2)
The substitution method
Claim: T(n) = O(n log n)
Proof: by induction on n
to show: there are constants c and n0 such that T(n) ≤ c n log n for all n ≥ n0
n = 1 ➨ T(1) = 2 ≤ c 1 log 1
n = n0 = 2 is a base case Need more base cases?
Base cases: n = 2: T(2) = 2T(1) + 2 = 2∙2 + 2 = 6 = c 2 log 2 for c = 3 n = 3: T(3) = 2T(1) + 3 = 2∙2 + 3 = 7 ≤ c 3 log 3
T(n) =2 if n = 1
2T( n/2 ) + n if n > 1
3/2 = 1, 4/2 = 2 ➨ 3 must also be base case
➨ n0 = 2
The substitution method
Claim: T(n) = O(n log n)
Proof: by induction on n
to show: there are constants c and n0 such that T(n) ≤ c n log n for all n ≥ n0
choose c = 3 and n0 = 2
Inductive step: n > 3T(n) = 2T( n/2 ) + n
≤ 2 c n/2 log n/2 + n (ind. hyp.) ≤ c n ((log n) - 1) + n
≤ c n log n ■
T(n) =2 if n = 1
2T( n/2 ) + n if n > 1
The substitution method
Claim: T(n) = O(n)
Proof: by induction on n
Base case: n = n0
T(2) = 2T(1) + 2 = 2c + 2 = O(2)
Inductive step: n > n0
T(n) = 2T( n/2 ) + n = 2O( n/2 ) + n (ind. hyp.)
= O(n) ■
T(n) =Θ(1) if n = 1
2T( n/2 ) + n if n > 1
Never use O, Θ, or Ω in a proof by induction!
Analysis of algorithms
one more example …
Example (A)
// A is an array of length n
1. n = A.length
2. if n=1
3. then return A[1]
4. else begin
5. Copy A[1… n/2 ] to auxiliary array B[1... n/2 ]
6. Copy A[1… n/2 ] to auxiliary array C[1… n/2 ]
7. b = Example(B); c = Example(C)
8. for i = 1 to n
9. do for j = 1 to i
10. do A[i] = A[j]
11. return 43
12. end
Example
Example (A)
// A is an array of length n
1. n = A.length
2. if n=1
3. then return A[1]
4. else begin
5. Copy A[1… n/2 ] to auxiliary array B[1... n/2 ]
6. Copy A[1… n/2 ] to auxiliary array C[1… n/2 ]
7. b = Example(B); c = Example(C)
8. for i = 1 to n
9. do for j = 1 to i
10. do A[i] = A[j]
11. return 43
12. end
Example
Let T(n) be the worst case running time of Example on an array of length n.
Lines 1,2,3,4,11, and 12 take Θ(1) time.Lines 5 and 6 take Θ(n) time.Line 7 takes Θ(1) + 2 T( n/2 ) time.Lines 8 until 10 take
time.
If n=1 lines 1,2,3 are executed,else lines 1,2, and 4 until 12 are executed.
➨ T(n):
➨ use master theorem …
)Θ(nΘ(i)Θ(1) 2n
1i
n
1i
i
1j
Θ(1) if n=1
2T(n/2) + Θ(n2) if n>1
Tips
Analysis of recursive algorithms:find the recursion and solve with master theorem if possible
Analysis of loops: summations
Some standard recurrences and sums:
T(n) = 2T(n/2) + Θ(n) ➨
½ n(n+1) = Θ(n2)
Θ(n3)
T(n) = Θ(n log n)
n
1i
i
n
1i
2i
Announcements
If you are not in a group yet, talk to me immediately!
Assignment A1 due on Sunday!
You can ask questions during the tutorial on Wednesday.
Email your assignment as .pdf to your tutor, not to me.
Top Related