Post on 31-Dec-2015
04/19/23 2
Exam (midterm 1)
• Closed book exam
• One cheat sheet allowed (limit to a single page of letter-size paper, double-sided)
• Tuesday, Feb 24, 10:00 – 11:25pm
• Basic calculator (no graphing) is allowed
04/19/23 3
Materials covered
• Up to Lecture 8 (Feb 6)• Comparing functions• O, Θ, Ω
– Definition, limit method, L’Hopital’s rule, sterling’s formula• Analyzing iterative algorithms
– Know how to count the number of basic operations, and express the running time as a sum of a series
– Know how to compute the sum of a series (geometric, arithmetic, or other frequently seen series)
• Analyzing recursive algorithms– Define recurrence– Solve recurrence using recursion tree / iteration method– Solve recurrence using master method– Prove using substitution method
04/19/23 5
Mathematical definitions
• O(g(n)) = {f(n): positive constants c and n0 such that 0 ≤ f(n) ≤ cg(n) n>n0}
• Ω(g(n)) = {f(n): positive constants c and n0 such that 0 ≤ cg(n) ≤ f(n) n>n0}
• Θ(g(n)) = {f(n): positive constants c1, c2, and n0 such that 0 c1 g(n) f(n) c2 g(n) n n0}
04/19/23 6
Big-Oh
• Claim: f(n) = 3n2 + 10n + 5 O(n2)
• Proof by definition:
f(n) = 3n2 + 10n + 5 3n2 + 10n2 + 5 , n > 1 3n2 + 10n2 + 5n2, n > 1
18 n2, n > 1If we let c = 18 and n0 = 1, we have f(n) c n2, n > n0. Therefore by definition, f(n) = O(n2).
04/19/23 7
Use limits to compare orders of growth
0• lim f(n) / g(n) = c > 0
∞n→∞
f(n) o(g(n))
f(n) Θ (g(n))
f(n) ω (g(n))
f(n) O(g(n))
f(n) Ω(g(n))
lim f(n) / g(n) = lim f(n)’ / g(n)’n→∞ n→∞
Condition:
If both lim f(n) and lim g(n) = ∞ or 0
L’ Hopital’s rule
Stirling’s formula
!n nn en 2/1(constant)
04/19/23 8
Useful rules for logarithms
For all a > 0, b > 0, c > 0, the following rules hold• logba = logca / logcb = lg a / lg b
– So: log10n = log2n / log2 10
• logban = n logba– So: log 3n = n log3 = (n)
•b
logba = a
– So: 2log
2n = n
• log (ab) = log a + log b– So: log (3n) = log 3 + log n = (log n)
• log (a/b) = log (a) – log(b)– So: log (n/2) = log n – log 2 = (log n)
• logba = 1 / logab• logb1 = 0
04/19/23 9
Useful rules for exponentials
• For all a > 0, b > 0, c > 0, the following rules hold• a0 = 1 (00 = ?) Answer: does not exist• a1 = a• a-1 = 1/a• (am)n = amn
• (am)n = (an)m
– So: (3n)2 = 32n = (32)n =9n
• aman = am+n
– So: n2 n3 = n5
– 2n 22 = 2n+2 = 4 * 2n = (2n)
04/19/23 10
More advanced dominance ranking
1)(logloglogloglog/log
loglogloglog/
!log~log23!
)3(
23
123
nnnnn
nnnnnnnn
nnnnnnnn nnn
04/19/23 11
Sum of arithmetic series
If a1, a2, …, an is an arithmetic series, then
2
)( 1
1
nn
ii
aana
04/19/23 12
Sum of geometric series
if r < 1
1
)1/()1(
)1/()1(1
1
0 n
rr
rr
r n
n
n
i
i if r > 1
if r = 1
?2
1lim
?2
1lim
?2
1
0
0
n
iin
n
iin
n
i
i
04/19/23 13
Sum manipulation rules
n
xii
x
mii
n
mii
i ii i
i ii ii ii
aaa
acca
baba
1
)(
Example:
n
ii
n
ii
n
i
n
i
nin
i
i
nnn
nnii
11
1 1
1
1
2
1
2
22)1(224)24(
04/19/23 14
Analyzing non-recursive algorithms
• Decide parameter (input size)
• Identify most executed line (basic operation)
• worst-case = average-case?
• T(n) = i ti
• T(n) = Θ (f(n))
04/19/23 15
Statement cost time__InsertionSort(A, n) {
for j = 2 to n { c1 n
key = A[j] c2 (n-1)
i = j - 1; c3 (n-1)
while (i > 0) and (A[i] > key) { c4 S
A[i+1] = A[i] c5 (S-(n-1))
i = i - 1 c6 (S-(n-1))
} 0
A[i+1] = key c7 (n-1)
} 0
}
Analysis of insertion Sort
04/19/23 16
Best case
• Array already sorted• S = j=1..n tj
• tj = 1 for all j• S = n. T(n) = Θ (n)
1 i j
sorted Key
Inner loop stops when A[i] <= key, or i = 0
04/19/23 17
Worst case
• Array originally in reverse order sorted• S = j=1..n tj
• tj = j• S = j=1..n j = 1 + 2 + 3 + … + n = n (n+1) / 2 = Θ (n2)
1 i j
sorted
Inner loop stops when A[i] <= key
Key
04/19/23 18
Average case
• Array in random order• S = j=1..n tj
• tj = j / 2 in average• S = j=1..n j/2 = ½ j=1..n j = n (n+1) / 4 = Θ (n2)
1 i j
sorted
Inner loop stops when A[i] <= key
Key
04/19/23 19
Analyzing recursive algorithms
• Defining recurrence relation
• Solving recurrence relation– Recursion tree (iteration) method– Substitution method– Master method
04/19/23 20
Analyzing merge sort
MERGE-SORT A[1 . . n]1. If n = 1, done.2. Recursively sort A[ 1 . . n/2 ]
and A[ n/2+1 . . n ] .3. “Merge” the 2 sorted lists
T(n)Θ(1)2T(n/2)
f(n)
T(n) = 2 T(n/2) + Θ(n)
04/19/23 21
Recursive Insertion Sort
RecursiveInsertionSort(A[1..n])
1. if (n == 1) do nothing;
2.RecursiveInsertionSort(A[1..n-1]);
3.Find index i in A such that A[i] <= A[n] < A[i+1];
4. Insert A[n] after A[i];
)(1)( nnTnT
04/19/23 22
Binary Search
BinarySearch (A[1..N], value) { if (N == 0)
return -1; // not foundmid = (1+N)/2;
if (A[mid] == value) return mid; // found
else if (A[mid] > value) return BinarySearch (A[1..mid-1], value); else
return BinarySearch (A[mid+1, N], value) }
)1(2
)(
nTnT
04/19/23 23
Recursion tree
Solve T(n) = 2T(n/2) + n.
n
n/4 n/4 n/4 n/4
n/2 n/2
(1)
…
h = log n
n
n
n
#leaves = n (n)
Total(n log n)
…
04/19/23 24
• Recurrence: T(n) = 2T(n/2) + n.• Guess: T(n) = O(n log n). (eg. by recursion tree
method)• To prove, have to show T(n) ≤ c n log n for
some c > 0 and for all n > n0
• Proof by induction: assume it is true for T(n/2), prove that it is also true for T(n). This means:
• Fact: T(n) = 2T(n/2) + n• Assumption: T(n/2)≤ cn/2 log (n/2)• Need to Prove: T(n)≤ c n log (n)
Substitution method
04/19/23 25
Proof
• To prove T(n) = O(n log n), we need to show that T(n) cn logn for some positive c and all sufficiently
large n. • Let’s assume this inequality is true for T(n/2), which
means T(n/2) cn/2 log(n/2)
• Substitute T(n/2) in the recurrence by the r.h.s. of the above inequality, we haveT(n) = 2 T(n/2) + n
2 * cn/2 log (n/2) + n cn (log n – 1) + n cn log n – (cn – n) cn log n for c ≥ 1 and all n ≥ 0.
Therefore, by definition, T(n) = O(n log n).
04/19/23 26
Master theoremT(n) = a T(n/b) + f (n)
CASE 1: f (n) = O(nlogba – ) T(n) = (nlogba) .
CASE 2: f (n) = (nlogba) T(n) = (nlogba log n) .
CASE 3: f (n) = (nlogba + ) and a f (n/b) c f (n)T(n) = ( f (n)) .
Optional: extended case 2
Key: compare f(n) with nlogba
Regularity Condition
04/19/23 28
Quick sort
• Another divide and conquer sorting algorithm – like merge sort
• Anyone remember the basic idea?
• The worst-case and average-case running time?
• Learn some new algorithm analysis tricks
04/19/23 29
Quick sort
Quicksort an n-element array:
1. Divide: Partition the array into two subarrays around a pivot x such that elements in lower subarray x elements in upper subarray.
2. Conquer: Recursively sort the two subarrays.
3. Combine: Trivial.
x x xx ≥ x≥ x
Key: Linear-time partitioning subroutine.
04/19/23 30
Partition
• All the action takes place in the partition() function– Rearranges the subarray in place– End result: two subarrays
• All values in first subarray all values in second
– Returns the index of the “pivot” element separating the two subarrays
x x xx ≥ x≥ xp rq
04/19/23 31
Pseudocode for quicksort
QUICKSORT(A, p, r)if p < r
then q PARTITION(A, p, r) QUICKSORT(A, p, q–1)QUICKSORT(A, q+1, r)
Initial call: QUICKSORT(A, 1, n)
04/19/23 32
Idea of partition
• If we are allowed to use a second array, it would be easy
66 1010 55 88 1313 33 22 1111
66 55 33 22 1111 1313 88 1010
22 55 33 66 1111 1313 88 1010
04/19/23 33
Another idea
• Keep two iterators: one from head, one from tail
66 1010 55 88 1313 33 22 1111
66 22 55 33 1313 88 1010 1111
33 22 55 66 1313 88 1010 1111
04/19/23 35
Partition In Words
• Partition(A, p, r):– Select an element to act as the “pivot” (which?)
– Grow two regions, A[p..i] and A[j..r]• All elements in A[p..i] <= pivot
• All elements in A[j..r] >= pivot
– Increment i until A[i] > pivot
– Decrement j until A[j] < pivot
– Swap A[i] and A[j]
– Repeat until i >= j
– Swap A[j] and A[p]
– Return j
Note: different from book’s partition(), which uses two iterators that both move forward.
04/19/23 36
Partition CodePartition(A, p, r) x = A[p]; // pivot is the first element i = p; j = r + 1; while (TRUE) {
repeat i++; until A[i] > x or i >= j; repeat j--; until A[j] < x or j < i; if (i < j) Swap (A[i], A[j]); else break;
} swap (A[p], A[j]); return j;
What is the running time of partition()?
partition() runs in (n) time
04/19/23 37
i j66 1010 55 88 1313 33 22 1111x = 6
p r
i j66 1010 55 88 1313 33 22 1111
i j66 22 55 88 1313 33 1010 1111
i j66 22 55 88 1313 33 1010 1111
i j66 22 55 33 1313 88 1010 1111
ij66 22 55 33 1313 88 1010 1111
33 22 55 66 1313 88 1010 1111qp r
scan
scan
scan
swap
swap
final swap
Partition example
04/19/23 38
66 1010 55 88 1111 33 22 1313
33 22 55 66 1111 88 1010 1313
22 33 55 66 88 1010 1111 1313
22 33 55 66 1010 88 1111 1313
22 33 55 66 88 1010 1111 1313
Quick sortexample
04/19/23 39
Analysis of quicksort
• Assume all input elements are distinct.
• In practice, there are better partitioning algorithms for when duplicate input elements may exist.
• Let T(n) = worst-case running time on an array of n elements.
04/19/23 40
Worst-case of quicksort
• Input sorted or reverse sorted.• Partition around min or max element.• One side of partition always has no elements.
)(
)()1(
)()1()1(
)()1()0()(
2n
nnT
nnT
nnTTnT
(arithmetic series)
04/19/23 46
n
T(0) (n–1)
Worst-case recursion treeT(n) = T(0) + T(n–1) + n
T(0) (n–2)
T(0)
T(0)
2
1
nkk
height
height = n
04/19/23 47
n
T(0) (n–1)
Worst-case recursion treeT(n) = T(0) + T(n–1) + n
T(0) (n–2)
T(0)
T(0)
2
1
nkk
n
height = n
04/19/23 48
n
(n–1)
Worst-case recursion treeT(n) = T(0) + T(n–1) + n
(n–2)
(1)
2
1
nkk
n
height = n
(1)
(1)
(1)T(n) = (n) + (n2)
= (n2)
04/19/23 49
Best-case analysis(For intuition only!)
If we’re lucky, PARTITION splits the array evenly:
T(n) = 2T(n/2) + (n)= (n log n) (same as merge sort)
What if the split is always 109
101 : ?
)()(109
101 nnTnTnT
What is the solution to this recurrence?
04/19/23 53
Analysis of “almost-best” case
n1001
(1)
(1)
… …log10/9n
n
n
n
…O(n) leavesO(n) leaves
n101 n10
9
n1009 n100
9 n10081
n
04/19/23 54
log10n
Analysis of “almost-best” case
n
(1)
(1)
… …log10/9n
n
n
n
T(n) n log10/9n + (n)
…
n log10n
O(n) leavesO(n) leaves
(n log n)
n1001
n101 n10
9
n1009 n100
9 n10081
04/19/23 55
Quicksort Runtimes
• Best-case runtime Tbest(n) (n log n)
• Worst-case runtime Tworst(n) (n2)
• Worse than mergesort? Why is it called quicksort then?
• Its average runtime Tavg(n) (n log n )
• Better even, the expected runtime of randomized quicksort is (n log n)
04/19/23 56
Randomized quicksort
• Randomly choose an element as pivot– Every time need to do a partition, throw a die to
decide which element to use as the pivot– Each element has 1/n probability to be selected
Rand-Partition(A, p, r) d = random(); // a random number between 0 and 1 index = p + floor((r-p+1) * d); // p<=index<=r swap(A[p], A[index]); Partition(A, p, r); // now do partition using A[p] as pivot
04/19/23 57
Running time of randomized quicksort
• The expected running time is an average of all cases
T(n) =
T(0) + T(n–1) + dn if 0 : n–1 split,T(1) + T(n–2) + dn if 1 : n–2 split,T(n–1) + T(0) + dn if n–1 : 0 split,
nknTkTn
nTn
k
1
0)1()(
1)(
Expectation
04/19/23 59
Solving recurrence
1. Recursion tree (iteration) method- Good for guessing an answer
2. Substitution method- Generic method, rigid, but may be hard
3. Master method- Easy to learn, useful in limited cases only
- Some tricks may help in other cases
04/19/23 60
Substitution method
1. Guess the form of the solution:(e.g. using recursion trees, or expansion)
2. Verify by induction (inductive step).
The most general method to solve a recurrence (prove O and separately):
04/19/23 61
Expected running time of Quicksort
• Guess
• We need to show that for some c and sufficiently large n
• Use T(n) instead of for convenience
nkTn
nTn
k
1
0)(
2)(
)log()( nnOnT ncnnT log)(
)(nT
04/19/23 62
• Fact:
• Need to show: T(n) ≤ c n log (n)
• Assume: T(k) ≤ ck log (k) for 0 ≤ k ≤ n-1
• Proof:
nkTn
nTn
k
1
0)(
2)(
nkkn
c n
k
1
0
log2
ncn
ncn 4
log
ncn log if c ≥ 4. Therefore, by defintion, T(n) = (nlogn)
using the fact that 8
log2
log221
0
nn
nkk
n
k
n
nn
n
n
c
8log
2
2 22
nkTn
nTn
k
1
0
)(2
)(
04/19/23 63
What are we doing here?
What are we doing here?
What are we doing here?
The lg k in the second term is bounded by lg n
Tightly Bounding The Key Summation
1
2
12
1
1
2
12
1
1
2
12
1
1
1
1
0
lglg
lglg
lglg
lglg
n
nk
n
k
n
nk
n
k
n
nk
n
k
n
k
n
k
knkk
nkkk
kkkk
kkkk
Move the lg n outside the summation
Split the summation for a tighter bound
0lglim0
xxx
04/19/23 64
The summation bound so far
Tightly BoundingThe Key Summation
1
2
12
1
1
2
12
1
1
2
12
1
1
2
12
1
1
1
lg1lg
lg1lg
lg2lg
lglglg
n
nk
n
k
n
nk
n
k
n
nk
n
k
n
nk
n
k
n
k
knkn
knnk
knnk
knkkkk
What are we doing here?The lg k in the first term is bounded by lg n/2
What are we doing here?lg n/2 = lg n - 1
What are we doing here?Move (lg n - 1) outside the summation
04/19/23 65
The summation bound so far
Tightly BoundingThe Key Summation
12
1
12
1
1
1
1
2
12
1
12
1
1
2
12
1
1
1
2
)(1lg
lg
lglg
lg1lglg
n
k
n
k
n
k
n
nk
n
k
n
k
n
nk
n
k
n
k
knn
n
kkn
knkkn
knknkk
What are we doing here?Distribute the (lg n - 1)
What are we doing here?The summations overlap in range; combine them
What are we doing here?The Guassian series
04/19/23 66
The summation bound so far
Tightly Bounding The Key Summation
48
1lglg
2
1
1222
1lg1
2
1
lg12
1
lg2
)(1lg
22
12
1
12
1
1
1
nnnnnn
nnnnn
knnn
knnn
kk
n
k
n
k
n
k
What are we doing here?Rearrange first term, place upper bound on second
What are we doing?Guassian series
What are we doing?Multiply it all out