CS 3343: Analysis of Algorithms Review for Exam 2.
-
Upload
cathleen-york -
Category
Documents
-
view
224 -
download
0
Transcript of CS 3343: Analysis of Algorithms Review for Exam 2.
CS 3343: Analysis of Algorithms
Review for Exam 2
Exam 2
• Closed book exam
• One cheat sheet allowed (limit to a single page of letter-size paper, double-sided)
• Tuesday, Nov 17, class time + 5 minutes
• Basic calculator (no graphing) is allowed but not necessary
Final grade calculation
• Three components: HQP, Midterm, Final• HQ = HW + Quiz + Participation
: (90% HW + 10% Quiz&Participation)
: One lowest HW dropped
• Midterm = 40% midterm1+ 60% midterm2• Max for each component is 100. • Sort three components (highest, middle, lowest)• Final grade:
– Case 1: HQP is not the highest
55% highest + 30% middle + 15% lowest
– Case 2: HQP is highest
55% middle + 30% HQP + 15% lowest
Materials covered
• Quick Sort• Heap sort, priority queue• Linear time sorting algorithms• Order statistics• Dynamic programming• Greedy algorithm• Graph basics
• Questions will be similar to homework / quizzes– Familiar with the algorithm procedure– Some analysis of time/space complexity– One or two problems on algorithm design
QuickSort
• Quicksort algorithm
• Randimized Quicksort algorithm
• Partition algorithm
• Quicksort/Randomized Quicksort time complexity analysis
Back
Quick sort
Quicksort an n-element array:
1. Divide: Partition the array into two subarrays around a pivot x such that elements in lower subarray x elements in upper subarray.
2. Conquer: Recursively sort the two subarrays.
3. Combine: Trivial.
x x xx ≥ x≥ x
Key: Linear-time partitioning subroutine.
Pseudocode for quicksort
QUICKSORT(A, p, r)if p < r
then q PARTITION(A, p, r) QUICKSORT(A, p, q–1)QUICKSORT(A, q+1, r)
Initial call: QUICKSORT(A, 1, n)
Partition CodePartition(A, p, r) x = A[p]; // pivot is the first element i = p; j = r + 1; while (TRUE) {
repeat i++; until A[i] > x | i >= j; repeat j--; until A[j] < x | j <= i; if (i < j) Swap (A[i], A[j]); else break;
} swap (A[p], A[j]); return j;
i j66 1010 55 88 1313 33 22 1111x = 6
p r
i j66 1010 55 88 1313 33 22 1111
i j66 22 55 88 1313 33 1010 1111
i j66 22 55 88 1313 33 1010 1111
i j66 22 55 33 1313 88 1010 1111
ij66 22 55 33 1313 88 1010 1111
33 22 55 66 1313 88 1010 1111qp r
scan
scan
scan
swap
swap
final swap
Partition example
66 1010 55 88 1111 33 22 1313
33 22 55 66 1111 88 1010 1313
22 33 55 66 88 1010 1111 1313
22 33 55 66 1010 88 1111 1313
22 33 55 66 88 1010 1111 1313
Quick sortexample
Quicksort Runtimes
• Best case runtime Tbest(n) O(n log n)
• Worst case runtime Tworst(n) O(n2)
• Average case runtime Tavg(n) O(n log n)
• Expected runtime of randomized quicksort is O(n log n)
Randomized Partition
• Randomly choose an element as pivot– Every time need to do a partition, throw a die to
decide which element to use as the pivot– Each element has 1/n probability to be selected
Rand-Partition(A, p, r){ d = random(); // draw a random number between 0 and 1 index = p + floor((r-p+1) * d); // p<=index<=q swap(A[p], A[index]); Partition(A, p, r); // now use A[p] as pivot}
Running time of randomized quicksort
• The expected running time is an average of all cases
T(n) =
T(0) + T(n–1) + dn if 0 : n–1 split,T(1) + T(n–2) + dn if 1 : n–2 split,T(n–1) + T(0) + dn if n–1 : 0 split,
nknTkTn
nTn
k
1
0)1()(
1)(
Expectation
nkTn
n
k
1
0)(
2
• Fact:
• Need to Prove: T(n) ≤ c n log (n)
• Assumption: T(k) ≤ ck log (k) for 0 ≤ k ≤ n-1
• Prove by induction
nkTn
nTn
k
1
0)(
2)(
nn
nn
n
cnkk
n
cnT
n
k
8log
2
2log
2)(
221
0
ncn
ncnnT 4
log)(
ncnnT log)( If c ≥ 4
Heaps
• Heap definition
• Heapify
• Buildheap– Procedure and running time (!)
• Heapsort– Comparison with other sorting algorithms in
terms of running time, stability, and in-place.
• Priority Queue operations
Back
• A heap can be seen as a complete binary tree:
Heaps
16
14 10
8 7 9 3
2 4 1
Perfect binary tree
16 14 10 8 7 9 3 2 4 1
Referencing Heap Elements
• So…Parent(i)
{return i/2;}
Left(i)
{return 2*i;}
right(i)
{return 2*i + 1;}
Heap Operations: Heapify()
Heapify(A, i){ // precondition: subtrees rooted at l and r are heaps
l = Left(i); r = Right(i);if (l <= heap_size(A) && A[l] > A[i])
largest = l;else
largest = i;if (r <= heap_size(A) && A[r] > A[largest])
largest = r;if (largest != i) {
Swap(A, i, largest);Heapify(A, largest);
}} // postcondition: subtree rooted at i is a heap
Among A[l], A[i], A[r],which one is largest?
If violation, fix it.
Heapify() Example
16
4 10
14 7 9 3
2 8 1
16 4 10 14 7 9 3 2 8 1A =
Heapify() Example
16
4 10
14 7 9 3
2 8 1
16 10 14 7 9 3 2 8 1A = 4
Heapify() Example
16
4 10
14 7 9 3
2 8 1
16 10 7 9 3 2 8 1A = 4 14
Heapify() Example
16
14 10
4 7 9 3
2 8 1
16 14 10 7 9 3 2 8 1A = 4
Heapify() Example
16
14 10
4 7 9 3
2 8 1
16 14 10 7 9 3 2 1A = 4 8
Heapify() Example
16
14 10
8 7 9 3
2 4 1
16 14 10 8 7 9 3 2 1A = 4
Heapify() Example
16
14 10
8 7 9 3
2 4 1
16 14 10 8 7 9 3 2 4 1A =
BuildHeap()
// given an unsorted array A, make A a heap
BuildHeap(A)
{
heap_size(A) = length(A);
for (i = length[A]/2 downto 1)Heapify(A, i);
}
BuildHeap() Example
• Work through exampleA = {4, 1, 3, 2, 16, 9, 10, 14, 8, 7}
4
1 3
2 16 9 10
14 8 7
4 1 3 2 16 9 10 14 8 7A =
4
1 3
2 16 9 10
14 8 7
4 1 3 2 16 9 10 14 8 7A =
4
1 3
2 16 9 10
14 8 7
4 1 3 2 16 9 10 14 8 7A =
4
1 3
14 16 9 10
2 8 7
4 1 3 14 16 9 10 2 8 7A =
4
1 3
14 16 9 10
2 8 7
4 1 3 14 16 9 10 2 8 7A =
4
1 10
14 16 9 3
2 8 7
4 1 10 14 16 9 3 2 8 7A =
4
1 10
14 16 9 3
2 8 7
4 1 10 14 16 9 3 2 8 7A =
4
16 10
14 1 9 3
2 8 7
4 16 10 14 1 9 3 2 8 7A =
4
16 10
14 7 9 3
2 8 1
4 16 10 14 7 9 3 2 8 1A =
4
16 10
14 7 9 3
2 8 1
4 16 10 14 7 9 3 2 8 1A =
16
4 10
14 7 9 3
2 8 1
16 4 10 14 7 9 3 2 8 1A =
16
14 10
4 7 9 3
2 8 1
16 14 10 4 7 9 3 2 8 1A =
16
14 10
8 7 9 3
2 4 1
16 14 10 8 7 9 3 2 4 1A =
Analyzing BuildHeap(): Tight
• To Heapify() a subtree takes O(h) time where h is the height of the subtree– h = O(lg m), m = # nodes in subtree– The height of most subtrees is small
• Fact: an n-element heap has at most n/2h+1 nodes of height h
• CLR 6.3 uses this fact to prove that BuildHeap() takes O(n) time
n
hh
n
hh
hn
nhnT
22 log
11
log
11 22
)(
Heapsort
Heapsort(A)
{
BuildHeap(A);
for (i = length(A) downto 2)
{
Swap(A[1], A[i]);
heap_size(A) -= 1;
Heapify(A, 1);
}
}
Heapsort Example
• Work through exampleA = {4, 1, 3, 2, 16, 9, 10, 14, 8, 7}
4
1 3
2 16 9 10
14 8 7
4 1 3 2 16 9 10 14 8 7A =
Heapsort Example
• First: build a heap
16
14 10
8 7 9 3
2 4 1
16 14 10 8 7 9 3 2 4 1A =
Heapsort Example
• Swap last and first
1
14 10
8 7 9 3
2 4 16
1 14 10 8 7 9 3 2 4 16A =
Heapsort Example
• Last element sorted
1
14 10
8 7 9 3
2 4 16
1 14 10 8 7 9 3 2 4 16A =
Heapsort Example
• Restore heap on remaining unsorted elements
14
8 10
4 7 9 3
2 1 16 Heapify
14 8 10 4 7 9 3 2 1 16A =
Heapsort Example
• Repeat: swap new last and first
1
8 10
4 7 9 3
2 14 16
1 8 10 4 7 9 3 2 14 16A =
Heapsort Example
• Restore heap
10
8 9
4 7 1 3
2 14 16
10 8 9 4 7 1 3 2 14 16A =
Heapsort Example
• Repeat
9
8 3
4 7 1 2
10 14 16
9 8 3 4 7 1 2 10 14 16A =
Heapsort Example
• Repeat
8
7 3
4 2 1 9
10 14 16
8 7 3 4 2 1 9 10 14 16A =
Heapsort Example
• Repeat
1
2 3
4 7 8 9
10 14 16
1 2 3 4 7 8 9 10 14 16A =
Implementing Priority Queues
HeapMaximum(A){
return A[1];}
Implementing Priority Queues
HeapExtractMax(A){ if (heap_size[A] < 1) { error; } max = A[1]; A[1] = A[heap_size[A]] heap_size[A] --; Heapify(A, 1); return max;}
HeapExtractMax Example
16
14 10
8 7 9 3
2 4 1
16 14 10 8 7 9 3 2 4 1A =
HeapExtractMax Example
Swap first and last, then remove last
1
14 10
8 7 9 3
2 4 16
14 10 8 7 9 3 2 4 16A = 1
HeapExtractMax Example
Heapify
14
8 10
4 7 9 3
2 1
10 7 9 3 2 16A =
16
14 8 4 1
Implementing Priority Queues
HeapChangeKey(A, i, key){ if (key ≤ A[i]){ // decrease key
A[i] = key; heapify(A, i);
} else { // increase key A[i] = key;
while (i>1 & A[parent(i)]<A[i])swap(A[i], A[parent(i)];
}}
Sift down
Bubble up
HeapChangeKey Example
Increase key
16
14 10
8 7 9 3
2 4 1
16 14 10 8 7 9 3 2 4 1A =
HeapChangeKey Example
Increase key
16
14 10
15 7 9 3
2 4 1
16 14 10 7 9 3 2 4 1A = 15
HeapChangeKey Example
Increase key
16
15 10
14 7 9 3
2 4 1
16 10 7 9 3 2 4 1A = 1415
Implementing Priority QueuesHeapInsert(A, key) { heap_size[A] ++; i = heap_size[A];
A[i] = -∞; HeapChangeKey(A, i, key);
}
HeapInsert Example
HeapInsert(A, 17)
16
14 10
8 7 9 3
2 4 1
16 14 10 8 7 9 3 2 4 1A =
HeapInsert Example
HeapInsert(A, 17)
16
14 10
8 7 9 3
2 4 1
16 14 10 8 7 9 3 2 4 1A =
-∞
-∞
-∞ makes it a valid heap
HeapInsert Example
HeapInsert(A, 17)
16
14 10
8 7 9 3
2 4 1
16 10 8 9 3 2 4 1A =
17
1714 7
Now call changeKey
HeapInsert Example
HeapInsert(A, 17)
17
16 10
8 14 9 3
2 4 1
17 10 8 9 3 2 4 1A =
7
716 14
Linear time sorting
• Counting sort– Algorithm and analysis– Conditions that make counting sort linear– Stable property and memory requirement
• Radix sort– Analysis
• Comparison of different sorting algorithms– Running time– Memory requirement– Stable property
Back
Counting sort
for i 1 to kdo C[i] 0
for j 1 to ndo C[A[ j]] C[A[ j]] + 1 ⊳ C[i] = |{key = i}|
for i 2 to kdo C[i] C[i] + C[i–1] ⊳ C[i] = |{key i}|
for j n downto 1do B[C[A[ j]]] A[ j]
C[A[ j]] C[A[ j]] – 1
1.
2.
3.
4.
Initialize
Count
Compute running sum
Re-arrange
Counting sort
A: 44 11 33 44 33
B:
1 2 3 4 5
C: 11 00 22 22
1 2 3 4
C': 11 11 33 55
for i 2 to kdo C[i] C[i] + C[i–1] ⊳ C[i] = |{key i}|
3.
Loop 4: re-arrange
A: 44 11 33 44 33
B: 33
1 2 3 4 5
C: 11 11 33 55
1 2 3 4
C': 11 11 33 55
for j n downto 1do B[C[A[ j]]] A[ j]
C[A[ j]] C[A[ j]] – 1
4.
Analysisfor i 1 to k
do C[i] 0
(n)
(k)
(n)
(k)
for j 1 to ndo C[A[ j]] C[A[ j]] + 1
for i 2 to kdo C[i] C[i] + C[i–1]
for j n downto 1do B[C[A[ j]]] A[ j]
C[A[ j]] C[A[ j]] – 1
(n + k)
1.
2.
3.
4.
Stable sorting
Counting sort is a stable sort: it preserves the input order among equal elements.
A: 44 11 33 44 33
B: 11 33 33 44 44
Why this is important?What other algorithms have this property?
Radix sort
• Similar to sorting the address books
• Treat each digit as a key
• Start from the least significant bit
198099109123518183599340199540380128115295384700101594539614696382408360201039258538614386507628681328936
Most significant Least significant
Time complexity
• Sort each of the d digits by counting sort• Total cost: d (n + k)
– k = 10– Total cost: Θ(dn)
• Partition the d digits into groups of 3– Total cost: (n+103)d/3
• We work with binaries rather than decimals– Partition a binary number into groups of r bits– Total cost: (n+2r)d/r– Choose r = log n– Total cost: dn / log n– Compare with dn log n
• Catch: faster than quicksort only when n is very large
Order statistics
• Select vs Randomize Select– Note: select is different from selection sort– Algorithm– Worst-case vs average-case (expected)
running time analysis
• Worst-case Linear-time Select– Algorithm and analysis– Comparison between sorting and select
algorithmsBack
Randomized select algorithm
RAND-SELECT(A, p, q, i) ⊳ i th smallest of A[ p . . q] if p = q & i > 1 then error!r RAND-PARTITION(A, p, q)k r – p + 1 ⊳ k = rank(A[r])if i = k then return A[ r]if i < k
then return RAND-SELECT( A, p, r – 1, i )else return RAND-SELECT( A, r + 1, q, i – k )
A[r] A[r] A[r] A[r]rp q
k
77 1010 55 88 1111 33 22 1313
33 22 55 77 1111 88 1010 1313
1010
1010 88 1111 1313
88 1010
Complete example: select the 6th smallest element.
i = 6
k = 4
i = 6 – 4 = 2
k = 3
i = 2 < k
k = 2
i = 2 = k
Note: here we always used first element as pivot to do the partition (instead of rand-partition).
Running time of randomized select
• For upper bound, assume ith element always falls in larger side of partition
• The expected running time is an average of all cases
T(n) ≤
T(max(0, n–1)) + n if 0 : n–1 split,T(max(1, n–2)) + n if 1 : n–2 split,T(max(n–1, 0)) + n if n–1 : 0 split,
nknkTn
nTn
k
1
0)1,max(
1)(
Expectation
Substitution method
Assume: T(k) ≤ ck for all k < n
nkTn
nknkTn
nTn
nk
n
k
1
2
1
0)(
2)1,max(
1)(
nknc
nkTn
nTn
nk
n
nk
1
2
1
2
2)(
2)(
cncn
ncnncn
nn
nc
nT )4
(4
38
32)(
2
if c ≥ 4Therefore, T(n) = O(n)
Want to show T(n) = O(n). So need to prove T(n) ≤ cn for n > n0
Worst-case linear-time select
if i = k then return xelseif i < k
then recursively SELECT the i th smallest element in the
lower partelse recursively SELECT the (i–
k)th smallest element in the upper part
SELECT(i, n)1. Divide the n elements into groups of 5. Find
the median of each 5-element group by rote.2. Recursively SELECT the median x of the n/5
group medians to be the pivot.3. Partition around the pivot x. Let k = rank(x).4.
Same as RAND-SELECT
Developing the recurrence
if i = k then return xelseif i < k
then recursively SELECT the i th smallest element in the
lower partelse recursively SELECT the (i–
k)th smallest element in the upper part
SELECT(i, n)1. Divide the n elements into groups of 5. Find
the median of each 5-element group by rote.2. Recursively SELECT the median x of the n/5
group medians to be the pivot.3. Partition around the pivot x. Let k = rank(x).4.
T(n)
(n)
T(n/5)
(n)
T(7n/10+3)
nnTnTnT
3
107
51
)(
Solving the recurrence
if c ≥ 20 and n ≥ 60cn
ncncn
ncn
ncncn
nncncnT
)20/(
20/19
4/35
)3107()5()(
Assumption: T(k) ck for all k < n
if n ≥ 60
Dynamic programming
• Proofs of optimal substructure property• Defining recurrence • Solve DP problem without recursive calls• Example DP problems:
– Shortest path problem– LCS– Restaurant location– Knapsack– Scheduling– Other similar problems
Back
Elements of dynamic programming
• Optimal sub-structures– Optimal solutions to the original problem
contains optimal solutions to sub-problems
• Overlapping sub-problems– Some sub-problems appear in many solutions
Two steps to dynamic programming
• Formulate the solution as a recurrence relation of solutions to subproblems.
• Specify an order to solve the subproblems so you always have what you need.
Optimal subpaths
• Claim: if a path startgoal is optimal, any sub-path, startx, or xgoal, or xy, where x, y is on the optimal path, is also the shortest.
• Proof by contradiction– If the subpath between x and y is not the shortest, we can
replace it with the shorter one, which will reduce the total length of the new path => the optimal path from start to goal is not the shortest => contradiction!
– Hence, the subpath xy must be the shortest among all paths from x to y
start goalx ya
bc
b’
a + b + c is shortest
b’ < b
a + b’ + c < a + b + c
Dynamic programming illustration3 9 1 2
3 2 5 2
2 4 2 3
3 6 3 3
1 2 3 2
5 3 3 3 3
2 3 3 9 3
6 2 3 7 4
4 6 3 1 3
3 12 13 15
6 8 13 15
9 11 13 16
11 14 17 20
17 17 18 20
0
5
7
13
17
S
G
F(i-1, j) + dist(i-1, j, i, j) F(i, j) = min
F(i, j-1) + dist(i, j-1, i, j)
Trace back
3 9 1 2
3 2 5 2
2 4 2 3
3 6 3 3
1 2 3 2
5 3 3 3 3
2 3 3 9 3
6 2 3 7 4
4 6 3 1 3
3 12 13 15
6 8 13 15
9 11 13 16
11 14 17 20
17 17 18 20
0
5
7
13
17
Longest Common Subsequence
• Given two sequences x[1 . . m] and y[1 . . n], find a longest subsequence common to them both.
x: A B C B D A B
y: B D C A B A
“a” not “the”
BCBA = LCS(x, y)
functional notation, but not a function
Optimal substructure
• Notice that the LCS problem has optimal substructure: parts of the final solution are solutions of subproblems.– If z = LCS(x, y), then any prefix of z is an LCS of a prefix of
x and a prefix of y.
• Subproblems: “find LCS of pairs of prefixes of x and y”
x
y
m
nz
i
j
Finding length of LCS
• Let c[i, j] be the length of LCS(x[1..i], y[1..j])=> c[m, n] is the length of LCS(x, y)
• If x[m] = y[n]c[m, n] = c[m-1, n-1] + 1
• If x[m] != y[n]c[m, n] = max { c[m-1, n], c[m, n-1] }
x
y
m
n
DP Algorithm
• Key: find out the correct order to solve the sub-problems• Total number of sub-problems: m * n
c[i, j] =c[i–1, j–1] + 1 if x[i] = y[j],max{c[i–1, j], c[i, j–1]} otherwise.
C(i, j)
0
m
0 n
i
j
LCS Example (0)j 0 1 2 3 4 5
0
1
2
3
4
i
X[i]
A
B
C
B
Y[j] BB ACD
X = ABCB; m = |X| = 4Y = BDCAB; n = |Y| = 5Allocate array c[5,6]
ABCBBDCAB
LCS Example (1)j 0 1 2 3 4 5
0
1
2
3
4
i
A
B
C
B
BB ACD
0
0
00000
0
0
0
for i = 1 to m c[i,0] = 0 for j = 1 to n c[0,j] = 0
ABCBBDCAB
X[i]
Y[j]
LCS Example (2)j 0 1 2 3 4 5
0
1
2
3
4
i
A
B
C
B
BB ACD
0
0
00000
0
0
0
if ( Xi == Yj ) c[i,j] = c[i-1,j-1] + 1 else c[i,j] = max( c[i-1,j], c[i,j-1] )
0
ABCBBDCAB
X[i]
Y[j]
LCS Example (3)j 0 1 2 3 4 5
0
1
2
3
4
i
A
B
C
B
BB ACD
0
0
00000
0
0
0
if ( Xi == Yj ) c[i,j] = c[i-1,j-1] + 1 else c[i,j] = max( c[i-1,j], c[i,j-1] )
0 0 0
ABCBBDCAB
X[i]
Y[j]
LCS Example (4)j 0 1 2 3 4 5
0
1
2
3
4
i
A
B
C
B
BB ACD
0
0
00000
0
0
0
if ( Xi == Yj ) c[i,j] = c[i-1,j-1] + 1 else c[i,j] = max( c[i-1,j], c[i,j-1] )
0 0 0 1
ABCBBDCAB
X[i]
Y[j]
LCS Example (5)j 0 1 2 3 4 5
0
1
2
3
4
i
A
B
C
B
BB ACD
0
0
00000
0
0
0
if ( Xi == Yj ) c[i,j] = c[i-1,j-1] + 1 else c[i,j] = max( c[i-1,j], c[i,j-1] )
000 1 1
ABCBBDCAB
X[i]
Y[j]
LCS Example (6)j 0 1 2 3 4 5
0
1
2
3
4
i
A
B
C
B
BB ACD
0
0
00000
0
0
0
if ( Xi == Yj ) c[i,j] = c[i-1,j-1] + 1 else c[i,j] = max( c[i-1,j], c[i,j-1] )
0 0 10 1
1
ABCBBDCAB
X[i]
Y[j]
LCS Example (7)j 0 1 2 3 4 5
0
1
2
3
4
i
A
B
C
B
BB ACD
0
0
00000
0
0
0
if ( Xi == Yj ) c[i,j] = c[i-1,j-1] + 1 else c[i,j] = max( c[i-1,j], c[i,j-1] )
1000 1
1 1 11
ABCBBDCAB
X[i]
Y[j]
LCS Example (8)j 0 1 2 3 4 5
0
1
2
3
4
i
A
B
C
B
BB ACD
0
0
00000
0
0
0
if ( Xi == Yj ) c[i,j] = c[i-1,j-1] + 1 else c[i,j] = max( c[i-1,j], c[i,j-1] )
1000 1
1 1 1 1 2
ABCBBDCAB
X[i]
Y[j]
LCS Example (9)j 0 1 2 3 4 5
0
1
2
3
4
i
A
B
C
B
BB ACD
0
0
00000
0
0
0
if ( Xi == Yj )c[i,j] = c[i-1,j-1] + 1
else c[i,j] = max( c[i-1,j], c[i,j-1] )
1000 1
21 1 11
1 1
ABCBBDCAB
X[i]
Y[j]
LCS Example (10)j 0 1 2 3 4 5
0
1
2
3
4
i
A
B
C
B
BB ACD
0
0
00000
0
0
0
if ( Xi == Yj ) c[i,j] = c[i-1,j-1] + 1 else c[i,j] = max( c[i-1,j], c[i,j-1] )
1000 1
1 21 11
1 1 2
ABCBBDCAB
X[i]
Y[j]
LCS Example (11)j 0 1 2 3 4 5
0
1
2
3
4
i
A
B
C
B
BB ACD
0
0
00000
0
0
0
if ( Xi == Yj )c[i,j] = c[i-1,j-1] + 1
else c[i,j] = max( c[i-1,j], c[i,j-1] )
1000 1
1 21 1
1 1 2
1
22
ABCBBDCAB
X[i]
Y[j]
LCS Example (12)j 0 1 2 3 4 5
0
1
2
3
4
i
A
B
C
B
BB ACD
0
0
00000
0
0
0
if ( Xi == Yj ) c[i,j] = c[i-1,j-1] + 1 else c[i,j] = max( c[i-1,j], c[i,j-1] )
1000 1
1 21 1
1 1 2
1
22
1
ABCBBDCAB
X[i]
Y[j]
LCS Example (13)j 0 1 2 3 4 5
0
1
2
3
4
i
A
B
C
B
BB ACD
0
0
00000
0
0
0
if ( Xi == Yj )c[i,j] = c[i-1,j-1] + 1
else c[i,j] = max( c[i-1,j], c[i,j-1] )
1000 1
1 21 1
1 1 2
1
22
1 1 2 2
ABCBBDCAB
X[i]
Y[j]
LCS Example (14)j 0 1 2 3 4 5
0
1
2
3
4
i
A
B
C
B
BB ACD
0
0
00000
0
0
0
if ( Xi == Yj ) c[i,j] = c[i-1,j-1] + 1 else c[i,j] = max( c[i-1,j], c[i,j-1] )
1000 1
1 21 1
1 1 2
1
22
1 1 2 2 3
ABCBBDCAB
X[i]
Y[j]
LCS Algorithm Running Time
• LCS algorithm calculates the values of each entry of the array c[m,n]
• So what is the running time?
O(m*n)
since each c[i,j] is calculated in constant time, and there are m*n elements in the array
How to find actual LCS
• The algorithm just found the length of LCS, but not LCS itself.• How to find the actual LCS?• For each c[i,j] we know how it was acquired:
• A match happens only when the first equation is taken
• So we can start from c[m,n] and go backwards, remember x[i] whenever c[i,j] = c[i-1, j-1]+1.
2
2 3
2 For example, here c[i,j] = c[i-1,j-1] +1 = 2+1=3
otherwise]),1[],1,[max(
],[][ if1]1,1[],[
jicjic
jyixjicjic
Finding LCSj 0 1 2 3 4 5
0
1
2
3
4
i
A
B
C
BB ACD
0
0
00000
0
0
0
1000 1
1 21 1
1 1 2
1
22
1 1 2 2 3B
X[i]
Y[j]
Time for trace back: O(m+n).
Finding LCS (2)j 0 1 2 3 4 5
0
1
2
3
4
i
A
B
C
BB ACD
0
0
00000
0
0
0
1000 1
1 21 1
1 1 2
1
22
1 1 2 2 3B
B C BLCS (reversed order):
LCS (straight order): B C B (this string turned out to be a palindrome)
X[i]
Y[j]
Restaurant location problem
• You work in the fast food business• Your company plans to open up new restaurants in
Texas along I-35
• Towns along the highway called t1, t2, …, tn
• Restaurants at ti has estimated annual profit pi
• No two restaurants can be located within 10 miles of each other due to some regulation
• Your boss wants to maximize the total profit• You want a big bonus
10 mile
A DP algorithm
• Suppose you’ve already found the optimal solution
• It will either include tn or not include tn
• Case 1: tn not included in optimal solution
– Best solution same as best solution for t1 , …, tn-1
• Case 2: tn included in optimal solution
– Best solution is pn + best solution for t1 , …, tj , where j < n is the largest index so that dist(tj, tn) ≥ 10
Recurrence formulation
• Let S(i) be the total profit of the optimal solution when the first i towns are considered (not necessarily selected)– S(n) is the optimal solution to the complete problem
S(n-1)
S(j) + pn j < n & dist (tj, tn) ≥ 10S(n) = max
S(i-1)
S(j) + pi j < i & dist (tj, ti) ≥ 10S(i) = max
Generalize
Number of sub-problems: n. Boundary condition: S(0) = 0.
Dependency: ii-1jS
Example
• Natural greedy 1: 6 + 3 + 4 + 12 = 25• Natural greedy 2: 12 + 9 + 3 = 24
5 2 2 6 6 63 10 7
6 7 9 8 3 3 2 4 12 5
Distance (mi)
Profit (100k)
6 7 9 9 10 12 12 14 26 26S(i)
S(i-1)
S(j) + pi j < i & dist (tj, ti) ≥ 10S(i) = max
100
07 3 4 12
dummy
Optimal: 26
Complexity
• Time: (nk), where k is the maximum number of towns that are within 10 miles to the left of any town– In the worst case, (n2)
– Can be improved to (n) with some preprocessing tricks
• Memory: Θ(n)
Knapsack problem
Three versions:
0-1 knapsack problem: take each item or leave it
Fractional knapsack problem: items are divisible
Unbounded knapsack problem: unlimited supplies of each item.
Which one is easiest to solve?
•Each item has a value and a weight•Objective: maximize value•Constraint: knapsack has a weight
limitation
We study the 0-1 problem today.
Formal definition (0-1 problem)
• Knapsack has weight limit W• Items labeled 1, 2, …, n (arbitrarily)
• Items have weights w1, w2, …, wn
– Assume all weights are integers
– For practical reason, only consider wi < W
• Items have values v1, v2, …, vn
• Objective: find a subset of items, S, such that iS wi W and iS vi is maximal among all such (feasible) subsets
A DP algorithm
• Suppose you’ve find the optimal solution S
• Case 1: item n is included
• Case 2: item n is not included
Total weight limit:W
wn
Total weight limit:W
Find an optimal solution using items 1, 2, …, n-1 with weight limit W - wn
wn
Find an optimal solution using items 1, 2, …, n-1 with weight limit W
Recursive formulation
• Let V[i, w] be the optimal total value when items 1, 2, …, i are considered for a knapsack with weight limit w
=> V[n, W] is the optimal solution
V[n, W] = maxV[n-1, W-wn] + vn
V[n-1, W]
Generalize
V[i, w] = maxV[i-1, w-wi] + vi item i is taken
V[i-1, w] item i not taken
V[i-1, w] if wi > w item i not taken
Boundary condition: V[i, 0] = 0, V[0, w] = 0. Number of sub-problems = ?
Example
• n = 6 (# of items)• W = 10 (weight limit)• Items (weight, value):
2 24 33 35 62 46 9
0
0
0
0
0
0
00000000000
w 0 1 2 3 4 5 6 7 8 9 10
425
6
4
3
2
1
i
96
65
33
34
22
viwi
maxV[i-1, w-wi] + vi item i is taken
V[i-1, w] item i not taken
V[i-1, w] if wi > w item i not taken
V[i, w] =
V[i, w]
V[i-1, w]V[i-1, w-wi]
6
wi
5
107400
1310764400
9633200
8653200
555532200
2222222200
00000000000
w 0 1 2 3 4 5 6 7 8 9 10
i wi vi
1 2 2
2 4 3
3 3 3
4 5 6
5 2 4
6 6 9
maxV[i-1, w-wi] + vi item i is taken
V[i-1, w] item i not taken
V[i-1, w] if wi > w item i not taken
V[i, w] =
2
4
3
6
5
6
7
5
9
6
8
10
9 11
8
3
12 13
13 15
107400
1310764400
9633200
8653200
555532200
2222222200
00000000000
w 0 1 2 3 4 5 6 7 8 9 10
i wi vi
1 2 2
2 4 3
3 3 3
4 5 6
5 2 4
6 6 9
2
4
3
6
5
6
7
5
9
6
8
10
9 11
8
3
12 13
13 15
Item: 6, 5, 1
Weight: 6 + 2 + 2 = 10
Value: 9 + 4 + 2 = 15
Optimal value: 15
Time complexity
• Θ (nW)• Polynomial?
– Pseudo-polynomial– Works well if W is small
• Consider following items (weight, value):(10, 5), (15, 6), (20, 5), (18, 6)
• Weight limit 35– Optimal solution: item 2, 4 (value = 12). Iterate: 2^4 = 16 subsets– Dynamic programming: fill up a 4 x 35 = 140 table entries
• What’s the problem?– Many entries are unused: no such weight combination– Top-down may be better
Use DP algorithm to solve new problems
• Directly map a new problem to a known problem• Modify an algorithm for a similar task• Design your own
– Think about the problem recursively– Optimal solution to a larger problem can be computed
from the optimal solution of one or more subproblems– These sub-problems can be solved in certain
manageable order– Works nicely for naturally ordered data such as
strings, trees, some special graphs– Trickier for general graphs
• The text book has some very good exercises.
Greedy algorithm
• Proofs that show locally optimal selection leads to globally optimal solution
• Example problems that can be solved by greedy algorithm:– Uniform-profit restaurant location problem– Fractional knap-sack problem– Other similar problems (e.g. event scheduling
without weights/values for different event)
Back
Greedy algorithm for restaurant location problem
select t1
d = 0;
for (i = 2 to n)
d = d + dist(ti, ti-1);
if (d >= min_dist)
select ti
d = 0;
end
end
5 2 2 6 6 63 10 7
d 0 5 7 9 150
6 9 150
100
7
Complexity
• Time: Θ(n)
• Memory: – Θ(n) to store the input– Θ(1) for greedy selection
Knapsack problem
Three versions:
0-1 knapsack problem: take each item or leave it
Fractional knapsack problem: items are divisible
Unbounded knapsack problem: unlimited supplies of each item.
Which one is easiest to solve?
•Each item has a value and a weight•Objective: maximize value•Constraint: knapsack has a weight
limitation
We can solve the fractional knapsack problem using greedy algorithm
Greedy algorithm for fractional knapsack problem
• Compute value/weight ratio for each item• Sort items by their value/weight ratio into
decreasing order– Call the remaining item with the highest ratio the most
valuable item (MVI)
• Iteratively: – If the weight limit can not be reached by adding MVI
• Select MVI
– Otherwise select MVI partially until weight limit
Example• Weight limit: 10
1.5
2
1.2
1
0.75
1
$ / LB
966
425
654
333
342
221
Value ($)
Weight (LB)
item
Example• Weight limit: 10
• Take item 5– 2 LB, $4
• Take item 6– 8 LB, $13
• Take 2 LB of item 4– 10 LB, 15.4
item Weight (LB)
Value ($)
$ / LB
5 2 4 2
6 6 9 1.5
4 5 6 1.2
1 2 2 1
3 3 3 1
2 4 3 0.75
Why is greedy algorithm for fractional knapsack problem valid?
• Claim: the optimal solution must contain the MVI as much as possible (either up to the weight limit or until MVI is exhausted)
• Proof by contradiction: suppose that the optimal solution does not use all available MVI (i.e., there is still w (w < W) units of MVI left while we choose other items)– We can replace w pounds of less valuable items by MVI– The total weight is the same, but with value higher than the
“optimal”– Contradiction
w w w w
Graphs
• Representations– Adjacency list vs adjacency matrix– Cons and pros
• MST– Kruskal’s– Prim’s
Back
Representing Graphs
• Assume V = {1, 2, …, n}• An adjacency matrix represents the graph as a n
x n matrix A:– A[i, j] = 1 if edge (i, j) E
= 0 if edge (i, j) E
• For weighted graph– A[i, j] = wij if edge (i, j) E
= 0 if edge (i, j) E
• For undirected graph– Matrix is symmetric: A[i, j] = A[j, i]
Graphs: Adjacency Matrix
• Example:
1
2 4
3
A 1 2 3 4
1
2
3 ??4
Graphs: Adjacency Matrix
• Example:
1
2 4
3
A 1 2 3 4
1 0 1 1 0
2 0 0 1 0
3 0 0 0 0
4 0 0 1 0
How much storage does the adjacency matrix require?A: O(V2)
Graphs: Adjacency Matrix
• Example:
1
2 4
3 4
3
2
0100
1011
0101
01101
4321A
Undirected graph
Graphs: Adjacency Matrix
• Example:
1
2 4
3
5
6
9 4
4
3
2
0400
4096
0905
06501
4321A
Weighted graph
Graphs: Adjacency Matrix
• Time to answer if there is an edge between vertex u and v: Θ(1)
• Memory required: Θ(n2) regardless of |E|– Usually too much storage for large graphs– But can be very efficient for small graphs
• Most large interesting graphs are sparse– E.g., road networks (due to limit on junctions)– For this reason the adjacency list is often a
more appropriate representation
Graphs: Adjacency List
• Adjacency list: for each vertex v V, store a list of vertices adjacent to v
• Example:– Adj[1] = {2,3}– Adj[2] = {3}– Adj[3] = {}– Adj[4] = {3}
• Variation: can also keep a list of edges coming into vertex
1
2 4
3
Graph representations
• Adjacency list
1
2 4
3
2 3
3
3
How much storage does the adjacency list require?A: O(V+E)
Graph representations
• Undirected graph
1
2 4
3 4
3
2
0100
1011
0101
01101
4321A
2 3
1
3
3
1 2 4
Graph representations
• Weighted graph
1
2 4
3
5
6
9 4 4
3
2
0400
4096
0905
06501
4321A
2,5 3,6
1,5 3,9
3,4
1,6 2,9 4,4
Graphs: Adjacency List
• How much storage is required?• For directed graphs
– |adj[v]| = out-degree(v)– Total # of items in adjacency lists is
out-degree(v) = |E|
• For undirected graphs– |adj[v]| = degree(v) – # items in adjacency lists is
degree(v) = 2 |E|
• So: Adjacency lists take (V+E) storage• Time needed to test if edge (u, v) E is O(n)
Tradeoffs between the two representations
Adj Matrix Adj List
test (u, v) E Θ(1) O(n)
Degree(u) Θ(n) O(n)
Memory Θ(n2) Θ(n+m)
Edge insertion Θ(1) Θ(1)
Edge deletion Θ(1) O(n)
Graph traversal Θ(n2) Θ(n+m)
|V| = n, |E| = m
Both representations are very useful and have different properties, although adjacency lists are probably better for more problems
Good luck with your exam!