CPSC 311, Fall 2009 1

CPSC 311Analysis of Algorithms

More Graph AlgorithmsProf. Jennifer Welch

Fall 2009

CPSC 311, Fall 2009 2

Minimum Spanning Tree

7

1645

6 8

11

15

14

17

10

13

3

12

29

18

find subset of edges that span all the nodes,create no cycle, and minimize sum of weights

CPSC 311, Fall 2009 3

Facts About MSTs There can be many spanning trees of

a graph In fact, there can be many minimum

spanning trees of a graph But if every edge has a unique

weight, then there is a unique MST

CPSC 311, Fall 2009 4

Uniqueness of MST Suppose in contradiction there are 2

MSTs, M1 and M2. Let e be edge with minimum weight that

is one but not the other (say it is in M1).

If e is added to M2, a cycle is formed. Let e' be an edge in the cycle that is not

in M1

CPSC 311, Fall 2009 5

Uniqueness of MST

e: in M1 but not M2

e': in M2 but not M1; wt is less than wt of e

M2:

Replacing e with e' creates a new MST M3 whose weight is less than that of M2

CPSC 311, Fall 2009 6

Generic MST Algorithm input: weighted undirected graph

G = (V,E,w) T := empty set while T is not yet a spanning tree of G

find an edge e in E s.t. T U {e} is a subgraph of some MST of G

add e to T return T (as MST of G)

CPSC 311, Fall 2009 7

Kruskal's MST algorithm

7

1645

6 8

11

15

14

17

10

13

3

12

29

18

consider the edges in increasing order of weight,add in an edge iff it does not cause a cycle

CPSC 311, Fall 2009 8

Kruskal's Algorithm as a Special Case of Generic Alg. Consider edges in increasing order of

weight Add the next edge iff it doesn't cause

a cycle At any point, T is a forest (set of

trees); eventually T is a single tree

CPSC 311, Fall 2009 9

Kruskal's Algorithm input: G = (V,E,w) // weighted graph T := Ø // subset of E sort E by increasing weights for each (u,v) in E in sorted order do

if T U {u,v} has no cycle then T := T U {(u,v)}

return (V,T)

CPSC 311, Fall 2009 10

Why is Kruskal's Greedy? Algorithm manages a set of edges s.t.

these edges are a subset of some MST At each iteration:

choose an edge so that the MST-subset property remains true

subproblem left is to do the same with the remaining edges

Always try to add cheapest available edge that will not violate the tree property locally optimal choice

CPSC 311, Fall 2009 11

Correctness of Kruskal's Alg. Let e1, e2, …, en-1 be sequence of

edges chosen Clearly they form a spanning tree Suppose it is not minimum weight Let ei be the edge where the

algorithm goes wrong {e1,…,ei-1} is part of some MST M but {e1,…,ei} is not part of any MST

CPSC 311, Fall 2009 12

Correctness of Kruskal's Alg.

white edges are part of MST M, which contains e1 to ei-1,but not ei

M: ei, forms a cycle in M

e* :min wt. edge in cycle not ine1 to ei-1

replacing e* w/ ei formsa spanning tree withsmaller weight than M,contradiction!

wt(e*) > wt(ei)

CPSC 311, Fall 2009 13

Note on Correctness Proof Argument on previous slide works for

case when every edge has a unique weight

Algorithm also works when edge weights are not necessarily correct

Modify proof on previous slide: contradiction is reached to assumption that ei is not part of any MST

CPSC 311, Fall 2009 14

Implementing Kruskal's Alg. Sort edges by weight

efficient algorithms known How to test quickly if adding in the

next edge would cause a cycle? use Disjoint Sets data structure…

CPSC 311, Fall 2009 15

Disjoint Sets in Kruskal's Alg. Use Disjoint Sets data structure to test whether

adding an edge to a set of edges would cause a cycle

Each set represents nodes that are connected using edges chosen so far

Initially put each node in a set by itself using Make-Set

When testing edge (u,v), call Find-Set to check if u and v are in the same set if so, then a cycle would result if (u,v) is chosen

When edge (u,v) is chosen, call Union(u,v)

CPSC 311, Fall 2009 16

Kruskal's Algorithm #2 input: G = (V,E,w) // weighted graph T := Ø // subset of E sort E by increasing weights for each v in V do MakeSet(v) for each (u,v) in E in sorted order do

a := FindSet(u) b := FindSet(v) if a ≠ b then

T := T U {(u,v)} Union(a,b)

return (V,T)

CPSC 311, Fall 2009 17

Running Time of Kruskal's MST Algorithm Sorting the edges takes O(E log E) time The rest of the time is proportional to the total time taken by all

the calls to MakeSet, FindSet, and Union. Number of calls to MakeSet is |V| If we unravel the for loop, we can see:

total number of calls to FindSet is O(E) total number of calls to Union is O(V)

So number of operations is O(V+E), with |V| MakeSet ops. If the Disjoint Sets data structure is implemented using union by

rank and path compression, the time for all the Disjoint Sets operations is O((V+E) log*V)

Total time is O(E log E), since graph is connected, so V ≤ E + 1, and this equals O(E log V), since E is O(V2) and log V2 = 2log V

CPSC 311, Fall 2009 18

Another Greedy MST Alg. Kruskal's algorithm maintains a forest

that grows until it forms a spanning tree Alternative idea is keep just one tree and

grow it until it spans all the nodes Prim's algorithm

At each iteration, choose the minimum weight outgoing edge to add greedy!

CPSC 311, Fall 2009 19

Idea of Prim's Algorithm Instead of growing the MST as possibly

multiple trees that eventually all merge, grow the MST from a single node, so that there is only one tree at any point.

Also a special case of the generic algorithm: at each step, add the minimum weight edge that goes out from the tree constructed so far.

CPSC 311, Fall 2009 20

Prim's Algorithm input: weighted undirected graph G = (V,e,w) T := empty set S := {any node in V} while |T| < |V| - 1 do

let (u,v) be a min wt. outgoing edge (u in S, v not in S) add (u,v) to T add v to S

return (S,T) (as MST of G)

CPSC 311, Fall 2009 21

Prim's Algorithm Example

a

b

h

i

c

g

d

f

e

4

8 7

8

11

2

7 6

1 2

4 14

9

10

CPSC 311, Fall 2009 22

Correctness of Prim's Algorithm Let Ti be the tree represented by (S,T)

at the end of iteration i. Show by induction on i that Ti is a

subtree of some MST of G. Basis: i = 0 (before first iteration). T0

contains just a single node, and thus is a subtree of every MST of G.

CPSC 311, Fall 2009 23

Correctness of Prim's Algorithm Induction: Assume Ti is a subtree of

some MST M. We must show Ti+1 is a subtree of some MST.

Let (u,v) be the edge added in iteration i+1.

u vTi

Ti+1

Case 1: (u,v) is in M.Then Ti+1 is also asubtree of M.

CPSC 311, Fall 2009 24

Correctness of Prim's AlgorithmCase 2: (u,v) is not in M. There is a path P in M from u to v, since M

spans G. Let (x,y) be the first edge in P with one

endpoint in Ti and the other not in Ti.

u v

Ti

x

yP

CPSC 311, Fall 2009 25

Correctness of Prim's Algorithm Let M' = M - {(x,y)} U {(u,v)} M' is also a spanning tree of G. w(M') = w(M) - w(x,y) + w(u,v) ≤ w(M)

since (u,v) is min wt outgoing edge So M' is also an MST and

Ti+1 is subtree

M' u v

Ti

x

y

Ti+1

CPSC 311, Fall 2009 26

Implementing Prim's Algorithm How do we find minimum weight

outgoing edge? First cut: scan all adjacency lists at

each iteration. Results in O(VE) time. Try to do better.

CPSC 311, Fall 2009 27

Implementing Prim's Algorithm Idea: have each node not yet in the

tree keep track of its best (cheapest) edge to the tree constructed so far.

To find min wt. outgoing edge, find minimum among these values use a priority queue to store the best edge

info (insert and extract-min operations)

CPSC 311, Fall 2009 28

Implementing Prim's Algorithm When a node v is added to T, some

other nodes might have their best edges affected, but only neighbors of v add decrease-key operation to the priority

queueu

vTi

Ti+1 w

w's best edge to Ti

check if this edge is cheaper for w

x

x's best edge to Ti

v's best edge to Ti

CPSC 311, Fall 2009 29

Details on Prim's AlgorithmAssociate with each node v two fields: best-wt[v] : if v is not yet in the tree, then

it holds the min. wt. of all edges from v to a node in the tree. Initially infinity.

best-node[v] : if v is not yet in the tree, then it holds the name of the node u in the tree s.t. w(v,u) is v's best-wt. Initially nil.

CPSC 311, Fall 2009 30

Details on Prim's Algorithm input: G = (V,E,w)// initialization initialize priority queue Q to contain all

nodes, using best-wt values as keys let v0 be any node in V decrease-key(Q,v0,0)

// last line means change best-wt[v0] to 0 and adjust Q accordingly

CPSC 311, Fall 2009 31

Details on Prim's Algorithm while Q is not empty do

u := extract-min(Q) // node w/ smallest best-wt

if u is not v0 then add (u,best-node[u]) to T for each neighbor v of u do

if v is in Q and w(u,v) < best-wt[v] then best-node[v] := u decrease-key(Q,v,w(u,v))

return (V,T) // as MST of G

CPSC 311, Fall 2009 32

Running Time of Prim's AlgorithmDepends on priority queue implementation. Let Tins be time for insert Tdec be time for decrease-key Tex be time for extract-minThen we have |V| inserts and one decrease-key in the

initialization: O(VTins+Tdec) |V| iterations of while

one extract-min per iteration: O(VTex) total

CPSC 311, Fall 2009 33

Running Time of Prim's Algorithm Each iteration of while includes a for

loop. Number of iterations of for loop varies,

depending on how many neighbors the current node has

Total number of iterations of for loop is O(E).

Each iteration of for loop: one decrease key, so O(ETdec) total

CPSC 311, Fall 2009 34

Running Time of Prim's Algorithm O(V(Tins + Tex) + ETdec) If priority queue is implemented with

a binary heap, then Tins = Tex = Tdec = O(log V) total time is O(E log V)

(Think about how to implement decrease-key in O(log V) time.)

CPSC 311, Fall 2009 35

Shortest Paths in a Graph We already saw the Floyd-Warshall

algorithm to compute all pairs of shortest paths

Let's review two important single-source shortest path algorithms: Dijkstra's algorithm Bellman-Ford algorithm

CPSC 311, Fall 2009 36

Single Source Shortest Path Problem Given: directed or undirected graph

G = (V,E,w) and source node s in V Find: For each t in V, a path in G from

s to t with minimum weight Warning! Negative weights are a

problem:

s t4

5

CPSC 311, Fall 2009 37

Shortest Path Tree Result of a SSSP algorithm can be

viewed as a tree rooted at the source Why not use breadth-first search? Works fine if all weights are the same:

weight of each path is (a multiple of) the number of edges in the path

Doesn't work when weights are different

CPSC 311, Fall 2009 38

Dijkstra's SSSP Algorithm Assumes all edge weights are nonnegative Similar to Prim's MST algorithm Start with source node s and iteratively

construct a tree rooted at s Each node keeps track of tree node that

provides cheapest path from s (not just cheapest path from any tree node)

At each iteration, include the node whose cheapest path from s is the overall cheapest

CPSC 311, Fall 2009 39

Prim's vs. Dijkstra's

s

5

4

1

6

Prim's MST

s

5

4

1

6

Dijkstra's SSSP

CPSC 311, Fall 2009 40

Implementing Dijkstra's Alg. How can each node u keep track of its best

path from s? Keep an estimate, d[u], of shortest path

distance from s to u Use d as a key in a priority queue When u is added to the tree, check each of

u's neighbors v to see if u provides v with a cheaper path from s: compare d[v] to d[u] + w(u,v)

CPSC 311, Fall 2009 41

Dijkstra's Algorithm input: G = (V,E,w) and source node s// initialization d[s] := 0 d[v] := infinity for all other nodes v initialize priority queue Q to contain

all nodes using d values as keys

CPSC 311, Fall 2009 42

Dijkstra's Algorithm while Q is not empty do

u := extract-min(Q) for each neighbor v of u do

if d[u] + w(u,v) < d[v] thend[v] := d[u] + w(u,v)decrease-key(Q,v,d[v])parent(v) := u

CPSC 311, Fall 2009 43

Dijkstra's Algorithm Example

a b

d e

c

2

84 9

2

4

12

10 6 3

source is node a

0 1 2 3 4 5Q abcd

ebcde cde de d Ø

d[a]

0 0 0 0 0 0

d[b]

∞ 2 2 2 2 2

d[c] ∞ 12 10 10 10 10

d[d]

∞ ∞ ∞ 16 13 13

d[e]

∞ ∞ 11 11 11 11

iteration

CPSC 311, Fall 2009 44

Correctness of Dijkstra's Alg. Let Ti be the tree constructed after i-th

iteration of while loop: nodes not in Q edges indicated by parent variables

Show by induction on i that the path in Ti

from s to u is a shortest path and has distance d[u], for all u in Ti.

Basis: i = 1. s is the only node in T1 and d[s] = 0.

CPSC 311, Fall 2009 45

Correctness of Dijkstra's Alg. Induction: Assume Ti-1 is a correct shortest path

tree and show for Ti. Let u be the node added in iteration i. Let x = parent(u).

s x

Ti-1

u

Ti

Need to show path in Ti from s to u is a shortest path, and has distance d[u]

CPSC 311, Fall 2009 46

Correctness of Dijkstra's Alg

s

x

Ti-1

u

Ti

P, path in Ti

from s to u

(a,b) is first edge in P' thatleaves Ti-1 (i.e., a is in Ti-1 but b is not)

a

b

P', anotherpath from s to u

CPSC 311, Fall 2009 47

Let P1 be part of P' before (a,b).

Let P2 be part of P' after (a,b).

w(P') = w(P1) + w(a,b) + w(P2)

≥ w(P1) + w(a,b) (nonneg wts)

≥ (wt of path in Ti-1 from s to a) + w(a,b) (inductive hypothesis)

≥ (wt of path in Ti-1 from s to x) + w(x,u) (alg chose u in

iteration i and d-values are accurate, by inductive hyp.)

= w(P).

So P is a shortest path, and d[u] is accurate after iteration i.

Correctness of Dijkstra's Alg

s

x

Ti-1

u

Ti

a

bP'

P

CPSC 311, Fall 2009 48

Running Time of Dijstra's Alg. initialization: insert each node once

O(V Tins)

O(V) iterations of while loop one extract-min per iteration => O(V Tex) for loop inside while loop has variable number of

iterations…

For loop has O(E) iterations total one decrease-key per iteration => O(E Tdec)

Total is O(V (Tins + Tex) + E Tdec)

CPSC 311, Fall 2009 49

Using Fancier Heap Implementations O(V(Tins + Tex) + E Tdec) If priority queue is implemented with a

binary heap, then Tins = Tex = Tdec = O(log V) total time is O(E log V)

There are fancier implementations of the priority queue, such as Fibonacci heap: Tins = O(1), Tex = O(log V), Tdec = O(1) (amortized) total time is O(V log V + E)

CPSC 311, Fall 2009 50

Using Simpler Heap Implementations O(V(Tins + Tex) + E Tdec) If graph is dense, so that |E| = (V2), then it

doesn't help to make Tins and Tex to be at most O(V).

Instead, focus on making Tdec be small, say constant.

Implement priority queue with an unsorted array: Tins = O(1), Tex = O(V), Tdec = O(1) total is O(V2)

CPSC 311, Fall 2009 51

What About Negative Edge Weights? Dijkstra's SSSP algorithm requires all

edge weights to be nonnegative even more restrictive than outlawing

negative weight cycles Bellman-Ford SSSP algorithm can

handle negative edge weights even "handles" negative weight cycles by

reporting they exist

CPSC 311, Fall 2009 52

Bellman-Ford Idea Consder each edge (u,v) and see if u

offers v a cheaper path from s compare d[v] to d[u] + w(u,v)

Repeat this process |V| - 1 times to ensure that accurate information propagates from s, no matter what order the edges are considered in

CPSC 311, Fall 2009 53

Bellman-Ford SSSP Algorithm input: directed or undirected graph G = (V,E,w)//initialization initialize d[v] to infinity and parent[v] to nil for all v in V other

than the source initialize d[s] to 0 and parent[s] to s// main body for i := 1 to |V| - 1 do

for each (u,v) in E do // consider in arbitrary order if d[u] + w(u,v) < d[v] then

d[v] := d[u] + w(u,v) parent[v] := u

// continued on next slide

CPSC 311, Fall 2009 54

Bellman-Ford SSSP Algorithm// check for negative weight cycles for each (u,v) in E do

if d[u] + w(u,v) < d[v] then output "negative weight cycle exists"

CPSC 311, Fall 2009 55

Running Time of Bellman-Ford O(V) iterations of outer for loop O(E) iterations of inner for loop O(VE) time total

CPSC 311, Fall 2009 56

Bellman-Ford Example

s

c

a

b3

—44

2

1

process edges in order(c,b)(a,b)(c,a)(s,a)(s,c)

<board work>

CPSC 311, Fall 2009 57

Correctness of Bellman-FordAssume no negative-weight cycles.Lemma: d[v] is never an underestimate of the

actual shortest path distance from s to v.Lemma: If there is a shortest s-to-v path

containing at most i edges, then after iteration i of the outer for loop, d[v] is at most the actual shortest path distance from s to v.

Theorem: Bellman-Ford is correct.Proof: Follows from these 2 lemmas and fact

that every shortest path has at most |V| - 1 edges.

CPSC 311, Fall 2009 58

Correctness of Bellman-Ford Suppose there is a negative weight

cycle. Then the distance will decrease even

after iteration |V| - 1 shortest path distance is negative infinity

This is what the last part of the code checks for.

CPSC 311, Fall 2009 59

Speeding Up Bellman-Ford The previous example would have

converged faster if we had considered the edges in a different order in the for loop move outward from s

If the graph is a DAG (no cycles), we can fully exploit this idea to speed up the running time

CPSC 311, Fall 2009 60

DAG Shortest Path Algorithm input: directed graph G = (V,E,w) and

source node s in V topologically sort G d[v] := infinity for all v in V d[s] := 0 for each u in V in topological sort order do

for each neighbor v of u do d[v] := min{d[v],d[u] + w(u,v)}

CPSC 311, Fall 2009 61

DAG Shortest Path Algorithm Running Time is O(V + E). Example: <board work>