January 28, 2015CS21 Lecture 101 CS21 Decidability and Tractability Lecture 10 January 28, 2015.
CS420 lecture one Problems, algorithms, decidability, tractability.
-
Upload
ira-simmons -
Category
Documents
-
view
222 -
download
1
Transcript of CS420 lecture one Problems, algorithms, decidability, tractability.
CS420 lecture oneProblems, algorithms, decidability, tractability
Problems, solutions, algorithms
• In this course we study questions such as:– given a problem, how do we find an (efficient)
algorithm solving it?– How do we measure the complexity (time, space
requirements) of an algorithm.• Problem: set of instances of a question– is 2 a prime?, .., is 4 a prime?, is 2100000 a prime
• Solution: set of answers– yes, .., no, no
Algorithm• Problem: question + input(s)• Algorithm: effective procedure– mapping input to output
• effective: unambiguous, executable– Turing defined it as: "like a Turing machine"– program = effective procedure
• the size of the problem: an integer n,• # inputs (for sorting problem)• #digits of input (for prime problem)• sometimes more than one integer
Is there an algorithm for each problem?
No1. the problem needs to be effectively specified
"how many angels can dance on the head of a pin?" not effective
2. even if it is effectively specified, there is not always an algorithm to provide an answer
Ulam's problem
f(n): if (n==1) return 1 else if (odd(n)) return f(3*n+1) else return f(n/2)
Try a few inputs, does f(n) always stop?
Ulam's problem
f(n): if (n==1) return 1 else if (odd(n)) return f(3*n+1) else return f(n/2)
f(1) f(2), f(4), f(8), f(2n) .... f(3) f(5) f(7) f(9)
Ulam's problem
f(n): if (n==1) return 1 else if (odd(n)) return f(3*n+1) else return f(n/2)
-Nobody has ever found an n for which f does not stop-Nobody has ever found a proof: (so there can be no algorithm deciding this.)
€
∀n : f (n) stops
Halting problem is undecidable
• Given a program P an an input x will P(x) stop? we can prove: the halting problem cannot be solvedie there is no algorithm Halt(P,x) that for any
program P and input x decides whether P(x) stops.
Undecidability
A problem P(x) is undecidable, if there is no algorithm solving P for any legal x
Halting problem is undecidableAssume there is a program P(P1,D) that for any P1 and D, outputs Yes (if P1(D) halts) or No (if P1(D) loops) Then construct Q'(P2): if (P(P2,P2)) loop else halt Now execute Q'(Q') if P(Q', Q') says Q'(Q') halts it loops if P(Q', Q') says Q'(Q') loops it haltsCONTRADICTION -> P does not exist!!
Verification/equivalence undecidable
• Given a specification S and a program P, there is no algorithm that decides whether P executes according to S
• given two programs P1 and P2, there is no algorithm that decides x: P1(x)=P2(x)
• Does this mean we should not build program verifiers? €
∀
Intractability
• Suppose we have a program, – does it execute a in a reasonable time? – Eg, towers of Hanoi.
Three pegs, one with n smaller and smaller disks, move (1 disk at the time) to another peg without ever placing a larger disk on a smaller
f(n) = # moves for tower of size n Monk: before a tower of Hanoi of size 100 is
moved, the world will have vanished
13
Example: Tower of Hanoi, move all disks to third peg without ever placing a larger disk on a smaller one.
14
Example: Tower of Hanoi, move all disks to third peg without ever placing a larger disk on a smaller one.
15
Example: Tower of Hanoi, move all disks to third peg without ever placing a larger disk on a smaller one.
f(n) = f(n-1) + …
16
Recursive AlgorithmsExample: Tower of Hanoi, move all disks to third peg without ever placing a larger disk on a smaller one.
f(n) = f(n-1) + 1 + …
17
Example: Tower of Hanoi, move all disks to third peg without ever placing a larger disk on a smaller one.
f(n) = f(n - 1) + 1 + f(n - 1) f(n) = 2f(n - 1) + 1, f(1) = 1
f(n): #moves
• f(n) = 2f(n-1) + 1, f(1)=1• f(1) = 1, f(2) = 3, f(3) = 7, f(4) = 15
• f(n) = 2n-1 How can you show that?
• Can you write an iterative Hanoi algorithm?
Was the monk right?
• 2100 moves, say 1 per second.....
How many years???
Was the monk right?
• 2100 moves, say 1 per second.....
2100 ~ 1030 ~ 1025 days ~ 3.1022 years
more than the age of the universe
Is there a better algorithm?
Is there a better algorithm?
• Pile(n-1) must be off peg1 and on one other peg before disk n can be moved to its destination
• so (inductively) all moves are necessary
Algorithm complexity
• Measures in units of time and space• Linear Search X in dictionary D i=1; while (not at end and X!= D[i]) {i++}• We don't know if X is in D, and we don't know
where it is, so we can only give worst or average time bounds
• We don't know the time for atomic actions, so we only determine Orders of Magnitude
time complexity
• In the worst case we search all of D, so the loop body is executed n times
• In the average case the loop body is executed about n/2 times. Why?
time complexity
• In the worst case we search all of D, so the loop body is executed n times
• In the average case the loop body is executed about n/2 times. In average case analysis we sum the products of the probability of each outcome and the cost of that outcome, here
€
1/n * i =1/n i = (n(n +1)i=1
n
∑ /2) /n ≈ n /2i=1
n
∑
Units of time
• 1 microsecond ?
• 1 machine instruction?
• #code fragments that take constant time?
Units of time
• 1 microsecond ? no, too specific and machine dependent• 1 machine instruction? no, still too specific• #code fragments that take constant time? yes
unit of space
• bit?
• int?
unit of space
• bit? very detailed but sometimes necessary
• int? nicer, but dangerous: we can code a whole
program or array (or disk) in one arbitrary int, so we have to be careful with space analysis (take value ranges into account when needed)
Orders of magnitude
• If an algorithm with input size n takes less than c.n2 steps to execute, we say it is an order n squared algorithm, notation O(n2)
• In general g(n) = O(f(n)) if there is a constant c such that g(n) <= c.f(n)
for all but a finite number of values for n. In other words, there is a n0 such that
g(n) <= c.f(n) for all n>n0
Worst / average case complexity
• Worst case : maximal number of steps / bits / words an algorithm needs for inputs of size n
• Average case: the expected number of steps / bits / words an algorithm needs:
€
Pii∈I n
∑ .Ci
Pi : probability input i occurs
Ci : complexity given input i
In : all possible inputs of size n
Is there a better algorithm?
• Is there a better algorithm than linear search?• What does that mean?
Is there a better algorithm?
• Is there a better algorithm than linear search?• What does that mean? 2*better? better order of magnitude? better worst case? better average case?
Is there a better algorithm?
• Is there a better algorithm than linear search?• What does that mean? 2*better? NO better order of magnitude? better worst case? better average case? usually better worst case
Binary Search• Because dictionaries are sorted BS(x,lo,hi): if (lo>hi) not found else { m = (lo+hi)/2; if (x==D[m]) found else if (x<D[m]) BS(x,lo,m-1) else BS(x,m+1,hi) }
Binary Search• Because dictionaries are sorted BS(x,lo,hi): if (lo>hi) not found else { m = (lo+hi)/2; if (x==D[m]) found else if (x<D[m]) BS(x,lo,m-1) else BS(x,m+1,hi) } why m-1 and m+1?
Binary search worst case time
• Every 'chop' takes half of the remaining possibilities away
• The number of times we can divide n>0 by 2 until it gets to 0:
log2(n)
so, worst case O(log(n)) why?
Is there an even better algorithm?
• uhhh...
Is there an even better algorithm?
• now we have to consider all possible algorithms• Lower bound of a problem– given a set of "allowed steps" • taking constant time
– lower bound is minimal worst case complexity of any algorithm, using only the allowed steps, solving it
Searching in an ordered array
• allowed steps:– comparison to /assignment of array element– arithmetic on index– some action on found or not found
eg NOT if (isElement(x,D)) ... why NOT?
Event / decision trees
• Given these steps each algorithm is associated with a set of event or decision trees:– comparison is an internal node with two children– (not) found is a leaf– other ops are branches leading to new decisions
or leaves• For some input a path through the tree to a
particular leaf is taken
event tree for linear searchcompare
foundnot found
foundfound action
comparefound
not foundfoundfound action
comparefound
not foundfoundfound action
...
foundnot found action
event tree for binary search
comparefound
not found
foundfound action
compare
found
not found
foundfound actioncomparefound
not found
foundfound action
Height of the event tree
• The event trees for any search are binary trees. Why?• Any tree associated with the search has at least n+1
leaves (n found, 1 not found)• Theorem: A binary tree with n leaves has a height of at
least log(n). Check out some cases.• There is at least one path from root to a leaf of length
at least log(n), so any search algorithm takes at least O(log(n)) time for some of its inputs. (In the next
lecture we will use big Omega for lower bounds.)
Lower bound for search
• Lower bound for search in an ordered array is O(log(n))
• There is no asymptotically better search than binary search
• But what if I know more, eg search in an array with a fixed range of values...
Upper and lower bounds
• Upper bound of a problem: – the lowest worst case complexity of any known
algorithm solving for the problem.– eg search: O(log(n)) binary search
• Lower bound of a problem– Maximum (the minimum number of steps we can
prove any algorithm will need to execute) – eg any search algorithm takes at least O(log(n))
steps
Closed problems, algorithmic gaps
• Closed problem lower bound = upper bound eg search in ordered array
• Algorithmic gap lower bound < upper bound eg NP Complete problems
NP Complete problems
• NPC: a certain class of problems with algorithmic gaps
upper bound: exponential lower bound: polynomial
Trial and error type algorithms are the only ones we have so far to find an exact solution
Some NPC problems
• TSP: Travelling Salesman given cities c1,c2,...,cn and distances between
all of these, find a minimal tour connecting all cities.
• SAT: Satisfiability given a boolean expression E with boolean
variables x1,x2,...,xn determine a truth assignment to all xi making E true
Back tracking
• Back tracking searches (walks) a state space, at each choice point it guesses a choice.
• In a leaf (no further choices) if solution found OK, else go back to last choice point and pick another move.
• A Non Deterministic algorithm is one with the ability to select the sequence of right guesses
(avoiding back track)
P vs NP
• NP is the class of problems solvable in polynomial time by a non deterministic algorithm
• P is the class of problems solvable in polynomial time by a deterministic algorithm
the $1,000,000 question: is P = NP ?
• a $1 question: is Hanoi in NP?
Coping with intractability• NP problems become intractable quickly TSP for 100 cities? How would you
enumerate all possible tours? How many?• Coping with intractability:– Approximation: Find a nearly optimal tour– Randomization: use a probabilistic algorithm using
"coin tosses" (eg prime witnesses)