Chapter 11 Limitations of Algorithm Power. Lower Bounds Lower bound: an estimate on a minimum amount...
-
Upload
sharlene-beasley -
Category
Documents
-
view
225 -
download
1
Transcript of Chapter 11 Limitations of Algorithm Power. Lower Bounds Lower bound: an estimate on a minimum amount...
Chapter 11Chapter 11
Limitations of Limitations of Algorithm PowerAlgorithm Power
Lower BoundsLower Bounds
Lower boundLower bound: an estimate on a minimum amount of work : an estimate on a minimum amount of work needed to solve a given problemneeded to solve a given problem
Examples:Examples: number of comparisons needed to find the largest element number of comparisons needed to find the largest element
in a set of in a set of nn numbers numbers number of comparisons needed to sort an array of size number of comparisons needed to sort an array of size nn number of comparisons necessary for searching in a sorted number of comparisons necessary for searching in a sorted
arrayarray number of multiplications needed to multiply two number of multiplications needed to multiply two nn-by--by-n n
matrices matrices
Lower Bounds (cont.)Lower Bounds (cont.)
Lower bound can beLower bound can be• an exact count an exact count • an efficiency class (an efficiency class ())
TightTight lower bound: there exists an algorithm with the same lower bound: there exists an algorithm with the same efficiency as the lower boundefficiency as the lower bound
ProblemProblem Lower bound Lower bound TightnessTightness sortingsorting ((nnloglog n n) yes) yes searching in a sorted arraysearching in a sorted array (log(log n n) yes) yes element uniqueness element uniqueness ((nnloglog n n) yes) yes nn-digit integer multiplication -digit integer multiplication ((nn) unknown) unknown multiplication of multiplication of nn-by--by-n n matrices matrices ((nn22) unknown) unknown
Methods for Establishing Lower BoundsMethods for Establishing Lower Bounds
trivial lower boundstrivial lower bounds
information-theoretic arguments (decision trees)information-theoretic arguments (decision trees)
adversary argumentsadversary arguments
problem reductionproblem reduction
Trivial Lower BoundsTrivial Lower Bounds
Trivial lower boundsTrivial lower bounds: based on counting the number of items that : based on counting the number of items that must be processed in input and generated as outputmust be processed in input and generated as output
ExamplesExamples finding max elementfinding max element
polynomial evaluationpolynomial evaluation
sortingsorting
element uniquenesselement uniqueness
Hamiltonian circuit existenceHamiltonian circuit existence
Conclusions Conclusions may and may not be usefulmay and may not be useful
be careful in deciding how many elements be careful in deciding how many elements mustmust be processed be processed
Decision TreesDecision Trees
Decision treeDecision tree — a convenient model of algorithms involving — a convenient model of algorithms involving comparisons in which:comparisons in which: internal nodes represent comparisonsinternal nodes represent comparisons leaves represent outcomes leaves represent outcomes
Decision tree for 3-element insertion sortDecision tree for 3-element insertion sort
a < b
b < c a < cyes
yes no
noyesno
a < c b < c
a < b < c
c < a < b
b < a < c
b < c < a
no yes
abc
abc bac
bcaacb
yes
a < c < b c < b < a
no
Decision Trees and Sorting AlgorithmsDecision Trees and Sorting Algorithms
Any comparison-based sorting algorithm can be represented Any comparison-based sorting algorithm can be represented by a decision treeby a decision tree
Number of leaves (outcomes) Number of leaves (outcomes) nn!!
Height of binary tree with Height of binary tree with nn! leaves ! leaves loglog22nn!!
Minimum number of comparisons in the worst case Minimum number of comparisons in the worst case loglog22nn!! for any comparison-based sorting algorithmfor any comparison-based sorting algorithm
loglog22nn!! n n loglog22nn
This lower bound is tight (mergesort)This lower bound is tight (mergesort)
Adversary ArgumentsAdversary Arguments
Adversary argumentAdversary argument: a method of proving a lower bound by : a method of proving a lower bound by playing role of adversary that makes algorithm work the hardestplaying role of adversary that makes algorithm work the hardestby adjusting input by adjusting input
Example 1: “Guessing” a number between 1 and Example 1: “Guessing” a number between 1 and n n with yes/nowith yes/no questionsquestionsAdversary: Puts the number in a larger of the two subsetsAdversary: Puts the number in a larger of the two subsets
generated by last question generated by last question
Example 2: Merging two sorted lists of size Example 2: Merging two sorted lists of size nn
aa11 < < aa22 < … < < … < aann and and bb11 < < bb22 < … < < … < bbnn
Adversary: Adversary: aai i < b< bjj iff iff ii < < jj
Output Output bb11 < < aa11 < < bb22 < < aa22 < … < < … < bbnn < < aann requires 2 requires 2nn-1 comparisons-1 comparisons
of adjacent elementsof adjacent elements
Lower Bounds by Problem ReductionLower Bounds by Problem Reduction
Idea: If problem Idea: If problem P P is at least as hard as problem is at least as hard as problem QQ, then a lower, then a lower
bound for bound for Q Q is also a lower bound for is also a lower bound for P. P.
Hence, find problem Hence, find problem Q Q with a known lower boundwith a known lower bound that canthat can be reduced to problem be reduced to problem PP in question. in question.
Example: Example: PP is finding MST for is finding MST for nn points in Cartesian plane points in Cartesian plane Q Q is element uniqueness problem (known to be in is element uniqueness problem (known to be in ((nnloglognn))))
11.3 P, NP, and NP-complete Problems11.3 P, NP, and NP-complete Problems
An algorithm solves a problem in polynomial time if its An algorithm solves a problem in polynomial time if its worst-case time efficiency belongs to O(p(n)) worst-case time efficiency belongs to O(p(n)) • Where p(n) is a polynomial of the problem’s input size n Where p(n) is a polynomial of the problem’s input size n
Problems that can be solved in polynomial time are called Problems that can be solved in polynomial time are called tractabletractable
Problems that cannot be solved in polynomial time are Problems that cannot be solved in polynomial time are called called intractableintractable..• Cannot solve intractable problems in a reasonable length of timeCannot solve intractable problems in a reasonable length of time
25 city Traveling Salesperson Problem25 city Traveling Salesperson Problem
There are 25! different possible paths to be considered. There are 25! different possible paths to be considered. • That is approximately 1.5 x 10That is approximately 1.5 x 102525 different paths. different paths.
Suppose the computer can analyze 10,000,000, or 10Suppose the computer can analyze 10,000,000, or 1077, paths , paths per second. per second. • The number of seconds required to check all possible paths is about The number of seconds required to check all possible paths is about
1.5 x 101.5 x 102525/10/1077, or about 1.5 x 10, or about 1.5 x 101818 seconds. seconds.
• That’s roughly 10That’s roughly 101212 years: about a trillion years. years: about a trillion years.
This would not be a feasible algorithm.This would not be a feasible algorithm.
The Halting ProblemThe Halting Problem
Turing – 1936Turing – 1936 Given a computer program and an input to it, determine whether the program will halt on Given a computer program and an input to it, determine whether the program will halt on
that input or continue working indefinitely on it.that input or continue working indefinitely on it.
Assume that A is an algorithm that solves the halting problem: A(P, I) = 1 if P halts on Assume that A is an algorithm that solves the halting problem: A(P, I) = 1 if P halts on input I, 0 otherwiseinput I, 0 otherwise
Consider P as an input to itself and use the output of A for pair (P,P) to construct a Consider P as an input to itself and use the output of A for pair (P,P) to construct a program Q as follows:program Q as follows:
Q(P) halts if A(P,P) = 0 (if program P does not halt on input P)Q(P) halts if A(P,P) = 0 (if program P does not halt on input P) Q(P) does not halt if A(P, P) = 1Q(P) does not halt if A(P, P) = 1
Then substituting Q for P we obtainThen substituting Q for P we obtain Q(Q) halts if A(Q, Q) = 0 i.e. if program Q does not halt on input QQ(Q) halts if A(Q, Q) = 0 i.e. if program Q does not halt on input Q Q(Q) does not halt if A(Q, Q) = 1 i.e. if program Q halts on input QQ(Q) does not halt if A(Q, Q) = 1 i.e. if program Q halts on input Q
This is a contradiction because neither of the two outcomes for program Q is possible. This is a contradiction because neither of the two outcomes for program Q is possible. QEDQED
Classifying Problem ComplexityClassifying Problem Complexity
Is the problemIs the problem tractabletractable,, i.e., is there a polynomial-time (O(i.e., is there a polynomial-time (O(pp((nn)) )) algorithm that solves it?algorithm that solves it?
Possible answers:Possible answers: yes (give examples)yes (give examples)
nono
• because it’s been proved that no algorithm exists at all because it’s been proved that no algorithm exists at all (e.g., Turing’s (e.g., Turing’s halting problemhalting problem))
• because it’s been be proved that any algorithm takes because it’s been be proved that any algorithm takes exponential timeexponential time
unknownunknown
Problem Types: Optimization and DecisionProblem Types: Optimization and Decision
Optimization problemOptimization problem: find a solution that maximizes or : find a solution that maximizes or minimizes some objective functionminimizes some objective function
Decision problemDecision problem:: answer yes/no to a question answer yes/no to a question
Many problems have decision and optimization versions.Many problems have decision and optimization versions.
E.g.: traveling salesman problemE.g.: traveling salesman problem optimizationoptimization: find Hamiltonian cycle of minimum length: find Hamiltonian cycle of minimum length decisiondecision: find Hamiltonian cycle of length : find Hamiltonian cycle of length mm
Decision problems are more convenient for formal investigation of Decision problems are more convenient for formal investigation of their complexity.their complexity.
Class Class PP
PP: the class of decision problems that are solvable in O(: the class of decision problems that are solvable in O(pp((nn)) time, )) time, where where pp((nn) is a polynomial of problem’s input size ) is a polynomial of problem’s input size nn
Examples:Examples: searchingsearching
element uniquenesselement uniqueness
graph connectivity graph connectivity
graph acyclicitygraph acyclicity
primality testing (finally proved in 2002)primality testing (finally proved in 2002)
Class Class NPNP
NPNP ( (nondeterministic polynomialnondeterministic polynomial): class of decision problems whose ): class of decision problems whose proposed solutions can be verified in polynomial time = solvable proposed solutions can be verified in polynomial time = solvable by a by a nondeterministicnondeterministic polynomial algorithmpolynomial algorithm
A A nondeterministic polynomial algorithmnondeterministic polynomial algorithm is an abstract two-stage is an abstract two-stage procedure that:procedure that:
generates a random string purported to solve the problemgenerates a random string purported to solve the problem checks whether this solution is correct in polynomial timechecks whether this solution is correct in polynomial timeBy definition, it solves the problem if it’s capable of generating and By definition, it solves the problem if it’s capable of generating and
verifying a solution on one of its tries verifying a solution on one of its tries
Why this definition?Why this definition? led to development of the rich theory called “computational led to development of the rich theory called “computational
complexity”complexity”
Example: CNF satisfiabilityExample: CNF satisfiability
Problem: Is a boolean expression in its conjunctive normal Problem: Is a boolean expression in its conjunctive normal form (CNF) satisfiable, i.e., are there values of its form (CNF) satisfiable, i.e., are there values of its variables that makes it true?variables that makes it true?
This problem is in This problem is in NPNP. Nondeterministic algorithm:. Nondeterministic algorithm: Guess truth assignmentGuess truth assignment Substitute the values into the CNF formula to see if it Substitute the values into the CNF formula to see if it
evaluates to trueevaluates to true
Example: Example: ((A | A | ¬B ¬B | | ¬C¬C)) & & ((AA | | BB)) & & ((¬¬B | B | ¬D |¬D | EE)) && ((¬D | ¬E¬D | ¬E))Truth assignments:Truth assignments:
A B C D EA B C D E0 0 0 0 00 0 0 0 0
. . .. . .1 1 1 1 11 1 1 1 1
Checking phase: Checking phase: O(O(nn))
What problems are in What problems are in NPNP??
Hamiltonian circuit existence Hamiltonian circuit existence Partition problem: Is it possible to partition a set of Partition problem: Is it possible to partition a set of nn
integers into two disjoint subsets with the same sum?integers into two disjoint subsets with the same sum? Decision versions of TSP, knapsack problem, graph Decision versions of TSP, knapsack problem, graph
coloring, and many other combinatorial optimization coloring, and many other combinatorial optimization problems. (Few exceptions include: MST, shortest paths)problems. (Few exceptions include: MST, shortest paths)
All the problems in All the problems in PP can also be solved in this manner (but can also be solved in this manner (but no guessing is necessary), so we have: no guessing is necessary), so we have:
PP NPNP
Big question: Big question: PP = = NP NP ??
NPNP-Complete Problems-Complete Problems
A decision problem A decision problem DD is is NPNP-complete-complete if it’s as hard as any if it’s as hard as any
problem in problem in NPNP, i.e.,, i.e., D D is in is in NPNP every problem in every problem in NPNP is polynomial-time reducible to is polynomial-time reducible to DD
Cook’s theorem (1971): CNF-sat is Cook’s theorem (1971): CNF-sat is NPNP-complete-complete
NP-completeproblem
NP problems
NPNP-Complete Problems (cont.)-Complete Problems (cont.)
Other Other NPNP-complete problems obtained through polynomial--complete problems obtained through polynomial-
time reductions from a known time reductions from a known NPNP-complete problem-complete problem
Examples: TSP, knapsack, partition, graph-coloring andExamples: TSP, knapsack, partition, graph-coloring and hundreds of other problems of combinatorial nature hundreds of other problems of combinatorial nature
knownNP-complete
problem
NP problems
candidate for NP -
completeness
P P = = NP NP ?? Dilemma RevisitedDilemma Revisited
P P = = NP NP would imply that every problem in would imply that every problem in NP, NP, including all including all NP-NP-complete problems,complete problems, could be solved in polynomial timecould be solved in polynomial time
If a polynomial-time algorithm for just one If a polynomial-time algorithm for just one NP-NP-complete complete problem is discovered, then every problem in problem is discovered, then every problem in NP NP can be solved can be solved in polynomial time, i.e., in polynomial time, i.e., P P = = NPNP
Most but not all researchers believe that Most but not all researchers believe that P P NP NP , i.e. , i.e. P P is a is a proper subset of proper subset of NPNP
NP-completeproblem
NP problems