Computational Methods forManagement and Economics

Carla P. Gomes

Introduction to Complexity TheoryModule 11

Just because we know how to solve a given problem, that does not mean that the problem can be solved.

The procedure (algorithm) may be so inefficient that it would not be possible to solve the problem within a useful period of time.

What is an algorithm?

Algorithms vs. recipes

• Baking a cake – process of producing a cake, from the ingredients, by the baker, aided by the oven and other utensils and equipment, according to the recipe.– Ingredients: input– Cake: output– Recipe: algorithm

• World of electronic computation:– Software: corresponds to the recipes (or algorithms)– Hardware: corresponds to the utensils and oven

(adapted from: Computer Ltd.- What they really can’t do, David Harel)

Baking a cake

Ingredients

RecipeAlgorithm

Oven,Utensils

(Software) (Hardware)

Cake

Summing Salaries

• We are given the list of personnel records;

• Each record contains: employee name, some other details and his/her salary (input);

• We are interested in the total sum of the salaries of all employees (output);

Algorithm for Summing salaries

Name Salary

John Brown

$21,000

Mary White

$34,400

Mike Smith

$35,000

… …

Joe Doe

$26,000

0 Value of noted number

$21,000

$55,400

$90,400

$17,821,000

Start

.

.

.

End

Algorithm for summing salaries

1. Make a note of the number 0;

2. Proceed through the list, adding the current employee’s salary to the noted number

3. At the end of the list, produce the noted number as output

Examples of algorithms presented in class

• Simplex Algorithm

• Transportation Simplex Algorithm

• Dijkstra’s Algorithm

Definition of an Algorithm

An algorithm is a well-ordered collection of unambiguous and effectively computable operations that when executed the algorithm produces a result and halts in a finite amount of time.

The word algorithm is derived from the last name of Muhammad Al-Khowarizmi (full name: Abu Ja'far Muhammad ibn-Musa Al-Khowarizmi), a famous Persian mathematician and author of the eighth and ninth centuries. (His most famous book is Kitab al jabr w'al muqabala.)

What are the five characteristics of an algorithm?   

1. Algorithms are well-ordered - we always know which instruction

should be performed next. There is no uncertainty regarding the execution order of instructions.

2. Algorithms have unambiguous operations - an ambiguous instruction lacks the details necessary for it to be executed.

For example, an ambiguous instruction for a person might be "Bake a chocolate cake." or “Melt sugar until it becomes caramel”. More detailed instructions are usually required for a person to solve the problem of making a chocolate cake and melting sugar.

By using primitives, we can write unambiguous algorithms for computers. A primitive is a basic operation used for writing an algorithm. Some primitives that computers understand are "Add these two numbers" or "Compare these two numbers to see which is smaller."

3. Algorithms have effectively computable operations – instructions should be doable. For example, an instructionsuch as “write the weather for April 5th, 2032” is notreally computable. No one can accurately predict the weather for this date

4. Algorithms produce a result.

5. Algorithms halt in a finite amount of time.

• Errors and correctness - given a valid input, the algorithm should return a correct answer

– An elderly lady rigth after her 107th birthday received a computerized letter from the local school authorities in the Danish country, with registration forms for first grade in elementary school. It turned out that only two digits had been alloted for the “age” field.

• Termination - We want to make sure that algorithms don’t run forever and don’t get into a loop.

The algorithm problem

Specification ofall legal inputs

and

Specification ofdesired output

as a function of the input

Any legalinput

The algorithm

The desiredoutput

Examples of algorithmic problems

Problem 1: Input: A list L, of integers Output: The sum of the integers on L

Problem 3: Input: A road map of citieswith distances attached to the road map, and two designated cities A and B Output: A description of theshortest path between A and B

Problem 2: Input: Two texts A and B in English Output: The list of common words in both texts

Problem 5:Input: A valid linear programming problemOutput: The optimal solution to the LP problem

Problem 4: Input: A road map of cities with distances attached to the road map, and a designated initial city A. Output: A description of the shortest tour that passes through all the cities starting and ending at A.

Examples of algorithmic problems

Instance of an algorithmic problemSize of an instance

• An instance of an algorithmic problem is a concrete case of such a problem with specific input. The size of an instance is given by the size of its input.

• Examples of instances:

– An instance of problem 1:

L= 2, 5, 26, 8, 170, 79, 1002

Problem 1: Input: A list L, of integers Output: The sum of the integers on L

Size of instance length of list

Size of instance = |L| = 7

Examples of instances

Problem 3: Input: A road map of citieswith distances attached to the road map, and two designated cities A and B Output: A description of theshortest path between A and B

1

2

3

4

5

6

2

4

2 1

3

4

2

3

2

Size of instance Number of cities and roads

A particular instance:Size of instance:6 nodes9 edges

The size of an instance is given by the size of its input.

An instance of problem 5:

max 3 x + 5 y

s.t

x ≤ 4

y ≤ 12

3 x + 2 y ≤ 18

x,y ≥ 0

Problem 5:Input: A valid linear programming problemOutput: The optimal solution to the LP problem

Size of instance:2 variables3 functional constraints

Size of instance Number of variablesand constraints (n,m)

The size of an instance is given by the size of its input.

Computer Programming

Programmer(human)

Compiler(software)

Algorithm

programming

Program in high-level language (C, Java, etc)

compilation

Equivalent program inassembly language

Equivalent program inmachine code

computer execution

Complexity of Algorithms

Complexity of Algorithms

• The complexity of an algorithm is the number of steps that it takes to transform the input data into the desired output.

• Each simple operation (+,-,*,/,=,if,etc) and each memory access corresponds to a step.(*)

• The complexity of an algorithm is a function of the size of the input (or size of the instance). We’ll denote the complexity of algorithm A by CA(n), where n is the size of the input.

(*) This model is a simplification but still valid to give us a good idea of the complexity of algorithms.

Example: Insertion Sort

From: Introduction to AlgorithmsCormen et al

best cost

worst cost

I n

I n

worst cost

best cost

I n – all possible instances of size n

Different notions of complexity

Worst case complexity of an algorithm A – the maximum number of computational steps required for the execution of Algorithm A, over all the inputs of the same size, s. It provides an upper bound for an algorithm. The worst that can happen given the most difficult instance – the pessimistic view.

Best case complexity of an algorithm A -the minimum number of computational steps required for the execution of Algorithm A, over all the inputs of the same size, s. The most optimistic view of an algorithm– it tells us the least work a particular algorithm could possibly get away with for some one input of a fixed size – we have the chance to pick the easiest input of a given size. 

Average case complexity of an algorithm A - i.e., the average amount of resources the algorithm consumes assuming some plausible frequency of occurrence of each input.

Figuring out the average cost is much more difficult than

figuring out either the worst-cost or best-cost e.g., we have to assume a given probability distribution for the types of inputs we get.

 

Different notions of complexity

In general thisis the notion that we use tocharacterize the complexity ofalgorithms

We performupper bound

analysis onalgorithms.

Growth RatesIn general we only worry about growth rates because:

1. Our main objective is to analyze the cost performance of algorithms.

2. Another obstacle to having the exact cost of algorithms is that sometimes the algorithm are quite complicated to analyze.

3. When analyzing an algorithm we are not that interested in the exact time the algorithm takes to run – often we only want to compare two algorithms for the same problem – the thing that makes one algorithm more desirable than another is its growth rate relative to the other algorithm’s growth rate.

Comparing algorithms wrt complexity

• Let us consider two algorithms A1 and A2,

with complexities:

CA1(n) = 0.5 n2

CA2(n) = 5 n

Which one is has larger complexity?

CA2(n) = 5 n ≥ CA1(n) = 0.5 n2 for n ≤ 10

CA1(n) = 0.5 n2 >CA2(n) = 5 n for n >10

When we look at the complexity of algorithms we think asymptotically –i.e., we compare two algorithms as the problem sizes tend to infinity!

Big Oh Notation

• Given two functions F and G, whose domain is the natural numbers, we say that the order of F is lower than or equal to the order of G if:

– F(n) ≤ K G(n) for all n > n0 (K and n0 are constants)

• We say F is O(G) (F is oh of G)Example:

(3 n3 + n2 + n ) is O(n3)

In practice we just look at the fastest growing term of the expression

CA1(n) = 0.5 n2 >CA2(n) = 5 n for n >10

The complexity of A2 is of lower order than that of A1. While A1 growsquadratically O(n2) A1 only grows linearly O(n).

x2 vs. (x2 + x) (x <=20)

x2 vs. (x2 + x) (x2 + x) is O(n2) (oh of n-squared)

Worst-case complexityDijkstra’s Algorithm

begin

d(s) : = 0 and pred(s) : = 0;

d(j) : = for each j N - {s};

LIST : = {s};

while LIST do

beginlet d(i) : = min {d(j) : j LIST};remove node i from LIST;update(i)if d(j) decreases, place j in LIST

endend

Complexity of Dijkstra’s Algorithm

• Node selections – the algorithm performs this operation n times and each such operation requires that it scans each temporarily labeled node. Therefore, the total node selection time is n + (n-1) + (n-2) + …+ 1 = O (n2)

• Distance updates.- the algorithm performs this operation | A(i)| times per iteration (i.e., the number of edges from i). Overall the algorithm performs this operation as many times as the number of edges in the graph, i.e., m

NiiA

|)(|

Since each distance update requires constant time i.e., O(1),The algorithm require O(m) total time for updating distances

Worst-case Complexity of Dijkstra’s Algorithm

• Node selections –O (n2)

• Distance updates.-

O(m)

Complexity of Dijkstra’s Algorithm is O (n2)

Total time is O (n2) + O(m) O (n2)

Note: this time bound can be improved with better data-structures…

How do computer scientists differentiate between

good (efficient) and bad (not efficient) algorithms?

How do computer scientists differentiate between good (efficient) and bad (not efficient) algorithms?

The yardstick is that any algorithm

that runs in no more than polynomial time is an efficient algorithm; everything

else is not.

Running times as n growsgrowth rates

1.0E+00

1.0E+03

1.0E+06

1.0E+09

1.0E+12

1.0E+15

1.0E+18

1 6 11 16 21 26 31 36 41 46

n

run

nin

g t

ime log n

n log n

n^2

2^n

Ordered functions by their growth rates

cOrder

constant 1

logarithmic 2

polylogarithmic 3

nr ,0<r<1

nsublinear 4

linear 5

nr ,1<r<2 subquadratic 6

quadratic 7

cubic 8

nc,c≥1

rn, r>1

polynomial 9

exponential 10

lg n

lgc n

n3

n2

Efficient algorithms

Not efficient algorithms

exponential

polynomial

N2

Binary B&B alg.Polynomial vs. exponential growth (Harel 2000)

LP’s interior pointMin. Cost Flow AlgsTransportation AlgAssignment AlgDijkstra’s alg.

Problem Complexity

Theory of NP-completeness or NP-hardnessEasy vs. hard problems

Overview of complexity• How can we show a problem is efficiently

solvable?– We can show it constructively. We provide an

algorithm and show that it solves the problem efficiently. E.g.:

• Shortest path problem - Dijkstra’s algorithm runs in polynomial time, O(n2). Therefore the shortest path problem can be solved efficiently.

• Linear Programming – The Interior Point method (not covered in class) has polynomial worst-case complexity. Therefore Linear programming can be solved efficiently.

(*) The simplex method has exponential worst case complexity/ However, in practice the simplex algorithm seems to scale as m3, where m is the number of functional constraints.

Overview of complexity

• How can we show a problem is not efficiently solvable?

– How do you prove a negative? Much harder!!!

– This is the aim of complexity theory.

Easy (efficiently solvable) problems vsHard Problems

• Easy Problems - we consider a problem X to be “easy” or efficiently solvable, if there is a polynomial time algorithm A for solving X. We denote by P the class of problems solvable in polynomial time.

• Hard problems --- everything else. Any problem for which there is no polynomial time algorithm is an intractable problem.

.

Class P Class of Problems solvable in Polynomial Time

•Finding a word in a dictionary with n entries. Time log n. Polynomial time

•Sorting n itemstime n log n

Polynomial time

•Finding the shortest path from s to ttime n2.Polynomial time

•Finding the solution to a Linear Programming problem

time n3. Polynomial time

Examples:

Other examples of problems in the class P include:

The assignment problem and transportation problem, finding the minimum cost flow problem and the max cost flow in in a directed graph.

Two problems

Problem 2: Input: A road map of citieswith distances attached to the road map, and two designated cities A and B Output: A description of thelongest path between A and B

Problem 1: Input: A road map of citieswith distances attached to the road map, and two designated cities A and B Output: A description of theshortest path between A and B

We know that it is easy – it can be solved efficiently!

Can we come up with an efficient algorithm for the longest path?

I’d like you to develop an effcient algorithm to find the longest path between two points in a graph.

Your Longest Path Algorithm between two nodes, u and v

1

2

3

4

5

6

2

4

2 1

3

4

2

3

2

Initialization: MaxPath none ;MaxPathLength 0;

For each path P starting at 1:if P is a simple path from u to v and length(P) > MaxPath

MaxPath P;MaxPathLength length(P)

Return: MaxPath; MaxPathLength;

G=(N,E)

u v

Is that the best you can do? -- that seems to be a bad algorithm!!!

I know…If each intersection leads to two roads and we have100 intersections that canlead to 2100 paths …

I cant’ find an efficient algorithm. I guess I’m too dumb.

I can’t find an efficient algorithm, but neither can these famous researchers.

George Dantzig

Invented Linear Programming Formulations:

max 3 x + 5 y

s.t

x ≤ 4

y ≤ 12

3 x + 2 y ≤ 18

x,y ≥ 0

Invented Simplex Algorithm

Coefficients of Variables

Bas Var equation Z x1 x2 x3 x4 x5 RHSZ 0 1 -3 -5 0 0 0 0x3 1 0 1 0 1 0 0 4x4 2 0 0 2 0 1 0 12x5 3 0 3 2 0 0 1 18

basic variables x3=4 x4=12 x5=18non-basic variables x1=0 x2=0

Alan Turing

In 1936, Alan Turing, a British mathematician, showed that there exists a relatively simple universal computing device that can perform any computational process. Computers use such a universal model.

Turing Machine(abstraction)

Turing also showed the limits of computation – some problems cannot be computed even with the most powerful computer and even with unlimited amount of time – e.g., Halting problem.

John Von Neumann

Brilliant mathematician, synthesizer, and promoterof the stored program concept, whose logical design of the Institute of Advanced Studies (Princeton)Computer became the prototype of today’s computer (*) - the von Neumann Architecture.

(*) sequential i.e., non-parallel computers

Richard Karp

Steven Cook

Invented the theory of NP-Completeness –proved that a simple problem - Satisfiabilityis NP-Complete.

Showed that several important problemsand applications are NP-Complete and NP-hard,including Integer Programming.

Given a propositional formula, is there an assignment to its

variables (a, b, and c – True or False) making the formula true?

)()()( cacacba

David Johnson

Michael Garey

Theory of NP-completeness and NP-hardness

Easy vs. hard problems

Can integer programming be solved in polynomial time?

• FACT: every algorithm that has ever been developed for integer programming takes exponential time.

• Hundreds of very smart researchers have tried to come up with polynomial time algorithms for integer programming, and failed.

• It is generally believed that there is no polynomial time algorithm for integer programming.

• Complexity theory: deals with proving that integer programming is hard.

Decision Problems NP-Completeness theory deals with decision problems. What is a decision problem? A problem for which there is a yes/no answer.

Examples:

Given a propositional formula, is there an assignment to its

variables (a, b, and c – True or False) making the formula true?

)()()( cacacba

Is there a path between two nodes in a graph shorter than k?(decision version of shortest path problem)

Is there a path between two nodes in a graph longer than k?(decision version of longest path problem)

Most optimization problems can be formulated as a decision problem

Class NP Class of Problems solvable in

Nondeterministic-Polynomial Time

• We say that a decision problem is solvable in Non-deterministic polynomial time if:

– The solution can be verified in polynomial time. (E.g., verifying that a path has length greater than K)

– If we imagine that we have an exponential number of processors, we can check all possible solutions simultaneously and therefore answer in polynomial time

NPP

Class NP-Complete

• The first problem to be shown to be NP-Complete was Satisfiability – Cook showed that all the problems in NP could be translated (in polynomial time) as Satisfiability problems;

• The word complete means that every problem in the class NP-complete can be transformed (in polynomial time) into another problem of the class NP-complete. For example all the problems in the class NP-complete can be written as Satisfiability problems.

• The class of NP-Complete problems is the class of the hardest computational problems in the class NP: every NP problem can be transformed into an NP-complete problem (the reverse is not true!!!)

NP

P NP-Complete

Is P not equal to NP?$1,000,000 question

• P not equal to NP? – Is that true that not all problems is NP can be solved in

polynomial time?

• Class of NP-Complete Problems– They all admit exponential time solutions;

– Nobody has ever been able to find a polynomial time solution for any single problem in the class;

– Nobody has ever been able to prove an exponential lower bound for any single problem in the class;

P not equal to NP?$1,000,000 question

Pictorial interpretation of this question

NP

P NP-Complete

Is this the right picture? There are problems in NP that are inherently intractableand cannot be solved in polynomial time.

P not equal to NP?$1,000,000 question

Pictorial interpretation of this question

Or is this the right picture? All the problems in NP can be solved in polynomial time. Even though at this point we don’t know of polynomial time algorithms to solve some problems in NP, they exist…

P = NP

P vs. NPOne Million Dollar Prize

http://www.claymath.org/Millennium_Prize_Problems/P_vs_NP/

Class of NP-Complete ProblemsOne Million Dollar Prize

• Completeness – if someone were to find a polynomial time

solution for a single problem in the class NP-complete all the problems could be solved in polynomial time!!!

– if someone were to prove an exponential lower bound for a single problem in the class NP-complete all the problems in the class would be intractable !!!

Conjecture

NP-Complete problems are inherently hard!

They are intractable !

NP

P NP-Complete

NP-Hard Problems

• The theory of NP-Completeness deals with decision problems.

• What about optimization problems?• An optimization problem whose

corresponding decision problem is NP-complete is said to be NP-Hard – it means that it is at least as hard as the problems in the class NP-complete.

Some examples of NP-hard problems

• Longest path• Traveling Salesman Problem• Capital Budgeting Problem (knapsack problem)• Independent Set Problem• Fire Station Problem (set covering)• 0-1 Integer programming• Integer Programming• Project management with resource constraints• and thousands more

Okay – Should we give up?

Here is why:

1. The theory of NP-completeness is only a worst case result. Not all problems instances are as hard as the worst case.

2. Real problems tend to have sub-problems that are tractable and by exploiting the structure of such sub-problems using efficient algorithms such LP, Min Cost Flow, Transportation. Assignment and shortest path methods we can solve much larger problem instances.

3. In the 1970’s we could only solve Binary Integer Programming instances with 100 variables. By exploiting the structure we can now solve real world instances with over 120,000 variables and 4000 functional constraints.

NO WAY!!!

EXPONENTIAL FUNCTION

POLYNOMIAL FUNCTIONHard Computational

ProblemsScale Exponentially

EXPONENTIAL-TIMEALGORITHMS

EXPLOSIVECOMBINATORICS

Take home message:A picture is worth 1000 words

ExperimentDesignGoal

Start

Software & HardwareVerification

Satisfiability

(A or B) (D or E or not A)

Data Analysis& Data Mining

Fiber optics routing

Capital BudgetingAnd Financial Appl. Information

Retrieval

Protein Folding

And Medical ApplicationsCombinatorial

Auctions

Planning and SchedulingAnd Supply Chain Management

Many more applications!!!

Require powerful computational and

mathematical tools!

NP-Complete andNP-Hard Problems

The End!

Thanks for your attention!It was a lot fun teaching this course!

I hope you enjoyed it too!

Beyond NP-Completeness

I’d like you to develop an algorithm to check if all the programs that we will ever use will always terminate.

The halting problem

The halting problem

XA

Potential Input

Program (algorithm)

We would like a program or algorithm to automatically answer

the question: Does A halt on X?

Yes No

I cant’ find such an algorithm. I guess I’m too dumb.

I can’t find such an algorithm, because Turing proved that such a problem is undecidable.

Beyond NP-CompletenessUndecidability

The halting problem

XA

Potential Input

Program (algorithm)

We would like a program or algorithm to automatically answer

the question: Does A halt on X?

Yes No

The halting cannot be solved by algorithmic means –It is undecidable!