Linear Programming The Simplex Algorithm: Part II Chapter...

Linear ProgrammingThe Simplex Algorithm: Part II

Chapter 5

University of ChicagoBooth School of Business

Kipp Martin

October 17, 2017

1

Outline

List of Files

Key Concepts

Revised Simplex

Revised Simplex with LU Decomposition

Other IssuesFinding a Starting BasisUnbounded ProblemsConvergenceThe Future

2

List of Files

I revisedsimplex.m

I revisedsimplexlu.m

I lptest1.m

I lptest2.m

I lptest3.m

I artificial.m

I unbounded.m

3

Key Concepts

I nonzero elements are the enemy

I density is our enemy

I zeros are our friend

I sparsity is our friend

I there is a big difference between toy textbook problems andproblems that need to get solved in practice

I deja vu – nonzero elements are the enemy – we will encountera number of other enemies throughout the quarter

4

Key Concepts

I revised Simplex

I revised Simplex with LU decomposition

I finding an initial basis

I unbounded linear programs

I degeneracy

5

Revised Simplex

In basicsimplex.m we partitioned the A matrix into basic andnonbasic columns

ABxB + ANxN = b

And then multiplied by the basis inverse

A−1B ABxB + A−1B ANxN = A−1B b

xB + A−1B ANxN = A−1B b

Here is the problem – even if AB and AN are sparse, A−1B AN isoften dense. Multiplying AN by A−1B often results in a very densematrix.

6

Revised Simplex

The non-basic columns of a covering problem before pivoting.

7

Revised SimplexThe non-basic columns after pivoting – note the nonzero fill in.

8

Revised Simplex

Bottom Line: No one ever, ever works with matrix A−1B AN inpractice. It is not practical. So we really have two problems ordragons to slay.

I we need the updated columns of AN (that is A−1B AN) in orderto do the minimum ratio test,

I we need A−1B to find the updated columns, the updatedright-hand side, and reduced costs.

9

Revised Simplex

In terms of basicsimplex.m we do the following:

T = inv(AB)*A;

bBAR = inv(AB)*b;

which is necessary for the minimum ratio test

if bBAR(k) / T( k, N(negRedCostIdx)) < minRatioVal

We also calculate

w = c’ - (cB’*inv(AB))*A;

which is necessary to determine the pivot column

if wN( k) < -.00001

so in general, T is VERY dense.

10

Revised Simplex

So let’s slay these two dragons one at a time. First, we eliminatethe need to explicitly keep T around. We store two things:

I The original A matrix

I The basis inverse (for now)

11

Revised SimplexStoring A: most real problems are actually quite sparse – that isthe ratio of nonzero elements in A to the number of elements in Ais quite small.

The vast majority of elements are zero. Consider for example atransportation problem.

A = [1 1 1 1 1 0 0 0 0 0;

0 0 0 0 0 1 1 1 1 1;

1 0 0 0 0 1 0 0 0 0;

0 1 0 0 0 0 1 0 0 0 ;

0 0 1 0 0 0 0 1 0 0;

0 0 0 1 0 0 0 0 1 0;

0 0 0 0 1 0 0 0 0 1]

Zeros are very good friends. We never need to use them inarithmetic operations such as plus or times so we don’t bother tokeep them around.

12

Revised Simplex

Sparse Matrix Storage: it is necessary to understand how amatrix is stored in practice.

Remember that real problems are sparse! Never store or operateon the zero elements.

Store the A matrix (constraint matrix) with three arrays (vectors):

I values – an array that holds the nonzero elements in the Amatrix, the size is equal to the number of nonzero elements inthe matrix

I indexes – an array that holds the row (column) indexes of thenonzero elements in the A matrix, the size is equal to thenumber of nonzero elements in the matrix

I starts – a pointer array that points the start of each column(row), the size is equal to the number of columns (rows) plusone in the matrix;

13

Revised Simplex

Column Major Storage:

Consider the following matrix.

A =

0 1.5 0 7.7 0

2.0 6.7 1.1 7.7 07.5 0 1.1 10 1

0 0 1.1 12 03.1 0 0 1.9 0

Then

values = 2.0, 7.5, 3.1|1.5, 6.7|1.1, 1.1, 1.1|7.7, 7.7, 10, 12, 1.9|1indexes = 1, 2, 4|0, 1|1, 2, 3|0, 1, 2, 3, 4|2starts = 0, 3, 5, 8, 13, 14

Note: we use 0-based counting for the array indexes

14

Revised Simplex

Key Idea: In revised simplex, we only store A and only update thecolumn we pivot on – none of the other columns! Seerevisedsimplex.m

aBAR = inv( AB)*AN(:,negRedCostIdx);

minRatioVal = inf;

for k = 1:numRows

if aBAR( k) > 0

if bBAR(k) / aBAR(k) < minRatioVal

minRatioVal = bBAR(k) /aBAR( k);

%record the index

minRatioIdx = k;

end

end

end

15

The Revised Simplex Algorithm

Here is a formal mathematical statement of the Revised SimplexAlgorithm.

Step 1: (Initialization) Initialize with a basis B corresponding toa basic feasible solution and for the corresponding basis matrixcalculate initial values of the right-hand-side vector

xBi= A−1B b

Step 2: (Pivot Column Selection) Calculate the reduced costs

wN = c>N − c>B A−1B AN

If wN ≥ 0, stop, the current solution is optimal; else, select q ∈ Nsuch that wq < 0.

16


How do we do the multiplication c>B A−1B AN in practice?

Matrix multiplication is associative.

(c>B A−1B )AN = c>B (A−1B AN)

Which do we prefer?

Why?

17

The Revised Simplex AlgorithmStep 3: (Minimum Ratio Test and Pivot Row Selection)Perform the minimum ratio test on column q that was selected inStep 2. First find the updated column q.

aq = A−1B aq

i∗ ← argmin {xBi/aiq | aiq > 0, i = 1, . . .m}

Step 4: (Update the right-hand-side)

xBi← xBi

− (xBi∗/ai∗q)aq

xBi∗ ← xBi∗/ai∗q

18


Step 5: (Update the Basic and Non-Basic Index Sets)

B ← (B\{Bi∗}) ∪ {q} N ← (N\{q}) ∪ {Bi∗}

Go to Step 2.

19


In Step 2 we calculate the reduced costs by

wN = c>N − (c>B A−1B )AN

not

wN = c>N − c>B (A−1B AN)

20

Revised Simplex

The MATLAB program, revisedsimplex.m, implements therevised Simplex algorithm as just described. (We assume a minproblem in standard form.)

Inputs:

I A – the constraint matrix

I b – the right-hand-side vector

I c – the objective function vector

I B – an index set of the basic variables (feasible)

Outputs:

I X – the optimal solution to the linear program

I objVal – the optimal objective function value

I B – the optimal basis

21

Revised SimplexA problem in standard form:

A = [.7 1 1 0 0 0

.5 5/6 0 1 0 0

1 2/3 0 0 1 0

1/10 1/4 0 0 0 1]

c = [-10; -9; 0; 0; 0; 0 ]

b = [630; 600; 708; 135]

B = [3 4 5 6]

Make the call

[X, objVal, B] = revisedsimplex( A, b, c, B)

22


Okay, so with revised simplex we avoid working with the updatedtableau,

xB + A−1B ANxN = A−1B b

Time to slay the second dragon – we don’t want to explicitly haveto calculate A−1B because it is too dense.

There are two places in the iterative process of revised simplex thatwe need A−1B .

I In Step 2 we need to calculate c>B A−1B

I In Step 3 we need to calculate aq = A−1B aq

23


We avoid calculating A−1B by using LU decomposition.

Reduced Cost Calculation: We need to calculate

u> = c>B A−1B

u>AB = c>B

(u>AB)> = (c>B )>

A>Bu = cB

In other words solve the system

A>Bu = cB

24


Solve the system A>Bu = cB with LU decomposition.

We know there exists a lower triangular matrix L, with diagonalelements of 1, and upper triangular matrix U, and permutationmatrix P such that

PA>B = LU

We solve

LUu = PcB

Let y = Uu. Do a forward solve on Ly = PcB to obtain y and thendo a backward solve on Uu = y .

25


Assume that the original A matrix is

A = [.7 1 1 0 0 0

.5 5/6 0 1 0 0

1 2/3 0 0 1 0

1/10 1/4 0 0 0 1]

A basis corresponding to a basic feasible solution is

B = [3, 4, 5, 2]

c> =[−10 −9 0 0 0 0

]Set up a system of equations to find the reduced costs.

26


>> [L, U, P] = mylu( AB’)

L =

1.0000 0 0 0

0 1.0000 0 0

0 0 1.0000 0

1.0000 0.8333 0.6667 1.0000

U =

1.0000 0 0 0

0 1.0000 0 0

0 0 1.0000 0

0 0 0 0.2500

P = I 27


We are solving

LUu = PcB

We set y = Uu and a forward solve on Ly = PcB

y1 = 0

y2 = 0

y3 = 0

y1 + (5/6)y2 + (2/3)y3 + y4 = −9

28


Do a backward solve on Uu = y

u1 = y1

u2 = y2

u3 = y3

(1/4)u1 = y4

29


This is implemented in revisedsimplexlu.m. In this code we do

% reduced cost calculations

u = lusolve(AB’, cB);

wN = cN’ - u’*AN;

Pretty easy!

30


We also need, aq = A−1B aq. That is, we need to solve

PABaq = Paq

LUaq = Paq

Let y = Uaq.

Do a forward solve on Ly = Paq to obtain y and then do abackward solve on Uaq = y .

In the revisedsimplexlu.m code this is implemented as

aBAR = lusolve(AB, AN(:,negRedCostIdx) );

31

Revised Simplex with LU DecompositionCalculate the updated column for variable 1 for use in theminimum ratio test.

>> [L, U, P] = mylu( AB)

L =

1 0 0 0

0 1 0 0

0 0 1 0

0 0 0 1

U =

1.0000 0 0 1.0000

0 1.0000 0 0.8333

0 0 1.0000 0.6667

0 0 0 0.2500

P = I32


We solve LUaq = Paq.

Do a forward solve on Ly = a1.

y1 = .7

y2 = .5

y3 = 1

y4 = .1

33


Do a backward solve on Uu = y

u1 + u4 = y1

u2 + (5/6)u4 = y2

u3 + (2/3)u4 = y3

u4 = y4

34

Revised Simplex

The MATLAB program, revisedsimplexlu.m, implements therevised Simplex algorithm as just described with LUdecomposition. (We assume a min problem in standard form.)

Inputs:

I A – the constraint matrix

I b – the right-hand-side vector

I c – the objective function vector

I B – an index set of the basic variables (feasible)

Outputs:

I X – the optimal solution to the linear program

I objVal – the optimal objective function value

I B – the optimal basis

35

Revised SimplexA problem in standard form:

A = [.7 1 1 0 0 0

.5 5/6 0 1 0 0

1 2/3 0 0 1 0

1/10 1/4 0 0 0 1]

c = [-10; -9; 0; 0; 0; 0 ]

b = [630; 600; 708; 135]

B = [3 4 5 6]

Make the call

[X, objVal, B] = revisedsimplexlu( A, b, c, B)

36


I actually have hidden a lot of detail from you.

In the MATLAB code revisedsimplexlu.m, at every iteration,we do a brand new LU decomposition of the basis matrix.

However, from iteration-to-iteration the basis matrix changes byonly one column. Hence an LU decomposition from scratch is ahuge waste!!!!

An incredible amount of research has gone into doing sparse LUupdates. We will not get into this.

If you understand what is in these slides you have a pretty goodfeeling for what is happening with real-world codes.

37

Finding a Starting BasisThe MATLAB simplex routines require as input, B, which indexesa basic feasible solution. For some problems it is easy to find astarting basic feasible solution. The linear program

max 10x1 + 9x2

.7x1 + x2 ≤ 630

.5x1 + (5/6)x2 ≤ 600

x1 + (2/3)x2 ≤ 708

.1x1 + .25x2 ≤ 135

x1, x2 ≥ 0

in standard form is

min −10x1 −9x2.7x1 +x2 +x3 = 630.5x1 +(5/6)x2 +x4 = 600x1 +(2/3)x2 +x5 = 708

.1x1 +.25x2 +x6 = 135x1, x2, x3, x4, x5, x6 ≥ 0

38

Finding a Starting Basis

Given the standard form

min −10x1 −9x2.7x1 +x2 +x3 = 630.5x1 +(5/6)x2 +x4 = 600x1 +(2/3)x2 +x5 = 708

.1x1 +.25x2 +x6 = 135x1, x2, x3, x4, x5, x6 ≥ 0

a trivial basic feasible solution is indexed by

B = [3 4 5 6]

Indeed, if the original linear program has all constraints of the formAx ≤ b and b ≥ 0, then we convert to standard form by adding anonnegative slack variable to each row and putting the slackvariable in the basis. The nonnegativity of b guarantees that this isa basic feasible solution.

39

Finding a Starting BasisFor problems with more general structure, finding a basic feasiblesolution by inspection is not so easy. Fortunately, there asystematic procedure for finding a starting basic feasible solution,but it requires solving a linear program. (See Section 6.5 of thetext).First convert the original linear program to standard form.

min c>x

s.t. Ax = b

x ≥ 0

Then convert the standard form to a Phase I problem.

min 0x + e>a

s.t. Ax + Ia = b

x , a ≥ 0

40

Finding a Starting BasisConverting to Phase I:

I Append to the constraint matrix A an m ×m identity matrixwhere m is the number of constraints. Thus, the newconstraint matrix is [A I ].

I Add m new variables to the problem. These are calledartificial variables. The coefficient of the new variablescorrespond to the identity matrix added.

I Give the original variables a cost of zero in the objectivefunction.

I Give each artificial variable a cost of 1.0 in the objectivefunction.

Result: If the simplex algorithm is applied to the Phase I problemfor a feasible and bounded linear program, then it will find a basicfeasible solution with no artificial variables and have optimal valueequal to zero.

41


Phase I Example: Consider the following linear program instandard form:

MIN 2 X1 + 11 X2 + 7 X3 + 7 X4 + 20 X5 +

2 X6 + 5 X7 + 5 X8

SUBJECT TO

X1 + X2 = 1

X3 + X4 = 1

X5 + X6 = 1

X7 + X8 = 1

3 X1 + 6 X3 + 5 X5 + 7 X7 + X9 = 13

2 X2 + 4 X4 + 10 X6 + 4 X8 + X10 = 10

Convert to a Phase I linear program.

42


Phase I Example: The Phase I version of this problem is:

MIN A1 + A2 + A3 + A4 + A5 + A6

SUBJECT TO

X1 + X2 + A1 = 1

X3 + X4 + A2 = 1

X5 + X6 + A3 = 1

X7 + X8 + A4= 1

3 X1 + 6 X3 + 5 X5 + 7 X7 + X9 + A5= 13

2 X2 + 4 X4 + 10 X6 + 4 X8 + X10 + A6 = 10

43

Finding a Starting BasisPhase I Example: The problem in MATLAB, artificial.m is

A = [1 1 0 0 0 0 0 0 0 0 1 0 0 0 0 0;

0 0 1 1 0 0 0 0 0 0 0 1 0 0 0 0;

0 0 0 0 1 1 0 0 0 0 0 0 1 0 0 0 ;

0 0 0 0 0 0 1 1 0 0 0 0 0 1 0 0 ;

3 0 6 0 5 0 7 0 1 0 0 0 0 0 1 0;

0 2 0 4 0 10 0 4 0 1 0 0 0 0 0 1]

b = [1; 1; 1; 1; 13; 10];

c = [0; 0; 0; 0; 0; 0; 0; 0;

0 ; 0; 1; 1; 1; 1; 1; 1];

B = [11 12 13 14 15 16];

[X, objVal, B] = revisedsimplexlu( A, b, c, B)

44


Phase I Example: The MATLAB solution is

objVal =

0

B =

1 3 6 5 7 4

45

Unbounded Problems

Recall that a linear program in standard form can be rewritten as

z0 = c>B A−1B b + w>NxN

xB = A−1B b − A−1B ANxN

xB , xN ≥ 0

What if there is a variable k ∈ N such that?

I wk < 0

I ak = A−1B ak ≤ 0

46

Unbounded Problems

We are unbounded!!! We currently are at the point x1 = [xB xN ]with objective function value c>B A−1B b. Move to the new point:

x2 = x1 + λr

for positive λ where component k of the r vector is 1, and theother nonzero components are given by ak .

This is tough. Let’s illustrate.

47

Unbounded Problems

Consider the following linear program in standard form.

min −2x1 −3x2x2 −x3 = 1

x1 +x2 −x4 = 2−.5x1 +x2 +x5 = 8−x1 +x2 +x6 = 6x1, x2, x3, x4, x5, x6 ≥ 0

The associated MATLAB file is unbounded.m.

48

Unbounded ProblemsThe MATLAB file is unbounded.m. defines the following:

% the constraint matrix

A = [0 1 -1 0 0 0

1 1 0 -1 0 0

-.5 1 0 0 1 0

-1 1 0 0 0 1]

% the objective function

c = [-2; -3; 0; 0; 0; 0 ]

% the right-hand-side

b = [1; 2; 8; 6]

%Initial Basis

B = [1 2 5 6]

49

Unbounded Problems

The associated MATLAB file is unbounded.m.

[tableau, B] = simplexpivot( A, b, c, B)

-0.00 -0.00 1.00 2.00 -0.00 -0.00 -5.00

----------------------------------------------------------------

1.00 0.00 1.00 -1.00 0.00 0.00 1.00

0.00 1.00 -1.00 0.00 0.00 0.00 1.00

0.00 0.00 1.50 -0.50 1.00 0.00 7.50

0.00 0.00 2.00 -1.00 0.00 1.00 6.00

Variable x4 has a reduced cost of -2. What is the effect of bringingthis variable into the basis?

50

Unbounded ProblemsIf variable x4 pivots in at any positive value, no other variablepivots out. The current solution is

x0 =

1100

7.56.0

If we increase x4 by λ the new solution is

x = x0 + λr =

1100

7.56.0

+ λ

1001.5

1.0

51

Unbounded ProblemsThe vector

r =

1001.5

1.0

is called an extreme ray or extreme direction of the linearprogram. I can take any positive multiple of this, add it to afeasible solution, and still be feasible.

Since c>r = −2,

c>(x0 + λr) = c>x0 + λc>r = −5− 2λ

52

Unbounded ProblemsExperiment a bit with unbounded.m to get a feeling for what isgoing on.

x0 = [1; 1; 0; 0; 7.5; 6.0]

r = [1; 0; 0; 1; .5; 1]

lambda = 100

%new solution

x = x0 + lambda*r

%new objective function value

objVal = c’*x

%should be feasible, i.e.

%this should evaluate to 0

A*x - b

53

Unbounded Problems

Here is the geometry again.

2 4 6 8x1

2

4

6

8

10

12x2

Feasible Region

Iteration 1

Iteration 2

Iteration 3

Iteration 4

54

Unbounded Problems

Every point (working in (x1, x2) space) in the feasible region can bewritten as:

x = λ1

[11

]+ λ2

[02

]+ λ3

[06

]+ λ4

[4

10

]+ λ5

[10

]+ λ6

[21

]

λ1 + λ2 + λ3 + λ4 = 1

λ1, λ2, λ3, λ4, λ5, λ6 ≥ 0

Observations:

I The feasible region is unbounded.

I How did I find the extreme ray corresponding to variable λ6?

55

Unbounded Problems

In practice, what does an unbounded problem tell you?

56

Convergence

Why does the Simplex algorithm terminate?

I Every basic feasible solution corresponds to an extreme point.

I There are a finite number of extreme points.

I Assuming no degenerate pivots the objective function valuestrictly decreases at every iteration. Hence we cannot repeatextreme points. Therefore we terminate with a finite numberof iterations.

Remember – if there is an optimal solution to the problem, there isan optimal extreme point solution. A linear program can fail tohave an optimal solution if and only if it is either infeasible orunbounded.

57

Convergence

Observations:

1. The change in the objective function value for a pivot oncolumn k in row i is

wk

(biaik

)2. Since wk < 0, aik > 0, and bi ≥ 0, this change can only be

zero if bi = 0.

A pivot on row i when bi = 0 is called a degenerate pivot. It istheoretically possible for the Simplex algorithm to cycle in thepresence of degeneracy. Cycling is not a problem in practice.

However, degeneracy is a real problem in that it can really slowthings down. Commercial codes have pivot selection rules that willprevent cycling and reduce the number of degenerate pivots.

58

The Future – Pricing

Here is where we are headed. Right now we explicitly price everycolumn when we make the calculationStep 2: (Pivot Column Selection) Calculate the reduced costs

wN = c>N − (c>B A−1B )AN

For many important applications this is not practical:

I there are so many columns in AN it would require too muchmemory and time to price

I it is not even practical to enumerate all of the columns in AN

We will do the pricing operation by solving another optimizationproblem or by implementing an algorithm.

59

The Future – Interior Point

The Simplex algorithm moves from extreme point to extreme pointon the boundary of the feasible region.

At the end of the quarter we will study interior point algorithmsthat move through the interior of the feasible region.

We will apply interior point algorithms to both linear andnon-linear programs.

Even for linear programs, interior point algorithms are based onideas from nonlinear programming so we need to cover nonlinearprogramming in order to understand how interior point algorithmswork.

60

Linear Programming The Simplex Algorithm: Part II Chapter...

Documents

Transcript of Linear Programming The Simplex Algorithm: Part II Chapter...