COMP3121/9101/3821/9801 Lecture Notes

Linear Programming

LiC: Aleks Ignjatovic

THE UNIVERSITY OFNEW SOUTH WALES

School of Computer Science and EngineeringThe University of New South Wales

Sydney 2052, Australia

Consider the following problem. You are given a list of food sources f1, f2, . . . , fn;for each source fi you are given its price per gram pi, the number of calories ciper gram, and for each of 13 vitamins V1, V2, . . . , V13 you are given the contentv(i, j) of milligrams of vitamin Vj in one gram of food source fi. Your goal isto find a combination of quantities of each food source such that:

1. the total number of calories in all of the chosen food is equal to a recom-mended daily value of 2000 calories;

2. the total intake of each vitamin Vj is at least the recommended amountof wj milligrams for all 1 ≤ j ≤ 13;

3. the price of all food per day is as small as possible.

To obtain the corresponding constraints let us assume that we take xi gramsof each food source fi. Then:

1. the total number of calories must satisfy

n∑i=1

xici = 2000;

2. for each vitamin Vj the total amount in all food must satisfy

n∑i=1

xiv(i, j) ≥ wj (1 ≤ j ≤ k);

3. an implicite assumption is that all the quantities must be non-negativenumbers,

xi ≥ 0, 1 ≤ i ≤ n.

Our goal is to minimise the objective function which is the total cost y =∑ni=1 xipi. Note that here all the equalities and inequalities, as well as the

objective function, are linear. This problem is a typical example of a LinearProgramming problem.

Assume now that you are the Shadow Treasurer and you want to makecertain promises to the electorate which will ensure your party will win in theforthcoming elections. You can promise that you will build certain number ofbridges, each 3 billion a piece, certain number of rural airports, each 2 billiona piece and a number of olympic swimming pools each a billion a piece. Eachbridge you promise brings you 5% of city votes, 7% of suburban votes and 9%of rural votes; each rural airport you promise brings you no city votes, 2% ofsuburban votes and 15% of rural votes; each olympic swimming pool promisedbrings you 12% of city votes, 3% of suburban votes and no rural votes. In orderto win, you have to get at least 51% of each of the city, suburban and ruralvotes. You wish to win the election by making a promise that will cost as littlebudget money as possible.

We can let the number of bridges to be built be xb, number of airports xaand the number of swimming pools xp. We now see that the problem amounts

1

to minimising the objective y = 3xb + 3xa + xp, while making sure that thefollowing constraints are satisfied:

0.05xb + 0.12xp ≥ 0.51 (securing majority of city votes)

0.07xb + 0.02xa + 0.03xp ≥ 0.51 (securing majority of suburban votes)

0.09xb + 0.15xa ≥ 0.51 (securing majority of rural votes)

xb, xa, xp ≥ 0.

However, there is a very significant difference with the first example: you caneat 1.56 grams of chocolate but (presumably) you cannot promise to build 1.56bridges, 2.83 airports and 0.57 swimming pools! The second example is anexample of an Integer Linear Programming problem, which requires all the so-lutions to be integers. Such problems are MUCH harder to solve than the plainLinear Programming problems whose solutions can be real numbers.

We now follow closely COMP3121 textbook by Cormen, Leiserson, Rivestand Stein, Introduction to Algorithms. We start by defining a common repre-sentations of Linear Programming optimisation problems.

In the standard form the objective to be maximized is given by

f(x) =

n∑j=1

cj xj ,

and the constraints are of the form

gi(x) ≤ bi, 1 ≤ i ≤ m; (1)

xj ≥ 0, 1 ≤ j ≤ n, (2)

where

gi(x) =

n∑j=1

aijxj ; 1 ≤ i ≤ m. (3)

Note that x represents a vector, x = 〈x1 . . . xn〉T, and aij are reals. Theabove somewhat messy formulation can be reformulated in a much more com-pact matrix form. To get a very compact representation of linear programslet us introduce a partial ordering on vectors x ∈ Rn by x ≤ y if and onlyif such inequalities hold coordinate-wise, i.e., if and only if xj ≤ yj for all1 ≤ i ≤ n. Recall that vectors are written as a single column matrices; thus,letting c = 〈c1, . . . , cn〉T ∈ Rn and b = 〈b1, . . . bm〉T ∈ Rm, and letting A be thematrix A = (aij) of size m×n, we get that the above problem can be formulatedsimply as:

maximize cTx

subject to the following two (matrix-vector) constraints:1 Ax ≤ b and x ≥ 0.

1Note that the inequality involved in the constraints is the partial ordering on vectorswhich we have just introduced above. Also, we are slightly abusing the notation because inthe non-negativity constraint, 0 actually represents a vector 0 ∈ Rn with all coordinates equalto 0.

2

Thus, to specify a Linear Programming optimization problem we just haveto provide a triplet (A, b, c), and the information that the triplet represents anLP problem in the standard form.

Wile in general the “natural formulation” of a problem into a Linear Pro-gram does not necessarily produce the non-negativity constrains (2) for all ofthe variables, in the standard form such constraints are indeed required for allof the variables. This poses no problem, because each occurrence of an uncon-strained variable xj can be replaced by the expression x′j − x∗j where x′j , x

∗j are

new variables satisfying the constraints x′j ≥ 0, x∗j ≥ 0.

Any vector x which satisfies the two constraints is called a feasible solution,regardless of what the corresponding objective value cTx might be. Note thatthe set of feasible solutions, i.e., the domain over which we seek to maximize theobjective, is a convex set because it is an intersection of the half spaces definedby each of the constraints.

As an example, let us consider the following optimization problem:

maximize 3x1 + x2 + 2x3 (4)

subject to the constraints

x1 + x2 + 3x3 ≤ 30 (5)

2x1 + 2x2 + 5x3 ≤ 24 (6)

4x1 + x2 + 2x3 ≤ 36 (7)

x1, x2, x3 ≥ 0 (8)

How large can the value of the objective z(x1, x2, x3) = 3x1 + x2 + 2x3 be,without violating the constraints? If we add inequalities (5) and (6), we get

3x1 + 3x2 + 8x3 ≤ 54

Since all variables are constrained to be non-negative, we are assured that

3x1 + x2 + 2x3 ≤ 3x1 + 3x2 + 8x3 ≤ 54

The far left hand side of this equation is just the objective (4); thus z(x1, x2, x3)is bounded above by 54, i.e., z(x1, x2, x3) ≤ 54.

Can we obtain a tighter bound? We could try to look for coefficientsy1, y2, y3 ≥ 0 to be used to for a linear combination of the constraints:

y1(x1 + x2 + 3x3) ≤ 30y1

y2(2x1 + 2x2 + 5x3) ≤ 24y2

y3(4x1 + x2 + 2x3) ≤ 36y3

Then, summing up all these inequalities and factoring, we get

x1(y1 + 2y2 + 4y3) + x2(y1 + 2y2 + y3) + x3(3y1 + 5y2 + 2y3)

≤ 30y1 + 24y2 + 36y3 (9)

3

If we compare this with our objective (4) we see that if we choose y1, y2 and y3so that:

y1 + 2y2 + 4y3 ≥ 3

y1 + 2y2 + y3 ≥ 1

3y1 + 5y2 + 2y3 ≥ 2

then

x1(y1 + 2y2 + 4y3) + x2(y1 + 2y2 + y3) + x3(3y1 + 5y2 + 2y3) ≥ 3x3 + x2 + 2x3

Combining this with (9) we get:

30y1 + 24y2 + 36y3 ≥ 3x1 + x2 + 2x3 = z(x1, x2, x3)

Consequently, in order to find as tight upper bound for our objective z(x1, x2, x3)we have to look for y1, y2, y3 which produce the smallest possible value ofz∗(y1, y2, y3) = 30y1 + 24y2 + 36y3, but which do not violate the constraints

y1 + 2y2 + 4y3 ≥ 3 (10)

y1 + 2y2 + y3 ≥ 1 (11)

3y1 + 5y2 + 2y3 ≥ 2 (12)

y1, y2, y3 ≥ 0 (13)

Thus, trying to find the best upper bound for our objective z(x1, x2, x3) obtainedby forming a linear combination of the constraints only reduces the originalmaximization problem to a minimization problem:

minimize 30y1 + 24y2 + 36y3

subject to the constraints (10-13)

Such minimization problem P ∗ is called the dual problem of the initial problemP .

Let us now repeat the whole procedure with P ∗ in place of P , i.e., let us findthe dual program (P ∗)∗ of P ∗. We are now looking for non negative multipliersz1, z2, z3 ≥ 0 to multiply inequalities (10-12) and obtain

z1(y1 + 2y2 + 4y3) ≥ 3z1

z2(y1 + 2y2 + y3) ≥ z2z3(3y1 + 5y2 + 2y3) ≥ 2z3

Summing these up and factoring produces

y1(z1 + z2 + 3z3) + y2(2z1 + 2z2 + 5z3) + y3(4z1 + z2 + 2z3) ≥ 3z1 + z2 + 2z3(14)

If we choose these multiples so that

z1 + z2 + 3z3 ≤ 30 (15)

2z1 + 2z2 + 5z3 ≤ 24 (16)

4z1 + z2 + 2z3 ≤ 36 (17)

4

we will have:

y1(z1 + z2 + 3z3) + y2(2z1 + 2z2 + 5z3) + y3(4z1 + z1 + 2z3) ≤ 30y1 + 24y2 + 36y3

Combining this with (14) we get

3z1 + z2 + 2z3 ≤ 30y1 + 24y2 + 36y3

Consequently, finding the dual program (P ∗)∗ of P ∗ amounts to maximizingthe objective 3z1 + z2 + 2z3 subject to the constraints (15-17). But notice that,except for having different variables, (P ∗)∗ is exactly our starting program P !Thus, the dual program (P ∗)∗ for program P ∗ is just P itself, i.e., (P ∗)∗ = P .

So, at the first sight, looking for the multipliers y1, y2, y3 did not help much,because it only reduced a maximization problem to an equally hard minimiza-tion problem.2

To find the maximal value of 3x1 + x2 + 2x3 subject to the constraints

x1 + x2 + 3x3 ≤ 30

2x1 + 2x2 + 5x3 ≤ 24

4x1 + x2 + 2x3 ≤ 36

let us start with x1 = x2 = x3 = 0 and ask ourselves: how much can we increasex1 without violating the constraints? Since x1 +x2 +3x3 ≤ 30 we can introducea new variable x4 such that x4 ≥ 0 and

x4 = 30− (x1 + x2 + 3x3) (18)

Since such variable measures how much “slack” we got between the actual valueof x1+x2+3x3 and its upper limit 30, such x4 is called a slack variable. Similarlywe introduce new variables x5, x6 requiring them to satisfy x5, x6 ≥ 0 and let

x5 = 24− 2x1 − 2x2 − 5x3 (19)

x6 = 36− 4x1 − x2 − 2x3 (20)

Since we started with the values x1 = x2 = x3 = 0, this implies that these newslack variables must have values x4 = 30, x5 = 24, x6 = 36, or as a single

vector, the initial basic feasible solution is (x1

0 ,x2

0 ,x3

0 ,x4

30,x5

24,x6

36). Note that “fea-sible” refers merely to the fact that all of the constraints are satisfied.3

Now we see that (18) implies that x1 cannot exceed 30, and (19) impliesthat 2x1 ≤ 24, i.e., x1 ≤ 12, while (20) implies that 4x1 ≤ 36, i.e., x1 ≤ 9. Since

2It is now useful to remember how we proved that the Ford - Fulkerson Max Flow algorithmin fact produces a maximal flow, by showing that it terminates only when we reach the capacityof a (minimal) cut...

3Clearly, setting all variables to 0 does not always produce a basic feasible solution becausethis might violate some of the constraints; this would happen, for example, if we had aconstraint of the form −x1 +x2 +x3 ≤ −3; choosing an initial basic feasible solution requiresa separate algorithm to “bootstrap” the SIMPLEX algorithm - see for the details our (C-L-R-S) textbbok.

5

all of these conditions must be satisfied we conclude that x1 cannot exceed 9,which is the upper limit coming from the constraint (20).

If we set x1 = 9, this forces x6 = 0. We now swap the roles of x1 and x6:since we cannot increase x1 any more, we eliminate x1 from the right hand sidesof the equations (18-20) and from the objective, introducing instead variable x6to the right hand side of the constraints and into the objective. To do that, wesolve equation (20) for x1:

x1 = 9− x24− x3

2− x6

4

and eliminate x1 from the right hand side of the remaining constraints and theobjective to get:

z = 3(

9− x24− x3

2− x6

4

)+ x2 + 2x3

= 27− 3

4x2 −

3

2x3 −

3

4x6 + x2 + 2x3

= 27 +1

4x2 +

1

2x3 −

3

4x6

x5 = 24− 2(

9− x24− x3

2− x6

4

)− 2x2 − 5x3

= 6 +x22

+ x3 +x62− 2x2 − 5x3

= 6− 3

2x2 − 4x3 +

x62

x4 = 30−(

9− x24− x3

2− x6

4

)− x2 − 3x3

= 21 +x24

+x32

+x64− x2 − 3x3

= 21− 3

4x2 +

5

2x3 +

x64

To summarise: the “new” objective is

z = 27 +1

4x2 +

1

2x3 −

3

4x6

and the “new constraints” are

x1 = 9− x24− x3

2− x6

5(21)

x4 = 21− 3

4x2 −

5

2x3 +

x64

(22)

x5 = 6− 3

2x2 − 4x3 +

x62

(23)

Our new basic feasible solution replacing (x1

0 ,x1

0 ,x3

0 ,x4

30,x5

24,x6

36) is obtained by set-

ting all the variables on the right to zero, thus obtaining (x1

9 ,x2

0 ,x3

0 ,x4

21,x5

6 ,x6

0 ).

NOTE: These are EQUIVALENT constraints and objectives; the old ones wereonly transformed to an equivalent form. Any values of the variables will produceexactly the same value in both forms of the objective and they will satisfy the

6

first set of constraints if and only if they satisfy the second set.

So x1 and x6 have switched their roles; x1 acts as a new basic variable,

and the new basic feasible solution is: (x1

9 ,x2

0 ,x3

0 ,x4

21,x5

6 ,x6

0 ); the new value of theobjective is z = 27 + 1

40 + 120− 3

40 = 27. We will continue this process of find-ing basic feasible solutions which increase the value of the objective, switchingwhich variables are used to measure the slack and which are on the right handside of the constraint equations and in the objective. The variables on the leftare called the basic variables and the variables on the right are the non basicvariables.

We now choose another variable with a positive coefficient in the objective,say x3 (we could also have chosen x2). How much can we increase it?

From (21) we see that x3

2 must not exceed 9, otherwise x1 will becomenegative. Thus x3 cannot be larger than 18. Similarly, 5

2x3 cannot exceed21, otherwise x4 will become negative, and so x3 ≤ 42

5 ; similarly, 4x3 cannotexceed 6, ie, x3 ≤ 3

2 . Thus, in order for all constraints to remain valid, x3cannot exceed 3

2 . Thus we increase x3 to 32 ; equation (23) now forces x5 to zero.

We now eliminate x3 from the right hand side of the constraints and from theobjective, taking it as a new basic variable:

4x3 = 6− 3

2x2 − x5 +

x62

i.e.,

x3 =3

2− 3

8x2 −

1

4x5 +

1

8x6 (24)

After eliminating x3 by substitution using (24), the objective now becomes:

z =27− 1

4x2 +

1

2

(3

2− 3

8x2 −

1

4x5 +

1

8x6

)− 3

4x6 =

111

4+

1

16x2 −

1

8x5 −

11

6x6 (25)

Using (24) to eliminate x3 from the constraints, after simplifications we get thenew constraints:

x1 =33

4− x2

16+x58− 5x6

16(26)

x3 =3

2− 3x2

8− x5

4+x68

(27)

x4 =69

4+

3x216

+5x38− x6

16(28)

Our new basic solution is again obtained using the fact that all variables on theright and in the objective, including the newly introduced non-basic variable

x5, are equal to zero, i.e., the new basic feasible solution is (x1334 ,

x2

0 ,x332 ,

x4694 ,

x5

0 ,x6

0 )and the new value of the objective is z = 111

4 .

Comparing this with the previous basic feasible solution (x1

9 ,x2

0 ,x3

0 ,x4

21,x5

6 ,x6

0 ) wesee that in the new basic feasible solution the value of x1 dropped from 9 to

7

33/4; however, this now has no effect on the value of the objective because x1no longer appears in the objective; all the variables appearing in the objective(thus, the non-basic variables) always have value 0.

We now see that the only variable in the objective (25) appearing with apositive coefficient is x2. How much can we increase it without violating the newconstraints? The first constraint (26) implies that x2

16 ≤334 , i.e., that x2 ≤ 132;

the second constraint (27) implies that 3x2

8 ≤32 , i.e., that x2 ≤ 4. Note that

the third constraint (28) does not impose any restrictions on how large x2 canbe, for as long as it is positive; thus, we conclude that the largest possible valueof x2 which does not cause violation of any of the constraints is x2 = 4, whichcorresponds to constraint (27). The value x2 = 4 forces x3 = 0; we now switchthe roles of x2 and x3 (this operation of switching the roles of two variables iscalled pivoting) by solving (27) for x2:

x2 = 4− 8x23− 2x5

3+x63

and then using this to eliminate x2 from the objective, obtaining

z = 28− x36− x5

6− 2x6

3

as well as from the constraints, obtaining after simplification

x1 = 8 +x36

+x56− x6

3(29)

x2 = 4− 8x33− 2x5

3+x63

(30)

x4 = 18− x32

+x52

(31)

which produces the new basic feasible solution (x1

8 ,x2

4 ,x3

0 ,x4

18,x5

0 ,x6

0 ) with the newvalue of the objective z = 28. Note that in the new objective all the variablesappear with a negative coefficient; thus our procedure terminates, but did itfind the maximum value of the objective? Maybe with a different choices ofvariables in pivoting we would have come up with another basic feasible solutionwhich would have different basic variables, also with all non basic variables inthe objective appearing with a negative coefficient, but for which the obtainedvalue of the objective is larger?

This is not the case: just as in the case of the Ford Fulkerson algorithmfor Max Flow, once the pivoting terminates, the solution must be optimal re-gardless of which particular variables where swapped in pivoting, because thepivoting terminates when the corresponding basic feasible solution of the pro-gram becomes equal to a basic feasible solution of the dual program. Sinceevery feasible solution of the dual is larger than every feasible solution of thestarting (or primal) program, we get that the SIMPLEX algorithm must returnthe optimal value after it terminates. We now explain this in more detail.

8

1 LP Duality

General setup

Comparing our initial program P with its dual P ∗:

P : maximize 3x1 + x2 + 2x3,

subject to the constraints

x1 + x2 + 3x3 ≤ 30

2x1 + 2x3 + 5x3 ≤ 24

4x1 + x2 + 2x3 ≤ 36

x1, x2, x3 ≥ 0;

P ∗ : minimize 30y1 + 24y2 + 36y3,

subject to the constraints

y1 + 2y2 + 4y3 ≥ 3

y1 + 2y2 + y3 ≥ 1

3y1 + 5y2 + 2y3 ≥ 2

y1, y2, y3 ≥ 0.

we see that the original, primal Linear Program P and its dual Linear Programare related as follows

P : maximize z(x) =

n∑j=1

cjxj ,

subject to the constraints

n∑j=1

aijxj ≤ bi; 1 ≤ i ≤ m

x1, . . . , xn ≥ 0;

P ∗ : minimize z∗(y) =

m∑i=1

biyi,

subject to the constraints

m∑i=1

aijyi ≥ cj ; 1 ≤ j ≤ n

y1, . . . , ym ≥ 0,

or, in matrix form,

P : maximize z(x) = cTx, subject to the constraints Ax ≤ b and x ≥ 0;

P ∗ : minimize z∗(y) = bTy, subject to the constraints ATy ≥ c and y ≥ 0.

9

Weak Duality Theorem If x = 〈x1, . . . , xn〉 is any basic feasible solutionfor P and y = 〈y1, . . . , ym〉 is any basic feasible solution for P ∗, then:

z(x) =

n∑j=1

cjxj ≤n∑

i=1

biyi = z∗(y)

Proof: Since x and y are basic feasible solutions for P and P ∗ respectively, wecan use the constraint inequalities, first from P ∗ and then from P to obtain

z(x) =

n∑j=1

cjxj ≤n∑

j=1

(m∑i=1

aijyi

)xj =

m∑i=1

n∑j=1

aijxj

yi ≤n∑

i=1

biyi = z∗(y)

Thus, every feasible solution of P ∗ is an upper bound for the set of all feasiblesolutions of P , and every feasible solution of P is a lower bound for the set offeasible solutions for P ∗; see the figure below.

Solutions for P

Solutions for P*

Consequently, if we find a feasible solution of P which is also a feasible solutionfor P ∗, such solution must be maximal feasible solution for P and minimal fea-sible solution for P ∗.

We now show that when the SIMPLEX algorithms terminates that it pro-duces a basic feasible solution x for P , and, implicitly, a basic feasible solutiony for the dual P ∗ for which z(x) = z∗(y); by the above, this will imply thatz(x) is the maximal value for the objective of P and that z∗(y) is the minimalvalue of the objective for P ∗.

Assume that the SIMPLEX algorithm has terminated; let B be such thatthe basic variables (variables on the left hand side of the constraint equations) inthe final form of P are variables xi for which i ∈ B; let N = {1, 2, . . . , n+m}\B;then xj for j ∈ N are all the non-basic variables in the final form of P . Since theSIMPLEX algorithm has terminated, we have also obtained a set of coefficientscj ≤ 0 for j ∈ N , as well as v such that the final form of the objective is

z(x) = v +∑j∈N

cjxj

If we set all the final non-basic variables xj , j ∈ N , to zero, we obtain a basicfeasible solution x for which z(x) = v.

10

Let us define cj = 0 for all j ∈ B; then

z(x) =

n∑j=1

cjxj

= v +

n+m∑j=1

cjxj

= v +

n∑j=1

cjxj +

n+m∑i=n+1

cixi

= v +

n∑j=1

cjxj +

m∑i=1

cn+ixn+i

Since the variables xn+i, (1 ≤ i ≤ m), are the initial slack variables, they satisfyxn+i = bi −

∑nj=1 aijxj ; thus we get

z(x) =n∑

j=1

cjxj

= v +

n∑j=1

cjxj +

m∑i=1

cn+i

bi − n∑j=1

aijxj

= v +

n∑j=1

cjxj +

m∑i=1

cn+ibi −m∑i=1

n∑j=1

cn+iaijxj

= v +

n∑j=1

cjxj +

m∑i=1

cn+ibi −n∑

j=1

m∑i=1

cn+iaijxj

= v +

m∑i=1

cn+ibi +

n∑j=1

(cj −

m∑i=1

cn+iaij

)xj

The above equations hold true for all values of x; thus, comparing the first andthe last equation we conclude that

v +

m∑i=1

cn+ibi = 0;

cj −m∑i=1

cn+iaij = cj , (1 ≤ j ≤ n).

i.e.,

m∑i=1

bi(−cn+i) = v;

m∑i=1

aij(−cn+i) = cj + (−cj), (1 ≤ j ≤ n).

11

We now see that if we set yi = −cn+i for all 1 ≤ i ≤ m, then, since theSIMPLEX terminates when all coefficients of the objective are either negativeor zero, then such y satisfies:

m∑i=1

biyi = v;

m∑i=1

aijyi = cj − cj ≥ cj , (1 ≤ j ≤ n),

yi ≥ 0, (1 ≤ i ≤ m).

Thus, such y is a basic feasible solution for the dual program P ∗ for whichthe dual objective has the same value v which the original, primal program Pachieves for the basic feasible solution x. Thus, by the Weak Duality Theorem,we conclude that v is the maximal feasible value of P and minimal feasible valuefor P ∗.

Note also that the basic and non basic variables of the final primal form ofthe problem and of its dual are complementary: for every 1 ≤ i ≤ m, variablexn+i is basic for the final form of the primal if and only if yi is non basic for thefinal form of the dual; (similarly, for every 1 ≤ j ≤ n, variable xj is basic forthe final form of the primal if and only if ym+j is not basic for the final formof the dual). Since the basic variables measure the slack of the correspondingbasic feasible solution, we get that if x and y are the extremal feasible solutions

12

for P and P ∗, respectively, then for all 1 ≤ j ≤ n and all 1 ≤ i ≤ m,

either xj = 0 or ym+j = 0, i.e.,

m∑i=0

aijyi = cj ;

either yi = 0 or xn+i = 0, i.e.,

n∑j=0

aijxj = bi.

Note that any equivalent form of P which is obtained through a pivotingoperation is uniquely determined by its corresponding set of basic variables.Assuming that we have n variables and m equations, then there are

(n+mm

)choices for the set of basic variables. Using the Stirling formula

n! ≈√

2πn(ne

)n⇒ lnn! ≈ ln(2πn)

2+ n lnn− n = n lnn− n+O(lnn)

we get

ln

(n+m

m

)= ln

(m+ n)!

m!n!= ln(m+ n)!− lnm!− lnn!

= (m+ n) ln(m+ n)− (m+ n)− n lnn−m lnm+m+ n

= m(ln(m+ n)− lnm) + n(ln(m+ n)− lnn)

≥ m+ n

Thus, the total number of choices for the set of the basic variables is(n+mm

)>

em+n. This implies that the SIMPLEX algorithm could potentially run in ex-ponential time, and in fact, one can construct examples of LP on which therun time of the SIMPLEX algorithm is exponential. However, in practice theSIMPLEX algorithm is extremely efficient, even for large problems with thou-sands of variables and constraints, and it tends to outperform algorithms for LPwhich do run in polynomial time (the Ellipsoid Method and the Interior PointsMethod).

1.1 Examples of dual programs: max flow

We would now like to formulate the Max Flow problem in a flow network as aLinear Program.

Thus, assume we are given a flow network G with capacities κij of edges(i, j) ∈ G. The max flow problem seeks to maximise the total flow

∑j:(s,j)∈E fsj

13

through a flow network graph G, subject to the constraints:

C∗

fij ≤ κij ; (i, j) ∈ G; (flow smaller than pipe’s capacity)∑

i : (i,j)∈G fij =∑

k : (j,k)∈G fjk; j ∈ G− {s, t}; (incoming flow equals outgoing)

fij ≥ 0; (i, j) ∈ G (no negative flows).

To eliminate the equality in the second constraint in C∗ while introducing onlyone rather than two inequalities, we use a “trick”: we make the flow circular, byconnecting the sink t with the source s with a pipe of infinite capacity. Thus,we now have a new graph G′, G ⊂ G′, with an additional edge (t, s) ∈ G′ withcapacity ∞.

We can now formulate the Max Flow problem as a Linear Program by replacingthe equality in the second constraint with a single but equivalent inequality:

P: maximize: ftssubject to the constraints:

fij ≤ κij ; (i, j) ∈ G;∑i : (i,j)∈G′

fij −∑

k : (j,k)∈G′

fjk ≤ 0; j ∈ G;

fij ≥ 0; (i, j) ∈ G′.

Thus, the coefficients cij of the objective of the primal P are zero for allvariables fij except for fts which is equal to 1, i.e.,

z(f) =∑ij

0 · fij + 1 · fts (1.1)

To obtain the dual of P we look for coefficients dij , (i, j) ∈ G correspondingto the first set of constraints, and coefficients pj , j ∈ G corresponding to thesecond set of constraints to use as multipliers of the constraints:

fijdij ≤ κij dij ; (i, j) ∈ G; (1.2)∑i : (i,j)∈G′

fijpj −∑

k : (j,k)∈G′

fjkpj ≤ 0; j ∈ G. (1.3)

14

note the two special cases of the second inequality involving fts are∑i : (i,t)∈G

fitpt − ftspt ≤ 0;

ftsps −∑

k : (s,k)∈G

fskps ≤ 0.

Summing all of inequalities from (1.2) and (1.3) and factoring out, we get∑(i,j)∈G

(dij − pi + pj)fij + (ps − pt)fts ≤∑

(i,j)∈G

κijdij

Thus, the dual objective is the right hand side of the above inequality, and, asbefore, the dual constraints are obtained by comparing the coefficients of theleft hand side with the coefficients of the objective of P :

P ∗ : minimize:∑

(i,j)∈G κijdij

subject to the constraints:

dij − pi + pj ≥ 0 (i, j) ∈ Gps − pt ≥ 1

dij ≥ 0 (i, j) ∈ Gpi ≥ 0 i ∈ G

Let us write the constraints of P using new slack variables ψij , ϕj :

ψij = κij − fij ; (i, j) ∈ G;

ϕj =∑

k : (j,k)∈G′

fjk −∑

i : (i,j)∈G′

fij ; j ∈ G;

fij ≥ 0; (i, j) ∈ G′;ψij ≥ 0; (i, j) ∈ G′;ϕj ≥ 0; j ∈ G

Note now an important feature of both P and P ∗: all variables fij , ψij , ϕj in P(and also all variables dij , pi in P ∗) appear only with the coefficient ±1. Onecan now see that, if we solve the above constraint equations for any subset ofthe set of all variables {fij , ψij , ϕj : (i, j) ∈ G∗, j ∈ G} as the set of the basicvariables, all the coefficients cij in the new objective which multiply ψij or ϕj

and which are obtained after the corresponding substitutions removing the ba-sic variables from the objective will have coefficients either 0 or 1.1 This meansthat in the corresponding basic feasible solution of P ′ at which the minimalvalue of the dual program is obtained, also all values dij of dij and all values pjof pj will be either 0 or 1.

What is the interpretation of such solution of the dual Linear Program P ∗?Let us consider the set A of all vertices j of G for which pj = 1 and the set B of

1This corresponds to the fact that such constraints define a polyhedra whose all verticeshave coordinates which are all either 0 or 1.

15

all vertices j for which pj = 0. Then A∪B = G and A∩B = ∅. The constraintps − pt ≥ 1 of P ∗ implies that ps = 1 and pt = 0, i.e., s ∈ A and t ∈ B. Thus,A and B define a cut in the flow network. Since at points p, d the objective∑

(i,j)∈G κijdij achieves the minimal value, the constraint dij − pi + pj ≥ 0 im-

plies that dij = 1 if and only if pi = 1 and pj = 0, i.e., if and only the edge(i, j) has crossed from set A into set B. Thus, the minimum value of the dualobjective

∑(i,j)∈G κijdij precisely corresponds to the capacity of the cut defined

by A,B. Since such value is equal to the maximal value of the flow defined bythe primal problem, we have obtained a maximal flow and minimal cut in G!

As we have mentioned, the extreme values of linear optimization problemsare always obtained on the vertices of the corresponding constraint polyhedra.In this particular case all vertices are with 0, 1 coordinates; however, in generalthis is false. For example, NP hard problems formulated as Linear Program-ming problems always result in polyhedra with non-integer vertices.

Theorem: Solving an Integer Linear Program (ILP), i.e., an LP with additionalconstraints that the values of all variables must be integers is NP hard, i.e., therecannot be a polynomial time algorithm for solving ILPs (unless an extremelyunlikely thing happens, namely that P = NP ).

16