University of Massachusetts at Amherst

665 Computer Algorithm Mathematical Programming

Professor Maciej CiesielskiSpring 2004

(DFG)

 OUTLINE

1. Non Linear Programming2. Quadratic Programming3. Linear Programming

A. Geometric SolutionsB. Basic SolutionsC. Simplex MethodD. LP modifications

4. Integer ProgrammingA. UnimodularityB. Network Flow Problems

5. Conclusions

Mathematical Programming Nonlinear Programming 

Minimize f(x) subject to:

g(x) = 0h(x) ≥ 0

  where x = [x1,...xn]T, and all functions f, g, h, are differentiable.

{

Convexity 

To obtain global optimum, it is important that the constraint set

 {x | g(x) = 0,h(x) ≥ 0}

 and the function f(x) be convex.

If the function is convex over a convex constraint set, then local minimun is also a globa1 minimum

Set convexity

Convex set non-convex set

Function convexity

Convex function non-convex function

Classical optimization deals with unconstrained optimization problems.

Solution: move the constraints to the function and solve the unconstrained optimization problem

For the minimization problem

Min f(x) subject to g(x) = 0

1. Penalty function method: replace f(x) by

F(x) = f(x)+ M( g(x))2, M >> 0

2. Lagrangian multipliers: replace f(x) by

L(x,λ) = f(x) + Σ λi gi(x)

At the optimum point, x*, L(x*, λ) = f(x*)

i

Quadratic Programming (QP)

Minimize xTQx

Subject to:

{Ax ≥ b

Where Q is an n x n symmetric matrix, A is a constraint matrix, and b is a constraint vector

Matrix Q is positive definite, i.e.,

xTQx > 0

For every x 0. Thus f is convex.

Linear Programming (LP)

Minimize cT

Subject to

Ax ≥ b

x ≥ 0

Where cT is a coefficient vector for f, A is a constraint matrix, and b es a constraint vector.

The constraint set

{x | Ax ≥ b}

Is a convex polyhedron, and f is linear, convex.

{

Linear Programming (LP)Linear Programming (LP)

Optimization problem:Optimization problem:  • 2 types of products are made by the factory• 2 types of products are made by the factory• 3 machines are needed to make each • 3 machines are needed to make each

productproduct  ConstraintsConstraints:: Machines are available for a Machines are available for a limited time during one production session:limited time during one production session: •• time on time on A A ≤≤ 50 50 hourshours • • time on time on B B ≤≤ 35 35 hourshours • • time on time on C C ≤≤ 80 hours 80 hours

To complete the product, each lot must be processed by all three machines for a certain number of hours  Required manufacturing time

Machines A B C 1

2

Time Constraint: 50 35 80

10 5 5

5 5 15

Profit•  Product 1 can be sold for $100 per lot• Product 2 can be sold for $80 per lot Optimization Problem: Design the production schedule which rnaximizes the profit of both products manufactured during one production session.

Formulation   x1 = number of lots of product 1

x2 = number of lots of product 2 

max f (x) = 100x1 + 80x2

10x1 + 5x2 ≤≤ 505x1 +   5x2 ≤≤ 355x1 +   15x2 ≤≤ 80

{

 Convert the problem to a standard LinearProgram (LP):

Max f(x) = 100x1 + 8Ox2

10x1 + 5x2+ s1 = 50

  s. to 5x1 + 5x2+ s2 = 35

5x1 + 15x2+ s3 = 80

Basis

New Matrix A

{

Geometric solutionGeometric solution

Family of curves, constant for Family of curves, constant for xx11,, xx22,,

F(x) = 100 xF(x) = 100 x1,1, ++ 80 x80 x22 = 800 = 800

(const)(const)

(a)(a)10x10x1,1, ++ 5x5x22 50  50 

(b)(b) 5x 5x1, 1, + 5x+ 5x22 35 35

(c)(c) 5x 5x1, 1, + 15x+ 15x22 80 80

a bxx11

xx22

15

10

5

15105 c

P2

P13

4

P0

F(x) = 800F(x) = 800

Basic Solution Given a system of equalities

Ax=b where x = n - vector, b = m - vector, A = m x n matrix, select a set of m linearly independent columns such that (m x m) matrix B is nonsingular, i.e. /B / 0. Then one may uniquely solve the equation 

Bxb= bwhere Xb is an m - element subvector of x.

By putting x=(xb,0) we obtain a solution to

Ax = b Such a solution (with n - m components of x not associated with columns of B m x m ) to the resulting set of equations is said to be a basic solution with regard to the basis B. The Xb variables, associated with columns of B, are called basic variables.

Basic Feasible SolutionBasic Feasible Solution

If a feasible solution (i.e. the one which satisfies all contraints) is also basic, it is called a basic feasible solution.

Fundamental Theorem of Linear Fundamental Theorem of Linear ProgrammingProgramming

Given an L.P. in standard formGiven an L.P. in standard form

min cmin cTTxx

Ax = bAx = b

x x 0 0

where where A A is is m m xx n n matrix of rank matrix of rank m m (i.e. (i.e. m < n m < n and and the the m m rows are linear independent),rows are linear independent),  

s. to

If there is a feasible solution, there is a If there is a feasible solution, there is a basic feasible solutionbasic feasible solution

If there is an optimal feasible solution, there If there is an optimal feasible solution, there is an optimal basic feasible solutionis an optimal basic feasible solution

This theorem shows that it is necessary only to This theorem shows that it is necessary only to consider consider basic feasible solutionsbasic feasible solutions when seeking an when seeking an optimal solution to a linear program, as the optimal solution to a linear program, as the optimal value is always achieved as such a optimal value is always achieved as such a solutionsolution

  

Ultimate goalUltimate goal::

find a basic feasible solution with a base find a basic feasible solution with a base B B composed of composed of originaloriginal variables only, and which is variables only, and which is optimum.optimum.

Solution: Solution: xx1 1 = 0, x= 0, x2 2 = 0, s= 0, s1 1 = 50, s= 50, s2 2 = 35, s= 35, s3 3 = 80,= 80,

is is a basic feasible solution (point a basic feasible solution (point PP00 (0,0) (0,0)))

Function value:Function value: F(0, 0. 50, 35, 80) = 0F(0, 0. 50, 35, 80) = 0

ƒ (x) = 100 xƒ (x) = 100 x1,1, ++ 80 x80 x22 + + 0 s0 s1 1 ++ 0 s0 s2 2 ++ 0 s0 s3 3

1010 5 5 100100

55 5 5 010010

55 1515 001001

xx11

xx22

ss11

ss2,2,

ss33

5 05 0

3 53 5

8 08 0

=

basis

Simplex MethodSimplex Method

Proceed from one basic feasible solution (extreme Proceed from one basic feasible solution (extreme point) to another, in such a way as to continually point) to another, in such a way as to continually decrease the value of the decrease the value of the f (x) f (x) until a minimum is until a minimum is reached. reached. 

General commentsGeneral comments

It is easy to find initial basic feasible solutions It is easy to find initial basic feasible solutions with with slack variablesslack variables..

Finding initial basic solution is part of the Simplex Finding initial basic solution is part of the Simplex method (Lue 84).method (Lue 84).

Simplex AlgorithmSimplex Algorithm

1.1. Select the column, such that the new resulting Select the column, such that the new resulting basic feasible solution will yield a basic feasible solution will yield a lower valuelower value to to f(x) f(x) than the previous one.than the previous one.

2.2. Select the pivot element in that column.Select the pivot element in that column.

By an elementary evaluation determineBy an elementary evaluation determine

• Which vector Which vector aajj should enter the basis (so that should enter the basis (so that f(x) f(x) is reduced), andis reduced), and

• Which vector should leave the basis. Which vector should leave the basis.

Construct a Simplex Tableau, [Construct a Simplex Tableau, [A, b A, b ] appended by ] appended by a row at the top with cost coefficients and current a row at the top with cost coefficients and current cost.cost.

j =j = 1 2 3 4 5 0

rrjj 100 80 0 0 0 0

yy1j1j 10 5 1 0 0 50

yy2j2j 5 5 0 1 0 35

yy3j3j 5 15 0 0 1 80

yi1 yi2 yi3 yi4 yi5 yi0

Point P0 (0,0)

F(0,0,50,35,80) = 0

Step 1Step 1.. Select Select columncolumn q q with with rrqq > 0. > 0.

((xxqq enters the basis)enters the basis)

q : rq : rqq = max r = max rii

Step 2.Step 2. Select Select pivot elementpivot element p p in column in column q q

Which minimizes ratio: Which minimizes ratio:

p :p : = min for positive = min for positive yyiqiq

yiq

yi0

ypq

yp0

yiq

yi0

i

Step 3Step 3.. PivotPivot on element on element yypqpq and update the and update the

tableau (update all rows, including the tableau (update all rows, including the top one)top one)

• In row In row pp divide divide yypjpj by by yypqpq , , j = 0, 1,…, nj = 0, 1,…, n

• Subtract from each row Subtract from each row i i p p

•• yypjpj , , j = 0, 1,…, nj = 0, 1,…, n yp q

yi q

Pivot on Pivot on yy1 11 1

j =j = 1 2 3 4 5 0

rrjj 100 80 0 0 0 0

yy1j1j 10 5 1 0 0 50

yy2j2j 5 5 0 1 0 35

yy3j3j 5 15 0 0 1 80

yi1 yi2 yi3 yi4 yi5 yi0

Update the tableUpdate the tablej =j = 1 2 3 4 5 0

rrjj 0 30 -10 0 0 -500

yy1j1j 1 0.5 0.1 0 0 5

yy2j2j 0 2.5 -0.5 1 0 10

yy3j3j 0 12.5 -0.5 0 1 55

yi1 yi2 yi3 yi4 yi5 yi0

Point P0 (0,0)F(0,0,50,35,80) = 0

Point P1 (5,0)F(5,0,0,10,55) = 500

Pivot on Pivot on yy2 22 2

Update the tableUpdate the tablej =j = 1 2 3 4 5 0

rrjj 0 0 -4 -12 0 -620

yy1j1j 1 0 0.2 -0.2 0 3

yy2j2j 0 1 -0.2 0.4 0 4

yy3j3j 0 0 2 -5 1 5

yi1 yi2 yi3 yi4 yi5 yi0

Point P1 (3,4)F(3,4,0,0,5) = 620

j =j = 1 2 3 4 5 0

rrjj 0 30 -10 0 0 -500

yy1j1j 1 0.5 1 0 0 5

yy2j2j 0 2.5 -0.5 1 0 10

yy3j3j 0 12.5 -0.5 0 1 55

yi1 yi2 yi3 yi4 yi5 yi0

Point P1 (5,0)F(5,0,0,10,55) = 500

LP ModificationsLP Modifications

1. Maximization1. Maximization

2. Unconstrained x2. Unconstrained x

max c max c t t x = min (-c x = min (-c t t x)x)

let x = xlet x = x++ - x - x -- ; x; x++, x , x -- 0) 0)

min (c min (c t t xx++ - c - c t t x x --))

[A, -A] = bxx++

x x --

xx++, x , x -- 0 0

s. tos. to

3. Constraint set:3. Constraint set: Ax Ax b b

Slack variablesSlack variables,, xxss : A : Axx + x + xss = b = b

[A, I ] = bxx

xxss

x , xx , xs s 0 0

min [cT , 0]

xx

xxss

Columns corresponding to Columns corresponding to xxss form a basic form a basic solutionsolution

4. Constraint set:4. Constraint set: Ax Ax b b

Surplus variablesSurplus variables,, xxpp : A : Ax x - x - xpp = b = b

[A, -I ] = bxx

xxpp

x , xx , xp p 0 0

min [cT , 0]

xx

xxpp

Artificial variablesArtificial variables,, xxaa : A: Axx - x - xpp + x + xaa = b = b

(cannot star with negative basis of(cannot star with negative basis of xxp p ))

[A, -I , I ] = b

x , xx , xpp , , xxaa 0 0

min [cT , 0 , k I ]

xxxxpp

xxaa

xxxxpp

xxaaS. to

Columns corresponding to Columns corresponding to xxaa form a basic form a basic solutionsolution

Integer ProgrammingInteger Programming

Integer ProgramInteger Program

max cmax cTTxx

Ax Ax b b

xxii integer integer

Typically, Typically, xxii = 0,1 = 0,1

(0,1 Integer Programming)(0,1 Integer Programming)

s. to

IP ExampleIP Example: : matching problemmatching problem

Given a graph Given a graph G G , find maximal set of edges in , find maximal set of edges in G, G, such that no two edges are adjacent to the same such that no two edges are adjacent to the same vertex.vertex.• Maximum matching: matching of maximum Maximum matching: matching of maximum

cardinality.cardinality.

• Weighted matching: matching with Weighted matching: matching with

)(max iew

1 3

2e1 e2

e3

otherwise

matchinginincludedisif

0

e1x i

i

1 3

2e1 e2

e3

321 xxx max

10x

1xx

1xx

1xx

tos

i

32

31

21

,

.

In matrix form:In matrix form:

10x

bAxtos

i ,.

xcTmax

110

101

011

e

e

e

e.g.

3

2

1

A

Where Where b = b = [ 1,…1][ 1,…1]T T , , A = A = incidence matrix of incidence matrix of GG

1 2 3

max max 1x1x11 + + 1x1x22 + + 1x1x33

1

1

1

110

101

011

s.to

3

2

1

x

x

x

One solution to this IP problem:One solution to this IP problem:

xx11 , , xx22 , , xx33 = 0, 1 = 0, 1

xx11 = 1 , x= 1 , x22 = 0 , x= 0 , x33 = 0= 0

Other possible solutions:Other possible solutions:

xx11 = 0 , x= 0 , x22 = 1 , x= 1 , x33 = 0, or x= 0, or x11 = 0 , x= 0 , x22 = 0 , x= 0 , x33 = = 1 1

QuestionQuestion

Can the solution to this IP problem be obtained Can the solution to this IP problem be obtained by dropping the integrality constraint:by dropping the integrality constraint:

xxii = 0, 1 = 0, 1

And solving the LP problem instead?And solving the LP problem instead?

In our example, solution to the IP is In our example, solution to the IP is notnot obtainable from LP.obtainable from LP.

ReasonReason:: matrix matrix A A does not have certain property does not have certain property (total unimodularity) needed to guarantee integer (total unimodularity) needed to guarantee integer solutions.solutions.

If If xxii = 0, 1 = 0, 1 is relaxed, such that is relaxed, such that xxii 0 , 0 , then then the solution to the associated LP problem is non – the solution to the associated LP problem is non – integer:integer:

2

1xxx 321

Reason: the Reason: the A A matrix matrix

110

101

011

Is not Is not totally unimodulartotally unimodular: |: |A| = - 2A| = - 2

Unimodularity: LP -> IPUnimodularity: LP -> IP

Given a constraint set in standard formGiven a constraint set in standard form

where where A, b A, b are integerare integer

Partition Partition A = A = [[B/N B/N ]; ]; x = x = [[xxBB, x, xN N ]]

B B is nonsingular is nonsingular m m xx m m basis, basis, NN is non-basic is non-basic

0x

bAx

i

bNxBxx

xNBAx NB

N

B

/

Basic solution is Basic solution is

bBx

0x1

B

N

In particular, when In particular, when B = I B = I and and B B -1-1 = = I thenI then

XXBB = = b b

a solution can be obtained by inspection (as in a solution can be obtained by inspection (as in the initial step of Simplex method).the initial step of Simplex method).

Since Since xxBB = B = B –1 –1 bb with with xxNN = 0, b = 0, b integerinteger

A sufficient condition for a basic solution A sufficient condition for a basic solution xxBB to be to be integer is thatinteger is that

B B –1 –1 be an integer matrixbe an integer matrix

UnimodularityUnimodularity

A square matrix A square matrix B B is called is called unimodularunimodular if if

D = |det B| = 1D = |det B| = 1

An integer matrix An integer matrix A A is is totally unimodulartotally unimodular if every if every square, nonsingular submatrix of square, nonsingular submatrix of A A is unimodular.is unimodular.

Equivalently: Equivalently: A A is totally unimodular if every is totally unimodular if every subdeterminant of subdeterminant of A A is 0, +1, or –1.is 0, +1, or –1.

Recall that for Recall that for B B nonsingular: nonsingular: Bted

BB 1

Where Where B B ++ , , adjoint matrix adjoint matrix

= [ = [ i j i j cofactor of element cofactor of element a a i ji j in in det det A A ]]TT

B B ++ and and det B det B are integer if are integer if BB –1 –1 is integer is integer

333231

232221

131211

bbb

bbb

bbb

B

T

211222111321231122132312

311232113113331113323312

312232213123332132233322

bbbbbbbbbbbb

bbbbbbbbbbbb

bbbbbbbbbbbb

B

)()()(

)()()(

)()()(

Cofactor of Cofactor of aai j i j :: Determinant obtained by Determinant obtained by omitting the iomitting the ithth row and the j row and the jthth column of column of AA and and then multiplying by (-1) then multiplying by (-1) i + ji + j . .

For For B B unimodularunimodular, , B B –1–1 integerinteger::

If If A A is is totally unimodulartotally unimodular, every basis matrix , every basis matrix BB is is unimodular and every basic solutionunimodular and every basic solution

( x( xBB , , yyNN) = ( B ) = ( B –1–1 b, 0 ) b, 0 )

Is Is integerinteger..

In particular, the optimal solution is integerIn particular, the optimal solution is integer

Theorem 1Theorem 1::

If If A A is totally unimodular then every basic is totally unimodular then every basic solution of solution of Ax = b Ax = b is is integerinteger..

For LP’s with equality constraints total For LP’s with equality constraints total unimodularity is sufficient but not necessary.unimodularity is sufficient but not necessary.

For LP with inequality constraints, For LP with inequality constraints, Ax Ax b b, , total unimodularity of total unimodularity of AA is both is both necessary necessary and sufficient and sufficient for all extreme points of for all extreme points of s = {x : Ax s = {x : Ax b , x b , x 0 } 0 } to be integer for to be integer for every integer vector every integer vector b.b.

ConclusionConclusion::

Any IP with totally unimodular constraint Any IP with totally unimodular constraint matrix can be solved as an LP.matrix can be solved as an LP.

Totally unimodular matrix : Totally unimodular matrix : a a i j i j = 0 , +1, -1 = 0 , +1, -1

Also, every determinant of Also, every determinant of A A must be must be 0, +1 or 0, +1 or -1-1

Special Linear Programs with Integer Special Linear Programs with Integer SolutionsSolutions

are LP with the property that they possess are LP with the property that they possess optimal solutions in integers.optimal solutions in integers.

• max flowmax flow

• min – cost flowmin – cost flow

• assignment problemassignment problem

• shortest pathshortest path

• transportation problemtransportation problem

Network flow problems:Network flow problems:

Some Totally Unimodular Linear ProgramsSome Totally Unimodular Linear Programs

Assignment problemAssignment problem(special case of min – cost capacitated flow (special case of min – cost capacitated flow problems)problems)

[ [ m m jobs x jobs x m m men ]men ]

otherwise

jobtoassignedismanif

0

ji1x ji

c i j = cost of assigning man i to job j

ijij

ij xcMax

s. to.s. to.

10,

)1(1,1

)1(1,1

1

1

ij

ij

m

j

ij

m

i

x

manperjobmix

jobpermanmjx

1

1

1

1

1

1

X

111

1

1

1

111

1

1

1

111

1

1

1

X11 X12 X13 X21 X22 X23 X31 X32 X33

xcTmaxIn matrix notation:In matrix notation:

Ax = 1 Ax = 1 wherewhere

11

0

1

1

0

11

0

1

1

0

11

1

1

A

m 2m m 2...

Exactly Exactly m m 1’s in each row1’s in each row

Exactly 2 1’s in each columnExactly 2 1’s in each column

Network Flow ProblemsNetwork Flow Problems::

• a a set of originsset of origins VV11 ; each origin i ; each origin i V V11 ; ; supplies asupplies a11 of of

commodity.commodity.

• a a set of destinationsset of destinations VV22 ; ; each destinationeach destination j j V V22 ; ; has a has a

demand demand bbii of commodity.of commodity.

• costcost per unit commodity; per unit commodity; ccijij associated with sending associated with sending

commodity through (commodity through (i, j i, j ).).

Given:Given:

Constraint SetConstraint Set: : (Totally unimodular)(Totally unimodular)

jiji

2i

1

1i

iΓij

ji

iΓji

ij

cx0

iΓVib

Vi0

iΓVia

xx

))((for

for

))((for

)(),()(),(

nodendestinatiofor

nodesernalintfor

nodesourcefor

)(),()(),( f

0

f

xxiΓij

ji

iΓji

ij

Special case: for single source, single Special case: for single source, single destination, max flow problemdestination, max flow problem

in any case:in any case: Ax Ax b b

No matter which problem it No matter which problem it isis

cutmin

flowmax

flowtcosmin

the general format is the general format is A’x A’x b b

It can be shown thatIt can be shown that• If If A A is totally unimodular then [is totally unimodular then [A/I A/I ] is also totally ] is also totally

unimodularunimodular• The transpose of totally unimodular matrix is also The transpose of totally unimodular matrix is also

totally unimodulartotally unimodular

• ..

I

AA'

where where A = A = incidence matrix of the corresponding incidence matrix of the corresponding digraph, totally unimodular. digraph, totally unimodular. I = I = identity matrix, identity matrix, A’ A’ is is totally unimodulartotally unimodular..

• Exploit a Exploit a special structurespecial structure of the problem of the problem (total unimodularity, etc.) to obtain integer (total unimodularity, etc.) to obtain integer solutions by solving simpler problems.solutions by solving simpler problems.

• TransformTransform the problem to another problem the problem to another problem for which an approximate solution is easier for which an approximate solution is easier to find.to find.

• Once the structure of the problem is well Once the structure of the problem is well understood, understood, use heuristicuse heuristic, but… stay away , but… stay away from brute force approach.from brute force approach.

No efficient algorithms exist for general Integer No efficient algorithms exist for general Integer Programming problemsProgramming problems

• Graph TheoryGraph Theory

• Mathematical OptimizationMathematical Optimization

A large number of CAD problems can be cast in A large number of CAD problems can be cast in analytical form:analytical form:

For some problems – efficient algorithms exist.For some problems – efficient algorithms exist.

For others – need to resort to For others – need to resort to heuristicheuristic, , suboptimal solutions.suboptimal solutions.

Some known successful heuristic approachesSome known successful heuristic approaches

• Simulated annealingSimulated annealing• LP roundingLP rounding

ConclusionsConclusions