111
CONSTRAINED OPTIMIZATION 1. EQUALITY CONSTRAINTS
Consider the problem (P1): Minimize f(x)
st hj(x) = 0, j=1,2,…,m
x Rn
Let us first examine the case where m=1 (i.e., a single constraint). Without this
constraint the necessary condition for optimality was f(x)=0. With the
constraint h(x)=0, we also require that x lie on the graph of the (nonlinear)
equation h(x)=0.
-f(x*) steepest descent direction, is
orthogonal to the tangent of the contour of f
through the point x*.
It must also be orthogonal to
the tangent of the curve
h(x)=0 at x*!
But we know that h(x*) is also orthogonal to the curve h(x)=0 at x*. Therefore...
-f(x*)
f(x*)
h(x)=0
x*
Contours of f(x)
Unconstrained Minimum
112
-f(x*) and h(x*) MUST BOTH LIE ALONG THE SAME LINE, i.e., for some
R, we must have
-f(x*) = h(x*)
So, if x* is a minimizer, the necessary condition reduces to
f(x*) + h(x*) = 0
We now proceed to the general case where the above necessary condition should
hold for each constraint.
DEFINITION: The Lagrangian function for Problem P1 is defined as
L(x,) = f(x) + j=1…,m j hj(x)
The KARUSH-KUHN-TUCKER Conditions
If the point x*Rn is a minimizer for problem P1, it must satisfy the following
necessary condition for some *Rm:
∗ ∗ 0 and ∗ 0∀
or equivalently,
x,L(x*, *) = 0.
113
EXAMPLE:
Minimize ½ x12 + ½ x2
2 –x1x2 - 3x2
st x1 + x2 = 3 ( h(x) = x1 + x2 - 3 = 0)
The Lagrangian is
L(x,) = (½ x12 + ½ x2
2 –x1x2 - 3x2) + ( x1 + x2 - 3)
The K-K-T conditions yield
0
3 0
3 0 originalconstraints
Solving these (in this particular case, linear) equations:
x1*= 0.75, x2
*= 2.25, *= 1.5, with f(x*) = -5.625
The above point thus satisfies the necessary conditions for it to be a minimum.
SUFFICIENT CONDITIONS: IF in addition to the K-K-T necessary conditions,
suppose that for arbitrary zRn with z0, the following statement is true:
“zT hj(x*) = 0, j=1,2,…,m implies that zTx
2L(x*,*)z > 0”
THEN f has a strict local minimum at x*, st hj(x*)=0, j=1,2,…,m.
114
For our example,
x2L(x*,*) = 1 1
1 1 and h(x*) = 1
1
Consider zR2 = 00.
Then zT hj(x*) = z1+z2 = 0 (i.e., z2 = -z1) implies that
zT x2L(x*,*) z = 1 1
1 1 = (z1 - z2)
2 = (2z1)2 > 0
x* is a strict local minimum.
As in unconstrained optimization, in practice, sufficient conditions become quite
complicated to verify, and most algorithms only look for points satisfying the
necessary conditions.
2. INEQUALITY-CONSTRAINED OPTIMIZATION
Consider the problem (P2): Minimize f(x)
st gj(x) 0, j=1,2,…,m
x Rn.
Again, let us first consider the case with a single inequality constraint (i.e., m=1).
115
Q. Can this be converted to an equivalent equality constrained problem?
1. Suppose that (as in LP) we write g(x)0 in the form g(x) + S =0, where S is
a nonnegative slack variable. Then we have added the constraint S0;
effectively, we have not accomplished anything (we just trade one
inequality for another)!
2. Instead, let us write g(x) + S2 = 0
(S20 is of course unnecessary…)
The Lagrangian is
L(x,S,) = f(x) + (g(x)+S2)
and the K-K-T optimality conditions become
∗ ∗ ∗ 0, 1,2, … ,
∗ ∗ 0 originalconstraints
2 ∗ ∗ 0 COMPLEMENTARYSLACKNESS
EXAMPLE: Minimize x12 + ½ x2
2 - 8x1 - 2x2 - 60
st 40x1 + 20x2 140 = 0
Converting to an equality constraint, we have
40x1 + 20x2 + S2 – 140 = 0
L(x1, x2, S, ) = (x12 + ½ x2
2 - 8x1 - 2x2 – 60) + (40x1 + 20x2 + S2 – 140)
116
The necessary conditions are therefore
2 8 40 0, 2 20 0
40 20 – 140 0 originalconstraints
2 0
Here we have nonlinear system of equations. Again note that these are only
necessary conditions, and there may be one or more points that satisfy these. We
seek an optimal solution to the original problem that is among the solution(s) to
the necessary conditions.
Rather than attempt a direct solution, let us use the complementary slackness
condition to investigate two cases: (i) =0, (ii) S=0.
Case (i) =0 2x1 -8=0 x1 =4
x2 -2=0 x2 =2
40x1 + 20x2 + S2 – 140 = 0 S2 = -60 INFEASIBLE!
Case (ii) S=0 2x1 -8+40=0 x1 =4
x2 -2+20=0 x2 =2
40x1 + 20x2– 140 = 0 x1 =3, x2 =1, =0.05
the only solution to the necessary conditions is the point x=(3,1). Thus, if the
original problem had a solution, then this must be it!
117
Geometric Interpretation
Consider Minimize f(x), st g(x)0.
Just as in equality constrained optimization, here too we have one of the
necessary conditions as
-f(x*) = .g(x*) for some value of
With equality constraints, recall that was just some real number with no sign
restrictions
(>0) (<0)
With inequality constraints we can “predict” the correct sign for
g(x)0, >0 g(x)0, <0
Feasible Region
Feasible Region
Thus for minimization, g(x)0 0; and g(x)0 0.
h
-f
h
h(x)=0 h(x)=0
OR
-f
g
g(x)<0
OR
-f
g
g(x)>0
g(x)=0 g(x)=0
dir. of increase of g
dir. of increase of g
-f
118
In practice, we dispense with the slack variable S and state the NECESSARY
CONDITIONS in a standard form that does not use the slack variable.
Let L(x,) = f(x) + g(x)
Then the necessary conditions are:
1 ∗, ∗ ∗ ∗ ∗ 0,∀
i.e., xL(x*,*) = f(x*) + *g(x*) = 0
(2) g(x*) 0
(3) *g(x*) 0 *0
Condition 2 yields the original constraints.
Condition 3 is complementary slackness: (g(x*)<0 * = 0; * >0 g(x*)=0).
For the general case with m inequality constraints (i.e., Problem P2) the K-K-T
necessary conditions are that if x* is a minimizer then j* 0 for j=1,2, ... ,m
such that for L(x,)=f(x) + j=1…m jgj(x)
1 ∗, ∗ ∗ ∑ ∗ ∗ 0 equations)
(2) gj(x*) 0, j=1,2,…, m
(3) {gj(x*)}j
* 0, j=1,2,…, m
NOTE: If a constraint were of the form g(x)0, we would require that j*0.
Alternatively, we could redefine gj(x)0 as -g(x)0 and retain j*0!
119
We now look at the general NLP problem (P3)
Minimize f(x)
st gj(x) 0, j=1,2,…,m,
gj(x) 0, j=m+1,…,p,
gj(x) 0, j=p+1,…,q,
xRn.
,
Karush-Kuhn-Tucker Necessary Conditions
If x* minimizes f(x) while satisfying the above constraints, then provided certain
regularity conditions are met, there exist vectors * and * such that
1 ∗ ∗ ∗ ∗ ∗ ∗ ∗
(2) All original constraints are satisfied
(3) j*{gj(x
*)} 0 j=1,2,…,p
(4) j*0 j=1,2,…,p
(5) j* are unrestricted in sign for j=p+1,…, q.
The regularity conditions referred to above are also known as constraint
qualifications. We will address these later…
120
NON-NEGATIVITY: Many engineering problems require the decision variables
to be non-negative. While each non-negativity requirement could be treated as a
constraint, we don't have to...
Consider the following problem (for simplicity, assume only constraints….):
Minimize f(x)
st gj(x) 0; j=1,2,…,m
qi(x) = xi 0; i=1,2,…,n.
Let us define Lagrange multipliers 1, 2,..., n corresponding to the non-negativity
constraints. Then in the K-K-T conditions we have via complementary slackness
i*x i
* = 0; i=1,2,…,n
and Condition (1) becomes
∗ ∗ ∗ ∗ ∗
⟹ ∗ ∗ ∗ ∗
100::00
∗
010::00
… ∗
000::10
∗
000::01
⟹ ∗ ∑ ∗ ∗ ∗, where * Rn = [1* 2
* … n*]T
Note that we want ∗
121
Thus the necessary conditions become
1 ∗ ∑ ∗ ∗ (n conditions)
2 Alloriginalconstraintsaresatisfied
3 j* gj x* 0 j 1,2,…,m
4 j*0 j 1,2,…,m
5 ∗ ∗ 0∀ , i. e., ∑ ∗ ∗ ∗ ∙ ∗ ∑ ∗ ∗ 0
NOTE THAT NO EXPLICIT MULTIPLIERS ARE USED FOR THE NONNEGATIVITY
CONSTRAINTS!
The Karush-Kuhn-Tucker conditions are the most commonly used ones to check
for optimality. However, they are actually not valid under all conditions.
There do exist a somewhat weaker set of necessary conditions (known as the Fritz
John conditions) that are valid at all times; however in many instances they do not
provide the same information as the Karush-Kuhn-Tucker conditions.
122
FRITZ JOHN CONDITIONS
These weaker necessary conditions for the problem
Minimize f(x)
st gj(x) 0, j=1,2,…,m
x Rn.
are based on the so-called weak Lagrangian
,
and may be stated as follows:
If x* is a minimizer then Rm+1 such that
1 ∗, ∗ ∗ ∗ ∗ ∗
(2) gj(x*) 0, j=1,2,…,m
(3) j*{ gj(x
*)} 0, j=1,2,…,m
(4) *0 and *0.
While the Fritz John conditions are always necessary for x* to be a solution, the
Karush-Kuhn-Tucker conditions are necessary provided certain conditions known
as CONSTRAINT QUALIFICATIONS are met. In effect:
Local Optimum Fritz John Karush-Kuhn-Tucker Constraint
Qualification
123
An example of a constraint qualification is:
The set S = {gj(x*) | jJ} is linearly independent, where J = {j|gj(x
*)=0}.
The above C.Q. states that the gradient vectors of all the constraints that are
binding (satisfied as strict equalities) at x* should be linearly independent.
In the Fritz John conditions, suppose that = 0. Then the conditions are satisfied
for any function f at the point x*, regardless of whether or not that function has a
minimum at x*!
This is the main weakness of the Fritz John conditions. If = 0, then they make
no use of the objective and they are of no practical use in locating an optimal
point.
Constraint qualifications essentially are constraints that ensure that >0, so that
if we redefine j by (j
/) for j=0,1,...,m, we can reduce the Fritz John
conditions to the Karush-Kuhn-Tucker conditions ( =1 now and may thus be
ignored).
124
EXAMPLE: Minimize –x1, st g1(x)=x2-(1-x1)30; g2(x)= -x20.
The optimum solution is at x*= 10
, where we also have
f(x*) = 10
; g1(x*) = 0
1; g2(x
*) = 01
Note that the constraint qualification is not met since both constraints are active at
x*, but the gradient vectors of these constraints are not linearly independent.
FRITZ JOHN CONDITIONS: 1
0 +
01+
01
= 00
Satisfied for 0=0, and = = arbitrary > 0.
KARUSH-KUHN-TUCKER CONDITIONS: -f(x*)= 10
= 01+
01
1(0) + 2(0) = 1 and 1(1) + 2(-1) = 0
Inconsistent! (cannot be satisfied)
‐2
0
2
4
6
8
‐1 ‐0.5 0 0.5 1 1.5 2 2.5
x2
x1
g1(x*)
g2(x*)
g1(x)=0
g2(x)=0 f(x*)
Feasible Region
125
CONVEX PROGRAMS
The following NLP (Program P4) is referred to as a convex program:
Minimize f(x)
st gj(x) 0, j=1,2,…,m,
gj(x) 0, j=m+1,…,p,
hj(x) 0, j=1,2,…,q,
x Rn.
where f is a proper convex function on Rn
gj are proper convex functions on Rn, j=1,...,m
gj are proper concave functions on Rn, j=m+1,...,p
hj are linear functions of the form
(Note: A proper function has at least one point in its domain that is <+ (i.e., its
effective domain is non-empty), and it is also >- everywhere in its domain.)
Thus a convex minimization program has a convex objective, and the set of
feasible solutions is a convex set.
EXERCISE: Show that the set of feasible points determined by the constraints of
program P4 is convex.
126
Q. Why are convex programs important?
A. Because they are 'well-behaved' programs with nice results!
Slater’s Condition: For convex programs, if x'Rn
gj(x') < 0 for j=1,2,...,m, and
gj(x') > 0 for j=m+1,...,p,
then the constraint qualification holds at x' (the relative interior is non-empty)
Furthermore, for a convex program where the constraint qualification holds, the
K-K-T necessary conditions are also sufficient.
Many algorithms have been designed specifically for convex programming
problems. A special case is (posynomial) geometric programming, where the
constrained local optima also turn out to be global optima.
127
LAGRANGIAN DUALITY
Just like LP, NLP also has a duality theory. For the problem
Min f(x), st gj(x)0, j=1,2,…,m; hj(x)0, j=1,2,…,p; xS
this is based upon the Lagrangian function L(x,) = f(x) + jjgj(x) + jjhj(x).
Primal Dual
MINIMAX
Minimize , st xS
where Max , , ,
if 0∀ and 0∀j∞if 0or 0forsome
and so
Min∈
Min∈
Max,
, ,
Min∈
| 0, 0∀
This is our original (Primal) problem!
MAXIMIN
Maximize , , st ,
where , Min , ,
In particular, under conditions of
convexity & differentiability this
reduces to
Max L(x,)
st , ,
0, unr xS,
WEAK DUALITY THEOREM: This states that
, , , for all 0 and xS
Max, {Min x L(x,,)} Min x {Max, L(x,,)}
The quantity , is referred to as the “duality gap.”
128
GEOMETRIC INTERPRETATION OF DUALITY
Consider the problem: Minimize f(x), st g(x) 0, xS.
Let z1 =g(x) and z2 =f(x)
G is the image of the set S under the map z1=g(x), z2=f(x). Thus the original
problem is equivalent to
Minimize z2 , st z1 0, z=(z1, z2) G
which yields the optimum solution shown above.
For 0 given, Min ∈ , Min ∈
= Min ∈
Thus the dual problem is to find the slope of a supporting (tangential) line (or
hyperplane) for which the intercept with the z2 axis is maximized.
x
z1
z2
G S
,
slope =
z2+z1 =
(0,0)
z=[g(x) f(x)]T
129
EXAMPLE 1: Minimize 4x12 + 2x1x2 + x2
2
st 3x1 + x2 6; x1, x2 0
Let S ={x | x1 0, x2 0}. The Lagrangian is
L(x,) = 4x12 + 2x1x2 + x2
2 + (-3x1 – x2 + 6)
The PRIMAL problem is
Minimize∈
Maximize , Min∈
i.e., Min 4x12 + 2x1x2 + x2
2
st 3x1 + x2 6; x1, x2 0
The corresponding DUAL problem is
Maximize Minimize∈
, Max
Consider the dual objective
Minimize∈
, 4 2 3 6
For this particular example, L(x,) (the objective function) is convex; therefore to
get the minimum the necessary and sufficient conditions are that
, 8 2 3 0
, 2 2 0i. e., ∗
3, ∗
6
Note that if 0, this satisfies xS.
130
Thus we have = L[(/3),(/6),] = 6 - (7/12)
The dual problem thus reduces to
Maximize 6 - (7/12)
st 0
The objective function for this subproblem turns out to be concave (CHECK and
VERIFY for yourself):
∴ 6712
2 0 ⟹ 367 0
Since 36/7 is not < 0, it solves the problem, i.e.,
DUAL SOLUTION: ∗ , ∗
PRIMAL SOLUTION: ∗ ∗
0; ∗ ∗
0;
and 3x1* + x2
* = (36/7) + (6/7) = 6 ---- feasible.
Complementary slackness is satisfied since *(6-3x1* + x2
*) = 0
(12/7, 6/7) is optimal for the Primal
Furthermore
4127
2127
67
67
1087
⟹ !
131
EXAMPLE 2: Min (-x2 –x3 )
(P) st x2 1
The Lagrangian for this problem is L(x,) = -x2 - x3 + (x2-1)
Consider the PRIMAL problem; optimality conditions are:
xL(x, ) = 0 -2x - 3x2 + 2x = 0
(x2-1)=0, 0; x2 1.
(a) =0 x = (-2/3) OR x = 0;
(b) >0 x2-1= 0 x = 1
x=+1 =2.5 x=-1 = -0.5 (infeasible…)
Therefore the solutions are
(x,) = (-2/3,0) or (0,0) or (1,2.5)
f(x) = -4/27 or 0 or -2
Thus the primal optimum is x* =1, f(x*) = -2.
‐6
‐5
‐4
‐3
‐2
‐1
0
1
2
‐2 ‐1.5 ‐1 ‐0.5 0 0.5 1 1.5 2
f(x)=-x2-x3
x
x*=1
f(x*)=-2
‐(2/3)
132
Let us look at the dual (D) and see if there is any duality gap. The dual is:
(D) Maximize ; st 0
where
Min , Min 1
= - for all values of !
∴ Max ∞
Thus the primal objective = -2; dual objective = -. There is a DUALITY GAP!
Looking at it geometrically, let z1=g(x)=x2–1, and z2 =f(x)= –x2 –x3.
The supporting hyperplane is the plane (line) through (-1,0) that intersects the z2-
axis at -.
‐10
‐8
‐6
‐4
‐2
0
2
4
6
‐2 ‐1 0 1 2 3 4
z2
z1
optimum primal value
G
133
Saddlepoint Sufficiency
Recall that from Weak Duality )()()(ˆ xλx,λ LLL .
Definition: The point )( λ,x is a saddlepoint of L(x,) if
1. SLL xλx,λ,x )()( , i.e., ( ) ( ) ( )SˆL Min L L xx,λ x,λ λ , and
2. ( ) ( )L L x,λ x,λ λ 0 , i.e., ( ) ( , ) ( )L Max L L x,λ x x
In other words, if )( λ,x is a saddlepoint of L(x,) then )(ˆ)()( λλ,xx LLL ,
i.e., primal objective = dual objective at the point and the duality gap = 0.
Now consider the following Primal problem:
Min f(x), st gj(x) 0, j=1,2,…,m; xS .
where f(x) and gj(x) are all convex functions, and S is a convex set.
Saddlepoint Sufficiency Condition: If 0and λx S , then )( λ,x is a
saddlepoint of L(x,) if, and only if,
x minimizes )()()( xgλxλx, TfL over S.
gj(x) 0 for each j=1,2,…,m
0)( xλ jj g , which implies that )()( λ,xx Lf
If )( λ,x is a saddlepoint of L(x,) then x solves the Primal problem above and
λ solves the Dual problem: Maximize )(ˆ λL st 0, where )()(ˆ λx,λ x LMinL S
134
Strong Duality Theorem
Consider the primal problem: Find
where SRn is nonempty and convex, f(x) and gj(x) are convex & hj(x) are linear.
Note: infimum may be replaced by minimum if the minimum is attained at some x
Define the dual problem: Find
where ( , ) ( ) ( ) ( )T TSL inf f g h xλ μ x λ x μ x .
Strong Duality Theorem: Assume that the following Constraint Qualification
holds:
where h(S) = h(x): xS.
Then = , i.e. there is no duality gap. Furthermore, if >-, then
= ),(ˆ ** μλL for some *0
If x*solves the primal, it satisfies complementary slackness: j*gj(x
*)=0 j.
1
2
inf ( )
st ( ) 0, 1,2,...,
( ) 0, 1,2,...,
j
j
f
g j m
h j m
x
x
x
x S
0λ
μλ
st
L ),(ˆsup
)(int0&
,...,2,1,0)ˆ(
,...,2,1,0)ˆ(
thatsuchˆexistsThere
2
1
Sh
x
x
x
mjh
mjg
j
j
135
QUADRATIC PROGRAMMING
Quadratic Programs are convex minimization programs that have a convex
(quadratic) objective and a convex constraint set formed by linear constraints. The
QP primal problem is:
(P) Minimize ½ xTQx + cTx
st Ax b ( b-Ax 0)
(any nonnegativity restrictions are included in Axb)
where
xRn, Q is a (nn), real, symmetric, positive definite matrix
cRn, A is a (mn) real matrix, bRm.
Thus we have a convex objective & linear constraints.
The Lagrangian function is
,12
The dual QP problem is
Max Min , ,i. e. ,Max , , st ,
This yields
(D) Maximize ,
st xL(x,) = Qx + c -AT = 0; 0.
136
The dual constraints imply that xT[Qx+ c –AT] = xT[0]
i.e., xTQx+ xTc – xT(AT) = xTQx+ xTc – TA x = 0 ()
The objective function can be rearranged as
Tb – ½ xTQx + (xTQx + xTc – TA x) = Tb – ½ xTQx from ()
Thus the dual reduces to
Maximize Tb – ½ xTQx
st Qx + c – AT = 0; 0.
If Q=[0], then (P) is Min cTx, st Axb
(D) is Max b, st AT = c
the regular LP primal-dual pair!
For Q [0]
(P) has n variables, m linear inequality constraints
(D) has (m+n) variables, n equality constraints, and m nonnegativity constraints
If Q is nonsingular (i.e., Q-1 exists) then from the dual constraints,
Qx + c – AT = 0 x + Q-1c - Q-1AT = 0
i.e., x = Q-1[AT - c]; thus we may eliminate x altogether from the dual
problem!
137
First, recall that given any two matrices U and V, (i) [UV]T=VTUT, (ii) [UT]T = U
and (iii) UTV=VTU (assuming compatibility). Also, (iv) Q, Q-1 are symmetric and
identical to their transposes.
Substituting this value of x into the dual objective function and rearranging:
Tb – ½ xTQx = Tb – ½ [Q-1(AT – c)]T Q [Q-1(AT – c)]
= bT – ½ (AT – c)T Q-1 Q Q-1 (AT – c) (i) and (iv)
= bT – ½ [(AT)T – cT] Q-1 (AT – c)
= bT – ½ [TAQ-1 – cT Q-1] (AT – c) (i) and (ii)
= bT – ½ [TAQ-1AT + cTQ-1c – cT Q-1 AT – TAQ-1c]
= bT – ½ [TAQ-1AT + cTQ-1c – cT Q-1 AT – (AQ-1c)T] (iii)
= bT – ½ [TAQ-1AT + cTQ-1c – cT Q-1 AT – cT(Q-1)TAT] (i)
= bT – ½ [TAQ-1AT + cTQ-1c – 2cT Q-1 AT] (iv)
= [bT + cT Q-1AT] – ½ [TAQ-1AT] – ½ [cTQ-1c]
= eT – ½ [TD] – ½ [cTQ-1c]
where
e = b + AQ-1c and D= -AQ-1AT
so that the dual is (as long as Q is nonsingular)
Maximize 12
12
138
EXAMPLE: Minimize ½ x12 + ½ x2
2 - 2x1 - 2x2
st 0 x1 1 1
0 ; 0 x2 1 1
0
i. e. ,Minimize12
1 00 1
2 2
xT Q x cT x
st
1 00 11 00 1
1100
A x b
In order to define the dual we have Q-1 (=Q) = 1 00 1
⟹
1100
1 00 11 00 1
1 00 1
22
1122
D= -AQ-1AT =
1 00 11 00 1
1 00 1
1 0 1 00 1 0 1
1 0 1 00 1 0 11 0 1 00 1 0 1
Therefore the dual is
Maximize 12
12
with the e and D as given above. The optimal primal solution x* = Q-1[AT*-c]
= 1 00 1
1 0 1 00 1 0 1
22
139
AN ALGORITHM FOR QUADRATIC PROGRAMMING
PROBLEMS: The Method of HILDETH AND D'ESPO
This is a "cyclic co-ordinate search" method applied to the QP dual:
Maximize 12
STEP 1: Start with an initial (e.g. =0)
STEP 2: For j=1,2,...,m do the following
Search for the maximum in the direction parallel to the j-axis by fixing k
for k≠j and solving 0 for j. If this j<0, then fix j=0.
STEP 3: If from this iteration the same as the from the previous iteration
STOP else go to STEP 2.
NOTE: In order to obtain we need
Consider the previous example where the dual was defined as:
Maximize 12
12
with
22; 1 0
0 1;
1122
;
1 0 1 00 1 0 11 0 1 00 1 0 1
140
Substituting the appropriate values of D, c, Q and e yields
Maximize 11 2 212
1 0 1 00 1 0 11 0 1 00 1 0 1
12
2 2 1 00 1
22
Maximize 2 212
2 2 4
The K-K-T conditions for this yield
1(1-1+3)=0
2(1-2+4)=0 Need to solve this system to obtain the optimal
3(-2-3+1)=0
4(-2-4+2)=0
Instead, let us try the method of Hildeth and D'Espo…
1 0 1 00 1 0 11 0 1 00 1 0 1
1122
1122
→→→→
/ ∂/ ∂/ ∂/ ∂
141
CYCLE-1 Let = [0 0 0 0]T
Solve∂
0 with 0 toobtain
-1+3+1=0 -1+0+1=0 1=1
= [1 0 0 0]T
Solve∂
0 with 1, 0 toobtain
-2+4+1=0 -2+0+1=0 2=1
= [1 1 0 0]T
Solve∂
0 with 1, 0 toobtain
1-3-2=0 1-3-2=0 3=-1 Fix 3=0
= [1 1 0 0]T
Solve∂
0 with 1, 0 toobtain
2-4-2=0 1-4-2=0 4=-1 Fix 4=0
= [1 1 0 0]T
END OF CYCLE 1 Another cycle is required since has changed from [0 0 0 0]T to [1 1 0 0]T.
142
1. -1+3+1=0; with 2=1, 3=0, 4=0 1=1
= [1 1 0 0]T
2. -2+4+1=0; with 1=1, 3=0, 4=0 2=1
= [1 1 0 0]T
3. 1-3-2=0; with 1=1, 2=1, 4=10 3=-1
= [1 1 0 0]T (3<0, so we set it to 0)
4. 2-4-2=0; with 1=1, 2=1, 3=1 4=-1
= [1 1 0 0]T (4<0, so we set it to 0)
END OF CYCLE 2
Since the value of has not changed from the previous cycle the algorithm has
converged.
Thus 1*= 2
*= 1; 3*= 4
* = 0. In order to now obtain the optimal primal vector x*
x* = Q-1[AT*-c] = 1 00 1
1 0 1 00 1 0 1
22
11
Dual Objective = Primal Objective = -3.
EXERCISE: Verify that K-K-T conditions for the dual are satisfied at x*.
143
WOLFE'S ALGORITHM FOR QP
Minimize12
,st , .
KARUSH-KUHN-TUCKER CONDITIONS (ignoring x0)
xL(x,) = 0 Qx + c - AT = 0 Qx - AT = -c
Original Constraints Ax ≥ b Ax - y = b (ysurplus)
Complementary Slackness Ty = 0
Nonnegativity y ≥ 0, ≥ 0.
Thus, we need to solve complementary slackness, nonnegativity for , y and x,
along with
This is solved in a manner similar to the PHASE-1 procedure of the simplex
method, while also using a "restricted basis entry" (RBE) rule:
“If yi is already in the basis at a level>0, do not consider i as a
candidate for entry into the basis, and vice versa. If yi (i) is in the
basis already at a 0 level, i (yi) may enter only if yi (i) remains at the
zero level.”
144
Let wj represent the artificial variable for the jth equality. Then, for the objective
we minimize the sum of the artificial variables (jwj). Thus the problem is to
Minimize W = jwj
st
while using the RBE rule.
If the optimum objective value of the LP is zero, we have an optimum
solution to the original problem; else the latter has no solution.
======================================
EXAMPLE: Minimize ½ x12 + x2
2 – x1x2 – x1 – x2
st x1+x2 ≤ 3 (i.e., – x1 – x2 ≥ -3)
2x1 +3x2 ≥ 6
x1, x2 ≥ 1.
i.e.,Min12
1 11 2
1 1
xT Q x cT x
st
1 12 30 01 1
3611
, 00
A x b x 0
Then it follows that
145
1 1 1 2 1 0 0 0 0 01 2 1 3 0 1 0 0 0 01 1 0 0 0 0 1 0 0 0
2 3 0 0 0 0 0 1 0 01 0 0 0 0 0 0 0 1 00 1 0 0 0 0 0 0 0 1
Therefore the simplex tableau for Wolfe's method is:
0 0 0 0 0 0 0 0 0 0 1 1 1 1 1 1 1 01 1 1 2 1 0 0 0 0 0 1 0 0 0 0 0 0 11 2 1 3 0 1 0 0 0 0 0 1 0 0 0 0 0 11 1 0 0 0 0 1 0 0 0 0 0 1 0 0 0 0 3
2 3 0 0 0 0 0 1 0 0 0 0 0 1 0 0 0 61 0 0 0 0 0 0 0 1 0 0 0 0 0 1 0 0 10 1 0 0 0 0 0 0 0 1 0 0 0 0 0 1 0 1
The simplex method requires that the RHS be non-negative, so that both sides of
equality in the 3rd constraint should be multiplied by -1.Thus we have
0 0 0 0 0 0 0 0 0 0 1 1 1 1 1 1 1 01 1 1 2 1 0 0 0 0 0 1 0 0 0 0 0 0 11 2 1 3 0 1 0 0 0 0 0 1 0 0 0 0 0 1
1 1 0 0 0 0 1 0 0 0 0 0 1 0 0 0 0 32 3 0 0 0 0 0 1 0 0 0 0 0 1 0 0 0 61 0 0 0 0 0 0 0 1 0 0 0 0 0 1 0 0 10 1 0 0 0 0 0 0 0 1 0 0 0 0 0 1 0 1
Adjusting the objective row via, Row 0 Row 0 + j,j≠3(Row j) yields the final
starting tableau below; now we can proceed with the Simplex method (with RBE)
4 6 2 5 1 1 0 1 1 1 0 0 0 0 0 0 0 111 1 1 2 1 0 0 0 0 0 1 0 0 0 0 0 0 11 2 1 3 0 1 0 0 0 0 0 1 0 0 0 0 0 11 1 0 0 0 0 1 0 0 0 0 0 1 0 0 0 0 32 3 0 0 0 0 0 1 0 0 0 0 0 1 0 0 0 61 0 0 0 0 0 0 0 1 0 0 0 0 0 1 0 0 10 1 0 0 0 0 0 0 0 1 0 0 0 0 0 1 0 1
146
BEALE'S ALGORITHM FOR QP
Minimize12
,st , .
Let BBasic; NNonbasic, so that ABxB + ANxN = b
| initialbasicfeasiblesolution, and
A = [AB | AN] where AB has columns corresponding to the basic variables xB and
is nonsingular. If xN =0, then xB = [AB]-1b.
In general, xB = [AB]-1b - [AB]-1ANxN for x that satisfies Ax=b.
Let us also partition c and Q as
| and where
EXAMPLE: Min ½ x12 + ½ x2
2 + x3 + 2x1x2 + 2x1x3 + x2x3 + 2x1 + x2 + x3
12
1 2 22 1 12 1 2
211
Q cT
st
810 ⟹ 1 0 1
0 1 1810
,
A b
147
An initial BFS might be x1=8, x2=10, x3=0, with the objective equal to 268 so that
| | 810|0
| 1 0 10 1 1
; | 21|1
1 2 22 1 12 1 2
Notethat 1 00 1
810
810
1 00 1
11
0 00
The objective function is f(x) = ½ xTQx + cTx
| |
|
Upon substituting xB = [AB]-1b - [AB]-1ANxN we obtain
12
12
148
Thus 12
Computing the partial derivatives, we have
for ϵ
= pi at xN = 0.
Choose the most negative pi, and increase the corresponding value of xi until
(1) One of the basic variables becomes zero (cf. the Simplex method...), OR
2 0, i. e. ,
(this results in a nonbasic but feasible solution)
For our example,
810 21
810 1 22 1
1 00 1
11
1 21 1 00 1
10
30
∴ 30
Increase x3 from 0 until one of the basic variables goes to zero, i.e., by 8 units
when x1=0, x2=2, x3 =8 (don't need R now since xN = 0). The new objective
function value is 92 (compared to 268 at the previous iteration).
At this point xB = [x2 x3 | x1] = [2 8 | 0] etc., etc., etc…
Top Related