Chapter 4 Sensitivity Analysis, Duality and Interior Point Methods.
-
Upload
mariah-berry -
Category
Documents
-
view
246 -
download
2
Transcript of Chapter 4 Sensitivity Analysis, Duality and Interior Point Methods.
Chapter 4
Sensitivity Analysis, Duality and Interior Point Methods
•Simplex methods reach the optimal solution by traversing a series of basic solutions. The path followed by the algorithm moves around the boundary of the feasible region.
•This can be cripplingly inefficient given that the number of extreme points grows exponentially with n for m fixed.
•The running time of an algorithm as a function of the problem size is known as its computational complexity.
• In practice, the simplex method works surprisingly well, exhibiting linear complexity—i.e., its running time is proportional to m + n.
•Researchers, however, have long tried to develop solution algorithms whose worst-case running times are polynomial functions of the problem size.
Interior Point Methods
•First Success: Russian Mathematician Khachian proposed the ellipsoid method with complexity O(n6).
It is efficient only theoretically, no one ever realize an implementation that matched the performance of current simplex codes.
•Big Success: Karmarkar(1984)
IPMs have produced solutions to many industrial problems that previously were intractable.
Interior Point Methods (IPMs)
Three major types of IPMs:
(1) Potential Reduction algorithm: most closely emb
odies the constructs of Karmarkar;
(2) Affine Scaling algorithm: the simplest to implem
ent;
(3) Path Following algorithms: combine excellent be
havior in theory and practice.
• We discuss a member of the third category known as the primal-dual path following algorithm. The algorithm operates simultaneously on the primal and dual linear programming problems.
Example:
Primal model Dual model
Max ZP = 2X1+ 3 X2 Min ZD = 8π1 + 6π2
s.t. 2 X1 + X2 + X3 =8 s.t. 2π1 +π2 - Z1 =2
X1 + 2X2 + X4 = 6 π1 + 2π2 - Z2 =3
Xj 0, j=l,.,.,4 π≧ 1 -Z3 =0
π2 -Z4 =0
Zj 0, j=1,...,4≧
Note: Zj’s denote the slack variables of the dual model.
• The algorithm iteratively modifies the primal and
dual solutions until the optimality conditions of
primal feasibility, dual feasibility, and
complementary slackness are satisfied to a sufficient
degree. This event is signaled when the duality gap—
the difference between the primal and dual objective
functions—is sufficiently small.
• For the purposes of the discussion, "interior of the
feasible region" is taken to mean that the values of
the primal and dual variables are always strictly
greater than zero.
The Primal and Dual Problems
Basic Ideas:
• The application of the Lagrange multiplier method to the transformation of a
n equality constrained optimization problem into an unconstrained one.
• The transformation of an inequality constrained optimization problem into a s
equence of unconstrained problems by incorporating the constraints in a logar
ithmic barrier function that imposes a growing penalty as the boundary { Xj =
0, and Zj = 0 for all j) is approached.
• The solution of a set of nonlinear equations using Newton's method, thereby a
rriving at a solution of the unconstrained optimization problem.
The Lagrangian Approach
Consider the general problem
Maximize f(x)
subject to gi (x) = 0, i= 1,...,m
where f(x) and gi (x) are scalar functions of the n-dimensio
nal vector x.
The Lagrangian for this problem is
L(x,π)=f(x)-
where the variables π = ( π1 , π2 ,..., πm ) are the Lagran
ge multipliers
m
iii xg
1
)(
Necessary conditions for a stationary point (maximum or minimum):
• For linear constraints (aix –bi = 0) , the conditions
are sufficient for a maximum if the function f(x) is concave and sufficient for a minimum if f(x) is convex.
, =1,..., and 0, 1, 2,...,j i
L Lj n i m
x
The Barrier Approach
• Considering the non-negativity conditions, the idea of the bar
rier approach, as developed by Fiacco and McCormick [196
8], is to start from a point in the strict interior of the inequalit
ies (xj > 0, and zj > 0 for all j) and construct a barrier that pre
vents any variable from reaching a boundary (e.g., xj = 0).
• Adding log(xj) to the objective function of the primal proble
m, for example, will cause the objective to decrease without
bound as Xj approaches 0.
• The difficulty is that if the constrained optimal solution is on the boundary (that is, one or more Xj*= 0, which is alw
ays the case in linear programming), then the barrier will prevent us from reaching it. To get around this difficulty, we use a barrier parameter μ that balances the contribution of the true objective function with that of the barrier term.
• The parameter μis required to be positive and controls the magnitude of the barrier term. Because function log(x) takes on very large negative values as x approaches zero from above, as long as the problem variables x and z remain positive the optimal solutions to the barrier problems will be interior to the nonnegative orthants (Xj > 0 and Zj > 0 for al
l j).
• The barrier term is added to the objective function for a maximization problem and subtracted for a minimization problem.
• The new formulations have nonlinear objective functions with linear equality constraints, and can be solved with the Lagrangian technique for μ > 0 fixed. The solutions to these problems will approach the solution to the original problem as μ approaches zero.
Note:• These conditions are also sufficient, because the primal La
grangian is concave and the dual Lagrangian is convex.
• The optimal conditions are nothing more than primal feasibility, dual feasibility, and complementary slackness satisfied to within a tolerance of μ.
To facilitate the presentation, we define two n × n diagonal matrices containing the components of x and z, respectively—i.e.,
X = diag{ x1, x2,.., xn }
Z = diag{ z1, z2,..., zn }
• Let e = (1,1,..., 1)T be a column vector of size n. The necessary and suffic
ient conditions can be written as
Primal feasibility: Ax - b = 0 (m linear equations)
Dual feasibility: A T πT - z - c T = 0 (n linear equations)
μ-Complementary slackness: XZe - μe = 0 (n nonlinear
equations)
We must now solve this set of nonlinear equations for the variables (x, π, z).
Stationary Solutions Using Newton's Method
• Consider the single variable problem of finding y to satisfy the nonlinear equation
f(y)=0
where f is once continuously differentiable. Let y* be the unknown solution.
• Taking the first-order Taylor series expansion of f(y) around yk yields
where = y△ k+1– yk.
• Setting the approximation f(yk+1) equal to zero and solving for yields△
)()()( '1 kkk yfyfyf
)(
)(' k
k
yf
yf
• The point yk+1 = yk + is an approximate solution of △the equation.
• It can be shown that if one starts at a point y° sufficiently close to y*, then yk→y* as k→∞.
• Newton's method can be extended to multidimensional functions. Consider the general problem of finding the r-dimensional vector y that solves the set of r equations
fi (y)=0, i=l,...,r or
f(y)=0
r
rrr
r
r
K
y
f
y
f
y
f
y
f
y
f
y
fy
f
y
f
y
f
yJ
21
2
2
2
1
2
1
2
1
1
1
:::
• The Jacobian at yk is
All the partial derivatives are evaluated at yk.
• Taking the first-order Taylor series expansion around the point yk and setting it equal to zero yields
f(yk)+ J(yk)d=0
where d = y k+1 - y k is an r-dimensional vector whose components represent the change in position for the (k+l)th iteration.
• Solving for d leads to
d= -J(yk) -1f(yk)
• The point yk+1 = yk + d is an approximation of the solution of the set of equations.
Newton's Method for Solving Barrier Problems:• We now use Newton's method to solve the optimality cond
itions for the barrier problems for a fixed value of μ. • For y = (x, π, z) and r = 2n + m, the corresponding equatio
ns and Jacobian are
where e is an n-dimensional column vector of 1's. • Assume that we have a starting point (x°, π°, z°) satisfying
x° > 0 and z° > 0, and denote by
δp=b-Ax°
δD= cT –AT(π°)T+ z°
XZ
IA
A
yJ
eXZe
czA
bAxTTTT
0
0
00
)(
0
0
0
• The optimality conditions can be written as
J(y)d = -f(y)
where the (2n + m)-dimensional vector d = (dX , dπ, dz)T is c
alled the Newton direction. We must now solve for the individual components of d.
• In explicit form, the preceding system is
XZeed
d
d
XZ
IA
A
D
p
z
xT
0
0
00
XZeeXdZd
ddA
Ad
Zx
DzT
px
• The first step is to find dπ. With some algebraic manipulation, we get
or
Note that Z-l = diag{ 1/z1, l/z2,.. , 1/zn} and is trivial to compute. • Further multiplications and substitutions lead to
dz= -δD+A T dπ and
dx = Z-l (μe-XZe-Xdz) • From these results, we can see in part why it is necessary to rema
in in the interior of the feasible region. In particular, if either Z-l or X-l does not exist, the procedure breaks down.
1 1 1 1( ) ( )TDd AZ XA b AZ e AZ X
1 1 1( )TDAZ XA d b AZ e AZ X
• We move from the current point (x°, π°, z°) to a new point (x1, π1, z1) by taking a step in the direction (dx, dπ , dz). The
step sizes αp and αD are chosen to preserve x > 0 and π > 0.
• This requires a ratio test similar to that performed in the simplex algorithm. The simplest approach is to use
min : ( ) 0( )
min : ( ) 0( )
jp x j
jx j
jD z j
z j
xd
d
zd
d
where γ is the step size factor that keeps us from actually touching the boundary. Typically, γ= 0.995. The notation (dx) j refers to t
he jth component of the vector dx
• The new point is
which completes one iteration.
• Ordinarily, we would now resolve using (x1, π1, z1) to find a new Newton direction and hence a new point. Rather than iterating in this manner until the system converges for the current value of μ, it is much more efficient to reduce μ at every iteration.
zD
D
xp
dzz
d
dxx
01
01
01
• It is straightforward to show that the Newton step reduces the duality gap—the difference between the dual and primal objective values at the current point. Assume that x° is primal feasible and that (π0, z°) is dual feasible.
• Let "gap(0)" denote the current duality gap.
gap(0) =π0b - cx°
=π0(Ax0) - (π0A - z0) T x° (primal and dual feasi
bility)
= (z0) T x°
• If we let α = min{ αp, αD}, then Tz
Tz dxdzgap )()()( 00
• It can be shown that gap(α) < gap(0) as long as
In our computations, we have used
which indicates that the value of μk is proportional to the duality gap.
• Termination occurs when the gap is sufficiently small—say, less than 10-8.
n
gap )0(
22
)()(
n
xz
n
gap kTkkk
Primal-Dual Interior Point Algorithm
Inputs:
1. The data of the problem (A, b, c), where the m x n matrix A has full row rank
2. Initial primal and dual feasible solutions x° > 0, z° > 0, π°
3. The optimality tolerance ε > 0 and the step size parameter γ (0,1)
Step 1: (Initialization) Start with some feasible point x° > 0, z° > 0, π° and set the iteration counter k = 0.
Step 2: (Optimality Test) If (zk) Txk < ε, stop; otherwise, go to Step 3.
Step 3: (Compute Newton Directions) Let
1 2 1 2 2
( ) x, ,..., , , ,..., ,
k T Kk k k k k k k k k
n n
zX diag x x x Z diag z z z
n
Solve the following linear system to get
Note that δP=0 and δD=0 as a result of the feasibility
of the initial point.
, , and k k kx zd d d
0
0
kx
T k kz
k kx z
Ad
A d d
Zd Xd e XZe
Step 4: (Find Step Lengths) Let
0)(:)(
min,0)(:)(
min jkz
jkz
kj
jDj
kx
jkx
kj
jP d
d
zd
d
x
Step 5: (Update Solution) Take a step in the Newton direction to get
Put k← k + 1 and go to Step 2.
kD
kk
KD
kk
kxP
kk
dzz
d
dxx
1
1
1
Example:
We have n = 4 and m = 2, with data
and variable definitions
6
8,
1021
0112),0,0,3,2( bAc
21
4
3
2
1
4
3
2
1
,,,
z
z
z
z
z
x
x
x
x
x
For an initial interior solution, we take x° = (1,1, 5, 3), z° = (4, 3, 2, 2), and π° = (2, 2) .
•The primal residual vectorδP and the dual residual vector δD are bot
h zero, since x° and z° are interior and feasible to their respective problems.
•The column labeled "Comp. slack" is the μ-complementary slackness vector computed as
XZe-μe
Since this vector is not zero, the solution is not optimal.