On detection of linear and nonlinear optimization problems

Volume 2, Number 4 OPERATIONS RESEARCH LETTERS November 1983

ON DETECTION OF LINEAR AND NONLINEAR OPTIMIZATION PROBLEMS

Asim ROY Quantitative Systems Department, College of Business Administration, Arizona State University, Tempe, AZ 8.5287, USA

Leon LASDON Department of General Business, University of Te.~as at Austin, TX 78712, USA

Received September 1982 Revised June 1983

Automatic classification of optimization problems as linear or nonlinear is now being demanded in some modeling and optimization systems to make such systems more user friendly. This requires that the system examine the model and make such determination. On such method of analysis is discussed in this paper.

1. Introduction

Different types of optimizers are being inter- faced with both problem oriented an general pur- pose languages in increasing numbers. Drud [3] has introduced an optimal control capability into a version of TSP and Berne et at. [1] have included nonlinear optimization in their chemical engineering design language. The algebraic modeling language GAMS [2] has both linear and nonlinear optimization capabilities. Optimization capabilities have been added to some financial planning languages, e.g., IFPS [6]. Through the planning languages, optimization is being introduced to a class of users who have very little knowledge about different categories of optimization problems and the algorithms required to solve them. Hence de- mands are being placed on system designers not to ask for information relating to problem classification. Thus, in the context of the work described in [6], which has both a linear and nonlinear opti- mizer, it was preferred that the user not be required to identify the nature of the problem, i.e., whether it is linear or nonlinear. Instead, the system itself was required to examine the model, internally detect the problem type, inform the user of its findings, and use the appropriate solution algorithm to solve it.

This paper describes the method and the results that are used to classify problems as linear or nonlinear in the systems described in [6]. However, the discussion in this paper is applicable to any language that allows multi-column models. In the next section we describe, ~Tith examples, the defi- nition of optimization problems in planning languages and define some terms. We also discuss the classification of variables and equations which must be done to determine if a model is linear or nonlinear. In Section 3, we discuss the derivative computation method, while Section 4 derives the results which diagnose linear and nonlinear problems. The method is illustrated in Section 5.

2. Optimization problems in planning lantu~es

Almost all planning languages have a matrix structure, where columns usually define time periods and rows specify the formulas for computing variables. We refer to columns henceforth as if they correspond to time periods. Table 1 shows an arbitrary 3 time period model, and Tables 2 and 3 each describe a separate optimization problem based on that model. In Table 2, variables At and B t are declared to be decision variables of the model. The values for At and Bt in the model serve

0167-6377/83/$3.00 O 1953, Elsevier Science Publishers B.V. (North-Holland) 149


Table 1 A 3-time period model

Optimization Problem I

3 Time Variable Period Model Classification

Endogeno,s Variable Number - l inear/ nonlinear

Optimization Problem 2

Variable Classification

Endogenous Variable Number - l inear/ nonlinear

A, : 1,2,3 Decision B, - 2,5,7 Decision Je "3,2,10 Exogenous K, ,= 5,10,11 Exogenous Nt - J, + K, Exogenous Ce - De + J, Ae Endogenous- Yt - linear

Simultaneous w/ D, De - - Ce - B , K , Endogenous- Y2 - linear

Simultaneous w/ G E e - ~ A, + F,= Endogenous- Y3 - nonlinear

Simultaneous w/F, F, - 2 E, - B, Endogenous- Y4 - linear

Simultaneous W / E , Xe - 2Ce + D, Endogenous-problem Ys - linear

function Y, =, E, + F~ Endogenous-problem Y6 - linear

function G, : 2B~ 2 Endogenous Y7 - nonlinear He = G, + A t Endogenous Ys - linear Z, = N,H, Endogenous.problem Y9 - linear

function L e = A,B, Endogenous- Yto - nonlinear

extraneous Pe" A, + B, Endogenous- Ytt - linear

extraneous Q," L, + 2P, Endogenous- Yt2 - linear

extraneous Wt - ~ " + Je Exogenous

X s u m - X t + X 2 + X3 Endogenous.problem Yt3 - linear function

Exogenous Exogenous Decision Decision Endogenous Endogenous- Simultaneous linear Endogenous- Simultaneous linear Exogenous

Exogenous

Endogenous-problem function Exogenous

Exogenous Exogenous Endogenous-problem function Exogenous

Exogenous

Exogenous

Yt - linear Y2 - linear

)'3 - linear.

Y4 - linear

Y5 - linear

Endogenous- ) '6- nonlinear extraneous Endogenous-problem Y7 - linear function

as the starting point for a nonlinear optimization algorithm if the problem is nonlinear. The variables J , K,, ~ and W, are exogenous variables of the model in this case. Exogenous variables are defined to be those variables which do not depend

on the decision variables, whereas endogenous variables are those variables which depend, directly or indirectly, on the decision variables. Thus, the remaining variables of the model are endogenous variables as shown in Table I for optimiza-

Table 2 Optimization problem I for model in Table 1

Objective Mav~nd~ Xsum

Decisions

0 ~ A,, B, ~ 10 t - 1,2,3 Constraints

x,. Y,. z , , ; 2o t--1,2,3

Table 3 Optimization problem 2 for model in Table 2

Objective Maximize Xsum

Decisions O ,~ ~, g, ~ lO

Constraints x . Z t ~ 10

t - 1.2.3

t - 1.2.3


tion problem 1. However, all the endogenous variables in the model are not directly constrained in the optimization formulation - only the variables X t, Y~ and 7-, are constrained, and Xsum is to be maximized. The other endogenous variables, i.e. C,, D t, E,, F t, G,, Hi, L,, Pt and Qt, are termed intermediate endogenous variables. On further ex- amination of the model it will be found that some of the intermediate endogenous variables, like L,, P, and 0,, do not affect any of the constraints or the objective (henceforth referred to as a problem function) either directly or indirectly. Such intermediate endogenous variables are called extraneous variables. The set of extraneous variables in a model depends on the optimization problem formulation. For example, if Q~ was added to the set of constraint functions in Table 2, then L,, Pt and Qt would no longer be extraneous. On the other hand, if X t was dropped from the constraint set, then, C t, D t and X t become extraneous. The concept of extraneous variables can be further extended from the point of view of a particular problem function. For example, for the objective Xsum in Table 2, the endogenous variables E t, F t, Y,, G t, H t and Z t are considered to be extraneous in addition to L t, Pt and Qt. The set of exogenous, endogenous and extraneous variables of the model are, of course, different when the optimization problem defines Jt and K t to be the decisions and only X t and Z t as constraints in Table 3. In that case only the variables N t, C~, D t, X,, Z,, Wt, and Xsum are endogenous and W t is extraneous.

An equation defining an endogenous variable is said to be linear if it is finear with respect to all the decision and endogenous variables on its right- hand side; otherwise it is nonlinear. Thus, for the optimization problem in Table 2, the equation defining C's is linear, since,/~ is exogenous; whereas the equation defining E t is nonlinear. Figure 1 shows the linear/nonlinear classification of each endogenous equation in the model for each optimization problem. A simultaneous equation set is nonlinear if at least one equation in the set is nonlinear; otherwise it is linear. Thus, for optimization problem 1, C~ and D t are a linear simul~ta- neous set, whereas E, and F, are a nonlinear simultaneous set though F~ is itself linear. In the actual computer implementation [6] the endogenous variables are assigned numbers for their separate identity, and such endogenous variable numbers are shown in Table 1 for each optimization

problem. It should be pointed out that the detection of linear/nonfinear problem functions would fail without proper variable classification (as exogenous or endogenous) and that equation classification (as linear or nonlinear) should follow variable classification.

The presence of nonlinear endogenous equations in a model does not mean that all or any of the problem functions are nonlinear. For the optimization problem defined in Table 2, an examina- tion of the model will find the functions X t and Xsum to be linear, and Y, and Z t to be nonlinear; hence the optimization problem is classified as nonlinear. However, the optimization problem defined in Table 3 is linear, since the problem functions X,, Z, and Xsum are all linear, even though W, is nonlinear. Thus, the same model can result in a linear/nonlinear optimization problem de- pending on the formulation of the optimization problem.

In the next two sections we present a method for deciding if a problem function is ~i~>ear or nonlinear based on the linear/nonlinear classification of each endogenous equation of the model, assuming the model might contain some nonlinear equations. Each of the model equations can be examined at the symbolic level to derive its linear/nonlinear classification, or linearity can be checked by evaluating the equation at several points [4]. The method presented here does not depend on the means of obtaining the linear/nonlinear classification of an equation. Our essential task is to find if the nonlinear effects 'cascade' into a problem function. One alternative to our procedure would be to use back substitution to obtain a closed form expression for a problem function in terms of the decision variables and then examine that expression to determine its linearity/nonlineadty. However, deriving such closed form expressions may not be possible when inter- media~te endogenous variables affecting that problem function are involved in simultaneous equations, as in the case of X~ in optimization problem 1. Of course, this problem ~an be overcome by developing a list of simultaneous variables affecting the function. Thus we would need to know that Ct and D~ affect Xt, and Et and Ft affect Yr. However, searching through a large model to obtain a separate list for each problem function is expensive. The method presented next shows how all the relevant information to classify a function

151


as linear or nonlinear can be obtained easily as a byproduct of its derivative computation (which is required in any case), thereby avoiding any extra effort.

3. Computing derivatives of problem functions

Let y and u be the vectors of endogenous and decision variables over all time periods of dimen- sions m and n respectively. Then, suppressing the exogenous variables, the model equations have the form:

O(y.u)=0 (t) where G is an m-vector of functions, assumed differentiable. As illustrated in the previous section, the constraints and the objective function are usually a subset of the functions in G. The other equations in G are intermediate equations involved in the computation of the objective or the constraints, or they may be equations computing extraneous information. But the extraneous equations are not identified as such by the user. We assume that (1) can be solved for y given any u of interest, and that the Jacobian matrix aG/ay is nonsingular for all y, u. Let f (y , u) be any differentiable function of y and u - representing a constraint or the objective. By the implicit function theorem, y is a differentiable function of u, say y(u), as is f(y(u), u)= F(u).

The gradient aF/au, called the reduced gradient, can be computed as follows [5,6]:

1. Solve * t ( aG/ay ) - af/ay (2)

for the Lagrange multiplier (row) vector *t.

2. Evaluate af/au by

aF/au = af/au-~(a~/au). O) In the above, all partial derivatwes are evaluated

at some point (y, u) satisfying (1), and (3) yields aF/au evaluated at u.

4. Detection of l i n e a r a n d nonlinear problem functions

In general, a model may contain both linear and nonlinear equations in its set of equations G(y, u). Thus, the Jacobians OG/ay and ag/au in equations (2) and (3) may contain some noncon-

stunt elements, i.e. elements which depend on y and u. However, all of the nonlinear equations may be extraneous to a particular problem function. In that case the reduced problem function F(u) will be linear. The following two theorems help to detect whether a reduced problem function is linear or not. The detection is done during the derivative computation of the function.

Theorem 1 If a f lay is independent of y and u, and if at any point (y(u°), u °) the set of Lagrange multi. pliers *t corresponding to nonlinear equations in G are zero, then they will be zero at any other point (y(u~j, u~.

Proof. Let ~r--(*t~, *t2) where *t~ - vector of Lagrange multipliers

corresponding to nonlinear equations in G, *t2 -vector of Lagrange multipliers

corresponding to linear equations in G. Let us similarly partition

r -

OG/Oy = I aG|/ay [aG2/ay •

Then,

[aGl /aY l -o f /Oy (*t~, *t,) at;2/ay

or,

aG ! aG 2 af "-G-y - a-7" (+)

Let ~r~, *t~ satisfy (4) when all partial derivatives are evaluated at (y(u ~), u +), i - 0 , 1. If at (y(uO), u0), *to - 0 then

aG 2 o° . . Oy - a l l a y . (5)

Here aG2/ay and 8flay consist of constant elements. At (y(u I ), u I ),

(aG') ' a6 2 a / ~l" "~y +*t~" ay =ay" (6)

This is a nonsingular system, hence its solution is unique. Setting ~r~ = 0, *t2 ! = ,r ° yields

at; 2 ~2°" ay = ay

which is true by (5). Hence (0, *to) satisfies (6), and is the vector of multipliers at any point (y(ul),ul).

152

Volume 2, Number 4 OPERATIONS RESEARCH LETrERS November 1983

Note that, under the hypothesis of this theorem, the multipliers 92 are also independent of u. This fact is used in proving the following theorem.

solution of (2)-(3), the optimization problem is labeled nonlinear.

Theorem 2. A problem function F(u) -~ f(y(u),u) is a linear function of u if (1) f(y, u) is linear in y and u, and (2) 9 ] - 0 at any point (y(u), u) where 91 is as defined in Theorem 1.

Proof. Let

[ "1 ao/a. L A G 2 / a u j "

Then, at any point (y(u), u)

aG I aG 2 a F / a u - a f / a u - 91 • ay 9'+'2" au • (7)

The proof of Theorem 1 showed that, under conditions (1) and (2) above, if (0, 9 2) was the multiplier vector at any point (y(u°) , u°), then it was the multiplier vector at any other point (y(u ~ ), u ~). Hence

aG 2 a F l a u - a f l a u - 9 ° . au " (8)

Since all quantities on the right hand side of (8) are independent of u, aF/au is independent of u. Hence F is a linear function of u.

The above two results show that the linearity/nonlinearity detection may be done at any point. In the system described in [6], it is done in the initialization phase. Nonconstant Jacobian elements are flagged, add if a nonzero component of .#r ever multiplies such an element during the

5. Illustration

Let us consider the simple one time period model shown in Table 4.

Table 5 shows the endogenous Jacobian for the above model. For problem function X, equation (2) yields the following set of multipliers at any point:

9~ - - 3 , 92 -- - 2 ,

93 -- 94 = 9s =96 = 9 7 = 9 s = 9 9 = 0 .

Since the multipliers for the nonlinear equations are all zero, and X satisfies all the other conditions of the above theorems, X can be classified as linear. For problem function Y, equation (2) yields

Table 5 Endogenous Jacobian aG/ay and al lay for x and Y

C D E F X Y G H Z

OG/aY G1 1 - 1 G2 - 2 1 G3 1 - 2 F G4 - 2 1 G5 - 1 - 1 1 G6 - 1 - 1 1 G7 1 G8 - 1 1 G9 - 5 1

al/ay f/= X) t 1

a//yy (1 = Y) ] i

Table 4

Model Variable Classification

A = I B - - 2 C = D + 2 A D = 2 C - B E - F 2 + A

F = 2 E - B X = C + D Y = E + F G = 2B 2 H=G+A Z = S H

(decision) (decision) (endogenous-simultaneous with D) (endogenous-simultaneous with C) (endogenous-simultaneous with F) (endogenous-simultaneous with E) (endogenous-problem function) (endogenous-problem function (endogenous) (endogenous) (endogenous-problem function)

Endogenous Equation Number - linear/nonlinear

Ol-linear O2-finear O3-nonlinear G4-1inear OS-linear G6-1inear OT-nonlinear Og-linear G9-1inear

153

Volume 2, Number 4 OPERATIONS RESEARCH LETI'ERS November 1983

the following set of multipliers:

z~ s -- ~2 = % ffi ~6 = ~7--- ~ s - ~9 ffi O,

~r3 = 3 / (1 - 4 F ) ~ 4 = ( l + 2 F ) / ( 1 - 4 F ) .

Though ~r 4 = 0 when F = - ½ , % is always nonzero. Thus, one of the multipliers (~r 3 or ~r 4) corresponding to the nonlinear simultaneous equation set will be nonzero at all points and hence Y is classified as nonlinear. Note that when F = ~, ~r 4 are undefined and the Jacobian is singular.

6. Conelusion

The results presented in this paper should be helpful in designing userfriendly optimization in- terfaces with planning oriented languages. Such languages are widely used in a simulation mode. The introduction of an easy-to-use optimization capability increases their power tremendously, and should expand the use of optimization models. The system described in [6], now called I F P S / O p - timum, is now used in over 20 large corporations, and the automatic problem detection feature has been essential to its success. See [7] for a descrip- tion of the current version of the system.

References

[I| T. Berne, M. Locke and A. Westerberg, "A new approach to optimization of chemical processes", internal Report, Chemical Engineering Department, Carnegie-Mellon Uni- versity (June 1978).

[2] 3. Bisschopp and A. Meeraus, "Towards a general algebraic modeling system", internal Report, Development Research Center, The World Bank (July 1977).

[3] A. Drud, "Combining an optimal control program with the time series processor system", Technical Note No. 15, Development Research Center, World Bank, 1818 H Street, NW, Washington, DC 20433 (September 1979).

[4] A. Drud, "Detecting or verifying sparsity structure in large scale nonlinear programming", Technical Note No. 14, Development Research Center, World Bank, 1818 H Street, NW, Washington, DC 20433 (August, 1979).

[5] J.B. Mantell and L.S. Lasdon, "A GRG algorithm for econometric control problems", Annals of Economic and Social Measuremew 6 (5), 581-597 (1978).

[6] Roy, A., "Optimizing problems formulated using the inter- active financial planning system (IFPS)", Ph.D. Disserta- tion, Department of General Business, University of Texas at Austin, Texas 78712 (December 1979).

[7l IFP$/Optimum User's Manual, Release 3.0, Execucom Sys- tems Corp., P.O. Box 9758, Austin, Texas 78766-9758 (1983).

154

On detection of linear and nonlinear optimization problems

Documents

Transcript of On detection of linear and nonlinear optimization problems