OD Nonlinear Programming LARGE 2010

download OD Nonlinear Programming LARGE 2010

of 78

Transcript of OD Nonlinear Programming LARGE 2010

  • 7/24/2019 OD Nonlinear Programming LARGE 2010

    1/78

    NONLINEARPROGRAMMING

  • 7/24/2019 OD Nonlinear Programming LARGE 2010

    2/78

    Nonlinear Programming

    Linear programming has a fundamental role in OR.

    In linear programmingall it s funct ions(objective

    function and constraint functions) are linear.

    This assumption frequently does not hold, and

    nonlinear programming problems are formulated:

    Findx = (x1,x2,...,xn) to

    Maximizef(x)

    subject to

    gi(x) bi , for i = 1, 2, ...,mandx0

    331

  • 7/24/2019 OD Nonlinear Programming LARGE 2010

    3/78

    Nonlinear Programming

    There are many types of nonlinear programming

    problems, depending onf(x) andgi(x). Different algorithms are used for different types.

    Some problems can be solved very efficiently, whilst

    others, even small, can be very difficult.

    Nonlinear programming is a particularly large subject

    (all the animals that are not elephants).

    Only some important types will be dealt with here.

    332

  • 7/24/2019 OD Nonlinear Programming LARGE 2010

    4/78

    Application: product-mix problem

    Inproduct-mixproblems (as Wyndor Glass Co.) the

    goal is to determine optimal mix of production levels. Sometimesprice elast icit yis present: the amount of

    sold product has an inverse relation to price charged:

    333

  • 7/24/2019 OD Nonlinear Programming LARGE 2010

    5/78

    Price elasticity

    p(x) is the price required to sellx units.

    c is the unit cost for producing and distributingproduct.

    Profit from producing and sellingx is:

    P(x) =xp(x) cx

    334

  • 7/24/2019 OD Nonlinear Programming LARGE 2010

    6/78

    Product-mix problem

    If each product has a similar profit function, overall

    objective function is

    Other nonlinearity:marginal costvary with productionlevel.

    It may decrease when production level is increased due

    to thelearning-curve effect.

    It may increase due to overtime or more expensiveproduction facilities when production increase.

    335

    1

    ( ) ( )n

    j j

    j

    f P x

    x

  • 7/24/2019 OD Nonlinear Programming LARGE 2010

    7/78

    Application: transportation problem

    Determine optimal plan for shipping goods from

    various sources to various destinations (see P&TCompany problem).

    Cost per unit shippedmay not be fixed.Volume

    discountsare sometimes available for large shipments.

    Marginal costcan have a pattern like in the figure.

    Cost of shippingx units is a piecewise linear function

    C(x), with slope equal to the marginal cost.

    336

  • 7/24/2019 OD Nonlinear Programming LARGE 2010

    8/78

    Volume discounts on shipping costs

    Marginal cost Cost of shipping

    337

  • 7/24/2019 OD Nonlinear Programming LARGE 2010

    9/78

    Transportation problem

    If each combination of source and destination has a

    similar shipping cost function, so that cost of shippingxij units from sourcei (i = 1, 2, ...,m)

    to destinationj (j = 1, 2, ...,n) is given by a nonlinear

    functionCij(xij).

    The overall objective function is

    338

    1 1

    Minimize ( ) ( )m n

    ij ij

    i j

    f C x

    x

  • 7/24/2019 OD Nonlinear Programming LARGE 2010

    10/78

    Graphical illustration

    Example: Wyndor Glass Co. problem with constraint

    339

    2 2

    1 29 5 216x x

  • 7/24/2019 OD Nonlinear Programming LARGE 2010

    11/78

    Graphical illustration

    Example: Wyndor Glass Co. problem with objective

    function

    340

    2 21 1 2 2126 9 128 13Z x x x x

  • 7/24/2019 OD Nonlinear Programming LARGE 2010

    12/78

    Example: Wyndor Glass Co. (3)

    341

  • 7/24/2019 OD Nonlinear Programming LARGE 2010

    13/78

    Global and local optimum

    Example:f(x) with three local maxima (x = 0, 2,4), and

    three local minima (x = 1, 3,5). Global?

    342

  • 7/24/2019 OD Nonlinear Programming LARGE 2010

    14/78

    Guaranteed local maximum

    Global maximum when:

    Function always curving downward is a concavefunction (concave downward).

    Function always curving upward is a convex function

    (concave upward).

    343

    2

    2

    ( )0, for all

    f x

    x

  • 7/24/2019 OD Nonlinear Programming LARGE 2010

    15/78

    Guaranteed local optimum

    Nonlinear programming with no constraints and

    concaveobjective function, a local maximum is theglobal maximum.

    Nonlinear programming with no constraints and

    convexobjective function, a local minimum is the

    global minimum.

    With constraints, this guarantee still holds if the

    feasible regionis a convex set.

    344

  • 7/24/2019 OD Nonlinear Programming LARGE 2010

    16/78

    Ex: Wyndor Glass with one concavegi(x)

    345

  • 7/24/2019 OD Nonlinear Programming LARGE 2010

    17/78

    Types of NP problems

    Unconstrained Optimization: no constraints

    necessarycondition for a solutionx* =x to be optimal:

    whenf(x) is aconcavefunction this condition is

    sufficient.

    whenxj has a constraintxj0, sufficient condition

    changes to:

    346

    Maximize ( )f x

    *( ) 0 at , for 1, 2, ,j

    fj n

    x

    xx x

    * *

    * *

    0 at , if 0( )

    0 at , if 0

    j

    jj

    xf

    xx

    x xx

    x x

  • 7/24/2019 OD Nonlinear Programming LARGE 2010

    18/78

    Example: nonnegative constraint

    347

  • 7/24/2019 OD Nonlinear Programming LARGE 2010

    19/78

    Types of NP problems

    Linearly Constrained Optimization

    All constraints are linear and objective function is nonlinear.

    Special case: Quadratic Programming

    Objective function isquadratic.

    Many applications, e.g. portfolio selection, predictive control

    Convex Programming assumptions for maximization

    f(x) is aconcavefunction.

    Eachgi(x) is aconvexfunction.

    For a minimization problem,f(x) must be a convex function.

    348

  • 7/24/2019 OD Nonlinear Programming LARGE 2010

    20/78

    Types of NP problems

    Separable Programming convex programming where

    f(x) andgi(x) are separable functions.

    A separable function is a function where each term

    involves only a single variable:

    Nonconvex Programming: local optimum is not

    assured to be a global optimum.

    349

    1( ) ( )

    n

    j j

    jf f x

    x

  • 7/24/2019 OD Nonlinear Programming LARGE 2010

    21/78

    Types of NP problems

    Geometric Programming is applied to engineering

    design as well as economics and statistics problems. Objective function and constraint functions are of the

    form:

    ci andaij are typically physical constraints.

    When they are all strictly positive, functions are

    generalized positive polynomials (posynomials), and a

    convex programming algorithm can be applied.

    350

    1 2

    1 21( ) ( ), where ( )

    i i in

    Na a a

    i i i nig c P P x x x x x x

  • 7/24/2019 OD Nonlinear Programming LARGE 2010

    22/78

    Types of NP problems

    Fractional Programming

    maximizesf1(x) and minimizesf2(x).

    whenf1(x) andf2(x) are linear:

    can be transformed into a linear programming problem.

    351

    0

    0

    ( ) c

    fd

    cxx

    dx

    1

    2

    ( )Maximize ( )

    ( )

    ff

    f xx

    x

  • 7/24/2019 OD Nonlinear Programming LARGE 2010

    23/78

    One-variable unconstrained optimization

    Methods for solving unconstrained optimization with

    only one variable (n = 1), where the differentiablefunctionf(x) isconcave.

    Necessary and sufficientcondition for optimum:

    352

    *( ) 0 at .f x

    x xx

    l i h i i i bl

  • 7/24/2019 OD Nonlinear Programming LARGE 2010

    24/78

    Solving the optimization problem

    Iff(x) is not simple, problem cannot be solved

    analytically. If not, search procedures can solve the problem

    numerically.

    We will describe two methods:

    Bisection method

    Newtons method

    353

    i i h d

  • 7/24/2019 OD Nonlinear Programming LARGE 2010

    25/78

    Bisection method

    We know that:

    If derivative ofxisposit ive,x is alower boundofx*.

    If derivative ofxisnegative,x is anupper boundofx*

    .

    354

    *( ) 0 if ,df x

    x xdx

    *( ) 0 if ,df x

    x xdx

    *( ) 0 if .df x

    x xdx

    Bi i h d

  • 7/24/2019 OD Nonlinear Programming LARGE 2010

    26/78

    Bisection method

    Having:

    In the bisection method, new trial solution is the

    midpont between the two current bounds.

    355

    *

    *

    *

    = current trial solution,

    = current lower bound on ,

    = current upper bound on ,

    = error tolerance for .

    x

    x x

    x x

    x

    Al i h f h Bi i M h d

  • 7/24/2019 OD Nonlinear Programming LARGE 2010

    27/78

    Algorithm of the Bisection Method

    Initialization:Select. Find initial upper and lower

    bounds. Select initial trial as: Iteration:

    1. Evaluate

    2. If

    3. If

    4. Select a new

    Stopping rule:If stop. Otherwise, go to 1.

    356

    2x xx

    *( ) at ,df x

    x xdx

    ( )

    0, reset ,df x

    x xdx

    ( ) 0, reset ,

    df xx x

    dx

    2

    x xx

    2x x

    E l

  • 7/24/2019 OD Nonlinear Programming LARGE 2010

    28/78

    Example

    Maximize

    357

    4 6 ( ) 12 3 2f x x x x

    S l ti

  • 7/24/2019 OD Nonlinear Programming LARGE 2010

    29/78

    Solution

    First two derivatives:

    358

    3 5( ) 12(1 )df x

    x x

    dx

    22 4

    2

    ( ) 12(3 5 )

    d f xx x

    dx

    Iteration x )/ x x x Newx x)

    0 0 2 1 7.0000

    1 12 0 1 0.5 5.7812

    2 +10.12 0.5 1 0.75 7.6948

    3 +4.09 0.75 1 0.875 7.8439

    4 2.19 0.75 0.875 0.8125 7.8672

    5 +1.31 0.8125 0.875 0.84375 7.8829

    6 0.34 0.8125 0.84375 0.828125 7.8815

    7 +0.51 0.828125 0.84375 0.8359375 7.8839

    S l ti

  • 7/24/2019 OD Nonlinear Programming LARGE 2010

    30/78

    Solution

    x* 0.836

    0.828125

  • 7/24/2019 OD Nonlinear Programming LARGE 2010

    31/78

    Newtons method

    This method approximatef(x) in neighborhood of

    current trial solution by a quadratic function. This quadratic approximation uses Taylor series

    truncated after second derivative term:

    Maximized by settingf (xi+1) equal to zero (xi,f(xi) and

    f (xi) are constants):

    360

    2

    1 1 1

    ( )( ) ( ) ( )( ) ( )

    2

    i

    i i i i i i i

    f xf x f x f x x x x x

    1 1( ) ( ) ( )( ) 0i i i i if x f x f x x x

    1

    ( )

    ( )i

    i i

    i

    f xx x

    f x

    Al ith f N t M th d

  • 7/24/2019 OD Nonlinear Programming LARGE 2010

    32/78

    Algorithm of Newtons Method

    Initialization:Select. Find initial trial solutionxi by

    inspection. Seti = 1. Iterationi:

    1.

    2. Set

    Stopping rule:If stop;xi+1 is optimal.

    Otherwise,i =i + 1 (another iteration).

    361

    Calculate ( ) and ( ).i if x f x

    1

    ( ) .

    ( )i

    i i

    i

    f xx x

    f x

    1 ,i ix x

    E l

  • 7/24/2019 OD Nonlinear Programming LARGE 2010

    33/78

    Example

    Maximize again

    New solution is given by:

    Selectingx1 = 1, and= 0.00001,

    362

    4 6 ( ) 12 3 2f x x x x

    3 5 3 5

    1 2 4 2 4

    ( ) 12(1 ) 1

    ( ) 12(3 5 ) 3 5i

    i i i i

    i

    f x x x x xx x x x

    f x x x x x

    Iteration i x

    i

    x

    i

    ) x

    i

    ) x

    i

    ) x

    i

    +1

    1 1 7 12 96 0.875

    2 0.875 7.8439 2.1940 62.733 0.84003

    3 0.84003 7.8838 0.1325 55.279 0.83763

    4 0.83763 7.8839 0.0006 54.790 0.83762

    M lti i bl t i d ti i ti

  • 7/24/2019 OD Nonlinear Programming LARGE 2010

    34/78

    Multivariable unconstrained optimization

    Problem: maximizing a concavefunctionf(x) of

    multiple variables x = (x1,x2,...,xn) with no constraints. Necessary and sufficient condition for optimality:

    partial derivatives equal to zero.

    No analytical solution numerical searchprocedure

    must be used.

    One of these is thegradient search procedure:

    It identifies the direction that maximizes the rate at

    whichf(x) is increased.

    363

    Gradient search procedure

  • 7/24/2019 OD Nonlinear Programming LARGE 2010

    35/78

    Gradient search procedure

    Use values of partial derivatives to select the specific

    direction to move, using the gradient. Gradient of a pointx =x is thevectorwithpartial

    derivativesevaluated atx =x:

    Moves in the direction of this gradient untilf(x) stops

    increasing. Each iteration changes the trial solutionx:

    364

    1 2( ) , , , at

    n

    f f f

    f x x x

    x x x

    *Next ( )t f x x x

    Gradient search procedure

  • 7/24/2019 OD Nonlinear Programming LARGE 2010

    36/78

    Gradient search procedure

    wheret* is the value oftthatmaximizesf(x+tf(x)):

    The functionf(x +tf(x)) is simplyf(x) where:

    Iterations continue untilf(x) = 0 withtolerance:

    365

    , for 1,2, , .j

    f j nx

    , for 1, 2, ,j jj

    fx x t j nx

    x x

    *

    0 ( ( )) max ( ( ))

    tf t f f t f

    x x x x

    Summary of gradient search procedure

  • 7/24/2019 OD Nonlinear Programming LARGE 2010

    37/78

    Summary of gradient search procedure

    Init ializat ion:Selectand any initial trial solutionx.

    Go to stopping rule.Iterat ion:

    1. Expressf(x +tf(x)) as a function oftby setting

    and substitute these expressions intof(x).

    366

    , for 1,2, ,j j

    j

    fx x t j n

    x

    x x

    Summary of gradient search procedure

  • 7/24/2019 OD Nonlinear Programming LARGE 2010

    38/78

    Summary of gradient search procedure

    Iterat ion (concl.):

    2. Use search procedure to findt=t* that maximizesf(x +tf(x)) overt0.

    3. Resetx =x +t* f(x). Go to stopping rule.

    Stopping rule:Evaluatef(x) atx =x. Check if:

    If so, stop with currentx as the approximation ofx*.

    Otherwise, perform another iteration.

    367

    , for 1,2, , .j

    fj n

    x

    Example

  • 7/24/2019 OD Nonlinear Programming LARGE 2010

    39/78

    Example

    Maximize

    Thus,

    Verify thatf(x) is concave (see Appendix 2 of Hilliers

    book).

    Suppose thatx = (0, 0) is initial trial solution. Thus,

    368

    2 2

    1 2 2 1 2( ) 2 2 2 .f x x x x x x

    2 1

    1

    2 2f

    x xx

    1 2

    2

    2 2 4f

    x xx

    (0, 0) (0, 2)f

    Example (2)

  • 7/24/2019 OD Nonlinear Programming LARGE 2010

    40/78

    Example (2)

    Iterat ion 1:Step 1 sets

    by substituting these expressions intof(x):

    Because

    369

    1

    2

    0 (0) 00 (2) 2

    x t

    x t t

    2 2

    2

    ( ( )) (0,2 )2(0)(2 ) 2(2 ) 0 2(2 )

    4 8

    f t f f tt t t

    t t

    x x

    * 2

    0 0 (0, 2 ) max (0, 2 ) max {4 8 }

    t tf t f t f t t

    Example (3)

  • 7/24/2019 OD Nonlinear Programming LARGE 2010

    41/78

    Example (3)

    and

    it follows that

    so

    This completes first iteration. For new trial gradient is:

    370

    24 8 4 16 0d

    t t tdt

    * 1

    4t

    1 1

    Reset (0,0) (0, 2) 0,4 2

    x

    10, (1,0)

    2

    f

    Example (4)

  • 7/24/2019 OD Nonlinear Programming LARGE 2010

    42/78

    Example (4)

    As < 1,Iterat ion 2:

    so

    371

    1 10, (1,0) ,2 2

    t t

    x

    2

    2

    2

    1 1 ( ( )) 0 , 0 ,

    2 21 1 1

    (2 ) 2 22 2 2

    1

    2

    f t f f t t f t

    t t

    t t

    x x

    * 2

    0 0 (0,2 ) max (0,2 ) max {4 8 }

    t tf t f t f t t

    Example (5)

  • 7/24/2019 OD Nonlinear Programming LARGE 2010

    43/78

    Example (5)

    Because

    and

    then

    so

    This completes second iteration. See figure.

    372

    * 2

    0 0

    1 1 1 , max , max

    2 2 2t t

    f t f t t t

    2 1 1 2 0

    2

    dt t t

    dt

    * 1

    2t

    1 1 1 1Reset 0, (1,0) ,

    2 2 2 2

    x

    Illustration of example

  • 7/24/2019 OD Nonlinear Programming LARGE 2010

    44/78

    Illustration of example

    Optimal solution is (1, 1), as f(1, 1) = (0, 0)

    373

    Newtons method

  • 7/24/2019 OD Nonlinear Programming LARGE 2010

    45/78

    Newton s method

    It is aquadratic approximat ionof objective function

    f(x). When objective function isconcaveand its gradient

    f(x) are written ascolumn vectors,

    The solutionx that maximizes the approximating

    quadratic function is:

    where2f(x) is thennHessian matrix.

    374

    2 1[ ( )] ( ),f f x x x x

    Newtons method

  • 7/24/2019 OD Nonlinear Programming LARGE 2010

    46/78

    Newton s method

    Theinverseof the Hessian matrix is commonly

    approximated in various ways. These approximations are referred to as quasi-Newton

    methods (orvariable metr ic methods).

    Recall that this topic was mentioned in the discipline

    Intelligent Systems, e.g. in neural network learning.

    375

    Conditions for optimality

  • 7/24/2019 OD Nonlinear Programming LARGE 2010

    47/78

    Conditions for optimality

    376

    Problem

    Necessary conditions

    for optimality

    Also sufficient if:

    One-variableunconstrained

    f(x) concave

    Multivariableunconstrained f(x) concave

    Constrained, nonnegativeconstraints only

    f(x) concave

    General constrainedproblem Karush-Kuhn-Tuckerconditions

    f(x) concave and gi(x)

    convex (j = 1, 2,...,n)

    Karush-Kuhn-Tucker conditions

  • 7/24/2019 OD Nonlinear Programming LARGE 2010

    48/78

    Karush-Kuhn-Tucker conditions

    Theorem: Assume thatf(x),g1(x),g2(x),...,gm(x) are

    differentiablefunctions satisfying regularityconditions. Then

    x = (x1*,x1

    *,...,xn*)

    can be anopt imal solutionfor the NP problem if

    there arem numbersu1,u2,...,um such thatalltheKKT condit ionsare satisfied:

    1.

    2.

    377

    1*

    *

    1

    0

    at , for 1, 2 , .

    0

    mi

    i

    ij j

    mi

    j i

    ij j

    gfu

    x xj n

    gfx ux x

    x x

    Karush-Kuhn-Tucker conditions

  • 7/24/2019 OD Nonlinear Programming LARGE 2010

    49/78

    Karush-Kuhn-Tucker conditions

    3.

    4.

    5.

    6.

    Conditions 2. and 4. require that one of the twoquantities must be zero.

    Thus, conditions 3. and 4. can be combined:

    (3,4)

    378

    *

    *

    ( ) 0for 1,2, , .

    [ ( ) ] 0

    i i

    i i i

    g bj m

    u g b

    x

    x

    * 0, for 1, 2, , .j

    x j n

    0, for 1,2, , .iu j m

    *( ) 0

    (or 0, if 0), for 1, 2, , .

    i i

    i

    g b

    u j m

    x

    Karush-Kuhn-Tucker conditions

  • 7/24/2019 OD Nonlinear Programming LARGE 2010

    50/78

    Karush-Kuhn-Tucker conditions

    Similarly, conditions 1. and 2. can be combined:

    (1,2)

    Corollary: assume thatf(x) isconcaveandg1(x),g2(x),...,gm(x) areconvexfunctions, where all

    functions satisfy the regularity conditions. Then,

    x = (x1*,x1

    *,...,xn*) is anopt imal solutionif and only if

    all the conditions of the theorem are satisfied.

    379

    1*

    0

    (or 0 if 0), for 1,2 , .

    m ii

    ij j

    j

    gf ux x

    x j n

    Example

  • 7/24/2019 OD Nonlinear Programming LARGE 2010

    51/78

    Example

    Thus,m = 1, andg1(x) = 2x1 +x2 is convex.

    Further,f(x) is concave (check it!).

    Thus, the KKT conditions gives conditions to find anoptimal solution.

    380

    1 2Maximize ( ) ln( 1)f x x x

    1 20, 0x x

    subject to

    1 22 3x x and

    Example: KKT conditions

  • 7/24/2019 OD Nonlinear Programming LARGE 2010

    52/78

    Example: KKT conditions

    1. (j= 1)

    2. (j= 1)

    1. (j= 2)

    2. (j= 2)3.

    4.

    5.

    6.

    381

    1

    1

    12 0

    1u

    x

    1 1

    1

    12 0

    1x u

    x

    11 0u

    2 11 0x u 1 22 3 0x x

    1 1 2(2 3) 0u x x

    1 2

    0, 0x x

    1 0u

    Example: solving KKT conditions

  • 7/24/2019 OD Nonlinear Programming LARGE 2010

    53/78

    Example: solving KKT conditions

    From condition 1 (j= 2)u11.x10 from condition 5.

    Therefore,

    Therefore,x1 = 0, from condition 2 (j= 1).

    u10 implies that 2x1 +x2 3 = 0 from condition 4.

    Two previous steps implies thatx2 = 3.

    x20 implies thatu1 = 1 from condition 2 (j= 2).

    No conditions are violated forx1 = 0,x2 = 3,u1 = 1.

    Consequentlyx*

    = (0,3).

    382

    1

    1

    1 2 0.1

    ux

    Quadratic Programming

  • 7/24/2019 OD Nonlinear Programming LARGE 2010

    54/78

    Quadratic Programming

    Objective function can be expressed as:

    383

    1Maximize ( ) 2

    Tf x cx x Qx

    subject to

    , and Ax b x 0

    1 1 1

    1 1( )

    2 2

    n n nT

    j j ij i j

    j i j

    f c x q x x

    x cx x Qx

    Example

  • 7/24/2019 OD Nonlinear Programming LARGE 2010

    55/78

    Example

    In this case,

    384

    2 2

    1 2 1 2 1 2Maximize ( ) 15 30 4 2 4f x x x x x x xsubject to

    1 2 1 22 30, and 0, 0x x x x

    [15 30]c

    [1 2]A

    1

    2

    x

    x

    x

    [30]b

    4 4

    4 8

    Q

    Solving QP problems

  • 7/24/2019 OD Nonlinear Programming LARGE 2010

    56/78

    Solving QP problems

    Some KKT conditions for quadratic programming

    problems can be transformed in equality constraintsby introducing slack variables.

    KKT conditions can be condensed due to the

    complementary variables, introducing

    complementary constraints. As this, except for the complementary constraints, all

    KKT conditions are linear programming constraints.

    385

    Solving QP problems

  • 7/24/2019 OD Nonlinear Programming LARGE 2010

    57/78

    Solving QP problems

    Using the previous properties, QP problems can be

    solved using a modified simplex method. See example of a QP problem in Hilliers book (pages

    580-581).

    Excel, LINGO, LINDO, Matlab and MPL/CPLEX all can

    solve quadratic programming problems.

    386

    Separable Programming

  • 7/24/2019 OD Nonlinear Programming LARGE 2010

    58/78

    Separable Programming

    Assumed thatf(x) is concave andgi(x) are convex.

    f(x) is a (concave)piecewise linear function (see

    example).

    Ifgi(x) are linear, this problem can be reformulated as

    an LP problem by using a separate variable for each

    line segment.

    The same technique can be used for nonlineargi(x).

    387

    1

    ( ) ( )n

    j j

    j

    f f x

    x

    Example

  • 7/24/2019 OD Nonlinear Programming LARGE 2010

    59/78

    Example

    388

    Example

  • 7/24/2019 OD Nonlinear Programming LARGE 2010

    60/78

    Example

    389

    Convex Programming

  • 7/24/2019 OD Nonlinear Programming LARGE 2010

    61/78

    Convex Programming

    Many algorithms can be used, falling into 3 categories:

    1. Gradient algorithms, where the gradient searchprocedure is modified to avoid violating a constraint.

    Example: generalized reduced gradient(GRG).

    2. Sequential unconstrained algorithms, includepenalty

    function and barrier function methods. Example: sequential unconstrained minimization

    technique (SUMT).

    3. Sequential approximation algorithms, include linearand quadratic approximation methods.

    Example: Frank-Wolfe algorithm.

    390

    Frank-Wolfe algorithm

  • 7/24/2019 OD Nonlinear Programming LARGE 2010

    62/78

    Frank Wolfe algorithm

    It is a sequential linear approximation algorithm.

    It replaces the objective functionf(x) by the first-orderTaylor expansion off(x) aroundx =x, namely:

    Asf(x) and f(x)x have fixed values, they can be

    dropped to give a linear objective function:

    391

    1

    ( )( ) ( ) ( ) ( ) ( )( )

    n

    j j

    j j

    ff f x x f f

    x

    x

    x x x x x x

    1

    ( )

    ( ) ( ) , where at .

    n

    j j jj j

    f

    g f c x c x

    x

    x x x x x

    Frank-Wolfe algorithm

  • 7/24/2019 OD Nonlinear Programming LARGE 2010

    63/78

    g

    Simplex method is applied to find a solutionxLP.

    Then, chose the point that maximizes the nonlinearobjective function along the line segment.

    This can be done using an one-variable unconstrained

    optimization algorithm.

    The algorithm continues the iterations until the stop

    condition is satisfied.

    392

    Summary of Frank-Wolfe algorithm

  • 7/24/2019 OD Nonlinear Programming LARGE 2010

    64/78

    y g

    Init ializat ion:Find feasible initial trial solutionx(0), e.g.

    using LP to find initial BF solution, Setk= 1.Iterat ion k:

    1. Forj = 1, 2, ...,n, evaluate

    and setcj equal to this value.

    2. Find optimal solution by solving LP problem:

    393

    ( )LP

    kx

    ( 1)( ) at .k

    j

    f

    x

    xx x

    1

    Maximize ( ) ,n

    j j

    j

    g c x

    x

    subject toand Ax b x 0

    Summary of Frank-Wolfe algorithm

  • 7/24/2019 OD Nonlinear Programming LARGE 2010

    65/78

    y g

    3. For the variablet [0,1], set

    so thath(t) gives value off(x) on line segment

    between (wheret= 0) and (wheret= 1).

    Use one-variable unconstrained optimization to

    maximizeh(t) to findx(k).

    Stopping rule:Ifx(k1) andx(k) are sufficiently close

    stop.x(k) is the estimate of optimal solution.

    Otherwise, resetk=k+ 1.

    394

    ( 1) ( ) ( 1)

    LP LP( ) ( ) for ( ),k k k

    h t f t

    x x x x x

    ( 1)kx

    ( )

    LP

    kx

    Example

  • 7/24/2019 OD Nonlinear Programming LARGE 2010

    66/78

    p

    As

    the unconstrained maximumx = (2.5, 2) violates the

    functional constraint.

    395

    2 2

    1 1 2 2Maximize ( ) 5 8 2f x x x x x

    subject to

    1 2 1 23 2 6, and 0, 0x x x x

    1 2

    1 2

    5 2 , 8 4f f

    x xx x

    Example (2)

  • 7/24/2019 OD Nonlinear Programming LARGE 2010

    67/78

    p ( )

    Iterat ion 1:x = (0, 0) is feasible (initial trialx(0)).

    Step 1 givesc1 = 5 andc2 = 8, sog(x) = 5x1 + 8x2. Step 2: solving graphically yields = (0, 3).

    Step 3: points between (0, 0) and (0, 3) are:

    This expression gives

    396

    (1)

    LPx

    1 2( , ) (0,0) [(0,3) (0,0)] for [0,1](0,3 )

    x x t tt

    2

    2

    ( ) (0,3 ) 8(3 ) 2(3 )

    24 18

    h t f t t t

    t t

    Example (3)

  • 7/24/2019 OD Nonlinear Programming LARGE 2010

    68/78

    p ( )

    the valuet=t* that maximizesh(t) is given by

    sot* = 2/3. This results leads to the next trial solution,

    (see figure):

    Iterat ion 2:following the same procedure leads to the

    next trial solutionx(2)

    =(5/6, 7/6).

    397

    ( ) 24 36 0dh t tdt

    (1) 2(0,0) [(0,3) (0,0)]3

    (0,2)

    x

    Example (4)

  • 7/24/2019 OD Nonlinear Programming LARGE 2010

    69/78

    p ( )

    398

    Example (5)

  • 7/24/2019 OD Nonlinear Programming LARGE 2010

    70/78

    p ( )

    Figure shows next iterations.

    Note that trial solutions alternate between twotrajectories that intersect at pointx = (1, 1.5).

    This is the optimal solution (satisfy KKT conditions).

    Usingquadraticinstead oflinearapproximations lead

    to a much faster convergence.

    399

    Sequential unconstrained minimization

  • 7/24/2019 OD Nonlinear Programming LARGE 2010

    71/78

    Main versions ofSUMT:

    exterior-pointalgorithm: deals withinfeasiblesolutionsand apenaltyfunction,

    interior-pointalgorithm: deals withfeasiblesolutions

    and abarrierfunction.

    Uses the advantage of solving unconstrainedproblems, which are much easier to solve.

    Each unconstrained problem in the sequence chooses

    a smaller and smaller value ofr, and solves forx to

    400

    Maximize ( ; ) ( ) ( )P r f rB x x x

    SUMT

  • 7/24/2019 OD Nonlinear Programming LARGE 2010

    72/78

    B(x) is a barrier function with following properties:

    1.B(x) issmallwhenx isfarfrom boundary of feasibleregion.

    2.B(x) islargewhenx isclosefrom boundary of feasible

    region.

    3.B(x) as distance from boundary of feasible region

    0.

    Most common choice ofB(x):

    401

    1 1

    1 1( )( )

    m n

    i ji i j

    Bb g x

    xx

    Summary of SUMT

  • 7/24/2019 OD Nonlinear Programming LARGE 2010

    73/78

    Init ializat ion:Find feasible initial trial solutionx(0), not

    on the boundary of feasible region. Setk= 1. Choosevalue forrand< 1 (e.g.r= 1 and= 0.01).

    Iterat ion k:starting fromx(k1), apply a multivariable

    unconstrained optimization procedure (e.g. gradient

    search procedure) to find local maximumx(k) of

    402

    1 1

    1 1( ; ) ( )

    ( )

    m n

    i ji i j

    P r f r b g x

    x x

    x

    Summary of SUMT

  • 7/24/2019 OD Nonlinear Programming LARGE 2010

    74/78

    Stopping rule:If change fromx(k1) tox(k) is very small

    stop and usesx(k)

    aslocal maximum. Otherwise, setk=k+ 1 andr= rfor other iteration.

    SUMT can be extended for equality constraints.

    Note that SUMT is quite sensitive tonumerical

    instability, so it should be applied cautiously.

    403

    Example

  • 7/24/2019 OD Nonlinear Programming LARGE 2010

    75/78

    is convex, but is not concave. Initialization: (x1,x2) =x

    (0) = (1, 1),r= 1 and= 0.01.

    For each iteration:

    404

    1 2

    Maximize ( )f x xx

    subject to2

    1 2 1 23, and 0, 0x x x x

    2

    1 1 2( )g x x x 1 2( )f x xx

    1 2 2

    1 2 1 2

    1 1 1( ; ) 3

    P r x x r x x x x

    x

    Example (2)

  • 7/24/2019 OD Nonlinear Programming LARGE 2010

    76/78

    Forr= 1, maximization leads tox(1) = (0.90, 1.36).

    Table below shows convergence to (1, 2).

    405

    k r x

    1

    k

    )

    x

    2

    k

    )

    0 1 1

    1 1 0.90 1.36

    2 102 0.987 1.925

    3 104 0.998 1.993

    1 2

    Nonconvex Programming

  • 7/24/2019 OD Nonlinear Programming LARGE 2010

    77/78

    Assumptions of convex programming often fail.

    Nonconvex programming problems can be much moredifficult to solve.

    Dealing with non differentiable and non continuous

    objective functions is usually very complicated.

    LINDO, LINGO and MPL have efficient algorithms to

    deal with these problems.

    Simple problems can be solved using hill-climbing to

    find alocal maximumseveral times.

    406

    Nonconvex Programming

  • 7/24/2019 OD Nonlinear Programming LARGE 2010

    78/78

    An example is given in Hilliers book using Excel Solver

    to solve simple problems. More difficult problems can use Evolutionary Solver.

    It uses metaheuristics based on genetics, evolution

    and survival of the fittest: a genetic algorithm.

    We presented several well known metaheuristics to

    solve this type of problems.