Deterministic Optimization and Design

Deterministic Optimization and Design Jay R. Lund UC Davis Fall 2017

1

Deterministic Optimization and Design

Class Notes for Civil Engineering 153

Department of Civil and Environmental Engineering University of California, Davis

Jay R. Lund

February 2016

© 2017 Jay R. Lund

"... engineers who can do only the best of what has already been done will be replaced by a computer system." Daniel C. Drucker

Engineering Education, Vol. 81, No. 5, July/August 1991, p. 480. "Good management is the art of making problems so interesting and their solutions so constructive that everyone wants to get to work and deal with them." Paul Hawken Acknowledgments: These notes have improved steadily over the years from corrections and suggestions by numerous students, TAs, and substitute instructors. Particularly useful improvements have come from Ken Kirby, Orit Kalman, Min Yip, Stacy Tanaka, David Watkins, and Rui Hui, with many others contributing as well.


2

Table of Contents Some Thoughts on Optimization ................................................................................................ 3 Introduction/Overview ................................................................................................................ 5 Design of A Bridge Over A Gorge ............................................................................................. 7 Systems Analysis ...................................................................................................................... 10 Steps to Solving a Problem Using Optimization ...................................................................... 12 Unconstrained Optimization Using Calculus ............................................................................ 16 Analytical vs. Search methods .................................................................................................. 21 Lagrange Multipliers ................................................................................................................. 24 Alternative Method for Inequality Constraints ......................................................................... 31 Mathematical Programming ...................................................................................................... 33 Karush-Kuhn-Tucker (KKT) Conditions for an Optimal Solution ........................................... 36 Linear Programming ................................................................................................................. 37 The Simplex Method: A Graphical Interpretation ................................................................... 41 The Simplex Tableau ................................................................................................................ 42 The Simplex Method - An Algorithm ....................................................................................... 44 Some Simplex Details ............................................................................................................... 48 Implementing the Big-M Method ............................................................................................. 52 Sensitivity Analysis in Linear Programming ............................................................................ 54 Duality....................................................................................................................................... 58 Example optimization: California Hydropower ........................................................................ 61 Transportation Problems ........................................................................................................... 61 Scheduling Problems ................................................................................................................ 63 LP Solution of Non-Linear Objectives with Linear Constraints .............................................. 65 Integer Programming and Mixed Integer-Linear Programming ............................................... 67 Summary of the BIP Branch and Bound Algorithm ................................................................. 69 More Non-Linear Objective Functions with Linear Constraints Using Integer-Linear Programming............................................................................................................................. 72 Computational Aspects ............................................................................................................. 73 Dynamic Programming (DP) .................................................................................................... 74 More DP Applications .............................................................................................................. 81 Non-Linear Programming ......................................................................................................... 84 Genetic Algorithms/Evolutionary Algorithms .......................................................................... 93 The Final Days ........................................................................................................................ 100 Appendices .............................................................................................................................. 101 PROBLEM FORMULATION: SOME GUIDELINES ......................................................... 103 INTERPRETATION OF RESULTS: SOME GUIDELINES ................................................ 104 INTRODUCTION TO SOLVING LINEAR PROGRAMS USING LINDO ........................ 105 THE SIMPLEX ALGORITHM .............................................................................................. 106 Selected Bibliography ............................................................................................................. 107


3

Some Thoughts on Optimization "All models are wrong, but some are useful." - G.E.P. Box (1979), Robustness in Statistics. "Operations research is the art of giving bad answers to problems to which otherwise worse answers are given." - T.L. Saaty (1959), Mathematical Methods of Operations Research, p. 3. "What would life be without arithmetic, but a scene of horrors." - Revd. Sydney Smith, 1835 “The purpose of mathematical programming is insight, not numbers” – A.M. Geoffrion 1978 "Decision analysis separates a large-scale problem into its sub-parts, each of which is simpler to manipulate and diagnose. After the separate elements are carefully examined, the results are synthesized to give insights into the original problem." - Harvey Wagner (1975), Principles of Operations Research. "The adoption of operations research calls for an act of faith in the potential benefits of a systematic approach to decision-making." - Harvey Wagner (1975), Principles of Operations Research. "Even when quantitative analysis is of central importance for a managerial decision process, an operations-research-oriented system never supplies all the information required for action, no matter how sophisticated the system's design. Furthermore, a truly successful implementation of an operations research system must apply behavioral as well as mathematical science, because the resultant system must interact with human beings. And finally, the very process of constructing an operations research system involves the exercise of judgement in addition to the logical manipulation of symbols and data." - Harvey Wagner (1975), Principles of Operations Research. "Le mieux est l'ennemi du bien." The best is the enemy of the good. - Voltaire “A great craftsman does not put aside the plumb-line for the benefit of a clumsy carpenter.” Mencius (China, c. 300 B.C., Book VII, Part A, 41) "Life must be lived forwards, but it can only be understood backwards" - Soren Kierkegaard "A man's gotta know his limitations." - Clint Eastwood "A problem well put is a problem half solved." - Anon. “The bottom line for mathematicians is that the architecture has to be right. In all the mathematics that I did, the essential point was to find the right architecture. It's like building a bridge. Once the main lines of the structure are right, then the details miraculously fit. The problem is the overall design.” Freeman Dyson, "Freeman Dyson: Mathematician, Physicist, and Writer" interview with Donald J. Albers, The College Mathematics Journal, vol 25, no. 1, January 1994. “An expert problem solver must be endowed with two incompatible qualities, a restless imagination and a patient pertinacity.” Howard W. Eves in Mathematical Circles, Boston: Prindle, Weber and Schmidt, 1969. “The errors of definitions multiply themselves according as the reckoning proceeds; and lead men into absurdities, which at last they see but cannot avoid, without reckoning anew from the beginning.” Thomas Hobbes, in J. R. Newman (ed.) The World of Mathematics, New York: Simon and Schuster, 1956.


4

“In order to translate a sentence from English into French two things are necessary. First, we must understand thoroughly the English sentence. Second, we must be familiar with the forms of expression peculiar to the French language. The situation is very similar when we attempt to express in mathematical symbols a condition proposed in words. First, we must understand thoroughly the condition. Second, we must be familiar with the forms of mathematical expression. George Polyá, How to Solve It. Princeton: Princeton University Press. 1945. People are often unhappy with complex system designs, but other choices are often worse.

xkcd.com Solving by enumeration "They always bury [treasure] under a haunted house, or on an island, or under a dead tree that's got one limb sticking out. Well, we've tried Jackson Island a little, and we can try it again some time; and there's lots of dead-limb trees - dead loads of em. Is it under all of them? How you talk! No! Then how you going to know which one to go for? Go for all of em. Why, Tom, it'll take all summer." - Mark Twain, The Adventures of Tom Sawyer, 1876.


5

Introduction/Overview What is "Deterministic Optimization"? 1) Optimization = A process of finding the "best" solution or design to a problem 2) Deterministic = Problems or systems that are well known; assume no probability or uncertainty Optimization: - finding the "best" solution or design to a problem 1. What do we mean by the "best"? cost, performance, aesthetics, social or individual "well-being", etc. The objective implied by "best" will vary with the problem and designer/client. Example: The "best" use of Mono Lake for L.A. might be to let it dry up. The "best" use of Mono Lake for Davis is for recreation. 2. Why can't Mono Lake serve both intensive water supply and recreation? The use of Mono Lake is constrained by: 1) limited water availability 2) the ecosystem of Mono Lake Can't make everybody perfectly happy, but have to do the "best" within constraints. Optimization selects the "best" decision from a constrained situation. Obviously, the "best" solution is driven by your objectives for solving a problem. A practical insight from optimization, enumeration theory:

xkcd.com


6

Course Mechanics Everyone needs computer programming for this course. Use whichever you are most comfortable with.

Get comfortable with one way to program, you'll be using it a lot! -) Use of the computer for this class: after ~3rd week you will be using it a lot. -) Use the computer to help you solve problems for other classes. Especially problems with trial-and-

error solutions. "Computers should work; people should think." -- But before this can happen you have to become familiar with getting computers to do your calculations.

-) Use of a computer language is required for this course (FORTRAN, BASIC, PASCAL, C, Visual Basic, MATLAB, etc., whatever you are most comfortable with). This is unavoidable for Project No. 2.

For those uncomfortable with programming, I recommend Visual Basic for Applications tutorials on the class web site, and elsewhere.

-) Weekly Homework or Projects (eight total) Homework dwells more on techniques and methods. Projects make you think a problem through more and so have more individual answers, and

generally require more work. Almost all assignments will require use of a computer. -) One mid-term -) One final The article passed out gives a useful philosophical perspective on this class, and how this class

should fit into your larger education as an engineer. Course Objectives: Classical Optimization Theory Formulates All Problems into the form:

Maximize or Minimize: an objective function Subject To: a set of mathematical constraints

This becomes the standard form for all problem statements for this class. The objectives of this class are:

a) to help you formulate engineering and other problems in this format,

b) to give you some mathematical and numerical tools for solving these problems,

c) to give you some hints as to how to interpret your results, which are often solutions of simplified versions of the real problems,

d) to provide you with opportunities to become more comfortable with the use of computers for solving problems, and

e) to demonstrate the potential value and limitations of these techniques in engineering practice.


7

Design of A Bridge Over A Gorge

We want to build a bridge to span a gorge.

How would we know a "good" bridge design? How would we know the "best" bridge design? By what process could we select a good design, or "the best" design? Let's formulate the problem into objectives, decisions, and constraints.

What objectives could we have for the design? How can we quantify these objectives? 1) minimize cost ($) 2) maximize traffic volume (capacity in vehicles/hour) 3) maximize reliability of bridge/ minimize the probability of structural failure (failure due to

earthquakes, floods, bombs, etc.) 4) make it pretty (hard to quantify, but often very important) 5) make it not cause flooding upstream (backwater effect) (Expected upstream flood damage) 6) Make it easy and safe to build. Who might choose some of these more specific objectives? Decisions: What types of design decisions can the engineer make? 1) Geometry of bridge 2) Material of bridge 3) Size/capacity of bridge Constraints: What factors constrain the design of any bridge over the gorge? -) Geometry of gorge (distance to be spanned and depth of gorge) -) Geology of gorge (ability to support bridge; earthquakes) -) Hydrology and hydraulics of gorge (floods, scour, etc.) -) Structural constraints (properties of materials, conservation of momentum, statics, dynamics) -) Budget? These constraints and objectives are all very complex. How could we ever hope to write complete

mathematical descriptions of it all? (We can't.)

High Water

Low Water

Gorge Width

Gorge Depth


8

Problem simplification Since we cannot solve the real problem in all its complexity, we'll have to simplify. Some ideas for

simplification: 1) Reducing the number of objectives: Can some objectives be formulated as constraints? Are some objectives accommodated by any

design or relatively unimportant, and can therefore be left out? 1) Bridge must cost < some budget amount. 2) Traffic capacity > some level of traffic demand. 3) Probability of structural failure < some annual reliability level. 4) It's cheap to make it pretty so we won't worry about this now. 5) If nothing valuable is upstream, any backwater is unimportant. If something very valuable is upstream, allow no backwater. 2) Simplifying the Constraints -) The different types of bridge geometries would be a mathematical nightmare. Let's look at only a

few types of bridge that are commonly successful for such gorges, optimize the design of each type for this gorge, and then compare the "best" of each bridge type to select the one bridge we'll build.

-) Can the mathematics of any constraints be approximated by a simple mathematical equation or

inequality? Formulation for a Simple Bridge Type: A simple geometry of spans supported by pilings.

Simplified Decisions: How many pairs of pilings and how many spans should be used? Simplified Objective: Minimize total construction cost, CT 1) Cost of a pair of pilings = a given function of compressive force, fp(F) Cp = fp(F), F = decision of compressive force on a single piling pair 2) Cost of span = a given function of its length, fs (L): Cs = fs (L), L = decision of length of each span. 3) If N pairs of pilings are used, N+1 spans will be needed. Therefore, total cost is approximated by: CT = N Cp + (N+1) Cs, N = decision of number of piling pairs

Spans

Pilings


9

4) Static design: The load on each piling pair is the weight of a span and the live load (vehicles). This

load is a function of the span's length, where we'll assume w is a constant. F = wL 5) The width of the gorge must be spanned (N + 1) L = g, g = gorge width (another constant)

or L = g

N + 1

Only one "decision variable" remains! Now, Minimize Cost for either L, F, or N and the other 2 values are known Reduce the known objective function from Step 3 to one independent decision variable. CT = N Cp + (N+1) Cs goes to: CT = N fp(F) + (N+1) fs(L) goes to: CT = N fp(wL) + (N+1) fs(L) which finally goes to:

CT = N fp

w g

N + 1 + (N+1) fs

g

N + 1

where the entire cost function is in terms only of the number of piling pairs N. What's the optimal N? Solution Methods 1) Trial and Error: Try different values of N and see which gives the lowest cost. 2) Enumeration: Try all or many values of N and see which gives the lowest cost. 3) Other more sophisticated methods (we'll learn later). Let's solve by enumeration. Solve the boxed equation for all reasonable values of N, and plot the results:

Number of Spans (N+1) This is a simple problem, but it illustrates the approach. The course presents a mathematical theory of optimal design. Even if the details of the theory will

sometimes be hard to implement, the approach should lead to better, if not "the best" designs. For some this will represent a new way of thinking about engineering problems.

1 2 3 4 5 6 7 8 9 10

Total Cost

($millions)


10

Systems Analysis Last time I tried to give a flavor of what is meant by "deterministic optimization." Things I'd like to do today: 1) Place this course in the larger context of Systems Analysis and Operations Research 2) Discuss the role of Operations Research in civil engineering: real cases where it's used. 3) Suggest a general approach to applying optimization to civil engineering problems. To codify what we did with the example last time and illustrate with an example. Systems Analysis/Operations Research Broadly, the use of mathematical descriptions of problems to help find better solutions. Formal origins: World War II, logistics and combat tactics problems. Utility of computers! (to test many solutions quickly and to implement tedious algorithms) Current applications in business (MBAs), military, engineering. EXCEL spreadsheets include optimization “Solve” with some elementary linear programing, nonlinear-programming, and evolutionary algorithm capabilities. Systems Analysis is more than just a bag of tricks and tools!

Systems and/Operations Research as a problem-solving discipline! A way of thinking about problems!

Optimization by any other Name, ...: Optimization goes under a number of different names and constitutes an important part of many fields. Here are some: Systems Analysis Operations Research (British: Operational Research) Industrial Engineering Management Science Engineering Economics.

I tend to use the term systems analysis, since it is fairly broad and to me it connotes a systematic analysis of design decisions. Here is a nice talk by David Rosenberg on optimization and the value of near-optimization. https://www.youtube.com/watch?v=b96LsqcJyHM


11

Some Uses of Systems Analysis in Civil Engineering (skipped in lecture) Water Resources Operating reservoirs, pipelines, and pumping plants (MWD of So. Cal., PG&E, Army COE) Determining optimal maintenance schedules (dredging, water meters, etc.) Optimal pumping schedules (groundwater and surface water) Scheduling of water conservation and supply facilities (San Diego County Water Authority) Optimal use of water purchases (San Diego County Water Authority) Most efficient hydraulic cross-section (textbook hydraulics) Environmental Engineering: Least-cost treatment levels for wastewater plants on a river subject to water quality constraints Least-cost sequencing of wastewater treatment processes Least-cost planning of solid waste recycling Locating transfer stations for solid waste collection Determining collection routes for garbage trucks Selecting a most profitable mix of recycled products Least-cost design of material recovery facilities Structures: Designing structures that accommodate loads with minimum weight Design of least-cost or minimum-weight structural cross-sections Geotechnical: Least-cost selection of aggregate sources for an aggregate mix Least-cost disposal of dredged material Transportation: Assigning aircraft to scheduled routes Assigning railroad cars to routes Selecting cargos to fit on a ship Traffic flow optimization and signal timing Routing ships, aircraft, trucks, and rail cars Construction: Selecting least-cost or completion-time-minimizing schedule and sequence of construction tasks to finish a

project (Critical Path Method - CPM) Least-cost cut and fill Construction equipment allocation between projects


12

ProblemFormulation

ProblemSolution

Interpretation andConclusions

1) Define the Problem

2) Express the Problem Mathematically

3) Simplify the Mathematical Problem to a Solvable Form

4) Solve the Simplified Mathematical Problem

5) Sensitivity Analysis

6) Interpretation of Results

7) Recommendations for Further Study or Implementation

THE MATHEMATICAL OPTIMIZATION PROCESS


13

Steps to Solving a Problem Using Optimization Here is a step-by-step approach to solving design problems with optimization. Hopefully, you'll find it

useful in approaching some of your homework problems and projects. 1) Define the Problem (in words) a) What decisions are to be made? b) What is an objective(s)? c) What are the constraints? 2) Express the Problem Mathematically (Simplification Step 1) a) Identify decision variables. b) Define objective function(s). c) Express constraints mathematically. 3) Simplify Mathematical Problem Statement to a Solvable Form (Simplification Step 2). Some Techniques: a) Express some objectives as constraints. b) Eliminate "minor" aspects of the problem. c) Adopt approximate expressions for objectives and constraints. d) Substitute equality constraints into the objective function. 4) Solve the Simplified Mathematical Problem Statement (using various methods presented in

the course) a) Analytical solution methods (Calculus) b) Search Methods c) Mixed Analytical and Search Methods (Efficient Search Methods) 5) Sensitivity Analysis No solution is perfect. Simplifications and assumptions were made. -) How stable is the solution if problematic assumptions are varied? -) Effects of uncertainties on the optimal solution and system performance? 6) Interpretation of Results This is what people pay for. The challenging part of this step is to reveal the design implications

of your solution, given that you have had to simplify the real problem before you could solve it. The solution to the math problem is not always the solution to the real problem. Sensitivity analysis and bit of thought can help a lot here.

7) More Analysis and/or Recommendations for Implementation?

Is more analysis needed? What recommendations would you make for implementing your conclusions? Are there particular aspects of implementing the design that merit more attention?


14

Example: Task Scheduling (Critical Path Method - CPM), 5-tasks in project. Background: Task Duration (weeks) Pre-requisite Activities 1) Excavation 2 None 2) Building Construction 10 1 3) Internal Finishing 4 2 4) Parking Lot 1 1 5) Landscaping 3 2,4 Step 1: Define Problem a) Decisions: At what time should each task be begun? (Schedule tasks) b) Objective: Min. time-to-completion of project. c) Constraints: Some tasks are pre-requisites of other tasks. Tasks also have durations. Step 2: Mathematical Statement a) Let Si = starting week of task i. i = index referring to activity Si are decision variables b) Min T = date of completion c) Constraints: S2 S1 + 2 T S1 + 2 S3 S2 + 10 T S2 + 10 S4 S1 + 2 T S3 + 4 S5 S2 + 10 T S4 + 1 S5 S4 + 1 T S5 + 3 Pre-requisites Definition of completion date (Completion occurs after every task is done) Step 3: Simplification - No more simplification is needed. Step 4: Solve - Today we will solve by common sense 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16

(Scheduling tens or hundreds of tasks is more easily done by computer.)

Excav. (1) Building Construction(2) Internal Finish(3) " Critical path"

ParkingLot (4) Slack Landscaping(5) Slack

discuss


15

Step 5: Sensitivity Analysis The durations of each activity are always somewhat uncertain. Maybe Fred's bulldozer

breaks down and takes an additional week to fix. Task Change in Tmin with small change in task duration 1 1 2 1 These activities are the "critical path" 3 1 4 0 5 0 Step 6: Interpretation 1) Do activities in the order 1, 2, and 3. Schedule activity 4 for anytime between weeks 2 and

11. Schedule activity 5 for either week 12 or 13. 2) Any "rush" of the project should concentrate on Tasks 1, 2, and 3. 3) No need to hurry on the parking lot. 4) Activity 5 should be begun within a week of the beginning of activity 3, or the completion

of the entire project will be delayed. 5) No need to hurry on landscaping unless Task 3 is accelerated by more than a week.


16

Unconstrained Optimization Using Calculus Now we'll start with the presentation of algorithms for solving mathematical statements of deterministic optimization problems. Let's begin with some definitions that will be used all quarter. Maxima and Minima Definitions

-) Local versus global optima -) Maximum versus minimum Note: A local maximum can be less than a local minimum. It is typically necessary to look at all "extreme" points (maxima or minima) to find the global optimum. Differentiability When solving an optimization problem by calculus, the objective function must be "piece-wise" continuous, and twice differentiable;

dfdx and

d2fdx2 must exist over the range of x.

Finding Unconstrained Maxima and Minima For Objective Functions of One Decision Variable Let xo be some particular value of x that we want to examine for optimality. First-Order Condition: (Necessary Condition)

0xdx

dz0 , evaluated at xo

Several values of x might satisfy this condition. Points satisfying this condition might be a maximum or a minimum. Points satisfying this condition are called critical points. Extreme points can also exist at discontinuities in the objective function and boundaries of the feasible region.

Z = f(X)Z = objective valueX = decision variable

Global Maximum

Z

Local Min

Local Maximum

Local MinimumGlobal Minimum

Local Max

X


17

Second-Order Condition:

If, in addition to the first-order condition, 0

xdx

zd

02

2 , the point xo is a maximum.

The change in dz/dx is negative at x. Therefore, z must decrease with further increase in x, making the point a maximum.

Necessary and Sufficient Conditions for Xo to be a Maximum

1) dzdx

xo

= 0

and

2) dz2

dx2 xo

< 0

Necessary and Sufficient Conditions for Xo to be a Minimum

1) dzdx

xo

= 0

2) d z2

dx2 xo

> 0

If, in addition to the first-order condition, d2zdx2 = 0, the point xo is on a plateau, with several equally-

optimal values of x. How does a blind man find the top or the bottom of a hill?


18

Finding Unconstrained Optima for Objective Functions of Several Decision Variables Now, let the objective function be a function of a vector of decision variables: z = f x , x = x1, x2, - - -, xn , a vector of n decision variable First-Order Condition: (Necessary Condition)

f xo =

f x o

x1

f x o

x2

f x o

xn

= 0

This is the same condition as before. The partial derivative of the objective function with respect to each decision variable must be zero. This is a necessary condition for a hill-top or a hole, no matter how many dimensions the geometry has. Just using these first-order conditions will greatly reduce the number of potentially optimal decisions that need to be evaluated. Second-Order Condition: The second-order conditions look a bit stranger, however. For a minimum:

0

If this inequality holds, H is calleda "Positive Definite" or “PositiveSemi-definite” matrix

Hessian Matrix

for all( )Xn

TH Xi

o Xn 0

Xn

T = arbitrary vector with n elements

Xn

This is how the Hessian matrix is defined:


19

H xo =

2f xo

x12

2

f xo

f1 f2

- - - 2

ff xo

x1 xn

|||||

|||||

2 xo

xn x1

2

f xo

xn x2

- - - 2

f xo

xn2

(

(

((

f( ( (

)))

)

) ))

x x

H positive definite if Xn T

HXn > 0.

H positive semidefinite if XnTHXn 0.

This is sometimes called a saddle point.

X 2

1X

•

f(X)

X2

1X

f(X)

•


20

Example

MIN f X = X1-1 2 + X22 + 1

f X = 0

0 =

f Xo

X1

f Xo

X2

= 2 X1-1

2X2

X1 = 1, X2 = 0.

H Xo

2f X

o

X12

2

f Xo

X1X 2

2f X

o

X2X1

2

f Xo

X22

= 2 00 2

Is H positive definite?

TiX

H iX

0 for all X 0

X1,X2 2 0

0 2 X1

X2

> 0?

2X1,2X2

X1

X2

> 0?

2X12 + 2X2

2 >0, for any X 0

H is positive definite.

Second-Order Condition for a Maximum:

0X

represents a maximum if f( 0X

) = 0 and 0X

H is "negative definite" or "negative semi-definite"

H( 0X

) is "negative definite" if X

T H X

< 0 for all X

0 (What a mess to calculate!) (But these are rigorous rules!)

Note: 0X

can satisfy f( 0X

) = 0 without being a minimum or a maximum, by being a "saddle point," or an unstable minimum-maximum. (i.e., min w.r.t. X1 & max w.r.t. X2). We won't use Hessians hardly at all.


21

Analytical vs. Search methods We have already seen some limitations to completely analytical methods: 1) often time-consuming to analyze 2) does not distinguish between global and local optima Still, analytical methods can sometimes easily solve problems, give more insight into the behavior of the optimal solution, and help simplify the solution of the problem to speed numerical searches. Example: Optimal Container Dimensions -) A cylinder must contain a given volume of oil, V. -) Find the height and radius of the cylinder that will minimize the cylinder's surface area.

Min S = 2rh + 2r2

S.T. V = r2h h = Vr2

Min S = 2r V

r2 + 2r2

or Min S = 2V

r + 2r2 . 1st-order condition:

dSdr = 0 0 =

-2V

r2 + 4r

0 = 4r3 - 2V V

23

= r*

2nd-order condition:

d2S

dr2 =

4V

r3 + 4 > 0

V

r3 > -

If V = 10ft3, r* = 1.167 ft from 1st-order condition, and V

r3 > - , satisfying 2nd-order condition for

a minimum of S.


22

Example 2: Optimal Scheduling of Water Meter Maintenance A water meter is used to charge a large industrial customer for water use. The water utility must decide how often to repair the meter, given that the accuracy deteriorates with time. The meter always deteriorates such that it registers less than the actual flow amount. The average yearly volume used is 10,000 x 100 cu.ft. (ccf). The price of water is 50¢/ccf. The cost of repairing the meter (which restores accuracy) is $500. The meter loses 1% of its remaining accuracy each year. Find how often the utility should repair the meter to minimize present value of total repair and revenue-loss costs. The interest rate is 5%. The present value of costs and losses over one period between repairs of length T is

C(T) = 500 + 0.5 0

T 1 0,000 ( )1 - e-0.01t e-0.05t dt

= 500 + 5,000

1

0.05 ( )1 - e-0.05T - 1

0.06 ( )1 - e-0.06T

C(T) = 500 + 5,000

3.333 -

e-0.05T

0.05 + e-0.06T

0.06

An infinite series of these repairs, occurring at intervals T, has a present value cost of:

PV(T) =

0

)(05.0)(k

kTeTC

This infinite series simplifies to PV(T) = C(T) + PV(T)e-0.05T, which further simplifies to:

PV(T) = C(T)

1 - e-0.05 T

The objective is to minimize this present value cost.

Min PV(T) = C(T)

1 - e-0.05 T

PV(T) = 500

1-e-0.05T + 100,000 - 83,333 1-e-0.06T

1-e-0.05T

Cost

timeT 2T 3T 4T …0


23

Solution by search/enumeration:

T*T

PVPlot

and select T.*

Change-out Period Present Value Cost

A B 1 10742.8 2 6229.3 3 5036.6

3.5 4796.0 4 4665.69

4.5* 4614.94* 5 4616.98

5.5 4656.7 6 4723.9

6.5 4811.9 7 4915.4 8 5155.8 9 5425.5 10 5712.6 11 6009.8 12 6312.3

Solution by Calculus:

1st order condition: Solve: dPV(T)

dT = 0

0.010.06

e-0.06T -e-0.01T = 0.05 500

5,000 - 0.05

0.06

0.167 e-0.06T -e-0.01T =-0.8283

Here, the first-order condition needs a numerical solution. T LHS 10 -0.8132 9 -0.8166 7 -0.8227 5 -0.8275 4.5 -0.8285 T* 4.5 years 4 -0.8294


24

Lagrange Multipliers A method of solving simple constrained optimization problems of the form: Max or Min f(X1, X2, ..., Xn) S.T. g1(X1, ..., Xn) = b1 g2(X1, ..., Xn) = b2 • • • gm(X1, ..., Xn) = bm Example: Max z = X1 + X2

S.T. 2 X12 + X22 = 1 One way to solve these problems is to use substitution to incorporate constraints directly into the objective function. This is not always possible. Also, substitution will not work for inequality constraints. Solving problems with a single constraint

Step 1: State Problem:

Max or Min Z = f X1, ..., XnS.T. g1 (X 1, ..., Xn) = b

Step 2: Form the "Lagrangian":

"Lagrange Multiplier",

Treat like just another variable.

Step 3: Treat the Lagrangian function like any other function to be maximized or minimized, adding as another decision variable. a) Solve for the first-order conditions:

L

Xi

= 0, Vi

L

= 0 (returns constraint)

b) Determine if solutions to first-order conditions are maxima or minima. -) Using Hessian matrices or evaluation of the objective function at each point. -) Select "best" solution

Step 4: Interpret , the Lagrange Multiplier

= f max or min

b

is the change in the best value of the objective function, if the constraint b is changed.

)),...,((),...,( 111 nn XXgbXXfL


25

A simple example: Step 1 Max f = X1 + X2

S.T. 2X12 + X22 = 1 Step 2 L = X1 + X2 + (1 - 2X12 - X22) Step 3

L

X1

= 1 - 4X1 = 0

(1)

L

X2

= 1 - 2X2 = 0 (

(2)

L

= 1 - 2X1

2 -X22 = 0

(3)

(1) = 1

4X1

(2) = 1

2X2

(1) & (2) 4X1 = 2X2 2X1 =X2

into (3) 1 =2X12 + 2/x1

2 X1 = 16

f = 16

+ 26

= 36

62

X2 = 26

= 3

2 6

2 6

X

Step 4:

= fmax

b = 3

2 6

fmax = b, for small b.

Raising b from 1 to 2 should raise fmax by 3

2 6.

is called a Lagrange multiplier, shadow price, or dual variable.


26

Solving Problems with Several Equality Constraints Step 1 Max or Min f(X1, X2, ..., Xn) S.T. g1(X1, ..., Xn) = b1 g2(X1, ..., Xn) = b2 • • • gm(X1, ..., Xn) = bm Step 2 Form Lagrangian

L = f X1,..., Xn +1 b1 - g1 X1,..., Xn

+2 b2 - g2 X1,..., Xn • • •

+m bm - gm X1,..., Xn Step 3 Solve for Xi and j

iX

L

= 0, i, there are n of these

L

j

= 0, j, there are m of these

This implies simultaneous solution of (n + m) equations. So this method is usually used on only small problems.

Step 4:

j = fmaximum

bj

same interpretation as before

For the skeptical reader, a derivation of the Lagrange multiplier method appears in an appendix.


27

Inequality Constraints Inequality constraints are very common. Given an inequality constraint; g(X1,..., Xn) b: The constraint is called non-binding if g(X*, …, Xn*) > b The constraint is binding if g(X*, …, Xn*) = b For a problem: Max f(X1, X2) S.T. g1(X1, X2) b1 g2(X1, X2) b2

L = f(X1,X2) + 1(b1-g1( X

)) + 2(b2-g2( X

)) If constraint 1 is non-binding, 1 = 0, then no small change in b1 will affect the "best" decision, removing the constraint from the Lagrangian. If constraint 2 is binding, 2 0, and the constraint remains in the Lagrangian. Making inequality into equality constraints: gj(X1,..., Xn) bj becomes gj (X1,..., Xn) + Sj = bj, where Sj is a slack variable. Sj = 0 if constraint is binding.

If Sj0,

j=0. "Complementary Slackness" Sj j = 0 .

We'll use this technique later as well. Example: Multiple inequality constraints Step 1: Max Z = X1 + X2

S.T. X12 + X22 25 X1 4 Step 2: Form the Lagrangian. L = X1 + X2 + 1 (25 - X12 - X22) + 2 (4 - X1) Step 3: Solve the Lagrangian for the first-order conditions, both for the Xis and the js.

Solve: (1) 1X

L

= 0 = 1 - 2 1X1 - 2

(2) 2X

L

= 0 = 1 - 2 1X2

X

X

1

2

12X + X = 252

2

1X = 4


28

(3) L

1

= 0, and

(4)L

2

= 0.

May go away, depending on value assumed

for 1 and 2

FOUR cases exist for the possible sets of Lagrange Multiplier values. For n inequality constraints, there are 2n cases. 1) 1 = 0 , 2 = 0 Solution: Z , X1 , X2 , but violates both constraints. 2) 1 = 0, 2 0: Solution: Z , X1 = 4, X2 , but violates 1st constraint. 3) 1 0, 2 = 0: 1st constraint binds; 2nd constraint is non-binding. There are two solutions for this case:

Solution 1: Z = 7.07, X1 = X2 = 252 3.54. OK with both constraints;

Solution 2: Z = -7.07, X1 = X2 = - 252 . OK with both constraints, but solution is a minimum.

4) 1 0, 2 0: Both constraints bind, OK with both constraints; Two solutions: Solution 1: Z = 7, X1 = 4, X2 = 3, Solution 2: Z = 1; X1 = 4, X2 = -3. FOR EACH CASE: 1) Solve for values of the decision variables which satisfy the first-order conditions. 2) Eliminate solutions that violate the constraints. 3) Select from remaining solutions, the solution X1*, X2*, with the best value of Z. 4) From all the cases, select the solution with the best value of Z. For many problems there are often several points that satisfy the 1st-order conditions of the Lagrangian, especially when there are inequality constraints. To solve these problems, solutions for the 1st-order conditions are examined systematically, by a search procedure to find the "best" solution.


29

Example Step 1 Max Z = 5X1 + 3X2 + 6X1X2

ST 1) X1 + X2 6 2) X1 7 3) X2 3 4) X1 0 Step 2 L = 5X1 + 3X2 + 6X1 X2 + 1 (6 - X1 - X2) + 2 (7 - X1) + 3 (3 - X2) + 4 (0 - X1) Step 3: For each combination of binding and non-binding constraints. (24 or 16 cases for this example) 1) Solve for X1 and X2

2) Eliminate solution if a solution X10, X20 violates a constraint that was assumed to be non-binding. Select the optimal solution from those remaining. (Optimal solution has best objective function value.)

iX

L

= 0, j

L

= 0, if j 0.

1X

L

= 0 = 5 + 6 X2 – 1 – 2– 4

2X

L

= 0 = 3 + 6 X1 – 1 – 3

1

L= 0 = 6 - X1 - X2, if 1 0

2

L= 0 = 7 - X1, if 2 0

3

L= 0 = 3 - X2, if 2 0

4

L= 0 = X1, if 4 0

Cases: At least 8 different solutions: 1) 1 0, 2 0, 3 0 2) 1 0, 2 0, 3 = 0

3) 1 0, 2 = 0, 3 = 0 4) 1 0, 2 = 0, 3 0

5) 1 = 0, 2 = 0, 3 0 6) 1 = 0, 2 0, 3 0

7) 1 = 0, 2 0, 3 = 0 8) 1 = 0, 2 = 0, 3 = 0

There are 16 cases if we consider 4 too. For a system of m inequality constraints, there are 2m cases. (You can see how this can get ugly.) Case 8: No constraints bind

Feasibleregion

(2)

70 X1

6

(1)3

6

(3)

X2

(4)


30

Solution A:

L

X1

= 0 X2 = -56

L

X2

= 0 X1 = -36

= -12

Z = -2.5 + -2.5 + 5/2 2.5 - -2.5

violates constraints 3 and 4

a saddle-point solution2.5 = -2.5

Solution B: not found using derivatives

X1 = X2 = Z =


The solution if no constraints were to bind.

Solutions C & D:

X1 = - X1 = + X2 = X2 = -

Z = -


Solution E:

X1 = X2 = - Z = -


If we go along this route, for all the cases, we might finish this example by the end of the course. So, we'll have to think a bit to shorten this process. Note: If X2 = 0, then 0 X1 6, using all constraints. And, if X1 = 0, then 3 X2 6, using all constraints.

Therefore, Constraint 2 will never be binding. 2 = 0

Therefore, we only need to worry about cases where 2 = 0. Consider only cases 3, 4, 5, and 8. Case 8 is already eliminated since all of its solutions violate constraints. Consider only cases: 3, 4, and 5. Now, look at feasible region, bounded by constraints. Probably 1 0 and (40 or 30).

To shorten the search process.

1X

L

= 0 = 5 + 6X2 – 1- 4


31

2X

L

= 0 = 3 + 6X1 + 1- 3

If 10: X1 = 6 - X2

30: X2 = 3

X40: X1 = 0

If 10 & 30: X2 = 3, X1 = 3 Z = 60

10 & 40: X2 = 6, X1 = 0 Z = 18

If 10 and 3 = 4 = 0: By substituting binding constraint 1 into objective function: Z = 5(6 - X2) + 3X2 + 6(6 - X2) X2

= 30 - 5X2 + 3X2 + 36X2 = 6X22

= 30 + 34X2 - 6X22

Z

X2

= 0 = 34 - 12 X2 X2 = 3412

= 2.83 which violatesX2 3.

This means there is no hill or valley for the plot of Z(X2) within the feasible range of X2. Let's plot this and see what it looks like.

3 X2 6 There is no max or min along the segment, except at the ends. Therefore, Z*max = 60, at X1* = 3, X2* = 3.

Alternative Method for Inequality Constraints Add (for ) or subtract (for ) a squared slack variable, and treat this slack variable like any other in the problem. (Reference?) Advantage: Avoids considering multiple cases. Disadvantages: More variables to solve for; the new variables are non-linear. Example: Step 1: Max Z = X1 + X2

S.T. X12 + X22 + X32 = 25

X1 + X42 = 4

Z


32

Now, instead of 4 unknowns, we have 6 unknowns (4 Xis and 2 js). This isn’t appealing if we have to solve by hand, but it’s not a problem if we let a computer do the work.


33

Mathematical Programming Our solutions of constrained optimization problems fall in the field of Mathematical Programming. A mathematical program is any method used to solve a problem of the form:

Max or Min fX

S.T. g X b any number of equality & inequality constrain

A linear program solves problems of the form:

0

.s.t

Max or Min T

x

bAx

xc

-) Linear objective function: f(X) = c1X1 + c2X2 + ...

-) Linear constraints: ai1X1 + ai2X2 + ... < bi , i = 1, …, m

-) There are linear programming methods that quickly and easily solve such problems. Other mathematical programs are called non-linear programming problems. Most texts study Lagrange multipliers as a non-linear program solution method. In fact, Lagrange multipliers are at the heart of the method for solving linear programs as well! Some terminology: -) A non-binding constraint is also called an inactive or slack constraint. -) A binding constraint is also called an active or tight constraint.

-) The Lagrange multiplier is also known as a shadow price. = b

fbest

Constraint

Lagrange multiplier, shadow price

Binding, tight, active

0

Non-binding, slack, inactive

= 0


34

Convex and Concave Functions (usually objective functions)

Convex and Concave Sets (usually constraint sets for feasible regions) Convex Set Concave Set

Does the line connecting any 2 points in the set contained by the set? If yes, the set is convex. For a convex constraint set: If the objective function is convex, any minimum is the global minimum. If the objective function is concave, any maximum is a global maximum.

Convex and ConcaveConvex Concave

X X X

f(X)

•

convexover entire range

f(X)f(X)•

•

•

X

f(X) • •

Convex or Concave over Certain Ranges

Neither Convex nor Concave as a Whole

••

Is the line connecting any 2 pts. onthe function always aboveintermediate points on the function?

If yes function is convex.

•

•

X

1

2

X

X

1

2

X

••


35

Unbound Maxima and Unbounded Minima Let's look at: Max z = X2 versus Min z = X2

Min Z

dZdX

= 0 = 2 X X = 0

d2ZdX2

= 2 > 0 X = 0 is a minimum.

Max Z

dZdX

= 0 = 2 X X = 0, But this is a minimum.

In the curve above, there are 2 maxima, X = + and X = - These are unbound maxima. Unbound maxima or minima occur where the value of the objective function + (unbound maximum)

or objective function value - (unbound minimum). dZ/dX = 0 exists only for finite minima or maxima.

Having an unbound optimum solution usually indicates an error in the problem formulation. Duality For every constrained mathematical program problem, there is a dual problem. They are re-statements

of the same problem. The original problem is called the primal problem. For a primal problem:

Minimize f X

S.T. g X b

The Wolfe dual problem is:

Maximize L ( X

,

)

Subject to: L ( X

,

) = 0

0 This is why is called a dual variable. In Linear programming the dual form is much simpler and more useful.

Z

Xo


36

Karush-Kuhn-Tucker (KKT) Conditions for an Optimal Solution

A series of conditions which must be met for a solution to a constrained optimization problem to be an

optimum (max. or min.).

For the Problem: Max or Min. f X S.T. g X b with n decision variables and m constraints.

The Lagrangian is: L( X

,

) = f( X

) + ]([1

m

iiii Xgb

And the KKT conditions are:

(1) L

Xi

= 0, i = 1,..., n Gradient conditions)

(2) gj X bj, j = 1,..., m Feasibility conditions

(3) j b - gj X = 0, j = 1,..., m Complementary slackness

(4)j 0, j = 1,..., m Non-negativity

(5) Xi 0, i = 1,..., n Non-negativity

Points Xo satisfying these conditions are maxima or minima. If, in addition to satisfying these conditions,

g X b is a convex set, and

f X is a convex function, Xo is a global minimum

or

g X b is a convex set, and

f X is a concave function, Xo is a global maximum


37

Linear Programming A linear program is a mathematical program where:

1) the objective function is linear and 2) all constraints are linear equalities or inequalities.

The standard linear program thus has the form:

Minimize Z = c1X1 + c2X2 + ... + cnXn (X1, ..., Xn) Subject To: a11X1 + a12X2 + ... + a1nXn < b1 a21X1 + a22X2 + ... + a2nXn < b2

... am1X1 + am2X2 + ... + amnXn < bm

X1, X2, ..., Xn 0 where, aji = unit requirement of Xi involved in constraint j (all constants), bj = resource availability (RHS constant) for constraint j, ci = constant coefficient (unit value) of objective for Xi, Xi = a decision variable, and Z = value of the objective function. This problem has n decision variables, m resource constraints, and n non-negativity constraints. Note that aji, bj, and ci are all constants (data or parameters) in the model. In matrix notation, this standard for becomes:

Max or Min c

T X

S.T. A X

< b

X

0 Is the objective function for a linear program convex or non-convex? Is the constraint set convex? What does this imply for global optimality of a local optimum to a linear program?


38

Today we have a graphical example of a product-mix type problem frequently solved by linear programs. Clyde Motor Company Example A company has 3 factories that together produce two products: cars and small trucks. Each factory specializes in certain components required for both products. Plant 1 makes truck bodies. Plant 2 makes car bodies. Plant 3 makes shared components and assembles trucks and cars. Each plant has a limited capacity. Each product is sold for a fixed price with a fixed profit per unit. Data: Product Capacity Available Plant 1 2 (units/hour) 1 1 0 4 2 0 2 12 3 3 2 18 Unit Profit $3 $5 -- Step 1: Define the Problem: -) Max profit -) S.T. limited plant capacities -) How many cars and trucks should be produced? Steps 2 and 3: Express the Problem Mathematically: Max Z = 3X1 + 5X2 = profit per hour of operation, say.

Subject To: (1) 1X1 4 (2) 2X2 12 (3) 3X1 + 2X2 18 (4) 1X1 0 (5) 1X2 0 X1 = Number of small trucks produced. X2 = Number of cars produced. Explanation of constraints:

(1) 1X1 4

(2) 2X2 12

(3) 3X1 +2X2 18

capacity constraintson production

(4) X1 0

(5) X2 0cannot produce

negative products


39

Step 4: Solve. A graphical solution: 1) Graph the constraint set and feasible region 2) Plot contour lines of the objective function

To solve, move contour lines out until they are no longer feasible. Note, Z > 36 is not feasible.

8 9 10

(2)

1

2

3

4

5

6

7

8

9

0 1 2 3 4 5 6 7

(3) (1)

(2,6)

(4)

(5)

Z = 36

Z = 30

Z = 15

X 1

X 2


40

Step 5: Sensitivity Analysis --- Graphical Sensitivity Analysis Several pertinent questions: 1) How much could unit profits change and leave the optimal solution unaffected? (How stable is

the optimal solution w.r.t. price fluctuations?). 2) How much would each of the non-binding constraints have to change before the solution would

be changed? (What if a machine breakdown lowered the capacity of a plant?) (I.e., what is the value of the slack variable assoc. with each non-binding constraint?)

3) How much should we be willing to pay to lessen each of the binding constraints? (How much should we pay to improve capacity at Plants 2 and 3, which are operating at full

capacity?) 4) What happens if plant production efficiencies change (or product input requirements change)? Modify earlier graph to answer each question. Binding constraints are: (2) capacity of Plant 2, to produce car bodies. (3) capacity of Plant 3, to cars and trucks. Assumptions for LP: 1) Proportionality: -) Producing twice as much costs (profits) twice as much -) Producing twice as much uses twice as many inputs (plant capacity) 2) Additivity: All objectives and LHS of constraints can be expressed as sums or

differences of linear terms. 3) Divisibility: Fractional decisions can be made. 2.34 doors can be produced. We will discuss relaxing this assumption in a few weeks. 4) Certainty: -) Rates of profit/unit, capacity req./unit, and capacity constraint levels are constant and well-known. -) All parameter values are constant and well-known. Additional Terms:

Feasible solution Feasible region Corner-point feasible solution

What’s the relationship between optimal solutions and corner-point feasible solutions?


41

The Simplex Method: A Graphical Interpretation The simplex method is the most common method for solving linear programs. It is an algorithm which efficiently searches through different combinations of binding and non-binding constraints. The Convexity of Linear Constraint Sets: Can you think of any set of linear constraints which is concave? Plot: Any linear constraint set will be convex or infeasible. Optimality on the Corner Add a linear objective function (Max or Min). Given a linear objective function and linear constraints, the solution will be either 1) a corner point, or 2) between and including adjacent (and equally optimal) corner points. The Simplex Method, Qualitatively 1) Initialization: Find a feasible corner-point to start with. 2) Iterative Step: Move to a better adjacent feasible corner-point solution. 3) Optimality test: STOP if no adjacent corner point solutions are better. Otherwise, repeat Step 2. Easier said than programmed. Run through an example. [Perhaps another extended example.]

X1

X2


42

The Simplex Tableau Here's how computer solution methods store the Linear Program.

For the problem: 0) Min Z =

n

iiic

1

X

Subject to: 1)

n

iiia

11 X b1

2)

n

iiia

12 X b2

3)

n

iiia

13 X b3

The inequality constraints are first made into equality constraints by adding slack variables:

1) 111

1 bSXan

iii

2) 221

2 bSXan

iii

3) 331

3 bSXan

iii

The "Initial Tableau" is then formed

Basic Eqn Coefficients of Variables Variable No. Z X1 X2 - - - Xn S1 S2 S3 RHS

Z 0 1 c1 c2 - - - cn 0 0 0 0 (objective function)

S1 1 0 a11 a12 - - - a1n 1 0 0 b1

S2 2 0 a21 a22 - - - a2n 0 1 0 b2 (constraints)

S3 3 0 a31 a32 - - - a3n 0 0 1 b3

This is simply a way of stating:

Max or Min c

T X

+ S

S.T. A X

+ I S

= b

The tableau is then manipulated using the simplex method and stores the intermediate and final solutions to the Linear Program. The manipulations are all linear algebra. In this initial tableau, the slack variables are basic (non-zero) and each of the original decision variables is non-basic (equal to zero). Non-basic variables define the binding constraints which define a corner point being examined.


43

The Initial Tableau for the Clyde Motor Co. Example:

Coefficient of Variable

Basic Variable

Eqn. No.

Z

X1 X2

S1 S2 S3

RHS

Z 0 1 -3 -5 0 0 0 0 S1 1 0 1 0 1 0 0 4 S2 2 0 0 2 0 1 0 12 S3 3 0 3 2 0 0 1 18

This is the standard form for a linear program: Problem is stated as a minimization problem. All constraints are equality constraints. All variables (decision and slack) are non-negative.

Basic and Non-Basic Variables All variables (decision and slack) are classified as either Basic or Non-Basic for each iteration of the Simplex algorithm. This classification of variables defines a particular feasible corner-point solution to the LP problem. Basic Variables all have non-zero values and are associated with non-binding constraints. (Slacks have non-zero values) Only one non-zero coefficient appears in the tableau for a basic variable and that coefficient = 1. (Show examples for initial tableau.) The objective function variable Z is always considered to be a basic variable. Non-Basic Variables all have values of zero and are those associated with binding constraints. (Slack variables values = 0) For a linear programming problem with n decision variables and m constraints, there will be m + 1 basic variables. Non-negativity constraints are not counted in the m constraints. Why m+1 basic variables? The objective function (Z) is always basic; that's one. A corner in an n-dimensional decision-space (feasible region) will be bound in n-dimensions (for each decision variable dimension to be bound). The total number of constraints (including non-negativity) is n+m. n of these bind together to form a corner-point. Therefore, there are n non-basic variables representing n binding constraints. Thus m constraints are left non-binding and have slack or decision variables which are basic. NOTE: There are also m + 1 rows in the simplex tableau. Here's a simple algorithm for the simplex method. A one-page version of the simplex algorithm appears in an appendix.


44

The Simplex Method - An Algorithm Step 1: Initialization: From the initial tableau. This also represents the first trial solution, where all slack variables are basic. (Note that each slack variable has only one non-zero coefficient and that coefficient = 1.) Optimality Test (Is trial solution optimal?) This current solution is optimal only if every coefficient in Eqn. 0 (objective function line) is non-negative ( 0). STOP if optimal. Why? If a coefficient in Eqn. 0 is zero, the associated variable is basic and zero. If a coefficient in Eqn. 0 is negative, the associated variable =zero and by increasing that variable (from zero) the value of the objective function can be lessened (improved). If all coefficients in Eqn. 0 are 0, no variable can be changed to decrease the value of the objective function. This is then the/an optimal solution. Step 2: Iterative Step: Part 1: Find the new basic variable. Rule: The new basic variable is the variable with the most negative coefficient in Eqn. 0. This variable produces the largest decrease in the objective function per unit increase in value (from zero). For the Clyde problem: X2 has the most negative coefficient in Equation 0 of the initial tableau and so is

the new basic variable. Since only m + 1 variables may be basic, one previously basic variable must become non-basic. Part 2: Find the new non-basic variable. The column of the new basic variable is called the pivot column. Rule: For each row (excluding Equation 0 and any rows where the pivot column elements are negative) divide the RHS by the coefficient in the pivot column.

Select the row with the minimum ratio of RHSiCpivot i .

. This is called the pivot row.

This pivot row constraint will bind first as the new basic variable increases from zero. This leads to a feasible adjacent corner. Rows with negative coefficients in the pivot column are excluded because this would imply decreasing the new basic variable to a negative, and therefore infeasible, value. The new non-basic variable is the old basic variable in the Basic Variable column of this row. The variable has a coefficient of 1 in this row and a coefficient of zero in all other rows.


45

The ratios for the first iteration of the Clyde problem are:

Associated Basic

Variable

Row/Eqn. Number

X2

coefficient

RHS

RHSi/CX2,i

Z 0 -5 0 Not Applicable S1 1 0 4 4/0 = S2 2 2 12 12/2=6

Min Ratio: S2 is new non-basic variable S3 3 2 18 9

The tableau must now be changed to reflect this new corner-point feasible solution. Part 3: Find the new tableau Rule: Use linear algebra operations to modify the tableau-matrix so that the New Tableau has zero coefs. in the column of the new basic variable, except for the pivot row where the coefficient = 1. Adding and subtracting multiples of the pivot row to the other rows is usually the most effective approach. For the Clyde example: 1) Update Row-Eqn. 3 by subtracting Row 2 from Row 3. This produces a zero in column X2 of Row 3.

2) Update Row 2 by dividing Row 2 by 2. This produces a 1 in column X2 of Row 2.

3) Row 1 already has a zero in column X2.

4) Add 5 times Row 2 to Row 0. This puts a zero in column X2 of Row 0.

It is an easier process if the new tableau is written directly beneath the old tableau. Add new tableau beneath old tableau. (Leave room for one more iteration). Coefficients

Basic Variable

Equation Number

Z

X1 X2

S1 S2 S3

RHS

Z 0 1 -3 -5 0 0 0 0 Iteration 0 S1 1 0 1 0 1 0 0 4

S2 2 0 0 2 0 1 0 12 S3 3 0 3 2 0 0 1 18

Z 0 1 -3 0 0 5/2 0 30 Iteration 1 S1 1 0 1 0 1 0 0 4

X2 2 0 0 1 0 1/2 0 6 S3 3 0 3 0 0 -1 1 6

Z 0 1 0 0 0 3/2 1 36 Iteration 2 S1 1 0 0 0 1 1/3 -1/3 2

X2 2 0 0 1 0 1/2 0 6 X1 3 0 1 0 0 -1/3 1/3 2

Talk about second iteration later.


46

Note: For the new tableau the value of each -) non-basic variable = 0 -) basic variable = RHS. Step 3: Optimality test (again). Are all coefficients in Eqn. 0 are non-negative? In example, No. A new iteration is needed. Repeat Step 2. ______________________________________________________________________________ These are the fundamental steps in the Simplex method. Now let’s finish the problem. Step 2 Iteration Step Part 1: X1 is the new basic variable (only variable with negative coefficient in Row 0).

Part 2: Basic

Variable Coef of

X1

RHS RHS/

Coefficient of X1

Z -3 30 Not Applicable S1 1 4 4/1=4

X2 0 6 6/0 = S3 3 6 6/3=2; Make S3 the new non-basic variable.

Part 3: Produce New Tableau 1) Add Row 3 to Row 0 to set coefficient for X1 = 0 in Row 0.

2) Divide Row 3 by 3 for the new Row 3. 3) Row 2 is OK as is. 4) Subtract the new Row 3 from Row 1 to set coefficient of X1 = 0 in new Row 1.

Add changes to tableau from p. 8. Step 3: Optimality Test All coefficients in Row 0 0, so no other adjacent solution is better. Therefore STOP. The final solution: Read from Tableau No. 2. From RHS Column: Z = 36 S1 = 2 X2 = 6

X1 = 2.

From Top Row (Equation 0): 1 = 0

2 = 3/2

3 = 1

A short one-page description of the simplex algorithm follows.


47

THE SIMPLEX ALGORITHM (in one page) The following steps form the fundamentals of the Simplex Algorithm for solving linear programs. 1) Set up an "initial tableau."

2) Is the trial solution represented by the initial tableau optimal? If all coefficients in Row 0 are 0, then the solution is optimal and you may STOP. If any coefficients in Row 0 of the tableau are < 0, the solution represented by this tableau is not the optimal solution and the set of basic variables must be changed.

3) Select a new basic variable. Select as a new basic variable the variable with the most negative coefficient in Row 0. The column for this variable is the "pivot column."

4) Select a new non-basic variable. Divide the RHS entry of each row by the coefficient in the column of the new basic variable. The new non-basic variable is the old basic variable associated with the row having the minimum ratio. This row is called the "pivot row."

5) Update the tableau. Form a new tableau that has zero for each entry in the pivot column, except for the coefficient in the pivot row. The entry in both the pivot row and pivot column must be equal to one. Use linear algebra and Gaussian elimination techniques to do this, centered on the pivot row. Replace the old basic variable with the new basic variable in the "Basic Variable" column. This new tableau represents a new corner-point solution.

6) Is this new solution optimal? If all coefficients in Row 0 of the new tableau are 0, then STOP; the solution is optimal. If any coefficients in Row 0 of the tableau are < 0, the solution represented by this tableau is not the optimal solution and the set of basic variables must be changed. GO TO STEP 3.

Reading the final tableau. The following tableau represents an optimal solution and final tableau for the problem: MAX 2X1 + X2 Subject To: 1) X1 + X2 4 2) X2 3

Basic Eqn. Coefficient of Variable . Variable No. Z X1 X2 S1 S2 RHS Z 0 1 0 1 2 0 8 X1 1 0 1 1 1 0 4 S2 2 0 0 1 0 1 3

The values of each basic variable is given by the RHS column. The value of each non-basic variable is zero. Z = 8, X1 = 4, S2 = 3, X2 = 0, S1 = 0. The value of the Lagrange multiplier (shadow price) for each constraint is given by the coefficient in Row 0 under the associated slack variable (S1 or S2). 1 = 2; 2 = 0. The Reduced Cost of each coefficient in the objective function is given by the coefficient in Row 0 under the associated decision variable (X1 or X2). Can you obtain these results?


48

Some Simplex Details The basic simplex method described earlier applies only to a relatively narrow set of linear programming problems. In particular, the problem must be a MINIMIZATION problem and all constraints must be of the form aij Xi bj, i.e., all constraints must be less than or equal to constraints. There are also some ambiguities in the application of the simplex method. This lecture will try to clear up these problems. Maximization The simplex method minimizes the value of an objective function. But note: Max z = Min (-z) Min z = Max (-z). So we can always make a maximization problem into a minimization problem, or vice versa. Ties for New Basic Variable If two or more variables have the same negative coefficient in Row 0 of the tableau, and that coefficient is the most negative coefficient in Row 0, then select any variable from those with the tie. The choice is arbitrary. Ties for New Non-Basic Variable What if 2 or more variables tie on the minimum ratio test for becoming the new non-basic variable?

Break the tie arbitrarily. Possibility of cycling problems (discussed in book). Very rare. Unbound Objective Function (Max z or Min z ) An unbound solution occurs if the new non-basic variable has a RHS/pivot ratio = , i.e., if Min(RHS/pivot) = . For this case that value of the new basic variable is not limited by any constraint and becomes infinite. Multiple Optima Where the optima fall on two or more corner-points, the optima will also include the edges of the feasible region between these points. These cases will appear only when, after the STOPPING criterion has been reached, at least one non-basic variable has a coefficient of zero in Row 0. If this variable is increased (from zero) it will not affect the objective function. The other corner optimal corner point solutions are found by performing extra tableau iterations.


49

Equality Constraints Equality constraints are of the form: aij Xi = bj, bj 0. The problem with equality constraints is that the initial solution, X = 0, is no longer feasible. To fix this, we add a new type of slack variable, called an artificial variable. This type of slack variable acts as a regular slack variable in the simplex method. The new equality constraint becomes: aij Xi + Rj = bj We also add a huge penalty to the objective function if Rj > 0. The new objective function becomes: Min z = ci Xi + M Rj. M is a HUGE number compared to the other coefficients ci. These changes force the simplex method to quickly move away from the initially infeasible initial solution to a real feasible corner point.

This approach is called the BIG M method. Inequality Constraints inequality constraints are of the form: aij Xi bj, or, with a slack variable, aij Xi - Sj = bj. The same problem arises with inequality constraints as with equality constraints, i.e., the initial tableau does not represent a feasible solution. Unless bj 0, X cannot be equal to zero and the initial trail solution is infeasible. We solve this problem again using the Big M method. First, an artificial variable is added to the constraint: aij Xi - Sj + Rj = bj. Second, a large penalty is given for having Rj > 0 by adding a HUGE coefficient for Rj in the objective function: Min z = ci Xi + M Rj., where M is very large. This quickly drives Rj 0 after the initial tableau and points the simplex method towards the feasible region. Essentially, the BIG M method allows infeasible solutions, but only at an overwhelming penalty.


50

Negative RHS Sometimes constraints have the form: aij Xi bj, where bj < 0. This can cause the initial tableau solution (all X = 0) to be infeasible. (This problem persists if we multiply both sides by negative one and reverse the inequality and becomes similar to solving a inequality constraint.) We solve this again using the Big M method. aij Xi + Sj - Rj = bj or - aij Xi - Sj + Rj = ABS(bj) Min z = ci Xi + M Rj., where M is very large. No Feasible Solution This case can only occur if there are equality constraints or inequality constraints in addition to regular constraints. Therefore, artificial variables must appear in the formulation. No Feasible Solution Two Parallel Equality Constraints No Feasible Solution

This case is discovered if the solution to the problem has at least one artificial variable not equal to zero. Allowing Negative Values for a Decision Variable How could we let a decision variable Xi have any value, including negative? Must divide Xi into two new decision variables: X+i and X-i. X+i will represent Xi if it has a positive value

and X-i will represent negative values of Xi. We then replace Xi in all constraints and the objective function with: Xi = X+i - X-i. We can similarly add a lower bound to the negative value of Xi by adding another constraint, where Li is the absolute value of the lower negative bound: X-i Li. This ensures that Xi -Li.


51

Minimizing a fractional objective Given the objective and constraints, Min z = ∑ / ∑ Subject to: ∑

0 Transform variables so: Yi = T Xi, and let T = 1/ ∑ Re-write the formulation in terms of Yi and T, and multiply all constraints by T (assuming T >0), so the formulation becomes: Min z = ∑ Subject to: ∑

1

0, 0 The optimal Xi are found by Xi = Yi /T. This clever approach was originally proposed by Charnes and Cooper (1962) and is distilled from AIMMS “Linear programming tricks”. A. Charnes and W.W. Cooper, Programming with linear fractional functional, Naval Research Logistics Quarterly 9 (1962), 181–186.


52

Implementing the Big-M Method We've suggested this method a lot. What does it do to the simplex method. A simple example: Min 5 X1 + 2 X2 Subject to: X1 1 2 X1 + X2 = 4 3 X1 + 4 X2 30 Change this to: Min 5 X1 + 2 X2 + M R1 + M R2 Subject to: X1 - S1 + R1 = 1 2 X1 + X2 + R2 = 4 3 X1 + 4 X2 + S3 = 30 Note that the 3rd constraint, a constraint, does not require an artificial variable. The first initial tableau: Basic Equation Coefficients of Variables Variable No. z X1 X2 S1 S3 R1 R2 RHS 0 1 5 2 0 0 M M 0 1 0 1 0 -1 0 1 0 1 2 0 2 1 0 0 0 1 4 3 0 3 4 0 1 0 0 30 Note that the "artificial variables" R1 and R2 are not yet basic variables, however. (We want to start at the origin.) They need to have a coefficient of zero in Row 0. We have to fix this before we begin the simplex method, using linear algebra operations (Gaussian elimination). Multiplying Row 1 by M and subtracting it from Row 0 makes R1 basic, and multiplying Row 2 by M and subtracting this from Row 0 makes R2 basic. This becomes the true initial tableau. This results in: Basic Equation Coefficients of Variables Variable No. z X1 X2 S1 S3 R1 R2 RHS z 0 1 5-3M 2-M M 0 0 0 -5M R1 1 0 1 0 -1 0 1 0 1 R2 2 0 2 1 0 0 0 1 4 S3 3 0 3 4 0 1 0 0 30 This tableau represents the corner-point "solution" at the origin (X1 = X2 = 0). This pseudo-solution is penalized by its poor performance of 5M. Ah! That's better. Now we start the simplex method using this as our initial tableau.


53

To complete the example, following the standard Simplex algorithm, Step 2. This solution is not optimal, several coefficients in Row 0 are negative. Step 3. The X1 column has the most negative value, making this the "pivot column" and X1 the new basic variable. Step 4. Eqn./Row 1 has the minimum ratio of RHS/coefficient in the X1 column (=1/1), making this the "pivot row," and R1 the new non-basic variable. Step 5. Updating the tableau to make X1 basic, the new tableau becomes: Basic Equation Coefficients of Variables Variable No. z X1 X2 S1 S3 R1 R2 RHS z 0 1 0 2-M 5-2M 0 -5+3M 0 -5-2M X1 1 0 1 0 -1 0 1 0 1 R2 2 0 0 1 2 0 -2 1 2 S3 3 0 0 4 3 1 -3 0 27 Step 6. This tableau is not optimal. Continue the algorithm with Step 3. Step 3. S1 is the new basic variable/pivot column. Step 4. Row 2 is the new non-basic variable. (2/2)=1 (Row 1 is excluded because of the negative coef.) Step 5. The new tableau is: Basic Equation Coefficients of Variables Variable No. z X1 X2 S1 S3 R1 R2 RHS z 0 1 0 -1/2 0 0 M -2.5+M -10 X1 1 0 1 1/2 0 0 0 1/2 2 S1 2 0 0 1/2 1 0 -1 1/2 1 S3 3 0 0 5/2 0 1 0 -3/2 24 Step 6. This tableau is not optimal. Continue the algorithm with Step 3. Step 3. X2 is the new basic variable/pivot column. Step 4. Row 2 is the new non-basic variable. (1/0.5)=2 Step 5. The new tableau is: Basic Equation Coefficients of Variables Variable No. z X1 X2 S1 S3 R1 R2 RHS z 0 1 0 0 1 0 -1+M -2+M -9 X1 1 0 1 0 -1 0 1 0 1 X2 2 0 0 1 2 0 -2 1 2 S3 3 0 0 0 -5 1 -5 -4 19 Step 6. This tableau is optimal, with the solution X1 = 1, X2 = 2, z = 9.


54

Sensitivity Analysis in Linear Programming

Sensitivity Analysis: How would the solution change if parameter or coefficient values are varied? Often for real problems parameter values are not known with certainty. Thus sensitivity analysis can tell us the relative importance of different sources of uncertainty.

Sensitivity analysis is the systematic variation of parameter and input data values in a model to assess the affect of uncertainties or variation in these variables on model results and designs based on model output.

Sensitivity analysis helps answer such questions as: 1) What is the effect if parameter ci, bj, or aij is off by some error ? 2) If input data values are accurate within some error , what are possible effects on model results? 3) Is further research into the value of certain parameters and input data warranted?

Since many models have large numbers of parameters and input data, conducting a complete sensitivity analysis on all parameters and data, and all possible combinations of co-varied errors, is impossible. This implies that sensitivity analysis must be somewhat judicious, concentrating on those parameters and inputs that seem most important and prone to error. Brute Force Sensitivity Analysis We could always re-solve the linear program with new parameter values. Linear programming solutions are usually quite fast, so this is often a very reasonable option. But, here are some simpler tricks! These are also discussed in Chapter 6 of Hillier and Lieberman. Lagrange Multipliers/Shadow Prices/Dual Prices The shadow price () associated with each constraint represents the change in the value of the objective function if the RHS constant of the constraint is changed by one unit. Note: Beware the sign of these Lagrange multipliers . A slight change in the formulation of the equations or in the solution method will change the sign of the Lagrange multiplier, although the absolute value should remain the same. It is common for two computer solution packages to give different signs for the shadow prices. In practice, I recommend interpreting the sign intuitively and then applying the absolute value for the magnitude. It is easy to intuit the sign; changing the constraint value to "loosen" the constraint should improve the value of the objective function. These appear as the coefficients in Row 0, under the slack variables, in the final simplex tableau. Slack Variables The value of the slack variable represents the amount or the "resource" remaining un-used in the constraint; i.e., the constraint can be tightened by up to the value of the slack variable before the optimal solution is changed. These values are given directly by the simplex solution. Reduced Cost The reduced cost is the amount by which the coefficient in the objective function would have to change to make a non-basic decision variable become basic (non-zero). These values are given by the coefficients in Row 0, under the decision variables, in the final simplex tableau. Ranges in Which the Basis is Unchanged Remember, the basis is the set of variables with non-zero values.

Often, linear program output will include a "range in which the basis is unchanged." This output indicates how much particular coefficients in the objective function, RHS constants, and occasionally coefficients in the LHS of the constraints can change before one or more of the basic variables must become non-basic, and vice versa.


55

Within the “range of objective function coefficient values for which the Basis is unchanged”, changes in these coefficient values will change the value of the objective function, but leave the optimal solution values for the decision variables completely unchanged. Within the “range of the RHS constants for which the Basis is unchanged”, changes in these RHS coefficient values for binding constraints will change both the solution values and the objective function value. For non-binding constraints, the range on the tightening side should be the same as the slack variable values. Example of LINDO Output MIN 5 X1 + 7 X2 SUBJECT TO 2) X1 + X2 >= 10000 3) - 0.2 X1 + 0.1 X2 >= 0 4) 0.1 X1 - 0.2 X2 <= 0 END LP OPTIMUM FOUND AT STEP 2 OBJECTIVE FUNCTION VALUE 1) 63333.3320 VARIABLE VALUE REDUCED COST X1 3333.333252 0.000000 X2 6666.666504 0.000000 ROW SLACK OR SURPLUS DUAL PRICES 2) 0.000000 -6.333333 3) 0.000000 -6.666667 4) 1000.000000 0.000000 NO. ITERATIONS= 2 RANGES IN WHICH THE BASIS IS UNCHANGED: OBJ COEFFICIENT RANGES VARIABLE CURRENT ALLOWABLE ALLOWABLE COEF INCREASE DECREASE X1 5.000000 2.000000 19.000000 X2 7.000000 INFINITY 2.000000 RIGHTHAND SIDE RANGES ROW CURRENT ALLOWABLE ALLOWABLE RHS INCREASE DECREASE 2 10000.000000 INFINITY 9999.999023 3 0.000000 1000.000000 1000.000000 4 0.000000 INFINITY 1000.000000


56

How Much can Coefficients, ci, bj, and aij Change Before the Solution Moves to Another

Cornerpoint? Consider the initial tableau for the problem: Max Z = cT X S.T. A X b

Eqn. No. Z Xi Sj RHS

0 1 -c 0 0 1 0 • • • m A I b

coefficients in the constraints Identity matrix

The final tableau will have the form: Equation No. Z Xi Sj RHS

[y* A - c] y* I Z* = y* b = cT X* B* A B* I B*b

NOTE: B is the basis matrix and y is the vector of Lagrange multiplier values. The “*” indicates values at the optimal solution. B* and y* together summarize all operations on the initial tableau needed to create the final optimal tableau. y* is the vector of Lagrange multiplier values at the optimal solution. How much can we change ci, bj, or aij before the Basis changes? Case 1: If the optimal solution remains at the same cornerpoint: (Basic variables remain basic) For changes in c that keep the same Basis, the solution values (and corner point) remain the same, although the Lagrange multiplier values will change. Let b' = b + b, A' = A + A, and c' = c + c, where b, A, and c represent changes in b, A and c, respectively. Then:

*'* ' * * '

*

' '* ' * * '

|

T T

Xb

S

z c X y b

B A B B


57

This implies The new final tableau has the form:

Equation No. Z Xi Xn Sj Sm RHS

0 1 y* A' - c' y* I Z* 1 0 B*A' B* I B*b'

m Case 2: If the optimal solution has changed corner-points: (Basic variables have changed.) The new simplex tableau found in Case 1 will no longer satisfy the optimality test, if the optimal corner-point solution has changed. This happens if either another corner-point is optimal or there is now no feasible solution. Using these insights to find changes in parameter values that keep the set of basic variables the same. Case 3: If the solution is infeasible: Some basic variable(s) will have negative values. I.e., a decision or slack variable will have a negative value. This can be quickly found because at least one element of B* b' will be negative.


58

Duality Duality is part of the mathematical theory behind linear programming that makes linear programming a power problem-solving tool. Some useful results of duality:

1) Helps interpretation of LP results. 2) Very useful for sensitivity analysis 3) Can be used to improve computational efficiency (reducing the number of constraints; simplex

solves faster with more decision variables and fewer constraints) 4) Useful for speeding multi-objective analysis

What is duality? For every original (primal) LP problem, there is an associated, dual problem. Compare the LP statements:

Min W = y b

Dual ProblemPrimal Problem

Max Z = c X

S.T. y A c y 0

S.T. A X b X 0

At X*, Z = c X At y*, W = b y At optimal solutions c X* = b y* = Z* Compare at Initial Tableaux: Primal: (Max) Dual: (Min) Xi Sj RHS yi SD

i RHS

-c 0 0 -b 0 0

A I b AT I c

Constraints Decision Variables Objective function coefficients. RHS

Compare the Final Tableaux: Primal: Dual: Xi Si yi SD

i

y*A -c y*i Z* A x* -b X*i Z*

X*i y*j

S*j SD*

j

Note: Solving either the dual or the primal by simplex solves both problems.


59

The relationship between primal and dual linear programming problems is shown by writing them as (Quirino Paris, n.d.): Primal Dual Max Z = c1x1 + c2x2 + c3x3 Min R = b1y1 + b2y2 Subject to: Subject to: a11x1 + a12x2 + a13x3 b1 y1 0

a21x1 + a22x2 + a23x3 b2 y2 0

x1 0 a11y1 + a21y2 c1

x2 0 a12y1 + a22y2 c2

x3 0 a13y1 + a23y2 c3.

Here, y1 and y2 are dual variables for the primal constraints and x1, x2, x3, are dual variables for the

dual constraints. The notion that Lagrange multipliers are in essence dual variables can be illustrated by defining the Lagrangean function for both linear programming problems. The use of the Lagrangean function in this context is of great help in understanding why and how the primal and the dual LP problems are so intimately related to each other. We now state "two" Lagrangean functions corresponding to the primal and dual LP problems and conclude that the two functions are identical and, therefore, only one Lagrangean function exists for a pair of primal and dual linear programming problems. We use y1, y2 as Lagrange multipliers for the primal problem and x1, x2, x3, as Lagrange multipliers for the dual.

Lagrangean for Primal Lagrangean for Dual

max L = c1x1 + c2x2 + c3x3 min L' = b1y1 + b2y2 + y1[b1 - a11x1 - a12x2 - a13x3] + x1[c1 - a11y1 - a21y2]

+ y2[b2 - a21x1 - a22x2 - a23x3] + x2[c2 - a12y1 - a22y2]

+ x3[c3 - a13y1 - a23y2]

The two Lagrangean functions just stated are actually the same. In this way, the primal and the dual LP problems are tied together by a unique Lagrangean function. Duality Example: For the Primal Problem: Find the Dual Problem: Max Z = 3X1 + 5X2 Min Z = 10Y1 + 12Y2 + 4Y3 Subject to: Subject to: X1 + X2 10 Y1 + 2Y2 + Y3 3 2X1 + X2 12 Y1 + Y2 5 X1 4 An economic interpretation: If the primal problem is to maximize profit subject to resource constraints, we can interpret Yj as the contribution to profit per unit of resource j. The objective of the dual problem, then, is to minimize the total implicit value of the resources consumed.


60

Yj = j = the marginal contribution to profit of resource j. Some Primal-Dual Relationships for LP 1) Weak Duality Property If both Xo and Yo are complementary feasible solutions for the primal and dual problems, respectively, then: c X Y b . 2) Strong Duality Property If both x* and y* are complementary optimal solutions for the primal and dual problems, respectively, then: c X* = Y* b . Note: Computationally, we would rather have variables than constraints. B* summarizes all the operations performed on the simplex tableau to get the final solution. The Final Tableau:

Coefs. of Variables Basic Var. Eqn. No. Z Xi Xn Sj Sm RHS

Z 0 1 [y* A' - c'] [y*] [Z*] 1 0 [A*] [B*] [b*] . . . m

Initial Tableau: The initial tableau consists of the stacked vector and matrix, tT

t = vector of Row 0 in initial tableau [c | 0 | 0] T = matrix of Rows 1 to m in the initial Tableau [A | I| b]

Final Tableau: The final tableau consists of the stacked vector and matrix,

t*

T*

t* = [Z* = y* A ], a consequence of operations done to Row 0 T* = [A* | B* | b* ]

A* = B* A b* = B* b "Fundamental Insight" (?) Xi coefs s RHS

t* = t + y*T = [y* A = c | y* | y* b] T* = B*T = [B* A | B* | B* b ]


61

Example optimization: California Hydropower Transportation Problems A Simple Transportation Problem Common problem of shipping good between multiple sources and destinations. Minimize cost. Let: n = number of locations Xij = units of good (flow) shipped from origin i and destination j cij = Cost of transporting a unit of good from i to j

dj = demand for good at each location j

si = supply of good at each source location i

Here's the Problem. How much flow should go where? What's the LP?

Min Z =

n

jijij

n

i

Xc11

S.T.

n

ijij dX

1

, j = 1, ..., n (satisfy demands at ea. location)

1

n

ij ij

X s

, i = 1, ..., n (limited sources at each location)

Xij ≥ 0, i and j There are n2 decision variables and 2n constraints for this problem, not counting the non-negativity constraints. Project homework is a transportation problem, plus a little bit more. Multi-Period Transportation Problem with Warehouse Storage Add time subscripts to each variable & coefficient to represent each of T time-periods in the plan. Let: T = number of scheduling periods. Xijt, djt, sit, cijt

?

1

?

?

? ?

?

? 5

4

3

2


62

cwit = unit cost of warehousing at location i and time t Kkt = Storage capacity of warehouse at location k and time t.

This problem is a little more complex. Since warehouses store goods in time, a time dimension (sub-script) is essential to this problem. The LP?

Min Z = 1 1 1 1

T n n n

ijt ijt wit itt i j i

Z c X c W

Subject To

, 11 1

n n

ijt jt jt jkt jt j ti k

X W S X D W

j,t (conservation of mass at all locations and times)

Djt djt, j,t (satisfy demands at all locations and times)

Sit sit, i,t (limited sources at all locations and times) Wkt bkt, k,t (limited storage capacity) Xijt rijt, ijt (limited route capacity on each route at each time) Xijt, Xkjt Xikt 0, i ,j,k,t (non-negativity) Number of decision variables = Tn2 Number of constraints = (4n + n2)T (not including non-negativity constraints) Number of parameters to estimate (c, cw, b, r, d, and s) = 2n2 + 4n Transportation with Warehouses and Travel Times/Delivery Lags Let: Xijt = departures from i to j at time t Aijt = arrivals from i to j at time t Lijt lag or travel time from i to j departing at time t Note: Xijt = Aij,t+Lijt since departures at time t arrive at time t + Lijt. The LP?

Min Z = 1 1 1 1

T n n n

ijt ijt wit itt i j i

Z c X c W

Subject To

, 11 1

n n

ijt jt jt jkt jt j ti k

A W S X D W

j,t (conservation of mass at all locations and times)

Xijt = Aij,t+Lijt i,t (arrivals lag departures by Lijt)

Djt djt, j,t (satisfy demands at all locations and times)

Sit sit, i,t (limited sources at all locations and times) Wkt bkt, k,t (limited storage capacity) Xijt rijt, ijt (limited route capacity on each route at each time) Xijt, Xkjt Xikt 0, i ,j,k,t (non-negativity) This “transportation” problem is now rather flexible and applies to lots of problems. Variations of this also can apply to other logistics problems, such as water and wastewater movements – Just consider reservoirs to be warehouses and you can model California’s water system this way.


63

Scheduling Problems Simple Scheduling Problems CPM and related problems. There are n activities to schedule to complete a project. Let: Si = the starting time of activity

di = duration of the activity pij = 1 if activity j is a prerequisite of activity i; equals zero otherwise.

Objective: Min. time to completion = T: The LP:

Min T S.T. T Si + di, i = 1, ..., n Si pijSj + pijdj, i = 1, ..., n and j = 1, ..., n

Scheduling where Activity Duration can be shortened at a cost: Fundamentally, this is a multi-objective optimization problem (Cohon 1978). The two objectives are 1) minimize cost and 2) minimize time to completion T. A weighting factor can be assigned to combine and weight these two objectives in the overall optimization objective function. If ci is max duration of activity

Let di = ci - ai Xi Xi = $ spent to speed completion of activity i

di bi bi is minimum duration of activity

Min Z = TXn

ii

1

S.T. T Si + di, i Si pijSj + pijdj, i,j, ij di bi, i di = ci - aiXi, i Minimize a combination of Cost and Time-duration Weighting Method: Here, we'll weight each of the two objectives. Making several LP runs with different weights will produce an efficient trade-off curve for the two objectives. Change objective function to:

Min Xii = 1

n

+ 1 - T .

Constraint Method: Another multi-objective optimization approach is the "constraint method". This would involve solving the cost-minimizing LP several times, with each LP run having a different constrained value for the completion time T, T . The constant will vary from Tmin (resulting from the completion-time minimizing formulation) to Tmax (the cost-minimizing solution). This "constraint method" is actually a bit more rigorous.


64

Assignment Problems Allocating resources on a one-to-one basis. Example allocating personnel to tasks on a project so as to 1) maximize overall performance or 2) make employees happier. Example: Allocate m graduate students to m available desks. Students have given preferences for particular desks (1 = most preferred, ..., m = least preferred). cij = student i's preference for desk j.

Xij = 1 if student i sits in desk j. Xij = 0 if not.

The Linear Program version of this problem:

Min i=1

m j=1

m cij Xij

S.T. i=1

m Xij = 1, for j = 1, ..., m

j=1

m Xij = 1, for i = 1, ..., m

Number of decision variables = m2

Number of constraints = 2 m. Other LP problems [Should include some more of these examples.] Regional Planning Reservoir operations Water Quality Regional environmental planning Air Quality Planning Product Mix Goal Programming

Read text


65

LP Solution of Non-Linear Objectives with Linear Constraints Sounds contradictory. But can be done for problems with objective functions of forms:

Min f(x): (convex) Max f(x): (concave)

o x

or f(x)

ox

f(x)

Piece-wise Linearization In each case, a linear approximation can be made of the function for specific ranges of X: Min f(x): fa(X) f(X) The resulting linearized non-linear program is:

Min fa(X) = f(0) + f(d1) - f(0)

d1 X1

+ f(d2) - f(d1)

d2 - d1 X2

+ f(d3) - f(d2)

d3 - d2 X3

Subject to: X = X1 + X2 + X3 X1 d1 X2 d2 - d1 X3 d3 - d2 followed by any other constraints

0

y = f(x) y=fa(x)

y

d1 d2 d3

X3

X2

X1

X


66

Note: The coefficients of X1, X2, and X3 are just slopes of parts of the objective function! Why it works: LP will always make X1 = d1 before X2 becomes > 0.

Similarly for X2 and X3.

To maximize this same function,

f(x)

o x it won't work.

The LP will want to max X3 and keep X1 and X2 = 0.

Maximizing Concave Objective Functions A similar linearization approach will work for maximizing a concave function subject to linear constraints. Max f(X): Later, we will see how any type of single-decision-variable can be linearized, with the addition of integer-linear programming. [Need a numerical example.] Example of Piecewise Linearization 1) Minimize a convex function,

Min Z = 5X2 S.T. constraints on X 2) Maximize a concave function, Max Z = 3X0.7 S.T. constraints on X

0

y = f(x)

y=fa(x) y

d1 d2 d3

X3

X2

X1

X


67

Integer Programming and Mixed Integer-Linear Programming For many problems, the decision variables can only take on integer values. For example, for a location decision, typically a facility can be located at only one site, a factory can sell only an integer number of airplanes, etc. Decision variables with integer values only are called 'integer variables". A common case is where a decision variable can have values of either 0 or 1. These are called "binary variables". "Yes" or "no" values for variable. Example: Capital Improvement Plan A city sewer utility must decide which capital improvement projects to select to minimize long-term operation and maintenance costs.

Decision No.

Decision

Decision Variable

Net Present Value

Capital Requirements

1 Sewer Rehab. #1 X1 $70 million $20 m

2 WWTP Rehab. #1 X2 $50 million $15 m 3 Sewer Rebuild #1 X3 $40 million $12 m

4 Sewer Rehab#2 X4 $30 million $10 m

Capital Avail. $25 million Cannot build everywhere because of capital availability constraint. Xi = yes or no, 0 = no, 1 = yes.

Max Z = 70X1 + 50X2 + 40X3 + 30X4 max. total new present value S.T. 1) 20X1 + 15X2 + 12X3 + 10X4 25 capital constraint 2) X1, X2, X3, X4 are binary (0,1) Now to solve? 1) By enumeration: Try all combinations of Xi = 0, 1. Choose combination with the best feasible value

of the objective function. For example: 24 = 16 combinations, for n binary decision variables where are 2n combinations. Gets to be lots of combinations fast.


68

2) Branch and Bound Methods: The concept: An efficient enumeration procedure. a) For the example, we can see that any solution where X1 = 1 and X2 = 1 is infeasible, violating the

budget constraint. For these values of X1 and X2, the solution is infeasible regardless of the values of X3 and X4.

So, by finding an infeasible subset of decision values, we eliminate many combinations of the larger set of values. Here, we eliminated: (1,1,0,0), (1,1,0,1), (1,1,1,0), and (1,1,1,1). b) Similarly, we can eliminate combinations of decision values which lead to inferior solutions. For the example: Let the decision be (0,1,0,1). This is feasible (just barely). Since the objective is one of maximization and the decrease in any decision variable only decreases the objective function, we know that (0,1,0,1) is better than (0,0,0,1), (0,0,0,0), and (0,1,0,0). The book and appendix detail methods for putting these ideas to efficient practice. Typically, a branch and bound approach will significantly limit the number of combinations to be evaluated. (This can still be a lot.) Mixed Integer-Linear Programming Often we must solve LP problems that have one or more binary or integer decision variables. The Hillier and Lieberman goes over several Branch-and-Bound approaches (pp. 515-541). Some major concepts for understanding such algorithms are:

LP relaxation Branching Bounding Fathoming

Appendix VIII provides a simple algorithm for solving integer-linear programs.

xkcd.com


69

Summary of the BIP Branch and Bound Algorithm [This section needs some work.] Initialization. Set Z* = - (for a maximization problem). Apply the bounding step, fathoming step, and optimality test described below to the entire problem. If not fathomed, classify this problem as the one remaining “subproblem.” Steps for each iteration: 1. Branching. Among the remaining (unfathomed) subproblems, select the one that was created

most recently. (Break ties according to which one has the larger bound). Branch from the node for this problem to create two new subproblems by fixing the next variable (the branching variable) to 0 or 1.

2. Bounding. For each new subproblem, obtain its bound by applying the simplex method to its

LP relaxation. [This section needs some work.] 3. Fathoming. For each new subproblem, apply the three fathoming tests described below, and

discard those subproblems that are fathomed by any of the tests. A subproblem is fathomed (dismissed from further consideration) if:

Its bound is less than or equal to Z* (i.e., we already know of a better solution). Its LP relaxation has no feasible solutions. The optimal solution to its LP relaxation satisfies the integer constraints. (If this solution is

better than the incumbent, it becomes the new incumbent.)

Optimality test: If there are no remaining subproblems, STOP. The incumbent is optimal. Otherwise, perform another iteration.

“NP-complete” problems cannot be provably solved in time polynomial time - O(nk) k=constant, n=size - in any known way, but useful approximations often exist. (Enumeration solutions are often O(2n).) Knapsack problems are to fill a knapsack capacity with the best selection of diverse things, without exceeding the capacity.

xkcd.com


70

INTEGER AND INTEGER-LINEAR PROGRAMMING: A BRANCH-AND-BOUND METHOD

The following algorithm can be used to solve a mixed integer-linear program using linear programming solution methods. Consider the problem: Maximize z = 5 X1 + 3 X2 Subject to: 4 X1 + 2 X2 25 X1 5 X2 8 Both X1 and X2 are integers.

Step 1: Find the optimal continuous solution. Relax the integer requirements for the decision variables and solve the resulting non-integer linear program. This gives an optimal continuous solution (OCS). If all originally integer variables have integer solutions in the OCS, then STOP; this is the optimal solution. Otherwise, continue with Step 2. For the problem above, the solution is X1 = 2.25, X2 = 8, z = 35.25, so this is not the optimal solution to the integer programming problem.

Step 2: Branch. Divide the problem into two sub-problems, searching for good integer-valued solutions. For our problem, X1 has a fractional value in Step 1 of 2.25. We produce the two sub-problems by adding new constraints on the non-integer value which keep it from its current non-integer value. Here, since X1 = 2.25 above, we add the constraints X1 2 and X1 3 to give the following two sub-problems. This is called branching. Sub-problem 1: Sub-problem 2: Maximize z = 5 X1 + 3 X2 Maximize z = 5 X1 + 3 X2 Subject to: 4 X1 + 2 X2 25 Subject to: 4 X1 + 2 X2 25 X1 5 (now redundant) X1 5 X2 8 X2 8 X1 2 X1 3

Step 3: Solve the branched problems. When these sub-problems are solved by linear programming, the solutions are compared. Sub-problem 1: X1 = 2, X2 = 8, Z1 = 34 Sub-problem 2: X1 = 3, X2 = 6.5, Z2 = 34.5

Step 4: Set the bound. The best solution to the sub-problem which satisfies all integer requirements is called the upper (or lower) bound. In this case, the sub-problem 1 solution, z = 34, is the lower bound of the optimal solution. If no sub-problem solution satisfies the integer conditions, then further branching and solution is required until a bound can be set. (If a bound has already been set, any new bound must have a better value than the existing bound.)

Step 5: Continued branching. Branch on sub-problems that both do not satisfy integer conditions and have superior solutions. In this case the solution to sub-problem 2 both does not satisfy the integer conditions and has a superior solution to the bound (bound = 34, Z2 = 34.5). If the solutions to the all other sub-problems are inferior to the bound, then STOP and the solution that sets the bound is the optimal solution which satisfies all constraints, including the integer conditions. Otherwise, GO TO Step 2.

Continuing with the solution to the example:


71

Following Step 2, continued branching for this problem (Sub-problem 2) gives the following two sub-problems: Subproblem 2a: Subproblem 2b: Maximize z = 5 X1 + 3 X2 Maximize z = 5 X1 + 3 X2 Subject to: 4 X1 + 2 X2 25 Subject to: 4 X1 + 2 X2 25 X1 5 X1 5 X2 8 (now redundant) X2 8

X1 3 X1 3

X2 6 (new constraint) X2 7 (new constraint) Repeating Step 3, the solutions to these new sub-problems are: Sub-problem 2a:X1 = 3.25, X2 = 6, Z2a = 34.25 Sub-problem 2b: infeasible

Step 4: Since neither of the new solutions both satisfies the integer conditions and has a better objective function value than the existing bound (existing bound = 34), the old bound remains.

Step 5: Since the objective function value for the solution to sub-problem 2a is superior to the bound, there remains a possibility that there is an integer solution better than the current bound (= 34). So branching on sub-problem 2a continues, using Step 2.

Step 2: Branching on sub-problem 2a gives the following new sub-problems: Subproblem 2a1: Subproblem 2a2: Maximize z = 5 X1 + 3 X2 Maximize z = 5 X1 + 3 X2 Subject to: 4 X1 + 2 X2 25 Subject to: 4 X1 + 2 X2 25 X1 5 X1 5 X2 8 (now redundant) X2 8 (now redundant)

X1 3 X1 3 (now redundant) X2 6 X2 6

X1 3 (new constraint) X1 4 (new constraint)

Repeating Step 3, the solutions to these new sub-problems are: Sub-problem 2a1: X1 = 3, X2 = 6, Z2a1 = 33 Sub-problem 2a2: X1 = 4, X2 = 4.5, Z2a2 = 33.5

Steps 4 and 5: Since neither solution to the new sub-problem is superior to the bound, there is no hope for an integer solution better than the bounded solution (Z = 34). (Since further branching adds constraints, and a solution is never improved by adding constraints, new sub-problems will never produce better solutions than their parent problems.) Therefore, we can STOP, knowing the optimal solution is that of the current bound, Z = 34, X1 = 2, X2 = 8.

Graphically, in solving the example we have created the following tree from the branches, with each branch terminating when we reach an inferior solution or an upper bound.

OCS

Optimal Continuous Solution

X1 = 2.25, X2 = 8, Z = 35.25

1 2

X1 2 X1 3

Lower Bound X1 = 2, X2 = 8, Z = 34

X1 = 3 X2 = 6.5 Z = 34.5

X2 7 X2 6

2b

Infeasible (inferior to Bound)

2aX1 = 3.25 X2 = 6 Z = 34.25

X1 3 X1 4

2a1

Z = 33 Inferior to Bound

2a2

Z = 33.5 Inferior to Bound


72

More Non-Linear Objective Functions with Linear Constraints Using Integer-Linear Programming.

Loucks, et al. (1981) have a nice presentation of linearizing any single-variable objective function, although often integer-linear programming is required. Consider the two functions:

Concave Function Convex Function

Method for Linear Programming (Max. concave or Min. convex objective functions) The first method of linearization is appropriate for linear programming and was described above. It is re-stated below.

Maximize f(X) = s1X1 + s2X2 + s3X3 = i=1

n siXi (for the concave function only)

Minimize f(X) = s1X1 + s2X2 + s3X3 = i=1

n siXi (for the convex function only)

Subject to: X = d0 + X1 + X2 + X3 Xj dj - dj-1, for each linear segment j and any original linear constraints in the problem

Method for Mixed Integer-Linear Programming (either convex or concave functions) Max or Min f(X) f(d0)Z0 + f(d1)Z1 + f(d2)Z2 + f(d3)Z3 + s1X1 + s2X2 + s3X3

1 10

( )n

i i i ii

f d Z s X

Subject to: X = d0Z0 + d1Z1 + d2Z2 + d3Z3 + X1 + X2 + X3 Z0 + Z1 + Z2 + Z3 = 1 Xj (dj - dj-1)Zj-1 for each linear segment j

Zj are integer variables for all j and any original linear constraints in the problem


73

Computational Aspects It is often possible to formulate problems that are too large for computers to solve. How large a problem can we formulate before we have to start worrying? Computational Aspects of Integer Programs Wagner (1975) (p. 490) -) Fewer than 100 binary variables can be solved "without great danger" -) "Real problems containing actual data often solve more rapidly than you might predict on theoretical grounds." Hillier and Lieberman (1986) (p. 400) -) "Even the fastest computers are incapable of performing exhaustive enumeration for binary integer problems with a few dozen variables..." LP Computation Requirements Schrage (p. 186) LP effort No. of decision variables (No. of constraints)2 Hillier and Lieberman (1986) (p. 92) LP effort: a) is less than proportional to the number of decision variables b) is roughly proportional to (No. of constraints)3 c) decreases with increasing "sparseness" of the constraint matrix (A) Problems with 5,000 constraints and 10,000 decision variables can be solved in < 1 hr. Recent LP Solver Experiences (1999): Problems with > 20,000 decision variables solved in 3 minutes. Specialized LP problems, such as Network Flow Programs: Solution time for some common specialized problems tends to increase about proportionately with the number of constraints squared or less. A recent reference on LP computational improvements: Bixby, Robert E. (2002), “Solving Real-World Linear Programs: A Decade and More of Progress”, Operations Research, Jan/Feb2002, Vol. 50 Issue 1, p3-15.


74

Dynamic Programming (DP) Dynamic programming is a method of solving a particular class of math. programming problems. The method is essentially an efficient form of enumeration, somewhat like the branch-and-bound method of integer programming. Illustrate method by way of a problem: Example: The stage-coach problem; (starts p.320 of Hillier and Leiberman text) A salesman wants to get from location 1 to location 10 with a minimum of risk, as represented by life insurance premiums on each intermediate journey. His map:

1

2

3

4

5

6

7

8

9

10

Note that the total trip is divided into 4 stages: 1 to 2,3, or 4; 2,3, or 4 to 5,6, or 7; 5,6, or 7 to 8 or 9; and 8 or 9 to 10. The life insurance cost for each possible trip segment. Cij of going from i to j.

Stage 1 Stage 2 Stage 3 Stage 4

2 3 4 5 6 7 8 9 10 1 2 5: 1 4 8: 3 3: 3 2 4 6 6 3 9: 4 4: 4 1 5 7: 3 3

How to select a route with minimum total life insurance cost? 1) By enumeration: Compare total cost of each possible combination of routes. a) 1,2,7,8,10 b) 1,2,7,9,10 c) 1,2,6,8,10 d) 1,2,6,9.10 e) 1,2,5,8,10 f) 1,2,5,9,10 g) 1,3,7,8,10 etc. (18 possible combinations of routes) Lots of combinations work. 2) Use a short-sighted method. Choose the cheapest route for the first stage, 1 2 ($2); then the cheapest route from 2 to the 2nd stage, 2 6 ($4); then again for the third stage, 6 9($3); and the fourth stage, 9 10 ($4). The total cost is then $13. 1 2 6 9 10: $13


75

But, 1 4 6 9 10 costs only 3 + 1 + 3 + 4 = $11 So this method will not guarantee the optimal solution. 3) Dynamic Programming Note that the routing decision is divided into 4 stages. At each stage only one next-step of several possible steps can be chosen. For Stage 4: The last decision: Make a table: n = 4 S f4

* (S )= C ij X 4*

8 3 10 9 4 10

X 4* is the decision variable of which location to go to next.

X 4* is always 10, the end.

S is the state in which the stage 4 decision is made (the entering state or condition). Go from 8 or 9. f4*(S) is the cost of going from the present possible state to the final state.

So far this isn't very interesting. Let's now work backwards, to the next-to-last stage, stage 3. f3(S3,X3)=c3(X3)+f4*(X3) n = 3 S X3: 8 9 f3

*(S) X 3*

5 4 8 4 8 6 9 7 7 9 7 6 7 6 8

-) Going from 5 to 10, it is best to pass through 8. -) Going from 6 to 10, it is best to pass through 9. -) Going from 7 to 10, it is best to pass through 8. fi() = recursive objective function, “cost-to-go” function, or accumulated objective function. It has two parts, 1) the direct costs of the decision in the current stage and state and 2) the best implied cost for all the later states, f*i+1(S i+1). Backwards again to Stage 2. f2(S2,X2)=c2(X2)+f3*(S3=X2) n = 2 S X2: 5 6 7 f2

*(S) X 2*

2 11 11 12 11 5 or 6 3 7 9 10 7 5 4 8 8 11 8 5 or 6

Same interpretation . Give examples. Finally, back again to Stage 1. f1(S1,X1)=c1(X1)+f2*(S2=X1) n = 1 S X1: 2 3 4 f1

*(S) X 1*

1 13 11 11 11 3 or 4


76

The best decision in Stage 1 is to go either to loc. 3 or 4. This is the overall optimum.

1

3

4

Looking at Stage 2 results, note that if coming from loc. 3, going to loc. 5 is preferred and if coming from loc. 4 go either to loc. 5 or 6.

1

3

4

5

6

Then for Stages 3 and 4:

8

9

101

3

4

5

6

There are 3 equally-optimal paths: 1,3,5,8,10 1,4,5,8,10, and 1,4,6,9,10. All have total costs of $11. Why did this work? Elimination of inferior subsets of decisions. Go through stages to show why. Travelling salesman problem: [Solution time is on the order of O(f(n))]

xkcd.com


77

DP Theory In solving this or any other DP problem we must formulate problems into: Stages: A decision must be made at each stage. States: When making a decision at a stage, there are a number of states (or conditions) we can be in when making the decision. (State variables must be discrete.) Decision Variable and options: At each stage we have a limited number of choices. (Decision variables must be discrete.) Objective function: Objective function Recursive Accumulated Objective Function: for usual "backwards recursion":

fn(S,Xn) = cn(S,Xn) + fn+1*(Sn+1= h(S,Xn)) Sn+1= h(S,Xn) is the “state transition” equation DP works by accumulating the optimal objective function as it moves through stages, eliminating bad decisions. (the "recursive relationship"). At the last stage: -) One optimal decision is specified, and -) The optimal objective function value is found. The rest of the optimal decisions are found by tracing from states to decisions for other stages. Backward DP Example: World Health Council Example (p. 339 in Hillier and Lieberman text). The WHC has 5 medical teams it can allocate among 3 countries. It would like to allocate these teams to maximize the additional person-years of life (A.P.Y.L.) for all countries combined. (A.P.Y.L. = increased life expectancy • Population) [NOTE: Another way is to increase population while keeping life expectancy the same.] Table 1 gives the return (in 1,000's of person-yrs) for each country. Table 1: 1,000's person-years return for each country [Pi(Xi)]

[Xi] No. of Teams 1 2 3 0 0 0 0 1 45 20 50 2 70 45 70 3 90 75 80 4 105 110 100 5 120 150 130

Formulated as a math program:

Max z = i=1

3 Pi(Xi )

S.T. i=1

3 Xi = 5

Xi = integer, i [Could solve by integer programming.]


78

For solution by DP: Stages: Let each country be a stage. The decision in Stage 3 will be how many teams go to Country

3? States: The incoming state of each will be the number of teams remaining to be allocated. Decision: How many teams should go to the country in a given stage if only 5 teams are remaining to be

allocated?

Objective function: Max z = i=1

3 Pi(Xi )

Accumulated objective function:

fn(S,Xn) = Pn (Xn) + f*n+1(S - Xn)

the "recursive relationship" Go over Again, we'll solve it backwards, starting with Stage/County 3. Stage 3 n = 3 S f*n(S) X*3 0 0 0 1 50 1 2 70 2 3 80 3 4 100 4 5 130 5

Essentially, allocate all that remains at the last Stage to Country 3. Stage 2 n = 2 f2(S,X2) = P2 (X2) + f*3(S – X2) S / X2: 0 1 2 3 4 5 f*2(S) X*2

0 0 -- -- -- -- -- 0 0 1 50 20 -- -- -- -- 50 0 2 70 70 45 -- -- -- 70 0 or 1 3 80 90 95 75 -- -- 95 2 4 100 100 115 125 110 -- 125 3 5 130 120 125 145 160 150 160 4

f3(4,2) = P2(2) + f3*(4-2=2) = 45 +70 = 115


79

Stage 1: At Stage 1, no allocations have been made, so S = 5. n = 1 f1(S,X1) = P1(X1) + f*2(S – X1) S: X1: 0 1 2 3 4 5 f*1(S) X*1 5 160 170 165 160 155 120 170 1

↓ =P1(1) + f*2(5-1=4) =45+125 = 170 The Solution The maximum number of additional person-years is 170,000. Trace the optimal decisions forward through the stages: Stage i Xi* S S - Xi* 1 1 5 4 = 5 - 1 2 3 4 1 = 4 - 3 3 1 1 0 = 1 - 1

Compare Calculations Needed for Enumeration Vs. DP Typical comparison is by number of cost calculations needed to solve the problem. For DP: No. of Cost Calculations =

Stage n = 1

nmax (number of states in stage n)*(number of decisions examined)

if Smax and dmax are same for all stages:

No. of cost cal. = nmax • Smax • dmax

For enumeration:

No. of cost cals. =

d n,max

n = 1

n max

If dmax is same for all stages:

No. cost cal. = dmax

nmax

For WHC problem: For DP: No. cost calc. = 6(1) + (6) (6) + 1(6) = 48 (including infeasible sol'ns)

= 6 + 21 + 6 = 33 (excluding infeasible sol'ns)

By Enumeration: No. of cost calc. - 6 • 6 • 6 = 63 = 216 (including infeasible sol'ns) DP requires 22% the calculation effort for this problem.


80

Forward DP WHC Example Revisited. Forward Accumulated Objective function:

fn(Sn+1,Xn) = Pn(Xn) + f*n-1(Sn = Sn+1 + Xn)

Concern of outgoing rather than incoming state. Stage 1: n = 1 Incoming state: S1 = 5

f1(X1) = P1(X1) = P1(S1 - S2) X1 = S1 - S2

Outgoing State S2 f*1 X*1 0 120 5 1 105 4 2 90 3 3 70 2 4 45 1 5 0 0

Stage 2 n = 2 f2 S3,X2 = P2 X2 + f1

* S3 + X2 = S2

Outgoing State S3

X2: 0 1 2 3 4 5

f2* S X 2

*

0

105 + 20 120 =125 135 145 155 150

155

4

1

70 + 45 105 110 =115 120 110 --

120

3

2

45 + 45 90 90 = 90 75 -- --

90

0,1 or 2

3

70 65 45 -- -- --

70

0

4

45 20 -- -- -- --

45

0

5

0 -- -- -- -- --

0

0

Stage 3 n = 3 f3 S4,X 3 = P3 X 3 + f2

* S4 + X 3 Outgoing State S4

X3: 0 1 2 3 4 5

f3* S 4 X 3

*

0

0+155 120+50 90+70 70+80 45+100 0+130 = 155 = 170 = 160 = 150 = 145 = 130

170

1

1,3,1 it works


81

More DP Applications Reservoir Releases Over Time from a Single Reservoir

given

I

S Q

t

tt

tttS + I1 = - +S tQ mass balance

The DP: Stages: time-periods, t States: Volume of water in reservoir at time t, St

Decisions: How much water to release at time t, Qt

Objective function: Max economic benefits =

1

,nt

t t tt

b S Q

Cumulative objective function: Backward moving:

ft(St, Qt)= b(St, Qt) + f*t+1(St+1=St - Qt + It)

Start with t = tn and move stages backwards to t = 1.

If Qt > St + It, then b(St, Qt) = - ; infeasible.

Stage tn n = tn ft(Stn, Qtn)= b(Stn, Qtn)

State Stn Qtn: 0 - - - - - - - - - - - - - - - - - - - - Qmax f*(Stn) Q*tn

0 | - - - - - - - - - - - - - - - - - - | Reservoir Capacity = Sp

Must discretize both -) Release values (m values)

-) Storage values (p values) Stage t n = t ft(St, Qt)= b(St, Qt) + f*

t+1(St+1=St - Qt + It)

State St Qt: 0 - - - - - - - - - - - - - - - - - - - - Qmax f*(St) Q*t

0 | - - - - - - - - - - - - - - - - - - | Reservoir Capacity = Sp


82

Stage t = 1 n = 1 f1(S1, Qt)= b(S1, Qt) + f*

2 (S2=S1 - Q1 + I1)

S1 Q1: 0 - - - - - - - - - - - - - - - - - - - - - - - Qmax f*(S1) Q*1

S1 = initial - - - - - - - - - - - - - - - - - - - - - -

storage value There is only one reservoir state in the present, S1. What would a forwards formulation be? ft(St, Qt)= bt(St, Qt) + f*

t-1(St-1=St + Qt - It), where St is the storage/state leaving stage t.

Computational effort to solve this reservoir problem: Let tn = number of stages m = number of decision alternatives per stage p = number of discretizations of the state variable Number of cost calculations by DP approx. tn *p*m Number of cost calculations by enumeration: approx. mt

n

Let's compare the computational effort for tn = 7 days, and p = m. Discretizations of m and p Calculations By DP Calculations by enumeration

10 7.10.10=700 10,000,000

100 70,000=7.104 1014 1,000 7,000,000=7.106 1021

It's possible to solve a much larger reservoir problem by DP than would otherwise be possible. Operating 2 Reservoirs in Series:

Stages: Time-periods t State 1: Storage in Reservoir 1: discretized values State 2: Storage in Reservoir 2: discretized values Decision 1: Release from Reservoir 1: discretized values Decision 2: Release from Reservoir 2: discretized values


83

Cumulative objective function: Backward moving:

ft(S1t,S2t, Q1t,Q2t) = bt(S1t,S2t, Q1t,Q2t) + f*t+1(S1t+It-Q1t, S2t+Q1t-Q2t)

Note: Conservation of mass equations for each reservoir determine the transition between states at each stage. S1t+1 = S1t+It-Q1t and S2t+1 = S2t+Q1t-Q2t. To set-up for each stage:

Stage t ft(S1t, S2t, Q1t, Q2t)

States Q1: 0 1 - - - - - - -

S1 S2 Q2: 0 1 2 3 - - - m 0 1 2 3 - - - m - - - - - - - f*t(S1t S2t) Q*1t Q*2t

0 0 1 2 | | p 1 0 1 2 | | p 2 3 | | p

Computational Effort:

No. of cost calculations by DP = tn • p1 • p2 • m1 • m2

For 2 reservoirs over 7 days, let p1 = p2 = m1 = m2 Discretizations of m Calculations By DP Calculations by enumeration

10 70,000 1014 100 700,000,000 = 7*108 1028

1,000 7*1012 1042 Calculations by enumeration = m2•7 General Comments: -) Multiple State Variables -) Multiple Decision Variables -) "Curse of Dimensionality"


84

Non-Linear Programming General Approach: Finding and comparing peaks and valleys in the objective function within a feasible region defined by the constraint set. Example Applications Profit maximization with price elasticity Transportation problems with volume discounts on shipping costs Portfolio selection with risky investments (also a multi-objective programming problem) Farm management considering agricultural production functions Hydropower scheduling Truss design Abbreviated Outline of Methods Unconstrained Nonlinear Optimization

Enumeration Analytical methods

Calculus Searching without derivatives

Golden Section and Fibonacci search methods Grid search method

Genetic algorithms Searching with first derivatives

One-dimensional (line) search procedure Gradient search (steepest descent) method (one of many “hill-climbing” methods) Quasi-Newton methods Conjugate gradient method

Searching with second derivatives Newton’s method Trust region methods

Constrained Nonlinear Optimization (Nominally, these are globally optimal for min. of convex functions over convex feasible regions.)

Analytical methods Lagrange multiplier method

Exterior penalty function methods Interior (barrier) function methods Gradient projection methods Generalized reduced gradient methods Successive linear or quadratic programming methods Genetic algorithms (constraints represented in “fitness” function)

Algorithms for Special Cases of NLP (Nominally, these are globally optimal.)

Separable programming (linear/mixed integer approximation) Quadratic programming (for minimizing convex functions) Dynamic programming


85

Non-Linear Search Procedures

Like the simplex method for linear programming, search procedures for nonlinear programming problems find a sequence of trial solutions that lead to an optimal solution. In each iteration, you begin with the current trial solution and proceed systematically to a new improved trial solution. Typically, you stop when either one of the following conditions are met:

(1) Your trial solutions have converged to a given value (i.e., successive solutions show little or no improvement).

(2) Your solution satisfies certain optimality criteria, such as the Karush-Kuhn-Tucker conditions. Fibonacci Search is an efficient method of finding the minimum of a single-variable function f(x) which strictly increases on either side of a single optimal value (i.e., is unimodal), considering a number of discrete values of the decision variable. In a sequence of n Fibonacci numbers, Fn, the next number in the sequence is the sum of the previous two numbers, e.g., 1, 1, 2, 3, 5, 8, 13, 21, 34.... Where there are (Fn - 1) ordered discrete values of the decision variable X with values xi, the Fibonacci search can find the minimum of the unimodal function f(X) in (n-1) iterations. For a minimization search over the interval a1 to b1: Initialization: Select a final solution tolerance, t > 0 and a value of n; (n-1) = number of desired iterations. Set index i = 1 L1 = a1 + (Fn-2/Fn)(b1 – a1) and U1 = a1 + (Fn-1/Fn)(b1 – a1) Step 1: If f(Li) < f(Ui), then:

The optimal value is below Ui. Set ai+1 = ai and bi+1 = Ui and Set Li+1 = ai+1 + (Fn-i-2/Fn-i)(bi+1 – ai+1) and Ui+1 = Li Step 2: If f(Li) >= f(Ui), then:

The optimal value is above Li. Set ai+1 = Li and bi+1 = bi and Set Li+1 = Ui and Ui+1 = ai+1 + (Fn-i-1/Fn-i)(bi+1 – ai+1) Step 3: Set i = i +1 Step 4: If i < n-1, go to Step 1 Step 5: If f(Li) > f(Ui), then: Final solution lies above Li, so final solution interval is [Li, bi]. ELSE, Final solution lies below Li, so final solution interval is [ai, Ui]. This search finds the optimal solution within a range of (b1-a1)/Fn. Fibonacci search is often used to set the step size (or number of steps, given a specified step size) in the gradient search method, described below. Wagner (1975, p. 455) has a nice example. Bazaraa, et al (1993) have excellent write-ups on this and related search methods. It can be derived that the Fibonacci search gives the narrowest interval for the optimal solution for any given number of objective function evaluations.


86

Golden Section Search is similar to the Fibonacci Search, but is a little simpler. Golden Section search is actually a special case of Fibonacci Search for large values of n. Let r = 0.5(SQRT(5) –1) ~ 0.618033…, which, while odd, turns out to be a very efficient way to divide up the search interval. For a minimization search over the interval a1 to b1: Initialization: Select a final solution tolerance, t > 0. The value of n is arbitrary, so long as it is large; (n-1) = number of desired iterations. Set index i = 1 L1 = a1 + (1-r)(b1 – a1) and U1 = a1 + r(b1 – a1) Step 1: If f(Li) < f(Ui), then:

The optimal value is below Ui. Set ai+1 = ai and bi+1 = Ui and Set Li+1 = ai+1 + (1-r)(bi+1 – ai+1) and Ui+1 = Li Step 2: If f(Li) >= f(Ui), then:

The optimal value is above Li. Set ai+1 = Li and bi+1 = bi and Set Li+1 = Ui and Ui+1 = ai+1 + r(bi+1 – ai+1) Step 3: Set i = i +1 Step 4: If bi – ai > t, go to Step 1 Step 5: If f(Li) > f(Ui), then: Final solution lies above Li, so final solution interval is [Li, bi]. ELSE, Final solution lies below Li, so final solution interval is [ai, Ui]. This search finds the optimal solution within a range of (b1-a1)*0.618(n-1). Note r2 = 1 - r. These search methods based on very odd numbers perform much better than simply choosing searching a number in the middle or other simple-minded approaches. Grid Search Method: A grid search evaluates points evenly spaced over a suitable grid in the decision space. While this method is easy to program, it is a form of approximate enumeration and too time-consuming for problems with more than a few decision variables. However, it can find an approximate solution or a good starting point for a more intensive search algorithms. Grid search methods are also useful for estimating the general shape of an objective function. The Search:


87

One-dimensional (line) search procedure: Starting with a trial solution to a minimization problem, one moves in the direction of the gradient (first derivative). (If the first derivative (slope) is positive, move to the left; if it is negative, move to the right.) Below the procedure is summarized using the mid-point rule:

Define: x’ = current trial solution x = current lower bound on x*

x = current upper bound on x* = error tolerance for x* Initialization: Select . Find initial values for x and x by inspection (or by finding any values

of x at which the derivatives are positive and then negative). Select an initial trial solution:

2'

xxx

Iteration: 1. Evaluate '

)(

xdx

xdf

2. If 0)'(

dx

xdf , set x = x’.

3. If 0)'(

dx

xdf , set x = x’.

4. Select a new 2

'xx

x

.

Stopping rule: If x - x < 2, Stop. Otherwise, perform another iteration.

This is a simple procedure in one dimension. For multiple decision variables, it can be applied to each variable in turn. This is also known as the univariate gradient search. Gradient search (steepest ascent/descent) method: This is another iterative method (based on first-order derivatives) which works easily for multiple dimensions. As in the one-dimensional search procedure, each iteration moves the trial solution in the direction of the gradient (i.e., downhill for a minimization problem). Additionally, one needs to decide how far to move downhill (the step size). At the optimal solution, 0*)( xf .

Consider:

nx

f

x

f

x

ff ,...,,)(

21x

t = step size (a single variable used in the 1-D search) The gradient search procedure for a minimization problem is summarized as: Initialization: Select and an initial trial solution x’. Go to the stopping rule. Iteration: 1. Express ))'( '( xx ftf as a function of t by setting

'

'

xx

j

jj x

ftxx for j = 1,….,n

and then substituting these expressions into f(x).


88

2. Use the one-dimensional line search procedure (or calculus or Fibonacci search) to find t = t* that minimizes ))'( '( xx ftf over t > 0.

3. Reset )'(* '' xxx ft and go to the stopping rule. Stopping rule: Evaluate )'(xf . Check if

jx

f for all j = 1,….,n.

If so, stop with the current x’ as the approximate optimal solution. Otherwise, perform another iteration.

u

u1

2

o

X1

1

X

X

X2

o

ouo

is to contour line of f(X ) .

X

As do most of the search methods discussed here, the gradient search only finds a local optimum. It also can have very slow convergence, especially for banana-shaped functions. Why?

X 1

2X


89

Newton's Method: An iterative search method that uses both first and second derivatives. Like many other search methods, the direction of the iteration is found using the first derivative (slope). The method then cleverly uses the second derivative at a point to predict how far to step to where the first derivative equals zero.

Derivation of Method:

Begin your search at a point X0, with f(X0) = value of the objective function.

The Taylor series of f(X) at X0 is:

f(X0 + X) = f(X0) + dX

Xdf )( 0 +

2

02 )(

2

1

dX

Xfd 2 + ...

Let X1 = X0 + X be your next estimate of the optimal solution X*.

Since

dX

df= 0 at any max or min, express

dX

df as a Taylor series, set = 0. The first-order Taylor series

of dX

df = 0 at X1 is:

dX

Xdf )( 1= 0 =

dX

Xdf )( 0+

2

02 )(

dX

Xfd X.

Manipulating, this becomes: X =

2

02

0

)(

)(

dX

Xfd

dX

Xdf

, or

X1 = X0 + X = X0

2

02

0

)(

)(

dX

Xfd

dX

Xdf

Repeat the use of this equation until

jx

f for all j = 1,…, n. Graphically,

What if there is more than one local optimum or more than one decision variable? The selection of the initial/starting solution is important.

1

f(X)

X

df/dx

X0 X1


90

Newton's Method for n-decision Variables This just applies the ideas of Newton’s method to n-decision dimensions. 1st-order condition for minimum or maximum of )(Xf :

0)(

iX

f X for i = 1,…, n.

Let 0X = initial trial solution, and consider the (multi-dimensional) Taylor series expansions:

,)()(

0)( 0200

XXXXX

T

jiii XX

f

X

f

X

f i = 1,…, n.

With n decision variables (Xi), there are n simultaneous equations like the one above:

,

)()(

)(

)()(

)()()(

)(0

2

02

1

02

2

02

22

02

12

02

1

02

21

02

21

02

0

...

X

XX

X

XX

XXX

X

T

nn

i

n

i

X

f

XX

f

X

f

X

f

XX

f

XX

f

XX

f

X

f

X

f

In the second term, the large matrix is H(X0), our old friend the Hessian matrix. We can solve for X using the inverse of the Hessian matrix:

i

f

X

XXHX

010 )(

Putting this into an iterative search scheme for n decision variables,

i

f

X

XXHXX

01001 )(

Continue iterations until:

i

f

X

X0

where is a convergence tolerance.


91

Sequential Linear Programming: An efficient technique for solving convex programming problems with nearly linear objective and constraint functions. As the name suggests, each of the approximating problems will be an LP problem that can be solved relatively efficiently. Also, successive problems will differ only by one constraint, so the dual simplex method can be used to solve the sequence of problems very efficiently. General procedure: 1. Start with an initial trial solution, xi (i = 0). 2. Use a first-order Taylor series approximation to linearize the objective and constraint functions about

the point xi:

iT

ii fff xxxxx )()()(

iT

ii ggg xxxxx )()()(

3. Formulate the approximating linear programming problem as

Minimize iT

ii ff xxxx )()(

Subject to

0)()( iT

ijij gg xxxx for j = 1,…, m.

4. Solve the approximating LP to find a new trial solution xi+1.

5. Evaluate the original constraints at xi+1; that is, find

)( 1ijg x , j = 1,…, m.

Example Sequential LP problem: Max 3X1 + 5X2 + 10 cos(X3) S.T. X1 + X2 + X3 20 -) Search over X3 (enumeration, Fibonacci, Newton, or Gradient Search)

-) LP for each value of X3 tried.


92

Quasi-Global Optimum Search Methods Relies on finding and comparing several local optima by starting local searches in different areas of the feasible region. Once a local optimum has been found, how can a new starting point be found that makes finding a different local optimum most likely? Approaches: 1) Eye-ball it, manually choosing starting points from different parts of the feasible region. 2) Randomly select new starting points. 3) Select a new starting point which maximizes the distance from all previously-examined points.

Step 1:Start a local optimum search starting from X0,1

and ending with X F1,1; X

F1,1 = X *,1.

Remember the best solution found X * and its objective function value f*.

Step 2: If k previous local searches have been undertaken over n decision variables; solve:

Max Z = j=1

k

i=1

n

l=1

Fj |Xi - Xil,k|

S.T. bXg

)(

Step 3: Use the solution to this linear program as a new starting point for a local search. Step 4: Go to step 2. The above linear program finds a point a maximum distance from all points previously examined. If Z* ≤ , STOP; Enough area has been searched. -) Computationally intensive -) Often only local optima are found. -) Harder to program; fewer nice solution packages. Example: Modeling to Generate Alternatives using HSJ. Brill, E.D., S.Y. Chang, and L.D. Hopkins (1982), “Modeling to generate alternatives: The HSJ approach,” Management Science, Vol. 28, No. 3, pp. 221-225.


93

Genetic Algorithms/Evolutionary Algorithms

xkcd.com

Genetic Algorithms Overview (Orit Kalman 2000) Genetic algorithms (GA) are adaptations of biological natural selection and adaptation processes to solving optimization problems. GA is based on the evolutionary concept that the characteristics of fit individuals in a population are promoted and evolved by means of recombination and mutations while weaker individuals die off. Fit individuals are promoted through selection based on environmental conditions, mating, and recombination of offspring carrying their parents’ traits (Holland, 1992). This natural optimization process is mimicked in solving a variety of operation, scheduling, and management problems.

GA is configured based on the structure of natural evolution. The first GA component is the representation of the environment in which a population or system is undergoing adaptation. In GA, the environment is represented by the fitness function that is used to evaluate the population. The fitness function is developed based on the problems’ objective function and constraints (Holland, 1992). Within the environment, a population exists based on an adaptive plan that includes selection, mating, and recombination. The population consists of strings (chromosomes) that define individuals in the population representing specific solutions to the problem. The strings are further divided into smaller bit-sets (genes) that define the unique characteristics of the individual and represent the population’s diversity. A comparison of the natural evolutionary process and the genetic algorithm elements appears below.

Evolution Genetic Algorithm Environmental selection Fitness function and Objective function Population Set of solutions Individual/Chromosome Solution string (set of decisions) Genes Bit sets (coded solution/decision variables)

Genetic Algorithm Operators GA is a modified spacious hill climbing optimization method. Common “hill climbing” methods search iteratively for an optimum but may be limited when the solution space has multiple


94

optima. GA addresses the challenge of multiple optima by searching through the whole solution space simultaneously through exploration. As in nature, GAs work by manipulating a population and creating future generations better adapted to the environment. Typical GA structure is shown in Figure 3-1. Four mechanisms control the GA: (1) initialization, (2) selection, (3) recombination, and (4) termination. Numerous GA operators have been described and used in the literature to perform these four mechanisms in efforts to enhance the reliability and convergence of GA solutions. Some of the promising and more commonly used GA operators are presented and reviewed.

In general, genetic or evolutionary algorithms have the following steps: 1. Initialization - Generate initial population of solutions 2. Evaluate fitness of individuals in population – Fitness function combines objective function and

constraints 3. Termination - Is fitness function value within termination limit? If so, STOP. 4. Selection and recombination - Create new population, then Go to Step 2.

Initialization The initialization of the genetic algorithm has two components: the manner of creating the initial population and the population’s size. Generally, initialization is accomplished by a random assignment based on a normal distribution. To improve the search process, the initial population may be seeded with individuals that are representative of the solution space or with extended random initialization (ERI). The process of ERI is based on a random selection of individuals that are forced to compete for a place in the initial population based on their fitness (Bramlette, 1991).

In addition to the importance of the initial population makeup, population size can greatly influence the efficiency and ability of the GA to locate the global optimum solution.

Typically, population size is constant and specified by the user. Some research has concentrated on developing methods to determine the optimal population size required to obtain reliable solutions (De Jong, 1975; Goldberg et al, 1993). Significant research has been done on methods such as crowding and generation gaps that provide dynamic population sizes, in efforts to improve GA efficiency (Mahfoud, 1995).

Selection Selection operators are used to improve the average quality of the population by selecting more fit individuals for the development of subsequent generations. Selection operators help focus the search on promising regions in the search space. Selection operators are important both in driving the search towards better individuals and in maintaining a high genotypic solution diversity of the population. These dual roles of selection operators create the challenge of balancing exploration versus exploitation. Exploitation is the passage of a gene or a solution trait from a parent to its offspring. Exploration is a new representation of a gene in an offspring (Eshelman and Schaffer, 1993). Exploiting the population by selecting the best individuals may lead to a narrow search and early convergence that often leads to a local rather than global optimum solution. On the other hand, exploration with little bias and random selection may lead to unfocused and inefficient search.

Several selection methods have been proposed to balance exploitation and exploration. Selection methods can be described as either preservative or extinctive. Preservative selections are indiscriminant selections in which each individual has a chance to contribute offspring to the


95

next generation while extinctive selections prohibit either weak or strong individuals from being selected for recombination. Generally, preservative selections produce higher diversity than extinctive selections at the risk of losing important information. Selection methods can be either Elitist or Pure. In pure selection, individuals do not compete with their offspring, but elitist selection allows some or all of the parents to undergo selection with their offspring, resulting in an ‘unlimited’ lifetime for super-fit individuals. The elitist selection in some cases may lead to premature loss of population diversity. Selection methods also can be generational in which a parent population is used solely for recombination, or steady state selection in which offspring that outperform their parents immediately replace their parents within the selection phase (Back and Hoffmeister, 1991).

The quality of the selection operators may be identified in terms of the time it takes the best individual to take over the population, the progress of the whole population, the average fitness changes, or the fitness distribution. Four common selections operators are described: the roulette wheel selection, random selection, tournament selection, and remainder stochastic sampling without replacement selection. Though the roulette wheel selection is most commonly used, the tournament selection method has been shown to be the most effective of the four operators in some cases (Goldberg and Deb, 1991).

The Roulette Wheel Selection The most common selection method is the fitness-proportionate roulette wheel algorithm. In proportionate selection methods, the probability of selection (pi) of an individual (i) can be calculated as the ratio of the individual’s fitness (fi) and the population average fitness becomes, pi = fi/f.

The Roulette wheel selection algorithm was named for its likeness to allocating pie-shaped slices on a roulette wheel to population members, with each slice proportional to the member’s fitness. Each roulette spin results in the selection of one parent. High performance individuals are assigned high selection probabilities (or larger slices on the roulette wheel) and therefore tend to be selected for generating new populations more often than individuals with low fitness values (Mitchell, 1996; Davis, 1991a; Bartlett, 1995). The roulette wheel selection algorithm is:

1. Determine Total fitness, the sum of all individuals’ fitness. 2. Choose a random number, R, uniformly distributed between 0 and total fitness. 3. Loop through the individuals in the population, summing the fitnesses until the sum is greater than or equal to R. The individual whose expected value puts the sum over this limit is the one selected.

Though roulette wheel selection works well for some GA applications, it has one major drawback that may reduce the GA’s reliability. The selection can cause an inadequate selective pressure in which extremely fit individuals take over the population, leading to a loss of diversity and premature convergence early in the search. This drawback can be partially addressed by linear normalization and ranking to replace selection by fitness with selection by ranking to improve the selection pressure in the population (Falkenauer, 1998).

Random Selection Random selection is the only selection method that does not consider fitness; individuals are randomly selected for recombination. This method provides high exploration but risks losing exploitation opportunities of highly fit solutions. Convergence to an optimal solution relies mainly on recombination and creation of new population operators.


96

Tournament Selection The roulette wheel selection and random selection provide two extreme selection choices; the roulette selection depends on performance whereas random mating is indiscriminant.

Tournament selection is an attempt to find a balance between the two methods. In tournament selection, n individuals randomly selected from the population (with or without replacement), compete for selection. The fittest selected individual is passed along to a parent generation that is used for recombination (Blickle and Thiele, 1995).

The tournament selection algorithm is: 1. Select n individuals from the population for the tournament. 2. Compare the selected individuals’ fitness and select the best for further genetic processing. 3. Repeat step 1 and 2 until the required number of parents to create a new population has been satisfied.

Since each tournament is performed independently, this selection method may suffer from the same sampling errors as the roulette wheel selection. Yet, the tournament selection has shown to work well when models perform selections and tournaments that are limited to sub populations (Hancock, 1995; Goldberg and Deb, 1991).

Remainder Stochastic Sampling without Replacement Selection This method attempts to improve the roulette wheel selection performance. As with the roulette wheel selection, fit individuals are allotted more chances to become parents. This selection method has two parts for selecting parents for recombination. First, an intermediate population is developed based on the fitness of the individuals in the population relative to the population’s average fitness (based on the roulette wheel selection). This step is then followed by a random selection of parents from the intermediate population (Goldberg, 1995). The remainder stochastic sampling without replacement selection algorithm is:

1. Select n individuals using a roulette wheel selection 2. Randomly select individuals from the pre-selected group to create a new population.

Recombination Recombination methods use selected parents to develop new individuals in an effort to improve subsequent generations. The most common recombination method is the crossover in which two individual parents exchange part of their code to produce a new individual. As with selection, recombination can significantly influence the balance between exploration and exploitation. In addition to crossover recombination, some GAs incorporate a mutation operator to improve genetic diversity.

Crossover Method The crossover method sections the chromosomes of two parents at randomly selected location(s) and switches the sections to create new offspring. The point of crossover is randomly selected and the crossover probability is generally set between 0.5 and 0.8. The rate of crossover and the population size have been shown to be critical in converging to an optimal solution. De Jong (1975) found that population size ranging between 50 and 100 with crossover rate of 0.6 and mutation rate of 0.001 per bit works best for his problem formulation. Yet, selecting crossover rates is highly dependent on the problem and research has been limited to empirical data (Davis, 1991b).


97

Some drawbacks have been identified with the use of a single point cross over. As the strings become longer, the single-point crossover method may lead to inefficient solutions since large portions of a string with information can be lost in the crossing process. Multiple point crossover operators have been suggested to resolve this problem. In addition, mutation is used to diversify the population by reintroducing solutions that may have been lost, particularly in advancing populations in which solutions are converging into optima. More generally, the success or failure of crossover largely depends on the fitness function, the encoding, and the interaction among all genetic algorithm operators (Mitchell, 1998).

Uniform Crossover Method The uniform crossover was developed in response to the loss of valuable information that was observed with a single-point crossover operator. Uniform crossover has been observed to be a reliable operator, particularly in combination with elitism (see description below) (Spears and De Jong, 1991; Schaffer et al, 1991). Uniform crossover can be regarded as a special case of the crossover operator. Each individual gene in the parents is randomly crossed to create two new offspring. For example, parents with five genes can be randomly crossed at five locations whereas with single-point crossover, random crossing would occur only at one location. Since uniform crossover selects genes randomly, mutation becomes less important in preserving solution information. The crossover probability can affect the GA reliability (probability that the final solution converges on the global optimum). Probability of 0.5 was chosen for the GA used in this research.

A range of crossover operators (single-point and multiple-points) has been compared with varying results depending on the fitness function used for testing. Spears and De Jong (1991) found that the uniform crossover, in which each gene in the solution is randomly selected for recombination, dominated over two-point crossover with small populations but the opposite was true with large populations. These results may have been due to high disruption exhibited by uniform crossover relative to the two-point crossover. Nevertheless, using low crossover probabilities, uniform crossover was superior to the two-point crossover.

Mutation Mutation is a “background” operator used to ensure that the population does not fixate on a limited gene pool that may lead to local optima. Each bit in the solution string may randomly be replaced by another to yield a new structure (Holland, 1992). A low probability of mutation applied with the crossover method has been shown to improve the GA reliability (Schaffer and Eshelman, 1991).

Creation of New Population Newly produced offspring are used to create subsequent generations. The creation of the new populations can be done by a simple replacement of parents with their offspring or by judicious comparison of existing and new individuals to ensure that the new generation is superior to the previous generations.

Replacement In replacement, offspring replace their parents to create a new population regardless of the offspring fitness (Cavicchio, 1970). The assumption behind this method is that offspring are developed from fit individuals and therefore will generally be similar or better than their parents. This assumption greatly depends on the function being optimized.


98

WeakParent In WeakParent, the offspring and their parents compete to be included in the new population. The offspring replace their parents only if they have higher fitness values. This comparison ensures that the offspring improve the fitness of the new population (Bartlett, 1995).

ChildRepWeak As with the WeakParent operator, offspring must compete to be included in the new population. Rather than comparing offspring to their parents, offspring are compared to the whole population and replace the weakest individuals in the population if they display a higher fitness value. The drawback of this method is that information carried by weak individuals may be lost and result in early convergence to a local optimum (Eshelman and Schaffer, 1993; Bartlett, 1995).

Restricted Tournament Restricted tournament is a crowding operator in which offspring are compared to a fixed number of randomly selected individuals in the population. The offspring are then compared to the individual that most closely resembles them in gene makeup (offspring and individual share the highest number of identical genes). The offspring are allowed to replace these individuals if their fitness is higher. The purpose of this operator is to improve the population without destroying information carried by other individuals prematurely (Harik, 1995). The tournament size can affect the convergence time to an optimal solution (Goldberg and Deb, 1991).

Other Operators In addition to the recombination operators, other operators have been suggested to improve convergence to global optimum. These operators are used to control and improve the convergence rate and reliability of the GA model particularly for problems with complex solution space that have several local optima.

Elitism Elitism was first introduced by De Jong (1975) to force the GA to retain a specified number of best individuals at each generation to ensure that they are not lost through selection or recombination. At the creation of a new generation, in addition to the new population operators, the elitism operator replaces the weakest individual with the fittest individual in the population with the hope of preserving fit information. Elitism was found by many users to improve GA performance. The main drawback of elitism is that although the population average fitness is improved, elitism may result in early conversion to a local optimum due to lost information (Mitchell, 1998).

Sharing and Niching Sharing and niching are used to define the solution landscape by developing subpopulations. Subpopulations are composed of individuals that share common traits such as gene characteristics or similar fitness values. In sharing, individuals that belong to the same subpopulations are penalized in the selection process to account for their similarities. Niching limits selection to occur within subpopulations to avoid premature convergence of sub-optimal individuals taking over the population. Niching encourages subpopulations convergence while keeping overall population diversity (Ryan 1995).

Termination GA termination is generally based on either the number of generations as specified by the user or the GA fitness function performance. GA performance can be defined based on average


99

population fitness, maximum fitness in the population, or rate of change of population fitness from one generation to the next.

Selection of Operators The success of a GA in any application can only be determined by experimentation since the performance of selected population size, selection, and recombination operators is highly dependent on the type of problem solved (Bagachi et al. 1991). Much research has been done on evaluation methods in an effort to define which operators will be most reliable for a host of problem types. In addition, to control premature convergence, some have suggested adaptive plans for the operators in which operators and their rates change during the GA search (Rosca and Ballard 1995; Davis 1991). References Bagachi, S, S. Uckun, Y. Miyabe, and K. Kawamura (1991). “Exploring Problem-Specific

Recombination Operators for Job Shop Scheduling.” In Proceedings of the Fourth International Conference on Genetic Algorithms. R.K. Belew and L.B.Booker (Ed) pp 10-17.

Bartlett, Geoff (1995). “Genie: A First G.A.” In Practical Handbook of Genetic Algorithms. Applications Volume I. Lance Chambers (Ed.) pp 31-56.

Cavicchio, D.J. (1970). Adaptive Search using Simulated Evolution, (Doctoral dissertation, University of Michigan).

Davis, Lawrence (1991a). “What is a Genetic Algorithm?” In The Handbook of Genetic Algorithms. L. Davis (Ed.) pp. 1-22. Van Nostrand Reinhold, New York.

Davis, Lawrence (1991b). “Parameterizing a Genetic Algorithm.” In The Handbook of Genetic Algorithms. L. Davis (Ed.) pp. 91-97. Van Nostrand Reinhold, New York.Harik, Georges R. (1995). “Finding Multimodal Solutions Using Restricted Tournament Selection.” In Proceedings of the Sixth International Conference on Genetic Algorithms. L.J. Eshelman (Ed.) pp 24-31.

De Jong, K.A. (1975). An Analysis of the Behavior of a Class of Genetic Adaptive Systems. Eshelman, Larry J. and J.D. Schaffer (1993). “Crossover’s Niche.” In Proceedings of the Fifth

International Conference on Genetic Algorithms. S. Forrest (Ed.) pp. 9-14. Goldberg, David E. (1989). Genetic Algorithms in Search, Optimization, and Machine Learning.

Addison-Wesley Publishing Company, Inc. Reading, Massachusetts. Goldberg, David E. and K. Deb (1991). “A Comparative Analysis of Selection Schemes Used in

Genetic Algorithms.” In Foundations of Genetic Algorithms. Bruce M. Spatz (Ed.) pp. 69-93. Morgan Kaufmann Publishers, Inc, San Mateo, CA.Holland, John, H. (1992). Adaptation in Natural and Artificial Systems. MIT Press, Cambridge, MA.

Mitchell, Melanie (1998). An Introduction to Genetic Algorithms. A Bradford Book, The MIT Press Cambridge, Massachusetts.

Rosca, Justinian P. and D.H. Ballard (1995). “Causality in Genetic Programming.” In Proceedings of the Sixth International Conference on Genetic Algorithms. Larry J. Eshelman (Ed.) pp. 256-263. Morgan Kaufmann Publishers, Inc. San Francisco CA.

Ryan (1995). Practical Handbook of Genetic Algorithms Vol. I. Lance Chambers (Ed.). Boca Raton: CRC Press.

Schaffer, David J. and L.J. Eshelman (1991). “On Crossover as an Evolutionarily Viable Strategy.” In Proceedings of the Fourth International Conference on Genetic Algorithms. Belew, R.K. and Booker L.B. (Ed.) pp. 61-67. University of California, Sand Diego. July 13-16, 1991.


100

The Final Days

"What we call results are beginnings." Ralph Waldo Emerson (1803-82) American writer, philosopher

-) Multi-Objective Optimization -) Viewing operations research in terms of Alternative Generation & Comparison • to suggest solutions • to get people talking • need to test and refine decisions suggested by optimization methods -) Review again the steps of O.R. application. -) Uses of operations research in practice Pre-Opt. & Post-Opt. Go back and put methods in context of Operations Research method. Pre-Opt.: (use notes from early in class) 1) Formulating the Problem as an O-R-problem -) What is the problem? -) Objective function definition(s) -) Decision variables -) Constraints -) Simplifications identified 2) Solve Mathematical Problem 3) Interpret Results (perhaps modify, re-solve, and re-interpret) Overview/Review of Solution Methods (Make a huge table) 1) Formulation limitations 2) Computational limitations 3) Advantages 4) Disadvantages 5) How does this method work? for: Trial & Error Enumeration Calculus Lagrange Multipliers Linear Programming (LP) Integer-LP Non-Linear Programming (NLP) Dynamic Programming (DP) Genetic Algorithms (GA)/Evolutionary Algorithms (EA) Discussion: why select one method over another.


101

Appendices I Derivation of the Lagrange Multiplier Method

II Problem Formulation: Some Guidelines

III Interpretation of Results: Some Guidelines

IV Introduction to Solving Linear Programs using LINDO

V The Simplex Algorithm

VI Selected Bibliography

xkcd.com


102

A Derivation of the Lagrange Multiplier Method Problem: Max Z = f(X1, X2) Subject To (S.T.): g(X1, X2) = b, b is a constant

If constraint could be reformulated as: X 2 = g1 X 1 , then the constraint could be substituted into f X 1, X 2 and solved as unconstrained optimization. (This is often not possible or expedient.)

First-order Conditions (2 of 'em) To Max Z, subject to the constraint, a first-order condition is:

(1) dfdX 1

= 0 = f

X1

+ f

X2

dX 2dX 1

The second part of the derivative accounts forthe second decision variable; dX 2dX 1

= 0 if there is no constraint.)

Another first-order condition comes from the constraint:

(2) dg

dX1 = 0 =

g

X1

+ g

X2

dX2dX1

This keeps us on the equality constraint; g(X1, X2) cannot vary (it must = b) as X1 varies.

Re-arranging both equations to isolate dX 2

dX 1:

Equation (3) contains two 1st order conditions and the constraint g(X1, X2) = b are now sufficient to find the extreme points of the problem. But they will often be awkward. Modifying (3), so derivatives on each side are w.r.t. only one decision variable:

(4) -f/X1

g/X1

= -f/X2

g/X2

= -f

b

Let Lagrange Multiplier . Breaking Equation (4) into two separate equations, both =, and re-arranging gives the final first-order conditions:

(5) 0 = f

X1

+ g

X1

and

(6) 0 = f

X2

+ g

X2

To this must be added the constraint

(7) g X1,X2 = b These are the same first-order conditions we get from maximizing or minimizing the Lagrangian: L = f(X1, X2) + [g(X1, X2) - b] That's why the Lagrange multiplier method works. Derivation is from: Donald A. Pierre (1969), Optimization Theory With Applications.


103

PROBLEM FORMULATION: SOME GUIDELINES Purpose of Formulation The formulation of a problem organizes the problem into a mathematical form that can be understood in terms of the real problem and, perhaps, solved by mathematical or numerical methods. Proper formulation is essential to solving the "right problem." Guidelines To formulate a problem as a mathematical program, decisions, objectives, and constraints must be identified and stated in mathematical terms. Some Steps for Problem Formulation Part I: Organize the problem into words by answering the following questions in words: Step 1) What decisions must be made? (E.g. Pipe diameters must be specified.) Step 2) What is the objective? What characterizes a "good" decision? (E.g., A "good" decision has a low total cost.) Step 3) What constraints restrict the decisions that can be made? (E.g., the pipe must convey a constant flow of 10 cfs.) Part II: Restate the decisions, objective(s), and constraints in mathematical terms. Step 4) Restate the decisions as decision variables. (E.g., D = pipe diameter (feet)) Define these decision variables completely with units. Step 5) Restate the objective as an objective function or objective functions. (E.g., MIN total cost = capital cost + present value of pumping cost = f(D) + p(D); Here, you would have to specify a capital cost function of pipeline diameter f(D) and a pumping cost function of pipeline diameter p(D)) Step 6) Restate the constraints as equations or inequalities. (E.g., Q = 10 = 2g D5/(8fL),

where L is the given pipe length and f is a (given?) friction factor) Note: The objective function and/or constraint equations must include decision variables. Part III: Summarize the mathematical formulation of the problem. Step 7) State the objective function. Step 8) State the constraints. This should look something like: MINIMIZE f(D) + p(D)

SUBJECT TO: Q = 2g D5/(8fL). Step 9) Add a list of variable definitions, including their units and specifying which variables are decision variables. Part IV: Look over the mathematical programming formulation of the problem and see if it makes sense. Does the math represent the logic inherent in the original word problem? Presenting Your Formulation Here are some ideas. • Use common-sense variable names. E.g., let QTW = the water flow for tomatoes in Winter. • Use capital letters for variables and small letters for constants in the formulation. • Organize your presentation in a systematic way. • Explain where the objective function and each constraint come from. But try to do this in a way that isn't tedious.


104

INTERPRETATION OF RESULTS: SOME GUIDELINES

Purposes of Interpreting Solutions An engineer's results must be used by other engineers and non-engineers, most of whom will be unfamiliar with any analytical approaches used to get the results. These clients or employers require your results in an informative, but simple and straight-forward form. They also need to know any important limitations of your analysis.

Some Guidelines for Presenting and Interpreting Results 1) Answer questions your client has. For our aggregate-mix problem, the client is likely to be a contractor or an engineer-estimator. They would want to know: -) How many cubic yards of material should be brought from each site? -) How much will this cost (i.e., the best value of the cost/objective function)? -) How much will it cost if the total aggregate volume required should change? -) How much will it cost if the specifications of the total aggregate should change? Answer these questions and any others you think are important. Note which important questions you cannot answer. Also qualify your answers when these qualifications are important or useful.

2) Present your results and interpretations in terms familiar to the your client. Who will be using your work and how can they best understand what your analysis has to say? For our aggregate mix problem, they would want these answers in terms of words and numbers with units. Telling them, "X1 = 3,333.33" is less useful than, "3,333 yd3 of aggregate should be taken from Site 1." For large problems, a tabular presentation is often best. Sensitivity analysis should also be presented in a useful form. Instead of, "1 = 6.33," it is much

more useful to write, "The cost of changing the total amount of aggregate required is $6.33/yd3, if the same sand and gravel specifications are maintained. This is given by 1." Since many engineers are not used to sensitivity analysis, it is often useful to mention where these numbers come from.

3) Structure your results. Often, the user first wants to know what the best solution is and how it performs on the objective function. The user of your work might then want to know how sensitive this solution is to changes in the problem's formulation, sensitivity analysis. You might draw the reader's attention to any important highly-sensitive parameters. You should comment on how well the real problem is represented by the optimization model. Finally, present an overview and highlight the most important aspects of your results and interpretations.

4) Sensitivity Analysis. Use Lagrange multipliers, slack variables, new model runs, etc. to answer important questions regarding the stability of your solutions and how the solution and its cost will change with reasonable changes in the formulation. Tables and figures are also useful for illustrating interpretations and sensitivities.

5) Tables, Figures, and Text. Use tables and figures to summarize results and text to highlight specific points.

6) Be brief and concise. Don't ramble. State your results in clear terms and move on. Use tables to present large quantities of results. This takes less time to read, and less time to write.

7) As a practicing engineer, your results and interpretation are your product. This product is only useful if it is right, understandable, and helps the client make an informed decision. A correct solution alone can be useless without a good presentation and interpretation.


105

INTRODUCTION TO SOLVING LINEAR PROGRAMS USING LINDO This exercise should help you get acquainted with the linear program solution package LINDO. In this exercise, you type the underlined words. Follow these commands with a return.

1) Open LINDO from the Programs menu under the Start menu.

2) You will be in an empty window called “untitled”.

3) Look under the Help menu, under “Contents”. This is an introduction to the help available within the LINDO program. Be sure to poke under the “File”>>”Take Commands” and “Reports” menus.

4) Return to the empty “untitled” window. You can use the “Window” menu to do this.

Type MIN 5X1 + 7X2

6) Type SUBJECT TO

7) Type X1 + X2 >= 10000. 8) Type -0.2X1 + 0.1X2 >= 0 9) Type 0.1X1 - 0.2X2 <= 0

10) Type END You have entered a linear program.

11) Under the Reports menu, hit Formulation 12) LINDO will ask you how much of the formulation you want to look at. Check “All” and then “OK”. LINDO will show you (in the Reports Window) the linear program you have just typed in. It will even

clean it up and standardize the formatting.

12) Under the Solve menu, hit Solve LINDO will now solve the LP and present some results (in the Reports Window). LINDO will then ask if you want to see more sensitivity analysis.

13) Hit “Yes” The sensitivity analysis will be presented in the Report window.

14) Under the File menu, hit Save as … This will save the reports window, including the formulation, solution, and sensitivity analysis. You specify the name and location of the file. 15) You can now quit LINDO. NOTES: a) The R.H.S. of all constraints must be only a constant. b) Often it is easier to use a text editor (e.g., Wordpad) to write out the formulation and “Take” the file into LINDO.


106

THE SIMPLEX ALGORITHM

The following steps form the fundamentals of the Simplex Algorithm for solving linear programs.

1) Set up an "initial tableau."

2) Is the trial solution represented by the initial tableau optimal? If all coefficients in Row 0 are 0, then the solution is optimal and you may STOP. If any coefficients in Row 0 of the tableau are < 0, the solution represented by this tableau is not the optimal solution and the set of basic variables must be changed.

3) Select a new basic variable. Select as a new basic variable the variable with the most negative coefficient in Row 0. The column for this variable is the "pivot column."

4) Select a new non-basic variable. For each row (except Row 0 and any rows where the pivot column elements are negative), divide the RHS entry of each row by the coefficient in the column of the new basic variable. The new non-basic variable is the old basic variable associated with the row having the minimum ratio. This row is called the "pivot row."

5) Update the tableau. Form a new tableau which has zero for each entry in the pivot column, except for the coefficient in the pivot row. The entry in both the pivot row and pivot column must be equal to one. Use linear algebra and Gaussian elimination techniques to do this. Replace the old basic variable with the new basic variable in the "Basic Variable" column. This new tableau represents a new corner-point solution.

6) Is this new solution optimal? If all coefficients in Row 0 of the new tableau are 0, then STOP; the solution is optimal. If any coefficients in Row 0 of the tableau are < 0, the solution represented by this tableau is not the optimal solution and the set of basic variables must be changed. GO TO STEP 3.

Reading the final tableau. The following tableau represents an optimal solution and final tableau for the problem: MAX 2X1 + X2 Subject To: 1) X1 + X2 4 2) X2 3

Basic Eqn. Coefficient of Variable Variable No. X1 X2 S1 S2 RHS Z 0 0 1 2 0 8 X1 1 1 1 1 0 4 S2 2 0 1 0 1 3

The values of each basic variable is given by the RHS column. The value of each non-basic variable is zero. Z = 8, X1 = 4, S2 = 3, X2 = 0, S1 = 0. The value of the Lagrange multiplier (shadow price) for each constraint is given by the coefficient in Row 0 under the associated slack variable (S1 or S2). 1 = 2; 2 = 0. The Reduced Cost of each coefficient in the objective function is given by the coefficient in Row 0 under the associated decision variable (X1 or X2). Can you obtain these results?


107

Selected Bibliography Here are some useful references for operations research practitioners. I used many of these books in putting together the notes for this class. If you read all of these books, you might discover some test or homework problems. Books on Methods: Sherali, Hanif D., C.M. Shetty, Mokhtar S. Bazaraa (1993), Nonlinear Programming Theory and Algorithms, John Wiley and Sons, N.Y. Excellent write-ups. Bertsekas, Dimitri P. (1999). Nonlinear Programming, Athena Publishing. Dantzig, George B. (1963), Linear Programming and Extensions, Princeton University Press, Princeton, NJ, 627 pp. The author is the inventor of the simplex method. It is a very good, if somewhat technical book on the ins and outs of the linear programming. Ecker, Joseph G. and Michael Kupferschmid (1988), Introduction to Operations Research, John Wiley and Sons, N.Y., 509 pp. A more mathematical, and consequently briefer and more straight-forward look at the most common operations research methods. Goicoechea, Ambrose, Don R. Hansen, and Lucien Duckstein (1982), Multiobjective Decision Analysis with Engineering and Business Applications, John Wiley and Sons, N.Y., 519 pp. An OK book on multi-objective optimization and decision-making methods. Hillier, Frederick S. and Gerald J. Lieberman (1990), Introduction to Operations Research, McGraw-Hill Publishing Co., N.Y., 954 pp. An excellent book, covering many topics in operations research with a minimum of mathematics. Not quite as comprehensive as Wagner's book, but with more exhaustive explanations. Intriligator, Michael D. (1971), Mathematical Optimization and Economic Theory, Prentice-Hall, Inc., Englewood Cliffs, NJ, 508 pp. A nice treatment of many optimization methods, with a nice introduction to optimal control theory. Written from the perspective of applied economic theory. Jensen, Paul A. and Honathan F. Baird (2003), Operations Research Models and Methods, John Wiley and Sons, Hoboken, NJ, 672 pp. Very nice, particularly on LP, network flow programming, and Markov chains. Taha, Hamdy A. (1992), Operations Research: An Introduction, MacMillan, N.Y., 822 pp. Another fine operations research text. Wagner, Harvey M. (1975), Principles of Operations Research, Prentice-Hall, Inc., Englewood Cliffs, NJ, 1039 pp. Still one of the most comprehensive books on optimization methods. Civil Engineering Applications: deNeufville, Richard (1990), Applied Systems Analysis: Engineering Planning and Technology Management, McGraw-Hill Publishing Co., N.Y., 470 pp. A nice mix of theory and applications. Hendrickson, Chris and Tung Au (1989), Project Management for Construction, Prentice-Hall, Inc., Englewood Cliffs, NJ, 537 pp. An excellent book on project management with some interesting applications of optimization.


108

Loucks, Daniel P., Jery R. Stedinger, and Douglas A. Haith (1981), Water Resources Systems Planning and Analysis, Prentice-Hall, Inc., Englewood Cliffs, NJ, 559 pp. An excellent overview of applications to water resource problems. Meredith, Dale D., Kam W. Wong, Ronald W. Woodhead, and Robert H. Wortman (1973), Design and Planning of Engineering Systems, Prentice-Hall, Englewood Cliffs, NJ, 393 pp. A nice overview of civil engineering applications of systems analysis. Ossenbruggen, Paul J. (1984), Systems Analysis for Civil Engineers, John Wiley and Sons, N.Y., 571 pp. A very nice book both introducing the theory of systems analysis and some applications in civil engineering. There are a fair numbers of errors in the text, however. Park, Chan S. and Gunter P. Sharp-Bette (1990), Advanced Engineering Economcs, John Wiley and Sons, N.Y., 740 pp. Nicely integrates optimization into engineering economics. Revelle, Charles S., Earl E. Whitlach, and Jeff R. Wright (1997), Civil and Environmental Systems Engineering, Prentice-Hall Upper Saddle R., NJ, 507 pp. Excellent treatment of linear programming, its applications, and engineering economics. Stark, Robert M. and Robert L. Nicholls (1972), Mathematical Foundations for Design: Civil Engineering Systems, McGraw-Hill, N.Y., 566 pp. A fine early text with interesting examples from a broad range of civil engineering applications. Some interesting papers: Higle, J.L. and S.W. Wallace (2003), “Sensitivity analysis and uncertainty in linear programming,”

Interfaces, Vol. 33, No. 4, pp. 53-60.

xkcd.com

Deterministic Optimization and Design

Documents

Transcript of Deterministic Optimization and Design