15.053 Thursday, April 25
-
Upload
calista-kennedy -
Category
Documents
-
view
29 -
download
3
description
Transcript of 15.053 Thursday, April 25
![Page 1: 15.053 Thursday, April 25](https://reader036.fdocuments.us/reader036/viewer/2022062301/568136df550346895d9e7a64/html5/thumbnails/1.jpg)
1
15.053 Thursday, April 25
• Nonlinear Programming Theory
• Separable programming
Handouts: Lecture Notes
![Page 2: 15.053 Thursday, April 25](https://reader036.fdocuments.us/reader036/viewer/2022062301/568136df550346895d9e7a64/html5/thumbnails/2.jpg)
2
Difficulties of NLP Models
LinearProgram:
NonlinearPrograms:
![Page 3: 15.053 Thursday, April 25](https://reader036.fdocuments.us/reader036/viewer/2022062301/568136df550346895d9e7a64/html5/thumbnails/3.jpg)
3
Graphical Analysis of Non-linear programsin two dimensions: An example
• Minimize
• subject to (x - 8)2 + (y - 9)2 ≤ 49 x ≥2 x ≤13 x + y ≤24
![Page 4: 15.053 Thursday, April 25](https://reader036.fdocuments.us/reader036/viewer/2022062301/568136df550346895d9e7a64/html5/thumbnails/4.jpg)
4
Where is the optimal solution?
Note: the optimalsolution is not at acorner point.It is where theisocontour first hitsthe feasible region.
![Page 5: 15.053 Thursday, April 25](https://reader036.fdocuments.us/reader036/viewer/2022062301/568136df550346895d9e7a64/html5/thumbnails/5.jpg)
5
Another example: Minimize(x-8)2 + (y-8)2
Then the globalunconstrainedminimum is alsofeasible.
The optimalsolution is not onthe boundary ofthe feasible region.
![Page 6: 15.053 Thursday, April 25](https://reader036.fdocuments.us/reader036/viewer/2022062301/568136df550346895d9e7a64/html5/thumbnails/6.jpg)
6
Local vs. Global Optima
Def’n: Let x be a feasible solution, then
– x is a global max if f(x) ≥ f(y) for every feasible y.
– x is a local max if f(x) ≥ f(y) for every feasible y sufficiently close to x (i.e., xj-ε ≤ yj ≤ xj+ ε for all j and some small ε).
There may be several locally optimal solutions.
![Page 7: 15.053 Thursday, April 25](https://reader036.fdocuments.us/reader036/viewer/2022062301/568136df550346895d9e7a64/html5/thumbnails/7.jpg)
7
When is a locally optimal solutionalso globally optimal?
• We are minimizing. The objective function is convex. The feasible region is convex.
![Page 8: 15.053 Thursday, April 25](https://reader036.fdocuments.us/reader036/viewer/2022062301/568136df550346895d9e7a64/html5/thumbnails/8.jpg)
8
Convexity and Extreme Points
We say that a set S is convex, if for everytwo points x and y in S, and for every realnumber λ in [0,1], λx + (1-λ)y ε S.
The feasible region of a linear program is convex.
We say that an element w ε S is anextreme point (vertex, corner point), if wis not the midpoint of any line segmentcontained in S.
![Page 9: 15.053 Thursday, April 25](https://reader036.fdocuments.us/reader036/viewer/2022062301/568136df550346895d9e7a64/html5/thumbnails/9.jpg)
9
Recognizing convex feasible regions
• If all constraints are linear, then the feasible region is convex
• The intersection of convex regions is convex
• If for all feasible x and y, the midpoint of x and y is feasible, then the region is convex (except in totally non-realistic examples. )
![Page 10: 15.053 Thursday, April 25](https://reader036.fdocuments.us/reader036/viewer/2022062301/568136df550346895d9e7a64/html5/thumbnails/10.jpg)
10
Which are convex?
![Page 11: 15.053 Thursday, April 25](https://reader036.fdocuments.us/reader036/viewer/2022062301/568136df550346895d9e7a64/html5/thumbnails/11.jpg)
11
Convex FunctionsConvex Functions: f( y + (1- )z) f(y) + (1- )f(z)for every y and z and for 0. e.g., f((y+z)/2) f(y)/2 + f(z)/2
We say “strict” convexity if signis “<” for 0.
Line joining any pointsis above the curve
![Page 12: 15.053 Thursday, April 25](https://reader036.fdocuments.us/reader036/viewer/2022062301/568136df550346895d9e7a64/html5/thumbnails/12.jpg)
12
Convex FunctionsConvex Functions: f( y + (1- )z) ≥ f(y) + (1- )f(z)for every y and z and for 0. e.g., f((y+z)/2) ≥ f(y)/2 + f(z)/2
We say “strict” convexity if signis “<” for 0.
Line joining any pointsis above the curve
![Page 13: 15.053 Thursday, April 25](https://reader036.fdocuments.us/reader036/viewer/2022062301/568136df550346895d9e7a64/html5/thumbnails/13.jpg)
13
Classify as convex or concave or bothor neither.
![Page 14: 15.053 Thursday, April 25](https://reader036.fdocuments.us/reader036/viewer/2022062301/568136df550346895d9e7a64/html5/thumbnails/14.jpg)
14
What functions are convex?
• f(x) = 4x + 7 all linear functions• f(x) = 4x2 – 13 some quadratic functions• f(x) = ex
• f(x) = 1/x for x > 0• f(x) = |x|• f(x) = - ln(x) for x > 0
Sufficient condition: f”(x) > 0 for all x.
![Page 15: 15.053 Thursday, April 25](https://reader036.fdocuments.us/reader036/viewer/2022062301/568136df550346895d9e7a64/html5/thumbnails/15.jpg)
15
What functions are convex?
• If f(x) is convex, and g(x) is convex. Then so is h(x) = a f(x) + b g(x) for a>0, b>0.
• If y = f(x) is convex, then {(x,y) : f(x) ≤ y} is a convex set
![Page 16: 15.053 Thursday, April 25](https://reader036.fdocuments.us/reader036/viewer/2022062301/568136df550346895d9e7a64/html5/thumbnails/16.jpg)
16
Local Maximum (Minimum)Property
• A local max of a concave function on a convex feasible region is also a global max.• A local min of a convex function on a convex feasible region is also a global min.• Strict convexity or concavity implies that the global optimum is unique.• Given this, we can exactly solve: – Maximization Problems with a concave objective function and linear constraints – Minimization Problems with a convex objective function and linear constraints
![Page 17: 15.053 Thursday, April 25](https://reader036.fdocuments.us/reader036/viewer/2022062301/568136df550346895d9e7a64/html5/thumbnails/17.jpg)
17
Which are convex feasible regions?
(x, y) : y ≤ x2 + 2
(x, y) : y ≥ x2 + 2
(x,y) : y = x2 + 2
![Page 18: 15.053 Thursday, April 25](https://reader036.fdocuments.us/reader036/viewer/2022062301/568136df550346895d9e7a64/html5/thumbnails/18.jpg)
18
More on local optimality
• The techniques for non-linear optimization minimization usually find local optima.• This is useful when a locally optimal solution is a globally optimal solution• It is not so useful in many situations.
• Conclusion: if you solve an NLP, try to find out how good the local optimal solutions are.
![Page 19: 15.053 Thursday, April 25](https://reader036.fdocuments.us/reader036/viewer/2022062301/568136df550346895d9e7a64/html5/thumbnails/19.jpg)
19
Solving NLP’s by Excel Solver
![Page 20: 15.053 Thursday, April 25](https://reader036.fdocuments.us/reader036/viewer/2022062301/568136df550346895d9e7a64/html5/thumbnails/20.jpg)
20
Finding a local optimal for a singlevariable NLP
Solving NLP's with One Variable:
max f(θ)s.t. a ≤ θ ≤ b
Optimal solution is either a boundary point or satisfies f′(θ∗) = 0 and f ″(θ∗) < 0.
![Page 21: 15.053 Thursday, April 25](https://reader036.fdocuments.us/reader036/viewer/2022062301/568136df550346895d9e7a64/html5/thumbnails/21.jpg)
21
Solving Single Variable NLP (contd.)
If f(θ) is concave (or simply unimodal) and differentiable
max f(θ)s.t. a ≤ θ ≤ b
Bisection (or Bolzano) Search:
• Step 1. Begin with the region of uncertainty for θ as [a,b]. Evaluate f ′ (θ) at the midpoint θΜ =(a+b)/2.
• Step 2. If f ′ (θΜ) > 0, then eliminate the interval up to θΜ.If f′(θΜ) < 0, then eliminate the interval beyond θΜ.
• Step 3. Evaluate f′(θ) at the midpoint of the newinterval. Return to Step 2 until the interval of uncertaintyis sufficiently small.
![Page 22: 15.053 Thursday, April 25](https://reader036.fdocuments.us/reader036/viewer/2022062301/568136df550346895d9e7a64/html5/thumbnails/22.jpg)
22
Unimodal Functions
• A single variable function f is unimodal if there is at most one local maximum (or at most one local
minimum) .
![Page 23: 15.053 Thursday, April 25](https://reader036.fdocuments.us/reader036/viewer/2022062301/568136df550346895d9e7a64/html5/thumbnails/23.jpg)
23
Other Search Techniques
• Instead of taking derivatives (which may be computationally intensive), use two function evaluations to determine updated interval.• Fibonacci Search• Step 1. Begin with the region of uncertainty for θ as [a, b]. Evaluate f (θ1) and f (θ2) for 2 symmetric points θ1<θ2.• Step 2. If f (θ1) ≤ f (θ2), then eliminate the interval up tθ1. If f (θ1) > f (θ2), then eliminate the interval beyond θ2.• Step 3. Select a second point symmetric to the point already in the new interval, rename these points θ1 and θ2 such that θ1<θ2 and evaluate f (θ1) and f (θ2). Return to Step 2 until the interval is sufficiently small.
![Page 24: 15.053 Thursday, April 25](https://reader036.fdocuments.us/reader036/viewer/2022062301/568136df550346895d9e7a64/html5/thumbnails/24.jpg)
24
On Fibonacci search
1, 1, 2, 3, 5, 8, 13, 21, 34At iteration 1, the length of the search interval is the kth fibonacci number for some kAt iteration j, the length of the search interval is the k-j+1 fibonacci number.The technique converges to the optimal when the function is unimodal.
![Page 25: 15.053 Thursday, April 25](https://reader036.fdocuments.us/reader036/viewer/2022062301/568136df550346895d9e7a64/html5/thumbnails/25.jpg)
25
Finding a local maximum usingFibonacci Search.
Length of search
interval
Where the maximum may be
![Page 26: 15.053 Thursday, April 25](https://reader036.fdocuments.us/reader036/viewer/2022062301/568136df550346895d9e7a64/html5/thumbnails/26.jpg)
26
The search finds a local maximum, butnot necessarily a global maximum.
![Page 27: 15.053 Thursday, April 25](https://reader036.fdocuments.us/reader036/viewer/2022062301/568136df550346895d9e7a64/html5/thumbnails/27.jpg)
27
The search finds a local maximum, butnot necessarily a global maximum.
![Page 28: 15.053 Thursday, April 25](https://reader036.fdocuments.us/reader036/viewer/2022062301/568136df550346895d9e7a64/html5/thumbnails/28.jpg)
28
Number of function evaluations inFibonacci Search
• As new point is chosen symmetrically, the length lk of successive search intervals is given by: lk= lk+1 + lk+2 .• Solving for these lengths given a final interval length of 1, ln = 1, gives the Fibonacci numbers: 1, 2, 3, 5, 8, 13, 21, 34,…• Thus, if the initial interval has length 34, it takes 8 function calculations to reduce the interval length to 1.
• Remark: if the function is convex or unimodal, then fibonacci search converges to the global maximum
![Page 29: 15.053 Thursday, April 25](https://reader036.fdocuments.us/reader036/viewer/2022062301/568136df550346895d9e7a64/html5/thumbnails/29.jpg)
29
Separable Programming
• Separable programs have the form:
Max
st
Each variable xj appears separately, one ineach function gij and one in each function fj inthe objective.
Each non-linear function is of a single variable.
![Page 30: 15.053 Thursday, April 25](https://reader036.fdocuments.us/reader036/viewer/2022062301/568136df550346895d9e7a64/html5/thumbnails/30.jpg)
30
Examples of SeparablePrograms
f(x1,x2) = x1(30-x1)+x2(35-x2)-2x12-3x2
2
f(x1,x2) = x15 + -18e-x2-4x2
3
x1
f(x1,x2,x2) = lnx15-sinx2-x3e-x3)+7x1
-4
![Page 31: 15.053 Thursday, April 25](https://reader036.fdocuments.us/reader036/viewer/2022062301/568136df550346895d9e7a64/html5/thumbnails/31.jpg)
31
Approximating a non-linear functionwith a piecewise linear function.
• Aspect 1. Choosing the approximation.• Aspect 2. When is the piecewise linear approximation a linear program in disguise?
![Page 32: 15.053 Thursday, April 25](https://reader036.fdocuments.us/reader036/viewer/2022062301/568136df550346895d9e7a64/html5/thumbnails/32.jpg)
32
Approximating a non-linearfunction of 1 variable
![Page 33: 15.053 Thursday, April 25](https://reader036.fdocuments.us/reader036/viewer/2022062301/568136df550346895d9e7a64/html5/thumbnails/33.jpg)
33
Approximating a non-linear functionof 1 variable: the λ method
Choosedifferentvalues of x toapproximatethe x-axis
Approximateusingpiecewiselinearsegments
![Page 34: 15.053 Thursday, April 25](https://reader036.fdocuments.us/reader036/viewer/2022062301/568136df550346895d9e7a64/html5/thumbnails/34.jpg)
34
More on the method
Suppose that for –3 ≤ x ≤ -1,
we represent x hasλ1 (-3) +λ2 (-1)
whereλ1 +λ2 = 1 andλ1,λ2 ≥ 0
Then we approximate f(x) as
λ1 (-20) +λ2 (-7 1/3)
a1 = -3, f(a1) = -20
a2 = -1 f(a2) = -7 1/3
![Page 35: 15.053 Thursday, April 25](https://reader036.fdocuments.us/reader036/viewer/2022062301/568136df550346895d9e7a64/html5/thumbnails/35.jpg)
35
More on the method
a2 = -1, f(a2) = -7 1 /3
a3 = 1 f(a3) = -2 2/3
Suppose that for –1 ≤ x ≤ 1,
we represent x hasλ2 (-3) +λ3 (-1)whereλ2 +λ3 = 1 andλ2,λ3 ≥ 0
How do weapproximate f( ) inthis interval?
What if –3 ≤ x ≤ 1?
![Page 36: 15.053 Thursday, April 25](https://reader036.fdocuments.us/reader036/viewer/2022062301/568136df550346895d9e7a64/html5/thumbnails/36.jpg)
36
Almost the method
Original problem:
min x3/3 + 2x – 5 + more terms
s.t. -3 ≤ x ≤ 3 + many more constraints
a1 = -3; a2 = - 1; a3 = 1; a4 = 3f(a1) = -20; f(a2) = -7 1/3; f(a3) = -2 2/3; f(a4) = 4
Approximated problem:
minλ1f(a1) +λ2f(a2) +λ3f(a3) +λ4f(a4)
more linear terms
s.t. λ1 + λ2 + λ3 + λ4= 1 ; λ ≥ 0
+ many more constraints
![Page 37: 15.053 Thursday, April 25](https://reader036.fdocuments.us/reader036/viewer/2022062301/568136df550346895d9e7a64/html5/thumbnails/37.jpg)
37
Why the approximation is incorrect
Approximated problem:
minλ1f(a1) +λ2f(a2) +λ3f(a3) +λ4f(a4)
more linear terms
s.t. λ1+λ2 +λ3+λ4 = 1 ; λ ≥ 0Considerλ1= 1/2 ;λ2=0 ;λ3=1/2 ;λ4 = 0;
The method givesthe correctapproximation ifonly twoconsecutiveλs arepositive.
![Page 38: 15.053 Thursday, April 25](https://reader036.fdocuments.us/reader036/viewer/2022062301/568136df550346895d9e7a64/html5/thumbnails/38.jpg)
38
Adjacency Condition
1. At most two weights (λs) are positive2. If exactly two weights (λs) are positive, then they are λj and λj+1 for some j3. The same condition applies to every approximated function.
![Page 39: 15.053 Thursday, April 25](https://reader036.fdocuments.us/reader036/viewer/2022062301/568136df550346895d9e7a64/html5/thumbnails/39.jpg)
39
Approximating a non-linear objectivefunction for a minimization NLP. original problem: minimize {f(y): y ∈P} Suppose that y = Σjjaj , where Σjj = 1 and >= 0 . Approximate f(y). minimize {Σjjf(aj): Σjjaj∈P}
• Note: when given a choice of representing y in alternative ways, the LP will choose one that leads to the least objective value for the approximation.
![Page 40: 15.053 Thursday, April 25](https://reader036.fdocuments.us/reader036/viewer/2022062301/568136df550346895d9e7a64/html5/thumbnails/40.jpg)
40
For minimizing a convex function, the-method automatically satisfies theadditional adjacency property.
min Z =λ1f(a1) +λ2f(a2) +λ3f(a3) +λ4f(a4)+λ5f(a5) s.t. λ1 + λ2 + λ3 + λ4 +λ5 = 1; ≥0 + adjacency condition + other constraints
![Page 41: 15.053 Thursday, April 25](https://reader036.fdocuments.us/reader036/viewer/2022062301/568136df550346895d9e7a64/html5/thumbnails/41.jpg)
41
Feasible approximated objective functions without the adjacency conditions
min Z =λ1f(a1) +λ2f(a2) +λ3f(a3) +λ4f(a4)+λ5f(a5) s.t. λ1 + λ2 + λ3 + λ4 +λ5 = 1; ≥0 + other constraints
![Page 42: 15.053 Thursday, April 25](https://reader036.fdocuments.us/reader036/viewer/2022062301/568136df550346895d9e7a64/html5/thumbnails/42.jpg)
42
But a minimum in this case always occurson the piecewise linear curve.
min Z =λ1f(a1) +λ2f(a2) +λ3f(a3) +λ4f(a4)+λ5f(a5) s.t. λ1 + λ2 + λ3 + λ4 +λ5 = 1; ≥0 + other constraints
![Page 43: 15.053 Thursday, April 25](https://reader036.fdocuments.us/reader036/viewer/2022062301/568136df550346895d9e7a64/html5/thumbnails/43.jpg)
43
Separable Programming (in the case oflinear constraints)
• Begin with an NLP:
• Transform to Separable:
• Approximate using the Method:
Max f ( )xs.t. Dx d x 0
Max ∑fj (xj) s.t. Dx d x
0
n
j=1
![Page 44: 15.053 Thursday, April 25](https://reader036.fdocuments.us/reader036/viewer/2022062301/568136df550346895d9e7a64/html5/thumbnails/44.jpg)
44
Approximation
• Re-express in terms of variables:
If the original problem is concave, then you can drop theadjacency conditions (they are automatically satisfied.)
and the adjacency conditions
![Page 45: 15.053 Thursday, April 25](https://reader036.fdocuments.us/reader036/viewer/2022062301/568136df550346895d9e7a64/html5/thumbnails/45.jpg)
45
How can one construct separable functions?
Term Substitution Constraints Restriction
None
None
None
![Page 46: 15.053 Thursday, April 25](https://reader036.fdocuments.us/reader036/viewer/2022062301/568136df550346895d9e7a64/html5/thumbnails/46.jpg)
46
Transformation Examples
Ex : (x1 + x2 + x3)6
Substitute y6 and let y = x1 + x2 + x3
x1x22
1+ x3
Ex : let y1and let y2
and add the constraint
![Page 47: 15.053 Thursday, April 25](https://reader036.fdocuments.us/reader036/viewer/2022062301/568136df550346895d9e7a64/html5/thumbnails/47.jpg)
47
NLP Summary
• Convex and Concave functions as well as convex sets are important properties• Bolzano and Fibonacci search techniques – used to solve single variable unimodal functions• Separable Programming – nonlinear objective and nonlinear constraints that are separable – General approximation technique