Introduction to Optimization - CERMICS – Centre d...

Introduction

Optimalityconditions

Introduction to Optimization

Pedro Gajardo1 and Eladio Ocana2

1Universidad Tecnica Federico Santa Marıa - Valparaıso -Chile

2IMCA-FC. Universidad Nacional de Ingenierıa - Lima - Peru

Mathematics of Bio-Economics (MABIES) - IHP

January 29, 2013 - Paris

Introduction

Contents

Introduction

Optimality conditions

Introduction

Optimization problems

In order to formulate an optimization problem, the followingconcepts must be very clear:

decision variables

restrictions

objective function

Introduction

decision variables

restrictions

objective function

Introduction

decision variables

restrictions

objective function

Introduction

Optimization problemsDecision variables

One or morevariableson whichwe can decide(harvestingrate or effort, level of investment, distribution of tasks,parameters)

Objective: to find thebestvalue for the decision variable

We denote byx the decision variable

x can be a number, a vector, a sequence, a function, ..........

In a general framework, we denote byX the set where thedecision variable is (x ∈ X)

Introduction

One or more variables on which we can decide (harvestingrate or effort, level of investment, distribution of tasks,parameters)

Objective:to find thebestvalue for the decision variable

Introduction

Wedenote byx the decision variable

Introduction

In a general framework,we denote byX the set where thedecision variable is (x ∈ X)

Introduction

Warning

This tutorial is onContinuous Optimization

That means that decision variables are continuous variables

What that means?

Firstly, the setX where the decision variable belongs, is a(linear) vector space

Introduction

Warning

What that means?

Introduction

Warning

What that means?

Introduction

Warning

What that means?

Introduction

Optimization problemsRestrictions

Constraints that the decision variable has to satisfy

If for a certain value of the decision variable therestrictions are satisfied, we say that it is a feasiblesolution

In a general framework, we denote byC ⊆ X the set of allfeasible solutions

If we have two decision variables,x1 andx2 and they haveto satisfy the following constraints:x1 ≥ 0, x2 ≥ 0,2 x1 + 3 x2 ≤ 5, we denote

C = {(x1, x2) ∈ R2 | x1 ≥ 0, x2 ≥ 0, 2 x1 + 3x2 ≤ 5}

Introduction

If for a certain value of thedecision variable therestrictions are satisfied, we say that it is a feasiblesolution

C = {(x1, x2) ∈ R2 | x1 ≥ 0, x2 ≥ 0, 2 x1 + 3x2 ≤ 5}

Introduction

In a general framework,we denote byC ⊆ X the set of allfeasible solutions

C = {(x1, x2) ∈ R2 | x1 ≥ 0, x2 ≥ 0, 2 x1 + 3x2 ≤ 5}

Introduction

C = {(x1, x2) ∈ R2 | x1 ≥ 0, x2 ≥ 0, 2 x1 + 3x2 ≤ 5}

Introduction

Warning

What that means?

Secondly, the setC of feasible solutions is not a discrete set (setof isolated points)

Introduction

Warning

What that means?

Secondly, the setC of feasible solutions is not a discrete set (setof isolated points)

Introduction

Optimization problemsObjective function

It is the mathematical representation formeasuring thegoodnessof values for the decision variable

This representation is done through a function

f : X −→ R

Introduction

Optimization problemsObjective function

It is the mathematical representation for measuring thegoodnessof values for the decision variable

This representation is done through a function

f : X −→ R

Introduction

To find thebestvalue for the decision variable from all feasiblesolutions

To find x ∈ C ⊆ X such that

f (x) ≤ f (x) (or f (x) ≥ f (x)) for all x ∈ C

minx∈X

or maxx∈X

subject to

x ∈ C

Introduction

minx∈X

or maxx∈X

subject to

x ∈ C

Introduction

minx∈X

or maxx∈X

subject to

x ∈ C

Introduction

Optimization problemsExample: maximizing a rectangular surface

With a given fence, to enclose the largest rectangular area

Objective: maximize the rectangular surface

Decision variables: lengths of the rectangular figure

Constraints:

positive lengths

length of the fence

Introduction

Constraints:

positive lengths

length of the fence

Introduction

Constraints:

positive lengths

length of the fence

Introduction

Constraints:

positive lengths

length of the fence

Introduction

Constraints:

positive lengths

length of the fence

Introduction

Decision variables: a (height) andb (width)

Objective: maximizes(a,b) = a b

Constraints:positive lengths:a > 0, b > 0

length of fence (L > 0): 2(a+ b) = L

b = L2 − a

New objective functionf (a) = a(

L2 − a

Rewrite restrictions:a > 0, a < L2

Introduction

Decision variables:a (height) andb (width)

b = L2 − a

L2 − a

Introduction

b = L2 − a

L2 − a

Introduction

b = L2 − a

L2 − a

Introduction

b = L2 − a

L2 − a

Introduction

b = L2 − a

L2 − a

Introduction

b = L2 − a

Newobjective functionf (a) = a(

L2 − a

Introduction

b = L2 − a

L2 − a

Rewriterestrictions:a > 0, a < L2

Introduction

maxa∈R

f (a) = a(

L2 − a

subject to

a < L2

Introduction

Optimization problemsExample: Optimizing a portafolio

A firm wishes to maximize the utilities of its portafolio

Objective: maximize profits

Decision variables: amount to invest in each fund

Constraints:

nonnegative investments

budget restrictions

minimal or maximal bounds for investments

Introduction

Constraints:

budget restrictions

Introduction

Constraints:

budget restrictions

Introduction

Constraints:

budget restrictions

Introduction

Constraints:

budget restrictions

Introduction

Constraints:

budget restrictions

Introduction

Decision variables: x1, x2, . . . , xn, wherexj the quantity toinvest in the fundj

Objective: maximize

(1+ r1)x1 + (1+ r2)x2 + . . .+ (1+ rn)xn

wherer j is the rentability of the fundj

Restrictions:

Non-negative investments:xj ≥ 0 j = 1, 2, . . . , n

Budget restriction:

x1 + x2 + . . .+ xn ≤ B

Minimal/maximal bounds for investments:

aj ≤ xj ≤ bj for j = 1, 2, . . . , n

Introduction

Decision variables:x1, x2, . . . , xn, wherexj the quantity toinvest in the fundj

Objective: maximize

(1+ r1)x1 + (1+ r2)x2 + . . .+ (1+ rn)xn

Restrictions:

Budget restriction:

x1 + x2 + . . .+ xn ≤ B

aj ≤ xj ≤ bj for j = 1, 2, . . . , n

Introduction

Objective: maximize

(1+ r1)x1 + (1+ r2)x2 + . . .+ (1+ rn)xn

Restrictions:

Budget restriction:

x1 + x2 + . . .+ xn ≤ B

aj ≤ xj ≤ bj for j = 1, 2, . . . , n

Introduction

Objective: maximize

(1+ r1)x1 + (1+ r2)x2 + . . .+ (1+ rn)xn

Restrictions:

Non-negative investments: xj ≥ 0 j = 1, 2, . . . , n

Budget restriction:

x1 + x2 + . . .+ xn ≤ B

aj ≤ xj ≤ bj for j = 1, 2, . . . , n

Introduction

Objective: maximize

(1+ r1)x1 + (1+ r2)x2 + . . .+ (1+ rn)xn

Restrictions:

Budget restriction:

x1 + x2 + . . .+ xn ≤ B

aj ≤ xj ≤ bj for j = 1, 2, . . . , n

Introduction

Objective: maximize

(1+ r1)x1 + (1+ r2)x2 + . . .+ (1+ rn)xn

Restrictions:

Budget restriction:

x1 + x2 + . . .+ xn ≤ B

aj ≤ xj ≤ bj for j = 1, 2, . . . , n

Introduction

maxx1,x2,...,xn∈R

(1+ r1)x1 + (1+ r2)x2 + . . .+ (1+ rn)xn

subject to

xj ≥ 0 j = 1,2, . . . ,n

x1 + x2 + . . .+ xn ≤ B

aj ≤ xj ≤ bj j = 1,2, . . . ,n

Introduction

Optimization problemsExample: Least square problem

Given points on the plane, to find theline of best fitthroughthese points

Objective: To minimize the (squared) error between a lineand the points

Decision variables: the parameters of a line

Introduction

Given points on the plane, to find theline of best fitthroughthese points

Objective: To minimize the (squared) error between a lineand the points

Decision variables: the parameters of a line

Introduction

Given points

(x1, y1); (x2, y2); . . . ; (xn, yn)

to find theline of best fitthrough these points

Decision variables (the parameters of a line): mandn,where the expression of a line in the plane is

y = mx+ n

Objective: To minimize

(mxk + n− yk)2

Introduction

Given points

(x1, y1); (x2, y2); . . . ; (xn, yn)

to find theline of best fitthrough these points

Decision variables (the parameters of a line):mandn,where the expression of a line in the plane is

y = mx+ n

Objective: To minimize

(mxk + n− yk)2

Introduction

maxm, n ∈ R

k=1(mxk + n− yk)

Introduction

Optimization problemsExample: Harvesting of a renewable resource

To maximize the benefit from the harvesting of a naturalresource

Objective: maximize present utilities

Decision variables: harvesting levels (or effort) at eachperiod

Constraints:

nonnegative harvesting

biology

relation between harvesting and the amount of the resource

environmental constraints

Introduction

Constraints:

biology

Introduction

Constraints:

biology

Introduction

Constraints:

biology

Introduction

Constraints:

biology

Introduction

Constraints:

biology

Introduction

Constraints:

biology

Introduction

x(t + 1) = F(x(t)) − h(t) t = t0, t0 + 1, . . . ,T − 1

x(t0) = x0 given (e.g. the current state of the resource)

x(t) ≥ 0 is the level of the resource at periodt

h(t) ≥ 0 is the harvesting at periodt

F : R −→ R is the biological growth function of theresource

Introduction

x(t + 1) = F(x(t)) − h(t) t = t0, t0 + 1, . . . ,T − 1

Introduction

x(t + 1) = F(x(t)) − h(t) t = t0, t0 + 1, . . . ,T − 1

Introduction

The total benefits can be represented by

B =T−1∑

ρ(t−t0)U(x(t),h(t))

U(x,h) is the instantaneous profit if we havex and weharvesth

0 ≤ ρ ≤ 1 is a descount factor

Introduction

B =T−1∑

Introduction

B =T−1∑

Introduction

maxh(t0),h(t0+1),...,h(T−1)

T−1∑

t=t0ρ(t−t0)U(x(t),h(t))

subject to

x(t + 1) = F(x(t)) − h(t) t = t0, t0 + 1, . . . ,T − 1

x(t0) = x0

x(t) ≥ 0 t = t0 + 1, t0 + 2, . . . ,T

0 ≤ h(t) ≤ F(x(t)) t = t0, t0 + 1, . . . ,T − 1

Introduction

Optimization problemsImportant remark: min is equivalent to max

We can restrict our study only to minimization problems

f (x) ≤ f (x) for all x ∈ C

minx∈X

subject to

x ∈ C

Introduction

We can restrict our study only to minimization problems

f (x) ≤ f (x) for all x ∈ C

minx∈X

subject to

x ∈ C

Introduction

f (x) ≥ f (x) for all x ∈ C

maxx∈X

subject to

x ∈ C

Introduction

f (x) ≥ f (x) for all x ∈ C

maxx∈X

subject to

x ∈ C

Introduction

(Pmax)

maxx∈X

subject to

x ∈ C

is equivalent to problem

(Pmin)

minx∈X

−f (x)

subject to

x ∈ C

In the following sense:

x is solution of(Pmax) if and only if it is solution of(Pmin)

The optimal value of(Pmax) is the negative of the optimalvalue of(Pmin)

Introduction

(Pmax)

maxx∈X

subject to

x ∈ C

(Pmin)

minx∈X

−f (x)

subject to

x ∈ C

Introduction

(Pmax)

maxx∈X

subject to

x ∈ C

(Pmin)

minx∈X

−f (x)

subject to

x ∈ C

Introduction

Optimization problemsExistence of solutions

minx∈X

subject to

x ∈ C

When(P) has solution?

If [a,b] = C ⊆ X = R

WhenX = Rn, there exists at least one solution ifC is

closed and bounded

Introduction

Optimization problemsExistence of solutions

minx∈X

subject to

x ∈ C

When(P) has solution?

If [a,b] = C ⊆ X = R

WhenX = Rn, there exists at least one solution ifC is

closed and bounded

Introduction

Contents

Introduction

minx∈X

subject to

x ∈ C

Optimality conditions are mathematical expressions: equationsor system of equations, system of inequalities, ordinarydifferential equations, partial differential equations

Solutions of the mathematical problems obtained fromoptimality conditions are related with the solutions of theoptimization problem

Introduction

minx∈X

subject to

x ∈ C

Introduction

minx∈X

subject to

x ∈ C

Introduction

minx∈X

subject to

x ∈ C

The idea is to obtain optimality conditions and then to solvetheassociated mathematical problems (analytically or numerically)

Introduction

minx∈X

subject to

x ∈ C

The idea isto obtain optimality conditionsand thento solve theassociated mathematical problems (analytically or numerically)

Introduction

Optimality conditionsNecessary conditions

minx∈X

subject to

x ∈ C

Suppose thatoptimality conditionsof this problem arerepresented by a system of equations(S)

We say that(S) is a necessary condition if a solution of(P) is a solution of(S)

The idea is to find solutions of(S) in order to have a list ofcandidates for the solutions of(P)

If (S) is solved only for one point, then it is solution of(P)or (P) does not have solution

Warning: A solution of(S) may be not a solution of(P)

Introduction

minx∈X

subject to

x ∈ C

Suppose that optimality conditions of this problem arerepresented by a system of equations(S)

Introduction

minx∈X

subject to

x ∈ C

The idea isto find solutions of(S) in orderto have a list ofcandidates for the solutions of(P)

Introduction

minx∈X

subject to

x ∈ C

Introduction

minx∈X

subject to

x ∈ C

Introduction

Optimality conditionsNecessary condition: unrestricted case

minx∈R

where f : R −→ R

A necessary condition isf ′(x) = 0 where

f ′(x) = limt→0

f (x+ t)− f (x)t

That means: Ifx is a solution of(P) thenf (x) = 0

If we know the expression off ′ we can solve the equationf ′(x) = 0 in order to have candidates of solutions

Warning!!!

Introduction

minx∈R

f ′(x) = limt→0

f (x+ t)− f (x)t

That means:If x is a solution of(P) thenf (x) = 0

Warning!!!

Introduction

minx∈R

f ′(x) = limt→0

f (x+ t)− f (x)t

Warning!!!

Introduction

minx∈R

f ′(x) = limt→0

f (x+ t)− f (x)t

Warning!!!

Introduction

Optimality conditionsNecessary conditions: restrictions

minx∈R

a ≤ x ≤ b

At a pointx observe thatf increase in the sense of (thesign of) f ′(x)

There are three posibilities:

an interior pointa < x < b is a solution. Thenf ′(x) = 0

x = a is a solution. Thenf ′(x) = f ′(a) ≥ 0

x = b is a solution. Thenf ′(x) = f ′(b) ≤ 0

Observe that iff ′(a) > 0 andf ′(b) < 0, then there existsa < x < b such thatf ′(x) = 0

Introduction

minx∈R

a ≤ x ≤ b

Introduction

minx∈R

a ≤ x ≤ b

Introduction

minx∈R

a ≤ x ≤ b

Introduction

minx∈R

a ≤ x ≤ b

Introduction

minx∈R

a ≤ x ≤ b

Observe thatif f ′(a) > 0 andf ′(b) < 0, thenthere existsa < x < b such thatf ′(x) = 0

Introduction

A necessary condition of the problem

minx∈R

a ≤ x ≤ b

Find x, α, β such that

f ′(x) − α+ β = 0

x ∈ [a,b]

α, β ≥ 0

α(x− a) = 0

β(x− b) = 0

Introduction

A necessary condition of the problem

minx∈R

a ≤ x ≤ b

Find x, α, β such that

f ′(x) − α+ β = 0

x ∈ [a,b]

α, β ≥ 0

α(x− a) = 0

β(x− b) = 0

Introduction

Optimality conditionsGlobal vs local minimum

minx∈X

subject to

x ∈ C

We say thatx ∈ C is a global minimum, if it is a solutionof (P)

We say thatx ∈ C is a local minimum, if for everyx ∈ Cclose enough, one hasf (x) ≤ f (x)

A global optimum is a local optimum

Introduction

minx∈X

subject to

x ∈ C

Introduction

minx∈X

subject to

x ∈ C

Introduction

Optimality conditionsSufficient condition: unrestricted case

minx∈R

We say that(S) is a sufficient condition if a solution of(S)is a solution of(P)

The idea is to find solutions of(S) in order to havesolutions of(P)

In the general case, this is very difficult

It is much realistic to have that solutions of(S) are localsolutions of(P)

A (local) sufficient condition to(P) is f ′(x) = 0 andf ′′(x) > 0

Introduction

minx∈R

The idea isto find solutions of(S) in order to havesolutions of(P)

Introduction

minx∈R

In the general case, this isvery difficult

Introduction

minx∈R

It is much realistic to have thatsolutions of(S) are localsolutions of(P)

Introduction

minx∈R

Introduction

Optimality conditionsSufficient condition

If we find x ∈ R such that

f ′(x) = 0

f ′′(x) > 0

Thenx is a local minimum of the problem

minx∈R

Introduction

Optimality conditionsSufficient condition

If we find x ∈ R such that

f ′(x) = 0

f ′′(x) > 0

Thenx is a local minimum of the problem

minx∈R

Introduction

Conclusions

minx∈X

subject to

x ∈ C

To formulate an optimization problem,the followingconcepts must be clear:

Decision variable(s):x ∈ X

Objective function:f : X −→ R

Restrictions:C ⊂ X

A maximization problem is equivalent to a minimizationproblem (deal with−f (x) instead off (x))

Introduction

Conclusions

minx∈X

subject to

x ∈ C

Decision variable(s): x ∈ X

Introduction

Conclusions

minx∈X

subject to

x ∈ C

Objective function: f : X −→ R

Introduction

Conclusions

minx∈X

subject to

x ∈ C

Restrictions: C ⊂ X

Introduction

Conclusions

minx∈X

subject to

x ∈ C

To formulate an optimization problem, the followingconcepts must be clear:

A maximization problem is equivalent to a minimizationproblem(deal with−f (x) instead off (x))

Introduction

Conclusions

minx∈X

subject to

x ∈ C

Optimality conditions are mathematical systems

The idea is to solve these systems in order to:

have candidates of solutions (if the system is a necessarycondition)

obtain a local solution (if the system is a sufficientcondition)

Introduction

Conclusions

minx∈X

subject to

x ∈ C

The idea isto solve these systemsin order to:

Introduction

Conclusions

minx∈X

subject to

x ∈ C

have candidates of solutions(if the system is anecessarycondition)

Introduction

Conclusions

minx∈X

subject to

x ∈ C

obtain a local solution(if the system is asufficientcondition)

Introduction

Next part

minx∈X

subject to

x ∈ C

Several decision variablesx1, x2, . . . , xn. That is

∈ Rn = X

Necessary and sufficient conditions

Why the convexity (or concavity) is relevant?

Introduction

Next part

minx∈X

subject to

x ∈ C

∈ Rn = X

Introduction

Next part

minx∈X

subject to

x ∈ C

∈ Rn = X

Introduction to Optimization - CERMICS – Centre d...

Documents

Transcript of Introduction to Optimization - CERMICS – Centre d...