EL6233 Lecture1

EL6233 System Optimization y p

Methods

Z. P. Jiang

ECE D t t R LC214ECE Department Rm LC214

NYU-Poly. ext. 3646

eeweb.poly.edu/faculty/jiang

1/25/2010 ECE Dept at Poly 1

Spring 2008 Z.P. Jiang. No reproduction permitted without written permission of the author.

Course OutlineCourse Outline

An introductory course to practical OptimizationAn introductory course to practical Optimization

Methods

* linear and nonlinear programming

* l l f i ti* calculus of variations

* dynamic programmingy p g g

* other optimization problems & methods


Features of this CourseFeatures of this Course

B l b t th d li ti• Balance between theory and applications

• Mix of academic examples and puzzlesMix of academic examples and puzzles

• Basic homework vs. open-end projectsp p j

• Independent reading vs. team work

• Numerical algorithms


Motivational Examples of

O i i iOptimization

! Linear Equations

" Linear Programming

# Least Squares Data Fitting

$ Nonlinear Programming


!Linear Equations!Linear Equations

Goal: a first pass to data fittingGoal: a first pass to data fitting

Given three points: (2, 1), (3, 6) and (5, 4), find the

corresponding quadratic functioncorresponding quadratic function

2

1 2 3( ) f t x x t x t! " "

w unknith w 1n ,o 3ix i# #


CommentCommentCommentComment

An optimization problem usually involvesAn optimization problem usually involves

! Objective function

" Variables or unknowns


"Linear Programming"Linear Programming

Manufacturing “bookshelf ( )”, “cabinets with1xManufacturing bookshelf ( ) , cabinets with

doors ( )”, “cabinets with drawers ( )”, and

“custom-designed cabinets ( )”.2x 3x

1x

4x

Objective: maximize the weekly revenue

4

j y


Cabinet Wood Labor Revenue

Bookshelf 10 2 100

With Doors 12 2 150

W. Drawers 25 8 200

Custom 20 12 400

Only5000 units of wood and 1500 units of labor are available


Only 5000 units of wood and 1500 units of labor are available.

LP ModelLP Model

Solve the following linear programming problemSolve the following linear programming problem

with constraints:

1 2 3 4maximize P=100 150 200 400x x x x" " "

1 2 3 4

1 2 3 4

subject to 10 12 25 20 5000

2 4 8 12 1500

x x x x

x x x x

" " " #

" " " #1 2 3 4

1 2 3 4 , , , 0.x x x x $



Optimization problems often involveOptimization problems often involve

# Constraints (on the unknowns)


#Least Squares Data Fitting#Least Squares Data Fitting

Assume that some, or all, measurements are inAssume that some, or all, measurements are in

error and that there are more data than the

number of variables.

Goal: minimize the observation errors

% &2

i i i ie m x x t x t! ' " "% &1 2 3

1 , 3

i i i ie m x x t x t

i N N

! " "

# # $


Formulation of the ProblemFormulation of the Problem

Mathematically, we want to solveMathematically, we want to solve

% &2

2 2

1 2 3

1 1

minimize

N N

i i i i

i ixe m x x t x t( )! ' " "* +, ,

1 1i ix ! !



Optimization problems may beOptimization problems may be

$ Nonlinear (without constraints)

In this case, the problem is referred to as:

Unconstrained Nonlinear ProgrammingUnconstrained Nonlinear Programming


$Nonlinear Programming$Nonlinear Programming

Electrical connections:

w1 w2

?

w3w4


NP ModelNP Model

4

i i i P ,

% & % &1

2 2

minimize i

iP w!

!,

% & % &

% & % &

2 2

0 0

2 2

subject to = , 1 4

1 4 4

i i iw x x y y i

x y

' " ' # #

' " ' #% & % &

% & % &1 1

2 2

2 2

1 4 4

9 5 1

x y

x y

' " ' #

' " ' #

3 3 2 4, 3 1

6 8 2 2

x y

x y

# # ' # # '

# # ' # #1/25/2010 ECE Dept at Poly 15

4 4 6 8, 2 2 . x y# # ' # #


Optimization problems may beOptimization problems may be

% Nonlinear with constraints

In this case, the problem is referred to as:

Constrained Nonlinear ProgrammingConstrained Nonlinear Programming


“Blood Diamond” GameBlood Diamond Game

Context: 5 pirates robbed 100 diamonds andContext: 5 pirates robbed 100 diamonds and decide to distribute these diamonds by the rules:

• Draw lots to give each pirate a number out of 1, 2, 3, 4, 5.

First No 1 pirate proposes a distribution plan• First, No. 1 pirate proposes a distribution plan for the group of fives to vote. Only when he gets the majority of votes, his proposal will be g j y , p papproved.

Otherwise, he will be thrown into the sea.


“Blood Diamond” (cont’d)Blood Diamond (cont d)

• After No. 1 pirate dies, No.2 proposes hisAfter No. 1 pirate dies, No.2 proposes his

distribution plan for the group of four to vote.

Only when he gets the majority of votes, his

proposal will be approved.

Otherwise, he too will be thrown into the sea.

• Continue this way for No. 3 and No. 4



Hypothesis:Hypothesis:

All five pirates are equally smart and can make

their right judgementtheir right judgement.



Problem:Problem:

What should No.1 pirate propose to maximize his

profit?profit?

Solution at the end of this semester &Solution at the end of this semester &


PreliminariesPreliminaries

B i N ti f O ti lit• Basic Notions of Optimality

• Convexity• Convexity

• Taylor SeriesTaylor Series

• The General Optimization Algorithmp g

• Rates of Convergence


Basic Notions of OptimalityBasic Notions of Optimality

Consider an optimization problemConsider an optimization problem

( ) ( ) performance indexminimize f x P f x!( ) ( )

( ) 0, bj

p

t t t i t

minimize

i

x S

f f

g x i I

-

! - ./0subject to

( ) 0,cons

traints

jg x j J0$ - /1

It is worth noting that

( ) ( ) maximize minimizef x f x''!


( ) ( )maximize minimizex S x S

f f- -

FeasibilityFeasibility

• A point satisfying all the constraints is said toA point satisfying all the constraints is said to

be feasible.

• Feasible set S: all feasible points.p

• Active constraints: when

• Inactive constraints: when

( ) 0.jg x !

( ) 0jg x 2Inactive constraints: when

• Boundary: The set of feasible points at which

at least one inequality is active.

( ) 0.jg x 2

at east o e equa ty s act e

• Interior points: All other points than boundary


An Example

X31 1 2 3( ) 2 3 6 0

( ) 0

g x x x x! " " ' !

$2 1

3 2

( ) 0

( ) 0

g x x

g x x

! $

! $A x

4 3( ) 0.g x x! $B

Cx

x

X2

Points:

X1

A = (0, 0, 2)

B = (3, 0, 1)


C = (1, 1, 1).

MinimizersMinimizers

• Global minimizer ifx3 ( ) ( ), .f x f x x S3 # 4 -• Strict global minimizer if, additionally,

x ( ) ( ), .f x f x x S# 4 -

• Local minimizer if

( ) ( ), , .f x f x x S x x3 35 4 - 6

( ) ( ), , such that | | .f x f x x S x x3 3# 4 - ' 57

• Strict local minimizer if

( ) ( ) s ch that | |f f S3 3 35 4 6 571/25/2010 ECE Dept at Poly 25

( ) ( ), , such that , | | .f x f x x S x x x x5 4 - 6 ' 57

Stationary PointsStationary Points

• For unconstrained problems, a stationary pointFor unconstrained problems, a stationary point

of f(x) is such that

'( ) ( ) 0f

f x x3 38!

• A minimizer (or a maximizer) is a stationary

'( ) ( ) 0f

f x xx

!8!

A minimizer (or a maximizer) is a stationary

point, but not vice versa.

• Saddle point: a stationary point that is not

minimizer nor maximizer.


ConvexityConvexity

• Convex set S, ifConvex set S, if

(1 ) for all 0 1.x y S9 " '9 - # 9 #

• Convex function f, if

( (1 ) ) ( ) (1 ) ( )f x y f x f y9 " '9 # 9 " '9

, and 0 1.x y S4 - # 9 #


Convex Programming ProblemConvex Programming Problem

For a convex set S and a convex function f over S,For a convex set S and a convex function f over S,

( )minx S

f x-x S-

For example such a set S can be defined via functionsconcave g

: ;For example, such a set S can be defined via functions ,

| ( ) 0

concav

.

e i

n

i

g

S x g x! - $"

Linear Programming,

where both the object

Special case of C

ive and the const

onvex Programming:

raints are linear


where both the objective and the constraints are linear.

An Important ResultAn Important Result

A local minimizer of a convex programmingx3A local minimizer of a convex programming

problem is a global minimizer.

Moreover if the cost function is strictly convex

x

Moreover, if the cost function is strictly convex,

then is the unique global minimizer. x3

Proof. Prove by contradiction. See the book.


Taylor SeriesTaylor Series

Goal: Approximate a (sufficiently smooth) functionGoal: Approximate a (sufficiently smooth) function

by a polynomial around a point 0.x

2 ( )

:

1( ) ( ) '( ) "( ) ( )

1-variable

nnp

f x p f x pf x p f x f x" " " " " "( )

0 0 0 0 0( ) ( ) ( ) ( ) (

n-variab

)2 !

:les

f x p f x pf x p f x f xn

" ! " " " " "# #

2

0 0 0 0

1( ) ( ) ( ) ( )

2

T Tf x p f x p f x p f x p" ! " < " < "#


Notation:

Gradient ( ) :f x<

otat o

Gradient ( ) :

( ) ( ) ( )( )

T

n

f x

f x f x f xf x

<

= >8 8 8< -? @ "

1 2

2

( ) ( ) ( ) ( ) , , ,

n

f f ff x

x x x< ! -? @8 8 8A B

# "

2

2

Jacobian ( ) :

( )

f x

f x

<

= >82 ( ) ( ) .n n

i j

f xf x

x x

C= >8

< ! -? @? @8 8A B"


jA B

ExampleExample

2 2

1 1 2 2( ) 3f x x x x x! " '

Compute its gradient ( )f x and<2

Compute its gradient ( )

its Jacobian ( ) (1, 1).

f x and

f x at x

<

< !


Necessary ConditionNecessary Condition

A necessary condition for optimization is:A necessary condition for optimization is:

( ) 0f x< !0( ) 0.f x< !


Sufficient ConditionsSufficient Conditions

Sufficient conditions for minimization are:Sufficient conditions for minimization are:

0( ) 0f x< !0

2

0

( ) 0,

( ) 0.

f x

f x

<

< 20( )f



A matrix is said to be ,

if , 0, it holds that

positive definiten n

n

M

x x

C-

4 - 6

"

" , 0, o ds

0.T

x x

x Mx 2

% &eigenvalueThe of are all such that

s

det 0.

M

I M

D

D ' !% &


General Optimization AlgorithmGeneral Optimization Algorithm

Step 0: Specify some initial guess of the solutionStep 0: Specify some initial guess of the solution

0x

Step k (k=0,1,…): If is optimal, STOP.

Otherwise determine an improved estimate of

0x

Otherwise, determine an improved estimate of

the solution: with step

length and search direction1k k k kx x p" ! " 9

k9 kpe gt a d sea c d ect ok9 kp


Rates of ConvergenceRates of Convergence

• Does an algorithm converge?Does an algorithm converge?

H f t d it ?• How fast does it converge?


: ;Sequence with and , ifrate rate constantkx x r C3E

•

: ;

1 lim , where

k

k kk

k

r

eC e x x

e

"3EF

! 5 F '$

ke

• Linear convergence if r = 1.

1) 0 < C < 1, the sequence converges.

2) C > 1 the sequence diverges2) C > 1, the sequence diverges.

• Superlinear convergence if r = 1 and C = 0.Superlinear convergence if r 1 and C 0.

• Quadratic convergence if r = 2.


A Brief Introduction to Lagrange

M l i li M h dMultiplier Method

Lagrange multiplier theory essentially targets

at constrained optimization problems. For example,

% &0

1

max ,

subject to the constraint ( ).

nP f x x

c f x

! -

!

"

1j ( )

To this end introduce the augmented performance inde

f

xTo this end, introduce the augmented performance inde

% & % &0 1

x

( )

ith th t f L lti li

T

aP f x c f x! " D '

D1/25/2010 ECE Dept at Poly 39

with the vector of Lagrange multipliers.D

A Brief Introduction to Lagrange

M l i li M h dMultiplier Method

Then, for optimality arenecessary conditions

P8 0aP

x

and

8!

8

1 ( ).

and

c f x!

For more details, wait for

the lectures on "nonlinear programming"


the lectures on nonlinear programming .

EL6233 Lecture1

Documents

Transcript of EL6233 Lecture1