Chpt13 Optimisation and Search

8/12/2019 Chpt13 Optimisation and Search

1/43

Optimisation and Search:

Gradient Descent and

the Nelder-Mead Simplex Algorithm


2/43

Why minimise a function numerically?

f(a,b)

a

by

Unknown!


3/43

Why minimise a function numerically?Background: linear regression


4/43

Straight Line: f(x) = 1x + 2



5/43


i



6/43


Error between f(xi) given bythe model andyi from thedata:

ii

iii

yx

yxf

+=

=

21

21 )(),(

i



7/43


Error between f(xi) given bythe model andyi from thedata:

ii

iii

yx

yxf

+=

=

21

21 )(),(

iTask: Find the parameters 1

and 2 that minimise the sumof squared errors!

=

=

+=

=

N

i

ii

N

i

i

yx

E

1

2

21

1

2

2121

)(

),(),(



8/43

Linear Regression:

Fitting function is linear with respect to theparameters can be solved analytically (see Wikipedia)

Non-linear Regression:

Fitting function is non-linear with respect to theparameters (e.g. f(x,1,2) = sin(1x)+cos(2 x)) Often no analytical solution Numerical optimisation or direct search

Why minimise a function numerically? non-linear regression


9/43

Gradient Descent: Example

1 2

E(1,2)


10/43

Gradient Descent

1. Choose initial parameters 1 and 22. Calculate the gradient

3. Step in the direction of the gradient with a stepsize proportional to the amplitude of thegradient

you get new parameters 1 and 24. Check if the parameters have changed at a rate

above a certain threshold

5. If yes, go to 2, else terminate


11/43


1 2

E(1,2)


12/43


1 2

E(1,2)


13/43


1 2

E(1,2)


14/43


1 2

E(1,2)


15/43


1 2

E(1,2)


16/43


1 2

E(1,2)


17/43


1 2

E(1,2)


18/43


1 2

E(1,2)


19/43


1 2

E(1,2)


20/43


1 2

E(1,2)


21/43


1 2

E(1,2)


22/43


1 2

E(1,2)


23/43


1 2

E(1,2)


24/43

Nelder-Mead Simplex Algorithm(for functions of 2 variables)1. Pick 3 parameter combinations Triangle

2. Evaluate the function for those combinations f

h,f

s,f

l: highest, second highest and lowest point

3. Update the triangleusing the best of thetransformations inthe figure

4. Check for endcondition

5. Go to 2 orterminate


25/43

Nelder-Mead Algorithm: Update Rulesr

srl

f

fff

accept


26/43

Nelder-Mead Algorithm: Example

1

2

E(1,2)


27/43


1

2

E(1,2)


28/43


1

2

E(1,2)


29/43


1

2

E(1,2)


30/43


1

2

E(1,2)


31/43


1

2

E(1,2)


32/43


1

2

E(1,2)


33/43


1

2

E(1,2)


34/43


1

2

E(1,2)


35/43


1

2

E(1,2)


36/43


1

2

E(1,2)


37/43


1

2

E(1,2)


38/43


1

2

E(1,2)


39/43


1

2

E(1,2)


40/43


1

2

E(1,2)


41/43


12

E(1,2)


42/43


12

E(1,2)


43/43


12

E(1,2)

Chpt13 Optimisation and Search

Documents

Transcript of Chpt13 Optimisation and Search