Extremum seeking control

Dragan Nešić

The University of Melbourne

Acknowledgements: Y. Tan, I. Mareels, A. Astolfi, G. Bastin, C. Manzie; A. Mohammadi; W. Moas Australian Research Council.

Outline

Motivating examples Background Ad hoc designs Black box:

- Problem formulation- Systematic design

Gray box:- Problem formulation- Systematic design

Conclusions & future directions

A Prelude

This is an approach for online optimisation of the steady-state system behaviour.

A standing assumption is that the plant model or the cost is not known.

The controller finds the extremum in closed-loop fashion.

Motivating examples

Continuously Stirred Tank (CST) Reactor

Substrate Product

u=Vol. flow rate Performance output y:

Productivity JP

Yield JY

Inflow Outflow

Overall JT

J T := ¸J P +(1¡ ¸)J Y ; ¸ 2 (0;1)

Single enzymatic reactionMichaelis-Menten Kinetics

Productivity and yield Total cost

is typically unknown!!

In steady-state, we would typically want to operate around u¤J T (¹u)

G. Bastin, D. Nešić, Y. Tan and I. Mareels, “On extremum seeking in bioprocesses with multivalued cost functions”, Biotechnology Progress, 2009.

Raman amplifiers

Fibre span

Power sensors

Pump lasers

u=laser power

P.M. Dower, P. Farrell and D. Nešić, “Extremum seeking control of cascaded optical Raman amplifiers”, IEEE Trans. Contr. Syst. Tech., 2008.

CostPerformance output y:

•Spectral flatness (equalization)• Desired power

¸

p

Pi (pi ¡ pd)2

Other engineering examples

Plant Performance output

Turbine Generated power

Solar cell Generated power

Variable cam timing Fuel consumption

Tokamak Reflected power during Lower Hybrid (LH) plasma heating experiments

Non-holonomic vehicles Distance from a source of a signal

Paper machine Retention of fines and fibers in the sheet

Ultrasonic/Sonic Driller/Corer Distance from resonance

Human Exercise Machine The user’s power output

ABS Magnitude of friction force

Examples from biology

E. Coli bacteria search for food in a similar manner to an extremum seeking algorithm (M. Krstic et al).

Some fish search for food in a similar manner to extremum seeking (M. Krstic et al).

k

kv

Background

Classification of approaches

NLP based ESC [Popović, Teel,…]

Adaptive ESC [Krstić, Ariyur, Guay, Tan, Nešić,…]

Deterministic Stochastic

Adaptive ESC [Krstić, Manzie,…]

NLP based ESC[Spall,..]

Also continuous-time versus discrete-time.

Brief history (deterministic):

1922 1950 2000

Firs

t ESC

?

1960 1970 2009

Vibra

nt re

sear

ch a

rea

Man

y ne

w s

chem

es p

ropo

sed

Espec

ially

Ada

ptiv

e

Firs

t loc

al s

tabi

lity

resu

lt fo

r ada

ptiv

e ESC

Syste

mat

ic d

esig

n di

scre

te-ti

me

NLP

.

Syste

mat

ic d

esig

n ad

aptiv

e

Schem

es.

Beginning Ad-hoc designs Rigorous analysis and design

Åströ

m &

Witt

enm

ark:

“one

of t

he m

ost

prom

isin

g ad

aptiv

e co

ntro

l tec

hniq

ues”

.

1995

Ad hoc adaptive designs

Adaptive ESC [Krstić & Wang 2000]

µ

asin(! t)

Ks

_x = f (x;u)

y = h(x)

y

+ x

asin(! t)

Extremum seeking controller

Wh(s)Wl(s)

u

Parameters:

a; K ; !

Static scalar case (gradient descent)

µ

u= µ+asin(! t)

asin(! t)

Ks

y

+ xµ

sin(! t)

Extremum seeking controller

a; K ;!Parameters:

Y. Tan, D. Nešić and I. Mareels, “On non-local stability properties of

extremum seeking control”, Automatica, 2006.

_x = f (x;u)

y = h(x)

Comments

Many similar adaptive algorithms proposed. Case-by-case convergence analysis. No clear relationship with optimization.

A unifying design approach is unavailable. A unifying convergence analysis is missing.

A unifying approach exists for another class of schemes [Teel and Popovic, 2000].

Black Box Approach

Problem formulation (black box)

Extremum Seeking

Controller

Assumption 1:

- Q(.) has an extremum (max)

- Q(.) is unknowny=Q(u) yu _x = f (x;u)

y = h(x)Dynamic case:

Problem:

Design ESC so that limsupt! 1 jy(t) ¡ y¤j ¼0

y¤ := Q(u¤) ¸ Q(u); 8u

9 (̀¢) ) 0 = f ( (̀u);u)

Q(u) := h± (̀u)

Assumption: u(t) ´ ¹u =) y(t) ! Q(¹u)

Systematic design(derivatives estimation)

D. Nešić, Y. Tan, W. Moas and C. Manzie, “A unifying approach to extremum seeking:adaptive schemes based on derivatives estimation”, IEEE Conf. Dec. Contr. 2010.

Continuous optimization (offline)

y=Q(u)

• No inputs & outputs

• Q(.) is known, so all derivatives of Q(.) are known

_µ= F (DN (µ)) limt! 1 jµ(t) ¡ u¤j = 0

DN (u) := [Q(u) DQ(u) D2Q(u) :: : DNQ(u)]

Examples

Gradient method

Continuous Newton method

_µ=DQ(µ) :

_µ= ¡ DQ(µ)D 2Q(µ) :

Extremum seeking (online)

y=Q(u)y

• Inputs & outputs available

• Q(.) is unknown

=µ+asin(t)

_µ= ²F (dDN (µ))

limsupt! 1 jµ(t) ¡ u¤j ¼0

dDN Derivativesestimator

asin(t)

+

• a, !L, ² are positive controller parameters

! L

u

Systematic design (use the previous block diagram)

Step 1: Choose an optimization scheme.

Step 2: Use an estimator for DN Q(¢).

Step 3: Adjust the controller parameters.

Estimator design

Estimating DQ(µ)

y=Q(u)yu= µ+asin(t)

! Ls+! L

£sin(t)

limt! 1 »1(t) ¼ a2DQ(µ)

»1

dDQ(µ) = 2a»1

Analysis

Q(µ+asin(t)) sin(t) ¼Q(µ) sin(t) +aDQ(µ) sin2(t) + a2

2 D2Q(µ) sin3(t)

where µ is assumed constant.

Average the right hand side of the model.

_»1 ¼¡ ! Lh»1 ¡ DQ(µ)a2

i

Model of the system:

_»1 = ¡ ! Lh»1 ¡ Q(µ+asin(t)) sin(t)

i

limt! 1 »1(t) ¼ a2DQ(µ)

Estimating D2Q(µ)

y=Q(u)yu= µ+asin(t)

! Ls+! L

£sin2(t)

limt! 1 »2(t) ¼ 12Q(µ) +

3a2

16 D2Q(µ)

»2

! Ls+! L

»0

limt! 1 »0(t) ¼Q(µ) + a2

4 D2Q(µ)

dD2Q(µ) = 8a2 (2»2 ¡ »0)

Higher order derivatives

y=Q(u)yu= µ+asin(t)

! Ls+! L

£sin(t)

! Ls+! L

g(a;»0; : : :;»N )

...

dDN (µ)

! Ls+! L

£sin(t)

......

»0

»1

»N

Convergence analysis

Model of the overall system

_̂µ = ²! L F (g(a;»0; : : : ;»N ))_»i = ¡ ! L (»i ¡ ³i (t;µ;a)); i = 0;1;: : : ;N

µ = µ̂+asin(t)

³i (t;µ;a) = Q(µ+asin(t)) sini (t)

!L, ² and a are controller parameters that need to be tuned to achieve appropriate convergence properties.

Slow:

Fast:

Assumption 1 (global max)

There exists a global maximum

DQ(µ) = 0 ( ) µ= µ¤

D2Q(µ¤) < 0

Assumption 2 (robust optimizer)

The solutions of

satisfy

for sufficiently small w(t).

_µ= F (DNQ(µ) +w(t))

limsupt! 1 jµ(t) ¡ µ¤j ¼0

Theorem

Suppose Assumptions 1-2 hold. Then

8(¢ ;º); 9(! ¤L ;a¤)

+

8! L 2 (0;! ¤L ); a 2 (0;a¤); 9²

+

j(µ(t0) ¡ µ¤;»(t0))j · ¢

+

limsupt! 1 j»(t) ¡ ¹ (µ(t);a)j · º;

limsupt! 1 jµ(t) ¡ µ¤j · º

Tuning guidelines

Geometrical interpretation

µ

»(»0;µ0)

Fast transient (estimator)

»= ¹ (µ;a)

Slow transient optimization

» ! L

» ²! L

lim supt! 1

jµ(t) ¡ µ¤j · º =) lim supt! 1

jy(t) ¡ y¤j · º1

B¢

Exist !L, ², a

Bº

µ¤

For any ¢, º

Comments

A systematic design approach proposed. Rigorous convergence analysis provided. Controller tuning proposed in general. Dynamic plants treated in the same way. Multi-input case is treated in a similar way. Averaging and singular perturbations used. Tradeoffs between the domain of attraction,

accuracy and speed of convergence!

Bioreactor example

All our assumptions hold – gradient method used.

Gray Box Approach

Problem formulation (gray box)

Extremum Seeking

Controller

Assumption 1:

- Q(.,p) has an extremum (max)

- Q(.,.) is known; p is unknowny=Q(u;p) yu _x = f (x;u;p)

y = h(x;p)Dynamic case:

Problem:

Design ESC so that limsupt! 1 jy(t) ¡ y¤j ¼0

y¤ := Q(u¤;p) ¸ Q(u;p); 8u

9 (̀¢;p) ) 0 = f ( (̀u);u;p)

Q(u;p) := h( (̀u;p);p)

Assumption: u(t) ´ ¹u =) y(t) ! Q(¹u;p)

Systematic design(parameter estimation)

D. Nešić, A. Mohamadi and C. Manzie, “A unifying approach to extremum seeking:adaptive schemes based on parameter estimation”, IEEE Conf. Dec. Contr. 2010.

Extremum seeking (online)

yu= µ+asin(t)

• Inputs & outputs available

• p is unknown

y=Q(u;p)

_µ= ²F (DN (µ; p̂))

limsupt! 1 jµ(t) ¡ u¤j ¼0

p̂Parameterestimator

asin(t)

+

• a, !L, ² are controller parameters

Comments

Similar systematic framework in this case. Similar convergence analysis holds. Classical adaptive parameter estimation

schemes can be used. Dynamic plants dealt with in the same way. Persistence of excitation is crucial for

convergence. Tradeoffs between domain of attraction,

accuracy and convergence speed.

Example

Consider the static plant:

We used the continuous Newton method.

Classical parameter estimation used.

Values p1=9 and p2=8 used in simulations.

y= u21+p1u1+p2u22

Simulations

Performance output Control inputs Parameters

Final remarks

Several tradeoffs exist; convergence slow. Many degrees of freedom: dither shape,

controller parameters, optimization algorithm, estimators.

Some global convergence results available (similar to simulated annealing).

Summary

A systematic design framework presented for two classes of adaptive control schemes.

Precise convergence analysis provided. Controller tuning and various tradeoffs

understood well. Applicable to a range of engineering and non-

engineering fields.

Future directions

Tradeoffs: convergence speed, domain of attraction and accuracy.

Various extensions: non-compact sets, global results, non-smooth systems, multi-valued cost functions.

Schemes robust although no formal proofs. Tailor the tools to specific problems. Exciting research area.

Thank you!