System Identification Parameter Estimation - TU … · 2/11/2010 Delft University of Technology...

2/11/2010

DelftUniversity ofTechnology

System Identification &

Parameter Estimation

Correlation functions in time & frequency domain

Alfred C. Schouten, Dept. of Biomechanical Engineering (BMechE), Fac. 3mE

Wb2301: SIPE lecture 2

2SIPE, lecture 2 | xx

Contents•

Correlation functions in time domain

• Autocorrelations (Lecture 1)• Cross-correlations

•

General properties of estimators• Bias / variance / consistency

•

Estimators for correlation functions

•

Fourier transform

•

Correlation functions in frequency domain• Auto-spectral density• Cross-spectral density• Estimators for spectral densities


Lecture 1: (auto)correlation

functions•

Autocorrelation function

•

Autocovariance

function

•

Autocorrelation coefficient

( )( ) 2τ τ μ μ τ μ⎡ ⎤= − − − = Φ −⎣ ⎦( ) ( ) ( ) ( )xx x x xx xC E x t x t

[ ]τ τΦ = −( ) ( ) ( )xx E x t x t

2

2 0τ μ μ τ μ ττσ σ σ

⎡ ⎤⎛ ⎞⎛ ⎞− − − Φ −= = =⎢ ⎥⎜ ⎟⎜ ⎟

⎝ ⎠⎝ ⎠⎣ ⎦

( ) ( ) ( ) ( )( )( )

x x xx x xxxx

x x x xx

x t x t Cr EC


Lecture 1: autocovariance

function•

Autocovariance

function:

•

If μ = 0, then autocovariance

and autocorrelation functions are identical

•

At zero lag:

( )( )[ ] [ ] [ ] 2

2 2 2

2

τ τ μ μ

τ τ μ μ μ

τ μ μ μ

τ μ

⎡ ⎤= − − −⎣ ⎦⎡ ⎤= − − − − + ⎣ ⎦

= Φ − − +

= Φ −

( ) ( ) ( )

( ) ( ) ( ) ( )

( )

( )

xx x x

x x x

xx x x x

xx x

C E x t x t

E x t x t E x t E x t E

( )2 20( ) ( )xx x xC E x t μ σ⎡ ⎤= − =⎣ ⎦


Example autocovariances•

Matlab

demo: Lec2_example_Cuu.m

0 1 2 3 4 5-5

0

5

whi

te

signal

-2 -1 0 1 2-2

0

2Cuu

0 1 2 3 4 5-1

0

1

low

-pas

s

-2 -1 0 1 2-0.05

0

0.05

0 1 2 3 4 5-1

0

1

sinu

soid

-2 -1 0 1 2-0.5

0

0.5

0 1 2 3 4 5-5

0

5

sin.

+ n

oise

time [s]-2 -1 0 1 2

-2

0

2

tau [s]


2D Probability density function

{ }dyytyydxxtxxdxdyyxf yx +≤+<+≤<= );( );(Pr);,( ζτζτ ∩


2D Probability density function•

Probability for certain values of y(t) given certain values of x(t)

•

Co-variance of y(t) with x(t): y(t) is related with x(t)• No co-variance between y(t) and x(t): 2D probability density function

is circular• Covariance between y(t) and x(t): 2D probability density function is

ellipsoidal

•

No co-variance between y(t) and x(t):• y(t) and x(t) are independent• no relation exist• transfer function is zero !


Cross-correlation functions•

Measure for common structure in two signals:

• Cross-correlation

• Cross-covariance

• Cross-correlation coefficient

[ ]( ) ( ) ( )xy E x t y tτ τΦ = −

( )( )τ τ μ μ τ μ μ⎡ ⎤= − − − = Φ −⎣ ⎦( ) ( ) ( ) ( )xy x y xy x yC E x t y t

0 0μ ττ μτ

σ σ

⎡ ⎤⎛ ⎞−⎛ ⎞− −= =⎢ ⎥⎜ ⎟⎜ ⎟⎜ ⎟⎢ ⎥⎝ ⎠⎝ ⎠⎣ ⎦

( ) ( )( )( )( ) ( )

y xyxxy

x y xx yy

y t Cx tr EC C


Example: Estimation of a Time Delay•

Consider the system:

• Additive noise v(t) is stochastic and uncorrelated to x(t),

•

Cross-covariance of the measured input and output in case:• input x(t) is white noise (stochastic)

0( ) ( ) ( )y t x t v tα τ= − +


Example: Estimation of a Time Delay•

Matlab

demo: Lec2_example_Cuy_delay.m

0 5 10 15 20 25 30 35 40 45 50-5

0

5u(

t)

time [s]

u

0 5 10 15 20 25 30 35 40 45 50-5

0

5

y(t)

time [s]

u

-2 -1.5 -1 -0.5 0 0.5 1 1.5 2-1

0

1

Cuy

( τ)

τ [s]


Effects of Noise on Autocorrelations•

Often, correlation functions must be estimated from measurements.

•

Consider x (t ) and y (t ), corrupted with noise n (t ) and v (t ) resp.:

•

Noises have zero mean and are independent of the signals and of each other.

•

Autocorrelation:

•

Additive noise will bias autocorrelation functions!

( ) ( ) ( )( ) ( ) ( )

w t x t n tz t y t v t

= += +

( )( )( ) ( ) ( ) ( ) ( )zz E y t v t y t v tτ τ τ⎡ ⎤Φ = − + − +⎣ ⎦

( ) ( ) ( ) ( )yy yv vy vvτ τ τ τ= Φ + Φ + Φ + Φ

( ) ( )yy vvτ τ= Φ + Φ


Effects of Noise on Cross-correlations•

Cross-correlation:

•

Additive noise will not bias cross-correlation functions!

( )( )τ τ τ

τ τ τ τ

τ

⎡ ⎤Φ = − + − +⎣ ⎦= Φ + Φ + Φ + Φ

= Φ

( ) ( ) ( ) ( ) ( )

( ) ( ) ( ) ( )

( )

wz

xy xv ny nv

xy

E x t n t y t v t


Noise Reduction

•

Longer recordings• More ‘information’, same amount of noise

•

Repeat the experiment• Noise cancels out by averaging from exactly the same inputs

•

Improve Signal-to-Noise Ratio• Concentrate power at specified frequencies, assuming the noise

power remains the same


Properties of Estimators•

Formula given (autocovariances

/ spectral densities) are

estimators for the ‘true’

relations.

•

What are the properties of these estimators?• bias / variance / consistency

•

Bias: structural error

•

Variance: random error

•

Consistent: A consistent estimator is an estimator that converges, in probability, to the quantity being estimated as the sample size grows.


Example: estimator for mean value and variance of a signal

•

Signal xk

, with

k=1…N

•

Estimator for the signal mean:

•

Estimator for the signal variance:

•

What are the expected values of both estimators? Expectation operator: E{.}

( )∑

∑

=

=

−=

=

N

kxkx

N

kkx

xN

xN

1

22

1

ˆ1ˆ

1ˆ

μσ

μ


Estimator for the mean value

{ } { }

{ } { }

{ } x

N

kxx

xk

N

kk

N

kkx

N

kkx

NE

xExE

xEN

xN

EE

xN

μμμ

μ

μ

μ

==

==

=⎭⎬⎫

⎩⎨⎧

=

=

∑

∑∑

∑

=

==

=

1

11

1

1ˆ

11ˆ

1ˆ


Estimator for the variance

•

Estimator is biased and consistent

( )

{ } { } { } { }

{ }

222

1 1 1

22

1 1 1 1

2 22

1 1 1 1

2 2 2 2 2 2 22

1

1 1 1ˆ ˆ

1 2 1

1 2 1ˆ

1 2 1ˆ 2k l k l

N N N

x k x k lk k l

N N N N

k k l l kk l l k

N N N N

x k k l l kk l l k

N

x x x x x x x x x x xk

x x xN N N

x x x x xN N N

E E x E x x E x xN N N

E K KN N N

σ μ

σ

σ σ μ σ μ σ μ

= = =

= = = =

= = = =

=

⎛ ⎞= − = −⎜ ⎟⎝ ⎠

⎛ ⎞= − +⎜ ⎟⎝ ⎠

⎛ ⎞= − +⎜ ⎟⎝ ⎠

= + − − + +

∑ ∑ ∑

∑ ∑ ∑∑

∑ ∑ ∑∑

∑

{ }1 1 1

2 2 2 22

2 1 1 1ˆ 1 1 1

N N N

k l l

x x x xNE N

N N N Nσ σ σ σ

= = =

⎡ ⎤⎢ ⎥⎣ ⎦

−⎡ ⎤ ⎛ ⎞ ⎛ ⎞= − + = − =⎜ ⎟ ⎜ ⎟⎢ ⎥⎣ ⎦ ⎝ ⎠ ⎝ ⎠

∑ ∑ ∑


Estimator for the variance•

‘biased’

estimator:

•

‘unbiased’

estimator: (default!)

( )

{ } 222

1

22

111ˆ

ˆ1ˆ

xxx

N

kxkx

NN

NE

xN

σσσ

μσ

⎟⎠⎞

⎜⎝⎛ −

=⎟⎠⎞

⎜⎝⎛ −=

−= ∑=

( )

{ } 221

22

ˆ

ˆ1

1ˆ

xx

N

kxkx

E

xN

σσ

μσ

=

−−

= ∑=


Estimates of Correlation Functions•

Cross-correlation:

•

Let x(t ) and y(t ) are finite time realizations of ergodic processes. Then:

•

Practically, signals are finite time and sampled every Δt. Signal x(t) is sampled at t=0, Δt, …, (N-1)Δt

giving x(i) with i=1, 2, …, N.

•

Then:

with i

=1, 2, …, N

12

ˆ ( ) lim ( ) ( )T

xy TT

x t y t dtT

τ τ→∞

−

Φ = −∫

1τ

τ ττ =

Φ = −− ∑ˆ ( ) ( ) ( )

N

xyi

x i y iN

[ ]( ) ( ) ( )xy E x t y tτ τΦ = −


Estimates of Correlation Functions•

Unbiased estimator:

•

Variance of the estimator increases with lag! To avoid this, divide by N (biased estimator):

•

Use large N to minimize bias! I.e.

•

Similar estimators can be derived for the covariance and correlation coefficient functions

1τ

τ ττ =

Φ = −− ∑ˆ ( ) ( ) ( )

N

xyi

x i y iN

1ˆ ( ) ( ) ( )N

xyi

x i y iN τ

τ τ=

Φ = −∑

1NN τ

→−


Additional demo’s•

Calculation of cross-covariance: Lec2_calculation_Cuy.m

1ˆ ( ) ( ) ( )N

xyi

x i y iN τ

τ τ=

Φ = −∑


Cross-covariance of some basic systems•

Example systems (Matlab

Demo: Lec2_example_systems.m):

• Time delay• 1st order• 2nd order

•

Input signals:• White noise (WN)• Low pass filtered noise (LP: 1/(0.5s+1) )


-5 0 5-0.5

0

0.5

1time delay

Cuy

(WN

)

-5 0 5-0.1

0

0.1

Cuy

(LP

)

-5 0 50

0.5

1

impu

lse

time [s]

-5 0 5-0.01

0

0.01

0.021st order system

-5 0 5-0.02

0

0.02

0.04

-5 0 50

1

2

time [s]

-5 0 5-0.05

0

0.052nd order system

-5 0 5-0.2

0

0.2

-5 0 5-5

0

5

10

time [s]


Intermezzo: Fourier Transform•

Fourier Transform:

•

Mapping between time-domain and frequency-domain• One-to-one mapping• Unique: inverse Fourier Transform exists• Linear technique

•

Inverse Fourier Transform:

( )( ) ( ) 2( ) j ftX f x t x t e dtπ∞

−

−∞

= ℑ = ∫

( )( ) ( )1 2( ) j tfx t X f X f e dfπ∞

−

−∞

= ℑ = ∫


Fourier transformation•

y(t) is an arbitrary signal

•

Euler formula:• symmetric part: cos(z)• anti-symmetric part: sin(z)

2( ) ( ) j ftY f y t e dtπ−= ∫

2 2 2( ) ( ) cos(2 ) *sin(2 )j ft j ft j fte re e im e ft j ftπ π π π π− − −= + = −

( ) ( )cos sinjze z j z= +


Example Fourier Transformation

0 0.5 1 1.5 2 2.5 3 3.5 4 4.5 5-1.5

-1

-0.5

0

0.5

1

1.5

( ) ( )HzHzty 5sin5.0sin)( +=

Y(0.5Hz): 5 Hz signal will be averaged out

2( ) ( ) j ftY f y t e dtπ−= ∫


Example Fourier Transformation

0 0.5 1 1.5 2 2.5 3 3.5 4 4.5 5-1.5

-1

-0.5

0

0.5

1

1.5

-0.5 0 0.5 1 1.50

0.5

1

1.5

2

2.5

3

3.5

4

4.5

5

y(t) = sin(0.5t) Y(ω) = δ(ω-0.5)


Frequency Domain Expressions•

Discrete Fourier Transform:

•

where f

takes values 0, 1, …, N-1

multiples of

•

Inverse Fourier Transform:

( )2

1

( ) ( ) ( )ftN jN

tU f u t u t e

π−

=

= ℑ = ∑

( )21

1

1( ) ( ) ( )ftN jN

fu t U f U f e

Nπ−

=

= ℑ = ∑

1fN t

Δ =Δ


Fourier transform of signals•

Discrete Fourier Transform (DFT)

•

DFT maps N real values in time domain to N complex values in frequency domain

•

Double information?• DFT is symmetric U(-r)=U(r)*• Information for N/2 complex values

( ) [ ]

( ){ } ( )

[ ]1,,1,0);(

)(

1,,1,0;1

0

/2

−∈

==

−∈

∑−

=

−

NrrU

ekukuDFTrU

NkkuN

k

Nrkj

…

…

π

⎥⎦⎤

⎢⎣⎡∈

2,,1,0 Nr …


Power spectrum or auto-spectral density

•

auto-spectral density Sxx

is Fourier Transform of autocorrelation

•

properties:• Real values (no imaginary part)• Symmetry: Sxx

(f)=Sxx

(-f) => only positive frequencies are analyzed• Cxx

(0) = ∫

Sxx

(f) df

= σx2

Area under Sxx

(f) is equal to variance of signal (Parseval’s

theorem)

( )

( ) 2

( ) ( ) ( )

( ) ( ) ( )

jxx xx xx

jfxx xx xx

S e d

S f e d

ωτ

π τ

ω τ τ τ

τ τ τ

∞−

−∞

∞−

−∞

= ℑ Φ = Φ

= ℑ Φ = Φ

∫

∫


Cross-spectrum or Cross-spectral density

•

cross-spectral density Sxy

:

•

properties:• Complex values• Sxy

*(f)=Sxy

(-f) => only positive frequencies are analyzed• Sxy

(f) describes the interdependency of signals x(t) and y(t) in frequency domain (gain and phase)

• if Θxy

(τ) = 0, then Sxy

(f) = 0 for all frequencies

( ) 2( ) ( ) ( ) j fxy xy xyS f e dπ ττ τ τ

∞−

−∞

= ℑ Φ = Φ∫


Estimators for the power spectrum•

Fourier transform of the autocorrelation function, the power spectrum:

•

using the time average estimate:

and multiplication with

•

gives:

* : complex conjugate:

1 2

0

ˆ ˆ( ) ( )fN jN

uu uuS f eτπ

τ

τ− −

=

= Φ∑

1 2 2

0

1 ( )ˆ ( ) ( ) ( )

i f ifN Nj jN N

uui

S f u i e u i eN

τπ π

τ τ

τ−− −

= =

= −∑ ∑

1ˆ ( ) ( ) ( )N

uui

u i u iN τ

τ τ=

Φ = −∑2 0 1

π −−

= =( )i i fj

Ne e

1 * ( ) ( )U n f U n fN

= Δ Δ* ( ) ( )U n f U n fΔ = − Δ

Indirect approach and direct approach give equal result!


Estimator for the cross-spectrum•

Fourier transform of the cross-correlation function is called the cross-spectrum:

1 *ˆ ( ) ( ) ( )uyS f U n f Y n fN

= Δ Δ

1 2

0

τπ

τ

τ− −

=

= Φ∑ˆ ˆ( ) ( )fN jN

uy uyS f e


Time-domain vs. Frequency-domain

x(t), y(t)

Φxy

(τ)

Cxy

(τ)

rxy

(τ)

X(f), Y(f)

Sxy

(f)

Γxy

(f)

input, output

cross-correlationfunction

cross-covariancefunction

correlation coefficient

input, output

cross-spectraldensity

coherence

Fourier TransformationTime

Domain Frequency Domain


Discrete and Continuous Signals

•

Reconstruction• DA conversion• Distortion: Hold Circuits

•

Sampling• AD conversion• Distortion: "Dirac Comb"

•

Continuous system: Plant

•

Digital Controller: Plant performance enhancement


Sampling, Dirac’s Comb


Models of Linear Systems •

System:

•

Linear systems obey both scaling and superposition property!

( ) ( ( ))y t N u t=

1 1 1 1

2 2 2 2

1 1 2 2 1 1 2 2

=

=

+ = +

( ) ( ( ))( ) ( ( ))( ) ( ) ( ( ) ( ))

k y t N k u tk y t N k u tk y t k y t N k u t k u t

Nu(t) y(t)


Time domain models•

Assume input is a pulse with:

•

Response of linear system N to pulse:

•

Assume u(t) is weighted sum of pulses:

( )1 for 0

, 20 otherwise

ttd t t t

⎧ Δ⎛ ⎞ <⎪⎜ ⎟Δ = Δ⎝ ⎠⎨⎪⎩

( )( ) ( ), ,N d t t h t tΔ = Δ

( ) ( ),kk

u t u d t k t t∞

=−∞

= − Δ Δ∑



The output is:

•

Limit case: unit-area pulse d(t,Δt) is an impulse:

•

Then h(t) is the impulse response function (IRF) and the output the convolution integral

( ) ( )0

lim ,td t t tδ

Δ →Δ =

( )( )

( )

∞

=−∞

∞

=−∞

=

= − Δ Δ

= − Δ Δ

∑

∑

( ) ( ( ))

,

,

kk

kk

y t N u t

u N d t k t t

u h t k t t

( ) ( )τ τ τ∞

−∞

= −∫( )y t h u t d



Causal system: h(t)=0 for t<0

•

Finite memory: h(t)=0 for t>T

•

Discrete ( ) ( )

1

0τ

τ τ τ−

=

= − Δ∑( )T

y t h u t

( ) ( )τ τ τ= −∫( )T

o

y t h u t d


Frequency domain models•

Time-domain: convolution integral

•

Fourier transform: ( ) ( ) ( )

( ) ( )

( ) ( )

( )

2

2

2

π

π

π τ

τ τ τ

τ τ τ

τ τ τ

τ τ

∞

−∞

∞ ∞−

−∞ −∞

∞ ∞−

−∞ −∞

∞−

−∞

⎛ ⎞= ℑ = ℑ −⎜ ⎟

⎝ ⎠⎛ ⎞

= −⎜ ⎟⎝ ⎠

= −

=

=

∫

∫ ∫

∫ ∫

∫

( ) ( )

( )

( ) ( ) ( )

j ft

ft

f

Y f y t h u t d

h u t d e dt

h u t e dtd

U f h e d

Y f H f U f

( ) ( )τ τ τ∞

−∞

= −∫( )y t h u t d

‘Convolution in time domain is multiplication in frequency domain’

(and vice-versa)


Estimators in time domain •

Cross-covariance

• When analyzing the dynamics of a dynamical system interested in variations of the signals around it mean (and not average value).

•

Assuming signals with zero-mean

( ) ( ){ }

( ) ( )( ) ( ') ( ') '

( ) ( ') ( ) ( ') 'uy

y t h u t h t u t t dt

C E u t y t E h t u t u t t dtτ τ τ

∞

−∞

∞

−∞

= ∗ = −

⎧ ⎫= − = − −⎨ ⎬

⎩ ⎭

∫

∫


Basic identification with cross- covariance

( ) ( ) ( ) ( )( ) ( ) ( ) ( ) ( ) ( )

( ) ( ) ( ) ( )

( ) ( )

'

multiply with ( ): ' ( )

' '

white noise: 0 for 0; 0 1

uy un uu

uu uu

y t n t h t u t - t' dt'

u t u t y t u t n t h t u t u t - t' dt'

C τ C h t' C t dt

C τ τ C

τ τ τ τ

τ τ

= +

− − = − + −

= + −

= ≠ =

∫∫

∫

( ) ( ) ( )

Other 'tricks' needed when u(t) is not white

uy unC τ C hτ τ= +

? y(t)u(t)

n(t)


Basic identification with spectral densities

( ) ( ) ( ) ( )( ) ( ) ( ) ( ) ( ) ( )

( ) ( ) ( ) ( )( ) ( ) ( ) ( )

( ) ( )

'

multiply with ( ): ' ( )

' '

Fourier transform:

if 0 :

uy un uu

uy un uu

un

y t n t h t u t - t' dt'

u t u t y t u t n t h t u t u t - t' dt'

C τ C h t' C t dt

S f S f H f S f

SS f H f

τ τ τ τ

τ τ

= +

− − = − + −

= + −

= +

= =

∫∫

∫

( )( )

uy

uu

fS f

? y(t)u(t)

n(t)


Identification: time-domain vs. frequency-domain

•

Time domain• Unknown system: impulse response function (IRF) of h(t’) over

limited time• Mostly: direct model parameterization (fit of physics model)

•

Frequency domain• Unknown system: frequency response function (FRF) H(f) for number

of frequencies

? y(t)u(t)

n(t)


Example IRF & FRF of typical systems

•

Time delay•

First order system

•

Second order system•

Unstable system

•

(try Matlab!)


Book: Westwick

& Kearney•

Lecture 1: signals

• Chapter 1, all• Chapter 2, sec. 2.1 –

2.3.1

•

Lecture 2: Correlation functions in time & frequency domain• Chapter 2, sec. 2.3.2 –

2.3.4

• Chapter 3, sec. 3.1 –

3.2

•

Lecture 3: Estimators for impulse & frequency response functions• Chapter 5, sec. 5.1 –

5.3


Next week: lecture 3•

Lecture 3:

• Estimation of IRF and FRF• Estimator for coherence• Open loop and closed loop system identification

System Identification Parameter Estimation - TU … · 2/11/2010 Delft University of Technology...

Documents

Transcript of System Identification Parameter Estimation - TU … · 2/11/2010 Delft University of Technology...