Post on 25-Jun-2020
Graz University of Technology – SPSC Laboratory
Lesson 3: Fading Memory Nonlinearities
Nonlinear Signal Processing – SS 2014
Paul MeissnerSignal Processing and Speech Communication Laboratory
Graz University of Technology
May 12, 2014
NLSP SS 2014 May 12, 2014 Slide 1/19
Graz University of Technology – SPSC Laboratory
Session contents
◮ Today:◮ Volterra Series as representation of fading-memory NL◮ System identification using different inputs◮ Time series modelling (Homework)
◮ Next time:◮ Higher-order statistics and spectral analysis
NLSP SS 2014 May 12, 2014 Slide 2/19
Graz University of Technology – SPSC Laboratory
Volterra series – Definition
◮ A finite Volterra series of order p and memory length M:
y [n] = h0 +
M−1∑
m1=0
h1[m1] x[n −m1]
+
M−1∑
m1=0
M−1∑
m2=0
h2[m1,m2] x[n −m1] x[n −m2]+
+M−1∑
m1=0
M−1∑
m2=0
M−1∑
m3=0
h3[m1,m2,m3] x[n −m1] x[n −m2] x[n −m3] + · · ·+
+
M−1∑
m1=0
· · ·
M−1∑
mp=0
hp[m1, . . . ,mp]
p∏
i=1
x[n −mi ]
◮ Universal approximator for time-invariant causal operatorswith fading memory (for bounded input signals)
NLSP SS 2014 May 12, 2014 Slide 3/19
Graz University of Technology – SPSC Laboratory
Volterra series – System identification (1)
◮ Only term with h1[·] is linear!
◮ Complexity grows exponentially with Mp
◮ Choice of input signal for system identification?
◮ Remember RBF-fits: Model also linear in coefficients →Least-squares fit easy
y [n] =∑
k
αkφk(x [n]) N equations
◮ Arrange in equation system
y [0]...
y [N − 1]
︸ ︷︷ ︸
y
=
φ1(x[0]) · · · φK (x[0])...
. . ....
φ1(x[N − 1]) · · · φK (x[N − 1])
︸ ︷︷ ︸
Φ, (N×K ), N≫K
α1
...αK
NLSP SS 2014 May 12, 2014 Slide 4/19
Graz University of Technology – SPSC Laboratory
Volterra series – System identification (2)
◮ Here we use the same method, just different basis functions
◮ Second order Volterra system as example (1 +M +M2 coeff.)◮ You need “quite some” data!
y [n] = h0 +
M−1∑
m1=0
h1[m1] x [n−m1] +
M−1∑
m1=0
M−1∑
m2=0
h2[m1,m2] x [n−m1] x [n−m2]
y [0]y [1]
.
.
.y [N − 1]
=
1 x [0] · · · x [−M + 1] x2[0] x [0]x [−1] · · · x2[−M + 1,−M + 1]
1 x [1] · · · x [−M + 2] x2[1] x [1]x [0] · · · x2[−M + 2,−M + 2]
.
.
.
.
.
.
h0h1 [0]
.
.
.h1[M − 1]h2 [0, 0]h2 [0, 1]
.
.
.h2[M − 1,M − 1]
◮ Basis vectors/functions are the data products
NLSP SS 2014 May 12, 2014 Slide 5/19
Graz University of Technology – SPSC Laboratory
Volterra series – Matlab files
◮ Function vkernels.m does the LS-fit (use p3 1.m as tutorial)◮ Nonlinearities defined in nlsystem1.m
y [n] = a0x[n] + a1x[n − 1] + a2x[n]2 + a3x[n]x[n − 1]
◮ Output of vkernels.m for this nonlinearity:
H{1} = 0
H{2} =[a0 a1
]T
H{3} =
x [n]2
︷︸︸︷a2
x [n]x [n−1]︷︸︸︷a3
a3︸︷︷︸
x [n−1]x [n]
0︸︷︷︸
x [n−1]2
◮ Or optionally a structure Vmodel that can be directly passedto vkernels o.m for Problem 3.2
NLSP SS 2014 May 12, 2014 Slide 6/19
Graz University of Technology – SPSC Laboratory
Volterra series – Time series modelling/forecasting
◮ Given a correlated (not necessarlily just second order!) timeseries s = [s1, . . . , sN ]
T
◮ Current sample sn depends on past samples
◮ Volterra series one way to model the dependence
◮ Input x and output y both generated from s
◮ Partitioning of s into training and validation sequences
→ Homework
NLSP SS 2014 May 12, 2014 Slide 7/19
Graz University of Technology – SPSC Laboratory
Higher order statistics and spectral analysis
◮ We are used to first- and second-order statistics◮ Mean: µ = E{x [n]}◮ ACF: m2[k ] = E{x [n]x [n + k ]}
◮ This is suitable as long as we deal with linear systems, e.g.
y [n] =
K∑
k=0
x[k]h[n − k]
◮ Standard example for a random process: Linear process, i.e.x [k] is a white-noise process driving a linear system
◮ But what if our system is nonlinear, e.g.
y [n] = a0x[n] + a1x[n − 1] + a2x[n]2 + a3x[n]x[n − 1]
NLSP SS 2014 May 12, 2014 Slide 8/19
Graz University of Technology – SPSC Laboratory
Higher order statistics and spectral analysis - Example
◮ Linear system w. K = 10, driven by white noise
−50 −40 −30 −20 −10 0 10 20 30 40 50−0.2
0
0.2
0.4
0.6
0.8
1
1.2
k
R
xx[k]
◮ Support of non-zero ACF indication for memory
◮ How do you evaluate nonlinear combinations of input samples?
→ Evaluate HOS, e.g. third-order E{x [n]x [n + k]x [n + l ]}
NLSP SS 2014 May 12, 2014 Slide 9/19
Graz University of Technology – SPSC Laboratory
Higher order statistics and spectral analysis - Example
◮ Linear system w. K = 10, driven by white noise
−50 −40 −30 −20 −10 0 10 20 30 40 50−0.2
0
0.2
0.4
0.6
0.8
1
1.2
k
R
xx[k]
Ryy,lin
[k]
◮ Support of non-zero ACF indication for memory
◮ How do you evaluate nonlinear combinations of input samples?
→ Evaluate HOS, e.g. third-order E{x [n]x [n + k]x [n + l ]}
NLSP SS 2014 May 12, 2014 Slide 9/19
Graz University of Technology – SPSC Laboratory
Higher order statistics and spectral analysis - Example
◮ Linear system w. K = 10, driven by white noise
−50 −40 −30 −20 −10 0 10 20 30 40 50−0.2
0
0.2
0.4
0.6
0.8
1
1.2
k
R
xx[k]
◮ Support of non-zero ACF indication for memory
◮ How do you evaluate nonlinear combinations of input samples?
→ Evaluate HOS, e.g. third-order E{x [n]x [n + k]x [n + l ]}
NLSP SS 2014 May 12, 2014 Slide 9/19
Graz University of Technology – SPSC Laboratory
Higher order statistics and spectral analysis - Example
◮ Linear system w. K = 10, driven by white noise
−50 −40 −30 −20 −10 0 10 20 30 40 50−0.2
0
0.2
0.4
0.6
0.8
1
1.2
k
R
xx[k]
Ryy,nonlin
[k]
◮ Support of non-zero ACF indication for memory
◮ How do you evaluate nonlinear combinations of input samples?
→ Evaluate HOS, e.g. third-order E{x [n]x [n + k]x [n + l ]}
NLSP SS 2014 May 12, 2014 Slide 9/19
Graz University of Technology – SPSC Laboratory
HOSA – Definitions
◮ Generalization of ACF: Non-central moments of order r forstationary process x [n]
mr [k1, . . . , kr−1] = E{x[n]x[n + k1] · · · x[n + kr−1]}
◮ Cumulant of order r again defined over characteristic function
◮ Can be expressed and estimated via moments
◮ Fourier transform of ACF is power spectral density
◮ Fourier transform (2D) of third order cumulant is theBispectrum
◮ cx3[k1, k2]DFT−→ Cx
3 [ω1, ω2]
NLSP SS 2014 May 12, 2014 Slide 10/19
Graz University of Technology – SPSC Laboratory
HOSA – Important Properties
◮ x [n] Gaussian:
cxr (k1, . . . , kr−1) = Cxr (ω1, . . . , ωr−1) = 0 for r > 2
◮ x [n] i.i.d.:
cxr (k1, . . . , kr−1) = a · δ(k1, . . . , kr−1)Cxr (ω1, . . . , ωr−1) = a
◮ x [n] symmetrically distributed around zero:
cxr (k1, . . . , kr−1) = Cxr (ω1, . . . , ωr−1) = 0 for r = 0, 3, 5, 7, . . .
◮ z [n] = x [n] + y [n], where x [n], y [n] jointly stationary andstatistically independent:
czr (·) = cxr (·) + cyr (·)Czr (·) = Cx
r (·) + Cyr (·)
NLSP SS 2014 May 12, 2014 Slide 11/19
Graz University of Technology – SPSC Laboratory
HOSA – Pros and Cons
Pros
◮ Analysis of nonlinearities
◮ Cumulants are additive for independent processes
◮ Gaussian noise: HOS zero (blind to Gaussian noise)
Cons
◮ Difficult to estimate from finite length data
◮ Influence of window
◮ Once you have them, how do you interpret them?
NLSP SS 2014 May 12, 2014 Slide 12/19
Graz University of Technology – SPSC Laboratory
HOSA – Example
◮ Example for a third-order cumulant
−15 −10 −5 0 5 10 15
−15
−10
−5
0
5
10
15
−0.02
−0.01
0
0.01
0.02
0.03
NLSP SS 2014 May 12, 2014 Slide 13/19
Graz University of Technology – SPSC Laboratory
HOSA – Example
◮ Example for a Bispectrum
−1 −0.8 −0.6 −0.4 −0.2 0 0.2 0.4 0.6 0.8 1−1
−0.8
−0.6
−0.4
−0.2
0
0.2
0.4
0.6
0.8
1
0.02
0.04
0.06
0.08
0.1
0.12
0.14
0.16
0.18
0.2
0.22
NLSP SS 2014 May 12, 2014 Slide 13/19
Graz University of Technology – SPSC Laboratory
HOSA – Example
◮ Example for a PSD
0 0.5 1 1.5 2 2.5 3−50
−45
−40
−35
−30
−25
−20
−15
−10
−5
0
ω, [rad/sample]
PS
D e
stim
ate
C2
x(ω)
C2
y(ω)
NLSP SS 2014 May 12, 2014 Slide 13/19
Graz University of Technology – SPSC Laboratory
HOSA – Matlab (1)
◮ HOSA toolbox, free, included in download-file
◮ HOSA toolbox manual is a great ressource (Matlab-central)!
◮ You will need:◮ cumest.m used to estimate cumulants◮ rpiid.m used to help generating input processes◮ gabrrao.m used to calculate window for 2D-FFT◮ viscumul3.m and◮ visbispec3.m for visualization
NLSP SS 2014 May 12, 2014 Slide 14/19
Graz University of Technology – SPSC Laboratory
HOSA – Matlab (2)
◮ cumest.m for third order cumulant calculates just one slice ofthe 2D correlation function
for k = -MaxLag : MaxLag
c3(:, k+MaxLag+1) = cumest(#, #, #, #, #, #, k);
end
◮ Bispectrum calculation: use fftshift(fft2( c3 .* w ))
to have a familiar picture
◮ Window w[n] obtained from gabrrao.m, optimal smoothingwindow, minimum bias in estimation
NLSP SS 2014 May 12, 2014 Slide 15/19
Graz University of Technology – SPSC Laboratory
Higher order statistics and spectral analysis - Problems (1)
◮ Limited amount of data leads to higher variance of estimators
◮ Problem even for ACF → Grows exponentially with order
◮ This makes visual interpretation much harder:◮ When is a Bispectrum zero?◮ When is a Bicoherence function flat?
NLSP SS 2014 May 12, 2014 Slide 16/19
Graz University of Technology – SPSC Laboratory
Higher order statistics and spectral analysis - Problems (2)
◮ Linear system w. K = 10, driven by white noise (500 samples)
−50 −40 −30 −20 −10 0 10 20 30 40 50−0.2
0
0.2
0.4
0.6
0.8
1
1.2
k
R
xx[k]
NLSP SS 2014 May 12, 2014 Slide 17/19
Graz University of Technology – SPSC Laboratory
Higher order statistics and spectral analysis - Problems (2)
◮ Linear system w. K = 10, driven by white noise (500 samples)
−50 −40 −30 −20 −10 0 10 20 30 40 50−0.4
−0.2
0
0.2
0.4
0.6
0.8
1
k
R
xx[k]
Ryy,lin
[k]
NLSP SS 2014 May 12, 2014 Slide 17/19
Graz University of Technology – SPSC Laboratory
Higher order statistics and spectral analysis - Problems (2)
◮ Linear system w. K = 10, driven by white noise (500 samples)
−50 −40 −30 −20 −10 0 10 20 30 40 50−0.2
0
0.2
0.4
0.6
0.8
1
1.2
k
R
xx[k]
NLSP SS 2014 May 12, 2014 Slide 17/19
Graz University of Technology – SPSC Laboratory
Higher order statistics and spectral analysis - Problems (2)
◮ Linear system w. K = 10, driven by white noise (500 samples)
−50 −40 −30 −20 −10 0 10 20 30 40 50−0.2
0
0.2
0.4
0.6
0.8
1
1.2
k
R
xx[k]
Ryy,nonlin
[k]
NLSP SS 2014 May 12, 2014 Slide 17/19
Graz University of Technology – SPSC Laboratory
HOSA – Problem 3.7, glstat.m (see HOSA manual)
... statistical tests for Gaussianity and Linearity using the squaredauto-bicoherence function |γ3x(ω1, ω2)|
2
... assumption: samples of Cx3 (ω1, ω2) asymptotically Gaussian
... remember: checks only one HOS (cx3) ⇒ no guarantee
Gaussian Process
... Cx3 (ω1, ω2) is zero
◮ auto-bicoherence |γ3x(ω1, ω2)|2 is ∼ χ2(0, 2) (central)
|γ3x(ω1, ω2)|2 =
|Cx3 (ω1, ω2)|
2
Cx2 (ω1 + ω2)Cx
2 (ω1)Cx2 (ω2)
(1)
◮ sum of P points of |γ3x(ω1, ω2)|2: ∼ χ2(0, 2P)
◮ Pfa: probability for sum being ∼ χ2(0, 2P)
NLSP SS 2014 May 12, 2014 Slide 18/19
Graz University of Technology – SPSC Laboratory
HOSA – Problem 3.7, glstat.m (see HOSA manual)
Linear Process
... |γ3x(ω1, ω2)|2 = λ is constant
◮ auto-bicoherence |γ3x(ω1, ω2)|2 is ∼ χ2(λ, 2) (noncentral)
◮ compares interquartile range to this distribution◮ R (estimated): difference between medians of values
above/below median◮ R (theory): χ2(λ, 2)◮ linear if approximately equal
x
min maxmedian (middle quartile)
median (lower quartile) median (upper quartile)interquartile range
NLSP SS 2014 May 12, 2014 Slide 19/19