E2C2-GIACS Advanced School on

‘Extreme Events: Nonlinear Dynamics and Time Series Analysis’

Smooth Time Series - Applications

Dmitri Kondrashov

University of California, Los Angeles

Joint work with M. Ghil, and many others

http://www.atmos.ucla.edu/tcd

1/32

Spectral Analysis

S(f) ~ f – p + poles

i.e. linear in log-log coordinates

For a 1st-order Markov process or “red noise” p = 2

“Pink” noise, p = 1 (1/f , flicker noise)

“White” noise, p = 0

Main challenge for short and noisy geophysical time series to distinguish between poles and (red) noise.

Tradeoff for spectral methods: resolution (smoothing, windowing) vs. spurious peaks & power leakage.

6/32

x = −ω2x vs. x = −λx

Synthetic example (Task & Prize!!!)

7/32

Datax101

0.11

0.21

0.00

-0.11

-0.21

1.0 40.8 80.6 120.4 160.2 200.0

Q: Is there a periodicity and what is its period?

A: What is the underlying noise “null hypothesis”?

Classical Spectral Methods

AR(1)95%

Blakman-Tukey FFT (window 80)

Frequency

0.00 0.10 0.20 0.30 0.40 0.50

10-1

10-0

101

AR(1)95%

Blakman-Tukey FFT (window 80)

Frequency

0.00 0.10 0.20 0.30 0.40 0.50

10-1

10-0

101

AR(1)95%

Blakman-Tukey FFT (window 120)

Frequency

0.00 0.10 0.20 0.30 0.40 0.50

10-1

10-0

101

AR(1)95%

Blakman-Tukey FFT (window 120)

Frequency

0.00 0.10 0.20 0.30 0.40 0.50

10-1

10-0

101

MEMMaximum Entropy Method (40)

Frequency

0.00 0.10 0.20 0.30 0.40 0.50

10-1

10-0

101

8/32

MEMMaximum Entropy Method (10)

Frequency

0.00 0.10 0.20 0.30 0.40 0.50

100

101

Advanced Spectral Methods

10/32

••

••

•••

•••

•

•••

••

•••

••

••

••

••••

••

••••••••

Singular Spectrum Analysis

Frequency

0.00 0.10 0.20 0.30 0.40 0.49

10-1

10-0

101

Multi-taper method

Frequency

AR(1) 99% 95% 90%

0.00 0.10 0.20 0.30 0.40 0.50

10-1

10-0

101

- Singular spectrum analysis (SSA)and Multi-taper method (MTM).

- detection of periodic signals: phase and amplitude modulation, intermittent behavior, large noise.

- use data-adaptive orthogonal basis in frequency domain (MTM) and time domain (SSA).

- significance tests for spectral peaks.

Right answer!

9/32

Signal

Synthetic Datax101

0.11

0.21

0.00

-0.11

-0.21

1.0 40.8 80.6 120.4 160.2 200.0

SSA Power Spectra & Reconstruction

A. Transform pair:

For given window M, ek’’s are adaptive filters (empirical orthogonal functions)

the ak’’s are filtered time series, principal components in time domain.

B. Power spectra

C. Reconstruction

in particular:

ak(t) =M∑

s=1

X(t + s)ek(s), ak(t)− PC

X(t + s) =M∑

k=1

ak(t)ek(s), ek(s)− EOF

XK(t) =1M

∑

k∈K

M∑

s=1

ak(t− s)ek(s);

12/32

K = {1, 2, ....., S} or K = {k} or K = {l, l + 1;λl ≈ λl+1}

SX(f) =M∑

k=1

Sk(f); Sk(f) = Rk(s); Rk(s) ≈1

T

∫ T

0

ak(t)ak(t + s)dt

SSA of Southern Oscillation Index (ENSO)

•Powerful noise filter: Break in slope of SSA spectrum distinguishes “significant” from “noise” EOFs

14/32

•Formal Monte-Carlo test identifies 4-yr and 2-yr ENSO oscillatory modes (SSA pairs). A window size of M = 60 is enough to “resolve” these modes in a monthly SOI time series.

SSA Forecast (ENSO)

Filter “signal” and forecast with AR(M) model.

Cross-validation to find optimum number of “signal” components and error bars of the forecast.

Correlations are both advantage and limitations of empirical models.

15/32

Real-time Plume of Climate Forecasts (ENSO)

16/32

IRI Multi-model forecast of Nino-34 index

UCLA-TCD: Kondrashov, D., S. Kravtsov, A. W. Robertson, and M. Ghil (2005), A hierarchy of data-based ENSO models, J. Climate, 18, 4425–4444. 495

!"" "#$ "%& %&' &'( '() ()" )"* "*! *!% !%! %!"!+,-

!+

!.,-

!.

!/,-

/

/,-

.

.,-

+

+,-

0

OBS FORECAST

(1('0,23&&43%5678$9:6;<

!6=>$3*6?>@8ABA36C3D(&'3C?673Aug 2007

IRI

+//E +//F

(%&%3G!%'

(;DH3;*&

"!%

&;I1HH&

J)D'

%K&LH'%!%

D;!M*

KN!'

N!%3&(K

D;O%!L!'!

;'J%3%('!

!>B*I%(;D

"H(!*I;G;

;H;3!IN'P

;);3J1!

;H;3;%

;H;3;;%

;&K3;J1HI

KQ;3((D4

*&K3IDGI

K;J%!4;)

)9587R@8$3!6=>$S

&B8BRABR@8$3!6=>$S

Dealing with

Missing

Data

17/32

Historical records are full of “gaps”....

Annual maxima and minima of the water level at the nilometer on Rodah Island, Cairo. 18/32

... nowdays on Earth...

SST (AMSR-E), daily 2x2, June 2002 – February 2007: 38.2% of missing points

Wind (QuikSCAT), daily 2x2, July 1999 -- February 2007:17.2% of missing points

Gaps: satellite coverage, precipitation and clouds.

20/32

23/32

1. Choose window M and set K=1. Flag fraction of dataset X(t)(t=1:N) as “missing” for cross-validation.

D =

X(1) X(2) . . X(M)X(2) X(3) . . X(M + 1)

. . . . .

X(N ′− 1) . . . X(N − 1)

X(N ′) X(N ′ + 1) . . X(N)

CX =1

N ′D

tD;CXEk = λkEk

Ak(t) =∑M

j=1X(t + j − 1)Ek(j)

RK(t) = 1

Mt

∑k∈K

∑Ut

j=LtAk(t − j + 1)Ek(j);

4. If convergence for missing points, K = K +1. Check cross-validation error, and Go to Step 2 if necessary.

2. Update mean and covariance, find leading K EOFs

3. Reconstruct missing points using K EOFs

Utilize (spatio) temporal correlations to iteratively compute self-consistent lag-covariance matrix => can be applied to very gappy data.

Follows expectation maximization (EM) procedure for finding maximum likelihood estimates of mean and covariance matrix.

A few K leading EOFs correspond to the “smooth” modes, while the rest is noise.

Provides both spectral analysis and estimates of missing data.

SSA gap-filling

D. Kondrashov and M. Ghil, 2006: Spatio-temporal filling of missing points in geophysical data sets,Nonl. Proc. Geophys., 13, 151-159.

Synthetic I: Gaps in Oscillatory Signal

• Very good gap filling for smooth modulation; OK for sudden modulation.

0 50 100 150 200!5

0

5a)SSA gap filling in [20:40] range

0 2 4 6 8 10

0.2

0.4

No. of modes

CVL

Erro

r

Optimum window M and number of modes101520

0 50 100 150 200!5

0

5b)SSA gap filling in [120:140] range

0 2 4 6 8

0.2

0.4

No. of modesCV

L Er

ror

Optimum window M and number of modes

152025

0 0.1 0.2 0.3 0.4 0.5

100

SSA spectrum, M=10

Frequency0 0.1 0.2 0.3 0.4 0.5

100

SSA spectrum, M=20

Frequency

24/32

Synthetic II: Gaps in Oscillatory Signal + Noise

x(t) = sin( 2π300 t) ∗ cos( 2π

40 t + π2 sin 2π

120 t)25/32

! "!! #!! $!!!%

!

%&'()*++*,-(-&.(/

0! "!! #!! $!!

!%

!

%

0

1'()*++*,-(-&.(//

2! %!! %2!

!"!"

0

3'()*++*,-(-&.(/

"!! "2! 4!! 42!

!"!"

0

5'()*++*,-(-&.(//

" # $ 6 %!

%

%7!2

897(9:(;95<=

<'(>?9==!@&+*5&0*9,(ABC(<??9?

! !7!" !7!#

%!!

:'(CCD(=.<30?E;F(BG"!!

)?<HE<,3I

Synthetic III: Synthetic gaps in SST data

1950-2004 IRI monthly SST dataset (10°x10°, 660x237 grid points) see improvement with MSSA; “random” pattern favors small M!Error is smaller in CE Pacific Sector where “signal” (ENSO) mode is dominant!

Synthetic IV: multivariate example (Prize!!!)

22/32

-2.00e+00

2.00e+00Dataset with 50% of missing data

Time

Spac

e

x102

0.01

0.07

0.14

0.20

1.0 20.8 40.6 60.4 80.2 100.0

-2.00e+00

2.00e+00Estimated smooth component

Time

Sp

ace

x102

0.01

0.07

0.14

0.20

1.0 20.8 40.6 60.4 80.2 100.0

-2.00e+00

2.00e+00Original noisy dataset

Time

Sp

ace

x102

0.01

0.07

0.14

0.20

1.0 20.8 40.6 60.4 80.2 100.0

-2.00e+00

2.00e+00Smooth component of full dataset

Time

Spac

e

x102

0.01

0.07

0.14

0.20

1.0 20.8 40.6 60.4 80.2 100.0

Filled-in Southern Ocean data

Gap-filling needs to respect physical limits

29/32

•Freeware ported to Sun, Dec, SGI, Linux, and Mac OS X: self-contained binary (~2-5Mb) depending on the Unix platform. •http://www.atmos.ucla.edu/tcd/ssa/ •Needs external graphics package: Grace (free, default) is a part of

standard Linux installation, may need compiling for other OS; IDL ($$)•Includes Blackman-Tukey FFT, Maximum Entropy Method, Multi-

Taper Method (MTM), SSA and M-SSA. •Spectral estimation, decomposition, reconstruction, gap-filling•Significance tests of “oscillatory modes” vs. “noise.”

30/32

SSA-MTM Toolkit

• Data management with named vectors & matrices.

• Default values. 31/32

SSA-MTM Toolkit (cont’d)

•Project files•SSA Forecasts•Automated tasks•Built-in plots•Animations (QuickTime)•Automation (Automator)•www.spectraworks.com

31/32

Mac OS X: kSpectra Toolkit

• Ghil M., R. M. Allen, M. D. Dettinger, K. Ide, D. Kondrashov, M. E. Mann, A. Robertson, A. Saunders, Y. Tian, F. Varadi, and P. Yiou, 2002: "Advanced spectral methods for climatic time series," Rev. Geophys.,40(1), pp. 3.1-3.41, 10.1029/2000RG000092.

• D. Kondrashov and M. Ghil, 2006: Spatio-temporal filling of missing points in geophysical data sets,Nonl. Proc. Geophys., 13, 151-159.

• more at http://www.atmos.ucla.edu/tcd/ssa.

• Computer Lab: SOI (ENSO), “small signal”, gap-filling, multivariate example (time permitting)

32/32

END