Data Assimilation with Overlapping Windows
Yannick Trémolet, Elias Hólm
ECMWF
ISDA, Reading, July 2016
Y. Trémolet Overlapping Window 4D-Var
Long Window Weak Constraint 4D-Var
�������
�� ��
�
�������
��� �
����
(1) Weak constraint 4D-Var
(2) Extended window
�������
�� ��
�
��
�������
��
� ���
(3) Initial term has converged (4) Assimilation window is moved forward
• This implementation is an approximation of weak constraint 4D-Var with anassimilation window that extends (almost) indefinitely in the past...
• ...which is equivalent to a full rank Kalman smoother (Fisher et al., 2005,Ménard and Daley, 1996) that has been running indefinitely.
• In principle B is a problem of the past, only the error characteristics of thefundamental ingredients of the DA problem remain.
Y. Trémolet Overlapping Window 4D-Var 1 / 12
Long Window Weak Constraint 4D-Var
�������
�� ��
�
�������
��� �
����
(1) Weak constraint 4D-Var (2) Extended window
�������
�� ��
�
��
�������
��
� ���
(3) Initial term has converged (4) Assimilation window is moved forward
• This implementation is an approximation of weak constraint 4D-Var with anassimilation window that extends (almost) indefinitely in the past...
• ...which is equivalent to a full rank Kalman smoother (Fisher et al., 2005,Ménard and Daley, 1996) that has been running indefinitely.
• In principle B is a problem of the past, only the error characteristics of thefundamental ingredients of the DA problem remain.
Y. Trémolet Overlapping Window 4D-Var 1 / 12
Long Window Weak Constraint 4D-Var
�������
�� ��
�
�������
��� �
����
(1) Weak constraint 4D-Var (2) Extended window
�������
�� ��
�
��
�������
��
� ���
(3) Initial term has converged
(4) Assimilation window is moved forward
• This implementation is an approximation of weak constraint 4D-Var with anassimilation window that extends (almost) indefinitely in the past...
• ...which is equivalent to a full rank Kalman smoother (Fisher et al., 2005,Ménard and Daley, 1996) that has been running indefinitely.
• In principle B is a problem of the past, only the error characteristics of thefundamental ingredients of the DA problem remain.
Y. Trémolet Overlapping Window 4D-Var 1 / 12
Long Window Weak Constraint 4D-Var
�������
�� ��
�
�������
��� �
����
(1) Weak constraint 4D-Var (2) Extended window
�������
�� ��
�
��
�������
��
� ���
(3) Initial term has converged (4) Assimilation window is moved forward
• This implementation is an approximation of weak constraint 4D-Var with anassimilation window that extends (almost) indefinitely in the past...
• ...which is equivalent to a full rank Kalman smoother (Fisher et al., 2005,Ménard and Daley, 1996) that has been running indefinitely.
• In principle B is a problem of the past, only the error characteristics of thefundamental ingredients of the DA problem remain.
Y. Trémolet Overlapping Window 4D-Var 1 / 12
Long Window Weak Constraint 4D-Var
�������
�� ��
�
�������
��� �
����
(1) Weak constraint 4D-Var (2) Extended window
�������
�� ��
�
��
�������
��
� ���
(3) Initial term has converged (4) Assimilation window is moved forward
• This implementation is an approximation of weak constraint 4D-Var with anassimilation window that extends (almost) indefinitely in the past...
• ...which is equivalent to a full rank Kalman smoother (Fisher et al., 2005,Ménard and Daley, 1996) that has been running indefinitely.
• In principle B is a problem of the past, only the error characteristics of thefundamental ingredients of the DA problem remain.
Y. Trémolet Overlapping Window 4D-Var 1 / 12
We are not there yet...
Weak constraint 4D-Var is difficult to implement in the IFS (both for scientificand technical reasons).
1. Overlapping windows with strong constraint 4D-Var
2. Results in reanalysis context
3. Results in NWP context
4. Comments about re-use of observations
Y. Trémolet Overlapping Window 4D-Var 2 / 12
Operational System: Observations Delay
• Observations are not availableinstantly
• Operational systems work with along-ish cut-off
• Issue: not all observations that havearrived are used!
• An early-delivery assimilation is run
• When cycling, information from theearly delivery run is lost
Y. Trémolet Overlapping Window 4D-Var 3 / 12
Operational System: Observations Delay
• Observations are not availableinstantly
• Operational systems work with along-ish cut-off
• Issue: not all observations that havearrived are used!
• An early-delivery assimilation is run
• When cycling, information from theearly delivery run is lost
Y. Trémolet Overlapping Window 4D-Var 3 / 12
Operational System: Observations Delay
• Observations are not availableinstantly
• Operational systems work with along-ish cut-off
• Issue: not all observations that havearrived are used!
• An early-delivery assimilation is run
• When cycling, information from theearly delivery run is lost
Y. Trémolet Overlapping Window 4D-Var 3 / 12
Operational System: Observations Delay
• Observations are not availableinstantly
• Operational systems work with along-ish cut-off
• Issue: not all observations that havearrived are used!
• An early-delivery assimilation is run
• When cycling, information from theearly delivery run is lost
Y. Trémolet Overlapping Window 4D-Var 3 / 12
Operational System: Observations Delay
• Observations are not availableinstantly
• Operational systems work with along-ish cut-off
• Issue: not all observations that havearrived are used!
• An early-delivery assimilation is run
• When cycling, information from theearly delivery run is lost
Y. Trémolet Overlapping Window 4D-Var 3 / 12
Overlapping System: Observations Delay
• A shorter cut-off is used
• The next overlapping cycle will usethe observations that have arrived inbetween
• No observations are lost
• The cost is the same: 4 × 12h4D-Var vs. 2 × 12h 4D-Var + 4 ×6h 4D-Var
• It is possible to use newly arrivedobservations only
• This algorithm is a quasi-smoother
Y. Trémolet Overlapping Window 4D-Var 4 / 12
Overlapping System: Observations Delay
• A shorter cut-off is used
• The next overlapping cycle will usethe observations that have arrived inbetween
• No observations are lost
• The cost is the same: 4 × 12h4D-Var vs. 2 × 12h 4D-Var + 4 ×6h 4D-Var
• It is possible to use newly arrivedobservations only
• This algorithm is a quasi-smoother
Y. Trémolet Overlapping Window 4D-Var 4 / 12
Overlapping System: Observations Delay
• A shorter cut-off is used
• The next overlapping cycle will usethe observations that have arrived inbetween
• No observations are lost
• The cost is the same: 4 × 12h4D-Var vs. 2 × 12h 4D-Var + 4 ×6h 4D-Var
• It is possible to use newly arrivedobservations only
• This algorithm is a quasi-smoother
Y. Trémolet Overlapping Window 4D-Var 4 / 12
Overlapping System: Observations Delay
• A shorter cut-off is used
• The next overlapping cycle will usethe observations that have arrived inbetween
• No observations are lost
• The cost is the same: 4 × 12h4D-Var vs. 2 × 12h 4D-Var + 4 ×6h 4D-Var
• It is possible to use newly arrivedobservations only
• This algorithm is a quasi-smoother
Y. Trémolet Overlapping Window 4D-Var 4 / 12
Future use of long windows
time
• The assimilation window can be moved in time frequently, for example by 15minutes every 15 minutes.
• Observations that have arrived in the last 15 minutes are added.– Very short cut-off time, without losing observations.
• Because of the large overlap, few inner iterations are needed at each stage– but many effective outer iterations.
• The assimilation becomes a service that runs continuously:– an up-to-date analysis is always available,– less pressure on the time critical path,– can be optimised for energy consumption rather than time to solution,– no peak in daily computer usage (smaller machine).
Y. Trémolet Overlapping Window 4D-Var 5 / 12
Overlapping Windows
• The assimilation needs updating several times daily: long assimilationwindows will overlap.
time
time
xb
– The background and observationerrors are uncorrelated
– The background is not the most upto date
– Two DA streams runningindependently
time
xb
– The background is the mostaccurate
– The background and observationerrors are correlated
• In practice, using the most recent background gives the best results.
Y. Trémolet Overlapping Window 4D-Var 6 / 12
Overlapping Windows
• The assimilation needs updating several times daily: long assimilationwindows will overlap.
time
xb
– The background and observationerrors are uncorrelated
– The background is not the most upto date
– Two DA streams runningindependently
time
xb
– The background is the mostaccurate
– The background and observationerrors are correlated
• In practice, using the most recent background gives the best results.
Y. Trémolet Overlapping Window 4D-Var 6 / 12
Overlapping Windows
• The assimilation needs updating several times daily: long assimilationwindows will overlap.
time
xb
– The background and observationerrors are uncorrelated
– The background is not the most upto date
– Two DA streams runningindependently
time
xb
– The background is the mostaccurate
– The background and observationerrors are correlated
• In practice, using the most recent background gives the best results.
Y. Trémolet Overlapping Window 4D-Var 6 / 12
Overlapping Windows
• The assimilation needs updating several times daily: long assimilationwindows will overlap.
time
xb
– The background and observationerrors are uncorrelated
– The background is not the most upto date
– Two DA streams runningindependently
time
xb
– The background is the mostaccurate
– The background and observationerrors are correlated
• In practice, using the most recent background gives the best results.
Y. Trémolet Overlapping Window 4D-Var 6 / 12
Ps-only Re-analysis
Background and Analysis fit to Observationsthroughout the assimilation window
2004-07-01 to 2005-04-09
0 3 6 9 12 15 18 21 24
Time in assimilation window (h)
0.5
1.0
1.5
2.0
Ps
obse
rvat
ion
fit(h
Pa)
Overlapping 24h 4D-Var 24h 4D-Var (ERA 20C) 12h 4D-Var
Paul PoliY. Trémolet Overlapping Window 4D-Var 7 / 12
Ps-only Re-analysis
Background and Analysis fit to Observationsthroughout the assimilation window
2004-07-01 to 2005-04-09
0 3 6 9 12 15 18 21 24
Time in assimilation window (h)
0.5
1.0
1.5
2.0
Ps
obse
rvat
ion
fit(h
Pa)
12h
Overlapping 24h 4D-Var 24h 4D-Var (ERA 20C) 12h 4D-Var
Paul PoliY. Trémolet Overlapping Window 4D-Var 7 / 12
Ps-only Re-analysis
Forecast scores vs. operational analysisZ500, NH, 2004-07-01 to 2005-04-09
0 1 2 3 4 5 6 7 8 9 10
Forecast Range (days)
20
30
40
50
60
70
80
90
100
Ano
mal
yC
orre
latio
n(%
)
Overlapping 24h 4D-Var 24h 4D-Var (ERA 20C) 12h 4D-Var
• Verification against independent (unused) observations:– confirms positive results with overlapping windows,– shows that 24h 4D-Var without overlap is slightly better than 12h 4D-Var.
Y. Trémolet Overlapping Window 4D-Var 8 / 12
Use of Observations (full system)
• Overlaping assimilation windows implies that observations are used twice.
0.4 0.6 0.8 1.0 1.2 1.4 1.6 1.8 2.0Std. dev.
0
5
10
15
20
25
30
35
40
45
Heig
ht
(km
)
GPSRO North Hemis.
12h 4D-Var24h 4D-Var (12-24h)24h 4D-Var (0-12h)
– The background fit the firsttime observations are seen issimilar to the 12h 4D-Varcontrol
– The analysis fit is equal orbetter than in 12h 4D-Var thesecond time observations areused
– It is possible to fitobservations at least as welleven with more constraints
Y. Trémolet Overlapping Window 4D-Var 9 / 12
Forecast Performance
• For operational use: 12h4D-Var every 6 hours
• Flow dependent B from6-hour EDA
• In practice, re-usingobservations gives betterperformance
• Not re-using observationswould require replacing Bby an analysis errorcovariance matrix
−0.2
−0.1
0.0
0.1
0.2
Diffe
rence in a
nom
aly
corr
ela
tion n
orm
alis
ed b
y 1
−A
C
T+24
−90 −60 −30 0 30 60 90Latitude
1000
700
400
100
10
1
Pre
ssure
, hP
a
T+48
−90 −60 −30 0 30 60 90Latitude
1000
700
400
100
10
1
Pre
ssure
, hP
a
T+72
−90 −60 −30 0 30 60 90Latitude
1000
700
400
100
10
1
Pre
ssure
, hP
a
T+84
−90 −60 −30 0 30 60 90Latitude
1000
700
400
100
10
1
Pre
ssure
, hP
a
T+96
−90 −60 −30 0 30 60 90Latitude
1000
700
400
100
10
1
Pre
ssure
, hP
a
T+120
−90 −60 −30 0 30 60 90Latitude
1000
700
400
100
10
1
Pre
ssure
, hP
a
T+144
−90 −60 −30 0 30 60 90Latitude
1000
700
400
100
10
1
Pre
ssure
, hP
a
T+168
−90 −60 −30 0 30 60 90Latitude
1000
700
400
100
10
1
Pre
ssure
, hP
a
T+192
−90 −60 −30 0 30 60 90Latitude
1000
700
400
100
10
1
Pre
ssure
, hP
a
T+216
−90 −60 −30 0 30 60 90Latitude
1000
700
400
100
10
1
Pre
ssure
, hP
a
New obs. only - obs. re-use, JJA 2014
Y. Trémolet Overlapping Window 4D-Var 10 / 12
Forecast Performance
• For operational use: 12h4D-Var every 6 hours
• Flow dependent B from6-hour EDA
• Without any change in B,comparison betweenoverlapping windows andearly delivery is mixed
• B will be re-tuned for theoverlapping system
−0.2
−0.1
0.0
0.1
0.2
Diffe
rence in a
nom
aly
corr
ela
tion n
orm
alis
ed b
y 1
−A
C
T+24
−90 −60 −30 0 30 60 90Latitude
1000
700
400
100
10
1
Pre
ssure
, hP
a
T+48
−90 −60 −30 0 30 60 90Latitude
1000
700
400
100
10
1
Pre
ssure
, hP
a
T+72
−90 −60 −30 0 30 60 90Latitude
1000
700
400
100
10
1
Pre
ssure
, hP
a
T+84
−90 −60 −30 0 30 60 90Latitude
1000
700
400
100
10
1
Pre
ssure
, hP
a
T+96
−90 −60 −30 0 30 60 90Latitude
1000
700
400
100
10
1
Pre
ssure
, hP
a
T+120
Overlap - Early delivery, JJA 2014
Y. Trémolet Overlapping Window 4D-Var 11 / 12
Background-Observations Errors Correlation
• Results improve when re-using observations
• Other errors in B and R might be (much) larger than the missingcross-correlation term
• Observation errors could be adjusted (Cf. talk Marc Bocquet)?
• The state used as the background is in fact already an analysis– B should be replaced by an analysis error covariance matrix
(the EDA can be modified for this)– Keeping both B and the observations anchors the guess where it has already
been pushed: (Jb + Jo)min acts as a Ja weight– Keeping B but removing the observations at the start of the window creates
an information leak (not enough weight on an accurate prior)
• In the long term, with weak constraint 4D-Var, there is no B, only Q– No issue of cross-correlation– Viewing B as an approximation of Q, ignoring cross correlations is sensible
Y. Trémolet Overlapping Window 4D-Var 12 / 12
Top Related