Proac’ve!Quality!Control!based!on! … · 2014-07-21 · Proac’ve!Quality!Control!based!on!...
Transcript of Proac’ve!Quality!Control!based!on! … · 2014-07-21 · Proac’ve!Quality!Control!based!on!...
![Page 1: Proac’ve!Quality!Control!based!on! … · 2014-07-21 · Proac’ve!Quality!Control!based!on! Ensemble!Forecast!Sensi’vity!to!Observa’ons! Daisuke!Ho=a1,2! EugeniaKalnay 1,](https://reader034.fdocuments.us/reader034/viewer/2022050418/5f8dd9e5d0ef02515e0c295a/html5/thumbnails/1.jpg)
Proac've Quality Control based on
Ensemble Forecast Sensi'vity to Observa'ons
Daisuke Ho=a1,2 Eugenia Kalnay1, Yoichiro Ota2
1 University of Maryland 2 Japan Meteorological Agency
July 16th, 2014 EMC Seminar
1
![Page 2: Proac’ve!Quality!Control!based!on! … · 2014-07-21 · Proac’ve!Quality!Control!based!on! Ensemble!Forecast!Sensi’vity!to!Observa’ons! Daisuke!Ho=a1,2! EugeniaKalnay 1,](https://reader034.fdocuments.us/reader034/viewer/2022050418/5f8dd9e5d0ef02515e0c295a/html5/thumbnails/2.jpg)
Outline
1. EFSO (Ensemble Forecast SensiIvity to ObservaIons)
and “ProacIve QC”
2. EFSR (Ensemble Forecast SensiIvity to ObservaIon Error Covariance matrix R)
and tuning of R
3. Future DirecIons: OperaIonal ApplicaIons
(Appendix: semi-‐implicit Lorenz N-‐cycle scheme) 2
![Page 3: Proac’ve!Quality!Control!based!on! … · 2014-07-21 · Proac’ve!Quality!Control!based!on! Ensemble!Forecast!Sensi’vity!to!Observa’ons! Daisuke!Ho=a1,2! EugeniaKalnay 1,](https://reader034.fdocuments.us/reader034/viewer/2022050418/5f8dd9e5d0ef02515e0c295a/html5/thumbnails/3.jpg)
Part 1: EFSO (Ensemble Forecast SensiIvity to ObservaIons)
and “Proac've QC”
3
![Page 4: Proac’ve!Quality!Control!based!on! … · 2014-07-21 · Proac’ve!Quality!Control!based!on! Ensemble!Forecast!Sensi’vity!to!Observa’ons! Daisuke!Ho=a1,2! EugeniaKalnay 1,](https://reader034.fdocuments.us/reader034/viewer/2022050418/5f8dd9e5d0ef02515e0c295a/html5/thumbnails/4.jpg)
MoIvaIon: The NCEP “forecast skill dropout” problem
From Kumar et al. (2009)
4
• NCEP’s 5-‐day Forecast skill is generally very high (∼ 0.9 level) • However, it occasionally drops to a low level (= “dropout”) • In some cases, all NWP centers suffer. • But in some cases, NCEP does suffer while ECMWF does not.
ECMWF (triangles)
NCEP
![Page 5: Proac’ve!Quality!Control!based!on! … · 2014-07-21 · Proac’ve!Quality!Control!based!on! Ensemble!Forecast!Sensi’vity!to!Observa’ons! Daisuke!Ho=a1,2! EugeniaKalnay 1,](https://reader034.fdocuments.us/reader034/viewer/2022050418/5f8dd9e5d0ef02515e0c295a/html5/thumbnails/5.jpg)
MoIvaIon: The NCEP “forecast skill dropout” problem
From Kumar et al. (2009)
GFS-‐GSI (OperaIonal)
ECMWF
GFS-‐GSI which assimilates “Pseudo-‐Obs” generated from ECMWF analysis
• “Culprit” is not the model but “bad observaIons” (or inability of DA system to properly assimilate them)
à How can we detect those “flawed” observa'ons?
5
![Page 6: Proac’ve!Quality!Control!based!on! … · 2014-07-21 · Proac’ve!Quality!Control!based!on! Ensemble!Forecast!Sensi’vity!to!Observa’ons! Daisuke!Ho=a1,2! EugeniaKalnay 1,](https://reader034.fdocuments.us/reader034/viewer/2022050418/5f8dd9e5d0ef02515e0c295a/html5/thumbnails/6.jpg)
EFSO: Ensemble Forecast Sensi'vity to Observa'ons • QuanIfies how much each
observa'on improved/degraded the forecast
• First invented for a variaIonal DA-‐system using the adjoint method by Langland and Baker (2004)
• Liu and Kalnay (2008) adapted it to LETKF (no adjoint)
• Kalnay et. al (2012) gave an improved, simpler formulaIon
• The new formulaIon is – more accurate – simpler and easier to implement – applicable to any formulaIon of EnKF
• Ota et al. (2013) successfully implemented the new EFSO into the NCEP’s operaIonal GFS system
ReducIon of forecast error by the assimilaIon of obs.
analysis spread in obs. space forecast spread O-‐B of the ens. mean
6
From Kalnay et al (2012)
![Page 7: Proac’ve!Quality!Control!based!on! … · 2014-07-21 · Proac’ve!Quality!Control!based!on! Ensemble!Forecast!Sensi’vity!to!Observa’ons! Daisuke!Ho=a1,2! EugeniaKalnay 1,](https://reader034.fdocuments.us/reader034/viewer/2022050418/5f8dd9e5d0ef02515e0c295a/html5/thumbnails/7.jpg)
• Identified 7 cases of potential “regional forecast skill dropouts”
• Rerun the analyses and forecasts without using “flawed” obs. identified by 24-hour EFSO
• The forecast errors were substantially reduced. 7
Ota et al. (2013): IdenIficaIon of “flawed” observaIons by 24-‐hour EFSO
![Page 8: Proac’ve!Quality!Control!based!on! … · 2014-07-21 · Proac’ve!Quality!Control!based!on! Ensemble!Forecast!Sensi’vity!to!Observa’ons! Daisuke!Ho=a1,2! EugeniaKalnay 1,](https://reader034.fdocuments.us/reader034/viewer/2022050418/5f8dd9e5d0ef02515e0c295a/html5/thumbnails/8.jpg)
“Proac've QC”: Proposed Algorithm
Suppose we wish to idenIfy and delete “flawed” obs. at 00h. ① Run regular DA cycle from -‐06h to 00h. ② Run regular DA cycle from 00h to 06h. ③ Detect “regional dropouts” using the informaIon
available from ① and ②. ④ Perform 6-‐hour EFSO to idenIfy “flawed” obs. at 00h. ⑤ If “flawed” obs. are idenIfied, repeat 00h analysis
without using the detected “flawed” obs.
8
![Page 9: Proac’ve!Quality!Control!based!on! … · 2014-07-21 · Proac’ve!Quality!Control!based!on! Ensemble!Forecast!Sensi’vity!to!Observa’ons! Daisuke!Ho=a1,2! EugeniaKalnay 1,](https://reader034.fdocuments.us/reader034/viewer/2022050418/5f8dd9e5d0ef02515e0c295a/html5/thumbnails/9.jpg)
Key quesIons to be addressed in order for the ProacIve QC to work
• Are 6 hours long enough for detec'ng “flawed observa'ons”? – Forecast errors are computed as Forecast-‐minus-‐Analysis – When compared to the errors of very short-‐term forecast, analysis errors might not be small.
à EsImaIon of forecast errors becomes more difficult. • What is the best criterion for rejec'on of observa'ons?
RejecIng too many observaIons might lead to forecast degradaIon, but too few would make liqle difference. à How to strike the best balance?
• Does rejec'on of those observa'ons really improve analysis and forecast?
9
![Page 10: Proac’ve!Quality!Control!based!on! … · 2014-07-21 · Proac’ve!Quality!Control!based!on! Ensemble!Forecast!Sensi’vity!to!Observa’ons! Daisuke!Ho=a1,2! EugeniaKalnay 1,](https://reader034.fdocuments.us/reader034/viewer/2022050418/5f8dd9e5d0ef02515e0c295a/html5/thumbnails/10.jpg)
Experiments with quasi-‐operaIonal NCEP’s GFS/GSI system Experimental Set-‐up
(Implemented on top of GFS/hybrid GSI ported to JCSDA’s S4 by Dr. Jim Jung)
• Forecast Model: NCEP’s GFS model • Resolu'on: half of the operaIonal:
– T254L64 (determinisIc), T126L64 (ensemble) • DA system: hybrid GSI (as in the operaIonal), but EnKF part replaced by LETKF
• Observa'ons: same as the NCEP operaIonal system • Period: 34 days (Jan – Feb, 2012) • LETKF: Covariance localizaIon and inflaIon (same as the operaIonal) • EFSO:
• Localiza'on: same as LETKF + moving localizaIon of Ota et al. (2013) • Error norm: Moist total energy norm
10
![Page 11: Proac’ve!Quality!Control!based!on! … · 2014-07-21 · Proac’ve!Quality!Control!based!on! Ensemble!Forecast!Sensi’vity!to!Observa’ons! Daisuke!Ho=a1,2! EugeniaKalnay 1,](https://reader034.fdocuments.us/reader034/viewer/2022050418/5f8dd9e5d0ef02515e0c295a/html5/thumbnails/11.jpg)
EFSO’s sensiIvity to forecast lead Ime (1) Time average
• EFSO results are not very sensiIve to the choice of evaluaIon lead Ime.
Lead-‐Time: 6 hrs. Lead-‐Time: 12 hrs. Lead-‐Time: 24 hrs.
Average net observa'on impact for each observa'on type
11
Units: J kg-‐1
![Page 12: Proac’ve!Quality!Control!based!on! … · 2014-07-21 · Proac’ve!Quality!Control!based!on! Ensemble!Forecast!Sensi’vity!to!Observa’ons! Daisuke!Ho=a1,2! EugeniaKalnay 1,](https://reader034.fdocuments.us/reader034/viewer/2022050418/5f8dd9e5d0ef02515e0c295a/html5/thumbnails/12.jpg)
EFSO’s sensiIvity to forecast lead Ime (2) Individual Cases
Example: MODIS wind near the North Pole on Feb 06 18UTC, 2012 Geographical distribuIon of EFSO
from each. obs. Red: nega've impact ; Blue: posi've impact
the size propor'onal to the magnitude
Lead-‐Time: 24 hrs.
• Again, EFSO results are not very sensiIve to the choice of evaluaIon lead Ime, even for individual cases.
• à 6 hours are long enough for detec'on of “flawed” observa'ons.
12
Lead-‐Time: 6 hrs.
![Page 13: Proac’ve!Quality!Control!based!on! … · 2014-07-21 · Proac’ve!Quality!Control!based!on! Ensemble!Forecast!Sensi’vity!to!Observa’ons! Daisuke!Ho=a1,2! EugeniaKalnay 1,](https://reader034.fdocuments.us/reader034/viewer/2022050418/5f8dd9e5d0ef02515e0c295a/html5/thumbnails/13.jpg)
Key quesIons to be answered
• Are 6 hours long enough for detecIng “flawed observaIons”? à Yes. 6-‐hr EFSO is equally capable of detec'ng “flawed” obs. as 24-‐hr EFSO.
• What is the best criterion for rejecIon of observaIons?
• Does rejecIon of those observaIons really improve analysis and forecast?
13
![Page 14: Proac’ve!Quality!Control!based!on! … · 2014-07-21 · Proac’ve!Quality!Control!based!on! Ensemble!Forecast!Sensi’vity!to!Observa’ons! Daisuke!Ho=a1,2! EugeniaKalnay 1,](https://reader034.fdocuments.us/reader034/viewer/2022050418/5f8dd9e5d0ef02515e0c295a/html5/thumbnails/14.jpg)
Units: J kg-‐1
Data Denial Experiments Case study (1): MODIS case
2012-‐Feb-‐06-‐18Z, [60oN—90oN] x [40oE—100oE]
6 hrs.
MODISà
Net EFSO Impact by obs. types measured with moist total energy norm
• MODIS wind idenIfied as “flawed” (i.e., with net negaIve impact). • There are both helpful and harmful observaIons. • How can we decide which / how many obs. should be denied?
ß Improve Degrade à
14
EFSO impact from each MODIS wind observaIon
![Page 15: Proac’ve!Quality!Control!based!on! … · 2014-07-21 · Proac’ve!Quality!Control!based!on! Ensemble!Forecast!Sensi’vity!to!Observa’ons! Daisuke!Ho=a1,2! EugeniaKalnay 1,](https://reader034.fdocuments.us/reader034/viewer/2022050418/5f8dd9e5d0ef02515e0c295a/html5/thumbnails/15.jpg)
DistribuIon of EFSO values from each observaIon Example: MODIS wind near the North Pole
15
allobs
allneg
one-‐sigma
netzero
Rank
EFSO
value
s
reduces fcst. error à posiIve impact
increases fcst. error à negaIve impact
Units: J kg-‐1
Data Denial Experiments SelecIon of the Obs. to be denied
à Try four criteria, perform data denial for each
mean + σ
![Page 16: Proac’ve!Quality!Control!based!on! … · 2014-07-21 · Proac’ve!Quality!Control!based!on! Ensemble!Forecast!Sensi’vity!to!Observa’ons! Daisuke!Ho=a1,2! EugeniaKalnay 1,](https://reader034.fdocuments.us/reader034/viewer/2022050418/5f8dd9e5d0ef02515e0c295a/html5/thumbnails/16.jpg)
How many obs. should we reject? Case study (1): MODIS case
2012-‐Feb-‐06-‐18Z, [60oN—90oN] x [40oE—100oE]
RelaIve 24-‐hr fcst. improvement:= (efbeforeQC – efa_erQC )/ efbeforeQC x 100 [%]
Data selecIon based on 6-‐hour EFSO
16
0%
-‐50%
50% allobs allneg one-‐sigma netzero
• allobs: overall improvement, but with several areas with degradaIon • allneg: enhanced improvement, reduced degradaIon • one-‐sigma & netzero: less improvement, but with further reduced
degradaIon
Improved
Degraded
![Page 17: Proac’ve!Quality!Control!based!on! … · 2014-07-21 · Proac’ve!Quality!Control!based!on! Ensemble!Forecast!Sensi’vity!to!Observa’ons! Daisuke!Ho=a1,2! EugeniaKalnay 1,](https://reader034.fdocuments.us/reader034/viewer/2022050418/5f8dd9e5d0ef02515e0c295a/html5/thumbnails/17.jpg)
Data Denial Experiment Summary of the relaIve 24-‐hr forecast improvement
for 20 cases (20x4x2=160 experiments) Case 6-hour 24-hour
# allobs allnegone-sigma netzero allobs allneg
one-sigma netzero
1
max.imp.max.deg.avg.imp.
12%-9%0.0%
11%-1%0.2%
4%-1%0.1%
5%-1%0.1%
12%-9%0.0%
20%-1%0.3%
0%-1%
-0.0%
6%0%
0.1%
2
max.imp.max.deg.avg.imp.
14%-5%
-0.1%
11%-4%0.3% N/A
4%0%
0.2% N/A
10%-5%0.1% N/A
2%0%
0.1%
3
max.imp.max.deg.avg.imp.
13%-15%0.0%
7%-5%0.2%
2%-1%0.0%
4%-2%0.0%
7%-8%
-0.1%
12%-7%0.1%
0%0%
0.0%
2%-3%
-0.1%
4
max.imp.max.deg.avg.imp.
25%-5%0.6%
27%-5%0.7%
15%-2%0.3%
13%-2%0.2%
3%-6%
-0.3%
4%-3%0.1%
1%-5%
-0.3%
0%0%
-0.0%
5
max.imp.max.deg.avg.imp.
15%-32%-0.2%
19%-81%-0.2%
23%-30%0.2%
22%-13%0.3%
12%-78%-1.3%
10%-21%-0.4%
1%-1%
-0.0%
1%-1%
-0.0%
6
max.imp.max.deg.avg.imp.
9%-9%0.0%
15%-6%0.4%
12%-3%0.3%
3%-1%0.1%
24%-38%0.0%
9%-10%0.1%
2%-2%0.0%
3%-2%0.0%
7
max.imp.max.deg.avg.imp.
17%-9%
-0.0%
13%-5%0.4%
2%-3%0.0%
0%0%
0.0%
19%-36%0.3%
26%-28%0.6%
0%0%
0.0%
4%-1%0.2%
8
max.imp.max.deg.avg.imp.
41%-18%0.9%
41%-14%1.1%
21%-5%0.8%
10%-2%0.4%
41%-18%0.9%
26%-10%1.2%
0%0%
-0.0%
4%-1%0.2%
9
max.imp.max.deg.avg.imp.
7%-21%-0.6%
8%-16%-0.4%
8%-3%0.0%
8%-4%0.1%
3%-2%
-0.1%
5%-1%0.1%
3%-1%0.0%
3%-1%0.0%
10
max.imp.max.deg.avg.imp.
25%-6%1.1%
19%-6%0.7% N/A
6%0%
0.2% N/A
17%-12%0.8% N/A
2%0%
0.2%
Table 6.4: Relative improvement or degradation of 24-hour forecast by the denial ofobservations. The first of the three rows of each case (labeled “max.imp.”) showsthe largest positive value of the local “relative forecast improvement.” The secondrow (labeled “max.deg.”) shows the largest negative value of the local “relativeforecast improvement” (i.e., the largest local “relative forecast degradation”). Thelast row (labeled “avg.imp.”) shows the average forecast improvement evaluated forthe extratropics of the hemisphere in which the target region is located. Continuedon Table 6.5.
132
17
Case 6-hour 24-hour
# allobs allnegone-sigma netzero allobs allneg
one-sigma netzero
11
max.imp.max.deg.avg.imp.
11%-6%0.5%
9%-5%0.3%
2%-2%0.1%
3%0%
0.1%
22%-5%0.9%
15%-6%0.9%
1%0%
0.0%
2%0%
0.2%
12
max.imp.max.deg.avg.imp.
37%-14%0.7%
39%-12%0.7%
19%-2%0.5%
19%-2%0.5%
37%-14%0.7%
38%-19%0.4%
1%0%
0.0%
12%-6%0.2%
13
max.imp.max.deg.avg.imp.
24%-9%1.4%
30%-9%0.8%
18%-10%0.3%
19%-12%0.4%
24%-9%1.3%
26%-10%1.1%
0%0%
0.0%
8%-6%0.1%
14
max.imp.max.deg.avg.imp.
5%0%
0.3%
3%0%
0.1%
1%0%
0.0%
1%0%
0.1%
5%0%
0.3%
3%-1%0.1%
0%0%
0.0%
0%0%
0.0%
15
max.imp.max.deg.avg.imp.
3%-2%0.1%
1%-1%0.1%
1%-1%
-0.0%
1%-1%0.0%
13%-16%-0.1%
35%-18%0.8%
1%-1%0.0%
7%-10%0.2%
16
max.imp.max.deg.avg.imp.
27%-15%1.9%
30%-21%1.8%
23%-4%1.3%
16%-2%0.7%
30%-20%2.1%
33%-43%1.2%
1%-1%0.0%
7%-1%0.3%
17
max.imp.max.deg.avg.imp.
39%-15%0.8%
48%-4%2.1%
26%-2%1.2%
20%-2%0.8%
45%-15%0.7%
51%-8%1.6%
0%-1%
-0.0%
15%-2%0.5%
18
max.imp.max.deg.avg.imp.
46%-9%2.4%
46%-8%2.2%
25%-3%1.0%
21%-2%0.8%
36%-14%1.6%
47%-13%2.1%
0%-1%
-0.0%
20%-4%0.6%
19
max.imp.max.deg.avg.imp.
44%-24%2.2%
37%-10%2.2%
17%-1%1.0%
14%-1%1.0%
6%-17%-0.2%
8%-7%0.2%
0%0%
0.0%
2%-1%0.0%
20
max.imp.max.deg.avg.imp.
12%-3%0.2%
10%-1%0.3%
5%-1%0.2%
3%-1%0.0%
12%-3%0.2% N/A
1%-2%
-0.0%
9%-1%0.2%
Table 6.5: Continued from Table 6.4.
133
• With allneg: • Hemispheric-‐scale forecast error reduced in 18 out of 20 cases. • Local improvement over 30% in 7 cases
![Page 18: Proac’ve!Quality!Control!based!on! … · 2014-07-21 · Proac’ve!Quality!Control!based!on! Ensemble!Forecast!Sensi’vity!to!Observa’ons! Daisuke!Ho=a1,2! EugeniaKalnay 1,](https://reader034.fdocuments.us/reader034/viewer/2022050418/5f8dd9e5d0ef02515e0c295a/html5/thumbnails/18.jpg)
Data Denial Experiment Summary of the results for 20 cases
18
• Data selecIon based on 6-‐hour EFSO: – allobs: improvement mixed with degradaIon – allneg: enhanced improvement, reduced degradaIon
Hemispheric-‐scale forecast error reduced in 18 out of 20 cases. Local improvement over 30% in 7 cases.
– one-‐sigma & netzero: diminished improvement, but with further reduced degradaIon
– For all of the 7 most successful cases, MODIS wind was idenIfied as “flawed.”
• Data selecIon based on 24-‐hour EFSO: Similar to 6-‐hour EFSO, but with less improvements.
![Page 19: Proac’ve!Quality!Control!based!on! … · 2014-07-21 · Proac’ve!Quality!Control!based!on! Ensemble!Forecast!Sensi’vity!to!Observa’ons! Daisuke!Ho=a1,2! EugeniaKalnay 1,](https://reader034.fdocuments.us/reader034/viewer/2022050418/5f8dd9e5d0ef02515e0c295a/html5/thumbnails/19.jpg)
Key quesIons to be answered • Are 6 hours long enough for detecIng “flawed observaIons”? • à Yes. 6-‐hr EFSO is equally capable of detec'ng “flawed”
obs. as 24-‐hr EFSO is. • What is the best criterion for rejecIon of observaIons? • à A ma=er of trade-‐off: if some degrada'on is
tolerable, “allneg” should be favorable; else “one-‐sigma” or “netzero” should be used.
• Does rejecIon of those observaIons really improve analysis and forecast? • à Yes, with >30% local improvement in 7 out of 20
cases.
19
![Page 20: Proac’ve!Quality!Control!based!on! … · 2014-07-21 · Proac’ve!Quality!Control!based!on! Ensemble!Forecast!Sensi’vity!to!Observa’ons! Daisuke!Ho=a1,2! EugeniaKalnay 1,](https://reader034.fdocuments.us/reader034/viewer/2022050418/5f8dd9e5d0ef02515e0c295a/html5/thumbnails/20.jpg)
Summary for Proac@ve QC
• “Flawed” observa'ons that potenIally lead to forecast skill dropouts can be detected by EFSO diagnosIcs a_er only 6 hours from the analysis.
• Proac've QC does improve forecast and analysis. • ProacIve QC is innova've:
– The first fully flow-‐dependent QC – based on whether observaIons actually improve/degrade forecast
20
![Page 21: Proac’ve!Quality!Control!based!on! … · 2014-07-21 · Proac’ve!Quality!Control!based!on! Ensemble!Forecast!Sensi’vity!to!Observa’ons! Daisuke!Ho=a1,2! EugeniaKalnay 1,](https://reader034.fdocuments.us/reader034/viewer/2022050418/5f8dd9e5d0ef02515e0c295a/html5/thumbnails/21.jpg)
Part 2: EFSR (Ensemble Forecast SensiIvity to
ObservaIon Error Covariance matrix R) and Tuning of R
21
![Page 22: Proac’ve!Quality!Control!based!on! … · 2014-07-21 · Proac’ve!Quality!Control!based!on! Ensemble!Forecast!Sensi’vity!to!Observa’ons! Daisuke!Ho=a1,2! EugeniaKalnay 1,](https://reader034.fdocuments.us/reader034/viewer/2022050418/5f8dd9e5d0ef02515e0c295a/html5/thumbnails/22.jpg)
MoIvaIon • Data AssimilaIon combines informaIon from background and observaIons with an “opImal weight.”
• The “opImal weight” is determined based on the background -‐and observaIon-‐ error covariances B and R.
• In EnKF, B (=Pb) is dynamically esImated, but R is sIll an external parameter. – Truth is unknown. à True R is also unknown. – NWP centers specify it empirically and subjecIvely.
• à We need a systemaIc method for tuning R. 22
![Page 23: Proac’ve!Quality!Control!based!on! … · 2014-07-21 · Proac’ve!Quality!Control!based!on! Ensemble!Forecast!Sensi’vity!to!Observa’ons! Daisuke!Ho=a1,2! EugeniaKalnay 1,](https://reader034.fdocuments.us/reader034/viewer/2022050418/5f8dd9e5d0ef02515e0c295a/html5/thumbnails/23.jpg)
EFSR FormulaIon
• Daescu and Langland (2013) proposed an adjoint-‐based formulaIon of forecast sensiIvity to R matrix.
• We can formulate an ensemble version based on EFSO by Kalnay et al. (2012) :
@e
@R
�
ij
⇡ @e
@yi
zj
⇡ � 1
K � 1
hR�1Ya
0XfTt|0C
�et|0 + et|�6
�i
i
⇥R�1�yoa
⇤j
where z is an ”intermediate analysis increment” in observation space
• We know whether fcst. will be improved or degraded by the increase or decrease of R.
à We can opImize R. 23
![Page 24: Proac’ve!Quality!Control!based!on! … · 2014-07-21 · Proac’ve!Quality!Control!based!on! Ensemble!Forecast!Sensi’vity!to!Observa’ons! Daisuke!Ho=a1,2! EugeniaKalnay 1,](https://reader034.fdocuments.us/reader034/viewer/2022050418/5f8dd9e5d0ef02515e0c295a/html5/thumbnails/24.jpg)
EFSR: Experiments • Perfect-‐model experiment with Lorenz ’96 system – Run two DA cycles, one with incorrect R, the other with correct R
– Perform EFSR to the two experiments. Examine if EFSR can detect mis-‐specificaIon of R.
• Real NWP system experiment & Tuning of R – Diagnose forecast R-‐sensiIvity for each observaIon type by EFSR.
– Tune R based on EFSR and run the DA cycle again. Examine if the tuning improves the EFSO impacts of the tuned observaIon types.
24
![Page 25: Proac’ve!Quality!Control!based!on! … · 2014-07-21 · Proac’ve!Quality!Control!based!on! Ensemble!Forecast!Sensi’vity!to!Observa’ons! Daisuke!Ho=a1,2! EugeniaKalnay 1,](https://reader034.fdocuments.us/reader034/viewer/2022050418/5f8dd9e5d0ef02515e0c295a/html5/thumbnails/25.jpg)
Perfect-‐model Experiment: Experimental Setup
• Model: Lorenz ’96 model with N=40 and F=8.0
• DA method: 40 member LETKF, no localizaIon • EFSR: no localizaIon • Observa'ons: available at every grid point. • Specifica'on of R:
25
7.4.1 Lorenz ’96 system
In our toy-system experiments, we use the Lorenz ’96 model as the forecast
model. It is a chaotic low-dimensional ODE system first introduced by Lorenz (1996)
and Lorenz and Emanuel (1998) to address a fundamental question in predictability
and DA study: In a system with insu�cient observations, how can we place a
supplementary observation that optimally improves the predictability of the system?
In the literature of predictability and DA study, this model has been widely adopted
as a benchmark system for new ideas.
The model is an N -dimensional ODE system defined by
dxj
dt= xj (xj+1
� xj�2
)� xj + F, j = 1, · · · , N (7.34)
with a cyclic boundary condition x�1
= xN�1
, x0
= xN and xN+1
= x1
. The first
term mimics the advection (�u ·ru) of fluid mechanical equations; the second and
third terms represent, respectively, dissipation (damping or di↵usion) and external
forcing, hence making the model a nonlinear, forced-dissipative system, a typical
genre of chaotic systems. As in the original study by Lorenz and Emanuel (1998),
for our study, we adopt N = 40 and F = 8.0. Note that, unlike Kalnay et al. (2012)
and Liu and Kalnay (2008), who used di↵erent values of F for the nature run and
DA cycles to account, to some extent, for model errors, we use the same parameter
F = 8.0 for both the nature run and the forecast model (so-called “identical twin”
or “perfect model” setting). The forecast model Eq. (7.34) is integrated by the
151
Name True obs error variance Prescribed error variance
SPIKE �o,truej
2
=
(
0.82 j = 11
0.22 j 6= 11�oj2 = 0.22 everywhere
STAGGERED �o,truej
2
=
(
0.12 j: odd
0.32 j: even�oj2 = 0.22 everywhere
LAND-OCEAN �o,truej
2
=
8
>
<
>
:
0.321 j 20(“land”)
0.1221 j 40(“ocean”)
�oj2 = 0.22 everywhere
Table 7.1: The true and specified observation error variances for the three experi-ments performed using the Lorenz ’96 system.
156
• Erroneous obs. variance only at the 11-‐th grid pt.
• DA system assumes constant R for all grid pts.
Design is inspired by Liu and Kalnay (2008)
![Page 26: Proac’ve!Quality!Control!based!on! … · 2014-07-21 · Proac’ve!Quality!Control!based!on! Ensemble!Forecast!Sensi’vity!to!Observa’ons! Daisuke!Ho=a1,2! EugeniaKalnay 1,](https://reader034.fdocuments.us/reader034/viewer/2022050418/5f8dd9e5d0ef02515e0c295a/html5/thumbnails/26.jpg)
Perfect-‐model Experiment: Result (SPIKE experiment)
24-‐hr. EFSR sensi'vity
-4
-3
-2
-1
0
1
0 5 10 15 20 25 30 35 40
de
f /dσ
o2
Grid Number
Ensemble 24hr-Fcst Sensitivity to R EFSR-REUSE SPIKE
Mis-specified σo2
Correct σo2
-14
-12
-10
-8
-6
-4
-2
0
0 5 10 15 20 25 30 35 40
de
f /dσ
o2
Grid Number
Ensemble 24hr-Fcst Sensitivity to R EFSR-NEW SPIKE
Mis-specified σo2
Correct σo2
Figure 7.2: The forecast sensitivity gradient to observation error variances@ef
t|0@�o
j
2
for the SPIKE experiment estimated using (top) “EFSR-REUSE” and (bottom)“EFSR-NEW” formulations. As in Figure 7.1, the blue and red lines represent, re-spectively, the DA runs with correctly specified observation error variances (correct-R run) and that with mis-specified observation error variances (incorrect-R run).As in Figure 7.1, the sensitivity gradients are averaged over the last 9 years of eachDA runs.
159
26
• For “incorrect-‐R,” EFSR detects the mis-‐specificaIon of R at the 11th grid point. à We can detect mis-‐specified R
• For “correct-‐R,” EFSR diagnoses almost-‐zero sensiIvity. à No “false alerts”
correct-‐R incorrect-‐R
NegaIve sensiIvity: forecast error can be reduced by increasing R à R is too small
![Page 27: Proac’ve!Quality!Control!based!on! … · 2014-07-21 · Proac’ve!Quality!Control!based!on! Ensemble!Forecast!Sensi’vity!to!Observa’ons! Daisuke!Ho=a1,2! EugeniaKalnay 1,](https://reader034.fdocuments.us/reader034/viewer/2022050418/5f8dd9e5d0ef02515e0c295a/html5/thumbnails/27.jpg)
EFSR for GFS / GSI-‐LETKF hybrid
Radiosonde à
MODIS wind à
AMSU-‐A à
Aircra| à
• Aircra_, Radiosonde and AMSU-‐A: large posiIve sensiIvity • MODIS wind : negaIve sensiIvity • à Tuning experiment:
• Aircra|, Radiosonde and AMSU-‐A: reduce R by 0.9 • MODIS wind: increase R by 1.1
27
R is too large (à should reduce R)
Units: J kg-‐1
![Page 28: Proac’ve!Quality!Control!based!on! … · 2014-07-21 · Proac’ve!Quality!Control!based!on! Ensemble!Forecast!Sensi’vity!to!Observa’ons! Daisuke!Ho=a1,2! EugeniaKalnay 1,](https://reader034.fdocuments.us/reader034/viewer/2022050418/5f8dd9e5d0ef02515e0c295a/html5/thumbnails/28.jpg)
Tuning Experiment: Result EFSO before/a_er tuning of R
Radiosonde
MODIS windà
Aircra_ à
Why no improvement in MODIS? • MODIS had “flawed” obs. along with “helpful” obs. • The “flawed” obs. might have resulted in incorrect esImaIon of EFSR.
AMSU-‐A à
28
• Aircra|, Radiosonde, AMSU-‐A: • significant improvement
of EFSO-‐impact (as expected)
• MODIS wind : • No improvement in EFSO
(contrary to expectaIon)
Units: J kg-‐1
![Page 29: Proac’ve!Quality!Control!based!on! … · 2014-07-21 · Proac’ve!Quality!Control!based!on! Ensemble!Forecast!Sensi’vity!to!Observa’ons! Daisuke!Ho=a1,2! EugeniaKalnay 1,](https://reader034.fdocuments.us/reader034/viewer/2022050418/5f8dd9e5d0ef02515e0c295a/html5/thumbnails/29.jpg)
Excluding cases where MODIS wind had negaIve impact
29
Figure 8.3: The EFSO impacts from MODIS wind observations for 6-hour lead-timemeasured with the moist total energy norm shown as a time series. The verificationis against the control GSI analysis. Note that positive values correspond to negativeimpacts because they increase the forecast errors.
185
Forecast Sensitivity to observation error variance scaling factor
6 hours
a)
c)
24 hours
b)
d)
Figure 8.4: As in Figure 8.1, but the average is taken without using cases in whichthe 6-hour EFSO impacts from MODIS wind measured with the moist total energynorm was negative.
186
ßRadiosonde à
ß MODIS wind à
ßAMSU-‐Aà
ß Aircra| à
• MODIS wind exhibited several negaIvely-‐impacIng cases.
• Exclude nega've cases • à EFSR for MODIS becomes
neutral • à Consistent with the result
of tuning experiment
Lesson: • Before performing
EFSR, we should remove “bad” obs.
Excluding “flawed” MODIS case
Including “flawed” MODIS case
Units: J kg-‐1 Units: J kg-‐1
Posi've Impact
Nega've Impact
![Page 30: Proac’ve!Quality!Control!based!on! … · 2014-07-21 · Proac’ve!Quality!Control!based!on! Ensemble!Forecast!Sensi’vity!to!Observa’ons! Daisuke!Ho=a1,2! EugeniaKalnay 1,](https://reader034.fdocuments.us/reader034/viewer/2022050418/5f8dd9e5d0ef02515e0c295a/html5/thumbnails/30.jpg)
Summary for EFSR
• EFSR gives informaIon on whether we should increase/reduce prescribed observaIon error covariance R.
• Tuning of R based on this diagnosIcs improves the EFSO.
• à EFSR can be used to systema:cally opImize R.
30
![Page 31: Proac’ve!Quality!Control!based!on! … · 2014-07-21 · Proac’ve!Quality!Control!based!on! Ensemble!Forecast!Sensi’vity!to!Observa’ons! Daisuke!Ho=a1,2! EugeniaKalnay 1,](https://reader034.fdocuments.us/reader034/viewer/2022050418/5f8dd9e5d0ef02515e0c295a/html5/thumbnails/31.jpg)
Future Direc'ons:
31
![Page 32: Proac’ve!Quality!Control!based!on! … · 2014-07-21 · Proac’ve!Quality!Control!based!on! Ensemble!Forecast!Sensi’vity!to!Observa’ons! Daisuke!Ho=a1,2! EugeniaKalnay 1,](https://reader034.fdocuments.us/reader034/viewer/2022050418/5f8dd9e5d0ef02515e0c295a/html5/thumbnails/32.jpg)
Future plans Immediate future: • ImplementaIon of ProacIve QC into the real operaIonal system – Can the opera'onal system
1. wait for 6 hours? 2. afford to do analysis again?
Long-‐term Future DirecIons: • ApplicaIons of EFSO and EFSR
– CollaboraIon with instrument developers – AcceleraIon of development for assimilaIon of new observing systems
32
![Page 33: Proac’ve!Quality!Control!based!on! … · 2014-07-21 · Proac’ve!Quality!Control!based!on! Ensemble!Forecast!Sensi’vity!to!Observa’ons! Daisuke!Ho=a1,2! EugeniaKalnay 1,](https://reader034.fdocuments.us/reader034/viewer/2022050418/5f8dd9e5d0ef02515e0c295a/html5/thumbnails/33.jpg)
ImplementaIon to the real operaIonal system (1) Can we wait for 6 hours?
Idea: Exploit the Ime lag between “early analysis” and “cycle (final) analysis” (suggested by Dr. John Derber, 2013)
33
Cycle anal. 00 UTC
Cycle anal. 06 UTC
Cycle anal. 12 UTC
Cycle anal. 18 UTC
Early anal. 00 UTC
Early anal. 12 UTC
Early anal. 18 UTC
Time
Early anal. 06 UTC
cycle (final) analysis: maintains analysis-‐forecast cycle early analysis: provides iniIal condiIon for extended forecast
![Page 34: Proac’ve!Quality!Control!based!on! … · 2014-07-21 · Proac’ve!Quality!Control!based!on! Ensemble!Forecast!Sensi’vity!to!Observa’ons! Daisuke!Ho=a1,2! EugeniaKalnay 1,](https://reader034.fdocuments.us/reader034/viewer/2022050418/5f8dd9e5d0ef02515e0c295a/html5/thumbnails/34.jpg)
ImplementaIon to the real operaIonal system (1) we don’t need to wait 6 hours!
34
Cycle anal. 00 UTC
Cycle anal. 06 UTC
Cycle anal. 12 UTC
Cycle anal. 18 UTC
Early anal. 00 UTC
Early anal. 06 UTC
Early anal. 12 UTC
Early anal. 18 UTC
PQC
PQC
Time
Idea: Exploit the Ime lag between “early analysis” and “cycle (final) analysis” (suggested by Dr. John Derber, 2013)
cycle (final) analysis: maintains analysis-‐forecast cycle early analysis: provides iniIal condiIon for extended forecast
![Page 35: Proac’ve!Quality!Control!based!on! … · 2014-07-21 · Proac’ve!Quality!Control!based!on! Ensemble!Forecast!Sensi’vity!to!Observa’ons! Daisuke!Ho=a1,2! EugeniaKalnay 1,](https://reader034.fdocuments.us/reader034/viewer/2022050418/5f8dd9e5d0ef02515e0c295a/html5/thumbnails/35.jpg)
ImplementaIon to the real operaIonal system (2) can we afford to do analysis twice?
Idea: Use approximated analysis rather than doing analysis again:
– Using the approximaIon to Kalman gain:
the change in analysis by the denial of observaIons can be approximated by: – As inexpensive as EFSO.
à No need to repeat analysis à Can minimize the Ime delay
35
l = 3, · · · , p. Assume further that the analysis obtained by not using the denied
observations can be approximated by the analysis obtained when those observations
coincide with the corresponding background (i.e., the innovation is zero). Let
xa,deny0
be the analysis that would be obtained without using the denied observations.
Then the analysis equation for xa,deny0
can be written as
xa,deny0
⇡ K⇣
�yob � �yob,deny0
⌘
(2.22)
Thus, from Eq. (2.1) and the approximate analysis equation Eq. (2.5), the change in
analysis that would occur by not assimilating denied observations can be estimated
by
xa,deny0
� xa0
⇡ �K�yob,deny0
(2.23)
⇡ � 1
K � 1Xa
0
Ya0
TR�1�yob,deny0
(2.24)
Similarly, by applying tangent linear approximation to the above equation, the
change in forecast that would occur by not assimilating denied observations can
be estimated by
xf,denyt|0 � xf
t|0 ⇡ Mt|0
⇣
xa,deny0
� xa0
⌘
⇡ �Mt|0K�yob,deny0
(2.25)
⇡ � 1
K � 1Mt|0X
a0
Ya0
TR�1�yob,deny0
(2.26)
⇡ � 1
K � 1Xf
t|0Ya0
TR�1�yob,deny0
(2.27)
35
l = 3, · · · , p. Assume further that the analysis obtained by not using the denied
observations can be approximated by the analysis obtained when those observations
coincide with the corresponding background (i.e., the innovation is zero). Let
xa,deny0
be the analysis that would be obtained without using the denied observations.
Then the analysis equation for xa,deny0
can be written as
xa,deny0
⇡ K⇣
�yob � �yob,deny0
⌘
(2.22)
Thus, from Eq. (2.1) and the approximate analysis equation Eq. (2.5), the change in
analysis that would occur by not assimilating denied observations can be estimated
by
xa,deny0
� xa0
⇡ �K�yob,deny0
(2.23)
⇡ � 1
K � 1Xa
0
Ya0
TR�1�yob,deny0
(2.24)
Similarly, by applying tangent linear approximation to the above equation, the
change in forecast that would occur by not assimilating denied observations can
be estimated by
xf,denyt|0 � xf
t|0 ⇡ Mt|0
⇣
xa,deny0
� xa0
⌘
⇡ �Mt|0K�yob,deny0
(2.25)
⇡ � 1
K � 1Mt|0X
a0
Ya0
TR�1�yob,deny0
(2.26)
⇡ � 1
K � 1Xf
t|0Ya0
TR�1�yob,deny0
(2.27)
35
covariance Pa0
, as:
K = Pa0
HTR�1 (2.2)
In EnKF, the analysis error covariance Pa0
is approximated by the sampled covari-
ance of analysis perturbation Xa0
, giving:
K ⇡ 1
K � 1Xa
0
Xa0
THTR�1 (2.3)
⇡ 1
K � 1Xa
0
Ya0
TR�1 (2.4)
where an approximation HXa0
⇡ Ya0
(ensemble perturbation of analysis at time 0 in
observation space) is used. As we will see later, this approximation turns out to be
extremely powerful and plays a crucial role in our EFSO and EFSR derivation (see
the next subsection and Section 7.3). Note that, in practical situations where the
ensemble size K is smaller than the number of degrees of freedom of the system, the
sampled covariance 1
K�1
Xa0
Xa0
T must be localized to avoid sampling error. Using
the above approximation, the analysis equation Eq. (2.1) can be approximated by:
xa0
� xb0
⇡ 1
K � 1Xa
0
Ya0
TR�1�yob0
(2.5)
2.2.2.3 Derivation of EFSO formulation
Now we proceed to deriving the EFSO formulation. The formulation presented
here is identical to that of Kalnay et al. (2012), except that they assumed the
31
covariance Pa0
, as:
K = Pa0
HTR�1 (2.2)
In EnKF, the analysis error covariance Pa0
is approximated by the sampled covari-
ance of analysis perturbation Xa0
, giving:
K ⇡ 1
K � 1Xa
0
Xa0
THTR�1 (2.3)
⇡ 1
K � 1Xa
0
Ya0
TR�1 (2.4)
where an approximation HXa0
⇡ Ya0
(ensemble perturbation of analysis at time 0 in
observation space) is used. As we will see later, this approximation turns out to be
extremely powerful and plays a crucial role in our EFSO and EFSR derivation (see
the next subsection and Section 7.3). Note that, in practical situations where the
ensemble size K is smaller than the number of degrees of freedom of the system, the
sampled covariance 1
K�1
Xa0
Xa0
T must be localized to avoid sampling error. Using
the above approximation, the analysis equation Eq. (2.1) can be approximated by:
xa0
� xb0
⇡ 1
K � 1Xa
0
Ya0
TR�1�yob0
(2.5)
2.2.2.3 Derivation of EFSO formulation
Now we proceed to deriving the EFSO formulation. The formulation presented
here is identical to that of Kalnay et al. (2012), except that they assumed the
31
![Page 36: Proac’ve!Quality!Control!based!on! … · 2014-07-21 · Proac’ve!Quality!Control!based!on! Ensemble!Forecast!Sensi’vity!to!Observa’ons! Daisuke!Ho=a1,2! EugeniaKalnay 1,](https://reader034.fdocuments.us/reader034/viewer/2022050418/5f8dd9e5d0ef02515e0c295a/html5/thumbnails/36.jpg)
Application of EFSO: (1) Collaboration with instrument developers
• In our experiments, in several cases, MODIS wind showed large negaIve impacts that caused “regional dropouts.”
• à EFSO can be used to build a database of “flawed” observaIons along with their relevant metadata. – Provide such database to instrument developers so that they can fix problems with their algorithms.
– Collaborate with instrument developers to determine which metadata would be helpful to them.
36
![Page 37: Proac’ve!Quality!Control!based!on! … · 2014-07-21 · Proac’ve!Quality!Control!based!on! Ensemble!Forecast!Sensi’vity!to!Observa’ons! Daisuke!Ho=a1,2! EugeniaKalnay 1,](https://reader034.fdocuments.us/reader034/viewer/2022050418/5f8dd9e5d0ef02515e0c295a/html5/thumbnails/37.jpg)
Application of EFSO: (2) Acceleration of development
for assimilation of new observing systems • TradiIonal approach: compare
– Test: with a new observing system – Control: without a new observing system
• Difficulty with this approach: – Signals from a new observing system is obscured by the many
observaIons that are already assimilated in Control. à Hard to obtain staIsIcally significant results
à EFSO-‐based data selec'on will enable efficient determina'on of an op'mal way to assimilate new observing systems.
• An opImal specificaIon of R is also a difficult issue for assimilaIon of new observing systems.
à Our EFSR diagnos'cs should provide useful guidance.
37
![Page 38: Proac’ve!Quality!Control!based!on! … · 2014-07-21 · Proac’ve!Quality!Control!based!on! Ensemble!Forecast!Sensi’vity!to!Observa’ons! Daisuke!Ho=a1,2! EugeniaKalnay 1,](https://reader034.fdocuments.us/reader034/viewer/2022050418/5f8dd9e5d0ef02515e0c295a/html5/thumbnails/38.jpg)
Conclusions • 6-‐hour EFSO can successfully idenIfy “flawed”
observaIons. • RejecIng them (Proac've QC) does improve the analysis
and forecast. • Database of “flawed” obs. can help instrument developers
to improve their algorithms.
• Proac've QC can readily be implemented into the operaIonal system.
• EFSR enables systemaIc tuning of R matrix.
• EFSO and EFSR together can accelerate development of the assimilaIon of new observing systems.
38
![Page 39: Proac’ve!Quality!Control!based!on! … · 2014-07-21 · Proac’ve!Quality!Control!based!on! Ensemble!Forecast!Sensi’vity!to!Observa’ons! Daisuke!Ho=a1,2! EugeniaKalnay 1,](https://reader034.fdocuments.us/reader034/viewer/2022050418/5f8dd9e5d0ef02515e0c295a/html5/thumbnails/39.jpg)
Acknowledgements • My advisor Prof. Eugenia Kalnay for advice and encouragement • Prof. Takemasa Miyoshi of RIKEN/AICS and University of Maryland for
guidance and help • JCSDA and its Director Dr. Sid Boukabara for access to the S4
supercomputer • Dr. Jim Jung of NCEP for kind guidance on S4 • JPSS program for support • Mr. Yoichiro Ota for EFSO and GFS-‐LETKF code • Government of Japan, in parIcular Japan Meteorological Agency (JMA)
for funding and support
• Dr. Jeff Whitaker, Dr. Sajal Kar and Dr. Jim Purser for discussion and advice (on the Appendix)
• … and many others and all of you.
39
THANK YOU!