Highlights from Operational Highlights from Operational Verification in COSMOVerification in COSMO
WG5 COSMO General Meeting, Rome 2011
Authors: ALL
Presented by Adriano Raspanti
WG5 COSMO General Meeting, Rome 2011
• Quick look to some common plots
• Intercomparison between driving model and high resolution
• Conditional verification
• Fuzzy verification
• Long term trends (mainly precipitation)
TEMPERATURE AT 2 M - SON 2010 - MAM 2011
MEAN SEA LEVEL PRESSURE - SON 2010 – MAM 2011
Grafikinhalte
BC from GME CEU,CPL,CRU !!!
BC from IFS C7,CI7,CGR !!!
BC from GME ???
BC from IFS ???
WIND SPEED AT 10 M - SON 2010 - MAM 2011
TOTAL CLOUD COVER - SON 2010 - MAM 2011
WG5 COSMO General Meeting, Rome 2011
• Quick look to some common plots
• Intercomparison between driving model and high resolution
• Conditional verification
• Fuzzy verification
• Long term trends (mainly precipitation)
COSMO General Meeting – Roma 05-09 Sept 2011
COSMOME vs ECMWF Temperature
SON
MAM
DJF
JJA
COSMO General Meeting – Roma 05-09 Sept 2011
COSMOME vs ECMWF Wind Speed
SON
JJA
MAM
DJF
COSMO General Meeting – Roma 05-09 Sept 2011
COSMOI7 vs ECMWF Temperature
SON
JJA
MAM
DJF
COSMO General Meeting – Roma 05-09 Sept 2011
COSMOI7 vs ECMWF Wind Speed
SON
JJA
MAM
DJF
COSMO General Meeting – Roma 05-09 Sept 2011
TemperatureCOSMOME vs COSMOIT
SON
JJA
MAM
DJF
COSMO General Meeting – Roma 05-09 Sept 2011
Wind SpeedCOSMOME vs COSMOIT
SON
JJA
MAM
DJF
Temp 2m - 7km vs 3kmTemp 2m - 7km vs 3km
WG5 COSMO General Meeting, Rome 2011
Fall Winter
Spring Summer
Underestimation of Temp, mainly in winter. error ~2o, worse with 7km by ~0.5o
Clear diurnal cycle
Wind Speed - 7km vs 3kmWind Speed - 7km vs 3km
Fall Winter
Spring Summer
Overestimation of wind (DJF,SON)2-2.5deg bias
similar attitude of 2 models
WG5 COSMO General Meeting, Rome 2011
17 Verification results at MeteoSwiss in 2011
COSMO GM / WG5 Parallel Session, 05.09.2011
Precipitation (12h-sums +36 to +48h):Spring 2011 over Switzerland (SYNOP‘s)frequency bias: COSMO-7 & IFS
observed frequency
V. Stauch
18 Verification results at MeteoSwiss in 2011
COSMO GM / WG5 Parallel Session, 05.09.2011
Precipitation (12h-sums +12 to +24h):Spring 2011 over Switzerland (SYNOP‘s)COSMO-7 & COSMO-2
V. Stauch
for both models mean over 9 gridpoints foreach station
T2m COSMO-RU 2.2 and 7 km, Sochi, station Krasnaya Polyana
19
2.2 km –Less overestimating
7 km
T2m in COSMO-RU 7 and 2.2 km, Krasnaya Polyana
20
Method: 1) nearest point 3D optimized !
COSMO-RU 2.2 km is better than COSMO-RU 7 km for Krasnaya Polyana
T2m in COSMO-RU 7 and 2.2 km, Moscow
21
Method: 1) nearest point 3D optimized !
COSMO-RU 2.2 km RMSE is even slightly higher than that of COSMO-RU 7 km for Moscow
COSMO General Meeting – Roma 05-09 Sept 2011
PERFORMANCE DIAGRAM
Period March 2010 - April 2011
COSMO General Meeting – Roma 05-09 Sept 2011
50% of points (median) > 1 mm/24h
COSMO General Meeting – Roma 05-09 Sept 2011
50% of points (median) > 5 mm/24h
COSMO General Meeting – Roma 05-09 Sept 2011
50% of points (median) > 10 mm/24h
COSMO General Meeting – Roma 05-09 Sept 2011
50% of points (median) > 20 mm/24h
COSMO General Meeting – Roma 05-09 Sept 2011
1 Point (maximum) > 1 mm/24h
COSMO General Meeting – Roma 05-09 Sept 2011
1 Point (maximum) > 5 mm/24h
COSMO General Meeting – Roma 05-09 Sept 2011
1 Point (maximum) > 10 mm/24h
COSMO General Meeting – Roma 05-09 Sept 2011
1 Point (maximum) > 20 mm/24h
COSMO General Meeting – Roma 05-09 Sept 2011
50% of points (median) > 1 mm/24h&
Maximum > 25 mm/24h
COSMO General Meeting – Roma 05-09 Sept 2011
50% of points (median) > 1 mm/24h&
Maximum > 50 mm/24h
COSMO General Meeting – Roma 05-09 Sept 2011
50% of points (median) > 1 mm/24h&
Maximum > 75 mm/24h
COSMO General Meeting – Roma 05-09 Sept 2011
50% of points (median) > 1 mm/24h&
Maximum > 100 mm/24h
COSMO General Meeting – Roma 05-09 Sept 2011
50% of points (median) > 5 mm/24h&
Maximum > 25 mm/24h
COSMO General Meeting – Roma 05-09 Sept 2011
50% of points (median) > 5 mm/24h&
Maximum > 50 mm/24h
COSMO General Meeting – Roma 05-09 Sept 2011
50% of points (median) > 5 mm/24h&
Maximum > 75 mm/24h
COSMO General Meeting – Roma 05-09 Sept 2011
50% of points (median) > 5 mm/24h&
Maximum > 100 mm/24h
COSMO General Meeting – Roma 05-09 Sept 2011
50% of points (median) > 10 mm/24h&
Maximum > 25 mm/24h
COSMO General Meeting – Roma 05-09 Sept 2011
50% of points (median) > 10 mm/24h&
Maximum > 50 mm/24h
COSMO General Meeting – Roma 05-09 Sept 2011
50% of points (median) > 10 mm/24h&
Maximum > 75 mm/24h
COSMO General Meeting – Roma 05-09 Sept 2011
50% of points (median) > 10 mm/24h&
Maximum > 100 mm/24h
COSMO General Meeting – Roma 05-09 Sept 2011
50% of points (median) > 20 mm/24h&
Maximum > 25 mm/24h
COSMO General Meeting – Roma 05-09 Sept 2011
50% of points (median) > 20 mm/24h&
Maximum > 50 mm/24h
COSMO General Meeting – Roma 05-09 Sept 2011
50% of points (median) > 20 mm/24h&
Maximum > 75 mm/24h
COSMO General Meeting – Roma 05-09 Sept 2011
50% of points (median) > 20 mm/24h&
Maximum > 75 mm/24h
WG5 COSMO General Meeting, Rome 2011
• Quick look to some common plots
• Intercomparison between driving model and high resolution
• Conditional verification
• Fuzzy verification
• Long term trends (mainly precipitation)
COSMO General Meeting – Roma 05-09 Sept 2011
Conditional Verification Temp – TCC obs <=25%
SON
MAM
DJF
JJA
Better behaviour for all the seasonsCompare to no condition model
COSMO General Meeting – Roma 05-09 Sept 2011
Conditional Verification Temp – TCC obs >=75%&Wind Speed (obs) <=2 m/s
SON
MAM
DJF
JJA
Similar. Differences in bias
COSMO General Meeting – Roma 05-09 Sept 2011 COSMO General Meeting – Roma 05-09 Sept 2011
Conditional Verification Temp – TCC obs <=25%&Wind Speed (obs) <=2
m/s
SON
MAM
DJF DJF
JJA
Similar. Differences in bias
2mT 2mT vsvs 2mT in overcast conditions2mT in overcast conditions 2mT overcast/no wind2mT overcast/no wind
Fall Winter
Spring Summer
2mT 2mT vsvs 2mT in skyclear conditions2mT in skyclear conditions 2mT skyclear/no wind2mT skyclear/no wind
Fall Winter
Spring Summer
CAPE>50
CAPE<50
Very high POD values for unstable conditions, FAR not so different
WG5 COSMO General Meeting, Rome 2011
WS - Z0>0.25
WS - Z0<0.25
Wind Speed with respect to Roughness Length
Strong underestimation of wind in positions with small roughness length and increased error
WG5 COSMO General Meeting, Rome 2011
Td, 2mT – dry soil
Td, 2mT – wet soil
W_SO Water content of first soil layer(kg/m2) 1cm.
Td: Higher error in dry soil and larger underestimation2mT: Higher error in wet soil and larger understimation
WG5 COSMO General Meeting, Rome 2011
• Quick look to some common plots
• Intercomparison between driving model and high resolution
• Conditional verification
• Fuzzy verification
• Long term trends (mainly precipitation)
57 Verification results at MeteoSwiss in 2011
COSMO GM / WG5 Parallel Session, 05.09.2011
results for 20103h accumulated precipitation sumsover the domain of the swiss radar composit
models: COSMO-2 and COSMO-7for all 8 daily forecast runs, precipitation sums from +3 to +6h
observation precipitation estimates of the swiss radar composit
in case of a missing value, the full date will not be evaluated
Neighborhood verification for precipitation
58 Verification results at MeteoSwiss in 2011
COSMO GM / WG5 Parallel Session, 05.09.2011
FSS, COSMO-2 minus COSMO-7, 2010
Winter Spring
AutumnSummer
numbers = FSS-Score of COSMO-2
colours = differences COSMO-2 minus COSMO-7
COSMO-2 better
COSMO-7 better
• COSMO-2: similar skill in all seasons
• COSMO-2 better than COSMO-7 on almost all scales
• COSMO-2 better than COSMO-7 especially in Winter and Summer
Tanja Weusthoff
59 Verification results at MeteoSwiss in 2011
COSMO GM / WG5 Parallel Session, 05.09.2011
3. „Fuzzy in Time“
• Extension of the spatial window with a window in time volume (dx * dy * dt)
• Evaluation of the forecasts in this volume• Time-window ntm = [1h,3h,5h,7h,9h]
dt
dx
dy
T. Weusthoff
60 Verification results at MeteoSwiss in 2011
COSMO GM / WG5 Parallel Session, 05.09.2011
dxdy
dx
dymodel
observation
e.g. FSS
fraction blue pxiels model = 12/75 (dt=3), 5/25 (dt=1)
fraction blue pxiels obs = 12/75 (dt=3), 3/25 (dt=1)
t0
t0-1
t0+1
t0
t0-1
t0+1
T. Weusthoff
61 Verification results at MeteoSwiss in 2011
COSMO GM / WG5 Parallel Session, 05.09.2011
00 01 02 03 04 05 06 07 08 09 10 11 12 13 1420 21 22 23
00-02
01-03
02-04
03-05
04-06
05-07
…
23-01
22-00
21-23
20-22
hourly accumulated precipitation
Evaluation of 3h accumulations, time window ntm = 5 (t0 +-2h)
00 UTC 03 UTC 06 UTC 08 UTC
22 UTC 02 UTC
01 UTC 05 UTC
3 hourly accumulated precipitation
Evaluation of 3h accumulations, time window ntm = 1
09 UTC00 UTC 03 UTC 06 UTC
23 UTC 01 UTC 02 UTC 04 UTC
T. Weusthoff
62 Verification results at MeteoSwiss in 2011
COSMO GM / WG5 Parallel Session, 05.09.2011
FSS for different time-windowsCOSMO-2, July 2010
T. Weusthoff
63 Verification results at MeteoSwiss in 2011
COSMO GM / WG5 Parallel Session, 05.09.2011 T. Weusthoff
FSS for different time-windowsCOSMO-7, July 2010
64 Verification results at MeteoSwiss in 2011
COSMO GM / WG5 Parallel Session, 05.09.2011
FSS for different time-windows COSMO-2 minus COSMO-7, July 2010
T. Weusthoff
65 Verification results at MeteoSwiss in 2011
COSMO GM / WG5 Parallel Session, 05.09.2011
Summary „fuzzy in time“
• FSS increases on all scales with increasing time-window• greatest effect for small spatical scales• lowest effect for high threshods
• Both models show a similar increase• difference COSMO-2 minus COSMO-7 stays equal, resp.
becomes littler for high time-windows
• For Upscaling the influence of a time tolerance is relatively low und restricted on low thresholds ( effect of the avergaing)
Application of time-windows on the gridscale would make sense; simultaneous application with space tolerance brings no great change
T. Weusthoff
Ulrich Damrath: Fuzzy and standard verification for COSMO-EU and COSMO-DE, COSMO GM, Rome September 2011
Configuration of precipitation verification with FUZZY-methods
Up to May 2011: Observation data: Radar data prepared by assimilation scheme
Model data: GME-, CEU- and CDE-GRIBS interpolated to CDE-grid (nearest gridpoint)
Run: 00 UTC
Forecast times: GME, CEU: 06-18, 06-30, CDE: 06-18 hours
Verification area: part of CDE that is covered by radar data
Since May 2011: Observation data as before, modell data: CEU- and CDE-GRIBS interpolated to CDE-grid
(nearest gridpoint)
Run: 00, 03, 06, 09, 12, 15, 18, 21
Forecast times: 01-04, 03-06, 06-12, 12-15, 15-18, 18-21 hours
Verification aread : CDE, Northern part of Germany, Southern part of Germany, North-Western part of Germany, North-Eastern part of Germany , South-Western part of Germany, South-Eastern part of Germany
Ulrich Damrath: Fuzzy and standard verification for COSMO-EU and COSMO-DE, COSMO GM, Rome September 2011
Application of Fuzzy-methods
Calculation of all Fuzzy-scores with the IDL-Program by Beth Ebert.
Monthly evaluation of data for Fractions Skill Score and Upscaling ETS
Generation of results for 8 (forecast runs)
* 7 (forecast intervals)
* 3 (2 models and one difference)
* 7 (regions)
* 2 (scores)
---------------------------------------------2352 Plots per time interval
Necessity to have a fast access to the data
Ulrich Damrath: Fuzzy and standard verification for COSMO-EU and COSMO-DE, COSMO GM, Rome September 2011
Some examples:ETS upscaling July 2011, Run: 00 UTC, forecast time 01-04 hours
Ulrich Damrath: Fuzzy and standard verification for COSMO-EU and COSMO-DE, COSMO GM, Rome September 2011
Some examples :FSS July 2011, Run: 00 UTC, forecast time 01-04 hours
Ulrich Damrath: Fuzzy and standard verification for COSMO-EU and COSMO-DE, COSMO GM, Rome September 2011
Some examples :ETS upscaling July 2011, Run: 00 UTC, forecast time 12-15 hours
Ulrich Damrath: Fuzzy and standard verification for COSMO-EU and COSMO-DE, COSMO GM, Rome September 2011
Some examples :FSS July 2011, Run: 00 UTC, forecast time 12-15 hours
Ulrich Damrath: Fuzzy and standard verification for COSMO-EU and COSMO-DE, COSMO GM, Rome September 2011
Some examples :ETS upscaling July 2011, Run: 00 UTC, forecast time 18-21 hours
Ulrich Damrath: Fuzzy and standard verification for COSMO-EU and COSMO-DE, COSMO GM, Rome September 2011
Some examples :FSS July 2011, Run: 00 UTC, forecast time 18-21 hours
WG5 COSMO General Meeting, Rome 2011
• Quick look to some common plots
• Intercomparison between driving model and high resolution
• Conditional verification
• Fuzzy verification
• Long term trends (mainly precipitation)
Ulrich Damrath: Fuzzy and standard verification for COSMO-EU and COSMO-DE, COSMO GM, Rome September 2011
COSI @ DWD
Ulrich Damrath: Fuzzy and standard verification for COSMO-EU and COSMO-DE, COSMO GM, Rome September 2011
COSI @ DWD
Ulrich Damrath: Fuzzy and standard verification for COSMO-EU and COSMO-DE, COSMO GM, Rome September 2011
COSI @ DWD
Ulrich Damrath: Fuzzy and standard verification for COSMO-EU and COSMO-DE, COSMO GM, Rome September 2011
COSI @ DWD
Seasonal trend 0.2mm/24h + ECMWF
All the versions present a seasonal cycle with an overestimation during summertime (except COSMO-7 and I2) COSMO-7 and I2 underestimate Overestimation error decreases in D+2 (spin-up effect vanished)Latest summer worsening EU and I2
Dataset: high resolution network of rain gauges coming from COSMO dataset and Civil Protection Department 1300 stationsMethod: 24h/6h averaged cumulated precipitation value over 90 meteo-hydrological basins
Seasonal trend 20mm/24h + ECMWF
• Slight bias reduction during latest seasons
• winter 2010: all the versions overestimate (probably due to lack of representativeness of the rain gauges over the plain during snowfall)
• Strong COSMO-7 underestimation BUT slight improvement during latest seasons
•General underestimation during latest seasons exc. I7
Seasonal trend 0.2mm/24h + ECMWF
• Very light improvement trend• Seasonal error cycle: lower ets during winter and summertime• no significant differences between D+1 and D+2 winter 2010 (very snowy particularly in Northern Italy): low ets value (D+1 and D+2) model error or lack of representativeness of the rain gauges over the plain during snowfall ?
Seasonal trend 20mm/24h + ECMWF
• Low values during summertime (in general)
•All the versions present two “big jump” at jja08 and jja09, after the values increase and become quite stationary
•Skill decreases with forecast time
Top Related