COSMO General Meeting – Moscow 06-10 Sept 2010 Some results from operational verification in Italy...

COSMO General Meeting – Moscow 06-10 Sept 2010

Some results from operational verification in Italy

Angela Celozzi - Federico GrazziniMassimo Milelli - Elena Oberto

Adriano Raspanti - Maria Stefania Tesini


Verification

CROSS MODEL COSMOME vs ECMWF CROSS MODEL COSMOI7 vs ECMWF CROSS MODEL COSMOME vs COSMOIT COSMOME –Upper Air COSMOI7 – Upper Air (OBS and

Analysis) CONDITIONAL VERIFICATIONS Long Term and Seasonal Precipitation


Verification

CROSS MODEL COSMOME vs ECMWF CROSS MODEL ECMWF vs COSMOI7 CROSS MODEL COSMOME vs COSMOIT COSMOME –Upper Air COSMOI7 – Upper Air CONDITIONAL VERIFICATIONS Long Term and Seasonal Precipitation


COSMOME vs ECMWF Temperature

SON

JJA

MAM

DJF


COSMOME vs ECMWF Dew Point Temperature

SON

JJA

MAM

DJF


COSMOME vs ECMWF Mean Sea Level Pressure

SON

JJA

MAM

DJF


COSMOME vs ECMWF Total Cloud Cover

SON

JJA

MAM

DJF


COSMOME vs ECMWF Wind Speed

SON

JJA

MAM

DJF


Verification



COSMOI7 vs ECMWF Dew Point Temperature

SON

JJA

MAM

DJF


COSMOI7 vs ECMWF Mean Sea Level Pressure

SON

JJA

MAM

DJF


COSMOI7 vs ECMWF Wind Speed

SON

JJA

MAM

DJF


COSMOI7 vs ECMWF Total Cloud Cover

SON

JJA

MAM

DJF


Verification



TemperatureCOSMOME vs COSMOIT

SON

JJA

MAM

DJF


Mean Sea Level PressureCOSMOME vs COSMOIT

SON

JJA

MAM

DJF


Dew PointTemperatureCOSMOME vs COSMOIT

SON

JJA

MAM

DJF


Total Cloud CoverCOSMOME vs COSMOIT

SON

JJA

MAM

DJF


Wind SpeedCOSMOME vs COSMOIT

SON

JJA

MAM

DJF


Conclusion

• COSMO - ME generally better than IFS, except MSLP

• COSMO – I7 better or almost the same than IFS

• Comparison COSMO-ME and COSMO-IT shows improvements for High-Res.


Verification

CROSS MODEL COSMOME vs COSMOIT CROSS MODEL COSMOME vs ECMWF CROSS MODEL ECMWF vs COSMOI7 COSMOME – Upper Air COSMOI7 – Upper Air CONDITIONAL VERIFICATIONS Long Term and Seasonal Precipitation


COSMOME –Upper AirTemperature


COSMOME –Upper AirWind Speed


Verification

CROSS MODEL COSMOME vs COSMOIT CROSS MODEL COSMOME vs ECMWF CROSS MODEL ECMWF vs COSMOI7 COSMOME –UpperAir COSMOI7 – Upper Air CONDITIONAL VERIFICATIONS Long Term and Seasonal Precipitation


COSMOI7 –Upper Air Temperature


COSMOI7 –Upper AirWind Speed


COSMO I7 COSMO BackUp ECMWF

MSL FC+48 – Each model is verified against its own analysis

Shaded contouring every 0.5 hPa, starting from 0.5. Red and blue lines representPositive/Negative bias, every 0.5 hPa

Spatial distribution of mean absolute error (MAE), computed over MAM 2010

UPPER-AIR (against analysis)


Forecast Step

MAE growth with forecast step, computed over COSMOI7 domainSpring 2010 (MAM 2010) – All models and analyses are interpolated onA regular grid at 0.25 * 0.25 deg of h-resolution. Everyone against its own analysis. MSL



Spatial distribution of mean absolute error (MAE), computed over MAM 2010

T850 FC+48 – Each model is verified against its own analysis

Shaded contouring every 0.5 C°, starting from 0.5. Red and blue lines representPositive/Negative bias, every 0.5 C°

COSMO I7 COSMO BackUp ECMWF



MAE growth with forecast step, computed over COSMOI7 domainSpring 2010 (MAM 2010) – All models and analyses are interpolated onA regular grid at 0.25 * 0.25 deg of h-resolution. Everyone against its own analysis. Z 700 hPa



Conclusion

• COSMO – ME and COSMO-I7 have a general good result in upper air Verification

• COSMO-ME seems better, but improvement from MAM for COSMO-I7 (bug in AOF file until march)


Verification

CROSS MODEL COSMOME vs COSMOIT CROSS MODEL COSMOME vs ECMWF CROSS MODEL ECMWF vs COSMOI7 COSMOME –Upper Air COSMOI7 – Upper Air CONDITIONAL VERIFICATIONS Long Term and Seasonal Precipitation


Conditional Verification Temp – TCC obs <=35%

Worse behaviour for all the seasonsCompare to no condition model

SON

MAM

DJF


Conditional Verification Temp – TCC obs >=75%

SON

MAM

DJF

Better behaviour for all the seasonsCompare to no condition model


Conditional Verification MSLP – MSLP >=mean

SON

MAM

DJF


Conditional Verification Tdew – Wind Speed (Obs) <=2 m/s

SON

MAM

DJF

Almost indifferent to the condition in obs space


Conditional Verification Tdew – Wind Speed (fcs) <=2 m/s

SON

MAM

DJF

Worse behaviour for all the seasonsIn fcs space


Conditional Verification

Temp – Prec +06 <= 0,5 Temp – Prec +06 <=10


Conditional VerificationTemp – MSLP >=mean

SON

MAM

DJF

Worse behaviour in DJF for RMSESimilar fo other seasons


Conditional VerificationTemp – MSLP <=mean

SON

MAM

DJF

General better behaviour for all theSeasons compare to NC


Conclusion

• Comparison between NC and Cond verification seems effective in most of the cases

• A standard set of Conditions should be decided by WG5 and produce on regular basis


Verification

CROSS MODEL COSMOME vs COSMOIT CROSS MODEL COSMOME vs ECMWF CROSS MODEL ECMWF vs COSMOI7 COSMOME –Upper Air COSMOI7 – Upper Air CONDITIONAL VERIFICATIONS Long Term and Seasonal Precipitation


Seasonal trend - low thresholds• All the versions present a seasonal cycle with an overestimation during summertime (except COSMO-7 and I2)• COSMO-7 and I2 underestimate• Overestimation error decreases in D+2 (spin-up effect vanished)

QPF verification of the 4 model versions at 7 km res. (COSMO-I7, COSMO-7, COSMO-EU, COSMO-ME) with the 2 model versions at 2.8 km res. (COSMO-I2, COSMO-IT)Dataset: high resolution network of rain gauges coming from COSMO dataset and Civil Protection Department 1300 stationsMethod: 24h/6h averaged cumulated precipitation value over 90 meteo-hydrological basins


• Very light improvement trend

• Seasonal error cycle: lower ets during winter and summertime

• no significant differences between D+1 and D+2

• Last winter (very snowy particularly in Northern Italy): low ets value (D+1 and D+2) model error or lack of representativeness of the rain gauges over the plain during snowfall ?

Seasonal trend - low thresholds


• Very light improvement trend

• Seasonal cycle with more false alarms in summertime (particularly for I7)




• Quite stationary during last seasons

• I2 has very low values during summer




Driving model comparison: ECMWF/COSMO-ME/COSMO-IT, low thresholds

• ECMWF tendency to forecast low rainfall amounts big overestimation, big false alarms, very low ets, quite good pod

• Better prediction for COSMO-models (no strong differences between ME and IT)


• ECMWF tendency to forecast low rainfall amounts big overestimation, big false alarms, very low ets, quite good pod

• Better prediction for COSMO-models BUT bad performance during summertime

Driving model comparison: ECMWF/COSMO-ME/COSMO-IT, low thresholds


• Slight bias reduction during latest seasons

• Last winter: all the versions overestimate (probably due to lack of representativeness of the rain gauges over the plain during snowfall)

• Strong COSMO-7 underestimation BUT slight improvement during latest seasons

Seasonal trend - high thresholds


• Low values during summertime

• In general, quite stationary error since son2008 up to now

• All the versions present a jump around son2008: ets increases from 0.2-0.4 up to 0.3-0.5 (cosmo-I7: son2008 introduction of 4.3 version with new T2m diagnostic)

• Skill decreases with forecast time




• Slight far reduction during last two years BUT high values during summer2009 and winter 2010 (probably due to lack of representativeness of the rain gauges over the plain during snowfall)

• Small far increase last spring


• Low probability of detection during intense convective events

• It is noticeable the improvement since son2008 BUT a subsequent worsening during 2009 and the first half of 2010

• Skill decreases with forecast time



• ECMWF difficulty to forecast high rainfall amounts bias around 1 BUT big false alarms, very low ets and pod

• Better prediction for COSMO-models

Driving model comparison: ECMWF/COSMO-ME/COSMO-IT, high thresholds


Diurnal cycle - low thresholds

• Little initial spin-up (especially for I7 and I2)

• No strong performance differences among the versions

• Slight diurnal cycle

• Slight worsening with forecast time


Diurnal cycle - high thresholds

• Little initial spin-up vanished with threshold increasing

• No strong performance differences among the versions except COSMO-7 underestimation

• Pronounced worsening with forecast time


COSMO-7 COSMO-I7 COSMO-ME

COSMO-EU COSMO-I2 COSMO-IT

Bias, 10mm/24h

200812-201005

• Systematic overestimation over Alpine areas, especially in the western part and in Veneto/Trentino-Alto Adige (incorrect representation of flow interaction with alpine chain during westerlies and north-easterlies ?)

• COSMO-7 underestimates especially in southern Italy (border of the domain ?)

• COSMO-I7 overestimates the Adriatic areas (especially during north-easterly flow forecasters experience)

• COSMO-I2 underestimates, COSMO-IT overestimates


Far, 10mm/24h

200812-201005



• More false alarms over Sardinia and Alpine areas, especially in the western part and in Veneto/Trentino-Alto Adige (incorrect representation of flow interaction with alpine chain during westerlies and north-easterlies ?)

• More false alarms for COSMO-I7 (and the other ones) over the Adriatic areas (especially during north-easterly flow forecasters experience)


Relative error %, SON 2009



• Too precipitation amount over Alpine areas, especially in the western part and in Veneto/Trentino-Alto Adige (incorrect representation of flow interaction with alpine chain during westerlies and north-easterlies ?)

• Few QPF for COSMO-7, COSMO-I2 and COSMO-IT

• Quite good QPF for COSMO-I7, COSMO-ME and COSMO-EU


Relative error %, DJF 2010



• Few precipitation amount (all the versions except COSMO-I2) in the Padana Plain: very snowy winter also in plain areas where there are no heated rain gauges lack of representativeness

• Few QPF for COSMO-7 in southern Italy


Relative error %, MAM 2010



• Too precipitation amount over Alpine areas, especially in the western part and in Veneto/Trentino-Alto Adige (incorrect representation of flow interaction with alpine chain during westerlies and north-easterlies ?)

• Few QPF for COSMO-7, COSMO-I2 and COSMO-IT

• Quite good QPF for COSMO-I7, COSMO-ME and COSMO-EU

COSMO General Meeting – Moscow 06-10 Sept 2010 Some results from operational verification in Italy...

Documents

Transcript of COSMO General Meeting – Moscow 06-10 Sept 2010 Some results from operational verification in Italy...