ICAM 2013 INCA-FVG verification · area, centroides,...) are compared. Different attributes are...

Regional Agency for Environmental Protection

ICAM conference 6 June 2013 ICAM conference 6 June 2013 Kranjska Gora (SLO)Kranjska Gora (SLO)

OSMER – ARPA FVGOSMER – ARPA FVG33040 Visco (UD), Italy33040 Visco (UD), Italy

Objective forecast verification of WRF compared to Objective forecast verification of WRF compared to ALARO and the derived INCA-FVG outputsALARO and the derived INCA-FVG outputs

Arturo Pucillo & Agostino Manzato


www.inca-ce.euwww.inca-ce.eu is an european project of the Central Europe area to apply and improve the INCA nowcasting software developed by ZAMG (A).

Introduction - OutlinesIntroduction - OutlinesINCA and other modelsINCA and other models

Rain forecasting models: - ALARO-5:ALARO-5: LAM at about 4 km, provided directly by ZAMG (A).- INCA-FVG:INCA-FVG: nowcasting software that downscales the initial ALARO-5 model output at 1 km and “fits” it with the surface stations and Fossalon radar observations. RR module runs every 15' on the INCA-FVG domain.

- WRF–ARW:WRF–ARW: LAM at about 3.5 km, initialized on ECMWF every 3h. Uses 3DVAR cold-run data assimilation to ingest about 170 surface stations on Northern-Italy and 5 high-resolution soundings, provided by the external expertise “CETEMPS – Università de L’Aquila”.

- Eulerian Persistence: Eulerian Persistence: also a simple “radar-frozen SRI” method has also a simple “radar-frozen SRI” method has been tried for some verification tests. been tried for some verification tests.

Rain observations for verification:- About 46 independent surface stations (different from those assimilated in WRF) have been used for the pointwise verification.- SRI maps (500m res) of the Fossalon di Grado radar “corrected” with surface OSMER raingauges on the FVG is used for the spatial verification.


The figure below shows in red the 170 stations assimilated by WRF (and INCA-FVG in its smaller domain) and in green the 60 stations used for an independent verification. In reality only about 46 green stations were placed inside the INCA-FVG domain.

DomainsDomains

There are two domains: a bigger one used for the pointwise verification (that is the INCA-FVG domain) and a smaller one (basically the Fossalon di Grado radar covered domain) used for the spatial verification.


The main software tool used for the verification is MET (Model Evaluation Tool) made by the Developmental Testbed Center of NCAR (Boulder, CO, USA), while other specific programs were written in R.

Time domain: the hourly rain was hincasted and verified only for the period 1/6/2011 - 31/7/2011, for a total of about 1450 hourly cases: not so many!Note that there are two runs per day: 00 UTC and 12 UTC (for INCA it were used the 01 and 13 UTC runs, that use the 00 and 12 UTC ALARO).Data Preprocessing: GRIB files were regridded at about 3.8 km using the copygb utility using a nearest neighbour algorithm (Accadia et al., 2003).

-> POINT verification (on the larger INCA-FVG domain):1. pointwise verification on single stations data (nearest neighbour algorithm as model interpolation method), using continuous (MSE, R, BIAS) and categorical (Peirce Skill Score) metrics.

-> SPATIAL verification (on the smaller FVG domain): 2. areal verification of nearest-neighbour using the Fractions Skill Score.3. object-oriented spatial verification using cells attributes (interest metric).

Verification methodsVerification methods


Rain > 1 mm/h

Rain > 2 mm/h

Even small rainfall like 2 mm/h are relatively rare (during these two months p< 4%, so that there are only few cases, ~55). One event has a peak at 6 UTC, while other cases were concentrated on the afternoon/evening period.

Rain climatology in 60 verification stationsRain climatology in 60 verification stations


MSE |R| BIAS

WRF:WRF: low MSE up tp +4h but R decreases until +7h, there is a dry BIAS;

ALARO:ALARO: similar to WRF, but lower MSE after +8h and better R after +5h;

INCA:INCA: higher MSE in the first +4h, high variability in R and wet BIAS.

1.1 Pointwise - continuous verification – run 001.1 Pointwise - continuous verification – run 00someway contrasting results...someway contrasting results...


If Rain > Threshold then the categorical variable is 1, otherwise it is 0. The verification has been repeated on more rain thresholds: >=0, >= 1, >= 2, >= 3, >= 4, >= 5 … mm/h

POD=a

( a+c ), POFD=

b( b+d )

PSS=POD−POFD , where

1.2 Pointwise - categorical verification1.2 Pointwise - categorical verificationcontingency table and derived indicescontingency table and derived indices

Peirce Skill Score chosen as verification metric (see Manzato 2007 WAF):

a = correct hitb = false allarmsc = missed eventsd = correct non-events


Rain > 1 mm/h Rain > 2 mm/h

WRFWRF: the small skill up to +4h seems a spin-up

- After +5hfter +5h ALAROALARO seems to have higher skill, while before , while before WRFWRF is better only for Rain>2mm/h. is better only for Rain>2mm/h.

- - INCAINCA seems better than LAMs in the first +3h (as expected) but EUL. PERS. EUL. PERS. seems even better?

ALARO ALARO seems to have more spin up up to +5h

Pointwise – categorical run 00Pointwise – categorical run 00PSS vs. lead timePSS vs. lead time


Tested with nearest neighbour method from 1 to 45 km windows width.Categorical thresholds: >=0, >= 1, >= 2, >= 3, >= 4, >= 5,... >= 12.

FSS (Roberts and Lean 2008) compares spatialized observations and forecasts on scales larger and larger, varying also the rain intensity threshold (same for obs and for).

The red box is the area under test and the number of gridboxes with rain > threshold is the only parameter considered (not the exact location), to avoid the double penalty problem.

FSS=1−

1

N∑i=1

N

(P fcst−Pobs )2

1

N∑i=1

N

Pf cs t2

+1

N∑i =1

N

Pobs 2

observed forecast

2. Areal Verification2. Areal Verificationnearest neighbour with Fractions Skill Score (FSS)nearest neighbour with Fractions Skill Score (FSS)

FSS=1−

1N∑i= 1

N

( P fcst−P obs)2

1N∑i=1

N

Pfcst 2

+1N∑i=1

N

Pobs2


WRF No skill for LAMs at +1h (spin up problem).

ALARO

Persist.Eul. INCA

Much better performance for INCAINCA and even better for Eul. Persist.Eul. Persist.

Fractions Skill Score run 00 @ +1hFractions Skill Score run 00 @ +1h


WRF

Still very low skill for the two LAMs.

ALARO

Persist.Eul.INCA

INCAINCA and Eul. Eul. Persist.Persist. show similar performance: quite lower than @ +1h lead time.



WRF

ALARO ALARO shows the best performance, followed by INCAINCA, that @ +6h should become very similar to its background model.

ALARO

Persist.Eul. INCA



WRF WRFWRF seems better than ALAROALARO, in particular at high rain thresholds.

ALARO

Persist.Eul. INCA

INCAINCA seems even better than ALAROALARO for the small rain thresholds (maybe due to the downscaling effect?). No skill for Eul. Eul. Persist.Persist...



WRF

Better skill @ +3h for the 12 UTC run than it was for the 00 UTC run. Probably because there were more rainy cases during the afternoon than during the night. That shows a dependence from the small sampling dataset. WRFWRF seems the best in this case.

ALARO

Persist.Eul. INCA



Identification

Merging

Matching

Comparison

Measure Attributes

Summarize

Identification

Measure Attributes

Matching: many objects are identified on the for and obs fields and their attributes (shape, area, centroides,...) are compared. Different attributes are summarized with the INTEREST index, that -of course- is very sensible to the parameters chosen.

INCAINCAWRFWRF

3. Spatial Verification3. Spatial Verificationobject-oriented attributes object-oriented attributes

OBSOBS OBSOBS


INCAINCA (run every 3h in this case) in the first +5h seems to have better description of the objects (because it starts from the radar SRI), but the simpler Eul. Eul. Persist.Persist. is also very good in the first +3h. WRFWRF is the worst in the first +6h, but then improves after +10h.

INTEREST vs. lead time - run 00 for Rain>1mm/h INTEREST vs. lead time - run 00 for Rain>1mm/h


In this case the two LAM perfromances are reversed: WRFWRF performs better in the first +6h, while ALAROALARO seems better after +10h.

The results of this verification seems to vary a lot with the rain threshold.

INTEREST vs. lead time - run 00 for Rain>2mm/hINTEREST vs. lead time - run 00 for Rain>2mm/h


- It is very difficult to make reliable conclusions given this small verification dataset (short period and small domain covered by radar). For this reason the statistical significance of these tests (computed in many cases but not shown) is very low;

- In general it seems that INCA FVGINCA FVG has good performances in the nowcasting range (up to +3h), but these performances seem not much better than the simpler Eulerian PersistenceEulerian Persistence;

- In the short-term forecasting it seems that WRFWRF seems better for the Rain>2 mm/h threshold, while ALAROALARO is better for the Rain > 1 mm/h events.

- Even WRFWRF with data assimilation (cold-run 3D-VAR) of local stations and high-resolution soundings shows spin up problems at least in the first 3h.

Preliminary ConclusionsPreliminary Conclusions

ICAM 2013 INCA-FVG verification · area, centroides,...) are compared. Different attributes are...

Documents

Transcript of ICAM 2013 INCA-FVG verification · area, centroides,...) are compared. Different attributes are...