Displaying the Verification Results of Terminal Aerodrome ... · • International Civil Aviation...

Displaying the Verification Results of Terminal Aerodrome Forecasts for Thunderstorms and Visibility

Jadran Jurković 1

Zoran Pasarić2 and Igor Kos 1 1 Croatia Control ltd 2 University of Zagreb, Dept. Of Geophys.

7th International Verification Methods Workshop | Berlin 2017

1

Outline

• Introduction

• Diagnostic purpose

• Displaying Results

– Thunderstorms

– Visibility

• Conclusion


2

Introduction: TAF

• Terminal aerodrome forecast

• Standard ICAO product

• but each country has its own style (Sharpe 2016) – „…as agreed by the meteorological authority with

the ATS authority and operators concerned”

– duration 24h (9-32h)

– Other: standards and recommendations , criteria for groups of changes (and/or AMD)


3

Introduction: TAF verification

• ICAO Annex 3 attachment B – Operationally desirable accuracy of forecasts – The same in EASA documents (without changes)

• Several basic approaches – Harris (2000), Mahringer (2008), Sharpe (2016)

• Other features for TAF verifications: – TAF statements are deterministic, probabilistic and temporal – Results: scores, contingency tables – Rare events

• forecast for a given point • the verification time period is just one hour • aviation requirements usually refer to high impact weather

– Climatological difference


4


Diagnostic verification

• Basic principles – G. Mahringer (2008)

– Best (FC and OBS) and worse (FC and OBS) conditions are verified

• Diagnostic verification

– Weakness and strength of forecast

– Verification of special problems for aviation in Croatia

• Convection (TS)

• Fog (reduced visibility)

• Wind (especially bora events)

5


Verification

• Verification scores – p0, Bias, TCC/PCC (tetrachoric/polychoric correlation

coefficient) – Juras and Pasarić (2006): Application of tetrachoric

and polychoric correlation coefficients to forecast verification (link) • Pearson (1900) • measure of association in contingency tables • TCC=-1, 1, TCC=0 for random • TCC/PCC do not depend on bias nor on marginal frequencies

=> independent information between them • TCC/PCC are particularly good for rare events (fig.2)

6

http://geofizika-journal.gfz.hr/vol_23/No1/juras.pdf


Verification

• Verification of hours with TS • FC: „worst conditions”=> all group of changes

• OBS: METAR 0:00 or 0:30

– Contingency table 2x2

• Frequency of observed hours with TS at airports in Croatia => 0.8-1.5% => rare event

• Relatively large false alarms 7


TS verification: hit a)

• Analysis of hit event – Months

– Hours

– Issue time

– Lead time

– Chronological

– Contingency table

– Forecast expression of TS in TAFs

Challenges in meteorology 3 21-22 November 2013.,Zagreb

ANALIZA PROGNOZE GRMLJAVINE (TS)Termini prognozirani: a = Točno (pogodak)

(Prognozirano: DA Motreno: DA) (interno)

Zračna Luka: LDZA Razdoblje: 2009-2013

Da Ne Ukupno

a= 1062 b= 8059 9121

c= 820 d= 164308 165128

1882 172367 174249

Načini i i izvještavanje za događaje "a" LDZA

Ne

... ...

Ukupno

METAR TAF70 TS

41 TSRETS

41 TS TS

25 TS

22 -TSRARETS -RA

20 -TSRA -TSRA

19 -TSRA TS

15 TSRA TSRA

15 TSRETS

15 VCTS

14 -TSRA TSRA

12 -TSRA -RA

12 TS TSRETS

11 -TSRA

9 -TSRARETS

8 RARETS RA

8 RETS

8 -TSRARETSRA

8 TS -TSRA

6 TSRARERA

6 TSRARETS RA

6 -TSRA -TSRARETS

6 TSRA -TSRARETS

5 -TSRARERA -

TSRARERA

5 TSRARETS

5 -TSRA TSRARERA

5 TSRETS RETS

4 +TSRARESHRA -TSRA

4 -RA -RARETS

4 -RA TS

183 NSW TE SHRA P40 TSRA

99 NSW TE TSRA

86 NSW P40 TSRA

78 NSW TE SHRA P30 TSRA

57 NSW P30 TSRA

27 NSW TE TSRA P30 TSGR

12 RA TE TSRA

10 NSW TE SHRA TE TSRA

9 NSW P40 SHRA P30 TSRA

6 NSW TE RA P30 TSRA

6 NSW TE TSRA BE RA

5 NSW TE TSRA P40 TSGR

4 NSW TE SHRA P40 TSRA BE

RA BR

4 NSW TE RA P40 TSRA

4 RA P30 TSRA

4 BR P40 BCFG P30 TSRA

3 NSW BE TSRA

3 NSW P40 TS

3 NSW TE SHRA P40 TSRA BE

RA

3 NSW TE TSRA P30 TSRAGR

3 NSW TE SHRA TE TSRA P30

TSGR

2 NSW FM RA TE SHRA P30

TSRA

2 TSRA

2 NSW TE TSRA P30 TSRA

2 NSW BE RA P30 TSRA

2 NSW P40 SHRA TE TSRA

2 RA P40 TSRA

Motreno

Prognozirano

Da

0 1 21 56

279 311265

17394

37 0 13

0

20

40

0

500

1 2 3 4 5 6 7 8 9 10 11 12 Pr.

br.

Kl.

TS

term

ina

M

ETA

R (

po

lusa

tni)

Br.

"Pro

pu

st"

ili

mo

tren

ih T

S sa

ti

Raspodjela prognoziranih sati "Točno (pogodak)" po mjesecima

KLIMA pros. br. TS termina Uk. verif. "Točno (pogodak)" sati 2009-2013 Uk. Motreni TS termini 2009-2013

29 25 22 8 13 12 23 13 9 1032 35

78101 95 86

117148

102 9851 36 50 57

0

50

100

150

200

0

100

200

300

0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 Pr.

br.

Kl.

TS

term

ina

M

ETA

R (

po

lusa

tni)

Br.

"Pro

pu

st" i

li

mo

tre

nih

TS

sati

Sat u danu (UTC)

Raspodjela prognoziranih sati "Točno (pogodak)" po satu (UTC) u danu

KLIMA pros. Br. TS termina Uk. verif. "Točno (pogodak)" sati 2009-2013 Uk. Motreni TS termini 2009-2013

74 7756

4467 65 55 52

37 3254

6552 54

39 3147

6851 54 45

3344 54

0

50

100

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24Sat nastupa TAF-a

Raspodjela prognoziranih sati "Točno (pogodak)" po vremenu nastupa(1-24) TAF-a

5 11 17 23 Ukupno Točno (pogodak)" sati

294332 336

288

200

300

400

5 11 17 23

Raspodjela prognoziranih sati "Točno (pogodak)" po vremenu izdanja TAF-a

0 0 02210

6993

353 5 0 4 0 0 9 0

19243257

1 0 0 0 0 0 0 8

75104

360 12 9 0 9 0 0 2 11

643930

9 2310 0 0 0 1 8 4

47364463

323 0 0

0

100

200

2009-1 2009-4 2009-7 2009-10 2010-1 2010-4 2010-7 2010-10 2011-1 2011-4 2011-7 2011-10 2012-1 2012-4 2012-7 2012-10 2013-1 2013-4 2013-7 2013-10

Br.

"Pro

pu

st" i

li

mo

tre

nih

TS

sati

Raspodjela prognoziranih sati "Točno (pogodak)" po mjesecima kronološki 2009-2011

Uk. verif. "Točno (pogodak)" sati 2009-2013 Uk. Motreni TS termini 2009-2013

8


Bias / TCC display

Example of contingeny tables for different bias and TCC

Example: frequency (a+c) of hours with TS je 1.3%

7 0,65% 9,10% 0,90% 9,10% 1,15% 9,10%

0,65% 89,60% 0,40% 89,60% 0,15% 89,60%

1,30% 98,70% 1,30% 98,70% 1,30% 98,70%

5 0,55% 6,50% 0,79% 6,50% 1,09% 6,50%

0,75% 92,20% 0,51% 92,20% 0,21% 92,20%

1,30% 98,70% 1,30% 98,70% 1,30% 98,70%

3 0,41% 3,90% 0,64% 3,90% 0,90% 3,90%

0,89% 94,80% 0,66% 94,80% 0,40% 94,80%

1,3% 98,7% 1,3% 98,7% 1,3% 98,7%

0,5 0,65 0,8

TCC

bia

s

a b

c d

9


Bias / TCC display example 2

Dispersive results are expected

10

Tablica kontningencije Tablica kontningencije Verifikacijski indeksi Ostali pomoćni pokazatelji

Razdoblje: 9.2.2012-31.12.2014. Aspolutne čestine [sat] Relativne čestine [%] Točnost Pristranost Pa+Pb Pa+Pc a/(a+c) a/(a+b) a/(a+b+c) (PSS+HSS)/2

Ime_EERS a b c d Pa Pb Pc Pd TCC Bias a-c/(a+c) Pfc Pobs POD pTSfc CSI KPI

Puhalo Dragović, Nataša(729) 480 93 521 66 10756 0,8% 4,6% 0,6% 94,1% 0,70 3,9 159 0,17 5,4% 1,4% 0,58 0,15 0,14 0,26SMB 4111 703 4684 549 92366 0,7% 4,8% 0,6% 94,0% 0,67 4,3 0,12 5,6% 1,3% 0,56 0,13 0,12 0,25

Raspodjela događaja a (pogodak), b (krivi alarm), c(propust)

Broj

TAFovaMotreni

TS sati

1,0

2,0

3,0

4,0

5,0

6,0

7,0

0,4 0,5 0,6 0,7 0,8 0,9 1,0

Pri

stra

no

st (

Bia

s)

Točnost (TCC)

Indeksi verifikacije TS prognoze u TAF-u po prognostičaru za razdoblje 9.2.2012-31.12.2014.

FC

Ostali

SMB

PO godinama

0

20

40

60

0 2 4 6 8 10 12 14 16 18 20 22

SAT U DANUb

c

a

0

20

40

60

1 3 5 7 9 11 13 15 17 19 21 23

NASTUPNI SATb

c

a

0

50

100

150

200

5 11 17 23

IZDANJE

0

50

100

150

200

1 2 3 4 5 6 7 8 9 10 11 12

MJESEC

XY


TS verification: forecaster LDZA

Challenges in meteorology 3 21-22 November 2013.,Zagreb

TCC

BIA

S

11



• Verification of visibility

– According to categories with limits 150, 350, 600, 800, 1500, 3000 and 5000m

– Best (FC and OBS) and worse (FC and OBS) conditions are verified

– 2 contingency tables NxN: polyhoric correlation coefficient is used

12



Better results when p0 is higher

Bias is larger - p0 smaller

13


Conclusions

• Diagnostic verification – Based on approach by Mahringer (2008)

• Verified elements in TAFs are rather rare • Displayed results are used with triplet

– marginal frequency, bias and tetrachoric coefficient (Juras and Pasarić 2006)

– Bias and TCC represents forecast – 4 examples for thunderstorms and visibility

• Additional conclusion : verification results depend on – climatology (frequency of an event) – forecaster

14


Literature

• Harris GR. 2000. Comparison of different scoring methods for TAFs and other

probabilistic forecasts. 15th AMS Conference on Probability and Statistics in the Atmospheric Sciences, 8–12 May 2000, Asheville, NC.

• International Civil Aviation Organization (ICAO). 2010. Meteorological Services for International Air Navigation. Annex 3 to the Convention on International Civil Aviation, 15th edn. ICAO: Montreal.

• Mahringer G. 2008. Terminal aerodrome forecast verification in Austro control using time windows and ranges of forecast conditions. Meterol. Appl. 15: 113–123.

• Pearson, K. 1900: Mathematical contributions to the theory of evolution. VII. On the correlation of characters not quantitatively measurable, Philos. Tr. R. Soc. S.–A, 195, 1–47.

• Juras J., Pasarić Z. 2006, Application of tetrachoric and polychoric correlation coefficients to forecast verification, Geofizika 23, 59-82

• A. Sharpe M., E. Bysouth C. , Trueman M. 2016, Towards an improved analysis of Terminal Aerodrome Forecasts, Meterol. Appl. 23, 698–704.

15

Displaying the Verification Results of Terminal Aerodrome ... · • International Civil Aviation...

Documents

Transcript of Displaying the Verification Results of Terminal Aerodrome ... · • International Civil Aviation...