NWS TAF Verification
description
Transcript of NWS TAF Verification
NWS TAF VerificationBrandi Richardson
NWS Shreveport, LA
Home
Previous
Next
Help
Do we care how our forecasts verify?
NO!
Home
Previous
Next
Help
Do we care how our forecasts verify?
• Yes!• The NWS measures verification by many
means– Probability of Detection (POD)– False Alarm Ratio (FAR)– Critical Success Index (CSI)– Percent Improvement
• Set goals for verification• Local offices add own flavor• Total IFR (IFR, LIFR, VLIFR)
Home
Previous
Next
Help
Why is verification important?
• Need to know what to improve– Lose credibility if too many forecasts are
wrong• Lose customers• Lose jobs
– Additional training– New techniques– Improved model guidance
• Need to know what we are doing well
Home
Previous
Next
Help
NWS TAF Verification
• TAFs evaluated 12 times per hour (every five minutes), or 288 times per 24-hour period
• TAFs compared to ASOS five-minute observations – ASOS = Automated Surface Observing
System, located at TAF airports• Stats calculated by flight category
– i.e., VFR, MVFR, IFR, LIFR, VLIFR
Home
Previous
Next
Help
Probability of Detection
• How often did we correctly forecast a particular flight category to occur?– Also known as “Accuracy”
• POD = V/(V+M)– V = forecasted and verified events
• Ex: IFR conditions forecasted…IFR conditions occurred– M = missed events
• Ex: VFR conditions forecasted…IFR conditions occurred• Ranges from 0 – 1, 1 being perfect
Home
Previous
Next
Help
False Alarm Ratio
• How often did we forecast a particular flight category to occur that did not occur?– i.e., how often did we “cry wolf”?
• FAR = U/(U+V)– U = forecasted and unverified
• Ex: IFR forecasted…VRF occurred– V = forecasted and verified events
• Ex: IFR conditions forecasted…IFR conditions occurred• Ranges from 0 – 1, 0 being perfect
Home
Previous
Next
Help
Critical Success Index
• CSI = V/(V+M+U)– V = forecasted and verified events
• Ex: IFR conditions forecasted…IFR conditions occurred– M = missed events
• Ex: VFR conditions forecasted…IFR conditions occurred– U = forecasted and unverified
• Ex: IFR forecasted…VRF occurred• Ranges from 0 – 1, 1 being perfect• Incorporates both POD and FAR• Overall score of performance
Home
Previous
Next
Help
Percent Improvement
• Forecaster CSI vs. Model Guidance CSI– Did we beat the model?
IFR will prevail…
IFR?! It’s July and dew points are in
the 20s! Take that!
Forecaster
GFS
Home
Previous
Next
Help
2009 NWS Goals
• The NWS has set goals for TAF forecasts– For total IFR (includes IFR, LIFR, and
VLIFR)• POD ≥ 0.640 (64%)• FAR ≤ 0.430 (43%)
• How do we measure up?...
Home
Previous
Next
Help
Examples of Local TAF Verification
4th qtr 05
1st qtr 06
2nd qtr 06
3rd qtr 06
4th qtr 06
1st qtr 07
2nd qtr 07
3rd qtr 07
4th qtr 07
1st qtr 08
2nd qtr 08
3rd qtr 08
4th qtr 08
1st qtr 09
2nd qtr 09
3rd qtr 09
4th qtr 09
2005 2006 2007 2008 2009
0.000
0.100
0.200
0.300
0.400
0.500
0.600
0.700
0.800
0.900
1.000
IFR Probability of Detection (POD) by QuarterSRSHVSHV GFS MOSGPRA Goals
Perc
ent
Home
Previous
Next
Help
Examples of Local TAF Verification
4th
qtr 0
51s
t qtr
062n
d qt
r 06
3rd
qtr 0
64t
h qt
r 06
1st q
tr 07
2nd
qtr 0
73r
d qt
r 07
4th
qtr 0
71s
t qtr
082n
d qt
r 08
3rd
qtr 0
84t
h qt
r 08
1st q
tr 09
2nd
qtr 0
93r
d qt
r 09
4th
qtr 0
91s
t qtr
102n
d qt
r 10
3rd
qtr 1
04t
h qt
r 10
1st q
tr 11
2nd
qtr 1
13r
d qt
r 11
4th
qtr 1
11s
t qtr
122n
d qt
r 12
3rd
qtr 1
24t
h qt
r 12
1st q
tr 13
2nd
qtr 1
33r
d qt
r 13
4th
qtr 1
3
2005
2006 2007 2008 2009 2010 2011 2012 2013
0.000
0.100
0.200
0.300
0.400
0.500
0.600
0.700
0.800
0.900
1.000
IFR False Alarm Ratio (FAR) by QuarterSRSHVSHV GFS MOSGPRA Goals
Perc
ent
Home
Previous
Next
Help
Examples of Local TAF Verification
4th
qtr 0
51s
t qtr
062n
d qt
r 06
3rd
qtr 0
64t
h qt
r 06
1st q
tr 07
2nd
qtr 0
73r
d qt
r 07
4th
qtr 0
71s
t qtr
082n
d qt
r 08
3rd
qtr 0
84t
h qt
r 08
1st q
tr 09
2nd
qtr 0
93r
d qt
r 09
4th
qtr 0
91s
t qtr
102n
d qt
r 10
3rd
qtr 1
04t
h qt
r 10
1st q
tr 11
2nd
qtr 1
13r
d qt
r 11
4th
qtr 1
11s
t qtr
122n
d qt
r 12
3rd
qtr 1
24t
h qt
r 12
1st q
tr 13
2nd
qtr 1
33r
d qt
r 13
4th
qtr 1
3
2005
2006 2007 2008 2009 2010 2011 2012 2013
0.000
0.100
0.200
0.300
0.400
0.500
0.600
IFR Critical Success Index (CSI) by QuarterSR
SHV
SHV GFS MOS
Perc
ent
Home
Previous
Next
Help
The Bottom Line
• Sometimes we do get the forecast wrong.
• Examination of TAF verification statistics helps to find our weaknesses and allows us to find ways to improve our forecasts.
• The NWS strives to provide quality products and services to our aviation customers and partners.