Verification of WAFS Global Icing Products

Jennifer Mahoney1 and Sean Madine1,2

1NOAA/Earth System Research Laboratory2Cooperative Institute for Research in the Atmosphere

This work was completed on behalf of the U.S. Federal Aviation AdministrationPresented at: WAFS Workshop on the Use and Visualisation of Gridded SIGWX Forecasts Paris, September 14-15 2009

Acknowledgements

This work was completed on behalf of the U.S Federal Aviation Administration

Project Team• Project Lead: Sean Madine• Project Analyst: Chungu Lu • Programming Support: Mike Kay• Program Chief: Jennifer Mahoney

Outline of Discussion

• Motivation, Background, and Challenges• Approach and Assessment Strategy

– Creation and validation of CloudSat Icing Product (CLIP)

• Forecast Analyses– Comparison of WAFC U.K. and WAFC U.S. Global Icing– Comparison of WAFC Global Icing to Significant

Weather (SigWx) Product• Summary

Motivation

• Icing forecasts are used in strategic planning

– One example: Planning for amount of fuel carried on board aircraft to manage possible icing encounters.

• The WAFC Washington (U.S.) and the WAFC London (U.K.) have developed Global Icing Products, updated frequently, that may improve icing information that is provided to the aviation community.

Background

• Project Goal: Evaluate the quality of the WAFC Global Icing Forecasts; with resource and time constraints:

– Altitudes that typically contain icing (i.e., mid-levels)

– Scoped evaluation to Nov 2008 – Jan 2009; winter in northern hemisphere

– Domains: Global, North Pacific, North Atlantic; with a North Atlantic focus

• Evaluation criteria : 1) accuracy of the products should be at least as good as the human generated Significant Weather (SigWx) forecasts produced today, and 2) WAFC U.S. and U.K icing products need to be compatible

Challenges

• Obtaining a representative observation dataset– Voice pilot reports (PIREPS) for icing are sparse in time and

space– PIREPs poorly sample at flight levels relevant to

operational decisions (enroute flight levels are typically above the occurrence of icing)

– PIREPs disproportionately sample icing conditions• Exploring new techniques for creating a proxy

observation of icing in the atmosphere

Assessment Approach

• Compare the quality of the WAFC U.K icing product to the WAFC U.S. icing products– Measure the quality– Qualitative examination of compatibility (i.e., incompatible

means ambiguity of the forecast in the planning process)• Compare the quality of the WAFC Global Icing

Products to the Significant Weather Product– Measure the quality

Assessment Strategy

• Build an icing verification proxy based on satellite products and use for evaluating the WAFC icing forecasts over global data-sparse domains

• Development of CloudSat Icing Product (CLIP)– Tuned and validated CLIP with respect to the Current Icing Potential (CIP),

current U.S. operational icing analysis

– Study period: November 2007 through January 2008 (Note: Same season, 1 year earlier)

• Use CLIP to assess the quality of the WAFC global icing products and SigWx products over global domains– Study period: November 2008 through January 2009

CloudSat Icing Product (CLIP) Strengths

• CloudSat cloud profiling radar very effective at diagnosing existence of clouds, particularly at levels relevant to our study

– CLIP highly accurate at detecting no cloud and therefore ‘no icing’

• Information for CLIP allows assessment of forecast risk (miss) and forecast efficiency (false alarms)

– PIREPs don’t allow the false alarm measure because of the imbalance of yes/no reporting

• CLIP’S radar input provides– An objective measure of cloud, at high resolution– Continuous reporting along orbiting track– An improvement upon the subjective nature and sparseness of using PIREPs

Comparison of the WAFC Global Icing Products

Analysis of WAFC Global Icing Products

• Many stratifications examined– Geographic domain

Global, N. Atlantic, N. Pacific– Flight level

• All, FL100, FL140– Cloud type (as diagnosed by CloudSat)

• All, Convective, Non-convective– Algorithm threshold

• 0.1, 0.3, and 0.5 – Lead time

• All leads 6, 12, …, 36-h individually

Analysis of WAFC Global Icing Products

• Main thread of investigation– North Atlantic domain

• Domain of interest during assessment winter time period and included significant amount of air traffic density

– Altitudes 10,000 - 20,000 ft • Operationally significant altitudes and typically contain ice

– Non-convective clouds • Products definition specifically includes non-convective icing

– Max icing attribute• Most compatible WAFC forecasts

Domains

Global, North Atlantic, North Pacific; 3 CloudSat orbits, each orbit ~1.5h,

WAFS Maximum Icing12 January 2009

UK/FL100

UK/FL140 US/FL140

US/FL100

Vertical Cross Sections Along CloudSat Path Atlantic Domain

WAFS-UK

WAFS-US

Cloud Classification

CLIP

12 January 2009

WAFS Maximum Icing23 December 2008

UK/FL100 US/FL100

UK/FL140 US/FL140

Vertical Cross Sections Along CloudSat Path Atlantic Domain

23 December 2008

Comparison of Performance: Flight Level(Statistics for 12- and 18-hr leads, Atlantic domain,

Non-convective clouds, Max icing attribute)

ROC Thresholds: 0.1, 0.3, and 0.5

Product

False Alarm Rate (1-PODn)

POD

Atlantic domain, 0.1 Threshold, FL100/140, 12- and 18-hr leads, Max icing intensity, Non-convective clouds

CLIP

12367

6684 54837 6430 3375

15356 2940 3640

Yes Icing No Icing No Cloud

No Icing Warm Cloud

No Icing Cold Cloud

UK

(fcst

)

8987

10063 58617 5709 4764

11576 3661 2251Yes

No

US

(fcst

)

Comparison WAFC Icing Forecasts in Operational Context

Yes Icing No Icing No Cloud

No Icing Warm Cloud

No Icing Cold Cloud

Yes

No

Blue – AgreementYellow – False alarmsRed - Misses

Comparison Summary

• The WAFC U.K and WAFC U.S. forecasts provide roughly the same forecast efficiency, but the WAFC U.K. provides lower operational risk

FAR POD BiasWAFC U.K 64% 65% 1.8WAFC U.s 66% 47% 1.4

Comparison of the WAFC Global Icing Products to the SigWx Product

WAFS Mid Level SIGWX Product

• Thunderstorms/CB• Tropical Cyclones• Mod/Sev Turb (CAT or

IC)• Mod/Sev Icing• Tropopause Heights• Jetstreams (80kt and

above depicted)• Jet Depth• Volcanic Eruptions• Widespread Sand/Dust

storms• Release of Radioactive

Materials

NOTE: Because of choice to use N. Atlantic domain, only U.S SigWx Product was evaluated

Analysis - Main Thread

• Atlantic domain• Altitudes 10,000 - 20,000 ft• Non-convective clouds• Max icing attribute

Comparison of Performance: SigWx Chart(FL100/140, SigWx issuances/lead, Atlantic domain)

ROC Thresholds: 0.1, 0.3, and 0.5

False Alarm Rate (1-PODn)

POD

Product

NOTE: Because of choice to use N. Atlantic domain, only U.S SigWx product was evaluated

Comparison Summary

• The WAFC U.K and WAFC U.S. forecasts provide roughly the same forecast efficiency, but the WAFC U.K. provides lower operational risk

FAR POD BiasWAFC U.K 64% 65% 1.8WAFC U.S. 66% 47% 1.4SigWx Product

69% 31% 1.2

Summary• The U.K. icing product slightly outperforms the U.S. icing product in an

operational; For similar FAR values, the U.K. has higher POD

• Qualitatively, the WAFC U.K. and U.S products are incompatible in volume extent and intensity of potential

• Both U.K. and U.S. products perform better than the SigWx Product in the operational context; For similar FAR, both WAFC products have higher POD values

• Based on this study and understanding of the algorithms, the automated WAFC product could be significantly improved with respect to volume coverage and calibration of potential

Appendix

• False alarm ratio = yn/yy+yn• False alarm rate = 1-PODn = yn/nn+yn• PODn = nn/nn+yn• POD = yy/yy+ny

2x2 Contingency TableObservation

Forecast Y N

Y YY YN

N NY NN