Evaluation of AMPSForecasts Using

Self-Organizing Maps (SOMs)

John J. Cassano

Cooperative Institute for Research in Environmental Science

and

Program in Atmospheric and Oceanic Sciences

University of Colorado at Boulder

Beardmore Glacier, Jan 2004

Outline

• What are SOMs?

• Application of SOMs for model evaluation studies

• Application of SOM Analysis to AMPS data

• Conclusions / Future Work

What are SOMs?

• SOM - Self-Organizing Map• SOM technique uses an unsupervised learning algorithm

(neural net)• Clusters data into a user selected number of nodes• SOM algorithm attempts to find nodes that are

representative of the data in the training set– More nodes in areas of observation space with many data points– Fewer nodes in areas of observation space with few data points

• SOMs are in use across a wide range of disciplines

Application of SOMsfor Model Evaluation Studies

• Synoptic pattern classification

• Frequency of occurrence of synoptic patterns

• Determine model errors for different synoptic patterns

Application of SOM Analysis to AMPS Data

• Train SOM with AMPS SLP data– Result is a synoptic pattern classification

• Evaluate frequency of occurrence of synoptic patterns predicted by AMPS as a function of forecast duration– Map 0h, 24h, and 48h forecasts to SOM

• Mis-mapping of AMPS forecasts• Model validation statistics for specific synoptic patterns

(ongoing work)– Calculate model error statistics at points of interest (Willie Field)

for different synoptic patterns– Are certain synoptic patterns prone to bias (e.g. error in predicted

wind speed or direction)?

AMPS Data for SOM Analysis

• SLP over Ross Sea sector of AMPS 30 km model domain

• Summer only (NDJ)

• 00Z AMPS simulations from Jan 2001 through Feb 2003 – 186 model simulations

• Evaluate 0, 24, and 48 h AMPS forecasts

AMPS SOM Analysis Domain

Synoptic Pattern Classification

Frequency of Occurrence

Misprediction of Synoptic Patterns

• Consider all of the time periods for which the model analyses map to a particular node– For these time periods determine which nodes the model

predictions map to

• From this analysis we can determine biases in the model predictions of specific synoptic patterns relative to the model analyses– Percent of cases that map to the correct node– Mis-mapping of model predictions between nodes

AMPS 24h Forecasts

1

1

11 42

1

2

100%71.1%81.1%

73.2%64.7%86.7%

AMPS 48h Forecasts

1

2

42 5

2

1

100%65.8%67.6%

63.4%58.8%86.7%

1

Model Errors for Synoptic Patterns

• Compare model predictions to in-situ atmospheric measurements

• Calculate model validation statistics for all time periods that map to each node

• Look for model errors that vary from node to node

• This is ongoing work using AMPS data– This technique has been applied to ARCMIP data

An ARCMIP exampleSHEBA Surface Pressure (DJF)

Psfc

-3.0

-1.0

1.0

3.0

5.0

7.0

djf (1,1) (1,2) (2,1) (2,2) (3,1) (3,2)

Bias (mb)

Polar MM5

Conclusions / Future Work

• The use of SOMs provides an alternate method of evaluating model performance– Identify synoptic patterns which are over or underpredicted– Determine model tendency for mis-prediction of certain

synoptic types– Provide information on model errors related to specific

synoptic patterns

• Complete SOM analysis for entire AMPS archive (Jan 2001 - present)

• Calculate model biases as a function of forecast time and synoptic patterns