WoF Workshop, April 3, 2014

Evaluation and Comparison of Multiple Convection-Allowing Ensembles Examined in Recent HWT Spring Forecasting Experiments

Israel Jirak, Steve Weiss, and Chris Melick Storm Prediction Center

• Convection-allowing ensembles (~4-km grid spacing) can provide important information to forecasters regarding the uncertainty of storm intensity, mode, location, timing, etc. on the outlook to watch scale

• These ensembles will play an important role in the ability of SPC to provide a more continuous flow of probabilistic hazard information in support of WoF2 March 2012 29 June 2012

24-h neighborhood prob UH ≥25 m2/s2 24-h ensemble max 10-m Wind Speed

Convection-Allowing EnsemblesOverview

• An experimental real-time Storm-Scale Ensemble Forecast (SSEF) system has been produced for the NOAA Hazardous Weather Testbed (HWT) Spring Forecasting Experiment (SFE) by OU CAPS since 2007 through CSTAR funding– Comprised of 4-km convection-allowing WRF and ARPS members:

• 2007: 10 members; 2008: 10 members; 2009: 20 members; 2010: 26 members (full CONUS to 30 h); 2011: 50 members (full CONUS to 36 h); 2012: 28 members; 2013: 27 members at 00Z (to 48 hours) and 8 members at 12Z (to 18 hours)

– Primarily examine explicit storm attributes, especially hourly maximum fields (HMFs): Updraft Helicity, Updraft Speed, 10-m AGL Wind Speed, 1-km AGL Reflectivity

– Ensemble display approaches include spaghetti plots, ensemble max, and neighborhood probabilities (more on next slide)

• SPC began processing deterministic high-resolution runs from EMC and NSSL as the Storm-Scale Ensemble of Opportunity (SSEO) in 2011

• The 4-km AFWA ensemble was made available to SPC in 2012

Convection-Allowing EnsemblesHistory

• Traditional ensemble probabilities of HMFs from high-resolution models are not especially useful, owing to poor agreement among members at the grid point of these fields.

• Applying a binary neighborhood approach to a storm-scale ensemble improves the statistical results of HMFs in forecasting severe weather– ROI=20-40 km– Sigma=30 grid points

• Same approach can also be applied to observations (e.g., radar reflectivity) for verification purposes

03 May 2008 (Harless 2010)

Convection-Allowing EnsemblesNeighborhood Probabilities

Convection-Allowing EnsemblesNeighborhood Probabilities

HM Updraft Helicity > 25 m2s-2

SSEO 24-hr fcst valid 00Z 28 April

Convection-Allowing EnsemblesNeighborhood Probabilities

Grid-Point Probability HM Updraft Helicity >25 m2s-2 SSEO 24-hr fcst valid 00Z 28 April

Convection-Allowing EnsemblesNeighborhood Probabilities

40-km Neighborhood Probability HM Updraft Helicity >25 m2s-2 SSEO 24-hr fcst valid 00Z 28 April

Convection-Allowing EnsemblesNeighborhood Probabilities

40-km Neighborhood Smoothed Prob HM Updraft Helicity >25 m2s-2 SSEO 24-hr fcst valid 00Z 28 April

• OU/CAPS Storm-Scale Ensemble Forecast (SSEF) System• Since 2007; 36-hr forecasts from 00z; 12z runs began in 2013• Primarily WRF-ARW; 4-km grid spacing; forecasts to 60hrs in 2014 • Multi-physics, multi-initial conditions: applies SREF perturbations to NAM ICs• Advanced physics, 3DVAR & radar data assimilation; available for HWT/SFE

• SPC Storm-Scale Ensemble of Opportunity (SSEO)• Since 2011; 36-hr forecasts at 00z & 12z; 7 members (2 time-lagged) • Multi-model (ARW, WRF-NMM & NMM-B), multi-physics; ~4-km grid spacing• Uses available deterministic models at SPC to process as an ensemble• Basic data assimilation through NDAS; available year-round in SPC

• Air Force Weather Agency (AFWA) Ensemble• Since 2012; 60-hr forecasts at 00z & 12z; 10 members; 4-km grid spacing• Single model (WRF-ARW), multi-physics, multi-initial conditions• Cold start from downscaled global model forecasts (GFS, UM, CMC)• No data assimilation; available year-round in SPC

Convection-Allowing EnsemblesSystem Comparison

• The Fractions Skill Score (FSS) was calculated for neighborhood probability (ROI=40 km; σ=40 km) of updraft helicity ≥ 25 m2s-2 for the SSEO/SSEF versus practically perfect hindcasts of preliminary severe weather reports (ROI=40 km; σ=120 km) during SE2011

• The SSEO had higher fractions skill score (FSS) for neighborhood probabilities of UH ≥25 m2/s2 during SFE2011 than the SSEF

• The number of members included in the SSEF did not seem to have a strong impact on the statistical results for neighborhood probabilities of UH during SE2011 when verified against severe weather reports

Convection-Allowing EnsemblesSFE2011 Results

Nprob UH ≥25 m2/s2SSEO SSEF – 24 memFSS = 0.84 FSS = 0.68

3-hr [NPRS]:UH ≥25 m2s-2 valid 06Z on 02 June 2011w/ verifying reports and practically perfect hindcast

FSS 3-h periods SFE2011

• Even for 6-h QPF, the SSEO received the highest subjective ratings relative to other operational and experimental models and ensembles during SFE2011

• Statistically, the probabilistic QPF forecasts (>0.5”) from the SSEO were typically as good as (if not better than) the SSEF during SFE2011 at various lead times

Convection-Allowing EnsemblesSFE2011 Results

from Tara Jensen, DTCfrom Dave Novak, WPC

SSEO favored over CAPS ensemble

• During SFE2012, the SSEO outperformed the SSEF and AFWA in terms of FSS for neighborhood probabilities of reflectivity ≥40 dBZ (bug later found in SSEF)

• Subjective ratings by the SFE2012 participants of HMF ensemble forecasts tended to favor the SSEO forecasts of UH over the SSEF and AFWA forecasts

Convection-Allowing EnsemblesSFE2012 Results

Hourly FSS Nprob Refl ≥40 dBZ 3-hr ensemble forecast ratings

(max, nprob) of UH

• The quality of the AFWA forecasts was less consistent than the SSEO forecasts

• Some UH forecasts from the AFWA ensemble were very good (bottom left) while others were poor (bottom right)

Convection-Allowing EnsemblesSFE2012 Results

• The impact of radar data assimilation in the CAPS SSEF was evident in the first 4 hours of the 12 UTC-initialized forecast.

• Otherwise, there was little statistical difference in the FSS among the 00 and 12 UTC SSEO and SSEF.

• Subjective ratings of 00Z ensemble HMFs were again favorable for the SSEO during SFE2013

Convection-Allowing EnsemblesSFE2013 Results

Hourly FSS Nprob Refl ≥40 dBZ

3-hr ensemble HMF forecast ratings

(max, nprob)

• Why is a “poor man’s ensemble” (i.e., SSEO) performing as well as formally designed ensembles? Let’s consider some aspects of configuration for convection-allowing ensembles– Single model vs. multi-model– Number of members– Initial conditions and IC/LBC perturbations– Physics

Convection-Allowing EnsemblesConfiguration

• Even with the same initial conditions, clustering often occurs by model core

• Generally more confident in a solution if different model cores are in agreement

• Is a multi-model approach a good way to address uncertainty in a convection-allowing ensemble?

Convection-Allowing EnsemblesConfiguration: Single model vs. multi-model

WRF-NMM WRF-ARW

SSEO

21Z on 16 April 2011

3-hr spaghetti plot of UH ≥25 m2s2

• Is the success of the SSEO a fortuitous balance of members with an underforecast bias and those with an overforecast bias (Row and Correia, 2014 AMS); not necessarily a result of using multiple model cores?

• Neighborhood verification of radar reflectivity reveals members with lower biases (e.g., NAM Nest) versus those with higher biases (e.g., NSSL-WRF)

• Biases will change with upcoming HiResW upgrade, so we may learn more this spring

Convection-Allowing EnsemblesConfiguration: Single model vs. multi-model

• For the way convection-allowing ensembles are currently configured, there does not appear to be a huge benefit to running more than ~10 members– Clark et al. (2011) objectively identified the “point of diminishing returns” at 9

members for 6-hr QPF at f30 and 2-km scale

Convection-Allowing EnsemblesConfiguration: Number of members

• Could run additional members to more effectively sample the forecast PDF, but is it worth the additional computational cost? Use a larger neighborhood?

from Clark et al. (2011)

• Currently, all members of the SSEO are initialized from the NAM (including two time-lagged members), so diversity primarily arises from multi-model/physics

• AFWA approach (single model, 3 different IC/LBCs) often leads to higher, overconfident probabilities

• SSEF approach not an obvious improvement over single, unperturbed IC (i.e. SSEO), suggesting ICs not properly perturbed at this scale

Convection-Allowing EnsemblesConfiguration: Initial conditions and IC/LBC perturbations

AFWA OBS

high probsnothing observed

• In four test runs for May 2013, Kong et al. (2014) found larger domain-average ensemble spread for multiple fields by directly using LBCs from SREF members rather than extracting perturbations and applying to the NAM (current strategy)

• NSSL-WRF ensemble this spring will directly utilize IC/LBCs from selected SREF members

Convection-Allowing EnsemblesConfiguration: Initial conditions and IC/LBC perturbations

Ense

mbl

e Sp

read

from Kong et al. (2014)

• Though spread in an ensemble with physics-only diversity is less than that from an ensemble that also includes IC/LBC perturbations, the contribution to spread from physics diversity can be large, including for instability fields (Clark et al. 2010)

Convection-Allowing EnsemblesConfiguration: Physics diversity

from Clark et al. (2010)

• Convection-allowing ensembles (~4-km grid spacing) can provide useful information to forecasters regarding the uncertainty of storm intensity, mode, location, timing, etc. in generating outlooks on Day 1; setting the stage for the continuous flow of probabilistic hazard information down toward the warning scale

• The SPC SSEO has proven to be as useful/skillful as formally designed convection-allowing ensembles, which raises questions about effective/proper configuration of these types of systems

• NSSL is running eight 00Z members this spring with only IC/LBC diversity directly from 21Z SREF members

• CAPS is planning to run an experimental 4-km EnKF SSEF system this year in near real-time for comparison with traditional SSEF forecasts

Convection-Allowing EnsemblesSummary