Slide 1

Evaluation and Comparison of Multiple Convection-Allowing Ensembles Examined in Recent HWT Spring Forecasting Experiments Israel Jirak, Steve Weiss, and Chris Melick Storm Prediction Center WoF Workshop, April 3, 2014

Slide 2

Convection-allowing ensembles ( ~ 4-km grid spacing) can provide important information to forecasters regarding the uncertainty of storm intensity, mode, location, timing, etc. on the outlook to watch scale These ensembles will play an important role in the ability of SPC to provide a more continuous flow of probabilistic hazard information in support of WoF 2 March 201229 June 2012 24-h neighborhood prob UH 25 m 2 /s 2 24-h ensemble max 10-m Wind Speed Convection-Allowing Ensembles Overview

Slide 3

An experimental real-time Storm-Scale Ensemble Forecast (SSEF) system has been produced for the NOAA Hazardous Weather Testbed (HWT) Spring Forecasting Experiment (SFE) by OU CAPS since 2007 through CSTAR funding Comprised of 4-km convection-allowing WRF and ARPS members: 2007: 10 members; 2008: 10 members; 2009: 20 members; 2010: 26 members (full CONUS to 30 h); 2011: 50 members (full CONUS to 36 h); 2012: 28 members; 2013: 27 members at 00Z (to 48 hours) and 8 members at 12Z (to 18 hours) Primarily examine explicit storm attributes, especially hourly maximum fields (HMFs):Updraft Helicity, Updraft Speed, 10-m AGL Wind Speed, 1-km AGL Reflectivity Ensemble display approaches include spaghetti plots, ensemble max, and neighborhood probabilities (more on next slide) SPC began processing deterministic high-resolution runs from EMC and NSSL as the Storm-Scale Ensemble of Opportunity (SSEO) in 2011 The 4-km AFWA ensemble was made available to SPC in 2012 Convection-Allowing Ensembles History

Slide 4

Traditional ensemble probabilities of HMFs from high-resolution models are not especially useful, owing to poor agreement among members at the grid point of these fields. Applying a binary neighborhood approach to a storm-scale ensemble improves the statistical results of HMFs in forecasting severe weather ROI=20-40 km Sigma=30 grid points Same approach can also be applied to observations (e.g., radar reflectivity) for verification purposes 03 May 2008 (Harless 2010) Convection-Allowing Ensembles Neighborhood Probabilities

Slide 5

HM Updraft Helicity > 25 m 2 s -2 SSEO 24-hr fcst valid 00Z 28 April

Slide 6

Convection-Allowing Ensembles Neighborhood Probabilities Grid-Point Probability HM Updraft Helicity >25 m 2 s -2 SSEO 24-hr fcst valid 00Z 28 April

Slide 7

Convection-Allowing Ensembles Neighborhood Probabilities 40-km Neighborhood Probability HM Updraft Helicity >25 m 2 s -2 SSEO 24-hr fcst valid 00Z 28 April

Slide 8

Convection-Allowing Ensembles Neighborhood Probabilities 40-km Neighborhood Smoothed Prob HM Updraft Helicity >25 m 2 s -2 SSEO 24-hr fcst valid 00Z 28 April

Slide 9

OU/CAPS Storm-Scale Ensemble Forecast (SSEF) System Since 2007; 36-hr forecasts from 00z; 12z runs began in 2013 Primarily WRF-ARW; 4-km grid spacing; forecasts to 60hrs in 2014 Multi-physics, multi-initial conditions: applies SREF perturbations to NAM ICs Advanced physics, 3DVAR & radar data assimilation; available for HWT/SFE SPC Storm-Scale Ensemble of Opportunity (SSEO) Since 2011; 36-hr forecasts at 00z & 12z; 7 members (2 time-lagged) Multi-model (ARW, WRF-NMM & NMM-B), multi-physics; ~4-km grid spacing Uses available deterministic models at SPC to process as an ensemble Basic data assimilation through NDAS; available year-round in SPC Air Force Weather Agency (AFWA) Ensemble Since 2012; 60-hr forecasts at 00z & 12z; 10 members; 4-km grid spacing Single model (WRF-ARW), multi-physics, multi-initial conditions Cold start from downscaled global model forecasts (GFS, UM, CMC) No data assimilation; available year-round in SPC Convection-Allowing Ensembles System Comparison

Slide 10

The Fractions Skill Score (FSS) was calculated for neighborhood probability (ROI=40 km; =40 km) of updraft helicity 25 m 2 s -2 for the SSEO/SSEF versus practically perfect hindcasts of preliminary severe weather reports (ROI=40 km; =120 km) during SE2011 The SSEO had higher fractions skill score (FSS) for neighborhood probabilities of UH 25 m 2 /s 2 during SFE2011 than the SSEF The number of members included in the SSEF did not seem to have a strong impact on the statistical results for neighborhood probabilities of UH during SE2011 when verified against severe weather reports Convection-Allowing Ensembles SFE2011 Results Nprob UH 25 m 2 /s 2 SSEO SSEF 24 memFSS = 0.84FSS = 0.68 3-hr [NPRS]:UH 25 m 2 s -2 valid 06Z on 02 June 2011 w/ verifying reports and practically perfect hindcast FSS 3-h periods SFE2011

Slide 11

Even for 6-h QPF, the SSEO received the highest subjective ratings relative to other operational and experimental models and ensembles during SFE2011 Statistically, the probabilistic QPF forecasts (>0.5) from the SSEO were typically as good as (if not better than) the SSEF during SFE2011 at various lead times Convection-Allowing Ensembles SFE2011 Results from Tara Jensen, DTC from Dave Novak, WPC SSEO favored over CAPS ensemble

Slide 12

During SFE2012, the SSEO outperformed the SSEF and AFWA in terms of FSS for neighborhood probabilities of reflectivity 40 dBZ (bug later found in SSEF) Subjective ratings by the SFE2012 participants of HMF ensemble forecasts tended to favor the SSEO forecasts of UH over the SSEF and AFWA forecasts Convection-Allowing Ensembles SFE2012 Results Hourly FSS Nprob Refl 40 dBZ 3-hr ensemble forecast ratings (max, nprob) of UH

Slide 13

The quality of the AFWA forecasts was less consistent than the SSEO forecasts Some UH forecasts from the AFWA ensemble were very good (bottom left) while others were poor (bottom right) Convection-Allowing Ensembles SFE2012 Results

Slide 14

The impact of radar data assimilation in the CAPS SSEF was evident in the first 4 hours of the 12 UTC-initialized forecast. Otherwise, there was little statistical difference in the FSS among the 00 and 12 UTC SSEO and SSEF. Subjective ratings of 00Z ensemble HMFs were again favorable for the SSEO during SFE2013 Convection-Allowing Ensembles SFE2013 Results Hourly FSS Nprob Refl 40 dBZ 3-hr ensemble HMF forecast ratings (max, nprob)

Slide 15

Why is a poor mans ensemble (i.e., SSEO) performing as well as formally designed ensembles? Lets consider some aspects of configuration for convection-allowing ensembles Single model vs. multi-model Number of members Initial conditions and IC/LBC perturbations Physics Convection-Allowing Ensembles Configuration

Slide 16

Even with the same initial conditions, clustering often occurs by model core Generally more confident in a solution if different model cores are in agreement Is a multi-model approach a good way to address uncertainty in a convection-allowing ensemble? Convection-Allowing Ensembles Configuration: Single model vs. multi-model WRF-NMM WRF-ARW SSEO 21Z on 16 April 2011 3-hr spaghetti plot of UH 25 m 2 s 2

Slide 17

Is the success of the SSEO a fortuitous balance of members with an underforecast bias and those with an overforecast bias (Row and Correia, 2014 AMS); not necessarily a result of using multiple model cores? Neighborhood verification of radar reflectivity reveals members with lower biases (e.g., NAM Nest) versus those with higher biases (e.g., NSSL-WRF) Biases will change with upcoming HiResW upgrade, so we may learn more this spring Convection-Allowing Ensembles Configuration: Single model vs. multi-model

Slide 18

For the way convection-allowing ensembles are currently configured, there does not appear to be a huge benefit to running more than ~10 members Clark et al. (2011) objectively identified the point of diminishing returns at 9 members for 6-hr QPF at f30 and 2-km scale Convection-Allowing Ensembles Configuration: Number of members Could run additional members to more effectively sample the forecast PDF, but is it worth the additional computational cost? Use a larger neighborhood? from Clark et al. (2011)

Slide 19

Currently, all members of the SSEO are initialized from the NAM (including two time-lagged members), so diversity primarily arises from multi-model/physics AFWA approach (single model, 3 different IC/LBCs) often leads to higher, overconfident probabilities SSEF approach not an obvious improvement over single, unperturbed IC (i.e. SSEO), suggesting ICs not properly perturbed at this scale Convection-Allowing Ensembles Configuration: Initial conditions and IC/LBC perturbations AFWAOBS high probs nothing observed

Slide 20

In four test runs for May 2013, Kong et al. (2014) found larger domain-average ensemble spread for multiple fields by directly using LBCs from SREF members rather than extracting perturbations and applying to the NAM (current strategy) NSSL-WRF ensemble this spring will directly utilize IC/LBCs from selected SREF members Convection-Allowing Ensembles Configuration: Initial conditions and IC/LBC perturbations Ensemble Spread from Kong et al. (2014)

Slide 21

Though spread in an ensemble with physics-only diversity is less than that from an ensemble that also includes IC/LBC perturbations, the contribution to spread from physics diversity can be large, including for instability fields (Clark et al. 2010) Convection-Allowing Ensembles Configuration: Physics diversity from Clark et al. (2010)

Slide 22

Convection-allowing ensembles (~4-km grid spacing) can provide useful information to forecasters regarding the uncertainty of storm intensity, mode, location, timing, etc. in generating outlooks on Day 1; setting the stage for the continuous flow of probabilistic hazard information down toward the warning scale The SPC SSEO has proven to be as useful/skillful as formally designed convection-allowing ensembles, which raises questions about effective/proper configuration of these types of systems NSSL is running eight 00Z members this spring with only IC/LBC diversity directly from 21Z SREF members CAPS is planning to run an experimental 4-km EnKF SSEF system this year in near real-time for comparison with traditional SSEF forecasts Convection-Allowing Ensembles Summary