Testbeds and Projects with Ongoing Ensemble Research:

Hydrometeorology Testbed (HMT) Hazardous Weather Testbed (HWT) Hurricane Forecast Improvement Project

(HFIP)

DET Collaboration with Other Projects: Past and Future

Ed Tollerud 1 and Tara Jensen 2

1 NOAA/GSD, Boulder, CO 2 NCAR/RAL, Boulder, CO

Acknowledgements: Tim Schneider, Marty Ralph, Gary Wick, Rob Cifelli, Dave Reynolds, Steve Weiss, Barbara Brown, Jack Kain, Ming Xue, Fanyou Kong, Adam Clark, Patrick Marsh, Mike Coniglio, Russ Schneider, Evan Kucera, Tom Hamill, Tony Eckel, Louisa Nance, Mark DeMaria, James Franklin, Ed Rappaport, Mike Fiorino and many others

MotivationMany testbeds now have an ensemble componentRobust and objective evaluation of promising

techniques neededDetermine potential utility for operations

Model A

Model C

Model E

Model B

Ensemble D

Forecast

Exercise

Testbed Operations

General Format of Many Testbeds

Research

GoalsLeverage off work done in other

testbedsExtend understanding of their

resultsHelp in planning for next year or

seasonFacilitate faster Research to Ops

(R2O)

Effo

rt

Time

Fore

cast

Exe

rcis

e

DET Retrospective Runs

There is much work to be done in the off-season -DET might be able to help

Most effort expended around R2O ForecastExercise

Recent Activities

HWT Participants Evaluate Models

Traditional Scores for HWT ensemble members and baselines

Object-Oriented Evaluation Of HMT members

2010 DTC/HMT Ensemble Activities

High-resolution 9 member WRF regional ensemble for HMT-West Winter Exercises ◦First DTC ensemble evaluation◦Individual deterministic models◦Simple ensemble mean calculated and evaluated

30-day Summary Scoring Displays Baseline Comparisons (with GFS)

2010 HMT Ensemble NWP9 member ensemble

◦Grid: 9 km outer - 3 km inner◦LBC: GFS Ensemble Members

North American domain 80 km resolution

◦IC: LAPS

• January Summary, ETS

• 24h forecast, full domain

• Verified at 24h gage sites

• Segregated by threshold

• Dramatic performance hit at higher thresholds

• No real ensemble member advantages in statistics but large inter-quartile range

Real-time QPF verification for HMT-West

2011 HMT Ensemble NWPDET 1st Benchmark (NCEP) will

hopefully be run for experiment◦ North American Domain◦ 20 km resolution◦ IC “cycling” option

9 member ensemble◦ Grid: 9 km outer – 3 km inner (or potentially

finer resolution)◦ IC: LAPS and/or Cycling◦ LBC: DET 1st Benchmark (NCEP)

Future DTC/HMT Activities◦Use DET Ensemble Configuration

Module to set-up and run◦First use of MET v3.0

including Ensemble-Stat Tool◦Additional Baselines◦Ensemble-based

Spatial Verification◦Estimate Uncertainty in

Verification Statistics

What can HMT do for DET• HMT used for test domain for DET

Module 2 (IC/LBC) Benchmark• Workflow Procedures will backbone of

DET system• Input data options for initialization and

verification (eg., QPF)• Ensemble Products Algorithms for

inclusion in Module 5 (Products/Display)

• Exploration of Spatial Verification Applications

What can DET do for HMTHelp determine optimal

ensemble configurationEvaluate Ensemble Products

◦Visualization utilities (Displays)◦Probability Products

State-of-the-art probabilistic verification measures

HWT 2008 •Introduce Objective

Evaluation

HWT 2009

•Realtime system•Address scientific

question

HMT 2010

•1st Ensemble evaluation

•Satellite data into MET

HWT 2010

•Add Ensemble methods

•AWC/HPC present

HMT 2011

•Refine Ensemble methods

•Data Impact Studies

ModelEvaluatio

nTools (MET)

DTCEnsembleTestbed(DET)

etc…

2010 DTC/HWT Ensemble ActivitiesEvaluated 26 member CAPS Storm-Scale

Ensemble Forecast System during 2010 Spring Experiment◦ Baselines: HRRR, SREF, NAM◦ Ensemble Products

generated by CAPS– using 15 memberswith radarassimilation

◦ First plots of BrierScore and Area Under Curve

NSSL Q2 QPE

CAPS SimplePQPF

SREF SimplePQPF

CAPSPN PQPF

NAM QPF

SSEFPMQPF

PQPF QPF

•Evaluation of Simple and Derived Ensemble Products: Probability fields using Neighborhood Method Probability matching method for QPF

Future DTC/HWT Activities◦ DET participates in planning of CAPS ensemble

constituents potentially with DET Ensemble Configuration Module, IC/LBC, and Physics Modules

◦ Real-time verification performed using DET Verification Module (HWT-DTC Collaboration)

◦ HWT products evaluated in DET Module 5 (Products/Display)

◦ Feedback to model developers

RETOP

How DET may help HWTModule 1 - Work with CAPS to define optimal

ensemble member configurationModule 2 – Work with DTC GSI task to identify

GSI-based IC perturbation capabilities for CAPS ensemble

Module 3 – Evaluate individual members in 2010 dataset to provide suggestions for physics improvements

Module 4 – Implement bias correction technique that requires limited historical data

Module 5 – Evaluate ensemble products in 2010 dataset

Module 6 – Used in all evaluation

HFIP Ensemble WorkMany institutions involvedMany approaches being investigated

Global Ensembles◦ Single Models with IC/LBC

and physics perturbations (GFS, NOGAPS, and FIM)

Regional Ensembles◦ Multi-model ensemble made up of regional

models run by different organizations statistically post-processed (FSU)

◦ Ensembles initialized from EnKF (PSU)Consensus ensembles (NHC)

How DET might contribute to HFIP

• DET could be a testing ground for most promising ensemble techniques

• Retrospective testing to address specific questions:

• Module 2 – Modify near storm environment or storm position?

• Module 3 – Which cumulus schemes provide best spread?

• Module 4 – Which is best way to bias correct track and intensity?

• Module 5 – How do we best display of uncertainty in track and intensity?

• Module 6 – Does spatial (object oriented) verification of ensembles improve understanding of output?

SOME IDEAS FOR THEFUTURE…

MODULE 1:Ensemble

Configuration

MODULE 2:IC/LBC Perturbations

MODULE 3:Physics Perturbations MODULE 4:

Statistical Post-Processing

MODULE 5:Products/Display

MODULE 6:Verification

Mostly Retrospective--------------------------------------

1) ROBUST Test & Evalof promising techniques uncovered

during forecast exercises

Use Near Real-time & Retrospectively

----------------------------------------1) Immediate Feedback during

forecast exercises2) ROBUST Test & Eval

of promising techniques uncoveredduring forecast exercises

Concept of Leveraging DET Modules

Questions for Working Group:

Should DET consider capability for realtime (or near realtime) demonstrations?

Where do we place the “overlap area” between DET and other ensemble-related testbeds? What does that area include?

What are the most critical testbed collaboration issues to focus on in the next year or two?

What interactions with other testbeds beyond HMT and HWT should be explored in the next few years?

HWT 2012 •Pre-Experiment

Evaluation

HWT 2011

•Ensemble membership

•Evaluate New Products

HFIP 2011

•Ensemble Membership

•Statistical Post-Processing

HMT 2012

•Ensemble Methods for Spatial Verification

HMT 2011

•Refine Ensemble Verification methods

•Data Impact Studies

DTCEnsembleTestbed

etc…

ModelEvaluatio

nTools (MET)