DSST Short-Term Ensemble Planning Conference Call #15/3/20061 Short-Term Ensemble Planning (STEP)...

5/3/2006 1DSST Short-Term Ensemble Planning Conference Call #1

Short-Term Ensemble Planning (STEP)

Hydrologic Ensemble Prediction (HEP) Group

Hydrologic Science and Modeling Branch

Hydrology Laboratory

Office of Hydrologic Development

NOAA/National Weather Service


Objective

• Develop operations concept and R&D plan for short-term ensemble forecasting (FY06 AHPS), referred to herein as the Short-Term Ensemble Plan, or STEP

• Seek RFCs’ input to and review of STEP

• Report progress on short-term ensemble, including data assimilation (DA), activities


STEP Time-Table• Call #1 May 3 (Wed) 2-4pm (Eastern) - DJ to present "why" and "what" of the activity, the current status of and the

existing near-term plan for short-term ensemble R&D at OHD

• Between Calls 1 & 2 (through email) - RFCs to prepare a few to several slides on short-term ensemble needs (service as well as operational) and operations concept envisioned, and email them to DJ, DJ to assemble them into a single slideshow and email out

• Call #2 May 17 (Wed) 2-4pm (Eastern) - RFCs to present their slides, participants to discuss issues

• Between Calls 2 & 3 (through email) - DJ to produce a skeleton operations concept and R&D plan, RFCs to review and provide DJ with a list of issues, comments, suggestions, etc., DJ to assemble them into a single list and email out

• Call #3 Jun 28 (Wed) 2-4pm (Eastern) - DJ to provide an update, participants to discuss the issues toward consensus/resolution

• Between Calls 3&4 - DJ to generate a draft operations concept and R&D plan, RFCs to review and provide DJ with suggested revisions, DJ to incorporate them and email out

• Call #4 Aug 30 (Wed) 2-4pm (Eastern), if necessary - DJ to provide an update, participants to discuss remaining issues

• By Sep 30, 2006 - OHD and DSST to produce the final version of the plan


In this presentation

• Where are we going?– Products and services envisioned– Capabilities being developed

– Operations concept (?)

• Where are we now?– Current capabilities

• How do we get there?– Operations concept– R&D plan

• Science• Software engineering


A Vision for Operational Hydrologic Short-Term Ensemble Forecasting - Rob Hartman (AHPS Theme

Team, June 2005)

• For many years, NWS customers have benefited from probabilistic long-range water supply forecasts in the Western U.S. The potential benefits of accurate short-term probabilistic flood forecasts are very significant. This becomes obvious when one considers the cost of local emergency management activities.

• For several years now, OHD has been working on short-term ensemble prototypes. These efforts have been concentrated on developing ensemble inputs through model downscaling or simulation based on the joint distribution of forecasts and observations. Additional activities such as post processing and data assimilation (DA) have been identified. To date, however, the operational environment for short-term ensemble based probabilistic forecasts has not been described.

• The time-line for AHPS implementation of short-term probabilistic forecasts is such that the forecasting environment will certainly still involve OFS and the IFP. As such, operational ensemble forecasts must function in this environment. This will require several changes to the ESP and OFS architecture. Here is a typical scenario:


• The RFC is experiencing moderate flooding in several watersheds. A forecast update is due out at 03Z. Data come in after 00Z for the second period of the day, the 5-day QPF is updated by the HAS function. Input data are QC’d and the forecaster starts his/her IFP run to update guidance.

• The first segment is a flood forecast point. The output includes the single-value forecast as well as shaded regions that depict the probabilistic forecast with user definable regions (10% EP , 25% EP, Ensemble Mean, 75% EP, 90% EP, etc). The output indicates that there is a 40% chance that the river will rise to 3 feet over flood stage by noon tomorrow (mouse tracker interpolation). Three feet over flood stage is a critical stage for local mitigation. But wait, the simulation appears to be a bit screwy. Upon examination, the forecaster sees that bad precipitation data made it through the QC process. The MAP for the 18-00Z period needs to be increased by 50%. A MOD is made. The forecaster reruns the segment. The ensembles are regenerated and included in the display as before. The single-value forecast and probabilities shift slightly. The guidance looks reasonable. The phone rings. It’s the WFO and their local EM needs a forecast update right now. The forecaster selects “Issue guidance” from the pull-down menu and the system initiates the process that generates and issues the single-value and probabilistic guidance for this location.


• This scenario identifies several issues that are not currently supported with OFS and ESP. These include:• 1. The notion of carryover must to be re-examined so that ESP can be run interactively from any

point in time and for individual components of a forecast group. Ensemble generation must become interactive rather than a batch process. Performance must support interactive use (rerun small pieces in a few seconds).

• 2. The ESP process must thoroughly re-examine the notion of MODs and their impacts on short-term ensembles. Some believe that DA and automatic state updating is the only solution to avoid MODs. In the midst of forecasting, this is unrealistic. There will always be times when a forecaster needs to drive the model to the appropriate outcome. That’s why we have forecasters.

• 3. Statistical post-processing techniques must to be fast and interactive.• 4. Visualization tools must to be developed within the IFP framework to support ensemble and

probabilistic information.• 5. Ensemble and probabilistic information must be managed to facilitate the generation of products

and guidance.• 6. OFS does not write information back to the processed DB until the segment is exited. This

prevents product and information generation while looking at the IFP display.• Without doubt, we’ll find lots of other issues as we attempt operational implementation of short-term

probabilistic forecasts. As such, it is important to being addressing the issues and developing an operational prototype.


Improved accuracy, Reliable

uncertainty estimates,

Benefit-cost effectiveness

maximized

Where are we going?

A

B


The More Local the Forecast, the BetterInteractive forecast maps desired

“I think this would be great if you

could pull up your state, zero in on your county, and

on your river gauges, and then go to look at that. If this is the first

thing I look at and I can just keep

clicking and start zooming in that’s

great.”

When shown River Flood Outlook maps….

From NWS Hydrologic Services Program Probability Focus Results Briefing, Apr 27, 2006

A


Tailor Probabilistic Forecast Products More sophisticated users want more

“I might want to put it in my own GIS format or do something else with it.” We’re not saying it’s not useful...I don’t think that anybody is saying that. It’s just that some people would like to have more in addition to this.”

H

Least Likely

Forecast Legend

Likely H

Least Likely

Forecast Legend

Likely H

Least Likely

Forecast Legend

Likely

Most Likely

_Median Fcst

? ObservedStage

Flood Stage

“Give it to me in this format. Give it to me in a tabular, numbers driven format.”

From NWS Hydrologic Services Program Probability Focus Results Briefing, Apr 27, 2006

B


Short-Term ESP: Developing Capabilities

Flood mapping


Short-Term ESP: Developing Capabilities

Reflect meteorol. & hydrol. uncertainties

Ensemble traces of precipitation, temperature

Ensemble traces of streamflow

Ensemble traces of streamflow

QPF, QTF

ESP Pre-Processor

Hydrologic model

ESP Post-Processor

12

Correct bias, account for meteorological uncertainty

Correct bias, account for residual hydrol. uncertainty

Account for uncertainties in I.C., model parameters

Data assimilator, parametric uncertainty processor under development


Short-Term Ensemble Streamflow Prediction (ESP)

Current Experimental Operations

Variational Data Assimilation (VAR)


EPP2

GFS Subsystem

ESP

Day 1-14 Precipitation and

Temperature Ensemble Forecasts

Day 1-14 Precipitation and

Temperature Ensemble Hindcasts

Streamflow

Ensemble Hindcasting

Streamflow

Ensemble Hindcasts

Ensemble Verification and ValidationESP

Ensemble V&V Statistics

Ensemble Hindcaster

Ensemble V&V

Real-time Streamflow

Ensemble Forecasts

ENS_PRE_S

Current Ensemble Projects and their Relationships


AnalysisHistorical Data

Calibration(Hydrologic andHydraulic Models)

Calibration System (CS)

Real-Time Observed and Forecast Data

Operational Forecast System (OFS)

Ensemble Streamflow

Prediction (ESP) System

Hydrologicand

Hydraulic Models

Analysis and DataAssimilation

Hydrologic andHydraulic Models

short term forecasts

current states

StatisticalAnalyses

ProbabilisticShort term toExtended

time

Analysiswindow

Interactive Forecast Program (IFP)

InteractiveAdjustments

flow

National Weather Service River Forecast System


Challenges

• Develop and maintain coherence and transparency between ensemble and deterministic forecasting– Deterministic forecast is a special case of ensemble forecast

• Develop and integrate (automatic) data assimilation (DA) capabilities in deterministic and ensemble forecasting, and maintain complementarity and coherence with (manual) MOD

• Achieve optimal balance between statistical and dynamical approaches– Keep data requirements in check– Leverage collaborations with NCEP and others (

http://wwwt.emc.ncep.noaa.gov/gmb/ens/pred_mtgs.html)• Improve reliability and skill by removing biases and accurately accounting

for uncertainties• Develop and maintain verification capabilities that will guide us


Short-Term Ensemble Pre-Processor (EPP2) Data Availability Survey

RFC Format of 6-hr MAP Archive

(Period of Record)

Format of 6-hr QPF Archive

(Period of Record)

Format of 6-hr MAT Archive

(Period of Record)

APRFC Archive Database(02/2006?)

Archive Database(02/2006 w/ winter

gaps)

Archive Database(02/2006)

LMRFC Database(1996-2006)

Database (2004-2006)

Files (1996-2006)

None

NCRFC SHEF .B(2000-2006 w/ gaps)

OFS mod or xmrg(2000-2006 w/ gaps)

None

SERFC Shef-coded Text Product

(05/1997 - present)

Shef-coded Text Product

(05/1997 - present)

None

WGRFC OHD DATACARD(1/1996 – 10/2005)

QPS – SHEF(12/2003 – present)

None


Uncertainties in hydrologic forecasts

Uncertainty in initial conditions

Parametric uncertainty

Lead Time

Uncertainty

Meteorological/Input uncertainty

Reduced uncertainty due to pre-processing

Reduced uncertainty due to calibration

Reduced uncertainty due to data assimilation


Need your input

• Prepare a few to several slides that capture your vision of operations concept for short-term ensemble forecasting, including data assimilation (DA)– Include ideas for short-term ensemble products and

services– List existing and anticipated operational needs associated

with short-term ensemble forecasting, including DA– Identify key issues (science and modeling, software

engineering, products and services, etc.)• Email the slideshow to DJ ([email protected])

by May 12 (Fri), 2006


The following slides were presented at the HIC meeting in Jan 2006


Ensemble Preprocessor II (EPP2) -FY05 Accomplishments

• Developed a prototype EPP2– Integrated the prototype GFS Subsystem

• To improve reliability of HPC/RFC forecast-based short-term precipitation and temperature ensembles by utilizing GFS reforecast data from CDC

• To extend lead time of precipitation and temperature ensembles to Day 14 by utilizing GFS forecast from NCEP

– Developed hindcasting capability for precipitation and temperature ensembles– Enhanced ENS_PRE_S (see Slide 16) to enable ingestion of MAPX data for

single-value precipitation forecast (released to ABRFC)– Identified and fixed bugs in ENS_PRE_S

• Supported development and experimental implementation of the prototype GFS Subsystem at CNRFC


Ensemble Preprocessor II – FY06 Planned Activities

• Release and support EPP2• Test and Integrate updates to the GFS Subsystem• Continue science validation of the software through

diagnostic verification of ensemble hindcasts• Develop and implement methodology for explicit modeling of

precipitation intermittency to improve reliability of precipitation ensembles

• Complete HOSIP Stage 3


Ensemble Hindcaster -FY05 Accomplishments

• Developed a prototype software for hindcasting of precipitation, temperature and streamflow ensembles– Uses EPP2 for hindcasting of precipitation and temperature

ensembles from• HPC/RFC forecast of MAP and MAT• Historical MAP and MAT time series (“raw” climatology)• “Resampled” historical MAP and MAT time series

– Uses carryover from FS5 files and ESP for hindcasting of streamflow ensembles

• Generated hindcasts of precipitation, temperature and streamflow ensembles for 5 and 1 segments in AB- and CNRFCs, respectively, and precipitation and temperature ensembles for 10 segments in MARFC


Ensemble Hindcaster – FY06 Planned Activities

• Develop and implement additional functionalities to support new verification requirements and capabilities– Generation of “reference” hindcasts for separation of input

and hydrologic uncertainties• Evaluate the sufficiency of existing archiving capabilities for

ensemble forecasting through interaction with the RFCs– Propose a plan for new needed archiving capabilities

• Improve robustness of the prototype software• Release and support the experimental version of the

Ensemble Hindcaster• Complete HOSIP Stage 3


Ensemble Verification & Validation – FY05 Accomplishments

• Developed a prototype software, the Ensemble Verification Program (EVP), for verification of ensemble hind-/forecasts of precipitation, temperature and streamflow

• EVP consists of– A suite of data processing and science algorithms that

• processes observed and hind-/forecast data and• calculates a suite of verification statistics

– A suite of R scripts for graphical display of verification statistics and data• scatter plots• bias ratio, mean error, root mean square error, correlation coefficient• Brier Score (BS), Brier Skill Score (BSS), reliability diagram, Ranked

Probability Score (RPS), Ranked Probability Skill Score (RPSS), and Relative Operating Characteristic (ROC)

• Generated and preliminarily examined verification results for 3 RFCs (Demargne et al. 2006)


Ensemble Verification & Validation - FY06 Planned Activities

• Improve ensemble verification capabilities (in coordination with Hydrologic Verification at RFCs)– Improve and expand science algorithms in the prototype Ensemble

Verification Program (EVP)• Calculation of aggregate (e.g. for multiple basin), conditional (e.g. on

occurrence of precipitation), and reference (e.g. against climatology, persistence, deterministic) verification statistics

• Estimation of confidence interval (including implementation of the Univ. of Iowa’s work)

– Improve R-based ensemble verification graphics capabilities– Release and support the experimental version of the Ensemble Verification

Program (EVP)• Complete HOSIP Gate 3• Identify a path for proper error analysis of deterministic forecasts

– Includes work on the operations concept, requirements development, and operational development plan for the "raw model" forecasts


Example: Brier Score and its Decomposition


Example: RPS & RPSS


Example: Reliability Diagram


Example: Bias Ratio, ME, RMSE, Correlation


Example: Relative Operating Characteristic


Example: Scatter Plots


GFS Subsystem – FY05 Accomplishments

• Produced prototype GFS Subsystem– Generates 6hr precipitation and temperature ensembles– Uses GFS ensemble mean and/or RFC fcsts– Accounts for temporal scale-dependencies

• Implemented at CNRFC for test basins


EPP2 GFS Subsystem – FY06 Planned Activities

• Funded primarily by GAPP Core Project (Restrepo, PI, Schaake, Seo, Co-PI’s)

• Produce Operational Prototype– In progress at CNRFC for test basins

• Support Additional RFC’s– Documentation– Expand CNRFC application to most basins

• Produce Gridded Prototype• HOSIP

– Complete Gate 3– Gridded version through Gates 1&2


VAR Verification, Validation & Enhancement – FY05

Accomplishments• Completed development of the prototype modeling tool for

Site Specific, AB_OPT (Kuzmin et al. 2006)• Improved low-flow performance of prototype VAR• Continued experimental operation of prototype VAR at

WGRFC• Completed comparative evaluation of prototype VAR at

WGRFC (see Slide 32 for summary of conclusion and recommendations, a larger-sample evaluation is pending)


Summary of Comparative Evaluation of prototype VAR at WGRFC (Seo et al. 2006)

Automatic state updating via VAR compares favorably with state updating via run-time modification (MOD) by human forecasters. Out of 22 basins, VAR performed better and worse for 13 and 3 basins, respectively, over the period of experiment.

Factors limiting the performance of VAR include large structural and/or parametric errors in soil moisture accounting and routing models, and lack of basin-specific modeling of uncertainty, particularly in streamflow observation.

The comparative conclusions drawn above are based on a limited size sample from a dry season. For confirmation based on a large sample and evaluation conditional on flow magnitude (e.g. above or below action and flood stages), it is recommended that the experiment continue.

Experience from the experimental operation indicates that the DA-aided forecasts and the verifying observations must be visualized to support error analysis and uncertainty assessment by the forecasters. It is recommended that the DA-aided forecasts be visualized as time-lagged ensembles.

In addition to improving model physics, additional improvements are needed for VAR in the following areas; modeling of uncertainty in observations, in particular streamflow, separation of phase and amplitude errors and explicit accounting of them in DA, and use of a priori information on model soil moisture states.

Operational implementation of VAR must consider strengths and weaknesses of automatic DA and manual MOD by human forecasters, and exploit the complementarities between them. The recommended strategy is that automatic DA operate in a continuous mode to produce baseline data-assimilated forecasts, and human forecasters analyze the forecasts generated with and without DA, and accept, reject or, if necessary, apply additional MOD to produce value-added forecasts.


Data Assimilator (DA) for Distributed SAC and Kinematic Wave Routing (of HL-RDHM) – FY05

Accomplishments

• Reviewed, formulated, coded and evaluated different techniques and approaches for operationally viable DA for distributed hydrologic models– Determined a hybrid approach of using variational assimilation

(4DVAR) and ensemble Kalman filter (ENS-KF) to be most promising• Developed prototype codes for variational assimilation, with varying

degrees of complexity, of streamflow, precipitation and PE data into distributed SAC and kinematic wave routing models of HL-RDHM

• Carried out initial evaluation of 4DVAR and preliminary investigation into ENS-KF application


WTTO2 in ABRFC WTTO2 channel network


Data Assimilation – FY06 Planned Activities

• Funded by GAPP Core Project– Continue enhancement, experimental operation and

evaluation of VAR at WGRFC, Complete Gate 3– Continue development of DA with distributed models,

Complete HOSIP Stage 3• GAPP External Project (Clark, Slater, Seo, PI’s)

– “Incorporating knowledge of observational uncertainties in streamflow forecasting applications for the western USA mountains”

– Keep RFCs informed


References• Demargne, J., S. Cong, L. Wu, D.-J. Seo, J. Schaake, R. Hartman, W. Lawrence, and

E. Pryor, 2006, Verification of experimental hydrometeorological and hydrological ensemble forecasts in the U.S. National Weather Service, manuscript under preparation.

• Duan, Q., S. Sorooshian, and V. K. Gupta, 1992, Effective and efficient global optimization for conceptual rainfall-runoff models. Water Resour. Res., 28(4), 265-284.

• Kuzmin, V., D.-J. Seo, and V. Koren, 2006, Fast and efficient optimization of hydrologic model parameters using a priori estimates and Stepwise Line Search, in internal review for submission to J. Hydrol.

• Schaake, J., J. Demargne, M. Mullusky, E. Welles, L. Wu, H. Herr, X. Fan, and D.-J. Seo, 2006, Precipitation and temperature short-term ensemble forecasts from existing operational single-value forecasts, submitted to the HEPEX Special Issue of Hydrology and Earth System Sciences.

• Seo, D.-J., L. Cajina, R. Corby, and T. Howieson, 2006, Automatic state updating for operational streamflow forecasting via variational data assimilation, to be manuscript under preparation for submission to ASCE J. Hydrol. Eng.

DSST Short-Term Ensemble Planning Conference Call #15/3/20061 Short-Term Ensemble Planning (STEP)...

Documents

Transcript of DSST Short-Term Ensemble Planning Conference Call #15/3/20061 Short-Term Ensemble Planning (STEP)...