Watersheds and the Hydrologic Cycle. The Global Hydrologic Cycle.
Hydrologic Verification Verification of Deterministic River Stage Forecasts
description
Transcript of Hydrologic Verification Verification of Deterministic River Stage Forecasts
![Page 1: Hydrologic Verification Verification of Deterministic River Stage Forecasts](https://reader034.fdocuments.us/reader034/viewer/2022051401/568143eb550346895db070af/html5/thumbnails/1.jpg)
Hydrologic Verification
Verification of Deterministic River Stage Forecasts
Edwin Welles
OST SeminarJune 6, 2007
![Page 2: Hydrologic Verification Verification of Deterministic River Stage Forecasts](https://reader034.fdocuments.us/reader034/viewer/2022051401/568143eb550346895db070af/html5/thumbnails/2.jpg)
Outline of This Talk● Introductory comments about river forecasts.
– Is there value in verifying stage forecasts?
● Results of evaluating limited sample of NWS stage forecasts.– Proposal: Hydrologists should verify their forecasts.
● Results of a hindcasting study on headwater basins.– Proposal 2: We can use verification information to guide our
forecast process development.
● Standardized Hydrologic Verification– Just advertising
2
![Page 3: Hydrologic Verification Verification of Deterministic River Stage Forecasts](https://reader034.fdocuments.us/reader034/viewer/2022051401/568143eb550346895db070af/html5/thumbnails/3.jpg)
The beneficial .....and not so beneficial uses of water.
3
![Page 4: Hydrologic Verification Verification of Deterministic River Stage Forecasts](https://reader034.fdocuments.us/reader034/viewer/2022051401/568143eb550346895db070af/html5/thumbnails/4.jpg)
Impacts of Floods and Droughts
– Floods kill about 100 people a year
– Approximate annual economic losses: Floods - $5 billion
Droughts - $6-8 billion
Flood
Hurricane
Winter Storm
Lightning
Tornado
Cold
Heat
Average Annual Deaths (1990-99): 5021414
HeatHeat
FloodFlood
193193
9999
5555
5757
5858
2626
4
![Page 5: Hydrologic Verification Verification of Deterministic River Stage Forecasts](https://reader034.fdocuments.us/reader034/viewer/2022051401/568143eb550346895db070af/html5/thumbnails/5.jpg)
Mitigating the Impacts
● Forecasts and data can help us manage our water resources and mitigate the impacts of floods.
● NWS started issuing hydrologic forecasts in 1890.
5
![Page 6: Hydrologic Verification Verification of Deterministic River Stage Forecasts](https://reader034.fdocuments.us/reader034/viewer/2022051401/568143eb550346895db070af/html5/thumbnails/6.jpg)
Data and Forecasts to Meet Diverse Needs
Radar Data
River Gage Data Weather Observations
Snow Cover/Melt DataPrecipitation
Forecasts
Climate PredictionsReservoirReleases
PrecipitationEstimates
Satellite Data
The Approach to River Forecasting
Models
6
![Page 7: Hydrologic Verification Verification of Deterministic River Stage Forecasts](https://reader034.fdocuments.us/reader034/viewer/2022051401/568143eb550346895db070af/html5/thumbnails/7.jpg)
Forecast Modeling Structure
INTERFLOWSURFACERUNOFF
INFILTRATIONTENSION
TENSION TENSION
PERCOLATION
LOWERZONE
UPPERZONE
PRIM ARYFREE
SUPPLE-M ENTAL
FREE
RESERVED RESERVED
FREE
EVAPOTRANSPIRATION
BASEFLOW
SUBSURFACEOUTFLOW
DIRECTRUNOFF
INTERFLOWSURFACERUNOFF
INFILTRATIONTENSION
TENSION TENSION
PERCOLATION
LOWERZONE
UPPERZONE
PRIM ARY
FREE
SUPPLE-
M ENTAL
FREE
RESERVED RESERVED
FREE
EVAPOTRANSPIRATION
BASEFLOW
SUBSURFACEOUTFLOW
DIRECTRUNOFF
INTERFLOWSURFACERUNOFF
INFILTRATIONTENSION
TENSION TENSION
PERCOLATION
LOWERZONE
UPPERZONE
PRIM ARY
FREE
SUPPLE-M ENTAL
FREE
RESERVED RESERVED
FREE
EVAPOTRANSPIRATION
BASEFLOW
SUBSURFACEOUTFLOW
DIRECTRUNOFF
INTERFLOWSURFACERUNOFF
INFILTRATIONTENSION
TENSION TENSION
PERCOLATION
LOWERZONE
UPPERZONE
PRIM ARY
FREE
SUPPLE-
M ENTAL
FREE
RESERVED RESERVED
FREE
EVAPOTRANSPIRATION
BASEFLOW
SUBSURFACEOUTFLOW
DIRECTRUNOFF
INTERFLOWSURFACERUNOFF
INFILTRATIONTENSION
TENSION TENSION
PERCOLATION
LOWERZONE
UPPERZONE
PRIM ARY
FREE
SUPPLE-M ENTAL
FREE
RESERVED RESERVED
FREE
EVAPOTRANSPIRATION
BASEFLOW
SUBSURFACEOUTFLOW
DIRECTRUNOFF
INTERFLOWSURFACERUNOFF
INFILTRATIONTENSION
TENSION TENSION
PERCOLATION
LOWERZONE
UPPERZONE
PRIM ARYFREE
SUPPLE-
M ENTALFREE
RESERVED RESERVED
FREE
EVAPOTRANSPIRATION
BASEFLOW
SUBSURFACEOUTFLOW
DIRECTRUNOFF
Snow
Runoff
Overland FlowRouting
ChannelRouting
Reservoir
...for many rivers,
with many reservoirs.
Elevation zoneboundary
ForecastPoint
RatingCurves
7
![Page 8: Hydrologic Verification Verification of Deterministic River Stage Forecasts](https://reader034.fdocuments.us/reader034/viewer/2022051401/568143eb550346895db070af/html5/thumbnails/8.jpg)
The Forecast Process
INTERFLOWSURFACERUNOFF
INFILTRATIONTENSION
TENSION TENSION
PERCOLATION
LOWERZONE
UPPERZONE
PRIM ARYFREE
SUPPLE-M ENTAL
FREE
RESERVED RESERVED
FREE
EVAPOTRANSPIRATION
BASEFLOW
SUBSURFACEOUTFLOW
DIRECTRUNOFF
BULLETINFLOOD WARNINGNATIONAL WEATHER SERVICE SHREVEPORT LA1033 AM CDT WED APR 13 2005
Data Assimilation
Forecast:p, T, PE, Q
res, Q
Observed:p, T, PE, Q
res, Stg
Verification
8
![Page 9: Hydrologic Verification Verification of Deterministic River Stage Forecasts](https://reader034.fdocuments.us/reader034/viewer/2022051401/568143eb550346895db070af/html5/thumbnails/9.jpg)
Where’s the …..?9
![Page 10: Hydrologic Verification Verification of Deterministic River Stage Forecasts](https://reader034.fdocuments.us/reader034/viewer/2022051401/568143eb550346895db070af/html5/thumbnails/10.jpg)
● … Verification
● Little verification of hydrologic forecasts has been conducted to date.
● My Main Point – Need to fill this void in both
Research and Operations.● So let’s look at some
verification metrics
10
Where’s the …..?
![Page 11: Hydrologic Verification Verification of Deterministic River Stage Forecasts](https://reader034.fdocuments.us/reader034/viewer/2022051401/568143eb550346895db070af/html5/thumbnails/11.jpg)
Description of the Data
● Two sets of River Stage forecasts from NWS RFCs
● OK Dataset– 4 Locations in Oklahoma– 1993 to 2002– 1 to 4 basins above the forecast point– Response time measured in hours
● MM Dataset– 11 locations along the mainstem of the Missouri river– 1983 to 2002 – 500 to 1000 basins above the forecast point– Response time measured in days
● Generated a Persistence forecast as a reference– Observation at time of forecast persisted into the future
11
![Page 12: Hydrologic Verification Verification of Deterministic River Stage Forecasts](https://reader034.fdocuments.us/reader034/viewer/2022051401/568143eb550346895db070af/html5/thumbnails/12.jpg)
Acrobat Document
12
![Page 13: Hydrologic Verification Verification of Deterministic River Stage Forecasts](https://reader034.fdocuments.us/reader034/viewer/2022051401/568143eb550346895db070af/html5/thumbnails/13.jpg)
13
![Page 14: Hydrologic Verification Verification of Deterministic River Stage Forecasts](https://reader034.fdocuments.us/reader034/viewer/2022051401/568143eb550346895db070af/html5/thumbnails/14.jpg)
Acrobat Document
14
![Page 15: Hydrologic Verification Verification of Deterministic River Stage Forecasts](https://reader034.fdocuments.us/reader034/viewer/2022051401/568143eb550346895db070af/html5/thumbnails/15.jpg)
Initial Observationsfrom this limited sample
• Day 1 and Day 2 forecasts are accurate and skillful (as compared to persistence).
• Little skill in day 3• Little change over the periods of record.
• Need to conduct a more complete study to establish a comprehensive baseline so we can answer this basic question.– What is the skill of hydrologic forecasts?
15
![Page 16: Hydrologic Verification Verification of Deterministic River Stage Forecasts](https://reader034.fdocuments.us/reader034/viewer/2022051401/568143eb550346895db070af/html5/thumbnails/16.jpg)
V E R I F I C A T I O N V E R I F I C A T I O N V E R I F I C A T I O N V E R I F I C A T I O N V E R I F I C A T I O N V E R I F I C A T I O N V E R I F I C A T I O N V E R I F I C A T I O N
Leverage Data and Systems from NOAA and Collaborators
Valuable Data and Forecasts to Satisfy Diverse Customer Needs
Radar Data
River Gage Data Weather Observations
Snow Cover/Melt DataPrecipitation
Forecasts
Climate PredictionsReservoirReleases
PrecipitationEstimates
Satellite Data
Improving The Approach to River Forecasting
Models
16
![Page 17: Hydrologic Verification Verification of Deterministic River Stage Forecasts](https://reader034.fdocuments.us/reader034/viewer/2022051401/568143eb550346895db070af/html5/thumbnails/17.jpg)
Some More Basic Questionsfor which we do not have answers
● How is new science improving the forecasts?
● What are the largest sources of error in the forecasts?
● What should be done to improve the forecasts?
How can verification can help us answer these questions?
17
![Page 18: Hydrologic Verification Verification of Deterministic River Stage Forecasts](https://reader034.fdocuments.us/reader034/viewer/2022051401/568143eb550346895db070af/html5/thumbnails/18.jpg)
A Hindcasting Experiment to Analyze Sources of Error
● Generated hindcasts – Three headwater basisns in Oklahoma and Missouri.– Out to three day lead times – Four years (1997 to 2000)
● Twelve total scenarios using…– 2 calibrations
● A “good” and an “a priori” calibration– 2 state updating methods
● Updated initial conditions and not-updated initial conditions– 3 QPF scenarios
● Perfect QPF– The observations
● Actual QPF – Computed QPF for 24 hours and then zero for days 2 and 3
● Zero QPF– Zero for all 3 days
18
![Page 19: Hydrologic Verification Verification of Deterministic River Stage Forecasts](https://reader034.fdocuments.us/reader034/viewer/2022051401/568143eb550346895db070af/html5/thumbnails/19.jpg)
Analysis Method● For each scenario
– Computed standard verifiction metrics on 4 subsets of the data
● Sorted into high and low stages● Sorted by observations and by forecasts
– Sorting by observations – Discrimination – Sorting by forecasts – Reliability
– Presenting just RMSE for High stage discrimination
● Compared scenarios with simple differences between the metrics.
● Evaluated error in the QPF also.
19
![Page 20: Hydrologic Verification Verification of Deterministic River Stage Forecasts](https://reader034.fdocuments.us/reader034/viewer/2022051401/568143eb550346895db070af/html5/thumbnails/20.jpg)
Comparing the Calibrations
Discrimination ReliabilityRMSE ME R RMSE ME R
Low: Calibrated 1.3 0.2 0.85 1.3 0.2 0.85Low: Uncalibrated 3.5 -0.1 0.51 2.6 -0.6 0.50
High: Calibrated 2.8 0.0 0.65 3.0 0.0 0.75High: Uncalibrated 5.8 3.2 0.55 11.5 10.5 0.35Table 8. Summary statistics to compare the model calibrations.
20
![Page 21: Hydrologic Verification Verification of Deterministic River Stage Forecasts](https://reader034.fdocuments.us/reader034/viewer/2022051401/568143eb550346895db070af/html5/thumbnails/21.jpg)
ME (byobs)
RMSE(by obs)
ME (byfcst)
RMSE(by fcst)
Samples(by obs)
Samples(by fcst)
Actual QPF <=25 mm 0.2 2.5 0.1 3.0 31800 31920Zero QPF <=25 mm -0.7 2.6 -0.7 3.5 31800 31946
Actual QPF >25 mm -22.8 25.7 15.5 20.3 146 26Zero QPF >25 mm -33.2 34.7 NA NA 146 NA
Table 9. The actual QPF compared to the zero QPF for the three hindcast basins.
ME(by obs)
RMSE(by obs)
ME (byfcst)
RMSE(by fcst)
FAR POD
National <=25 mm 0.1 2.4 0.1 2.4 na na
Hindcast basins <=25 mm 0.2 2.5 0.1 3.0 na naNational >25 mm -24.7 29.1 16 23 0.76 0.10
Hindcast basins >25 mm -22.8 25.7 15.5 20.3 0.77 0.04
Table 10. National Precipitation Verification Unit QPF statistics and the QPF statisticsfor the 3 hindcast basins.
The Skill of the Input Forecasts 21
![Page 22: Hydrologic Verification Verification of Deterministic River Stage Forecasts](https://reader034.fdocuments.us/reader034/viewer/2022051401/568143eb550346895db070af/html5/thumbnails/22.jpg)
Comparing the State Updating Scenarios
● State updating appears to bring similar skill to the hindcasts as the calibration in the early periods.
● QPF and state updating skill appear independent.
22
![Page 23: Hydrologic Verification Verification of Deterministic River Stage Forecasts](https://reader034.fdocuments.us/reader034/viewer/2022051401/568143eb550346895db070af/html5/thumbnails/23.jpg)
23
![Page 24: Hydrologic Verification Verification of Deterministic River Stage Forecasts](https://reader034.fdocuments.us/reader034/viewer/2022051401/568143eb550346895db070af/html5/thumbnails/24.jpg)
Comparing theCalibration Scenarios
● State updating appears to bring similar skill to the hindcasts as the calibration in the early periods.
● Improving the calibration may degrade forecast skill depending upon the QPF characteristics.
● Need good QPF to realize benefits of improved calibration.
24
![Page 25: Hydrologic Verification Verification of Deterministic River Stage Forecasts](https://reader034.fdocuments.us/reader034/viewer/2022051401/568143eb550346895db070af/html5/thumbnails/25.jpg)
25
![Page 26: Hydrologic Verification Verification of Deterministic River Stage Forecasts](https://reader034.fdocuments.us/reader034/viewer/2022051401/568143eb550346895db070af/html5/thumbnails/26.jpg)
Comparing the QPF Scenarios
● Using some QPF almost always improves the forecasts (1 exception) for the High Stages– Not so for the low stages.
● Can capitalize on the improved QPF even with poor calibration.– But get more improvement with better calibration
26
![Page 27: Hydrologic Verification Verification of Deterministic River Stage Forecasts](https://reader034.fdocuments.us/reader034/viewer/2022051401/568143eb550346895db070af/html5/thumbnails/27.jpg)
27
![Page 28: Hydrologic Verification Verification of Deterministic River Stage Forecasts](https://reader034.fdocuments.us/reader034/viewer/2022051401/568143eb550346895db070af/html5/thumbnails/28.jpg)
28
![Page 29: Hydrologic Verification Verification of Deterministic River Stage Forecasts](https://reader034.fdocuments.us/reader034/viewer/2022051401/568143eb550346895db070af/html5/thumbnails/29.jpg)
Improving The Forecast Process
INTERFLOWSURFACERUNOFF
INFILTRATIONTENSION
TENSION TENSION
PERCOLATION
LOWERZONE
UPPERZONE
PRIM ARYFREE
SUPPLE-M ENTAL
FREE
RESERVED RESERVED
FREE
EVAPOTRANSPIRATION
BASEFLOW
SUBSURFACEOUTFLOW
DIRECTRUNOFF
BULLETINFLOOD WARNINGNATIONAL WEATHER SERVICE SHREVEPORT LA1033 AM CDT WED APR 13 2005
Data Assimilation
Forecast:p, T, PE, Q
res, Q
Observed:p, T, PE, Q
res, Stg
Verification
Verification
VerificationVerification
Verification
Verification
Verification
Verification
Verification
Verification
Help needed for > day 1 forecasts
The hindcasts tell us ....
29
Critical for < day 1 forecasts
![Page 30: Hydrologic Verification Verification of Deterministic River Stage Forecasts](https://reader034.fdocuments.us/reader034/viewer/2022051401/568143eb550346895db070af/html5/thumbnails/30.jpg)
Standardized Verification for Hydrologic Forecasts
U N W SNWSU
Without Verification With Verification
It is hard to communicate without a common language. StandardizedVerification is the common language forecasters and researchers need.
This Professor
has a really
good idea.
This one used
verification to
explain her idea.
30
![Page 31: Hydrologic Verification Verification of Deterministic River Stage Forecasts](https://reader034.fdocuments.us/reader034/viewer/2022051401/568143eb550346895db070af/html5/thumbnails/31.jpg)
Standardized Verification for Hydrologic Forecasts
● Supports research by identifying needs AND by clarifying the value of results.
● Supports operational agencies by defining acceptable methods.
● They are expected to evolve, but we must start somewhere.
● Use peer review to establish validity.
31
![Page 32: Hydrologic Verification Verification of Deterministic River Stage Forecasts](https://reader034.fdocuments.us/reader034/viewer/2022051401/568143eb550346895db070af/html5/thumbnails/32.jpg)
HydrologicForecastProcess
Supported by a complete verificationsystem.
ModelsRuntime
Observations
Parameters
Model States
Final Forecast
Automatic Forecast
DataAssimilation
ForecasterAnalysisReivew
Forecasts Models
DataQualityControl
ModelCalibration
HistoricalObservations
Models
Model Simulationwith No DA
Model Stateswith No DA
Automatic Forecastwith No DA
AutomaticForecast
InputForecasts
Model Setup
Data Assimilation
ForecastGeneration
ForecastReview
FinalForecast
Veri f icat ion
32
![Page 33: Hydrologic Verification Verification of Deterministic River Stage Forecasts](https://reader034.fdocuments.us/reader034/viewer/2022051401/568143eb550346895db070af/html5/thumbnails/33.jpg)
Summary
● Verification methods can identify sources of error.– Demonstrated with hindcast experiment
● Propose Standardized Verification will enhance verification efforts and moving research to operations.
● Identified a need● Fill the hydrologic verification void.
● Documented an initial description of NWS river stage forecast skill.
● Need to verify more forecasts to update baseline
33