RFC Short-Term Ensemble Workshop Ensemble verification ... · Three components 1) Graphical User...
Transcript of RFC Short-Term Ensemble Workshop Ensemble verification ... · Three components 1) Graphical User...
![Page 1: RFC Short-Term Ensemble Workshop Ensemble verification ... · Three components 1) Graphical User Interface (GUI) in Java 2) Statistics engine in FORTRAN 3) Plotting engine, based](https://reader033.fdocuments.us/reader033/viewer/2022042004/5e6eceddb21002337c3077c8/html5/thumbnails/1.jpg)
1
James [email protected]
Ensemble verification session, part 1: introduction to the Ensemble
Verification System (EVS)
RFC Short-Term Ensemble Workshop
![Page 2: RFC Short-Term Ensemble Workshop Ensemble verification ... · Three components 1) Graphical User Interface (GUI) in Java 2) Statistics engine in FORTRAN 3) Plotting engine, based](https://reader033.fdocuments.us/reader033/viewer/2022042004/5e6eceddb21002337c3077c8/html5/thumbnails/2.jpg)
2
1. What is EVS?• The purpose of EVS• Overview of the tools available
2. Detailed examples• Verification of precipitation• Verification and aggregation of streamflow
3. Plans for the future• Plans for EVS (…depend on your feedback!)
Contents
![Page 3: RFC Short-Term Ensemble Workshop Ensemble verification ... · Three components 1) Graphical User Interface (GUI) in Java 2) Statistics engine in FORTRAN 3) Plotting engine, based](https://reader033.fdocuments.us/reader033/viewer/2022042004/5e6eceddb21002337c3077c8/html5/thumbnails/3.jpg)
3
Purpose of EVS
![Page 4: RFC Short-Term Ensemble Workshop Ensemble verification ... · Three components 1) Graphical User Interface (GUI) in Java 2) Statistics engine in FORTRAN 3) Plotting engine, based](https://reader033.fdocuments.us/reader033/viewer/2022042004/5e6eceddb21002337c3077c8/html5/thumbnails/4.jpg)
4
Transition to ensemble forecasts• Forecasts increasingly allow for uncertainty• This adds another layer of complexity….• ….so the benefits must be demonstrated….
Need for verification• Understanding model strengths/weaknesses• Identifying systematic errors and causes• Communicating quality (confidence building)
Purpose of EVS
![Page 5: RFC Short-Term Ensemble Workshop Ensemble verification ... · Three components 1) Graphical User Interface (GUI) in Java 2) Statistics engine in FORTRAN 3) Plotting engine, based](https://reader033.fdocuments.us/reader033/viewer/2022042004/5e6eceddb21002337c3077c8/html5/thumbnails/5.jpg)
5
Verification by operational forecasters• Purpose built tool that guides users• Statistics and plots that make sense • Transparency about how they are computed• Clarity on limitations and need for judgement
Purpose of EVS
![Page 6: RFC Short-Term Ensemble Workshop Ensemble verification ... · Three components 1) Graphical User Interface (GUI) in Java 2) Statistics engine in FORTRAN 3) Plotting engine, based](https://reader033.fdocuments.us/reader033/viewer/2022042004/5e6eceddb21002337c3077c8/html5/thumbnails/6.jpg)
6
Components of EVS
![Page 7: RFC Short-Term Ensemble Workshop Ensemble verification ... · Three components 1) Graphical User Interface (GUI) in Java 2) Statistics engine in FORTRAN 3) Plotting engine, based](https://reader033.fdocuments.us/reader033/viewer/2022042004/5e6eceddb21002337c3077c8/html5/thumbnails/7.jpg)
7
Three components1) Graphical User Interface (GUI) in Java2) Statistics engine in FORTRAN3) Plotting engine, based on R scripts
User control• User interactions controlled by GUI• Statistics and plotting engines called by GUI• But ‘control files’ (e.g. R scripts) are available
Components of EVS
![Page 8: RFC Short-Term Ensemble Workshop Ensemble verification ... · Three components 1) Graphical User Interface (GUI) in Java 2) Statistics engine in FORTRAN 3) Plotting engine, based](https://reader033.fdocuments.us/reader033/viewer/2022042004/5e6eceddb21002337c3077c8/html5/thumbnails/8.jpg)
8
Staged working environment• Tabs (high level), windows, panels (low level)• Navigate using tabs and Next/Back• Administrative functions always visible
Three stages of verification (as ‘tabs’)1. Verification of one variable on one segment2. Aggregation of statistics across segments3. Display of original/aggregated statistics
Graphical User Interface
![Page 9: RFC Short-Term Ensemble Workshop Ensemble verification ... · Three components 1) Graphical User Interface (GUI) in Java 2) Statistics engine in FORTRAN 3) Plotting engine, based](https://reader033.fdocuments.us/reader033/viewer/2022042004/5e6eceddb21002337c3077c8/html5/thumbnails/9.jpg)
9
A. Defining a Verification Unit (VU)• Identifiers: one variable on one river segment• Input data: path to forecasts/observations
Pairing process (observed vs. forecast)• Temporal parameters• Output data: directory to store statistics
B. Selecting verification statistics• Tests for mean and ensemble members
Stage 1: ‘verification’
![Page 10: RFC Short-Term Ensemble Workshop Ensemble verification ... · Three components 1) Graphical User Interface (GUI) in Java 2) Statistics engine in FORTRAN 3) Plotting engine, based](https://reader033.fdocuments.us/reader033/viewer/2022042004/5e6eceddb21002337c3077c8/html5/thumbnails/10.jpg)
10
Defining an Aggregation Unit (AU)• Assumes that verification stats. available• Requires VUs with ‘common’ properties:
– Common environmental variable– Common temporal parameter values– Common statistics (and their parameter values)
• Candidates added automatically to window• Requires selection of VUs and output path
Stage 2: ‘aggregation’
![Page 11: RFC Short-Term Ensemble Workshop Ensemble verification ... · Three components 1) Graphical User Interface (GUI) in Java 2) Statistics engine in FORTRAN 3) Plotting engine, based](https://reader033.fdocuments.us/reader033/viewer/2022042004/5e6eceddb21002337c3077c8/html5/thumbnails/11.jpg)
11
Behaviour on editing VUs• VUs may belong to one or many AUs• If a VU changes, it is removed from all AUs
Calculation of statistics• Weighted average of input VUs• Weighted by number of observed events
Stage 2: ‘aggregation’
![Page 12: RFC Short-Term Ensemble Workshop Ensemble verification ... · Three components 1) Graphical User Interface (GUI) in Java 2) Statistics engine in FORTRAN 3) Plotting engine, based](https://reader033.fdocuments.us/reader033/viewer/2022042004/5e6eceddb21002337c3077c8/html5/thumbnails/12.jpg)
12
Plots of verification statistics (using R)• Various plots depending on stats. computed• Some plots display composite information• Plots organised by analysis units (VUs, AUs)• Allows sub-selection of month or season
Examples of plots• Reliability diagram• Scatter plot
Stage 3: ‘display’
![Page 13: RFC Short-Term Ensemble Workshop Ensemble verification ... · Three components 1) Graphical User Interface (GUI) in Java 2) Statistics engine in FORTRAN 3) Plotting engine, based](https://reader033.fdocuments.us/reader033/viewer/2022042004/5e6eceddb21002337c3077c8/html5/thumbnails/13.jpg)
13
Statistics engine (FORTRAN)• Verification statistics (verif*.exe) • Aggregation of statistics (agg_verif*.exe)• Driven by command files (written by GUI)• Example of verif*.exe command file
Plotting engine (R)• R statistics and graphics (www.r-project.org)• R plotting scripts written by GUI
Other components of EVS
![Page 14: RFC Short-Term Ensemble Workshop Ensemble verification ... · Three components 1) Graphical User Interface (GUI) in Java 2) Statistics engine in FORTRAN 3) Plotting engine, based](https://reader033.fdocuments.us/reader033/viewer/2022042004/5e6eceddb21002337c3077c8/html5/thumbnails/14.jpg)
14
Documentation!• User’s manual for the GUI• Appendices with data formats, statistics etc.• Few examples at present• Will expand (inc. basic verification concepts)• Also, code is documented in html
Other components of EVS
![Page 15: RFC Short-Term Ensemble Workshop Ensemble verification ... · Three components 1) Graphical User Interface (GUI) in Java 2) Statistics engine in FORTRAN 3) Plotting engine, based](https://reader033.fdocuments.us/reader033/viewer/2022042004/5e6eceddb21002337c3077c8/html5/thumbnails/15.jpg)
15
Detailed examples
![Page 16: RFC Short-Term Ensemble Workshop Ensemble verification ... · Three components 1) Graphical User Interface (GUI) in Java 2) Statistics engine in FORTRAN 3) Plotting engine, based](https://reader033.fdocuments.us/reader033/viewer/2022042004/5e6eceddb21002337c3077c8/html5/thumbnails/16.jpg)
16
Verification of precipitation forecasts• ABRFC region precipitation (CHTM7)• 6-hr observed and forecast values• 01/01/2004 to 31/12/2004• Lead days 1-14
Example verification• Verify to lead day 14 at ‘annual scale’……..
Example I
![Page 17: RFC Short-Term Ensemble Workshop Ensemble verification ... · Three components 1) Graphical User Interface (GUI) in Java 2) Statistics engine in FORTRAN 3) Plotting engine, based](https://reader033.fdocuments.us/reader033/viewer/2022042004/5e6eceddb21002337c3077c8/html5/thumbnails/17.jpg)
17
Verification of streamflow forecasts• ABRFC region streamflow (CHTM7/JOPM7)• 6-hr observed and forecast values• 01/01/2004 to 31/12/2004• Lead days 1-14
Verification and aggregation• Verify to lead day 14 at ‘monthly scale’• Aggregate the two segments……
Example II
![Page 18: RFC Short-Term Ensemble Workshop Ensemble verification ... · Three components 1) Graphical User Interface (GUI) in Java 2) Statistics engine in FORTRAN 3) Plotting engine, based](https://reader033.fdocuments.us/reader033/viewer/2022042004/5e6eceddb21002337c3077c8/html5/thumbnails/18.jpg)
18
Plans for the future
![Page 19: RFC Short-Term Ensemble Workshop Ensemble verification ... · Three components 1) Graphical User Interface (GUI) in Java 2) Statistics engine in FORTRAN 3) Plotting engine, based](https://reader033.fdocuments.us/reader033/viewer/2022042004/5e6eceddb21002337c3077c8/html5/thumbnails/19.jpg)
19
Short/medium term• Improved verification measures (stats/graph)• Support for long-term forecasts (> lead 14)• Confidence intervals for measures• Improved display in EVS• Streamflow verification by river stage
Long-term• One tool for determ./probabilistic forecasts
Plans for the future
![Page 20: RFC Short-Term Ensemble Workshop Ensemble verification ... · Three components 1) Graphical User Interface (GUI) in Java 2) Statistics engine in FORTRAN 3) Plotting engine, based](https://reader033.fdocuments.us/reader033/viewer/2022042004/5e6eceddb21002337c3077c8/html5/thumbnails/20.jpg)
20
Long-term (cont.)• One tool for all space-time scales• One tool for all forecast variables (including
joint verification of multiple variables)
….your input required!
Plans for the future
![Page 21: RFC Short-Term Ensemble Workshop Ensemble verification ... · Three components 1) Graphical User Interface (GUI) in Java 2) Statistics engine in FORTRAN 3) Plotting engine, based](https://reader033.fdocuments.us/reader033/viewer/2022042004/5e6eceddb21002337c3077c8/html5/thumbnails/21.jpg)
21
???
![Page 22: RFC Short-Term Ensemble Workshop Ensemble verification ... · Three components 1) Graphical User Interface (GUI) in Java 2) Statistics engine in FORTRAN 3) Plotting engine, based](https://reader033.fdocuments.us/reader033/viewer/2022042004/5e6eceddb21002337c3077c8/html5/thumbnails/22.jpg)
22
Forecast data file format
![Page 23: RFC Short-Term Ensemble Workshop Ensemble verification ... · Three components 1) Graphical User Interface (GUI) in Java 2) Statistics engine in FORTRAN 3) Plotting engine, based](https://reader033.fdocuments.us/reader033/viewer/2022042004/5e6eceddb21002337c3077c8/html5/thumbnails/23.jpg)
23
Observed data file format
![Page 24: RFC Short-Term Ensemble Workshop Ensemble verification ... · Three components 1) Graphical User Interface (GUI) in Java 2) Statistics engine in FORTRAN 3) Plotting engine, based](https://reader033.fdocuments.us/reader033/viewer/2022042004/5e6eceddb21002337c3077c8/html5/thumbnails/24.jpg)
24
paired_ts file format
…….
![Page 25: RFC Short-Term Ensemble Workshop Ensemble verification ... · Three components 1) Graphical User Interface (GUI) in Java 2) Statistics engine in FORTRAN 3) Plotting engine, based](https://reader033.fdocuments.us/reader033/viewer/2022042004/5e6eceddb21002337c3077c8/html5/thumbnails/25.jpg)
25
Reliability diagram
![Page 26: RFC Short-Term Ensemble Workshop Ensemble verification ... · Three components 1) Graphical User Interface (GUI) in Java 2) Statistics engine in FORTRAN 3) Plotting engine, based](https://reader033.fdocuments.us/reader033/viewer/2022042004/5e6eceddb21002337c3077c8/html5/thumbnails/26.jpg)
26
Scatter plot
![Page 27: RFC Short-Term Ensemble Workshop Ensemble verification ... · Three components 1) Graphical User Interface (GUI) in Java 2) Statistics engine in FORTRAN 3) Plotting engine, based](https://reader033.fdocuments.us/reader033/viewer/2022042004/5e6eceddb21002337c3077c8/html5/thumbnails/27.jpg)
27
Scatter plot
![Page 28: RFC Short-Term Ensemble Workshop Ensemble verification ... · Three components 1) Graphical User Interface (GUI) in Java 2) Statistics engine in FORTRAN 3) Plotting engine, based](https://reader033.fdocuments.us/reader033/viewer/2022042004/5e6eceddb21002337c3077c8/html5/thumbnails/28.jpg)
28
Verif*.exe command file
……….