Interactive Verification ProgramInteractive Verification ProgramTraining for AWIPS OB8.2Training for AWIPS OB8.2
Hank HerrHank Herr
Office of Hydrologic DevelopmentOffice of Hydrologic Development
RFC Verification Workshop, 08/15/2007
22
OverviewOverview
IntroductionIntroduction IVP TerminologyIVP Terminology Software ComponentsSoftware Components Database UsageDatabase Usage Output FilesOutput Files DesignDesign DemonstrationDemonstration TipsTips Future PossibilitiesFuture Possibilities ExercisesExercises DiscussionDiscussion
33
Original IVP written around 2000-2001 for old HP based verify softwareOriginal IVP written around 2000-2001 for old HP based verify software Verification moved to archive database for OB4Verification moved to archive database for OB4
– IVP Batch Program software writtenIVP Batch Program software written– Old IVP GUI still ran, but used pairs files created during batch processing; Old IVP GUI still ran, but used pairs files created during batch processing;
it did not access the archive database directlyit did not access the archive database directly New IVP GUI written for OB7.2New IVP GUI written for OB7.2
– Provided ability to build verification products via GUI, modify components Provided ability to build verification products via GUI, modify components of a chart and save modifications, create batch file to reconstruct graphics of a chart and save modifications, create batch file to reconstruct graphics via IVP Batch Programvia IVP Batch Program
– Accessed archive database directlyAccessed archive database directly– Limitations:Limitations:
Designed for stage dataDesigned for stage data Accessed only three data tables of the archive databaseAccessed only three data tables of the archive database Limited number of statisticsLimited number of statistics No quality control of pairsNo quality control of pairs Inflexibility in building batch files from GUIInflexibility in building batch files from GUI JClass Desktop Views used to render charts (limited charting capabilities; JClass Desktop Views used to render charts (limited charting capabilities;
upgrade cost too much money)upgrade cost too much money)
IntroductionIntroductionHistory of IVPHistory of IVP
44
IntroductionIntroductionNew for OB8.2New for OB8.2
Designed for verification of stage, flow, temperature, and precipitation Designed for verification of stage, flow, temperature, and precipitation forecastsforecasts– Can do other data types as long as data is stored within accessible tables Can do other data types as long as data is stored within accessible tables – Accesses 10 data tables of the archive databaseAccesses 10 data tables of the archive database
Uses all data tables except pempsep, pemrsep, and pairedvaluesUses all data tables except pempsep, pemrsep, and pairedvalues Additional statisticsAdditional statistics
– Moments, CDF Plot, ROC, etc.Moments, CDF Plot, ROC, etc. Quality control of pairs via GUIQuality control of pairs via GUI
– View pairs in a time series plot, in addition to existing scatter plotView pairs in a time series plot, in addition to existing scatter plot– Mark pairs as ‘bad’ so that verification ignores themMark pairs as ‘bad’ so that verification ignores them– Change forecast or observed values within a pair and save changes to Change forecast or observed values within a pair and save changes to
databasedatabase More flexible batch file constructionMore flexible batch file construction
– Batch file can be built that will construct current chart for other Batch file can be built that will construct current chart for other combinations of locations and analysis periodscombinations of locations and analysis periods
Chart Director used for renderingChart Director used for rendering– Relatively quick, memory efficient chart renderingRelatively quick, memory efficient chart rendering– Used previously for HydroGen, so a run-time license already existsUsed previously for HydroGen, so a run-time license already exists
55
IVP TerminologyIVP Terminology
Verification LocationVerification Location– A forecast point or location and a data type for which verification is A forecast point or location and a data type for which verification is
to be performedto be performed– Uniquely specified by location id (lid within database) and SHEF Uniquely specified by location id (lid within database) and SHEF
pedtsep code.pedtsep code.– Includes following information:Includes following information:
Forecast and observed categoriesForecast and observed categories Observation type (raw or processed)Observation type (raw or processed) Active flag (yes or no)Active flag (yes or no) National flag (yes or no)National flag (yes or no) Sensor preferences (ordered list of observed type source, representing Sensor preferences (ordered list of observed type source, representing
preferences for which observed values to pair with a forecast)preferences for which observed values to pair with a forecast)
66
IVP TerminologyIVP Terminology
Verification GroupVerification Group– A set of locations and parameters that define the data to load and A set of locations and parameters that define the data to load and
other parameters of verificationother parameters of verification Locations must have the same number of categories and must be of Locations must have the same number of categories and must be of
the same type: stage (height), flow (discharge), temperature, the same type: stage (height), flow (discharge), temperature, precipitationprecipitation
– Includes following parameters:Includes following parameters: Analysis period and intervalsAnalysis period and intervals Lead time period and intervalsLead time period and intervals Issuance time-of-day (0-24 hours) period and intervalsIssuance time-of-day (0-24 hours) period and intervals Restrictions on location response time, active flag, and SHEF Restrictions on location response time, active flag, and SHEF
components (forecast type source, physical element, duration, and components (forecast type source, physical element, duration, and extremum)extremum)
Whether or not to compute statistics independently for each lid or Whether or not to compute statistics independently for each lid or forecast type sourceforecast type source
77
IVP TerminologyIVP Terminology
Forecast-Observed Data PairForecast-Observed Data Pair– A forecast value paired with an observed valueA forecast value paired with an observed value
Observed value can be “raw” or “processed” depending on if the value Observed value can be “raw” or “processed” depending on if the value was posted to the database by the raw or processed SHEF decoderwas posted to the database by the raw or processed SHEF decoder
Each forecast can have one raw pair and one processed pair Each forecast can have one raw pair and one processed pair associated with itassociated with it
– ““Best” candidate observation is chosen for each forecast based on Best” candidate observation is chosen for each forecast based on following criteria:following criteria: SHEF qualifier code (G, M, P, V, S, Z, T, F, and Q, in order)SHEF qualifier code (G, M, P, V, S, Z, T, F, and Q, in order) sensor preference (observation type source) ranksensor preference (observation type source) rank difference between forecast validtime and observation obstimedifference between forecast validtime and observation obstime
– A pair can be “good” or “bad”A pair can be “good” or “bad” Bad pairs are not used in verification computationsBad pairs are not used in verification computations
88
Software ComponentsSoftware Components
Vfyruninfo Editor GUIVfyruninfo Editor GUI– Manage vfyruninfo tableManage vfyruninfo table
IVPIVP– Data quality control and define verification products graphicallyData quality control and define verification products graphically
IVP Batch ProgramIVP Batch Program– Build pairs and verification products via a batch languageBuild pairs and verification products via a batch language
IVP Batch BuilderIVP Batch Builder– Edit IVP Batch Program batch files graphicallyEdit IVP Batch Program batch files graphically
Verify Pairs IngestorVerify Pairs Ingestor– Ingest pairs directly from text files of specific format into the Ingest pairs directly from text files of specific format into the
vfypairs or vfyprocpairs tablesvfypairs or vfyprocpairs tables del_bad_pairsdel_bad_pairs
– Removes bad pairs from the vfypairs and vfyprocpairs tableRemoves bad pairs from the vfypairs and vfyprocpairs table
99
Database UsageDatabase Usage
Verification Location InformationVerification Location Information– Vfyruninfo tableVfyruninfo table
Defines locations for which verification can be performed.Defines locations for which verification can be performed. Individual entries for each verification location and sensor Individual entries for each verification location and sensor
preference component (observed type source)preference component (observed type source) Edited via Vfyruninfo Editor GUIEdited via Vfyruninfo Editor GUI
– Location table Location table Information used in building labels, includes RFC name, full Information used in building labels, includes RFC name, full
location name, etc.location name, etc.
– Rivercrit table Rivercrit table Flood stages and flows (if location is of appropriate data type).Flood stages and flows (if location is of appropriate data type).
1010
Database UsageDatabase Usage
Building PairsBuilding Pairs– Tables of data for pairing:Tables of data for pairing:
Forecasts: pedfsep, pehfsep, peqfsep, pedcsepForecasts: pedfsep, pehfsep, peqfsep, pedcsep Raw Observations: pecrsep, pedrsep, peoosepRaw Observations: pecrsep, pedrsep, peoosep Processed Observations: pedpsep, pehpsep, peqpsep, peoosepProcessed Observations: pedpsep, pehpsep, peqpsep, peoosep
– Vaddadjust tableVaddadjust table Adjust observations prior to pairing with a forecastAdjust observations prior to pairing with a forecast Only affects forecast-observed pairsOnly affects forecast-observed pairs Is not dated; there can be only one entry per locationIs not dated; there can be only one entry per location
– Sensok tableSensok table Allows for specifying time ranges during which sensors should Allows for specifying time ranges during which sensors should
not be used (i.e. observations should not be used for pairing)not be used (i.e. observations should not be used for pairing)
1111
Database UsageDatabase Usage
Storing PairsStoring Pairs– Vfypairs tableVfypairs table
Stores pairs constructed using raw observationsStores pairs constructed using raw observations Field quality_code defines if the pair is good or badField quality_code defines if the pair is good or bad Can be very large (millions of records)Can be very large (millions of records)
– Vfyprocpairs tableVfyprocpairs table Stores pairs constructed using processed observationsStores pairs constructed using processed observations Field quality_code defines if the pair is good or badField quality_code defines if the pair is good or bad Can be very large, but will likely be a fraction of the size of Can be very large, but will likely be a fraction of the size of
vfypairsvfypairs
– A forecast can have up to two pairs built for it: one A forecast can have up to two pairs built for it: one stored in vfypairs and one stored in vfyprocpairsstored in vfypairs and one stored in vfyprocpairs Only one of the two pairs can be analyzed at one timeOnly one of the two pairs can be analyzed at one time
1212
Output FilesOutput Files
Output Statistics File (batch only)Output Statistics File (batch only)– Verification statistics output in an ASCII tabular formatVerification statistics output in an ASCII tabular format
Pairs FilePairs File– Pairs used to compute statisticsPairs used to compute statistics– ASCII tabular formatASCII tabular format
Output Image File (.png, .jpg)Output Image File (.png, .jpg)– PNG or JPG files displaying a plot of verification PNG or JPG files displaying a plot of verification
statisticsstatistics Image Data File (.dat)Image Data File (.dat)
– Statistic values comprising an output image plotStatistic values comprising an output image plot– ASCII tabular formatASCII tabular format
1313
DesignDesignFoundationFoundation
Archive Database
Archive database provides the following:Archive database provides the following:– Forecasts and observationsForecasts and observations
– Verification location informationVerification location information
– Forecast-observed data pairsForecast-observed data pairs
User
1414
DesignDesignVfyruninfo EditorVfyruninfo Editor
Archive Database
Vfyruninfo Editor
User
Vfyruninfo rows
Vfyruninfo Editor is used to specify verification Vfyruninfo Editor is used to specify verification locations and saves them to the vfyruninfo locations and saves them to the vfyruninfo table of the archive databasetable of the archive database
1515
PairingProcessor
Archive Database
User
pairs
Design Design IVP Batch ProgramIVP Batch Program
Vfyruninfo Editor
Vfyruninfo rows
forecasts,observations,location info
IVP Batch Program provides the user the IVP Batch Program provides the user the ability to:ability to:
– Pair forecasts with observed values according Pair forecasts with observed values according to user specified parametersto user specified parameters
parameters
Batch
Batch file
1616
DesignDesignIVP Batch ProgramIVP Batch Program
VerificationProcessor
PairingProcessor
Archive Database
User
pairs
ASCII
pairs, location info
forecasts,observations,location info
Vfyruninfo Editor
Vfyruninfo rows
pairs, statistics
IVP Batch Program provides the user the IVP Batch Program provides the user the ability to:ability to:
– Calculate statistics and output results to tabular Calculate statistics and output results to tabular ASCII filesASCII files
parameters
Batch
Batch file
1717
DesignDesignIVP Batch ProgramIVP Batch Program
VerificationProcessor
PairingProcessor
Archive Database
User
ASCII
pairs
GraphicsProcessor
Graphics
statisticsparams
Vfyruninfo Editor
Vfyruninfo rows
forecasts,observations,location info
pairs, location info
pairs, statistics
IVP Batch Program provides the user the IVP Batch Program provides the user the ability to:ability to:
– Generate user customizable graphical Generate user customizable graphical verification products displaying statisticsverification products displaying statistics
parameters
Batch
Batch file
1818
params
VerificationProcessor
PairingProcessor
Archive Database
User
GUIGraphics
ASCII
pairs
pairs, statistics
DesignDesignIVP GUIIVP GUI
Vfyruninfo Editor
Vfyruninfo rows
forecasts,observations,location info
pairs, location info
IVP provides the user the ability to:IVP provides the user the ability to:– Specify, build, and view graphicsSpecify, build, and view graphics
GraphicsProcessor
statisticsparams
parameters
Batch
Batch file
1919
params
VerificationProcessor
PairingProcessor
Archive Database
User
GUIGraphics
ASCII
pairs
pairs, statistics
DesignDesignIVP GUIIVP GUI
Vfyruninfo Editor
Vfyruninfo rows
forecasts,observations,location info
pairs, location info
IVP provides the user the ability to:IVP provides the user the ability to:– Modify forecast-observed data pairsModify forecast-observed data pairs
GraphicsProcessor
statisticsparams
pairs
parameters
Batch
Batch file
2020
params
VerificationProcessor
PairingProcessor
Archive Database
User
GUIGraphics
ASCII
pairs
parameters
pairs, statistics
DesignDesignIVP GUIIVP GUI
Batch
Vfyruninfo Editor
Vfyruninfo rows
forecasts,observations,location info
pairs, location info
IVP provides the user the ability to:IVP provides the user the ability to:– Create batch files to produce graphics via IVP Create batch files to produce graphics via IVP
Batch ProgramBatch Program
GraphicsProcessor
statisticsparams
pairs
Batch file batchtokens
2121
DesignDesignIVP Window LayoutIVP Window Layout
1. Load Pairs 2. View and QC Pairs
3. Define Products4. View Products
2222
DemonstrationDemonstration
Feature availability:Feature availability:– Both OB7.2 and OB8.2Both OB7.2 and OB8.2– OB8.2 only*OB8.2 only*
2323
Start Vfyruninfo EditorStart Vfyruninfo Editor Filter ingestfilter table contents*Filter ingestfilter table contents* Select locations from ingestfilter table*Select locations from ingestfilter table* Create locations from scratchCreate locations from scratch Specify location parametersSpecify location parameters
– Active*Active*– National*National*– Response timeResponse time– Sensor preferencesSensor preferences
Save to archive databaseSave to archive database
DemonstrationDemonstrationVfyruninfo EditorVfyruninfo Editor
ref: Vfyruninfo Editor User’s Manual
2424
Commands:Commands:– Time window: START_TIME, END_TIMETime window: START_TIME, END_TIME– Specify locations: LOCATION, Specify locations: LOCATION, DUR*DUR*, , EXTREMUM*EXTREMUM*, ,
FCST_TS*FCST_TS*, , PE*PE*– Pairing window: PAIRING_WINDOWPairing window: PAIRING_WINDOW– Observation type: Observation type: OBS_TYPE (replaces OBS_TABLE)*OBS_TYPE (replaces OBS_TABLE)*– Persistence flag: PERSISTENCEPersistence flag: PERSISTENCE
Action:Action:– Build the pairs: BUILD_PAIRSBuild the pairs: BUILD_PAIRS
DemonstrationDemonstrationPairing Batch FilePairing Batch File
ref: IVP Batch Program User’s Manual for Pairing
2525
Example (buildpairs_template.bat):Example (buildpairs_template.bat):
DemonstrationDemonstrationPairing Batch FilePairing Batch File
# Pair all locations. Use default settings of # ALL for all other location specifying commandslocation = ALL
# Build pairs for a two week window prior to # current system timestart_time = "* - 14 days"end_time = "*"
# Parameters pairing_window = 1obs_type = RAW
build_pairs = true
ref: IVP Batch Program User’s Manual for Pairing
2626
Commands:Commands:– Time Window: START_TIME, END_TIMETime Window: START_TIME, END_TIME– Specify Locations: Specify Locations: DUR*DUR*, , EXTREMUM*EXTREMUM*, FCST_TS, PE, FCST_TS, PE
Action:Action:– Compute statistics: NATLSTATSCompute statistics: NATLSTATS
Recommendation*:Recommendation*:– Set the national flag for all locations that are part of the Set the national flag for all locations that are part of the
national program to ‘Y’ (use Vfyruninfo Editor)national program to ‘Y’ (use Vfyruninfo Editor)– Give the NATLSTATS action a value of NATIONALGive the NATLSTATS action a value of NATIONAL
DemonstrationDemonstrationNational Statistics Batch FileNational Statistics Batch File
ref: IVP Batch Program User’s Manual for Verification
2727
Example (natlstats_template.bat):Example (natlstats_template.bat):
DemonstrationDemonstrationNational Statistics Batch FileNational Statistics Batch File
# Compute stats for September, 2003.START_TIME = “2003-09-01 00:00:00”END_TIME = “2003-09-30 23:59:59”
# Default Values are ‘ALL’# FCST_TS = ALL# PE = ALL# DUR = ALL# EXTREMUM = ALL
NATLSTATS = NATIONAL
ref: IVP Batch Program User’s Manual for Verification
2828
Start IVPStart IVPgo vsys_scriptsgo vsys_scripts
ivpivp
Specify parameters of a verification groupSpecify parameters of a verification group– Analysis period and intervalsAnalysis period and intervals– Lead time period and intervalsLead time period and intervals– Issuance time period and intervals*Issuance time period and intervals*– Miscellaneous parametersMiscellaneous parameters
DemonstrationDemonstrationIVP: Verification Group ManagerIVP: Verification Group Manager
ref: IVP User’s Manual, Section 9
2929
Specify verification locations that comprise a Specify verification locations that comprise a groupgroup– Filter available locations*Filter available locations*– Choose locationsChoose locations– Specify categoriesSpecify categories– Specify observation type (from which pairs table Specify observation type (from which pairs table
to acquire data)*to acquire data)*
DemonstrationDemonstrationIVP: Verification Location ManagerIVP: Verification Location Manager
ref: IVP User’s Manual, Section 13
3030
Create IVP Data DisplayCreate IVP Data Display– Loading data progress windowLoading data progress window
Open the Verification Group Manager and Open the Verification Group Manager and Verification Location ManagerVerification Location Manager
Save the imageSave the image
DemonstrationDemonstrationIVP: IVP Data DisplayIVP: IVP Data Display
ref: IVP User’s Manual, Section 15
3131
Examine data through two views:Examine data through two views:– Scatter Plot ViewScatter Plot View
Select a location*Select a location* – In OB7.2, use Show button in Verification Location ManagerIn OB7.2, use Show button in Verification Location Manager
Display info about a location*Display info about a location* Line x = yLine x = y
– Time Series View*Time Series View* Specify time series to viewSpecify time series to view Emphasize a time seriesEmphasize a time series
DemonstrationDemonstrationIVP: IVP Data DisplayIVP: IVP Data Display
ref: IVP User’s Manual, Sections 15.1, 15.2
3232
Draw a selected region rectangleDraw a selected region rectangle Zoom and the navigator panelZoom and the navigator panel
– Zoom in on scatter viewZoom in on scatter view– Zoom in on time series view*Zoom in on time series view*
Panning through the time series by zoomingPanning through the time series by zooming
– Shift the zoomed region via the navigator panel Shift the zoomed region via the navigator panel and keyboard*and keyboard*
– Zoom outZoom out
DemonstrationDemonstrationIVP: IVP Data DisplayIVP: IVP Data Display
ref: IVP User’s Manual, Sections 15.6, 15.7
3333
View Pairs via the Verify Pairs ViewerView Pairs via the Verify Pairs Viewer– Select pairsSelect pairs– Sort tableSort table– Create a pairs fileCreate a pairs file
WARNING: Do not view very large WARNING: Do not view very large (30,000+) sets of pairs… the table rendering (30,000+) sets of pairs… the table rendering may be too slowmay be too slow
DemonstrationDemonstrationIVP: IVP Data DisplayIVP: IVP Data Display
ref: IVP User’s Manual, Section 16
3434
Mark pairs as good or bad via IVP Data Display*Mark pairs as good or bad via IVP Data Display* Edit pairs via the Verify Pairs Editor*Edit pairs via the Verify Pairs Editor*
– Select pairsSelect pairs– Mark pairs as good or badMark pairs as good or bad– Edit forecast or observed value manuallyEdit forecast or observed value manually– Focus on a pairFocus on a pair– Edit forecast or observed value graphicallyEdit forecast or observed value graphically
Scatter viewScatter view Time series viewTime series view
Change the mouse mode*Change the mouse mode*
WARNING: Do not select large sets (1000+) of pairs for WARNING: Do not select large sets (1000+) of pairs for editing… the table rendering speed may be too slowediting… the table rendering speed may be too slow
DemonstrationDemonstrationIVP: IVP Data DisplayIVP: IVP Data Display
ref: IVP User’s Manual, Sections 15.11, 17
3535
Open the Plot Definition ManagerOpen the Plot Definition Manager Select statistics Select statistics
– IVP Statistic Chooser ManagerIVP Statistic Chooser Manager Select plot typeSelect plot type Select x-axis variableSelect x-axis variable Select comparison variable*Select comparison variable* Edit statistics parameters*Edit statistics parameters* Edit group parameters*Edit group parameters*
– Spawns a window nearly identical to Verification Group ManagerSpawns a window nearly identical to Verification Group Manager
Note how variable selection affects analysis, lead time, and issuance Note how variable selection affects analysis, lead time, and issuance time-of-day intervalstime-of-day intervals
Note how variable selection affects category choice boxesNote how variable selection affects category choice boxes
DemonstrationDemonstrationIVP: Verification Plot Definition ManagerIVP: Verification Plot Definition Manager
ref: IVP User’s Manual, Sections 18, 19
3636
ErrorsErrors Categorical StatisticsCategorical Statistics Quantiles and ExtremesQuantiles and Extremes
– Computed relative to whichever variable whose category used is Computed relative to whichever variable whose category used is set to “Do Not Use”set to “Do Not Use”
Moments*Moments* Sample SizeSample Size
– The number of pairs used in the computations The number of pairs used in the computations ––or–or–
– Number of pairs within the category (if x-axis or comparison Number of pairs within the category (if x-axis or comparison variable is forecast or observed categories)variable is forecast or observed categories)
Misc.*Misc.* Special Plots*Special Plots*
DemonstrationDemonstrationIVP: The StatisticsIVP: The Statistics
ref: IVP Batch Builder User’s Manual, Appendix B
3737
Categorical Statistics use 3x3 Contingency TableCategorical Statistics use 3x3 Contingency Table
Examples:Examples:– HFAR = D/(D + E + F)HFAR = D/(D + E + F)– TFAR = (D + F)/(D + E + F)TFAR = (D + F)/(D + E + F)– CSI = (A+B+D+E)/(A+B+D+E+G+H+C+F) CSI = (A+B+D+E)/(A+B+D+E+G+H+C+F)
In a 2x2 contingency tableIn a 2x2 contingency table– HFAR is always 0 for the lowermost categoryHFAR is always 0 for the lowermost category– TFAR and HFAR are equivalent for uppermost categoryTFAR and HFAR are equivalent for uppermost category
POD + UFR + OFR = 1 in all casesPOD + UFR + OFR = 1 in all cases
DemonstrationDemonstrationIVP: The StatisticsIVP: The Statistics
ref: IVP Batch Builder User’s Manual, Appendix B
Observed was...
Forecast was... below within above
below A B C
within D E F
above G H I
3838
DemonstrationDemonstrationIVP: Verification Plot Definition ManagerIVP: Verification Plot Definition Manager
x-axis
primary y-axis
comparisonvariable
secondary y-axis
Statisticshort name(matches outputfiles)
3939
Choose categories usedChoose categories used– Disabled if x-axis or comparison variable is forecast or Disabled if x-axis or comparison variable is forecast or
observed categoriesobserved categories– Select “Do Not Use”Select “Do Not Use”
Only one of the two choice boxes can be “Do Not Use”Only one of the two choice boxes can be “Do Not Use”
– Impact of selections on how statistics are computedImpact of selections on how statistics are computed Category used restricts pairs used in computations to only Category used restricts pairs used in computations to only
those within the categorythose within the category Some statistics are calculated relative to the variable not used Some statistics are calculated relative to the variable not used
to categorize data (e.g. quantiles, to categorize data (e.g. quantiles, cdf plot*cdf plot*, , pdf plot*pdf plot*))
– Special case: ROC plot*Special case: ROC plot* Category used defines observation thresholdCategory used defines observation threshold
DemonstrationDemonstrationIVP: Verification Plot Definition ManagerIVP: Verification Plot Definition Manager
ref: IVP User’s Manual, Section 18.7
4040
Create IVP Statistic DisplayCreate IVP Statistic Display– Data is always loaded from memory even if the Data is always loaded from memory even if the
overridden group parameters imply additional data is overridden group parameters imply additional data is needed (such as a wider analysis interval, for example)needed (such as a wider analysis interval, for example)
– Allows for relatively quick plot generationAllows for relatively quick plot generation
Save the imageSave the image View the data via the IVP Statistics Data ViewerView the data via the IVP Statistics Data Viewer
– Create an ASCII format image data fileCreate an ASCII format image data file
DemonstrationDemonstrationIVP: IVP Statistic DisplayIVP: IVP Statistic Display
ref: IVP User’s Manual, Section 21
4141
Create a batch fileCreate a batch file– Change the analysis period*Change the analysis period*– Change the included locations*Change the included locations*– Create other groups for which to generate Create other groups for which to generate
products*products*– Specify output and template file namesSpecify output and template file names
Replacement strings are needed if images are to be Replacement strings are needed if images are to be produced for multiple groups in order to avoid produced for multiple groups in order to avoid overwriting output files*overwriting output files*
– Specify batch file nameSpecify batch file name
DemonstrationDemonstrationIVP: IVP Statistics DisplayIVP: IVP Statistics Display
ref: IVP User’s Manual, Section 23
4242
Open up the Chart Property ManagerOpen up the Chart Property Manager– Click on menu item or tool bar itemClick on menu item or tool bar item– Click on plot component*Click on plot component*
Change labelsChange labels Change axis limits or chart dimensionsChange axis limits or chart dimensions Change legend componentsChange legend components Change x-axis labels (for IVP Statistic Display)Change x-axis labels (for IVP Statistic Display) Create a template fileCreate a template file
– Don’t modify manually unless absolutely necessaryDon’t modify manually unless absolutely necessary Apply changes from a template fileApply changes from a template file
DemonstrationDemonstrationIVP: Chart Property ManagerIVP: Chart Property Manager
ref: IVP User’s Manual, Sections 24, Appendix D
4343
DemonstrationDemonstrationIVP: Chart Property ManagerIVP: Chart Property Manager
X-Axis Title
Y-Axis 1Title
Legend
Y-Axis 2Title
LegendTitle
Tick Mark Font
Plot Title
4444
Start IVP Batch BuilderStart IVP Batch Builder Load batch fileLoad batch file Insert, append, delete command linesInsert, append, delete command lines Edit command linesEdit command lines Comment and uncomment command linesComment and uncomment command lines Check for errorsCheck for errors Run batch fileRun batch file
DemonstrationDemonstrationIVP Batch Builder and the IVP Batch ProgramIVP Batch Builder and the IVP Batch Program
ref: IVP Batch Builder User’s Manual
4545
Create one batch file to generate products Create one batch file to generate products for many locationsfor many locations– @FILE Command to refer to other files@FILE Command to refer to other files– Replacement strings to avoid overwriting output Replacement strings to avoid overwriting output
filesfiles
DemonstrationDemonstrationIVP Batch Builder and the IVP Batch ProgramIVP Batch Builder and the IVP Batch Program
ref: IVP Batch Builder User’s Manual
4646
Organize output by using subdirectories of $Organize output by using subdirectories of $(vsys_output)*(vsys_output)*– If producing graphics for each location, use If producing graphics for each location, use
@ONELID or @ONELIDSHEF to define @ONELID or @ONELIDSHEF to define directories: directories:
OUTPUT_FILE = “@ONELID/output.txt,c”OUTPUT_FILE = “@ONELID/output.txt,c”– If producing graphics for time periods, use If producing graphics for time periods, use
@STARTTIME or @ENDTIME:@STARTTIME or @ENDTIME:OUTPUT_FILE = OUTPUT_FILE =
“@STARTTIME{[CCYY]}/output.txt,c”“@STARTTIME{[CCYY]}/output.txt,c”
DemonstrationDemonstrationIVP Batch Builder and the IVP Batch ProgramIVP Batch Builder and the IVP Batch Program
ref: IVP Batch Builder User’s Manual
4747
To minimize time spent querying database, load all needed To minimize time spent querying database, load all needed pairs at oncepairs at once– Only specify the overall analysis period and choose the locations Only specify the overall analysis period and choose the locations
within the Verification Group Manager and Verification Location within the Verification Group Manager and Verification Location ManagerManager Specify other parameters via the Plot Definition version of the Specify other parameters via the Plot Definition version of the
Verification Group Manager (click on Edit Group Parameters in the Plot Verification Group Manager (click on Edit Group Parameters in the Plot Definition Manager)Definition Manager)
– Important when generating many productsImportant when generating many products To minimize the chart rendering time for the IVP Data To minimize the chart rendering time for the IVP Data
Display, keep the number of pairs smallDisplay, keep the number of pairs small– Do one location at a time for a couple yearsDo one location at a time for a couple years– Do several locations at a time for one monthDo several locations at a time for one month– Important when quality controlling pairs dataImportant when quality controlling pairs data
See the Memory Limitations Tips later for recommended See the Memory Limitations Tips later for recommended numbers of pairs to loadnumbers of pairs to load
Tips:Tips:Verification Group Manager UsageVerification Group Manager Usage
4848
Use OBS_TYPE “RAW” for paired observations Use OBS_TYPE “RAW” for paired observations if…if…– Observed data was posted to archive database via raw Observed data was posted to archive database via raw
SHEF decoder, SHEF decoder, ANDAND– First letter of type source is ‘R’.First letter of type source is ‘R’.
Use OBS_TYPE “PROCESSED” if…Use OBS_TYPE “PROCESSED” if…– Observed data was posted to archive database via Observed data was posted to archive database via
processed SHEF decoder, processed SHEF decoder, OROR– First letter of type source is not ‘R’First letter of type source is not ‘R’
Example:Example:– If processed observations were posted via raw SHEF If processed observations were posted via raw SHEF
decoder (to peoosep table), then specify PROCESSEDdecoder (to peoosep table), then specify PROCESSED
Tips:Tips:Deciding Between PROCESSED or RAWDeciding Between PROCESSED or RAW
4949
Specifies parameters of graphical user interfacesSpecifies parameters of graphical user interfaces– FontsFonts– Window SizesWindow Sizes– ColorsColors– OthersOthers
LocationLocation– Default:Default:
$(get_apps_defaults vsys_dir)/app-defaults/*_SYSTEM_FILE.txt$(get_apps_defaults vsys_dir)/app-defaults/*_SYSTEM_FILE.txt
– Directory can be overridden by exporting environment variable Directory can be overridden by exporting environment variable SYSTEM_SETTINGS_FILE_DIRSYSTEM_SETTINGS_FILE_DIR Can allow for user specific system settingsCan allow for user specific system settings
FormatFormat– ESPADP/IVP batch language standard “<token> = <value>” formatESPADP/IVP batch language standard “<token> = <value>” format
Tips:Tips:System Settings FilesSystem Settings Files
ref: IVP User’s Manual, IVP Batch Builder User’s Manual, Vfyruninfo Editor User’s Manual, Appendix A
5050
Tips: Tips: Memory LimitationsMemory Limitations
Recommended number of pairs:Recommended number of pairs:
– Limits can be changed by using the ivp and ivpbatch script –m and –s Limits can be changed by using the ivp and ivpbatch script –m and –s options (run “ivp –hh” or “ivpbatch –hh” to read about these options)options (run “ivp –hh” or “ivpbatch –hh” to read about these options)
– Limits above are for IVP GUI; IVP Batch Program can deal with more Limits above are for IVP GUI; IVP Batch Program can deal with more pairs because not all pairs need to be stored in memory at one timepairs because not all pairs need to be stored in memory at one time
If IVP GUI runs out of memory, an error message will pop-up and If IVP GUI runs out of memory, an error message will pop-up and the pairs will be discarded (in most cases)the pairs will be discarded (in most cases)– The software may crash in other casesThe software may crash in other cases
Verify Pairs Viewer window is a memory hogVerify Pairs Viewer window is a memory hog– Add about 25% to the recommended value if not using the ViewerAdd about 25% to the recommended value if not using the Viewer
ref: IVP User’s Manual, IVP Batch Builder User’s Manual, Vfyruninfo Editor User’s Manual, Appendix A
System pairs QC product generation Approx. MaximumAX 20,000 20,000 40-60,000LX 100,000 400,000 600-800,000
Recommendation
5151
Debug problemsDebug problems– Export vsys_debug token prior to execution to see additional outputExport vsys_debug token prior to execution to see additional output
Useful levels are 0 (no output), 1, 20, 40, 60, 80, 100, 120, 128.Useful levels are 0 (no output), 1, 20, 40, 60, 80, 100, 120, 128. ““export vsys_debug=20” to see additional progress outputexport vsys_debug=20” to see additional progress output ““export vsys_debug=40” to see database query where clauses and export vsys_debug=40” to see database query where clauses and
timing results for chart renderingtiming results for chart rendering ““export vsys_debug=128” to see everything (lots and lots of output)export vsys_debug=128” to see everything (lots and lots of output)
PerformancePerformance– If IVP Data Display chart rendering is slow…If IVP Data Display chart rendering is slow…
Reduce number of pairs viewedReduce number of pairs viewed Zoom in on regions (points outside current zoom area are not Zoom in on regions (points outside current zoom area are not
rendered)rendered) Run on faster machines (port code to LX and execute software there)Run on faster machines (port code to LX and execute software there)
– If database query speed is slow…If database query speed is slow… Turn off SHEF decoders for a short time (if you can)Turn off SHEF decoders for a short time (if you can) Reduce number of locations or width of analysis periodReduce number of locations or width of analysis period
Tips:Tips:Debugging and PerformanceDebugging and Performance
ref: IVP User’s Manual, Section 25
5252
Future PossibilitiesFuture Possibilities Confidence IntervalsConfidence Intervals
– Algorithms need to be developedAlgorithms need to be developed– Ability to produce CI values and plot them needs to be addedAbility to produce CI values and plot them needs to be added
GUI redesignGUI redesign– Reduce the number of windowsReduce the number of windows– Reorganize window so that the Verification Group Manager does not need Reorganize window so that the Verification Group Manager does not need
to be in two placesto be in two places Add ability to handle multiple comparison variablesAdd ability to handle multiple comparison variables Add ability to modify other properties of a chartAdd ability to modify other properties of a chart
– Tick spacing, line style (solid, dashed, etc), line width, etc.Tick spacing, line style (solid, dashed, etc), line width, etc.– Requires additions to Chart Property ManagerRequires additions to Chart Property Manager
Create baseline batch files requiring no changes in order to produce Create baseline batch files requiring no changes in order to produce standard graphics standard graphics – Standard graphics need to be determinedStandard graphics need to be determined
Add ability to put changed forecasts and observed values back into Add ability to put changed forecasts and observed values back into original data tables and rebuild all affected pairs on-the-flyoriginal data tables and rebuild all affected pairs on-the-fly– IVP would become a more powerful, general QC tool for archive dataIVP would become a more powerful, general QC tool for archive data
5353
DocumentationDocumentation
Available on the webAvailable on the web
http://www.nws.noaa.gov/oh/hrl/verification/verification_doc_ob82.phphttp://www.nws.noaa.gov/oh/hrl/verification/verification_doc_ob82.php
5454
ExercisesExercises
Described in hand-outsDescribed in hand-outs– The data sets are very large, so while the pairs are The data sets are very large, so while the pairs are
loading, discuss the plots that are needed in order to loading, discuss the plots that are needed in order to answer the exercise questionsanswer the exercise questions What statistics are to be displayed?What statistics are to be displayed? What are the x-axis and comparison variables?What are the x-axis and comparison variables? Are there any observed or forecast category restrictions?Are there any observed or forecast category restrictions?
Group assignmentsGroup assignments
Group LX Machine Exercise 1 Location Exercise 2 Location1 lx1 BLKO2 TIFM72 lx3 GLOO2 WTTO23 lx4 MORA4 TIFM74 lx5 QUAO2 WTTO25 lx6 WTTO2 TIFM7
5555
DiscussionDiscussion
Review exercisesReview exercises Pre-workshop questionsPre-workshop questions
– Have they been answered?Have they been answered?
General comments on softwareGeneral comments on software
Top Related