Fritz Heidenreich President Heidenreich Innovations LLC fritz@heidenreich +1.203.413.2030
Hydrologic Modeling: Verification, Validation, Calibration, and Sensitivity Analysis Fritz R....
-
Upload
donald-flowers -
Category
Documents
-
view
231 -
download
3
Transcript of Hydrologic Modeling: Verification, Validation, Calibration, and Sensitivity Analysis Fritz R....
Hydrologic Modeling: Verification, Validation, Calibration, and
Sensitivity Analysis
Fritz R. Fiedler, P.E., Ph.D.
Definitions (review)
• Verification: check if code solves equations correctly
• Validation: check if model reasonably represents physical process
• Calibration: adjust model parameters to match observations
• Sensitivity Analysis: relative effect of parameter changes on output
Verification
• Compare numerical results to analytical results
Level 1 Validation
• Compare model results to simple experiments (can estimate parameters a priori)
Calibration
• Adjust parameters to match observations
Level 2 Validation
• Compare model results to observations for a different input data set post-calibration– Reserve some data (do not use in calibration)– After finding parameters that result in “best fit,”
run model with reserved input and compare to output
• Problems with this?• What happens in practice?
Sensitivity Analysis
• Explore how parameter changes affect output
• Sensitivity index:
measureeperformancofvalueZ
iparameterofvaluex
x
dxdZ
S
i
i
ii
Calibration TargetsModel Parameters Observations
Channel Models n, S0, h-B relationship
Q, h
Watershed Models
watershed and channel
Q
Richards Equation
K(), h() head, moisture
Green-Ampt K, , flux, moisture (?)
Can physically based model parameters be measured? Why or why not?
Goodness of Fit
• Visual comparison between simulated and observed – look for trends in errors– A learned art– Use appropriate graph scales
• Statistical performance measures– Consider mean daily discharge as calibration
target– Q = observed– S = simulated
Means and Bias
N
SS
N
ii
1
N
N
ii
1
)100(
1
1
N
ii
N
iii
Q
QSbiaspercent
Common calibration strategy: fix bias first, revisit periodically, goal of no bias
• Maximum Error:
• Percent Average Absolute Error
NtoiforQSME ii 1)max(
)100(
1
1
Q
QSN
PAAE
N
iii
Sum of Squares of Errors
• Most common basis for statistical goodness of fit– e.g., least squares regression, seek to minimize
N
iii QS
1
2)(
Root Mean Squared Error
• Size of error usually related to size of events or values, thus RMSE typically smaller for dry periods, small watersheds (for example)
• How would you modify RMSE to facilitate comparison?
2/1
1
2)(
N
QSRMSE
N
iii
Percent RMSE
• Normalize RMSE by mean observed
• Because the magnitude of RMSE varies with magnitude of values, by minimizing RMSE only, which part of hydrographs are primarily best fit in calibration?
• How can this tendency be addressed?
)100(Q
RMSEPRMSE
Nash-Sutcliffe
• Very popular method of evaluating calibration
• Reading: McCuen, R. H., Evaluation of the Nash—Sutcliffe efficiency index, Journal of Hydrologic Engineering, 11(6), 597-602, 2006 (note: author uses different variables)
N
ii
N
iii
NS
QSR
1
2
1
2
2
)(
)(1
Line of Best Fit
ABSQ
Analyze as in regression: hypothesis testing on A and B, residual analysis, correlation coefficient…
Line of Best Fit – Correlation Coefficient
2/12222
iiii
iiii
QQNSSN
QSQSNR
How to Use Statistical Measures
• For a given time period, e.g., 1 year, and/or averages over multiple years
• Look for seasonal trends
Month Q S PB Bias ME PAAE RMSE PRMSE Jan Feb March … Annual
How to Use Statistical Measures
• By flow interval (value interval)
• Errors as f(Q) – aim for no systematic variation• How would you pick the intervals?
Interval N Q S Bias ME RMSE PRMSE 0-20 20-40 40-60 …
Exceedance Plots
Q, S
0 100percent days exceeded
x
x
x
x
x
x
xxx
x
Generalized Calibration Strategies
• Set realistic parameter bounds before starting• Fix insensitive parameters first; focus on most
sensitive• Eliminate most bias early in process, revisit• Use regionalized variables as appropriate• Combine manual and automatic techniques
Equifinality
• Multiple combinations of parameters can lead to similar results
• Issue with both multi-parameter lumped models (e.g., SAC-SMA) and spatially distributed models (e.g., CASC-2D)
• Reading: Ebel, B. A. and K. Loague, Physics-based hydrologic-response simulation: Seeing through the fog of equifinality, Hydrological Processes, 20(13), 2887–2900, 2006