Advances in Water Resources 31 (2008) 1387–1398

Contents lists available at ScienceDirect

Advances in Water Resources

journal homepage: www.elsevier .com/ locate/advwatres

Incorporating multiple observations for distributed hydrologic model calibration:An approach using a multi-objective evolutionary algorithm and clustering

Soon-Thiam Khu a,*, Henrik Madsen b, Francesco di Pierro a

a Centre for Water Systems, School of Engineering, Computer Science and Mathematics, University of Exeter, North Park Road, Exeter EX4 4QF, United Kingdomb DHI, Water, Environment and Health, Agern Allé 5, DK-2970 Horsholm, Denmark

a r t i c l e i n f o

Article history:Received 23 January 2007Received in revised form 11 July 2008Accepted 13 July 2008Available online 26 July 2008

Keywords:CalibrationDistributed modellingMulti-objectiveSelf-organising map (SOM)Multiple observationsEvolutionary algorithmsGroundwater

0309-1708/$ - see front matter � 2008 Elsevier Ltd. Adoi:10.1016/j.advwatres.2008.07.011

* Corresponding author.E-mail address: s.t.khu@exeter.ac.uk (S.-T. Khu).

a b s t r a c t

The use of distributed data for model calibration is becoming more popular in the advent of the availabil-ity of spatially distributed observations. Hydrological model calibration has traditionally been carried outusing single objective optimisation and only recently has been extended to a multi-objective optimisa-tion domain. By formulating the calibration problem with several objectives, each objective relating toa set of observations, the parameter sets can be constrained more effectively. However, many previousmulti-objective calibration studies do not consider individual observations or catchment responses sep-arately, but instead utilises some form of aggregation of objectives. This paper proposes a multi-objectivecalibration approach that can efficiently handle many objectives using both clustering and preferenceordered ranking. The algorithm is applied to calibrate the MIKE SHE distributed hydrologic model andtested on the Karup catchment in Denmark. The results indicate that the preferred solutions selectedusing the proposed algorithm are good compromise solutions and the parameter values are well defined.Clustering with Kohonen mapping was able to reduce the number of objective functions from 18 to 5.Calibration using the standard deviation of groundwater level residuals enabled us to identify a groupof wells that may not be simulated properly, thus highlighting potential problems with the modelparameterisation.

� 2008 Elsevier Ltd. All rights reserved.

1. Introduction

Application of distributed hydrological modelling has putemphasis on the use of data for model calibration and validation(e.g. [40,32]). Use of distributed measurements of different statevariables is essential in order to document the predictive capabilityand credibility of the model for prediction of internal state vari-ables within a catchment. This calls for formulation of the calibra-tion problem within a multi-objective context [30].

The idea of using multiple observations as sources of data forcalibration and validation of hydrological models is not new. Re-cently, the US National Weather Service conducted an extensivecomparative study on different distributed hydrologic models onseveral catchments in the US, known as the Distributed Model In-ter-comparison Project (DMIP). The broad aim of this study was todetermine the conditions to which distributed models are mostsuitable for implementation [45]. An interesting fact that cameout from DMIP is that although interior observations were avail-able, none of the participants took advantage of their distributed

ll rights reserved.

model to perform multi-site calibration using a multi-objectiveframework. Di Luzio and Arnold [28] undertook further calibrationexercises on the same catchments with interior observations. How-ever, they adopted a two stage process instead of a general, one-step multi-objective calibration approach. In the calibration of agroundwater model of 125 sub-catchments within the NorthRhine-Westphalia region of Germany, Bogena et al. [3] proposedto correlate the baseflow indices of the sub-catchments for theease of calibration rather than solving the complete calibrationproblem within a multi-objective framework.

As mentioned in McCabe et al. [32], multi-objective calibrationis intrinsically different from single objective model calibration.While conventional single objective calibration usually tries toidentify a set of model parameters based on the model’s abilityto reproduce a single independent observation record, multi-objec-tive calibration inherently recognises that models have multipleoutputs [32]. For instance, Madsen [30] proposed a multi-objectiveframework for the automatic calibration of a distributed hydro-logic catchment model, MIKE SHE [12], using both groundwater le-vel and runoff observations. In this case, performance indices fromindividual groundwater wells were aggregated into one objectivefunction and optimised with the performance index of the

1388 S.-T. Khu et al. / Advances in Water Resources 31 (2008) 1387–1398

catchment runoff in a two-objective optimisation framework.Meixner et al. [34] applied a multi-objective algorithm, MOCOM-UA [50], to calibrate a hydrochemical model with a total of 21hydrologic and chemical criteria to evaluate the model perfor-mance. They tested different combinations of performance criteriaand found that some combinations of four criteria gave better per-formance than others. van Griensven and Bauwens [13] proposed amethodology that handles multiple observations by convertingmultiple objectives functions into a single objective function, sothat it is amenable to an existing single objective global optimisa-tion algorithm. This is a very neat way to handle multiple objec-tives but it does not provide any information of the trade-offbetween different simulated outputs. Such trade-off informationmay provide the modeller with vital knowledge on the deficienciesof the model parameterisation and model structure.

Model calibration using multiple sources of data has severalimportant advantages. First, it better constrains the calibrationprocess, resulting in better defined model parameter estimates[35,44,17,18]. In addition, by incorporating new type of informa-tion in the calibration the model prediction uncertainty may be re-duced (e.g. [27,19]). Parameter non-uniqueness in terms ofequifinality [2] can be partly attributed to single objective calibra-tion where multiple model outputs (either temporal or spatial) aremapped to some form of singular index that is optimised. Multi-objective analysis may be seen as a way to unfold the equifinalityproblem by looking at model performance with respect to differentmodel responses [31].

From a calibration and optimisation perspective, a multi-objec-tive formulation of a problem could have an attractive capability ofcreating a smooth transition for parts of the objective functions, acharacteristic known as ‘‘multi-objectivisation” [24,48]. Suchsmoothing effect could allow the calibration algorithm to escapefrom multiple local optima, which is characteristic for many hydro-logical calibration problems (e.g. [10]). Although there is no math-ematical proof that this is the case for water resources calibrationproblems, empirical studies have indicated accelerated searchcapabilities by converting a single objective problem to a multipleproblem [41,23].

A further justification of adopting the multi-objective approachis that it allows the model results to be incorporated into a multi-criteria decision support framework. The resultant output of themulti-objective calibration is some form of Pareto trade-off curvebetween different objective functions, and each combination ofobjective functions is a possible option or choice for the modelleror decision maker. But very often, the number of choices presentedto the decision maker should be constrained to a limited number inorder to assist the decision maker assessing the implication of eachoption. In Khu and Madsen [22], it has been shown that preferenceordering is an effective method that could sieve through myriadoptions and promote the selection of a small group of good com-promised solutions.

However, the answer to the question of how to effectively usemultiple observations for calibration remains elusive despite manyyears of research in hydrological modelling. This paper investigateshow to incorporate multiple responses (multi-variable and multi-site measurements) and multiple performance criteria within amulti-objective calibration framework. The paper proposes thateach set of observations should be considered independently andformulated as a different objective function when performing cal-ibration. However, such a radical formulation cannot be effectivelysolved by any current optimisation algorithm [20,39]. In order toeffectively handle several spatially distributed observations, anew approach using a non-linear classifier based on artificial neu-ral networks and a multi-objective genetic algorithm based onpreference ordered ranking is proposed and applied to a distrib-uted hydrological model.

2. Multi-objective calibration framework

In a multi-objective context, model calibration can, in general,be performed for multiple responses. These responses can be di-vided into the following three groups [30]:

� Multi-variable measurements. These refer to different types ofobservations within the modelling domain such as groundwaterlevel, sub-catchment runoff, water quality parameters, soilmoisture content in the unsaturated zone, etc. Usuallydistributed hydrological models simulate several of these vari-ables. In such conditions, each of these measurements can beformulated as a performance objective for multi-objectivecalibration.

� Multi-site measurements. These refer to the same type of vari-able observed or measured at different locations distributedwithin the modelling domain. Several variables simulated byhydrological models are site-specific (such as groundwater lev-els and soil moisture) and measurements of these variablescan be formulated as a performance objective for multi-objec-tive calibration. Besides point measurements it is also importantto include runoff measurements in the calibration to evaluatethe water balance simulations at sub-catchment level. Individ-ual runoff measurements in the catchment can be formulatedas a performance objective.

� Multi-criteria modes. These refer to formulating different per-formance indices for either the same time series or for differentparts of the time series. For example, different objective func-tions can be formulate that (i) measure various responses ofthe hydrological processes such as, e.g. the general water bal-ance, peak flows, and low flows [29]; (ii) partition the observedhydrograph into different components [4]; or (iii) using differentmathematical formulations to analyse residual errors, such asroot-mean-square errors, maximum error, etc. [14]. The ASCETask committee [1] provides a comprehensive list of differentways of evaluating model performance.

Usually the performance measures to be used as objective func-tions in the calibration are derived from the single time series, suchas the root-mean-squared error (RMSE), mean average error, Nash–Sutcliffe coefficient [36], etc. Recently, Wealands et al. [49] high-lighted several problems associated with current practices inassessing spatial predictions from distributed models. They pro-posed a number of comparative measures such as ‘‘importancemap”, ‘‘weighted local variance”, ‘‘category comparison” and ‘‘fuz-zy map” for comparing distributed model outputs with spatialobservations. These comparative measures can all be taken into ac-count if the calibration is performed within a multi-objectivecontext.

To solve the multi-objective calibration problem numericaloptimisation tools can be applied. Mathematically, the multi-objective optimisation problem can be formulated as

Min fF1ðhÞ; F2ðhÞ; . . . ; FmðhÞg h 2 H; ð1Þ

where m is the number of objective functions, Fi(h), i = 1,2, . . . ,m arethe individual objective functions, and h is the set of model param-eters to be optimised. We consider a constrained optimisation prob-lem with a feasible parameter space H, which reflects the à prioriinformation of the model parameters. Due to trade-offs betweenthe different objectives, the solution to Eq. (1) will not, in general,be a single unique parameter set. Instead, it will consist of severalnon-dominated or Pareto optimal solutions according to thetrade-offs. For a Pareto optimal solution none of the objectivescan be further improved without deterioration of one or more ofthe other objectives.

Input raw data

Estimate the no. of clusters (range)

Cluster the raw data using Kohonen self-

organising map

Formulate objective functions for each cluster of data

Run POGA to calibrate the model

S.-T. Khu et al. / Advances in Water Resources 31 (2008) 1387–1398 1389

There are a number of computing algorithms which can handlemultiple objective functions without the need to resort to any formof aggregation or weighting. Examples are a large class of algo-rithms collectively known as multi-objective genetic algorithms(MOGA) [11], multi-objective variants of the shuffled complex evo-lution algorithm (the multi-objective complex evolution (MOCOM)algorithm [50] and the multi-objective shuffled complex evolutionmetropolis (MOSCEM) algorithm [47]), and others [42]. These algo-rithms have recently been applied for multi-objective calibrationof distributed hydrological models (e.g. [43,31,46]) and also forother types of rainfall runoff models [7,6,5].

Output: sets of compromise Pareto optimal parameters

Fig. 1. Flowchart of proposed methodology.

3. Proposed methodology

To solve the multi-objective optimisation problem we considerhere multiple objective functions in terms of performance indicesrelated to individual observed time series of hydrological variablesand possible individual response modes. Thus, potentially we mayhave a very large number of objective functions to be optimised.State-of-the-art multi-objective optimisation algorithms incurserious performance deterioration when exploited to solve prob-lems consisting of more than 3–4 objectives [38]. The reasons aretwofold. Firstly, finding a good representation of the optimaltrade-off curve of a problem consisting of a large number of objec-tive functions requires an exponentially high number of functionalevaluations, and this leads to computation times that are oftenunaffordable. Secondly, most of the traditional population-basedalgorithms for multi-objective optimisation suffer from a lack ofdiscriminating power when assessing the relative quality of solu-tions that must be evaluated on a large set of objectives, and thishas a serious impact on their performance.

To effectively solve a multi-objective optimisation problemwith a large number of objectives a two-step approach is here pro-posed. This includes:

(i) classification of multi-site measurements into groupsaccording to temporal dynamics using an artificial neuralnetworks (ANN); and

(ii) calibration using an automatic scheme utilising the latestdevelopments in multi-objective optimisation with manyobjectives.

The approach is illustrated in Fig. 1 and explained herein.

3.1. Grouping of multi-site measurements

To better constrain the model calibration it is, in general, pref-erable to include new data of a different variable rather than moredata of the same variable, and new data at a different locationrather than more data at existing locations (e.g. [33]). However,when considering multi-site measurements, each new data seriesdoes not provide independent, non-commensurable information.Thus, rather than considering each data series independently inthe optimisation, one should utilise the dependency between thedata series to diminish the inclusion of redundant information. Itis therefore necessary to investigate the amount of useful informa-tion content present in the observation records [19]. The depen-dency between the observation records is also a usefulcharacteristic that can be exploited for grouping some of the obser-vations in order to reduce the complexity of the problem. As such,grouping of measurement sets can be seen as both crucial andpragmatic.

Grouping could be performed in various ways, such as based onphysical proximity, expert knowledge, or using data mining tech-niques. In this study, a data mining technique known as artificial

neural network is used. We have chosen an unsupervised ANN,the Kohonen self-organising map (SOM) [26], as a tool to assistin classifying the groundwater level observations from differentwells since it has the capability of preserving the topological struc-ture of the original data, and it creates a topology preserving mapin the training process. A topological map is simply a mapping thatpreserves neighbourhood relations.

The Kohonen SOM is a highly effective tool for visualizing high-dimensional, complex data with inherent relationships betweenthe various features comprising the data. As such, it can be usedto extract salient, multi-scale features from the raw data and there-by constructing or automatic formatting data into clusters. Formore information on Kohonen SOM and its applications, see Hay-kin [15].

3.2. Multi-objective calibration using preference ordering

The grouping of multi-site measurements will effectively re-duce the number of objective functions to be included in the mul-ti-objective optimisation. However, still the number of objectivefunctions may be too large (say more than 3–4) for current mul-ti-objective optimisation algorithms to be effectively applied. Themulti-objective calibration is here performed using a newly devel-oped multi-objective genetic algorithm known as preference ordergenetic algorithm (POGA) [37]. POGA has been found to be veryeffective in dealing with many objectives (i.e. more than threeobjective functions) both for hypothetical problems as well as forwater resources calibration problems [22,21].

The basic working principles of POGA, i.e. the use of multiplegenerations where good solutions are probabilistically more oftenselected to exploit their genetic materials and form new good can-didate solutions, are similar to those of the elitist non-dominatedsorted genetic algorithm, NSGA-II, [8], a well-known algorithm thathas been successfully used to solve many difficult test problemsand complex applications. POGA differs from NSGA-II in the wayelitism, i.e. the maintenance of good solutions across generations,is performed. As opposed to NSGA-II, which uses Pareto dominanceto assess the relative quality of solutions, POGA resorts to Prefer-ence Ordering. Preference Ordering is a generalisation of Paretodominance, however it is more stringent, and therefore more effec-tive in achieving a better grading of a set of solutions to a problemthat consists of many objective functions. With Preference OrderingPareto optimal solutions that are also Pareto optimal in differentsubspace combinations of the objective functions space are pre-ferred. For instance, consider a three-objective optimisation prob-lem. A Pareto optimal point in the three-objective space that isalso a Pareto optimal point in all of the three subspaces of

1390 S.-T. Khu et al. / Advances in Water Resources 31 (2008) 1387–1398

two-objective combinations has higher efficiency than (or domi-nates) three-objective Pareto optimal points that are only Paretooptimal in two of the three subspaces. Similarly, the points thatare Pareto optimal in two of the three subspaces of two objectivesdominate points that are Pareto optimal in only one of the subspac-es, etc.

The use of Preference Ordering has shown to have a consider-able impact on the convergence properties of the optimisationalgorithm, which is less affected by the lack of discriminatingpower of commonly applied multi-objective ranking procedures(such as those used in NSGA-II). For a detailed description of thePOGA algorithm the reader is referred to di Pierro et al. [37]. Theconcept of Preference Ordering and its use in model calibration isdescribed in Khu and Madsen [22].

Fig. 2. Map of Karup catchment showing the locations of runoff stations andgroundwater wells.

Table 1Model parameters included in the calibration and parameter bounds

Modelcomponent

Model parameter Symbol Unit Lowerlimit

Upperlimit

Saturatedzone

Hydraulicconductivity of soiltype 1

Kh1 m/s 0.00005 0.005

Hydraulicconductivity of soiltype 3

Kh3 m/s 0.0001 0.01

Unsaturatedzone

Saturated hydraulicconductivity of soil 1

Ks1 m/s 0.000001 0.0001

Van Genuchten N-parameter of soil 1

N1 – 1.2 2.5

Van Genuchten a-parameter of soil 1

a1 m�1 0.05 0.5

Saturated hydraulicconductivity of soil 2

Ks2 m/s 0.00005 0.005

Van Genuchten N-parameter of soil 2

N2 – 1.2 2.5

Van Genuchten a-parameter of soil 2

a2 m�1 0.01 0.1

Drainage Drainage level DrainLevel m �1.3 �0.8Drainage coefficient DrainCoef m/s 1 � 10�8 1 � 10�6

River–aquiferinteraction

Leakage coefficient LeakCoef m/s 1 � 10�8 1 � 10�6

4. Application example: catchment and model setup

4.1. Catchment

The proposed calibration approach was used to calibrate theMIKE SHE model applied to the Danish Karup catchment, whichhas previously been used in Madsen [30] for a two-objective cali-bration, considering an aggregate performance measure of allgroundwater level measurements and a performance measure ofthe runoff at the catchment outlet. The Karup catchment has anarea of 440 km2 and is located in the western part of Denmark.The catchment elevation varies from about 20 m to 100 m. Thegeology is relatively homogeneous with highly permeable sandand gravel deposits and small lenses of moraine clay. The aquiferis mainly unconfined and varies in thickness from about 10 m atthe western and central part to more than 90 m at the upstreameastern water divide. The depth of the unsaturated zone variesfrom 25 m at the eastern water divide to less than 1 m in the wet-land areas along the river. The land use consists of agriculture(67%), forest (18%), heath (10%), and wetland areas (5%). The catch-ment is drained by the Karup River and about 20 tributaries. Amore detailed description of the Karup catchment can be foundin Refsgaard [40].

Daily precipitation data are available from nine measurementstations in the catchment, and data from these stations are spatiallydistributed in the model using Thiesen polygons. Daily potentialevapotranspiration and average temperature data are availablefrom one measurement station and are used as spatial homoge-neous input in the model. For each of the four vegetation types timeseries of leaf area index and root depth are available [40].

The available measurements for calibration consist of ground-water level data sampled every 2 weeks from 35 locations in thecatchment and daily discharge data from four stations in the riversystem, including the runoff at the catchment outlet. Similar to thecalibration performed by Madsen [30], groundwater level datafrom 17 wells as well as runoff data from the catchment outletare used (Fig. 2) for the multi-objective calibration in this study.The remaining 18 groundwater wells are used for validation ofthe multi-site classification procedure (see below). Data in the per-iod 1 January 1971–31 December 1974 are used in the calibration.To minimise the effect from the initial conditions for calculation ofthe objective functions, a 2-year warm-up period is applied in thesimulations. For validation, data in the period 1 January 1975–31December 1977 are used.

4.2. Model setup and parameterisation

MIKE SHE is a flexible, integrated hydrological modellingsystem that combines different process-oriented modelling com-ponents within the same modelling framework. For each compo-

nent several model descriptions are available ranging fromcomplex, physically-based descriptions that solve the governingpartial differential equations to simple, conceptual models. In thepresent application the following model descriptions are applied:(i) The Kristensen–Jensen model for calculating actual evapotrans-piration [25], (ii) 2D diffusive wave approximation of the Saint Ve-nant equations for overland flow, (iii) Muskingum routing for flowin the river system, (iv) 3D Boussinesq equation for flow in the sat-urated zone, (v) 1D Richards’ equation for vertical flow in the

Well Group A

0

0.2

0.4

0.6

0.8

1

1.2

01/06

/71

01/01

/72

01/08

/72

01/03

/73

01/10

/73

01/05

/74

01/12

/74

01/07

/75

01/02

/76

01/09

/76

date

Nor

mal

ised

wat

er le

vels

Well Group B

0

0.2

0.4

0.6

0.8

1

1.2

01/06

/71

01/01

/72

01/08

/72

01/03

/73

01/10

/73

01/05

/74

01/12

/74

01/07

/75

01/02

/76

01/09

/76

date

Nor

mal

ised

wat

er le

vels

Well Group C

0

0.2

0.4

0.6

0.8

1

1.2

01/06

/71

01/01

/72

01/08

/72

01/03

/73

01/10

/73

01/05

/74

01/12

/74

01/07

/75

01/02

/76

01/09

/76

date

Nor

mal

ised

wat

er le

vels

Well Group D

0

0.2

0.4

0.6

0.8

1

1.2

01/06

/71

01/01

/72

01/08

/72

01/03

/73

01/10

/73

01/05

/74

01/12

/74

01/07

/75

01/02

/76

01/09

/76

date

Nor

mal

ised

wat

er le

vels

Fig. 3. Normalised water levels in the four groups of the groundwater wells using the Kohonen SOM classification scheme (grey lines: calibration wells; red dotted lines:validation wells). (For interpretation of the references in colour in this figure legend, the reader is referred to the web version of this article.)

S.-T. Khu et al. / Advances in Water Resources 31 (2008) 1387–1398 1391

Table 2Classification of wells according to observed dynamics in piezometric levels

Groups Calibration wells identificationnumber

Validation wells identificationnumber

A 9, 22, 39, 78 8, 38, 41, 72B 5, 12, 21, 24, 64 11C 27, 35, 37, 45, 69 6, 34, 44, 52, 62, 63D 49, 55, 56 25, 36, 46, 47, 51, 54, 66

0.0

0.5

1.0STD_Well5

STD_Well9

STD_Well27STD_Well49

RMSE_20.05

Fig. 5. Radar plot of normalised values for the five- and nine-objective calibrationproblem. STD_Well5, STD_Well9, STD_Well27 and STD_Well49 are standarddeviation of residuals for the different wells representing, respectively, group A,B, C and D. RMSE_20.05 is the RMSE of runoff at the catchment outlet. (grey lines:74 Pareto optimal parameters for the five-objective calibration, i.e. set A; red dottedlines: preferred parameters according to preference ordering for the five-objectivecalibration problem, i.e. set B; black lines: preferred parameters according topreference ordering for set nine-objective calibration problem, i.e. set C). (Forinterpretation of the references in colour in this figure legend, the reader is referredto the web version of this article.)

1392 S.-T. Khu et al. / Advances in Water Resources 31 (2008) 1387–1398

unsaturated zone, (vi) Linear reservoir model for flow in drains,(vii) Darcy equation for the river–aquifer interaction, and (viii) Adegree-day approach for snow melt. For a detailed description ofthe MIKE SHE modelling system, the reader is referred to Grahamand Butts [12].

The model is discretised in a 1 � 1 km horizontal grid. For thesaturated zone modelling the geological conceptualisation is takenfrom the Danish National Water Resources model (DK-model)which is defined in 1 � 1 km grids and 10 m thick layers [16]. Foreach grid element a soil type is assigned (a total of five soil typesare identified for the Karup catchment), and the hydraulic proper-ties of these soils are included in the calibration. For the unsatu-rated zone the parameterisation used by Refsgaard [40] isapplied. This includes definition of two soil profiles for the entirecatchment, each defined with two soil types (a top layer rangingfrom 55 to 100 cm and a homogenous layer below). For each ofthe resulting four soil types van Genuchten retention and conduc-tivity curve parameters are specified, and the parameters of theseequations are included in the calibration.

The Karup River and the main tributaries are included in theriver model. To describe river–aquifer interaction a thin permeablelayer is assumed between the river and the main aquifer. The leak-age coefficient that characterises this layer is assumed homoge-nous in the catchment and is included in the calibration. Thewetland areas are drained by ditches and drain pipes which aremodelled conceptually using a linear reservoir description in eachcell. The drainage level (relative to ground surface) and the timeconstant of the linear reservoir model are assumed homogeneousin the catchment and are included in the calibration. The empiricalparameters in the Kristensen–Jensen model for the evapotranspira-tion calculation are based on experience values [40] and are notsubject to calibration.

0.0

0.2

0.4

0.6

0.8

1.0

Kh_1

Kh_3

DrainL

evel

DrainC

onst

Leak

Coef

Alpha_

1

Parame

No

rmal

ised

val

ues

Fig. 4. Normalised parameters for Pareto optimal sets (grey lines: 74 Pareto optimal pparameters according to preference ordering for the five-objective calibration problem,nine-objective calibration problem, i.e. set C). (For interpretation of the references in co

A preliminary sensitivity analysis was carried out to identify themost sensitive parameters [9]. Based on this analysis 11 parame-ters were included in the multi-objective calibration (see Table 1).

4.3. Classification of groundwater wells

In order to reduce the complexity of the multi-objective calibra-tion problem, the 17 calibration wells were classified into distinc-tive groups according to the fluctuations of observed piezometrichead using the Kohonen SOM classification procedure describedabove. To facilitate the classification, the groundwater level datawere transformed using the ground level as the datum and norma-lised using the mean water level in each well. The Kohonen SOMprocedure was able to classify the 17 calibration wells into fourgroups (Fig. 3). This Kohonen network was later used to sort out

N_1Ks_

1

Alpha_

2N_2

Ks_2

ters

arameters for the five-objective calibration, i.e. set A; red dotted lines: preferredi.e. set B; black lines: preferred parameters according to preference ordering for setlour in this figure legend, the reader is referred to the web version of this article.)

0.00

0.50

1.00

1.50

2.00

2.50

0.00 0.50 1.00

STD_Well group B

RM

SE_r

unof

f_20

.05

0.00

0.50

1.00

1.50

2.00

2.50

0.00 0.50 1.00 1.50STD_Well group A

RM

SE_r

unof

f_20

.05

0.00

0.50

1.00

1.50

2.00

2.50

0.00 0.20 0.40 0.60 0.80

STD_Well group D

RM

SE_r

unof

f_20

.05

0.00

0.50

1.00

1.50

2.00

2.50

0.00 0.20 0.40 0.60

STD_Well group C

RM

SE_r

unof

f_20

.05

0.00

0.10

0.20

0.30

0.40

0.50

0.60

0.70

0.80

0.90

0.00 0.50 1.00 1.50

STD_Well group A

STD

_Wel

l gro

up B

0.00

0.10

0.20

0.30

0.40

0.50

0.60

0.00 0.50 1.00 1.50

STD_Well group A

STD

_Wel

l gro

up C

0.00

0.10

0.20

0.30

0.40

0.50

0.60

0.70

0.00 0.50 1.00 1.50

STD_Well group A

STD

_Wel

l gro

up D

0.00

0.10

0.20

0.30

0.40

0.50

0.60

0.00 0.20 0.40 0.60 0.80 1.00

STD_Well group B

STD

_Wel

l gro

up C

0.00

0.10

0.20

0.30

0.40

0.50

0.60

0.70

0.80

0.90

0.00 0.20 0.40 0.60 0.80STD_Well group D

STD

_Wel

l gro

up B

0.00

0.10

0.20

0.30

0.40

0.50

0.60

0.00 0.20 0.40 0.60 0.80STD_Well group D

STD

_Wel

l gro

up C

a b

dc

e f

g h

i j

Fig. 6. Various plots showing different combinations of any two of the five objectives (dots: all Pareto efficient points, i.e. set A; circles: set B preferred points; crosses: set Cpreferred points).

S.-T. Khu et al. / Advances in Water Resources 31 (2008) 1387–1398 1393

1394 S.-T. Khu et al. / Advances in Water Resources 31 (2008) 1387–1398

the remaining 18 validation wells into one of these groupings. Thereason for doing so is to confirm that the trained Kohonen networkwas indeed able to perform the classification of groundwater wells.The resultant groupings are shown in Table 2, and Fig. 3 shows thegraphical representations of the water levels in the validation wellsnormalised by the mean water level in each well. From Fig. 3, thewater level dynamics of each group is quite distinct: group A wellsare less dynamic compared to other groups as there are only twomajor trends; from June 1971 to December 1974, and from July1975 to January 1977. Group B wells are similar to group A wells

runoff@ locat

0

1

2

3

4

5

6

7

8

9

10

11

12

1970 1971 19time

disc

harg

e (m

3 /s)

Fig. 7. Hydrograph at catchment outlet compared with observations for all 74 optimal

Fig. 8. Simulation results of water levels in some calibration groundwater wells comsimulations, i.e. set A; blue lines: set B preferred solutions; black lines: set C preferred soluis referred to the web version of this article.)

except for some slight fluctuations at around February 1976. GroupC and group D wells behave similarly in that they fluctuate quite alot during the study period but, in general, group D wells fluctuatewith a larger magnitude compared to group C wells.

4.4. Formulation of objective functions based on well groupings

Once the wells have been classified, a representative well fromeach group is selected for use in the calibration. Since the dynamicsof the wells within each group are similar to each other, the overall

ion 20.05

72 1973 1974(year)

simulations (crosses: observed values; grey lines: 74 Pareto optimal simulations).

pared with observations (crosses: observed values; grey lines: 74 Pareto optimaltions). (For interpretation of the references in colour in this figure legend, the reader

S.-T. Khu et al. / Advances in Water Resources 31 (2008) 1387–1398 1395

results are not sensitive to the well selected for calibration. Analy-sis of the model residuals from the calibrated model at the differ-ent wells also support this finding. In this paper, wells 9, 5, 27 and49 are selected as calibration wells to represent groups A, B, C andD, respectively. A total of nine objective functions were formu-lated: two for each group of wells and one for the runoff measure-ments at the catchment outlet. The objective functions weredefined as:

(i) root-mean-square error (RMSE) of the groundwater levels;(ii) standard deviations of the groundwater level residuals;

(iii) RMSE of the runoff at the catchment outlet.

The use of the standard deviations of residuals as objectivefunctions to evaluate groundwater level simulation is motivatedby the fact that large biases may exist due to scaling problems,i.e. the dissimilarity between the measurement scale (at a point)and the modelling scale (on a grid) and the associated heterogene-ity within the model grid. The standard deviation of model residu-als measures the dynamic behaviour of the model response, andhence when used in the calibration allows ignoring the bias. Thebias problem is important to take into account when comparingany point measurement with a grid-based model response. Onthe other hand, catchment aggregated values such as catchment(or sub-catchment) runoff may be directly compared since theyare referring to the same scale (the catchment or sub-catchmentscale). Two different calibration setups were tested using POGA,one using five objective functions (from (ii) and (iii)) and the otherwith all the nine objective functions. This was done in order toevaluate the effect of including more objective functions in the cal-ibration to better constrain the parameter optimisation.

4.5. Setup of optimisation algorithm

The 11 parameters of the simulation model were encoded asbinary variables in POGA. Here, binary coding was preferred to real

Fig. 9. Simulation results of water levels in some validation groundwater wells compsimulations, i.e. set A; blue lines: set B preferred solutions; black lines: set C preferred soluis referred to the web version of this article.)

coding because preliminary sensitivity-type analysis on algorith-mic parameterisation was not viable due to the considerable sim-ulation time and the authors had prior expertise on parameterisingthe binary-coded optimisation algorithm herein used on similarproblems. The variables with a feasible range covering several dec-ades (see Table 1) were logarithmic transformed to better repre-sent the search space. After a number of trial runs with differentselection and recombination operators, the bit wise tournamentselection was implemented together with the uniform crossoverand uniform mutation operators. The population size, the probabil-ity of crossover and mutation were set to 200, 0.5 and 1/200,respectively. The maximum number of iterations was set to 100.

The simulation time of each model setup required approxi-mately 5 min on a Pentium dual core 2.0 GHz, leading to a totalduration of the optimisation process of approximately 1650 h. Thisprevented running a series of independent optimisation runs to as-sess and filter out the impact of randomness affecting POGA(mainly in the form of probability of occurrence of the geneticoperators) on the results.

5. Results and discussion

In the five-objective calibration, there were 74 sets of parametercombinations that satisfy the condition of Pareto efficiency. Fig. 4shows the variations of the set of Pareto optimal parameters (setA) scaled with respect to the feasible ranges given in Table 1. Whenthe conditions of preference ordering were applied to distinguishbetween these 74 sets of parameters, five sets of parameters wereidentified as non-dominated (set B, indicated as dotted red lines inFig. 4). Fig. 5 shows a radar plot with all 74 sets of the five objectivefunction values standardised with respect to their maximum val-ues. It can be seen that the objective functions STD_Well9,STD_Well5 and STD_Well27 (representing standard deviations ofwell groups A, B and C, respectively) behaved very much in tandemwith each other, i.e. changing a parameter set will either increaseor decrease all these three objective functions most of the time.

ared with observations (crosses: observed values; grey lines: 74 Pareto optimaltions). (For interpretation of the references in colour in this figure legend, the reader

0.0

0.2

0.4

0.6

0.8

1.0RMSE_Well9

RMSE_Well5

RMSE_Well27RMSE_Well49

RMSE_20.05

Fig. 10. Radar plot of normalised values for RMSE of wells and runoff. RMSE_Well9,RMSE_Well5, RMSE_Well27 and RMSE_Well49 are RMSE for the different wellsrepresenting, respectively, group A, B, C and D. RMSE_20.05 is the RMSE of runoff atthe catchment outlet (grey lines: 74 Pareto optimal parameters, i.e. set A; reddotted lines: set B preferred parameters; black lines: set C preferred parameters).(For interpretation of the references in colour in this figure legend, the reader isreferred to the web version of this article.)

1396 S.-T. Khu et al. / Advances in Water Resources 31 (2008) 1387–1398

It can be seen that the preferred set B parameters (dotted red lines)have very good performance in general with small and similarRMSE values for runoff and standard deviations for selectedgroundwater wells. However, as shown in Fig. 4, there is still alarge variability in some parameter values in set B. In other words,these parameters are quite ill-defined with respect to the objectivefunctions selected. It is expected that when more performancemeasures are considered as objective functions, the calibrationproblem will be better constrained, resulting in better identifiableparameters.

Fig. 11. Simulation results compared with observations in a groundwater well that exhibset A; blue lines: set B preferred solutions; black lines: set C preferred solutions). (For intthe web version of this article.)

In the nine-objective calibration where both RMSE and standarddeviation of groundwater levels as well as the RMSE of runoff wereconsidered, 3 sets of parameters were identified using POGA (set C,shown in black lines in Fig. 4) and their ranges were more narrowcompared to those from the five-objective calibration. This indi-cates that the preferred solutions are better defined compared tothe five-objective function case. The ranges of some of theseparameters (Kh1, Kh3, DrainConst, LeakCoef, N1, Ks1) were consid-erably narrower than those in the five-objective calibration. Thisshows that through POGA, we were able to identify 7 out of 11parameters. There are still considerable variations in some param-eters (such as DrainLevel, a2, N2 and Ks2). Fig. 5 also shows the nor-malised objective function values of this calibration (in back lines)compared to those from the five-objective calibration.

Fig. 6 shows the different combinations of two objectives for thefive-objective calibration problem. It can be seen clearly that pref-erence ordering offers a useful approach to sieve through all thePareto efficient points and select points that are good compromisesolutions in each of the two-objective combinatorial plots (Fig. 6a–j). However, when compared with the preferred solutions from thenine-objective calibration, most of the preferred five-objectivesolutions are dominated by the preferred nine-objective solutionsexcept for those in Fig. 6a, b and e. Hence, the use of more objectivefunctions has clear benefits in discriminating very good solutionsfrom the preferred solutions based on fewer objective functions.Fig. 7 compares the simulated runoff at the catchment outlet forthe 74 Pareto optimal parameter sets of the five-objective calibra-tion (set A) with the observed hydrograph. The range of simulatedrunoff is seen to bracket well the observed runoff.

Fig. 8 compares the groundwater level simulations for the set APareto optimal solutions with the observations in four wells usedfor calibration. It can be seen that for all wells, the dynamics ofthe water levels can be captured reasonably well by all Pareto opti-mal parameter sets. This also applies to the groundwater levels inother calibration wells. Fig. 9 compares the set A groundwater levelsimulations with observations for some validation wells. Since

its phase error (crosses: observed values; grey lines: 74 Pareto optimal simulations,erpretation of the references in colour in this figure legend, the reader is referred to

S.-T. Khu et al. / Advances in Water Resources 31 (2008) 1387–1398 1397

these wells were not used for calibration, we would expect that theresults are not as good as for calibration wells, but neverthelessgood. The RMSE of the validation wells are around 1.19–1.44(m)-compared to 0.98–1.26 (m) for the calibration wells.

At some well locations (see e.g. well 27 in Fig. 8), there are con-siderable differences between simulated and observed water lev-els. These differences may be up to about 3 m. In the Karupcatchment, the groundwater table is characterized by a high spatialgradient, up to about 3.5 m per km. Since model results are repre-sentative of a 1 � 1 km grid scale, the errors in simulated ground-water levels are within the acceptable limits when scalinguncertainties are taken into account. The radar plot of RMSE ofgroundwater levels (Fig. 10) clearly indicates that there is consid-erable difficulty in reducing the RMSE of well group C representedby well 27. On the other hand, Fig. 5 shows a small standard devi-ation of the preferred solutions for well 27, indicating a good dy-namic description (see also Fig. 8).

It can also be noted that there are a few groundwater wells (cal-ibration wells 49 and 55, and validation well 51) that cannot beadequately simulated in terms of temporal dynamics, and themodel results always contain some form of phase error as shownin Fig. 11. These wells all belong to group D and are located atthe upstream part of a tributary in the western part of the catch-ment (see Fig. 2). The phase shift is also illustrated in the relativelypoor standard deviation results for this group of wells compared tothe other well groups (Fig. 5). Thus, this may indicate errors in themodel related to the parameterisation or, more generally, modelstructural errors in this part of the model domain which shouldbe further investigated.

6. Conclusions

A new multi-objective calibration approach incorporating mul-ti-variable, multi-site and multi-response measurements has beenproposed in this paper. The methodology consists of two steps: (i)grouping of multi-site measurements into clusters using KohonenSOM, and (ii) multi-objective optimisation with POGA, which isparticularly powerful for dealing with many objectives.

The methodology has been demonstrated on calibration of aMIKE SHE model setup of the Karup catchment in Denmark. Mea-surements of groundwater levels and runoff were considered in themulti-objective context. The groundwater observations were takenfrom different measurement locations within the catchment, andtwo different criteria for assessing goodness-of-fit (i.e. RMSE andstandard deviations of simulation residuals) were considered. Itwas found that clustering played a vital role in the methodologyby: (i) reducing the number of representative observation wellsby grouping wells in clusters of similar behaviour, and (ii) reducingthe dimensionality of the multi-objective calibration problem.Hence, it provided a balanced approach between using a largenumber of observation well data with individual objective func-tions and a single objective function that aggregates all data intoone performance measure. It was also found that the wells werenot clustered solely based on geographical locations but alsoaccording to dynamical behaviour of the observed water levels.Although through clustering, we were able to reduce the numberof objective functions from 18 to 5 (considering only standarddeviations of the simulated groundwater residuals) and from 35to 9 (considering also RMSE of groundwater levels), automatic cal-ibration would not be possible without the use of a powerful mul-ti-objective optimisation algorithm, POGA.

Overall, the results obtained were very encouraging in that thedynamics in the groundwater level observations can be modelledadequately together with the catchment runoff. At the same time,the multi-objective calibration approach provides the modeller a

choice of calibrated parameter sets, which the modeller can selectbased on other considerations.

For some of the simulated groundwater levels that exhibitphase shifts, and which cannot be reduced by the proposed meth-odology, one should re-examine the observations and the appliedmodel parameterisations of the different process descriptions. Thiswas, however, not further elaborated in the current project.

References

[1] ASCE Task Committee. Definition of criteria of evaluation of watershed models.J Irrig Drain Eng, ASCE 1993;119(3):429–42.

[2] Beven K, Binley AM. The future of distributed models: model calibration anduncertainty prediction. Hydrol Process 1992;6:279–98.

[3] Bogena H, Kunkel R, Schobel T, Schrey HP, Wendland F. Distributed modellingof groundwater recharge at the macroscale. Ecol Model 2005;187:15–26.

[4] Boyle DP, Gupta HV, Sorooshian S. Toward improved calibration of hydrologicmodels: combining the strengths of manual and automatic methods. WaterResour Res 2000;36(12):3663–74.

[5] Cheng C-T, Zhao M, Chau KW, Wu X. Using genetic algorithm and TOPSIS forXinanjing model calibration with a single procedure. J Hydrol 2006;316(1–4):129–40.

[6] Cheng C-T, Wu X, Chau KW. Multiple criteria rainfall-runoff model calibrationusing a parallel genetic algorithm in a cluster of computer. Hydrol Sci J2005;50(6):1069–88.

[7] Cheng C-T, Oh C, Chau KW. Combining a fuzzy optimal model with a geneticalgorithm to solve multiobjective rainfall-runoff model calibration. J Hydrol2002;268(1–4):72–86.

[8] Deb K, Pratap A, Agarwal S, Meyarivan T. A fast elitist non-dominated sortinggenetic algorithm for multi-objective optimisation: NSGA-II. Report No.200001. Indian Institute of Technology, Kanpur Genetic AlgorithmsLaboratory; 2000. p. 20.

[9] Dornes PF. Calibration strategy for parameter estimation in the MIKE SHEmodelling system. Master of Science thesis. International Institute forInfrastructural, Hydraulic and Environmental Engineering, IHE, DELFT, TheNetherlands.

[10] Duan QY, Sorooshian S, Gupta V. Effective and efficient global optimisation forconceptual rainfall-runoff models. Water Resour Res 1992;28(4):1015–31.

[11] Fonseca CM, Fleming PJ. Multiobjective optimisation. In: Handbook ofevolutionary computation. Oxford University Press, IOP Publishing Ltd.; 1995.

[12] Graham DN, Butts MB. Flexible, integrated watershed modelling with MIKESHE. In: Singh VP, Frevert DK, editors. Watershed models, 2006. p. 245–72.

[13] van Griensven A, Bauwens W. Multiobjective autocalibration forsemidistributed water quality models. Water Resour Res 2005;39(12):1348.doi:10.1029/2003WR002284.

[14] Gupta HV, Sorooshian S, Yapo PO. Toward improved calibration of hydrologicalmodels: multiple and non-commensurable measures of information. WaterResour Res 1998;34(4):751–63.

[15] Haykin S. Neural networks: a comprehensive foundation. Prentice Hall; 1999.[16] Henriksen HJ, Troldborg L, Nyegaard P, Sonnenborg TO, Refsgaard JC, Madsen

B. Methodology for construction, calibration and validation of a nationalhydrological model for Denmark. J Hydrol 2003;280(1–4):52–71.

[17] Horritt M. Calibration of a two-dimensional finite element flood flow modelusing satellite imagery. Water Resour Res 2000;36(11):3279–91.

[18] Hunter NM, Bates P, Horritt M, de Roo A, Werner M. Utility of different datatypes for calibrating flood inundation models within a GLUE framework.Hydrol Earth Syst Sci 2005;9(4):412–30.

[19] Khadam IM, Kaluarachchi JJ. Use of soft information to describe the relativeuncertainty of calibration data in hydrologic models. Water Resour Res2004;40:W11505. doi:10.1029/2003WR002939.

[20] Khare V, Yao X, Deb K. Performance scaling of multi-objective evolutionaryalgorithms. In: Fonseca Carlos M, Fleming Peter J, Zitzler Eckart, DebKalyanmoy, Thiele Lothar, editors. Evolutionary multi-criterion optimization.Second international conference, EMO 2003, Lecture notes in computerscience, vol. 2632. Faro, Portugal: Springer; 2003. p. 376–90.

[21] Khu S-T, diPierro F, Savic D, Djordjevic S, Walters GA. Incorporating spatial andtemporal information for urban drainage model calibration: an approach usingpreference ordering genetic algorithm. Adv Water Resour 2006;29(8).doi:10.1016/j.advwatres.2005.09.009.

[22] Khu S-T, Madsen H. Multiobjective calibration with Pareto preferenceordering: an application to rainfall-runoff model calibration. Water ResourRes 2005;41(3). doi:10.1029/2004WR003041.

[23] Khu S-T, Keedwell E. Introducing choices (flexibility) in upgrading of waterdistribution network: The New York City tunnel network example. Eng Opt2005;37(3):291–305.

[24] Knowles J, Watson A, Corne D. Reducing local optima in single-objectiveproblems by multiobjectivization. In: 1st International conference onevolutionary multi-criterion optimization, 2001. p. 268–82.

[25] Kristensen KJ, Jensen SE. A model for estimating actual evapotranspirationfrom potential evapotranspiration. Nordic Hydrol 1975;6:70–88.

[26] Kohonen T. Self-organized formation of topologically correct feature maps.Biol Cybern 1982;43:59–69.

1398 S.-T. Khu et al. / Advances in Water Resources 31 (2008) 1387–1398

[27] Kuczera G, Mroczkowski M. Assessment of hydrologic parameter uncertaintyand the worth of multi-response data. Water Resour Res 1998;35:1481–9.

[28] Di Luzio M, Arnold JG. Formulation of a hybrid calibration approach for aphysically based distributed model with NEXRAD data input. J Hydrol2004;298:136–54.

[29] Madsen H. Automatic calibration of a conceptual rainfall-runoff model usingmultiple objectives. J Hydrol 2000;235:276–88.

[30] Madsen H. Parameter estimation in distributed hydrological catchmentmodelling using automatic calibration with multiple objectives. Adv WaterResour 2003;26:205–16.

[31] Madsen H, Khu ST. On the use of Pareto optimization for multi-criteriacalibration of hydrological models, calibration and reliability in groundwatermodelling: from uncertainty to decision making. In: Bierkens MFP, Gehrels JC,Kovar K, editors. Proceedings of ModelCARE 2005, The Hague, TheNetherlands, June 2005. IAHS Publication No. 304, 2006. p. 93–9.

[32] McCabe MF, Franks SW, Kalma JD. Calibration of a land surface model usingmultiple data sets. J Hydrol 2005;302:209–22.

[33] McLaughlin D, Townley LR. A reassessment of the groundwater inverseproblem. Water Resour Res 1996;32(5):1131–61.

[34] Meixner T, Bastidas LA, Gupta H, Bales RC. Multicriteria parameter estimationfor models of stream chemical composition. Water Resour Res2002;38(3):1027. doi:10.1029/2000WR000112.

[35] Mroczkowski M, Raper GP, Kuczera G. The quest for more powerfulvalidation of conceptual catchment models. Water Resour Res 1997;33(10):2325–35.

[36] Nash JE, Sutcliffe J. River flow forecasting through conceptual models, Part I: adiscussions of principles. J Hydrol 1970;10:282–90.

[37] di Pierro F, Khu S-T, Savic DA. An investigation on preference ordering rankingscheme in multiobjective evolutionary optimization, IEEE Trans Evol Comput2007;11(1):17–45. doi:10.1109/TEVC.2006.876362.

[38] di Pierro F. Evolutionary many-objectives optimisation and applications towater resources engineering. PhD dissertation. University of Exeter, 2006.

[39] Purshouse RC, Fleming PJ. Evolutionary multi-objective optimisation: anexploratory analysis. In: Proceedings of the 2003 congress on evolutionary

computation (CEC’2003), vol. 3. Canberra, Australia: IEEE Press; 2003. p. 2066–73.

[40] Refsgaard JC. Parameterisation, calibration and validation of distributedhydrological models. J Hydrol 1997;198:69–97.

[41] Savic DA. Single-objective vs. multiobjective optimisation for integrateddecision support. Integrated assessment and decision support. In: Rizzoli AE,Jakeman AJ, editors. Proceedings of the 1st Biennial Mtg. of int. env. modellingand software society, vol. 1. Switzerland: Lugano; 2001. p. 7–12.

[42] Savic DA, Khu S-T. Applications of evolutionary computing in hydrologicalsciences. In: Anderson MG, editor. Encyclopaedia of hydrological sciences. JohnWiley publishers; 2005. p. 177–8.

[43] Schoups G, Hopmans JW, Young CA, Vrugt JA, Wallender WW. Multi-criteriaoptimization of a regional spatially-distributed subsurface water flow model. JHydrol 2005;311:20–48.

[44] Seibert J, Uhlenbrook S, Leibundgut C, Haldin S. Multiscale calibration andvalidation of a conceptual rainfall-runoff model. Phys Chem Earth, Part B:Hydrol Oceans Atmos 2000;25(1):59–64.

[45] Smith MB, Seo D-J, Koren VI, Reed SM, Zhang Z, Duan Q, et al. The distributedmodel intercomparison project (DMIP): motivation and experiment design. JHydrol 2004;298:4–26.

[46] Tang Y, Reed P, Wagener T. How effective and efficient are multiobjectiveevolutionary algorithms at hydrologic model calibration? Hydrol Earth Syst Sci2006;10:289–307.

[47] Vrugt JA, Gupta HV, Bastidas LA, Bouten W, Sorooshian S. Effective andefficient algorithm for multi-objective optimization of hydrologic models.Water Resour Res 2003;39(8):1214. doi:10.1029/2002WR001746.

[48] Watanabe S, Sakakibara K. The effectiveness of multiobjective optimizer insingle-objective optimization environment. GECCO conference, June 25–29,Washington, DC, USA. ACM 1-59593-010-8/05/0006, 2005.

[49] Wealands SR, Grayson RB, Walker JP. Quantitative comparison of spatial fieldsfor hydrological model assessment – some promising approaches. Adv WaterResour 2005;28:15–32.

[50] Yapo PO, Gupta HV, Sorooshian S. Multi-objective global optimization forhydrologic models. J Hydrol 1998;204(1–4):83–97.