SINGH, REED, and KELLER: MULTI-OBJECTIVE FRAMEWORK 1
Many-objective robust decision making for managing an ecosystem 1
with a deeply uncertain threshold response 2
Riddhi Singha,b, Patrick M. Reedc, and Klaus Kellerd,e,f 3
aDepartment of Civil Engineering, Indian Institute of Technology Hyderabad, Yeddumailaram, India. Ph: (+91)40-4
2301-8455, (Corresponding Author) email: [email protected] 5
bFormerly: Earth and Environmental Systems Institute, The Pennsylvania State University, 217F EES Building, 6
University Park, PA 16802, USA. 7
cSchool of Civil and Environmental Engineering, Cornell University, 212 Hollister Hall, Ithaca, NY 14583-3501, 8
USA. email: [email protected] 9
dDepartment of Geosciences, The Pennsylvania State University, 436 Deike Building, University Park, PA 16802, 10
USA. email: [email protected] 11
eEarth and Environmental Systems Institute, The Pennsylvania State University, University Park, PA 16802, USA. 12
fDepartment of Engineering and Public Policy, Carnegie Mellon University, 5000 Forbes Ave, Pittsburgh, PA 13
15213, USA 14
SINGH, REED, and KELLER: MULTI-OBJECTIVE FRAMEWORK 2
Abstract 15
Managing ecosystems with deeply uncertain threshold responses and multiple decision makers 16
poses nontrivial decision analytical challenges. The problem is imbued with deep uncertainties 17
because decision makers do not know or cannot converge on a single probability density 18
function for each key parameter, a perfect model structure, or a single adequate objective. The 19
existing literature on managing multi-state ecosystems has generally followed a normative 20
decision-making approach based on expected utility maximization (MEU). This approach has 21
simple and intuitive axiomatic foundations, but faces at least two limitations. First, a pre-22
specified utility function is often unable to capture the preferences of diverse decision makers. 23
Second, decision makers’ preferences depart from MEU in the presence of deep uncertainty. 24
Here, we introduce a framework that allows decision makers to pose multiple objectives, 25
explore the trade-offs between potentially conflicting preferences of diverse decision makers, 26
and to identify strategies that are robust to deep uncertainties. The framework — referred to as 27
many objective robust decision making (MORDM) — employs multi-objective evolutionary 28
search to identify trade-offs between strategies, re-evaluates their performance under deep 29
uncertainty, and uses interactive visual analytics to support the selection of robust management 30
strategies. We demonstrate MORDM on a stylized decision problem posed by the management 31
of a shallow lake where surpassing a pollution threshold causes eutrophication. Our results 32
illustrate how framing the lake problem in terms of MEU can fail to represent key trade-offs 33
between phosphorus levels in the lake and expected economic benefits. Moreover, the MEU 34
strategy deteriorates severely in performance for all objectives under deep uncertainties. 35
Alternatively, the MORDM framework enables the discovery of strategies that balance multiple 36
preferences and perform well under deep uncertainty. This decision analytic framework allows 37
the decision makers to select strategies with a better understanding of their expected trade-offs 38
(traditional uncertainty) as well as their robustness (deep uncertainty). 39
Keywords: a posteriori decision making; deep uncertainty; lake eutrophication; many 40
objective; robustness analysis; utility 41
SINGH, REED, and KELLER: MULTI-OBJECTIVE FRAMEWORK 3
INTRODUCTION 42
Managing ecosystems poses challenging decision analysis problems due to (i) the 43
presence of several decision makers, (ii) the potential for highly nonlinear threshold responses, 44
and (iii) the underlying deep uncertainties. Deep uncertainties emerge when planners are unable 45
to agree on or identify the full scope of possible future events including their associated 46
probability distributions (Knight 1921, Lempert et al. 2006). Representing the diverse 47
preferences of decision makers can be challenging if they have conflicting objectives and values. 48
In cases of conflicting preferences, explicit trade-offs between alternative management actions 49
exist. Mapping these trade-offs poses nontrivial conceptual and computational challenges when 50
defining ecosystem management problems (e.g., choosing objectives, management decisions, 51
planning horizons, representations of preferences, etc.) as well as predicting the impacts of 52
actions (e.g., imperfect knowledge of system dynamics, external forcings, or environmental 53
thresholds (Clark 2005, Lempert and Collins 2007, Keller and McInerney 2008, McInerney et al. 54
2012)). Representing the potential for highly nonlinear threshold responses (or tipping points) 55
can be crucial because surpassing such thresholds can lead to regime shifts that significantly 56
degrade the objectives of some or all decision makers (Bennett et al. 2008). Current decision 57
support tools for ecosystem management are limited in their ability to address these interacting 58
challenges (Lempert and Collins 2007). Throughout this study, we refer to the diverse set of 59
stakeholders in a decision-making process as ‘decision makers’. 60
Figure 1 classifies two main challenges that emerge when defining ecosystem 61
management problems: (i) appropriately accounting for uncertainty, and (ii) accounting for 62
diverse decision maker perspectives. Approaches for representing uncertainty are arranged on 63
the x-axis with increasing conceptual and computational complexity: beginning from 64
deterministic (perfect foresight) assumptions, moving towards well-characterized uncertainty 65
captured using a single probability distribution function (PDF), next transitioning to deep 66
uncertainties represented with multiple PDFs, and finally, learning based on new observations to 67
actively reduce uncertainties. The y-axis arranges methods for incorporating decision makers’ 68
preferences in increasing complexity: starting from the evaluation of management actions using a 69
single a priori abstraction of preferences in a utility function (Ramsey 1928, Von Neumann and 70
Morgenstern 1944), moving towards weighting schemes aggregating multiple objectives into a 71
single measure as typified by traditional multi criteria decision analysis (Brink 1994, Ralph 72
2012, Köksalan et al. 2013), and finally, many-objective analysis where decision makers’ 73
preferences are elicited after an explicit mapping of trade-offs between alternatives. 74
Prior decision analytical frameworks have been mapped onto the uncertainty and 75
preference axes based on their underlying approaches when addressing the lake problem as 76
illustrated in Figure 1. The lake problem has a long history and represents a hypothetical lake 77
with a nonlinear uncertain threshold response. The lake problem seeks to proxy a broad class of 78
environmental management problems such as fishery collapse, and abrupt climate change 79
(Carpenter et al. 1999, Brozovic and Schlenker 2011). The problem’s narrative describes a lake 80
near a town. The citizens have to identify a pollution strategy that meets conflicting objectives 81
such as maximizing economic gain while maintaining a healthy lake. Here, we build on the lake 82
problem’s legacy as a test bed for comparing and contrasting decision-making frameworks for 83
environmental management. Through this example, we first show how standard approaches 84
limit the decision makers’ ability to navigate the preference-uncertainty landscape, together 85
SINGH, REED, and KELLER: MULTI-OBJECTIVE FRAMEWORK 4
termed as the twin challenges. We then discuss avenues to overcome these limitations. Finally, 86
we propose a decision analysis framework that enables the decision makers to tackle the twin 87
challenges while contrasting it with the more traditional approaches. 88
Twin challenges in ecosystem management 89
The ecosystem management literature in general and analyses of the lake problem in 90
particular focus on the economic value represented through the maximization of expected utility 91
(MEU) as the sole objective to evaluate alternative strategies (Carpenter et al. 1999, Maler et al. 92
2003, Bond 2010, Brozovic and Schlenker 2011, Horan et al. 2011, Johnson et al. 2013). These 93
standard approaches position themselves in the lower left side of Figure 1. Despite its long 94
legacy of applications, MEU faces limitations when applied to ecosystem management problems. 95
First, MEU evaluates alternative actions with an a priori aggregate utility function. This limits 96
the ability to provide insights into the trade-offs between the often divergent objectives of 97
different decision makers (e.g., maximizing net economic benefits vs. ensuring sustainability) 98
(Common and Perrings 1992, Ludwig 2000, Morse-Jones et al. 2011, Admiraal et al. 2013). 99
Value functions are also insensitive to ecosystem attributes that do not affect them directly (e.g., 100
loss of species) and can suggest strategies resulting in ecosystem collapse for non-linear 101
threshold based systems (Admiraal et al. 2013). 102
Secondly, the standard implementation of the MEU approach assumes that all 103
uncertainties can be represented with a single joint probability density function and decision 104
makers choose alternatives that maximize the expected value of the merit function across this 105
function (Carpenter et al. 1999, Peterson et al. 2003, Bond and Loomis 2009). This is a poor 106
assumption in the presence of deep uncertainties that emerge when unique distributions of 107
system parameters or forcings cannot be specified (Knight 1921, Lempert et al. 2006). Also, it 108
has been shown that decision makers are ambiguity averse – they prefer a profit of known 109
probability instead of a profit of unknown probability despite equal expected payoffs (Ellsberg 110
1961). Hence, decision makers do not always choose alternatives that maximize their expected 111
utility. Aversion to ambiguity exists in many application areas including health, environment, 112
negotiation, and more (Becker and Brownson 1964, Curley and Yates 1985, Hogarth and 113
Kunreuther 1989, Kuhn and Budescu 1996, Budescu et al. 2002). 114
Potential solutions to the twin challenges 115
Moving from the single-objective formulation in the MEU framework to a multi-116
objective framework can provide insights about tensions and tradeoffs (Brill et al. 1990). If two 117
or more objectives are in conflict, optimization for multiple objectives yields multiple non-118
dominated solutions that comprise the Pareto front (or a trade-off curve). Non-domination 119
implies that solution performance in any given objective cannot be improved without a 120
performance loss in at least one other objective. This provides decision makers with several 121
alternatives instead of a single optimal strategy (Pareto 1896, Cohon and Marks 1975). An 122
explicit understanding of the trade-offs across alternatives can significantly change decision 123
makers’ preferences and broaden the scope of values encompassed in the planning process (Brill 124
et al. 1990, Fleming et al. 2005, Kasprzyk et al. 2009, Kasprzyk et al. 2012, Woodruff et al. 125
2013, Zeff et al. 2014). 126
SINGH, REED, and KELLER: MULTI-OBJECTIVE FRAMEWORK 5
Identifying Pareto optimal strategies poses significant computational challenges that have 127
severely limited the number of studies that move beyond the MEU approach to explore a richer 128
set of objectives (McInerney et al. 2012, White et al. 2012). White et al. (2012) present such a 129
trade-off analysis for management of shared marine resources between sectors such as offshore 130
wind energy, commercial fishing, and whale watching. They show that including several 131
objectives in the analysis prevents significant losses (>$1million), generates extra value (>$10 132
billion), provides managers with a means to incorporate sectors that cannot be measured in 133
dollars (conservation), increases transparency in the decision-making process and helps avoid 134
unnecessary conflicts due to perceived but weak trade-offs. McInerney et al. (2012) illustrate an 135
approach to map the trade-off curve defined by a transition of preferences between two values 136
that enables decision makers to choose their desired level of compromise a posteriori. Both 137
studies generate trade-off curves by varying the weights on various objectives to identify one 138
point on the Pareto front with each optimization (Chankong and Haimes 1983). However, this 139
becomes computationally expensive (and arguably infeasible) for three or more objectives due to 140
the exponential growth in the number of sub problems solved when increasing objective counts 141
(Teytaud 2007). 142
Beyond the challenge of quantifying key trade-offs, it is also critical to evaluate the 143
component solution’s robustness to deep uncertainties (Lempert et al. 2003, Dixon et al. 2007, 144
Lempert and Collins 2007). When selecting a preferred trade-off solution, decision makers may 145
prefer a strategy with reduced but consistent multi-objective performance across deeply 146
uncertain states-of-the-world (SOWs) versus a strategy that is optimal in the expected SOW 147
attained by assuming well characterized uncertainty (Lempert and Collins 2007). Robustness 148
analysis often employs minimization of average loss of utility (regret) under different SOWs 149
representing deep uncertainty (Lempert et al. 2003, Dixon et al. 2007, Lempert and Collins 150
2007). The above cited robustness studies all adopt a univariate objective function and 151
consistently show that single objective performance degrades in the presence of deep 152
uncertainty. However, these studies are typically silent on the implicit degradation of other 153
potential objectives (for example, multi-objective regrets). Robustness analysis based on a single 154
objective allows decision makers to progress along the uncertainty axis on Figure 1 but limits 155
movement along the preference axis. 156
Defining robustness in a multi-objective context allows decision makers to identify 157
solutions that perform well across multiple objectives as well as under deep uncertainties, thus 158
allowing them to traverse Figure 1 along both axes. Such analysis may also reveal that the trade-159
offs are extremely sensitive to the baseline modeling assumptions, stressing the need for 160
alternative problem formulations (add/remove objectives, decisions, constraints, or modeled 161
uncertainties). The focus on multiple, competing problem formulation hypotheses links well to 162
decision analytical approaches termed constructive decision aiding, where problem framing is 163
acknowledged as a key focus and challenge in real decision contexts (Tsoukias 2008). This 164
primary focus on iterative problem formulation where decision makers repeat the process of 165
identifying the objectives and decisions until a satisfactory set of strategies are discovered is 166
often neglected in the MEU approach. Here, we implement a constructive decision aiding 167
approach for the lake problem and compare it with the MEU approach. 168
Many objective robust decision making: a framework for constructive decision aiding 169
SINGH, REED, and KELLER: MULTI-OBJECTIVE FRAMEWORK 6
Many objective robust decision making (MORDM) has emerged as a framework for 170
implementing the constructive decision aiding approach in a variety of real-world problems 171
(Kasprzyk et al. 2013). It combines the robust decision making (RDM) framework (Lempert and 172
Collins 2007) with the many objective (MO) search capabilities of evolutionary algorithms (Hall 173
et al. 2012, Reed et al. 2013). The MO search enables decision makers to discover management 174
problems’ trade-offs where each component point is a potential strategy alternative. The trade-175
offs obtained under well-characterized uncertainties provide a baseline view for the best 176
available non-dominated strategies under the assumption that the model as well as associated 177
probability assumptions adequately represent performance in the future conditions. RDM 178
complements the MO search by testing strategies against alternative futures attained through 179
sampling deeply uncertain factors to identify robust solutions. This facilitates a posteriori 180
decision making, meaning that decision makers’ choices occur after they have an explicit 181
understanding of a system’s expected trade-offs and robustness. MORDM exploits recent 182
technological advances in high dimensional visual analytics (Thomas et al. 2005, Kollat and 183
Reed 2007, Keim 2010) as well as multi-objective optimization. The framework employs the 184
BORG multi-objective evolutionary algorithm (MOEA) that can provide high quality 185
approximations of the trade-offs for problems with non-linear, threshold based dynamics across a 186
range of objectives and representations of uncertainty (Coello Coello et al. 2005, Fleming et al. 187
2005, Hadka and Reed 2012, Reed et al. 2013, Woodruff et al. 2013). 188
In this study, we use MORDM to address four main questions for the lake problem: (i) 189
does the incorporation of more planning objectives help to avoid collapse of the lake? (ii) what 190
are the main conflicts in decision makers’ perspectives? (iii) how do the strategies attained 191
under well-characterized uncertainty perform when re-evaluated for their robustness? and (iv) 192
what are the implications of alternative decision makers’ preferences and definitions of 193
robustness for managing the lake? 194
THE CONSTRUCTIVE DECISION ANALYTIC FRAMEWORK 195
There is a growing recognition that decision aiding frameworks need to move towards a 196
more flexible process of problem formulation wherein the decision makers can add/remove 197
objectives as needed, explore the consequences of each formulation, and identify novel 198
alternatives that were not visible before (Brill et al. 1990, Tsoukias 2008, Kasprzyk et al. 2013, 199
Woodruff et al. 2013). Here, we demonstrate many objective robust decision making (MORDM) 200
as a potential framework to facilitate constructive decision aiding in the lake problem. MORDM 201
combines the strengths of many objective evolutionary search and robust decision making 202
(Lempert and Collins 2007, Kasprzyk et al. 2013, Reed et al. 2013). MOEAs help the decision 203
makers to search the space of feasible strategies and discover alternatives that compose optimal 204
trade-offs while robust decision making helps to test the performance of strategies under 205
changing assumptions of uncertainty. 206
If uncertainties are well characterized, a posteriori trade-off analysis can be used to elicit 207
which management action decision makers prefer based on its expected performance under 208
baseline uncertainty. However, when uncertainties are deep, decision makers might change their 209
preferred management strategies if it is necessary to trade-off high performance under the 210
baseline well-characterized uncertainty with satisficing multi-objective performance across 211
broader ensemble of SOWs representing deep uncertainties (Simon 1959). We refer to Pareto 212
SINGH, REED, and KELLER: MULTI-OBJECTIVE FRAMEWORK 7
satisficing strategies that perform acceptably across their component objectives as well as under 213
deep uncertainties as ‘robust’. There is a third possibility that none of the strategies identified 214
using the particular problem framing perform well under deep uncertainty, which may indicate 215
the need to fundamentally alter the problem formulation, the evaluative simulation, or both. 216
In this way, MORDM facilitates well-informed a posteriori trade-off decisions under 217
well-characterized uncertainties, minimization of multi-objective performance losses under deep 218
uncertainties, and problem falsification when no robust strategies can be identified. Other 219
robustness approaches such as info-gap (Ben-Haim 2001) and decision scaling (Brown et al. 220
2012) represent local sensitivity analyses around a small set of user specified actions. They lack 221
global multi-objective exploration of alternatives and have limited ability to quantify deeply 222
uncertain trade-offs. 223
MORDM has been found to be successful in several other decision contexts. For 224
example, Kasprzyk et al. (2013) found that including robustness in a multi-objective context as a 225
decision criterion dramatically altered the choice of an urban water supply planning problem’s 226
formulation as well as its component trade-off alternatives. Since then, MORDM has also been 227
applied in designing engineered systems. Woodruff et al. (2013) discovered two families of 228
aircraft designs using a ten-objective problem formulation that were not evident by using prior 229
highly aggregated goal programming formulations. In a real multi-stakeholder application, 230
Herman et al. (2014) identified key trade-offs and demand-based failure modes for four cities 231
seeking to coordinate the mitigation of their water supply risks to drought. The recent growth in 232
availability of open source software and comprehensive testing of MORDM in a range of 233
engineering applications sets the stage for its use in a wider array of environmental problems 234
(Haasnoot et al. 2013, Haasnoot et al. 2014, Kwakkel et al. 2014). For example, it could be used 235
for analyses highly challenging environmental management problems, such as geoengineering, 236
climate change adaptation, etc. 237
238
As noted by Herman et al. (2015) in their taxonomy of robustness techniques, the 239
MORDM framework’s flexibility and emphasis on exploring competing hypotheses with respect 240
to uncertainties and preferences, serves to make it possible to incorporate rapidly growing 241
methodological innovations in this area. As an example, frameworks employing dynamic 242
programming (see for example Webster et al. (2012)) provide their own suite of problem 243
formulations that could be explored on a comparative basis with multi-objective explorations of 244
the same problems. There is emerging work in multi-objective control under uncertainty that has 245
significant potential to aid users confronting the curse-of-dimensionality as the number of states 246
under consideration while also making it possible integrate a broader array of information 247
sources (Castelletti et al. 2008, Giuliani et al. 2014). Overall MORDM offers two core benefits 248
beyond unifying existing methods: 249
1. MORDM can employ a large number of objective functions (at this time up to ten with 250
relative ease) without the need to aggregate them. MORDM can hence visualize the 251
multi-dimensional trade-offs to provide a broader decision making context relative to a 252
single optimal strategy. 253
2. MORDM supports the exploration of broader definitions of robustness that account for 254
multi-objective performance requirements. 255
Summary of framework 256
SINGH, REED, and KELLER: MULTI-OBJECTIVE FRAMEWORK 8
MORDM is typically implemented in an iterative approach (Figure 3). The first step is to 257
elicit decision makers’ objectives, potential management decisions, constraints, uncertainties, 258
and system planning models through surveys or interviews. The second step is to generate 259
alternative problem formulation hypotheses by experimenting with different objectives or their 260
combinations, the decisions, and system uncertainty characterizations. Once alternative problem 261
formulation hypotheses are identified and baseline well-characterized uncertainties are specified, 262
the potential trade-offs between different objectives can be assessed in Step 3. At this step, the 263
results are presented to the decision makers. On viewing the trade-offs between objectives and 264
understanding the compromise required in one objective to maintain a satisfactory performance 265
in another, decision makers can choose one or more alternatives. If none of the alternatives are 266
acceptable, decision makers may choose to alter the problem formulations as represented by the 267
gray arrows between Step 3 and Step 2. 268
Following this, the robustness component of the framework requires an elicitation step 269
where the decision makers identify the minimum levels of system performance that they are 270
willing to accept. These measures can be related to modeled objectives or constraints as well as 271
other measures that were not included in the optimization process, e.g., new constraints or 272
measures that emerge after viewing the trade-offs. Since performance requirements are elicited 273
after the trade-off analysis, decision makers have an understanding of the consequences of their 274
preferred management strategies (i.e., a posteriori decision making). Building on this, 275
robustness of the solutions with respect to decision makers’ performance requirements can be 276
assessed in Step 5. Carefully elicited measures of robustness provide decision makers with an 277
understanding of fragility of management actions under well-characterized uncertainties. At this 278
stage, if very few or none of the solutions are found to be robust, the decision makers can be 279
asked to either alter their performance thresholds (e.g., reduce their risk aversion and go back to 280
Step 4), or the problem formulations by adding/removing objectives/decision variables (go back 281
to Step 2). 282
Once robust solutions are identified under well-characterized uncertainty, they are tested 283
under deep uncertainty scenarios, and their robustness is re-assessed in Step 6. At this stage, 284
decision makers can again go back to Step 2 or Step 4 in case they are not satisfied with the 285
solution’s performance and robustness under deep uncertainty. At the end of this process, the 286
decision makers should have identified a problem formulation and associated solution strategies 287
that satisfy various preferences, and are robust under both well-characterized and deep 288
uncertainty. Finally, the strategy is implemented and the ecosystem is monitored for the impact 289
of the selected strategy. In case of unexpected changes in decision makers’ objectives, the entire 290
process can be repeated (i.e., the constructive decision aiding as discussed by Tsoukias (2008)). 291
It is important to stress here that visual analytics are a key component of the MORDM 292
process without which the goal of enabling decision makers to explore diverse alternatives and 293
consequences of their different choices would not be feasible (Stump et al. 2003, Kollat and 294
Reed 2007, Castelletti et al. 2010, Reed et al. 2013, Woodruff et al. 2013). In the beginning of 295
the decision making process, decision makers may not have a complete understanding of the 296
complex interactions between their objectives. They may also not be aware of the ranges of 297
performance that are possible across different measures. Selecting candidate strategies after 298
visualizing the trade-off curve avoids the biases associated with a priori weighting methods, 299
which can hide key insights. 300
SINGH, REED, and KELLER: MULTI-OBJECTIVE FRAMEWORK 9
APPLICATION OF MORDM TO THE LAKE PROBLEM 301
Our results and analysis follow the MORDM steps illustrated in Figure 3. We begin by 302
identifying three problem formulation hypotheses (P1, P2, and P3) as described in the sub-303
section on ‘Objectives and problem formulations’. Optimal strategies for these multiple problem 304
hypotheses are identified using the optimization framework described in supplementary text 305
‘Evolutionary multi-objective optimization’ (Appendix A1). Following this, we discuss the 306
methods of estimating robustness in the sub-section on ‘Assessing robustness’. We present the 307
baseline expected trade-offs in the result sub-section on ‘Trade-off analysis under well-308
characterized uncertainty’. Next, the result sub-section on ‘Navigating trade-offs to explore 309
diverse alternatives’ illustrates how decision makers can visually navigate the trade-off space and 310
select solutions that represent competing preferences. We then use the simulated decision 311
makers’ performance requirements to assess the robustness of solutions under well-characterized 312
uncertainty, and then under deep uncertainty in the result sub-section on ‘Robustness analysis’. 313
The lake model 314
The lake problem is a hypothetical decision problem faced by a town located in the 315
vicinity of a lake (Figure 2). The model was developed by Carpenter et al. (1999) by analyzing 316
the behavior of lakes. The inhabitants of the town have to decide on the amount of allowable 317
pollution that can be emitted into the town’s lake for a given planning horizon (i.e., they have to 318
formulate a pollution control strategy). At each time step, the town releases a certain amount of 319
anthropogenic pollution to the lake in the form of phosphorus. There is also some amount of 320
uncontrolled natural inflow to the lake. The lake is able to remove a part of this pollution 321
(phosphorus) based on its properties represented by parameters b and q. The phosphorus 322
dynamics in the lake is approximated by: 323
1 ln( , )1
q
tt t t tq
t
XX X a bX
X
, (1) 324
,where, Xt represents the (dimensionless) phosphorus in the lake at time t (in years), at 325
(dimensionless) represents the allowed anthropogenic pollution in the lake at time t, b and q are 326
parameters of the lake model. Together, they represent the lake’s ability to remove phosphorus 327
(through sedimentation, outflow, and sequestration in biomass of consumers or benthic plants) 328
and recycle phosphorus (primarily through sediments). The last term represents natural inflows 329
into the lake that are outside the decision maker’s control. They are represented in this study by 330
lognormal distributions with specified mean and standard deviation. Depending upon the 331
particular formulation of the lake problem being analyzed, we exclude the uncertainties in 332
natural inflows term (deterministic case), specify this term with fixed choices for μ and σ 333
(stochastic case that assumes well-characterized uncertainty), and test robustness of pollution 334
control policies for varying values of μ and σ (the deep uncertainty case). 335
The lake can exist in two states: oligotrophic or eutrophic. In the oligotrophic state, the 336
lake has low concentrations of phosphorus with clear water. In the eutrophic states, the 337
phosphorus concentration is high and algae can bloom. In the eutrophic state, the lake is 338
assumed to be unable to support fisheries, or tourism, and also to be severely degraded in 339
aesthetic value. It is also much harder to revert the lake from the eutrophic to the oligotrophic 340
SINGH, REED, and KELLER: MULTI-OBJECTIVE FRAMEWORK 10
state in a short time by reducing pollution alone. Therefore, the transition to eutrophic state has 341
multiple disadvantages, besides loss of economic activity. 342
Depending upon the ease with which a lake in eutrophic state can be brought back to its 343
oligotrophic state, lakes can be classified as reversible, hysteretic, or irreversible. Irreversible 344
lakes are most vulnerable since it is impossible to bring them back to an oligotrophic stage by 345
reducing phosphorus concentrations alone once they exceed a threshold. In this study, the 346
parameters of the lake model are such that the lake is irreversible and therefore represents an 347
ecosystem with two possible states. Once the lake turns eutrophic, it is not possible to return it to 348
an oligotrophic state by reducing phosphorus inputs alone. In reality, these conditions are most 349
likely to occur in shallow lakes, lakes in phosphorus rich regions, or lakes that have received 350
extreme phosphorus inputs for an extended period of time. 351
In the simple model, the parameter b determines whether the lake is reversible, hysteretic 352
or irreversible for a given value of the recycling parameter q. Higher values of b suggest a lake 353
that has a high capacity to remove pollution and vice-versa. If recycling occurs, the 354
concentration of phosphorus in the lake increases suddenly over a period of time, the rate of this 355
change is governed by the recycling parameter q. Higher values of q correspond to fast 356
transitions and vice-versa. We adopt this formulation from the pioneering study by Carpenter et 357
al. (1999). Carpenter et al. (1999) also provides a very careful and much more detailed 358
description for the lake (model) system. Table 1 lists the parameter values for the lake model 359
used in our study. 360
The lake problem formulation of Carpenter et al. (1999) is simple, yet conceptually rich. 361
It has the ability to represent tipping points, nonlinearity, and deep uncertainties (Carpenter et al. 362
1999, Lempert and Collins 2007). Carpenter et al. (1999) demonstrate the impact of uncertainty, 363
lags, etc. on the optimal strategy. In a groundbreaking study, Peterson et al. (2003) show that 364
decision makers maximizing their expected utility can cause periodic collapses of the lake 365
ecosystem if there is uncertainty about the lake’s eutrophication thresholds. Lempert and Collins 366
(2007) analyze the decision problem to identify robust solutions that compromise optimality for 367
acceptable performance under a broader envelope of uncertainty assumptions. 368
Uncertainty 369
Uncertainty may arise in the lake model due to several sources – parameters governing 370
the lake dynamics and economics, stochastic random inflows, mapping decision makers’ 371
preference to the objective function, etc. For this study, we focus on the uncertainty in the 372
inflow of natural pollution into the lake. We analyze the impact of two types of uncertainties on 373
the selected strategy – well-characterized uncertainty and deep uncertainty. Uncertainty in 374
random inflows to the lake is modeled stochastically using lognormal distributions (Equation 1). 375
When the parameters of the lognormal distribution are pre-specified, it represents the case of 376
well-characterized uncertainty. On the other hand, if the decision makers are uncertain about the 377
characterization of the uncertainty (in this case the parameters of the lognormal distribution) they 378
face a simple case of deep uncertainty. 379
Deep uncertainty is incorporated into the analysis to demonstrate the impact of a classic 380
risk-based analysis vs. a multiple scenarios based robustness analysis. The use of multiple 381
SINGH, REED, and KELLER: MULTI-OBJECTIVE FRAMEWORK 11
probability distribution functions (PDFs) is similar to the approaches used by previous studies 382
for characterizing deep uncertainty (Epstein and Wang 1994, Krishnamurti et al. 1999, Lempert 383
and Collins 2007). We chose nine distributions for representing deep uncertainty using a 384
standard sampling technique: by moving in increments above and below the baseline uncertainty 385
distribution. The mean of the baseline uncertainty distribution was chosen such that it is half of 386
the mean pollution inflow under the strategy obtained from single objective optimization. The 387
variance of the baseline distribution was chosen by analysis of the random samples generated by 388
the chosen mean/variance combination. The variance estimates that produced estimates of 389
pollution between the next higher/lower mean were selected. The lognormal distributions that 390
represent well-characterized uncertainty (or the baseline risk scenario) and deep uncertainty are 391
illustrated in Figure 4. Corresponding parameters are listed in Table 2. Each PDF is used to 392
generate 10000 random SOWs resulting in 90000 total SOWs used in this study. 393
Objectives and problem formulations 394
A sole focus on economic valuation when identifying environmental management 395
strategies can lead to a restrictive problem framing that can limit the types of alternatives that 396
decision makers’ explore (Brill et al. 1990). This concern has been termed cognitive myopia in 397
the literature exploring biases in decision making (Hogarth and Kunreuther 1989, Brill et al. 398
1990). Recent examples of how cognitive myopia can negatively influence decisions have been 399
published for water management as well as the design of complex engineered systems (Kasprzyk 400
et al. 2012, Woodruff et al. 2013). MORDM provides a broader trade-off context as well as the 401
potential to discover a more diverse suite of management policies to overcome cognitive myopia 402
as suggested by Brill et al. (1990). Hence, we illustrate the MORDM framework in the study 403
using a diverse suite of objectives that represent different decision makers’ perspectives in the 404
imaginary town and assess the potential for myopia in the classical expected utility maximization 405
formulation of the problem. 406
We begin with a widely used objective in the analysis of the lake problem - the 407
discounted net present value of utility (O1) given by, 408
1
1
1 N
i
i
O VN
, where, (2)409
,
0
Tt
i t i
t
V u
, and, (3)410
2
, , ,t i t i t iu a X . (4) 411
In these equations, Vi (to be maximized) is the discounted net present value of utility for 412
the ith SOW, ut,i is the utility, at,i is the allowed anthropogenic pollution and Xt,i is the level of 413
phosphorus in the lake at time step t and ith SOW. The economic parameters α and β capture the 414
willingness to pay for pollution and the compensation lake users are willing to accept to tolerate 415
a given state of the lake respectively. α and β are fixed at 0.4 and 0.08 respectively following the 416
analysis in Carpenter et al. (1999). For simplicity, we neglect the considerable uncertainty about 417
the values for the discount rate and the economic parameters (Chichilnisky 1996, Dasgupta 418
2008). The discount factor, δ translates future to present utilities. 419
SINGH, REED, and KELLER: MULTI-OBJECTIVE FRAMEWORK 12
The time index, t, varies from 1 to T years (T = 100 years), and there are N SOWs. The 420
SOWs are sampled from the lognormal distribution in Equation (1) and their total number (N) 421
varies from 0 to 90000 based on the type of uncertainty being considered as described in the 422
section on ‘Uncertainty’. N is 0 for the deterministic case, 10000 for well-characterized 423
uncertainty and 90000 for deep uncertainty. The allowed anthropogenic pollution flow a is only 424
decision variable that controls the objective function. The decision maker can change a only 425
every 5 years. As a result, there are 20 planning periods across a planning horizon of 100 years 426
and the optimization framework needs to identify the 20 values of a that satisfy selected decision 427
makers’ preferences such as maximizing a chosen objective function. 428
To contrast the strategy that maximizes the discounted net present value of utility (O1) 429
given by equation (2) with strategies based on other value preferences, we introduce additional 430
objectives that represent decision makers more strongly focus on the long term environmental 431
quality of the lake or that vary in their inter-temporal preferences. Decision makers often assess 432
outcomes using a diverse set of values and objectives (Kasprzyk et al. 2009, McInerney et al. 433
2012, White et al. 2012, Herman et al. 2014). Farber et al. (2006) for example, argue that the 434
linking of ecology and economics requires identification of ecosystem services that are likely to 435
be in conflict. Our objective formulation is to a large part motivated by this assessment. 436
Identifying key objectives that represent diverse stakeholders is challenging and a 437
potentially iterative process. Some of these objectives are a proxy for ecosystem services 438
(recreation, fishery), while others serve as proxies for alternative perspectives with regard to 439
valuing economic services (utility). This approach can be interpreted as representing the 440
perspectives of five hypothetical stakeholder groups in the fictitious town. The values and 441
preferences of each assumed stakeholder group were then mapped to an objective. This resulted 442
in the following objectives considered in our analysis: 443
1. Minimize the level of phosphorus in the lake (O2) – Admiraal et al. (2013) point out that 444
the utility function is strongly biased towards anthropogenic services which is a key 445
limitation in identifying ecosystem management strategies that adequately protect 446
environmental values. Here, we introduce this objective to represent a regulatory 447
perspective related to an indicator of the health of the lake. This objective can be 448
interpreted as one key concern of individuals that aim to preserve the lake as it is and 449
therefore their sole preference is to reduce the levels of phosphorus in the lake. The 450
objective function is, 451
2 ,
1 1
1 N T
t i
i t
O XNxT
, (5) 452
,where, Xt,i is the phosphorus in the lake at time step, t and ith SOW. This objective aims 453
to minimize the average levels of phosphorus in the lake. 454
2. Maximize the utility of the present decision makers (O3) – This objective represents 455
utility of the current decision makers. The objective function is 456
3 1,
1
1 N
i
i
O UN
, (6) 457
SINGH, REED, and KELLER: MULTI-OBJECTIVE FRAMEWORK 13
,where, U1,i is the utility of the first year in the 100 year planning horizon for the ith SOW 458
and is to be maximized. 459
3. Maximize the utility of the future decision makers (O4) – The objective was motivated by 460
the definition of sustainability adopted by past studies (Holling 1973, United Nations 461
1987, Cato 2009). These definitions represent the interest of present and future 462
generations quite differently than the discounted expected utility framework. While 463
discounting has been the classic approach to analyze inter-temporal trade-offs, several 464
reports, even governmental decisions have been based on objectives that are not subject 465
to discounting. One simple example is the design of flood defenses in the Netherlands 466
that are subject to an acceptable level of risk (Jonkman 2013). Therefore, we explicitly 467
model inter-temporal preferences in separate objective functions. To approximate this 468
perspective, we choose two example stakeholder groups (i) current generation and (ii) 469
generations in the far future. (Far here is represented as the second half of the planning 470
horizon of the problem). This objective represents the utility of future decision makers 471
who exist in the last 50 years of the 100-year planning horizon. The objective function is 472
4 50 100,
1
1 N
i
i
O UN
, (7) 473
,where, U50-100,i is the sum of undiscounted utilities for the generations spanning years 50 474
to 100 in the 100-year planning horizon for the ith SOW. This objective function is to be 475
maximized. 476
4. Maximize reliability (O5) – One of the goals of this study is to capture the behavior of 477
multi-state ecosystems when some states are far less preferable to the decision maker. 478
The reliability objective seeks to ensure that the lake remains below critical pollution 479
levels to avoid eutrophication. This objective also represents key concerns of 480
stakeholders who either depend directly on the ecosystem services provided by the lake, 481
or those who aim to maintain the ecosystem itself while being able to accept some levels 482
of pollution. In addition, this formulation approximates a common risk-based 483
engineering metric that has been widely employed across many contexts (Hashimoto et 484
al. 1982). Maximizing the reliability of avoiding a tipping point response captures the 485
strong aversion to irreversible losses of key economic and ecosystem services. The 486
objective is, 487
5 ,
1 1
,
,
,
1
1 ,
0
crit
crit
N T
t i
i t
t i
t i
t i
ONxT
if X Xwhere
if X X
. (8) 488
In these equations, θt,i is the reliability index which is 1 if the level of phosphorus 489
in the lake (Xt,i) is below the specified critical threshold (Xcrit) and 0 otherwise. The 490
critical threshold is set at 0.5 based on the parameters of the lake model. Xcrit is the 491
minimum steady state pollution value at which the lake transitions from an oligotrophic 492
to eutrophic state. A reliability of 1 represents a pollution strategy that successfully 493
keeps the phosphorus levels in the lake below the specified critical thresholds across the 494
SINGH, REED, and KELLER: MULTI-OBJECTIVE FRAMEWORK 14
entire planning horizon and across all SOWs. Table 3 lists the objectives used in this 495
study. 496
We combine these five objectives in three different problem formulations to represent 497
three potential problem frameworks to identify a suitable pollution strategy for the lake – 498
1. Deterministic single objective (P1) – This formulation maximizes the net present value of 499
utility (O1) ignoring all uncertainties and identifies a single optimal management strategy 500
with the assumption of perfect foresight. 501
2. Stochastic single objective (P2) – This formulation maximizes the net present value of 502
expected utility (O1) assuming well-characterized uncertainties in natural inflows and 503
identifies a single optimal management strategy. The formulations P1 and P2 are 504
summarized as, 505
1
1 2 20
( ) ,
( , ,..., )
1 1
10000 2
NF x O
x
x a a a
for PN
for P
.
(9) 506
In these equations, F(x) is the objective function vector to be optimized, x is the 507
decision vector, and N is the number of SOWs over which O1 is optimized. 508
3. Stochastic five objectives (P3) – This formulation identifies a Pareto approximate front 509
that is obtained after considering all the objectives (O1, O2, O3, O4, and O5) 510
simultaneously. The Pareto approximate front consists of all strategies that are non-511
dominated (i.e., it is not possible to maximize one objective without degrading the value 512
of another across these strategies). The Pareto approximate front consists of a range of 513
solutions that are presented to the decision makers for negotiations and deliberations. 514
The formulation is, 515
1 2 3 4 5
1 2 20
5
( ) ( , , , , ) ,
( , ,..., )
10000
, : 0.9
N
rel
F x O O O O O
x
x a a a
N
subject to c O
. (10) 516
Here, crel is the constraint on the reliability metric that allows only those solution 517
to be feasible that perform above 0.9 in the reliability objective. Solving problem 518
formulations P1, P2, and P3 demonstrates constructive decision aiding approach where 519
multiple problem framing hypotheses are being comprehensively evaluated under well-520
characterized uncertainty. This initial stage of the MORDM framework allows for a 521
baseline elicitation of decision makers’ preferred management actions assuming that all 522
SINGH, REED, and KELLER: MULTI-OBJECTIVE FRAMEWORK 15
model projections are of high fidelity. The second stage of the MORDM framework 523
evaluates the ‘robustness’ of the management strategies obtained after solving P1, P2, 524
and P3 under deep uncertainty. 525
Assessing robustness 526
The resulting pollution control strategies that emerge from optimizing formulations P1, 527
P2, and P3 present decision makers with the critical task of negotiating their preferences to 528
select their preferred problem as well as candidate robust management strategies. Visual 529
exploration of problems’ relative trade-offs can facilitate a negotiation process by demonstrating 530
explicitly how alternative management actions balance their performance across objectives (O1-5) 531
under well-characterized and deep uncertainties. This a posteriori evaluation of alternatives to 532
identify the strategy for implementation constitutes Step 5 and Step 6 of robustness estimation in 533
Figure 3. Defining robustness necessitates the elicitation of appropriate performance 534
requirements by decision makers. Once these requirements are obtained, the performance of the 535
solution across objectives based on the elicited requirements and under uncertainty can be 536
assessed through a quantitative measure of robustness. 537
Several robustness metrics have been proposed (Starr 1963, Schneller and Sphicas 1983, 538
Lempert 2003, Dixon et al. 2007, Lempert and Collins 2007, Kasprzyk et al. 2013) to evaluate if 539
a management action remain viable across the alternative SOWs sampled for deeply uncertain 540
factors. Lempert (2003) defines robustness as ‘satisficing over a wide range of futures’. They 541
propose metrics such as regret defined as the difference between performance of a selected 542
strategy in an uncertain scenario and the optimal strategy for that scenario. Strategies that have 543
small average regrets are then categorized as robust. Lempert and Collins (2007) further 544
compare various alternatives to assess robustness – trading a small amount of optimal 545
performance for less sensitivity to violated assumptions, reasonably well performance across a 546
wide range of plausible futures and leaving options open. Leaving options open seeks to keep 547
the system in a state that allows future decision makers to make similar choices as the current 548
decision makers. Herman et al. (2015) provide a comprehensive assessment of robustness 549
measures. While most studies agree that robustness is related to the performance of a strategy 550
across uncertain states of the world, to our knowledge only a few studies (Kasprzyk et al. 2013, 551
Herman et al. 2014) cast robustness as multi-criterion regrets while others focusing on a single 552
objective measure. 553
In our analysis, the presence of multiple objectives necessitates that robustness is linked 554
with the performance across all objectives as well as multiple SOWs. This forms the basis of our 555
robustness metric introduced in Equation 11 that estimates robustness as the percentage of total 556
SOWs in which a strategy satisfies performance requirements for all objectives. The estimation 557
of robustness index requires specification two criteria: 558
1. Performance thresholds: Stakeholders identify the minimum acceptable performance for 559
variables of interest, a deterioration of performance below this threshold is considered 560
unacceptable. These variables can be the same as or different from objectives based on decision 561
makers’ requirements. 562
2. Uncertainty represented through a set of states-of-the-world (SOWs): We consider two 563
types of uncertainty representations: well-characterized and deep uncertainty. Both 564
representations of uncertainty are quantified through a set of scenarios or SOWs sampled 565
SINGH, REED, and KELLER: MULTI-OBJECTIVE FRAMEWORK 16
from lognormal distributions of fixed (variable) parameters for well-characterized (deep) 566
uncertainty. 567
A strategy that satisfies performance thresholds for all variables in a given SOW scores 1 568
for that SOW, otherwise 0. Strategies that achieve acceptable performance across a range of 569
variables instead of being acceptable or high achieving in one variable (while being sub-optimal 570
in others) are termed as satisficing. Robustness is then defined as the fraction of the SOWs in 571
which a strategy achieves satisficing performance. Based on this framing, strategies that perform 572
reasonably well across all variables will achieve higher robustness than those that achieve 573
optimal performance in one or more (but not all) variables or those that deteriorate heavily in any 574
variable under changing scenarios. 575
We define performance requirements for key variables and a strategy that equal or 576
exceeds these requirements across a range of uncertain scenarios is considered to be robust. An 577
overall measure of robustness is thus defined as – 578
1
,
1% 100,
1 ,
0
N
i
i
o i O
i
Robust r xN
when P requirement owhere r
otherwise
. (11) 579
Here, ri is 1 if the strategy performs above all requirements otherwise 0, Po,i is the value 580
of the oth variable of interest under the ith SOW, requiremento is the performance requirement for 581
the oth variable, and N is the total number of SOWs (10000 for well-characterized uncertainty and 582
90000 for deep uncertainty). 583
The performance requirements represent criteria that actual decision makers may 584
consider as performance levels that could not be compromised further. For example, decision 585
makers are likely to have a factor of safety associated with the critical phosphorus levels. Here, 586
we fix that factor of safety at 0.75. Similarly, a high reliability of keeping the lake in the 587
oligotrophic state is binding due to obvious economic and environmental consequences. Thus, 588
the performance requirement on reliability was fixed at 99%. While maintaining the lake in an 589
oligotrophic state is important, a minimum level of economic activity is also required. This level 590
was set at 50% of the value obtained in the optimal strategy for expected utility maximization 591
(P2). Our proposed definition of robustness is illustrative and the MORDM framework is highly 592
flexible in accommodating alternative definitions. The primary intent of our example is to 593
emphasize that system performance requirements are themselves likely to be multi-objective, 594
complex in their effects on filtering solutions, and should be carefully elicited in any real 595
application of MORDM. 596
Note that this definition of robustness spanning multiple preferences prevents a decision 597
maker heavily biased towards one objective (say utility) from selecting strategies that favor their 598
preferences. For the robustness index of a strategy to be high, all performance criteria need to be 599
simultaneously satisfied across multiple SOWs. So, if a high performance requirement is 600
selected for the utility function, strategies satisfying it may not satisfy the reliability or 601
phosphorus requirements. Thus, it is likely that none of the strategies emerges as robust forcing 602
decision makers to revise their threshold specifications. 603
SINGH, REED, and KELLER: MULTI-OBJECTIVE FRAMEWORK 17
RESULTS 604
Trade-offs under well-characterized uncertainty 605
The multi-objective framework enabled us to explore the conflicts between objectives for 606
problem formulations with multiple objectives (P3) as well as map the location of strategies 607
discovered by using multiple problem formulations (P1, P2, and P3). Jointly exploring the 608
consequences, trade-offs, and insights of alternative problem formulations is an important step in 609
MORDM analysis. In Figure 3, this corresponds to Step 3. We explore three alternative 610
hypotheses on how to formulate the lake problem using P1, P2, and P3. P1 assumes a 611
deterministic lake model with no natural inflows whereas P2 and P3 assume well-characterized 612
stochastic uncertainties in natural inflows. P1 and P2 represent a single objective utility-based 613
analysis while P3 represents multiple decision makers in a five-objective formulation. We obtain 614
a single optimal pollution control strategies for P1 and P2, while 399 Pareto approximate 615
strategies are obtained for P3 (399 Pareto approximate strategies). Since the optimization 616
framework in Figure A1 optimizes for only 100 SOWs for each evaluation, we re-evaluate the 617
obtained strategies against all 10000 SOWs representing well-characterized uncertainty to obtain 618
the average performance. This re-assessment is carried out in order to assess the consequences 619
of the deterministic assumption used in P1 as well as to verify that the optimization framework 620
was able to identify strategies that perform well across all SOWs for P2 and P3. The stochastic 621
alternatives (P2 and P3) based on the lognormal distribution of natural inflows in Figure 4 had 622
consistent performance under large well-characterized ensemble validating the sampling strategy 623
used during the BORG MOEAs search. This has been observed in several prior studies where 624
the expected objective values attained during the optimization with small Monte Carlo samples 625
matched closely with those attained with the full uncertainty ensemble (Miller and Goldberg 626
1996, Smalley et al. 2000, Kasprzyk et al. 2009). 627
The decision myopia inherent in the single objective problem formulations as well as the 628
consequences of not including uncertainty in the decision analysis is clearly shown in Figure 5. 629
We assess 401 strategies under well-characterized uncertainties and across all objectives in 630
Figure 5. The re-evaluated deterministic strategy from P1 is plotted in the right panel and the 631
stochastic strategies from P2 and P3 are plotted in the left panel. In each of the two panels for 632
Figure 5, two cubes are plotted. Each cube in Figure 5 represents a 3 dimensional space with 633
each dimension mapping performance for an objective function. Color represents the fourth 634
dimension, or performance for the fourth objective function. The primary axes of the cubes 635
represent the objectives of expected utility (x-axis), utility of future decision makers (y-axis) and 636
phosphorus in the lake (z-axis). The color of the solutions represents the utility of the current 637
decision makers. The red star represents the ideal point that shows the best possible values that 638
can be attained across all the objectives simultaneously. Although the ideal solution is not 639
actually feasible for the problem formulations considered here because of conflicting objectives, 640
it provides a visual reference to assess potential compromises. In the left panel, the arrows along 641
the axes designate the directions of increasing preference for the plotted objectives. The right 642
hand side cube represents the same objectives; therefore, we have dropped all descriptions and 643
arrows. Only the red star has been retained to highlight the ideal solution. The yellow solution 644
is the deterministic single objective solution that has been reevaluated under the baseline 645
uncertainty assumption. In the right hand side cube, the larger box spanning the red star and the 646
yellow solution represents the range of objective function values spanning P1, P2, and P3. The 647
SINGH, REED, and KELLER: MULTI-OBJECTIVE FRAMEWORK 18
smaller box spans the range of objective function values spanning P2 and P3 only. For clarity, 648
the left hand side cube is a zoomed view of the performance trade-offs for P2 and P3. 649
Comparison between the two boxes indicates that the entire Pareto approximate front along with 650
the green solution falls within a much smaller region when viewed with respect to the 651
performance of the deterministic solution. 652
The strategy based on P1 (deterministic utility) deteriorates drastically in performance 653
across all objectives when re-evaluated under the broader scope of the uncertainty and additional 654
objectives. This is of potential concern since it is quite common to assume a deterministic 655
problem formulation in environmental planning studies (Maler et al. 2003, Chen et al. 2012). 656
Both P2 and P3 perform relatively well across objectives as they are optimized over the baseline 657
uncertainty. Since high values of reliability (>0.98) were obtained for all the solutions under 658
well-characterized uncertainty for P2 and P3, reliability is not shown in Figure 5. The 659
deterministic P1 formulation yielded an extremely poor performance of 11% for the reliability 660
objective (O5). Alternatively, the MEU solution from P2 is able to maintain high reliability 661
(98%) under all the 10,000 SOWs representing well-characterized uncertainty. P3 strategies 662
have the highest performance in the reliability objective, maintaining a reliability of 99%-100% 663
across all solutions. 664
The trade-offs between different decision makers’ preferences are well evident in the P3 665
strategies plotted under the stochastic case in Figure 5. A total of 399 solutions form the Pareto 666
approximate set for P3. Recall that each strategy in this set represents a non-dominated solution 667
(i.e., it is not possible to improve one objective without degrading another objective). Hence, 668
they encapsulate many potential preference combinations for managing the lake that are 669
mathematically non-inferior (i.e., optimal trade-offs). These trade-offs provide significant 670
contextual information and solution diversity to aid decision makers in avoiding myopia when 671
selecting a strategy for implementation. Moreover, they can facilitate negotiated compromise 672
across conflicting preferences. For example, the trade-off between expected utility and 673
phosphorus in the lake can be visualized by the backward sloping surface of the Pareto 674
approximate set. Increasing expected utility simultaneously increases the levels of phosphorus in 675
the lake, causing the Pareto front to move away from the ideal red star. This is anticipated since 676
the utility function requires some pollution emissions to obtain economic gains. 677
Simultaneous maximization of expected utility and the utility of future decision makers is 678
only feasible in our analysis if phosphorous levels are not actively reduced. This is indicated by 679
the presence of solutions near the maximum of both objectives only on the bottom right of the 680
plot. The inter-temporal tension between utility of the current decision makers and expected 681
utility is also highlighted by the change of color from red (high current decision makers’ utility) 682
to yellow/green shades (reduced current decision makers’ utility) along the increasing direction 683
of the expected utility axis. This reflects that utility of the current decision makers aims to 684
pollute the lake at its maximal capacity in the first time period so as to maximize near term 685
economic gains, which has consequences for the later periods causing a reduction in the expected 686
utility (which is estimated across the entire planning horizon). Readers are directed to the 687
Appendix Figure B1 that illustrates the trade-offs between each pair of objectives for P3 688
explicitly. 689
SINGH, REED, and KELLER: MULTI-OBJECTIVE FRAMEWORK 19
The potential for cognitive myopia to influence decision makers focused solely on the 690
utility-based formulations is evident in Figure 5. The strategy based on P2 (termed MEU) places 691
the decision makers in an extreme region of the available trade-off space, which can only be 692
comprehended by contrasting it with the multi-objective formulations. Despite a range of 693
possible compromises between expected utility and phosphorus in the lake, the P2 strategy 694
focuses solely on expected utility maximization in exclusion of understanding environmental or 695
explicit inter-temporal trade-offs. The P3 formulation permits exploration of trade-offs and 696
evaluation of the broader consequences of the single objective P1 and P2 formulations. Had we 697
used only P1 or P2 to perform the optimization, their highly restrictive definitions of optimality 698
would severely reduce decision makers understanding of lake management alternatives. The 699
trade-offs captured by P3 (stochastic five objective with a reliability constraint) provide decision 700
makers with diverse strategies ranging from those that maximize utility (lower right) towards 701
those that focus solely on maintaining low phosphorous levels in the lake (upper left) (as 702
recommended by (Brill et al. 1990). 703
Figure 5 highlights the role that visual analytics play in the selection of a suitable 704
compromise strategy/strategies. The figure presents a range of Pareto-optimal solutions that 705
comprise of solutions maximizing each individual objective along with solutions spanning the 706
entire range of possible trade-offs between these optimal values. This is in stark contrast with 707
the a priori weighing of different objectives, which results in a single a solution, thus, pre-708
determining the location of the compromise in the multi-dimensional objective space (Woodruff 709
et al. 2013). A typical weighting based solution would form just one point among the large 710
number of strategies shown in Figure 5. This process of identifying candidate solution after 711
visualization forms the next step of the MORDM process. 712
Navigating trade-offs to explore diverse alternatives 713
The multi-objective optimization of alternative problem formulation hypotheses P1, P2, 714
and P3 presents a diverse set of alternative pollution management strategies for decision makers 715
to consider. This section illustrates a posteriori constructive decision aiding where decision 716
makers with different problem conceptions and preferences can falsify formulations while 717
selecting candidate strategies of interest. These selected solutions serve as prototypes for the 718
types of preferred lake management actions decision makers would choose under well-719
characterized uncertainty (i.e., assuming model projections are of high fidelity). Since the 720
strategy obtained using the deterministic utility maximization (P1) has severe multi-objective 721
regrets even under well-characterized uncertainty (Figure 5), this formulation has been falsified 722
and we do not assess it any further. The MEU strategy obtained using P2 performed relatively 723
well in the utility-based objectives and is retained in further analysis as representative of decision 724
makers focused solely on this objective. 725
As discussed in the introduction, solving the five objective P3 formulation has the 726
implicit advantage of allowing decision makers to explore the trade-offs in the individual sub-727
problems or objective combinations as illustrated in Figure 6. The central plot in Figure 6 is 728
identical to the left panel plot in Figure 5. The four objectives are plotted using a combination of 729
three axes and color. The sides of the cubes correspond to the objectives of expected utility (x-730
axis), utility of future decision makers (y-axis) and phosphorus in the lake (z-axis). The color of 731
the solutions represents the utility of the current decision makers. The red star represents the 732
SINGH, REED, and KELLER: MULTI-OBJECTIVE FRAMEWORK 20
ideal point showing the best possible values that can be attained for all the objectives. Arrows 733
indicate the direction of increasing preference for each objective. 734
The projection plots on each side of the cube highlight three pair wise trade-offs between 735
phosphorus and the future decision makers, phosphorus and expected utility, and the trade-off 736
that emerges between expected utility and utility of future decision makers with decreasing 737
phosphorus levels. There is also a trade-off between utility of the current decision makers and 738
expected utility (Appendix Figure B1). Overall, 4 out of 10 possible pairs of objectives for P3 739
have significant trade-offs associated with them (Appendix Figure B1). These trade-offs occur 740
across the values of decision makers (preference to maintain the lake in a pristine state vs. 741
preference for economic growth), as well as differences in benefits accrued temporally (current 742
decision makers vs. decision makers across the entire planning horizon). Since there are five 743
objectives in P3, there are a total of 30 sub-problems (5 single objective, 10 two objective, 10 744
three objective, and 5 four objective) whose solutions are simultaneously plotted in Figure 6. 745
These 399 solutions presented in Figure 6 capture the trade-offs in all the sub problems, and the 746
projection plots show how these multi-dimensional trade-offs can be projected back to reduced 747
dimensions to assess individual tensions between each pair of objectives. 748
The exploration of the trade-offs and consequences of conflicting values and preferences 749
among decision makers is facilitated by Figure 6. For example, the highlighted sky blue lake 750
management strategy termed as low phosphorus represents decision makers whose sole emphasis 751
is on maintaining low phosphorus levels in the lake. This strategy represents the minimal 752
expected levels of phosphorus that can be attained in the lake. The highlighted orange lake 753
management strategy in Figure 6 termed the utility solution represents decision makers that seek 754
high values of expected utility, utility of current generation and utility of future generations. 755
This is clear from the location of this strategy in the bottom right corner of the trade-off space, 756
and its orange color. 757
The identification of compromises will invariably involve negotiations between decision 758
makers with different preferences and can also involve potentially multiple iterations on the 759
problem formulation, objectives and uncertainties considered. Negotiations and iterations on 760
problem formulation can continue until all decision makers are ‘on-board’ or have identified a 761
sub-set of mutually agreed strategies. Since it is not possible to mimic the exact process of 762
arriving at potential compromises without a set of real decision makers, we used a conceptual 763
abstraction of hypothetical decision makers. In order to represent a balance between the 764
opposing preferences captured by the low phosphorous and the utility management strategies, we 765
highlight a yellow compromise strategy that lies at the center of the trade-off region and 766
represents mid-range performance in all objectives. This strategy is termed as the compromise 767
lake management strategy. The compromise strategy was selected by maximizing the worst 768
performance across all objectives and attempts to ensure that the performance across all 769
objectives is well represented. Although this selection strategy for the illustrated compromise 770
solution is a simplified abstraction of the potential for negotiation, the broader conceptual 771
contribution is that Figure 6 encapsulates a broad suite of potential preference combinations that 772
can be explored explicitly (i.e., decision makers can choose what they prefer with knowledge of 773
what is possible). In real decision contexts, decision makers are likely to have a far more 774
nuanced evaluation of their preferences while seeking to facilitate compromises (e.g., see 775
SINGH, REED, and KELLER: MULTI-OBJECTIVE FRAMEWORK 21
Basdekas (2014)). Table 4 lists the expected objective function values obtained for the four 776
selected strategies under well-characterized uncertainties. 777
Robustness analysis 778
The a posteriori trade-off analysis on P1, P2, and P3 provides decision makers with 401 779
diverse management alternatives that capture a broad range of values or preferences. This 780
potentially helps them in identifying a smaller set of strategies that are of interest to them. As a 781
consequence of that analysis, we select four strategies (MEU, low phosphorus, compromise, and 782
utility) to represent diverse decision makers in this toy problem. In particular, the compromise 783
strategy was selected as an abstraction of the negotiation process using a simple, but ad hoc best-784
worst heuristic rule. The difficulty with this rule and other commonly employed simple multi-785
criterion weighting strategies (e.g., minimizing the distance from the ideal solution), is that they 786
assume every decision maker would be willing to make substantial sacrifices across all 787
performance objectives equally (Keeney and Raiffa 1993, Brink 1994, Köksalan et al. 2013). 788
This is a strongly questionable assumption for many resource management applications that 789
show high levels of risk aversion and a strong emphasis on the reliability of services (Brown et 790
al. 2012, Admiraal et al. 2013, Kasprzyk et al. 2013, Giuliani et al. 2014, Zeff et al. 2014). In 791
this study, we showcase a robustness analysis that assesses the performance of strategies with 792
respect to diverse performance requirements under well-characterized and deep uncertainty. To 793
this end, we first, evaluate the robustness of lake management strategies under well-characterized 794
uncertainty, and then re-assess their robustness under deep uncertainty. 795
We assess the robustness of strategies under baseline uncertainty using the performance 796
requirements discussed above. We choose three such performance requirements to approximate 797
decision makers with diverse preferences. The first and second requirements represent the 798
perspectives of decision makers that prefer on a healthy lake (whether for aesthetics or 799
economy). The first requirement is to keep the average phosphorus in the lake a factor of safety 800
(0.75) below the concentrations that may cause an irreversible collapse and the second 801
requirement is to maintain a high reliability (99%) of keeping the lake in the oligotrophic state. 802
While the first requirement is an expected performance, the second is far more stringent, 803
assessing performance across random SOWs and time. Finally, keeping in mind that some 804
decision makers may heavily prefer high economic productivity of the town, a critical minimum 805
level of economic activity was set at 50% of the value obtained in the optimal strategy for 806
expected utility maximization (P2). Note that in a real decision-support application, this process 807
of setting performance requirements can be repeated several times as new insights are gained 808
through the MORDM analysis. 809
Visualizing the performance of strategies w.r.t their robustness revealed that the 810
strategies can be classified into two disparate groups – one with very high and another with very 811
low robustness. Strategies are plotted based on their baseline expected performance for each 812
objective and colored based on their robustness under well-characterized uncertainty across 813
10000 SOWs in Figure 7. The horizontal axis represents the five objectives and the vertical axis 814
plots the objective values (normalized across the range of the objective). Each colored line is a 815
solution from Figure 6. Markers are used to distinguish the highlighted solutions from Figure 6. 816
An ideal strategy would perform optimally across all objectives, thus creating a horizontal line 817
running through the top of the plot. The worst strategy would be a horizontal line running 818
SINGH, REED, and KELLER: MULTI-OBJECTIVE FRAMEWORK 22
through the bottom of the plot. Note that the vertical scale is reversed for the phosphorus 819
objective, as minimization is preferred. Diagonal lines represent explicit trade-offs between 820
objective pairs. Colors show the estimated robustness metric for each solution. 821
We find that both low phosphorus and MEU have 0% robustness even under baseline 822
uncertainty. The low phosphorus solution is not robust as it fails to satisfy the performance 823
requirements on economic activity while the MEU solution fails to satisfy the 99% reliability 824
requirement for avoiding the eutrophic threshold in the lake. Both the utility and compromise 825
solutions have 100% robustness under baseline well-characterized uncertainty. In reference to 826
the broader set of trade-off solutions, two families of solutions are identified distinguished by the 827
red vs. blue colors. We found that all solutions were able to satisfy the phosphorus and 828
reliability performance requirements (except MEU). It is the requirement on economic activity 829
that mainly distinguishes the two families of solutions. 830
Once the robustness of solutions is assessed for baseline well-characterized uncertainty, 831
decision makers can either go back to Step 2 or Step 4 (of Figure 3) in case they do not find 832
solutions with satisfactory robustness. However, in this case, we are able to identify a significant 833
number of solutions with high robustness. Following this, we re-evaluate the performance of all 834
solutions across eight varying assumptions of uncertainty that characterize our representation of 835
deep uncertainty (Figure 4a). We explore the impact of these varying distributions from Figure 4 836
on the Pareto approximate front in Figure 8. We find that the baseline Pareto approximate front 837
transitions significantly for alternative distributions representing uncertain pollution inflow 838
changes. The front moves closer to the ideal point as the mean of the uncertain pollution inflow 839
decreases (distributions i, ii and iii), and the impact of altering the variance is negligible as seen 840
from the overlapping fronts from distributions i, ii and iii. When the mean of the pollution 841
inflow increases (distributions vii, viii, and ix), we observe severe degradation in all objectives 842
(not just the levels of phosphorus in the lake) accompanied by an increased variation across 843
alternative assumptions of variance. This implies that there are multi-objective opportunity costs 844
if the mean of the uncertain phosphorous inflows is over-estimated and multi-objective regrets 845
otherwise. This information can be helpful in providing decision makers with an understanding 846
of the effects of their assumptions and inform their risk attitudes. 847
The significant shift in the Pareto front across varying assumptions of uncertainty 848
motivates the need for assessing robustness of solution under deep uncertainty (i.e., across 90000 849
SOWs). The robustness analysis is carried out using the same performance requirements as 850
before, except for the expected levels of phosphorus in the lake. While assessing robustness 851
under well-characterized uncertainty, we found that all solutions were able to satisfy the 852
phosphorus requirements. If the same requirement is used in the further analysis of robustness 853
under deep uncertainty, it is likely that solutions remain robust despite a significant deterioration 854
in performance of phosphorus when compared to the performance under well-characterized 855
uncertainty. Since decision makers are likely to prefer solutions that maintain phosphorus levels 856
at least above than the worse performing solution in the baseline uncertainty, a stricter 857
requirement for phosphorus was set for robustness analysis under deep uncertainty. The revised 858
requirement for phosphorus was fixed at the maximum levels of phosphorus obtained across the 859
solutions under the baseline uncertainty. This is one example of how decision makers can go 860
back to Step 4 from Step 5 in Figure 3. 861
SINGH, REED, and KELLER: MULTI-OBJECTIVE FRAMEWORK 23
By assessing the robustness of strategies under deep uncertainty, we discovered that there 862
were three families of solutions with very low, moderate and high robustness respectively 863
(Figure 9). We evaluate the robustness of all strategies under the full suite of 90000 SOWs 864
sampled from all nine lognormal distributions representing deep uncertainty using the robustness 865
definition in Equation 11. The results are plotted in Figure 9(a), which is designed to be 866
comparable to the robustness evaluation under well-characterized uncertainty in Figure 7. 867
Strategies are plotted based on their re-evaluated average performance for each objective and 868
colored based on their robustness metric across 90000 SOWs. The horizontal axis represents the 869
five objectives and the vertical axis plots the objective values (normalized across the range of the 870
objective). The ranges on the y-axis were expressed as percentage of the maximum and 871
minimum objective values obtained under the baseline well-characterized uncertainty. For four 872
out of five objectives, there is a significant change in the objective function ranges when 873
compared to the solution performance in the baseline case. We find an overall deterioration in 874
the best and worst objective function values when assessing performance under deep uncertainty. 875
Colors represent robustness and markers are used to distinguish the highlighted solutions from 876
Figure 6. Since many solutions failed to satisfy the performance requirements for phosphorus 877
and reliability under deep uncertainty, these requirements are now shown as crosses on the 878
vertical axis associated with the respective objective. The pollution strategies associated with 879
selected solutions are shown as a function of time in Figure 9(b). 880
The main contrast between solution performance under well-characterized (Figure 7) and 881
deep uncertainty (Figure 9(a)) is the heavy deterioration in many solutions that were found to be 882
robust under well-characterized uncertainty. Only a small set of solutions with high robustness 883
(blue color) remain under deep uncertainty indicating that it is much harder to identify solutions 884
that can preserve performance under changing assumptions of uncertainty. The solutions that 885
were not robust under baseline uncertainty degraded even more under deep uncertainty. Many 886
solutions that were nearly 100% robust under baseline uncertainty now deteriorated to reduced 887
levels of robustness (approx. 67%). While under baseline uncertainty only the economic activity 888
requirement distinguished robust solutions, under deep uncertainty it was hard to satisfy 889
performance requirements for all three variables. Under deep uncertainty, 3 out of 4 selected 890
strategies failed to maintain high levels of robustness, while only the compromise strategy 891
remained highly robust (approx. 100%). We also find that the compromise solution is 892
surrounded by medium to low robust solutions. Therefore, it would be incorrect to assume that 893
the solutions in the vicinity of the robust solutions are likely to be robust. In contrast to Figure 7, 894
three families of solutions are identified distinguished by the red, light blue and blue colors. 895
Therefore, the robustness evaluation provides a means for both negotiating performance 896
requirements and providing a formalized approach for selection of acceptable solutions with a 897
data rich a posteriori analysis. 898
Finally, we show the performance of selected strategies under deep uncertainty in terms 899
of the distribution of water quality dynamics for the lake system (i.e., the amount of phosphorus 900
in the lake as a function of time). This shows whether or not selected strategies prevent 901
eutrophication of the lake under various parameterizations of the lognormal distributions 902
representing natural inflows. The time series of phosphorus in the lake under each 903
parameterization of the natural inflows is shown in Figure 10. The strategies are colored based 904
on their robustness under deep uncertainty in Figure 9(a). We find no considerable impact of 905
altering variance of the lognormal distribution at low to medium mean values of random 906
SINGH, REED, and KELLER: MULTI-OBJECTIVE FRAMEWORK 24
pollution inflow. Impact of altering the variance of the lognormal distribution is evident at high 907
mean values of random pollution inflow. The compromise strategy outperforms utility and MEU 908
as the mean and variance of the random pollution inflow increases. Eutrophication occurs for 909
only 2 out of 90000 SOWs for the compromise strategy. These cases occur when the mean and 910
variance of the lognormal distribution describing random inflows are highest. These two 911
strategies also fail much farther into the planning period (after year 40). On the other hand, the 912
MEU and utility strategies fail under all SOWs very early into the planning horizon (before year 913
20) when the mean of the lognormal distribution is the highest (30000 out of 90000 SOWs 914
considered) irrespective of the variance. 915
The re-evaluation of strategies under deep uncertainties highlighted the multi-objective 916
regrets that are likely to occur if utility-based frameworks are used for managing ecosystems. 917
The utility-based strategies (utility and MEU) not only lead to eutrophic collapse of the lake in 918
significant number of cases (Figure 10), they also fail to maintain high levels of robustness 919
(Figure 9). Since economic activity is also included in the performance requirements for 920
robustness, this implies that the utility-based strategies ultimately lead to a loss of both 921
environmental and economic goals of the decision makers. This is expected since the eutrophic 922
lake will also have low utility. We have falsified the utility-based problem framing in this study 923
by exposing its failure to maintain performance across objectives and under deep uncertainty. 924
On the other hand, the compromise strategy with high robustness also prevented 925
eutrophic collapse of the lake in almost all cases (Figures 9 and 10). Note the crucial role that 926
MORDM played in identifying the compromise strategy. It was the multi-objective (MO) aspect 927
of MORDM that allowed the discovery of strategies with varying levels of compromises. The 928
MO analysis also revealed the myopic location of P1 and P2 when compared to P3 strategies. 929
Then using robust decision making (RDM), we were able to test the strategies further and 930
demonstrate additional weaknesses in P1 and P2 problem framings. Finally, only a small set of 931
initially available strategies were identified as robust across a range of decision makers’ 932
preferences and deep uncertainties. Therefore, MORDM provides a means to the decision 933
makers’ to not only falsify problem formulations but also to identify new strategies that will 934
satisfy their diverse preferences and risk attitudes. 935
CAVEATS AND FUTURE RESEARCH NEEDS 936
While MORDM has several potential advantages over the classical utility-based 937
approach, there are limitations of the method in general and our approach in this study in 938
particular. First, the analyzed Pareto set is approximate. Even though this can be categorized as 939
a weakness of this approach, it is well known that most real world problems are ‘wicked’ and ill 940
defined (Rittel and Webber 1973). Consequently, small refinements of optimality are likely less 941
important than substantial structural changes in alternative problem framing hypotheses. 942
Second, the strategy obtained through our analysis is not adaptive, (i.e., there does not exist a 943
way to incorporate new knowledge about the system within this framework yet). This forms the 944
basis of ongoing work, which aims to incorporate learning within the MORDM framework. 945
Third, our specific problem framings still does not explore the full suite of uncertainties relevant 946
for the lake problem and rely heavily on the utility function. For example, we do not consider 947
the uncertainties in the lake model itself (structural as well as parametric), the parameters of the 948
utility function, etc. We did not perform an extensive uncertainty analysis to incorporate all 949
SINGH, REED, and KELLER: MULTI-OBJECTIVE FRAMEWORK 25
possible sources of uncertainties in this study, our proposed framework is well-equipped to 950
tackle a wide array of uncertainties as had been demonstrated elsewhere (Kasprzyk et al. 2013, 951
Woodruff et al. 2013). 952
Finally, communicating the high dimensional Pareto fronts as well as deep uncertainty 953
are key challenges for the MORDM approach. There have been many real application successes 954
that have used high-dimensional a posteriori trade-off analysis in complex environmental and 955
engineering design contexts which is encouraging (Fleming et al. 2005, Ferringer et al. 2009, 956
Reed et al. 2013, Basdekas 2014). We hypothesize that MORDM can change the decision 957
making process and outcomes when stakeholders have varied preferences, do not know a priori 958
the levels of compromise they can accept, and there are deep uncertainties about the system’s 959
future. To be clear, we are not aware of any formal (academic, peer reviewed, or laboratory 960
based) based study in the field of judgment and decision making that assesses the effect of 961
confronting decision makers with the kind of multi-objective trade-off displays produced in this 962
manuscript and that tests this hypothesis. 963
Successful applications of MORDM till date highlight the importance of highly 964
interactive many objective visual analytics frameworks. Some aspects of how the display of 965
deeply uncertain information affects decision making is given in Budescu et al. (2014). Also, the 966
many objective visual analytics framework by Woodruff et al. (2013) provides an approach to 967
identifying compromise strategies from high dimensional trade-off. As a last point, we would 968
like to point out that the real world usefulness of any decision aiding tool will remain dependent 969
on the effective formulation of objectives, robustness criterion, representativeness of the system 970
model, and the identifications of key sources of uncertainties. 971
CONCLUSIONS 972
We implemented a constructive decision aiding approach to identify a robust strategy for 973
the management of an irreversible lake. We presented a step-by-step procedure to implement 974
MORDM. Then, using the BORG MOEA, we were able to solve a wide variety of problem 975
formulations across a range of uncertainty assumptions. Our results revealed the failure of 976
utility-based frameworks in satisfying multiple objectives of diverse decision makers. We term 977
this failure as ‘myopia’ and display it using high dimensional visualizations. We also identified 978
a strategy that balances the various environmental, economic and inter temporal objectives not 979
only under well-characterized uncertainty but also under deep uncertainty. We were able to 980
identify this strategy by using a comprehensive definition of robustness that assesses strategy 981
performance against minimum acceptable performance requirements under various assumptions 982
of uncertainty. Our framework allows environmental planners to move beyond utility-based 983
approaches to methods that incorporate (potentially multiple) indicators of sustainability into the 984
problem framing. 985
We obtained management strategies for the lake by using three different framings (P1, 986
P2, and P3) of the lake problem across well-characterized and deep uncertainties. The most 987
important finding of this study is the evident cognitive myopia in the commonly used framing of 988
the lake problem, and by extension, most environmental management problems. We show that 989
the strategies that aim only to maximize the utility function will place decision makers in an 990
extreme region of the trade-off space. On the other hand, MORDM allows decision makers to 991
SINGH, REED, and KELLER: MULTI-OBJECTIVE FRAMEWORK 26
explore the entire trade-off space and select a posteriori the level of compromises they wish to 992
attain between various environmental, economic and inter-temporal objectives. It was 993
impossible for decision makers to discover these implicit choices without the facility of a flexible 994
problem framing provided by MORDM. 995
We also showed the crucial role played by visual analytics in the decision making 996
process. High dimensional data visualization enabled us to showcase the myopia inherent in the 997
MEU formulation (Figure 5). Each high dimensional visual allowed us to communicate the steps 998
of the posteriori decision making process: (1) exploration of multiple formulation hypotheses and 999
understanding key trade-offs under well characterized uncertainties in Figure 5, (2) exploring the 1000
set of feasible alternatives to identify candidate solutions from different regions of the trade-off 1001
space (Figure 6), (3) re-evaluation of trade-off under deep uncertainty (Figure 8), and (4) 1002
assessing robustness under well-characterized and deep uncertainty (Figure 7 and Figure 9). 1003
We employed a definition of robustness that was based on decision makers’ requirements 1004
and highlighted how the robustness of strategies changed as the assumptions of uncertainty are 1005
varied. Using this analysis, we found that the utility strategy degrades severely in robustness 1006
under deep uncertainty. It also leads to eutrophication in the lake for 33% of the deeply 1007
uncertain SOWs. Our results are also in agreement with the recent findings by Admiraal et al. 1008
(2013) who show that traditional problem framings generally fail to identify sustainable 1009
management policies for environmental systems. They attribute this to the inability of the utility 1010
function to capture the key features of a natural system that make it sustainable. We provide a 1011
framework for environmental planners to incorporate such elements (once identified) into the 1012
analysis. Embedding the requirements of sustainability within the definition of robustness, and 1013
then applying MORDM to search for robust (hence, sustainable) strategies, provides a promising 1014
avenue for environmental management problems. 1015
ACKNOWLEDGEMENTS 1016
We thank David Hadka, the developer of the BORG MOEA for his insights on 1017
implementation of the algorithm for the lake problem. This work was partially supported by the 1018
National Science Foundation through the Network for Sustainable Climate Risk Management 1019
(SCRiM) under NSF cooperative agreement GEO-1240507 and the Penn State Center for 1020
Climate Risk Management. Any opinions, findings, and conclusions or recommendations 1021
expressed in this material are those of the authors and do not necessarily reflect the views of the 1022
National Science Foundation. We thank Stephen R. Carpenter and W.A. Brock for their useful 1023
inputs on the initial stages of the analysis. We also thank Robert J. Lempert, Nancy Tuana, 1024
Gregory G. Garner, the Reed and Keller research groups, and the SCRiM external advisory team 1025
for their inputs via many discussions. Any errors or omissions are the responsibility of the 1026
authors alone. 1027
AUTHOR CONTRIBUTIONS 1028
This study was initiated by the third author. The study design evolved as an outcome of 1029
interaction between all co-authors. The analysis was performed by the first author. The first 1030
author wrote the first draft of the manuscript. All co-authors contributed towards interpretation 1031
of results and writing of the manuscript. 1032
SINGH, REED, and KELLER: MULTI-OBJECTIVE FRAMEWORK 27
LITERATURE CITED 1033
Admiraal, J. F., A. Wossink, W. T. de Groot, and G. R. de Snoo. 2013. More than total economic 1034
value: How to combine economic valuation of biodiversity with ecological resilience. 1035
Ecological Economics 89:115-122. 1036
Basdekas, L. 2014. Is Multiobjective Optimization Ready for Water Resources Practitioners? 1037
Utility’s Drought Policy Investigation. Journal of Water Resources Planning and 1038
Management 140:275-276. 1039
Becker, S. W., and F. O. Brownson. 1964. What Price Ambiguity? or the Role of Ambiguity in 1040
Decision-Making. The University of Chicago Press. 1041
Ben-Haim, Y. 2001. Information-gap decision theory: decisions under severe uncertainty. 1042
Academic Press. 1043
Bennett, E., S. Carpenter, and J. Cardille. 2008. Estimating the Risk of Exceeding Thresholds in 1044
Environmental Systems. Water, Air, and Soil Pollution 191:131-138. 1045
Bond, C. A. 2010. On the potential use of adaptive control methods for improving adaptive 1046
natural resource management. Optimal Control Applications and Methods 31:55-66. 1047
Bond, C. A., and J. B. Loomis. 2009. Using Numerical Dynamic Programming to Compare 1048
Passive and Active Learning in the Adaptive Management of Nutrients in Shallow Lakes. 1049
Canadian Journal of Agricultural Economics/Revue canadienne d'agroeconomie 57:555-1050
573. 1051
Brill, E. D., J. M. Flach, L. D. Hopkins, and S. Ranjithan. 1990. MGA: A Decision Support 1052
System for Complex, Incompletely Defined Problems. IEEE Transactions on Systems, 1053
Man, and Cybernetics 20:745-757. 1054
Brink, T. L. 1994. R.L. Keeney, H. Raiffa: Decisions with multiple objectives–preferences and 1055
value tradeoffs, Cambridge University Press, Cambridge & New York, 1993, 569 pages, 1056
ISBN 0-521-44185-4 (hardback), 0-521-43883-7 (paperback). Behavioral Science 1057
39:169-170. 1058
Brown, C., Y. Ghile, M. Laverty, and K. Li. 2012. Decision scaling: Linking bottom-up 1059
vulnerability analysis with climate projections in the water sector. Water Resources 1060
Research 48:W09537. 1061
Brozovic, N., and W. Schlenker. 2011. Optimal management of an ecosystem with an unknown 1062
threshold. Ecological Economics 70:627-640. 1063
Budescu, D., S. Broomell, R. Lempert, and K. Keller. 2014. Aided and unaided decisions with 1064
imprecise probabilities in the domain of losses. EURO Journal on Decision Processes 1065
2:31-62. 1066
Budescu, D. V., K. M. Kuhn, K. M. Kramer, and T. R. Johnson. 2002. Modeling certianty 1067
equivalents of imprecise gambles. Organizational Behavior and Human Decision 1068
Processes 88:748-768. 1069
Carpenter, S., D. Ludwig, and W. A. Brock. 1999. Management of Eutrophication for Lakes 1070
Subject to Potentially Irreversible Change. Ecological Applications 9:751-771. 1071
Castelletti, A., A. V. Lotov, and R. Soncini-Sessa. 2010. Visualization-based multi-objective 1072
improvement of environmental decision-making using linearization of response surfaces. 1073
Environmental Modelling & Software 25:1552-1564. 1074
Castelletti, A., F. Pianosi, and R. Soncini-Sessa. 2008. Water reservoir control under economic, 1075
social and environmental constraints. Automatica 44:1595-1607. 1076
Cato, M. S. 2009. Green Economics: An Introduction to Theory, Policy and Practice. Earthscan. 1077
SINGH, REED, and KELLER: MULTI-OBJECTIVE FRAMEWORK 28
Chankong, V., and Y. Y. U. h. b. g. c. i. b. i. Z.-A. Haimes. 1983. Multiobjective decision 1078
making: theory and methodology. North Holland. 1079
Chen, Y., C. Jayaprakash, and E. Irwin. 2012. Threshold management in a coupled 1080
economic–ecological system. Journal of Environmental Economics and Management 1081
64:442-455. 1082
Chichilnisky, G. 1996. An axiomatic approach to sustainable development. Social Choice and 1083
Welfare 13:231-257. 1084
Clark, J. S. 2005. Why environmental scientists are becoming Bayesians. Ecology Letters 8:2-14. 1085
Coello Coello, C., A. Hern√°ndez Aguirre, E. Zitzler, P. Fleming, R. Purshouse, and R. Lygoe. 1086
2005. Many-Objective Optimization: An Engineering Design Perspective. Pages 14-32 1087
Evolutionary Multi-Criterion Optimization. Springer Berlin Heidelberg. 1088
Cohon, J. L., and D. H. Marks. 1975. A review and evaluation of multiobjective programing 1089
techniques. Water Resources Research 11:208-220. 1090
Common, M. S., and C. Perrings. 1992. Towards an Ecological Economics of Sustainability. 1091
Beijer International Institute of Ecological Economics, the Royal Swedish Academy of 1092
Sciences. 1093
Curley, S. P., and J. F. Yates. 1985. The center and range of the probability interval as factors 1094
affecting ambiguity preferences. Organizational Behavior and Human Decision Processes 1095
36:273-287. 1096
Dasgupta, P. 2008. Discounting climate change. Journal of Risk and Uncertainty 37:141-169. 1097
Dixon, L., R. J. Lempert, T. LaTourrette, and R. T. Reville. 2007. The Federal Role in Terrorism 1098
Insurance: Evaluating Alternatives in an Uncertain World. RAND Corporation. 1099
Ellsberg, D. 1961. Risk, Ambiguity, and the Savage Axioms. The Quarterly Journal of 1100
Economics 75:643-669. 1101
Epstein, L. G., and T. Wang. 1994. Intertemporal Asset Pricing Under Knightian Uncertainty. 1102
Econometrica 62:283-322. 1103
Farber, S., R. Costanza, D. L. Childers, J. Erickson, K. Gross, M. Grove, C. S. Hopkinson, J. 1104
Kahn, S. Pincetl, A. Troy, P. Warren, and M. Wilson. 2006. Linking Ecology and 1105
Economics for Ecosystem Management. BioScience 56:121-133. 1106
Ferringer, M. P., D. B. Spencer, and P. Reed. 2009. Many-objective reconfiguration of 1107
operational satellite constellations with the Large-Cluster Epsilon Non-dominated Sorting 1108
Genetic Algorithm-II. Pages 340-349. IEEE. 1109
Fleming, P., R. Purshouse, and R. Lygoe. 2005. - Many-Objective Optimization: An Engineering 1110
Design Perspective. - 3410. 1111
Giuliani, M., J. D. Herman, A. Castelletti, and P. Reed. 2014. Many-objective reservoir policy 1112
identification and refinement to reduce policy inertia and myopia in water management. 1113
Water Resources Research 50:3355-3377. 1114
Haasnoot, M., J. H. Kwakkel, W. E. Walker, and J. ter Maat. 2013. Dynamic adaptive policy 1115
pathways: A method for crafting robust decisions for a deeply uncertain world. Global 1116
Environmental Change 23:485-498. 1117
Haasnoot, M., W. P. A. van Deursen, J. H. A. Guillaume, J. H. Kwakkel, E. van Beek, and H. 1118
Middelkoop. 2014. Fit for purpose? Building and evaluating a fast, integrated model for 1119
exploring water policy pathways. Environmental Modelling & Software 60:99-120. 1120
Hadka, D., and P. Reed. 2012. Diagnostic assessment of search controls and failure modes in 1121
many-objective evolutionary optimization. Evol. Comput. 20:423-452. 1122
SINGH, REED, and KELLER: MULTI-OBJECTIVE FRAMEWORK 29
Hall, J. W., R. J. Lempert, K. Keller, A. Hackbarth, C. Mijere, and D. J. McInerney. 2012. 1123
Robust Climate Policies Under Uncertainty: A Comparison of Robust Decision Making 1124
and Info-Gap Methods. Risk Analysis 32:1657-1672. 1125
Hashimoto, T., J. R. Stedinger, and D. P. Loucks. 1982. Reliability, resiliency, and vulnerability 1126
criteria for water resource system performance evaluation. Water Resources Research 1127
18:14-20. 1128
Herman, J. D., P. M. Reed, H. B. Zeff, and G. W. Characklis. 2015. How Should Robustness Be 1129
Defined for Water Systems Planning under Change? Journal of Water Resources 1130
Planning and Management. 1131
Herman, J. D., H. B. Zeff, P. M. Reed, and G. W. Characklis. 2014. Beyond optimality: 1132
Multistakeholder robustness tradeoffs for regional water portfolio planning under deep 1133
uncertainty. Water Resources Research 50:7692-7713. 1134
Hogarth, R., and H. Kunreuther. 1989. Risk, ambiguity, and insurance. Journal of Risk and 1135
Uncertainty 2:5-35. 1136
Holling, C. S. 1973. Resilience and Stability of Ecological Systems. Annual Review of Ecology 1137
and Systematics 4:1-23. 1138
Horan, R. D., E. P. Fenichel, K. L. S. Drury, D. M. Lodge, and S. Polasky. 2011. Managing 1139
ecological thresholds in couple environmental-human systems. Proceedings of the 1140
National Academy of Sciences 108:7333-7338. 1141
Johnson, F. A., B. K. WIlliams, and J. D. Nichols. 2013. Resilience Thinking and a Decision-1142
Analytic Approach to Conservation: Strange Bedfellows or Essential Partners? Ecology 1143
and Society 18:27. 1144
Jonkman, S. N. 2013. Advanced flood risk analysis required. Nature Clim. Change 3:1004-1004. 1145
Kasprzyk, J. R., S. Nataraj, P. M. Reed, and R. J. Lempert. 2013. Many objective robust decision 1146
making for complex environmental systems undergoing change. Environmental 1147
Modelling & Software 42:55-71. 1148
Kasprzyk, J. R., P. M. Reed, G. W. Characklis, and B. R. Kirsch. 2012. Many-objective de Novo 1149
water supply portfolio planning under deep uncertainty. Environ. Model. Softw. 34:87-1150
104. 1151
Kasprzyk, J. R., P. M. Reed, B. R. Kirsch, and G. W. Characklis. 2009. Managing population 1152
and drought risks using many-objective water portfolio planning under uncertainty. 1153
Water Resources Research 45:W12401. 1154
Keeney, R. L., and H. Raiffa. 1993. Decisions with Multiple Objectives: Preferences and Value 1155
Trade-Offs. Cambridge University Press. 1156
Keim, D. 2010. Mastering the information age: solving problems with visual analytics. 1157
Eurographics Association, Goslar. 1158
Keller, K., and D. McInerney. 2008. The dynamics of learning about a climate threshold. 1159
Climate Dynamics 30:321-332. 1160
Knight, F. H. 1921. Risk, Uncertianty and Profit. Hart, Schnaffer & Marx; Houghton Mifflin 1161
Company, Boston, MA. 1162
Köksalan, M., J. Wallenius, and S. Zionts. 2013. An Early History of Multiple Criteria Decision 1163
Making. Journal of Multi-Criteria Decision Analysis 20:87-94. 1164
Kollat, J. B., and P. Reed. 2007. A framework for Visually Interactive Decision-making and 1165
Design using Evolutionary Multi-objective Optimization (VIDEO). Environmental 1166
Modelling & Software 22:1691-1704. 1167
SINGH, REED, and KELLER: MULTI-OBJECTIVE FRAMEWORK 30
Krishnamurti, T. N., C. M. Kishtawal, T. LaRow, D. Bachiochi, Z. Zhang, E. Williford, S. 1168
Gadgil, and S. Surendran. 1999. Improved Weather and Seasonal Climate Forecasts from 1169
Multimodel Superensemble. Science 285:1548-1550. 1170
Kuhn, K. M., and D. V. Budescu. 1996. The Relative Importance of Probabilities, Outcomes, and 1171
Vagueness in Hazard Risk Decisions. Organizational Behavior and Human Decision 1172
Processes 68:301-317. 1173
Kwakkel, J., M. Haasnoot, and W. Walker. 2014. Developing dynamic adaptive policy 1174
pathways: a computer-assisted approach for developing adaptive strategies for a deeply 1175
uncertain world. Climatic Change:1-14. 1176
Lempert, R. J. 2003. Shaping the Next One Hundred Years: New Methods for Quantitative, 1177
Long-Term Policy Analysis. RAND Corporation. 1178
Lempert, R. J., and M. T. Collins. 2007. Managing the Risk of Uncertain Threshold Responses: 1179
Comparison of Robust, Optimum, and Precautionary Approaches. Risk Analysis 1180
27:1009-1026. 1181
Lempert, R. J., D. G. Groves, S. W. Popper, and S. C. Bankes. 2006. A General, Analytic 1182
Method for Generating Robust Strategies and Narrative Scenarios. Management Science 1183
52:514-528. 1184
Lempert, R. J., S. W. Popper, and S. C. Bankes. 2003. Shaping the next one hundred years new 1185
methods for quantitative, long-term policy analysis and bibliography. RAND, Santa 1186
Monica, CA. 1187
Ludwig, D. 2000. Limitations of Economic Valuation of Ecosystems. Ecosystems 3:31-35. 1188
Maler, K.-G., A. Xepapadeas, and A. D. Zeeuw. 2003. The Economics of Shallow Lakes. 1189
Environmental and Resource Economics 26:603-624. 1190
McInerney, D., R. Lempert, and K. Keller. 2012. What are robust strategies in the face of 1191
uncertain climate threshold responses? Climatic Change 112:547-568. 1192
Morse-Jones, S., T. Luisetti, R. K. Turner, and B. Fisher. 2011. Ecosystem valuation: some 1193
principles and a partial application. Environmetrics 22:675-685. 1194
Pareto, V. 1896. Cours d'economie politique. . Lausanne: F. Rouge. Tome I. 1195
Peterson, G. D., S. R. Carpenter, and W. A. Brock. 2003. Uncertainty and the Management of 1196
Multistate Ecosystems: An Apparently Rational Route to Collapse. Ecology 84:1403-1197
1411. 1198
Ralph, L. K. 2012. Value-Focused Brainstorming. Decision Analysis 9:303-313. 1199
Ramsey, F. P. 1928. A mathematical theory of saving. Economic Journal 38:543-559. 1200
Reed, P. M., D. Hadka, J. D. Herman, J. R. Kasprzyk, and J. B. Kollat. 2013. Evolutionary 1201
multiobjective optimization in water resources: The past, present, and future. Advances in 1202
Water Resources 51:438-456. 1203
Rittel, H. J., and M. Webber. 1973. Dilemmas in a general theory of planning. Policy Sciences 1204
4:155-169. 1205
Schneller, G. O., and G. P. Sphicas. 1983. Decision making under uncertainty: Starr's Domain 1206
criterion. Theory and Decision 15:321-336. 1207
Simon, H. 1959. Theories of decision-making in economics and behavioral Science. American 1208
Economic Review 49:253-283. 1209
Starr, M. K. 1963. Product Design and Decision Theory. Prentice-Hall. 1210
Stump, G. M., M. Yukish, T. W. Simpson, and E. N. Harris. 2003. Design Space Visualization 1211
and Its Application to a Design by Shopping Paradigm. Pages 795-804. ASME. 1212
SINGH, REED, and KELLER: MULTI-OBJECTIVE FRAMEWORK 31
Teytaud, O. 2007. On the hardness of offline multi-objective optimization. Evol. Comput. 1213
15:475-491. 1214
Thomas, J. J., K. A. Cook, V. National, and C. Analytics. 2005. Illuminating the path. IEEE 1215
Computer Society, Los Alamitos, Calif. 1216
Tsoukias, A. 2008. From decision theory to decision aiding methodology. European Journal of 1217
Operational Research 187:138-161. 1218
United Nations. 1987. Our common future - Brundtland Report. Pages 204. Oxford University 1219
Press, Oxford; New York. 1220
Von Neumann, J., and O. Morgenstern. 1944. Theory of games and economic behavior. 1221
Princeton university press. 1222
Webster, M., N. Santen, and P. Parpas. 2012. An approximate dynamic programming framework 1223
for modeling global climate policy under decision-dependent uncertainty. Computational 1224
Management Science 9:339-362. 1225
White, C., B. S. Halpern, and C. V. Kappel. 2012. Ecosystem service tradeoff analysis reveals 1226
the value of marine spatial planning for multiple ocean uses. Proceedings of the National 1227
Academy of Sciences 109:4696-4701. 1228
Woodruff, M., P. Reed, and T. Simpson. 2013. Many objective visual analytics: rethinking the 1229
design of complex engineered systems. Structural and Multidisciplinary Optimization 1230
48:201-219. 1231
Zeff, H. B., J. R. Kasprzyk, J. D. Herman, P. M. Reed, and G. W. Characklis. 2014. Navigating 1232
financial and supply reliability tradeoffs in regional drought management portfolios. 1233
Water Resources Research 50:4906-4923. 1234
1235
1236
SINGH, REED, and KELLER: MULTI-OBJECTIVE FRAMEWORK 32
Table I Parameters related to the lake model, uncertainty characterization, reliability estimation and 1237
stochastic sampling in BORG. 1238
Category Name Parameter Value Dimensions
Lake model Phosphorus removal rate b 0.42 dimensionless
Steepness factor q 2 dimensionless
Number of years T 100 years
Utility estimation Cost multiplier α 0.4 dimensionless
Damages multiplier β 0.08 dimensionless
Discount factor δ 0.98 dimensionless
Uncertainty estimation Number of stochastic samples
per distribution
N 10000 dimensionless
Reliability estimation Critical phosphorous level Xcrit 0.5 dimensionless
BORG algorithm Stochastic optimization
sampling frequency
- 100 dimensionless
Discretization of decision - 5 years
Significant precision of
decision
- 10-2 dimensionless
1239
SINGH, REED, and KELLER: MULTI-OBJECTIVE FRAMEWORK 33
Table II Mean and standard deviation of the lognormal distributions used in the analysis of deep 1240
uncertainty. 1241
1242
Color Mean Standard deviation
Dark blue 0.01 10-6
Blue 0.01 10-5.5
Light blue 0.01 10-5
Dark green 0.02 10-6
Green 0.02 10-5.5
Light green 0.02 10-5
Yellow 0.03 10-6
Orange 0.03 10-5.5
Red 0.03 10-5
1243
SINGH, REED, and KELLER: MULTI-OBJECTIVE FRAMEWORK 34
Table III The five objective functions and the three problem formulations used in the study. 1244
1245
Objectives Uncertainty Problem formulations
Discounted Utility Deterministic P1
Stochastic P2
Utility of 1st gen Stochastic
Utility of last 20 gens Stochastic P3
Phosphorous in the lake Stochastic
Reliability Stochastic
1246
SINGH, REED, and KELLER: MULTI-OBJECTIVE FRAMEWORK 35
Table IV Performance of four distinct types of solutions from the approximate Pareto set for the 1247
five objective problem (P3) and the single objective stochastic formulation (P2) under well-1248
characterized uncertainty. The compromise strategy is obtained by using the max-min approach 1249
discussed in the text. The utility solution is identified as the one for which performance around 1250
95% can be achieved for the expected utility (phosphorous) objectives. The low phosphorus 1251
solution has the lowest average concentration of phosphorus in the lake. 1252
1253
Objective Performance [% of range]
Phosphorus 11.9 59.0 100.0 0.8
Utility 92.6 62.1 5.1 103.7
Utility of current decision maker 86.1 72.1 22.1 52.6
Utility of future decision maker 93.6 78.1 4.5 98.2
Reliability 98.6 100.0 100.0 0.0
Assigned code Utility Compromise Low phosphorus MEU 1254
SINGH, REED, and KELLER: MULTI-OBJECTIVE FRAMEWORK 36
1255 Figure 1. The decision analytic approaches used in literature to identify management strategies 1256
for a multi-state lake with deeply uncertain variables. Studies are classified on the basis of the 1257
number of objectives in their problem formulation (y-axis) and characterization of uncertainty (x-1258
axis). 1259
SINGH, REED, and KELLER: MULTI-OBJECTIVE FRAMEWORK 37
1260 1261
Figure 2. Schematic of the lake problem with variables highlighted in bold italic letters, see text 1262
for equations and variable description. 1263
SINGH, REED, and KELLER: MULTI-OBJECTIVE FRAMEWORK 38
1264 1265
Figure 3. Flowchart for implementing many objective robust decision making (MORDM). Each 1266
box represents a step in the analysis. Solid arrows show the direction of the decision-making 1267
process. Gray arrows connect steps with the potential to alter the problem formulations with Step 1268
2. Dashed black arrows connect steps with the potential to alter decision makers’ performance 1269
requirements with Step 4. 1270
SINGH, REED, and KELLER: MULTI-OBJECTIVE FRAMEWORK 39
1271 1272
Figure 4. (a) Parameters of the lognormal distribution used for generating deep uncertainty 1273
scenarios. (b) Associated lognormal distributions. The lognormal curves in (b) are colored based 1274
on the associated circle in (a) that locates their mean and variance. The distribution shown by the 1275
black circle and the black dasehd line in (a) and (b) respectively is used to represent well-1276
characterized uncertainty. Each lognormal distribution is used to generate time series of natural 1277
inflows to the lake. 10000 such random time series are generated per lognormal distribution 1278
resulting in 10000 states of the world (SOWs) for well-characterized uncertainty, and 90000 SOWs 1279
for deep uncertainty. 1280
SINGH, REED, and KELLER: MULTI-OBJECTIVE FRAMEWORK 40
1281
1282 1283
Figure 5. Visualizing the alternative solution strategies for the lake problem obtained by using 1284
the three problem formulations (P1, P2, and P3). The formulations are – (i) maximization of utility 1285
with no uncertainty (P1), (ii) maximization of expected utility under stochastic uncertainty (P2), 1286
and (iii) simultaneous maximization of five decision makers’ objectives under stochastic 1287
uncertainty (P3). The five-objectives are expected utility maximization (x-axis), maximization of 1288
utility of the future decision makers (y-axis), minimization of phosphorus in the lake (z-axis), 1289
maximization of the utility of current decision makers (color), and maximization of reliability (not 1290
shown here since all solutions achieved reliability>98%). The total number of alternative 1291
strategies plotted is 401 (one for P1 and P2 and 399 for P3). The arrows indicate direction of 1292
increasing preference for each objective and the red star shows the ideal solution. All solutions 1293
have been re-evaluated under 10000 SOWs that represent well-characterized baseline uncertainty. 1294
SINGH, REED, and KELLER: MULTI-OBJECTIVE FRAMEWORK 41
1295 1296
Figure 6. Illustrating the process of selecting solution strategies from the set of 401 obtained in 1297
Figure 5. The axis and color are same as Figure 5. The red star locates the ideal but unattainable 1298
solution. Highlighted solutions are MEU (green), low phosphorus (light blue), compromise 1299
(yellow), and utility (orange). Marginal projections showing pair-wise relationships between 1300
objectives are shown on three sides of the cube. 1301
SINGH, REED, and KELLER: MULTI-OBJECTIVE FRAMEWORK 42
1302 1303
Figure 7. (a) Parallel coordinate plots showing the performance of solutions from P2 and P3 1304
across five objectives (x-axis) averaged across 10000 SOWs under well-characterized uncertainty. 1305
MEU (single objective), low phosphorus, compromise and utility strategies from Figure 5 are 1306
highlighted through markers. Solutions are colored based on their robustness, which is defined 1307
using the performance thresholds mentioned in the figure. The ranges of objectives shown as 1308
[minb, maxb] are [0.06, 0.21], [0.03, 0.36], [0.00, 0.04], [0.00, 0.01], [0.98, 1.00], rounded to two 1309
decimal places, for phosphorus, expected utility, utility of current decision maker, utility of future 1310
decision makers and reliability respectively. 1311
SINGH, REED, and KELLER: MULTI-OBJECTIVE FRAMEWORK 43
1312 1313
Figure 8. Re-evaluating 399 solutions from P3 under deep uncertainty. The color of the Pareto 1314
approximate fronts corresponds to the color of the lognormal distributions in Figure 4 the solutions 1315
are re-evaluated against. Expected utility, utiltiy of future decision maker, and phosphorus in the 1316
lake are plotted on x-axis, y-axis and z-axis respectively. The red star shows the location of the 1317
ideal solution and arrows indicate direction of increasing preference. Severaly degraded solutions 1318
are indicated through an arrow. 1319
SINGH, REED, and KELLER: MULTI-OBJECTIVE FRAMEWORK 44
1320 Figure 9. (a) Parallel coordinate plots showing the performance of solutions from P2 and P3 across 1321
five objectives (x-axis) averaged across 90000 SOWs under deep uncertainty. MEU (single 1322
objective), low phosphorus, compromise and utility strategies from Figure 5 are highlighted 1323
through markers. Solutions are colored based on their robustness, which is defined using the 1324
performance thresholds mentioned in the text. The ranges for objectives are shown as deviation 1325
from the [minb, maxb] ranges in Figure 7. (b) The amount of allowed anthropogenic pollution in 1326
the lake as a function of time for selected solution strategies. The strategies are colored based on 1327
their robustness and distinguished through markers. 1328
SINGH, REED, and KELLER: MULTI-OBJECTIVE FRAMEWORK 45
1329 1330
Figure 10. (i-ix) Phosphorus in the lake for MEU (single objective), low phosphorus, 1331
compromise, and utility strategies under different assumptions of uncertainty in natural inflows. 1332
Main plots show the levels of phosphorus in the lake as a function of time in years. The 1333
strategies are colored based on their robustness in Figure 9. A dashed line is used to represent 1334
the single objective (MEU) while solid lines represent the three multi-objective solutions. The 1335
side plots on the right of each main plot show the lognormal distribution of natural inflows used 1336
for estimating the phosphorus in the lake (black) against the baseline well-characterized 1337
distribution used for optimization (gray). The plots are numbered according to the distributions 1338
in Figure 4. Also note that the y-axis contains an axis break in the main plot for distributions i-1339
vi. 1340
1341
SINGH, REED, and KELLER: MULTI-OBJECTIVE FRAMEWORK 46
APPENDIX A 1342
The appendix includes: 1343
1. A detailed description of the multi-objective optimization procedure (A1) 1344
2. The impact of random seeds on the results (A2) 1345
3. Runtime dynamics of the multi-objective evolutionary algorithm (A3) 1346
A1. EVOLUTIONARY MULTI-OBJECTIVE OPTIMIZATION 1347
We use MOEAs to optimize the three problem formulations (P1, P2, and P3). The 1348
optimization finds the optimal strategies for single objective formulations (P1 and P2) and the 1349
Pareto approximate front that captures the trade-offs between all objectives for P3. Here, we use 1350
the recently developed and benchmarked MOEA, the BORG MOEA (Hadka and Reed 2013). 1351
While the advantage of using MOEAs lies in their ability to identify Pareto approximate fronts 1352
for multi-objective problems, the BORG MOEA can also be used to solve the single objective 1353
formulations such as P1 and P2. Solving the single objective formulations (P1 and P2) yields a 1354
single approximately optimal pollution strategy per formulation while solving the multi-objective 1355
formulation (P3) yields a wide range of strategies that span the trade-off between various 1356
objectives. If np3 solutions form the Pareto approximate set for P3, optimizing P1, P2, and P3 1357
together present the decision makers with a total of np3+2 potential management strategies to 1358
choose from. This process of generation of alternative strategies for decision makers to compare 1359
and contrast forms the first part of MORDM. 1360
There is a rapidly growing body of literature where MOEAs are being used to solve 1361
many-objective problems because their population-based search enables the direct approximation 1362
of problems’ Pareto frontiers in a single optimization run ((Purshouse and Fleming 2003, 1363
Fleming et al. 2005, Aguirre et al. 2013, Lygoe et al. 2013, Morino and Obayashi 2013, Reed et 1364
al. 2013, Woodruff et al. 2013). For example, while solving a five-objective problem, the 1365
MOEAs simultaneously solve the five single-objective problems, ten two-objective problems, 1366
ten three-objective problems, and five four-objective problems. This has significant 1367
computational advantages over weight-based approaches to solve multi-objective problems that 1368
assign different weights to each objective and solve a single objective problem by maximizing 1369
the aggregated weighted objective. In order to capture the entire trade-off space, such 1370
approaches need to perform the optimization many times by varying the weights, which becomes 1371
intractable as the number of objectives goes beyond three (Teytaud 2007). 1372
The BORG MOEA is able to solve high dimensional problems for non-linear threshold 1373
based models by adaptively using multiple (in this case six) search strategies. Search strategies 1374
refer to the operators that the algorithm uses to search the space of feasible solutions. Most 1375
multi-objective optimization algorithms employ a single search operator. But the BORG MOEA 1376
simultaneously uses six search strategies and assigns higher probability of use to a search 1377
strategy based on its ability to identify solutions on or close to the Pareto approximate front. 1378
Other features include random re-start of the search when no significant progress is observed (or 1379
the algorithm is trapped in a local optima), epsilon-dominance archiving (Laumanns et al. 2002), 1380
adaptive population sizing (Kollat and Reed 2007), and a steady state algorithm structure (Deb et 1381
al. 2005). Comparative analysis has demonstrated the efficacy of the BORG MOEA on a range 1382
of test problems as well as real world problems many of which require optimization under 1383
SINGH, REED, and KELLER: MULTI-OBJECTIVE FRAMEWORK 47
uncertainty (Hadka and Reed 2012, Reed et al. 2013). The BORG MOEA is also relatively easy-1384
to-use, has an underlying theoretical proof of convergence, and is highly scalable on parallel 1385
computing systems, thus increasing its potential usage across a wide variety of disciplines. 1386
The framework used for performing multi-objective optimization of the lake problem is 1387
shown in Figure 5. In order to optimize under stochastic uncertainty, we randomly sample 100 1388
SOWs out of 10000 SOWs for each evaluation of the lake problem in a function call of the 1389
BORG MOEA. In any given evaluation of a lake’s pollution strategy (parameter ‘a’ in Equation 1390
1), the relatively small number of SOWs (100) sampled is likely to yield ‘noisy’ objective 1391
function values. However, the small sample size drastically reduces computational demands. 1392
Evolutionary heuristics of the Borg MOEA underlying the Darwinian selection reward those 1393
solutions whose performance minimally varies across these small number of samples and have 1394
been demonstrated to be capable of maintaining high quality search in uncertain spaces (Miller 1395
and Goldberg 1996, Smalley et al. 2000, Reed et al. 2013). In other words, even if the BORG 1396
MOEA optimizes the objective function values across only 100 randomly sampled SOWs from 1397
the total space of 10000 SOWs at every function call, its underlying structure enables it to 1398
identify strategies that perform well across the entire ensemble of uncertain futures. This 1399
framework therefore allows for multi-objective optimization under uncertainty (i.e., identifies a 1400
strategy that performs well over many SOWs drawn from the baseline well-characterized 1401
uncertainty distribution). The robustness analysis under well-characterized uncertainty tests each 1402
solution against all 10000 SOWs, serving as a validation of this computational savings using 1403
small number of samples. Moreover, all results presented in the study are objective performance 1404
re-evaluated against all 10000 SOWs. We employ a parallelized multi-master version of the 1405
BORG MOEA that runs on a cluster with 8 islands (different evolving populations that search 1406
the space) across 64 nodes to speed up the optimization procedure. The algorithm’s search 1407
operator parameters were set at default values based on recommendations discussed in (Hadka 1408
and Reed 2013). 1409
SINGH, REED, and KELLER: MULTI-OBJECTIVE FRAMEWORK 48
1410
Figure A1. The robust optimization framework implemented using the BORG MOEA. This 1411
framework is used to optimize the various formulations of the lake model. The procedure within 1412
the solid black rectangle is repeated until a stopping criterion is met. Gray components show 1413
how the BORG MOEA implements optimization in the presence of stochastic uncertainty. 1414
SINGH, REED, and KELLER: MULTI-OBJECTIVE FRAMEWORK 49
A2. IMPACT OF RANDOM SEEDS ON RESULTING COMPROMISE STRATEGY 1415
This analysis was carried out in order to establish the reliability of the results, i.e., to ensure that the 1416
conclusions of the study are independent of a random start of the algorithm. 1417
1418 Figure A2. Analyzing the impact of random seeds on the results. Using ten random seeds to start the 1419
BORG MOEA, ten different Pareto approximate fronts are obtained for the stochastic multi-objective 1420
formulation (P3). Each Pareto approximate front is evaluated to arrive at the compromise strategy using 1421
the minimum tolerable windows approach. Each strategy is associated with five objectives - expected 1422
utility, utility of current generation, utility of future generation, phosphorus in the lake, reliability. The 1423
strategy that maximizes the minimum objective among the across all strategies is identified as the 1424
compromise strategy. Figure shows the identified compromise strategies across ten random seeds, and the 1425
envelope bounding their lower and upper limits. 1426
0 20 40 60 80
0.0
00.0
20.0
40
.06
0.0
80.1
0
Time [years]
Po
llutio
n [−
]
Random Seeds
Upper/lower limits
SINGH, REED, and KELLER: MULTI-OBJECTIVE FRAMEWORK 50
A3. RUNTIME DYNAMICS OF THE BORG MOEA 1427
The appendix presents the runtime dynamics of the algorithm. Runtime dynamics show that the 1428
algorithm converged to the Pareto approximate set within one million function evaluations. 1429
1430
1431
1432 Figure A3. Runtime dynamics for the BORG MOEA - The figure shows the evolution of the Pareto 1433
approximate front with number of function evaluations (NFE) as the BORG MOEA explores the 1434
objective space. Each plot shows expected utility on the x-axis, utility of future generations on the y-axis, 1435
SINGH, REED, and KELLER: MULTI-OBJECTIVE FRAMEWORK 51
amount of phosphorus in the lake on the z-axis. The color represents the utility of the first generation. 1436
Reliability objective is not shown here. The ideal point is shown as the red point on the bottom right 1437
corner. Snapshots of the algorithm’s search are plotted at the following NFEs – one hundred, one 1438
thousand, ten thousand, hundred thousand, one million, ten million, and 180 million. The figure shows 1439
that the algorithm converges to the Pareto approximate front within 1 million function evaluations. For 1440
the results presented in this study, we ran the algorithm on a parallel version with 8 nodes, each node 1441
running approximately 180 million evaluations. 1442
References 1443
Aguirre, H., A. Oyana, and K. Tanaka. 2013. Adaptive ε-Sampling and ε-Hood for Evolutionary 1444
Many-Objective Optimization.in R. C. Purshouse, P. J. Fleming, C. M. Fonseca, S. 1445
Greco, and J. Shaw, editors. Evolutionary Multi-Criterion Optimization. Springer Berlin 1446
Heidelberg, Sheffield. 1447
Deb, K., M. Mohan, and S. Mishra. 2005. Evaluating the epsilon-Domination Based Multi-1448
Objective Evolutionary Algorithm for a Quick Computation of Pareto-Optimal Solutions. 1449
Evolutionary Computation 13:501-525. 1450
Fleming, P., R. Purshouse, and R. Lygoe. 2005. - Many-Objective Optimization: An Engineering 1451
Design Perspective. - 3410. 1452
Hadka, D., and P. Reed. 2012. Diagnostic assessment of search controls and failure modes in 1453
many-objective evolutionary optimization. Evol. Comput. 20:423-452. 1454
Hadka, D., and P. Reed. 2013. Borg: An auto-adaptive many-objective evolutionary computing 1455
framework. Evol. Comput. 21:231-259. 1456
Kollat, J. B., and P. Reed. 2007. A framework for Visually Interactive Decision-making and 1457
Design using Evolutionary Multi-objective Optimization (VIDEO). Environmental 1458
Modelling & Software 22:1691-1704. 1459
Laumanns, M., L. Thiele, K. Deb, and E. Zitzler. 2002. Combining convergence and diversity in 1460
evolutionary multiobjective optimization. Evol. Comput. 10:263-282. 1461
Lygoe, R., M. Cary, and P. Fleming. 2013. A Real-World Application of a Many-Objective 1462
Optimisation Complexity Reduction Process. 7811. 1463
Miller, B. L., and D. E. Goldberg. 1996. Optimal Sampling for Genetic Algorithms. ASME 1464
Press. 1465
Morino, H., and S. Obayashi. 2013. Knowledge Extraction for Structural Design of Regional Jet 1466
Horizontal Tail Using Multi-Objective Design Exploration (MODE). 7811:656-668. 1467
Purshouse, R. C., and P. J. Fleming. 2003. Evolutionary many-objective optimisation: an 1468
exploratory analysis. Pages 2066-2073 Vol.2063 in Evolutionary Computation, 2003. 1469
CEC '03. The 2003 Congress on. 1470
Reed, P. M., D. Hadka, J. D. Herman, J. R. Kasprzyk, and J. B. Kollat. 2013. Evolutionary 1471
multiobjective optimization in water resources: The past, present, and future. Advances in 1472
Water Resources 51:438-456. 1473
Smalley, J. B., B. S. Minsker, and D. E. Goldberg. 2000. Risk-based in situ bioremediation 1474
design using a noisy genetic algorithm. Water Resources Research 36:3043-3052. 1475
Teytaud, O. 2007. On the hardness of offline multi-objective optimization. Evol. Comput. 1476
15:475-491. 1477
Woodruff, M., P. Reed, and T. Simpson. 2013. Many objective visual analytics: rethinking the 1478
design of complex engineered systems. Structural and Multidisciplinary Optimization 1479
48:201-219. 1480
SINGH, REED, and KELLER: MULTI-OBJECTIVE FRAMEWORK 52
APPENDIX B 1481
The appendix illustrates the trade-off plots each pair of objectives in the P3 problem formulation. 1482
1483
1484
1485
1486
1487
0.05 0.10 0.15 0.20 0.25 0.30 0.35
0.2
00.1
60
.12
0.0
8
Expected Utility [utility units]
Pho
sp
ho
rus [
dim
en
sio
nle
ss]
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●● ●●
●
●
●
●
● ●
●
● ●
●
●
●
●
●
●
●
●●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
● ●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●●●
●
●
●
●
●
●
●
●
● ●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
● ●
● ●
● ●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
● ●
●●
●
0.05 0.10 0.15 0.20 0.25 0.30 0.35
0.0
05
0.0
15
0.0
25
0.0
35
Expected Utility [utility units]
Cu
rre
nt D
M [
utilit
y u
nits]
●
●
●
●●
●●
●
●
●
●
●
●
●
●
●
●●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
● ●
●
●
●
●
● ●
●●
●
●
●
●
●
●
●
● ●
●
●
● ●●
●
●
●
●
●●●● ● ●
●●
●
●
●
●● ●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●●
●
●●
●
●
●
●
●
●
● ●
●
●
●
●
●
● ●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
● ●
● ●●
●●
●
●
●●
●
●
●●
●
●
●● ●
●●
●
●
●
●●●
●●
●
●●
●●●
●
●
●
●
●●●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
● ●
●
●
●●
●
●
●●
● ●
●
●
●
●
●
●●
●●
●
●
●
●
●●
●●
●●
●●
●
●
●
●
●
●●
●
●
●●
●
●
●●
●●
●
●
●●
●
● ●● ●
●
●
●●
●
●
● ●
●
●●
●
●
●
●
●
●●
●
●●
●
●●
●
●
● ●
●
● ●
●●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
● ●
●
●
●
●
●
● ●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●●
●
●
●
●
●
●
●●
● ●
●●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
0.05 0.10 0.15 0.20 0.25 0.30 0.35
0.0
00
0.0
04
0.0
08
Expected Utility [utility units]
Fu
ture
DM
[utilit
y u
nits]
●
●
●
● ●
●
●
●
●
● ●
●
●
●
●
●
●
●
●
●
●
● ●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●●
●
●
●
●
●
●
●
●
●
● ●
●
● ●
●
● ●
●●
●
●
●
● ●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
● ●
●
●
●
●
●
●
●
●
● ● ●
●
●
●
●
●
● ●
●●
●
●
● ●
● ●
●
●
●
●
●
●
●
●
●
●
●
●
● ●
● ●●
●●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●● ●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
● ●
●
●
● ●
●
●
●
●
●
●
● ●
●
●●
●
●
●
●
●
●●
●
● ●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
● ●●
● ●
●
●
●
●
●
●
●
●
● ●
●
●
●
●
●
●
● ●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●●
●●
●
●
●
●
●
● ●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●●
●
0.05 0.10 0.15 0.20 0.25 0.30 0.35
0.9
92
0.9
96
1.0
00
Expected Utility [utility units]
Re
liab
ility
[dim
en
sio
nle
ss] ●●●● ●●●● ●● ●●
●
● ●● ●●●●●● ●●● ● ●
●
● ● ●● ● ●● ●●● ●●
●●
●
●
●● ●
●
● ●● ● ●●
●
●
● ● ●
●
●
●●● ● ●
●
●● ● ●●
●
● ●● ●● ●●● ●●
●
● ●●● ●● ●●
● ●● ●● ●●●●● ●●● ●● ●● ●● ● ● ●● ●● ●
●
●●●● ●●● ●
●
●● ●● ●● ● ●●● ● ●
● ●
● ●
●
●●● ● ●
●
●● ●● ●
●
●
● ●●
●
●●●●●
●
●● ● ●● ● ● ●● ●● ●●●● ●
●
●●● ●● ● ●● ●●● ●● ● ●● ● ●●● ● ●●
●●
● ●● ●●●
●
●●● ● ●
●
●●●
●
●● ● ● ●●● ●●
●
●● ● ●● ●●● ●● ●●
●
●
●
● ●● ●● ● ● ●● ●● ●● ● ●●● ●● ●●●● ●●
●
●
●
● ●●● ●●● ●● ●●
●
●● ● ●●
●
● ●
●
● ●●●● ●●● ● ●●● ●
●●●
● ●●● ● ●● ● ● ● ● ●● ●● ● ●●● ●●● ●●● ●●
●
● ●
●
●
●
● ● ●● ● ●●● ● ● ● ●
●
●
●
● ●
●
● ●● ●
●
●● ● ●●● ● ●
●
● ● ●●●●●●●
● ●● ● ●
●
●
●
0.20 0.18 0.16 0.14 0.12 0.10 0.08 0.06
0.0
05
0.0
15
0.0
25
0.0
35
Phosphorus [dimensionless]
Cu
rre
nt D
M [
utilit
y u
nits]
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●●
●
●●●
●
●
●
●
●
●
●
●
●
●
●
● ●
●
●
●●
●
●
●
●
●●
●●
●
●
●
●
●
●
●
●●
●
●
●● ●
●
●
●
●
●●
● ●●●
●●
●
●
●
●
●●
●
●
●●
●
●
●
●
●
●
●
●●
●
● ●
●
●●
●
●
●
●
●
●
●●
●
●
●
●
●●●
●●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●●
●●●
●●
●
●
●●
●
●
● ●
●
●
●●●
● ●
●
●
●
●● ●
● ●
●
●●
● ●●
●
●
●
●
● ●●
●
●
●
●
●
● ●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●●
●
●
● ●
●●
●
●
●
●
●
● ●
●●
●
●
●
●
●●
●●
●
●
●●
●
●
●
●
●
●●
●
●
●●
●●
● ●
● ●●
●
●●
●
●●●●
●
●
●●
●
●
●●
●
●●
●
●
●
●
●
●●
●
●●
●
●●
●
●
●●
●
●●
●●
●
●
●
●
●
●
●
● ●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●●
●
●
●
●
●
●
●●
●●
● ●
●
●
●
●
●
●
● ●
●
●
●
●
●
●
●
●
●
●
●
●
●
0.20 0.18 0.16 0.14 0.12 0.10 0.08 0.06
0.0
00
0.0
04
0.0
08
Phosphorus [dimensionless]
Fu
ture
DM
[utilit
y u
nits]
●
●
●
●●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●●
●
●
●
●
●
●
●
●
●
●●
●
●●
●
●●
● ●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
● ● ●
●
●
●
●
●
● ●
● ●
●
●
● ●
●●
●
●
●
●
●
●
●
●
●
●
●
●
● ●
●● ●
●●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
● ● ●
●
●
●
●
●
●
●
●
●
●
●
●
● ●
●
●
●
●
●
●
●
●
● ●
●
●
●●
●
●
●
●
●
●
●●
●
●●
●
●
●
●
●
●●
●
●● ●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
● ●
●
●
●
●
●
●
●
●
●
●● ●
●●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●●
● ●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●●
● ●
●
●
●
●
●
●●
●●
●
●
●
●
●
●
●
●
●
●
●
●
● ●
●
●
●
●
●
●
●
●
●
●
●●
●
0.20 0.18 0.16 0.14 0.12 0.10 0.08 0.06
0.9
92
0.9
96
1.0
00
Phosphorus [dimensionless]
Re
liab
ility
[dim
en
sio
nle
ss] ●● ●●● ● ● ●● ●● ●
●
●●●● ●● ● ●●●● ●● ●
●
●●● ●●● ●● ● ●● ●
●●
●
●
● ●●
●
●● ●●● ●
●
●
● ●●
●
●
●● ●●●
●
● ●●● ●
●
●● ●● ●● ● ●● ●
●
●● ● ●● ●●●
●● ●● ●● ● ●●●● ●●● ●● ●● ●●●● ●● ●●
●
● ● ● ●●● ●●
●
● ●● ●●●● ● ● ● ●●
● ●
●●
●
● ●● ●●
●
● ●● ●●
●
●
●● ●
●
● ●● ● ●
●
●● ●● ●● ●● ●● ●● ●●● ●
●
● ●● ● ●●● ●● ● ●● ●●●● ●●● ●●● ●
● ●
●● ●● ● ●
●
●● ●●●
●
●● ●
●
● ● ●●● ● ●● ●
●
● ● ●● ●● ● ●● ●●●
●
●
●
● ● ●● ●●●●●● ●● ●●● ●●● ●● ●●● ● ●
●
●
●
●● ●●● ● ●● ●●●
●
●●●● ●
●
●●
●
●● ● ● ●● ● ●● ●● ●●
●● ●
●● ●● ●● ●●● ●●● ●● ●●● ● ●● ● ●●● ●●●
●
●●
●
●
●
●● ● ●●● ● ● ●●●●
●
●
●
● ●
●
●●●●
●
● ●●● ●● ●●
●
●●● ●● ●● ●●
●●●●●
●
●
●
0.000 0.002 0.004 0.006 0.008
0.0
05
0.0
15
0.0
25
0.0
35
Future DM [utility units]
Cu
rre
nt D
M [
utilit
y u
nits]
●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●●
●
●● ●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
● ●
●
●
●
●
●●
●●
●
●
●
●
●
●
●
●●
●
●
● ●●
●
●
●
●
●●
●● ● ●
●●
●
●
●
●
● ●
●
●
●●
●
●
●
●
●
●
●
●●
●
●●
●
●●
●
●
●
●
●
●
● ●
●
●
●
●
●● ●
●●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●
● ●
● ●●
●●
●
●
●●
●
●
●●
●
●
●● ●
●●
●
●
●
●●●
●●
●
●●
●●●
●
●
●
●
●●●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
● ●
●
●
●●
●
●
●●
● ●
●
●
●
●
●
●●
●●
●
●
●
●
●●
●●
●
●
●●
●
●
●
●
●
●●
●
●
●●
●●
●●
●●●
●
●●
●
●●● ●
●
●
●●
●
●
● ●
●
●●
●
●
●
●
●
●●
●
●●
●
●●
●
●
● ●
●
● ●
●●
●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●●
●
●
●
●
●
● ●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
●
●
●●
●
●
●
●
●
●
●●
● ●
●●
●
●
●
●
●
●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
0.000 0.002 0.004 0.006 0.008
0.9
92
0.9
96
1.0
00
Future DM [utility units]
Re
liab
ility
[dim
en
sio
nle
ss] ● ●● ●●● ●● ●●●●
●
● ●● ●● ●●● ●●●● ●●
●
●●●● ●●● ●●● ●●
●●
●
●
●●●
●
● ●● ●●●
●
●
●●●
●
●
● ●● ●●
●
●● ● ●●
●
● ●● ●● ●●● ●●
●
● ●●● ●● ●●
● ●● ●● ●●● ●● ●● ● ●● ●● ●● ● ● ●● ●●●
●
●●●● ●● ● ●
●
●● ●● ●● ●●●●● ●
●●
●●
●
●● ●● ●
●
●● ●● ●
●
●
●●●
●
●●●●●
●
● ●● ●● ●● ●● ●● ●● ●●●
●
●●●●● ● ●● ●●● ●● ● ● ●● ●●● ● ●●
●●
● ●● ●●●
●
● ●● ● ●
●
●●●
●
●●●● ●●● ●●
●
● ●● ●● ●●● ●● ●●
●
●
●
●●● ●● ● ●● ●●● ●●● ●●● ●● ●● ● ●●●
●
●
●
● ● ●● ●● ●●● ● ●
●
●● ● ●●
●
● ●
●
●● ●●● ●● ● ●● ●● ●
●●●
●●●●● ● ● ● ●● ●●● ●●● ●● ●●●● ●●● ●●
●
● ●
●
●
●
●●●● ● ●● ●● ● ● ●
●
●
●
●●
●
●● ● ●
●
●● ● ●● ●● ●
●
● ● ●● ●● ● ●●
● ● ●●●
●
●
●
0.005 0.015 0.025 0.035
0.9
92
0.9
96
1.0
00
Current DM [utility units]
Re
liab
ility
[dim
en
sio
nle
ss] ● ● ● ● ●● ●● ● ● ●●
●
●● ● ●● ●●●●● ●● ●●
●
● ● ● ●●●● ● ●●●●
● ●
●
●
●●●
●
● ●● ●●●
●
●
●● ●
●
●
● ●● ●●
●
●●●● ●
●
●● ● ● ●●●● ●●
●
●● ●● ● ● ●●
● ●●● ●● ●●● ●●● ●● ●● ●● ●●● ●●●● ●
●
●● ●● ● ●● ●
●
●● ●●● ●●● ●●● ●
●●
● ●
●
●● ●● ●
●
●● ●● ●
●
●
● ●●
●
● ●●●
●
●
● ●● ●●●●● ● ● ●●● ● ●●
●
● ● ●●● ● ● ● ● ●● ●● ●● ●●● ●● ●● ●
● ●
●●● ●● ●
●
● ● ● ●●
●
● ●●
●
●●● ●● ●●●●
●
●●● ●●● ● ●●● ● ●
●
●
●
●●● ●●●● ● ●● ●●● ● ●● ●● ●●● ● ●●●
●
●
●
● ●● ●● ●●●● ● ●
●
● ●●● ●
●
● ●
●
● ●●●●● ●● ●● ●● ●
●● ●
● ●● ●●●● ●● ● ● ●● ●● ●● ●● ●●● ● ●● ● ●
●
● ●
●
●
●
● ●● ●●● ●● ● ● ●●
●
●
●
●●
●
● ●● ●
●
●●● ●●● ● ●
●
● ●●● ●● ●●●
●●● ●●
●
●
●
SINGH, REED, and KELLER: MULTI-OBJECTIVE FRAMEWORK 53
Figure B1. Tradeoffs between each pair of objectives for the five-objective formulation P3. The 1488
objectives are – phosphorus in the lake (minimize), expected utility (maximize), utility of the present 1489
generation (maximize), utility of the future generations (maximize), and reliability (maximize). The 399 1490
solutions from the five objective optimization are plotted as gray points. Non-dominated sorting is 1491
carried out for each pair of objectives across the 399 points to identify the Pareto approximate front for 1492
each pair of objectives. If there is tension between the two objectives, a front is identified and plotted as 1493
think black line. If there is no tension between two objectives, it implies that both can be simultaneously 1494
optimized and the ideal point is attainable, shown by the black circle around the red diamond. The red 1495
diamond represents the ideal point. Gray shading represents the dominated region; solutions in this 1496
region are inferior to those at the Pareto approximate front. Note that all 399 solutions are non-dominated 1497
w.r.t each other in the five-dimensional objective space, but when the space is collapsed to two 1498
dimensions, the non-dominated sorting is done again to identify the new Pareto front for two objectives. 1499
Top Related