Seediscussions,stats,andauthorprofilesforthispublicationat:http://www.researchgate.net/publication/220535093

AssessingJointDistributionswithIsoprobabilityContours.

ARTICLE·JANUARY2010

DOI:10.1287/mnsc.1100.1161·Source:DBLP

CITATIONS

4

DOWNLOADS

296

VIEWS

172

3AUTHORS,INCLUDING:

DavidV.Budescu

FordhamUniversity

170PUBLICATIONS4,268CITATIONS

SEEPROFILE

YuhongGu

UniversityofIllinois,Urbana-Champaign

1PUBLICATION4CITATIONS

SEEPROFILE

Availablefrom:DavidV.Budescu

Retrievedon:18June2015

MANAGEMENT SCIENCEVol. 56, No. 6, June 2010, pp. 997–1011issn 0025-1909 �eissn 1526-5501 �10 �5606 �0997

informs ®

doi 10.1287/mnsc.1100.1161©2010 INFORMS

Assessing Joint Distributions withIsoprobability Contours

Ali E. AbbasDepartment of Industrial and Enterprise Systems Engineering, College of Engineering,University of Illinois at Urbana–Champaign, Urbana, Illinois 61801, aliabbas@uiuc.edu

David V. BudescuDepartment of Psychology, Fordham University, Bronx, New York 10458; and Department of Psychology,

University of Illinois at Urbana–Champaign, Urbana, Illinois 61801, budescu@fordham.edu

Yuhong (Rola) GuDepartment of Psychology, University of Illinois at Urbana–Champaign, Urbana, Illinois 61801,

gyh1224@gmail.com

We present a new method for constructing joint probability distributions of continuous random variablesusing isoprobability contours—sets of points with the same joint cumulative probability. This approach

reduces the joint probability assessment into a one-dimensional cumulative probability assessment using asequence of binary choices between various combinations of the variables of interest. The approach eliminatesthe need to assess directly the dependence, or association, between the variables. We discuss properties ofisoprobability contours and methods for their assessment in practice. We also report results of a study in whichsubjects assessed the 50th percentile isoprobability contour of the joint distribution of weight and height. Weuse the data to show how to use the assessed contours to construct the joint distribution and to infer (indirectly)the dependence between the variables.

Key words : isoprobability contours; joint probability elicitation; probability encoding; correlation; dependenceHistory : Received November 19, 2008; accepted December 29, 2009, by George Wu, decision analysis.

Published online in Articles in Advance April 23, 2010.

1. IntroductionThe construction of representative probability distri-butions from a decision maker (DM) is a fundamentalstep in decision analysis and has engendered a sub-stantial literature. It is well known that the probabilityelicitation process for simple events or continuous ran-dom variables can be subject to many cognitive andmotivational biases (see, for example, Hogarth 1987,Kahneman and Tversky 1973, Spetzler and Staël vonHolstein 1975, Tversky and Kahneman 1974, Wallstenand Budescu 1983). When eliciting a joint probabilitydistribution, the magnitude of this task is much largerbecause we are faced with the need to incorporatethe dependence among the random variables into theanalysis. Several authors have discussed these issuesand have presented methods to facilitate the elicita-tion process (see O’Hagan et al. 2006 for a review).For example, Howard (1989) proposed evocative andredundant knowledge maps to graphically representthe dependence relations and provided examples ofhow they simplify the construction of joint probabil-ity distributions. Other methods approximate the joint

distribution in terms of lower-order joint probabil-ity assessments. For example, Keefer (2004) presenteda binary event model for approximating dependencerelations between the variables, and Abbas (2006)presented a maximum entropy approach for con-structing a discrete joint probability distribution usinglower-order joint probability assessments. Some meth-ods for constructing joint probability distributions ofcontinuous variables require information about thecross-moments or the correlation coefficients betweenthe variables. For example, Cooke (1991) proposeda continuous maximum entropy joint distributionusing moments and correlation coefficients, Yi andBier (1998) constructed joint distributions using cop-ula structures and certain dependence parameters forrisk analysis, and Clemen and Reilly (1999) used mul-tivariate normal copulas to construct a joint distribu-tion using pairwise correlation coefficients.Several “direct” methods for assessing dependence

between variables have also been proposed. For exam-ple, Gokhale and Press (1982) elicited concordanceprobabilities from subjects. Under the assumption of

997

Abbas, Budescu, and Gu: Assessing Joint Distributions with Isoprobability Contours998 Management Science 56(6), pp. 997–1011, © 2010 INFORMS

bivariate normality, these probabilities can be relatedto the Pearson correlation coefficient.1 Clemen et al.(2000) conducted a study comparing various methodsof eliciting the correlation between pairs of variables.Under the assumption of bivariate normality, it is pos-sible to use these estimates to infer the joint probabil-ity distribution of the two variables. They found thatdirect assessments of correlation coefficients on a scaleof −1 to 1 outperformed several other methods. Thisresult contradicts previous studies (such as Morganand Henrion 1990, Kadane and Wolfson 1998).The judgment literature shows that people are noto-

riously poor estimators of association and correlationbetween variables. For example, Meyer and Shinar(1992) and Meyer et al. (1997) studied the inferenceof correlations based on visual displays (scatterplots)and found that the dispersion of the data and thepresence of outliers greatly affected the perception ofthe correlation and led to large underestimation of thecorrelation between the variables. Other researchers(e.g., Alloy and Tabachnik 1984, Arkes and Harkness1983, Beyth-Marom 1982, Hamilton and Gifford 1976,Trolier and Hamilton 1986) showed that judgmentsof covariation and correlations are highly sensitiveto (a) the method and format of data presentation,(b) the instruction for judgments, and (c) the judges’prior expectations. In particular, people tend to detect“illusory correlations,” i.e., systematic covariation incases where it is not present (e.g., Allan and Jenkins1980, Chapman 1967).In this paper, we present a new and fundamen-

tally different method for constructing joint proba-bility distributions of continuous random variables.This approach dispenses with the need to assessconditional probability distributions, correlation coef-ficients, concordance probabilities, or conditionalpercentiles. The proposed procedure reduces thejoint probability assessment into a one-dimensionalmarginal probability assessment of one of the vari-ables, and the elicitation of one or more “isoprobabilitycontours.”An isoprobability contour of a joint cumulative dis-

tribution is the collection of all the points that havethe same cumulative probability, and is analogous toan isopreference contour for a multiattribute utilityfunction (Keeney and Raiffa 1976). Although therehas been work on assessing isopreference contours(e.g., the classic paper by MacCrimmon and Toda(1969) on assessing isopreference contours of bundlesof money and ballpoint pens), and work by Mathesonand Abbas (2005) on deriving the relation betweenconditional utility functions of attributes using iso-preference contours and a one-dimensional utilityfunction over value, we are not aware of theoretical

1 Pr�x < �x�y < �y� = 0�25+ sin−1��xy/2��.

or empirical work on assessing isoprobability con-tours of joint cumulative distributions. As we shownext, isoprobability contours have some additionalproperties that facilitate their assessment in practice.The differences between the isopreference and iso-probability contours are due to the different levels ofmeasurement of the utilities (typically, interval scalesunique up to a positive linear transformation) andprobabilities (absolute scales). Thus, this work goesbeyond simple reparameterization of the isoprefer-ence contours.We also propose a method to assess isoprobability

contours in practice. This approach does not requirethe subjects to provide any numeric judgments, butsimply to state their preferences over binary lot-teries having identical outcomes. The approach cantherefore be used with judges having less technicalexpertise in probability or statistics. The probabilitiesdefining the two lotteries involve the two variablesof interest and, as such, cause the judge to considerthe relevant trade-offs between them. To the best ofour knowledge, this is the only method that treatsthese probability trade-offs directly and explicitly andmakes them an integral part of the elicitation pro-cess. Once the isoprobability contours are determined,a one-dimensional marginal probability assessmentover any of the variables (or over the contours them-selves) is sufficient to determine the complete jointdistribution of all the variables. As a result, isoprob-ability contours can also be used to determine thedependence parameters between the variables of thedecision situation.The remainder of this paper is structured as fol-

lows. Section 2 presents some properties of isoproba-bility contours that facilitate their elicitation. Section 3presents methods to assess isoprobability contoursand methods to construct the joint distribution. Sec-tion 4 describes the experimental procedure. Section 5reports results of a study designed to test the feasi-bility of assessing isoprobability contours. Section 6presents methods to curve fit isoprobability contoursto functional forms to simplify the construction of thejoint distribution. Section 7 curve fits isoprobabilitycontours to a bivariate Gaussian copula and estimatesthe dependence between the two variables. Section 8summarizes our results.

2. Properties of IsoprobabilityContours

To start, consider two continuous variables, X andY , having a joint cumulative probability distribution,F �x�y�, over a connected bounded domain, i.e., x ∈xmin�xmax, y ∈ ymin�ymax. An isoprobability contourfunction, Ci�x�y�, is the set of all points in the x–yplane that have the same cumulative probability, Ci:

Ci�x�y� = ��x�y�� F �x�y� = Ci � (1)

Abbas, Budescu, and Gu: Assessing Joint Distributions with Isoprobability ContoursManagement Science 56(6), pp. 997–1011, © 2010 INFORMS 999

Figure 1 Joint Cumulative Distribution and Isoprobability Contours

Varia

ble

x

Variable y

Joint cumulative distribution function

h

0 0.1 0.2

0.3

0.4 0.5

0.6

0.7

0.8 0.9

1.0

0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1.0

0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1.000.050.100.150.200.250.300.350.400.450.500.550.600.650.700.750.800.850.900.951.00

Variable x

Var

iabl

ey

Isoprobable contours

h

Figure 1 shows an example of a joint cumulativedistribution and the corresponding two-dimensionalprojection of some of its isoprobability contours.Isoprobability contours have several properties that

can facilitate their elicitation in practice and can serveas consistency checks during the elicitation process.We discuss some of these properties below.

Property 1. The left and bottom boundaries of thedomain are isoprobability contours.

Proof. A joint cumulative distribution has a valueof zero if any of the variables is at the minimal valueof the domain. This is known as the “grounding prop-erty,” where

F �x�ymin� = F �xmin�y� = 0

∀x ∈ xmin�xmax� y ∈ ymin�ymax� (2)

Equation (2) implies that all points on the axesx = xmin and y = ymin have a joint cumulative probabil-ity of zero and therefore lie on the same isoprobabilitycontour. �

Property 2. If differentiable, isoprobability contourshave a nonpositive slope.

Proof. Recall that the change in probability alongan isoprobability contour must be zero. Hence,

dF �x�y� �Ci=[

�F �x�y�

�xdx + �F �x�y�

�ydy

]Ci

= 0� (3)

Rearranging (3) gives

dy

dx

∣∣∣∣Ci

= −�F �x�y�/�x

�F �x�y�/�y= −Fy�y���Fx �y�x � y�/�x�

Fx�x���Fy �x�y �x�/�y�� (4)

where Fx �y�x �y� � P�X ≤ x � Y ≤ y�. The right-hand side of (4) is nonpositive, because a cumula-tive distribution function (CDF) is nonnegative andnondecreasing. �

Given some assessed points on an isoprobabil-ity contour, Property 2 can be tested by calculat-ing Kendall’s �b rank-order correlation between thepoints, as we illustrate in §5.The following properties are also useful and fol-

low directly from the definition of isoprobabilitycontours.

Property 3. Isoprobability contours do not intersect.

Property 4. Isoprobability contours connect pointswith the same marginal probability, as shown in Figure 2.

In the next section we use these properties to assessisoprobability contours and to verify the quality of theassessments.

Figure 2 Isoprobability Contours Connect Points with IdenticalMarginal Probabilities

x

yF (x ) = p

F(y ) = p

F (x, ymax) = F (x )

F(xmax, y ) = F (y )

Abbas, Budescu, and Gu: Assessing Joint Distributions with Isoprobability Contours1000 Management Science 56(6), pp. 997–1011, © 2010 INFORMS

3. Assessing an IsoprobabilityContour and Constructing theJoint Distribution

3.1. Assessing an Isoprobability ContourSuppose we have assessed the median (50th per-centile) of variable X, which we denote as x50, aswell as other fractiles of X, such as x75 and x90. Wenow wish to assess the 50% isoprobability contour.We start by offering the DM a choice between twodeals with identical outcomes. Thus, the DM’s choiceshould depend solely on the probabilities. Considerfor example, the following two deals.Deal A: The DM receives a fixed amount, $, if the

outcome of variable X is less than x50 and variable Ytakes any value (i.e., Y ≤ ymax�.

Deal B: The DM receives the same fixed amount, $,if the outcome of variable X is less than x75 and theoutcome of variable Y is less than y1 (where y1 < ymax).Based on the DM’s response, we adjust the value

of y1 sequentially until the user expresses indifferencebetween �x50�ymax� and �x75�y1�. If the DM does notexpress indifference between the two deals after a pre-determined number of binary questions, we settle forupper and lower bounds for the value of y1, which wedenote as �x75�yU

1 � and �x75�yL1 �, respectively, so we

may have a narrow band around the contour. We con-sider the midpoint of this band as the estimate of y1.We then move on to the next point and ask the DM tocompare deals based on �x75�y1� and �x90�y2� (wherey2 < y1�, and change y2 in a similar fashion until wereach indifference between them. Through this chain-ing process we trace the 50% isoprobability contourof the DM. This process is illustrated in Figure 3. It ispossible to supplement this process with consistencychecks involving previously elicited points lying onthe contour.This process is symmetric across variables, i.e., the

roles of the X’s and Y ’s can be reversed.

Figure 3 Assessing Points on an Isoprobability Contour by Indifference Choices

Q = (x75, y1)Q = (x75, y1) Q = (x90, y2)

x

y

x50

y50

x75 x

y

x50

y50

x75 x90

3.2. Probability Trade-OffsFor any two points �x1�y1� and �x2�y2� along the sameisoprobability contour, consider the ratio

�y12

�x12

∣∣∣∣Ci

= y2 − y1

x2 − x1� (5)

This ratio is nonpositive (Property 3) and representsthe increase in the value of one variable needed tocompensate for a decrease in another to achieve thesame joint probability. As the increment in one vari-able decreases (and �x → 0�, this ratio approachesthe slope of the isoprobability contour (the proba-bility trade-off). This provides a simple and intu-itive interpretation for the slope of the isoprobabilitycontours—the change in one variable needed to com-pensate for a change in the other, to achieve the samecumulative probability. We can also define the ratio ofpercentage change in one variable needed to compen-sate a percentage change in the other,

��x�y� = −�y/y

�x/x

∣∣∣∣Ci

� (6)

We have chosen to add a negative sign in this def-inition to yield nonnegative values of the probabilitytrade-off. This ratio is easier to think about than theabsolute value of the increments because it is dimen-sionless and the DM needs only to provide a percent-age between 0% and 100%. As the increment �x → 0in (6), a fractional change corresponds to the deriva-tive, � = −�dy/dx��x/y�. Contours having a constant� define a family of functional forms whose shapedepends on this parameter and are often referred toas the constant elasticity of substitution curves.

3.3. Constructing the Joint ProbabilityDistribution

Once the isoprobability contours have been deter-mined, the construction of the joint distribution

Abbas, Budescu, and Gu: Assessing Joint Distributions with Isoprobability ContoursManagement Science 56(6), pp. 997–1011, © 2010 INFORMS 1001

requires only the assessment of one of the marginalprobability functions, say Fx�x�. We can determine thejoint cumulative distribution of any point, �x�y� ∈xmin�xmax×ymin�ymax, by finding the point �x1�ymax�that lies on its isoprobability contour. By definition,

F �x�y� = F �x1�ymax�� Fx�x1�� (7)

This formulation reduces the joint probability intoa one-dimensional marginal probability. The marginaldistribution function, Fx, can be assessed directly usingany of the standard techniques of probability encodingused in decision analysis (Spetzler and Staël von Hol-stein 1975, von Winterfeldt and Edwards 1986, Abbaset al. 2008), or by assuming a particular functionalform (e.g., a beta distribution) and assessing its param-eters by assessing a few probability points using indif-ference to binary lotteries. This process is symmetricacross variables, i.e., we can replace Fx by Fy .Define

x1�x�y� = x1 such that C�x�y� = C�x1�ymax��

For any (x�y), define x1 such that C�x�y� =C�x1�ymax�. Then, F �x�y�� Fx�x1�, and the conditionalprobability, F �x � y�� is

F �x � y� = Pr�X ≤ x � Y ≤ y� = F �x�y�

Fy�y�= Fx�x1�

Fy�y�� (8)

The conditional probability functions at any pointin the plane can also be related explicitly to theassessed trade-off functions of (55) and the marginaldistributions using the same transversality relationsthat were developed in Matheson and Abbas (2005).

4. An Empirical Study4.1. DesignThe goal of this study is to determine whether thejudges can make sensible probability trade-off judg-ments that can be used to infer meaningful joint dis-tributions. We asked a number of judges to assessfractiles of the marginal distribution and the 50% iso-probability contours of two variables with which theywere highly familiar.We elicited the joint probability of the height and

weight of male undergraduate students at the Uni-versity of Illinois at Urbana–Champaign (UIUC).We selected these variables because all our sub-jects2 (UIUC students) were highly familiar withthem. Twenty-five students volunteered to partici-pate. They were guaranteed $8 for their participationand could also earn an additional $20, based on one of

2 We use the terms “subject” and “judge” interchangeably through-out this section.

their choices. This provided an incentive to respondaccurately and honestly. Subjects were between 19and 28 years old, with a median age of 24. Nine ofthem were female. They were from various depart-ments, and 19 of them (75%) reported that they hadcompleted at least one class in probability.The experiment was conducted in two different set-

tings. Ten subjects were run in a face-to-face setting,whereas the other 15 were run in the computerizedcondition. The two groups were very similar in mostrespects. We did not have specific hypotheses aboutthe performance of the two groups. Our goal was sim-ply to test the feasibility of the method under variouscircumstances. The face-to-face condition was similarto a probability encoding session in decision analysisin the sense that the experimenter3 had more flexi-bility and could adjust the nature and sequence ofthe questions as a function of the subject’s responsesto converge to the key values. As such, this groupprovides the gold standard against which the resultsof the computerized group, which was run withoutdirect interaction between the subjects and the exper-imenter, can be evaluated.All subjects were presented with pairs of gambles,

each of which offered some chance to win $20. Thetwo gambles in a pair described two different events(see details below). In the face-to-face group, sub-jects followed the instructions given by the experi-menter. In the computerized group, the instructionswere included in the computer program and a hardcopy of the instructions was also available for refer-ence. Subjects were asked first to select their preferredunits (pounds or kilograms for weight and centime-ters or inches for height) and to specify their rangeof values for both variables. Next they estimated themarginal distributions and then assessed the contourof the joint distribution.

4.2. Eliciting the Fractiles of the MarginalDistributions

To elicit the marginal probability distributions of theheight and weight variables, we used pairs of gam-bles involving (a) a probability wheel and (b) oneevent related to the weight or the height of a ran-domly selected male student from UIUC. For exam-ple, a wheel showing a 50% chance of winning wascompared with the possibility that the weight of arandomly selected male student at UIUC will be lessthan 120, 140, 160 lbs, etc.In the face-to-face groups, we used the fixed variable

(FV) method for eliciting the fractiles of the marginaldistributions, where the variable values are fixed andthe wheel setting is adjusted based on the responsesobtained (for more information on this approach, see

3 The first and third authors served as experimenters for theseelicitations.

Abbas, Budescu, and Gu: Assessing Joint Distributions with Isoprobability Contours1002 Management Science 56(6), pp. 997–1011, © 2010 INFORMS

Figure 4 An Example of a Binary Choice Used to Assess the MarginalDistribution of Height

You win $20 if

The wheel is spun and it landson orange

Choose Choose

The height of a randomlyselected UIUC male

undergraduate student is lessthan 5'10"

Indifferent

<5'10"

Spetzler and Staël von Holstein 1975, Abbas et al.2008). Subjects were presented with five weights orheights which they compared with various settings ofthe probability wheel until they reached (or were closeto) indifference between the two options. We elicitedbetween 4 and 12 points for each distribution based onthe experimenter’s judgment. The order of elicitationof the two variables was randomized.In the computerized group, we used the fixed

probability (FP) method for eliciting the fractiles ofthe marginal distributions (see Spetzler and Staëlvon Holstein 1975, Abbas et al. 2008). With thisapproach, the probability wheel setting is fixed andcorresponding values of the variables are elicited. Weused five fixed settings of the wheel (5%, 25%, 50%,75%, and 95%), which were compared with variousweights or heights until the judges reached (or wereclose to) indifference between the two options. Fig-ure 4 presents an example of a display for themarginal distribution elicitation task.Each point was based on a sequence of interrelated

choices. The subject’ choice at step t was used to deter-mine the pair shown on step �t+1�� which was alwaysselected to make the two options more similar (i.e.,closer to indifference). In the face-to-face group, weasked questions until subjects reached indifference.In the computerized group, a series was terminatedwhen the subject expressed indifference between thetwo options, or when the number of comparisonsexceeded six. Thus, in some cases we ended with arange of values, rather than a point. Overall, subjectsreached indifference in 54% of the judgments. At theend of each series, subjects were presented with a“prediction” of their preferences (based on their pre-vious choices), and were asked to confirm it beforemoving to the next series.

4.3. Eliciting the Isoprobability Contours of theJoint Distribution

To elicit isoprobability contours of the joint distri-bution the subjects (in both groups) were asked toconsider pairs of events related to the weight and the

Figure 5 An Example of a Binary Choice Used to Assess anIsoprobability Contour of Weight and Height

You win $20 if

Choose Choose

The height of a randomlyselected UIUC male

undergraduate student is lessthan 6'1" and his weight is less

than 148 Ibs

The height of a randomlyselected UIUC male

undergraduate student is lessthan 6'2" and his weight is less

than 112 Ibs

Indifferent

<6'1"

<148 Ibs <112 Ibs

and

<6'2"

and

height of randomly selected male students at UIUC. Ineach pair, one of the events was fixed but the descrip-tion of the other event changed. For example, the eventof choosing a UIUC student “whose height is less then6’1” and who weighs less then 148 lbs” was comparedwith the event of selecting a UIUC student “whoseheight is less then 6’2” and who weighs less then Xlbs,” where the weight varied from one stage to thenext (for example, X could be 112, 125, or 140 lbs,etc.). Subjects performed comparisons with variousweights (values of X) in the second lottery until theyreached (or were close to) indifference between thetwo options. Figure 5 presents an example of such atrial.We obtained two series of five points on the 50%

isoprobability contour—once by fixing the weight andanother by fixing the height. In the fixed height(weight) series, subjects performed comparison withvarious weights (heights) of the second lottery whilethe height (weight) of the second lottery was fixed. Forexample, in a fixed weight case, the decision makerwas offered a choice to bet on (a) the probability ofchoosing a UIUC student “whose height is less then5’9” and who weighs less then 150 lbs” and (b) thechance of selecting a UIUC student “whose weightis less then 160 lbs and whose heights is less then xinches,” and the height varied from one trial to thenext. For the fixed weight (height) series, the initialoption was a student with “any height (weight)” and“weight (height) less than the value corresponding toa marginal cumulative probability of 50% elicited bythe subject in the marginal section.”In the face-to-face group, we asked questions

until subjects reached indifference. In the computer-ized group, a series was terminated when the sub-ject expressed indifference between the two options,or when the number of comparisons exceeded six.Subjects reached indifference in 59% of the cases.

Abbas, Budescu, and Gu: Assessing Joint Distributions with Isoprobability ContoursManagement Science 56(6), pp. 997–1011, © 2010 INFORMS 1003

There were wide individual differences in this respect.For instance, 3 of the 15 subjects reached indifferenceon all series, and 3 other subjects never expressedindifference. In the face-to-face setting, subjects hadan extra pause (of about five minutes) between themarginal and each of the joint assessments.At the end of the experiment, one pair of gambles

was selected and was played according to the sub-ject’s preference to determine the payment.4 Half thesubjects won the lotteries. In the face-to-face group wealso asked the subjects to rate the difficulty of the esti-mations tasks and to provide a direct estimate of thecorrelation they perceive between height and weight.Participants took between 50 to 60 minutes to com-plete all stages of the study.

5. Experiment ResultsWe now present the results of both groups. In thosecases where we wish to highlight individual results,we focus on the face-to-face group where all sub-jects reached indifference on all series. Figure 6 showsthe joint assessments for the 10 subjects in the face-to-face group (all converted to centimeters and kilo-grams). For each subject, we plot all the points thatwere elicited using the two sequences (fixed weightsand fixed heights) jointly, but we use different symbolsto identify the two.

5.1. MonotonicityOur first analysis is concerned with the monotonicityof the assessments. We use Kendall’s �b as a measureof monotonicity. For any n assessments on a contour,let C be the number of points that are concordant (i.e.,an increase in weight corresponds to an increase inheight and the monotonicity condition of Property 2 isviolated), and let D be the number of pairs that are dis-cordant (i.e., the monotonicity condition of Property 2is satisfied). Kendall’s �b is the difference between theproportion of concordant pairs and the proportion ofdiscordant pairs. Formally,5

�b =∑n−1

i=1∑n

j=i+1sgn�heighti −heightj �sgn�weighti −weightj �(n2

)

= C−D(n2

) = C−D

C+D� (9)

where sgn is the sign function.

4 Once a pair of gambles was chosen and the subject’s responsewas identified, we compared it with the weight and height of a“randomly selected undergraduate student.” This undergraduatestudent was in fact the programmer for the experiment. The weightand height of this undergraduate student were fixed in advance inthe program to determine the payoff.5 In the presence of ties, the denominator of the formula is√

�C + D + Tx��C + D + Ty�, where Tx is the number of pairs with tieson X (but not on Y ), and Ty is the number of pairs with ties on Y(but not on X).

Table 1 Median Kendall’s �b Rank-Order Correlation for theVarious Distributions in the Two Groups

Marginal distributions Joint distributions

Sample Height WeightGroup size Height Weight fixed fixed

Computerized 15 1�00 1�00 −0�95 −0�95Face-to-face 10 0�99 0�90 −1�00 −1�00All 25 1�00 1�00 −1�00 −1�00

Kendall’s �b is a measure of rank-order correlationthat ranges from −1 (all pairs are discordant) to 1(all pairs are concordant), and it is 0 when there areequal numbers of concordant and discordant pairs.For comparison purposes, we also present mea-

sures of monotonicity for the two marginal distri-butions (in this case, perfect monotonicity implies�b = 1). Table 1 shows the median Kendall’s �b for eachdistribution for both groups. All of the (absolute) val-ues are very high, confirming the monotonicity of thejudgments for the marginal distributions and the iso-probability contours. They are highly similar in allcases, indicating that subjects were equally adept inmaking the univariate and the joint assessments.6

5.2. Estimating the Implied Probability Trade-OffsFor each two adjacent points, �xi� yi� and �xj� yj�, onthe assessed contour, we calculated the increments�yij = yj − yi and �xij = xj − xi, and then the ratio

�ij = −�yij/yi

�xij/xi

= − �height/height�weight/weight

(see Equation (6)). We then calculated for each sub-ject a robust measure of the spread of the �ij ratios—the median absolute deviation (MAD) around theirmedian value, MAD(�ij �, across the various points onthe assessed contour. A constant probability trade-offimplies MAD(�ij � = 0. In the face-to-face group, theindividual MADs range from 0.05 to 0.77, with amedian of 0.19. In the computerized group, we havehighly similar results—the 15 MADs range from 0.04to 0.73, with a median value of 0.24. These valuessuggest that for many subjects, a constant probabilitytrade-off provides a reasonable first-order approxima-tion of the contour function.

5.3. Internal ConsistencyRecall that we assessed the 50th percentile isoprob-ability contour in both directions: On one occasionwe started with the median value of the weight(obtained from the marginal distribution assessments)

6 In the face-to-face group, we also asked the subjects to rate howeasy it was to assess the marginal and the joint distributions tocompare the perceived difficulty of the two tasks. The split was 6/4(which is not significantly different from chance, by a sign test),indicating that, on average, the two tasks are equally easy.

Abbas, Budescu, and Gu: Assessing Joint Distributions with Isoprobability Contours1004 Management Science 56(6), pp. 997–1011, © 2010 INFORMS

Figure 6 50% Isoprobability Curves Assessed by the 10 Subjects in the Face-to-Face Group

170

180

190

200

210

70 90 110 130

Hei

ght (

cm)

Weight (kg)

Subject 1

Fixed height

Fixed weight

170

180

190

200

210

70 90 110 130

Hei

ght (

cm)

Weight (kg)

Subject 6

170

180

190

200

210

70 90 110 130

Hei

ght (

cm)

Weight (kg)

Subject 2

170

180

190

200

210

70 90 110 130

Hei

ght (

cm)

Weight (kg)

Subject 7

170

180

190

200

210

70 90 110 130

Hei

ght (

cm)

Weight (kg)

Subject 3

170

180

190

200

210

70 90 110 130

Hei

ght (

cm)

Weight (kg)

Subject 8

170

180

190

200

210

70 90 110 130

Hei

ght (

cm)

Weight (kg)

Subject 4

170

180

190

200

210

70 90 110 130

Hei

ght (

cm)

Weight (kg)

Subject 9

170

180

190

200

210

70 90 110 130

Hei

ght (

cm)

Weight (kg)

Subject 5

170

180

190

200

210

70 90 110 130

Hei

ght (

cm)

Weight (kg)

Subject 10

with no constraints on the height, and then offereddeals with increased weights and reduced heights.We repeated the assessment starting with the medianheight (no constraints on the weight), and then offereddeals with increasing heights and decreasing weights.Consistency with Property 4 requires that the assessed

contours connect the median values of each variable,i.e., the median height assessed from the marginal dis-tribution, for example, should equal the value of theheight intersecting the 50% contour assessment withthe vertical axis at the upper bound of the weightvariable (Figure 2).

Abbas, Budescu, and Gu: Assessing Joint Distributions with Isoprobability ContoursManagement Science 56(6), pp. 997–1011, © 2010 INFORMS 1005

Figure 7 Assessed and Inferred Median Heights and Weights for all the Subjects in the Face-to-Face Group

Height comparison

107

9 6 83

1

4

5

2

170

175

180

185

170 175 180 185

H_ joint (cm)

H_

med

(cm

)

Weight comparison

6

7

510

3 9

4 18

2

70

75

80

85

90

95

70 75 80 85 90 95

W_ joint (kg)

W_

med

(kg

)

Figure 7 displays these consistency results forthe 10 subjects in the face-to-face group. For eachvariable, we plot the median of the variable obtainedfrom the assessment of its marginal distribution andthe median of the same variable as inferred from thecontour assessment. Perfect consistency requires thatall points would lie on the identity line. In addi-tion to the identity line, the plots include narrowbands of ±5 units (centimeters for height and kilo-grams for weight),7 and each subject is identified byhis/her number. Note that most points are, indeed,within this narrow band and close to the ideal withonly two exceptions (weight judgments of Subject 1and height judgments for Subject 10). We concludethat the observed consistency between the assessedand inferred medians is impressively high for bothvariables.

6. Some Functional Forms ofIsoprobability Contours

The experimental results indicate that people can rea-son about the isoprobability contours. To simplifytheir assessment, one could assume a particular func-tional form for the contours and assess its parameters.Of course, the choice of the functional form dependson the qualitative properties of the contours providedby the DM. For example, suppose that for a given DMwe assume that the percentage increase in one vari-able needed to compensate for a percentage decreasein the other to achieve the same cumulative proba-bility is constant. This implies that the value of �ij isconstant (��, and the DM’s subjective contours can be

7 We selected this ±5 margin to reflect the natural rounding ten-dencies of most people in reporting heights and weights.

described by constant elasticity of substitution func-tions of the form

C�x�y� = �x� + �y��1/�� 0< a < 1�

where C�x�y� is a contour function, special cases ofwhich are the Cobb-Douglas-type contour functions8

C�x�y� = �x − xmin��y − ymin��� (10)

Figure 8 shows, as an example, the isoprobability con-tours implied by the Cobb-Douglas contour functionsC�x�y� = xy� for � = 0�8 in the left panel, and � =2�4 in the right panel. The contours in both panelsconnect the marginal probability values of each vari-able starting at �xmax�y� and ending at �x�ymax�. Themarginal distribution of Y is the same for both casesbut, as the value of � increases, the contours end atpoints �x�ymax� with higher values of the variable X.This models a joint distribution whose marginal prob-ability mass for variable X is concentrated at highervalues.In many cases, the assumption that � is constant

can provide a good first-order approximation, and itis convenient, because it determines the functionalform of the contours in terms of a single parameter.Re-arranging terms in (10) gives

log�x − xmin� = logC�x�y� − � log�y − ymin�� (11)

This is easily recognized as a linear regression equa-tion, and the parameters can be estimated by ordinaryleast squares (OLS) regression. In particular, the slopeestimates the (negative) probability trade-off coeffi-cient. We illustrate this approach for one subject (Sub-ject 4 in the face-to-face group).

8 Cobb-Douglas-type functions are a special case of the constantelasticity of substitution family when we take the limit when � → 0and using L’hopital’s rule.

Abbas, Budescu, and Gu: Assessing Joint Distributions with Isoprobability Contours1006 Management Science 56(6), pp. 997–1011, © 2010 INFORMS

Figure 8 Isoprobability Contours for Cobb-Douglas-Type Functions with � = 0�8 and � = 2�4

0.8–1.00.6–0.80.4–0.60.2–0.40–0.2

0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1.000.050.100.150.200.250.300.350.400.450.500.550.600.650.700.750.800.850.900.951.00

Variable x

Isoprobable contours

0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1.00

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1.0

Variable x

Var

iabl

ey

Var

iabl

ey

Isoprobable contours

To illustrate this approach, we present in Figure 9the elicited points of one of our subjects and the fittedfunction on a log-log scale, as well as the parametersestimated by OLS. The figure shows the estimatedprobability trade-off coefficient obtained by fixing theweight, fixing the height, and by using both assess-ment methods for the estimation. The linear functionfits the data points well in all three panels (whichyield highly similar parameters), suggesting a con-stant probability trade-off, and supports the choice ofthe one-parameter function used in this case.Once the value of � is estimated, the contour

function is fully determined and the joint cumula-tive probability at any point �x�y� can be obtainedby determining the boundary value using eithermarginal distribution:(i) Using the marginal distribution of X, calcu-

late the value of x1 that corresponds to �x − xmin��y−ymin�

� = �x1−xmin��ymax−ymin�� and determine the

cumulative probability of x1 using its marginal distri-bution, Fx�x1�.(ii) Using to the marginal distribution of Y , cal-

culate the value of y1 that corresponds to �x − xmin��y−ymin�

� = �xmax−xmin��y1−ymin�� and determine the

cumulative probability of y1 using its marginal distri-bution, Fy�y1�.

6.1. Numerical ExampleConsider one point (x1 = 88�45 kg, y1 = 177�8 cm) onthe assessed contour of Subject 4 (see Figure 9), wherexmax = 91�72 kg and ymax = 190�5 cm were establishedfrom the initial boundary values supplied. We esti-mated � = 1�38. The boundary values of weight andheight that correspond to this contour are obtained bysolving

�x1 − xmin��y1 − ymin� = �x2 − xmin��ymax − ymin��

= �xmax − xmin��y2 − ymin���

We get x2 = 68�42 kg, y2 = 177�57 cm. The joint cumu-lative probability of (88.45 kg, 177.8 cm) is equal to themarginal cumulative probability of weight≤ 68�42 kg,and the marginal cumulative probability of height ≤177�57 cm. From the assessed marginal distributionswe estimate Fx�68�42� = 0�43 and Fy�177�57� = 0�45,which are very similar.

6.2. Sensitivity AnalysisBecause we need only one marginal distribution anda contour function to calculate the joint probability,consistency requires that calculations obtained usingeither marginal distribution should provide the sameresult. Figure 10 plots the joint probability of thepoint (x = 88�45 kg, y = 177�8 cm) obtained usingboth marginal distributions (as we did in the previousexample) for different values of �. The figure showsseveral interesting results. First, note that when theestimate of � is between 1.38 and 1.66 (the valuesestimated from the two sets in Figure 9), there is lit-tle change in the joint cumulative probability usingeither marginal distribution. In other words, the twoprobability trade-offs �� = 1�38�� = 1�66� yield highlysimilar joint probabilities. Perfectly consistent assess-ments (for both marginal distributions and isoproba-bility contours) occur at � = 1�32 (where the curvescross).If we change the value of � while keeping the

marginal distributions fixed, however, we get largerdiscrepancies. The explanation is that for any givenvalue of � and one fixed marginal distribution, say ofX, we completely characterize the joint distribution,and we are determining the marginal distribution forvariable Y (refer back to Figure 8, which shows howthe contours connect different values of variable Y aswe vary ��.

Abbas, Budescu, and Gu: Assessing Joint Distributions with Isoprobability ContoursManagement Science 56(6), pp. 997–1011, © 2010 INFORMS 1007

Figure 9 Estimating the Parameter � for the Assumed ContourFunction

Subject 4 fixed height (face to face)

5.05

5.10

5.15

5.20

5.25

5.30

5.35

4.40 4.45 4.50 4.55 4.60 4.65 4.70 4.75 4.80

Log weight

Log

heig

ht

Experiment points

Subject 4 fixed weight (face to face)

5.05

5.10

5.15

5.20

5.25

5.30

5.35

5.40

4.2 4.3 4.4 4.5 4.6 4.7 4.8

Log weight

Log

heig

ht

Subject 4 both sets (face to face)

5.05

5.10

5.15

5.20

5.25

5.30

5.35

5.40

4.30 4.35 4.40 4.45 4.50 4.55 4.60 4.65 4.70 4.75

Log weight

Log

heig

ht

� = 1.38

� = 1.66

� = 1.38

7. Inferring Dependence from theAssessed Contours

Once the joint distribution is specified, we candetermine any measure of association between thevariables or any concordance probability by directcalculation. Isoprobability contours can also be usedto infer dependence indirectly by curve fitting anappropriate joint probability model. For example, ifthe assessed curves fit a bivariate Gaussian distri-bution (or bivariate Gaussian copula), the estimatedparameters of the curve fit will in fact be the correla-tion coefficients. Figure 11 shows the 50% isoprobabil-ity contours for several bivariate standard (i.e., �x =�y = 0 and �x = �y = 1) Gaussian distributions. Theseven contours correspond to distributions with dif-ferent correlation coefficients, �xy . The purpose of thissection is to show how to determine the correlationscoefficients between the variables if the assessed iso-probability contours match those of a bivariate nor-

Figure 10 Joint Probability with Two Marginal Distributions forDifferent Values of �

Join

t cu

mul

ativ

e pr

obab

ility

0

0.1

0.2

0.3

0.4

0.5

1 1.3 1.6 1.9 2.2 2.5 2.8 3.1 3.4 3.7 40

Using the marginaldistribution of Y

Using the marginaldistribution of X

�

Figure 11 Selected 50% Contours for Standard Bivariate GaussianDistributions as a Function of �

C = 0.5

–0.5

0.5

1.0

1.5

2.0

2.5

3.0

3.5

–0.5 0.5 1.0 1.5 2.0 2.5 3.0

X

Y

� = 0.9� = 0.6� = 0.3� = 0� = –0.3� = –0.6� = –0.9

00

mal copula. Similar procedures can be used with otherfamilies of joint distributions if the assessed contoursdo not match those of the Gaussian distribution.

7.1. Estimation of Correlation Coefficient byCurve Fitting to a Bivariate Gaussian Copula

To infer the correlation coefficient implied by theassessed isoprobability contour we curve fitted thedata to a bivariate Gaussian copula. The curve-fittingprocedure is as follows:(1) Determine the marginal distributions by fitting

the elicited marginal fractiles into functional forms ofcumulative distributions for weight and height, Fw�w�and Fh�h�, respectively. We fitted beta distributions tothe marginal fractiles because they allow a lot of flexi-bility in matching a variety of fractile assessments forbounded distributions. However, one could also useother distributions depending on their goodness of thefit.(2) Using the fitted marginal distributions, trans-

form the elicited weight and height points �w�h� on theassessed contour into corresponding points on stan-dardized normal distributions �wN �hN � by

�wN �hN � = ��−1�Fw�w����−1�Fh�h����

where � is the standard normal distribution.

Abbas, Budescu, and Gu: Assessing Joint Distributions with Isoprobability Contours1008 Management Science 56(6), pp. 997–1011, © 2010 INFORMS

Table 2 Estimated Correlations Between Height and Weight and theGoodness of Fit (RMSE) of the Estimates

Subject RMSE � ��s = 6/�� sin−1��/2� �b = �2/�� sin−1

���

(a) Estimates from face-to-face group1 0�17 −0�48 −0�46 −0�322 0�10 0�80 0�79 0�593 0�06 0�39 0�37 0�254 0�03 0�92 0�91 0�745 0�07 0�89 0�88 0�706 0�21 0�89 0�88 0�707 0�13 0�69 0�67 0�488 0�08 0�22 0�21 0�149 0�06 0�88 0�87 0�6810 0�07 0�99 0�99 0�91Median 0�08 0�84 0�83 0�64

(b) Estimates from computerized group1 0�74 0�99 0�99 0�912 0�28 0�83 0�82 0�623 0�31 −0�74 −0�72 −0�534 1�71 0�99 0�99 0�915 0�46 0�99 0�99 0�916 0�24 0�99 0�99 0�917 1�02 0�73 0�71 0�528 0�33 0�80 0�79 0�599 0�09 0�58 0�56 0�3910 0�16 0�51 0�49 0�3411 0�07 −0�17 −0�16 −0�1112 0�04 0�70 0�68 0�4913 0�12 0�36 0�35 0�2314 0�04 −0�05 −0�05 −0�0315 0�35 0�60 0�58 0�41Median 0�28 0�70 0�68 0�49

(3) Determine the correlation between weight andheight by curve fitting the transformed points �wN �hN �on the isoprobability contour to a bivariate Gaussiandistribution. We searched for the value of � that min-imizes the root mean square error (RMSE) betweenthe standardized (transformed) points on the empir-ical 0.5 contour and the 0.5 contour of the bivariateGaussian distributions (see examples in Figure 11). Wesearched over the range −0�99 to 0.99 in incrementsof 0.01.Table 2 shows the estimated correlation coefficients

and the RMSEs for all the subjects in both groups.The number of assessed points underlying this esti-mation was 10 for all the subjects in the computerizedgroup. The number of points assessed in the face-to-face group varied from one person to another becauseit was determined by the responses obtained duringthe elicitation and the comfort level (of the subjectsand the experimenter) with the quality of data. Formost subjects we elicited 8 or 9 points (median, 9).Subject 10 required 10 points, and Subject 8 requiredonly 7 points.With a few noticeable exceptions (see Subjects 1, 4,

and 7 in the computerized group), the RMSEs are verylow, indicating good fit, especially in the face-to-face

group, where the fit is significantly better then in thecomputerized group (t�23� = 2�06; p < 0�05). However,if the three outliers are eliminated, the goodness of fitin the two groups is comparable.The median correlation in the face-to-face group

was slightly higher than in the computerized experi-ment (0.84 versus 0.70), but the two distributions arenot significantly different (t�23� = −0�39; p > 0�05).Given that we are focusing on the copula, we alsotransformed these correlations to rank-order correla-tions that do not depend on the marginal distributions.Kruskal (1958) shows that in a bivariate normal distri-bution, the Spearman �s and Kendall’s �b are relatedto Pearson’s �:

�s = 6�

sin−1(

�

2

)� (12)

�b = 2�

sin−1���� (13)

The results (especially Spearman’s �s� are very simi-lar. Although the correlations in the face-to-face groupappear to be high (Table 2), recall that we also askedthe subjects in this group to estimate the correlationat the end of the experiment. Eight of the subjectsdid (two of them did not provide numbers but sim-ply said that it is positive, which, incidentally, rein-forces our point that such correlation judgments areoften difficult to provide directly). Among the eightfor whom we have direct estimates, the fitted estimatewas higher in five cases than the direct estimate, butit was lower for the other three. The medians of theeight correlations were, essentially, identical (mediancorrelation of 0.75 using the isoprobability contour-based approach and 0.73 for the direct estimates).Thus, although it is possible that “objectively” thesesubjects overestimate the weight–height correlation inthe student population; this does not seem to be asso-ciated with a particular estimation method.Note that in four cases (Subject 1 in the face-to-face

group and Subjects 3, 11, and 14 in the computerizedgroup), the estimation yielded negative estimates ofthe correlation coefficient. This counterintuitive resultdoes not reflect an estimation problem. Rather, it canbe attributed to the inappropriateness of the bivariateGaussian assumption for the points assessed for thesesubjects, as shown in Figure 12. The left panel showsthe assessed points for Subject 1 (estimate � = −0�48)and the contours of the various bivariate Gaussiandistributions. Clearly, the assessed points are incon-sistent with the model. In sharp contrast, the rightpanel shows one of the best-fitting subjects (Subject 4,estimate � = 0�92), for which many of the points lieexactly on the 0.9 contour.

Abbas, Budescu, and Gu: Assessing Joint Distributions with Isoprobability ContoursManagement Science 56(6), pp. 997–1011, © 2010 INFORMS 1009

Figure 12 Isoprobability Contours of Bivariate Gaussian Distributions with Data from Two Subjects

CDF = 0.5 bivariate Gaussian isoprobability contourswith Subject 1 experiment data

Y Y

X X

0.0

1.0

2.0

3.0

4.0

0.0 1.0 2.0 3.0 4.0

� = 0.6

� = 0.9

� = 0.3

� = 0

� = –0.3

� = –0.6

� = –0.9

Subject 1

CDF = 0.5 bivariate Gaussian isoprobability contourswith Subject 4 experiment data

0.0

1.0

2.0

3.0

4.0

0.0 1.0 2.0 3.0 4.0

� = 0.6

� = 0.9

� = 0.3

� = 0

� = –0.3

� = –0.6

� = –0.9

Subject 4

7.2. Numerical Example: Estimating theJoint Distribution Using a BivariateGaussian Copula

Having determined that the isoprobability contour ofSubject 4 in the face-to-face group matches a bivariateGaussian (see Figures 11 and 12), we can estimate thejoint cumulative distribution by curve fitting it to abivariate Gaussian copula. The marginal distributionassessment of weight indicates that Fx�88�45� = 0�77.Under the assumption of standard Gaussian vari-able, this implies X = 0�74. The marginal distributionassessment of height indicates that Fy�177�8� = 0�46.This corresponds to Y = −0�10 in a standard Gaussiandistribution. The estimated correlation coefficient was0.92 (see Table 2). The joint cumulative probability ofa standard bivariate Gaussian distribution where X =0�74, Y = −0�10, and � = 0�92 is 0.46, which is remark-ably close to the previous values obtained by curvefitting to the contour function.

8. ConclusionsWe presented a method for direct elicitation and con-struction of joint probability distributions of continu-ous variables using isoprobability contours. Our keycontribution is the development and empirical vali-dation of a practical, easy to implement assessmentmethod. In addition, we described two methods forconstructing the joint probability distribution from theassessed contours and illustrated how to determinebivariate correlations from these values.The main advantage of the newly proposed

approach compared to other approaches such as mar-ginal conditional assessments or direct assessment ofdependence is that the judges do not have to provideany numerical responses, or estimate complex jointprobabilities (e.g., the quadrant probability) or param-eters (e.g., the covariance or correlation between vari-ables) that are not intuitive, prone to judgment errors,

and could be intimidating for judges who lack properstatistical training. In our approach, judges only needto express preferences over simple binary gambleswith identical payoffs whose probabilities depend on,and reflect, the probability trade-offs between the twovariables. A second major advantage of our proce-dure is that the events being compared involve bothvariables, so the judges are forced to consider therelevant trade-offs. Thus, the use of isoprobabilitycontours reduces the assessment of joint probabilitydistributions to a one-dimensional cumulative prob-ability assessment (which is cognitively simpler toreason about than conditional assessments) and theelicitation of the probability trade-offs (inferred byexpressing preferences over binary deals).We reported results of a simple feasibility study. The

key empirical question we addressed is whether sub-jects are able to reason about probability trade-offs.The results of the empirical study showed that this is asensible and reasonable task that was performed suc-cessfully. The judgments regarding the joint distribu-tions matched the quality of the standard probabilityelicitation (FP and FV) for the marginal distributionswith respect to difficulty and monotonicity. The mono-tonicity level was excellent, and the level of internalconsistency was satisfactory. Finally, most correlationcoefficients estimated from the judges’ assessmentswere reasonable, and the few outlying results are dueto the fact that the judgments of some subjects do notfit the assumptions of the model (in this case, bivari-ate Gaussian), rather than elicitation or statistical esti-mation problems. In these types of situations, othermodels of joint probability distributions may providebetter fits.All these results suggest that people can indeed rea-

son about the newly defined probability trade-off coef-ficients in assessing the isoprobability contours. It is

Abbas, Budescu, and Gu: Assessing Joint Distributions with Isoprobability Contours1010 Management Science 56(6), pp. 997–1011, © 2010 INFORMS

important to stress that these results appear to be quiterobust. Recall that we obtained comparable resultsin the computerized group, where about 40% of theseries ended before converging to indifference, and weapproximated the location of the point on the contourby the middle of the range of values. The one excep-tion to this generalization is that for a minority (3 outof 15) of the subjects in the computerized groups, thefit of the inferred correlations was unsatisfactory.One of the nice features of the new method is that,

unlike the standard methods of assessing dependence,it generates data that can be used in various ways toestimate the target quantities. We discussed and illus-trated two ways to estimate the joint probability dis-tribution and infer the bivariate correlation from thejudges’ assessments. This flexibility allows us to dou-ble check and validate the quality of our procedure.Both methods are easily implemented and generatedsimilar results in our simple example, but in our opin-ion it is easier to fit contour functions than curve fitbivariate copulas. The former approach requires onlythe estimation of one probability trade-off and one ofthe marginal distributions. However, it is also impor-tant, as a consistency check, to test that the joint prob-ability density obtained by the contours method isnonnegative, as we do when constructing joint cumu-lative distributions using cumulative marginal condi-tional assessments.The notion of an isoprobability contour leads nat-

urally to several directions for future research. Thefirst is the derivation of different functional forms ofcontour functions that allow for variable probabilitytrade-offs. For example, all the contour functions wepresented and used in our examples involve constantelasticity of probability trade-offs. Other functionalforms relaxing this condition, the simplest being a lin-ear function of the variables, i.e., ��x�y� = �0 + �1x +�2y, which generalizes the forms used in this paper,should be considered.The second direction pertains to extensions into

higher dimensions. The basic principle underlyingour approach can be extended beyond the bivari-ate case, but the complexity of the problem is con-siderably higher and increases rapidly (as does thecomplexity of conditioning on a larger number ofvariables for conditional probability assessments). Forexample, in the trivariate case, one could estimate rel-atively easily the probability trade-offs between eachpair of variables at fixed values of the third vari-able. This would lead to conditional probabilities ofthe form F �xi� xj � xk� for all distinct permutations ofi, j , k, and works perfectly for constructing the jointdistribution when the dependence between the vari-ables is of the second order, i.e., when this condi-tional probability does not depend on xk. Cases withmore complex relationships would require estimation

of more complicated three-way probability trade-offs(or, under certain parametric assumptions, repeatingthe assessment for different values of the variable Xk�,which may be harder to implement and judge. Onecan also assume a certain functional form of thecontour function that extends to higher dimensionsand then use the probability trade-offs to estimate itsparameters directly. For example, higher-order func-tions of the form C�x�y�z� = xy�z� can also be consid-ered using pairwise trade-offs and their parametersassessed at fixed values of the third variable.Finally, there are some important empirical direc-

tions for future research. The first one is to go beyondthe feasibility demonstration of this paper and pro-vide evidence-based guidelines for implementing thenew method in practice. This requires answeringquestions such as the following: How many contoursshould be assessed? Does it matter which contours areassessed? What is the minimal number of points thatshould be assessed for each contour, and is there anoptimal way of choosing them? It is equally importantto compare empirically results based on assessmentsof isoprobability contours with other direct elicitationmethods of dependence. It would be useful to deter-mine which method (or methods) are perceived to bemore natural, easier to implement, and more comfort-able for (and hence preferred by) the DMs, and verifywhich method (or methods) generates better (morestable, reliable, accurate, valid, etc.) results in practice.

AcknowledgmentsThis material is based upon work supported by the NationalScience Foundation under Award SES 06-20008. The authorsthank Sean Jordan for programming the study.

ReferencesAbbas, A. E. 2006. Entropy methods for joint distributions in deci-

sion analysis. IEEE Trans. Engrg. Management 53(1) 146–159Abbas, A. E., D. V. Budescu, H.-T. Yu, R. Haggerty. 2008. A compar-

ison of two probability encoding methods: Fixed probabilityvs. fixed variable values. Decision Anal. 5(4) 190–202.

Allan, L. G., H. M. Jenkins. 1980. The judgment of contingencyand the nature of the response alternatives. Canadian J. Psych.34(1) 1–11.

Alloy, L. B., N. Tabachnik. 1984. Assessment of covariation byhumans and animals: The joint influence of prior expectationsand current situational information. Psych. Rev. 91(1) 112–149.

Arkes, H. R., A. R. Harkness. 1983. Estimates of contingencybetween two dichotomous variables. J. Experiment. Psych. Gen-eral 112(1) 117–135.

Beyth-Marom, R. 1982. Perception of correlation reexamined. Mem-ory Cognition 10(6) 511–519.

Chapman, L. J. 1967. Illusory correlation in observational report.J. Verbal Learn. Verbal Behav. 6(5) 151–156.

Clemen, R. T., T. Reilly. 1999. Correlations and copulas for decisionand risk analysis. Management Sci. 45(2) 208–224.

Clemen, R. T., G. W. Fischer, R. L. Winkler. 2000. Assessingdependence: Some experimental results. Management Sci. 46(8)1100–1115.

Abbas, Budescu, and Gu: Assessing Joint Distributions with Isoprobability ContoursManagement Science 56(6), pp. 997–1011, © 2010 INFORMS 1011

Cooke, R. M. 1991. Experts in Uncertainty: Opinion and SubjectiveProbability in Science. Oxford University Press, Oxford, UK.

Gokhale, D. V., S. J. Press. 1982. Assessment of a prior distributionfor the correlation coefficient in a bivariate normal distribution.J. Roy. Statist. Soc., Ser. A 145 237–249.

Hamilton, D. L., R. K. Guifford. 1976. Illusory correlations in inter-personal perception: A cognitive bias of stereotypic judgments.J. Experiment. Soc. Psych. 12(4) 392–407.

Hogarth, R. M. 1987. Judgement and Choice, 2nd ed. Wiley, Chich-ester, UK.

Howard, R. A. 1989. Knowledge maps. Management Sci. 35(8)903–922.

Kadane, J. B., L. J. Wolfson. 1998. Experiences in elicitation (withdiscussion). Statistician 47(1) 1–20.

Kahneman, D., A. Tversky. 1973. On the psychology of prediction.Psych. Rev. 80(4) 237–251.

Keefer, D. 2004. The underlying event model for approximatingprobabilistic dependence among binary events. IEEE Trans.Engrg. Management 51(2) 173–182.

Keeney, R. L., H. Raiffa. 1976. Decisions With Multiple Objectives:Preferences and Value Tradeoffs. John Wiley & Sons, New York.

Kruskal, W. H. 1958. Ordinal measures of association. J. Amer.Statist. Assoc. 53(284) 814–861.

MacCrimmon, K. R., M. Toda. 1969. The experimental deter-mination of indifference curves. Rev. Econom. Stud. 36(4)433–451.

Matheson, J., A. E. Abbas. 2005. Utility transversality: A value-based approach. J. Multicriteria Decision Anal. 13(5–6) 229–238.

Meyer, J., D. Shinar. 1992. Estimating correlations from scatterplots.Human Factors 34(3) 335–349.

Meyer, J., M. Taieb, I. Flasher. 1997. Correlation estimation as per-ceptual judgments. J. Experiment. Psych. Appl. 3(1) 3–20.

Morgan, M. G., M. Henrion. 1990. Uncertainty: A guide to Deal-ing with Uncertainty in Quantitative Risk and Policy Analysis.Cambridge University Press, Cambridge, UK.

O’Hagan, A., C. E. Buck, A. Daneshkhah, J. R. Eiser, P. H.Garthwaite, D. J. Jenkinson, J. E. Oakley, T. Rakow. 2006. Uncer-tain Judgments: Eliciting Experts’ Probabilities. John Wiley &Sons, West Sussex, UK.

Spetzler, C. S., C.-A. S. Staël von Holstein. 1975. Probability encod-ing in decision analysis. Management Sci. 22(3) 340–358.

Trolier, T. K., D. L. Hamilton. 1986. Variables influencing judgmentsof correlational relations. J. Personality Soc. Psych. 50(5) 879–888.

Tversky, A., D. Kahneman. 1974. Judgment under uncertainty:Heuristics and biases. Science 185(4157) 1124–1131.

von Winterfeldt, D., W. Edwards. 1986. Decision Analysis and Behav-ioral Research. Cambridge University Press, New York.

Wallsten, T. S., D. V. Budescu. 1983. Encoding subjective probabil-ities: A psychological and psychometric review. ManagementSci. 29(2) 151–173.

Yi, W., V. M. Bier. 1998. An application of copulas to accident pre-cursor analysis. Management Sci. 44(12, part 2 of 2) S257–S270.