Assessing Joint Distributions with Isoprobability...
Embed Size (px)
Transcript of Assessing Joint Distributions with Isoprobability...

Seediscussions,stats,andauthorprofilesforthispublicationat:http://www.researchgate.net/publication/220535093
AssessingJointDistributionswithIsoprobabilityContours.
ARTICLE·JANUARY2010
DOI:10.1287/mnsc.1100.1161·Source:DBLP
CITATIONS
4
DOWNLOADS
296
VIEWS
172
3AUTHORS,INCLUDING:
DavidV.Budescu
FordhamUniversity
170PUBLICATIONS4,268CITATIONS
SEEPROFILE
YuhongGu
UniversityofIllinois,UrbanaChampaign
1PUBLICATION4CITATIONS
SEEPROFILE
Availablefrom:DavidV.Budescu
Retrievedon:18June2015
http://www.researchgate.net/publication/220535093_Assessing_Joint_Distributions_with_Isoprobability_Contours?enrichId=rgreqe275465c259f43628b9b93869724848b&enrichSource=Y292ZXJQYWdlOzIyMDUzNTA5MztBUzoxMDI1OTMzNzYyOTI4NjhAMTQwMTQ3MTU3NTUyNQ%3D%3D&el=1_x_2http://www.researchgate.net/publication/220535093_Assessing_Joint_Distributions_with_Isoprobability_Contours?enrichId=rgreqe275465c259f43628b9b93869724848b&enrichSource=Y292ZXJQYWdlOzIyMDUzNTA5MztBUzoxMDI1OTMzNzYyOTI4NjhAMTQwMTQ3MTU3NTUyNQ%3D%3D&el=1_x_3http://www.researchgate.net/?enrichId=rgreqe275465c259f43628b9b93869724848b&enrichSource=Y292ZXJQYWdlOzIyMDUzNTA5MztBUzoxMDI1OTMzNzYyOTI4NjhAMTQwMTQ3MTU3NTUyNQ%3D%3D&el=1_x_1http://www.researchgate.net/profile/David_Budescu?enrichId=rgreqe275465c259f43628b9b93869724848b&enrichSource=Y292ZXJQYWdlOzIyMDUzNTA5MztBUzoxMDI1OTMzNzYyOTI4NjhAMTQwMTQ3MTU3NTUyNQ%3D%3D&el=1_x_4http://www.researchgate.net/profile/David_Budescu?enrichId=rgreqe275465c259f43628b9b93869724848b&enrichSource=Y292ZXJQYWdlOzIyMDUzNTA5MztBUzoxMDI1OTMzNzYyOTI4NjhAMTQwMTQ3MTU3NTUyNQ%3D%3D&el=1_x_5http://www.researchgate.net/institution/Fordham_University?enrichId=rgreqe275465c259f43628b9b93869724848b&enrichSource=Y292ZXJQYWdlOzIyMDUzNTA5MztBUzoxMDI1OTMzNzYyOTI4NjhAMTQwMTQ3MTU3NTUyNQ%3D%3D&el=1_x_6http://www.researchgate.net/profile/David_Budescu?enrichId=rgreqe275465c259f43628b9b93869724848b&enrichSource=Y292ZXJQYWdlOzIyMDUzNTA5MztBUzoxMDI1OTMzNzYyOTI4NjhAMTQwMTQ3MTU3NTUyNQ%3D%3D&el=1_x_7http://www.researchgate.net/profile/Yuhong_Gu?enrichId=rgreqe275465c259f43628b9b93869724848b&enrichSource=Y292ZXJQYWdlOzIyMDUzNTA5MztBUzoxMDI1OTMzNzYyOTI4NjhAMTQwMTQ3MTU3NTUyNQ%3D%3D&el=1_x_4http://www.researchgate.net/profile/Yuhong_Gu?enrichId=rgreqe275465c259f43628b9b93869724848b&enrichSource=Y292ZXJQYWdlOzIyMDUzNTA5MztBUzoxMDI1OTMzNzYyOTI4NjhAMTQwMTQ3MTU3NTUyNQ%3D%3D&el=1_x_5http://www.researchgate.net/institution/University_of_Illinois_UrbanaChampaign?enrichId=rgreqe275465c259f43628b9b93869724848b&enrichSource=Y292ZXJQYWdlOzIyMDUzNTA5MztBUzoxMDI1OTMzNzYyOTI4NjhAMTQwMTQ3MTU3NTUyNQ%3D%3D&el=1_x_6http://www.researchgate.net/profile/Yuhong_Gu?enrichId=rgreqe275465c259f43628b9b93869724848b&enrichSource=Y292ZXJQYWdlOzIyMDUzNTA5MztBUzoxMDI1OTMzNzYyOTI4NjhAMTQwMTQ3MTU3NTUyNQ%3D%3D&el=1_x_7

MANAGEMENT SCIENCEVol. 56, No. 6, June 2010, pp. 997–1011issn 00251909 �eissn 15265501 �10 �5606 �0997
informs ®doi 10.1287/mnsc.1100.1161
©2010 INFORMS
Assessing Joint Distributions withIsoprobability Contours
Ali E. AbbasDepartment of Industrial and Enterprise Systems Engineering, College of Engineering,University of Illinois at Urbana–Champaign, Urbana, Illinois 61801, [email protected]
David V. BudescuDepartment of Psychology, Fordham University, Bronx, New York 10458; and Department of Psychology,
University of Illinois at Urbana–Champaign, Urbana, Illinois 61801, [email protected]
Yuhong (Rola) GuDepartment of Psychology, University of Illinois at Urbana–Champaign, Urbana, Illinois 61801,
We present a new method for constructing joint probability distributions of continuous random variablesusing isoprobability contours—sets of points with the same joint cumulative probability. This approachreduces the joint probability assessment into a onedimensional cumulative probability assessment using asequence of binary choices between various combinations of the variables of interest. The approach eliminatesthe need to assess directly the dependence, or association, between the variables. We discuss properties ofisoprobability contours and methods for their assessment in practice. We also report results of a study in whichsubjects assessed the 50th percentile isoprobability contour of the joint distribution of weight and height. Weuse the data to show how to use the assessed contours to construct the joint distribution and to infer (indirectly)the dependence between the variables.
Key words : isoprobability contours; joint probability elicitation; probability encoding; correlation; dependenceHistory : Received November 19, 2008; accepted December 29, 2009, by George Wu, decision analysis.
Published online in Articles in Advance April 23, 2010.
1. IntroductionThe construction of representative probability distributions from a decision maker (DM) is a fundamentalstep in decision analysis and has engendered a substantial literature. It is well known that the probabilityelicitation process for simple events or continuous random variables can be subject to many cognitive andmotivational biases (see, for example, Hogarth 1987,Kahneman and Tversky 1973, Spetzler and Staël vonHolstein 1975, Tversky and Kahneman 1974, Wallstenand Budescu 1983). When eliciting a joint probabilitydistribution, the magnitude of this task is much largerbecause we are faced with the need to incorporatethe dependence among the random variables into theanalysis. Several authors have discussed these issuesand have presented methods to facilitate the elicitation process (see O’Hagan et al. 2006 for a review).For example, Howard (1989) proposed evocative andredundant knowledge maps to graphically representthe dependence relations and provided examples ofhow they simplify the construction of joint probability distributions. Other methods approximate the joint
distribution in terms of lowerorder joint probability assessments. For example, Keefer (2004) presenteda binary event model for approximating dependencerelations between the variables, and Abbas (2006)presented a maximum entropy approach for constructing a discrete joint probability distribution usinglowerorder joint probability assessments. Some methods for constructing joint probability distributions ofcontinuous variables require information about thecrossmoments or the correlation coefficients betweenthe variables. For example, Cooke (1991) proposeda continuous maximum entropy joint distributionusing moments and correlation coefficients, Yi andBier (1998) constructed joint distributions using copula structures and certain dependence parameters forrisk analysis, and Clemen and Reilly (1999) used multivariate normal copulas to construct a joint distribution using pairwise correlation coefficients.Several “direct” methods for assessing dependence
between variables have also been proposed. For example, Gokhale and Press (1982) elicited concordanceprobabilities from subjects. Under the assumption of
997

Abbas, Budescu, and Gu: Assessing Joint Distributions with Isoprobability Contours998 Management Science 56(6), pp. 997–1011, © 2010 INFORMS
bivariate normality, these probabilities can be relatedto the Pearson correlation coefficient.1 Clemen et al.(2000) conducted a study comparing various methodsof eliciting the correlation between pairs of variables.Under the assumption of bivariate normality, it is possible to use these estimates to infer the joint probability distribution of the two variables. They found thatdirect assessments of correlation coefficients on a scaleof −1 to 1 outperformed several other methods. Thisresult contradicts previous studies (such as Morganand Henrion 1990, Kadane and Wolfson 1998).The judgment literature shows that people are noto
riously poor estimators of association and correlationbetween variables. For example, Meyer and Shinar(1992) and Meyer et al. (1997) studied the inferenceof correlations based on visual displays (scatterplots)and found that the dispersion of the data and thepresence of outliers greatly affected the perception ofthe correlation and led to large underestimation of thecorrelation between the variables. Other researchers(e.g., Alloy and Tabachnik 1984, Arkes and Harkness1983, BeythMarom 1982, Hamilton and Gifford 1976,Trolier and Hamilton 1986) showed that judgmentsof covariation and correlations are highly sensitiveto (a) the method and format of data presentation,(b) the instruction for judgments, and (c) the judges’prior expectations. In particular, people tend to detect“illusory correlations,” i.e., systematic covariation incases where it is not present (e.g., Allan and Jenkins1980, Chapman 1967).In this paper, we present a new and fundamen
tally different method for constructing joint probability distributions of continuous random variables.This approach dispenses with the need to assessconditional probability distributions, correlation coefficients, concordance probabilities, or conditionalpercentiles. The proposed procedure reduces thejoint probability assessment into a onedimensionalmarginal probability assessment of one of the variables, and the elicitation of one or more “isoprobabilitycontours.”An isoprobability contour of a joint cumulative dis
tribution is the collection of all the points that havethe same cumulative probability, and is analogous toan isopreference contour for a multiattribute utilityfunction (Keeney and Raiffa 1976). Although therehas been work on assessing isopreference contours(e.g., the classic paper by MacCrimmon and Toda(1969) on assessing isopreference contours of bundlesof money and ballpoint pens), and work by Mathesonand Abbas (2005) on deriving the relation betweenconditional utility functions of attributes using isopreference contours and a onedimensional utilityfunction over value, we are not aware of theoretical
1 Pr�x < �x�y < �y� = 0�25+ sin−1��xy/2��.
or empirical work on assessing isoprobability contours of joint cumulative distributions. As we shownext, isoprobability contours have some additionalproperties that facilitate their assessment in practice.The differences between the isopreference and isoprobability contours are due to the different levels ofmeasurement of the utilities (typically, interval scalesunique up to a positive linear transformation) andprobabilities (absolute scales). Thus, this work goesbeyond simple reparameterization of the isopreference contours.We also propose a method to assess isoprobability
contours in practice. This approach does not requirethe subjects to provide any numeric judgments, butsimply to state their preferences over binary lotteries having identical outcomes. The approach cantherefore be used with judges having less technicalexpertise in probability or statistics. The probabilitiesdefining the two lotteries involve the two variablesof interest and, as such, cause the judge to considerthe relevant tradeoffs between them. To the best ofour knowledge, this is the only method that treatsthese probability tradeoffs directly and explicitly andmakes them an integral part of the elicitation process. Once the isoprobability contours are determined,a onedimensional marginal probability assessmentover any of the variables (or over the contours themselves) is sufficient to determine the complete jointdistribution of all the variables. As a result, isoprobability contours can also be used to determine thedependence parameters between the variables of thedecision situation.The remainder of this paper is structured as fol
lows. Section 2 presents some properties of isoprobability contours that facilitate their elicitation. Section 3presents methods to assess isoprobability contoursand methods to construct the joint distribution. Section 4 describes the experimental procedure. Section 5reports results of a study designed to test the feasibility of assessing isoprobability contours. Section 6presents methods to curve fit isoprobability contoursto functional forms to simplify the construction of thejoint distribution. Section 7 curve fits isoprobabilitycontours to a bivariate Gaussian copula and estimatesthe dependence between the two variables. Section 8summarizes our results.
2. Properties of IsoprobabilityContours
To start, consider two continuous variables, X andY , having a joint cumulative probability distribution,F �x�y�, over a connected bounded domain, i.e., x ∈xmin�xmax, y ∈ ymin�ymax. An isoprobability contourfunction, Ci�x�y�, is the set of all points in the x–yplane that have the same cumulative probability, Ci:
Ci�x�y� = ��x�y�� F �x�y� = Ci� (1)

Abbas, Budescu, and Gu: Assessing Joint Distributions with Isoprobability ContoursManagement Science 56(6), pp. 997–1011, © 2010 INFORMS 999
Figure 1 Joint Cumulative Distribution and Isoprobability Contours
Varia
ble
x
Variabley
Joint cumulative distribution function
h
0 0.1 0.2
0.3
0.4 0.5
0.6
0.7
0.8 0.9
1.0
0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1.0
0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1.000.050.100.150.200.250.300.350.400.450.500.550.600.650.700.750.800.850.900.951.00
Variable x
Var
iabl
ey
Isoprobable contours
h
Figure 1 shows an example of a joint cumulativedistribution and the corresponding twodimensionalprojection of some of its isoprobability contours.Isoprobability contours have several properties that
can facilitate their elicitation in practice and can serveas consistency checks during the elicitation process.We discuss some of these properties below.
Property 1. The left and bottom boundaries of thedomain are isoprobability contours.
Proof. A joint cumulative distribution has a valueof zero if any of the variables is at the minimal valueof the domain. This is known as the “grounding property,” where
F �x�ymin� = F �xmin�y� = 0∀x ∈ xmin�xmax� y ∈ ymin�ymax� (2)
Equation (2) implies that all points on the axesx = xmin and y = ymin have a joint cumulative probability of zero and therefore lie on the same isoprobabilitycontour. �
Property 2. If differentiable, isoprobability contourshave a nonpositive slope.
Proof. Recall that the change in probability alongan isoprobability contour must be zero. Hence,
dF �x�y� �Ci=[
�F �x�y�
�xdx + �F �x�y�
�ydy
]Ci
= 0� (3)
Rearranging (3) gives
dy
dx
∣∣∣∣Ci
= −�F �x�y�/�x�F �x�y�/�y
= −Fy�y���Fx �y�x � y�/�x�Fx�x���Fy �x�y �x�/�y�
� (4)
where Fx �y�x �y� � P�X ≤ x � Y ≤ y�. The righthand side of (4) is nonpositive, because a cumulative distribution function (CDF) is nonnegative andnondecreasing. �Given some assessed points on an isoprobabil
ity contour, Property 2 can be tested by calculating Kendall’s �b rankorder correlation between thepoints, as we illustrate in §5.The following properties are also useful and fol
low directly from the definition of isoprobabilitycontours.
Property 3. Isoprobability contours do not intersect.
Property 4. Isoprobability contours connect pointswith the same marginal probability, as shown in Figure 2.
In the next section we use these properties to assessisoprobability contours and to verify the quality of theassessments.
Figure 2 Isoprobability Contours Connect Points with IdenticalMarginal Probabilities
x
yF (x ) = p
F(y ) = p
F (x, ymax) = F (x )
F(xmax, y ) = F (y )

Abbas, Budescu, and Gu: Assessing Joint Distributions with Isoprobability Contours1000 Management Science 56(6), pp. 997–1011, © 2010 INFORMS
3. Assessing an IsoprobabilityContour and Constructing theJoint Distribution
3.1. Assessing an Isoprobability ContourSuppose we have assessed the median (50th percentile) of variable X, which we denote as x50, aswell as other fractiles of X, such as x75 and x90. Wenow wish to assess the 50% isoprobability contour.We start by offering the DM a choice between twodeals with identical outcomes. Thus, the DM’s choiceshould depend solely on the probabilities. Considerfor example, the following two deals.Deal A: The DM receives a fixed amount, $, if the
outcome of variable X is less than x50 and variable Ytakes any value (i.e., Y ≤ ymax�.
Deal B: The DM receives the same fixed amount, $,if the outcome of variable X is less than x75 and theoutcome of variable Y is less than y1 (where y1 < ymax).Based on the DM’s response, we adjust the value
of y1 sequentially until the user expresses indifferencebetween �x50�ymax� and �x75�y1�. If the DM does notexpress indifference between the two deals after a predetermined number of binary questions, we settle forupper and lower bounds for the value of y1, which wedenote as �x75�yU1 � and �x75�y
L1 �, respectively, so we
may have a narrow band around the contour. We consider the midpoint of this band as the estimate of y1.We then move on to the next point and ask the DM tocompare deals based on �x75�y1� and �x90�y2� (wherey2 < y1�, and change y2 in a similar fashion until wereach indifference between them. Through this chaining process we trace the 50% isoprobability contourof the DM. This process is illustrated in Figure 3. It ispossible to supplement this process with consistencychecks involving previously elicited points lying onthe contour.This process is symmetric across variables, i.e., the
roles of the X’s and Y ’s can be reversed.
Figure 3 Assessing Points on an Isoprobability Contour by Indifference Choices
Q = (x75, y1)Q = (x75, y1) Q = (x90, y2)
x
y
x50
y50
x75 x
y
x50
y50
x75 x90
3.2. Probability TradeOffsFor any two points �x1�y1� and �x2�y2� along the sameisoprobability contour, consider the ratio
�y12�x12
∣∣∣∣Ci
= y2 − y1x2 − x1
� (5)
This ratio is nonpositive (Property 3) and representsthe increase in the value of one variable needed tocompensate for a decrease in another to achieve thesame joint probability. As the increment in one variable decreases (and �x → 0�, this ratio approachesthe slope of the isoprobability contour (the probability tradeoff). This provides a simple and intuitive interpretation for the slope of the isoprobabilitycontours—the change in one variable needed to compensate for a change in the other, to achieve the samecumulative probability. We can also define the ratio ofpercentage change in one variable needed to compensate a percentage change in the other,
��x�y� = −�y/y�x/x
∣∣∣∣Ci
� (6)
We have chosen to add a negative sign in this definition to yield nonnegative values of the probabilitytradeoff. This ratio is easier to think about than theabsolute value of the increments because it is dimensionless and the DM needs only to provide a percentage between 0% and 100%. As the increment �x → 0in (6), a fractional change corresponds to the derivative, � = −�dy/dx��x/y�. Contours having a constant� define a family of functional forms whose shapedepends on this parameter and are often referred toas the constant elasticity of substitution curves.
3.3. Constructing the Joint ProbabilityDistribution
Once the isoprobability contours have been determined, the construction of the joint distribution

Abbas, Budescu, and Gu: Assessing Joint Distributions with Isoprobability ContoursManagement Science 56(6), pp. 997–1011, © 2010 INFORMS 1001
requires only the assessment of one of the marginalprobability functions, say Fx�x�. We can determine thejoint cumulative distribution of any point, �x�y� ∈xmin�xmax×ymin�ymax, by finding the point �x1�ymax�that lies on its isoprobability contour. By definition,
F �x�y� = F �x1�ymax�� Fx�x1�� (7)This formulation reduces the joint probability into
a onedimensional marginal probability. The marginaldistribution function, Fx, can be assessed directly usingany of the standard techniques of probability encodingused in decision analysis (Spetzler and Staël von Holstein 1975, von Winterfeldt and Edwards 1986, Abbaset al. 2008), or by assuming a particular functionalform (e.g., a beta distribution) and assessing its parameters by assessing a few probability points using indifference to binary lotteries. This process is symmetricacross variables, i.e., we can replace Fx by Fy .Define
x1�x�y� = x1 such that C�x�y� = C�x1�ymax��For any (x�y), define x1 such that C�x�y� =C�x1�ymax�. Then, F �x�y�� Fx�x1�, and the conditionalprobability, F �x � y�� is
F �x � y� = Pr�X ≤ x � Y ≤ y� = F �x�y�Fy�y�
= Fx�x1�Fy�y�
� (8)
The conditional probability functions at any pointin the plane can also be related explicitly to theassessed tradeoff functions of (55) and the marginaldistributions using the same transversality relationsthat were developed in Matheson and Abbas (2005).
4. An Empirical Study4.1. DesignThe goal of this study is to determine whether thejudges can make sensible probability tradeoff judgments that can be used to infer meaningful joint distributions. We asked a number of judges to assessfractiles of the marginal distribution and the 50% isoprobability contours of two variables with which theywere highly familiar.We elicited the joint probability of the height and
weight of male undergraduate students at the University of Illinois at Urbana–Champaign (UIUC).We selected these variables because all our subjects2 (UIUC students) were highly familiar withthem. Twentyfive students volunteered to participate. They were guaranteed $8 for their participationand could also earn an additional $20, based on one of
2 We use the terms “subject” and “judge” interchangeably throughout this section.
their choices. This provided an incentive to respondaccurately and honestly. Subjects were between 19and 28 years old, with a median age of 24. Nine ofthem were female. They were from various departments, and 19 of them (75%) reported that they hadcompleted at least one class in probability.The experiment was conducted in two different set
tings. Ten subjects were run in a facetoface setting,whereas the other 15 were run in the computerizedcondition. The two groups were very similar in mostrespects. We did not have specific hypotheses aboutthe performance of the two groups. Our goal was simply to test the feasibility of the method under variouscircumstances. The facetoface condition was similarto a probability encoding session in decision analysisin the sense that the experimenter3 had more flexibility and could adjust the nature and sequence ofthe questions as a function of the subject’s responsesto converge to the key values. As such, this groupprovides the gold standard against which the resultsof the computerized group, which was run withoutdirect interaction between the subjects and the experimenter, can be evaluated.All subjects were presented with pairs of gambles,
each of which offered some chance to win $20. Thetwo gambles in a pair described two different events(see details below). In the facetoface group, subjects followed the instructions given by the experimenter. In the computerized group, the instructionswere included in the computer program and a hardcopy of the instructions was also available for reference. Subjects were asked first to select their preferredunits (pounds or kilograms for weight and centimeters or inches for height) and to specify their rangeof values for both variables. Next they estimated themarginal distributions and then assessed the contourof the joint distribution.
4.2. Eliciting the Fractiles of the MarginalDistributions
To elicit the marginal probability distributions of theheight and weight variables, we used pairs of gambles involving (a) a probability wheel and (b) oneevent related to the weight or the height of a randomly selected male student from UIUC. For example, a wheel showing a 50% chance of winning wascompared with the possibility that the weight of arandomly selected male student at UIUC will be lessthan 120, 140, 160 lbs, etc.In the facetoface groups, we used the fixed variable
(FV) method for eliciting the fractiles of the marginaldistributions, where the variable values are fixed andthe wheel setting is adjusted based on the responsesobtained (for more information on this approach, see
3 The first and third authors served as experimenters for theseelicitations.

Abbas, Budescu, and Gu: Assessing Joint Distributions with Isoprobability Contours1002 Management Science 56(6), pp. 997–1011, © 2010 INFORMS
Figure 4 An Example of a Binary Choice Used to Assess the MarginalDistribution of Height
You win $20 if
The wheel is spun and it landson orange
Choose Choose
The height of a randomlyselected UIUC male
undergraduate student is lessthan 5'10"
Indifferent

Abbas, Budescu, and Gu: Assessing Joint Distributions with Isoprobability ContoursManagement Science 56(6), pp. 997–1011, © 2010 INFORMS 1003
There were wide individual differences in this respect.For instance, 3 of the 15 subjects reached indifferenceon all series, and 3 other subjects never expressedindifference. In the facetoface setting, subjects hadan extra pause (of about five minutes) between themarginal and each of the joint assessments.At the end of the experiment, one pair of gambles
was selected and was played according to the subject’s preference to determine the payment.4 Half thesubjects won the lotteries. In the facetoface group wealso asked the subjects to rate the difficulty of the estimations tasks and to provide a direct estimate of thecorrelation they perceive between height and weight.Participants took between 50 to 60 minutes to complete all stages of the study.
5. Experiment ResultsWe now present the results of both groups. In thosecases where we wish to highlight individual results,we focus on the facetoface group where all subjects reached indifference on all series. Figure 6 showsthe joint assessments for the 10 subjects in the facetoface group (all converted to centimeters and kilograms). For each subject, we plot all the points thatwere elicited using the two sequences (fixed weightsand fixed heights) jointly, but we use different symbolsto identify the two.
5.1. MonotonicityOur first analysis is concerned with the monotonicityof the assessments. We use Kendall’s �b as a measureof monotonicity. For any n assessments on a contour,let C be the number of points that are concordant (i.e.,an increase in weight corresponds to an increase inheight and the monotonicity condition of Property 2 isviolated), and let D be the number of pairs that are discordant (i.e., the monotonicity condition of Property 2is satisfied). Kendall’s �b is the difference between theproportion of concordant pairs and the proportion ofdiscordant pairs. Formally,5
�b =∑n−1
i=1∑n
j=i+1sgn�heighti −heightj �sgn�weighti −weightj �(n2
)
= C−D(n2
) = C−DC+D � (9)
where sgn is the sign function.
4 Once a pair of gambles was chosen and the subject’s responsewas identified, we compared it with the weight and height of a“randomly selected undergraduate student.” This undergraduatestudent was in fact the programmer for the experiment. The weightand height of this undergraduate student were fixed in advance inthe program to determine the payoff.5 In the presence of ties, the denominator of the formula is√
�C + D + Tx��C + D + Ty�, where Tx is the number of pairs with tieson X (but not on Y ), and Ty is the number of pairs with ties on Y(but not on X).
Table 1 Median Kendall’s �b RankOrder Correlation for theVarious Distributions in the Two Groups
Marginal distributions Joint distributions
Sample Height WeightGroup size Height Weight fixed fixed
Computerized 15 1�00 1�00 −0�95 −0�95Facetoface 10 0�99 0�90 −1�00 −1�00All 25 1�00 1�00 −1�00 −1�00
Kendall’s �b is a measure of rankorder correlationthat ranges from −1 (all pairs are discordant) to 1(all pairs are concordant), and it is 0 when there areequal numbers of concordant and discordant pairs.For comparison purposes, we also present mea
sures of monotonicity for the two marginal distributions (in this case, perfect monotonicity implies�b = 1). Table 1 shows the median Kendall’s �b for eachdistribution for both groups. All of the (absolute) values are very high, confirming the monotonicity of thejudgments for the marginal distributions and the isoprobability contours. They are highly similar in allcases, indicating that subjects were equally adept inmaking the univariate and the joint assessments.6
5.2. Estimating the Implied Probability TradeOffsFor each two adjacent points, �xi� yi� and �xj� yj�, onthe assessed contour, we calculated the increments�yij = yj − yi and �xij = xj − xi, and then the ratio
�ij = −�yij/yi
�xij/xi= − �height/height
�weight/weight
(see Equation (6)). We then calculated for each subject a robust measure of the spread of the �ij ratios—the median absolute deviation (MAD) around theirmedian value, MAD(�ij �, across the various points onthe assessed contour. A constant probability tradeoffimplies MAD(�ij � = 0. In the facetoface group, theindividual MADs range from 0.05 to 0.77, with amedian of 0.19. In the computerized group, we havehighly similar results—the 15 MADs range from 0.04to 0.73, with a median value of 0.24. These valuessuggest that for many subjects, a constant probabilitytradeoff provides a reasonable firstorder approximation of the contour function.
5.3. Internal ConsistencyRecall that we assessed the 50th percentile isoprobability contour in both directions: On one occasionwe started with the median value of the weight(obtained from the marginal distribution assessments)
6 In the facetoface group, we also asked the subjects to rate howeasy it was to assess the marginal and the joint distributions tocompare the perceived difficulty of the two tasks. The split was 6/4(which is not significantly different from chance, by a sign test),indicating that, on average, the two tasks are equally easy.

Abbas, Budescu, and Gu: Assessing Joint Distributions with Isoprobability Contours1004 Management Science 56(6), pp. 997–1011, © 2010 INFORMS
Figure 6 50% Isoprobability Curves Assessed by the 10 Subjects in the FacetoFace Group
170
180
190
200
210
70 90 110 130
Hei
ght (
cm)
Weight (kg)
Subject 1
Fixed height
Fixed weight
170
180
190
200
210
70 90 110 130
Hei
ght (
cm)
Weight (kg)
Subject 6
170
180
190
200
210
70 90 110 130
Hei
ght (
cm)
Weight (kg)
Subject 2
170
180
190
200
210
70 90 110 130
Hei
ght (
cm)
Weight (kg)
Subject 7
170
180
190
200
210
70 90 110 130
Hei
ght (
cm)
Weight (kg)
Subject 3
170
180
190
200
210
70 90 110 130
Hei
ght (
cm)
Weight (kg)
Subject 8
170
180
190
200
210
70 90 110 130
Hei
ght (
cm)
Weight (kg)
Subject 4
170
180
190
200
210
70 90 110 130
Hei
ght (
cm)
Weight (kg)
Subject 9
170
180
190
200
210
70 90 110 130
Hei
ght (
cm)
Weight (kg)
Subject 5
170
180
190
200
210
70 90 110 130
Hei
ght (
cm)
Weight (kg)
Subject 10
with no constraints on the height, and then offereddeals with increased weights and reduced heights.We repeated the assessment starting with the medianheight (no constraints on the weight), and then offereddeals with increasing heights and decreasing weights.Consistency with Property 4 requires that the assessed
contours connect the median values of each variable,i.e., the median height assessed from the marginal distribution, for example, should equal the value of theheight intersecting the 50% contour assessment withthe vertical axis at the upper bound of the weightvariable (Figure 2).

Abbas, Budescu, and Gu: Assessing Joint Distributions with Isoprobability ContoursManagement Science 56(6), pp. 997–1011, © 2010 INFORMS 1005
Figure 7 Assessed and Inferred Median Heights and Weights for all the Subjects in the FacetoFace Group
Height comparison
107
9 6 83
1
4
5
2
170
175
180
185
170 175 180 185
H_ joint (cm)
H_
med
(cm
)
Weight comparison
6
7
510
3 9
4 18
2
70
75
80
85
90
95
70 75 80 85 90 95
W_ joint (kg)
W_
med
(kg
)
Figure 7 displays these consistency results forthe 10 subjects in the facetoface group. For eachvariable, we plot the median of the variable obtainedfrom the assessment of its marginal distribution andthe median of the same variable as inferred from thecontour assessment. Perfect consistency requires thatall points would lie on the identity line. In addition to the identity line, the plots include narrowbands of ±5 units (centimeters for height and kilograms for weight),7 and each subject is identified byhis/her number. Note that most points are, indeed,within this narrow band and close to the ideal withonly two exceptions (weight judgments of Subject 1and height judgments for Subject 10). We concludethat the observed consistency between the assessedand inferred medians is impressively high for bothvariables.
6. Some Functional Forms ofIsoprobability Contours
The experimental results indicate that people can reason about the isoprobability contours. To simplifytheir assessment, one could assume a particular functional form for the contours and assess its parameters.Of course, the choice of the functional form dependson the qualitative properties of the contours providedby the DM. For example, suppose that for a given DMwe assume that the percentage increase in one variable needed to compensate for a percentage decreasein the other to achieve the same cumulative probability is constant. This implies that the value of �ij isconstant (��, and the DM’s subjective contours can be
7 We selected this ±5 margin to reflect the natural rounding tendencies of most people in reporting heights and weights.
described by constant elasticity of substitution functions of the form
C�x�y� = �x� + �y��1/�� 0< a < 1�where C�x�y� is a contour function, special cases ofwhich are the CobbDouglastype contour functions8
C�x�y� = �x − xmin��y − ymin��� (10)Figure 8 shows, as an example, the isoprobability contours implied by the CobbDouglas contour functionsC�x�y� = xy� for � = 0�8 in the left panel, and � =2�4 in the right panel. The contours in both panelsconnect the marginal probability values of each variable starting at �xmax�y� and ending at �x�ymax�. Themarginal distribution of Y is the same for both casesbut, as the value of � increases, the contours end atpoints �x�ymax� with higher values of the variable X.This models a joint distribution whose marginal probability mass for variable X is concentrated at highervalues.In many cases, the assumption that � is constant
can provide a good firstorder approximation, and itis convenient, because it determines the functionalform of the contours in terms of a single parameter.Rearranging terms in (10) gives
log�x − xmin� = logC�x�y� − � log�y − ymin�� (11)This is easily recognized as a linear regression equa
tion, and the parameters can be estimated by ordinaryleast squares (OLS) regression. In particular, the slopeestimates the (negative) probability tradeoff coefficient. We illustrate this approach for one subject (Subject 4 in the facetoface group).
8 CobbDouglastype functions are a special case of the constantelasticity of substitution family when we take the limit when � → 0and using L’hopital’s rule.

Abbas, Budescu, and Gu: Assessing Joint Distributions with Isoprobability Contours1006 Management Science 56(6), pp. 997–1011, © 2010 INFORMS
Figure 8 Isoprobability Contours for CobbDouglasType Functions with � = 0�8 and � = 2�4
0.8–1.00.6–0.80.4–0.60.2–0.40–0.2
0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1.000.050.100.150.200.250.300.350.400.450.500.550.600.650.700.750.800.850.900.951.00
Variable x
Isoprobable contours
0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1.00
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1.0
Variable x
Var
iabl
ey
Var
iabl
ey
Isoprobable contours
To illustrate this approach, we present in Figure 9the elicited points of one of our subjects and the fittedfunction on a loglog scale, as well as the parametersestimated by OLS. The figure shows the estimatedprobability tradeoff coefficient obtained by fixing theweight, fixing the height, and by using both assessment methods for the estimation. The linear functionfits the data points well in all three panels (whichyield highly similar parameters), suggesting a constant probability tradeoff, and supports the choice ofthe oneparameter function used in this case.Once the value of � is estimated, the contour
function is fully determined and the joint cumulative probability at any point �x�y� can be obtainedby determining the boundary value using eithermarginal distribution:(i) Using the marginal distribution of X, calcu
late the value of x1 that corresponds to �x − xmin��y−ymin�� = �x1−xmin��ymax−ymin�� and determine thecumulative probability of x1 using its marginal distribution, Fx�x1�.(ii) Using to the marginal distribution of Y , cal
culate the value of y1 that corresponds to �x − xmin��y−ymin�� = �xmax−xmin��y1−ymin�� and determine thecumulative probability of y1 using its marginal distribution, Fy�y1�.
6.1. Numerical ExampleConsider one point (x1 = 88�45 kg, y1 = 177�8 cm) onthe assessed contour of Subject 4 (see Figure 9), wherexmax = 91�72 kg and ymax = 190�5 cm were establishedfrom the initial boundary values supplied. We estimated � = 1�38. The boundary values of weight andheight that correspond to this contour are obtained bysolving
�x1 − xmin��y1 − ymin� = �x2 − xmin��ymax − ymin��= �xmax − xmin��y2 − ymin���
We get x2 = 68�42 kg, y2 = 177�57 cm. The joint cumulative probability of (88.45 kg, 177.8 cm) is equal to themarginal cumulative probability of weight≤ 68�42 kg,and the marginal cumulative probability of height ≤177�57 cm. From the assessed marginal distributionswe estimate Fx�68�42� = 0�43 and Fy�177�57� = 0�45,which are very similar.
6.2. Sensitivity AnalysisBecause we need only one marginal distribution anda contour function to calculate the joint probability,consistency requires that calculations obtained usingeither marginal distribution should provide the sameresult. Figure 10 plots the joint probability of thepoint (x = 88�45 kg, y = 177�8 cm) obtained usingboth marginal distributions (as we did in the previousexample) for different values of �. The figure showsseveral interesting results. First, note that when theestimate of � is between 1.38 and 1.66 (the valuesestimated from the two sets in Figure 9), there is little change in the joint cumulative probability usingeither marginal distribution. In other words, the twoprobability tradeoffs �� = 1�38�� = 1�66� yield highlysimilar joint probabilities. Perfectly consistent assessments (for both marginal distributions and isoprobability contours) occur at � = 1�32 (where the curvescross).If we change the value of � while keeping the
marginal distributions fixed, however, we get largerdiscrepancies. The explanation is that for any givenvalue of � and one fixed marginal distribution, say ofX, we completely characterize the joint distribution,and we are determining the marginal distribution forvariable Y (refer back to Figure 8, which shows howthe contours connect different values of variable Y aswe vary ��.

Abbas, Budescu, and Gu: Assessing Joint Distributions with Isoprobability ContoursManagement Science 56(6), pp. 997–1011, © 2010 INFORMS 1007
Figure 9 Estimating the Parameter � for the Assumed ContourFunction
Subject 4 fixed height (face to face)
5.05
5.10
5.15
5.20
5.25
5.30
5.35
4.40 4.45 4.50 4.55 4.60 4.65 4.70 4.75 4.80
Log weight
Log
heig
ht
Experiment points
Subject 4 fixed weight (face to face)
5.05
5.10
5.15
5.20
5.25
5.30
5.35
5.40
4.2 4.3 4.4 4.5 4.6 4.7 4.8
Log weight
Log
heig
ht
Subject 4 both sets (face to face)
5.05
5.10
5.15
5.20
5.25
5.30
5.35
5.40
4.30 4.35 4.40 4.45 4.50 4.55 4.60 4.65 4.70 4.75
Log weight
Log
heig
ht
� = 1.38
� = 1.66
� = 1.38
7. Inferring Dependence from theAssessed Contours
Once the joint distribution is specified, we candetermine any measure of association between thevariables or any concordance probability by directcalculation. Isoprobability contours can also be usedto infer dependence indirectly by curve fitting anappropriate joint probability model. For example, ifthe assessed curves fit a bivariate Gaussian distribution (or bivariate Gaussian copula), the estimatedparameters of the curve fit will in fact be the correlation coefficients. Figure 11 shows the 50% isoprobability contours for several bivariate standard (i.e., �x =�y = 0 and �x = �y = 1) Gaussian distributions. Theseven contours correspond to distributions with different correlation coefficients, �xy . The purpose of thissection is to show how to determine the correlationscoefficients between the variables if the assessed isoprobability contours match those of a bivariate nor
Figure 10 Joint Probability with Two Marginal Distributions forDifferent Values of �
Join
t cu
mul
ativ
e pr
obab
ility
0
0.1
0.2
0.3
0.4
0.5
1 1.3 1.6 1.9 2.2 2.5 2.8 3.1 3.4 3.7 40
Using the marginaldistribution of Y
Using the marginaldistribution of X
�
Figure 11 Selected 50% Contours for Standard Bivariate GaussianDistributions as a Function of �
C = 0.5
–0.5
0.5
1.0
1.5
2.0
2.5
3.0
3.5
–0.5 0.5 1.0 1.5 2.0 2.5 3.0
X
Y
� = 0.9� = 0.6� = 0.3� = 0� = –0.3� = –0.6� = –0.9
00
mal copula. Similar procedures can be used with otherfamilies of joint distributions if the assessed contoursdo not match those of the Gaussian distribution.
7.1. Estimation of Correlation Coefficient byCurve Fitting to a Bivariate Gaussian Copula
To infer the correlation coefficient implied by theassessed isoprobability contour we curve fitted thedata to a bivariate Gaussian copula. The curvefittingprocedure is as follows:(1) Determine the marginal distributions by fitting
the elicited marginal fractiles into functional forms ofcumulative distributions for weight and height, Fw�w�and Fh�h�, respectively. We fitted beta distributions tothe marginal fractiles because they allow a lot of flexibility in matching a variety of fractile assessments forbounded distributions. However, one could also useother distributions depending on their goodness of thefit.(2) Using the fitted marginal distributions, trans
form the elicited weight and height points �w�h� on theassessed contour into corresponding points on standardized normal distributions �wN �hN � by
�wN �hN � = ��−1�Fw�w����−1�Fh�h����where � is the standard normal distribution.

Abbas, Budescu, and Gu: Assessing Joint Distributions with Isoprobability Contours1008 Management Science 56(6), pp. 997–1011, © 2010 INFORMS
Table 2 Estimated Correlations Between Height and Weight and theGoodness of Fit (RMSE) of the Estimates
Subject RMSE � ��s = 6/�� sin−1��/2� �b = �2/�� sin−1���(a) Estimates from facetoface group
1 0�17 −0�48 −0�46 −0�322 0�10 0�80 0�79 0�593 0�06 0�39 0�37 0�254 0�03 0�92 0�91 0�745 0�07 0�89 0�88 0�706 0�21 0�89 0�88 0�707 0�13 0�69 0�67 0�488 0�08 0�22 0�21 0�149 0�06 0�88 0�87 0�6810 0�07 0�99 0�99 0�91Median 0�08 0�84 0�83 0�64
(b) Estimates from computerized group1 0�74 0�99 0�99 0�912 0�28 0�83 0�82 0�623 0�31 −0�74 −0�72 −0�534 1�71 0�99 0�99 0�915 0�46 0�99 0�99 0�916 0�24 0�99 0�99 0�917 1�02 0�73 0�71 0�528 0�33 0�80 0�79 0�599 0�09 0�58 0�56 0�3910 0�16 0�51 0�49 0�3411 0�07 −0�17 −0�16 −0�1112 0�04 0�70 0�68 0�4913 0�12 0�36 0�35 0�2314 0�04 −0�05 −0�05 −0�0315 0�35 0�60 0�58 0�41Median 0�28 0�70 0�68 0�49
(3) Determine the correlation between weight andheight by curve fitting the transformed points �wN �hN �on the isoprobability contour to a bivariate Gaussiandistribution. We searched for the value of � that minimizes the root mean square error (RMSE) betweenthe standardized (transformed) points on the empirical 0.5 contour and the 0.5 contour of the bivariateGaussian distributions (see examples in Figure 11). Wesearched over the range −0�99 to 0.99 in incrementsof 0.01.Table 2 shows the estimated correlation coefficients
and the RMSEs for all the subjects in both groups.The number of assessed points underlying this estimation was 10 for all the subjects in the computerizedgroup. The number of points assessed in the facetoface group varied from one person to another becauseit was determined by the responses obtained duringthe elicitation and the comfort level (of the subjectsand the experimenter) with the quality of data. Formost subjects we elicited 8 or 9 points (median, 9).Subject 10 required 10 points, and Subject 8 requiredonly 7 points.With a few noticeable exceptions (see Subjects 1, 4,
and 7 in the computerized group), the RMSEs are verylow, indicating good fit, especially in the facetoface
group, where the fit is significantly better then in thecomputerized group (t�23� = 2�06; p < 0�05). However,if the three outliers are eliminated, the goodness of fitin the two groups is comparable.The median correlation in the facetoface group
was slightly higher than in the computerized experiment (0.84 versus 0.70), but the two distributions arenot significantly different (t�23� = −0�39; p > 0�05).Given that we are focusing on the copula, we alsotransformed these correlations to rankorder correlations that do not depend on the marginal distributions.Kruskal (1958) shows that in a bivariate normal distribution, the Spearman �s and Kendall’s �b are relatedto Pearson’s �:
�s =6�
sin−1(
�
2
)� (12)
�b =2�
sin−1���� (13)
The results (especially Spearman’s �s� are very similar. Although the correlations in the facetoface groupappear to be high (Table 2), recall that we also askedthe subjects in this group to estimate the correlationat the end of the experiment. Eight of the subjectsdid (two of them did not provide numbers but simply said that it is positive, which, incidentally, reinforces our point that such correlation judgments areoften difficult to provide directly). Among the eightfor whom we have direct estimates, the fitted estimatewas higher in five cases than the direct estimate, butit was lower for the other three. The medians of theeight correlations were, essentially, identical (mediancorrelation of 0.75 using the isoprobability contourbased approach and 0.73 for the direct estimates).Thus, although it is possible that “objectively” thesesubjects overestimate the weight–height correlation inthe student population; this does not seem to be associated with a particular estimation method.Note that in four cases (Subject 1 in the facetoface
group and Subjects 3, 11, and 14 in the computerizedgroup), the estimation yielded negative estimates ofthe correlation coefficient. This counterintuitive resultdoes not reflect an estimation problem. Rather, it canbe attributed to the inappropriateness of the bivariateGaussian assumption for the points assessed for thesesubjects, as shown in Figure 12. The left panel showsthe assessed points for Subject 1 (estimate � = −0�48)and the contours of the various bivariate Gaussiandistributions. Clearly, the assessed points are inconsistent with the model. In sharp contrast, the rightpanel shows one of the bestfitting subjects (Subject 4,estimate � = 0�92), for which many of the points lieexactly on the 0.9 contour.

Abbas, Budescu, and Gu: Assessing Joint Distributions with Isoprobability ContoursManagement Science 56(6), pp. 997–1011, © 2010 INFORMS 1009
Figure 12 Isoprobability Contours of Bivariate Gaussian Distributions with Data from Two Subjects
CDF = 0.5 bivariate Gaussian isoprobability contourswith Subject 1 experiment data
Y Y
X X
0.0
1.0
2.0
3.0
4.0
0.0 1.0 2.0 3.0 4.0
� = 0.6
� = 0.9
� = 0.3
� = 0
� = –0.3
� = –0.6
� = –0.9
Subject 1
CDF = 0.5 bivariate Gaussian isoprobability contourswith Subject 4 experiment data
0.0
1.0
2.0
3.0
4.0
0.0 1.0 2.0 3.0 4.0
� = 0.6
� = 0.9
� = 0.3
� = 0
� = –0.3
� = –0.6
� = –0.9
Subject 4
7.2. Numerical Example: Estimating theJoint Distribution Using a BivariateGaussian Copula
Having determined that the isoprobability contour ofSubject 4 in the facetoface group matches a bivariateGaussian (see Figures 11 and 12), we can estimate thejoint cumulative distribution by curve fitting it to abivariate Gaussian copula. The marginal distributionassessment of weight indicates that Fx�88�45� = 0�77.Under the assumption of standard Gaussian variable, this implies X = 0�74. The marginal distributionassessment of height indicates that Fy�177�8� = 0�46.This corresponds to Y = −0�10 in a standard Gaussiandistribution. The estimated correlation coefficient was0.92 (see Table 2). The joint cumulative probability ofa standard bivariate Gaussian distribution where X =0�74, Y = −0�10, and � = 0�92 is 0.46, which is remarkably close to the previous values obtained by curvefitting to the contour function.
8. ConclusionsWe presented a method for direct elicitation and construction of joint probability distributions of continuous variables using isoprobability contours. Our keycontribution is the development and empirical validation of a practical, easy to implement assessmentmethod. In addition, we described two methods forconstructing the joint probability distribution from theassessed contours and illustrated how to determinebivariate correlations from these values.The main advantage of the newly proposed
approach compared to other approaches such as marginal conditional assessments or direct assessment ofdependence is that the judges do not have to provideany numerical responses, or estimate complex jointprobabilities (e.g., the quadrant probability) or parameters (e.g., the covariance or correlation between variables) that are not intuitive, prone to judgment errors,
and could be intimidating for judges who lack properstatistical training. In our approach, judges only needto express preferences over simple binary gambleswith identical payoffs whose probabilities depend on,and reflect, the probability tradeoffs between the twovariables. A second major advantage of our procedure is that the events being compared involve bothvariables, so the judges are forced to consider therelevant tradeoffs. Thus, the use of isoprobabilitycontours reduces the assessment of joint probabilitydistributions to a onedimensional cumulative probability assessment (which is cognitively simpler toreason about than conditional assessments) and theelicitation of the probability tradeoffs (inferred byexpressing preferences over binary deals).We reported results of a simple feasibility study. The
key empirical question we addressed is whether subjects are able to reason about probability tradeoffs.The results of the empirical study showed that this is asensible and reasonable task that was performed successfully. The judgments regarding the joint distributions matched the quality of the standard probabilityelicitation (FP and FV) for the marginal distributionswith respect to difficulty and monotonicity. The monotonicity level was excellent, and the level of internalconsistency was satisfactory. Finally, most correlationcoefficients estimated from the judges’ assessmentswere reasonable, and the few outlying results are dueto the fact that the judgments of some subjects do notfit the assumptions of the model (in this case, bivariate Gaussian), rather than elicitation or statistical estimation problems. In these types of situations, othermodels of joint probability distributions may providebetter fits.All these results suggest that people can indeed rea
son about the newly defined probability tradeoff coefficients in assessing the isoprobability contours. It is

Abbas, Budescu, and Gu: Assessing Joint Distributions with Isoprobability Contours1010 Management Science 56(6), pp. 997–1011, © 2010 INFORMS
important to stress that these results appear to be quiterobust. Recall that we obtained comparable resultsin the computerized group, where about 40% of theseries ended before converging to indifference, and weapproximated the location of the point on the contourby the middle of the range of values. The one exception to this generalization is that for a minority (3 outof 15) of the subjects in the computerized groups, thefit of the inferred correlations was unsatisfactory.One of the nice features of the new method is that,
unlike the standard methods of assessing dependence,it generates data that can be used in various ways toestimate the target quantities. We discussed and illustrated two ways to estimate the joint probability distribution and infer the bivariate correlation from thejudges’ assessments. This flexibility allows us to double check and validate the quality of our procedure.Both methods are easily implemented and generatedsimilar results in our simple example, but in our opinion it is easier to fit contour functions than curve fitbivariate copulas. The former approach requires onlythe estimation of one probability tradeoff and one ofthe marginal distributions. However, it is also important, as a consistency check, to test that the joint probability density obtained by the contours method isnonnegative, as we do when constructing joint cumulative distributions using cumulative marginal conditional assessments.The notion of an isoprobability contour leads nat
urally to several directions for future research. Thefirst is the derivation of different functional forms ofcontour functions that allow for variable probabilitytradeoffs. For example, all the contour functions wepresented and used in our examples involve constantelasticity of probability tradeoffs. Other functionalforms relaxing this condition, the simplest being a linear function of the variables, i.e., ��x�y� = �0 + �1x +�2y, which generalizes the forms used in this paper,should be considered.The second direction pertains to extensions into
higher dimensions. The basic principle underlyingour approach can be extended beyond the bivariate case, but the complexity of the problem is considerably higher and increases rapidly (as does thecomplexity of conditioning on a larger number ofvariables for conditional probability assessments). Forexample, in the trivariate case, one could estimate relatively easily the probability tradeoffs between eachpair of variables at fixed values of the third variable. This would lead to conditional probabilities ofthe form F �xi� xj � xk� for all distinct permutations ofi, j , k, and works perfectly for constructing the jointdistribution when the dependence between the variables is of the second order, i.e., when this conditional probability does not depend on xk. Cases withmore complex relationships would require estimation
of more complicated threeway probability tradeoffs(or, under certain parametric assumptions, repeatingthe assessment for different values of the variable Xk�,which may be harder to implement and judge. Onecan also assume a certain functional form of thecontour function that extends to higher dimensionsand then use the probability tradeoffs to estimate itsparameters directly. For example, higherorder functions of the form C�x�y�z� = xy�z� can also be considered using pairwise tradeoffs and their parametersassessed at fixed values of the third variable.Finally, there are some important empirical direc
tions for future research. The first one is to go beyondthe feasibility demonstration of this paper and provide evidencebased guidelines for implementing thenew method in practice. This requires answeringquestions such as the following: How many contoursshould be assessed? Does it matter which contours areassessed? What is the minimal number of points thatshould be assessed for each contour, and is there anoptimal way of choosing them? It is equally importantto compare empirically results based on assessmentsof isoprobability contours with other direct elicitationmethods of dependence. It would be useful to determine which method (or methods) are perceived to bemore natural, easier to implement, and more comfortable for (and hence preferred by) the DMs, and verifywhich method (or methods) generates better (morestable, reliable, accurate, valid, etc.) results in practice.
AcknowledgmentsThis material is based upon work supported by the NationalScience Foundation under Award SES 0620008. The authorsthank Sean Jordan for programming the study.
ReferencesAbbas, A. E. 2006. Entropy methods for joint distributions in deci
sion analysis. IEEE Trans. Engrg. Management 53(1) 146–159Abbas, A. E., D. V. Budescu, H.T. Yu, R. Haggerty. 2008. A compar
ison of two probability encoding methods: Fixed probabilityvs. fixed variable values. Decision Anal. 5(4) 190–202.
Allan, L. G., H. M. Jenkins. 1980. The judgment of contingencyand the nature of the response alternatives. Canadian J. Psych.34(1) 1–11.
Alloy, L. B., N. Tabachnik. 1984. Assessment of covariation byhumans and animals: The joint influence of prior expectationsand current situational information. Psych. Rev. 91(1) 112–149.
Arkes, H. R., A. R. Harkness. 1983. Estimates of contingencybetween two dichotomous variables. J. Experiment. Psych. General 112(1) 117–135.
BeythMarom, R. 1982. Perception of correlation reexamined. Memory Cognition 10(6) 511–519.
Chapman, L. J. 1967. Illusory correlation in observational report.J. Verbal Learn. Verbal Behav. 6(5) 151–156.
Clemen, R. T., T. Reilly. 1999. Correlations and copulas for decisionand risk analysis. Management Sci. 45(2) 208–224.
Clemen, R. T., G. W. Fischer, R. L. Winkler. 2000. Assessingdependence: Some experimental results. Management Sci. 46(8)1100–1115.

Abbas, Budescu, and Gu: Assessing Joint Distributions with Isoprobability ContoursManagement Science 56(6), pp. 997–1011, © 2010 INFORMS 1011
Cooke, R. M. 1991. Experts in Uncertainty: Opinion and SubjectiveProbability in Science. Oxford University Press, Oxford, UK.
Gokhale, D. V., S. J. Press. 1982. Assessment of a prior distributionfor the correlation coefficient in a bivariate normal distribution.J. Roy. Statist. Soc., Ser. A 145 237–249.
Hamilton, D. L., R. K. Guifford. 1976. Illusory correlations in interpersonal perception: A cognitive bias of stereotypic judgments.J. Experiment. Soc. Psych. 12(4) 392–407.
Hogarth, R. M. 1987. Judgement and Choice, 2nd ed. Wiley, Chichester, UK.
Howard, R. A. 1989. Knowledge maps. Management Sci. 35(8)903–922.
Kadane, J. B., L. J. Wolfson. 1998. Experiences in elicitation (withdiscussion). Statistician 47(1) 1–20.
Kahneman, D., A. Tversky. 1973. On the psychology of prediction.Psych. Rev. 80(4) 237–251.
Keefer, D. 2004. The underlying event model for approximatingprobabilistic dependence among binary events. IEEE Trans.Engrg. Management 51(2) 173–182.
Keeney, R. L., H. Raiffa. 1976. Decisions With Multiple Objectives:Preferences and Value Tradeoffs. John Wiley & Sons, New York.
Kruskal, W. H. 1958. Ordinal measures of association. J. Amer.Statist. Assoc. 53(284) 814–861.
MacCrimmon, K. R., M. Toda. 1969. The experimental determination of indifference curves. Rev. Econom. Stud. 36(4)433–451.
Matheson, J., A. E. Abbas. 2005. Utility transversality: A valuebased approach. J. Multicriteria Decision Anal. 13(5–6) 229–238.
Meyer, J., D. Shinar. 1992. Estimating correlations from scatterplots.Human Factors 34(3) 335–349.
Meyer, J., M. Taieb, I. Flasher. 1997. Correlation estimation as perceptual judgments. J. Experiment. Psych. Appl. 3(1) 3–20.
Morgan, M. G., M. Henrion. 1990. Uncertainty: A guide to Dealing with Uncertainty in Quantitative Risk and Policy Analysis.Cambridge University Press, Cambridge, UK.
O’Hagan, A., C. E. Buck, A. Daneshkhah, J. R. Eiser, P. H.Garthwaite, D. J. Jenkinson, J. E. Oakley, T. Rakow. 2006. Uncertain Judgments: Eliciting Experts’ Probabilities. John Wiley &Sons, West Sussex, UK.
Spetzler, C. S., C.A. S. Staël von Holstein. 1975. Probability encoding in decision analysis. Management Sci. 22(3) 340–358.
Trolier, T. K., D. L. Hamilton. 1986. Variables influencing judgmentsof correlational relations. J. Personality Soc. Psych. 50(5) 879–888.
Tversky, A., D. Kahneman. 1974. Judgment under uncertainty:Heuristics and biases. Science 185(4157) 1124–1131.
von Winterfeldt, D., W. Edwards. 1986. Decision Analysis and Behavioral Research. Cambridge University Press, New York.
Wallsten, T. S., D. V. Budescu. 1983. Encoding subjective probabilities: A psychological and psychometric review. ManagementSci. 29(2) 151–173.
Yi, W., V. M. Bier. 1998. An application of copulas to accident precursor analysis. Management Sci. 44(12, part 2 of 2) S257–S270.
/ColorImageDict > /JPEG2000ColorACSImageDict > /JPEG2000ColorImageDict > /AntiAliasGrayImages false /CropGrayImages true /GrayImageMinResolution 300 /GrayImageMinResolutionPolicy /OK /DownsampleGrayImages true /GrayImageDownsampleType /Bicubic /GrayImageResolution 300 /GrayImageDepth 1 /GrayImageMinDownsampleDepth 2 /GrayImageDownsampleThreshold 1.50000 /EncodeGrayImages true /GrayImageFilter /DCTEncode /AutoFilterGrayImages true /GrayImageAutoFilterStrategy /JPEG /GrayACSImageDict > /GrayImageDict > /JPEG2000GrayACSImageDict > /JPEG2000GrayImageDict > /AntiAliasMonoImages false /CropMonoImages true /MonoImageMinResolution 1200 /MonoImageMinResolutionPolicy /OK /DownsampleMonoImages true /MonoImageDownsampleType /Bicubic /MonoImageResolution 1200 /MonoImageDepth 1 /MonoImageDownsampleThreshold 1.50000 /EncodeMonoImages true /MonoImageFilter /CCITTFaxEncode /MonoImageDict > /AllowPSXObjects false /CheckCompliance [ /None ] /PDFX1aCheck false /PDFX3Check false /PDFXCompliantPDFOnly false /PDFXNoTrimBoxError true /PDFXTrimBoxToMediaBoxOffset [ 0.00000 0.00000 0.00000 0.00000 ] /PDFXSetBleedBoxToMediaBox true /PDFXBleedBoxToTrimBoxOffset [ 0.00000 0.00000 0.00000 0.00000 ] /PDFXOutputIntentProfile () /PDFXOutputConditionIdentifier () /PDFXOutputCondition () /PDFXRegistryName () /PDFXTrapped /False
/Description > /Namespace [ (Adobe) (Common) (1.0) ] /OtherNamespaces [ > /FormElements false /GenerateStructure false /IncludeBookmarks false /IncludeHyperlinks false /IncludeInteractive false /IncludeLayers false /IncludeProfiles false /MultimediaHandling /UseObjectSettings /Namespace [ (Adobe) (CreativeSuite) (2.0) ] /PDFXOutputIntentProfileSelector /DocumentCMYK /PreserveEditing true /UntaggedCMYKHandling /LeaveUntagged /UntaggedRGBHandling /UseDocumentProfile /UseDocumentBleed false >> ]>> setdistillerparams> setpagedevice