Using imprecise estimates for weights

Alan JessopDurham Business School

Motivation

In a weighted value function model weights are inferred from judgements.

Judgements are imprecise and so, therefore, weight estimates must be imprecise.

Probabilistic weight estimates enable the usual inferential methods, such as confidence intervals, to be used to decide whether weights or alternatives may justifiably be differentiated.

Testing & sensitivity

Single parameter. Results easily shown and understood. But partial.

Multi parameter. Examine pairs (or more) to get a feel for interaction.

Global (eg. Monte Carlo). Comprehensive but results may be hard to show simply.

Using some familiar methods uncertainty can be inferred from judgements and the effects of global imprecision can be shown.

An analytical approach rather than a simulation.

Sources of imprecision

Statements made by the judge are inexact. This is imprecise articulation: variance = σa²

The same judgements may be made in different circumstances, using different methods or at different times, for instance. This is circumstantial imprecision: variance = σc²

articulationsingle point three-point

degrees of freedomimplicit in questions

0 none available a2

>0 c2 c

2 + a2

Sources of imprecision

Redundancye.g. ask at different times reciprocal matrix

No redundancye.g. simple rating

related?

3 point estimate: a2

μ = aM + (1-a)(L+H)/2σa = b(H-L)

Previous studies for PERT analyses. Generalise as a = 1.800x10 -12 c 5.751

b = 1.066 - 0.00853c

Beta distribution

But because w = 1 variances will be inconsistent.Solution: fit a Dirichlet distribution.

Dirichlet

f(W) = ki wiui -1 ; 0<wi<1, iwi = 1, ui0, i

where k = (i ui) / i(ui)

which has Beta marginal distributions with

mean i = ui / vvariance i² = ui(v- ui) / v²(v+1) = i(1-i) / (v+1)

and covariance ij = -uiuj / v²(v+1) = -ij / (v+1) ; i≠j

where v = i ui

Relative values of parameters ui determine means.Absolute values determine variance via their sum, v.

Dirichletset parameters weight values

Dirichletjudgements: marginal characteristicsmean ei and variance si

2

consistent variances i²

Usually used by specifying parameters (eg in Monte Carlo simulation)

But can also be used to ensure compatibility:

Put i = ei

Then get least squares best fit to minimise S = (i² - si2 )2

S/v = 0 → v+1 = [ei(1- ei )]2 / ei(1- ei )si2

so i² = ei(1-ei) / (v+1)

( NOTE: only have to know mean values and v )

Sum over available estimates si² so can tolerate missing values

Experiment: FT full-time MBA rankingattribute weightWeighted salary (US$’000) 20Salary increase(%) 20FT research rank 10 rankInternational mobility rank 6 rankFaculty with doctorates (%) 5FT doctoral rank 5 rankInternational faculty (%) 4International students (%) 4Aims achieved (%) 3 rankValue for money rank 3 rankCareer progress rank 3 rankEmployed at three months (%) 2Women faculty (%) 2Women students (%) 2International board (%) 2Languages (number) 2Placement success rank 2 rankAlumni recommend rank 2 rankInternational experience rank 2 rankWomen board (%) 1

100

54

11118

555

7 variables used in

experiment

Experiment: 3 point estimate → a2

low mode high low mean high s σa

Salary increase (%) 70 90 95 0.153 0.185 0.208 0.016 0.021Aims achieved (%) 100 0.219 0.022Employment at 3 months (%) 70 90 100 0.153 0.189 0.219 0.020 0.021Women faculty (%) 10 40 50 0.022 0.072 0.109 0.026 0.014Women students (%) 40 50 60 0.087 0.109 0.131 0.013 0.017International faculty (%) 10 30 40 0.022 0.058 0.087 0.020 0.012International students (%) 60 80 90 0.131 0.167 0.197 0.020 0.020

260 480 435 0.568 1 0.951

3 point judgement scaled

from which, mean and standard deviation

Dirichlet consistentmissing value tolerated

Summarising discrimination between programmes

y = i wixi

var(y) = ij σijxixj

= [ i wi(1-wi)xi² - 2ij wiwjxixj ] / (v+1) j>i

For two alternatives replace x values with differences (xa-xb)

Northern Europe: UK, France, Belgium, Netherlands, Ireland

Summarising discrimination between programmes

Summaries

(v+1) = 351.77

Proportion of all pairwise differences significantly different at p = 0.1:

discrimination = 81%

Lines show indistinguishable pairs; p = 10%

Lines show indistinguishable pairs; p = 1%

Lines show indistinguishable pairs; p = 25%

Weights. discrimination = 71%

Weights; p = 10%

best 1 2 3 4 5 6 71 1 7 1 7 7 8 62 1 6 6 73 8 1 7 7 7 74 15 5 1 56 5 17 7 7 9 7 1

Different circumstances → c2. Reciprocal matrix.

1 2 3 4 5 6 7 mean1 0.37 0.30 0.37 0.18 0.23 0.23 0.41 0.302 0.05 0.04 0.05 0.16 0.20 0.20 0.01 0.103 0.37 0.34 0.37 0.18 0.23 0.20 0.48 0.314 0.05 0.01 0.05 0.03 0.01 0.01 0.01 0.025 0.05 0.01 0.05 0.13 0.03 0.14 0.01 0.066 0.05 0.01 0.05 0.13 0.01 0.03 0.01 0.047 0.06 0.30 0.05 0.18 0.30 0.20 0.07 0.17

1 1 1 1 1 1 1

σc

0.090.080.110.020.060.040.11

1 2 3 4 5 6 71 1 7 1 7 7 8 62 0.14 1 0.13 6 6 7 0.143 1 8 1 7 7 7 74 0.14 0.17 0.14 1 0.2 0.2 0.145 0.14 0.17 0.14 5 1 5 0.116 0.13 0.14 0.14 5 0.20 1 0.147 0.17 7 0.14 7 9 7 1

2.72 23.48 2.70 38.00 30.40 35.20 14.54

a2 and c

2. Reciprocal matrix.

Give each judgement in a reciprocal matrix as a 3-point evaluation.

Then treat each column as a separate 3-point evaluation and find Dirichlet compatible a

2 as before.

For each weight the mean of these variances is the value of a2

as in aggregating expert judgements (Clemen & Winkler, 2007).

The mean of the column means is the weight and the variance of the means is c

2.

Results from 10 MBA students

FT C A B F G H I J E D mean σ/μSalary increase 54 40 36 34 31 27 19 17 16 13 9 24 0.44Aims achieved 8 20 25 25 14 10 34 20 20 11 8 19 0.43

Employed at 3 months 5 18 18 17 32 35 15 17 43 26 29 25 0.38Women faculty 5 3 2 3 4 2 3 5 3 3 3 3 0.25

Women students 5 4 3 6 7 6 4 12 2 3 4 5 0.54International faculty 11 6 7 4 5 4 6 3 7 21 23 9 0.82

International students 11 9 9 11 9 16 18 26 10 22 24 15 0.45

standard deviations

σ = [a2 + c

2]½

Are the two sources of uncertainty related?

0

0.01

0.02

0.03

0.04

0.05

0.06

0.07

0.08

0 0.002 0.004 0.006 0.008 0.01 0.012 0.014

variance from 3 point estimates - σa2

A

B

C

D

E

F

G

H

I

J

variance from mean estimates - σc2

c r7.3 0.91

4.6 0.84

4.7 0.83

15.9 0.62

1.5 0.43

2.9 0.30

1.4 0.72

1.5 0.77

4.9 0.79

16.0 0.82

3.2 0.33 ALL

σc2 = c.σa

2

mean r = 0.70

taken togetherr = 0.33

consistently σc > σa

FT C A B F G H I J E D mean σ/μSalary increase 54 40 36 34 31 27 19 17 16 13 9 24 0.44Aims achieved 8 20 25 25 14 10 34 20 20 11 8 19 0.43

Employed at 3 months 5 18 18 17 32 35 15 17 43 26 29 25 0.38Women faculty 5 3 2 3 4 2 3 5 3 3 3 3 0.25

Women students 5 4 3 6 7 6 4 12 2 3 4 5 0.54International faculty 11 6 7 4 5 4 6 3 7 21 23 9 0.82

International students 11 9 9 11 9 16 18 26 10 22 24 15 0.45

Student G is representative

Scores. (v+1) = 8.54. discrimination = 30%

Lines show indistinguishable pairs; p = 10%

Lines show indistinguishable pairs; p = 50%

decide that 1 & 5 can be distinguished

A possible form of interaction

Assume that new discrimination is due to increased precision rather than difference in scores: statistical significance rather than material significance.

So, change precision by changing (v+1) and leave weights unaltered.

z is directly proportional to √(v+1).

In this case (v+1) = 8.54 → z1,5 = 0.55

p = 50% → z* = 0.67

(v+1)new = (z*/z1,5)² × (v+1) = (0.67 / 0.55)² × 8.54 = 12.67

and so ...

Weights. discrimination = 14%

Group aggregation

A

μ, σa²,σc²

A B C D

μ, σa²,σc²

Do this for all ten assessors.

Scores. (v+1) = 7.18. discrimination = 16%

Lines show indistinguishable pairs; p = 10%

Lines show indistinguishable pairs; p = 50%

Weights. discrimination = 0%

Tentative conclusions

Even though results are imprecise there may still exist enough discrimination to be useful, as in forming a short list.

May give ordering of clusters.

Makes explicit what may justifiably be discriminated.

Choosing confidence levels and significance values is, as ever, sensible but arbitrary. Explore different values.

Once a short list is identified, further analysis needed, probably using some form of what-if interaction to see the effect of greater precision.

Variation between circumstances seems to be consistently greater than self-assessed uncertainty. Does this matter? Do we want to justify one decision now or address circumstantial (temporal?) variation?

end