Naval Training EquiRment'Center, Orlandd, Fla ...DOCUMENT, RESUME. JR' 002 541. Nickeriom, R. S::...

4

.,

4

'46, ED 1114437'

. AUTHORTITLE

INSTITUTIONREPORT NOTUB DATE,..NOTE

DOCUMENT, RESUME

JR' 002 541

Nickeriom, R. S:: Fee hrer, C.4pecisiba Making and Training: A Review of neoreticaland Empirical Studies of pecrsion Making and their

"Implications four. the Training of Decision Makeri%. Naval Training EquiRment'Center, Orlandd, Fla.

NUTRAEQUIPCEN-73-C-0128-1Aug 75227p.; Xechi;lical Re'port,

,EDRS. PRICE MF-$0.76, HC-$12.05 Plus ,PostageDESCRSPTORS Cognd.tiv.e Processes; *Decision Making Skills;

Educational Games; *LeadershiR Training; LiteratureReviews; Banagement Sy#ems; Military Training;

'.b - Problem Solving;'- Professional Education; *Skill.

Development;, Training'Objecti s;, *Tra fei ofTraining. ''

..\-.. e.

ABSTRACT. ,. . , ThLretical and empirical studiesof decision-making

we reviewedreviewed to identify results applicable. to trainIng 4 ..

decisiop'-makers. the review is organized in terms.of the following 1,,componefit tasks: information gathering', data evaluation, problemstructuring, hypothesis generation, hypothesis evalUatioD, preferencespecification, action selection, and-decisibn evaluation.

..

Implications of research findings for each task are described. It isconcluded that decision - malting ±s not sufficiently well 'understood to

, permit the design of an effective general-purposeCtraining system,but that systems and programs could be developed foetraining inspecifid decision:-making skills.'(SK)' a :

4

*);********A**.*****i******4*1c****************************************** Documents acquired by ERIC include many informal unpublished *

*'.materials not available from other sources. ERIC makes every effort ** td obtain the best copy available..nevertheless, items of marginal *

/i reproducibility are often encountered and this. affects the quality *,

* of the microfiche. and hardcopy reproductions ERIC .makes available ** vi'a the ERIC Document Reproduction Service (EDRS). EDRS is no* responsible for the pality of the original document. Reproductions ** supplied by EDRS are.the best that can' be made from the original.***********************************************************************

.-- .

S

tts 4.

k

Report: NVTRAEQUIPCEN 73-C-0128-1

r34 DECISION MAKING AND TRAINING':

, r1

A Review of Theoretical and Empirical Studies

of.Decision Making And Their Implications for

the_Training of DeeisEtin Makers .

R. S. Nickersoh

C. E. Feehrer

Bolt Beranek.. And piewman,

Cambridge, Massachusetts 02138Contract 116.1339-7;p4-0128

-1 Auguslit 1975

DOD4istribution Statement'

Approved for public release;distribution unlimited.

4

AT.

4

U S DEPARTMENT OF HEALTH,EDUCATION& WELFARENATIONAL INSTITUTE OF

EDUCATIONTioS DOCUMENT HAS BEEN REPRODUCE EXACTLY AS RECEIVED FROMTHE PERSON OR ORGANIZATION ORIGINATING IT POINTS OF VIEW OR OPINIONSSTATED DO NOT NECESSARILY REPRESENT OFFICIAL NATIONAL INSTITUTE OFEDUCATION POSITION OR POLICY

NAVAL TRAINING EQUIPMENT CENTER

ORLANDO, FLORIDA 32813T

2

U

pAVTRAEQUIFCEN 73 -C- 01,28-11

ABSTRACT

, This repi5i.t.review6 theoretical and empirical studies ofdecision making. The purpose of the 'review -was to identify resultsth&t.would be applicable to the problem of training decision makers.

4 .The literature on decision making is extensive. However,

relatively few studies haNie dealt explicit]' with the problem of' training in decision-making skills. The task, therefore, was togather from the general: literature on decision making any impli-cations that Could be found fdr training.

Dessionqiaking is conceptualized here as a type of probleino'solving, and the review is organized in termdrof the f011owing ,

'component tasks: information gathering, data evaluation, problemstructuring, hypothesis generation, hypothesis evaluation, pref-erence specification, action selection, and decis4on eValuation.Implications of research findingaafoi training ar&discussed inthe context 'of descriptions of each of these,tasks.

IA general conclusion drawn from the study is that'decisionmakingis probably not sufficiently well understood to Permit thedesign of,an effective general-purpose training system for decisionmakers. Systems-and programs could be developed, however, tofacilitate training with,respect to kpecikic decision-making skills.'The- ',development of more generally applicable training techniquesor systems should proceed-in an evolutionary fashion.

4Training is one way tq improve decision-making perfprmance;

another it to provide the decision maker with aidsfor variousaspects of his task. .Because training and the provision of decisionaids are viewed as domplementary approaches to the same problem,

j the report ends with a discussion of several decition-aiding tech-niques that are in one or another-stage of study or development.

Governmentpla hts In Data Statement

Repbduction of this publicationin whol4e or in part is permittedfor any purpose of the UnitedStates Government.

'Mt

s,

a.

4

' UNCLASSIFIEDSECURITY CLASSIFICATION OF THIS PAGE Men Data Eniored,

f.----,

REPORT DOCUMENTATION P'AGE. READ INSTRUCTIONS /

B EFORE tCOMPLETING FORM'. REPORT NUMBER 2. GOVT ACCESSION NO. 3. RECIPIENT'S CATALOG NUIII3ER....;)

NAVTRAEQUIPCEN 73=C-0128-1 . .4. TITLE (and Subtlile) , 1 5. TYPE OF REPORT A PERIOD COVEREOM.

Decision Making and Training,* A Review of Thed- 1

rekical and Empirical Studies of Decision Making Research 6/73 - 7/75

and their Ipiplications for the Training of Decisio 3 PERFOF;IMING,ORG. REPORT NUMBER, \Makers

7$ AU pOR(s) 3. CONTRACT OR GRANT NOMBER(a)

NAVTRAEQUIPCEN,

R. S. NICKERSON.

N61339-73-C-0128-1Cr E. FEE.HRER '

9. \PERFORMING ORpANIZATION NAME AND ADDRESS 10:. PROGRAM ELEMENT, PROJECT, TASK-AREA & WORK UNIT NUMBERS

Bolt Beranek and Newman, Inc.. , NAVTRAEQUIPCEN

50 Moulton Street , Task No 3751-01P02 'Cambridge, Massachusetts02138

II. CONTROLL NG OFFICE NAME AND ADDRESS 12. REPORT DATE

Naval Trai ing Equipment Center , August'1975Code N-215 13. NUMBER.OF PAGES

Orlando, Florida' 32813, NO 1 '

14. MONITORING AGENcY NAME & ADDRESS(ff different from Controlling Office) 15. SECURITY,CLASS. (of rhis report)

"\' . .. 'Unclassified

15a. DECLASSIFICATION /DOWNGRADING,. SCHEDULE

16. DISTRIBUTION STATEMENT (of Ipla 120.2:1) .

Approved for public release; distributron.unlimited.- \

.

. ,ia

17. DISTRIBUTION,STATEMENT (of the abstraci warned In Block 20, If Oltenia! from Riiport),. *. i. . -,

.

I N

,13. SUPPLEMENTARY NOTES . ,

I. aat , . r.. './ . .

.19. KEY WORDS (Conilnuo on revere() side 11 necessary and Identify by block number)

Deci'sion making' Preference Specification TrainingProblemsolving. Action Selectpn Training Device .

Information gaithering ,Decislin Aids' Technology

Hypothesis Generation' PrObabiyity theory *

A othesis Evalua is silo. Is 1 I e I II

20. ABSTRACT (Continue on reverse side If nee...eery and Idenilfy by block number)

This report reviews th6oretical and.empirical studies of daision making. Thepurpose of the review was to identkiy results that would be applicable to theproblem of training decision makers.

The literatre on decision making is(extensivee .However, relatively fewstudies hay.e dealt explicitly with the problem of training in decision-making

.skills. The task, therefore, was.to gathert.fromthe.general litergture ondecision making any implications that could be found for training., (Cdnt)

DD 1 F.,c,),F:;3 '1473 EDITION OF 1 NOV 65 IS OBSOLETES/P,r 0 10 2-014.660 1

4

UNCLASSIFIEDSECURITY CLASSIFICATION OF THIS PAGE (171rn Date Knitted)

t(

I

.6*.4.ttt. LWRf r Y CLASSIFICATION OF THIS PiOt(Whon Data Entered)

TeeislonmakiAg.ts,conceptualized here as a type of problem solving, and the'.

review is organized in. terms of the-followilig component tasks: informationgathering, data evaluation, problem structuring, hypothesis generation, hypothsis evaluation, prefeee-noe spectfication, action selection, and,decision eval-uation. Implications of research findings for training are discussed in thecontext.of,Aescriptions of each of these tasks.

i

A general concluSion drawn from the study is that decision making is probablynot ,sufficiently well understood to permit the design of'an effective general-purpose training system for decision maker, Systemg and programs could bedeveloped,; however, to facilitate training with respect to specific decision -

=making skills. The development of more generally applicable training tech-niques or systems shouldlpraceed in an evolutionary fashion.

Training is one way to Amprove decision-making performanCe; another is:to,provide the dqision maker with aids for various aspects of his task. Becausetraining and the provision of decision aids are viewed as complementary ap-proaches.to the same problehi, the report ends with a discussion of severaldecision- siding techniques that arse in one or another stage of study ordevelopment. 4 - e'

S

1'

0

,

<

44.

UNCLASSIFJEDsecunity CLASSIFICATION OF$THIS PAGE(Whon Data Entered)

IPS

I

oG

4

3

a

.

NAIPI1RAEQUIPCEN 73-C=0128-1 `

FOTIDOR

The uman Factors Laboratory of the Naval Training Eq uipment's,

..

,Center has been involved in decision-making research with theae.

Objebtive of developing an approach to decision-making trainiAkwhich Will iMprdve the decision-making and tactical performancecapabilities of Navy. commanders. This report is the,result of an '

\analytical review of decision-making research which wa performedis

to identify information pertinent to the training of deisipn-making skills. .

4The outcome, of ''his effort corroborated an ippression that

very little of the great amount of decision=ffiaking fesearch.hasdirectly addressed the problem of training indecision making.The review has identified-implications fir ,the training of decisionmakeks and areas for research which could provide insight for the .

development of effective training procedures and programs. . .

4

Imre'

d.21 WILLIAM P. LANE.Acquisition Director

A s

I

.

.

45

I4

NAVTRAEQUIPCRil 73-C-0128-1

TABLE OF CONTENTS

SECTION. I. ", INTRODUCTION

4

Palle

1

-

SECTION'IT: SOME COMMENTS ON DECISION THEORY.,.'.. 4

2.1 Prescriptive versus Descriptive Models 42.2 Worth, Probability and Expectation 72.3 Game Theory 0

152.4 :Decision Theory and Treing 16 '

SECTION IiI. CONCEPTUALIZATIONS OF DECISION',SITUATIONS AND TASKS - 18

3.Y.

3.2

Classifications of Decision Situations orDecision Types3.1.1 Edwatdos .

3.1.2 Howard %

..

'43.1.3 Sidorsky .,.' t

Classifications of Decision Tasks

181818

.1919

Ao

110

3.2.1' Howard .

-., 32:2 Adelson -. .

.1

, . 3.2.? Drucker .1

Soelberg .3.2.4*.

g1...

t 1.2.5 -Hill and .11.1artin 4. ..'3.2.6 Edwards3.2,7 Schrenk. . . 0. . . ..-.. ... ... ._......-.. ,

,....

r

.

.

%

1

1920202020222

25

29

29-'

.

30

-33

34

3536

38,

383940'

41

)13.3 Decision Making as a Collection of Problem-Solving Tasks

..#

- .

...-0-/

SECTION IV. INFORMATION GATHERING 1

. ,

4.1 Information Seeking versus Information PurchasingM.2 Optional-Stopping Experiments -

4.3. .Decision Revision and-Effect of Commitmenton Information Gatherin

4.X Quantity of Informati and Quality of Decision4.5 A Conceptualization of Inforittion Gathering

. in the Real World4.6 Information Gathering and Training.

1

. SECTION V- DATA EVALUATION

5.1 The Evaluation versus the Use of Data5.'2 Studies of` Data Evaluation5.-3 The Use of Nonquantitative Qualifiers5.4 Data Evaluation And TrAining

0s'

a

NAVTRAEQUIPCEN 73-C-0128!

.>

.

fSECTION VI. PROBLEM STRUCTURING4'2.

$ ._,-

i

6.1 St'ate-Aa n Matrices. 1 '436.2 AlterpAt, e Strugturings of a 'Giveri Situatiori 446.3 Structd5in(as-an Iterative Process 45'6..4 Pxoblem Strdcturing'and Training 46

. '

SECTION VII. HYPOTHESIS GENERATXON.. . / '48.

(..

7.1 Hypothesis Generation/versus Hypothesis Testing M87.2 Importance of Hypothesis Generation....d 49

.

'

4 , 7.3 Experiments, qnHypothesis Generation I- .... 507.4 Hypothesis Generation and Training -'" , 53

..J

'

t

.

.

Q ..

SECTION VIII. HYPOTHESIS EVALUATION 'S

8.1 Serial versus Parallel Processing8.2 Subconscious Processes Li ---se .

8.3 Man as an Ilftuitive Ldgician.\:",,8.4 Man as ari IntuitiVe Statistician , t

7 8.5 Intuitive Probability Theory ..

8.5.1 Representativeness8.5.2 Availability 4

8.5.3 A Methodological Consideratfon. ,. -.

8.5.4 Training and IntuitiVe4ProbabilitY Theory8.6 Bayesian Inference., ,. 4-

_..8.6.1 Bayes Rule .

.,''

8.6.2 Likelihood Rat i . -,

8.6.3 OtheeiMethods Obtaining ProbabilityEstiMates ..:: ,

'8.6.4 DiagnOstiel,V of Data - .',6'8.6.5 Odds

- 8.6.6 Applications of Bayes rule,in theTwo-Hypothesis Case ..

.

8.6.7 .Expected fffects.of Observations' on Hypotheses..8.6'.81 The SyMthetrical Two-Hypothesis Case8.6.9 The Several-Hypothesis Case

.18.6.10 Man as a Baypsian Hypothesis Evaldator48.6.11 'Bayesian Hypothesis Evaluation and Training

8.7 The Measurement of Subjective Probability8.7.1 The Logarithmic Loss Function8.14 The Quadratic Loss Function 1.

;,8..7.3 /the Spherical-Gain Function .

8.7.4 4Ftplementat-ion Vf Admissible Probability4: Me sures ' .

8.7.5 The fficacy of Admissible Probability Measures.8.7.6 Silbje ive Probability Measurement and Training.

54

4545556586062

1'6364

1 65666668

186969

707596102115118'120122123'123

125126126

iv ti

4

4

t

. .. \ '. .

s, V r,.

NAVTRAEQUIPCEN 73-Ce0128-1 ." , .....)

fh.,

.

. , ia

(4

4

8.8 The Use of Unreliable Data. .. 127,.

.

8.'8.1 Prescriptive Approachea f

- 1278.8.2 Some Empirical Results .1368.8.3 SoMe AtteMpts to Derelop'Desctiptive Modelsf,.

.%of Cascaded Inference

0.38 -89 Some'Comtents on .Bayesian Hypothesis 'Evaluation ,142.

.. .

,

- . .

SECTION IX. PREFERENCE SPECIFICATION; .. ..... 146 .-i. 0 .

..

9.1 A Di -fficult and- Pecilliailvlipman Task'9:2 Some EarlyPrescription foRChoice'9.3 'Simple Models' of Worth ComposLtion 4' =

9.4, The Problem of Identifying Worths iCoMponents9,5 'Studies of Choice,tehavior

4

5.

9.6 Procedures for Specifying Worth A

Prgfer6nee4 among gamblds...,9.8 Preference, Specification and Training.

,

-SECTION.X. ACTION SELEI--.

SECTION XI. dECISIQN EVALUATION

-11.1 Effec-qiveness versus Lgoga cal Soundness, 11.2 'Evaluation Criteria --'.4

I.

A

S5

= 147::149151151152154156

!AECTION XII. SOME FURTHER COMMENTS OWTRAINING OF= DECISION MAKERS

12.1 Performance Deficiencies ver as'ioetformance Limitations.12.'2 Simulation as an ApKoach 'Training12.3 4.0n the_ .Idea of a G.4neral-Purpose Trainling Syst

\ for Decision Makers

AS

I

SEtTI6N-XIII. DECISION( .

.AIDS. \14e.

',13.1 LineaWrogramming A 0 v kn 171, 13.2 Decision Trees. 4 91 Flow Diagrams 172

13.3 Delphi, an Aid to CrRup Decision Making\

1743.4 Computer-Based poecision Aids 176

13.4.1 'Computer-Based Aids for. Medical Decision Making 178;. 013.4.2 Computer-Based Aids for Tactical-7NDecision Making 179 ,

,13.442.1, AESOP 4 ...e k 0-...4 13 4 2 2 TACTRAIN

. / ** .

, ,

, 2(313.4.3 .computer -Based Dedision,Aids d Training..)..! 284' ', p

,

c.

165

165166

169

170

N'1.

NAVTRhEQUIPCEN 73;-c 0128:7;1 .

4'

ICKNOVLEDdgENTS40.:** ..,' q..

0' 186\-, / "

.

REFERENCES ,187

<

tS

sr

1

V.

.4.

a

-I

61.

-1

'I

rt

1

R

Table

4

,

NAVTRAEQUIPCEN 73-C-0128 -1

pList. of Tables

Page.

1 Four basic TyRes'of Expectation 8 /

2 Serpitivity,

of Bayesian Analysis ..

95

3 , OddsrFavd-rina 411 Giyen the Indicated V alues of ,

ip(UAH1), p (D I H2) qnd'd.

..

. 103

. \I

Expect&I.Values of posterior Probabiiities and Uncertainty(in bits),, Given that Chips-are Sampled from the Uit forwhich the Indicated Hypothesis iSTrue 110 4

ft I

EiPlo(Hilf)1Hj_is True] for All CombInations,of4i and j

and the Two Indicated Hypothesis Sets.

111...

- - .' -E[p10 (H

i[1:)1jffs is True] f9r Tw9,Five-Hypothesis Sets 112

. 3-.

.,4.

I

J

ti

I

Si A'

11t

r

t

r.

NAVTRAEQUIPCEN/73-C-0128-1

List of ilaustrations

,/

1 Changes in Posteriorid a'Result, of the I

k

Probabilities, p(kilD) anddicated Observations of Red and Wick Chips... 72

2 Changes in odds, SI as a Result of the Indicated Observations

of Red and Black Chips,

3 Changes in Uncertainty Concerning Hypotheses as a Reslilt of

th, fIndicated Observations o Red'and Black Chips ti

4 Changes ip Posterior Probabilities, p(HilD)Juld;p(H2ID) as a

Result of the Enditated 0bserliations of -Red and Black.ChiRs

5 Effects,of Indicated Observations on p(Hilb) for Different

Values.of

6 Effects of Vindicatedpdicated Observafionb.onInitial Values of p(H

11

/7 Effects of Inditated Observations on

Initial Values of p(111).......,,

Odds, SI, 1 for Different

Uncertainty4or.DifferentA

8 All Possible Values of p(1111D)after N Observations

9 Graph Illustrating the Computation of Expected Values of

Posterior .Probabilities

10 Expected )lalue of Posterior Probability Hi, Given H4 is True,,

as a Function of Number of Observations. e

11 Expected Uncertainty Concerning whith Hypothesis is True,,as

a Function of Number of ,Observations

73 f

74

76'

77 IN.

78

79,

81

84

,86

88

1

*1.

-

12 Changes in Posterfan P obabilities, p(HilD) and p(H21D),Iaa a' aRgshlt of the Indicated Observations of Red aneBlack Chips 89

13

14

'15

Changes in Vneertainty Cohcerning Hypotheses as a'Result of the

Indicated Observations of Red and Black Chips., . ...

EFpected Vlue of Posterior Probability of HA, Given H is True,

as a Function of Number ofObservations.....:"Z.

Expected' Uncertainty ConcerRing which Hypothesis ±d True, as

a Function of Number of Observations' '

.41(

16 Expected Value of Posterior Probability Hi, Given H is True, as

A Function of Number of Observations

re

t.

i .0

N.

"91 N-

92

93'

,-1

.

.

t.. Figure .

.. . .

0e.

1 .17' E4ected Uncertainty Concerning which Hypothesis is True,.

as a Function of Number of Observations ,p4. ..F;

18 The Effec=t of Drawing a

,

Single Red Chip, Given, Various COmbina- 1- tiona of Prior Hypotheses-Concerning the Proportion of Reds in

'the Urn .. i . y 97

NAVTRAEQUIPCEN 7 3 -C -0 l 28 -.1

1List pfIllustretions (Cont)

.. -0

19 The Rate of Growth in Posterior Odds as a Function oft theDifference, d, betweelkthelf4mber of Observations Favoring H1and the Number Favoring H2 in-the SymMetricalAcate

0

.'2,0 The Sizef-d Required to Realize a Given Odds Ratio, as, aFunction of the Larger of tbe' Two Conditional Probabilities

''in the Symmetrical Case

#.21" 'Menges in Po'Sterior Probabilities p(HilD) as a Result of theIndicated Observationsof Red and glack Chips

* ,4104

0 .4c--

i22 Changes d.n Uncertainty as a'

Result of Indicated Observaitons pda , g

.23 Changes in Pairwise Odds as a Result of the indicatedObservations 107

,

ioo

101

24 Changes in Absolute Odds for Each Hypothesis.as a Resultsof the Indicated Observations 108

25 u Expected Value of Posterior Probability of H4,'diven that H2is True, as a Function of NuMber of. Observations. , 114

0

26 Graphical Representation of 'Derivation of p(4[114.)andAdjusted Likelihood` atios for Le'ss than Completely ReliableData 7, , .. f 130

27 Adjusted Likelihood Ratio (A) as a Function of Data ReliabilityCr) for Several Values of Unadjusted Likelihood Ratio (L),

- for Sebum and.DuCtiarme's Case.I. .

28 A and W as Functions of r for SeVeral Values of L 140

29 The Ratio, of.

n to A as a Functidn of r for Several4Valueal

Aof L -.) .

.4

ix /xr

. , 13

141

,

NAVTRAEQUIPCEN

V

SECTION I

INTRODUCTION

%

'Much,has been written about the importance of4

decision makingfor industry, for government, for the military and for rational--or

at least reasonablepeople in general. Moreover, a.great deal ofreseakchshis been conducted on decision-making behavior. In spiteof these factsor perhaps because of them --there `is not generalagreement concerning what dec4ion making. is, how it should be done,

how it is done, how to tell.whether it is .done well' or poorly, and

how to train people to lo it better.

The.. term "decision making" has'been 'applied to a very broad

range of behaviors., The detedtion oT weak sensory stimuli has been

-viewed, in part, as a decision process (Green & SWets, 1966), as

has perception} by huans more. generally (Bruner, 1957).. Pattern

classi4cation by machines (Sebestyng 1962)/ the retriseval, of

information from memory (Egan, 1958), the performance of skilled

tasks such as automobile driving ',(Algea, 1964) and airplane pilot-

ing (Szafran, 1970), the production'of speech (Rochester ,& Gill,

197 ), educational counseling (Stewart & Winborn,,l9)73), the pur- A

cha ing of industrial prodi3cts (Reingep, 1973), the evaluation of

the erformance of salesmen .(Sheridan,& Carlson, 1972),-and the

. con ucting of 'a laboratory experiment ,(Edwards,. 19,56) -are'also

.representative allthe types of processes that have been discussed

under.the rubric of decision making. Probably when the term is used

in industrial, governmental and military contexts, however, what

the user has in mind '.s something close -to,what SChrenk (1969) 4

.describes',,as "situationscharacteriZed by fairly well- defined

objectives, significant action alternatives, relatively high

stakes, inconclusive informatioh and limited time for decisio

(p. 544). We hasten to add that to-limit one's attention-tosituations that-Have all of these characteristics would preclude

consideration of the large majority of,enDerimental'±nvestigations

'of decision,making;.in particular, in very few laboratoy studies

of decision making have the stakes been,high; and one may question

in many--if not most--cases the sigpificahce of the action alter-

natives to the experimental subjeeEs. It does not necessarily

follow that the results.of laboratory studies have no relevance to

real-life decision'making, of course. The dggree to which one is

willing to extrapoldte from the one situation. to the other depends

4 incOnsequentidl decision-proble s are-solved--at least in principleon the extent to which one subscribes to the view that simple and

--in the same ways as are'those that are complex and consequential.

. As Schrenk(1g0) has pointed p ut, there are three ways to

improve the performance of the human decision element in a system:

`(1) selection (insui'e'that decisions are made only by individuals

who are competent to make them), (2) training (attempt to improve

the decision-related skills,of people in decision-making positrons)1

'

14

NAVTRAEQUIPAN 73-C-0128-1

and (3) decisi4 aidiiisq (provide decision makers with proceduraland technical adds to'compensate for their own limiiations). Tothe extent that performance of a decision - making system is of in-terest, as opposed to that of a human being, another possibilitythat deserves consideration is that of automation (have Machines

,perform those decision tasks that they can perform better than'Fieople). ".

The nUmber.of -asks that are now performed by machines thatA were once thought to require humin skills is growing and will

continue to do so. Many tasks that involve decision making bysome definition should.be--indeed, many have been--automated.There' is little justification for wasting a good,human brain tomake what Soelberg (1967) calls "programmed decisions," decisionsthat are made with suffidient frequency and unddr sufficientlyspecifiable conditions to permit the detailed description of pro-cedures for making them. Thermostats, governors, regulators,stabilizers, Complite.r algorithms, and such things, are the pre-ferred "depision makers",for these types of situations. Thesituations with which we are priMarily concerned are not of thisstraightforward programmed type. They are situations that arenovel, unstructured or unplanned fob, or,they involve human pref-erenpes that are, not easily ,specified, or potential action con-sequences that are not .known with certainty. Clearly, thesetypes of situations arse the more interesting objects of study,and are probably more representative of what people view as bonafide decision making. .

.. It is imPortant to recognize that: the objectives of muchdecision- making research are to make novel situations less novel

4by pro.O.ding prototypes in terms of which the novel situations

. can be perceived, to facilitate the imposition of structure onsituations when apparent structure is lacking, and to providetechniques for qecreasing the probability of surprises and for

,coping with unplanned-for situations as though they had beenanticipated all along. But the reader who might think that suchobjectives could, if realized, take the chatm out of decisionmaking Maydrest easy'. There seems little danger of success tothe point of reducing all decision making to an algorithmicprocess in the near future. Indeed, there are some aspects ofdecision making that men may never feel comfortable turning overto machines. Hence, the needs for selection, training and de,-cision aiding are still real, and are likely to continue to befor some time to come, Moreover, as more and more of the pro-cedurizable tasks that were once performed by men do become auto-mated, the tasks that are' left to be performed by men--or perhapsby men and machines in collaboration--take on added interest andsignificance by virtue of their very ipsistande to automation.Should not those tasks which seem to require the attention ofhuman brains be the tasks that hold a unique fascination for usas human beings?

a 2

, ,

S

NAVTRAEQUIPCEN 7,3-C-0128-1

The general question that motivates this study As the questionof whether individuals can.be trained to be effective decision

- makers in unprogrammed situations. And if the answer to thatquestion appears to be yes, the next question that presents itselfis that of how that training canbe accomplished most'-geffectively.Immediately, one is 1paito,more specific questions. Does it makesense toithink of decision making as a skill, or as a collectionof skills, that -can be developed in a sufficiently general, way thatthey can be applied in a variety of specific contexts? What is itthat the decision maker -needs to be taught? Concepts? Facts?Principles? Attitudes? Procddures? Heuristics?

The literature on decision-making research is volumious, butdespite numerous references to the importance of the training of.decision makers (e.g.., Edwards, 1962; Evans & Cody, 1969; Fleming,197-07 -Hammell & Mara, 1970; Kanarick, 1969; Kepner & Tregoe, 1965;

Scalzi, 1970; Sidorsky & Simpneau; 1970), the number of studies

that have explicitly addressed the uestion of exactly what should

by taught an& how the teaching can best be accomplished is remark-

ably small. The central interest in the area continues to be

with. parameterization of the decision maker and his environment,_,/

and with generation of specific aids to'the decision prbcqss.

This review is not limited, therefore, to studies that have

focused specifically on the issue oftdecision training. We have

attempted instead to look at a rather broad cross section of the

.general decision-making research literature with a view to finding,

wherever we could, implications For the training of, ,decision 'makers'

and clues concerning "what further research might lead to moreeffective.training procedures or programs.

J

3

j

NAVTRAEQUIPCEN 73-C-0°128-1

SECTION ZI

SOME COMMENTS ON IECISIdN THEORY

One can distinguish two rather diferent approaches thathave been taken to the study of decision making. 'One is analyti-

cal; the other is basically empirical. A common goal of bothapproaches, however, is .the development' of formal models ofdecision processes. .In the firsoase-, one tries to analyzed/Acision situations--often hypothetical situatiOns--abstracting,from them their common elements. One then attempts to producee%model of the decision,.making process, using the Constructs thathave been identified in the process of an4ysis. In tile empiricalapproach, one begins by observing individuals making decisions inrealrlife situations, and attempts, on the basis of these obser-vations, to develop parsimonious descriptions of decision-makingbehavior.

Each approachhas its Strengths andits weaknesies.- Themodgl,s generated by analysis are likely to be more abstract thanthbse developed through observation. AS a consequence, they' aretypically more general. Howelier, there may be considerable dif-'fioulty in applying such models in specific cases. TN.'S is truebecausesreal-life decision situations frequently are not easily.describable in terms that an application of a model would require.

In contrast, a model of a decision-making process that is developedby observing decision makers in action is likely to be applicable,at least to situations highly similar to that front which the model

is derived. Such models may,lack generality, howevery and proveto be inapplicable outside the Context/in which they are developed.

2.1-. Prescriptive versus Descriptive Models

A prescriptive model indicates what one should do in a given'idecision situation; a descriptive model is intended to describe

what one actually-does. Typically, prescriptive models are theoutcomes of analytical approaches to the study of decision making,whereas empirical approaches generally lead to descriptive models,In theory at least, a prescriptive model may be used either as aguide for decision makers or as a standard against which to assessthe extent to which,.. decision-making performance approaches opti-

mality. Descriptive-models differ from prescriptive mbdels inso-far as human decision makers perform in a lessnthan optimal fash-

ions. ,Were a dedision maker to behave in an optimal fashion, adescription of his behavior would.constitute a prescrip4ve model.Comparisons between .prescriptive and descriptive models can beinstructive in suggesting the reasons why human behavior is some-

times not optimal.

Prescriptive models are generally associated with economistsand mathematical statisticians. Among the ddveloper and expositors

194

qrikVTRAEQUIPCEN 73-C-0128-1

of prescriptiVe decision theory are, Bernoulli (1738), Neyman andPearson (1933), Samuelson (1947),' vonNeumann and Morgenstern(1947), Wald. (1947, 1950) Good (1952), Blackwell and Girshick(1954), Savage (1954),.. Luce and RaiOa (1957), and SChlaifer (1959) .Such models typically postulate^an"economic, or at least a

0 "rational," man who behaves in a way that is 'entirely consistentwith his decision objectives and who does not have some of thelimitations of-teal people.,

Descriptive models wer introduced primarily by psychologistsand other students Qf human 'behavior, notably Edwards,(1954, 1961);PetersOn, Birdsall, and Fox (1954).; Thrall, Coombs, and Davis(1954); Simon (1954, 1955); Tanner (1956); Davidgon, .Suppes, andSiegel (1957); Festinger (1957); Luce (1959); Siegel.(1959);Rapoport (1960); Estes (1961); and Edwards, Lindman, and Savage(1963): The objective in this case has been to discover byexperiment And observation how human beings, given their limita-tions,'perform in decision-making situations. It is importantto no'e that descriptive models have been viewed' as descriptiveonly of the behavior of the decision maker, and not necessarily -

of the\thinking that leads to that behavior. ,For example, thefinding that an individual's choice between two gamble 'can bepredicted on the basis of which has thezost favorable "expected,outcome" is not taken as evidence that in making the choice the,individual actually goes through the process of calculatingexpected values and picking the alternative with the largest one(Edwards, 1955; Ellsberg, -1961).

The two lines of development--prescriPtiveland descripgive-model§,--have not proceeded,independently of each other. Severalof the investigators mentioned above have made significant con-tributions of both prescriptive and descriptive types. Moreover,one approach that has been taken to the -study of human, limitationsis that of attempting to modify prescriptive models sb that theyare'in fact more dessriptive.. Typically, what this involves isthe imposition of constraints on the model that represent specificlimitations of the human. For example, a prescriptiVe model that.assumes an infallible memory unlimited capaCity is unlikelyto be very descriptive of WM behavior; to modify such a modelfor the purpose of increasing its descriptiveness would necessi-

rtatb at least the a ition.of some constraints that representsuch factorsAs a mitation on memory capacity and degradationof stored inifoxitation over time.

i

'

The distinction betweenprescripitive and descriptive modelsis sometImes blurred in the literature and one cannot always besure in which way a proponent of a model intends for it to betaken. On the other hand, many writers have observed that themodels deriving from theories of economics do, in fact, fail todescribe behaVior,'or at least to d6 so very accurately. Miller

r-5 18

.

.0

P NAVTRAEQUIPCEN 73-C-0128-1

,--

. . .

and Starr (1967) poigt out, for example, that in the economist'sview of decision.making, the objective of the decision ffiaker is tomaximize the "'utility" that he can achieve within the limitations,of his resources. They ,note', however', that the lssumption thatindividuhls to act so as to maximize utility has been cftaliengedby many investigators of decision making. If rationality isdefined in terms of the extent to which behavior is appropriateto the, maximization of utility, theysnote, then when people do

/

not maximize utility,' they, by definition, are acti,pg irrationally. ?".

Miller and Starr list several factOrs that have been suggested aspossible reasons for the failure of'decision makers to behave inan optimal way: "the iiability of the individual to duplicate therather. recondite'mathematics which economists have used to solvethe problem of maximization of utility; the existence of othervalues which, though apt readily quantifiable, do cause divergencesfrom the maximization of utility in the marketplace; the effect ofhabit; the iifluence of social ertiulation; the. effect of socialinstitutions" (. 25).

,/,,, ) k \

"While interest in prescriptive models stems at least inpart from the 'assumption that they can provide guidance fordecision makers in real-life situations, their application oftenproves to be less than straightforward. Haythorn (1961) notesthe difficulty that operations analysts and operations researchersoften encounter in trying to analyze decision situations-in .com-

applied. He ascribes the difficulty to sever 1 factors: "First

.com-plex organizations to the point that prescriW49 models can be

is the fact that organizations are constructed by men with some'purposes in mind, although these are not usually stated veryexplicitly. Analytic solutions must assume that the decisionmaker is rational, that the parameters relevant to the decisionare quantifiable, and, that the information necessary to make anoptimum decision is available. A careful look at the view ofthe world held by critical decision makers reveals that they areby no means completely rational; that some of their objectivesare not easily quantifiable, and perhaps even incompatible withother objectives; that they do not have all of the informationneeded in many cases; and that frequently the informatiOn theyhave is inaccurate",(p. 23). 5

Schrenk (1969) has argued that progre\ss on the developmentof 'technique's for aiding decision makers w 11 be impeded untila model of "optimum" decisibp processes th t makeS realisti.cassumptions about human capabilities is fo thcoming. Such amodel, Schrenk suggests, should reflect the behavior of " "reasoningman," 1 concept that 1.i4 istinguishes from the rational man of

ILeconomic decision theory. "The idea is not to specify an 'ideal'decision procedure which '11 prodube perfect choices in abstractor laboratory situations, but rather to develop a process thatwill yield better decisions in real situations" (p. 548). Schrenk

6

.

t.NANTRAE'QdIPCEN 73-C-0128-1

1

sees foga: p urposes that such a model might serve: (1) it couldproVidea framework for the classification and integration of theresdlts of decision-making research; (.2) it comad provide guidancefor further research; (3) it. could help syTia/designers to,'structure depisign tasks and to allocate de ision functions tomen Nand machiAbs; and (4) it could.help guide the development ofdeci.gion-aiding concepts.

. %

'2.2 worth, . Prbbability and Expectation

Sometimes a Acision mAker klas the tesk of choosing one fromarming several alternative courseS of acti n, knowing what theeffect of any choice would be. (This situation, which is referred -

to.as decision making under certainty, is ditcuss'ed in Section IX.)Often, however, one must make a choice when t consequences ofthat choice cannot be anticipated with.aertai y. In the lattersituation, the decision maker is said to be m ing a decision .

"under risk." The most common,way of dealing th risky dtcisionsquantitatively has been with models ' that make use of the Conceptoft mathematical expectation..- )

. a

The "expectation" associated with a choice is calculated byc obtaining the product,9ffisome measure of worthtof. each outcome''

and a measure of the probability of that outcome, and summing overall outcomes that could result from the, choice of interest. It.has sometimes been assumed that the decision maker attempts tomake a chOice "ghat maximizes his "expected" gain. More precisely,it is assumed that the decision maker behaves as though he calcu-

. lated for each action alternative, the sum of the products of theworths and probabilities of the possible outcomes associated withthat alternative, and picked the alternati e for which this sum'was greatest. The "as though" in the preceding statement is,important. No one contends that decision makers, as a rule, reallyperform the arithmetic necessarf to compute expectation;` it isonly suggested that choices are made as though they were based onsuch ca culations.

. .

, Each ok'the factors in the expectation equatiOn,-worth andprobabilityvariable.

The four posSibie combinations of objective and sub-.can be treated as either an objective or a subjective

jeative indicants of worth combined with objective and subjective.measures of probability define four classes of-expectation modelstht have been studied. Table 1 gives expressions, in the nota-tion used by Coombs, Bezembinder, and Goode (1967), for expecta-tions representing each of these models. Much Ofe,- fie research ondecision making under risk has been concerned with determiningwhich.of these models is most descriptive of human behaNiior, andwith developing techniques for measuring subjective yorths.andprobabilities. t

r 20

NAVTRAEQUIPCEN -C -0128 -1

TABLE 1. FOUR BASICATYPES OF EXPECTATIONMODELB.

ModelType of°worthmeasure

L

;`

4 ExpatationType of associatedprobability with jtkmeasure possible

outcome

A.

Expectationassociatedwith kth e,choice (whichhal n possible"utcomes) r,

Expectedvalue

Expectedutility

Subjectivelyexpectedvalue ,

Subjectively".expected

utility

objective

subjective

objeCtive

subjective

objective pj vj

)

objective pi uj

subjective tPj vj

%

1

subjective .

Jd)

Pi

tPi

vi

uj

ti

an objective probability

a subjective piobaidlity

an objective measure of value (e.g. amount of

measure of worth (or utility)a subjective

I

1'

uj

-n

E ),Avj=i

nE tV u=-1

money)

NAVTIIEQUIPCEN 73-C-0128-1 /

I

The first of the models listed in Table 1, the Expected Valuemodel, is to least complex conceptually, and the most easily ap-plied, inasmuch as.both of 'its parameters are objectively defined.Although-bhis model has some appeal as a prescriptive model, ithas proved not to be generally descriptive of how real declionmakers behave (s s, for example, Coombs, Dawes, Tversky, 1970;Edwards, 1961;' ichtenstein & Slovic, 1971; iiichtenstein, Slovic,-& Zink; 1969). 4

The inadequacy of the Expected Value Model'as a descriptiveTodel'is.clearly illustrated"by the well-known St. Petersburgparadox. Suppose one were offered an opportunity to purchase thefollowing gamble. A fair coin is to be tossed until it comes uptails, at which time the coin tossing is terminated and the winningsare collected. If the coin conies up heads on the first toss, the-purchaser will receive $2.00; if it comes up heads on both thefirst and second toss, he will receive $6.00 (or $2.00 for thefirst toss and $4.003 for the sepond). More generally., if 'it comesup heads*fot n consecutive tosses`, he will receive $2.00 for the,

°first toss, $4.0C'for the second, $8.00 for the'third"... and $2for the kth, for 'a total of

n4 1 E Ik ddllars.

!II ta,-,

k=1.

Since, by definition, the successive tosses'are independent, theexpected value of this gamble in'dollars is siven by,

EV =1 1

' 2 + 4 + ... + 1 2n

+'... = 1 * 1 -12 1 4- ...4211

which is to say, it is infinite. If one were attempting to maximizeexpected value, therefore, one should be willing to pay a large ,

amount of money indeed to play thiS game. It would be surprising,however, if many people could be found who would be willing torisk'their life savings, say, which would be small by comparisonwith the expected gain, to purchase this gamble. In general, itis clear that the attractiveness of a gamble depends not only onthe expected value of the outcome but on such factors as the amountthat one coul possibly lose, and the nature of the distributionof probabiliti s over, the possible outcomes. In the gaMble de-scribed above, or example, the probability is .5'that the, purchaserwill win nothing, and .75 that he will win at most $2.Qb. -..

1

1 _spite of thd inadequacy of the Expected Value model siaenera 1 valid description of behavibr, it Should be noted tat

the model does a creditably good job of describing behavior, atleast grosply, in many ddcision situations. Evep in the case ofgambling behavior, it does not-invariapAy fail; "about 88% of thejob" of,explaining the behavior of the\Las Vegas gamblers studiedby Edwards, for example, could...be done on the basis of a knowledge

,) of the expeCted valUe of each bet (Rapoport & 111,allsten, 1972).

r

1-

11.

.

NAVTRAEdUT.PCEN, 73-C-0128-1

.

Implicit to the Expected Value model is,tIe assumption thatthe monetary value of a decision outcome represents its real worthto the decision maker, and tilat,..thig, worth is the same for all.individuals. Recognition that such an assumption is undoubtedlyfalse led to the formulation of the Expected Utility model in

,monetary value is repldbed by a measure of the "utility" of :.

an utcbme for the particular decision maker involved. Accordingto this formulation the. same tlecition outcome may appeal to dif-ferent individuals to different devees, and, consequently, prefer-enc'es.among decision alternaties with uncertain outcomes maydiffer from one decision maker to ano ther. The Expected Utilitymodel was first proposed by Bernoulli (4738) 'and aiven its modetnaxiomatic form by von Neumann. and Morgellstern (1947).

Given that the worthfactot,

r in the expectation equation isdefined as a subjective variable, the question arises concerninghow probability shouldAbe defined. Although a review o2 the con-,troversy would take us,Itoo far afield, it should be noted that thequestion of what'the concept of probability "really means" hasbeen the subject of'endless philosophical debate. It is suffid.j:ent.for our'purposes to recognize that statements of the type "theprobability of the occurrence of event X is equal to It" have, beenused in a variety of ways. Such a sEatement is sometimes used torefer to the relative frequency with which X has been_observedover the course of many similar situations. Or it,can have refer-enco to, a ratio in which the numerator represents the tcttal number6f-ways in which the outcome of an hypothetical experimentkcansatisfy some criterion and the denominator representsthe totalnumber oftdifferent outcomes (as when one says the proba-bility of rolling a 2 oriless on a fair die is 2/6).*" Sometimesa probability statement is uses to refer to the strefigth of one'sconfidence, or the degree of one's belief, that an event X, asopposed to the other events that are considered possibilities,will occur. It is this connotation thht we here refer to as"'subjective probability."

MN/

In some Situations it;makes 2ittle if any pra ticaldifferencewhich of these connotations one gives to the concept of probability,

*Related to this usage of the term is the so-called "Principle ofInsufficient Reason," whichclirects the decision maker to considerall possible outcomes tq be of equal likelihood in the absence ofinformation which indicates such a consideration to be inappro-priate. See Rapoport (1964) for an'interesting discussion of thelimitations of this prescription in defense of an assertion thatthe six faces of a die'are equally likely When one has no reasonto assert otherwise.

10 0

23

d's

1

N.

NAVTRAEQUIPCEN 73-d-012877.'

t L.

inasra4ch as they will all yield the .bame,numii,erS. Most peoplewoul perhaps agree, for example, that the probability of tossing

1 headd on a fair coin Is .5,'irrespeCtive of their philosophibal,

position concerning how probability should be defined. ManyG "probabilistic" situations of interest to invesbigaters of ddci4

sion making do At easily admit of an analysis in terms of rela-..tive frequencies, dr even of theoretiAl ratios, however, and itis perhaps for this re4son that many decision theorists subscribeto the notion thatfprobability is best defined" in terms of-degreeof belief. Rapoport (1964) defends this positionithe following way."We are told'that decisions involving the probability of the but -break, of a nuclear war'are based on 'calculated risks,' by which

ilities. Since of an Oent suchterm those who recommend or makeeciions must imply c#1culationg

4 involving probabas the outbreak of a nuclRar war can have nothing to do with thefrequenc such events 'tsinc at this writing none has occurred,and, i 11 likelihodd; no mor than very few can occur), either'the rase the probability of a nuclear war has no meaning at

1011

. all, in.which case the notion of the 'calculated risk' iS onlyeyewash, or else 'probability' has another meaning, having nothing.whatsoever to do with frequency" (p. 25.

The argument that-probability often'cannot be defined mean-

411ingfully in terms of relative frequencies or ratios' is a strongone for, resorting ito a definition in terms of subjective uncer-_tainty. Even when an objective definition is easy to comeoy,however, one may question, whether it should be used by any,theorythat purports to be djcriptive of the behavior of real decisionmakers. It i the decision maker's own expectation that pre-

." sumably important in determining his behavior and his expectationmust be calculated in terms of the,probabilities as he perceivesthem. Moreover, it is required of a rational man that his behaviorbefconsistent with the information at his disposal, but not thathe have perfectly accurate informatiori. Thus, two'decision makerscould behave optimally.,..,but quite differently, in the same situa-tion if their perceptions_of the situation differed, a fact thatis easy to accommodate wheriprobability is detailed as degree ofbelief but nbt when it is defined strictly in terms of the ob-jective details of the situation.

In the foregoing discussion of Expected Valup and Expected. Utility models, it was tacitly assumed that the probability factorin the. expectation equation was objectively defined. As suggestedby Table 1, two additional types of expectation models might berealized by combining subjective probabilities with both objedtivoand subjective measures of worth. 'The resulting models might bereferred to, respectively, as SUbjectively Expiected Value and Sub-jectively Expected Utility models: Although both of these types of c>models have been considered, the latter is by far the more widely ac-

11/-cepted,and.used. This model has been presented by Savage (1954) andby Edwards (1955). Among the four models liSted in Table 1 which

11

24

4.

tiave been referredit has 'receive theat the moment, ran1972).

44

I ,NAVTRUQLTCEN 73-C-0128-1

4

o as singie-stge algebraic decision models--reatest amount of empirical support, andas the most influential (Rapoport & Wallsten,

V Savages (1954) formulation of decision theory iderAPies anuMbet of "seemingly agreeable" (Tversky, 1969) rules thatshould be satisfied b?foge it is appiopriate to assign a singlefixed number denotingworth to each possible decision outcome anda single fed number denoting judged4aikelihood,of occurrenceand then to'select maximum products. These rules (set Becker.&McClintock, 1967)" 4,re\as follows:

I-4 Ru 0 1: Transitivity. If, in 'a choice situation, the de-

cision aker prefers Outcome A to Outcome B and Outcome B toOutco C he should prefer, Outcome 4 to Outcome C.

4

Rule 2: Comparability. The decision maker should b' willingto compare two possible outcomes and decide either that he prefersone to the other or that he has ho preference between them.

Rule 3: Dominance. If thedecision make determines that,under every possible Condition, a' choice of 'on of flis alternative

t agtions resultsin an outcome at leasta8 de irable as that whichwould result from the,choice of a second alt rnative action, and,results in'a more detirable outcqme'undqr at least one possiblecondition than would ,the second pCitiori, the second action shouldnot be preferred tb the first.'

.

Rule 4: Irrelevance of-nonaffected outcomes. If the de-cision maker determines, that, for a phrticular state of the world,two or more of the actions open to him result in the same outcome.,his preferences among such actions should'not be affe6ted by the'outcome associated with that stater

Rule 5L Independence of be liefs and rewards. The decisionmaker's statement concerning the likelihood of occurrence of.agiven outcome shouldinot be affected by wha he hopes will occur.

Some of these rules seem to'be honored as much in the breachas in the bbservance4(see, for example, MacCrimmon, 1968). Vio-lations of Rule 1. are of maj signifibance. This is so becausethe assumption of transitiv y of preferences is a necessaryrequirement for the-cons ction 6f a consistent ordinal utility

. function. (For a discussion of the problem of generating utilityfunctions from preference j'udgments, seeRoberts, 1970.) Tversky11969) refers to transitity,as "the corperstqne" ofirdecisiontheory and points out that it underlies measurement models ofsensation And .value As well., He also notes that decision'makersoften do violate the transitivity rule in specific situations.

,e)

1225

NAVTRAEQUIPCA EN 73 -C -014-41--

I* -1/4

Another rule which seems difficult to satisfy is that_re-quiring independence of beliefs, and rewards (Rale 5). MacCrimmon(19.68) has found a strong dependency between an individual'sestimates of. the likelihoods of events and his fitastes"--theworths he assigns to those events. AS might be anticipated, suchan association may pose difficdlt analytic problems, since, fora given set of choices, one cannot assume that a distribution of(stated) preferences arises simply opt of differences associatedwith bPt one of the two parameters,in the expectation equation.In principle, this problem is similar to the so-called "Conjointmeasurements problem" which has received major attention in thecontext of Subjectively Expectdd Utility theOry.

--A-variant of the dominance principle (Rule 3) has been statedby MacCrimmon so as to apply ,to the problem of comparing alternatives that'differ with respect to several attributes.whenpreferences can be stated with resp.tct to single attributesindividually: . "When 'comparing all alternatives, if some alter-native has higher attribute values for all attributed, we saythat this alternative 'dominates' the others. We can weaken thisnotj.on somewhat and pay that if one alternative is at least s

good as the other alternatives on all attributes, and is act lly

better on at east one of them, then this can still be considered

the dominant 'a ernative. Conversely, `if one alternative isworse than some ther alternative for at least one attribute, and

is no better than equivalent tsor `a.11 other attributes, then we

can say the former alternative is dominated by the latter0 (p. 18).

Schewriters have noted that the dominance criterion is inconsis-tent with the makimin Criterion of game theory (Marschak, 1959)Luce & Raiffa, 1957). Ellsberg (1961) has discussed additional

S

s f

I

problems with this rule.

Some other assumptions that have usually been considerednecessary to the use of expectation models are the following: D.)that the act of gambling has no utility iteelf;0(2) that the sub-jective probabilities associated with the alternative decisionoutcomes sum to nity; (3)..that preferences are independent of the

method by which they are'meadsured. It has not been possible'todemonstrate that the first two of these.assumptions are simulta-

neously valid. Moreover, SloVic (1966) and others (Lichtenstein& Slovic, 1971; Lindman, 197.0) have shown that' preferences amonggambles may indded depend in part on-the method by which they

are obtained (e.g., rating, procedure versus a bidding procedure).In spite of these limitations, expectation models, and in par-

. ticular the Subjectively Expected Utility model, have proven tobe reasonably predictive of at least certain types of choicebehavior (Coombs; Bezembinder, & Goode, 1967). They clearly donot, however, tell the whole story of how to account for human

chOice behavior.

r - 13-26

O

NAVTRAEQUIPCEN 73-C-0128-1.

The demonstration that expectation models such as those de-scribed are unable to account for choice behavior'.donsistentlyand completely has led some theorists to seek to modify (which hasinvariably meant to'complicate) the) models to make tIem more de-scriptive. Other-theorists have simply rejected them out of hand.Payne (1973`) points out that Models1such as those we have con-sidered involve the representation of risky alternatiVes as proba-bility distributioA over sets of decision outcomes, and attributethe choice among the decision alternatives to some function of eachdistribution's central tendency. In the hope of developing modelswith greater predictive power, some theorists have looked not only )

to central tendency measures, such as expected or mean values,bl4t to variances and higher moments of these distributions as well(Becker & McClintock, 1967). Still others have made modificationsthat relax the requirement *that the decision maker's choice be .0'

N_invariably dictated by which of his alternatives representS thegreatett expectation; ."random utility" models have been proposed,for example, which assume that the utility of a given outcome'isa random variable and that variations in this variable producevariations in choice (Becker, DeGroot, & Marschak, 1963).

S

Shackle's (1967) assessment of 'expectation models is represen-tative of the opinions of theorists who reject such model.s out ofhand. He argues that the concept of mathematical expectation,and, indeed, the concept.of probability as well, are irrelevantto' the assessment of one-of-a-kind decition situations. Further-more,',he contends, most real -life decision situations of interestare, to those who face them, unique events; never before has theindividual beemcalled upon to.make exactly the choice that hefaces And never again will he have to select from among the sameset of action alternatives under precipely the same circumstances.In such cases, Shackle argues, the decision maker is concernedwith what can happen as aresult of his choice, not with whatwould happen if the experjment were repeated a large number oftimes "he is concerned with possibility'and not probability"ip. 40). We sho d note that the argument implies a relative-.frequency conno ation of probability, a connotation that not alldecision'theori is accept.

'4

Miller and Starr (1969) suggest that one can always find away :s View a dedision problem as maximization problem if onewants to do so: the quantity at, the decision maker wishes tomaximize is the degiee of at ainment of his objective. But thisis not Awry helpful as a definition: indeed, it comes close tokeinq tautological. Millet and Sta22 apparently do not intendto assert as an empirical fact that decision makers 4o attempt tomaximize anything. More generally,* whether decision makers attempt Ato find optimum solutions to their decision problems Miller andStarr consider be questionable. Simon (1955) has taken theposition that they usually do not. According to his "principle

N47T1AEQUIPCEN 73-C-0l262lk

of lounded rationality" what they do instead is to define a -

, limited,set of acceptable, or "good enough," decision' outcomesand then select a strategy that they consider to be likely toachieve one of these.

The current status of expectation models among investigatorsof decision making is, reasonably well summarized by three obser-vations. (1) The models that are seriously advocated as descrig-tive,of human behavior are rather more complex than the straight-forward Expected Value model that was origihally proposed:. ,Thehistory of the development of expectation models may be fairlycharacterized as a progression from the simple'to the more complex:objectively defined yariables have been replaced with variablesdefined in subjective terms, and the, number of_model parametershas been increased. (2)- Even the most complicated models havenot proven to be totally descriptive of behavior and some theoristshave challenged the validity of the basic, assumption of this classof modelA, na,nely that the decision maker is motivated to maximize.an expectation, no matter how the,factors from which expectationis computed are defined_ (3) Their limitations notwithstanding,expectation mGdels--even the least sophisticated*Expectea-ValueModel--do a reasonably good job of predicting choice behavior inmany situations. The challenge is to come up with models that

*"can handle the situations for which these models fail, as wellas those for which they succeed. Meanwhile, when'the maximizationof expectation is recognized as the decision objective, thenexpectation models can be used prescriptively to guide thedecision process. -

2.3 GA-me Theory

The theory of games was developed to deal with situationsin which the outcomes of an individual's decisions depend not onlyupon his own actions but also upon those of one or more "opponents"--decision makers whose objectives con+flict to some degree withhis own. Of special,,interest is the so-called "zero-sum" situa-tion -in which the worths of the outcomes to the opponents sum tozero; one loses what another wins. ,A,commonly prescribed strategyfor each "player" of a zero-sum game is to make choices in sucha way as to minimize his maximum possible loss, the so-called,minimax rule.

The assumptions,of game theory are open to a number of criti-cisms. Shackle (1967), for example, characterizes the theory dfgames, as developed by vonNeumann and Morgenstern, as "essentialla study of the logic of hoW to present as impregnable a frontas possible to an infallibly, wise and rational opponent" (p. 61).The assumption that one is in a confij.ct and that one's opponentis rational and infallibly wise leads directly to the minimaxdoctrine. Shackle questions to what extent this conceptualization

3g8

NAVTRAEQUIPCEN 73-C-0128-1

can be taken as a reasonable approximation to reality. Is theimpersonal world of nature or even that of business actively con-cerned to defeat us? Is the human opponent reasonably assumedto be infallible? Is there no essential and inerae.icable uncer-tainty in the outcomes of such few big experiments, large in timescale in comparison with the human life-span, that any of us hastime to make? Rather than minimax our lOsses, is it not morereasonable to fix for them some maximum tolerable numerical size,to avoid any action-schethe which would bring losses larger than .

this within the range of possible or,ftoo-possible' outcomes, andsubject to this, constraint to choose that action-scheme whichbrings within the range of,possible br 'sufficiently possible'outcomes, as high a positive success as we can find?" (p. 65).

In a similar vein, Beckr. and McClintock (1967) questionwhat they refer to as game theory's "principle psychological assump-tions." They point out that the theory assumes, on the one hand,that both decision makers will attempt to maximize their awnutility and, on the other hand, will attempt to minimize theirmaximum losses. These assumptions are, inconsistent unless thedecision makers lock at the game from each Other's points ofview--a requirement which Morin (1960) finds unsupportable onempirical ground - -and unless the utilities of each decision makerare known to the other and sum to zero for each possible outcome.

Despite its limitations, game theory has provided a valuableframework within which to view decision making.in suclAr-fields aseconomics, political science, social psychology and militarystrategy. The theory has been.extended to cover non-zero-sumsituations, situations permitting cooperation or collaboration

, among subsets of players of multi person games. In addition tominimax, other strategies have been identified as either prescrip-tively appropriate, or descriptive of behavior, in particularsituations:

A short and very readable exposition of, the basic conceptsof game theory may be found in Edwards (1954). A comprehensivetutorial treatment' is provided by Luce and Raiffa (1957).

2.4 Decision Theory and Training

It is a reasonable question to rase whether one may hope tobe an effective decision maker in a variety of situations without'some intellectual appreciation for the decision-making process,as it is, represented by theoretical treatments of decision making.One would guess that there would be some advantage to being famil-iar, at'least with certain of the key concepts that decisiontheorists employ. In practice, this would mean providing would-be

c "4ecision makers with a basic introduction to probability theory,a.twell as a working familiarity with notions of rationality,'value, utility, mathematical expectation, risk, risk preferences,and so on.

16

29

S.


In fact, one could make the case that failure to provide anadequate grounding in theory might deprive the decision maker ofthe sorts of insights that would lead to productive use of avail-able decision-aiding techniques. The demonstration by MacCrimmon(1968) that decision aids developed in quite disparate contextscan be effectively brought together in the solution of problemsinvolving multi-attribute alternatives, suggests the utility ofbroad acquaintance with basic concepts and principles.

In reporting one effort,to develop a system to assist cor-porate decision makers by enabling, them to manj.pulate parameters'(entered as distribution functions) on preprogrammed tree mOdtls,Beville, Wagner, and ,Zanatbs (1970) made some observations thatare relevant to this point. They noted that the use of subjectiveprobability distributions as inputs to models is novel even toexperienced decision makers, and must be carefully, taught. Moregenerally, they concluded that a black-box approach to utilization'of'the system would have been markedly inferior to one in whichthe workings of the system were explained to the user.

The teaching of decision theory should, of course, distin-guish what is intended to be prescriptive from what is considereddescriptive of the behavior of huthan decision makers. It shouldalso clearly identify the limitations of the models that areconsidered. Tutorial treatments of decision theory and sametheory are readily available sources of training material (Edwards,1954;'Edwards & Tversky, 1967; Howard, 1968; Lee, 1971; Luce& Raiffa, 1957; Miller & Starr, '1967; North, 1968; Rapoport, 1960;Schleifer, 1969). A comprehensive bibliography of research reportshas been prepared by Edwards (1969).

Whether familitization with theoretical treatments of de-cision making will In fact improve decision-making behavior is aquestion for empirical research. Our guess is that the answerwill be a qualified yes. Such training will be efficacious forsome people performing certain types of decision tasks but perhapsnot for all people or all tasks. One objective of training researchshould be to identify those conditions under which such trainingwould be effective and those under which it would by a waste oftime:

4.4

17

204".


SECTION III

CONCEPTUALIZATIONS OF DECISION SITUATIONS AND TASKS"

Numerous ways of conceptualizing decision processes have beenproposed by different investigators: Some conceptualizationsemphasize differences among decision situations; others focus onthe tasks that decision makers are required to perform. All ofthem have the same purpose, namely that of simplifying the problemof thinking about decision making by identifying a'few "types,"each of whichsis representative, in terms of some critical aspects,of a variety of specific situations or tasks. We review briefly inthis section a numberof proposed simplifying conceptualizations.The,re ist,x10 attempt to be exhaustive. The intent is simply toillustrate by means of a few examples some of the ways,, in whichinvestigators of decision making have characterized or categorizedthe object of their S 1study. -

3.1 Classifications of Decision Situations or Decision Types

3.1.1 Edwards

Edwards (1967) makes'a distinction between static and siElais_decision situations. In the former case, a one-time decision isrequired, whereas in the latter, sequences of decisions are made,earlier decisions and their outcomes having implications for sub-sequent ones. Six types of dynamic decision situations aredistinguished on the basis of such factors .as whether the environ-ment is stationary or nonstationary, whether or not the environmentis affected ley the decisions that are made, and whether or not theinformation about the environment is affected or controlled bythose decisions. Edwards further classifies psychological researchrelating to decision making under four topics: informationseeking, intuitive statistics, sequential prediction, and Bayesianprocessing.

3.1.2 Howard

Howard (1968) characterizes decision situations in terms ofthree orthogonal dimensions: degree of uncertainty, degree ofcomplexity (number of relevant variables), and degree of timedependence. The various combinations of the extreme values onthese dimensions are taken as representative'of eight prototypicalsituations, for each of which there'is an appropriate set ofanalytidal tools. An example of a deterministic (no uncertainty),single variable, static (time=independent) problem would be todetermine the largest rectangular area that can be enclosed witha fixed amount of fencing. The appropriate mathematical tool wouldbe the calculus. Decision problems like assigning customers towarehouses or jobs to men would, in Howard's taxonomy, be in thecategory defined as deterministic, complex (many variables), and

18

31


static. Matrix algebra and-linear optimization are appropriatemathematical techniques.

3.1.3 Sidorsky

Sidorsky and,,his,colleagues have proposed a taxonomy of typesof decisions encountered in military situations (Sidorsky,Houseman, & Ferguson; 1964; Sidorsky & Simoneau, 1970; Hammell &Mara, 1970). The acronym ACADIA is used as a mnemonic for the sixtypes of "situational demands" identified by the taxonomy:Acceptance, Change, Anticipation, Designation, Implementation, andAdaptation.

An acceptance-type decision has to do with applying data tothe acceptance or rejection of a hypothesis concerning some char-acteristic of the enemy. Detection, classification and localiza-tion are associated operations or objectives. The acceptance-decision idea seems to be close to what some other investigatorshalm referred to as situation diagnosis. A change-type decisioninv4iies the decision. maker in a choice between initiating a newtactical operation or continuing the course of action on which heis already launched. An anticipation-type decision is requiredwhen a decision maker must predict what the state or intention ofan enemy force will be sometime in the future.

A designation-type deCision involves the choice of one fromamong a seeof possible action alternatiVes. An implementdtion-type decision has to do, not with the _selection of an actionaltqrnative, but with the determinatioh of the proper time to'execute it. An adaptation -type decision is called for when thedecision maker is faced suddenly with unexpected and perhapspotentially disastrous circumstances.

3.2 Classifications of Decision:Tasks

3.2.1 Howard

Howard conceives of the decision process as ing composedof threelphases: (1) the deterministic phase, (2) the proba-bilistic phase, and (3) the information phase. In the deterministibphase, the decision analyst identifies the state and decisionvariables and constructs a model of the decision problem. In theprobabilistic phase, he assigns.probability distributions on thestate variables. In the information phase, he det6rmines whatadditional information should be gathered to reduce uncertaintyfurther. Howard estimates that the first phase represents about60% of the total effort of the decisiort maker, while the secondand third phases represent about 25% and 15%, respectively.

r - 19

32

3.2.2 Adelson

NAVTRAEQUIPCEN 73 -0128-1

4e.

A taxonomof decision tasks that are carried'out in modernmilitary command-and-control systems is proposed by Adelson (1961).Four, types of tasks are distinguished: (1) characterization of thestate of the world, (2) determination of the available actionalternatives', (3) outcome prediction and (4) choice rationalization.The first task type refers to the need of the decision maker tocharacterize the current state of the world in a way .that isrelevant to his decision problem. The definition of the variablesin terms of which the characterization should he made, and theassessment of the relative stability of the world that is beingobserved are seen as significant problems. The second task typeacknowledges the need to make explicit the courses of action thatare open to the qecision maker. The difficulty -6f thiS.-task maydepend somewhat on how rapidly the situation is changinei and on thecost of obtaining information. Outcome prediction refers to theprocess of attempting to anticipate what the consequences would beif specific.action alternatives were selected. ehe final task typeinvolves the need to justify one's choice of action in terms of theobjectives of the command - and - control system.

3.2.3 Drucker

Drucker (1967) has identified six steps.that he considers tobe involved in the, process of making the types of decisions thatconfront business executives: (1).the classification of the problem,(2) the definition of the problem, (3) the specifications which theanswer to the problem must satisfy, (4) the decision as to what is"right (as distinguished from what is acceptable in order to meetthe boundary conditions), (5) the building into the decision of-the

1

action/ to carry it out, .and (6) the feedback hich tests thevalidity arid effectiveness of the decision ag inst the actual courseof events.

3.2.4 Soelberg

Soelberq's (1966) taxonomy, like Dnucker's identifies sixaspects of the decision making process: (1) problem recognition,-(2) problem definition, (3) planning, (4) search,. (5), confirmationand (6) implementation.

3.2.5 Hill and Martin

A model proposed by Hill andtMartin (1971) also recognizessix different categories of activities in the decision-makingprocess: (1) identification of concern, (2) diagnosis of situation,(3) formulation of action alternatives, (4) test of feasibility ofselected alternatives, (5) adoption of alternative,, and (6) assess-ment of consequences of adopted alternative. The model assumes that

20

.

4

i'NAVTRAEQUIPCEN 73-C-01228-1

the decision maker's behavior at each of these steps is influencedby what he knows Of the theory and practice,of decision taking aswell as by what he knows about the setting in which the decisionproblem exists. Hill'and Martin identify nineteen skills thatthey consider to be implict in these six generic activitycategorieA:

"1. Asking for and receiving feedback

2. Assembling the facts (including past experience as itbearson the decision)

3.-Identifying the courses,of action available

4. Identifying forces for and against the alternatives

5. Ranking and rating alternativeg (includes.putting a valueon applicable risk factors)

6. Assessing the people-task ratio

7. Identifying the latest and expected consequences of thealternative courses of.action

8. Determining the advantages and disadvantages of each action,.alternative

t,

. Testing the validity and effectiveness of the consequencegof the decision against the actual course of events to.evaluate the decision maker's judgffient and to modify hissubsequent deciSion-making behavior

10. Brainstorming action alternatiVes

-11. Classifying and defining thpproblem requiring a decision

12. Analyzing and evaluating stimuli and decisions coming in

4 from the outside

13. Defining the goal at which the decision is directed

14. Communicating the decision in written or verbal composition

1!5.. Identifying resources belring on the making of the deCision

16. Recognizing the need for a detision

17. Utilizing minor, relatively simple decisions to contributeto making the more complex one (includes determining thehierarchy of order in which minor decisions will-be dealtwith and coping with timing as alternatives come into focusand seemingly demand attention at 'the sameltime)

18. Obtaining informAion

19. Specifying the boundary conditions the decision mustsatisfy" (p. 433) .

-,1

NAVTRAEQU IPCEN 73 -C-01 i8-1

a3:2.6 Edwards

Edwards (1965b) lists the following thirteen steps that must'be carried out by any Bayesian decision system:

"1. Recognize theexistence of a decision problem

2. 'Identify available acts

3. Identify relevant states that 'determine payofffor acts

4. Identify the value dimensions to be aggregated0 into the payoff matrix

Y

'5. Judge the vayle,pf each outcome o n each dimension

6.. Aggregate value judgments into a compositepayoff matrix'

7. Identify information'tgourcegrelevant todiscrimination among states

Col lect'data frOm information sources

:9. Filter data, put.intO s- tandard format,and display to likelihood estimators

10: Estimate likelihood ratios (or somp.otherquantity, indidating the impact of the datumon the hypotheses)

11. Aggregate impact estimates into posteriordistributions

12. Decide among acts by using' pFinciple ofmaximizing expected value .

13. Implement the decision" (p. 142, Table 1).

Steps 1 through 5, and 7 and 10, Edwards suggests, are best per-formed by men, Steps 6, 11 and 12 by maChinesp,and Steps 8, 9 and13 by both men and machines. Steps 1 through 7 may be .done in

.

advance of the decision time; Steps 8 through.l3 must be done atthe time that'the decision is to be made. ,(See Section VIIi,fora discussion of Bayesian information pi.ocessing.)

2235


&

3.2.7 Schrenk

' A conceptualization ofrthe decision-process that we f dparticularly interesting is one proposed by Schrenk (1969). Themotivation for developing this qonceptualization was to prdvide arepresentation of the decision-making process that is prescriptivein the sense that it can be used 'as a guide for the structuringof decision-making tasks of man-machine systems,/but which does notmake unrealistic assumptions about human capabilities. The pn-ceptualiZation is viewed by Schrenk as tentative, and in need offurther development; however, even as it stands it provides thesystem designer with a great deal of food for thought concerninghow to allocate decision functions among men andmachined..

Three major categories of decision tasks, or phases'ol thedecision process are distinguished: (1) problem recognition,(2) problem diagnosis, and (3) action selection. EaCh of thesephases is further broken down into several components, and flow-

. diagrams are given which show, where the components appear in the'"overall process. The following is a paraphr.isihg of Schrenk'sdescription of each of these components.

.

Problem ReCognition: :Determination that a problemrequiring a decision eXists.

- Acquire information: Receipt of iniormation indicatingthat actual situation differs from the desired situation.

- Recognize objectives: 'The decision.makerls purpose ormission.

Perceive decision need: Perception of difference betweenobjectives and current situation; may result from changein situation or in objectives.

- Assess problem urgency and importance: Establishment ofpriority. of-problem, relative to other.problems demandingattention, and allocation of resources for Isolving it.

Problem Dia nosis: Determination of the situation that iscausang prob em.

- Define Eossible situations: Generation of hypothesesregarding situation.

- Evaluate situation likeliboods: Assignment of a prioriprobabilities to alterhatiTie hypothesed-

,

4e

r

NAVTRAEQUIPCEN

- Determine whether more information;iS needed: AssessMentof adequacy of information in hand; a continu.ng process.

- Identify possible data_sources: If more inforMation isdesired.

Judge value versus cost: To determine whether, or how,desired information should be acquired.

- Seek more information: Assuming value judged to be greaterthan cdst.

P

- Re-evaluate situation likelihoods: Iterate.

- Determine whether alternatives under consideration accountfor all the data: Recognition cifpossible' need to modify,set of hypotheses, being considered.

-'Make diagnostic decision: Selection of favored hypothesis,or possibly of small set of weighted alternatives.

Action Selection: Choice of course of action.-4--

- Define action goals: Specification of explicit goals,including interim or suArdinate objectives.

fEecifyvalue and time criteria: IdentificatiOn ofrelevant dimensions of multidim nsional goals andspecification of time constrai s within which decisionmust be made.

-1Weight decision criteria: Establishment of relativeimportance' of various decision criteria.

- Specify risk EhilosoEla: Specification of strategy ofaction selection insofa6 as it is dictated by considerationsof balancing risks against pdtential gains.

- input 222Iating_doctrine: 'Consideation of any rules ordodtrine by which the decision maker's behavior should beguided.

Geperate\action alternatives: Explicit listing of reasonableset or courses &F actian-agEn to decision maker.

- Predict possible outcomes: Specification of the possibleoutcome associated with each of the potential actionalternatives.

- Estimate outcome gains and losses: Determination of valueof pgble decision outcomes.

- Estimate outcome likelihoods: Estimation of probabilitiesof occurrence of possiEre outcomes for each actionalternative.

24

37

V

.1

'I

4

\NAVTRAEQUIPCEN 73-'C -0128 -1

- Evaluate expected values df actions versus their costs:Derivation, from preceding two EEeps,,Of expected value ofeach possible action, and estimatioxi of associated 'cost.

- Evaluate actions }a ristahilosophz: Assessment of eachaction alternative in terms of itsmiklEgis.Jo_r_the-----risk philosophy that the decisiorirTaker has adopted./

- Determine whether1 more information is needed: As underDiagnosis; a continaig question; newIiiTaiiitiOn might beuseful either for identifying additional action possibili-ties, or to improve predictions concerning possibledecision outcomes.

1

- Seek information: I f desired, and worth cost of acquisition.

Re-evaluate action alternatives: Iterate

,-. Determine whether best action is acceptable: Review of mostdesirable action alternative to assure its acceptability,in terms of the decision goals and oriteria, the expectedgains from the choice and the cost of making it.

.

- Choose course of action: ThendeciaRion."

- Implement action: Initiation 6Pwiltever steps arenecessary assure that the selected action is carried out.

t

? The main fault that we have to ifind with Schrenk's model isthat it may be overly elaborate.. It is doubtful that manyindividuals go through anything approaching this multistep pro-cedure in the process of making a decision. This is perhaps anunjustified criticism, inasmuch as Scbrenk intended the model tobe more prescriptive than descriptive. And whether such a modelcan serve as a proitotype 'procedure for decision makers to folloaremains to be seenk. fIn any case, the representation does serve theuseful function of making explicit many of the asgls Of decisionmaking and it stands asa-reminder that decision m king may beviewed.as a complex and multifaceted process indeed.

'\3.3 Decision Making as a Collection of Problem - Solving Tasks

We take the puition that decision making is best conceived'as a form of probldffi solving; or, more specifically, that itinvolves a variety of aspects each of which may be viewed as aproblem-solving task in its own right. In the most, general terms,the decision maker's problem is to.behave in a rational, or atleast a reasonable, manner. To be sure, the distinctive character-istic of'the specific problems"with which the decision maker deals*is the element of choice; he must at some point decide upon onefrom among two or more alternative courses of action. While the

-.25 28 I

a

1

e.

NAVTRAHQUIPCEN 7P-,O-01.28-1

act of choosing among alterngtives is central to decision making,--it is by no means the only problem--or even necessar the mostdifficult

,oneL-that the decision maker must solv . We wish to

emphasize the,importance of,making explicit the ther things thatmust be donecif one is;motivated to make the bes pbssibleorat least a satisfactory---,decisiont,given th6 res urces at one'sdisposal. In many real-life situations, the pro lem of choosingamong possible courses.of action is far simpler than that ofdiscovering what one's options are in'the first place, or ofassigning preferences to possible decision outcomes in a con-sistent way.. 'Also, the decision maker may find it necessary tomake many preliminary decisionb simply by 'WaS7 of setting the stagefor making the decision'mhich is his primary doncern. For example,he will want to reduce his uncertainty about the decision situa-

:. tion or about 141e consequences of the various choices .hat areopen to him.* However, the acduisitiOn of information takes time,

' and may be costly in other ways, so he will continually be facedwith the problem of deciding whether any additional informationthat he may wish to get is worth the cost of getting it.

It is clear from the foregoing that there are many ways toclassify the various tasks that tIle decision maker may be requiredto perform. The scheme that we flhd most satisfactory recognizeseight aspects of'decision making: rinformati gathering, dataevaluation, hypothesis generatiOn, problems cturing, hypothesisevaluation, preference specification, action s lection, and decisionevaluation.

This cdnceptualization has an element of arbitrariness aboutlitas does any other. There are four points that we would liketo make in this regard. First, the decision to dOnceptualize theprocess in terms of eight type's of tasks,. as opposed to some othernumber, is itself somewhat arbitrary, and reflects our own biasesconcerning what constitutes a useful level of Organization. Onemight _conceptualize the decision process at a much coarser leveland distinguish two major types of task's-- diagnosis and actionselection- -that would encompass all of those that we wish todistinguish. This app;oach has been taken by several investigators

i (Bowen, Nickerson, Spooner & Triggs, 1970), Kanarick, 1969;Williams & Hopkins, 1958)." Bowen et al. (1970) point out that inthe military, diagnosis is, the proper function of intelligence, and

.o.

NAVTRAEQUIPCEN 73-C-Di28-1

action selection that of command. t the other extreme, onemight attempt a much finer ,grained representation and identify amuch larger number of activities that a decision maker may be

. called upon-to perform. In this case, each of the tasks we haveidentified might be replaced with several more detailed tasks.These are not mutually exclusive approaches, of course, and we willhave occasion to consider how some of the tasks we have identifiedmay be further broken down: However, this level of analysisappears to..us to be the most useful one for our present purpose,and possibly for serving'as a'general framework in terms of whichto think abOut decision making as a whole.

Second, 'our taxonomy is not orthogonal. to other conc.Iptuali-zations such .as, those discussed in the precedng section. it haselements in common, with most of them. Indeed', the intent is notto' take issue with other taxonomies, but to propose one that rep-resents what, in our view, are the best aspects of all of them.

.

Third, we ,do not mean to suggest that whenever an individualgfinds himself performing,the role of a decision mak,he explicitly'runs through this set of tasks in serial fashion, or even that he%performs each of these tasks explicitly at all,-Mordover, whenhe does perform these tasks it is not necessgfily the case that heis fully aware of doing so. it is characteristic of human beingsthat they often can solve problems quite effectiVely4without havingany C;ear idea how they do it. This characteristic has been afrustration to researchers in artificial intelligence, who havefound it exceedingly difficult to program computers to perform sometasks that human beings seem to be able to perform with ease.What we do mean to suggest by the proposed taxonomy is that pll ofthese types of activities are implicated in decision making andthat any attempt at a thorough discussibn of the decision-makingprocess must take account of them..

Finally, viewing decision making as a problem - solving proCessthat is composed of several Phases or subrirocessesifemphasizes thefact that in any given decision situation, different decision taskscould be performed by different individuals or groups (or machines)."An implication for training is that it ma' be less appropriate to,think of training decision makers per.se than of training individualsto play specific roles in the decision-making prOcess. On the other.hand, there will undoubtedly always,be some situations in which allthe various aspects of a decision problem will be handled by thesame .individual. But whatever the case, there is perhaps somethingto be gained by making decision makers--or specialist members ofdecision-making groups7-aware of the Many facets of the general task.

""

27.

40


In the next few sections of this report, we consider each of,the compohents of our task taxonomy in turn. The order in whichthe tasks are discussed represents a nAura1 progression; however,in real life decision situations, an individual, in a decision-making system, may "perform several of these tasks more or lesssimultaneously. Or he may skig from one to another in a varietyof,orders, and may perform any given type of task many times inthe course of attempting to solve a single decision problem.

.^


SECTION IV

INFORMATION GATHERING

From the point of view of the decision, maker, most decisionsituations are characterized by some degree of uncertainty. Thisuncertainty may involve the current "state of the world," thedecision alternatives that are,available, the possible consequencesof selecting any given one of them, and even the decision maker'spreferences with respect to the possible decision outcomes. Oneof the major problems facing the decision maker,rtherefpre, isthat of acquiring information in order to seduce his uncertainty6oncerning such factors, thereby increasing his chancespf makinga decision that will have a desirable outcome.

What makes the problem interstingt and nontrivial,,is thefact that information acquisition can be costly, both in terms of

. time and money. Therefore, the decision maker must determinewhether the value of the information that could be obtained throughany given data-collection effort is likely to be greater than thecost of obtaining it. And therein lies a decision problem in itsown right.

In theory, one can see an infinite regress here. In order todecide whether to initiate any information-collecting effort, onemust determine the worth of the information to be collected andthe cost of collecting if. But in order to determine that, onemay have to collect some information--at some.cost, and so on. Inpractice, of course, infinite regresses never occur; and in thiscase, one very quickly gets to a point at which the decision makerrelies on information n hand, or appealsito his own intuitions.

4.1 Information Seeking versus Information Purchasing

Information gathering, may be thought of as involving two quitedifferent activities: (1) information seeking (locating the infor-mation that ope needs or wants), and (2) informat}on purchasing(deciding whether information, the location of which is known,. isworth what it will cost to acquire it). This distinction is some-thing of an oversimplification, inasmuch as the act of seekingitself typically involves some cost, and one often Must Aecidewhether to incur that cpst without any assurance that the searchwill yield the information that is desired. .The aspect of "seek-ing" that we wish to emphasize, however, is the need for identi-fying and actively searching' ollt information sources, of findingopt where the desired information is and going after it. The term"purchasing" is used to connote a more passive role on the part ofthe decision maker, the opportunity to acquire information is pre-sented to him and he need only indicate whether or not he wantsto avail himself--at some cost--of the information that isoffered.

29r kt

42F

,

a


The distinction between information seeking and informationpurchasing is a useful one because it highlights the fact thatexperimental studies have focused almost exclusively on the latterprocess; although investigators often have not made the distinctionand have frequently discussed their results as though they had todo with the former. Typicallx, the decision maker is presentedWith all the information that he needs--although he may have todecide how much of it to purchabe--and the process of seeking in-formation is not studied. The world outside the laboratory is,not nearly so accommodating, however, and one must either seekout the information one wants, or go without.it. Moreover, studiesOf information purchasing, while they tell us something*about howeffectively people can judge the worth of information that is madeavailable to them, shed little light, on information-seeking behavior.

Perhaps the main reason why information-seeking behavior hasnot been widely studied is the difficulty of manufacturing situa-tions in the iaboratory that are representative of those faced bydecision makers in the real world. In any case, whatever theseasons, information seeking per se has not received the attentionfrom investigators of decision making_that it deserves. The ex-peiiments that we have reviewed that purport to deal with this .

topic invariably have actually studied information purchasing aswe have defined that term.

III4.2 Optional-Stopping Experiments

An experimental pa f adigm that has often been usedto studyinformation-purchasing behavior is one in which the decision makeris provided with the opportunity on each trial either of purchasingmore data that are relevant to the decision that he is requiredto make, or of making the decision. The terms "deferred decision""optional stopping" and "optimal stopping" have all been used to

I% refer-to this paradigm. "Deferred decision" and "optional stop-ping" connote the fact that the subject in lsuch an experiment hasthe option on each trial of making a terminal decision or deferringit in,order to obtain more data. 4"Optimal stopping" refers tothe fact that when the situation is sufficiently well-structuredso that the costs and payoffs associated With possible decisioncutcoMes, the cost and informativeness of data, and the decisionmaker's objectives are all known, the point can be determined atwhich information purchasing. should be stopped and the decision

' made. The "optional-stopping" paradigni is to be contrasted bothwith the more familiar paradigm in which the, experimenter deter-mines how much information'the decision maker will be given, andwhat is usually called the "fixed-stopping" paradigm in which thedecision maker specifies how much infOrmation he wishes to urchasepin advance of receiving any.

..

411

30

43


/

. //

Often, in optional-stopping experiments, the required de-cisioncision concerns the parametexs of the distribution tom which

'4the observational data are being drawn. For examp , one mayhave to decide whether a sequence of red and black poker chipsthat one observes is drawn 4om a population in which the propor-tion of reds to blacks is, say, 60-40 or 30-70./ The question ofinterest in such experimentsis whether the su,ject's information-purchasing behavior deviates from optimality,/ and if so, in whatway? ,

I

/ ,..

/What constitutes optimal performance. has been worked out for

a variety of specific situations (Birdsall & Roberts, 1965;Blackwell & Girshick, 19544 Raiffa & Schiaifer, 1961Y. For ourpurposes it suffices to tecognize that, in general, the amount ofinformation (number of observations) that should be purchased,will vary directly with, the magnitude of the costs and valuesassociated with the dedision outcomes, and inversely, with thecost and "diagnostiltly" of the data that are purchased. Diag-nosticity refers to the degree to which the data should reducethe decision maker's uncertainty gout which of the terminal de-ci on alternatives should be selected. The diagnostic' value ofa datum depends on several factors (some of which are discussedin Section VIII), and typically decreases as the number of datathat have already been collected increases. A factor that usually

410is not taken into consideration in optional-stopping experimentsbut'can be critical in real-life situations is the importance oftime itself. In some situations the potential consequences of adecision are highly time-dependent. Thi/s fact can be incorporatedin an optimal-stopping rule by making the cost of an observation,or the stopping criterion, a function of time.

Typically, performance in optional-stOpping experiments hasbeen found not to be optimal. Moreover, as illustrated by astudy by Green, Halbert, and Minas (1964), the deviation fromoptimality may be in either direction. In one experiment, Green,et al. found that the number of observations purchased increasedwith the a priori uncertainty concerning the correct decision--as would be expected of an efficient Bayesian pkodessor--howeven,subjects tended to purchase too many observations when the a prioriuncertainty was maximized by providing no prior information con-derning'the likelihoods of the correctness of the possible 4e-'cisions. In combination, the results of these experiments suggestthat decision makers may sometimes purchase too much information,and sometimes too little. In particular, it would Appear that .

they may purchase too much information if the a priori uncertaintyis small, and too little if the a priori uncertainty is large.

Many investigators have used the optional-stopping paradigm(Becker,' 1958; Edwards, 1967; EdOards & Slovic, 1965; Fried &

110

Peterson, 1969; Howell, 196,i 6; Irwin & Smith, 1957; Pitz, 1968, 1969;

- 31

44

NAVTRAEQUIPCEN 73-C-0128-1%It

Pruitt, 1961; Schrenk, 1964; Snapper & Peterson, 1971; Swets &Birdsall, 1967). The results of most of these studies suggestthat althouyh information seeking may approach optimal levels(Becker, 1958; Howell, 1966; Pruitt, 1961), there are reasonablysystematic departures from perfect performance. The generalfinding seems to be that too little.information is sought when(theoretically) muc-Iiiis required, and that too much is soughtwhen little is required. The latteD finding fits well with theconservatism or inertia effect often noted in studies of Bayesianinference, but the 'former clearly does not.

A few descriptive models of optional-stopping behavior havebeen developed. (see, for examples, EdwardS, 1965a; Pitz, 1968;Pitz, Reinhold, & Geller, 1969). These models have been developedIn a Bayesian context (Rapoport K Wallsten, 1972) and tend to besituation specific (see, for example, the "World Series Model"of Pitz, Reinhold, & Geller, 1969).

Noting that most optimal-stopping experiments had been con-cerned only with the question of when to stop acquiring informationfrom a single source, Kanarick, Huntington, and Petersen (1969)suggested that a more valid simulation of some decision-makingsituations, e.g., tactical situations, would recognize that thedecision maker must deal with information from more than one source.In keeping with this observation, Kanarick et ai. did an optional-stopping study in which the decision maker had the option on eachtrial of acquiring data from his choiceofthtee sources, or ofmaking a terminal decision. The terminal decision that was re-quired involved the presence or absence of an enemy submarine inthe vicinity. The information sources differed, both with respectto the cost of obtaining information from them and with respect tothe reliability of the information obtained. (The topic of reli-ability of information will be discussed more fully in SectionsV and VIII.) Costs associated with incorrect decisions were alsomanipulated. Although the behavior of the subjects was consistentwith the rational model in many ways--they were willing to pay morefor more reliable information; how much information they collectedbefore making a particular decision depended on how bad the con-sequences would be if that decision proved to be incorrect- -performance was less than optimal in several respects. The sub-jects tended, for example, to consult the most reliable (and mostcostly) sources less frequently and the less reliable (and lesscostly) sources more frequently than they should have. Kanaricket al. characterized this behavior as a form of conservatism, "areluctance to expend the resources necessary to obtain the bestinformation in a choice situation" (p. 382). The subjects alsotended to pdrchase less data in general than they should have, and,consequently, made more incorrect decisions and won fewer pointsthan did a Bayesian model that was used to represent optimal be-havior.

35

NAVTRAEQUIPCEN. 73- C- 0128 -1

Levine and Samet (1973) have also studied information gather-, ing--information purchasing in our terms--in a simulated tactical

situation. The scenario was a military action and the subjects'task was to decide which of, eight locations was the target of ahypothetical enqmy. advance. On each trial, a subject could eithermake a terminal decision or request additional information fromeach of three intelligence sources concerning the present where-aboutt of the advancing force. A sequence of reports from a givensource represented the path that the advancing force had takenover a period of time, According to that source. Among the vari-

.ables,that were manipalated were the reliability of the intel,li-gence sources, the degree of conflict among ieports from differentsources, and the probability that a request for information wouldyield an updated report (as opposed to a repetition of the preced.-

ng report). Performance was sensitive to each of the'variables.In particular, fewer reports were 'requested and'decisions weremore often correct when all tht sources were reliable, and thequality of performance,tended to decline as the percentage of the

sources that were unreliable was increased. Increasing the degree

to which the sources were in conflict also had the effect of de-creasing the number of reports requested. (This counterintuitive;result may be due in part to the fact that as conflict increasedin this experiment, so did the probability that the correct targetVas indicated by at least one of the sources on a given trial.)The number of requests for reports decreased as the probability thata given report would yield new information increased; the relation-ship was'such, however, that the amount of information (number ofupdates) received increased with this variable.

In a subsequent experiment, j_11 which the same decision problem

was used, Levine, Samet, and Brahlek (1974) varied the rate atwhich-new reports were given to the subject, whether the reports

were delivered automatically or in response to the subject'srequdst, the possibility of revising an initial decision and the

payoff scheme. In this case, performance was/better for the fasterrates of information acquisition, but was not highly sensitive towhether the rate was self- or force-paced. Increasing the,oppor-tunity for revising a decision had the effect of decreasing theaccuracy of first decisions and the subjects' confidence in them.

4.3 Decision'Revision and Effect of Commitment onInformation Gathering

The results of a few studies suggest that one's information-gathering behavior may be different after making a decision thanbefore, particularly if the making of the decision involves some-sort of public acknowledgment or commitment (Geller & Pitz, 1968;Gibson & Nichol, 1964; Pruitt, 1961; Soelberg, 1967). People mayrequire more information, for example, to change a decision thanwas required to arrive at a decisioh in the first place (Gibson &Nichol, 1964; Pruitt, 1961). This observation is in keeping withthe results of several studies that suggest that evidence that tends

to confirm a favored hypothesis is often given more credence thanevidence that tends to disconfirm it (Brody, 1965; Geller, & Pitz,

3346

NAVTRAEQUIPCFN 73-C-0128-1

1968; Pitz, Downing, & Reinhold, 1967). And sometimes disconfirm-ing evidence may even be misinterpreted as supportive of a decisionthat has already been made (Grabitz & Jochen, 1972).

The !motivation for acquiring infprmation may change, followinga decislon, from that of trying to increase the probability ofmaking a good decision tothat of justifying or rationalizing-adecision that has'already been made. Soelberg (1967) has concludedfrom a study of the job-seeking behavior of graduates of the SloaneSchool that people frequently make an implicit selection from amongthe existing opportunities, following which "a great deal of per-ceptual and interpretational distortion takes place in favor ofthe choice canc4date" (p. 29). In,a somewhat similar vein, Morganand Morton (1944) have asserted that people oftentaccept conc1,usionsthat are consistent with their convictions without regard for thevalidity of the inferences on which those conclusions are based,and that "the only circumstance under which we can be zalativelysure thatIthe inferences of a person will be logical is wen they :lead to a, conclusion which he hqs already accepted" (p. 39). Wewill return-to the question of the logicality of thought inSection 8.3).

One suspects that in real-world situations the information-seekina behavior that follows the making of a decision may oftendiffer considerably from that that precedes it. In particular,one would guess that to the degree that the motive of the informa-tion seeker is the rationalization of a decision already made, theprocess would become highly selective as to the sources consulted.

4.4 Quantity of Information and, QualiLy. of Decision

It is quite natural to assume that the more data one has thatare relevant to a choice that he must make, the better his choicewill be. The assumption, without qualification, is not valid'(Ackoff, 1967; Fleming, 1970; Hayes, 19641 Hoepfl & Huber, 1970;Sidorsky & Houseman, 1966). Ithis posdible, indeed easy, to providean individual with more information than he can assimilate and use- -especially if he is.operating under some time pressure. The pointis illustrated nicely by an experiment by Hayes.

Hayes had naval enlisted men make decisions concerning which ofseveral airplanes to displatch to investigate a reported submarinesighting in a simulated tactical situation. The available airplanesdiffered with respect to such characteristic-6 as speed, distance ofits base from the target, delay before it could take off, qualityof its pilot, quality of its radar, and so on. Each characteristic'could take on any of eight (not necessarily numerical) "values,"which could be ranked unequivocally from best to worst. The numberof available airplanes from which a subjeqit had to choose was varied(4 or 8) as*was'the number of characteristics (2, 4, 6 or 8) on

34

47

I


which he was to base his choice. The effect of the latter variable'

is of particular interest. Decision time increased markedly with

this variable; however, the decision quality--which was defined ob-

jectively in two ways--did not. Hayes hypothesized that, o-Oher,

things equal, one's sensitivity to the way two alternatives differ

with respect to individual characteristics decreases as the number

of characteristics that must.be considered increases. Of particular

relevance to this review is the fact that Hayes trained a second setof subjects for several days to see if they would learn to makebetter decisions with the larger amounts of information. Although

the quality,of decisions was generally somewhat higher after training

than before, the relationship between decision quality and number of

characteristics on which a decision, was based did not change:

We should not conclude from this study that one should never,

under any circumstances, be provided with more than a very few items

of information that are relevant ta any choide that one may have to

make. One mightconclude)-however; (1) that decision makers should

be trained to recognize their limitations for assimilating informa-

tion, and to avoid attempting to operate beyond them, and (2) that

to the extent that the functional relationship between the desira-

bility of the various choice alternatives that are open to the de-

cision maker and the values of the factors that determine it is known,

the implidation of particular sets of factor values should probably

be computed, and not estimated by men. The problem of determining,

or discovering, such functional relationships is a nontrivial one.

(See Section IX.)0

4.5 A Conceptualization or Information Gathering in the Real World

What makes the real-world decision maker's task particularly

difficult is the fact that the information that he would like to

have typically is distributed among a variety of sources. One'way

of characteriling these sources is in terms of the two properties:degree of passivity an'd degree of cooperativeness. According to

this conceptualization, a source is either active, or passive, land

either cooperative or uncooperative.

An actively. cooperative sourcethe/preferred type--volunteersinformatON, and seeks ways to get it to the decision maker. In

the military context, an intelligence offider would be an activelycooperative source for a commander.

A passively cooperative source is one that would. provide in-

formation if solicited, but does not volunteer it. A possible

reason for not volunteering information in this case is.a'failureof the source to recognize itself as such. An example, again 'from

a military context, would be friendly inhabitants of an area of

operations who have infortation that would be valuable to a

. NAVTRAEQUIPCEN 73-C-0128-1

military commnder, but are unaware of the fact. The problem thatthe decision maker has vis-a-vis passively cooperative sources isto identify and find them.

An actively uncooeerative source has information that wouldbe of use to the decision maker, but being motivated to thwart.the decision maker's objectives if possible, volunteers informationthat is misleading. A propagandist is an example of such a source.The decision make'r's problem with respect to actively uncooperativesources is to recognize them as such and to assess the informationobtained from them accordingly.

A passive uncooperative source is one that withholds in-formatiO7Wom the decision maker, and further will not prol)ideit if.asked. Hostile noncombatants in an area of military operationsmight it this description, as might espionage agents. The decisionmaker's problem with espect to passively uncooperative sources isto persuade them to change their status and become activelycooperative. History, both real and fictitious, is replete withaccounts of the unsavory methods that have been employed to this\ end. ,s

To the extent that. laboratory studies of decision making havebeen concerned with ipformation gathering, they have involvedactively cooperative sources almost exclnsively. The problem offinding sources that are nonobvious and that of coping with thosethat are noncooperative have received very little attention fromexperimenters. In part this is undoubtedly due to the fact thatcapturing the essence of these aspects of information gathering inlaboratory situations is a very difficult thing to do. And thealternative of studying these processes in situ is hardly lessdifficult. Until such studies are performed, however, our under-.

standing of how ,decision makers go about gathering - especiallyseeking - information so as to increase their chances of makingeffective, decisions will remain very incomplete.

4.6 ormationlGathering and Trainia2

We stress again that laboratory studies of informationgathering have failed to capture the complexity of the problemthat often faces the information seeker outside the laboratory.Consequently,very little.is known about information seekingbehavior as it occurs in the real world. This is unfortunate

,because information seeking constitutes a particularly criticalaspect of many real-life decision problems and so long as thisbehavioris not well understood, our understanding of decisionmaking will be incomplete.

36

49

NAVTRAEQUIPCEN 73-C-0128-1 /*

)

The implications for training are obvious: training proceduresthat are based on a solid foundation qf factual knowledge abouthuman capabilities and limitations cannot be developed if the foun-dation does not exist. The need is fOr research that is designedto answer some of the questions that labRratory, experiments here-tofore have, failed to address effectively Such questions includethe following. How good are people at ide tifying sources ofinformation that is relevant to their decis on problems? How dothey go about discovering such sources? Ho \capable are they ofassessing the cost of acquiring information -Oat may be difficultto get and the worth of the information that th4.ght be obtained?To what extent can useful principles and procedures for informationseeking be made explicit and taught? It is probbly fair to saythat with respect to such questions there is insufficient basis for

. even an educated guess as to the answer. Clearly\there is need forsome imaginative research on this aspect of the deFision-makingprocess.

Laboratory studies such as those reviewed above do shed somelight on information purchasing behavior. In particular they tellus something out human capabilities and limitations in assessiigthe worth of information .in Bell structured situations. Althoughit would be risky to generalize many of the conclusions uncriticallyto.nonlaboratory situations, the conclusions nonetheless are sug-gestive of what should perhaps be done by way of training or train-ing research.

)

37

50


SECTION V

DATA EVALUATION'

In the preceding section we used the words data and informa-tion more or less synonymously. It will be helpful at this pointto make a distinqtion. The t rm data is perhaps best used torefer to what one coll'ects, a i d the term information to connotewhatever conclusions or infer. nces one draws from data. The dataand the information extracted therefrom can be identical, but,theYneed not be. For example, if a military commander receives datato the effect that the troop strength of an opposing tacticalforce is 15,000 men, and he considers the source to be _a reliableone, he will undoubtedly accept the data as accurate and concludethat the enemy troop strength is indeed 15,000 men. On the otherhand, if he has less than full confidence in the source of thisreport, he may tentatively conclude that the troop strE.Agth issomewhere between 5,000 and 25,000 men, and attempt to gel moredatafrom which he can derive a more precise estimate.

The point is that as part of the process of attempting toreduce' his uncertainty about his depisiOn situation, the decisionmaker must evaluate the data tgat he receives as to their per-tinence and trustworthiness. In other words, the first decisionthat the decision'maker must make..with respect to any'new datumis how seriously he should take it. He may not explicitly dothis in all cases, but to fail to do so at least implicitly istantamount to judging his sources as completely trustworthy andtheir inputs as equally important.

5.1 The Evaluation versus the Use of Data,

There are two questions relating to data quality that deserveattention: (1) how well can people judge and report the quality8f the data on which.decisions are to be based, and (2) howeffectiVely can they;utilize information concerning quality ofdata when that information is provided for them? The first ofthese. questions concerns what we are referring co as the task ofdata evaluation, and is discussed in this sectioft: The secondhas to do with data utilization and is more appropriately dis-cussed in connection with, hypothesis evaluation in Section VIII.

In anticipation of the latt4r discussion, we'note here simplythat several experiMents have beep addressed to the question o£how effectively decision makers use knowledge of data quality.In most s5ch studies th2 performance of subjects has,-been comparedwith that of some ideal (usually Bayesian) model (see, for examples,Funaro, 1974: Johnson, 1974; Schum, DuCharme, & Deitts, 1973;Snapper & Fryback, ..1971; Steiger & Gettys, 1972)'. What is mostgermane to he topic of this sectionis the fact that the modelsthat are used to represent optimal behavior typically distinguish

4

38

51

NAVTRAEQUIPCE14 73-C-0128-1

two separate steps. The first step entails an adjustment of thenominal diagnostic value of a datum, the value that the datum wouldhave if ie were known to have been reliably observed or'reported.The seQpnd step involves the application of the modified datum tothe hyp6heses,of interest. The first step is whatLwe ard"calling,data evaluation, and it is important to note that the failure ofsubjects to perform this step properly appears to be one of thereasons why they typically acquire less information from data thatare not perfectly reliable than is there to be acquired.

5.2 Studies of Data Evaluation

Data evaluation has been recognized by the U.S. Army as. beingof sufficient importance to warrant the development of a ratingprocedure for use by tactical intelligence personnel to evaluateall incoming "spot reports" (Combat Intelligence Field Manual,FM30-5). ,The procedure, which has been standardized for use byNATO army forces, requires that a sender of a report explicitlyrate the report both with respect to the reliability of its sourceand the accuracy of its contents. The letters A through F areused to designate estimates of reliability, and th6 numbers 1through 6 to represent judge accuracy, The.first five ratingsrepresent a scale going from "completely reliable" (A) to "un-

reliable" (E) in one case, and from "confirmed by other sources"(1) to "improbable" (5) in the ,)ther. The lowest rating in eachcase is used to indiqate 'thatla judgment cannot be made; "relia-bility'cannot be judged" (F), "truth cannot be judged" (6).

Obviously, the purpose of using such a rating procedure isto provide the receiver,of a report with some indication of howmuch confidence he should have in its contents. How effective theprocedure has been, however-, is open to question. Data collectedduring field exercises have indicated that ratings often areomitted. from spot reports, and that the ratings that are used aretoo consistently high (Baker, McKendry, & Mace, 1968). The samestudy also revealed that the reliability'and the accuracy ratingstend to be highly correlated. One possible` explanation of thiscorrelation is that reliable sources tend to produce accuratereports. This is an intuitively plausible explanation, and itraises the'questidh of the need for two ratings. The other pos-sible explanation for the correlation is that the rater finds itdifficult to treat reliability and accuracy as independent dimen-

sions. The results of a subsequent laboratory study of ratingbehavior were interpreted as supporting the latter possibility(Samet, 1975a). On the basis of his results, Samet proposed thatan attempt be made to design and validate an improved procedurefor evaluating intelligence data. Specifically; he suggested theposLbility of assigning to a report a single number that wouldrepresent the evaluator's estimate oflthe likelihood of the reportbeing true, based on alL.the.information available to him thatwas relevant to that judgment.

0

NAVTRAEQUIPCEN 73-C-6128-1I

5.3 The Use of Nonquantitative Qualifiers_

Probably most peopl who evaluate data or data sources do notdo so according' to a fo 1 procedure or irt.quantitative terms.More typically, theytuse such qualifiers as "usp lly reliable,""not very dependable,' "prone to exaggerationd ' "very precise,""a bit.careless,":"very likely, " "a rodgh est mate " and so forth.Such_phrases are certainly meaningful and undoubtedly can conveyimportant qualifying information. The.problem is that not Allpeople mean the same thing when they use one of these phras &s, andwhat complicates matters is the fact that even a given individualmay use the same term to mean somewhat different things at dif-ferent times.

4A number of effortb'have been made to measure the extent ofagreement between individual's in their use of such qualifyingterms. A common experiment* paradigm s that of provi.ling sub"jecti with lists of terms or phraseS and requiring them, to trans-late the degree of certainty or uncertainty, denoted into 44 numeric

(typically probabilistic) estimate. The variance observed amongand within subjects in the translatothri then provides ameasurement

of agreement. Results of these studies (see, for example, Lich-tenstein & Newman, 1967; Johnson, 1973; Samet, 1975a, 1975b)typically show very low levels of agreepent among subjects, andthe potential foi considerable misunderstanding when large vo6abu-

r- laries of qualifiers are, used.

iWhat factors influence the translation of a qualifier into

a numeric estimate? Thee seem to be no clear answers to thisquestion. Cohen; Dearnley, and Hansel (1958) suggested that con-text in which a word is used might play a role, but a recent qtudyby Johnson (1973) in which the encoding of 15 different probabilitywords (or phrases) contained in each of three different sentencecontexts was explored failed to uncover any significant contexteffect. On the other hand, a study dy, Rigby and Swain (1971) inwhich magnitude-denoting terms such as "couple," "lots," and"bunch" were used did suggeSt such an effect. For example, a"bunch of missiles" had an average assignmenteof 7.73, while a"bunch of tents" had an average assignment of 12.32. It seemsobvious on the face of it that nonquantitative terms denotihgphysical magnitudes must be subject to enormous context effects."Small" distances are measured in angstrom units by nuclearphysicists and in Iightyears by astronomers. Indeed, it isdifficult to see how, in the absence of context, such terms canbe considered meaningful at all. Probability terms are diferentfrom magnitude terms in that probabilities are bounded whereasmagnitudes are not. Perhaps this helps to account for the former'sgreater independence of context. 'It should be oted that neitherJohnson nor Rigby and Swain found significant di ferences in theuse of these terms due to group membership (ar y enlisted men and


0

graduate students, in the former study; army helicopter, Air Forceprop, Air Force jet, and Navy attack bomber pilots in the latter).

5.4 Data Evaluation and Training

It seems clear that full exploitation of computer-based tac-tical data-analysis systems will ultimately require the use ofnumeric values in place of qualitative estimates, if the relia-bility of da,ta is to be taken into account when they are used.How best to arrive at these values is, at this point, a matter ofconjecture. One could attempt to establish a formal vocab4lary.of qualitative terms and phrases, associite With each term orphrase a specific numerical value (or range of values), and trainpersonnel to use the resultant is morphisms in encoding and de=coding communications. This is he essence of a proposal madesome years ago by. Kent (see P tt, 1957). Considering, however,that formal training would b a requirement in any caLa, a pre-ferred alternative to this apptoach is to instruct deci...ion.inakersin the use of probability (and magnitude) scales and requireestimates to be communicated in explicitly quantitative terms(Johnson, 1973; Samet, 1975), Thq obvious problem for trainingresearch is that of developing effective procedures for trainingpeople to evaluate data quantitatively and for increasing theinfra., and inter-perpon consistency with which quantitativeassessments are made.

J

1

NAVTRABQUICEN 73-C-0128-1

SECTION VI

PROBLEM STRUCTURING

An exceedingly important step, in solving any problem is to-be-- quite explicit- about what the problem is that one is to solve.And one way to be explicit is to attempt to represent the problemin terms of a formal structure. While the need to be explicitmay appear to be too obvious to deserve comment, it is also ap-.parent that sAtisfying that need is, not always an easy thing to do.Attempts to apply computers to problem-solving tasks have high-lighted both the need for explicitness and the difficulty of ob-taining it. Armer (1964) has commented on the frustration thatis sometimes entailed when one tries to formulate a problem insuch a way that a computer can help solve it. He illustrates hispoint with reference to a bank official who stated, after havinghis banking procedures mechanized "that 65 percent of the data-procesbinggroup's:effort went to deciding in detail wh t problemthey were'solving"(p. 250). Presumably, the investment ,as worthit; without it, they could not have recognized a solution had theyfound one.

The, act of trying to make the structure of a problem explicitcan be an instructive experience for a problem solver, inasmuchas it forces on him the realization of what he does and does notknow about the problem on which he is working - -or thinks he is.Essentially, this observation is made by.Cloot (1968) vis-a-visthe application of computers to the decision problems of manage-ment. He takes the position that one of the benefits thatis to be derived from an attempt to implement a computer-basedmanagement information system is not the help that one would getfrom a functioning system; but what one can learn about the prac-tice of management from the implementation effort.. "It can evenbe'argued-that the successful use of a computer-based MIS shouldbe measured by the extent to which managers learn to improvetheir performance so that they can discard it again... There isno doubt that the changes that do come about will be due more tomanagers having abetter understanding of theL decision processes,than, to the technical facilities of the computer" (p. 280).

A major contribution of theoretical treatments of decisionmaking is the provision of formal models in terms of which a de-cision maker can attempt to structure his own decision problems.Invariably, such models are simplified abstractions, and conse-quently may not do justice to'the full details of any given situa-tion. Nevertheless, they do provide one with structured ways ofviewing things,. which may make the problems easier to think about,and ase consequence--hopefully--easier to solve. It has beensuggested that this is the way in. which quantitative models willhave their primary'effectr "I believe that the greatest impact ofthe quantitative approach will not be in the area of problem sol-viny, although it will have growilTai usefulness there. Its greatest.

42

NAVT.RAEQUIPCEN 73-C-01281

impact will be on problem formulation: the way managers thinkabout t eir problems--how they size them up, bring new insightsto bear n them, and gather information for analyzing theM. Inthis sens the results that 'quantitative people' have producedare beginning to contribute in a really significant way to theart of management" (Hayes, 1969, p. 108).

6.1 State-Action Matrices

Probably the most well-known way of representing decision situ-ations is in terms of state-action matrices. .Such matrices make -

three aspects of decision situations explicit: the hypothesizedpossible "states of the world," the action alternatives that areopen to the decision maker, and the decision maker's preferenceswith respect to the various possible state- action combinations.'Sometimes such matrices are referred to as payoff matrices inasmuchas each cell of the matrix represents the cost or val'ie--or utility--to the decision maker of the outcome of a particulaL action se-.lectiOn, given'that the associated state hypothesis is true. Adecision problem may be represented in this way as as follows:

Action AlternativesAl A

2A. A

n

Hypothesized H1

U11

U12

States

of the H2

U21

022

World

H. U . .1)

Hm Umn

Much of the theoretical-analytical work on decision makinghas been concerned with optimal strategies for selecting actionalternatives once the situation has been formally structured.Given an explicit decision goal (e.g., minimization of risk,maximization of expected gain, and a formal representation*of the situation, prescriptive models can provide useful guidance \for action selection. The process of representing real-lifedecision situations formally, however, is at the present time moreof an art than a science. Examples of decision situations thatare easily structured can always be found; however, not all de-cision problems can. readily be forced to fit the same mold.

NAVTRAEQUIPCEN 73'-C-0128-1'

Evdn given that the structure shown above is an appropriateItone for a particular problem, it is clear that in order g use itone must be able to specify, as a minimum, what the hypo esizedstates of the world are, what one's action options are, and how-the various possible decision outcomes (state-action pairs) relateto the value system that will determine the desirability of theactual outcome. One may find it necessary to engage in a consider-able amount of information seeking, in order to fill out-such astructure. Moreover, how one does fill out the structure is deter-mined in part by exogenous variables over which one has no control,and in part by self-impbsed constraints. The state of the worldtends to be beyond one's control; all one can do is attempt todetermine what it is likely to be. One's action alternatives,'

`however, may be constrained in part by limits that are self-imposed.What are viewed as viable strategic military options, for example,may depend min the particulAr military doctrine in vogue at the time.Benington (1964) points out that the basic concept beh'.nd thedevelopment of such automated, or sefilautomated, systems ls theSAGE system in the 1950's was the concept of "set-piece. warfare.""Set-piece warfare is characteri2edby warning of threat, totaland preplanned goals, speed of,response, and detailed and precisemanagement of the Campaign" (p. 9). Emphasis is on massive re-taliation totally preplanned, or "spasm" response. During theearly 19150's, the set-piece warfare idea lost faor. PresidentKennedy and Secretary of Defense McNarriara began to emphasize the-importance of 'flexibility and adaptability, the'ability to makeselected and controlled responses, directed toward military (non-civilian) targets and appropriate to the (not always foreseeable)contingencies that elicit them. Clearly, the set of action alter-natives that the strategist will consider ,under one of these re-taliation doctrines is quite different from that that he willconsider under the other.

6.2 Alternative Structurin s of a Given Situation

It is apparent that to think in terms of the structure of adecision space IS to oversimplify matters greatly. Usually anygiven situation can be structured in a variety of ways. Moreover,how one chooses to representja particular situation may not beincidental. It seems to be true of problem solving in generalthat how one represents a problem can be an important factor indetermining how easily one can then solve it. This point has oftenbeen made by individuals engaged in efforts to program computersto perform intellectually demanding tasks (see, for example,

4 Nilsson, 1971). The same probldm may yield to'attempts to solveit when represented in one way while resisting such attempts whenrepresented in another.

An important aspect of developing a useful structure is thatof conceptualizing a situation at an appropriate level of detail.

NAVTRAEQUIPCEN.73-C,-0128-11

Too simple a structure may violate the complexity of an actualsituation. On the other hand,'ICharles Pierce's maxim that "a fewclear ideas are worth more than many confused ones" seems particu-larly apropos ere. We state as a conjecture that a necessaryrequisite for effective decision making is the ability to getquickly to tlie heart of a problem, to concentrate on.essentials,and to ignore irrelevancies. What this often' means ip practice isbeing able to see.through superfi6ialities that frequently obsgure'underlying issues. Moreover, even when the situation, stripped ofincidentals, is itiAerently complex, there may be some merlt in asimplified conceptualization of it, provided that the fact that theconceptualization is a simplification is not then promptly forgotten.There is little to be gained by representing, a situation A sucha complex way that the decision maker cannot grasp the representationintellectually. What constitutes an optimal level of detail mayvary from situation to situation and frOm individual to individual,but variability in this regard may not by very great. We suspectthat for the vast majority of situations and decision meo'srs arepresentation that involves more than eight or ten hypothesizedstates of the world and as many action alternatives, at any givenlevel of' description, will prove to be an unwieldy, one.

6.3 -Structuring as an,Iterative Process

On the basis of an anlaysis of protocols obtained in hxsclassical study of groblem solving, Duncker (1945) reached a con-clusion that is gerrane to the issue of problem structuring. Theproblem that he used most frequently in his studies was the nowwell-known radiation problem: "given a human being with an inoperablestomach tumor, and rays which destroy organic tissue at sufficientintensity, by what procedure can one free him of the tumor by,. theserays and at the same time avoid destroying the healthy tissue whichsurrounds it?" (p. 28).. The conclusion that Duncker came to afterobserving the efforts of many people to solve such problems wasthat the development of a solution typically proceeds frbm the moregeneral to the more specific. (On this point, see also Hogarth,1974, and Kleinmute, 18643.) The principle by which the problemis, hopefully, to be solved merges first, and the dettils of thesolution come later. It often happens that a principle may bevalid, but there turns out to be no feasible way tbiimplement it.A principle that was frequently identified in the case of the radi-ation problem, for example, was "avoid contact between rays'vldhealthy tissue." When the problem solver could think of no way todo this and still get the rays to the tumor, he had to abandon theprinciple itself--even though it was a sound one--and search foranother that was not only sound but practicable.

The finding of--a new principle, or a general property of asolution, always involves, Duncker suggests, a reformilation ofthe original problem. In the case of the example just given, havingaccepted "avoiding contact" as a valid principle, one has in effect

45ti


defined his problem as that of finding a way to do just this. Whenforced to reject a given principle as impractical, the substitutionof another (e.g., "lower the intensity of the rays on their waythrough healthy tissue") in effect defines another how-to-do-itproblem to be solved. "We can accordingly describe a process ofsolution either as development of.the solution or as developmentof the problem. Every solution - principle found in the process,which is itself not yet ripe for concrete realization... functionsfrom then on as reformulation, as sharpening of the original settingof the problem. It is therefore meaningful to say that.what isreally done in any solution of problems consists in formulating theproblem more productively. To sum up: The final-form of a solutionis typically attained by way of mediating phases of the process,of. which each one, in retrospect, possesses...the character of asolution, and, in prospect, that of a problem" (p.34, italics his).

It is,probably the case that complex decision problems, likeother types of complex, problems, yield Grudgingly to attempts tostructdre them. Moreover, a decision maker may find it necessaryto.formulate and reformulate a decision space several times beforearriving at a structure that he feels adequately represents thedecision problem that he must solve and okes so in a way thatfacilitates arriving at aisolution. The willingness to discarda favored conceptual framework when it is seen no longer to fitthe facts in hand has been considered by some to be one of thedefining characteristics of original thinking (Mackworth, 1965;Polyani, 1963).

6.4 Problem Structuring and Training

The question of how to train decision makers to structuredecision problems effectively has received very little attention.Moreover, if it is true, as Edwards (1973) has suggested, that ofthe several aspects of decision analysis the process of problemstructuring is least amenable to formal prescription, exactly whatshould be taught is not clear.

It seems likely, however, that something is to be gained byfamiliarizing decision makers with such formal representations7-models--of decision situations, as are provided by decision theoryand game theory. Such training should be conducted in such 'a wayas not to leave the student with the unrealistic idea that alldecision situations are readily. represented -- without distortion- -by the same model.

Practice in representing specific situations in terms of suchmodels, and oiteria for judging the relative merits of differentmodels for different problems should probably be part of anytraining program in decision making. Practice in representing agiven decision problem at different levels of detail also would

46

59

NAVTRABQUIPCEN 73-C-012E3a-1

probably be beneficial. Duncker's work suggests that one approachto problem structuring that might usefully be taught is that ofzeroing in on -an appropriate formulation_ by a -series of approxima-tions, proceeding from the more general to the more detailed.

But these-are only conjectures. The fact is that little isknown about how to train a person to be good at'imposing structure

on a problem--whether it be a decision problem or a problem ofany other kind. Mackworth (1965) has noted that one of the char-acteristics of creative individuals is an exceptionally strongneed to find order where none appears on the surface. If this '

is so, then one way to train people to be better problem structurersis to train them to be more creative. If only we knew how to do

that!.4

An alternative to training decision makers to formalize theirdecision problems is to.provide them with models that are appro-priate to their particular, situations, and that can then be used

as decision aids. wry (1970) hat suggested this possibility.A model that is to be used by a decision maker need not be genera-ted by him but, Gorry points outwit may be derived from hisdescription of the situation, and it must be thoroughly under-standable by him. In this case the training task becomes that of

teaching an individual to make effective use of the structure that

someone else -has imposed upon his problem. -At least one study has been addressed to the question of the

subtasks in terms of which one class of decision makers sees deci-sion-making and how this view would change ass a result of training.Hill and Martin (3,971) gave secondary-school teachers problem-solving exercises designed to' train them with respect to some of

0 nineteen specific skills that they associated with decision makingand to acquaint them with a particular model' of the decision-makingprocess (see Section III). Both before and after training, the 1

subjects were asked to list the specific steps that they wouldtake in an effort to solve a hypothetical prcblem involving an inter-person conflict. 'Perhaps the most striking aspect of the resultswas how large a proportion of the steps that subjects' listed fellin the "formulating-action-alternatives" category. Before,training,more of the listed steps fell in this category than in the otherfive combined. The main effect of training was to reducesthe num;ber of steps-in this category by about two-thirds and to increasethe usage of some of the other categories slightly; but formulatingalternatives still remained the largest category. The investigators,concluded that training had made the participants more aware of theseveral activities involved in decision making, but pointed outthat their results shed rib light on the question of whether much asincreased awareness would produce better decision making.

47

3O

NAVTRAEQtIPCEN 73-C-0128-1

SECTION VII

HYPOTHESIS GENERATION

7

HypdtheSis generation is closely associated with problemstructuring. We find it convenient to consider it separately,however, because it is a more narrowly focused type of activity..Problem structuring is always important. Even when complete in-formation is available concerning the state of the world, theaction alternatives and all the possible decision outcomes, it isstill necessary to cast the problem into some mold, and the moldthat is chosen may have much to do with the decision that is made.Hypothesis generation, on the other hand, is a necessary activityin those decisign situations characterized by uncertainty aboutsuch things as the state of the world and,the implications ofselecting specific' decision alternatives. Often, in spiteof one's best efforts to gather information, it is not possibleto eliminate uncertainty about thes& things completely:, In suchcases, it is convenient to conceptualize the decision'makar's viewof the situation as a set of conjectures, or hypotheses.

7.1 Hypothesis Generation versus Hypothesis Testing

Investigators of cognitive processes have long recognizedtwo rather different types of thinking. Bartlett (1958) speaksof closed versus adventurous thinking, Guilford (1963) of-conver-gent versus divergent thought. Mackworth (1965) distinguishesproblem solvers and problem finders. the one kind of thitkingtends to be deductive and analytical; the other inductive andanalogical. The first has to do with evaluating hypotheses, thesecond' with generating them. The history 'of science attests tothe fact that the ability to evaluate hypotheses, to deduce theimplications of theories and put them to empirical test, is a farmore common quality among men than is the ability to generatehypotheses, to construct theories that organize and structurefacts that were not perceived as related before.

Some formal treatments of debision making require that theSituation, as viewed by the decision maker, be conceptualized asa Set of mutually exclusive and exhaustive hypotheses, each ofwhich reprdsents one of the possible states of the world. As da,taare gathered, they are used to modify a set of probabilities, eachof which represents the decision maker's estimate of the likelihoodthat a given hypothesis is true. Much laboratory experimentationhas been devoted to the question of how effectively man can assi-

' milate 'data and use it to modify his view of the world as impliedby the probabilities that he associates with the hypotheses thathe is entertaining. (We will consider that problem in the fol-lowing section.) However, very little attention has been givento the question of howcapable people are of generating a reason-able set of hypotheses to begin withi'or of modifying the setwhen the need to do so arises.


Typically, all of the hypotheses that are to be consideredare provided for the,decision maker in advance, so the process ofhypothesis generation is_not studied_. Moreover, formal decision,procedures usually permit the decision maker only to update theprobabilities that have been assigned to the previously establishedset of hypotheses: 'They fail to recognize the fact that it maybe the case in real-life situations that a set of hypotheses thatis originally developed may not contain the hypothesis that willeventually prove to be the true 'one. It often occurs in real-lifesituations that incoming data suggest to the decision maker new 1

hypotheses that have not yet been considered. Any decision- makingprocedure that purports to be generally valid must provide forestablishment of new hypotheses whenever the information in handindicates the need for them.

7.2 Importance of Hypothesis Generation

The importance of the function of hypothesis generaclon canhardly be overemphasized. To be sure, one may think of some de-cision contexts for which all the potentially interesting hypothesescan be specified in advance. For example, it may be the case forsome straightforward troubleshooting situations that an exhaustiveset of the hypotheses of interest can be listed prior to the per-formance of any tests. More typical of complex decision problems,however, is the case in which the set of possibilities is eithernot fully known, or too large to be listed exhaustively. Theproblem of the physician who is attempting to diagnose an illnesswith a set of.symptoms that does not fit a common pattern, or theinvestor who is trying to gauge the risks and potential gains ina speculative financial venture, or the computer programmer who istracking down an elusive bugi or the tactician who is trying toassess the significance of some unorthodox behavidr on the partof a wily opponent is less that of testing prespecified hypothesesthan that of defining hypotheses that it would make sense to con-sider.

The difficulty is not so much that of representing a decisionsituation in terms of a set of pOssible states of the world thatis exhaustive and mutually exclusive. The problem is that ofcoming up with a set of possibilities that is useful from the'decision maker's point of view. A military commander can alwaysrepresent the alternatives that are open to an adversary in termsof such gross action categories as attack, defend, and withdraw,and the ability to distinguish among these' possibilities wouldundoubtedly be of interest. However, a commander's decision-makingresponsibilities typically require much more precise informationthan would be provided by the resolution of the uncertainty implicitin these three possibilities. That is to say, he wants to know notonly whether, enemy forces plan to attack, but at what time, in whatstrength, at what locations, and so forth. It is' at this level ofrepresentation that the commander's or perhaps his intelligenceofficer's) hypothesis-generation capabilities are put to the test.

49 62


7.3 Experiments on Hypothesis Generation

The study of Hypothesis generation in the laboratory hasoften involved "concept attainment" or "discover the rule" typetasks. The work of Bruner, Goodnow, and Austin (1956) illustratesthe use-of concept attainment tasks to study this aspect of think-ing. In a typical experiment, a subject attempts to identify aconcept that an experimenter has in mind. The concgpt usually isdefined in terms of conjunctions or disjunctions oespecific stim-ulus attributes (e.g., "red and square"; "blue or yellow, and notcircular"). In some situations the subject is shown stimuli, someof which belong to the conceptual category that he is attemptingto identify and some of which do nbt. He is told which stimuliare which and from this "exemplar" information he is to attempt toidentify the concept. Sometimes the subject chooses the stimufithat he sees, in which case the task can also be used to study aform of information-gathering behavior.

Obviously, the performance of this task involves hypothesistesting (a topic to which we will turn in the following section),but the key problem 14 that of hypothesis generation. Unless onecomes up with the right hypothesis to test, the testing that hedoes will only eliminate some of the untenable possibilities, ofwhich there may be many.

A basic conclusion that Bruner et al. draw from their experi-mental results is that the strategies that subjects employ in these,sorts of tasks can be isolated and describbd. They identify foursuch strategies, for example, that subjects-use when they have thejob of discovering a conjunctive concept by selecting stimuli andbeing told, concerning each stimulus selected, whether or not itis an exemplar of the concept that they are attempting to identify.These strategies differ in terms of the balance they strike amongthree parameters: the amount of information obtained from an ob-servation, the cognitive strain imposed on the subject (amount ofintormation that must be carried in memory, extent to which involvedinferenbes must be made), and the risk that the strategy will fail.The strategies are defined in terms of the nature of the hypothesesthat are generated and put to the test. In one case, for example("successive scanning"), one specific concept is hypothesizes ata time, and stimuli are chosen in such a way as to test that hy-pothesis directly. In another case ("conservative focusing"),the initial hypothesis, in effect, includes several possible con-cepts and an attempt is made to discover the defining attributessystematically one at a time. Which of the several strategies ismost appropriate depends on the details of the experimental situa-tion.

Bruner et al. found that the strategies that subjects usetend to change appropriately in response to changes in the


experimental situation; and, on balance, these investigators con-sidered the performance of their subjects to be quite good. Intheir words: "In general, we are-struck -by -the notable-flexibilityand Intelligence of our subjects in adapting their strategies tothe information, capacity, and risk requirements we have imposedon them. They have altered their strategies to take into accountthe increased difficulty of the problems being tackled, choosingmethods of information gathering that were abstractly less thanideal but that lightened pressures imposed on them by the tasksset them. They have changed from safe-but-slow to risky-but-faststrategies in the light of the number of moves allowed them. Theyhave shown themselves able to adapt to cues that sabre less thanperfect in' validity and have shown good judgment in dealing withvar.ous kinds of payoff matrices. They have shown an ability toco ine partially valid cues and to resolve conflicting cues"(P. 8) .

Performance was not ideal, however.' Among the limitations 'that were noted were a tendency to persist in focusing on cuesthat had proved to be useful in the past even if they were notuseful in the present, and an inability to make as effective useof information gained from noninstances of a category as of thatgained from category exemplars.

Bruner et al. also found that concepts defined in terms ofdisjunctions of stimulus attributes were more difficult to discoverthan those that were conjunctively defined. This finding has beencorroborated by Neisser and Weene (1962) who used a large variety ofattribute-combination rules. Not surprisingly, concepts definedin terms of the presence or absence of a single attribute areeasier to attain than are those defined in term of conjunctionsor disjunctions of two or more attributes, whic in turn areeasier than those defined in terms of more com ex rules involv-ing combinations of conjunctions and/or disjunctions (Haygood& Bourne, 105;'Neisser & Weene, 1962).

Another experimental task that has been used to study hypo-thesis generation is that of discovering the rule by which aspecific sequende of numbers or letters'was generated. Typically,the subject is shown one or more sequences (or segments of se-quehces) that satisfy the rtkle. He then can propose other se-quences, or continuations of the segment, in order to test thevalidity of tentative hypotheses that he may wish to consider.Each time he proposes a possibility he is told whether it satis-fies the rule; and when he feels he has obtained enough inform'ationto justify doing so, he is to state the rule.

Again, performance of this task obviously involves informationgathering and hypothesis testing as well as hypothesis generation,but hypothesis generation is in some sense central. What information

-" 51

64


is sought is likely to depend strongly on what rule is being con-sidered.. Moreover, unless the correct rule is hypothesized at

;some- point, it cannot be-tested- alra-va'llttated.

The results of experiments along these lines have revealedsome interesting deficiencies in hypothesis-generation behaviorwhich appear to stem from a lack of understanding of some basis -rules of logic. Wason (1974) has described some results thatsuggest that people may have particular difficulty in discoveringrules that are sufficiently, general that they subsume many rulesthat are more specific. For example, the rule "any three numbersin increasing order of magnitude" proved to be particularly dif-ficult for his subjects to discover. If, as examples of triadsthat conform to this rule, a subject were given (8 10 12), (1416 18) and (20 22 24), he might quickly generate the hypothesis"successive even numbers," test it with other sequences thatsatisfy it, and then announce this rule with confidence. What isdisappointing about this behavior is the failure to ,hypothesizealternative rules to which the given sequences also conform, andthen to consider sequences that would discriminate between thealternatives hypothesized. More disturbing, however, is thefinding that even when told.of the incorrectness of a hypothesis,and presented with conclusive infirm evidence, subjects some-

- times insisted that their hypothesized rule was validated by thefact that all the test sequences that they generated conformedto it.

Two other results noted by Wason are relevant to the problemof hypothesis generation, because they also demonstrate how the.process can get bogged down. First is the possibility of perse-veration with an invalidated hypothesis without recognizing thatone is persever4ing. He notes, in this regard,. that what subjectsoften do when informed that a hypothesized rule is not the correct

0 one is to generate additional 'triads' that are consistent with thatrule and then announcethe same rule expressed in different terms.Second is.a tendency, when hypothesized rules are invalidated, togenerate more and more complex rules rather than simpler ones.The following example is given of a third generation rule producedby one subject: "The rule is that the second number is random,and either the first number equals the second minus two, and thethird is random but greater than the second; or the thitd numberequals the second plus two, and the first Is random but less thanthe second" (p. 382). Recall that the correct rile was "any threenumbers in increasing order of magnitude." One donclusion thatmay be drawn from this type of experimental finding is that thediscovery of a general rule, even though conceptually simple, maybe impeded by the discovery of more specific rules whos_ exemplarsare also exemplars of the more general rule.

5265

NAVTRAEQUIPCENO

7.4 Hypothesis Generation and Training

HypOthesis generation represents the same sort of challengeto training and training research as does problem structuring.The basic nged in both cases is for a_ reater understanding ofhow to promote creative thinking.

A specific problem that deservesrattention from trainingspecialists is that of perseveration, Results such as those ob-tained by Bruner, Goodnow, and Austin (1955) and by Wason (1974)indicate the need for training Kocedures designed to improvethe ability, or increase the willingness, of decision xakers togenerate alternatives to the hypothesis, or hypotheses, underconsideration. They demonstrate the importance of sensitizingdecision makers to the danger of accepting a hypothesis on thebasis of insufficient evidence, and to the fact that the best wayto avoid this mistake is to attempt to generate plaus.ble alter-natives and to seek the kind of data that will be most likely todiscriminate among them.

53

66se


SECTION VIII

HYPOTHESIS-EVALUATZON-(

4Narrowly defined, hypOthesis evaluation refers to the process

cf applying data to the assessment/of the likelihoods of one'shypotheses concerning the unknowns`of the situation. More generaly,the term might be used to connote the process of extracting.informa.7tion from data, of attempting tb reduce one's degree of uncertaintyabout the parameters'of the decision space._ In some formally struc-tured approaches to decision making, hypothesis evaluation may involvethe revisionsoof numerical probability estimates or other, quantita-tive indicants of relative likelihoods. In other cases the processmay be less explicit, but it is not for that reason less important.'We assume that even in situations that have been given little formalstructure, the decision maker attempts to make use of aL least someof the data that are available to him, in order to clarify his view,or perhaps to confirm his assessment, .6f the situation.

The following discussion takes a rather broad view of ,

hypothesis evaluation. It touches on a number of topics thatrelate to man's abilities, limitations, biases and predilections-as a processor of information or a user of eviderice. In some cases,it may appear to range beyond the specific subject of hypdtheSis,evaluation, and deal with "thinking more generally. Our reasonfor including this material is that it seems to us televant to,theproblem of decision making, and it appears to fit mare readily ,herethan elsewhere within our conceptual framework. In Section 8.6,the discussion bacomes,orrowly focused on the problem of revisingprobabilities in situations that have been formalized to the extentthat a Bayesian dgta-aggregation algorithm might be applied.

'8.1 Serial versus Parallel' Pro\gessin;q,

One gliestion of interest concerning the way people evaluatehypotheses is whether they consider them one, or several, at atime. Empirical'data are lacking on the question of which ofthese alternatiIes best characterizes man's approach to hypothesisevaluation. It is our impression that the prevailing consensus isthat the assumption of seriality is the more plausible of the two,insofar as the conscious consideration of hypotheses is concerned.

If the serial model is t4e more nearly correct, this mustrepresent a basic limitation of man. It is difficult to think ofa convincing reason why one should evaluate the,hypotheses seriallyif he is able to treat them in parallel.

But even if we, assume that one cannot test several hypothesesat once, there is still a question about'the order in which testingis done. One might apply an incoming datUm to each of the

(r

6754

AS

A

:*)


hypotheses in turn. AlteTnaEively, one might focus exclusively onone hypothesis until one had enough confirming data to accept it,or until the evideAceagainqit was sufficient to Warrant itsrejection,,. ih which case attention would be shifted to another'possibility. Note that-in this latter case a datum cannot be dis-carded after being applied to the evaluation of one hypothesisbecause it-may be germane to the evaluation of, others later.

One putative advantage of the Bayesian approach (seeSection 8.6) is that it forces the decision maker to apply anincoming daum_to each of the candidate hypotheses in turn. Oneof thQ implioatios,of this fact is that it minimizes, the needfor the decision maker or system to retain data. Assuilfing that'the set of'hypotheseh with which the. decision maker is workingis complete, and will not be extended, a datum cap be discardedonce it has been assimilated and the probabilities asqociatedwith all the hypotheses,, revised.

43.2 Subconscious Processes

What is happening at a.subsconscious level is, of course,even less kNrell-understood. The belief has been expressed thatthe brain carries on problem-solving activity even when one is notconsciously thinking about a problem. Wallas (1926) elaboratedand popularized the notion, which he, credits to Helmholtz, thatcreative thinking often involves a period of "incubation," whichTollowg a period of "preparation," and precedes a period of"illumination." During the preparation period, according to thisview, the problem solver conscious 1y. labors on the problem,; duringthe illumination period the problem solver becomes aware of thesolution for which he was seeking. No conscious attention isigiven to the problem during the incubation period, but, Wallacesuggests, much subsconscious exploiation of the problem takes place,

While the idea has primarily'anecdotalisupport, thetestimony of creative thinkers about the wayJthey have arrivedAt solutions to difficult problems is fairly compelling evidencethat something of this sort does occur. We mention it in thiscontext to make the point' that the fact (if it is a *fact) thatdecision makers tend to apply newly acquired data to the evaluationof only one hypothesis at a time, should probably not be taken asconclusive evidence'that the credibility of a hypothesis not underconsideration has not been affected by those data. Moreover, itis at least a plausible conjecture that the likelihood that anygiven hypothesis will"suggest itself" for expliciticonsiderationMay depend to some degree on suah subconscious activity (Maier,1931).

55

68

A

NAVTRAEQUIPCEN73-C-0128 -1

Dreyfus (1961) has argued that such subconscious, ormarginally conscious, activity is a general and difficult'-to-

simulate characteristic of man 4s a problem solver. It is this

ability that makes it possible for him to consider consciouslyonly the "interesting" moves. in a game of chess without explicitlyconsidering all possible moves and rejecting those that are not

worth pursuing. But subconscious processes are beyond the scopeof this report, so we will not pursue the topic further,

8.3 Man As An Intuitive Luician

Tdchnically, logic is the discipline which deals with therules of valid inference. The term is used colloquially, however,as a synonym for reasoning. It is of some relevance to thegeneral problem of decision making, and in particular to theproblem of training decision makers. to consider whether reasoning

as it is practiced by people is logical in the technical sense;

and, to the extent th4t it is illogical, whether it is illogical

in consistent ways. A further question of interest is whether

training in formal logic can reasonably be expected to improve

decision-making performance.

Philosophers have not been in agreement on the first

question. Henle (1962) points out that some of the 19th century

writers (e.g., Boole, 1854; Kant, 1885; Mill, 1874) viewed logic

as the science of the laws of thought. ,Some more recent writers

(e.g., Cohen, 1944; Rusell, 1904; Schiller, 1930) have treated

logic as something quite independent of thought processes and to

reject the notion that thinking necessarily conforms to logical

principles.* A middle-of-thd-road view is that thinking sometimes

conforms to logical. principles -- especially when one's explicit

purpose is to reason carefully and deductively- -and sometimes

does not.

*A cynic might assert that few arguments are won or lost on

logical grounds. Certainly, the alogical strategems that canbe applied to arguments are numerous, and perhaps are better

learned in the course of normal development than are the rules

of inference. The disputatious reader who feels his arsenal

of such strategems is deficient is referred to Schopenhauer(no date) who provides a veritable cornicopia.of them.

56

NAVTRAEQUIPCEN 73-C-0128-1.

Whether or not thinking is logical may be difficult todetermine empirically in any particular case, because the stepsby which one arrives at a conclusion usually are not availablefor observation. As Mill (1874) points out, since "the premisesare seldom formAly set out,... it is almost always to a certaindegree optional in what- manner the suppressed link shall be filledup... (A person] has it almost always in his power to 'Make hissyllogism good'by introducing a false premise; and hence it isscarcely ever possible decidely to affirm that any argument involvesa bad syllogism" (p. 560; from Henle, 1962).

Individuals undoubtedly differ greatly in their ability tothink logically, and any charaCterization of human strengths andweaknesses in this regard is bound to be only partially correct.There are many ways in which reasoning can be illogical, however,and it is not unreasonable to ask whether some of the many possiblieevidences of fallibility are appreciably more common ,.han others.Several ways in which human reasoning does seem to depart from theranks of logic have been discussed by Henle (1962). These include:failure to distinguish between thefactual truth of ,a conclusionand the logical validity of the argument on which it is based;restatement of a premise or a conclusion, which may have theeffect of preserving a logically valid. form, while changing thesubstance of the argument; the omission of premises from an argument,or the addition of spurious premises. The fallacy'of the "undis-tributed middle" is one that has long been recognized asparticularly bothersome, and involves the assignment of differentmeanings to the same term when it appears in different premises.

Another type of logical error that seemg.to be commonlymade involves a misunderstanding of the syllogiOtic form: "If Athen B; A; therefore B," or "If A then B; not 8; therefore not A."These forms may be perverted either as "If A then B; not A;therefore not B," or "If A then.B; B; therefore A." Bbth of theseforms are invalid; nevertheless most,readers- will probably recognize-them as forms that.they have encountered, and perhaps used, inarguments.

Wason (1974) describes a failure in reasoning that he hasobserved that seems to be related to this type of misunderstanding.Four cards are placed on a table so the subject can see only oneside of each of them. The cards contain retpectively a vowel, aconsonant, an even number and an odd number. The subject is told,that each card has a letter on one,side and a number on the other,and is asked which cards would have to be turned over to determinethe truth or falsity of the statement: "If a card has a vowel onone side, then it has an even number on the other." The majority ofWason's subjects chose either the card showing the vowel and theone showing the even number, or just the card showing the vowel.The correct answer is: the card that shows the vowel and the onethat shows the odd number. Only by finding an odd number behind

5 770

4,


the vowel or a vowel behind the odd number would the statement.befalsified. The students' choice of'the card with the even numberis

(a form of the fallacy known as asserting the consequent: "If

A then B; B; therefore A.",

This type of reasoning error occurs With sufficient,-consis-tency (at least among college students) to have prompted'investi-gation by several researchers. A completely satisfactory explana- rtion has not yet been forthcoming. Wason seems to favor the view

g that the choice of cards 1.4 made on an intuitive basis and thatthb "reasons" for the choice - which subjects give in response tothe experimenter's inquiries - are really rationalizations., "Thishypothesis is consistent with our crude knowledge.about intuition.A verdict may occur to a judge before the grounds which supportit have been spelled out; a chess player may "see" a good move-randthen analyze the continuations which validate it. Such thoughtsuggests a processing mechanism which operates at different levels"(p.385).

The last chapter on the topic of the relationship betweenlogic and thought has not been written. And it cannot be untilmuch more is knOwn about the workings of the human mind. The /immediate challenge for training research is to identify ways toimprove the capability of individuals to reason logically, or atfleast4to recognize and be able to avoid the more common illogicalpitfalls.

ti

8.4 Man as an Intuititatistician

It isequite clear that most individuals couldAanage to get,through life without e:er explicitly assigning a numericalprobability to an event. Undoubtedly, the vast majocr4ty of peopledo so. It seems safe to assume, however, that people do makejudgments of likelihoods, and that these judgments--even thoughnonnumeric, and often implicit--condition their behavior. Anindividual carries ah umbrella because he thinks there is a goodchance of rain, or buys stock that he expects to appreciate. Onepurchases.life insurance before boarding an airplane because one,in effect, has considered the likelihood that the plane will godown ddiag thatiflight to be nonnegligible; the fact that heboards the plane at all is probably evidence that he also considersthat likelihood to be something less than certainty. One choosesone among three job opportunities, because the chances of successand advancement are perceived as greater in the case of the selectedjob than in that of the others. In short, although most of us donot attempt to assign numeric probabilities to possible situationsor events, we behave as though our choices had been dictated byreasoning of the sort: this event is more likely than that, or thelikelihood of this situation is great enough so that I had betterdo thus and so in order to be prepared if it should occur.

NAVTBAEQUIPCEN 73-C-0128-1

A question of some practical interest, therefore, is that ofhow effectively such judgments are made. For many situations,there is no way to answer 'this question objectively. The individ-ual who selects one job from among'three possibilities because heconsiders the likelihood of success to be highest for that casewill never know for certain whether his judgment was correct.There are also, however, many situations for which the "objective"probabilities of events are known or can be determined, and wecan at least ask how well people do when asked to estimateprobabilities explicitly in thdse cases. The literature that isrelevant to this question falls fairly naturally into three cate-gOries. First are the studies that deal with people's ability toestimate the statistical properties of samples that they arepermitted to observe. Such studies concern relative frequenciesrather than probabilities, but to the degree that our ideasaboutprobabilities are based on, or influenced by, percei%,d frequenciesthey are germane. Second are some studies that have to do withthe extent to which.people's intuitive notions about the probabil-ities of events correspond to, or conflict with, the implicatiOnsof the theory of probability as represented in the probability cal-culus. Third are numerous recent experiments that consider thespecific question of how effectively people function as Bayesiandata aggregators. In this section we will consider briefly thefirst of these three categories of studies; in Sections 8.4 and 8.5we will consider the last two.

People appear to be reasonably good at perceiving proportions,or the relative frequencies of occurrence, of both sequential andsimultaneous events (Attheave, 1953; Peterson & Beach, 19.67;Schrenk & Kanarick, 1967; Erlich, 1964; Vlek, 1970) and at esti-mating the means of number sequences (Beach & Swensson, 1966;Edwards, 1967)., Inferences'concerning the median or mode of askewed distribution (assuming the subject knows the definitionsof these terms) are fairly accurate, and the estimated mean of suchdistributions tends tb be biased in the direction of the median(Peterson & Beach, 1967). One's confidence is one's estimate ofthe mean or the variance of a population appears to increase as thesample size increases (Peterson & Beach, 1967; but see also Pitz,(1967).

Estimates of the variability of a set of data often tend todecrease as the mean increases (Hofstatter, 1939; Lathrop, 196/;Peterson & Beach; 1967). Peterson and Beach (1967) point out thatwhile the notion that variability is necessarily inversely related'to the mean is erroneous, it is intuitively compelling. "Thinkof the top of a forest. The tree tops seem to form a fairly smoothsurface, considering that the tree may be 60 or 70 feet tall. Now,look at your desk top. In all probability it is littered with manyobjects and if a cloth were thrown over it the surface would seemvery bumpy and variable. The forest top id far more variable thanthe surface of your desk, but not relative to the sizes of theobjects being considered" (p. 31). One is led to wonder whether

59

/


the finding that estimated variability tends to decrease withincreasing*mean might be due in part'to failure by the subjectto understand that it is:an estimate of absolute variability thathe is to produce. Relative variability probably often does de-crease as the mean increases (to cite Peterson and Beach's treetop example), and without explicit instructions to the contrary itwould not be unreasonable for a subject to suit the terms to thecontext, as one does when one speaks both of a small skyscraperand a large dog.

8.5 Intuitive Probability Theory

How closely do man's intuitions about probabilities corre-spond to the implications of probability theory? The questioncannot be answered decisively, but a number of pertinent observa-tions can be made. For example, people often seem to find itdifificult to believe that, the outcome of an event can bt.independent of what has preceded it. This difficulty is sometimesmanifested in the "gambler's fallacy", (a fallacy, that competentgamblers probably would not make), one form of which holds that arun of successes increases the likelihood of a failure, or viceversa (Cohen & Hansel, 1956). Another example of assumed dependenceamong successive events has been noted by Jarvik (1951), who foundthat when given a two-alternative prediction task, subjects oftentended to predict the more frequent event after one occurrence ofthe less frequent event and to predict the less frequent after twoconsecutive occurrences of the more frequent event.

Several experimenters have &land that man does not estimatethe probability of compound events very accurately. In particular,when assessing the likelihood of the joint occurrence of severalindependent events, he tends to produce estimates that are toohigh (Cohen, Chesnick, & Haran, 1972; Fleming, 1970; Slovic, 1969).Conversely, when estimating the probability of disjunctive events--the pro4bility that any one of several specified events will occur--

/ he tends\to produce estimates that are too low (Cohen, Chesnick,& Haran,` 972; Tversky & Hahneman, 1974). The overestimation oftheproba of conjunctive events is consistent with the ob-servation t at people frequently base judgments of the degree ofcorrelation between two events on those cases in which the outcomesof interest dd,occur together without giving sufficient considera-tion to those cases in which they do not (Peterson & Beach, 1967).

What is of more interest than the fact that man's intuitionssometimes lead to incorrect judgments about event probabilities isthe question of the extent to which the failings of intuition--at least insofar as they are systematic--are explainable in termsof identifiable ways in which such judgments are made. In arecent series of studies, Tvers:cy and Kahneman (1971,.1973, 1974;

7360

1

NAVTRAEQUIPCEN 73-C-0128-1"1

Kahneman & Tversky, 1972, 1973) 'have explored this question t Thegeneral approach in these studies and in those of others whaspaveconducted similar investigations (e.g., Alberoni, 1962; Tund,41964;Wagenaar, 1970) has been to ask people to estimate the probabilityof the occurrence of a hypothetical event, or, perhaps more commonly,to indicate which of two such events is the more probable. Onemight be asked, for example, to indicate which of the two followingsequences of coin tosses is the more likely, HHHHTTTT or HHTHTTHT;or to indicate which of two hospitals -- .which record approximately15 and 45 births a day, respectively--would have the largestfrequency of days on which more than 60% of the babies born areboys.

The results of these studies have revealed a number of waysin which the answers that people give to such questions departsystematically from the objective probabilities of t1 events asinferred from the application of probability mathematics. Tverskyand Kahneman attribute such failures in judment to the heuristicprinciples that people often use when attempting to. estimateprobabilities or relative likelihoods.

It will be helpful, before considering some of Tversky andKahneman's specific results to digress briefly to consider thenotion of A heuristic principle or procedure. The term "heuristic,"which comes from the Greek heuriskin, meaning' "serving to dis-,

Icover," appears sporadically in the literature of philosophy andlogic as the name of a branch of study dealing with the methodsof inductive reasoning. It was revived by Poiya (1957) in hisclassic treatise on problem solving, and used to connote inductiveand'analogioal reasoning leading to plausible conclusions, asopposed to the deductive developments of rigorous proofs. Inrecent years, computer scientists, and especially researchers inthe area of machine intelligence, have appropriated the term toconnote "a rule of thumb, strategy, trick, simplification, or otherkind of device which drastically limits search for solutions inlarge problem spaces" (Feigenbaum & Feldman, 1963, p. 6). In short,a heuristic principle or procedure, usually referred to simply as aheuristic, is a means of making an inherently difficult problem moretractable. The criterion by which a heuristic is measured is itsusefulness. It is important to bear in mind, however, thatheuristics are not expected to lead invariably to correct solutions."A 'heuristic programs,' to be considered successful, must work wellon a variety of problems, and may often be excused if it fails onsome (Minsky, 1963, p. 408) .

6,1/


8.5.1 Representativeness

Tversk and Kahneman describe two heuristic principles ---representati eness and availability -which they 'feel account formany ofthe systematic judmental biases that they and otherinvestigators have ieserved. According to the representativeneSsprinciple, "the subjective probability of an event, or a sample,is determined by the,degree to which it: (i) is similar ineSsential gharacteristics to Vts parent population; and (ii)reflects the salient features of the process by which it is gen-

_yrated" (KahneMan & Tversky, 1972, p. 430). Several examples ofthe application of this principle are given; two will suffice forour purposes, one illustrating each of theSubprinciples.

The importance of the similarity between the judged eventand the parent population is illustrated by the follow2ng question:"All families of six children in a city were surveyed. In 72families, the exact order of births of boys and girls was GBGBBG.What is your estimate of the number of families surveyed in whichthe exact order of births was BOMBE?" (Kahneman & Tversky, 1972,p. 432). If the probabilities of male and female births wereexactly equal,,the two birth sequences would be equally probable.(Apparently, the frequency of male births is slightly higher thanthat of female births, so the lattet sequence is slightly moreprobable than the former.) About 80% of the subjects (high-school-students) who were asked this question judged the latter sequence.

to be less likely than the former; the median estimated number offamilies with this birth order was 30. Kahneman and.Tverskyattributed this result to the fact that the two birth Sequences,while about equally likely, are not equally.representative offamilies i the population. The former sequence is more similarto a larger proportion of the population, both in terms of the,relative number of girls and boys, and in terms of the length ofruns of births of the same sex.

The second way in which the representativeness heuristicmanifests itself--in sensitivity to the degree to which. an eventrefledts the salient features. of the process that generated it --is illustrated by the tendency of people to consider regdlaritiesin small samples to be inconsistent with the assumption that suchsamples were generated by a random process. Thus, when people areasked to produce random sequences such as the results of an imaginedseries of coin tosses, they tend to produce fewer long runs thanwould a truly random process. Moreover, in judging the randomness,of small samples, they are, likely to reject as nonrandom many ofthe samples that a random process does generate. Kahneman andTversky characterize the intuition that produces such judgmentalbiases as a belief that a representative sample should representthe essential characteristics of the parent population, not onlyglobally, but locall§ in each of its parts. In other words the

NAVTRAEQUIPCEN 73:-C-0128-1

observed behavior is consistent with the belief:that the law'oflarge numbers applies to small numbers as well (Tversky & Kahneman,1971).

The application of this heuristic could lead ?!ne to the sortof fallacious thin'-ing illustrated by the conclusion that theprobability of finuing more than 600 boys in a random sample of 1000children is the same as that of finding,more than 60 boys in a randomsample of 100 children. The probability of the latter event is,bf course, much greater than that of the former, Kahneman andTversky ,(1972) showed that people (at least high-school students)do virtually ignore the effect of sample size when estimating theprobabilities. of random events of this sort. In general, the '

estimates made by Kahneman and Tversky's subjects., when asked tojudge the probability of events that have a binomial distribution, ,

were much mqre appropriate for small samples (e.g., than forlarge samples (e.g., 100 or 1D00). In other words, foi large samples,subjects tended ,to underkstimate grossly the prqPability of high-pro4bility events 4nd overestimate the probability of low-proba-bility events, and the magnitude of the miss increased with thesize of the sample.

8.5:2 Availability

The availability principle,' according to Tversky and Kahneman(1973) is used whenever one bases estimate's oif frequency or prob-ability on the ease with which instances or associations arecalled to mind. For example, when asked to estimate the relativelikelihoods of heart attacks for men and women, one might think ofmale and female victims of heart.attack among one's personal acquain-tances an take the ratio as. an estimate of the relative likelihoodsin the population. Or, if asked to judge which of two letters occursthe More frequently as the first letter of English words, one mightattempt to think of a few *ords of each class and make the judgmenton the basis of the rapidity with which examples come to mind.

Tversky and Kahneman point out that "availability" is anecologically valid cue for the judgment of frequency because, ingeneral, more frequent events are easier to recall or ima ine thaninfrequent ones. However, availability is also affected'13y variousfactors, which are unrelated to actual frequency. I the availabilityheuristic is applied, then such factors_ will affe the perceivedfrequency of classes and the subjective probabili y of events.Consequently- the use of.the availability heuristic leads tosystematic biases" (1973, p. 209).

As one example of how application of the availability heuristibcan lead to an erroneous judgment, Tversky and Kahneman report thefollowing experiment. Subjects were asked to estimate the number ofdifferent remember committees that can be formed from a group of 10

63 76


people. The estimates tended to decrease with increasing r forvalues of r between 2 and 8. In particular, subjects typicallyjudged it to be possible to form many more committees of size,2than of size' 8, when in, fact the same number is,possible in both'cases.. (Similar results were obtained when subjects were askedto estimate the number of different,patteins of r stops that a buscould make while traversing a route with 10 stations between startand finish.) The explanation for this result, according to Tverskyand Kahneman, lies in the fact that committees/of two members aremore readily imagined than those of eight, and, consequently, appeart6 be more numetous.

The major difference between the heuristic principles,ofrepresentativeness and availability, Kahneman and Tversky suggest,is in the nature of the judgments on which the subjective prob-ability estimates are based. "According to the representativenessheuristic,one evaluates subjective probability by the degree ofcorrespondence between the sample and the population, or betweenan occurrence and a model. This heuristic, therefore, emphasizesthe generic features, or the connotation, of the event. Accordingto the availability heuristic, on the other hand, subjectiveprobability is evaluated by the difficulty of retrieval and con-struction of instances': It focuses, therefore, on, the particularinstances, or the deriotation, of the event. Thus, the represen-tativeness heuristic is more likely to be employed when events arecharacterized in terms of their general properties; whereas, theavaiaability hepristic is more likely to be employed when eventsare more naturally thought of in terms of specific occurrences"(Kahneman &.Tversky, 1972, p. 452). A feature commqn to bothheuristics is their reliance on mental effort as an indicant ofsubjective probability. "It is certainly harder to-imagine anuncertain prcicess yielding a nonrepresentative outcome than toimagine the same process yielding a highly representative outcome.Similarly, the Jess available the instances of an event, the harderit Is to retrieve and construct them" (ibid, p. 452).

8.5.3 A Methodological Consideration

There is a methodological consideration relating tosome of the findings of judgmental biases that deserves moreattention than it has received. This has to do with the possiblerole of language ambiguities. We have already alluded more than onceto the well known fact that the meaning of language is conditionedby the situation in which it occurs.* To borrow an example fromDreyfus (1961), "a phrase like 'stay near me' can mean anything from'press up against me' to 'stand one mile away,' depending uponwhether it is addressed to a child in-- a- -crowd or a fellow astronautexploring the moon" (p. 20). Although it seems unlikely that manyof the results that have been mentioned above can be attributed tothe imprecision of language, the possibility that some of them

64

77


may be based, at least in part, on this factor should not 'be

overlooked. The finding .that the estimated variability of a set

of.data tends to decrease as the mean increases was mentioned in

a preceding section as one possible case in point. Tversky and

Kahneman's finding that people judge it to be possible to form a c

larger number of different 2-man.committees than 8-man committees

from a pool of 10 men may be another. There is a way of defining

"different" (e.g., "having no people in common") such that the

judgment would be valid, and before one can take the results as

evidence of faulty intuitions concerningcombinatorics, one must

be certain that none of the'subjects is using such a definition.

Our guess is that language ambiguities will not go far toward

explaining the results obtained by Tversky and Kahneman, but it

seems conceivable that they may have played some role, and some

further research.might be directed toward determining the extent

of that role.

8.5.4. Training and Intuitive PtSbability Theory

We have reviewed these results at some length because this

general line of research strikes us as being not only exceptionally

interesting from a theoretical point of view, but of considerable

practical significance. To the extent that the heuristics that

have been identified are representative of the ways in which people

generally make judgments of likelihood, it is clearly important to

determine those conditions under which they lead to erroneous

judgments and those under which they do not. Tversky and,Kahneman '

have demonstrated that there are at least some situations in which

judgments, that are presumably based on identifiable heuristics,

err in systematic ways. This )toes not, of course, establish that

these heuristicS are, on balance, bad, as they are careful to

point out. What one would like to know is the relative frequency

with which they lead to erroneous decisions in practical real-life

situations. From the point of view of the training of decision

makers the question is how to foster the use of such heuristics in

situations in which they are most likely to be effective, whilediscouraging their use in situations in which they are likely to

lead to erroneous judgments. Perhaps at least a small step in

that direction would be to make decision makers explicitly aware

of the nature of.the heuristics that tend to be used in estimating

probabilities, and of the types of- erroneous decisions to which

they'can some, imes lead.

:65

78

NAVTRAEQUIPCEN 73-C-0128-;

8.6 Bayesian Inference

Undoubtedly the most widely advocated formal approach to theapplication Of incoming data to the evaluation of hypotheses isthe "Bayesian" approach. Because it has attracted so much atten-tion and has been the focus of so much research, we will considerit in som' detail.

8.6.1 Bayes Rule

It is necessary to begin with a set of mutually exclusiveand exhaustive hypotheses, Hi, concerning the state of the world.To each of these hypotheses one must assign .a probability, p(Hi),that that hypothesis is true. Because these hypotheses are, bydefinition, mutually exclusive and xhaustive, it follows that thea priori probabilities sum to one,

Ep(H.) = 1.(1)

Inasmuch as the hypotheses that one is considering are likely6) have different,implications concerning what might be observed-under specified conditions, it seems intuitively reasonable thatone should be able to increase one's degree of certainty concern-,ing the truth or falsity of any given hypothesis by making appro-priate observations. For example, if Hi implies D, and if D isobserved, then the credibility of H1 might reasonably be expectedto be increased. (The truth of Hi is not proved by such an obser-vation, of course, inasmuch as it does not follow from the factthat Hi implies D that D implies Hi; as was pointed out in Sec-"tion 8.3, inferring the truth of Hi from the observation of Dwould involve the logical fallacy known as "asserting the con-sequent.")IfbothH.and H. could lead to D, but the likelihoodof D given H. is greater thaik its likelihood given H., then ourintuitive notions about evidence suggest that the observation ofD should increase our confidence in Hi somethat more than ourconfidence in 114. These notions were expressed formally by the18th Century British minister, Thomas Bayes, in the so-called"inverse probability theorem"--a theorem or rule that has beenthe subject of much'debate.

Bayes rule expresses p(HilD), the probability that Hi is truegiven the observation, or datum, D, as a function of p(DIHi), theprobability that D will be observed given-Hi is true, and p(Hi),the probability that Hi is true as determined prior to the obser-vation of D. The probability of an observation given a hypothesis,p(DIH) is usually referred to as a conditional probability; the

66

79

NAVTRAEQUIPCEN 73-Cr01211-1

probability'of a hypothesis given an observation, p(4), is usuallycalled.a posterior probability. Bayes rule defines a procedure for

,using the fact that D has been observed, to adjust one's estimate ofthe probability that Hi is true. The rule may be written as

p(HilD)

p(DIHi) p(Hi

np(H.)

j=1 %

(2)

where n is the total number of hypotheses in the set. BecauseEp(DIH.) p(H.) = P(D), equation (2) may be simplified ,t0

p(DIHi) p(Hi)p(HilD)

p (D)

(3)

When a sequence of observations is made, the rule is applied

recu ively, and the value of p(HilD) that is computed as theresu t of one observation becomes the p(Hi) for the followingcompu ation. That is to say, the posterior probabilities result-ing from one observation become the prior probabilities for thenext one. Thus, equation (3) may be written more appropriately as:

p(DIHi) pn-1

(HiID)

pn(HilD)

Pn-1

(D)

(4)

where pn(HiID) represents p(HilD) after the nth observation, andpo(HilD) or, more appropriately, po(Hi), is understood to be theprobability of Hi before any observations are made. We will followthe convention of using subscripts only when they are essentialfor clarity.

Bayes rule states, in effect, that if the prior probabilityof a hypothesis being true, p(Hi) and the probability of observinga particular datum given that hypoINgsis is true, p(DIHi) are knownfor all i, then the probability that the hypothesis is true given

67

r


that the datum has been observed, p(HilD), can be calculated ina straightforward way. ,In many decision situations, p(Hi) andp(DIHi) are not known, and cannot be determined objectively;therefore, they must be estimated. The significance of the rulesterns from the assumption, for which there is some evidence thatwill be considered later, that people are better at estimatingconditional probabilities, p(DIH), than at estimating posteriorprobabilities, p(HID). Obviously, if they were invariably verygood at estimating p(HID) there would be no need to make use ofBayes rule to calculate this value; it would suffice to have thedecision maker estimate it directly.

8.6.2 Likelihood Ratio

In order totlake use of Bayes rule it is not necessary torequire that an individual estimate probabilities explicitly. Analternative procedure is to have him judge the ratios of pairs ofconditional probabilities. Such ratios are referred to aslikelihood ratios. The likelihood ratio of D given H1 relativeto D given H2 may be expressed as follows:

p(DIH1)

L1,2

p(DIH2)

(4)

The.attractiveness of likelihood ratio stems from the factthat people often find it easier to make the-implied judgmentthan to estimate conditional probabilities directly. The type ofjudgment that is required in this case is of the sort "Event Dis X times as likely if Hi is true than if Ho is true." Neitherof the conditional probabilities need be spe8ified explicitly.A disadvantage associated with its use is the fact that a greatmany more judgments are required with respect to each observation.

8.6.3 Other Methods for Obtaining Probability Estimates r

Other methods have been used to obtain-probability estimateswithout having the subject explicitly produce numerical values.For chips-in-urn problems, foF example, Peterson and Phillips(1966) have had subjects adjust markers on a scaled 0-to-1 con-tinuum so that each interval is equally likely to contain thetrue proportion of chips of a specified color. Organist (1964)developed a simple answer chart which forced &subject to makehis distribution of probabilities over the possible hypothesessum to one and also specified what his payoff would be for each

68

r 81


hypothesis if it were'correct, given theoprobability that heattached to it. Shuford 1967) describes a computer-controlledsystem which presents a subject with a set of hypotheses andallows him to specify probabilitibs by adjusting the lengths oflines associated with the hypotheses by pointing at theM with alight pen. When one line is lengthened or shortened, compensatoryAdjustments are automatically made in the remaining line so thatthe probabilities always sum to one. This system also pt vides theuser with information concerning the implications-of his probabilityassignments vis-a-vis his payoff, given the truth of anyparticular

8761-4Diaptirfidity-TIC-Trata

Intuition suggests that the more dispatate the implidltionsof two hypothesest the more informative data shcSuld b, concerningwhich of-the hypotheses is likely to be true. In a Bay-siancontext the informativeness, or "diagnosticity,"of data is definedin terms of the likelihood ratio. Specifically,, the magnitude ofa likelihood ratio is said to represent the diagnosticityof adatum with respect to the two particular hypotheses involved. Themore the ratio differs from 1:11'in either direction, the moreinformative the datum is with respect to which of the. hypothesesunder consideration is correct, and the more the distribution ofprobabilities over'these hypotheses will change as a consequence.,

8.6.5 *Odds

The ratio of two posterior probabilities is referred to asthe posterior "odds" with respect to the associated hypotheses.The posterior odds of H1 with respect to H2 may be expressed as

pn

(1:1

1ID) rp(DIH

I) p n-1(H1 ID)

Pn(H2ID) (D)

or, equivalently, as

[

p(DIH2) Pn-1('H21D)

(6)

p(D)

Pn(HlID) p(DIH1Pn-l(H11E4

Pn(H2ID) P(DIH2) pn.01(H2140)

,69'

(7)


which Is to say that the, posterior odds is, simply the prior oddsmultiplied by the likelihood ratio. Letting represent theodds of Hi with respect to Hi after the nth observation, we mayexpress the relationship as follows;

Qn;i,j Li,j'%-1;i,A (8)

Obviously-1QJ . = Q. . (9), 1 1, 3

andL. . = L. -1 ;13).

3,1 1,3

Often it is clear from the context which of the two terms ofeither an odds ratio or a likelihood ratio is to be the numeratorand which the denominator, so the subscripts are omitted and theexpression is written more simply as

Qn LQn-1. ¶11)

It is essential, however, that the same hypothesis, whether H. or H4,be represented in the same position (numerator or denominatort inboth.ratics.

8.6.6 Applications of Bayes rule in The Two-Hypothesis Case.

To summarize what has been said so far, Bayes rule representsa procedurefor evaluating hypotheses in situations that have thefollowing characteristics: (a) the possible states of the worldcan be explicitly represented by an exhaustive, and mutuallyexclusive set of possibilities; (b) discrete observations may bemade in an effort to find more information about the actual stateof the world; and (c) for the data obtained from each observation,it must be reasonable to assign a number that represents theprobability that those data would have been obtained, given thetruth of any specific one of the hypothesized states of the world.In order to get an appreciation ofihow Bayes rule extractsinformati6n from data, it will be helpful to consider some concreteexamples of decision tasks to which the rule might be applied. Wewill focus first on the simple case in which the hypothesis setcontains only two alternatives.

83


Imagine an urn containingAred and black chips. Suppose twbhypotheses, H1 and IL), are stated, one and only one of which 1strue, concerning what proportion'of the chips in the urn are red.The task is to decide which of these hypotheses is the true one.Data are obtained by sampling chips one at a time, replacing eachchip after it is-examined. Assume that the'ohips are thoroughlymixed before each observation and that the probabilii of drawj.nga red chip on' a trial is exactly R/R+B, where R is the number ofred chips, and B the number of black chips, in the urn.

Suppose the first hypothesis, Hi, is that 70% of.the chipsare red4 and that the second hypothegis, H2, is that 20% of thechips are red. Suppose further that the prior probabilities areequal, that is, p0(H1) = p0(H2) = .5. ,Figure 1 shows how

p(Hi(D) and p(H2JD) change as a result of applying Bares rule to,

the data obtained in the following ten successive observations:RRBBRRBRRR. Figures 2 and 3 show the odds, and the uncertainty,in the information theoretic sense of the word, change from ob-servation to observation. Uncertainty is, of course, a monotonebut nonlinear function of the difference between the probabilitiesassociated with the two hypothess.

Note that the effect of,..an observation is no necessarily todecrease the amount of uncertainty concerning whi"611 hypothesis istrue. If the distribution of p(Hi) favors the incorrect hypothesis,uncertainty is very likely'to ihcrease as a result of observing .1

dataebefore it decreases. Even if the distribution of p(Hi)favors the correct hypothesis, or weights both hypotheseg equally,uncertainty maincrease on individual trials. In this case, how-ever, it will decrease on the average, assuming unbiased sampling.

Another interesting and perhaps counterintuitive observationconcerning figure 1 is the very large effect that the one or twoinitial observations can have In our example, the initialdrawing of two successive reds had the result of making one of the(initially equally likely) hypotheses over twelve times more likelythan the other.

Intuitively, one would expect that the degree of confidencethat one should have that the proportion of reds and blacks in one'ssample reflects the true proportion in the population should dependon the sample size. That the application of Bayes theorem does notviolate this intuition may be seen by comparing the probabilitydistribution after the third observation and after the sixth obser-vation (figure 1). In both cases, red, chips have comprised 67 percent

I

71


.1

N_

v-s-

/ \/ \2k\\ '/ ' .-6.Iti

I L 1 1 I - -AN--A3

_c_111-.

4 5 6 ,7 8 9 10 11

OBSERVATION NUMBER

R'RB AR R eR R

DATUM

100 100 67 50 60 67 59 62 67 70PERCENT OF REDS IN $AMPLE

H1. 70% Red; H2: 20% Red;

P0.(H1)p0 (H2)

.5

Figure 1. Changes in posterior probabilities,

p(HilD) and p(H2ID) as a result of

the indicated observations of Red and

Black chips

s5 72

NAVTRAEQUIPCEN.73-C-0128-1

3

R R

100 100,67PERCENT OF REDS IN SAMPLE

4 5 6 7 8 9 10 11

OBSERVATION NUMBER

88R R 8R R RDATUM

50 60 67 59 62 67 70

H1: 70% .ed; H2: 20% Red;

P0(H1) P0(H2) .5

Figure 2. Changes in odds, 521, as a result of the

indicated observations of Red and Black chips

/ V.


921 .;

/I

10 3 113 4 5 6 7OBSERVATION NUMBER

8

R R BB'R R BR R R-DATUM

100 100 67 50 60 67 59 62 67 70PERCENT OF RE DS IN SAMPLE

H1: 70% Red; H2: 20% Red;

p0 (H1) p0 (H2).5

Figure 3. Changes in uncertainty concerning hypotheses

as a result of the indicated observations of

Red and Black chips


of the sample; however, the uncertainty'is less following the.,sixth

observation than followirig the third, reflecting the fact that the

sample size was larger in the former case.

Figure 4 shows how the probabilities change over the course

of ten observations in/which reds and blacks occur, with the same

frequency but in a different order. In particular, the first twoobservations in this case produced blacks, and the second two, reds.$5 bservations 5 through 10 are assumed to be the same as in the

original example. Note that by the end of the fourth trial, the

proportion of red and black draws was the same its poth examples;'

consequently, the probability distributions are the same at this

point and thereafter. This illustrates the fact that the Bayesian

calculation of p(H11D) is Pah-independent, in the sense that theeffect'of an observation is strictly dependent on the-Current value

of-p(H.), and independent of the particular sequence of observations

on which that value is based. The calculation is also ,ndependent

of the number of observations on which the current value of p(Hi)

is based. Note that, this point is different from the one made aboveconcerning the effect of sample size.on uncertainty. The point

that was made above was that the probability that ,a given proportion

of reds in;one's sample accurately reflects the proportion in thepopulation'tincreases with sample size. The Point here is that theeffect,that an observation will have is independent of howP(Hi) got to be whateverit is.

4

Figures 5, 6 and 7 illustrate' the effects of setting

the initial values of p(H1) and p(1-1,.0. to something other than .5.

The sequence cef draws is identical to that in figure 1, and cOng-

sistent with what might be expected if the true hypothesis were H1.

In each figure, one curve shows the effect of&these observationsgiven that pn(H.1) = .8; another shows the effect given that pri(Hi

Initial

)

J-2.2, and th6 tftirdlrepresent6 p(H,).= .5. The main thing t61,

notice is that the effect of an nitial incorredt bias is larqfly.44

nulled out by relatively few observations. This point is frequently

made by proponents of Bayesian information processing systems in

response to the observation that a,priori probabilities are some-.1.

times difficult to assign on anything other than an arbitrary basis.

A fact that usually is not pointed out is illustrated in figure 7:

changing the distribUtion of a priori probabilities shifts thfunction relating log odds to data by a constant.

8.6.7 Expected Effects of Observations on Hypotheses

In the foregoing examples of the application of Bayeshave considered how probabilities may chage as a result of asequence of specific observations. It has been apparent from theseexamples that the effect of an observation sometimes is to increasethe probability associated with the true hypothesis and sometimes to

decrease it. On the average, however, we expect the probabili4ty

75

olt

NAVTRAEQUIPCEN 73-C-012B-1

O PC Hi ID)

P( Hz ID)

0

3 4 5 6 7 9 9 10 11

OBSERVATION NUMBER

B R R R R BR R RDATUM

O 25 50 60 67 59 62 67 70PERCENT OF REDS IN SAMPLE

H1: 70% Red; H2: 20% Rea;

p0 (H1) = p0 (H2) = .5

Figure 4. Changes in posterior probabilities, p(H1ID)

and p(H2ID) asa result of the indicated

observations'of Red and Black chips

(Note that the results of the observationsare the same as in figure 1 except that thefirst four produce a different ordering ofReds and Blacks.)

76

f


F0),(H1)...90

o P(N1) -.50E F:,(1-11)..,20

4 5 6 10 11

OBSERVATION NUMBER' __

R R B B R R B R R R

DATUM

100 100 67 50 . 60 67 59 62 67 .70PERCENT OF REDS IN SAMPLE

H1: 70% Red; H2:_20$

4

Figure 5. Effects of indicated observations on

p(HilD) for different values of p0 (11)

NAVTRAEQUIPCEN 13 -C- 0128 -1

3 4 6 6 7OBSERVATION NUMBER

R R-BEIRRFIRRDATUM

* 100 100 67 60 60 67 '69 '62 67 70PERCENT,OF REDS IN SAMPLF

HI: 70% Red,; H2: 20% Red

a

Figure 6. Effects of indicated observations on odds,

511,2 for different initial values of p(H1

)

978

.1

ti

f1AVTRAEQUIPCEN 73-C-0128-1

7

940, 4 5 6 7 8 9 10 11

OBSERVATION NUMBERBRRBR RRDATUM

100 100 67 50 60 67 69 62 67 70PERCENT OF REDS IN SAMPLE

H1: 70%,Red; H2: 20% Red

Figure 7. Effects of indicated observations on

uncertainty flaNdifferent initial vallesof p(Hi)

4


7( .4

associated with the true hypothesis to increase with each observa-tion, and that associated with a false hypothesis to decrease, as-suming an unbiased sampling of the data. We turn now to a con-sideration of how p(HilD) can be expected to change, on the average,as aresult of applying Bayes rule, if chips are drawn from an urnthat contains reds and blacks in the proportion indicated by aspecified hypohesis, say H ..

It will facilitate the discpssion to begin by considering allpossible outcomes of a specific experiment. Figure 8 shows allpossible effects of four observations on p(HilD), in the case of)our example of 111:' 70% ft, H2: 20% R, 'arid p(Hi) =.5. Each nodein the graph reptesents one possible value of p(1111D) after thenumber of observations indicated on the abscissa; no values arePossible other than those represented by nodes, (By rotating thegAph in figure 8 about a horizontal axis passing trough the .5pointlon the ordinate, one would produce the,graph of p(H21D);which is to say,each of'the points in the graph ofp(H21D) it thecomplement of a,point in the graph of p(H1D).) In general; afterN observations, p(HilD) will have one of N-4-1tkossible values. Aftertwo observations, for example, p(HilD) will have one of the, three values.925, .568, or .123. The number above each node indicates the.number of ways to arrive at that node. There are three ways, forexample; to arrive at the node at p(HilD) = .821, N = 3: RRB, RBRand BRR. The set of numbers associates with a given value of Nwill be recognized as the coefficients of the terms of the expansionof (a+b), the so-called "binomial coefficients." In our appli-cation, each of these coefficidnts, which may be written as (g),represents the number of ways that N ekrents can be composed,of in events of one type and N-m. of another. The events of interestin our case are draWs of chips from an urn, 'and tie two types'are draws-of red and black chips, respectively. The sum of thesecoefficients for given N,

E IT=2)

m =0

represents the number of uniquely ordered sequences of reds and 4

blacks t5at'can result from NStays. Inasmuch as the effect of )applying Bayes rule to a sequencecrf data is insensitive to .theorder in which the data are considered, it is convenient to tqinkof all sequences having the samd canbination of reds and blacks.

. as the same outcome, irrespecti yl. of the order in which.the reds-and)placks have occurred. Thus;"'t'he effective number 9f possibleoutcomes of N 'draws is N-1-1 rather Johan 2N

Figure showk the graph of possible oat omes for ourhypothetical "experiment as they peTEalff-E6 p(HilD). By the

9380

6

NAVTRACENEQUIP 73-C-4128-4

Figuire 8.

H1 ': 70% Red; H2: 20% Red;

)

Po(Hir =,.7

ra.

All possible values of p(HIID)

(.7after N observations

tr,


algebra of expectation, the expected, or mean value of p(HlID)after N draws is the weighted sum of all_possible values, 6achvalue being weighted by its probability of occurrence. In termsOf figure 8, the expected value of p(H1ID) after N draws may'befound by multiplying the product of the value of each node above.N on the abscissa by the probability\qf arriving at that node,and summing over the products.

Suppose that the probability that an observation will yielda red chip (solid lines) is q, and the probability that, it willyield a black one (dotted lines) .ig 17q. The probability ofarriving at a given node in the graph, via a particular path, isthe product of the probabilities associated with the links in thatpath. The probability of arriving at a given node, irrespectiveof the path, is thp sum of.the probabilities associated wlth allpo'ssible paths to that node. But every path leading to a commonnode has exactly the sate probability of being.traverset.., becauseeach is composed of the same combination of R and B links. 'So,the easy way to calculate the prdbability,of arriving at a nodeis to take the product of the probability of traversing any spe-cific path to that node and the number of path leading to that node.Figure 9 shows expressions for these probabilities for eachof the nodes in our sample graph. In general, the probability ofarriving at a given node via a specific path composed of'm R ljnksand N-m ,B'links, is given by

qm(1-q)N-m

and the probability of arriving at a node via any such path by

TrNm (g)in:(1-q)N-m(13)

The expected value of p(HilD) after N observations, then,N, is given by

E [pm(HilD)N

EPN,mE Tr i4,rn(14)

where P represents the posterior probability of Hi after NobservaEiRns, m of which have yielded red chips.

e

82

NAyTRAEQUIPCEN 73-C70128-1

The following iterative formula can be used to compute fl :

(N,m)q7TN,m+1 (m+1) (1-q) N,m

where

= (1,q)N,

and computation can be,simplified by taking logarithms:

4

where

and,

=log log x + log TrN,m7rN,m+1

AN-m).(q)x (+1) (1.-q)

log TrN,0 = N log (1-0.-" (19)

The value of q in equation (13) id ends, pf course, on whichof theypotheses under consideration appens to be true. The -

expectatio4 can be computed, however, for each of the possibilities.The general expression may be written as follows:

E {pm (Hid Hi

'

]is trueN

. m=0

m ((I:1)p(RIH.)11mp(B1 (20)14,m,

1

I

1

C

4

t

1-11 R./ Hz:.-20t4e

S

%

NAVTRAEQUIPCEN 73C-0128-1. 1

S

.

54

Ns o

2t0CO)N.

\\141,2(1 -cb

fp'

.\ ,-N

4,

,..46-13)4

.1

1

Ac

Figure 9'. Graph illustrating,the'computetion of expectedvalues of posterior probabilities

I (The expression above each node.represents,e..

.the probability of arriving'at that npde, givenq is the probability of drawing a red chips The..0- it expected value of the posterior probabilityfollowing N observations is the sum of the valuesofthe nodes above .N, each weighted by its

1 "drrival" probability.

J84

/.1

ti

J

NAVTRAEQUIPCEN 73-C-0128-1 ,

Q Agure IN shows E[pN (H.ID)] for'our example (H1. 70% red,

HH2: 20% red), given Hi is true (top cjirve) and given H. is not trde

( bottom curve): The top curve shows the expected growth ofp(H1ID) wheh H1 is true antrof,p(H2ID) when H2 is true.' tonversely,the bottom curve represen the expected decline of p(ailD) whenH2.is true and of p(H2ID) tw Zen Hi is t rue. Thus, in theotWo-

c.......,,

alteuative case,.

H true..

= E fp(H2ID)1H2 is true] . (21)4

I

. ,/'

-To compute the. expected uncertainty following N olservations,one must compute the uncertainty associated-with each of,:the possibleoutcomes of the observatipns, and then take a weighted average ofthe'se.uncertainties, th e-weights bging the Kobabiltities of,occur-rence of the specific outcomes, The untertainty associated with a -

specific outcome, says the outcome N' observations yielding m redchips, 'is giVien by ,

\ b..h.t = - E P. lo P.

uN,m -

i=1i;N,m /.

g2 i. N,m

r

where h)is the number of hypotheses un er consideration and Pis the probability associated with the 3. h hypothesis after.N observations yielding m red chips. The expected uncertaintyafter'N observations, then, is obtained by weighting each UN4mby its probability of occurrence, and summing over allpossibleoutcomes. Thus,' 7

MN) 711;0 ir N uti,m,

(23)

.where JrN,m is4deffhed as before. Again, inasmuch es the value

of q in)equation'(13) depends on which hypothesis is true, thegeneral expression for E(U

N) conditional up6n which hypothesis

is true may be written1

.

is true) r/11 -p(k1Hi/D-CBIHJ)N-mUN,mb (24)m=0

(22).

E(UNIH.

r.

-85

(1,p

tt

.i /......,

L

'4

NAVtRAEbtaP EN 73-C-0128-I

A

')

H1

: 70% Red; H2:t20%.Red;

4

Figure, 10.

POMP

Expected value of posyeior probability Hi,

given H3 .Als true, as "a function of number

of observations

fig

a

9

I. . .

,,

.NAVTRAEQUOCEN 73.-C-0128=1; -, ,. .

* N

I...I

. The computatiOn of expected'unCertainty,relatesito figLe 8

i1 the folloWing wayy. Imagine that` such an outcome graph, weredeveloped for each of the hypotheses under considetation7 con- .

# ditional upon the truth of a.specified hypothesis.,: The value ofUm is found by sumbing...-p Iogip across grgphO,for given N,m,

.I.the values of-p being the values of the nqdes on the g?aphs-(. (equation 22). The value of E(Um) is thep obtained by weighting

each of .these sums by the probability of arriving at node N,m, .

given the truth of the specified' hypothesis, and summing over m,(equation 23). Figure 11 shows how th6 expedted uncertainty, con-cerning which of the two hypotheses is true changes as a re'Sult of

. observations in the case of our example.(HI: 70% red, H2: 20% red)', i

given the.truth of each hypothesis in turn. m..

, .

,

The 'examples that. we have been considering 14p all converged-,rather quicklY to a state of. relatively low, unce tainty. This wasdue to the fact that H1 -.pd H2 were quite disparate. But suppose'l and Ho were simi1 n4r with reispect to their for

a data. Sdppose, for example, that we-let H1 be the Sa as before(that is, that 70% of the chips are red) afid H2 be the\4hypothesisthat 60% of the,chips are'red. Again, setting the initial.proba- ,

4Iities equal t9 .5 and, assuming the same sequence of observationsas indicates in figure 1, figures 12 and 13 show the effects ofthese Observations on the distribution of probabilities over thetwo hypotheses, and on unoertainty. Figures 14--ana 15 show _the ,

expected effects of data on posterior probabilities and uncertaintyfdr this case, Obviously, the ekpected!effectsiof observationsare much smaller--the data have less di'agnostic impacto-uten the,hypotheSes are-similar thn when they are very different. Or, tosay the same thing in other words, asjarger sample is needed to.produce th _same degree of certainty %ith respect to which hypo-.thesis is-t ue. This illustrates tie intuitively compelling ideathat the smaller't the differences between two statistical distribu-tions, the closer one must examineithem to tell which is whip.,Continued sampling will eventually make the probabikities divergeand the uncertainty decrease, assuming, of course, that the samp-ling is random %rid one of the hypotheses is in fact true. Figures16 an617 show the eicpected qanges.inp(Hi[D) and uncertaintyover the course of 200 observations, giy.en H1: 7Q'% red, Ho: 60% fed,pn(H.) = .5j../ Two hpndred observations would noti on the Average,,r6du6e the uncertaihty in this case 1M the.amount that ten ob-servations would reduce it,,given-the more disparate hypotheses,

,-1

H1

: 70% red and H2: 20% Fed..

Table 2 (page 95) shows, for various combinations of H and Ho,the expected posterior probabiAty of Hi.after ten observations, giventhat chips are, sampled from an ur41 containing reds and flecks in,.theproportionsspegifiedbyHi ,And.assuming$OH1) = p0(H2) = .5.

87

'190-c

a

a

'A 2

I

."

SQ

'qAVVAEQUikEti 73-C-01282;1

4

0

4

4 5 6 7- 8 9 10 11

. OBSERVATION NUMBERA

H1: 70% Red; 20% Red;

1. p0 (hid).5 ,

y4 s,

Figure 12: Expected uncertainty concerning whichhypothesis is true: as a function ofnumber' of observa.;:%ons 0

88

101

r`.

to .t

.3

.

A

NAVTRAEQUIPCEN 73-C70128-1

7

'7

R 13, R R RDATUM

100 100 67 50 60 '.67 59 621 4_67 .70PERCENT OF REPS IN SAMPLE .

H1: 76% RedH2

: 60% Red;"

P0(1r1)%= p0 (H2)

:.

Figure 12.! Ch'afiges in.posterior probabilities,)(H1lD)-

0- and p0121D), as a result of the indicated,

observations of Red and Black chips

891

. 102

411

4

NAVTRAEQUIPCEN 73-C-0128-I

4

0

3 4 5 4, 8 10 11

..OBSERVATION, NUMBER. ,.. .

R , E) 8 R - R .8 R R R-

"̀-'" DATUM'

1

100 100 67 50 60 67' $9 62 67.PERCENT or REDS IN SAMPLE

1

11

4

: 70% Red; H2: 60% Red;

P0

(IH2. ) = .5

70 ,

4

Fiquke'13. Charpges in uncertainty concerning.hypotheses as a iesult of the indicatedObservations of Red and-Black chips

90

103

1

A

$

. ,

NAVTRAEQUIpCEN 73-C70128 -1

....,

Figu're 14. Expected value of'posterior probability

r . or H1., given Hj

is true, as a function

of number

Of!observations

91

104,

N

.

.4

VAVTRAEQUIP4N 73-C-0128-1.

V

p

r

fa

r' IIIler

1

1

I

0

Figure

4"

4

Hl: 70% Red; H2: 6O% Red;

PO(Hi)t= .5

. .

1J. Expected uncertainty concerning whichhypothesis is trtiboajLik function ofnumber of observatio5i *ff

w" jti

. )PP

1 .1

..^...NAVTRAEQUIPCEN 73-C-0128-1.

r

4'V

M1

41,

.4

A

ti

40,

A

6

40 60 ,80 100 120 '140 160 180 200 .

N 4

k. .A

vIiti% Red; H2: 00% Red:

. PO (Hi) =.5-.

..1, .

t

Figure 16. Expected value,of posterior probabilitv H

- ,.

1 H. , H3 , is true, ,as a function of \4

number of ob erVations 4

1

93106

I

A

ts

.NAVTRAEUIPCEN 73-C-0128-1

IT-7

/-0° .3' <IX th.

$5

`

A

..$

CR`

I)

L- I

.20 100 ,120 140 160 180 200 220

: 70%*Red;

Jr 'p0p (H.)

Nr

H2

: 60% Red;

= 5

,,

.4 Figure 17.- .Expected pncertain

,

y conceeninfj which r ,

. hypothsig is true', as a function of z,,,,,--

number of observations - / .

so

a

1

r

A

I W

NTiVtRAEQUXPCEN 73- 0k:28 -1

)

TABLE 2. SENS*IVITYOF,BAYESIAN/ANALYSI8:

.---

4 a

. /

Percentage of Rdds accbrding. tc H2

00I10

i

20

030

140.."

50

)60

70

-80

90

100

00 10 20 30 40 50 60 "_70 80

.50

'..74

, .90

.97

.99

1.00

1.60

1.00

1.00

1.00

.74'

.50

.59

.72

.83.

.90

25:98

.99

1.00

1.00

.90,,

.59

.50

.56

.67

.71

.87

.94/,

.97

.99

1.00

.97

.72

.56

.50

.55

.66

.77

-.87

.94

.98

1.00

.99

.83

.67

..55

.50

.55

.65

.77

.87

.95

1.00

.1.00 1.00

\ .90 :95

.79 .87

.66 .77

- .35' 265

.50 s.55

,55 , .50

.66 .55

.79 ..67

.90 .83

1.00 .99

Att

1.06

.98

404

.87

..7.7

.66'

.55

.56

-56

'.72'

.97

1.00

-.99

.97

. .94

%.,87

.79

.67

:56

.50

.59

.90

90 100

LOD .1.00

iroo

.99 1.,00

.*8 1:00

.64. 1.00

.9Q,. 1.0

.83 9 .

.72 .97 -

.59 .90

.50 .74

.74. .50

Cells represent e*pected values of posterior probabilitiesassociated with correct hypothesis after 10 dr4ws ftom (either)one of the urns.

4

95

a

I'

73-C-0128-1-

There are several things to notice about this. table. First,'it ,represents the expected value of the prpbability associated withthetrue hypothesis, so it.represents p(H11D) if data are sampled.from41Urn for which H-rig'true, and p(H21D) if the sample istaken. from an urn for which H2 is' true. A second thing to noticeabout the table is the fact that it is symmetric about the minordiagonal. Thus, for example, the probability associated with,thetrue hypothesis is'the sae for H1: x'90- red, H2: y% red-as forH1: y% red, H2: x%, red.,. This is a trivial point, and simplyindicatesAhat the' expected effect of a gequence of observations .is sttia:lY a function of the.diagnoticity of data and is inde-l'" pendent which hypothesieis which: Third, except when one of thehypotheses is extreme (say, hypothesizes that 10% or less, or 90%or morei,ofhe chips,are'of one color) , the expegted impact qtdata is largely a function of the' differedce between the hypo,Ehe-.sized percentages and relatively independent of their a' solutemagnitudes. r .-

,It' was pointed out above that for various combi nations of H1and 5.H2, .the 'first one or two observtions can'have a remarkably.large effect. HOwmuch effect, they will have depends, how6rer,on what those observations are and'on the disparity between H1and 42. This point is illustrated by figure 18. The figure showsthe probability of H1,'. giVen a single'observation that yield's a

"''red chip. In all cases,, it is assumed that the hypotheses wereequally probable before the observation. Note that if the hypothesesare disparate, for example, '111: 90% red and H2: 10% red,' or Hi: 10%red a'nd,H2i 90% red, a,single obgervation will change the proba-bilities associated with Hl and, H2 from .5 and .5 to .9 and .1,or to .1 and .9. On the other hand, if the initial probabilitiesare very close, say.5 and.6, a single..obsery Lion will changethem very little.

.

8.6.8 The Symmetrical Two-Hypothesis Case

1)

A

A two-hypothesis'case of special interest is thatfor whichone of.two possible observations has the same probability givenone hypothelsis.as does the other observation given the otherhypothesis.% That is, we ,are concerned with the situation in whichp(D

q 1 0) = p(D.IH

2 equivalently,,in4wiriCh p(Dapil) = 1-pADulH2).This ig sometimes referred to as the "symmetricalq.case, reflectingthe fact that one of-the two possible observations

prov,ides exactlyas much stipport for one of the hypotheses as does the other obser-vation for the other hypothesis: This situation holds in thechips-in-urn context when both hypotheses involve the same pro-portional plit of chips of different colors, but one identifiesred chips, and the other black-chips, as being the more numerous.

4,

et'

r

an

A.

NAVTRAWUIPCEN 73-CL0128-4,1/

t

4

.1 .2 .3 .4 .6 .6 .8PROPORTION OF REDS ACCORDING TO H

Q0 (Hl) p0 (H2).5

1.0

,Fisur 18. The effect of drawing a single Red chip,,, given various combinations-of prior,

hypotheses concerning the proportionof Reds in the urn

00.

97

1.10. '

. s

. 4NAVT4AEQUIkEN 73-C-01281

The hypOihesis pair H1: 70% R, 39%-B; Hi: 30% R, 70%B satisfiesthis condition, for example; whereas thp pair Hy 70% R, B;

.H 20% R, 80% B does not.2'

.-The sSrmmetrical case is of special interest because of the .

fact that,t1le effect of a series of observations .on the odds/favor-Eng one hypothesis over other can be calculated in a triviallysimple way. If Stn represents the odds prior to the obserliationsof interest, and t represents the, likelihood ratio, then '

g = Lgdd 0(25)

"where d represents the difference between the number of observa--tions of D -(say, red chips) andbf'Dol (say, black chips),-an-d---gA represents the odds following the observations. Note that thesize of the eaiple--the number of observations--does no enter intothis, calculation'. Suppose," for example, that go = 1,and L =(as would be the case if p(RIH1) = .75 and p(RIH1) = .25, and the-..odds likelih od ratio were expressed Hi relative to H1), then,given a sequence of observ$tions'yielaing tour more red chipsthan black chi. s, theo.postetior,odds would be

rf-Y 4

g = 34!1 = 81,4 '

(26).

and the same result would hold whether the difference of four wasobtained from a sample containing 8 reds and 4 blacks,or one con-

,

taining 100 reds and 9.6 blacks.

The exclithive dependence of Std on d follows directly fromthe fact that the likelihood ratio for one of the two possible ob-servations is the reciprocal of that for the other observation.Recall from egwation -(11) that the posterior odds following asingle observatiop, is simply the prior odds multiplied bythe

4 likelihood ratio associated with the observation

SZn L%-1. (11)

'Recall, too, however, that the likelihood ratio is conditional onthe observation. Thus, if Da is observed,

.p(DallyL

p(D I.H )

ct 2

'whereas if U is obServed,

p(D IH )

1LTTIET-JH )

,s0 2

98

(27)

(28)

1.

ti


r (

Letting La represent the likelihood ratio when Da is observed,,

and Lo,the 1ikelihood ratio when Do is observed, we may representthe effect 6W a bpecific sequence of observations, sayDaDcc 0

1 5 D'D D4

, on tht odds as ..BccLsl

11.= /L

ccStn_ (29)

,.: f3 6-

.

In the symmetrical case, however,

1= ra \a,

) (30)

so the effect of the same specific sequence of obsetvations may',,be written as

-St = L4 L. = L 2

SIn a a2

SI

n-6 a n-6.

and in general

a = L nd

n a n-k

c2

where d is the number of observations of Da minus the,observations of DB. But, inasmuch as neither n nor kthe calculation, we may express S/ as a function of d,the expression as in equation a5) .

(31)

(32)

number ofis used inand write

We see then that in the symmetrical case, the odds increaseexponentially with the difference between the number of observa-

wl

tiqns of the one type and that of the other type that have beenyobtained. Figure 19 'shows how the' rate of growth of function.14-r

depends on the disparity between the conditional probabilities, or, i

equivalently, on the size of the likelihood ratio. Figure 20shows how the size of the difference that is 4equired to realize

h

a given odds varies with the fargeY of the probabilitiof which the likelihood ratio is'comprised.typically tend to be conservative Bayesians ie finding that peopl

n their use of datato revise theirestimates of the likelihoods of thefpossible statesof the world suggests that many people would find the relationships

I

that are shown in the'se figures to be counterintuitive. The fact, ,/ '

\.

f 99

112 c.

100,0

100

6

10

NAVTkAEQUIPCEN',73-C.L.0128-

4,

.4

.. ........... _A__I___I

..

...,,

TT,

....--- ...........a.

:

.. .. : . .=4--.---

: 1

.

i

, 1

__.. _ .i

..... -. .

!

.1 ,

_ ..i.__*.i...._".

1

7 1 1

'" I *

_

_....;._"--1 _.

% ; 1 . 1.*

:I .17

Ti::.....i.,..

1I

=1777.

.........- -

. r .

. .

i. : .. , . r- " : ...; .:....' : ! i::777112

14

X!!:..1::i... 1 i ,,...

...

. ;....--: ::-.1-', :. ! 4.!. ..!... 4...

' 1

_. ..

.. . .

'

'14: ..

;-i 7"1 ' 1

....... p.

. . ; ..

I f .-....1..................

a ;.I

! - .. ''I

--'F2*

" i .Y., ...

- . . a. . a ..

_ .. .

....I

. t " 1 i i

1 :

I

. . .i

:

i. ..I !

:,... ; .. ;.

i I: ....4/, . -" ' ". 1

--,-.:..

:"'.

.

.....:_:._--'4t:. i ... " , .

i. - . . I ...

6 8 10

d

12 14 16

Figure 19. The rat-of growth in posterior odds as afunction of the difference, d, betweenthe' number of.obssrvations favoring H1and the num er favorir H2, in thesymmetrica cas 4(The parameter islikelihoo ratio, 1,1,2)

1.00

113

.

itv

0.4

NAVTRAE' QU/PCEN 73-C-0128-1

.50 .60

110

.70/ .80

p(D(H)

.90 1.,00

Figure 20. The size of d required to realize a givenodds ratio as a function. of the larger' ofthe two ponditional probabilities in thesymmetrical case

tv

1


y,

.

for example, that with conditional probabilitkds of p(D 1H1) = .7 .,'

andvp(Da1H2) = ,3, a sample that containp three more obaervations ofDa than of Do will favor H1 over H2 by a factor of More than 10 .Taybe surprising; as may t'he fact that with a difference of six,the odds are,greater. than 100 to 1. - ,

/ -.

Another, aspect of the symmetrical case that some readers mayfind counterintuitive is the fact that the total effect Of a seriesof observations on the .odds depends only on the difference between

ofthe number ofobservations of the two types and is independent,

the total number of observations-made. Both intuition'and statistical training suggest that one's confidence in any inference thatis to be drawn from the properties 'of a sample should,increase with

Itthe2saMple.size. The apparent parado is resolved by a recognitionof the fact that,' except under the hypothesis that each observationis equally likely, th4 absolute difference (thOligh not uhe relativedifference) between the frequelicies of occurrence of the two types

of observation is expected to increase-with sample size. Speci-'fically, if H: x% R, (1-4x)% B is true, the difference betWeen thenumber of Rs and 1th in'a sample of size N should be (2x-1)N, on 1'

the average. 1

Consider, .for example, the symmetrical hypotheses H 70% R,1.

30% B and H2: 30 %'R, 703 B. If H1

were true, samples of ten drawswould be expected to produce four more'reds thhn blacks on the'average; and the odds following a ten-draw sample wit1r foUr more-reds than blacks .would be'about 30 to 1 in favor of H1. Samples of

100 draws, given should produce 40 more reds than blacks, onthe average, a.difference that would. drive the odds to more than/523 trillion to 1. Thus, the odds dp ,tend to increase with samplesize because.d tends to increase with sample siKe. A sample of100 draws that produced four mare reds than blacks would be quite

unlikely if H1 were true, and thus would not constitute strongevidence in favor of that hypothesis. It would be even lesslikely,,however, if H2'were true, so it does constitute someevidence for H1,,but only as mu h as one would expect to obtainfrom a much smaller sample. Ta le 3 shows the odds favoring H1,giyen various combinations of p( 1H1) and p(D1112,) and several

values of d. !

8.6.9 The Several-HypothesiS' ease

So far, the examples that we have considered to llustratethe fuse of Bayes rule have involved only two hypothes s. We turnnow to consideration of a few cases in which there ar more thantwo hypotheses. Figure 21 illustrates a. case in which 1,and H1 represent, respectively,. the hypotheses that the percentageof red chips in the urn is 90, 70 and 50, and shows how the pos-terior probabilities associated with these hypotheses would change

102

- 115

.r.NAVTRAEQUIPCEN 73-C-0128-1

,-

T ABLE,3. ODDS FAYPRING H1GIVEN THE INDICATED VALUES OF

.

p(DIH2) AND d. ti

t is

p(Dly p(DIR ) L-4

I . 2 4

d

8 16 32

..55 .45 1.22 1,2E0 1.5E0 2.2E0 )4.9E0 2.4E1 .5.8E2

.60 x.40 1.50. 1.5E0 2.3E0 '5.1ED 2.6E1 6.6E2 4.3E5

/65 .35 1.85 1.9E0 3.4E0 1.2E1 1.4E2 1.9E4 3.5E8

.70 .30 2.33 2.3E0 5.4E0 3.0E1 8.8E2 7.7E5 6.0E11

.75 .25 3.00 5.0E0 9:0E0 8.1E1 6-.6E3 4.3E6 1.9E15

.80 .20 400' 4.0E0 1.6E1 2.6E2 6.6E4 4.3E9 1.8E19

%85 ./15 5.67 57E0 3.2E1 1.0E30 1.1E6 11E12 1.3E24

.90 .10 9.00 9.0E0 8.1E1 6%6E3,, 4.3E7 1.9E15 3.4E30

". 95 .05 19.00 1.9E1 3.6E2 1.3E5 1.7E10 2..9E20 8.3E40.t.

,

,...'... \,..

.

All odds vanes are rounded to two significant digits and expressin exponential form. To obtain the approximate value of R, multiplythe number to the left of the E by ten raised to the power indicatedby the number to the right of the E. For example, 4.3E7 = 4.3 x 107)-

103

116

S

'

D

NAVTRAEQUIPCEN 73-C-OC28-1"

4

4

).-Cr

X 'b'( ,g4c

"-t-$

R

100

4 5 6 7OBSERVATION NUMBER

B .B R R B R

DATUM

10

R R

00 67 50 60 67 59 62 67 70'PERCENT OF REDSPZN SAMPLE

't

Hl: 90% Red; H

H3: 50% Red; p

7Figure 21. Changes in posterior probabilities p (Hi I D)

as '''result of the indicated. observations

of Red and Black chips

: 70% Red;

(Hi) .333

104

117 Per

1

*1,

Q

$

P

at


as a result of the, indicated sequence of observations, giveh p0(H.)

1--77 .333. Figure 22 shows the change in uncertainty concerningwhich hypothesis is.correct as a result of the same sequence ofobservations.

, When only two hypotheses4are under consideration, there is 1/only one odds ratio (or its reciprocal) that can be expressed.The iiambe'r of odds ratios that can be expressed grows rapidly,however, as the number_ of hypotheses is increased beyond two. In

general, given N hypotheses; there arerfl or N(N-1)/2 odds.that

can be expresSed considering only pairs of hypotheses. Thus in thethree-hypothesis case we might consider Q

1,2, 2

1,3-or Q2)3' each

of which is shown infigure 23 for our example.

It may be of interest to consider lother than pairwise oddsratiosin the several-altdrnative case,, however. Given five t

hy?)otheses,.for example, one might wish o consider the odds pfH1 relative to the combination of H3 and H4, which would be ()b-

eg tamed by taking4the ratio of p(HilD) to the sum Of p/H3ID) andp(H4rD). It may often be of particular Interest to consider theodds-of a given hypothesis, Hi relAtive to all the remaininghypotheses in combination. Such an odds would give the rat ofthe probability that.HJ.- is true to the probability that one ofthe remaining hypotheses is true, i.e., that Hi is false. We #might refer to such an odds as the absolute odd^ of Hi and repre-sent it as f011ows:

p(HilD) p(HilD)Q.1,1 E .p (Hj I D) 1-71:7117TUT

(:33)

Figure 24 shows how the indicated ohservations affect the ab-solute odds of each of the hypotheses ofeour example.

Expected values of posterioi probabilities and of uncertaintymay be calculated in the same way when there Are several hypothesesas when there are only two. An outcome graph such as those shown(in figured 8 and 9 could be,used to specify all possible posteriorprobabilities for a given hypqthesis, andand their probabilitiesof att4inmenton the assumption that a specified hypothesis, H. istrue. The weighted sum of the nodes above a particular value 3of N,would'represent, as before, 4e expected value, after N obser-vations, of the posterior probability of Hi, .given that Hi is reallytrue. Also as before, computation of expected uncertainty involvessumming over both i and, m, for given N. Inasmuch as it is possibletocomputearlexpectationpfp(Hi HD) given that HI is true for all

b

MO

t

.1NAVTRAEOUIPCEN 73-C-0128-1

/

Pt

v.8 I I J t I 1 I I

t 2 a 4 5 6 7 8 9 10 _11

OBSERVATION NUMER

R R 8.8'R R BR RRDATUM

100 100 67 60 60 67 59 62 67 70PERCENT OF REDS IN SAMPLE

aH1: 90% Red; H2: /0% Red;

H3:,50% Red; p0 {Hi) = .333

Figure 22. Changes ikuncertainty as a result ofindicated bbsevatiOns

106

119

ti

NAVTRAEQUZRCEN 73-C-01211

A

S

IR R R 13DATUM

leo 100 67 60 .60 67r 69 62 67 70PERCENT OF REDS IN SAMPLE

H1: po% Red; H2: 70% Red;

H3: 50% Red.; po(Hil = .333

ti

4 6 '6 7 B 9 '10 if ,

ERVATION WISER

Figure 23. Changes in pairwise odds as a result ofthe indicated observations

p

107

x 20

A

.4:."Not

is

NAVTRAEQUIPCEN 73-C-012a-1

arS

possible combinat ons of values of i and j, the numbek of outoomegraphs that coul be of interest increases with the square of thenumber of hypot ses under ,consideration.,

Table 4 shows the expected values of p,/(1.1i1D) and Um, 6rN = 1,2, ...ro, given that Hj is true for all com4anations"ofand j in the case of our example (H1: 90% redr,H2: 70% red, v! 50%red, pn(H4) = .333)., As might be expected, given that Hi rud,E[p(HiID)T increases most rapidly when i = i;.which is tdo ay, . -'the expected value of the'prObability associated with the truehypothesis grows faster thap that of the probability associatedwith either false hypothesis. Counter to intuition, however, thisis not a necessary condition. An example will be.consideredpresently in which the expected probability associated, with a falsehypothesis grows, for a White, at a greater rate, than does:theexpdcted Rrobability associated with the true hypothesis,, evenwhen both hypotheses are equally probable a priori. ith coDtinuedsampling, however, the probability of the true hypothesis evbntuallygets larger than that of any of the false hypotheses Another pointof interest 'concerning table 4 is the fact that each of, threecolumns of values occurs twice: the second and fourth columnsare tical, as are the third and seventh, and the sixth ancl eighth.This it ustratep the following relationship:

Efp(HilD)Iiii is true] = Efp(Hi(D))Hi is true], (34)1

that is, the expected posterior probability of Hi, given that H4is true, is the same as the expected posterior probability of ,J

giyen that .Hi is true. This ,relationship holds in general, andindependenly of the number of hypotheses under consideration.

-As in the two-alternative case; th9 rate et which the expectedvalues of the posterior probabilities approach pne or zero--and,consequently, the rate at which uncertainty is expeqIed to decrease--depends on the disParity-amonc the hypotheses. The point is illus-trated in table 5, which shoes all values of Eflpicl(Hi(D)1H-C4.'true] for two sets of hypotheSes: H1, H2 and H3,:,-50,-70 and 501%red, and'ISO,' 60 and 30% red'. The table' also shows the expected',uncertainty after ten observations, E(U101,concerning whichhypothesis is trqe, as a function of which hypothesis actually istrue.

Table.6 shows E (1310(Hil'D)N is true] and E(U101Hj is truerfor two sets of five hypotheses. his table illustrates some of thesame points as does table 4. The rate at which the,probabilitieschange from their original values, and the rate at which uncertainty,decreases depend on the disparity among the hypotheses. The valueof Efp(HillioXpj is' true] is always equal to that of Elp(HjID)IHiis true] , whicn is seen by t e fact that each array, if consideredas a matrix, is equaleto its/ transpose.

109

122

6

NAVTRAEQUIPCEN 73-C-01.28-1.

,/TABLE 4. EXPECTED VALUA OF POSTERIOR PRQBABILITIES, AND

: UNCERTAINTY (IN BITS), GIVEN THAT CHIPS ARE SAMPLED

FROM THE URN FOR WHICH-THE'INDICATED HYPOTHESIS

IS TRUE.

4.

H1

'Hue Hypothlli5

H2 H3

Hypothesis-for which Expectation Computed

H1

H2

H3

t(U) H1* H2

H3

1E(U)1

H2

H3

1 .397 .333 :270 J.53 .333 .333 .333 1.49 .270 .333 .397

0 2 .453 .327 .220 1.44 '.327 .338 .33.4 1.41 .220 .334 .446

4)ro

3 .501 .319 .18U 1.35. .319 .347 .334 1.36 .180 .334 .486

wtt)

4

5

.542

.577

.310

.300

.149

.123

1'.26

1.18

.310

.300

.358

.370

.332

.330

1.32

1.28

.149

.123

.332 .51'9

.547.4 ,.330

0 6 .607 .291 .102 1.10 .291 .383 .326 1.25 .102 .326.--4720 634 .281 .084 1.03 .2'81 .396 ..322 1.22 .084 .322 .593

(11 8 .658. .272 .070 .0.96 .272 .411 .318 1.19 .070 .318 ,.612

V 9 .680 .263 .058 0.90. .262 .425 .312 1.17 .058 :312 .630

10 .699 .253 0.48 0.85 .253 .440 .307 1.14 .048 .307 .645

4c

I

E(U)

1.45

1.35

1.26'

1.19

1.12

1.06

1.00

0.9,5

O.91

0.87

H1: % red, H2: 70% red, H3: '50% red; p0 (Hi) = .333.

(No e: expected uncertainty, E(U),.is not the same as the

uncertainty calculated from the Axpected posterior probabilities.

lip

K

.7

ti

4..

-NAVTiAtQUIPtEN 73-C-0128-1

TABLE 5. IE[p10

(H.ID)1H, PS TRUE] FOR ALL COMBINATIONS OF3.2,. j

t and j'AND'THE TWO INDICATED HYPOTHESIS SETS.

H1

: 90 %, Red, H2

: 70% -Red,' H3: 50% Repl

1 2 - 3

1 - .99/ .254 - .048

i 2 .253 .440 .307

3 .048 .3b7 .645

E(U)(in bits)

0.85 1.14 0.87

H1: 90% Red, H2: 60% Red, H3: 30% Red

-1

):2 .

, 4,

1 .824 .171 .005

le

1i 2 .171 .603

.

.226

3 .005 .226 .769

E (U) 0.48 0.86 0.55

(in bits)

In both cases p0(Hi) = .333.

111

124


TABLE 6. E[p10(HilD)(H IS TRUE) FOR TWO FIVE-HYPOTHESIS SETS.

H1- 7 boo, 80, 70, 60, 50% Red, respectively

1 ' 2 3 c 4 5

1 .475 .282 .148 .068 :.026L

2 .282c

.271 .216 .146 .085

i a .148 .216 .2391 .221 . .1764,

4 .068 .146 .221 .273 .292

/5 .026 .085 .176 .292 y .421

E(U)(in bits)

1.64 1.90 1.96 1.86 1.67

H1

-1.1 90,.75, 60; 45, 30% Red, respectively

1 2 3 1 4 5

1 .604 .280 .094 .021 .002

2 .280 .337, .240 .112 .031i

3 .094 .240 :300 .240 .126

4 .021 .11'2 .240 .323 .304

5 .002 .031 .126 .304 .537

E(U) 1.19(in bits) ;

1.63' /1.75

In both cases p0(Hi) = ,2.

112

125

NAVTRAEQUIPCEN 73-=C -0128 -1

For the hypothesis set,represented by the bottom half oftable 6, it is true that the expectation is maximum when i = j,-which is to say that after ten observations the prObability ofthe true hypothesis is always larger than that of any of the falseones. Note, however, that this property does not characterize thevalues for the hypothesis set represented by the top half of thetable.' In particular, given this hypothesis set, the expectedprobability of H1 is greater than that of 111 after ten observations,'even if chips are drawn fiom an urn fir whift 1 is true. Similarly,E(p (H ID)] is greaterthan E[p.10(HAID)) when'n4 is true. WithcontInugd sampling the expected Probability of the true hypothesiswill continue to grow, finally approaching one, whereas that of,each of the false hypotheses will at some point begin to decreaseand will eventually approach zero. The fact that the expectedvalue of the probability, of a false hypothesis. is higher at anytime than that of the true hypothesis may be quite counterintuitive,however. Figure 25 Mows the way in which the expected values;of each of the posterior probabilities of the example representedin the top-half of table 6 change over twenty observationt, giVenthat 12 j.sireally true. -Note that p(H2ID) is initially'smallerthan plHilD), but eventually overtakes and surpasses it; with.,further gampling p(111ID) would continue to increase,, whereasp(H

1INt would decreage.

(.

411 A comparisofi of tables 5 and 6 \illustrates several additlionalpoints. The hypothesis sets represented in table 5 are containedwithin those reprebented in table 6.. Considering only those,hypotheses that are represented in both tables, it may be seenthat the expected posterior probabilities associated with hypotheseswithin the smaller set are invariably larger than those associatedwith the same hypotheses within the rarger set.. It may alsoi beseen that the expected amount of,uncertainty remaining after, tenobservations, given the truth of'a specific hypothesig, issgreaterwhen the hypothesis set contains five alternatives than when itcontains three. Of course, the a priori uncertainty is alsogreater in the former case (2.32 bits versus 1.58 bits), so,whatis of greater significance is the fact that'a larger proportion ofthe original uncertainty is resolved in the three-alternative case.

0e.113

126

X


4

.n .ce .,...0---

I cr- __0---

4121-"" /

f

HI)7121-13

Ma

I "510 12 14 16 18 20, 22 21

N

H1: 90% Red; H2: 80% Red;

H3: 70% Red; H4: 60% Red;

H5I 50% Red; p0 (Hi). = .2

VF'i'gure 25. Expected value of posterior probability of

Hi, given that H2 is true, as a function ofnumber of observations

114

127

NAVTIZAEQUIPCEN 73-C-0128-1

ti

8.6.10 Man as a Bayesian HypOthesis Evaluator

A considerable amount of experimentation has been done tpdetermine how well Bayes rule predicts behavior when an indvi-dual attempts to process probabilistic information in situationslike those illustrated. For example, given the task pfdeciding, on the basis of a sequence of observations, which ofseveral hypotheses about the nature of the source of those ob-servations is true, how closely will the estimates produced by thehuman decision")maker correspond to those produced by the applicationof Bayes theorem? Obviously, in situations as highly structured asthose described, it would be of little interest to do such experi-ments with an individual who linderstood Bayes rule and was permittedto do the calculations necessary to use it. Sucl a test would donothing but demonstrate one's ability to do.arithmet3n. Experi-ments on Bayesian information processing typically Are done withpeople who. are not formally aware of Bays rule, or if.they-are,they are not provided with the time to perform the necessary calcu-lations. It is an interesting question, in this case, whether anindividual's intuitive, or at least informal, notions about evidencewill lead him to adjust his probability estimates in a way similarto that that would result froth an application of Bayes rule. And

. if-the answer to this question is no, it is of interest to determinewhether his performahce deviates from that of Bayes rule in con-.sistent way's.

Perhaps the question that has been of grpatest interest to,and received most attention from, experimenters is whether hypotheses'are more effectively evaluated by having decision makers estimateposterior probabilitiesp(HID), directly upon acquiring-incomingdata, or to have them eNtimate conditional probabilities, p(DIH),`and then to use these estimates to update the posterior probabilitieswith the use of Bayes rule. Much of the evidence favors the con-clusion that hypotheses are evaluated more efficiently when thelatter approach is taken, that is, when humans make estimates ofi-,(DOH) and these estimates are used along with Bayes rule 'Eo cal-culate estimates of p(HID). Although the directional effects ofdata on posterior probability estimates produced by humans are -similar to those on estimates revised in accordance with Bayes rule,the magnitudes of the effects tend to be smaller in the former case.In particular, the posterior probabilities tend to obtain more ex-treme values and to reach asymptote faster when they are calculated. according to Bayes theorem than when they are estimated directlyby humans (Edwards, Lindman, & Phillips, 1965; Howell & Getty, 1968;Kaplan & Newman, 1963; Peterson & DuCharme, 1967; Peterson & Miller,1965; Peterson, Schneider, & Miller, 1965; Phillips & Edwards, 1966.It appears, therefore, that humans tend to extract less informationfrom data than the data contain; they require more evidence thandoes a- Bayesian process to arrive at a given level of certainty

NAVTRAEQUIPCEN 73-C-0128 1

concepning which of the competing hypotheees is true. That is oneof the findings that has led to the characterization of man as a"conservative" Bayesian. In other words, men tend to undekestimatehigh posterior probabilities and overestimate low ones. A similar,butless pronounced, tendency is found when men estimate oddsrather than posterior probabilities (PhilliPS & Edwards, 1966).

Slovic and Lichtenstein (1971)refer to the conservatism ofman in his use of probabilistic data as the primary finding ofBayesian research. They-review three competing explanations ofthe result: (1) misperception, or misunderstanding, by the subjectof the process by which the data are generated; (2) inability of

` subjects to aggregate, or put together, the impacts of severaldata to produce a single response; and (3) an inability, or un-willingness, to assign-extreme odds, e.g., odds outside the rangeof 1:10 to 10:1. Whether any of 'these explanations is adequatehas yet to be determined.

It was The finding of conservatism that prompted Edwards (1963,1965) and his colleagues (Edwards, Phillips, Hays, &'Goodman, 1968)to experiment with probabilistic information-prOcessing systemsthat use experts to judge the-likelihoods of the data reaching thesystem, given each hypothesis under.cbnsideration, and machinesto calculate posterior probabilities on the basis of these estimates,and the data.

.

Not all of the evidehce'that is relevant to the question favorsthe conclusion that humans are invariably much better at estimatingp(DIU) than p(HID). Southard, Schum, and Briggs J1964b), forexample, obtained some results that challenge the generality of thefinding that humans tend to underestimate high posterior proba-bilities, and overestimate low ones. In particular, given a smallhypothesis set and a frequentistic environment, the estimates ofp(HID) produced by humans were' lose to, and sometimes more extremethan, those produced by Bayesian methods. Other studies, severalfrom the same laboratory, have also yielded results that questionthe validity of the general conclusion that better decisions resultwhen values'of p(HID) are derived by applying Bayes rule to men'sestimates-of p(DIH) (Schum, Goldstein, & Southard, 1966; Howell, 1967;.Kaplan & Newman, 1966; Southard, Schum, & Briggs,. 1964a). Ofteneven when evidence of conservatism has beeh,found, 'the degree towhich the human's estimate .of P(HID) has differed from an estimateproduced by Bayes rule has been very slight (Petersen & Phillips,1966; Schum, Southard, & Wombolt, 1969).

These findings do not'permit one to conclude that estimatesof p.(HID) are never better when derived from estimates of p(D1H)than when produced directly, but they do call into question the,opposite notion, namely'that of the invariable suPeriority of theindirect approach. Moreover, they suggest that the direction thatresearch should take id that of determining the conditions under

116

129

NAVITAEM/PCfEN 73-C-0128-1

which each approach is warranted. .Sebum, Goldstein, andSouthard (1966). present some data, for example, that suggest'thatestimates of'p(HID) that are produced directly are more adverselyaffected by degradation in'the fidelity of the incoming informationthan are those that are derived from estimates of p(DIH).

Another finding that is relevant to the'-,,question of man'scapabilities as a Bayesian hypothesis evaluatOr is that evidencethat tends to confirm a favored hypothesis may be given Morecredence than evidence that tends to disconfirm.it (Brpdy, 1965;Geller & Pitz, 1968; .Pitz, Downing, & Reinhold, 1567; Slovic,1966). This finding raises the more general question of whethera vested interest in a decision outcome impairs one':s ability toevaluate data objectively.' If itis the case, as Bacon (1955)long ago suggested, that "what a man.had rather were true, thathe more readily believes," at least-one of Savage's basicrules for the application of decision theory is gerie:ally violated.

The possibility that an individual's.preferences among

khypotheses may impair his ability to evalua e them in an unbiasedway is closely related to the findin7 that eople tend to bereluctant to change a decision once it has been made (see Section-4.3).

CO

Each of these tendencies--conservatism, partiality, andpqrseverativeness--has been viewed as a fault, or as, evidencethat'man applies data to thes.valuation of hypotheses in.aninefficient way. And, in t-context of most laboratory experi-ments in which it has been observed, itundoubtedly is. Thesetendencies may sometimes be less patently unjustifiable outsidethe laboratory, however. An insistence on having coppellingevidence before changing an established opinion may have astabilizing effect that is not altogether bad. Many opinions

thatformed slowly over a period of years, and all the factorsthat may have contributed to their formation cannot always berecalled at will. the individual who is quick to change anopinion every time he encounters an argument that he cannotimmediately refute"may find himself constantly shifting from onepdsition to another, always a proponent of the view that he lastheard capably expounded.

Hypothesis evaluation has been studied more than most aspectsof decision making in the laboratory. This is due in part to theexistence of a simple prescriptive model (Bayes rule) for per-forming this task, given an appropriately structured problem, andin part to the fact that it lends itself to laboratory explorationmore readily than some of the other decision-making functions.Much has been learned about make's capabilities and limitationsin applying evidence to the resolution of uncertainties about thevarious aspects of a decision situation. Much remains to be

117

,. e

NAVTRAEQUIPCEN 73-C-0128-1,

determined, however. Along several issu at deserve furtherstudy are the following: the ,possibility gat information-displayformats-and response techniques may e subjective proba-bility estimates that are obtained (Dam Goodman, &yeterson,1972; Herman, Ornstein, & Bahrick, 1964); the appar nt lack ofunderstanding of how to combine probabilities arising from inde-pendent sources of information (Fleming, 1970); the possibilitythat the weight that one attaches to data may depend on when thosedata occur during the hypothesig-evaluation process (Chenzoff,Crittendon, Flores, Frances, Mackworth, &Tolcott, 1960; Dale,1968; Peterson & DuCharme, 1967); and, the possibility that one'sability to deal with uncertainty in a conflict situation may de-Pend on whether one is operating with an advantage or a disadvan-tage with respect to one's. opponent (Sidorsky & Simoneau, 1970).

8.6.11 Bayesian Hypothesis Evaluation and Training

One way to interpret some of the results that 4iave been de-,scribed above--for example, th finding. that men of en extract-less information from data than does a Bayesian aggregator--isto see them as indications that man's intuitive notions concerningthe uses of evidence are not entirelS, consistent with the implica-tions of Bayes rule. Perhaps the thing to do, if this is, the case,is to disabuse would-be decision makers of those faulty intuitions,.

I

Such a task might be approached in two way's. On the onehand is the cognitive approach of teaching the decision maker aboutBayes rule and its implications. An alternative possibility isto expose the decision maker to a variety of situations, in whichhis behavior is evaluated and immediate feedback is provided tohim concerning the way in which it departs from optimality, if itdoeS. This is the behavior-shaping approach; in essence, it isaimed at modifying one's.intuitions without necessarily providingan intellectual understanding of how optimality is defined. Thesetwo approaches are not mutually exclusive, of course, and it seemsreasonable to assute that a training program would,be more likely,to be effective if it used both. That is to say, the dec.i,sionmaker should probably be given a good understanding of the notionof inverse probability and how Bayes rule, aggregates data; and heshould also be provided with considerable practice in attemptingto apply the rule in situations that are sufficiently well-struc-tured that his performance conibe 'evaluated and "compared to anobjective criterion of optimality. The selection of trainingscenarios should put special emphasis on those situations forwhich'man's intuitions have bden shown to be. most misleading, e.g.,especially small or especially large levels of a priori uncertaintyand situations in which the direction of evidence changes aftera tentative decision has been reached.

118

1

NAVTRAEQUIPCEN

-

The results of some studies have indicated that such .trainingcan be at least partially effective. Fleming (1930), for example,explored the queltion of the effectiveness of feedback concerningthe outcome of a selected action in improving the decision maker'sperformance on subseqhent de6ision tasks. The context of theStudy was'a simulated tactical decision-making situation. Subjectswere required to combine probabilistic data from three independentsources in order to arrive at an estimate of the relative-likeli-hood of attack on each of three ships. Initially? subjects demon-strated an"igriorance o'f the proper combining rule (multiplicatiop),and were conservative in estimating the overall probabilities .ofattack. The investigator concluded that these data-aggregationand'Probabilityestimation tasks should be automated. He alsoshotged, howev,er, that, although the subjects where unable to gen-erate the correct probabilities on the basis of feedback, the didrevise their estimates 'over the course of trials in such.a wayas to correct for conservatism (apparently by adding a constant).

. iOther investigators have al8O,ShOWn that experience in estimating

posterior probabilities can produce behavior which, if not optimal,is more nearly so than before the training began (Edwards, 1967;Hoffman & Peterson, 1974 Southard, Schum, & Bridges, 1964b).Such studies establish that. certain aspects of hypothesis evalua-tion, in particular posterior probability estimation, can be im-proved somewhat as a result of practice. What they do not indicate,however, is how much can be expected of training or how the train-ing should be done in order to obtain optimal results.

Another issue that relates to training involves the questionof how well people can make the p(DIH) judgments that they are ""%

required to make in some Bayesian systems. It seems to be generallyassumed that people have less trouble making these judgments thanthey do making judgments of pailD). In at least one study, how-ever, this was not the case.. Bowen, Feehrer, Nickerson, Spooner,and Triggs (1971) encountered a fairly strong resistance on thepart of experienced military intelligence officers to the idea ofmaking judgments of the sort: "If.it is assumed that 'Attack' isthe enemy commander's course of.actionowhat is the probabilitythat one will observe the traditional 'indication YMassing of Tanks%2"These investigators pointed out that, the "generally negative re-

. action to the possibility of estimating probabilities of the typethat would be required in a Bayesian system must be tempered bythe fact that the participants were not familiar with the conceptof Bayesian inference and had not been trained to make the requiredjudgments" (p. 103): There is, therefore, the question of thedegree to which training in Bayesian analysis would be effectivein overcoming the relatively strong preferences that some decision,makers seem to have for estimating posterior probabilities them-selves.

119

132

1

'NAVTRAEQUIPCEN 73-C-0128-1

Several other questions concerning man's capabilities as theyapply to hypothesis evaluation have been noted above. These ques-tions have arisen because of the results of experi)nental studies.They are questions which, fon,the most pirt, have not yet beenadequately answered; The questions, in most cases, suggest somelimitation or deficiency in man's hypothesis-evaluation skills. .

To the extent.that these limitations or deficiencies are demon- rstrated by further research to be genuine, they represent challengesto designers of training programs. If it is the case, for example,that probabiaity estimations are senTitive to the format in whichinformation is displayed or the mode in which the response is given,as sdme studies have indicated, the callestion is whether such effectscan be eliminated by training.- If they cannot be, then the need.to be restrictive with respect to display formats and response.Dmodes is so much the greater. Or, to take another example, if theway one applies data to the evaluation of a hypothesis is differentfor a favored hypothesis than for an unfavored one, as )ther studieshave suggested, this constitutes another challenge to training..Can one be trained to apply data to all possible hypotheses-in anunbiased way without regard for his preferences? Similar questionsconcetning the potential effectiveness of training can be raised ,

concerning each of the other limitations and deficiencies that.'have been noted. More research will be required in order to answerthese questions.

8.7 The Measurement of ective Probability

Throughout this report we have made frequent reference tosubjective probabilities, and it has been tacitly assumed that'such things can be accurately measured. In fact, how to assureaccuracy in measurements of this quantity haS been a question ofSome interest. The problem is a problft,because of the fact thattheeprobabilit4es that one obtains may depend on the way in whiphthey are obtained; or es Today (1963)(puts it, subjective proba-bility is esgentially defined by, the measuring= technique that isused. Toda further suggests teveral criteria that such a measu-tring technique should satisfy: "First, the logical nature`of thetask presented to the subject should be thordughly understood bythe experimenter, and, hopefully, by an intelligent subject.Second, the tack should involve well-defined payoffs to the subject.Third, the task should be so structured that it is to the disad-vantage of a subject to respond in a manner inconsistent with hisexpectations. Fourth, since our interest in measuring subjectiveprobability is related to its use,in\decision theory, the measure-ment technique should not be inconsistent with decision theory" (p. 1).

The third of these criteria is perhaps the most subtle, andhas received the greatest amount of attention. Stated in otherterms, the requirement'is that it be in the subject's best interestto state his probability estimates honesty.- That this can be aproblem may be illustrated by a'simple example of a situation in

120

133

1

NAVTRAEQUIPCEN 73.-C-0128,-1

which the requii.ement is not met. Consider the case of a studenttaking a multiple-choice examination. Suppose he has been in-structed that in answering 4ach question he is to assign a numberto each of the alternatives associated' with that question in sucha way as to reflect his estimation of the probability that thatalternative is the correct one. When he is very certain of whichalternative isxcorrect, all of the numbers except bne will be zero;when he is less than 100% certain, however, he would assign non-zero numbers to more than one alternative. Suppose furthef thatthe score that he is to receive for any given question is somelinear function of the ratio of the number placed on the 'correct-alternative to the sum of the numbers used on all the alternativesassociated with that question. Given this scoring rule, the studentshould not distribute numbers in accordance with his true estima-tion of the probabilities; instead, he should put zeros on all.the alternatives except the orie that he considers most, likely,even if he is not very certain that that alternatiT7T-Is indeed thecorrect one.

This is easily seen by considering a two-alternative case.Suppose that the student really thinks that the chances are 7.in10 in favor of A being the corect alternative. If he is honest,then he'will assign 7/10, of whatever points he is going to use,on alternative A and 3,410 on B. Given our scoring rule, and assu-ming that our hypothetical student assigns numbers to the twoalternatives in the ratio of 7 to 3, then the two values that hisscore may assume are 7/10 and 3/10. Moreover, from the student'spoint of view, the probability of getting a score of 7/10 is 7/10

the probability that A is correct), and the probability ofgetting a score of 3/10 is 310. Thus the subjectively expectedvalue of his score is..(7/10) + (3/10) = 8. But suppose thatour student were a gambler, and decided to put all hig7chances onthe altern4tive that he considered most li ely to be correct. Nowthe two values that his score can assume, a e 1 and 0, and the ax;pected value of his score (assuming that he really believes thatA's chances are 7 in 10, rather than 10 in 10, as his answer would ,-indicate) is 7/10 x 1 + 3/10 x 0 = .70. Thus, whereas the studentwas instructed to assign numbers to alternatives in accordance withhis judgment of the likelihood'of their being correct, the scoringrule is such that he can expect to obtain a higher score by-ignor-ing/the Astructions than by following them.

A scoring rule that is to satisfy Toda's "honesty is the bestpolicy" requirement must have what has been referred to as a"matching property." In'formal terms, the matching property maybe stated as follows.: Suppose that a subject reports n non-negativevalues,

presumablyto reflect thei=1

subjective probabilities that he associates with alternativepossibilities, xl, x2, ...xn. Assume a liscrete subjective

121

134

C

NATRAEQUIPCEN 73-C-0128-1

probability distribution,,pi, p , ...psubject's true probability dstiltates r gardi xi, xl, ...x .

Letting P, R and X represent, respective , the M'otors n(pl, R72, .pn), (1'1, r2, ...rn) and (xi x ...x )

2, n.'and W(R, X) the payoff to the subject, given the response vector Rand'the probability vector, X, the Matchi g property is realized byany payoff function for which the following statement -is true:._The response *vector, R maximizes the subjectively expected payoffE[W(R, X)1, if"and onlF if R ==. kP, k being a Scalar constant.That:is-to say, a payoff scheme, or a scoring rule, has the matching

, property if and only if the subject maximizes his subjectivelyexpected payoff when the weights that he assigns to the possibilities

* differ from his true subjective probabilities at, most by the,samemultiplicative fadtor. Nkte that when the relationship R = kPdoes hcLd, the calchlation`of odds will be the same whether basedon-11 or on"P. .4 -

Subjective- probability measure ment procedures and responsescoring techniques that make use of functions that have this matchingprdperty have been referred. to as "admissible probability measures"(Shuford, Alberti & Massengill, 1966), and "proper scoring" rules(Winkler & Murphy, 1968).\SI-Several functions with the matchingproperty, have been defined and investigated, among them the "loga-rithmic loss" (pobd, 1952; Toda, 196 the "quadratic loss"(Brier,/1950; deFinetti, 1962; Toda, 63; van Naerssen, 1962,and the "spherical gain" (Toda, .1963; °by,' 1964, 1965).

8.7.1 -The Logarithmic Loss Function

that represents the

The logarithmic'los,function is unique among these functions_in its exclusive dependence on the value'df the component of Rthat is assigned to the correct alternative. It is not affectedby how numbers are distributed over the other components of R.The function is given by

WL

Rix.) = log r - E

1r

j=

where k is a positive constant, and (RJx.) is read "responsevector R, given that xi is the correct alternative. The subjec-"tivelv expected paysoff, given this function, is

(35)

E(WL) = kE log r. - E r3

which is maximiced when ri = pi,

Max E(WL) kEpi log pi - 1 (Todaf 1963). .(37)

.4

(36) -

122

i3

NAVTRAEQUIPC1N 73-C-0128-1

p

Because, the maximum subjectively expected value is negative--henceits designation as a "loss" function--a constant is often addedto the function in order to make the payoff positive.' Also, because.the function becomes -coat r. = 0; a truncated version of it isusually employed in practice.

,8.7.2 The Quadratic Loss Function

l

The quadrdtic Toss, function is given by. .

\.14 'Mix.) = E r 2 + (1 - r )2Q k i, Ici

when the,number of alternatives is 4reater than two,/and by .

.2 'WQ(Rlx.)

lc%, k i

(.38),

(39)

.

for the two-alternative case; 1This,function is negative in thetwo-alternative case (although not necessarily'when the Number ofalternatives is greater than ,two), 5:10-as in the case of thelogarithmic loss, a:constant is often added to tHe.function toassure a positive payoff.

*

8.7.3 The Spherical-Gain Function ,

The spherical-gain function, which hash been elaborated byRoby (196) will be considered in somewhat more deai:1, becauseit has some useful properties that the,other rules/do not have,and a particularly elegant geometrical representatai as wellt.The payoff function is given by

2(Rlx.) = r. E r.

.,.-3=1

Ws 2(40)

A

For a proof that Ws is maximized only when R = kP see Snuford,Albert, and Massengill (1966). A reference to tHe example that wasused earlier should be sufficient to make the assertion plausible.Consider again the two-alternative examination item for which a

.student thinks the chances are 7 in 10 in favor of alternative A.Recall that if his score ES a linear function of the proportionof points he assigned to the correct alternative, theh nis beststrategy is to put zero on every alternative except the one heconsiders most likely to be correct; in which case, his expectedscore would be .70. To see that this is not true in.the case ofthe spheridal gain scoring rule, note that if the student putsall his stakes, say n points, on alternative A, his expected scorewill be:

123

136

NAVGRAEQUIPCEN 73.1C-0128-1

7/10 x (n/vn/2+0 2 ) + 3/10 x (0/vn/2 +0 2 ) = .70.

If, however, he weights the alternatives in accordance with hisjudgment of what-the chances really are,, his expected score willbe:

7/10 x ( 7//7C3Y) + 3/10 x (3/71-32) .76.

Te,

It should be noted that the pi-ocedure permits the student toassign weights to the various alternatives in any way'he sees fit.There might appear to be some advantage in forcing the numbersassigned to the alternatii/es for a given item to add to one,inasmuch as they'could then be interpreted directly as probabilityestimates. The student could,be instructed to make his assign-ments so that they would indeed add to one; however, this is anunnecessary demand since the score is unaffected by a change ofscale. MoreoVe, if we wish to treat the assignments' as proba-bility estimates, as we shall in what follows, we can easilynormalize them by simply dividing each assigned number by the sumof the numbers associated'with that question. When this is done,and each of the original numbers is replaced with the resultingquotient, then each of the resulting numbers will be referred toas a probability estimate, and the collection of numbers associatedwith a given item as a probability vector.

A nice feature of the spherical gain scoring rule is that itprovides an easy and intuitively meaningful way of distinguishingbetween one's confidence in the truth of a particular hypothesis(or correctness of a test item) and one's general degree of"resolution" with respect to the overall decision space (or .to thewhole test item).- Roby defined, as a "resolution index,"

2

( n3=1 ri

(41)

where RI represents an individual's confidence in his answer.Equation 41 is simply the denominator of equation 40 after thelatter has been normalized..

As in the case of Ws, the maximum value of RI is 1. It shouldbe clear that RI = 1 only if ri = 1 for one value of j and 0 forall others. That is to say, iu keeping with our intuitive notionsabout how an index of confidence should behave, it assumes itsmaximum value when one has put all his chances on a single alter-native. (Note that whether that alternative is correct or

124

137


incorrect is irrelevant to this measure -'-as it should be.) UnlikeWs

, RI canrDt assume the value 0. Its minimum value depends on'thenumber of alternative hypotheses under consideration, or--in thecase of the examination example--the number of candidate answersassociated with a question. It is obtained when . 1

r -3 E .rj

for all j; that is, the index gets its lowest:value when the samenumber is assigned to every alterhative. Agaih, this is consistentwith our intuitive ideas about confidence. The fact that theminimum value of the index depends on the numbeF of alternatives isalso in keeping with our intuitions about how al measure of confidenceshould behave: one should have less confidence,in a guess amongthree equally likely alternatives than in a guess betweenAwo ofthem.

8.7.4 Implementation of Admissibld4Probability MeasL.es

One of the practical difficulties in applying scoring ruleswith the matching property is that of providing subjects withintuitively meaningful information concerning the implications oftheir probability assignments vis-a-vis the scores that could re-sult from them. It is clear that simply providing individuals withformal expressions of the rules will not suffice, at least for-those who are not mathematically trained. One approach to thisproblem is that of illustrating the implications of any given rule-witb, concrete examples that make clear the advantages of beinghonest. Another, and perhaps prferredg approach is that of pro-viding the individual with an explicit representation of the payoffthat he would receive, given the truth of any specific hypothesisand the way in which he had distributed probabilities over thealternatives.

'Organist an4 Shuford designed .a pappr and pencil procedurefor providing this' information in the case of the logarithmicloss function (Baker, 1964; Organist, 1964; Organist & Shuford,1964). Shuford (1967) and Baker (1968) have also described acomputbr-based technique for providing similar information in adynamic way. 'In this case the alternatives open to the decisionmaker are shown on a computer- dxiven display. Associated witheach alternative is a line,thb length of which represents theuser's relative confidence that that alternative is the correctone. The user adjusts the lengths of the lines by means of alight pen. When the length of one line is changed by the user,the lengths of all the others,are adjusted by the computer so asto constrain the sum of the lengths to add to one, at all times.Also displayed with each line is a number which indicates to theuser what his payoff would be if the alternative associated withthat line were the correct one The logarithmic loss functiondeterthined the ,relationship between the number representing

125

:128


potehtial payoffs and the lengths of the lines in the applicationsof the system that are reported. However, the relationship couldas well have been determined by any other scoring rule of interest.

8.7.5 The Efficacy of Admissible Probability Measures

It would seem clear from the mathematics of the,situation*that scoring rules that have the matching property should be psedin preference to those that do not. It has not been clearly de-monstrated empirically, however, that subjects' tend to behavedishonestly if such rules are not used, or that their responsesare free of biases if they ,are (Aczel & Pfanzagl, 1966; Jensen &Peterson, 1-973; Samet, 1971; Schum, Goldstein,,Howell, & Southard,1967). Moreover, it is also apparent that many of the probabilityestimation situations of interest to investigators of decisionmaking are situations in which the only scoring rules that areoperative that'are imposed by nature. The s.tuationsin which ubj e probabilities are of greatest practi.,a1 sig-nificance ten o be those in which the payoffs are beyond the

: experimenter's control.S

8.7.6 ,SUbjective Probability Measurement and Training

One question of interest that relates to training researchis whether individuals who have had experience'in making probability 411,judgments in controlled situations with scoring rules that havethe matching property are more effective at judging probabilitiesin real-world situations than those who have had experience atestimating probabilities. but have not been exposed to matching-property rules. As has already been noted, some investigatorshave advocated the use of experts to estimate conditional proba-bilities to be used in Bayesian aggregation systems (Bond & Rigney,1966; Edwards, 1965b). Often, however, it is not possible todetermine how accurately such estimates are na 'de. If one had anobjective indicant of the probabilities of interest that was inde-pendent of the experts' judgments, it would not be necessary toget the judgments. It would be.of interest, however, to determinewhether the behavior of experts on such tasks would be sensitiveto the type of experience they had had in estimating probabilitiesin controllda situations, and in particular td their exposure toadmissible or inadTissible probability measurement techniques.Savage (1971) has suggested the early introduction of admissiblescorini 'rules to'children, albng with careful training in theassessment of opinion strength, could have the salutory effect ofdispelling some of the myths concerning the relationships IRetweencertainty, belief and action--e.4., the idea that one should speakand act as though certain, even when one is not, and the notionthat weakly held opinions are worthless--that are fostered byconventional educational testing methods.

NAVTRAEQUIPCEN 73rC7018-1'

Scoring rules With the matching prope ty answer to one aspectof the problem of measuring subjective pr babilitiesisnamely thatof structuring the ituation so that hohesty in reporting is thebest policy. There, are other as*Pectso;fthe problem, however, that .

are not so readily Solved. Expressiohs of certitude, have been shownto vary considerably as a function o the way in which they arereported (Samet, 1911) and of the.c text in which they are obtained(NickersoR & McGol4ick, 1963). Typically, when subjects are askedto rate their confidence in their/own performance on a perceptualor cognitive task,/a positive cotrelation between these variablesis'found--confidenice is highest when performance is best- -but thestrength of the relationship is not always great, acid the signifi-cance of -a given confidence Fating depends on the situation and theperson making it (Andrews &Aingel, 1964; Nickerson,& McGoldrick,-1965). A fund6ental question that is raised by these results iswhether suchtactors affeCt certitude itself, or only its expression.A further question is whether such variability-rwhatever its basis- -can be eliminated, or at,least significantly reduced, as a resultof appropriate training.'

8.8 The Use of-Unreliable Data

In the foregoing discussions of the use of Bayes rule, it hasbeen tacitly assumed that the data used in estimating conditionalor posterior probabilities had been accurately observed and re-ported. In the chips-in-urn illustrations, for example, it wasassumed that, one could examine a chip and determine its color easily,or that someone else determined the color and reported it accurately.Thus, the decision maker could operate with complete confidence inthe data at his disposal. In the real world of decision making,things often are not this way. Frequently, the observation or thereporting of events is faulty, and the decision makei is obligedto take this fact into account when making use of the data that hehas obtained.

We naturally assume that data from a trustworthy source willbe more useful to a decision maker than will data from a sourcethat has not inspired confidence in the past. The use of anexplicit reliability rating procedure for intelligence reports byNATO army forces (see Section 5.2) is based on such an assumption.Few attempts have been made, however, either to validate thisassumption or to determine in a quantitative way exactly how con-fidence in a data source does affect the way in which the datafrom that source are applied to a-decision prioblem.

8.8.1 Prescriptive Approaches

One class of prescriptive models for taking into account the..-Treliability of data has come to be known as "cascaded" or "multi-

stage" inference, suggesting a process of hypothesis evaluationthat involves more than one step. Schum and DuCharme (1971) pointout that research on cascaded inferepce has been focused on two

127

140

NAVTRABQUIPCEN 73-C-0128-1

situations: one, in which the observer or reporter of an event ex-presses his degree of certainty concerning whether or-not the.eventactually occurred (see, for example, Dodson, 1961; Gettys & Willke,1959; Steiger & Gettys, 1972), and a second, in which the report ofan event is made without qualification by a source that is known tobe less than perfectly reliable (see, for example, Schum & DuCharme,1971; Schum, DuCharme, & DePitts, 1973; Snapper & Fryback, 1971).

In both cases, attention has been confined primarily torelatively simple situations, e.g., those in which (1) the decision.maker's task is to determine which of two hypotheses, H1 or HI, istrue, and observations have only two possible outcomes, D,

'and D2,

and (2) the reliability of a report is independent of thehypotheses that are being, considered, that is to say, event D1and D

2 are neither more nor less likely to be confused underH1

than under H2.

Dodson (1961) considered the Situation in which an observeris not certain which of two mutually exclusive events, D1 andand n

-""has occurred, but may be able to make a probability or certitudg.judgment on the question. He suggested that in order to calculatethe posterior probability of a hypothesis in this case, one shouldcalculate its value, given each of the possible' events, and thentake a weighted sum of these values, the weights being the proba-bilities'that the observer attaches to the event possibilities.Given only two possible data, the calculation may be representedas follows:

g(HIID) = *(D1)P(HiD1) + W2)p(HilD2) (42)

where (HiID) is the posterior probability of Hi, taking the ob-server's uncertainty into account, and i(D.) is the probabilitythat...the observer attaches to the possibility that he has observedevent D.. More generally, given n possible events and the assump-%tion that the observer can attach a probability-to each of them,the formula might be written as

n(H. 1D) = Ip(D )p(H.ID

j).

(43)j=1

Substituting the Bayesian formula for p(HilDj) we have

(Hi [D) = EV(D.).77; POD.1H.)p(H.).3 i 1

p(DjiHi)p(Hi)(44)

Using Dodson's work as a point of departure, Gettys and Willke(1969) and Schum and DuCharme (1971) gave the process of dealingwith unreliable data a more explicit two-stage form. The following

128

141

NAVTRAEQUIPCEN 73-C-01.28-1

-----

discussion roughly follows Schum and DuCharme. These writers fo-cused on the case in which a decision maker obtains informationabout data via a 'source tiat sometimes incorrectly reports whathas adtually occurred. (It is irrelevant to this discussion whetherthe source's errors' are assumed to be errors of observation orerrors of report.) Fot each of the decision problems that wereanalyzed there were two hypotheses, H1 and H1, two possible dataevents, D., and Dl

1

, and two possible reports By the source, d1

and dWhat one *ants tO.determine is p(H.Id

j).*

ACcording to Bayes ruleu

P(dillii)p(H1)

p(d.)3

aft

(45)

The problem then is to determine p(dilH.). If the pr..bability ofa datum conditional on a hypothesis,Jp(k1H.), and the ,irobabilityof a report, conditional jointly on a hypothesis and a datum(p(d.i1H.9Dk), are known, then the probability of a report, .condi-tion41 6n a hypothesis p(ddifi) can be easily calculated. Therelationship- is given by

p(d.IH.) = ),k " jik (46)

a graphical representation.of which is shown in figure 26. When,by assumption, the reliability of a report is independent of thehypothesis that is being considered,

p(dilHinDk) = p(dilDk) (.47)

so, in effect,

p(d j IH.) =,Ep(Dk IH.)p(d.ID

k) .

Schum and DuCharme refer to

(48)

p(d.1Hi)A

-p(d3.

rHk )

(49)

*Our notation differs slightly from that used by Schum and DuCharme:we use D

1and D

2to represent the two possible data events, whereas

they used D and D, and we use d1 and d2 to represent reported datawhereas they used D* and D*.

129

i42


Figure 26. Graphical representation of derivation of

p(dilHi) and' adjusted likelihood ratios

for less than completely reliable data

130

143


as the "adjusted likelihood ratio," the likelihood ratio that takeiinto account the'degree of reliability of the source. Of course,A reduces to the standard likelihood ratio when the source isassumed to give completely 'reliable reports, inasmuch as, dn thiscase

I for j = kP (:dij I Dk ) '

0 for j k.-

The way to deal with the problem of unreliable data then, accor-diftsto Schum and DuCilarme is with a two-step process: (1) adjustthe diagnosticity of the data by determining p(d.IDIF) or A, and(2) apply the adjusted data to revise the distrigution of pro-babilities over the hypotheses via Bayes rule.

What one must be able to measure r estimate in order touse this procedure are p(DIH), the sta and conditior.11 probabilitiesof Bayes theorem, and p(dID), the indic s of source reliability.Schum and DuCharme define source reliability in terms of

r = p(dirDi)

the probability that the source will report a data event accurately.'They distinguish four different decision "rases" in terms of cer-tain symmetries and asymmetries involving p(d1H)Jand p(d1D), andthey develop the implications of their prescription for dealingwith unreliability for each case. The cases that they distinguishare:

Case I: Symthetric p(DIH): Symmetric p(d1D)

= p(D21H2); = p(d21D2)'

Case II: Asymmetric p.44.1H);

p(D11H

1) 0 (1)21H2);

Symmetric p(dID)

p(dilDi) = p(d2ID2).

Case III: Symmetric p(D1H): Asymmetric p(dID)

= p(D2IH2); p(dilDi) p(d21D2).

Case It: Asymmetric p(DIH); Asymmetric p(dID)

p(b2IH2); ,p(d21D2).

131'

44

a

NAVTRAEQUIDCEN 73-C-0128

In brder'to avoid the use of conditional probability nota-tion, Schum and DuCharme introduced the following notationalequivalencies:

For symmetric p(DIH): p E p(DilHi)

For asymmetric p(DIH):A

For symmetric p(dID):

1=p Es(DilHi), ji.

p1 E p(DilHi)

P2 P(D11142)

1-p1 E p(D21111)

1-p2 E p(D2IH2)

r E p(dilDi)

1-r E p(dilDi),ji.

For asymmetric p(dID): r1 E p(d1ID1)

r2 E p(d2ID2)

1-r1E p(d

2ID

1).

1-r2 E p(dilD2) .

0

Letting the subscripts on A represent symmetry or asymmetrywith respect to p(DIH) and p(dID), respectively, and maRing theabove substitutions into equation (46), as appropriate, we obtainSchum and DuCharme's expressions for the prescribed use of dataof imperfect, but known, reliability for each of the four casesthey considered., All adjusted likelihood ratios represent

p(difH2)

132

145

,NAVTRAEQUIPCEN 73-C-0128-1

pr+( )1 (' -r) .Case I: As,s (1-p) +p(1-r)

Plx*(1-13Case II: Aa,s p2r+(l-p

or, equivalently,

Where

4

Case III:

or if 13l and

where

p1+k

Aas p

2+k

1-tr

2r-1'r 5 .

pki+(l-p)(1-r2)

As,a 117-T)11-11)(1-4.2)

c-11r +1As,a = Ll-P

c+-2-11-pj

(50)

(51)

(52)

(53)

(54)

(55)

Case IV:

or if r1V(l-r2)

where

c =

Aa a

Aa,a

b

r1 .

1-r2

P1r14.(1-p1)(1-r2)p2r1+(1 -p2)(1-r2)

131443

p2+b

1-:2

1-(1-r

2

133

146

(56)

(57)

(58)

(59)


It follows from the definitions of unadjusted and' adjustedlikelihood ratio that the latter is always cloer to unity thanthe former and that the difference between them increases asreliability, r, is decreased from 1.0 to 0.5* (except when thedata, are completely uninformative to begin with and the unadjustedratio is 1). This is consistent with the intuitively compellingrequirement that the less "reliable the data, the less diagnosticimpact it should have.** Figure 27 shows how the difference be-tween unadjusted and adjusted likelihood ratio grows as reliabilityis decreased, and how the, adjusted ratio goes to 1 for r = .5, forthe case in which both p(D(li) and r are Symmetric, i.e., Schum and'.DuCharme's Case I.

Figure 27 also illustrates the fact that the greater thediagnostic impact of data (when reported by a completely reliablesource), the greater is the effect of a decrease in reliability ofa report. This also is an intuitively reasonable relat'onship:the less informative data are to begin with, the less there is tolose if they are reported unreliably. What is less intuitivelyapparent is the fact that even avery small decrease in reliabilitymay have an extremely large eetect on likelihood ratio if the un-adjusted ratio is very high. Schum and DuCharme (1971) point out,for example, that in Case I, if a datum with an unadjusted likeli-hood ratio of 100,000 is-reported by a source with a reliability,of .99, the adjusted ratio is reduced by about four orders ofmagnitude to slightly less than .99.

The results of Schum and DuCharme's analysis bear on issuesrelating to the design of information and decision-making systemsand on the role of humans therein. For example, they show that underCase I conditions, there is a reasonably straightforward tradeoff

A

*Decreasing r below 0.5 has the effect of making the adjusted,likelihood atiO depart again from unity, although it still remainscloser to unity than does the unadjusted ratio. In otherewords,decreasing the reliability quotient below 0.5 increases thdiag-nosticity of the data, but in sunport of the alternative hypothesis.This is consistent with the idea that a source that is consistentlywrong may be very informative; one need only interpret its reportas evidence of the oppo4te of what it says. In this, discussion,we will confine our attention to the case in which 1.0.> r > 0.5!- -* *Schum and DuCharme (1971) point out, however, that when thereliability of report is not independent of which hypothesis'isbeing considered, it is possible for A to differ more from 1 thandoes L; that is to say, it is possible for a decrease in relid-bility, in that case, to indraase the diagnosticity of data.

134


0.8 0.6 0.4

Reliability r

0.2 0.0

Figure 27. Adjusted likelihood ratio (A) As a functionof data reliability (r) for valuesof unadjusted likelihood ratio L), for Schumand DuCharme's Case

1'

NAVTRAEQUaPCEN 731-C-0128-1

between p(DIH) and, p(d(12). And' the tradeoff is such that if onewants to',increage the diagnostic,impact of information flowingthrough.a% stenr, and the costs of increasing the conditionalsp(DIH) an p(dID) arejequal, one should increase the smaller ofthe two.

/

Also, the analyses show that in Cases II.and IV A is dependentvupon specific values of pl and p, rather than on 'their ratio. 'Rlus,

despite the fact that earlier results have suggested that peoplefind. it easier to make judgments of likelihOod ratios than ofConditional probabilitie's, there may be situations in which esti-mates of the latter should be required.

8.8.2 Some Empirical Results

e..

The models ddveloped.by Schum and DuCharme are prescriptive,providing for optimal adjustment of the likelihood ratio underconditionsLin which .data are reported with lesg'than total, butknowx'i, reliability. We now.turn to a consideration of severalstudies aimed at comparing actual performance against that prescribedby these modelq. In the next section we'then present a,briefaccount of some descriptive models suggested by these results.All experiments and models ttlt will be considered ,in these sectionaddress situations where input to the decision process is an eventor set of events reported bia !Lila!e unreliable source.

Snapper and Ffyback (1971) present,the results Of a study inwhich the experimenter reported to the subject with (symmetric)reliabilities of 1.0, 0.9, and 0.7 the outcomes of events concep-tually similar to the draws of chips from an urn. The probabilitiesof events conditional on hypotheses, p(DilH1), p(D211.11) andp(D11111), p(D11H0, were, respectively, as follows: 1a)p.33, 0.67ran0.67, 0.33; lb) 0.80, (1.40 and 0.60, 0.40; (c) 0.90, 0.10 and0.45, 0.55; (d) 0,.25, 0.751'dnd 0.75, 0.25. For conditions inwhich the experimenter's reliability was egua1 to unity, only (a)and (b) were used. Subjects were required to indicate which ofthe hypotheses they considered ilioreclikely as a result of the ex-gerimenter's report, and how much more likely than the alternativehypothesis they considered it to be. Under conditions of unitreliability, subjects' estimates corresponded very closely to theactual likelihdod ratio, but when reliability was less than unitythey represented slight underestimates of the impact of the least

'diagnostic reports and overestimates of the impact of the remainingreports. The extent, of this overestimation, moreover, increasedwith the magnitude of A.

Johnson (1974; see'als -Johnson,1973),has'utilized a similar task andstudy the effects of four different vinference: (1) sample size, the numbaxcumulative outcome report (er.g:, "

136

149

avanagh, Spooner, & Sainet,response structure toriables on cascaded

r of draws which under:layhree reds, two blacks");

-"NAXTRAEQUIPCEN 73-C-0128-1

ata&

(2) data generator diagnosticity, the relative composition of redand black chips in the, urn; (3) sample diagnosticity, the diagnos-tic value. defined by the difference between total numbers of redand of black Chips underlying a report; and (4) source reliability.Posterior-odds estimates that were obtained in thisCacaT-77E5E51to be sensitive to different values of sample size, data generatordiagnosticity and source reliability, tending to decrease as thevalues of these variables decreased. When the report was known,to be perfectly reliable, estimates of posterior olds were generallymore conservative than those computed from Bayee..thedrem; however,they became progressively less conservative and approached optimalvalues at intermediate levels of reliability (.8-.7), and thenbecame slightly excess ve at lower levels (.7-..6).

The diagnosticity and reliability of reported events weremanipulated by Youseff and Peterson (1973) in sucha way that thevalue of A in a situation requiring multistage inference wasequal to the value of the standard likelihood ratio in .a single-stage situation (that is, one with report reliability, equal.to.unity). Subject'' estimates. tended to be conservative for highvalues, both of f'and of L, as compared with the Bayesian mbdel,and tended to be excessive.for low values. 'The odds estimatedin conditions requiring multistage inference were consistentlygreater than those estimated in single-stage conditions and, asa result, were excessi 've compared to the optimal odds over awider range than were single-stage odds. t

Schum, DuCharme, and DePitts (1971) conducted a study in whichthe accuracy of subjects' own. observations of ,the number of Xscontained in tachistoscopically presented 4 x 4 matrices of Xsand Os constituted the reliability levels. Subjects were requiredto estimate the relative likelihood of twti possible hypothesesre sting to the data generator after each of five stimulus 4

pk sentations; Under conditions in which sufficient time wasav ilable for totally accurate observation of the stimuli, e ti-mates became increasingly conservative compared to the optimal,model as the diagnosticity of each observed event.and the infe-rential consistency over a set of five events increased. Underconditions in which insufficient time was available for accurateobservations, the subjects' estimates were generally close tooptimal or slightly excessive when diagnosticity and .consistencywere high, and became more conservative as either of these para-meters assumed low pr values. In a second phase of this same study',subjects estimated/ directly the diagnosticity of data based onbrief obsbrvations of each slide. Compared with the optimal model,such estimates become increasingly excessive as L increased.

The results of these studies establish that the behavior ofdecision makers is indeed influenqed by the degree of reliability,of their data sources. They alsodemonstrate, hoWever, that

'J


pefformance tends not to.be consistent with that prescribed bythe formally appropriate rule for adjusting data diagnosticity.Further, performance with unreliable data often differs in oneimportant respect from that that has been observed in classicalBayesian inference situations in which events are observed, orreported,'With perfect accuracy.. Whereas in the latter case thedecision mper's estimates, though revised in the appropriate di-rectiod, tend to be conservative as compared with Bayes theorem,his estimates based on less-than-completely-reliable data fre-quently appear. to be excessive as compared with Schum and Du-Charme's prescription for optimality. Because the value of A asdefined by the Schum and DuCharme model, in effect, makes an adjust-ment in the direction of increasing conservatism (produces a valuecloser to unity), the two effects--conservatism vis-a-vis L andexcessiveness vis-a-vis A- -can offset each other, if conditionsare just right.

-8.8.3 Some Attempts to Develop Descriptive Models ofCascaded Inference

As we have noted, the model developed by Schum and DuCharme(1971) for dealing with unreliable data prescribes two steps, orstages: in the first stage, the nominal diagnosticity of a datumis discoUnted to reflect the degree of reliability of the source,and in the second, the adjusted datum is applied to the hypotheses'under evaluation in accordance with Bayes rule. If hypothesesare being evaluated in terms of odds, the process can be repre-sented as,follows:

Stage 1: compute A

Stage 2: compute Qi=

P(d1H1)p(d7-17

where A represents the adjusted likelihood ratio, and Ql and Q0represent the posterior and prior odds, respectively.

. The experimental results that were reviewed briefly abovemake it clear that people typically do not behave in accordancewith this prescription. Several investigators have attempted todevelop models that do describe behavior.

The results obtained by Snapper and Fryback (1971), usingsymmetric reliabilities, suggest that in dealing with unreliabledata, decision makers estimate the likelihood ratio as though thedata were completely reliablp, adjust the resulting ratio by mul-tiplying it by the reliability quotient, and then apply the ad-justed ratio to the calculat_on of posterior odds. The processmay be represented as follows: 'v

138

. 151

NAVTRAEQUIPCEN 731-0128-1

Stage 1: compute W = rL

Uage 2: compute k = 110.

Snapper and Fryback note that the optimal rule for the first stageof the process is neither 'apparent nor intuitive, whereas the rulethat seemed to describe the behavior of thdircsubjects has someintuitive appeal and is easily applied. Its use leads, however, tosubjective estimates of likelihood ratio that are excess ve incompdrison with those prescribed by A. That is to say, A leads tooverestimation of the diagnostic Wadt of a given (urireliable)datum.

The extent to which W overestimates A--for Schum and Ducharme'sCase T--is illustrated in figures 28 and 29. Figure 28 showsboth A and A as eunptions of ,r for several values of L; figure29 show's the ratio A/A for the same,conditrons. The figures show 'only cases for which L_> 1 and r > .5. For L < 1, one obtains thesame relationships by simply expressing-'the lkkelihobd ratio H2re HI rather than H re H.

2' The case of r <.5 is of little'.interest for the reason explained in the first footnote on page 134.As may be seen from these figures,'the,degree to which ? over-

estimates A depends both on L and t: for given L it tends tovary inversely with r (given i >.5) and for given r it increasessharply with L.

'Gettysi' Kelly, and Peterson (1973) have suggested a model

that is slightly' different from that,of Snapper and Fryback. Itassumes that the decision maker estimates posterior'odds on theassumption that the most likely event is true, anNthen adjuststhe odds to reflect the reliabiliity of the data ;"51.1tce. Thismodel may be represented as follows:

Stage 1: compute P, = Ln_ v

Stagd 2: compute k = rP1*

It is apparent that although the process by which the pos-terior odds are estimated differs in the two cases, the resultsare precisely the same. Edwards and Phillips (1966) have presentedevidence, however, suggesting that the way in.which people estimateposterior odds may be better described by

Q LcQ1 0' (60)

where c varies with L, than by the prescribed Q. = L2 n. Funaro(1974) points out that the models of Snapper and FrybNck, and of

139

152

$2?

S

VAVTRAEQUIPCEN 73-C-0128-1

111=114 . ;

,

- .

, .

1 ,

_ .' i__t____. ...:__ :., . -1-1 : J - 1

t . i

., : 1 1 . - -:- - 1- , . ...- --r----.,- - -- g's, -~ --- - I-- ...... .............................- -I - L--1 -i- : _E '_`t-Nt . . 1 ......, .... ----...... -...-.1.......-

1 '.... , .- I ..111:___- '

- . ... %I L - .., .."I

1111

1

, .S

alI1Y

4/1

1.0 0.9 0.8 0.7

Rellanlity r

, 0.6 0.5

11/ 4Figure 28. A and A as functions of r for several

values of L

ti

0

00T4

1


mmnamMe~mlao,1

; -,i I

Oar

;

I I-1-±-''

1.0 0.9 0.8 0.7

Reliability r

0.6 0.5

Figure 29. The ratio of W to A as a function ofr for several values of L

bt

,


Gettys, Kelly, and Peterson make different predictions if the oddsare calculated according to Phillips and Edwards' expression. Theformer leads to

SI1

= (rL)cO0

and the latter to

1= rLcct

(61)

(62)

Funaro (1974) has recently attempted to evaluate the pre-dictive power of Snapper and Fryback's model and of that of Gettys,Kelly, ancKPeterson, using both L and.Lc as unadjusted likelihoodratios in each base. A symmetric p(DIH)--symmetric'r task (Schumand DuCharme, Case I) was used. Subjects were require., to reviseodds' estimates under both single-stage (perfect-source reliabilityAssumed) and cascaded-inference conditions. Values of c wereestimated separately for individual subjects from data obtainedin the single -stage conditions.

The results were not consistent with any of the models de-scribed above. They were predicted best by another model thatFunaro proposed. This model, which Funaro called the empiricalmodel, assumes that subjects accurately estimate A, and then applythis estimate to the revision of odds with the same degree of ef-fectiveness, or ineffectiveness, with which they apply L in single-stage tasks. The conclusion appears to be inconsistent with theresults of Youssef and Peterson (1973) who found that odds's es-timates made under cascaded conditions were consistently excessiverelative to those made in single-stage tasks, given A = L.Funaro notes, however, that subjects in his experiment couldhave acquired a direct appreciation for A from the proportionof successes and-failures in a series of reports obtainedfrom the source during the course of the experiment. (In a sym-metrical p(DIH) chips-in-urn situation, one can unambiguouslydefine a "success" as the drawing--or in this case reporting--ofa chip of the predominant color.) To the extent that subjectswere able to develop a direct awareness of A, the effect wouldhave been to eliminate the need for a two-stage process and totransform the task into the simpler problem of revising odds onthe basis of totally reliable data. The suggestion is an eminentlyplausible one and the possibility that this(is in'fact the wayunreliable data are often accommodated in real-world situationsdeserves further study.

. 8.9 Some Comments on Bayesian Hypothesis Evaluation

Inasmuch as the Bayesian approach to hypothesis evaluationhas received so much attention by decision theorists And investi-gators of decision making, ieseems important to consider some of

142

155


the limitations of thj_s approach. To point out limitations is not,of course, to deny that the approaqh has merit. Among its advantagesare the fact that it places minimal demands on memory_because_datacan be discarded after being used to'update the distribution ofprobabilities over hypotheses, the fact that it provides a meansof aggregating qualitatively different data in a meaningful way andthe fact that the procedure for applying data to the evaluation ofhypotheses automatically weights data in terms of their relevanceto the hypotheses being evaluated. It is precisely because the ap-proach does work well in some Contexts that there is a danger ofuncritically concluding .that it is appropriate in all cases. Thefollowing observations are based largely on a diiscussion by Bowen,Nickerson, Spooner, Triggs (1970).

First, Bayes rule itself applies to only one of the several taspects of decision making; namely, hypothesis evaluation or, moreprecisely, -the resolution of unceAainty concerning tae state ofthe world. Whatever its efficqcy-For that particular task, it isnot the grand solution tO the problem of decision making.

Second, application of Bayes rule requires that the decisionproblem be structured-in a very precise way. In particular, itrequires that one's uncertainty about the state of the world berepresented as an exhaustive set of mutually eRclusive possibilities.It does not, however, provide any help in identifying these possi-bilities.

Third, the requirement for an exhaustive set of mutually ex-clusive hypotheses about the state of the world precludes thepossibility of exipanding,one's hypothesis space as one goes alon.It clearly often is the case, in real-life situations, that new 'hypotheses are suggested t,y incoming data. That is to say, obser-vations may have the effect not only of'modifying the credibilityof existing hypotheses, but of suggesting new hypotheses as well.

FOurth, the fact that use of Bayes rule presupposes a set ofmutually exclusive hypotheses has another implication. 'By defini-

. tion, one and only one of these hypotheses can be.true; all theothers must be false. The probabilities that are associated withthese hypotheses do not, of course, represent their truth values,but, rather, the decision maker's opinion concerning their truth,or falsity. It was pointed out in the preceding paragraph that noprovision is made for the possibility that the hypothesis set doesnot contain the true hypothesis. It is also the case that provisionis not made gor the Possibility...that more than one of the hypo-theses are true, or that one or more is partially true.

Fifth, application of Bayes rule is a recursive process: eachtime that a new observation is to be used to update a posteriorprobability estimate, the posterior probability from the preceding

rT143


'NJ

update is used as the prior probability for the current update.however -- before the fiist observation is made--the

prior; probabilities must be estimated, and Bayes rule does nothelp ih thi-S regard. Investigators are not entirely agreed on howthese prior probabilities should be assigned--or on what they mean.It is often pointed out that how prior probabilities are assignedmay make little difference (provided values very close to 0 or 1are not used), because the effect of the initial values will belargely nulled after several observations have been made. Theproblem can be a significant one, however, when hypotheses must beevaluated on the basis of relatively few data. In such cases, theinitial prior probabilities can have a very strong effect on thefinal posteriors, and thus the way in which they are assigned'isof considerable concern.

Sixth, the basic assumption that justifies a Bayesian approachts) hypothesis evaluation is the assumption that man is Netter atestimating p(DIH) than at estimating p(HID). We have noted inpreceding sections some experimental evidence that tends to supportthis assumption. We have also noted some studies, hOwever, thathave shown that this result is not always found. Moreover, thereis a question concerning how far the evidence that does supportthis assumption can be pushed. The only way that one can determinehow accurately a man can estimate p(DIH) is to observe his perfor7mance in experimeAtal situations in which p(D(4) is objectivelydefined or can be determined empirically. But, typically, in real-life situations of greatest interest, p(DIH) is not known, andcannot be determined empirically-which is why is must bedefined or can be determined empirically- -which is why it must beestimated. The question arises then, if it is not known, hoy canwe be sure that one's estiMate of it is accurate? And the answeris that we cannot. How much confidence one should have in the.on-clusion that man is better at estimating p(DIH) than at estimatingp(HID) in real-world situations depdnds in large part on the extentto which one is willing to assume that what is known about perfor-mance in laboratory situations in which p(DIH) usually has astraightforward relative - frequency interpretation.is generalizableto real-world situations in which it does not.

Seventh, Bayes rule does not provide the decision maker witha criterion concerning when to stop processing incoming data andto make a decision. Inasmuch as data gathering can be costly interms of both time and money, it is'essential that any completelyadequate prescriptive model of'decision making have an explicitstopping rule to indicate when hypothesis evaluation should be ter-minated and a decision made:

144

15'2V

j

NAVTRAEQUIi3CEN 73-C-0128-1

We emphasize that these comments deal with limitations ofBayes rule. One might argue that the observations are unnecessary,on the gtounds that proponents of Bayesian diagnosis have neverclaimed that these limitations do not exist. It seems to usimportant to make these limitations explicit, however, becausethey help to place the notion of Bayesian decision making inperspective. The idea of obtaining estimates of p(DIH)lor oflikelihood ratios from humans and using these estimates ,o updateposterior probability distributionslin accordance with Bayes theoremis undoptedly a reasonable approach to evaluation in some situa-tions. It is not always appropriate or practicable, however, assOme,,Bayesians have been careful to point out. Edwards (1967) de-scribes the situations for which the approach is most appropriateas those that have one or more of the folic:riling three characteris-tics: "the input information is.fallible, or the relc.tion of inputinformation to output diagnostic categories is ambiguous or uncertain,or the output is required to be.in explicitly probabilistic form"(p. 71).

Ma

145


SECTION IX

PREFERENCE SPECIFICATION

It is generally assumed that .a decision maker is not indif-ferent to which of the various possible decision_outcomes_ occurs.AS-we hae,already noted, in some formal representations of de-cision situations, the decision maker's perferences with respectto the possible outcomes are made explicit in a payoff matrix.The contents of a cell of such a matrix is the worth to the de-cision maker of the choice of a specific action-alternative, giventhe truth of a specific hypothesis concerning the state of theworld. The entire matrix presumably' represents the situation

'14fully: it identifies all the decision maker's action alternativesas well as all the possible states of the world, aftd shows for eachalternative-state combination its worth to the decision maker.

9.1 A Difficult and Peculiarly Human Task

The problem is how to determine these worths. There are twoobservations to make in this regard. The first is that this task,more than any other associated with decision making, is peduliarlyhuman. One would expect that many of the decision-related tasksthat now must be performed by humans will in time be performed bycomputers. However, the specification of preferences for decisionoutcomes iiivolves value judgments. To say that one decision out-come is, better than, worth more than, or preferred to, another isto say that it represents a greater good within the context of the .

decision maker's own value system. Such judgments must come, atleast indirectly, from man.

The second observation is that to specify one's preferencesobjectively is not necessarily an easy thing for an individual to

41q -do. Even when all of the action alternatives have been made ex-plicit and the outcome of each possibility is known--that is, evenwhen uncertainty is minimal--the decision task may still'be a verydifficult one. This is particularly true when the worths ofpossible decision outcomes are intangible or depend on many factors.Consider, for exaree, the problem of choosing a house for purchase._.Even assuming- tba one confines his attention to' a few houses thathe knows are available, and that he has all the information thathe wants about each one, he has the problem of somehow derivingfrom many factors (purchase price, number of rooms, design, generalcondition, extras--porch, garage, storage space, extra baths,fireplace--lot location and layout, distance from work, tax ratein town, services and public facilities in town) a common figureof merit in terms of which one house can be judged to be more orless preferred than another.

e

In military situations, the specification of preferences maybe especially difficult. It may often happen that nonegof thepossible decision outcomes is intrinsically desirable, and the

146

159


decision maker may find himself faced with the necessity of attempt-ing to choose the least undesirable one. The problem is aggravatedby the fact that the assignment of preferences may necessitate theweighting of time, materiel, territory and human lives. One balksat the.idea oftrying_to_specify the value/of human lives and thatof a piece of territory in terms of a common metric, but this iswhat is done, at least implicitly, when a decision is made to at-tempt to.gain a territorial objective when it is known that theendeavor is likely to result in the loss of a certain number of men.Or, consider the private transportation system in the United States.The builders, users and regulators of. automobiles and highways haveimplicitly expressed a preference for a system that provides certaincapabilities and convenkences at a cost of approximately 60,000traffic fatalities perear. One suspects that the exercise ofmaking explicit how the various factors-that contribute to humanpreferences are traded off against each other in specific decisionsituations would often be revealing to decision maket. themselves,who sometimes may have little conscious appreciation, w.Lthout goingthrough such an exercise, of how such factors do combine to deter-mine their own preferences.

Among the eight aspects of decision making in terms of whichthis report is organized, preference specification is one of thetwo (the other is hypothesis evaluation) that have received thegreatest amount of attention from.philosophers and researchersalike. In the case of decision making under certainty,,the studyof preferences and the study of choice behavior amount to the samething. Prebumably one chooses what one prefers--and vice versa- -if he can know for certain what the decision outcome will be.

9;2 Some Early Prescriptions for Choice

In order to make choices arpng alternatives that differ withrespect to several incommensurate variables, ,one must, at leastimplicitly, derive from the several Variables involved a singlefigure of merit with respect to which the alternatives pan becompared. That is to say, one must be able to decide that in someglobal sense Alternative A is preferred to Alternative B. _How thisis-generally-done is not known; how it should be done is a matterof some dispute. Undoubtedly, individual methods for dealing withthe problem range )

from highly intuitive impressionistic approaches(I just consider all the factors and decide that I like this com-bination better than that) to formal quantitative algorithms.

.Benjamin Franklin was familiar with the problem, and his wayof dealing with it is at least of historical interest: "I cannot,for want-of sufficient premises, advise you what to determine, butif you please I will tell you how... My wayig-to divide half asheet of paper by a line into two columns; writing over the one Pro,and over the other Con. Then,during three or four days' considera-tion, I put down under the different heads short hints of the

147

160


different motives, that at different times occur to me for oragainst the measure. When I have thus got them all together inone view, I endeavor to estimate therespective weights...[to] find,at length where the balance lies... -And, the weight ofreasons cannot be taken with the precision of algebraic quantities,yet, when each/ is thus considered, separately and comparatively,and the whole matter lies before me, I think I can judge better,and am less liable to make a rash step; and in fact I have foundgreat advantage for this kind of equation, in what may be calledmoral or prudential allttEa..*

A more formal attempt to procedurize choice behavior was madeat about the same time by the British philosopher and social re-former, Jeremy Bentham. Starting with the basic premise thatchoices should be dictated by the extent to which their outcomesaugment or diminish the happiness of the party or parties whoseinterest is in question (the "principle of utility"), Benthamattempted to define a quasi-quantitative procedure--a "hedonisticcalculus"--the use of which would assure that the choices that are,made would be consistent with this principle:

"To take an exact account then of the general tendencyof any act by which the interests of a community are affectedproceed as follows. Begin with any one person of those whoseinterests seem most immediately to be affected by it,,andtale an account:

(1) Of the value of each distinguishable pleasurewhich appears to be produced by it in the first instance.

(2) Of the value of each pain which appears to be pro-duced by it in the first instance.

(a) Of the value of each pleasure which appears to beproduced by it after the first. This constitutes the fecundityof the first pleasure and the impurity of the first pain.

(4) Of the value of each pain which appears to be pro-duced by it after the first. This constitutes the fecundityof the first pain, and the impurity of the first pleasure.

(5) Sum,up all the values of all the pleasures on the oneside, and those of all the pains on.the other. The balance,if it be on the side of pleasure, will ive the good tendency

*41

*This account of Franklin's approp.ch to decision making was quotedby Dawes and Corrigan (1974), who found it in .a letter from Franklinto his friend Joseph Priestly, dated September 19, 1772.

148

1.61-

NAVTRAEQUIPCEN 73-C-0,128-1

of tbe act upon the whole, 'with respect to the interests ofthat individual person; if on the side of pain, the badtendency of it upon the whole.

(6) Take an account of the number of persons whoseinterests appear to be concerned, and.reptat the above pro-cess with respect to leach. Sum up the, numbers expressiveof the degrees of good tendency which the act has, with re-.spect to each individual in regard to whom the tendency ofit is good upon the whole; do this again with'respect to eachindividual in regard to whom the tendency of it is bad uponthe whole. Take the balance; which, if on the side ofpleasure, Will give the general good tendency of the-act,with respect to the total number or community of individualsconcerned; if on the side.of pain, the general evil tendency,with respect to the same community" (Bentham, 1939, p. 804).

The value of a pleasure or pain, Bentham assumed, wetild dependon four factors: .

" (1) Its intensity.(2),Its duration.(3) Its certainty or uncertainty.(4) Its propinquity or remoteness."

Bentham did not expect that the procedure he defined would be"strictly pursued previously to every moral judgment, or to everylegislative or judicial operation"; but he did contend that itrepresented.a model of how judgments should be made, and a stan-dard against which whatever procedures are used might be evaluated.

Bentham's approach to choice behavior can be, and has been,,criticized on philosophical grounds. The principle of "the greatestpleasure for the greatest number" is itself open to criticism,because it appears to place no limits on the extent to whicb themany can prosper at the expense of the few, provided only that the"bottom line" of the calculation,of the net happiness is increased /in the process. For our pttrposes, the important_point is the fact__

-.- that-Bentham-attempted to-r-educe the process of making choices.toa stepwise procedure.

9.3 Simple Models of 118i.th Composition

Although he used language that suggested that he believedthat worth could be quantified and his procedure formalized as 0a port of calculus for computing the worth of any given decisionoutcome, Bentham did not himsqd_f express his notions in mathe-matical form. His conceptualization of the choice process, how-

, ever, is clearly suggestive of a linear model which expresses the,- \worth of a decision alternative as a function of the sum of the

149

162


values of the various components of pleasure (or pain) that thatalternative represents, weighted by the number of .people thatwould be affected by the decision outcome. ,The choice would, ofcourse, be the alternative with the greatest calculated worth.

Implicit in BenthaWs prescription is the assumption thatthe total worth of a debision outcome is a monotonically increasingfunction of each of the factors which contribute. to the worth, andthat the monotone character of this relationship for any givenfactor is independent of the values of the other factors. Yntemaand Torgprson (1961) have suggested that there are probably manypractical choice situations in which this is evalid assumption.For example, the worth of a vocational choice probably increasesmonotonically with the attractiveness to the individual Of thework involved, whatever the status of the other factors to be con-,sidered. Yntema and Torgerson present some data that suggestthat when this is,the case, the decision maker's choice behaviorcan often be matched, if not improved upon, by a selection algo-rithm that takes account only of how worth relates to each of thefactors individually and ignores the ways in which the factorsinteract. To develop such an algorithm it is necessary only todetermine how worth varies' with the individual factorg. Severalways of making.this determination'are suggested. An importantpoint for our Npurposes is that the relationships of interest maybe inferred from the behavior of the decision maker when confrontedwith'the task of choosing between pairs of hypothetical alternativesselected to represent specific (in partjcular, extreme) combina-tions of 'the relevant factors,.

4%

Dawes and Corrigari (1974) have recently taken an even strongef'positiOn with respect to thekpracticality and, the validity of simplelinear decision algorithms in'a wide variety'of choice situations..They have shown that if, each of the factors contributing to theworth of a decision outcome has a conditiOnally monotone* relation -ship, to that worth, and the measUretent of these factors is subjectto error, then not only are decisions that are based on weightedliner combinations of the factors likely to be better than thosemade by human decision makers, but in some cases this is true evenif the weights are egual'or randomly chosen. Data from severalstudies of judgmental and choice behavior are reviewed in supportof this conclusion. Of the situations neviewed by Dawes and Cor-rigan, the only ones in which a linear weighting algorithm didmore poorly than a iliman decision maker were those in, which thehuman's judgment= was based on information not taken into accountby the algorithm.

*A conditionally Monotone relationship is one that is monotone, orcan be made monotone by a scaling transformation.

ISO

163

,to

ivAVTRADEQUIPCEN 71-C-012,8-1

9.4 The Problem of Identifying Worth Components

The implication is that if one can identify the factors "interms of which worth is determineA,, one frequently can improvesignificantly upon f jlumanudgmentJby application of a simple linear

'model. /the problem, according to this view, is not in the develop-.

ment of arcane mathematical decision algorithms,, or even -In theapplication of complex weighting functions to,a linear combinationrule, but that of identifying the dimensions of the choice spaceand of determining how these dimensions relate, individually, tothe worth of the possible decision outcomes.

The danger in this line of reasoning is that of assuming thatidentification of the factors in terms of which judgments are, or.should be, made is a trivial task. As we have already suggested,such an assumption is almost certainly false fbr many, if not most,real-life decision situations. Most people can proba.ly recallchoices that they have made which they realize in retro-pect weremade without consideration of some factor that they would haverecognized as relevant and important if only they had thought tothink of it. An individual buys a house, fon example, and realizestoo late that he failed to determine whether the cellar leaks.Had the question occurred to ;him, he would have recognized it nonly as a relevant consideration but as one that would have figureheavily in his assessment of the relative worths of candidate pur-chases. A potentially important aid to a decision maker would bea procedure that would facilitate the identification of the dimen-sions of his choice space. Having determined the factors uponwhich the relevant worths of possible choices depend, and how. thesefactors relate functionally to worth, a simple linear model of thetype espoused by Yntema and Torgerson might then. be used to inferthe decisionhimaker's behavior'in a choice situation. The experi-mental results reviewed by Dawes and Corrigan suggest that such amodel m' t even be used in place of the decision maker to effectthe choite.

9.5 Studies of Choice Behavior

In using the choices of a human as the standard against whichto compare the performance of a model, one is assuming that humansbehave in at least a consistent, if not an optimal, fashion. Onlyrecently has the assumption that decision makers are able to makeconsistent choices amoig alternatives that differ on many d imen-sions without recourse to formal analytical procedures been tested.

dlovic and Lichtenstein (1971) have reviewed several approachesthat have been taken to the problem of describing how people do infact make such choices. They divide these approaches into twomajor categories: those that make use of correlational or regreS-sion analysis or the closely related analysis of variance, Tid those

151

- 164

k

NAVTRAEQUIpCEN 73-C-0128-1

that make use of Bayes theorem. Among the nonBayesian approaches,..,that are reviewed are the correlation model of Hoffman (1960; 19681/,the lens model of Brunswik (1952, 1956), the integration theory ofAnderson (1968, 19'69), and the theory of conjoint measurement ofLuce and Tukey (1964).and Krantz and Tversky (1971). The objectivein all of this work is to discover and describe how a human "judge"combines information concerning different attributes of a choicealternative to arrive at a judgment of its Overall desirabilityrelative to the other alternatives iartiong which a choice is to bemade.

The results of many of the studies reviewed by Slovic andLichtenstein(1971) suggest that, although people can make "wholisticevaluations" (Fischer, 1972), they tend to focus their considera-tions on less:than the full set of dimensions, and, as a conse-quence, frequently ignore potentially important information. Also,there appears to be a degree of xandom error in the evaluationprocess which increases as the decision maker attempts tc, considerincreasing numbers of relevant attributes (Hayes, 1964; Kanarick,Huntington, & Petersen, 1969; Rigney & Debow, 1966).

On the igasis of results obtained in his study of job-seekingbehavior, Soelberg (1967) challenged the idea that people generallydo make choices in accordance with worth-calculation models inreal-world situations. In his words, "The decision maker believesa priori that he will make his decision by weighting all relevant,factors with respect to each alternative, and then 'add up num-bers' in order to identify the best one. In fact, he does notgenerally do this, and if he does,-it is done after he has madean 'implicit' selection'among alternatives" (p. 28)). Soelbergdraws a number of other conclusions from his study which; in theaggregate, seem to suggest that much of the effort that goes intodecision

,making is calculated to rationalize--rather-than.arrive

at--a choice. It's as though the decision maker were in cahootswith himself to deceive himself into perceiving his choices aswell- founded when in fact the real basis for them may be unknown.

9.6 Procedures for Specifying Worth

Obviously people can--people do--make choices among multi-dimensional stimuli; the results mentioned above suggest, however,that our ability to handle many dimensions simultaneously in aconsistent and reliable way without the aid of a formal procedureis somewhat limited. Given that the problem seems to be one ofexceeding man's ability to process information, it is not sur-prising that some of the solutions that have been proposed takethe form of ways of restructuring unmanageable problems So as tomake them into problems of simpler proportions. Such proceduresare sometimes referred to as decomposition procedures becausethey divide the task into subtasks that presumably are within the

152


decisio maker s information-processing capabilities. The solutionsto the ubtasks are then used as a basis for inducing a solution tothe original pr blem.

These prc res typically involve a number of steps (e.g.,Fischer, 1972) uch as specifying the alternatives to be compared,specifying the 4imensions or factors with respect to which thealternatives are to be compared, assessing the worth of each alter-native with respect to each dimension,'and combining the results ofthe dimension-by-dimension assessments into some overall indicantof worth for each alternative. The first of these steps has notbeen a focus of attention in studies of preference specification;the alternatives usually are provided. In the real world, identi-fying these alternatives can be a nontrivial problem, but it isperhaps better thought of as a problem of information gatheringthan one of specifying preferences. The second step llso hasipotreceived much research attention.

ti

A great deal of attention has been given to the third- of thesteps mentioned by Fischer (e.g., Becker & McClintock, 1967; Coombs,1967;-Fischer& Peterson, 1972; Fishburn, ]!967; Hammond, 1967; Huber,Sahney, & Ford, 1969; Luce & Tukey, 1964;/MacCrimmon, i1968;. Miller,,Kaplan, & Edwards, 1967; Raiffa,' 1968). NumerouS techniques havebeen proposed and studied for assessing the worths of alternativeswith respect to individual dimensions or factors. These techniquesrange from simple, qualitative pair - comparison procedures thatyield ordinally scaled preferences to relatively complex methodsfor deriving ratio scales for interdependent factors.

MacCrimmon (1968) has reviewed several prescriptive techniquesfor choosing among alternatives that differ withrespect to multiple.factors. The techniques that he considers are discussed under thefollowing rubrics: (1) dominance, (2) satisficing, (3) maximin, (4)maximax, (5) lexicography, (6) additive weighting, (7) effectivenessindex, (8) utility theory, ('9) tradeoffs, and (10) nonmetric Scaling.In each case, he describes the necessary assumptions and informationrequirements, and presents a formal mathematipal representation.ofthe optimal (or best) choice defined by the technique. Considera-tion is also given to the possibi4ty,of using several methods, incombination on a given choice problem, as suggested earlier byPinkel (1967).

A more recent review of worth-assessment techniques has beenprepared by Kneppreth,.Gustafson, Leifer, and Johnson (1574), Inthis review, methods are classified in terms of five properties:.(1) whether probabilities are used, (2) what kind of judgment isrequired (e.g., simple preference, numerical assignment), (3) num-ber of factors involved in a single judgment, (4) whether appropriatefor continuous or discrete factors,,,and (5) nature of output,pro-duced (e.g., ranking of worth, quantitative indicant of worth).

153

166

]


-4

Especially helpful features of this review are explicit discussionsof what the authors seetas the primary advantages and disadvantagesassociated with each of the meth-J*6s described, and the provision of

'references for the theoretical bases of these techniques. Of par-ticular relevance to this report is the stress that Krieppreth, et al.put on the need for training before some of these techniques can beused effectiValy.

The fourth step mentioned by Fischer--that of combining theresults of factor-by-factor assessments into overall worth esti-mates--has proven not to be a difficult one in many practical situa-tions because of the fact that a simple'linear combination ruleseems to work remarkably well in so many cases (see Section 9.3).

Prescriptive techniques for preference specification, or worthassessment, are of considerable interest because of the potentialthat they represent for procedurizing--and thereby, hopefully,simplifying--the solutions for complex choice problems. A lesstangible but perhaps no less important benefit 'that can result fromattempts to apply such prescriptive techniques in real-world situa-tions steps from the fact that these procedures,force the decisionmaker to be explicit concerning his own value system as it relatesto the problem at hand. This fact has obvious ramifications vis-a-vis the problem of evaluating the performance of decision makerswho make choices that affect the lives of others; one clearly wantsto''know, in such cases, not only what the choices are; but thebase3 on which they are made. .Being forced to be explicit concern-ing the factors that determine his choice nd the relative importancettat.he attaches to each of'tthem may be a revealing to the de-cision maker himself as to an independent observer.

9.7 Preferences among Gambles

So far, we have considered only the problem of specifyingpreferences among stimuli, that differ perhaps in many, but in known,ways. In this case the decision maker knows what the effect of anychoice that he may make will be. Another type of preference speci-ficatiori that has been studied involves preferences among gambles,or between gambles and "sure things," The general procedure insuch studies is to present the decision maker with a choice, eitherbetween two wagers, or, more typically, between a wager and a surething, and then to adjust either the possible outcomes of thewager(s) or the probabilities of these outcomes until the decisionmaker is indifferent to the alternatives from which he must choose.By repeating this process a number of times with different wagers,one can generate the kind of data from which worth functions canbe inferred.

154

i&7


Typically,'-the wagers that have been used in these studiesare such that one of the possible outcomes,is more desirable thanthe other, and the probability of the less desirable outcome isunity minus the probability of the more desirable one. Slovic,(1967, 1969), however, has studied preference behavior in so-calledduplex gambles in which the probabilities of "winning" or "losing"can be varied independently of respective payoffs. In this situa-tion, the decision 15aker can win and not lose, lose and not win,win and lose, or neither win nor lose. As Slovic points out, "Itcan be argued that this type of gamble is as faithful an abstrac-tion of real-life decision situations as its more commonly studiedcounterpart in which the probability of-losing is equal to unityminus the probability of winning (p.r. = 1-p.). For example, phechoice of a particular-job might offer soma probability (pw) of apromotion and some probability (p L) of a transfer to an undesirablelocation, and it is possible that one of these event-. both of 4

them, or .neither of them, will occur" (p.223).

In the first of Slovic's studies, two differerit methods ofindicating the attractiveness/unattractiveness of a' wager wereexplored. One method required the subjects to rate strength ofpreference directly on a scale ranging from 4-5 (strong preferencefor playing) to -5 (strong preference for not playing). The secondrequired the subject to equate the attractiveness of this gamblewith an amount of monkey such that he would be indifferent to play-ing the gamble or receiving the stated amount. One third of thesubjects assigned to the second method were required to state thelargest amount they would be willing to pay the experimenter inorder to play each bet, and, for an undesirable bet, the smallestamount the experimenter would have to pay thembefore they wouldp1ay.it. Another third of the subjects were given ownership of aticket for each gamble and required to state the least amount ofmoney for which they would sell the ticket. The subjects in thefinal third were required to state a fair price for a given gamblein the absence of information as to whether they or the experimenterowned the right to play it.

Slovic demonstrated that subjects did not weight the riskdimensions in the same way when bidding as when rating. Variationin the ratings was influenced primarily by variation in-probabilityof winning (p )r while variation in bidding was influenced primarilyby variation Yn probability of loping (p L). Also, payoff.dimen-sions--dollars won ($W) and dollars lost ($L) produced more effecton bids than on ratings, while probability dimensions producedmore effect on ratings than on bids. Finally, it was found thatwhen a person in the bidding group considered a bet to be attractive,his judgment of its degree of attractiveness was determined. pri-marily by the amount ($W); when he disliked a bet, the primarydeterminant of the degree of dislike was ($L). This finding hasparticularly important methodological implications, because, asSlovic points out, no existing prescriptive theory of decision

b:t55


making woul4lconsider that response mode should be a determinant ofthe way in which decision makers utilize probabilities and payoffsin making decisions, under risk, and he argues that behavior in suchcircumstances may be strongly influenced by information-processingconsiderations.

9.8' Preference Specification and Training

On first thought, preference specification--among all thetasks associated with decision making--might appear to pose theleast challenge for training research. One might. assumethat if there is an aspect of decision making that comesnaturally, it should be that of sayihg what one's preferences are.Things clearly are not that, simple, however, and the evidence isabundant that people do not always know what their'preferences are,or at least howto specirg-them in an unambiguous and consistentway.

The research reviewed in this report suggests at least fourproblems that relate to training and preference specification.First is the question of how to train people to make judgments ofsubjective probability that are independent of the worths of pos-sible decision outcomes, as the use of subjective expected utilitymodels requires (see Section 2.2. A second and closely relatedquestion is that of how to train people to make worth judgmentsthat are invariant across different measuring techniques.

The development of decomposition methods has been motivatedby an interest in simplifying the process of making preferences,and their bases, explicit. As Kneppreth, Gustafson, Leifer, andJohnson (1974) have pointed out, however, some of these procedures,particularly those that yield the most quantitative results, areworkable only with relatively sophisticated users. A third challengefor training research, therefore, is to develop methods for pro-viding the necessary training in cost-effective ways.

A fourth problem relates to two aspects of decision making,preference specification and information gathering. In laboratorystudies of choice, the dimensions in terms of which preferencesare to be specified typically are given. In real-world situations,however, the dimensions of choice are often determined by thedecision maker himself; in other words, the factors that are con-sidered in attenipting'to assess the relative merits of the choicealternatives are those that the decision maker happens to thinkabout. Surprisingly little attention has been given by researchersto the' question of how capable people are at enumerating on demandthe factors that they would consider important in any particularchoice situation. It is not even clear whether, when provided witha list of such factors, one can say with confidence whether thelist is complete. Much more research is needed, both to determine

156

169

e


human limitations and performance characteristics in this regard,and to explore how training might improve one's ability.to make__one's worth space explicit vis-a-vis specific,choice problems.

157

170

NAVTRAEQUIPCEN 73-C-0128,-1

SECTION ,X

ACTION SELECTION

Selection, or choice, is often thought of as representing theessence of decision making. And obviously, if one has no options,then he has no decisions to make. Paradoxically, however, the actof choosing per se as the least interesting of the aspects ofdecision making that are considered in this report. This is be-cause of the fact that when the other aspects have been realazed----when information has been obtained, the decision space structured,hypotheses generated and evaluated, and preferences stated--thechoice may, in effect, have been kaetermined. This is, of course,as it should be. One's goal in all of these activities is to remove,insofar as possible, doubt about what the choice should be.

In spite of his best' efforts to reduce, uncertainty to a minimum,and thereby to discover what his decision ought to,be, however, thedecision maker may, on occasion, feel very much "left to his owndevices" when forced to make a choice, Ellsberg (1961) rathergraphically' described the sense of frustration that one can feelwhen he faces his moment of truth and is not entirely convinced ofthe adequacy of the basis on which the choice will have to be made."(This) judgment of the ambiguity of one's information of the over-all credibility of one's composite estimates, of ones confidencein them, cannot be expressed in terms of relative likelihoods orevents (if it could, it would simply affect the final, compoundprobabilities). Any scrap of evidence bearing on relative likeli-hood should already be represented in those estimates. But havingexploited knowledge, guess, rumor, assumption, advice, to arriveat a final judgment that one event is more likely than another orthat they are equally likely, one can still stand back from thisprocess and ask: 'How much, in the end, is all this worth? Howmuch do I really know about the problem? How firm a basis forchoice, for appropriate decision and action, do.I have'?' Theanswer, "I don't know very much, and I can't rely on that,: maysound rather familiar, even in connection with markedly unequalestimates 'of relative likelihood. If 'completOignorance' is rareor non-existent, 'considerable' ignorance is surely not" (pp. 20,21).*

Most of the decision situations that we have considered inthis report involve the problem of choosing one from among several

*This statement IA contained within a larger discussion of circum-stances in which it May be "sensible" to act in conflict with theprescription of the Savage (1954) axioms (see Section 2.2). Thereader is referred to the full discussion for an interesting anal-ysis of the problem of ambiguity in choice behavior.


courses of action. It is important to note, however, that peoplesometimes find themselves faced with the task of deciding not whatto do, but when to do it:, The required action may be dictatedby circumstances, or predetermined in one way or' another, butthe individual is left with the job of deciding on the best timeto act. This type of decision problem is nicely illustrated bythe following situation.

Consider a pistol duel in which the duelists are instructedto turn to face each other on signal and to fire one shot at will.Suppose that once the men have faced each other, each may walktoward the other, reducing the distance between them if he wishes.We may assume.that the accuracy of each duelist improves, althoughnot necessarily at the same rate, as the distance between them'decreases. Clearly, each man faces a dilemma: every second thathe. delays firing in crder to decrease the distance between him andhis opponent and to increase his chances of an, accur,te .shot, healso increases the chances of success'for hisopponent; on theother hand, if he fires too soon; he risks missing, in which casehis opponent is free to advance on him until his shot will be cer-tain to findits mark.

This type of situation is representative of what Sidorsky,Hougeman, and Ferguson (.1964) have characterized as "implementation -type decision tasks." In.S4dorsky's experiments the duelists weresimulated navy tactical units, bttt the problem was essentially thesame as that of the individual antagonists. The decision makerhad to decide when to fire a missile, knowing that bo.L the proba-bility of hitting his opponent and the probability of being hit byhim were increasing (but at different rates) in time.

A particularly interesting result from this work is the find-ring that subjects urformed less appropriately when operating ata disadvantage than when operating at an advantage. One of theconclusions that Sidorsky and his colleagues drew from the resultsof a seriestof studies (Sidorsky & Houseman, 1966; Sidorsky, HoAse-man, & Ferguson, 1964; Sidorsky & Simoneau, 1970) was that "the1inability to analyze and respond appropriately in disadvantageous

0 .

situations iS a major cause of poor performance in tactical de-cision making" (Sidorsky &'Simoneau, 1970, p. 57). If this obser-vation is.generally valid, its implications for tactical decisionmaking are clearly very significant. The implications for trainingare also apparent, namely, the need for extensive decisidn-makingexperience in disadvantageous situations.

159

172

a

'"NAVTRAEQUIPCEN 73- C- 0128 -1

SECTION XI

DECISION EVALUATION

The problem of evaluating the performance of decision makersis a difficult one and it is critically important to the task oftraining. Without an evaluation scheme, there is no way of as-certaining whether training has resulted in an improvement indecision-making performance. Trainirlq assessment is not the only

ireason for an interest in evaluation of decision-making performance,howeller. Anyone who finds himself in a position of having to passjudgment on the performance of a decision maker is in need of aset of criteria in terms of which that judgment can be made. More-over, a decision maker himself might wish to evaluate a particulardecision that he has made in terms of a set of objective` criteria.

Unfortunately, a completely satisfactory set of jectivecriteria against which performance can be compared has not beendeveloped. As Kanarick (1969) has pointed out, "unlike otherbehaviors, there is no standard dependent variable, such as time-on-target, trials to criterion, or percent correct." One can, of

, course, choose for study in the laboratory only tasks for whichloerforthance can be objectively evaluated (e.g., probability esti-m4tion for frequentistic events); however, one runs the risk ofthereby excluding from study a large percentage of the problems ofinterest. Certainly, in most real-life decision situations inwhich the objectives are comprext the stakes are real, and theinformation is incomplete, evaluation is an extremely difficulttask.

11.1 Effectiveness versus Logical Soundness

Of central importance to a discussion of evaluation of de-cision making is the distinction between effectiveness and logicalsoundness. Failure to make this distinction sharply--sometimesto make it at all--has resulted in much confusion in the litera-ture. Effectiveness and logical soundness are quite differentthings. One might be willingto assume that logically sound de-cisions will, on the average, tend to be more effective thandecisions that are not logically sound. However, the assumptionthat the correspondence will necessarily hold in any particularinstance is manifestly not valid.

A decision is effective to the extent that the result towhich it leads is one which the decision maker desires. Effective-ness usually is easily determined after the fact. The logicalsoundness of.a decision depends on the extent to which the de-cision maker's choice'of action is consistent with the informationavailable.to him at the time the decision was made, and with thedecision maker's own preferences and goals. That these are quitedifferent factors is clear from a simple example. Suppose that

160

1.73

4


one it given the option of betting $5 against $20 that the nextroll of a fair die will come up 6, or betting $10 against $12 thatthe up face on the next roll will have an odd number of dots. Ifhe elects to make the first bet and the roll produces a 6, we wouldsay that the decision was an effective one. However, whether itcould be considered a logically sound one would depend on what thedecision maker's objectives were. If his intent was to maximizehis potential gain, or to minimize his potential loss, the decisionwas sound. If his intent was to maximize his expected gain, itwas not.

Decision-making behavior,,should be evaluated in terms of itslogical defensibility and not in terms of its effectiveness, inas-much as effectiveness is found to be determined in part by factorsbeyond a decision maker's control, and usually beyond his knowledgeas well.* It often appears not to work this way in practice, how-ever. Evaluation of decisions in terms of their outc...mes seemsto be the rule, for example, in the world of finance and business.Investment counselors are hired and fired on the basis of the con-sequences of their portfolio recommendations,.and corporate manage-ments are frequently juggled as a result of unsatisfactory profitand loss statements. Although the cliche "it's the results thatcount" has particularly strong intuitive appeal in this context,decision outcome is no more justified as the basis for evaluationof decision making in the financial world than in any other. AsKrolak (1971) asserts in a discussion of portfolio managementevaluation: "The real question to be answered is how well did [I)do with the information, capital, strategy and ability to assumerisk as compared with others who might possess the same resources?"(g 235).

That decision-making performance in military-training situationsis not always evaluated in terms of its logicality, has been notedby Hammell and Mara (1970). In discussing some of the mission

*Commenting on Fuchida and Okumiya's account of the WWII Battle of

Midway, Admiral Spruance (1955) made the following interestingobservation: "In reading the account of what happened on 4 June,I am more tharr ever impressed with the part that good or bad for-`tune sometimes plays in tactical engagements. The authors give uscredit; where no credit is due, for being able to choose the exacttime for our attack on the Japanese carriers when they were at agreat disadvantage--flight decks full of aircraft fueled, armedand ready to go. All that I can &laim credit for, myself, is avery keen sense of the urgent need for surprise and a strongdesire to hit the enemy carriers with our full strength as earlyas we-could reach them."

161

NAVTRAEQUIPCEN 73 -C-0128-P

training that*is carried out in ASW tactical training programs,they point out that performance evaluation is based, in many in-stances, on the simple effectPfehess indicator of whether or notthe team scores A hit. If it does, performance is judged to beacceptable. Commenting on specific training exercises that theyobserved they note: "If a hit was made, regardless of circumstances,each team member's performance was usually considered good... Insome instances a hit was scored because the target would make apredetermined maneuver intd the path of a torpedo which had beenobviously fired in a wrong direction"(p. 9).

It is probably safe to assume that most people in decision-making positions are more likely to be rewarded, or censured, asthe case may be, on the b9is of the effectiveness of their de-cisions than on that of their logical quality. This is due inpart perhaps to the fact that society is far more interested inthe results produced by its decision makers than in the reasonsfor which decisions were made. It is undoubtedly W.so true, however,that-it is easier to determine the outcome of a decision than todetermine whether the decision was logically justified at the timethat itwas taken. One wonders how many heroes have been made, notin spite of, but because of, very pobr decisions which have had -

happy outcomes, and, conversely, how many "bumblers" owe theirreputations not to the illogicality of critical decisions theyhave made, but to fortuitous turns of events that have blessedsound choices with disastrous results.

We may note in passing that even if one wishes to evaluate adecision in terms of its effectiveness, rather than its logicalsoundness, the task may be less than straightforward. Miller andStarr (1969) make the point that decision objectives are not alwayssingular. Often, one is attempting to realize several objectivessimultaneously, and seldom is it possible to optimize with respectto all objectives at the same time. It is difficult in such casesto evaluate a decision outcome unless its implications with respectto all the objectives can be combined Into a single figure of merit.

One attempt to develop a procedure for combining performancescores on various decision-effectiveness criteria into a singlefigure of merit was made by Sidorsky (1972), and Sidorsky and hiscolleagues (1968, 1970). A set of operational criteria that wereintended to be used to evaluate the decision performance of amilitary tactical unit was identified as follows: spatial rela-tionsh' s (the spatial interface between own and enemy tacticalunits , self-concealment (the degree of success in keeping theene uninformed concerning,own unit), information generation(th degree of success in keeping informed concerning enemyuni , weapon utilization (destLoy or counterattack capability),and conservation of resources (adequacy of supplied). Suchcriteria have been used by Sidorsky to rate the quality of

1

162

NAVTRAEQUIPCEN 73-C-0128-1,

decisions made during experimental tactical exercises. ADecision Response Evaluation Matrix was developed which, whenusdd in conjunction with an algorithm for combining scores withrespect to all five operational criteria, permitted the qualityof a decision to be expressed as a single measure.

11.2 Evaluation Criteria

Granted that logical soundness is the appropriate basis onwhich to evaluate decisions, the problem then is to translatethat principle into a set of lective criteria against whichdecision-making performance can be judged. In view of the hugeliterature on decision making, surprisingly little attentionhas been given to this problem.

Sidorsky and his colleagues (1964, 1966; 1968, 1970) andHammell_andMara (1970) have suggested five behavioral factors interms of which an individual decision-maker's performam..e might

'be judged: stereotopy (the tendency of a decision maker to respondin an unnecessarily predictable way), perservation (the tendencyto persist when persistence is unwarranted)4, timeliness (the,extent to which the decision-maker's behavior is reasonable interms of the time constraints imposed by the situation), It*

completeness (the extent to which all available relevant informa-tion is used), and series consistency (the consistency of thedecision-maker's behavior within the context of a series ofinterrelated actions). The first two factors are liabilitiesfor a decision maker; the last three are assets. In contrastwith the operational criteria mentioned in the preceding section,these behavioral criteria are more concerned with the logicalityof'a decision than with its effectiveness.

The conceptualization of the decision-making process that hasprovided the structure of this report suggests a number of dimensionswith respect to which the quality of a decision-making activitymight be evaluated: 'the adequacy of the information- gatheringprocesp; the sensitivity of data evaluatibn; the appropriatenessof the structure that is given to a decision problem; the facilitywith which plausible hypotheses are generated; the optimality ofhypothesis evaluation; the sufficiency with which preferencesspecified; the completeness of the set of decision alternativthat is considered; the timeliness' of action selection and itconsistency with the decision maker's preferepces, objectives, andinfOrmation in hand. The development of techniques for assessingthee aspe-:ts of decision making quantitatively and unambiguouslyrepresents a challenge to investigators of decision-making behavior.

11.3 'A Methodological Problem

It is worth noting that to determine after a decision ryas beenmade whether its basis was logically sound may be a very difficult

163

176


task. People usually can give plausible reasons for choices theyhave made. One may be permitted a certain amount of skepticism,however, concerning whether reasons that are given after the factare the reasons that prevailed at the time of the making of thechoice (Soelberg (1967). This is not to suggest that people neces-

- sarily misrepresent the bases for their decisions intentionally.It seems nob unlikely, however, that we frequently convince our-selves, without being conscious of doing so, that choices havebeen determined by certain rational considerations, when in factthose considerations were discovered or invented only after thechoice was made. One might argue that even though the allegedbasis of adecision may not have been verbalized, or even consciouslyappreciated by the decision maker, it could still have been opera-tive at a subconscious level at decision time. But this is adifficult, if not impossible, point to confirm or invalidate ex-perimentally, and for that reason it is not a very useful hypothesis.Pascal (1910) expressed his skepticism concerning the credibilityof after-the-fact introspective explanations of behavior over threehundred years ago: "M. de Roannex said: 'Reasons come to me after-wards, but at first a thing pleases or shocks me without my know-ing the reason-, and yet it shocks me for the reason which I onlydiscover afterwards., But I believe, not that it shocked him forthe reasons which were found afterwards, but that these reasonswere only found because it shocks him" (p. 98).

164

177

NAVTRAEQUIPCENr--

SECTION' XII

SOME FURTHER COMMENTS ON TRAINING OF.DECISI'ON MAKERS

_ThroughoUt this report we have commentedical notions and research findihgs that haveto issues of training and training research..been made within the contexts4of the discussiopertain. It is not our purpose in this sectiomarine these comments, but rather to turn to stopics that have not been addressed elsewhere

1

n how the theoret-een reviewed relatehese comments haves.to which theyto review or stem-

me training-relatedthe, report.

12.1 Performance Deficiencies versus Pe mance Limitation4 _4,

Some investigators (Hammell & Mara, 1970) have advocated,(,the approach of identifyirig "behavioral deficiencies" and. Adeveloping training programs that are designed to amelioratethem. Similarly, Kanarick (1969) has suggested that Ona.componehaf a training program for decision makers= should be= that of makingithem, -aware of some of the common reasons for the making of poordeci'ions.

The term "deficiencies" has been used in ctwo ways in the ,

lite ature: to refer to stereotyped ways of behaving suboptimally,and to refer to basic human limitations. In what follows, we willrefer to the second type of "deficiencies" as limitatiohs., anduse the word -deficiency only to dencit2 suboptimal but presumably.correctable behaviors. An example of a behavioral deficiencywould be the tendency of humans to be overly conservative in theirapplication of probabilistic information to the evaluation of hy-potheses. A possible example of a limitation would beithe in-ability of most people to weigh more than some small number offactors, without some procedural help, in arriving at a preferenceamong choice alternatives.

The distinction etween deficiencies and limitations hasimportant implications for training. Deficiencies may be "trainedout"; basic limitations must be 'trained around."

The first problem in dealing faith either a putative deficiencyOr a limitation, however, is to verify that it indeed exists,. Itis obviously imperative, when a deficiency or limitation is iden-_tined by a single experimental study, that the finding be cor-roborated by further research. More istTportant, however, and more.-difficult, is the probJem of establishing that the conclusionsdrawn from experimental studies are valid beyond the laboratory.environments in which the results were obtained. It is exceed-ingly difficult to capture some of the aspectg of many real-worlddecision problems (e.g., very high stakes) in laboratory situa-tions. And what may constitute appropriate behavior in the onesituation may prove to be inappropriate in the other.

(4-

165

1'78


Assuming, however, that one-is able to identify some examplesof deficient behavior that5m5ear to be fairly universal amongdecision makers, the question is how to go about training them out.One obvious possibility is to expose trainees to decision-makingsituations in which a given deficiency is likely to show itselfif it is ever going to do so, and then provide the individual withsome immediate feedback concerning the appropriateness, of his be-havior. One would probably want to provide numerous opportunitiesfor the same deficiency to show itself in a variety of contexts,providing feedback to the trainee each time that the deficiency isdisplayed. Probably, too, feedback should be provided for sometime after performance has improved to the point that the deficienCyis no longer apparent.

When dealing with basic human limitations, the goal should beto 'educate the decision maker concerning what those limitationsare and to provide him with the means for working around them.For example, if it is the case that without the help of some ex-plicit procedure, a decision maker cannot effectively weigh morethan n variables in attempting to optimize his choice of an actionalternative, it may be futile to try to train him to.make effectiveuse of more variables; however, if that is the case, he should bemade aware of his limitation and be trained to perform within it.

Another approach to dealing with deficiencies.and limitations- -in addition to training--is that of providing the decision makerwith aide to facilitate variousiaspects of the decision process.Ae goals of training and of dec3ision aiding are not viewed by thewriters as mutually exclusive, but rather as complementary, ap-proaches to the improvement of decision making. Moreover, thefact that decision aids are being developed has implications fortraining, a point to which we will return in Section XIII.

12.2 Simulation as an A proach to Trainin

A common approach to the problem of training decision makersis that of simulation (Bellman, Clark, Malcolm, Craft, &Ricciardi, 1957; Cohen & Rehman, 1961). The idea is to placethe decision maker in contrived situations that are similarin certain critical respects with the decision-making situationsthat they are likely to encounter in the real-world. The approachhas been used in efforts to train business executives (Martin,1959) prospective hig'r-school principals (Alexander, 1967),iesearch and development project managers (Dillman & Cook, 1969),military strategists and Aacticians (Carr, Pyrwes, Bursky, Linzen,& Hull, 1970; Paxson, 1963), high-school history and scienceteachers (Abt, 1970), vocational-education leaders (Rice & Meckley,1970), and government planners (Abt, 1970).

166,

r2s

8


Most business colleges and graduate schools today make someuse of simulation and gaming' teChhiques to teach management anddecision-making skills. Also, as a result of early efforts by theAmerican Management Association to develop & decision-makingcourse, corporations such as General Electric, Pillsbury, Westing-house, and Standard Oil, of New Jersey have devised in-house trainingprograms that make use of simulation techniques.

Two different forms'of management-training games are discussedby Cohen and Rhenfian (1961) in their survey of the present and fu-ture roles of such games in education and research. The firstform--the "general-management" game--attempts to provide experiencein the making of business decisions at a top-executive level, whilethe second form--the "functional" business came-- focuses on specificdecision situations within a limited _functional level of the organ-ization. Because oftthe complexity of interactions among organi-zational entities and the multidimensionality of the (4,,licision envi-ronment simulated in the general-management games, the possibilityof defining and utilizing optimal strategies has not yet beendemonstrated. The functional gaMe situations,'.on the other hand,which are typically lower in complexity, allow for the specificationand application of optimal or "best" strategies.

A variety of views have been expressed concerning the strengthsand weaknesses of simulation as an approach to training. Kibbee(1959)-suggests the following advantages:

"1) It (simulation) can provide a dynamic opportunity forlearning such management skills as organization, planning,control, appraisal, and communication.

2) Simulation can provide an executive with an appreciationof overall company operations and the interaction between man,money and materials. It helps make a generalist out of aspecialist who has never had the opportunity of reviewing hisdecisions as they affect the organization as a whole.

3) Simulation can provide executives with practice, insightand improvement of their main function: making decisions.Faced with realistic decisions about typical business problems,they can experience years of business activity in a matter ofhours, in an environment similar to that they face in everydaylife.

4) Simulation can exhibit what Dr. Forrester of M.I.T.the 'dynamic, ever-changing forces which shape the destiny ofa company.' The general business principles that are illus-trated can be studied and understood by the participants"(p. 8).

167

1'6'0


Similar themes are expressed by Abt (1970) concerning the efficacyof management games:

"Games are effective teaching and training devices for stu-dents of all ages and in many situations becausd they are,highly motivating, and because they communicate very effi-ciently the concepts and facts of many subjects. They createdramatic representatives of the real problem being studied.The players assume' realistic roles, face problems, formulatestrategies, make decisions, and get fast feedback on theconsequences of their action.

In short, serious games offer .us a rich fie'd for a risk-free,active exploration\of serious intellectual and social problems"(p 13)

Simulation,- as a general approach to'training of decisionmakers is-not without its critics, however. Martin (1959), whogenerally endorses the approach, volunteers several caveats. Hepoints out, for example;, that many of the qualitative dimensionsof a situation, such as.personnel quality and morale in an organi-zation being modelled, are difficult to reflect in a game. Further,in order to make a game administratively manageable, it may benecessary to limit the degrees of freedom one has with respect toinnovation, which is an unfortunate constraint. Finally, he pointsout thatit is not always clear exactly what students are learningin a-simulation situation. ."There is no doubt that the simulationtechnique is a powerful teaching device, and therefore is poten-tially dangerous unless we are relatively, sure of what is beingtaught."

One wonders, in connection with the last point, if definitionof what should be taught and learned can really be expected priorto development of an adequate prescriptive theory of managementdecision making. Moreover, it seems clear that so long as decisionsare evaluated in terms of effectiveness rather than in terms oflogical soundness, the answer to the queStion of whether any train-ing program is teaching individuals to make optimal decisions will.remain a matter of conjecture. Apropos the point of .how to insurethat simulations have some realism, Freedy, May, Weisbrod, and4eltman (174) have proposed a technique for generating decision- -

task scenarios that utilize expert judgments concerning statevariables and transformations in much the same way that a Bayesianaggregator Would make use of expert judgments of conditional pro-babilities.

We would summarize our own attitude toward simulation trainingin the following way. The approach has many advantages. The stu-dent can be exposed to a variety of decision situations. Situa-tion parameters can be varied systematically, thus permitting the

168

4)


study of their effects on decision-making performance. The conse-quences of incorrect decisions are not catastrophic, as they couldbe in some real-life situations of interest. The student's per-formance can be evaluated and immediate feedback can be providedto him, thus, presumably, improving his chances of learning.

On the negative side of the ledger, there is first the diffi-culty of the task of deciding what aspects of a situation to simulate.Any simulition is a simplification, and if one wishes tc assuretransfer of what is learned in the simulated 'situation to real-lifesituations, it is imperative that the simulation preserve thoseaspects of the real-life situation that are relevant to the skillthat is being trained. Moreover, the difficulty of assuring theveridicality of a simulation is likely to increase greatly withthe complexity of the situation that is being simulated. 'Sedond,there is the problem of generality. Situations are specific. Onewant's the student to carry away from training se,ssions skills which,will be applicable in a variety of contexts. Simulation itselfdoes not guarantee that that will occur. In fact, one might guessthat there would be the danger of focusing on specific aspects ofparticular situations which could have a tendency to impair thelearning of general principles.

12.3 On the Idea of a General-Purpose Training System forDecision Makers

A training system for decision makers that has a reasonabledegree of general,ity is bound to be a relatively complex system.Moreover, given the current level of understanding of decisionprocesses, it is unlikely that anyone would be able to design asystem that would be certain to be satisfactory. The approachthat seems to us most likely to produce a useful system is anexplicitly evolutionary one, and one that involves potential usersof the system iu its development from the earliest stages. Whatone needs to do is build a working system that represents one'sbest guess concerning what capabilities such a system should have,and then elaborate, extend, and improve the system in accordancewith the insights that are gained through attempts to make use of

The idea that many complex systems are best developed throughan evolutionary process is not a new one. Benington (1964) hasargued strongly for such an approach in the development of command -and- control systems. Commenting on the fact that many systems be-come obsolete even before they are operational, he notes that "Theprincipal cause of this situation is the fact that until recentlythe proposed users of these systems did not take many interim stepsthat would have helped them; instead, they waited for the grandsolution. When the development of these command-and-control systemswas undertaken, it-was thought that the design team could analyzepresent operations, project changes over many years, design a systemfor the far-off future, and then implement. Now most agree thatthis process just won't work" (p. 16).

NAVTFAEQUIPCEN 73-C-0128-1

SECTION XIII

DECISION AIDS

The recognition thatwhether because of behavioral defi-ciencies or basic limitationsmen often do not perform optimallyas decision makers has motivated the development of numerous de-cision-aiding procedures and techniques. The existence of deci-,sion aids has two somewhat opposing implications for the trainingof decision makers: On the one hand, insofar as ah aid succeedsin simplifying or otherwise facilitating the performance of somespecific task, its existence may lessen the training demands vis-a-vis that tpsk; on the other hand, users of decision aids mustbe trained to use those aids. It does not follow from, the factthat some training may be required before an aid can be usedeffectively that the aid is therefore a failure; if a traineduser'of,an aid can make better decisions than a trained decisionmaker who does not use that aid, then the aid May be sa.Ld to bean effective one.-e,

Given the view of' decision making as comprised of a varietyof tasks,and processes, it seems reasonable to expect that initialdecision-aiding techniques will be more successfully applied tosome of these tasks than to others. The goal should be, not todeVelop the grand aid for the decision hake', .but, rather, todevelop Tiiariety of aids to facilitate performahce of the varioustasks. Together, a.group such aids might be thought of aS-a"decision support system" (Levit, Alden, Erickson,. & Heaton, 174Meadow & Ness, 1975; Morton,, 1973), but the iridividual aids, andnot the system,"are probably the more rasonabld'objectives towardwhich to work initially.

Another factor that some researchers have argued is highlyrelevant to the design of decision aids is that of individual dif-ferences. One group of investigators, for example, has character-ized "decision styles" in terms of three dimensions with respectto which individuals are assumed to vary: abstract-concrete,logical-intuitive, active-passive (Henke, Alden, & Levit, 1972;Levit, Alden, Erickson, & Heaton, 1974). All possible combinationsof the extremes of these dimensions are viewed as eight "puredecision styles" that and representative of the types of individual-ized apprbaches to decision making that decision-aiding systemsmust take into account. The point that these investigators makeis that decision aids or decision support complexes, should bedesigned with particular users, or user types, in mind. Systemsdesigned for one type of decision style, they claim, may degradethe performance of a user who operates according to a differentstyle.

170

i3

}4

NAVTRAQUIPCEN 7-CL0128-1

Decision aids run the.gamut from /the types of heuristicprinciples discussed by Polya.(1957) to explicit paper and pencilprocedures for working through some aspect of a decision problem,to interactive computer-based techniques. In this section, weConsider only a few of the many 4ids to decision making that havebeen developed. The intent is not to provide an exhaustive reviewbut a representative sampling of what has been done in this regard.

13.1 Linear Programming

Linear programming is a mathematical technique for determi-ning a set of deciSion* parameter values that maximizes or minimizesspecified functions within certain linear constraints. The tech-nique is particularly.uSeful in solving such problems as resourceallocation, production mix and ,industrial cost contn.l. It isbest illustrated by a simple example.

Suppose a manufacturer produces three products. We willdesignate the monthly quantities of these products as xl, x2 andx3. The ,products have different unit production costs, say,al, a2, and al, and different unit sale prices, say, b1, b2, and .

b3. To keep the illustration simple, we ignore the problem ofinventories. Raw material limitations restrict the number of unitsof products 1 and 3 that can be produced per month to cl and c3,respectively, The total number of man-hours available fo theproducer is n per month,. and it requires d1, d2, and d3 man-hours'to produce one unit of,products 1, 2, and 3, respectively. Theproblem is to determine the, number of units of each4pZoduct thatthe manuf.acturer should produce permonth,in ordet to maximizehis profit.

Linear programming is a technique for solving such problems,when solutions exist. The technique involves expressing the con-1straints'as a set of simultaneous linear equations, ana-ERn..?

1",

(

si


searching within the ranges of the values of the independentvariables that satisfy the equations for those values that op-timize the desired function. In thd case of our example, thefunction to be optimized in this case, maximized) would be theprofit function, i.e,

(b1-a

1)x

1+ .(b2 -a2)x2 + (b

3-a

35x

3.

When the problem involves only two or three decision vari-ab1es, a geometrical model of the situation can give the decitionmaker an intuitively meaningful representation of the significanceof the various factors and, in pa4ticular, of the sensitivitofthe decision outcome to a less than optimal selection of valuesfor the decision variables. When certain boundary conditions aremet, the set of parameter values that satisfies the 3inear con-straints within which the decision must be made is reprsentedby convex polygons or polyhedra (in the two- and three-variablecases, respectively), and the solution to the optimization probleminvariably is .(or at least contains) one of the fi?ure's vertices.The same principle holds in cases of more than three /ariables,but, of course, the geometrical model is no longer helpful.

One of the limitations of linear programming is the factthat it is applicable only to-situations in which the decisionseace has been fully represented numerically and the outcomes ofall of the admissible decisions are known. Another is the factthat it can be used only when the effects of the individual deci-sion variables combine in an additive (linear) fashion. One canimagine real-life decision situations in which the effect of achange in the value of one decision variable depends in some wayon the value of another variable. For example, how much impor-tince one Would attach to a difference in salary between two jabsmight depenl on whether the jobs also differed significantly interms of the extent to which they placed one's life in .danger.As has already been noted in SeCtion IX of this report, however,several investigators of decision making have argued that theassumption of additivity Appears to be a reasonable one in many,if not most, real-life situations. Probably the more difficultrequirement to satisfy in order to use linear programmingoech-niques is that of adequately struceuring the decision spa andquantifying the salient variables. When the necessary conditionscan be met, however, there can be no doubt of the effectivenessof the technique.

13.2 Decision Trees and Flow Diagrams

Sometimes it it possible to convert an apparently complexset of written or verbal instructions concerning a problem-solvingprocedure into a decision tree or flow diagram. When such a con-version can be accomplished, it is often found that the desired

172

185

NAVTRAEQUIPGEN 73-C-0128-1

procedure is more easily and efficiently followed with the aid ofthe diagram than with the original set of instructions (Blaines,1973; Raiffa,1968; Wason, 1968; Wright, 1971) .

The following distinction between decision trees and flowdiagrams is made by Triggs (1973): "A decision tree is an assemblyof individual patht in a-structure organized so that no path everreturns or proceeds to another part of the diagram. A decisionflow diagram may, on the other hand, contain paths that returnto early parts of the diagram or feed to other common elements.A decisibn flow diagram can be more operationally directive in itsstructure,, and less concerned with the explicit details of thedecision process. In a tree structure, at every node of the tree,the user of the diagram can exactly state by what set of chanceevents and decisions one arrived there. The flow diagram structureis not always 6rganized so that each such path can be uniquelyspecified" (p. 3).

The clarity and efficiency gained by representing proceduresrequiring sequentialAdecisiont in diagrammatic form have beenrecognized for some time. In, such fields, as computer programmingand systems analysis, graphic techniques-have been employed inthe teaching and Conduct of specific programming, debugging, main-tenance, and troubleshooting tasks. Only recently, however, haveformal attempts been made to assess the benefitsto be derived. Inan entertaining article by Davies (1970) the results of a relevantexperiment by B. N. Lewis are discussed. The latter investigatorpresented a series of six problems nVolving a tax regulation toeach of 60 subjects. One third of the subjects worked with theoriginal (prose) statement of the regulation, a second third workedwith a simplified (prose) statement, and the final third workedwith an algor±thmic ( decision tree) form. The mean time requiredby the original prose group to solve all six problems was 23.4minutes, compared.to 11.8 minutes required by the simplified prosegroup and 9.2 minutes required by the algorithm group. Mean errorsin problem solution followed a similar pattern: 29%, 10%, and 8%for the respective groups.

More recently, Blaiwes.(1973) compared ti-0 performance ofdecision makers who had been given instructions concerning the )construction and use of decision trees with that of decisionmakers who had hot been so instructed. Only one of the ten subjectsin the uninstructed group-gave evidence of using a decision-treeapproach to the 'solution of the four experimental tasks, whereasall -,,h of the instructed subjects used it. Subjects using thedecision-tree approach initially required more time thanuninstructed subjects; but their performance improved as theygained facility with the approach. Most importantly, subjectsin the instructed group performed at a higher level of

173ittol;

NAVTRAEQUIPCEN 73 -C-- 0128 --1 ,

accuracy than subjects in the uninstructed group. Although thepossible effects due to practice cannot be separated from thosedue to problem difficulty because of the.particular design usedby Blaines, we regard the experiment as a demonstration of theease with which the decision-tree.approach can be taught toindividuals who have not previously encountered it.

A review of numerous attempts to apply decision trees andflow diagrams, to the solution of decision problems (e.g., Baker,1967; Clarkson, 1963; Dutton & Starbuck, 1971; Horabin, 1972;Howard, Matheson, & North, 1972; Rousseau & Zambra, 1972; Tudden-ham, a968) has been prepared by Triggs (1973). He points outthat the degree to which such aids can be useful to a decisionmaker will depend o4 the nature of the problem that is faced.They tend to be most useful for situations that are easily struc-tured, perhaps by means of decomposition techniques advocatedby Raiffa (1968). Triggs cautions againat the temptation "to makea complex problem tractable by forcing it into a conceptual repre-sentation.witfi which one knows how 'to cope," at the.expense ofignoring or eliminating critical aspects of the real problem.He also points out that'the task of. imposing the type of structureon a decision picblem that is necessary if decision trees or flowdiagrams are to be used,to advantage, may be sufficiently time-consuming and expensive to assure Ets*impracticality in some dy-namic situations in which the time for analysis is limitee. More-0,117e, forcing the decision make to think about his problem interms of a specific structure may inhibit his use of cognitiveskills that he otherwise might bring to the task. Triggs concludes,

0 however, that on balance these cautions do not negate the efficacyof the approach. Citing Zadeh's (1973) work, he notes that "evenin systems that are too complex or too ill-defined to admit of

. precise guantitative analysis, 'fuzzy' algorithms and diagramshavethepotential of being useful to the human decision maker"(P. 171.

A lucid tutorial treatment of deciSion trees and their use' is presented by Peterson, Kelly, Barclay, Hazard, and Brown (1973)

in Chapters 2 and 3 of a Handbook for Decision Analysis. Thehandbook has been prepared for the express purpose of aiding theindividual who is faced with substantive decision problems toapply concepts and procedures of decislon theory to the solutionof those problems.

6

13.3 Delphi, an Aid to Group Decision Making

e

The decision maker of most prescriptive models of decisionmaking could be an individual, a committee, a corporation, or amachine, inasmuch as such mode,ls are concerned with the decisionmaking process and are indifferent to the nature of its embodiment.11Pst empirical stud'ies of decision making, however, have focused

174

187

1

NAVTRAEQUIkEN 73-C-0128-1

on the behavior of individuals. Relatively little attention hasbeen given to the question of how decisions are, or should be,made bytn-person groups. There are; of course, large liter-atures dealing with related topics such as the effects of grouporgargzation and communication channels on problem solving, andthe effects of grout) pressures on individual behavior.

One generalization that it seems safe to make is that thedecision-making perfdrmance of groups may bejnfluenced by anumber of factors that are not obviously telated.to decisionquality in any straightforward way. Especially'is this true-ulfengroup-members are required to resolve problems about which thereexist conflicting views. As Helmer (1967) puts it:

"Round-table discussions for such purposes have certain.psychological drawbacks in that the outcomg is apt to be acompromise between divergent views, arrived at ail too oftenunder the undue influence of certain fqtors inherent in thy'face-to-face situation. T1-4se may include such things asthe purely specious persuasion of others by the member withthe greatest supposed authority Jor even merely the loudestvoice, an unwillingness to abandon publicly expressedopinions, and the bandwagon effect of majority opinion" (p.19).

As one.leans of remedying these types of problems, and ofprovicling a rationale by which to combine "expert" opinions, theDelphi method, was created .(Brown, 1968; Dalkey & Helmer, 1963;Helmer, 1967; Rescher, 1969). This technique requires each memberof the group to write down his independent assessment of the prob-lem or solution under study. The set of assessments is thenrevealed to all members but without identification of which parti-cular assessment was made by which. member. The pros and cons ofeach response are then openly debated and each member files asecond assessment. Following n repetitions of this procedure,the median assessment is then adopted.

The Delphi procedure-is reputed to be usable:

"1) To detprmine what the operative values of a group are,what relative weight they have, what sorts of possible trade-offs obtain among them, and the like.

2) To explore the sphere of value crite-riologv, clarifyingby what criteria the values of a group come to be brought tobear upon actual c4hVs.

3) To discover divergences of value posture within a groupand the existence of subgroups with aberrant value structures.

4 175

1,

NAVTRAEQUIPCEN 73-C-0128.,1

4) To serve as a tool for seeking` out areas of valueconsensus--or agreement as to actions and preferences--that may exist even when there are conflicts of value.

5) To provide a tool for the third-party evaluation ofconflicts of interest.

6) To assess the correctness of value ascriptions to givengroups.

7) To assess the correctness of value judgments in the areaof 'Mans-values" (Rescher,-1969, p. 17).

The use of a modifi4d version of the Delphi technique isillustrated in a recent effort by 'O'Connor (1972) to apply expertjudgment to the scaling of water quality. The problem was toassess the quality of water to be used (1)' as a public Lupply,and (2) for the maintenance of a fishband wildlife population.

r.

Eight experts made iterative judgments as to the parameterstoincluded, the relative importance weights to be assigned, and the'rules for combination- of indices.' Good consensus was- obtainedwith respect to sets of judgment parameters and Combination rules,but there was considerable disagreement on weightings.. O'Connorfound, however, that this'disagreement was not critical in thedevelopment of the final indices.

An important,

feature of the Delphi technique is the fact thatit provides a means for,achieving group consensus wIhout the needfor the face-to-face discussion of issues'which typifies mostgroup problem-solving methods. This characteristic was exploitedin the O'Connor study, where the experts were geographically widelyseparated and were never in direct cOmmunication with eac,h other.

,13.4 Computer -Based Decision Aids y'

, ,

The potential, advantages to be gained from car/plying the /general computational capabilities of digital computers to deci-,ion problems have been recognized for some time. Severa writershave made very convino.i,h'' arguments to the effect that bo menand computers have something to offer to the decisionl-maki gprocess, and that the nled is for the development of (Recisionsystems that assure a symdlOtic coupling of the capabili ties ofman and machine (Briggs AfSchum, 1965; Wards, )965b; Li4klider,1961; Shuford, 1965;. Yntema & Klem, 19,6-

'Yntpma'& Torgerpon, 1961).

'1

t4.1It is not diffiCult t% imagineS,,,computer system being usedto aid a decision maker irk the perfoim-alice of essentially all ofthe aspects of decision making that we have considered in fore-going sections of this eport. Such a system might provide th,Fdec4ion maker wittl,A 4 to base of ,facts or obsehlations t. at


relevant to his decision problem. It could serve as an extensionsof his own memory by keeping a record of factors that he had in- 1.

dicated he ought to "keep in mind" in making a decision. It couldhelp him generate hypotheses, and to structure and present thedecis:;nn space. It could help him discover what his preferencesare and to express them in a quantitative way. It cold provide

'graphical representations of the deciSion situation. It might(assuming d.v4lid model of the decision problem) project the prob-able consequences of various action selections. It might serveas an interface between two Or more decision makers' collaboratingon the same problem and facilitate the application of group deci-sion, techniques. It could do whatever computation was required.It could prod the decision maker to consider aspects of the problemthat he otherwise plight overlook. It could suggest approaches orstrategies that have been found to be useful in similar problemsituations. It could make explicit to the decision maker (either

'byj0.nference or by questioning of the decision maker himself) someaspects of the .situation or the decision maker's thinking thatotherwise would only be implicit. And so on.

It is in fact So easy to imagine ways in which the computercould be,used as an aid for decision making that one can be seducedto Oinking that the implementation of such capabilities is astraightforward thing. In,some instances this is perhaps the case;in others, it,assuredly is not. The important point is, however,that computer -based decision alas are being, developed and quite

No trai 'ng program for Ocision makers can afford to ignore thissophist'` aced are likely to be operational in the near future.

fact. \

k1

Ina preceding section of this report some commen ts were'madeConcerning simulation as an-approach to training. Given the avail-ability of coniputer:systems to decision makers, another way thatsimulation may be used to advantage is as an operational decisionaid. In this case the effects, or probable effects, of selectingspeckiic action alternativgis can be explored by 'the decision' makerbefor he actually makes hjis choice (Ferguson & Jones, 1969). Theprojections or predictions of the aid will only be as good, ofcourse, as is the model, of the situation that produces them, andit is not necessaTily the case that-the use of such predictive ,

aids will invariably lead-to imioroved performance (Sidorsky & Mara,.1968Y. The potential for this type of simulation is, great, however,

and deServes-more attention that it'has rebspived to date. At thevery least, such an aid can be 'used to helpfd9;termine what ispo'ssib1 and what isinot, giving an accurate representation of thecurrent state of affairs. The point isillustrated by an experi-mental decision aid' designed to monitor andtti maritpetraffics(Elmal , Prywes, & Gu tafemo, 967.) . The system was com-posed ci A a for atte data base,_ a se of-, "worker programs" whi=choperate k) on the da base, and a quo y language which allowed the

0 NN.--P

177


user to interact with the ,data base on line. Information thatcould beextracted from thd data base on request included"'"(1)past, present, or future locations of ships, (2) the number, type,or names of ships in any geographic area of the North Atlanticat a past,, present, or future time, and (3) how far is a shipfrom some particular place'and if ordered to change course whencan it get there?" (p. 206). The system could provide informationon sets of. ships satisfying some class description; for example,

-it could provide the distances of all ships of a given type,, froma given destination,,and the time required to reach that detina-tion, assuming the necessary, change in course. The System 'illus-trates a nice allocation of function between man and Machine.The computer does the bookkeeping and arcthmeticr the man exer-ceses judgment and makes choice's._ Hopefully, the choices tpat._the man makes will be the better because of the bookkeeping and 1,

arithmetic that the machine does.I

Two of the more prominent problem areas for which computer-.based decision aids have been developed oar planned are medicineand military tactics.

13.4.1 Computer-Based Aids for Medical Decision Making

Among the' first investigators to attempt to apply moderndecision theory to medical decision making were Ledley and,Lusted(1959). During the subsequent fifteen years, many such applica-tions of decision theoretic techniques were proposed and tried;and within the past ten years, several experimental computer-basedsystems have been developed for the purpose of facilitatingvarious aspects of decision making in the medical context. Ap-plications that have been explored include initial patient inter-viewing ar4 symptom identification (Griest, Klein, & VanCura, 1973;Whitehead & Castleman, 1974), analysis organization and presenta-tion of'the results of laboratory tests (Button & Gambino, 1973),personality analysis (Kleinmuntz, 1968; Lusted, 1965), storage andretrieval of individual-patient data (Collen, 1970; Greene, 1969),on-derdand provision to practitiohers of clinical informationtSiegel CStrom, 1972), automated and computer-aided diagnosis ofmedical problems JCumberbatch & Heaps, 1973; Fisher, Fox, & Newman,1973; Fleiss, Spitler, Cohen, & Endicott, 1972; Gledhill, Mathews

',& Mackay, 1972; Horrocks & deDombal, 1973; Jacquez, 1972; Locwick,1965; Lusted, 1965; McGirr, 1969; Yeh, Betyae, & Hon, 1972).management and graphical representations of data to aid researchin pharmacology and medicinal chemistry' (Castleman, RusSell, Webb,Hollister, Siegel, Zdonik,,& Fram, 1974), modelling"of physiologicalsystems and exploration via simulation of the effects of alternativecourses of treatment (Seigel & Farrell, 1973), and training(Feurzeigc 1964; Feurzeig, Minter, Swets, & Breen, 1964).

178


The results of one recent study of computer-assisted diagnosisare particularly relevant to the question of when expert judgmentshould or should not be used in the decision process. Leaperj11972,1975) compared two methods of computer- a =ssisted diagnosis of dis-orders for which_ abdominal pain wasa priMary sympto, (e,g., appen-dicitis, diverticulitis, perforated ulcer). Comput r ided Bayesiandiagnose were petforme0 using estimates of-probab: 'es that wereeither (a) ingarred from frequency data collected frdm 600 patients'or, (b) produced' by,,a group of, clinicians. The diagnoes that resultedfrom, the computer-aided method that used"-ihe clinicians' proba- . .

bility' estimates were'marginally more accurate than those producedbyunaided clinicians (82% versus 80%). The method that made useof probabilities inferred from incidence data, however,savessig-..nificantly more accurate results (91%). A secondary result of '

this study that' is of some interest is the fact''thap mosf'ilinicians.

, insisted on retaining their own pr4bability estimates, even when.those estimates were greatly different from the,survey data and,r they had been informed of this; fact.

/.e.-7.

`N:s. These'regults strongly suggest that relative frequericy datashould 1.e used) s a basis for probability estimates in preferenceto expert opini.atns, if such date are available. The principle'should not be"applied,-of dcurse, withoutadue re4ard for such fac-

0 'tors as the size and repres ativenegg,of th6 samples from whiph/

the relative frequency data re obtained': As a general rule, themost defensible strategy inbestimating robabilities would seemto be: use expert judgments only if a ore objective method is notfeasible, as would be the case when es mating the probabilitiesof very low-frequency events or events that are not reasonablythought of as "frequentistic" in nature.

,

13.4.2 Computer-Based Aids for Tactical Decision Making)

Much has been written about the use of computer-based aids.7- to facilitate decision making in the context of tactical operations

Widen, Levit, & Henke, 1973; Baker, 1970; Bennett, Degah, & Spiegel,1964; Bowen, Feehrer, Nickerson, & Triggs, 1975; Bowen, Feehrer,Nickerson, Spooner, & Triggs, 1971; Bowen, Halpin, Long, 4,ukasiMull rkey, & Briggs, 1973; Freedy, Weisbrod, May, Schwartz, & Wett-\man, 973; Gagliardi,'Hussey, Kaplan, & Matten, 1965;.Hanes & Gebhard,1966; Levit, Alden, & Henke, 1973; Levit, Alden, Erickson, & Heaton,1974; Sidorsky & Simoneau, 1970). The extent to which such systemsandt-aid,:s have led to improved decision making is probably impdssibleto determine. Xt(is easy to be critical of this iaork, however,beCause-progresS has certainly not ple9a..spectacular. And it may

some of the decision-aidimg efforts have been poorlybe thcdrIceived.

But tacticar decision making is complicated and notb,,t

thoroughly understood. It is not surprising that there would bpsome false starts before significant progress i made on thisproblem. EVen faAse stares can provide usetul insights, into a

Y

f.

179

1,

s 'NAVTRAEQUIPCEN "73-:C-0128-1

. . .... ..problem, however; if pothing.more, tlhey should help to clarify ---what the Wimensionsof thq problem are and to provide some clues

}`° concerning tbe requirements for a solution. '.

-Bowen, Nickerson, Spooner, and-Triggse(1Z70) have despi.ibed- several computer-basbd system6 that have. been,. or, are being, de-

veloped by the military services td aid the decision- making processin tactical situations. Among the systems that were revIewed.are:

., the Army's ZactlicalOperations.Systems(TOS)--in particular, TOS-7th Army-,--and Tactical Fire-Direction System (TACFIRE), the AirForce and Marine COrps' Tactical Information Processing and Inter-

',-pretation System (TIPI),"the Air Torce'.24ntelIigence Data Hand-ling 'SystemifiDHS), and the Navy's integrated Optional Intel-ligence SysteM (IOIS) . These systems' are intended to, improve -

tactical deciSion making by.facilitating data managementand mani-puFation, message routing, display, generation, report preparation-fire control, plannThg, resource allocation, and other-tasks andfunctions that fall within the ,purview of tactical dpgrations.

There are two motivations for bringing siCh system into thetactical situation. One is to unburden the deSsiorf make; of tasksthat are just as well performed by machines,.and thereby make itposgible for him to devote more time to those aspects of decisionmaking'that require hanan judgment.and expertise. The other is

'to upgrade the quality and adequacy' of the ihfoxmation on.wh4ich'decisions are based. This involvesinot,only the problem ofpp3ces-sing and integrating large amounts of information, but also that.of packagingand presenting information in ways that ar. well-

.

:suited to the information-processing..capabilities of the human '

being who.must make use of it. Hbid effectively existing or con-teMplated.systems realize these objectives is difficult to,deter-mine with much prpciSion.

.It is not the purpose of this review to describe particular-

.; systems in detail. We will, however,..coOider briefly two systemsas illustrative of those that have been developed, one intendedfor/operational use, and one for {Ise as a. training instrument.

13.4.2.1 AESOP

An intensive program to develop an on-linsinformation:-,control.

system of'value to military decision makers i!nthe planning of-tactical and gtrhtegic resource allocations was begun by the MitreCorporation in 1964. On completion in 1969, the prototype, called"An Evolutionary System for On-line Plannifig.(AES0p) to emphasizeits.incremental approach to the gpneTation of-computer-based/management and planning assistance, made available to system usetsa range of techniques which could aid in such diverse activities

,as data acquisition, aggregation, plan assessment and reportpr,eparation, *

180

193

,

-/

'NAVTRAEQUIPCEN 73-C-0128-1

The AESOP,system.consIsts of4tWb major parts. One of .theseis a-set of capabilities for storing, modifying, retrieving, anddisplaying dat4, and for performing various sorts of symbolic and_arithmetic mfnipulations with the aid of a,flexible disPlay-oriented user language, a light pen, typewriter and,push-buttOns.Details olq these aspects are covered in a variety of program pub-lications, the most informative of which, are. Bennett, Hains, andSummers (1965) and Summers .and Bennett (1967). ,

The second part of the system consists of g'sset of simulatedstrategic and tactical military applications which provide a ' *

context for exercising the capabilities mento ed above. One ofthe moreNsigniflaNh,,t of these is that of a The al Air Control.Center (TAZt41 in which the resource allocation to kd of a FighterSectiop/Current Plans Division are simulated. Since this parti-cular application alsg served as a testbed for the formal.test andevaluation of AESOP principles*, it provides'tbe most -omprehensivepi,:ture of the strengthssand'weaknesses of the system. The.remain-der.df our current summary, will relate to this applic'ati'on and tothe results df evaluation studies. More detailed treatments ofthe simulation and evaluation can be found in Doughty (167),Doughty and Feehrer (1969), and Doughty, Feehrer, Bachand andGreen (1969).

IAs'simulated :1-1 the AESO program,,the basic task ofLa Fighter

Section revolves about the allo tion (on request by higher head-quarters) of tactical aircraft to each of three mission calegolies:(1) oh-call bloge air. support, (2) preplanned close air-support/

circumstances the total number of ready aircra t in near proximityand (3) preplanned counter -air And interdictis Under "normal"

% to prescribed,Earget areas is less'than,the slumber of aircraft ,.requested, soYthe planner is forced to make tradeoffs relating to

%' such factors as sortie rate, flying time, time over target, andprobable degree of target destruction. The cumulative consequencesof these tradeoffs are: (1) that some requeqts for support fail tobe satisfied atall, (2) some requests fail to be satisfied on atimely basis., and.(3)(,some requests, though satisfied on a timelybasis,lare not satisgied at the required level. .

csI,,n this contest, the tactical version of AESOP has two d'interrelated- goals: (1)* the elimination Of much of the laborivandinaccuracy assoia,ted with manual computation and display af'readyiresources, portie rates, flying times and weapons' effects, andwith the preparation Of formal orderq (Fragmentary Orders) tosquadrons implicated in a planned allocation, and (2) facilitationof the problein-dolving activity of decision makers, that is, ofIthejudicious selection of squadrons, aircraft tpes,"weapons categories,and so on. -

For purposed of evaluation, the actual resources allocationsproduced by planners using the AESOP system 'A-re compared with

14)*

4-

LS

L

f

.7

NAVTRAEQUIPCEN 13-C=012871

those .produced by,planners'using a sim ated version of the,stan-'dard system in an-integrated series of tactical exercises depictingthe military-maneuvers of loyalists and insurgents during a ten-day*limited war. Experimental seSt'ions began with briefings re-liting to orders' of battle, political and military activity, andJoint Task Force requests for support of loyalist objectives.AESOP and Manua. Planning teams -then adjourned to commence allo-cation activities i'n response to the simulated JTF requests. Theexperirttent ended each day with the (automated or manual) produc-tion of squadron Fragmentary Orders.

Each planner, whetheloperating the manual or AESOP system,ed, in his ,

ed in de-wAs required to. generate an allocation which represjudgment, .the best tradeoff among four criteria (liscreasing order of importance):

1. Sdsfaction of requested level of damage.

2. SatisfacticE of requested time over target

3.' N1i imization of use of recycled aircraft(i.e., of sortie rate)

. (

1.4. minimizatiOir of (total) flying time.f I ..1

The results Oi the evalu ation study contained few surprises.In those,aspects of planning activity'for which-AESOP providgd

,direct assistance, performance of those usirk the system was su-perior. In :those aspects ,,for which assistance was not rovided,'planners'in the two systems performed at approximately equal levelt.The net performance of AESOP planners was superior to, that of'manual planners with respect to plan quality Rd production ef-ficiency,'a findinethat must be assessed in light of the factthat the larger portion of 'the task was fairly routine and requiredlittle creative abpity.

I is important to note that the AESOP system provided hoformal procedural aids to the decision"'maker such as decision al-gqFithmp, linear prograwing solutions, etch. ,What benefits accruedto user of. the ,Sylstem duing the more creative phases of theirtask seemed to result from a combination of:..indirect factors. Itappeared to be the case, for example, that planners could moreeasily comprehend the4extent to:Which resources would be "strained"and, thefeby, develop a better "feel" for the nominal form of theirplan prior to its production. This appreciation for the difficultyof the prqblem with which they' were faced on a particular day wasmaterially aided by the concise nature of theJdisplays provlded

' by thepsysettl. Planners who used the system were in a muqii better,position to monitor their own progress while solving the problem'than were those whose appreciation, of the demands of the situationhad to be assembled from groups of formal documents.

)

I

182,

3

.

NAVTRAQUIPCEN 73-C-012a-1'

.

,.

It 'appeared also to be the case t, hat., since the system per::: 1.,

-formed routine aspects'automa i6ally, more time was-avaiil le fors/creative - activities and for rel. ration Of'plans. ,On severaloccasions AESOP planners attempted successfully to produce series.of,allocations of progressively greater meritand stopped only 1

when they were totally satisfied with their efforts*

,11

)3.4.2..2 TACTRAIN

;ThesTactical Training (ENCTRAIV),facility was developed...;1ythe Electric 5qat 5Eirsion of GeneralDynamics, partly as ademonstration that a modest computer with a CRT display couldbe employedin thtraining'of decision-makingskills and partlyes-an experimental tool for evaluation of alternative tacticaldisplay/interrogation formats. Details regarding computer anddisplay equipment, software and tactical prqblem parameters usedin the system and a*,specificconfiguration-employed iarheystemevaluation are discussed by Si4Orsky and Simoneau (1973). Thesummary' presentation below draws heavily on their discussion:

The TACTRAIN system provides an opportUnity for the deAsiOnmaker to take on the role of:a commanding officer of a submarineon an ASW search-avad-,destroy mission. His specific .task is tomaneuver- in such a way that, he siMbltaneously, maximizes the prob-ability of destroying a simulated enemy ship and minimizes theprqbability that the enemy'ship will destroy him. 'HeIp.hooses amaneuver by selecting a speed,' a depth, a firingirange, and aquantity of terpedoes, each froM amongaive alternatives. Thechoices are donstrained to be consistent with the operatingcharacteristicS of own and enemy ships, thd parameters ofav6ship's weapons complement, and specific souftchannel, topo-graphic and bathythermal conditions. The maneuver implied bythe alternatives that are chosen is then evaluated with respect,to each Of four Criter,ia: (1) the probability that own ship wouldbe able to detect'the enemy ship, (2) thp probability that theenemy ship would be able, to detect own ship, (3) the probabilitythat own ship would be .able tcmdestroy the enemy ship, gi.Oon themaneuver and weapon characteristics, and (4).the probability thatthe enemy..ship, would Wable to destroy own ship.

While solving,a particular tactical Problem, the officercan retrieve inforMation Oared in the systwiby interrogatingthe display with a light gen. Appropriate interrogations lead

.*A quasi-linwAs later deactivity.

ar program to aid this strategy at a.forinal level,

loped by Feehres, (1968) f'cir the AESOP viqc planriing,

.183

18; t

A

)k.

1

INAVTRAEQUIPttN 737C-0128-1

ti

. , , ' .

to one of two categories o displaw (1) prior to d "command de-.cLsion," graphic displays of the "tactical effectiveness" asso-ciated with theohoice of'a particular alternative on each tac- ''ticai,dimepsio (speed, range, etc.) with.respect to each of the

5K1a/21four ciitef.' -=available prior to a c and decision, and (?),alphanum c displays revealing the, outcome. of the,maneuver; the-number of (quality) points to be,assigned to the outcome, ,andthe Cumulative number of points acquired as of the end of the .experimental. trial i4 question--avaiiable following a commanddecision.cisibm.

,.

-

..

/ \ . ,

,Theideveiopers of TACTRAIN see it, .

making at least two valvgbiefnput.§ to the learning process:x4.the deci-sion maker. First,-it)rovic s immediateknowledge of. the conselquencg Ofs. a decision.The de

iision maker discovers very quickly whether he destroyed

the enemy ship and whether his own ship was destroyed in theiprocess. Moreover, he is provided with an a .ithmetic..leasure,however arbitrarily derived, of his pumulati e 'performance. #

'15.6:,.-"e ecis 4ion,maker is provided, viaithe-4SP1,with_a graphic* rtra:yal of the interactions bdtween tAFtical andenvironmental ariables and their re] ationship to tactical ef-fectiveness as relpresenedidetection/connter detection andhit/miss outcomes. 221d, inasmuch as_thd-tactical problem unfoldover time, the decision' maker also gains an appreciation for thechanging.cpmplexities of"these interaotions and for the needfor timeliness in hiss' decision.'

, ,

1-'3%4.3 Ccimputer-Based Decidon Aids and Training. . .

I , 4.

. -

It seems highly probable that Many attempts to develop com-,puter-based decision aids' wil`l fail in the sense that the aids,that are produced will not measurd up to the expectations oftheir developers,. This is `not neCessarilylailure in. a largerview, hbwever, if these attempts.leadto a better understanding.of the decision-making processas ond Might reasonably IThpe that .

they will. To the extent t'hat these efforts-,ao lead to newinsights into various aspects of theAecision-making process, theywill have direct impac on,training curricula.

4D..the extent tha't specific systems proye to be effective ,raids in operational situations, they will constitute new tc,,lslaith which decision makers wilq have to work. TIZus,/their.existence will represent a nertraining need,' namely the needto train the users of-these aids.

Perhaps the most challenging way in which the development of.

increasingly Sophisticated Computer-based systems relates totraining is An the potential that these 'systems represent forproviding training for their users. Critics of the'idea of

184

, 1.7

,

I

4AVTRAEQIPCEN 73-C-0128-1'

1

of1

computev-AsdAted instruction can correctly point out ,that".

results of endeavors in this area have not measured tw to title 'isexpectations that were fostered by many of the early enthusiasts,

. fO.,.r this use of computei. Very real progress in the area .is.being made, however, and it may proxie to be the case that the

'

early enthusiasts erred only in failing t2 appreciate the ' "difficulty of some of the probiems.that had to be solmed 'and thetime that would be equired.to solve them. There is no4Restionbut that. computer ustems that are intended to be used by people-interactively on_complex problem-solVing tasks can ,be given. the

$,capability-to providemuchfof the, trainirig..that-rE required, .

both to initiate users and.to-bring users froth nehyte to expert ystatus. The potentialfgains td berealized,by_bdilding such,/training capabilities into operational systems sugge6t that this

'possibility is worth far more attention than it has yet received,

N.1

a

0 0

44

A 1

r

8

. 185

198

0 00

10 .

..tA

(

eve$

ti I

,

NAVTRAEQUI4CEN1

t

ACKNOWIDGMENTS

6r

$

a

41

We wish ,td acknowledge , 'and express our appreciatiOri for,the assistance of severallpeople 41 this project.. Charlene- Long,Walter' Hawkins an'd George Luka$ helped to locate and screenarticles: Ann Rollins wrote the computer programs that wereused to generate the results reported in Section 8.6.. Michael.Samet made several useful comments on a draft'of the report.,Flckrence Maurer typed the anugcript and helped over the courseof the project 'in many ways. Sheila Allen put the reterenpeson a cotputeropfkle and thereby facilitated' the preparation pfthe bibliography:

. .

,Q

r'

I

,

r"

\

.

186

X99

t

z

N

I4

o ' J

. ..,K ,' / , A

% J. IA "o I . . *

esIS I a

o ta

It . .', ' ,NAVTRAEQUIPCEN 73LC-0128-1

4. C .r. .

Abt, C.D. Serious games.. New York .>

:Me Viking. Press, 1976..A

1t ,

4 or

Adkoff, R.L. Management misinfornati,an' Systems. Management

,.

-.Science, 1967, T4, b -147;156. ..

.:,c 1,

Aczel, --=-414& Pfanzagl, J. Remark's' on.....hed measurement Of

subjective probability ,and information. Metrika, 1966, 11,

- 81-105. .

1.

, .

O

A

o

Adelson, M: Human decisions -in command control .centers'. Annals

New York Academy of Science, 1961, 82, '726 -731.

beroni, F. Contribution to the study' Of ,.,''subjective

4probility. Part I.. Journal of General Psychology, A962,

66,.24.1=26,4:

Alden, 'D.,' Levit, R: Henke, A. Development of operational'

decision aids for. naval task force application. iloneywell TR

701-404,8., 1973. . I.

.,

'`'

.,'

Alxander, L.T. A demonsqation of the use of simulation,in thCP e-)

training a school admini trators.. Ney York: City University (

of New Yerk, Division of ear,:her Education, 1967, 114, AD -

014157..

Algea, -C.W. A development of a conceptual. framework of the

driving task. 'Human Factors, 1964, 6, 575-385.!--. I

inderson, N. H. A simple model(for infonmatibm Integra ion. In

R. P. 'Abelson, et . al (Esis.),- Theoies of \cognitive

.constancy: A source book: Chicago: Rand :McNally, 1968.

Anderson, N. i. Comment on an analysis -of- variance. model for the

assessment of configural --cue utilization in clinical

judgmedt. Psyb.hologv-Bulletin, 1.969,/2, 63-65.

Andrews, R1 S., Jr.' & ngel, S. Certitude' judgment, And

accuracy of .information assimilation from. visual displays.

U.S Army Personnel Research'Dffice ,Technical Research Note

No. 145,-1964

rmer, P. The use Qf computers. In E. S. Quade (Ed.), Analysis

for military decisions. Chicago: Rand-McNally, 1964.

Attneave, F. Psychological probability as, a function of

'.experienced frequency. Journal of Exieriiental Psychology,

1958, 46, 81-86,. .

*Bacon, Ft The new organon.. In . Dick '& Francis 5acon

(Eds.), Selected Writings, M n Library Edition, New York:

Random House, 1955.

.... 7

Z187

.

4

F

,

.t

,

. 4.-4)

.N.AVTRAEQUIPCEN 7\ 3-C-0128-1l

. ., ,

.:

er, F.B. The internal organization of computer models of

( ecignitive-behavidr: Behavioral Science, 1967, 12, 156-161..

Baker, J.D A technque for obtaining non-dichotomous .measuresof short term memory. Decision Sciences LaboratorY'Report,

' ESD-TR-64,6W4, 1968.. 4

'Baker, J.D. Tkie uncertain student and ' the I un erstandingcomputer. Paper presented at a -colloquium on ProgrammedLearning Research; Nice, Fratipe) 1968.

-*,;

Baker, J. D. Quantitative modelling of i -perfoymbnce in

information systems- Ergonomics, 19.70, 12, 645-664;.

Biker, J.D., McKendry, J.M', & Mace, D.J. 'CertiUlde judgments in. Sn operational enviato; ment. U.S. .Army B6havAbral Science

Research Laboratory, teihnical Research Note 2CO, November;

1968. ' .:, \1 -.-

,

An-L.exnex4imental arld 'sbpial vstudy.-

)

Bartlett, F0. ` ThinLondon: ,Allen' and

Beach , L.R SwensgonPsyqhonomic Science;

- f, 41

j IntUitive esti atioq of ,meahs.1966 '415, 161-162: ti

\pecker, G.M..- Sequential decisibn making:

r

-laid's' Model and

eStimates*Qf parameters. Journal of Experimental -Pychplogv,, 4

195B, 55: 628-636.-

Bepker,G.M:, Degroot, M.H.,'&Marscak, J. -Stochastic models of

choice behavior. BWavioral Sci nce, 19'63, 8, 41-55.I

Becker,, G.M., & McClintock, G.G: . V lue :. Behavioral deciapD5

theory. Annual Review Psychology, 1967, 18,., 239-286.

Be).1man, R,E., Clark, C.E.,

...Ricciardi, F.M.- 1.Qn ,the

lUltiperson business game.\ 469-503. f

D.G., Grafwt, D.J., &

construction of a multistage,'Operations Research, 1957; .5,

Bpnington, H.D., ionformatipn - recentl'y' LIT presently.ifn- E. Bennett, J. Degan, and J. Spiegel, (Eds.), Military.inTorma:4on systems. Praeger, 1964.'1

Bennett, E.,systems:New Xoilv

144

Degan, -J., f& Spliegqi, J. MilitAryThe design,' 24: computer -aided systems!raegpr, 1964C42:

Bennett, E. 11%, Haines, E. c., & Summersprototype fpr ,on-line user control.

'Proceedings:,'Fall Joint Oomputer Conferences

435-4554-

informationfor command.

J. K.', P:,$A-. FIPS Con erence'

:19651 , 22, (

188

201e- 1

J.*

.

v.

e

, a

...' NA713:AEQIPCEN 73-C-0128-i1 ,

.,

.

.Bentham, J. An introduction to the principles of moralS and .

. legislation,. (originally published in 1789). In E.A. Burtt r/(Ed.),' The english philosophers from Bacon to Mill. Modern:Library Edition, New York: Random House, 1939. , ,;' J'

; , .

. ,* 4-* ,.

Bernotali1, D.., 'Specimen theorise norae de mensui2' sortis.Comentarii Acede:oae Scientiarum Imperiales Petr6Politanae,17.3.8 5, 175-192, (TranSlated by L. Sommer:- in Econometrica,1954, 22, f3-26).

.; 1

'1 :'Ber f.fle, J.; WagneraY%f. H., &*Zannetot, Z. S. Theq'velopmentoof

1f

...t. aq4',inter,asti've /graphical risk analysis system: Mi?Ii T.-,

, .:, Slaari Schvql,of/Management, 1970,'502-570.,. i

4.121-4.t . 4., t, . / Li\ . 1 ..7.1, 1 *Bir sail; ,z. G.r.,' & Roberts, 'R... A. Thedry of signal

..

0,etecta ility:. Deferred-decision theory. TheJournar of the. c dstical Society of lmbrica, 1965, 31, 1064-10,74.. '

. F.. ,

,

f.

.. ....1 .,

BlaLell b., & Giraick;ir. 'Theory of sames and statistical

_

.

deeisibns. ;New tork:'Wiley-,;1954. ,

iaiwesll'A. . Some ttraining/ factors related to prOcedural

perlsrman e. Journal Applied Psychology, 1973, 51,

21A-218 -. :' .. p.i;. ,

.

Bpnd, N. IT., Jr. 'RigneS/, Bayesian aspects of

, shooting b avior.^.HUman'FacWA, 1966, 8,.377-385.

Boole 'G. A n investigatioh dt the raws o hought_. Macmillan,

184u .

t,-

trouble

owen, R. 4., Feeheer, C. Oick son, R. S., Oooner, R. L., A.. Triggs, T. J. Army ta. ticapintelligence: An examination, of

"ckiimputer-dhSed displays and a'seructured 'decision _procedure, as Aids to tWe intelligence analy b.. BBN Report 2073-tinderContract.DAHC-19-69-C-0026, it Army Behavior And SystemsReSearch Lab',' 197'3,

A ,

A. ) w. 4. q,A '

1

Bowen, R. J.", Feehrer, d. E.1, Nickerson,P43..". S.., &" Triggs, T. J:

Computer-baVe displays as aids slin'ehe production of army,

,.tactical .e'd.ritligence,. 'USA Resehrch Institute for e

Behavioral an Social' Sciences. Technical Paper 258, February1975. t

4 /.

X i

r *

Bowen, . 4J... Halpin, J: A., Long', C. A., Lukas, G.', Mullarkey,M. M:1' & Triggs,, T. J. Decision flow, diagrams and dataaggregation in army tactical intelligence. BBN Report # 2570

. ..to USA Reselarch, Institute for,,r&be Behayioral and SocielSciences, Jude 1973. (

.---. -'..-,

"B-6Wen,. R..0J. Ificketwon, R. -E.*, Spoosaer, R. L., and T., :.iggs, T. ;.

'Army: tactical intelligence: An investigation ,of current

methods of production and possible computer-basedp,7-11` .

*t. I .. t ,

4..

's) It A

I 18

2(32%MO

A

sI

.

/ cr

.

1

NAVTRAIQUIPOEU73-d:0128:1

innovations.: BBN- Re art No. t945, 16 Sept. 1970, to USA,Behavior and Sydtems esearch Laboratory under ContractDAHC-19-69-c0020; . P ' .

. . Brier, G.W", Verification o forecasts expressed in terms of A. probabilib. Monthly Weatherher Review, 1950, La, 1.2.3.,

%, .- , -

Briggs., G. E., , & r 'Schum,' D. A. Automated . Bayesian4 -, hypothesis- selection inia simulated theat-diagnosis system.

In J. gptegel& D. E., Walker. (Eds.),. Information systemssciences: Proceedings of the second congress. Washington,-ID . 9 .. Spartan -Books, 1965, 169-17'6.

: ..

\i

Brody, N..

.

-The effect Of commitment to correct and incOrPect.decisions on confi4ence in a sequenti'd1 decision task.American Journal of Psychology, 11965, il, 251-256.

z,,

uner, J. S. On perceptual :readiness. Psychological Review,. ...

perceptual

. .,

1957; 642123-152. i ,

% 4 . N 4 e ..

Brdner, Je S., Googinon J,J..., & Austin, G. A. A study of.thinking. -NN York: Idiley, 1956. *'

..

4

. ee

4 grown, 8: Delphi procbss;.,The Rand Corporatiod,p-3925, 1968.. 0

,. ,, 7. ...\--7-, 1. 1

'Brunswick, E. The conceptual framework of. psychology. Chicago.:

t 0, University of-Chicago Press,'1952. . -,: *

e,1. . ,

'Brunswick, 8, / Pericek ption and the, representative, design of

ex erime ts. ,Berke'ley: Uniwerpity of_California Press, 1956.,

, ,

)(1iP ..

$. .

But ton, K F., & -GambinoiJ S'. H. ' Laboratory 'diagnOsis, by0

'compute. Computers id'

3i'

qlogv and ftedidine, 1973; 3,

131-136. , ... ,,

,4 ., , . ,' . .

Carr, J. W., ,III,,Prywes, IT. S., BursWyP., 1.4.nzsn, S., & Lull,' ,B. Man,- computer j1roblem solving ,in real -time naval duels.

UniVersiV:yof Pennsylvania, Moore School Report No.% 71-08,

. -1970. .r-

../. .

Castleman,,P.pA., Russell C.*R.; Webb, F. ,...N....) Hollister, .,C. A.,,

Siege; 'J..R., Zdonik,l, S. R., & .Fraell D. Mt Title

implepientation, of tble PROPHET system. Proceedings !o.f..

. ,National Computer Conferen e, 1974.

Cienzo'ff, A:,- Cri.tten en,.R. .tFlores, I., & 'Tolcott -, M. A.. Human dpcision mpking-as.elted to airurveillance systems:

NTbchnical Report/d. 1, A surveSrpf literature ,,and currentresearch: , AFCCDD-W-60-25. June 10, 1960. Issued by Dunlap

..11

and Associates. .

Clarkson,.G. P.E. A modelI

of the trust investment process. In ):4

,

E, A. , Feigenbaum 1: 'J. Feldman (Eds.), ,Com.uters ana-.

190 10

,

A 2a3,

.k

41,

fi

%I I

1

NAVTRAEQUdIPGEN 73 -C -0128-;

thought. New ;ork: Mepraw-Hill, 1963.,

..s. f-COhen' K. JC., & Rhenman, E. The role of management games in, .

education and research; Management Sciences 196.1, .1;131-166. ,

..\

, , ti

.'

Cohen, 4., Chesni1c0 E. I., & Haran,-D-, -Confirmation ofinertial,.,.

- PSI effet in sequential chpica and decision'.. BritishJournal of Psychology, 1972, .3, 41-46; ,

j.i.t ,

Cohen, J, DearnaleY, E. J., A Hansel, C. 8. M.'*Rilk and .,hazard.'Oper. Research Quarterly, 1956, j, 3, 67-8a. % '

Cohen,:J.r, Dearnaley, 'E. J. lc Hansel, C. E. M. Skill withchance: 'Variations in 'stimates Of skill with'anincrea6ingelement of.chance. ,British Journal of Psychology, 1958,4( .112r.r31'9-323:

. , .

. MI.

Cohen; j., &- Hansen, C. E. M. Risk and gambling: The study. o'i' subjective probability. New York: Philosophical Librar

19564,-'' . t

....

.

.

Cohen, 'M. R.0 A preface to logic.- Hole, 1944:, t

/ I'.

....

C.

'Collen, M. F. (Ed.).Redical, InformatiOn Systems, Springfield,a.,, National Technicd1 information Services,. 1970.

Coombs, C.H.r,Inconsistency of preferences test 'cif unfolding. theory. In W. Edwards & A.. Tver'sky (Eds.), Decision .

making. Baltimore: Penguin, 1967.. ., ,,

y, .

Coombs, C. H., Bezembander, V. G., 'Goode, F.44.. .Testin,expectation theories of decision making without utility or -

subjective proba 'lity. Journal of Mathematical Esighologa, -.1*\ .

1967, 4, -72-1031 a

, : r

Coombs, C. H., Dawes, R. M., & Tver,sky, . MathematicalPsychology: An Elementary.' Introduction, New Jersey:Prenticb-Ha11, 1970 ... 1,, , e

.

systems-\

>r

loot, P. L., Management informtion syetems- Can computers help?1 The Computer-Bulletin, 1968, 11, 316,=?81. N

.. q

.Cumberbatch, J. & Heaps, H. S. Applic4ions 'of .a' non-Bayesian,approach to computer-aided diagnosis for upper abdominalpain. International Journal of Biomedical 'Computing, 973,4', 105-115.

, 4,6

Dale, -H. .C..A. 'Weighing evidence: An attempt to assess theefficiency of the human ogera on. Ergonomics; 1968, 11,-215-230.

ti

.

s191

2:01,

ti

4 .,

NAVTRAEQUIPCEN .0-0128-1

Dalkey, & Helmer, 0.*.' An experimental application of theDelphi method to the use PT experts. Management Science,1963, 4, 45,8-46.7.

' %t . D..- s- ,

amas, P., N, Goodmap, B. t., & Peterson, C.. R. Bayes theoreM:Response scales and feedbaek% ' University ofMidhigan project,

..number,NR 197.414, Sept. ^. 197a.- ...,t

.. , ..

.

Davidson, D., Suppes, 5., & Siegel, S. Decision making,: Alnexperimental Approach. Stapford: Stanford,University'Press,1957.

Ayres, I. Get imMediate relief With. an A),gorithm., PsychologyToday, April, 197Q,,53-69. 4

s%<*.Daye§,, R. M., &rCpet.agan, B. Linear models in decisidtr-making.Psychological Bulletin'; 1974;--811

'DeFipetti, B. *Does it make sense'to speak of "good probabilityappraisers"? In J. Gootl, (Ed.) The scientist speculates:An anthology of partly -baked ideas. New York:' Basic Books,

Dillman, DP H., & Cook, D,. L. .SiMulatio4.in the training of,R&Dproject managers. Paper pres,anted at the Annual Meeftng ofthe American Educational ffesearch Association, Lop Angeles,/Calif.; Feb., 8, 1969, 20 p. ED 209373

Dodson, J. D., Simulation system design for a TEAS simulatio nresearch facility.. Report AFCRL1112.and PRC 194. PlanningResearch,Corporation, Los Angeles; November 15, 1961.

Doughty, J. M. The,,AESOR testbed: Test sehies 1/2. In. D. E.''Walker (Ed.), Thformation' system sci.encet and echnologa.

Washington D.C.'s!Thompson Book Co., 69 -8* 107.'4

Doughty, J. M., & Feehrer, C. E. The AESOP testbed:' Test Series1/2 (Summary Report). Mitre Technical Report No., aaa, 1969.

Doughty, J. I4., Fe*ehrer, C. E., Bachand, T. E., & Green, E. R.The AESOP testbed, Test series 1/2: Supplementary technicalinformation. 'Vol. 1, SysteM descriptions, teSt procedures,and data analysis .Mitr'e Technical Report No. 824, 1969.

Dreyfus, S. Dynamic Prog-ramming. 4:In R. L. Adkoh, (Ed.),Progress in. operations eg:search., New York: Wiley, 1"961.'

Drucker, P. F. The effective decision. Harvard Business Review,1967, A5, 92-98.

Dunckei-, K. On peoblem-solVing. Wason & P.N.Johnson-Laird (Eds.), Thinking, and Reasoning,- Baltimore:_Penguin Bpoks, 1968, 28-43.

192

NAVTRAEQUIP,CEN 73-C-0128,1

J . c

0 Dutton, J. M. & tarbvck, W. H. Finding Charlie 4s run time-, estimator. In 'J. U. Dutton & W.z*H. Starbdck (Eds.),

COmputer, simulation of human behavior. New York: Wiley,,

1971%; ..,.

Edwards, J1. The theory pf decision making. Psychological: /-Bulletin, 1954, 51, 380-417.

Edwards, V. The use of statistical decision functions in ,makingspractical decisions., Paper given at Symposium.on Air Force'Human Engineering! Personnel, and Tr'aining Research, November14-16-, 055, Washington, D.c: Publication 455., NAS-NRC1956.

Edwards, W. , "Behaviaral decision theory, Ann al .Review ofPsychology, 1961, 12, 473-493%

Edwards/W. Probabilistic information processing, by men,mach/ines and man-machine systems. System DevelopmentCorporation TM-1418/000/01 August 1963. (Paper presented atlath Irternational Congress of Psychology, Washington, D.C.,August 1963.

cEAwards, W. Optimal strategies for seeking information: Models'for statii.st4s, choiceregction times, and human information ,

.procesSigg. aourbal of Mathematical Psychology, '196a; 2, .

312-329. -

Edwards, W. Probabilistic information processing system, fordieagnosis and action selection. In J. Spiegel & D., Walker(Eds.), Information system sciences. Proceedings "of the4econd Congress. Washington, D.C.: Spar tan Books, Inc.,1965b, 14T-155.

Edwards, W. Dynamic decision theory and probabilisticinformation processing. Human Factors, 1967, 4, 59-73.

trEdwards, W. A bibliography of research on behavioral decision

process to 4968 ,Memorandum Report No. 7,, Human Performance,.Center, University- of Michigan, Ann,Ahor, Michigan, 1969.

Edwards,." W. Reearch on the technology of inference and4 <decision. University off Michigan NTIS AD 771 463, 1973

Edwards, W., Lindman,' H., and Phillips, L. D. Emergingtechnologies for making decisions. New Direction's iCt

:----Psychology New York: 111olt, linehart, & WAnston, 1965,261-325.. Edwards, W., Phillips, L. D., Hays, W. L., &Goodman, B. C. Probpilistic information procegsing systems:Design and evaluation. IEEE Transactions on Systems Scienceand Cybernetics, 1968, SSC.L4, 248-265.

193.Zit;

4

(".

--NAVTRAEQUIPCEN 73-9-0128=14

Edwards`, W., & Slovie, P. Seeking information,to reduceitherkskof decisions. Ameeican Journal of Psychology, 1965, /1,1B8-197.

i. ---,, %

, , ,

Edwards,-W., & Tversky; A. Decision Making. Middlese *: PenguinBooks,.1967.' , .

. .

Egan, J. P., Recbgnition memory and the operating characteristic... AFGRL-TN-58-51,. Hearing' and Communication'Lab, Indiana

'U,niVersity,- Bloomington, Indiana,1958.0'4 t

.

ELlsberg, D. Risk, ambiguity andthe Savage axioms. Rand Corp.memo no. P-2173) :19'61. , .

gp. .

Elmaleh, J. S., Prywes, N. S., and Gustafemo, J. F,. Conversationwith computers in the opeeation'of sea traffic control. IEEEInternational Convention Record' Part71, 1967', 203-209.

"

Erlick,,D. E.,..Aeso.lute judgments of discrete quantities eandomfydistribUted over tide. Journal of Experimental Psychology,.1964,'61, 475-482.

Estes, W. JK. A descriptiye,apprpach to the dynamics 'of choiceibehavior. Behavioral Science, 1961, 6, 17T-184.

Evans, J. B., & Cody, J. J. Transfer of decision-making skillslearned in a counseling-like setting to similar and

situations. Journal of Counseling Psychology,1969, 16; A27-432.

Feehrer, C. E. TACC Planning Aid. The Mitre Corp. Memo No.

.D74-M-473, Febrgary 6, 1968.C

Feigenbaum, E. A. & Feldman, J. (Eds.). Computers and thought.New York: McGraw-Hill, Inc, 1963, 535 pages.

Ferguson, R. 1.,' & Jones, C. H. , A computer aided decisionsystem. `Management Science, 1969, l, 10, B-550 - B-561.

Festinger, L. A theory of cognitive dissonance. Evanston: Row,Peterson, 1957.'

Feurzeig, W. Conversational teaching machine. Datamation,Magasine, June, 1964.

Feurzeig, W., MUneer, P., Swets, J., & Been, M. Computer-aidedteaching in medical diagnosis. Journal of Medical Education,1964, 21, 746-754.

Fischer, G. W. Pour methods for assessing multi-attributeutilities: An experimental 'validation. University ofMichigan project number NR 197-014, September 1972.

'194

). 207

,

1

NAVTRAE QUIP CEN '73-C-0128-1

Fischer, G.W., $c Peterson, C.'R., Ratio versus magnitudeestimates of importance factors.' Ufliversity e orMipihiganReport. No. 037230-3-T, June, 1972.

Fishburn, P. Methods of estimating additip utilities.---litrentScience,, 1967, 13, 435-453.

Fisher; M., Fox, & Newman, A. Computer diagnosis of theacutely 'ill patient ,with fever and rash. InternationalJournal of Dermatology, 1973,41Z, 59-63. .N

Fleiss, J.,L., Spitzer, Cohen, J., & Endicott, J. Threecomputer diagnosis methods compared. Archives of GeneralPsychiatry, 1972,,2/, 643.649.

Fleming, R. A. The processing of conflicting information- in asimulated tactical- decision-making task. Human Factors,1970, 12, 375-385.

jFreedy, A., May, D., Weisbrod, R., &I Weltman, GI Adaptivecomputer aiding in dynamic decision process: Scenariogeneration by elicited expert probabilities, PerceptronicsInc., Technical Report No. PTR-T1016774-5 (II),*may 1, 1974.

.Freedy., A., Weisbrod, R., May, D., SchwartZ, S., & Weltman, G.Adaptive computer aiding in dynamic decision process..Technical Report PTR-73-101 Perceptronics, NTIS-AD 769 113,1973.

Fried, L. S., & Peterson, C. R. Information seeking: Optionalvs. fixed Stopping. Journal of Experimental Psychology;1969, 80,525-529.

Funaro, J. F. An empirical analysis of five descriptive modelcsfor cascaded inference. Rice UniverSity, Depai7tment ofPsychologyResearchReport Series, Report No. 74.2, May1974.

Gagliardi, U. 0., Hussey, R. A., Kaplan, T. T. & Matteis, R. J.

Man-computer interac\tions in idealized tactical problem1 solving. Nonr-3602(00), 1965.0

Geller, S. & Pitz, G.'F. Confidence and decisio4 speed inthe revision of opinion. Organizational Behavior and HumanPerformance, 1968, a, 190-201.

Gettys, C. F., Kelly, C. W., & Peterion, C. R. i Best guesshypothesis in multi-stage inference. Organizatiional Behaviorand Hunan Performance, 1973, 10, 364-373

Gettys, C. F: & Wqke, T."A. The application or Bayes's theoremwhen the true data state is uncertain. OrganizationalBehavior and Human Performance, 1969, 4, 125-141.

195

;&{)8.,

4

NAVTRAEQUIITgN 73-A128-1

.

Gibson, R. S. & Nicol, E. H: The modifiability of decisionsmade in a changing environment. ESD-TR-64-657, December,1964.

p

Gledhill, V. X., Mathews, J. D., ,and Mackay, J. R.Computer-aided diagnosis: A study of bronchitis. Methods ofInformation in Medicihe, 1972, II, 228-232.

,

Good, I. J. Rational decisions., J% Royal, Statistical Society.Series B, 1952, '14, 107-114.

Gorry, G. A. The development of managerial models. M.I.T.,Alfred P. Sloan Sdhool of Management, 1970.

Grabiti, H. J., & Jochen, H. An evaluation of. confirmin-g anddisconfirming information in decision making (Grier). Archivfur Piycholok'ie, 1972; 124, 133-144.

Green, D. M., & Swets, J.A. Am*,1, detection theory andosychophysics. New York: 'Robert r.". Krieger Publishing Co.,Inc., 1974. (Origipally published, 1966.)

Green, P. E., Halbert, M. H. :and Minas, J. S. An experiment ininformation buyj..ng., Journal of Advertising Research, 19644

Greene, R. A. A computer -based system for medical record keepinghy physicians. Harvard University, 1969.

Greist, J: H., .Klein, M. H.,' & Van Cura, L. J. A computerinterview for psychiatric patients target symptops. Archivesof Generalpayetiahry, 1973, 22, 247-253.

Guilford, T.P. Intellectual resources and their values'as seenby 'scientists. In C. W. Taylor & F. Barron (Eds.),Scientific creativity: Its recognition and development.' Nevi

York: Wiley, 1963.

Hammell, T.J., and,Mara, T. D. Application of decision makingand team training research to 'operational training: Atranslative technique. NAVTRADEVCEN 68-C-024201, 1970.

Hammond, J.1"S. Better decisions with preference theory.Harvard Business Review, 11967, Aa, 123-141..

Hanes, R. M., & Gebhard, J. W.1, The computer's role in commanddecision. U.S.- Naval Institute Proceedings, 1966, 22,60-68. 4.

Hayes, J. R. M. Human data processing I.imits in decision making.In E. Bennett (Ed.), Information system science and

..' engineering. Proceedings pi,. the First Congress on theInformation Systems Sciences, New York: McGraw-Hill, 1964.

/ 196

209

A

t4(

).

vNAVTRAEQUIPCEN 73-C-0128-1

Haygood, R.*C., &Bourne., L. E., Jr. %Attribute and rule-learning: aspects of conceptual behavthr. Psychological Review, 1965,/a, 175-195:

Haythorn, W. W. Human factorsin system research. Rand Corp..'0 Report No. P-2337, June 1961. ,(To be A chapter in E. M.

Bennett (Ed,), Human factors.in technology.-

Helmer, O. Systematic-use of expert.opinions. Rand Corp. memo

. u t ,dr,.

7 No. P- 3721, "1967... .

Henke: A. H., Alden, D. G., & Levit, R. 11%. An analysis of humancognitive styles. Honeywell, Inc., Document "F0141, 2

volumes, 172. .,. ,

I

,nlHee, M.. On the relation between logic and thinking.

Psychology Review, 1962, 62, 366-378. ,

..,,,,

,,

HenMan, L. M., Ornstein, N., & Bahrick, H. P. Operatordecision performance sing probablistic displays of objectlocation. IEEE Transactions on Human Factors ireElectronic6,1964, a, 13-19.

Hill, J., & Martin, Jr R. Trainingfeffor educational deciionmaking. Journal of Teacher Education; 1971, 22141, 443-447.

Hoepfl, R. T.,'& Huber, G. P. A study of self- exblicated utility. ,models. Behavioral Science, 1970, 12, 408-414.

Hoffman, P. J. The paramo phio repre sentation ofjudgment: .Psychological Bulletin, 1960; 5J, 116.

Hdffman, P. J. Cue-consistency and configurality in human/--''` judgment. In B. Kleihmuntz (Ed.), Formal representation of

"human .judgment. New ,York: Wiley, 1968.

Hoffman, P.J. & Peterson, C. R., A scoring rule to trainprobability assessors. University of Michigan Project ao.NR 197-014, 1972.

.Hofstatter, P. R. . Uber die Schatzung von gruppeneigenschaftbn.Zeitschrift fur peychologie,, 1939, 145, 1-44.

Hogarth, R. M. Process tracing in clinical. judgment. BehavioralScience, 1974, 12, 248-313,

Horabin, I. Algorithms.- Improving Human Performance: A ResearchQuarterly, 1972, 1, 28-41. e

Horrocks, J. C., 4.de Dombal, F. T. quman and computer-aideddiagnosis of 'dyspepsia , British Journal of Surgery, 1973,60, 910.

.. 4

I '. r. "I.)

1 t

aNAVTRAEQUIPCEN 73:C70128-1

.

..

Howard, R. A, The foundations of decision analyiis. fEEE' 1

Transaction on Systems Sdtence andCvUernetics,. 1968, S8C-4,

211-219. 's ''',,

,

. - e ,

Go'

I Howarld, Fit,. A., AatteSon, J. E., & North, D. W. The decision to

seed hurricanes, Science,' 1972, 176, 1191-1202. ,,

. . ..- .

Howell, W.C. Task characteristics in sequential de isionbehavior. Journal 'of Experimental Psychology, 19 6, fl,124-131.

Howell, W. C., & Gettys, C. F. Some principles for design of.44

decision systems: A review of the fin base of research on" a command contol system simulation. 'USAF AMRL Tech. Report

No. 68-158, 1968. 4

Huber, G. .§ahney, V, & Ford; D. l' study of subjective

Devaluation models. Behavioral Science, 1969, 14, 463-489.

4

Irwin, F. W. & Smith, W. A. S. Value, cost, and information\ asdeterminers of decision. Journal of Experimental Psycholegv,1957, 5111 229-232.

Jacques,J. A. (Ed'.) Computers diagnosis and diagnostic methods.

Springfield, Illinois, Charles C. Thomas, 1972...

Jarvik, M. E. Probability learnting and a negative recency ef febt

in the serial anticipation of alternative symbols. 4,ournal.

of Experimental Psychology, 1951, 41, .291-297.

'Jensen, F. A., & Peterson, C. R. -Psychological effects of properscoring rules, Organizational Behavior and Human Performance,

1973,,2, 307-317-. .4

Johnson, E.M. Numerical encoding of qualitive expressions of

uncertainty. U.aS. Army Research Institute for the

Behavioral and Social Sciences Technical Paper No. 250,

December 1973

Johnson, E. M. The effect of data sotace on'intuitiveinference. ARI Technical Paper\ No. 251, AD 784097, July1974:

t

Johnsop, E. M., Cavanagh', R. C., Spoort, R. C., & Samet, M. G.

Utilization of reliability. ,m surements in Bayesianinference: Models and human performance. IEEE TransactionsOn Reliability, 1973, r-22, 176-183.

Kaplan, R. J., & Newman, J. R. A study in probabilisticinformation processing (PIP). System DevelopmentCorporation, TM-1150/000/00, April 1963.

198

211

sea

1

r

470

7 .c


Kaplan, 'R. J., & Nipwman, J. R. Studies in probabilisticeinfdrmation processing. IEEE Transactions in Human Factorsin Electronics, 1966, 1, 49-63.

IKahneman, D. & Tversky, A. Subjetive probability:. A judgmentof representativeness; Cognitive Psychology, 1972, a,

434-454.4

Kahneman, D, & Tversky, A; On the psychology of mediction.cr Psvcho]ogical Review, 1973, 80'1'4237-251.

AKanarick, A. F. The learning, retention and transfer of decision

making. Paper prdSented at meetings on Learning,- Retentionand Transfer, sponsorbd by Honeywell Systems and ResearchCenter, and the Naval Training Devices Center, Winter Park,4 Florida, February 19,69..4/

Kanarick, A'. F., Huntington, J. M., & Pqersen, R. C.

Multi-source information acquisition with opt'ion..1 stopping.Human Factors, 1969, 11, 379 -386. ,

Kant, I. Introduction to logic and essay on the mistakensubtilq- of the four figures. 4. K. 4 Abbott; trans.),Longman s, Green, 1885.

4

Kepner, C. H., & Tr.egoe, ,B. B. The rational sanaglni A

systematic approach to problem solving and decision making.New.York4 4cGraw Hill, 1965..

.P-

Kibbee,A M. Dress rehearsel for decision making: Thy` growingusb of business, games. The Management Review, 1959, 48,4473.

Kleinmutz, B. The processing of clinical information by/men andmachines. In Kleinmutz,. B. (Ed.),, Formal representation ofhuman judgment. New York:.Wiley, 1968.

Kneppreih, N. P., Gutafson, D. H., Leifer, R. P., & -Johrlson,

E., M.' Techniques for assessment of worth.' U.S. ArmyResearch Institute for the Behavioral and 'Social SciencesTechnical Paper. No. 254, .1974. .i#

Krantz, D.H., & Tversky, A. Conjoint - measurement analysil ofcomposition rules in psychology. Psych'o'logy Review, 1971, 8,157-169,

Lathrop, R. G. Perceived variability. Join4n

Psychology. 1967, 12, 498-502.

Leaper, D. J. Computer-:asetisted diagnosis of abdominal painusing "estimates" of data provided by clinicians and surveydata obtained from real life, In T. J. Triggs',& R. ti.4Pickett (Eds.), Human factors in health care, Lexington,

:1:2112

of Experimental

199

$

NAVTRAEQUIPCEN

Massachusetts: D.C. Heath and Compny, 1975

5

Leaper, D. J., de DoMbal, F, T., Horrocks, J. C., & .0taniland,J. R. Computer - assisted diagnosis of abdominal pain using, . ceestipates provided b clinitans. British Journal of

-IrSurgerx, 1972, 52, 897-89

Ledley, R. S., & Lusted, L. B. Reasoning foundations of medicaldiagnosis. Science, 1959, 130., 9-21. ,

\Lee, W. Decision theory and human behavior. New York: John

Wiley & Sons, 1.971.4

Levine, J. M., & Samet, M.'U. *Information-seeking with multiplesources of conflicting and unreliableinfbrmation. HumanFlebrs,1973, 15:, 407-419.

Levine, J. , Samet, M, G., & BraheV; R. E. Informationseeking with input pacing and. multiple decisionopportunities. ,Human Factors, 1974, 16, 3847394.

s,

Levit, R. A., Alden, D. G., Erickson; J. M., ?. Heaton, B. J.

Development and application of a decision aid for tacticalcontrol of battlefield operations. Vol. 1: A conceptualstructure Cor -decision support in tactical 1!,operationssystems. Honeywell,)Inc. Contract number DAHC 19 -73 -C -0069,1974. tt,

'Levit, R. A., Alden, D. G., A Henke, A. H. Development apa

application of a decision aid for tactical control ofbattlefi,ld operations. HOneywell Document No. F6049-AA,

1973.

Lichtenstein,,S. & Newman, J.R. Empirical scaling of common',

verbal phrases assoc'atel Nith numerical probabilitieS.Psvchonomic Science, 19o7, 11 56S-564.

Lichtenstein, S. & Slovic, P. Reversals of preferences betweenbids a choices in gambling decision's. Journal OfExperimental Psychology, 1971, 82,'46-55.

Lichtenstein, S. , Slovic, P. & Zink, 'D. Effecrof instructionin expected value on optimality of gambling decisions.Journal of Experimental Psychology, 1969, /2,236-240.

N

. Licklider, J. C. R. Man-computer symbiosis. IRE Transactions onHuman Factors in Electronics, March 1960, 4-11.

Lindman, H. Inconsistent preferences among gambled. Indiana,Mathematical Psychology Program Report 70-2, 1970.

Lodwick, G. S. A probabilistic approach to dignosis of bonetumors. Ridioaogic Clinics- of North America, 1965, a,

200

7

4

ti

487-497.

e

NAVTRAE9UUCEN 73-0-0128j1

, .

s,

Luce, R. D. Individual choibe behavior. \lew York: Wiley, 1959.

4

ti

Luce, R. D. & Raiffa, H: Games and decisions: Introduction andcritical survey. New York: Wiley, 1957.

Luce, R. D.? kTukey, J. W. Simultaneous conjoint measurement: Anew type-Of fundamental measurement. Journal of Mathematical-Psychology,, 1964, 1, 1-27.

sl

Ludted, B.' Computer techniques' inl medical diagnosis. InIL W. Stacy And B. \Waxman (Eds.), Computers in biomedicalresearch, Volume 1, 1965, New York: Academic Pr7ess, 319-338.

MacCrimmon,? K. R. Decision making among multiple-attributealternafiives: 'A survey,. and consolidated approach. ROdCorporation Memo RM 4823-ARPA, December 1968.

,

Mackworth, N. H. Originality.ft American Psychologist, 1965, 20,

51-66.

,Maier, N. R. F. Re asoning in humans. II.. Th4e solution of a

alproblem anck its appearance i consciousness. Journal ofComparative Psychology, 1931, 1 181-194..

a

Marschak, National" behavior, uncertain prospects, andmeasurable:utility. EOonometrica, 1950, 18, 111-141.

Martin, E. W., Jr.' Teaching executives via simulation. BusinessHorizon, 1959, 2, 100-109.

McGirr, E. M. Computers in clinical diagnosis. In J. Rose(Ed.), Computers in medicine, 1969, London: J.A. Churchill,19-29.

Meadow, C. L. & Ness, D. N. A case. study of the use of a

decision-support system. M.I.T. Report No 674--P73, 1973.

Mill, A system of logic. Harper, 8th edition, 1874.

Miller, D. W., & Starr, M. K. The structure Of human decisions.New Jefsey: Prentice-Hall, Inc., 1967.

Miller, L.W., Kaplan, R.J., & Edwards, W. JUDGE: A

value-tjudgmeht-based tactical ,command system. OrganizationalBehavior and Human Performance, A967; 2, 329-374. .

Minsky; M. Steps toward artificial intelligence. In E. A..Feigenbaum & J. Feldman (Eds.), Co utera and thought. New:York: McGraw-Hill, Inc., 1963, 406 -450.

201

214

1

A Page' 16

.' NAVTRAEQUIPCEN 73-9=01 8.;-1

Mo rgan, J. 4. B., Mo ton J. T. The aistortion of syrlogisticI

db.

v.reasoning produce by pe/rsonal convictions, Journal of SocialPsycliology, 1944* 20, 39-59. .

in, R. E. Strategies` *im/ games with saddle-points.Psychological Report, 1960, /, 479-485.

...

.

4

Morton, M. S. S. Decision6support sy)tems: the design pig ceg.

Sloan School of Management, Report No. 686-73, 1973.. .

in .concept attainme4.1962, 64, 640 -645.( 1

.NeIsser, U., & Weene, P. :HierarchiesJournal of Experimental Psychology,

.Neyman, J., & PearSon, E. S. The testing, of statistical.hypotheses, in relation to probability a priori. Proceedingsof the Cambridge PhilosOpifical Society, 1933, 22, 492-510.

Nickerson, R.- S. & McGoldrick, C. C. Confidence, ci:rreatness,and difficulty 4ith non-puchophysical comparative judgments.Perceptual and Motor SkiLis%1963, 1/, 159-167.

Nilsson, N.J. Problem-solving methods inA intelligence. McGraw-Hill, Inc., 1971.

,.,

North, D. W. A tutorial introduction-t'o decision theory. IEEETransactions on Systems Science andCillerpetics, 1968, SSC-4,200-210. ,'

,.40 . 1 0..

G

O'Connor, M. F. The assessment of worth for multilattributedalternatives: A survey!. Unpublished report, Ugiversity ofMichigan, 1972. .i

artificial

orgitnist, W. E. Use of subjective probabilities as a responsetechnique in multiple choice, behavior. Paper presented atthe Fifth Annual Meeting of the Psychonomic Society,. Ontario,Canada, October, 17,-1964.

Crganist, W. E., & Shuford, E. H. Bayesian management systems.Paper present:ed at the University of Michigan,. SecondBayesian Systems Conference, May, 1964'.

. Pascal, B. Thoughts. In C.W. Eliot (Ed.), The Harvardclassics, 19101 48, *New York: Collier:

Paxson, E. W. War gaming. Rand Corp.1963.

memo no. RM-3489-PR,

Payne, J. W., Alternative 'approad4es to decision making underri*: Moments versus risk dimensions, NPsychological Bulletin,19a3, 80, 439-45-3., .

',,

'

.. IP

Peterson, C. R.', & Beach, L. R. Man as an intuitive`statistician. Psychological Bulletin, 1967, 68, 29-46.

202. 215

ti

.4

ik

4*

Pe } erson, & bliCharme, W. M.subjective probability reviion.Psychology,- 1967,, /3) 61-65.

Zeterson, C..' R., et al . andbook

, NAVTRAEQ1.11iRCEN 73-C=01.28z-1

Peterson, .C. R. & Miller A. J.probability rrevdsion. Journal1965, 70, 117-121.

A- primacy effect inJodrhil of Experimental

for' decision analysis.1973) ,ARPA/ONR 4No.

'

' Decisions -and A Designs, Inc.,Nonr-N000114-73Cr0149.

Sensitivity of subj-Activeof Experimental Psychology,

Peterson, C. R.,' ec. Phillips, L. D. Revision of continuousprobability 'distributions. (IEEE Transactions on HumanFactors ih Electronics, 1966,.HFE-7, 19-22.

Peterson, C. R. Schneider; H. J., & Miller, A. J. Sample sizeand .the revision of su jectivet probability. .JournalofExperimental PsybholOgy, 1 65, 62,'522-527.

Peterson, W. W., Birdsall, T:signal detectability,171 -212.

Phillips, L. D. & Edwardp, W.probability inference task.Psychology, 1966, la, 345=3:5.7.

4

& Fox,` W. C. The theory ofRE Transactions, lqp, PGIT-Y,

(

"Pitz, G.F. Sa le size; likelih1

iod and confidence in a decision.

Conservation in a simple .

Journal o Experimental

Ps chop mic4plim.se, 1967, 1 (6),'257=-258.

Pitz, £: F.ihe limited.' 25-34.

Information seeking when available information. isJournal of Experimental Psychology, 1968, 36,

Pitz, G.F. An inertia effect (resistance to change) in therevision of a 'nion. Canadian Journal of Psychology: 1969,23, 24-33.

r. d/ , , ?,

Pitz, G. F., Downing, L., & Reinhold, H. -Sequential effects inthe reisionf of subjective,probabilities. Canadian Journalof PsychBlp)zv, 1967, 21, 381-393.

--',.

Pitz, G. F., 'Rerrigald, H., & Geller, E. S. Strategies of,information % seeking in- deferred decision making,Organizational Behavior ind Human Performance, 1969.

Platt, W. Strategic intelligence production. New York: Praeger,1957.

PQlya, G. How to solve it. Princeton, New Jersey: Princeton.University Press, (2nd edition, Anchor Press),.1957.

203

216

0

1%,

.4 1

#

NAVTR QUIPCEN 73 --6L01.28-'1

I

r

'Polyani, V. Experience and the .perception pattern. In' K. M.

' Sayre &441" F. Crossman' (Eds.);The modeling ,of mind:

Computers and .intelligence. Notre Dame: University of Notre

Dame.Pressi-1963...

. Prui(t, P'. G. Informational requirements in making decisions.American Journal of Psychology, 1961, /11, 433-439x,

r* Raiffa H. .pecistpn 'analysis. 4assalusetts: Addison-Asley,

1968. / .

.,

% r 4

Raiffa, H. Preferences for multi - attributed alternatives. Rand

.,CorporatiOn Me1116 RM-586,8-DOT-RC, April, 1.969.

I ,..,

Raiffal H. & Schlaifer, R. Applied statistical decision theory;.

Boston: garvard Urliver'Sity, 1-/

961. i , *

,

.

,

,.

`Rapoport, A. Flights, games, and 'debates. Ann Arbor: University

° of Michigan Press, 1960. ,

.

- . - .

Rapoport, A. Sequential depis. tbn-making in a .computer- -controlled`

task. Mattiematical Psv.chdlogv., 1964, 1, 351-374 -

Rapoport, A., '& allseep, T.-4. 4 Individu decision behavior,.,

.

4 Annual Review of Psychology, 1972,, 131-176.

,fieingen, P. H.. Risk-taking by individu s and informal groups

with the use of industpial-prdduct.purchasing situations as

stimuli. Journ0..,of Psv/Ch-olagv, 1973, Aa, 339-345 1

. /

Rescher,- Delphi, and values. Rand CorP.tMemo #P-4182, Sept.,1969.

Rice, D. C., & Meckley,' R. F. Supervision and, -decision-making

skills .in vocational education: *A traihing program utilizingsimulation tech'iques. Office,of Education (DHEW), .Bureau of

Res6arch, Waishirecon, b. C., Final Report No. RS-51, Marcy,

1)970, ED 038501.

tRikhey, J. W., & DeBow., C. H. Decisibetstrategies "in AAW: I.

'Analysis of air threat judgments and weapons assignments.ONR Technical Report No.. 47,

Rigby,'L. V., & Swain, A. D. In--Flight target reporting..-How

much llaybunzth"? Human $actors, 1971, 13, 177-182.-

Roberts, F. S.* What if utility functions .do not exist? Rand

Corp. memo no. ar528 -ARPA, 1970..

Roby, T. B. Belief statesevidence and action. In,D. P. Htn t

& D. L. Zink redectsional processes in decisionmaking: Proceedings 'o,P a' Symposium, USAF AMRL- TDR- 64 -77,

1964, 27-46.I

4

, 204"

2i?

0,

t


r

andsaBelief States the uses of evidence. Behavioral,Science, 1965, 10,'255 -270.

Rochester, S. R. Gill, J. Production of complex sentepces inmonolbgues an .dialogues. Journal of Verbal and-n---- 4

Verbal Behav or,-,1973, 1.2, 203,210. a

Rousseau, W. F., & Zamora, R. M. user's manual for the.generation and analysis orprob biliy and-decisiOn trees.SRI Final Report, Project 464531-307, 1972.

Russell,.B. The axiom of infillity: Hibbert J., 1904, 8097812.

Samet, M. G: Effects of the lOgarithmic, quadratic and spherical A

payoff functions on response behavior. Doctoral .

Dissertation, Tufts University, 197.1.. 4

Samet, M. 9uantitativ4intbrpretatiQn of two qualitive scalesused to Ate military Intelligence. Human Factors, 1975a,2/,'76-86.

,

, . .iole-- , 11

Amet', M. Subjective interpretation of reliability and accuracyScales for evA7ivating military intelligence. U.S. ArmyReseantynstitute for the BehaVioral and Social. sciencesTechnida. Paper No.-260; 1975b.

C.

-dSamuelson, P. A.Foundations ofeconomic, -Analysis. Massachusetts:

Haryard,University Press, 1947.

Savage, L. J. Foundation of statistics. New York': Wiley, 1954.

%Savage. L.

\

J. Elicitation of personal probabilities andexpectations. Journal of the American Statistical

4 Association, 1971, 66, 83-801.

Scalzi, J. B. CO decision making be taught? Civil Engineer ingASCE, 1970',' 40,' 37-38.

Schiller,' F. C. S.. Logic for use. Harcourt,.Brace, 1930.

Schleifer, R. Probability and statistics for business decisions.New York: 'McGraw-Hill; 1959.. '

'Schleifer, R. Anarysis of 'decisions under uncertainity. New. York: McGraw-Hill, 1969. 90

Schopenhauer, A. The art of controversy. In J. B. Saunders, (Ed.), The Essays of Arthur Schopenhauer, New York: Wiley

Book Co., no date.

Schrenk, L. Pt, .Objective difficulty and input' history- insequentia),. Acision making. Human Factors, 1964, 9, 49-55.

205

INANTRAE QU P GEN 73-C-0128-1

a

. .

Schrehk, L. P. -.Aiding the decision maker--A decision processmodel. Ergonomics, 1969, 12, 543-557..

Scheenk, L. P., & Kanarick, A. F1 Diagnostic decision making in. ----. q, 'Bayesian framework. Proceedings of the .5th Annual

7.SgmposiuM on Human Factors in Eldctronics, 19,6,

Schum, D, A., & DuCharmer. W. M. Comments on the relationship'.. . between the- impact, and the rgliahility of evidence.

Organizational Behavior and Human Performance, 1971, 6,111-131.

La ). 4,

A

/Schum, D. A.;'' DuCharme, W. M. & DePitts, K. E. ReSearch on

human multi-stgge probabilistic inference processes.Organizational Behavior and Human Performance, 1973, 10,318-348.

.Schum, D. A., Goldstein, I. L.,'HoWell, W. C., & Sou-thLrd, J. F.Subjective probability revisions under several cost payoff,arrangements.. Organizational Behavior & Human Performance,t1967, 2, 84-104.,

Schum, D. A., Goldste,in, I. L., & Southard, J. F. Research on asimulated Bayesian'information processing system.. IEEETransactions on Human Factors in Electronics, 1966, HF5-7,37 -8.

Schum, D. A.,, Southard, J. F., ,& Wombol't, L. F. Aided humanOocessing of inconclusive evidence =in diagnostic systems: Asummar-k of experimental evaluations. AMRL-4TR-69-11,`Aeromedical Research Laboratories, Wright Patterson, 'AFB,1969.

Sebestyen, G. S. *Decision making processes in patternrecognition. Nq,w York: Macmillan, 1962.

4.

it

Seigel, J. & Farrell, E. J. A compute!" simulation model tostudy he clinical observability of ventilation and perfusionabnorm lities in human shock states. Surgery, 1973.

Shackle, G. L. S. Timesin economics. -Amsterdam:, North, HollandPublishing Co., 1958; second printing 1967.

Sheridan, .A. J., & Carlson, R. E. Decision - making in - a*perormance appraisal situation. Personnel Ps cholo 1972,

aa, 339-359.

Shufor'd, E.H. A computei.-based system for aiding decisionmaking. In J. Spiegel & D. Walker (Eds.), Informationsystem sciences, Proceedings of the Second Congress.

, Washington, D.C.: Spartan Books, Inc., 1965, t5g7168.

206

gig

J

I

;

v,NAVTRAEQ9CEN,737C-0128-1

Shuford, E. H. Cybernetic testing. Electronic Systems Division,U.S.A.F. Repoyt No., ESD-TR1-67-229, 1967.

Shuford, E.H., Albeit, A., '&. Massengill, H.E. - Admissiblejprobabilityjmeasurement procedures. Psvchometrika, 1966,. 31,125-147.'

a

. Sidorsky, R.C. Sote determinants of computer-aided decisidneffectiveness. Proceedings of 80th Annual Convention,American Psychological Association, 1-972, '697-698.

Sidorsky,'R. C., 4&* Houseman, J. F. Research on generaliRedskill's related tai. tactical decisions making. U.S. NavyTraining Device Center Report NAVTRADEVCEN TR 1329-2,-1966r,

Sidorsky, R. C., Housema., J. F., & Ferguson, D. E. Behavioraland operational sects of tactical decision making in AAWand ASW. NAVTRA N 1329-1, 1.964':

Sidorsky, R. C. Mara, T. D. Training aspects ofcomputer-aided decision making: I. Comparisen of, twoapproaches to man-computer interaction. NAVTRADEVCEN 1329-3,July 1968.

Siegel, J. H. &" Strom, B. L. The Computer as a '1±vingtextbook' applied to the care of the critically injuredpatient. Journal of Trauma, 1972, 12, 739-755.

Siegel, S. Theoretical models of choice and strategy behavior:Stable state behavior in the two-choice uncertain outcomesituation. Eamehometrika, 1959, 24, 303-316.

Simon, H. ..11 behavioral model of rational choice. ,QuarterlyJournal of Economics, 1955, 62, 99-,118.

Slovic, P.' Value as a determiner of subjective probability.IEEE Transactions on Human Factors in Electronics, 1966,giFE-7-22-28.

A

Slovic,' P. Influence of response mode upon the relativeimportance of probabilities and payoffs in risk taking.Proceedings ofT the 75th Annual Convention, AmericanPsychological Association, 1967, 33-34. s

Slovic, P. Manipulating the attractiveness of a gamble withoutchanging its expected value. Journal of ExperimentalPsychology, 1969, 12, 139-145.

Slovic, P., & Lichtenstein, S. Comparison of Bayesian andregression approaches to the study of information processingin jddgment. 0 ganizational Behavior and Human Performance,W1, 6, 649-7 .

207

2O

NAVTRA.EQUZPCEN- 73 -C,0128 71.

110

1

Snapper, . J., & Fryback, D. G. Inference based on unreliablereports Journal of Experimental Psychology, 1971, 81,401-404.

Snapper,*K. J., & Petersen, C. R. Information seeking and datadiagnosticity. Journal of Experimental Psychology, 1,971,IT,429-433. .

Soelberg,P. First generalizable decision_ process model:- HIT:Sloan School, 1966.

-Soelberg, P. O. Unprogrammed decision making. Industrial

Management Review, 196-7k8, 19-29.

Southard, J. F., Ochum,' D. A. & Briggs, G. E Application ofBayes theorein as a hypothesis--Selectibn aid in a complexinformatidn processing system. Ohio State ,University,AMRL-TDR-64-51, 1964a.

Southard, J. F., Schum, D. A., & Briggs, G. E. Subject controlOver a Bayesian hypothesis-selection aid in a complexinformation processing system. Ohio University,AMRL-TR-64-95, 1964b. ,

Spruance, R. A. Foreword i4i Fuchida, H.,-& Okumiya4 M.1 MIDWAY:The Japanese Navy's story of the battle that docOed Japan.U.S. Naval Institute: Ballantin,e Books,,- '0955.

Steiger, J. H.,. & Gettyp, C. P. Best-guess errors inmulti-stage inference. Journal of Experimental Psychold,'1972, 22, 1-7.

Stewart, N.R., & Winborn, B. B. A model of de6ision making insystematic counseling. Educational Technology, 1973, 1R,13-15.

Summers, J. K., & Bennett; E. AESOP--A final report: A prototype.on-line interactive information control system. In D. E.Walker (Ed.), Information system science and technology,Washington, D.C.: Thompson Book Co., 1967, 349-357.

Swets, J. A., & Birdsall, T. G. Deferred decision in humansign&l detection: A preliminary experiment. Perception &Psychophysics, 1967, 2, 15-18.

Szafran, J. The effects of ageing on professional pilots. In'-Price, J. H. (Ed.), Modern trends in psychological medicineII, New York: Appleton-Century-Crofts, 1970.

Tanner, W. P., Jr. A theory of recognition. Journal of theAcoustical Society of America, 1956, 28082-888.

208

r 221

NAVTRAEQUIRCEN 73-0-0128-1

Thrall, R. M., Coombs, C. H., & Davis, R. L. Decision processes.John Wiley & Sons, 1954.

Toda, M. Measurement of subjective probability distributions.Decision Sciences Laboratory Technical Documentary Report No.ESD-TDR-63-407, July 1963:

Triggs,, T. J. Decibion flow diagrams in' information managementJ.4systems. BBN Report 2617, 1973, Submitted to U.S. Army"9Research Institute for the Behavioral and Social Sciences.under contract DAHC-19-72=G.-001,9. t

T dokenham, W. J. ,The use of logical flow charts as an aid in'teaching roentgen diagnosis. American Journal ofRoentgenology, 1968, 102, 797, 803.

r .

Tune`, G. S. Response preferences: A .review Of some relevant,literature. Psychological Bulletin, 1964,61, 28u-302.

Tversky, A., & Kahneman, D. Belief in the law of small numbers.Psychological Bulletin, 1971, /6, 105-110. \

Tversky, A. & Kahneman, D. Availability: A heuristic forjudging frequenCy and probability. Cognitive Psychology,1973, 5, 207-232.. e,

Tversky, A., .& Kahneman, ,D. Judgment under uncertainty:Heuristics and biases. Science, 19,74, 185, 112A-11:31.

Van Naerssen, R. F. A scale for the measurement of subjectiveprobability. Acta pAigholgalga, 1962, 20, 159-166.

Vlek, C. A. Multiple probability learning: Associating eventswith their probabilities of occurrence. Acta *Psychologica,1970, 33, 207-232:

Von Neumann, J. Morgenstern, 0. Theory of games, and economicbehavior, ,2nd ed., New Jersey: Princeton University Press',1947.

Wagenaar, W. A.' Subjective randomness and the capacity tto

generate information. In A. F. Sanders (Ed.), Attention andeperformance III, Acta Psychologica, 1970, 31,233-242.

Wald, A. Sequential analysis. New York: Wiley, 1947.

Wald, A. Statistical decision f notions. New York: Wiley, 1950.

Wallas, G. The art of thoug . London: Jonathan Cape, 1925.

Wason, P. The drafting of rules. New Law ,Journal, 1968, 118,548-549.

I.

, I**

.NAVTRAEQUZPCEN 73-C-0128-1

Wason, P. The psychology of-deceptive problems. New Scientist,- 1974, _63, 382-385.

Whitehead, S. F.,-& Castleman, P. A. Evaluation 'of an automatedmedical history in medical practice. Proceedings.of the 12/AConference of the Society for Computer Medicine (to bepublished). 4.

Williams, A. C., Jr. & Hopkins, C. 0. Aspects of pilotdecision-making. WADC TEchnical Report 58522, 1958.

Winkler, R.L., & Murphy= A.H. Evaluation of subjectiveprecipitation probability forecasts. In Proceedings of theFirst National Conference on Statistical Meteorology,American Meteorology Society, Boston, 1968, 148-157.

Wright, P. Writing to be understood:Ugly use sentences? Applied

grE2aMiga, 1,* 207-209..

Yeh, S.Y., Betyar, L., & Hon', E.H. Computer diagnosis of fetalheart rate patterns. American Journal of Obstetrics andGynecology, 1972, 114, 890-897.

Yntema, D, B., & KleM, L. Telling a computer how to dvaluatemultidimensional situations. IEEE Transactions on HumanFactors Engineering, 1965, 3-13.

Yntema, D. B. & Torgerson, W. S. Man-computer cooperation in

decisions requiring common sense. IRE Transactions on HumanFactors in Electronics, 1961, 20-26.

Zadeh, L. A. Outline of a new approach to the analysis ofcomplex systems and decision processes. IEEE Transytions onSystems: Man and Cybernetics, 1973, 5MC-3, 28-44.

210

223

IP.

a NAVTRAEQUIPCEN 73-C-0128-1

HUMAN gACTORS DISTRIBUTION LIST

O

NOT

Mailing labels eke prepared, when needed, as a computerlisting, the source of which is updated on a weekly basis.It is not practical to prepare distribution lists each timelabels are prepared. .Therefore such lists are preparedsemiannually, and' a slight discrepancy may exist`betweenthe addressees on'this list and. those. appearing on the labelsused to distribute this publication.

Chief of Naval TrainingAttn: Captain A. E. McMicha.el, N-3Pensacola, FL 32508

Chief of Naval TrainingAttn: Captain B. C. StonePensacola,' FL 32508

A

Chief trf Naval Training .Attn: Dr. W. Maloy, Code 01APensacola, FL 32508

Chief of Naval MaterialAttn: Mr. A. L. Rubinstein, MAT-03424Navy DepartmentWashington DC 20360

Commanding OfficerNaval Submarine Base, New LondonAttn: Psychology SectionBOX 00Groton, CT 06340

Chief of 'Naval Air TrainingAttn: Joseph L. UlatoskiNaval Air StationCorpus Christi, TX 78419

Commander Training CommandAttn: Educational AdvisorU.S. Pacific FleetSan Diego, CA 92147

(1 of 4)

Chief Naval ResearchAttn: Code 458 Navy DepartmentArlington, VA 22217

COMDT of Marine C. -psCode A030WashingtOn, DC 20380

Air Force Human Resources Lab/DOJZ.Brooks Air Force BaseTexas 78235

CDR NAVAIR DEVEL CTRAttn: Human Engrg BrWarminister, PA 18974

CommanderNaval Electronics Systems CommandCode 03Washington, DC 20360

NAVAERSPMEDINSTNAVAEROSPRE GMEDCENATTN: Ch Avia Psych DivPensacola, FL 32512

Chief Naval OperationsAttn: CDR H. J. ConneryOP-987 M42 Navy DeptWashington, DC 20350

224


HUMAN FACTORS DISTRIBUTION LIST

Commander Training CommandAttn: Educational AdvisorU. S. Atlantic FleetNorfolk, VA 23511

Assistant Secretary of the Navy (R&D)Attn: Dr.,. Samuel Kos lov, 4E741Navy DepartmentWashington, DC 20350

Director of Defense Research & EngineeringAttn: Lt Col Henry Taylor, OAD (R&D)Washington, DC 20301

Dir Hinman Res Rsch Org300 N Washington StAlexandria, 'VA 22314

CommanderNAVAiRSYS COMCode 03Washington, DC 20360

CDR NAVSHIPS SYS CMDNAVSHIPS SYS CMD INSSHIPS 031-1 Geo. N. GraineWashington, DC 20360

Chief Naval OperationsAttn: Dr. J. J. Collins ,OP-987F Navy DeptWashington, DC 20350

ERIC Clearinghouse UNEDUCAT MEDIATECHStanford UnivStanford, CA 94305

Bureau Naval PersonnelAttn: PERS A3Arlington AnnexWashington, DC 20370

225

(2'of 4)

Chief Naval ResearchPsychological SciencesCode 450 Navy DeptArlington, IA 22217

Chief Naval MaterialM4 J' 031MWashington, DC 20360

USIF Human Res LabPersonnel Rsch DivLackland AFB, TX 78236

Human ResOurces ResellOrgztn Div 6 AviationP. O. Box 428Fort Rucker, AL 3630

.National Science FoundationAttn: Dr. Henry S. Odbert1800 G.St-NWWashington, DC 20550

Chief Naval OperationsAttn: M. k. MalehornOP-14C Navy DeptWashfngton,' DC 20350

USAF Human Res LabAFHRL-FTFlying Trng DivWilliam AFB, AZ 85224

USAF Human Res LabAFHRLLTTTech Tng Div ,Lowry AFB, CO 80230

CommanderNAVORDSYSCMDCode 03Washington, DC 20360


HUMAN FACTORS DISTRIBUTION,LIST

USAF Human Res LabAFH111,- AS

Advance Systems DivWright-Patterson AFB, OH 45433

U.S. Army Research InstituteResearch LabCommonwealth Bldg (Rm 239)1320 Wilson BlvdArlington, VA 22209

Commanding OfficerPERS-TRNG RESH DEV LABSan Diego, CA 92152

CNETSCode 'N-241NAS Pensacola, FL 32508

Naval Education & Training SupportCenter (Pacific)

Code N1Fleet Station Post Office BldgSan Diego, Ca 92132

Mr. Sam CanipbellGrumman Aerospace CorporationPlant 47Bethpage, L. I. , New York 11714

Mr. Robert E. CowardChief, Instructional Technology DivADC/DOTIEnt AFB CO 80912

Commanding Officer 88Naval Training Equipment CenterOrlando, FL 32813

Defense Documentation Center 12Cameron StationAlexandria, VA 22314

(4 of 4)

NAV PERSQNNEL RESCH ANDDEVELOPMENT LABORATORYAttn: LibraryBldg 200 Rm 3307 WNYWashington, DC 20390

CNETSCode N-2, Bldg 45(Dr. Charles Havens)NAS 'Pensacola, FL 32508

O

Dr. Mark A. HofmannU. S. Army Aeromedical Research LabP. O. Box 577Fort Rucker, Alabama 36360

MT. Harold KcittmannASD/SMSEWright-Pattersokkir Force BaseOhio 45433

Mr. Arthur DotyWright 'Patterson AFB (ENCT)Dayton, Ohio

,

C OMNAVAIRPACBox 1210 USNASATTN: Code 316North IslandSan Diego, CA' 92135

The Field Artillery SchoolTarget Acquisition Department(Mr. Eugene C. Rogers)Ft, Sill, Oklahoma 73503

USAF Human'Res LabAFHRL/OROccupational & Manpower Research DivLackland AFB TX 78236

226

1

NAVTRA2QUIPCEN 73-C-0128-1

HUMAN.FACTORS DISTRIBUTION 1,IST

HQS-AF SYS CMD D1SLOfc Scientific RschAndrews AFBWashington, DC 20331

CoramrnderNAVSUPSYSCMD -

Code 03Washingtofi; DC 20360

USAF Human- Res 'LabAFHRL /SMomputational Sciences' DivLaolcland-'AFB TX'7823.5

Haman Res Bach OrgDiv No. 1 Sys Oper200 N.' Washington ST .

Alexandria, VA 22314

CO NAV MISSILE CTRAttn: Hd Human FactorsEngrg Br .Point Mugu, CA -93042

C6mmanding.OfficerNavyiVIedicalNeuropsychiatric Resell UnitSan Diego, CA 92152

CO NAVAIR& TECH TRNGNAS MemphisAttn: Dr. G. D. MayoHd Research BrMillington, TN 38054

Dir DEF RSCH-ENGRGARPABehavioral Science DivAttn: 'LCOL A. W. KiblerWashington, DC 20301

(3 of 4)

3*

r 227

,

Scientific TechnicalInformation OfficeNASAWashington, DC 20546

rCH RSCH.OFC.Ofc Dep Ch of Staff for PetsDept of Army -,

Washington, DC 20310

Chief of Naval Technical TrainingNAS Memphis 75Attn: Code 34 Dr. HardingMillington, TN 3864

Dr. John MeyerHDQTRS Air Training, CommandXP TRandolph AFB, TX'78148

.Joseph J. CowanCh PERS RSCH BRANCHUSCG HQ P0-1 STA 3-12'400 Seventh St'. S. W.Washingtoni DC 20590

Executive Editor .

Psycho AbstractsAmerican Psych Assoc1200 17th St NW'Washington, DC 20036

Die. Ralph R. CanterDir MIL MANPWR RSCHOSAD M-RA MR-UPentagdn RM 3D960Washingt4m, DC 20301

Dr. John W. WeiszDir Human Engrg LabUSA Aberdeen Rsch Devel CtrAberdeen Proving GroundsMD 21005

3,

Naval Training EquiRment'Center, Orlandd, Fla ...DOCUMENT, RESUME. JR' 002 541. Nickeriom, R. S::...

Documents

Transcript of Naval Training EquiRment'Center, Orlandd, Fla ...DOCUMENT, RESUME. JR' 002 541. Nickeriom, R. S::...