Biased About Biases: The Theory of the Handicapped Mind in the...

© Robert R. Hoffman All rights reserved

Biased About Biases:

The "Theory of the Handicapped Mind" in the Psychology of Intelligence Analysis

Robert R. Hoffman

Originally Presented at

The Annual Meeting of the Human Factors and Ergonomics Society

Sept 2004


Volumes of volumes!

  Kahneman, D., & Tversky, A. (Eds.) (2000). Values, choices, and frames. Cambridge: Cambridge University Press.

  Arkes, H. R., & Hammond, K. R. (Eds.) (1986). Judgment and decision making. Cambridge: Cambridge University Press.

  Juslin, P., & Montgomery, H. (1999). Judgment and decision making. Mahwah, NJ: Erlbaum.

  Plous, S. (1993). The psychology of judgment and decision making. New York: McGraw-Hill.


Bushels of biases!

•  Perception •  Memory

•  Logical Reasoning •  Probabilistic reasoning

40+ in the literature, and counting!


Reasoning about probability & uncertainty Availability bias Anchoring and adjustment Representativeness (or Familiarity) Salience Bias The Gambler' fallacy Framing effects Recency Bias

Reasoning about Cause-effect relations Illusory correlation Representativeness Framing effects


Logical Reasoning

Recency Bias Salience Bias Reductive bias Confirmation bias Overconfidence bias Consistency bias Hindsight bias Attentional limitation (1 or 2 hypotheses at a time) Biases in syllogistic reasoning (affirming the consequent, denying the antecedent)

Reasoning about costs & benefits Loss aversion Risk-aversion or Over- conservatism The endowment bias Framing effects


And in the "popular" press

Ariley, D. (2008). Predictably irrational. New York: Harper Collins.

Brafman, O., and Brafman, R. (2008). Sway: the irresistible pull of irrational behavior. New York: Doubleday.

Burton, R.A. On Being Certain: Believing You Are Right Even When You’re Not. New York: St. Martin’s Press, 2008.

Fine, C. (2006). A mind of its own: How your brain distorts and deceives. Neew York: W.W. Norton.

Freedman, D. (2010). Wrong: Why Experts Keep Failing Us-And How to Know When Not to Trust Them. Boston, MA: Little Brown & Company.


... and more ...

Kaplan, M. and Kaplan, E. (3009). Bozo sapiens. New York: Bloomsbury Press.

Pohl, R. F. (Ed.) (2004). Cognitive illusions. New York: Psychology Press.

Russo, E.I and Schoemaker, P.J.H. (1990). Decision traps. New York: Simon and Schuster.

Taleb, N.M. (2008). Fooled by Randomness: The Hidden Role of Chance in Life and in the Markets, 2nd revised edition. New York: Random House.

van Hecke, M.L. (2007). Blind Spots: Why smart people do dumb things. Amherst, NY: Prometheus Books.

And the reviews are in! All of these are rated "highly entertaining"

by journalist J. Jones of the Jones Newsletter (who is rational)


This literature has had a wide impact

Richards Heuer (1999)

The psychology of intelligence analysis

And recently:

Cooper, J. (2005). “Curing Analytic Pathologies: Pathways to Improved Intelligence Analysis.”

Washington, DC: Center for the Study of Intelligence.


The Theory of

“The Handicapped Mind”

“Weaknesses and biases [are] inherent in human thinking processes“ (Heuer, Ch. 1).

The more instances a person can recall of a phenomenon, the more probable that phenomenon seems to be . . . unrelated to the actual probability. . . ” (Heuer, Ch. 1).


“Simultaneous evaluation of multiple, competing hypotheses is very difficult to do. To retain three to five or even seven hypotheses in working memory and note how each item of information fits into each hypothesis is beyond the mental capabilities of most people.” (Heuer, Ch. 8).

“Psychological research into how people go about generating hypotheses shows that people are actually rather poor at thinking of all the possibilities.” (Heuer, Ch. 8).


Richards Heuer would be the first to insist that we be systematic and critical in a retrospective.

What light has been shed by studies of expertise and naturalistic decision making since 1980s?


What is a “bias?”

Statistical definition

  Deviation of a value from a statistical estimate   Systematic error introduced in sampling by

encouraging a particular outcome

This fits with how the word is used in the laboratory research (e.g., probability puzzles).



Social-psychological definition

  An inclination of temperament or outlook, especially a highly personal and unreasoned distortion of judgment.



PIA's definition

A cognitive bias does not result from any emotional or intellectual predisposition toward a certain judgment, but rather from subconscious mental procedures for processing information. A cognitive bias is a mental error that is consistent and predictable. (Ch. 9).

- Rules out the social-psychological aspects.

- Places the bias in the reasoning strategy/process.


The Theory of the Handicapped Mind



Cognitivist definition

The view is that bias is not a matter of temperament but a consequence of natural or inherent tendencies of the human cognitive faculty.

The assumption is that judgment is reasoned, it is just bad reasoning.


What this means, however....

The fact that a reasoning sequence gives rise to wrong answers it not in itself sufficient to show that there is “bias” in a reasoning sequence.


PIA’s Example (Ch.9)

“ The apparent distance of an object is determined in part by its clarity. . .”

So . . . . . .

Even depth perception based on visual information possessing some degree of ecological validity can involve “bias.”

Where does this end? ANYTHING can be seen as being biased!

Bounded rationality Unbounded irrationality!


Example:

Combine the framing bias and the hypothesis limitation

Framing bias:

Our “mind sets” keep us from recognizing our assumptions

Hypothesis limitation:

We are limited in our ability to consider alternative hypotheses

We could not lay out alternatives even if we wanted to because we cannnot recognize our own assumptions!

+


Binds and contradictions!

Both confirmation and disconfirmation are nearly impossible because:

1.  One cannot ever have "all" of the confirming evidence and,

3.  All evidence has only a probabilistic relationship with the hypotheses (making disconfirmation necessarily tentative).


“The human mind has difficulty coping with complicated probabilistic relationships . . . [The analyst should] make a judgment based on an assumption that the available evidence is perfectly accurate and reliable, then reduce the confidence in this judgment by a factor determined by the assessed validity of the information. For example, available evidence may indicate that an event probably (75 percent) will occur, but the analyst cannot be certain that the evidence on which this judgment is based is wholly accurate or reliable. Therefore, the analyst reduces the assessed probability of the event (say, down to 60 percent) to take into account the uncertainty concerning the evidence.” (Heuer, Ch, 10)

If the major premise is true, the last thing we should do is turn analysts into probability jugglers!

Another Bind


Let’s return to the statistical definition:

Systematic error introduced in sampling by encouraging a particular outcome

That is precisely what laboratory research has done!!!

That is “The Bias Bias”!


•  Most biases are revealed and studied in artificial probability estimation tasks -- these are “puzzles,” not problems. •  Dependent measures often assume omniscience as the benchmark, i.e., that people can actually

know the ”objective” frequencies. •  Participants tend to be college students, not domain journeymen or experts. •  Rationality is defined in terms of the rules for deductive and probabilistic inference rather than the “rules of thought”--normative versus descriptive. •  Failure to see reasoning strategies as skills. It is no surprise that people come into the lab “biased!”


Take the definitions and substitute “poorly reasoned” for “unreasoned.” What does it take for a reasoning sequence to be “biased?”

1. There are alternative decisions or decision paths,

2. The reasoner knows that there are alternatives, 3. The reasoner knows what the alternatives are,

4. The reasoner makes the "bad" decision anyway.


“Cognitive biases are similar to optical illusions in that the error remains compelling even when one is fully aware of its nature. Awareness of the bias, by itself, does not produce a more accurate perception. Cognitive biases, therefore, are, exceedingly difficult to overcome.” (Heuer, Ch. 9).

Not quite. Lab work also shows that even with itsy hints, people can mitigate some of the biases.

1. Biases are unavoidable


2. Confirmation bias is a bad thing

People tend to seek evidence that supports their current hypothesis and tend to fail to seek evidence that disconfirms it.

Irrelevant premise: Why would we ever really expect people to not seek confirming evidence?

When does this “bias” show up? •  In lab studies, the “bias” disappears with even a bit of coaching. •  In cognitive field research, disconfirmatory reasoning is often

observed (e.g., thunderstorms)


Disconfirmation is a skill just like any other reasoning strategy:

 Explicit instruction

 Deliberative practice

Is there any empirical base on disconfirmatory reasoning in analytical practice?


People judge the frequency of events in terms of the ease with which examples come to mind.

Example: Read people a list of names of people. If the listed men’s names are all famous ones, then people say that male names were more frequent in the list than female names.

Is this a bias? 1. Assumes memory is non-Aristotelian 2. Go back to the criteria for “bias” - People are not told beforehand what the alternatives are. If they were, they would surely count.

3. Availability Bias


The effect is not what we are led to believe it is.

“I knew it all along.”

4. Hindsight Bias


What would you say is the likelihood that upcoming negotiations with North Korea will result in their complete dismantling of their WMD programs?


A posteriori judgments of event likelihood tend to be higher (lower)for events that had been given low (high) a priori likelihoods but that had (had not) transpired.

Fischoff, B., & Beyth, R. (1975). "I knew it would happen": Remembered probabilities of once –future things. Organizational Behavior and Human Performance, 13, 1-16.

Winman, A., & Juslin, P. (1999) " I was well-calibrated all along:" Assessing accuracy in retrospect. In Juslin, P., & Montgomery, H. (1999). Judgment and decision making. Mahwah, NJ: Erlbaum. pp. 97-120.


There is an interesting effect here (directionality of a posteriori judgments), but it may not be an effect that fits the received definition of cognitive bias:

1). Emotional aspect (desire for self-consistency), which we already ruled out,is a likely explanation.

2). Memory failure (Why expect people to remember minutae of their life, i.e.,probability estimates given 6 months earlier?


The hindsight bias effect shows in artificial probability estimation tasks which assume omniscience as the benchmark.

If you gave probability estimates for each of 15 possible situation outcomes, would you naturally think of remembering them so that you could reconstruct them (accurately!) six months later?


• People are said to "act as if they knew the correct answer all along”--- but do they?

• The critical comparison is between one group frequency % of people who know that Bulgariahas a lower mortality rate than Egypt and another group frequency

% of people who are told that Bulgaria has a lower mortality rate than Egypt, and are then asked to estimate their confidence prior to having been told the correct answer.

• The conclusion does not really apply to individual minds! • And worse still, do the researchers bother to ask what the participants are

thinking?

Another reason for caution:

Within vs. between designs


When you look at experts who are engaged in their familiar tasks. . . .

At best, one must say that the evidence for hindsight bias is mixed.

Christensen-Szalanski, J. J., & Willham, C. F. (1991). The hindsight bias: A meta-analysis. Organizational Behavior and Human Decision Processes, 48, 147-168.


What people do engage in, all the time, is

hindsight analysis

With more information one reconsiders previous hypotheses.

That is rational!

It is rational for people to change their minds!

"With the benefit of hindsight, factors previously considered relevant may become irrelevant, and factors previously thought to have little relevance may be seen as determinative" (Heuer, Ch., 9)

The hindsight “bias” effect is a

particular laboratory effect


5. Reductive Bias

Mental models tend to simplify

Things that are . . . Are seen as . . . nonlinear linear dyamic static multiply caused singly caused dependent independent simultaneous sequential heterogeneous homogeneous conditional universal


These are the “dimensions of difficulty” that make problems hard, even for experts.

They typify the challenges for analysts!

Feltovich, P.J., Coulson, R.L., Spiro, R.J (2001). Learners’ (mis)understanding of important and difficult concepts: A challenge for smart machines in education. In K. D. Forbus & P.J. Feltovich (Eds.), Smart machines in education: The coming revolution in educational technology (pp. 349-375 ). Menlo Park, CA: AAAI/MIT Press.


But this is not bias! It is an epiphenomenon of learning!

  At any given time, our knowledge of anything must be incomplete (even in domains of expertise).

  Concept formation always starts simple,and works from there (the simplicity principle).

  Hey, gang, we have known about assimilation and accommodation for decades!


Not everything boils down to logic!

The probability that a physician continued to smoke was directly related to the distance of the physician's specialty from the lungs.

6. Another aspect of the bias bias

The tendency to ignore the social-psychological definition


People do not have much intuitive feel for how large a sample has to be before they can draw valid conclusions from it. We make the mistake of treating small samples as though they were large ones. (Heuer, Ch. 7)

Is it "intuitive feel" ( = voodoo psychology) or is it a matter of skilled practice?

Ahem, Jim Shanteau showed that this classic Tversky & Kaheneman finding doesnot replicate for expert statisticians! They even disagreed with the "correct" answers in the T&K study

7. Law of small numbers


You can fool some of the people some of the time,

And you can fool some of the people some of the time,

But you can’t fool some of the people some of the time.

(Who else? - Groucho!)


Bounded rationality Unbounded irrationality!

Heuer’s Plea

Analysts need tools that help them avoid or overcome all of the cognitive biases and traps--Selective perception, confirmation bias, fixation, “mental ruts,” the whole enchilada.


Focus on biases

You build models of biases

+ You use those to shape the tools!


If humans are so limited and biased, how come it is always humans who make it possible to overcome error, recover from automation surprises, and build work-arounds?

Nothing would get done without the human abilities to perceive, reason, and collaborate.

Flach, J., & Hoffman, R. R. (2003/January-February). The limitations of limitations. IEEE Intelligent Systems. (Pp. 94-97).


An alternative view

•  Humans have characteristics, but these need not be seen as limitations or biases.

•  The human headbone is the only “machine” capable of dealing with problems at the level of knowledge, meaning, and context.

•  The "semsemaking" approach, and Data-Frame Theory


An alternative approach

Build models across the proficiency span (initiate, apprentice, journeyman, expert)

Use those to inform the creation of tools that:

 Amplify and extend the human ability to perceive, reason, collaborate,

 Amplify and extend the human ability to achieve and exercise expertise.

www.ihmc.us/users/rhoffman/main www.ihmc.us

Thank you!


Biased About Biases: The Theory of the Handicapped Mind in the...

Documents

Transcript of Biased About Biases: The Theory of the Handicapped Mind in the...