Ph.D. in Political Science - Comparative and European PoliticsAcademic year 2012-2013Part 2 Experimental Design

EXPERIMENTAL METHODS IN POLITICAL AND SOCIAL SCIENCES Alessandro Innocenti

alessandroinnocenti@live.it

OUTLINEPart 1 Laboratory Methods To provide a basic introduction to experimental methodology both from a theoretical and

an empirical point of view. Part 2 Experimental Design To learn how to design an experiment and to understand that experiments in

political sciences share many features from cognitive and experimental economics.

 Part 3 Applied ExperimentsTo understand the differences between different kinds of experimental designs by

discussing weaknesses and strengths of some experimental papers and the specificities of their designs.

1

Part 2 Experimental DesignBIASES IN JUDGMENT

  “People rely on heuristic principles which reduce the complex tasks of assessing probabilities and predicting values to simpler judgmental operations. In general, these heuristics are quite useful, but sometimes they lead to severe and systematic errors”

(Tversky and Kahneman 1974)  CONFIRMATION BIAS

Once individuals devise a strong hypothesis they will tend to misinterpret or even misread new information unfavorable to this hypothesis

Also production of treatment effects: when a researcher believes an hypothesis is true, he often produces a biased sample of evidence that reinforces his or her belief (unconsciously?)

Consequence is obvious: confirmation bias inhibit learning whether one’s underlying belief is false

But fresh thinkers may be better at seeing solutions to problems than people who have meditated at length on the problems, because the fresh thinkers are not overwhelmed by the “interference” of old hypotheses.

2

Correlated phenomenaFALSE CONSENSUS

People use their own tastes and beliefs as information in guessing what others like and believeApplication: to put in other people’s shoes is not useful to find focal points

HINDSIGHT BIASCurrent recollections of past judgments tend to be biased by what actually happened since then

WYSIATI RULEwhat you see is all there is

We often fail to allow for the possibility that evidence that should be critical to our judgnebt is missin (what you see is all there is)Applications

Jumping to conclusions on the basis of limited evidence The halo effect judgments of a person’s character can be influenced by one's overall

impression of him or her (attractiveness) Overconfidence. The WYSIATI rule implies that neither the quantity or the quality of

the evidence counts for much in subjective confidence

  

3

AN ILLUSTRATIVE EXPERIMENT

  Martin Jones and Robert Sugden “Positive confirmation bias in the acquisition of information”,Theory and Decision, 50, 2001, 59-99

 Positive confirmation bias: tendency, when testing an existing belief, to search for evidence which could confirm that belief, rather than for evidence which could disconfirm it

 Wason’s (1968) selection task

 Four double-sided cards. Subjects are told that each card has a letter on one side and a number on the other, but they can see only the upper faces of the four cards

 Four cards uncovered show ‘A’, ‘D’, ‘4’ and ‘7’

Each subject is asked to consider the following rule, as applied to the four cards: ‘If a card has a vowel on one side, then it has an even number on the other side’

 Instruction: ‘Your task is to say which of the cards you need to turn over to find out whether the rule is true or false’

  

4

 Two most common responses ‘A’ card alone ‘A’ and ‘4’ cards in combination

 The correct answer to the question posed is, of course, the combination of ‘A’ and ‘7’.

The frequently-chosen ‘4’ card can provide no information which is relevant to the issue of whether the rule is true or false The ‘A’ and ‘4’ cards are the ones that are capable of providing evidence which confirms the rule: by turning over either of these cards, the subject may find a card with a vowel on one side and an even number on the other In contrast, the ‘7’ card can only disconfirm the rule (i.e. by revealing a card which has a vowel on one side but not an even number on the other) In this sense, the evidence from the selection task can be interpreted as consistent with positive confirmation bias

  

5

Criticism

The original selection task was formulated in highly abstract terms 

Counterargument

Correct response might be facilitated by adding thematic content to the task, i.e. by providing a cover story which accounts for the statement and gives some point to the selection task

  Jones and Sudgen’s design

  Subjects have to pay a fixed cost per card turned over

After they have made this choice, the chosen cards are turned over

Then they make the judgment that the statement is ‘true’ or ‘false’

Finally the remaining cards are turned over and they receive a fixed reward if and only if their judgment was in fact correct

  

  

6

Experiment carried out at the University of East Anglia in Norwich  120 students recruited on the campus (wide range of courses) Computerized experiment No communication between subjects

Each task is presented by means of a sequence of six screens The screen presents first the cover story, then the statement and finally four cards to

choose Each object has two characteristics, each of which can take one of two values that

correspond with p, p, q, and q (as before vowel and consonant, even and odd) Each subject perform seven different tasks <p, q> or < q, p>, if turned over, is a disconfirmation of the experimental HP <p> <p, q> and <q, p> are confirmations   

  

7

Exemplificative Tasks 1. Relatives . A survey is taken of 100 people in Los Angeles, Seattle, London and Norwich

who have relatives living in other cities. Each person in the survey living in Britain has relatives in Los Angeles or Seattle and each person living in America has relatives in Norwich or London. No one has relatives in more than one city. The details of the survey are written down on report cards by putting the city each person lives in on one side of the card and the city their relatives live in on the other side. A sample of four report cards is selected. Look at whichever cards you wish to test the statement:

[Standard statement] Every person in the sample who lives in London also has a relative who lives in Los Angeles.

[Contraposed statement] Every person in the sample who lives in Seattle also has a relative who lives in Norwich.

 2. Drinkers . Only people over the age of eighteen are allowed to drink alcohol in a pub in

Britain. A survey is carried out of 100 people in a large public house which identifies their age and whether they are drinking alcohol or a soft drink. Each person’s details are put down on a report card with the person’s age on one side and their drinking behaviour on the other. A sample of four report cards is selected. To find out if the four people in the sample are obeying the law, look at whichever cards you wish to test the statement:

[Standard statement] Every person in the sample who is drinking alcohol is also over eighteen.

[Contraposed statement] Every person in the sample who is under eighteen is also drinking a soft drink.

  

  

8

Results

 In favour of the confirming bias hypothesis: 62% of the choices (445/720)<No cards> 18%<p> 14%<p, q> 18% 

Conclusions  Overwhelming evidence that subjects’ information-gathering decisions are

systematically biased in favor of information which is potentially confirming

But behavior seems to have been closer to Bayesian rationality than in many other selection task experiments

Especially the drinkers story facilitates Bayesian rationality (why?)

What is the effect of financial incentives?

Application to economic learning: an agent who repeatedly faces the same set of options might retain the false belief that a particular option was optimal, even after long exposure to evidence which, rationally interpreted, would indicate the contrary

  

9

II. Conceptions of rationality 

What do we mean by rational choice? Lots of formulations, involving assumptions of different strength  Different forms of rationality imply different experiments to test them 

Goal oriented Satisficing behavior Maximizing behavior Ordinal utility maximization Expected utility maximization Subjective expected utility maximization  

Experimental economics reveals the hidden or implicit assumption by showing anomalies in the formulation of rationality

 Consequence: there is a variety of definitions of rational individual

And what about heterogeneity?

  

10

Risk neutral economic man: never buys insurance, but would be willing to pay any finite amount to participate in Petersburg paradox.  Expected utility maximizing man: buys insurance, but ignores sunk costs, and is immune to framing effects.  Almost rational economic man (e.g. prospect theory man) has malleable reference points and probability perceptions, but still has preferences - comfortable with non-utility Allais choices, but doesn’t exhibit preference reversals.  Psychological man doesn’t have preferences, has mental processes. Different frames and contexts, and different choice procedures elicit different processes - So he may sometimes exhibit preference reversals because choosing and pricing elicit different mental procedures.

Neurobiological man: doesn't (even) have a fixed collection of mental processes, in the sense of psychological man. He has biological and chemical processes which influence his behavior. Different blood chemistry leads to different mental processes; e.g. depending on the level of lithium (or Valium or Prozac) in his blood, he makes different decisions (on both routine matters and matters of great consequence - even life and death). An understanding of how chemistry interacts with mental processes has proved to be very useful, for instance in treating depression.

  

11

 CONSTRUCTIVE REACTIONS

 

12

Expected utility theory prospect theory

asymmetric response to price increases, downward-sloping labor supply among cab drivers

Exponential discounting

hyperbolic discounting addition and procrastination

Self-seeking behavior social utilities

trust and reciprocity in financial relationship

Equilibrium processes of equilibration drift effect, automata

ranked preferences constructed preferences

information manipulation in horse race betting, void informational cascades in the artistic markets

Bayesian probability judgments confirmation bias Self-fulfilling expectations in

financial markets, focal points

PART 2 Experimental DesignAPPLICATIONS

 Trust gameTrust game (or investment game):  

Two players are paired off anonymously and respectively named as the sender and the responde

The sender is given a certain amount of money and told that he or she can keep the entire amount or send some or all of it to the responder.

Any money passed from the sender to the responder is tripled by the experimenter and then given to the responder.

The responder can keep the entire amount or give back some or all of it to the sender.

When the sender receives the amount sent back by the responder the game ends.  It measure of the propensities to trust, which is the proportion of the initial endowment sent by the sender, and to reciprocate, which is the ratio between the amount returned and the amount received by the responder.

Backward induction solution: the responder will not send any money back. anticipating the responder’s decision, the sender will not send any money to the responder.

 

  

13

Results from earlier experiments are inconsistent with the conventional game theory prediction.

 

  

14

Table 1. Experimental results on the trust game Berg-Dickhaut-

McCabe (1995) Buchan-Croson-Johnson (2000)

Schwieren-Sutter (2003)

Burks - Carpenter -Verhoogen (2003)

Chaudhuri - Gangadharan

(2003)

Innocenti - Pazienza (2006)

No history

Social history

Only U.S.

All Single role

Both roles, no

prior

Both roles, prior

Double blind procedure

Double blind procedure

Trust

51.6 53.6 49 67 65.7 65.0 65.2 47.3 43.3 42.2

Reciprocity

30.1 40.2 22 37 37.6 43.6 25.9 17.1 17.5 16.2

Trust = Average fraction sent (Amount sent / Initial endowment); Reciprocity = Average fraction returned (Amount sent back/ Amount received)

15

ApplicationsOXYTOCIN AND TRUSTING BEHAVIOR

Michael Kosfeld, Markus Heinrichs, Paul J. Zak, Urs Fischbacher & Ernst Fehr “Oxytocin increases trust in humans“ Nature 2005   In non-human mammals, the neuropeptide oxytocin has a key role in general

behavioural regulation, particularly in positive social interactions.  

Oxytocin receptors are distributed in various brain regions associated with behaviour, including pair bonding, maternal care, sexual behaviour, and the ability to form normal social attachments.

 Thus, oxytocin seems to permit animals to facilitate approach behaviour.

HP.: oxytocin might also promote prosocial approach behaviours (such as trust) in humans.

 Recent neuroscientific finding: neuropeptides cross the blood-brain barrier after intranasal administration  

 16

17

Double-blind study design to compare: - trusting behaviour in a group of subjects who receive a dose of intranasal oxytocin - trusting behaviour in a control group of subjects who receive placebo. Experiment’s object trust game with real monetary stakes (29+29 subjects)

Hypothesis to test: oxytocin increases the trusting behaviour of investors the investors in the oxytocin group will show higher money transfers than those in the placebo group.

Two treatments

standard trust game risk trust game

 In B the investor faced the same choices as in the trust game but in which a random mechanism, not the trustee's decision, determined the investor's risk.

The random mechanism in the risk experiment replicated the trustees' decisions in the trust experiment.

 Therefore, the investors faced exactly the same risk as in the trust experiment

However, their transfer decisions were not embedded in a social interaction because there were no trustees in the risk experiment.

 

18

Experimental design

194 male students (mean age s.d., 22.0 3.4 yr) from different universities in Zurich 128 participants in the trust experiment and 66 subjects participated in the risk

experiment Exclusion criteria: medical or psychiatric illness, medication, smoking, drug or alcohol

abuse Subjects were instructed to abstain from food and drink (other than water) for 2 h

before the experiment, and from alcohol, smoking and caffeine for 24 h before the experiment

Participants were informed at the time of recruitment that the experiment would evaluate the effects of a hormone on decision making

16 individuals out of the original sample of 194 were excluded because of incorrect substance administration (7 in the trust experiment, 5 in the risk experiment) or their stated disbelief that the opponent in the trust game was actually a human being (4 participants)

Subjects received a single intranasal dose of 24 IU oxytocin (Syntocinon-Spray, Novartis; 3 puffs per nostril, each with 4 IU oxytocin) or placebo 50 min before the start of the experiment

Subjects were randomly assigned to the oxytocin or placebo group In order to avoid any subjective substance effects (for example, olfactory effects)

other than those caused by oxytocin, the placebo contained all inactive ingredients except for the neuropeptide.

19

20

Relative frequency of investors' average transfers in oxytocin (filled bars) and placebo (open bars) groups in the trust experiment (n = 58). Subjects given oxytocin show significantly higher transfer levels.

The investors' average transfer is 17% higher in the oxytocin group (Mann-Whitney U-test; z = -1.897, P = 0.029, one-sided).

Median transfer: 10 MU (oxytocin group) > 8 MU (placebo group)

 

21

Relative frequency of investors' average transfers in oxytocin (filled bars) and placebo (open bars) groups in the risk experiment (n = 61). Subjects in the oxytocin and the placebo group show statistically identical transfer levels.

Median transfer: 8 MU (in both groups)

Average transfer 7.5 MU (in both groups) (Mann-Whitney U-test; z = 0.022, P = 0.983; two-sided test, n = 31 in oxytocin group, n = 30 in placebo group).

Conclusion oxytocin increases the investors' transfer levels in the trust experiment but not in the riskexperiment oxytocin specifically affects trust in interpersonal interactions. 

Explanationsa) oxytocin causes a general increase in prosocial inclinationsOxytocin should affect not only the prosocial behaviour of the investors but also that of the trustees.But trustees given oxytocin do not show more trustworthy behaviour. At every positive transfer level (4, 8 or 12 MU), their back transfers are statistically indistinguishable from those of placebo trustees (Mann Whitney U-tests; P > 0.243, two-sided tests for each positive transfer level). b) oxytocin does not increase the general inclination to behave prosocially. Rather, oxytocin specifically affects the trusting behaviour of investors.  c) effect of subjects' beliefs. Oxytocin might render subjects more optimistic about the likelihood of a good outcome. In order to address this question, we measured the investor's subjective expectation about the trustee's back transfer after every transfer decision. A Mann-Whitney U-test indicates that these expectations do not differ significantly between oxytocin and placebo groups at every feasible positive transfer level  d) oxytocin helps subjects to overcome their betrayal aversion in social interactions. This explanation is consistent with the differing effects of oxytocin across the trust and the risk experiments, and is further supported by the fact that investors faced a considerable betrayal risk.

22