And The Link With Theory Of MindCLASS 8

STRATEGIC THINKING IN GAMES AND MARKETS

1

FIRST: THE INSPECTION GAME

2

UNCERTAINTY:• Employee does not know what employer will do

• Employer does not know what employee will do

3

PREDICTING UNDER THIS UNCERTAINTY

• This skill is extremely important in modern economic life

• Yet we don’t understand why (many) humans are good at it

• Humans seem to apply the right “intuition”

• Who?• What is this intuition?

4

NASH

• Subjects do not play the equilibrium strategies from the one-shot game

• So, forecasting what the opponent will do cannot be based on this prediction

• (Of course, you can try to figure out the Nash equilibrium of the multi-stage game, but there are lots, lots,...)

5

PSYCHOLOGISTS TALK ABOUT: THEORY OF MIND

• “Theory of Mind” (or “mentalizing”) is the ability to recognize and understand intentions or goal-directness in patterns in one’s environment

• … as opposed to mere expression of physical laws

• Intention:

• Malevolent • Benevolent

• Involves (“new”) regions of the cortex, distinct from formal mathematical and probabilistic brain regions

6

Example: The intentional stance of a moving object

Base Situation

New Situation 1Physics OK

New Situation 2:?!

(Uller, Nichols, Cognition, 2000)

7

REMARKS

• One-year old infant (and apes) may be able to recognize goal-directness but cannot use it to its advantage (E.g., chocolate-in-drawer experiment)

• But:

• What does Theory of Mind mean formally?

• Enter Economics…

8

THEORY OF MIND ENGAGES PART OF THE “SOCIAL BRAIN”

9

IN GAMES: • Recently, Theory of Mind brain

regions have been found to be activated also when playing strategic games

• … such as rock, paper, scissors (Gallager-Frith, 2003: contrast between playing against human and against simple computer-based rule)

10

BUT…

• These are simple contrasts (affectionately called “blobology”)

• What computations are involved?

Hampton, Bossaerts, O’Doherty, PNAS, 2008

11

MATHEMATICS: REINFORCEMENT LEARNING

Prediction ErrorAction Value

Logit Model for Probability of Action a

12

MATHEMATICS: FICTITIOUS PLAY

Stochastic (Logit) Best Response Given Beliefs

Learning of Opponent’s

Strategy

Prediction Error

13

MATHEMATICS: “INFLUENCE”Taylor Expansion of

StrategiesGetting

Opponent’s Beliefs

Learning From Action Of Opponent (“Fictitious Play”) Predicted Change In Opponent

Actions (“Influence”)14

THE REWARD ERROR FROM THE “INFLUENCE” PREDICTION IS ENCODED

IN PCC:

Hampton, ea, PNAS 2008

15

(HERE IS THE FICTITIOUS PLAY REWARD PREDICTION ERROR:)

Striatum: Traditionally involved in reinforcement learning

(Based on “dopamine”)

16

DIFFERENCE WITH CLASSICAL GAME THEORY:

• (Static) Nash equilibrium predicts:

• Employee works with probability 1/5; shirks with probability 4/5• Employer inspects with probability 1/2

• No need to predict effect of own actions on opponent’s beliefs and actions!

17

RELATED TO HIGHER-ORDER THINKING

NO correlation with accuracy in the calculation task (r = .009, P >.5)Coricelli and Nagel (2009)

18

STAG-HUNT

19

therefore, there is no need to infer the other’sstrategy. The players in the ToM model assumethe other player might change his/her strategyand optimize their own after each move. Wecalculated the evidence using k1 ! 1, . . . , 6for the fixed-strategy model and K1 ! 1, . . . , 6for the ToM model; i.e., we used 12 models intotal. This is because the difference betweensuccessive value functions becomes smallerwith increasing order, and the value functionssaturate at k1 ! 6 (supplemental Fig. S1, avail-able at www.jneurosci.org as supplementalmaterial). In the stag hunt game, the agent’sbehavior depends on the order of strategieskcom; the first-, third-, or fifth-order and thepositions of other agents. The average cooper-ation rates simulated with two agents withmatched sophistication level were 9.6, 20.8,and 83.9% for the fist-, third-, and fifth-orderlevels from all possible sets of initial states. In-terestingly, the third- and fifth-order strategiesshow quite different behaviors (supplementalFig. S2, available at www.jneurosci.org as sup-plemental material). For the model comparison,we used the true values of the strategies of com-puter agent; i.e., k1 ! {1, 3, 5}. Supplemental Fig-ure 3 (available at www.jneurosci.org assupplemental material) shows the results ofBayesian model selection (Stephan et al., 2009).Over model space of the ToM model, we inferredthat the sophistication level of the subjects is K1 !5. This is reasonable since the subjects did nothave to use the strategies higher than k1 !6, giventhat the computer agent’s strategies never ex-ceeded five.

Under the inferred (or best) model with K1

! 5, we optimized two model parameters. Oneis a subjective utility parameter ", which scalesthe utility of a rabbit relative to the utility of astag: if the subjects overestimate a rabbit’s utility, this parameter is largerand induces more competitive behaviors; if they underestimate a rabbit’sutility, this parameter is smaller and leads to cooperation. The maximumlikelihood estimation showed that the optimal value was in the range of0.39 # " # 0.43 for each subject (mean " SD ! 0.41 " 0.01) and " !0.41 for all subjects. The other parameter is the forgetting parameter $(Eq. 1), and it was in the range of 0.51 # " # 0.84 for each subject(mean " SD ! 0.68 " 0.11) and estimated as 0.75 from all subjects’ data.To test the efficacy of the forgetting parameter, we compared the logevidences of the ToM model with and without forgetting effect and ver-ified that the ToM model with forgetting effect shows significantly betterfit with the behavioral data (data not shown). To calculate the regressionfunctions for the imaging analysis, we used the optimal utility parameter" ! 0.4 for all subjects and optimized the forgetting parameter k for eachsubject.

The recursive or hierarchical approaches to multiplayer games havebeen adopted in behavioral economics (Stahl and Wilson, 1995; Costa-Gomes et al., 2001) in which individual decision strategies systematicallyexploit embedded levels of inference. The sophistication that we specifi-cally address here pertains to the recursive representation of the otherplayer’s intentions, and this is distinct from the number of “thinkingsteps” in other models (Camerer et al., 2004), which corresponds to thedepth of tree search.

In this article, we wanted to establish the neuronal correlates of ourBayes optimal model of cooperative play. However, it is important tonote that our conclusions are conditioned upon the model that we use. Itis possible that our subjects used different models, in particular belieflearning models with a dynamic game (Fudenberg and Levine, 1998). Wesuggest that subsequent work might use Bayesian model comparison toadjudicate among Bayes optimal models and a family of heuristic ap-

proximators (Gigerenzer et al., 1999). Such an evaluation in terms of theevidence for alternative models from both behavioral and physiological(fMRI) data is clearly an important avenue to pursue.

fMRI acquisition. A Siemens 3T Trio whole-body scanner with stan-dard transmit–receive head coil was used to acquire functional data witha single-shot gradient echo isotropic high-resolution echo-planar imag-ing (EPI) sequence (matrix size: 128 # 128; field of view: 192 # 192mm 2; in-plane resolution: 1.5 # 1.5 mm 2; 40 slices with interleavedacquisition; slice thickness: 1.5 mm with no gap between slices; echotime: 30 ms; asymmetric echo shifted forward by 26 phase-encodinglines; acquisition time per slice: 68 ms; reaction time: 2720 ms). Thenumber of volumes acquired depended on the behavior of the subject.The mean number of volumes of each session was 318, giving a totalexperiment time of $14.4 min. A high-resolution T1-weighted struc-tural scan was obtained for each subject (1 mm isotropic resolutionthree-dimensional modified driven equilibrium Fourier transformation)and coregistered to the subject’s mean EPI. The mean of all individualstructural images permitted the anatomical localization of the functionalactivations at the group level.

fMRI analysis. Statistical parametric mapping (SPM) with SPM5 software(Wellcome Trust Centre for Neuroimaging, UCL) was used to preprocess allfMRI data, which included spatial realignment, normalization, and smooth-ing. To control for motion, all functional volumes were realigned to themean volume. Images were spatially normalized to a standard space Mon-treal Neurological Institute template with a resample voxel size of 2 # 2 # 2mm and smoothed using a Gaussian kernel with an isotropic full width athalf maximum of 8 mm. In addition, high-pass temporal filtering with acutoff of 128 s was applied to remove low-frequency drifts in signal, andglobal changes were removed by proportional scaling.

Figure 1. Stag hunt game and theory of mind model. A, Two players (hunters), a subject (green circle) and a computer agent(blue circle), try to catch prey: a mobile stag (big square) or two stationary rabbits (small squares), in a maze. From an arbitraryinitial state, they move to the adjacent states in sequential manner in each trial; the stag moves first, the subject moves, and thenthe computer agent moves. The players can capture either a small payoff (rabbit hunt; bottom left) or a big payoff (stag hunt,bottom right). Cooperation is necessary to hunt a stag successfully. At the end of each game, both players receive points equal tothe sum of prey and points relating to the remaining time (see Materials and Methods). B, In our theory of mind model, the optimalstrategies differ in the degree of recursion (i.e., sophistication): first-order strategies assume that other players behave randomly,second-order strategies are optimized under the assumption that other players use a first-order strategy, and third-order strate-gies pertain to an assumption that the other player assumes you are using a first-order strategy, and so on. Here, we assume boundsor constraints on the strategies available to each player and their prior expectations about these constraints. C, The subjects changetheir behavior based on the other player’s sophisticated level (k) which was unknown. When the computer agents used fifth-orderstrategies, the rate of cooperative games (stag hunt) out of the total (mean " variance ! 0.405 " 0.034) was significantly higherthan when they used the lower third-order strategies (mean " variance ! 0.089 " 0.012, p % 0.0001) and first-order strategies(mean " variance ! 0.026 " 0.002, p % 0.00001).

10746 • J. Neurosci., August 11, 2010 • 30(32):10744 –10751 Yoshida et al. • Prefrontal Cortex and Cooperation

• Complex stag-hunt game

• Against sophisticated (order k) computer

• E.g., Order 2 = “I think that you think that I will move left”

• (Cooperation, i.e., stag hunt, increases when order k increases)

UNCERTAINTY ABOUT INFERENCE OF COMPUTER’S LEVEL IS ENCODED IN

MPFC (YOSHIDA EA)

20

the other player’s strategy using Bayesian belief learning and thenoptimize their own strategy. Accordingly, we used two principalstatistics for belief inference as parametric regressors. The firstwas the entropy of the belief about the other player’s strategies(Fig. 2C), by which we mean the uncertainty or average surpriseof belief inference. The second was the sophistication level ofsubjects’ strategies (Fig. 2B, green line), which corresponds totheir level of strategic thinking and, implicitly, the expected levelof the other player’s sophistication.

We found that the entropy of belief inference was correlatedwith activity in an anterior part of rostral medial prefrontal(paracingulate) cortex (MPFC), a region consistently identifiedin psychological tasks requiring mentalizing (McCabe et al.,2001; Gallagher et al., 2002) (Fig. 5A). That is, activity was greaterwhen the subjects were more uncertain about the other player’slevel of sophistication.

By contrast, we observed that the level of strategic thinkingcorrelated with the BOLD signal in the left dorsolateral prefrontalcortex (DLPFC), bilateral frontal eye field (FEF) on the superiorfrontal sulcus, and the left superior parietal lobule (SPL) (Fig.5B). Equivalent signals were present in the right DLPFC at thesame threshold but did not pass our cluster extent criterion. Allresults are reported in areas of interest at p ! 0.001 (uncorrected)and k ! 100 (Table 2).

DiscussionIn summary, brain activity in distinct prefrontal regions, includingpart of the classical theory of mind network, expresses a dynamicpattern of activity that correlates strongly with the statistical compo-nents of belief inference.

In the field of ToM in general, previous neuroimaging studies(Frith and Frith, 2003) have found that the anterior MPFC isimportant for representing the mental states of others (McCabeet al., 2001; Gallagher et al., 2002; King-Casas et al., 2008). How-ever, these studies have left unanswered the question as to theprecise computations invoked during the application of ToM(Wolpert et al., 2003; Lee, 2008) and, hence, which of the manysubprocesses might be implemented in classical ToM regions.Indeed, it remains possible that the mere presence of a humanconspecific engages ToM regions and, thus, that the correspond-ing activation might not be functionally related to the executionof the social games per se. Here, we looked specifically at beliefinference, which is likely to be a central (but by no means theonly) component of ToM. The fact that subjects knew they wereplaying a computer suggests that anterior MPFC activity relates(at least in part) to a computational component of belief infer-ence, as described above, and is not purely related to belief aboutthe humanness of the other players.

Recent evidence from the “inspection” game (Hampton et al.,2008) has suggested that ToM in games involves a prediction ofhow the other player changes his/her strategy as a result of one’sown play. In the Hampton et al. (2008) study, neural responses inanterior MPFC related to predicting such changes are notablegiven that game theory has seldom considered “higher-orderthinking” in learning during repeated games, although Bayesianupdating of types has been proposed in reputation formation andteaching (Fudenberg and Levine, 1998; Camerer et al., 2002).Both Hampton’s influence model and our ToM model involvethe learning of the other’s strategy, where the other individual

Figure 5. Statistical parametric maps showing where BOLD activity correlated with discrete components of the model parameters at the time of computer move. A, The uncertainty of beliefinference (the entropy of computer agent’s strategies in Fig. 2 A) shows a significant correlation with the anterior part of the MPFC ("6, 54, 14 mm, Z # 4.76; p ! 0.01 SVC). B, Based on the beliefinference, the subjects optimize the level of their own strategies. The estimated sophistication level of subjects’ strategy from the theory of mind model showed a significant correlation with BOLDresponses in the left DLPFC ("50, 28, 32, Z # 4.26), the bilateral FEF on the superior frontal sulcus ("20, 4, 50, Z # 4.25; 30, 6, 64, Z # 4.17), and the left SPL ("16, "56, 66, Z # 4.22). L, Left;R, right; BA, Brodmann’s area.

Yoshida et al. • Prefrontal Cortex and Cooperation J. Neurosci., August 11, 2010 • 30(32):10744 –10751 • 10749

OWN LEVEL OF K CORRELATES WITH NEURAL ACTIVITY

• Lateral PFC, Superior Parietal Lobule, Superior Frontal Sulcus

21the other player’s strategy using Bayesian belief learning and thenoptimize their own strategy. Accordingly, we used two principalstatistics for belief inference as parametric regressors. The firstwas the entropy of the belief about the other player’s strategies(Fig. 2C), by which we mean the uncertainty or average surpriseof belief inference. The second was the sophistication level ofsubjects’ strategies (Fig. 2B, green line), which corresponds totheir level of strategic thinking and, implicitly, the expected levelof the other player’s sophistication.

We found that the entropy of belief inference was correlatedwith activity in an anterior part of rostral medial prefrontal(paracingulate) cortex (MPFC), a region consistently identifiedin psychological tasks requiring mentalizing (McCabe et al.,2001; Gallagher et al., 2002) (Fig. 5A). That is, activity was greaterwhen the subjects were more uncertain about the other player’slevel of sophistication.

By contrast, we observed that the level of strategic thinkingcorrelated with the BOLD signal in the left dorsolateral prefrontalcortex (DLPFC), bilateral frontal eye field (FEF) on the superiorfrontal sulcus, and the left superior parietal lobule (SPL) (Fig.5B). Equivalent signals were present in the right DLPFC at thesame threshold but did not pass our cluster extent criterion. Allresults are reported in areas of interest at p ! 0.001 (uncorrected)and k ! 100 (Table 2).

DiscussionIn summary, brain activity in distinct prefrontal regions, includingpart of the classical theory of mind network, expresses a dynamicpattern of activity that correlates strongly with the statistical compo-nents of belief inference.

In the field of ToM in general, previous neuroimaging studies(Frith and Frith, 2003) have found that the anterior MPFC isimportant for representing the mental states of others (McCabeet al., 2001; Gallagher et al., 2002; King-Casas et al., 2008). How-ever, these studies have left unanswered the question as to theprecise computations invoked during the application of ToM(Wolpert et al., 2003; Lee, 2008) and, hence, which of the manysubprocesses might be implemented in classical ToM regions.Indeed, it remains possible that the mere presence of a humanconspecific engages ToM regions and, thus, that the correspond-ing activation might not be functionally related to the executionof the social games per se. Here, we looked specifically at beliefinference, which is likely to be a central (but by no means theonly) component of ToM. The fact that subjects knew they wereplaying a computer suggests that anterior MPFC activity relates(at least in part) to a computational component of belief infer-ence, as described above, and is not purely related to belief aboutthe humanness of the other players.

Recent evidence from the “inspection” game (Hampton et al.,2008) has suggested that ToM in games involves a prediction ofhow the other player changes his/her strategy as a result of one’sown play. In the Hampton et al. (2008) study, neural responses inanterior MPFC related to predicting such changes are notablegiven that game theory has seldom considered “higher-orderthinking” in learning during repeated games, although Bayesianupdating of types has been proposed in reputation formation andteaching (Fudenberg and Levine, 1998; Camerer et al., 2002).Both Hampton’s influence model and our ToM model involvethe learning of the other’s strategy, where the other individual

Figure 5. Statistical parametric maps showing where BOLD activity correlated with discrete components of the model parameters at the time of computer move. A, The uncertainty of beliefinference (the entropy of computer agent’s strategies in Fig. 2 A) shows a significant correlation with the anterior part of the MPFC ("6, 54, 14 mm, Z # 4.76; p ! 0.01 SVC). B, Based on the beliefinference, the subjects optimize the level of their own strategies. The estimated sophistication level of subjects’ strategy from the theory of mind model showed a significant correlation with BOLDresponses in the left DLPFC ("50, 28, 32, Z # 4.26), the bilateral FEF on the superior frontal sulcus ("20, 4, 50, Z # 4.25; 30, 6, 64, Z # 4.17), and the left SPL ("16, "56, 66, Z # 4.22). L, Left;R, right; BA, Brodmann’s area.

Yoshida et al. • Prefrontal Cortex and Cooperation J. Neurosci., August 11, 2010 • 30(32):10744 –10751 • 10749

COMPARING AUTISTIC PATIENTS WITH CONTROLS

• Patients cannot track changes in sophistication (order k) of computer

• Model that assumes fixed k for computer fits choices better than model that tracks changes in k (“ToM model”)

• Equivalently: “forgetting rate” (discarding old observations) is higher for controls

22

calculated the marginalized posterior probabilities of the fixed strategymodel and the ToM model by accumulating evidence for different mod-els specified by different k-level and bound K; i.e., 6 models each, as!k

p!MkFIX" and !

kp!Mk

TOM".

To estimate the cognitive flexibility, we used the average forgetting effect,which is calculated as the weighted sum of the parameter! with the posteriorprobability of each model. The forgetting parameter was optimized for eachToM model with different K, and set at one for the fixed strategy models:

"k

p!MkFIX" " "

k

!Kp!MkTOM".

The average sophistication for the ASC participants was estimated as theweighted sum of parameter k with the model probability of each fixedstrategy model.

ResultsTo first confirm a basic cognitive understanding of the task de-mands, we examined behavioral strategies in the ASC group.Overall behavioral measurements including the average reactiontimes, total earnings, and cooperation rates of the ASC group didnot differ from the control group, providing evidence that anunderstanding of the social task was intact (Chiu et al., 2008;Uddin et al., 2008). The participants in the control group attemptedto catch a stag, i.e., they behaved more cooperatively, when the com-puter agent was more sophisticated, and this tendency was also ob-served for the ASC group on average (Fig. 2B). The behavioralprofiles, however, differed greatly in individuals with ASCs. Thus,while all control participants dealt with both hunting a stag andhunting a rabbit according to the situation (e.g., initial positions),there are extreme participants in the ASC group who never cooper-ate or compete (Fig. 2C). This suggests a behavioral diversity or evenmultiple phenotypes among the ASC participants.

To identify functional abnormalities inthe computational processes involved inthe task, we used the ToM model and thefixed strategy model. The ToM model in-cludes two model parameters characteriz-ing the cognitive processing: one is theupper bound of sophistication (K), whichdefines the capacity of strategic planning,and the other is a forgetting effect, whichcontrols how quickly a player respondsto changes in the other’s sophistication,thereby representing a measure of cogni-tive flexibility. For the fixed strategymodel, as it is assumed that players do notchange their strategy, only the sophistica-tion level (k) is estimated.

First, to compare the behavioral fit ofthese two models, we calculated the loglikelihoods of the models for the controland the ASC participants (Control: ToM#5776, fixed #6060; ASC: ToM #5330,fixed #4937; greater values indicate betterfit). Bayesian model selection based onthese log likelihoods showed that, for thecontrol participants, the ToM model withbelief inference accounted for the behav-ior significantly better than the fixed strat-egy model without belief inference.Conversely, the fixed strategy model ex-plained individual behavior better for 12of 17 participants with ASCs (Fig. 3A). Wealso evaluated the quality of model fitting

with a pseudo-r 2 statistic, defined as (m0 # m)/m0, where m andm0 are the log likelihood of the data under the model and underpurely random choices, respectively. The results showed that theToM model (r 2 $ 0.230) provided a better fit than the fixedstrategy model (0.194) for the controls, while the fixed model(0.262) provided a better fit than the ToM model (0.203) for theASC participants.

We divided the ASC participants into two subgroups based onthe better explained models, and compared their diagnosis andintellectual scores (Fig. 3B). The diagnosis scores (the ADI-R andASDI) were significantly higher for the ASC participants whosebehavior fit better with the fixed strategy model (n $ 12) thanothers with the ToM model (n $ 5), while there was no differenceon the intellectual (IQ) scores. Furthermore, the probability ofthe ToM model, how likely recursive belief inference is used,correlated negatively with individual autistic symptomatology asmeasured by the sum of scores on the ADI-R and the ASDI (Fig.3C). The correlation was not significant ( p $ 0.055) with therather small number of sample size (n $ 14); however, this negativecorrelation was also observed with the ADI-R (n $ 14, r $ #0.45,p $ 0.105) and the ASDI score (n $ 16, r $ #0.44, p $ 0.090)individually. We thereby characterized unobservable computationalprocesses implicitly involved in ToM quantifying the individual abil-ity for recursive belief inference.

In terms of the model parameters, the mean estimated forget-ting effect in the ASC group was significantly higher than that ofthe control group (Fig. 3D). The higher value of the forgettingeffect means that participants are tied to their past strategies, andthis result indicates that the ASC participants had an additionalimpairment in cognitive flexibility. For the sophistication level,

Figure 3. Model-based behavioral results. A, The probability for the ToM model (98.2%) was higher than that for the fixedstrategy model for controls, while the fixed strategy model was dominant (78.6%) in individuals with ASCs. B, Diagnosis measure-ment scores, ADI-R and ASDI, were significantly higher for the ASC participants whose behavior fit better with the fixed strategymodel (n $ 12) than those showing a better fit with the ToM model (n $ 5). C, In the ASC group, the greater the expectation ofrecursive belief inference, the more severe was the autism symptomatology (n $ 14, r $#0.52, p $ 0.055), as measured by thesum of scores on the ADI-R and the ASDI. D, The estimated forgetting parameter for the ASC group (mean % SD $ 0.57 % 0.19)was significantly higher than that for the control group (0.93 % 0.13) ( p & 0.1 ' 10 #6). E, The estimated sophistication for theindividuals with autism showed significant positive correlation with individual IQ scores (left panel: n $ 17, r $ 0.54, p $ 0.026),while there was no correlation for the control participants (right panel: n $ 17, r $ 0.02).

Yoshida et al. • Cooperation and Heterogeneity of the Autistic Mind J. Neurosci., June 30, 2010 • 30(26):8815– 8818 • 8817

(ANALOGOUS EFFECTS IN TRUST GAME FOR BORDERLINE PERSONALITY

DISORDER PATIENTS)

• King Cases ea: When trust is broken in trust game, BPD patients cannot re-establish trust

• Related to different activation in anterior insula (they do not expect trust to go away, but once it does, they don’t know anymore what world they live in

• COMPUTATIONAL (NEURO)PSYCHIATRY

23

TOWARDS “MANIPULATION”

24

• Bhatt ea, PNAS (2010)

• Beyond influence (taking into account that one is learning about a moving target)

• Really “teaching” (belief manipulation)

sophisticated strategic deception also involves the execution ofa long-term strategy, which requires prospective thinking, activegoal maintenance, and cognitive control—suggesting the involve-ment of BA10 and DLPFC.In this game, the “no feedback bargaining task,” two players,

a buyer and a seller, play 60 rounds of a bargaining task (Fig. 1). Atthe beginning of each round the “buyer” is informed of her privatevalue v of a hypothetical object. She is then asked to “suggesta price” to the seller (values and prices are integers, 1–10). Theseller then receives this suggestion and is asked to set a price p. Ifthe seller’s price is less than the private value v (which is knownonly to the buyer), the trade executes and the seller receives p,whereas the buyer receives v ! p, the difference between the pri-vate value and the selling price. If the seller’s price exceeds thebuyer’s value, the trade does not execute and both parties receivenothing. No feedback about whether the trade occurred is pro-vided to either player.The tradeable object has no value to either player if a trade

does not occur. However, if a trade does occur, each player pre-fers a sale price that favors them. Buyers prefer lower prices andsellers prefer higher prices. This misalignment of incentivesimplies that the only equilibrium solution of the one-round ver-sion of this game is for no information transfer to occur (21). Thebuyer should “babble” and send suggestions with no informativerelationship to her private value, and the seller should ignore thissuggestion and set a price of either 5 or 6 (to maximize theexpected revenue). However, this is the mutually optimal solutiononly if both players believe that the other is also playing in equi-librium. That is, babbling is only optimal if the seller is in factignoring buyer suggestions, and ignoring buyer suggestions is onlyoptimal if they contain no meaningful information. In actuality, inthese types of games players’ beliefs about what others are actu-ally likely to do are often not accurate (i.e., they are out of equi-librium). Therefore, descriptive models of belief formation andadjustment will be more involved than the simple equilibriumones (5, 6, 21, 22). Models of this cognitive hierarchy type predictthe existence of different behavioral types on the basis of thedepth to which they model their opponent.

ResultsBehavioral Results.Given these conditions, how do buyers actuallyplay this game? Simple buyer strategies can be detected by re-gressing the buyer’s private value v against their suggestion s sentto the seller. We restricted our analysis to the second half of theexperiment to allow strategies to stabilize. From these linearregressions, we extracted two behavioral descriptors, the slopeand the R2 of the regression, and used these to cluster buyers

into types (Fig. 2). The slope of this regression tracks roughlywith the credibility of the buyer’s suggestion (i.e., this slopetracks with how “good” the information contained in the sug-gestions is). If the slope is high, sellers should trust the in-formation, and conversely if the slope is near zero, suggestionscontain no information. In the interesting case in which the slopeis negative, suggestions are actively misleading. Thus, we refer tothis slope as a buyer’s “information revelation” coefficient (IR).The buyers fall into three distinct clusters: the “incrementalist”

group (n = 32, blue) is characterized by a relatively high IR andhigh fit (large R2). They are relatively honest with their pricesuggestions (the group mean slope is 0.57, consistent with sug-gesting prices equal to approximately half of the buyer’s value,possibly to share the gains from trade equally). The “conservative”group (n = 28, green) generally show IRs close to zero and in-termediate or low fit. Their suggestions may still contain in-formation about the underlying value, but notmuch. Some of theseactually had constant strategies and always sent a suggestion of 1.The third group, the strategic deceivers, or “strategists” (n = 16,red), are the most interesting. These buyers send suggestions thatare negatively correlated with their private value. Because there isno feedback to either player after each trade, these strategic typeshave surmised that as long as they send a sequence of suggestionsthat mimics an incrementalist type, they should be able to makehigher profits. For example, if they receive a value of 2, they willforego an immediate profit (which would be low at best) and senda high suggestion (for example, 8). Then when they get a highvalue, they can credibly send a low suggestion and reap a highprofit from an unsuspecting seller. In addition to this data-drivenclustering, we developed and estimated a model of belief forma-tion, described in SI Methods, that predicts the existence of thesethree types of players independently, with each type possessingdifferent depths or “levels” of theory of mind. The most sophisti-cated of these, the ones who reasoned most deeply about theiropponents, should exhibit a negative IR. We designated these as“level-2” players. When we estimated this model on our subjects,we found that 14 of our 16 strategists were correctly classified aslevel-2 players. These level-2 subjects are designated by the tri-angles in Fig. 2A.We assessed intelligence quotient (IQ) in 30 of our 76 subjects

(11 incrementalists, 9 conservatives, and 10 strategists) and foundthat although incrementalists IQs were suggestively lower thanconservative and strategist IQs, there were no significant differ-ences among the three groups using a one-way ANOVA. Moreimportantly, there was significant overlap among the three dis-tributions, and there was no significant difference between con-servative and strategist IQs. This shows that IQ alone does notaccount for the differences in behavior, and although above-average IQ seems to be a necessary condition for strategist be-havior, it is not sufficient (Fig. 3A). We also assessed socioeco-nomic status in 65 of our 76 subjects and found no significantdifferences among the three groups (Fig. 3B). Overall, earningswere significantly lower in the incrementalists group than in theother two. Although there was no significant difference in meanearnings between conservatives and strategists, conservativeearnings fell over a larger range, including many of the lowest andhighest earnings overall (Fig. 3C). This is consistent with thecognitive hierarchy model because the superiority of the strategistor conservative approach to the game is dependent on the so-phistication of the seller (i.e., the conservative approach does wellagainst a credulous seller but does poorly against a more sophis-ticated, “level-1” seller).Subject debriefing in the form of a free-response question ad-

ministered after the experiment confirmed that the strategicdeceivers were aware and deliberate in mimicking a distribution ofsuggestions that might be expected from a more truthful in-dividual. One subject wrote “I tried to throw off [the] seller bysaying the low things were high. . ..” This comment, and other

Fig. 1. Experimental task. At the beginning of each round the computerassigns a value for the widget to the buyer. The buyer “suggests a price” tothe seller, who uses this information to set a final price for the object. Thecomputer automates whether the deal occurs—if the price is less than orequal to the buyer’s value, the seller receives the price, p, and the buyerreceives the difference between the price and his private value, v ! p.Otherwise, the deal fails and neither party receives anything. Neither party isinformed of the outcome of the previous trial; payoffs are just added toa running tally of points kept by the computer.

Bhatt et al. PNAS | November 16, 2010 | vol. 107 | no. 46 | 19721

ECONOMIC

SCIENCE

SPS

YCHO

LOGICALAND

COGNITIVESC

IENCE

S

OF INCREMENTALISTS, STRATEGISTS AND RATIONALISTS

• Incrementalists: correlate price suggestion with value

• Rationalists: think everyone is rational, and induce no correlation in relation with value...

• Strategist: react to presence of incrementalists by anti-correlating suggested price and value!

25

similar comments, showed a conscious and sophisticated model ofhow their suggestionsmight be processed by the seller over time. Incontrast, the conservatives often simply stated that they alwaysdesired lower prices and therefore sent low suggestions, reflectinga simpler model of seller behavior. Finally, incrementalists tendedto be more vague in their descriptions of their strategies than theother two groups.

Functional MRI Results. To probe the neural underpinnings ofstrategic behavior in buyers, we performed two sets of analyses.First, we performed between-group (defined by the behavioral

clustering) comparisons of neural activity at various trial epochs.Second, we regressed buyers’ neural activity against the buyer’sIR coefficient at those same epochs. As with the behavioral data,we restricted our analysis of the functional MRI (fMRI) data tothe second half of the experiment to allow strategies to stabilize.Between-group comparisons over the individual subject first-

level boxcar regressors over the entire trial (onset to decision)revealed two significant main effects of behavioral type that sur-vive correction for multiple comparisons at the P < 0.05 level(either corrected for familywise error over gray matter at peakvoxel, or cluster-level correction at P < 0.001, k > 5). First,

Fig. 2. Behavioral analysis. (A) Behavioral clustering in buyers. Incrementalists (blue) send suggestions that are highly correlated with their true value.Strategists (red) send suggestions that are negatively correlated with value. Strategists appear similar to incrementalists and thus reap the surplus from high-value trials. Conservative buyers (green) play closest to an economically rational actor and reveal no information about their value with their suggestions.Triangles indicate subjects who were classified as sophisticated “level-2” buyers according to a generative model. (B) Mean Kullback-Leibler (KL) distances ofthe players’ choice distribution from the uniform distribution. Incrementalists and strategists are both significantly closer to the uniform distribution thanconservatives but are not significantly different from each other. (C) Histograms showing suggestion frequencies for a single incrementalist (Left) and a singlestrategist (Right). Note that from the perspective of the seller the two are indistinguishable.

Fig. 3. Group differences were not explained by differences in IQ or socioeconomic status. (A) Although incrementalists had suggestively lower IQs thanstrategists or conservatives, this was not quite significant (P = 0.07, one-way ANOVA). However, both conservative and strategist IQs were significantly higherthan average, whereas incrementalist IQs were not (11 incrementalists, 9 conservatives, and 10 strategists came back to take the IQ test); there was nosignificant difference between strategist and conservative IQs. (B) We also assessed socioeconomic status on 65 subjects using income, occupation, andeducation level and found there were no significant differences among the three groups according to one-way ANOVA. (C) Subject earnings by behavioraltype: incrementalists had significantly lower final payments for the experiment (P = 0.02, one-way ANOVA); there was no significant difference in meansbetween conservative and strategist earnings. Colored sections of the box-plots indicate the interquartile interval of the data, and whiskers show the totaldata extent excepting outliers, which are shown as red crosses. The black lines indicate the median data, and the notched section of the box gives a 95%confidence interval for the median.

19722 | www.pnas.org/cgi/doi/10.1073/pnas.1009625107 Bhatt et al.

IQ, SOCIOECONOMIC STATUS AND EARNINGS

26

similar comments, showed a conscious and sophisticated model ofhow their suggestionsmight be processed by the seller over time. Incontrast, the conservatives often simply stated that they alwaysdesired lower prices and therefore sent low suggestions, reflectinga simpler model of seller behavior. Finally, incrementalists tendedto be more vague in their descriptions of their strategies than theother two groups.

Functional MRI Results. To probe the neural underpinnings ofstrategic behavior in buyers, we performed two sets of analyses.First, we performed between-group (defined by the behavioral

clustering) comparisons of neural activity at various trial epochs.Second, we regressed buyers’ neural activity against the buyer’sIR coefficient at those same epochs. As with the behavioral data,we restricted our analysis of the functional MRI (fMRI) data tothe second half of the experiment to allow strategies to stabilize.Between-group comparisons over the individual subject first-

level boxcar regressors over the entire trial (onset to decision)revealed two significant main effects of behavioral type that sur-vive correction for multiple comparisons at the P < 0.05 level(either corrected for familywise error over gray matter at peakvoxel, or cluster-level correction at P < 0.001, k > 5). First,

Fig. 2. Behavioral analysis. (A) Behavioral clustering in buyers. Incrementalists (blue) send suggestions that are highly correlated with their true value.Strategists (red) send suggestions that are negatively correlated with value. Strategists appear similar to incrementalists and thus reap the surplus from high-value trials. Conservative buyers (green) play closest to an economically rational actor and reveal no information about their value with their suggestions.Triangles indicate subjects who were classified as sophisticated “level-2” buyers according to a generative model. (B) Mean Kullback-Leibler (KL) distances ofthe players’ choice distribution from the uniform distribution. Incrementalists and strategists are both significantly closer to the uniform distribution thanconservatives but are not significantly different from each other. (C) Histograms showing suggestion frequencies for a single incrementalist (Left) and a singlestrategist (Right). Note that from the perspective of the seller the two are indistinguishable.

Fig. 3. Group differences were not explained by differences in IQ or socioeconomic status. (A) Although incrementalists had suggestively lower IQs thanstrategists or conservatives, this was not quite significant (P = 0.07, one-way ANOVA). However, both conservative and strategist IQs were significantly higherthan average, whereas incrementalist IQs were not (11 incrementalists, 9 conservatives, and 10 strategists came back to take the IQ test); there was nosignificant difference between strategist and conservative IQs. (B) We also assessed socioeconomic status on 65 subjects using income, occupation, andeducation level and found there were no significant differences among the three groups according to one-way ANOVA. (C) Subject earnings by behavioraltype: incrementalists had significantly lower final payments for the experiment (P = 0.02, one-way ANOVA); there was no significant difference in meansbetween conservative and strategist earnings. Colored sections of the box-plots indicate the interquartile interval of the data, and whiskers show the totaldata extent excepting outliers, which are shown as red crosses. The black lines indicate the median data, and the notched section of the box gives a 95%confidence interval for the median.

19722 | www.pnas.org/cgi/doi/10.1073/pnas.1009625107 Bhatt et al.

IMAGING

• Differential activation for strategists (between-group comparison)

• Compare to Yoshida’s imaging results!

27

strategists show a differentially higher activation in left rostralprefrontal cortex (BA10) shown in Fig. 4A, whereas incremen-talists show differentially lower activation in the right DLPFC(rDLPFC) (Fig. 4B). In addition to these main effects, we founda significant (P < 0.005 at peak voxel, corrected for familywiseerror over gray matter) group ! value (individual subject !s forthe value at onset regressor) interaction in right TPJ (rTPJ) (Fig.5). All three of these activations were significant at the P < 0.001(uncorrected) level in the secondary analysis using IR as a con-tinuous between-subject regressor. Both the rDLPFC and rTPJactivations also survived correction for multiple comparisons inthis secondary analysis. Full statistics for both analyses arereported in SI Methods.To further understand the nature of these activations we ex-

amined the time series of activity across the three groups in theidentified regions. Confirming the whole-brain analysis, strategistactivity in BA10 (Fig. 4A, Right) is significantly greater than forboth conservatives and incrementalists, whereas conservative andincrementalist time courses were essentially identical. On theother hand, in the rDLPFC (Fig. 4B, Right), time course analysisreveals that although the area was characterized by decreasedactivation in the incrementalists, conservatives show an in-termediate level of activation, between the activities of the incre-mentalists and the strategists. Finally, in the rTPJ (Fig. 5, Rightpanel), strategists showed a strong relationship between activationand value, which was absent in the other two groups.

DiscussionFrom a neural standpoint our understanding of social interactionsis in its infancy. The present study attempts to shed light on animportant aspect of social interaction: the understanding andmanipulation of others’ perceptions of us. Second-order beliefformation is particularly interesting because there seems to bea wide range of abilities within the scope of normal human be-havior. Indeed, in this simple bargaining task we used task be-

havior to uncover three distinct clusters of buyers. Further, fMRIrevealed distinct neural correlates associated with these clusters.To display strategic deception, subjects had to be able to con-

sider the implications of current decisions on future payoffs andespecially consider the counterfactual situation of what mighthappen if they chose the conservative strategy and engendered toomuch suspicion in the seller. This also requires the maintenanceand continual updating of the “false beliefs” of their opponent, aswell as cognitive control mechanisms necessary to inhibit the im-pulse to transfer information. In contrast, the conservatives onlyneed to inhibit information transfer, and the incrementalists’ naïvestrategy simply anchors suggestions directly to the true value.TPJ has been found repeatedly to be active in theory of mind

tasks, particularly in the attribution of false or incongruent beliefsto another person (16, 19, 23). It is interesting that rTPJ activity ismodulated by value rather than simply being more active instrategists, as might be expected. Saxe andWexler (17) found thatsignals in the rTPJ were modulated by the degree of incongruenceamong multiple facts known about a target’s mind. This findingshapes our interpretation of the modulation of activity in rTPJby value in strategists: it is during the high-value trials that thestrategists’ bluff really matters. Even though strategists are de-ceiving during both high- and low-value trials, and their sugges-tions are always incongruent with their true value, the payoff onlycomes during the high-value trials. Additionally, strategists are ef-fectively switching between twomodes of behavior: reputation build-ing, which occurs during low-value trials, and reward-collection,which occurs during high-value trials. This switch between atten-tion to one’s reputation and attention to one’s actual payoffs is anexample of attentional reorienting that has been associated withactivity in TPJ (20). This activation also bears striking similaritiesto a nearby activation in superior temporal sulcus (STS) foundby Hampton et al. [The reported peaks are close together, butdo not appear to overlap: (52, !48, 20) for right TPJ in thisstudy; (60, !54, 9) for STS in Hampton et al.] (13). In this article,

Fig. 4. Strategists differentially activate rDLPFC and left BA10. (A) Between-group analysis: strategist activation over the entire trial vs. other groups. Left: LeftBA10. Peak voxel at (!32, 48, 20), k = 14, t = 4.72, P = 0.049 at peak voxel (corrected for familywise error over graymatter). Right: Time courses in BA10 by group.(B) Between-group analysis: incrementalist activation over the entire trial vs. other groups. Left: rDLPFC. Peak voxel at (36, 28, 36), k = 27, t = 4.62 at peak voxel,cluster-level P = 0.044 (corrected). Right: Time courses in rDLPFC by group. For both regions clusters are shown at P < 0.001, uncorrected. Cluster extents andcluster-level Ps are reported at this threshold as well. Full statistics are reported in SI Methods. For time courses, all data are normalized to trial onset, dottedblack line indicates average decision time, and asterisks indicate significance of the one-way ANOVA on activation at peristimilus time at the P < 0.01 level.

Bhatt et al. PNAS | November 16, 2010 | vol. 107 | no. 46 | 19723

ECONOMIC

SCIENCE

SPS

YCHO

LOGICALAND

COGNITIVESC

IENCE

S

bilateral STS activation was correlated with an “influence” pa-rameter in a behavioral model. Interestingly, this parametercorrelated with both expected reward and strategic switching intheir task, much as high-value trials correlate to expected returnand strategic switching in the strategists.BA10 has been implicated in long-term goal maintenance and

the use of prospective memory (10, 24), vital aspects of thestrategists’ forward-looking behavior. Burgess et al. proposea medial/lateral functional mapping of this region according towhether mental processing is stimulus oriented or stimulus in-dependent, with the latter being associated with more lateralactivations. This is consistent with our interpretation of therelatively lateral activation (x = !32) in BA10 as correspondingto the need for prospective thinking and goal maintenance inthe strategist approach. As mentioned above, unlike incremen-talists or conservatives, strategists have a distinct intermediategoal in the pursuit of reward: reputation building. The mainte-nance of the relatively long-term goal as a means to greateroverall and future rewards is consistent with the stimulus-independent processing attributed to the area. The relative lackof activation at this locus for the other two groups (as well as thesimilarity of activity levels between the two other groups) ac-curately reflects the leap in task complexity between the con-servative and strategist approaches.On the other hand, rDLPFC shows a more continuous re-

lationship between activation and strategic sophistication. Ofthe three areas highlighted here, it is the only one where theanalysis using IR as a continuous between-subject regressoryielded a larger activation than the between-group analysis (k =31 vs. k = 27; SI Methods provides full statistics on both anal-yses). The area has been consistently implicated in tasks in-volving working memory and cognitive control (25, 26). Both ofthese are functions that should be used by strategists, who mustkeep track of their previous suggestions to infer what theirreputation is with the seller and inhibit the impulse to simplyanchor their suggestion on their true value. The cognitive con-trol function of rDLPFC in this task is further highlighted by thefact that conservatives also have elevated activity in the area ascompared with incrementalists. One transcranial magneticstimulation (TMS) study showing that disruption in rDLPFC re-duces intertemporal building of a trustworthy reputation (butonly when other people are highly trusting) is consistent withthis cognitive control function of rDLPFC in strategizing (27).The patterns of activity uncovered in the strategists uncover

a set of regions involved in the successful manipulation of others’beliefs over time. BA10 is strongly recruited in strategists but notin incrementalists or conservatives. rDLPFC is recruited stronglyby strategists, with some variance by conservatives, and weakly by

incrementalists. Finally, the rTPJ, important for the attributionof false or incongruent beliefs to others and attentional reor-ienting, is strongly activated during strategist bluffing.Human strategic thinking is both complicated and highly adap-

tive, driven by the coevolution of complex artificial socioeconomicenvironments and the mind’s capacity to navigate those environ-ments. Earlier studies have established neural correlates of thecapacity to reason about other agents’ likely behavior (14, 28) andthe ability to learn from social information (29). Our study goes animportant step further, by shedding light on how some agentsmake choices to manipulate other agents’ perceptions of their ownstrategies. The pairing of the behavioral and neural data stronglysuggests that strategists are guided in their deceit by the moreschematic, forward-looking computations of BA10 and TPJ, inconcert with heightenedmemory and control provided byDLPFC.It remains to be seen how a given individual finds herself in onegroup or the other: is strategic ability inherent, or can we trainindividuals to more easily identify strategic solutions by empha-sizing the use of schematic representations and counterfactualanalysis? Is strategic ability context dependent? Whatever thecase, opportunities for strategic deception of this sort are possibleonly because of the existence, and in fact likely relative prevalence,of people with the tendency to be honest even when such honestyis not in their interest. However, in our admittedly circumscribedsituation it is clear that there are three distinct classes of individ-uals who approach this strategic interaction in completely differ-ent ways and that these differences are manifest in qualitativelydifferent neural signatures.Our results suggest a method of understanding and quantify-

ing individual differences—as clusters of behavior in an eco-nomic game (9)—and point to applications for the definition anddiagnosis of mental disorders. Economic games can provideobjective quantitative measures of strategic thinking (as in thisstudy), social preferences (30, 31), risk preferences (32, 33), anda host of other potentially interesting characteristics. A betterunderstanding of the range and joint distributions of these fac-tors in the population could provide insight into those individualswho fall at the extremes of these distributions (i.e., those withmental disorders).

MethodsWe regressed buyers’ suggestions on their private values over the secondhalf of the experiment, yielding three descriptive strategy parameters foreach buyer—the slope, intercept, and fit (R2). We normalized these threestatistics across subjects by subtracting means and dividing by SDs. Clusterswere identified using the k-means algorithm (34). The clusters did notchange significantly when intercepts were excluded, therefore the results inthe text are clustered using only slope and fit.

Fig. 5. Value modulates rTPJ activation only in the strategists. Between-group analysis: interaction of activation at trial onset with value, incrementalists vs.other groups. Left: rTPJ. Peak voxel at (52, !48, 20), k = 10, t = 5.41, P = 0.004 at peak voxel (corrected for familywise error over gray matter). Full statistics arereported in SI Methods. Right: Time courses in rTPJ by group.

19724 | www.pnas.org/cgi/doi/10.1073/pnas.1009625107 Bhatt et al.

NEXT: FINANCIAL MARKETS WITH “INSIDERS”

• Even large-scale, anonymous financial markets may be perceived as intentional, goal-oriented – or “strategic:” there may be “insiders”

• Questions:

1. Is this within scope of Theory of Mind?2. Does Theory of Mind help?3. What does it do?

28

EXPERIMENT:

• Markets are re-played in scanner while subject is exposed to risk

• Subject:

• Predicts price changes in market replay; • Performs Theory of Mind tests; • Performs (financial) mathematics test

20

29

MARKET REPLAY

30

22

Graphical replay of market

31

IMAGING RESULTS

23

Effect of |price-0.25|,

contrast between insider

and no-insider treatment

Bruguier, ea, J Finance 201032

EXPLANATORY VARIABLES FOR BRAIN ACTIVATION

33

PREDICTION PERFORMANCE RESULTS

Against Heider test

0 2 4 6 8 10 12 1445

50

55

60

65

70

75

80

85

!=0.348247, p=0.022110, n=43

Heider Test

Fin

ancia

l M

ark

et P

redic

tion

Against Math test

−1 0 1 2 3 4 5 6 7 845

50

55

60

65

70

75

!=0.060792, p=0.698569, n=43

Mathematical puzzles

Fin

ancia

l M

ark

et P

redic

tion

34

WHAT DOES THIS MEAN FORMALLY?

! Theory of Mind brain regions are engaged and financial performance correlates with Theory of Mind skills

! Now, Theory of Mind = pattern recognition! So: What patterns are attended to?! First of all: What distinguishes presence of insiders?

35

PRICING PATTERNS WHEN THERE ARE INSIDERS: VOLATILITY CLUSTERING

36

“GENERALIZED AUTOREGRESSIVE CONDITIONAL

HETEROSKEDASTICITY” (GARCH)

! Typical autocorrelations of absolute price change: GREEN=Insiders; BLUE=No Insiders

37

(CONTRAST WITH ECONOMIC THEORY)

! Rational Expectations Equilibrium: Uninformed are supposed to know how prices relate to information of insiders

! Where does this knowledge come from?!! Here we provide a partial answer

38

ASSET PRICE BUBBLES

• Similar activations when there is mis-pricing...

39

TAKE-HOME MESSAGES I

! Playing strategic games engages “Theory of Mind”! Theory of Mind is about predicting how the opponent

changes strategies because she is learning! Goes beyond fictitious play and reinforcement learning

(opponent is intentional)! Theory of Mind is mathematics: pattern recognition! … which is not the mathematics of Nash equilibrium

40

TAKE-HOME MESSAGES II

! Theory of Mind also applies to understanding “social systems” that don’t really have a mind on their own

! We see this in brain activation when exposed to risk in markets with insiders

! We also see this in correlation of performance between predicting prices in such markets and traditional Theory of Mind tasks

! … and in the absence of correlation with mathematical skill (as when playing strategic games)

! Patterns being recognized = GARCH?

41