COGNITIVE SCIENCE

Using large-scale experiments and machine learningto discover theories of human decision-makingJoshua C. Peterson1*, David D. Bourgin1†, Mayank Agrawal2,3,Daniel Reichman4, Thomas L. Griffiths1,2

Predicting and understanding how people make decisions has been a long-standing goal in manyfields, with quantitative models of human decision-making informing research in both the socialsciences and engineering. We show how progress toward this goal can be accelerated by using largedatasets to power machine-learning algorithms that are constrained to produce interpretablepsychological theories. Conducting the largest experiment on risky choice to date and analyzing theresults using gradient-based optimization of differentiable decision theories implemented throughartificial neural networks, we were able to recapitulate historical discoveries, establish that thereis room to improve on existing theories, and discover a new, more accurate model of humandecision-making in a form that preserves the insights from centuries of research.

Understanding howpeoplemake decisionsis a central problem in psychology andeconomics (1–3). Having quantitativemodels that can predict these decisionshas become increasingly important as

automated systems interact more closely withpeople (4, 5). The search for such models goesback almost 300 years (6) but intensified inthe latter half of the 20th century (7, 8) as em-pirical findings revealed the limitations of theidea that peoplemake decisions bymaximizingexpected utility (EU) (9–11). This led to the de-velopment of new models such as prospecttheory (PT) (8, 12). Recently, this theory-drivenenterprise has been complemented by data-driven research using machine learning topredict human decisions (13–19). Althoughmachine learning has the potential to accel-erate the discovery of predictive models of hu-man judgments (20–22), the resulting modelsare limited by small datasets and are oftenuninterpretable (5). To overcome these chal-lenges, we introduce a new approach based ondefining classes of machine-learning modelsthat embody constraints based on psycholog-ical theory. We present the largest experimentstudying people’s choices to date, allowing usto use our approach to systematically evalu-ate existing theories, identify a lower boundon optimal prediction performance, and pro-pose a new descriptive theory that reachesthis bound and contains classic theories asspecial cases.We focus on risky choice, one of the most

basic and extensively studied problems in de-cision theory (8, 23). Risky choice has largelybeen examined using “choice problems,” sce-

narios in which decision-makers face a choicebetween two gambles, each of which has a setof outcomes that differ in their payoffs andprobabilities (Fig. 1A). Researchers studyingrisky choice seek a theory, which we formal-ize as a function that maps from a pair ofgambles, A and B, to the probability P(A) thata decision-maker chooses gamble A over gambleB, that is consistent with human decisionsfor as many choice problems as possible. Dis-covering the best theory is a formidable chal-lenge for two reasons. First, the space of choiceproblems is large. The value and probability ofeach outcome for each gamble define the di-mensions of this space, meaning that describ-ing a pair of gambles could potentially requiredozens of dimensions. Second, the space ofpossible theories is even larger, with theoriesof choice between two options spanning allpossible functions mapping choice problemsin ℝ2d to ℝ, i.e., from a vector of d gambleoutcomes and d associated probabilities to achoice probability.Machine-learning methods such as deep

neural networks (24) excel at function approx-imation (25, 26) and thus provide a tool thatcould potentially be used to automate theorysearch. However, thesemethods typically requirelarge amounts of data. Historically, datasets onrisky choice have been small: Influential papersfocused on a few dozen choice problems (27) andthe largest previous dataset featured <300 (28).Consequently, off-the-shelf methods have per-formed poorly in predicting human choices (29).Furthermore, even when data are abundant,the functions discovered by machine-learningalgorithms are notoriously hard to interpret(30), making for poor explanatory scientificmodels.To address these challenges, we collected a

large dataset of human decisions for almost10,000 choice problems presented in a formatthat has been used in previous evaluations ofmodels of decision-making (27–29) (Fig. 1A).

This dataset includes >30 times the numberof problems in the largest previous dataset(27) (Fig. 1B). We then used this dataset toevaluate differentiable decision theories thatexploit the flexibility of deep neural networksbut use psychologically meaningful constraintsto pick out a smooth, searchable landscape ofcandidate theories with shared assumptions.Differentiable decision theories allow the intu-itions of theorists to be combinedwith gradient-based optimization methods frommachinelearning to broadly search the space of theoriesin a way that yields interpretable scientificexplanations.More formally, we define a hierarchy over

decision theories (Fig. 1C) reflecting the addi-tion of an increasing number of constraintson the space of functions. These constraintsexpress psychologically meaningful theoret-ical commitments. For example, one class oftheories contains all functions in which thevalue that people assign to one gamble can beinfluenced by the contents of the other gamble.If theories in this class are more predictivethan those that belong to the simpler classescontained within it (e.g., where the value ofgambles are independent), then we knowthat these simpler theories should be elimi-nated. We enforce each constraint by modify-ing the architecture of artificial neural networks,resulting in differentiable decision theories. Thistheory-driven approach to defining constraintscontrasts with genericmethods for constrainingneural networks, such as restricting their size orthe ranges of their weights (31). After optimizinga differentiable theory to best fit human be-havior, it will ideally have picked out the optimaltheory in its class.The lowest levels of our hierarchy contain

the simplest theories, including classic modelsof choice. Objectively, gambles that yield higherpayouts in the long run are those with higherexpected value (EV), with the value V(A) ofgamble A being

Pi xipi, where outcome i of

gamble A has payoff xi and probability pi.In our hierarchical partitioning, this is thesimplest possible theory because it has nodescendants. Moving up the hierarchy, andfollowing expected utility (EU) theory (6, 32),we can ask the question of whether payouts xiare viewed subjectively by decision-makers:V(A) =

Pi u(xi)pi. When u(•) is the identity

function u(x) = x, EU reduces to EV and thuscontains it. Theories based on EU have his-torically relied on explicit proposals for theform of u(•), which are typically simple, non-linear parametric functions (33). By contrast,we search the entire class by learning the op-timal u(•) with a neural network (we call theresulting model “neural EU”), and use auto-matic differentiation to optimize the modelP(A)º exp{h

Piu(xi)pi}, where h captures the

degree of determinism in people’s responses(34). This can be viewed as a neural network

RESEARCH

Peterson et al., Science 372, 1209–1214 (2021) 11 June 2021 1 of 6

1Department of Computer Science, Princeton University,Princeton, NJ 08540, USA. 2Department of Psychology,Princeton University, Princeton, NJ 08540, USA. 3PrincetonNeuroscience Institute, Princeton University, Princeton, NJ08544, USA. 4Department of Computer Science, WorcesterPolytechnic Institute, Worcester, MA 01609, USA.*Corresponding author. Email: joshuacp@princeton.edu†Present affiliation: Spotify.

Corrected 11 June 2021. See full text. on June 12, 2021

http://science.sciencemag.org/

Dow

nloaded from

architecture in which the output layer is asoftmax function ehzj=Skehzk , there is onenode for each gamble, the hidden units inthe second-to-last layer encode the utilities ofthe outcomes, and the final layer of weightscorresponds to their probabilities (Fig. 1D).Figure 2 shows that the discovered form ofu(•) is similar to those proposed by humantheorists (i.e., decreasing marginal utility andasymmetry) but outperforms any of those the-ories and can be learned using only a quarterof our data. [All theories are evaluated on theircross-validated generalization performance,meaning that model complexity is already im-plicitly accounted for in our analyses; we focuson mean-squared error (MSE) for consis-tency with previous evaluations of models ofdecision-making (28, 29) but also include analy-ses of cross-entropy in the supplementarymaterials.] The decision preference accuracy[i.e., the proportion of problems in which themodel prediction for P(A) is >0.5 when theobserved proportions are also >0.5] for thismodel was 81.41%.Next, mirroring subjective EU (7) and PT, we

can ask the question of whether the proba-bilities (pi) are also viewed subjectively by

decision-makers: v(A) = Siu(xi)p(pi). Again,p(•) can take on classic forms or be learnedfromdata (“neural PT”). Figure 2B shows thata form of p(•) that outperforms all proposalsby human theorists can be learned using one-fifth of our data and exhibits overweightingof events with medium to low probability.This overweighting is much smaller than istypically found in applications of PT, in partreflecting the difference in the range of choiceproblems that we consider relative to classicstudies. We will return to this point later. Thedecision preference accuracy for this modelwas 82.33%.Allowing separate p(•) functions for posi-

tive and negative outcomes, respectively, andapplying them cumulatively to an ordered setof outcomes corresponds to the most popularmodern variant of PT: cumulative PT (CPT)(12, 35) (Fig. 2B; see the materials and meth-ods). Notably, “neural CPT” does not containneural PT because the former cannot violatestochastic dominance.With small amounts ofdata, corresponding to the largest previousexperiments (28, 29), CPT outperforms PT,accounting for its popularity. However, thistrend reverses as the amount of data is in-

creased, which illustrates that suitably largedatasets, in addition to aiding machine learn-ing, provide more robust evaluation.Next, we can ask whether the possible out-

comes of a gamble affect the perception ofeach other and their probabilities and viceversa. More formally, we learn a neural networkf(•,•) such that P(A) º exp{f(xA,pA)}, wherexA and pA are the vector of payoffs and prob-abilities associated with gamble A, respec-tively. This function class computes the valueof a gamble (“value-based”) like PT and othersbut does not enforce linearity when combiningpayoffs and probabilities. Notably, this class ofmodels includes those that violate the in-dependence axiom in decision-making (32).Figure 3A shows that there exists a value-based theory that results in a greater improvementin performance over PT than PT does over EU.Relaxing the constraint that each gamble

is valued independently results in our mostgeneral class of functions, “context-dependent”functions g(•)whereP(A) = g(xA,pA, xB,pB). Thisclass of models includes those that violate boththe independence and transitivity axioms indecision-making (32). This formulation pro-vides a way to estimate the performance of the

Peterson et al., Science 372, 1209–1214 (2021) 11 June 2021 2 of 6

Fig. 1. Applying large-scale experimentation andtheory-driven machine learning to risky choice.(A) Experiment interface in which participantsmade choices between pairs of gambles (“choiceproblems”) and were paid at the end of the experimentbased on their choice in a single randomly selectedgamble. (B) Each pair of gambles can be described bya vector of payoffs and probabilities. Reducing theresulting space to two dimensions (2D) allows us tovisualize coverage by different experiments. Each pointis a different choice problem, and colors showreconstructions of the problems used in influentialexperiments (green), the previous largest dataset(red), and our 9831 problems, which provide muchbroader coverage of the problem space. This 2Dembedding results from applying t-distributedstochastic neighbor embedding (t-SNE) to the hidden-layer representation of our best-performing neuralnetwork model. (C) We define a hierarchy oftheoretical assumptions expressed as partitions overfunction space that can be searched. More complexclasses of functions contain simpler classes asspecial cases, allowing us to systematically search thespace of theories and identify the impact of con-straints that correspond to psychologically meaningfultheoretical commitments. All model classes aredescribed in the main text. (D) Differentiable decisiontheories use the formal structure of classic theoriesto constrain the architecture of the neural network.For example, our EU model uses a neural network todefine the utility function but combines thoseutilities in a classic form, resulting in a fullydifferentiable model that can be optimized bygradient descent.

RESEARCH | REPORT

Corrected 11 June 2021. See full text. on June 12, 2021

http://science.sciencemag.org/

Dow

nloaded from

near-optimal theory because it has no con-straints (except that it must be differentia-ble). In particular, this model is simply a fullyunconstrained neural network that takes allinformation about both gambles as input andoutputs P(A). The move from value-based tocontext-dependent functions results in thelargest improvement yet in prediction per-formance and a decision preference accuracyof 84.81%. Because value-based theories assignvalues to gambles independently, comparingtheir performance with this less-constrainedneural network implicitly asks whether thevalue of gambles is contextual (i.e., whetherone gamble and its parameters affect one’sperception of the other gamble); our resultsindicate that this is the case and, further, thatthe impact of this effect is substantial.One context-dependent psychological model

that has outperformed others in recent evalua-tions is a process model called Best Estimateand Sampling Tools (BEAST) (27), whichmakesdecisions by combining EV with the results offour different kinds of simulations, a subset ofwhich integrates information about both out-comes and probabilities across gambles (27).Despite outperforming all other models withsmall amounts of data, Fig. 3A shows thatBEAST falls short of our best-performingmodel in the context-dependent class as theamount of data increases. Further, we found

that several othermodels fromdiversemodelingtraditions in risky choice that also incorporatevarious forms of context performed no betterthan PT on our dataset (see Fig. 3B and thesupplementary materials). We also replicatedour core findings in a second dataset of 1000problems collected from a new group of par-ticipants, including showing that we see sim-ilar results when evaluating themodels at thelevel of individual participants (see the sup-plementary materials).Having evaluated the most competitive the-

ories at each level of our hierarchy, we findthat the best-performing theory belongs tothe most complex class we defined and liesoutside of all simpler classes. This impliesthat the best predictions of human choicesresult from viewing payoffs and their prob-abilities subjectively but, more importantly, inways that are sensitive to the context of thecompeting gamble. However, as a relativelyunconstrained neural network, this modelprovides limited psychological insight andis highly susceptible to overfitting the noise insmall datasets because of its expressivepower.To better understand which aspects of con-

text were responsible for bettermodel perform-ance, we conduct a second pass of our method.In particular, we define a class of models(“contextual multiplicative”) where V(A) =

Si∈Au(xi, c1)p(pi, c2) and c1 and c2 are vectorspotentially containing information from xA,xB, pA, and pB that condition u(•) and p(•),allowing subjective rewards and probabilitiesto vary depending on the problem (see thesupplementary materials for more details).Results are shown in Fig. 3A. When condi-tioning utilities on other outcomes withina gamble (c1 = xA; “intra-gamble outcomecontext”), performance improves only mar-ginally and does not match the value-basedclass. However, when instead conditioningutilities on all other outcomes across bothgambles (c1 = {xA, xB}; “inter-gamble outcomecontext”), performance improvesmarkedly. Fur-ther allowing probabilities to interact (c2 ={pA, pB}; “inter-gamble outcome/probabilitycontext”) provides marginal improvement. Ex-isting theories that fall into these classes per-formed no better than PT (i.e., where c1 and c2contain no information; see the supplemen-tarymaterials). Finally, a “fully contextual mul-tiplicative” model, where c1 = c2 = {xA, xB, pA,pB} fails to improve performance, likely becauseit has twice as many parameters as the context-dependentmodel as a result of conditioning twonetworks on all information fromboth gambles.This analysis helps to illuminate the keyproperties of the best-performing theoryfoundwithin the context-dependent class: Out-comes and probabilities are largely combined

Peterson et al., Science 372, 1209–1214 (2021) 11 June 2021 3 of 6

Fig. 2. Comparing classic theories proposed by human researchers withdifferentiable decision theories discovered through machine learning.(A) EU. In the left panel, EU with a learned optimal utility function outperformsclassic models (gray lines) given enough data. Performance is assessed interms of prediction error (MSE) on ~1000 unseen choice problems as afunction of the amount of training data used for model fitting (~9000 choiceproblems, average of 50 runs). The right panel shows that the utility function

identified through this optimization method reproduces many of thecharacteristics suggested by human theorists. (B) PT. In the left panel,PT with learned utility and probability weighting functions outperforms classicmodels (gray lines) given enough data. CPT, the modern variant of PT,performs better with small amounts of data, on the scale of previousexperiments, but slightly worse with more data. The right panel shows theoptimized utility and probability weighting functions for PT.

RESEARCH | REPORT

Corrected 11 June 2021. See full text. on June 12, 2021

http://science.sciencemag.org/

Dow

nloaded from

multiplicatively to form an average but aresubjectively transformed in ways that dependon information across both gambles, especial-ly outcomes.To identify a theory that has these charac-

teristics, and is more continuous with previousaccounts of human decision-making, and ableto be trained with fewer data, we comparedEU theory with the best-performing context-dependent model, identifying patterns in thetypes of problems where PT performed lesswell. We found that the largest differencesconcerned dominated gambles (i.e., where alloutcomes of one gamble are better than alloutcomes of the other) and pairs of gamblesthat pit likely losses against high uncertainty(see the supplementary materials). To addressthis, we defined a new model based on thehypothesis that people use different strategies

for different choice problems, reminiscent ofthe dual process theories of risky choice (36).These strategies can correspond to classic mod-els, preserving their insights. Further, if we al-low the selection of these strategies based onthe properties of both gambles, then this mod-el is a simpler, more specific form of the con-textual multiplicative class, which normallyselects from a theoretically infinite set of mod-els (i.e., utility and weighting functions) in-stead of a relatively compact, fixed set.The resulting mixture of theories (MOT)

model learns to apply one of two utility func-tions and one of two probability weightingfunctions. Two accompanying neural networkstake both gambles as input and output convexmixture weights for both subjective functions,which are learned jointly in a “mixture of ex-perts” architecture (37). For dominated gambles,

the gamble values determined by thesemixturesare bypassed and a learned, fixed probability ofchoosing the dominated gamble is taken as theprediction. Figure 4A shows that this modelgeneralizes to unseen problems as well as ourbest, fully unconstrained neural network (context-dependent), with a similar decision prefer-ence accuracy of 84.15%, and is able to achievebetter performance with fewer data. Thefunctional forms rediscovered by the mix-ture model correspond closely to classic EUand PT models (Fig. 4B). In particular, onelearned utility function clearly produces lossaversion (8) and one probability weightingfunction overweights low probabilities in away that is more consistent with the classiceffects modeled by PT (8). By looking at theproblems where the different components areused, we can seewhen these characteristics aremost important to capturing human decisions(Fig. 4, lower panels). The best predictors of theutility function used, which an ablation anal-ysis reveals is the most important contexteffect (see the supplementary materials), weremaximum outcome, minimum outcome, andoutcome variability. The best predictors forprobability weighting functions were mini-mum outcome and number of losses (see thesupplementary materials). This focus on thecontext of outcomes across gambles is cor-roborated by similar performance being pro-duced by a variant ofMOT inwhich themixturenetwork is only given outcomes as input (Fig.4A, dashed green line). The clusters of problemsassigned to each subjective function emerge in arepresentation based on hidden activationsfrom the context-dependent model, suggestingthat the MOT is capturing a structure similarto this unconstrained model.We also found that MOT provides a com-

petitive model of individual-level behavior. Inparticular, we fine-tuned the parameters ofthe MOT model above trained on aggregatedata to the behavior of individual participants,obtaining an average MSE of 0.058 on out-of-sample decisions for each participant. Forcomparison, we repeated this procedure usingthe best-performing parametric form of PT, acommon choice for individual-level modelingof risky choice, obtaining an average MSE of0.063. See the supplementary materials foradditional details and results.Our results illustrate the successes of human

ingenuity, in particular, finding good functionalforms for the EU and PTmodels. However, theyalso illustrate that this ingenuity can be sup-plemented by an automated search overmodelsgiven enough data, and that as the class ofmodels becomes less restrictive and the datasetbecomes larger, this automated approach be-gins to substantially outperform the best mod-els of decision-making developed by humanresearchers. This does not mean that theoriesdeveloped by psychologists and economists are

Peterson et al., Science 372, 1209–1214 (2021) 11 June 2021 4 of 6

Fig. 3. Complex decision theories exhibit better predictive performance than simpler ones. (A) Performanceof differentiable decision theories. As model flexibility increases, performance increases, along with datarequirements. (B) Performance comparison between differentiable decision theories and 21 other well-knowntheories across the risky choice literature, none of which outperforms PT (see the supplementarymaterials for more details).

RESEARCH | REPORT

Corrected 11 June 2021. See full text. on June 12, 2021

http://science.sciencemag.org/

Dow

nloaded from

not valuable for predicting human behavior,even in the big-data regime: Our MOT modelwas able to leverage large amounts of data tolearnwhich classic theory best describes humanbehavior in different contexts. These models,variants of EU and PT, have been subject torigorous mathematical analyses over decades,which are preserved in this new theory.Further, whereasMOT is a special case of the

contextual multiplicative model, it performsbetter due to theory-driven simplifications andis thus more sample efficient. Human ingenu-ity will also be required for potentially trans-lating this descriptive theory into normativeand processmodels (38, 39). We anticipate thatour approach of defining differentiable theoriesthat express meaningful psychological con-straints can be applied in other settings aswe continue to gather more data on humandecision-making.Finally, it is noteworthy that models of

decision-making developed byhuman researchers

tend to outperformourmachine-learningmod-els when we only consider amounts of datathat are consistent with the scale of previousbehavioral research, but this trend reverseswhen more data are available. This patternmay imply that the complexity of psycholog-ical theories has been constrained by limiteddata. As we begin tomove into a regime of bigbehavioral data, our theories are going to haveto become increasingly complex to be able tocapture the systematic variation that theselarger datasets reveal. The use of large datasets(40, 41) has revolutionized machine learning,computer vision, and artificial intelligence. Ourstudy is one of the first to use a similar meth-odology in systematically investigating theoriesof human cognition. We believe that the use oflarge datasets coupled with machine-learningalgorithms offers enormous potential for uncov-ering new cognitive and behavioral phenomenathat would be difficult to identify without suchtools (42).

REFERENCES AND NOTES

1. N. C. Barberis, J. Econ. Perspect. 27, 173–196 (2013).

2. R. Hastie, R. M. Dawes, Rational Choice in an Uncertain World:The Psychology of Judgment and Decision Making (Sage,2009).

3. I. Gilboa, Theory of Decision Under Uncertainty (CambridgeUniv. Press, 2009), vol. 45.

4. A. Jameson, “Choices and decisions of computer users,” in TheHuman-Computer Interaction Handbook: Fundamentals,Evolving Technologies and Emerging Applications, J. A. Jacko,Ed. (CRC Press, 2012), pp. 77–94.

5. I. Rahwan et al., Nature 568, 477–486 (2019).

6. D. Bernoulli, Econometrica 22, 23–36 (1954).

7. L. J. Savage, The Foundations of Statistics (Courier, 1972).

8. D. Kahneman, A. Tversky, Econometrica 47, 263–292 (1979).9. M. Allais, Econometrica 21, 503–546 (1953).10. H. J. Einhorn, R. M. Hogarth, J. Bus. 59 (S4), S225–S250

(1986).

11. D. Ellsberg, Q. J. Econ. 75, 643–669 (1961).

12. A. Tversky, D. Kahneman, J. Risk Uncertain. 5, 297–323 (1992).

13. D. Fudenberg, J. Kleinberg, A. Liang, S. Mullainathan,Measuring the completeness of theories. arXiv:1910.07022[econ.TH] (15 October 2019).

14. G. Noti, E. Levi, Y. Kolumbus, A. Daniely, Behavior-basedmachine-learning: A hybrid approach for predictinghuman decision making. arXiv:1611.10228 [cs.LG](30 November 2016).

Peterson et al., Science 372, 1209–1214 (2021) 11 June 2021 5 of 6

Fig. 4. The MOT model assigns a fixed learned decision probability forgambles that are dominated and determines the value of gambles in allother choice problems using mixtures of two utility and two probability-weighting functions. (A) MOT obtains slightly better performance than the bestlearned decision theory, even when the mixture networks and thus contexteffects are limited to information about outcomes only. (B) Components of themodel are interpretable by design and resemble the classic forms for EU and PT:

Utility function 1 (UF 1) is symmetric, whereas UF 2 shows loss aversion.Probability weighting function 1 (PWF 1) is linear, whereas PWF 2 showsoverweighting of lower probabilities. 2D t-SNE embeddings of the choiceproblems based on the hidden unit representations of the best-performingcontext-dependent neural network exhibit distinct clusters that correspond toproblems where MOT applied UF 2 (C), PWF 2 (D), and the dominatedcomponent (E).

RESEARCH | REPORT

Corrected 11 June 2021. See full text. on June 12, 2021

http://science.sciencemag.org/

Dow

nloaded from

15. T. Yarkoni, J. Westfall, Perspect. Psychol. Sci. 12, 1100–1122(2017).

16. A. Peysakhovich, J. Naecker, J. Econ. Behav. Organ. 133,373–384 (2017).

17. J. Kleinberg, H. Lakkaraju, J. Leskovec, J. Ludwig,S. Mullainathan, Q. J. Econ. 133, 237–293 (2018).

18. J. S. Hartford, J. R. Wright, K. Leyton-Brown, Adv. Neural Inf.Process. Syst. 29, 2424–2432 (2016).

19. L. He, P. P. Pantelis, S. Bhatia, Manage. Sci., in press.20. J.-F. Bonnefon, A. Shariff, I. Rahwan, Science 352, 1573–1576

(2016).

21. A. Rosenfeld, S. Kraus, “Predicting human decision-making:From prediction to action,” in Synthesis Lectures on ArtificialIntelligence and Machine Learning, R. Brachman, F. Rossi,P. Stone, Eds. (Morgan & Claypool, 2018), vol. 12, no. 1,pp. 1–150; https://doi.org/10.2200/S00820ED1V01Y201712AIM036.

22. C. F. Camerer, “Artificial intelligence and behavioraleconomics,” in The Economics of Artificial Intelligence: AnAgenda, A. Agrawal, J. Gans, A. Goldfarb, Eds. (Univ. of ChicagoPress, 2018), pp. 587–608.

23. W. Edwards, Psychol. Bull. 51, 380–417 (1954).

24. Y. LeCun, Y. Bengio, G. Hinton, Nature 521, 436–444 (2015).

25. G. Cybenko, Math. Contr. Signals Syst. 2, 303–314 (1989).

26. K. Hornik, Neural Netw. 4, 251–257 (1991).

27. I. Erev, E. Ert, O. Plonsky, D. Cohen, O. Cohen, Psychol. Rev.124, 369–409 (2017).

28. O. Plonsky et al., Predicting human decisions with behavioraltheories and machine learning. arXiv:1904.06866 [cs.AI](15 April 2019).

29. O. Plonsky, I. Erev, T. Hazan, M. Tennenholtz, “Psychologicalforest: Predicting human behavior,” in Thirty-First AAAI

Conference on Artificial Intelligence, San Francisco, CA,4–9 February 2017; https://aaai.org/ocs/index.php/AAAI/AAAI17/paper/view/14925.

30. F. Doshi-Velez, B. Kim, Towards a rigorous science ofinterpretable machine learning. arXiv:1702.08608 [stat.ML](28 February 2017).

31. J. Schmidhuber, Neural Netw. 61, 85–117 (2015).

32. J. Von Neumann, O. Morgenstern, Theory of Games andEconomic Behavior (Princeton Univ. Press, 1944).

33. P. P. Wakker, Prospect Theory: For Risk and Ambiguity(Cambridge Univ. Press, 2010).

34. D. McFadden, “Conditional logit analysis of qualitative choicebehaviour,” in Frontiers in Econometrics, P. Zarembka, Ed.(Academic, 1973), pp. 105–142.

35. H. Fennema, P. Wakker, J. Behav. Decis. Making 10, 53–64(1997).

36. K. Mukherjee, Psychol. Rev. 117, 243–255 (2010).

37. R. A. Jacobs, M. I. Jordan, S. J. Nowlan, G. E. Hinton, NeuralComput. 3, 79–87 (1991).

38. N. Stewart, N. Chater, G. D. Brown, Cognit. Psychol. 53, 1–26(2006).

39. R. Bhui, S. J. Gershman, Psychol. Rev. 125, 985–1001(2018).

40. J. Deng, W. Dong, R. Socher, L. Li, K. Li, L. Fei-Fei, “ImageNet:A large-scale hierarchical image database,” in IEEE Conferenceon Computer Vision and Pattern Recognition (IEEE, 2009),pp. 248–255.

41. A. Krizhevsky, I. Sutskever, G. E. Hinton, ImageNetclassification with deep convolutional neural networks. Adv.Neural Inf. Process. Syst. 25, 1097–1105 (2012).

42. T. L. Griffiths, Cognition 135, 21–23 (2015).

ACKNOWLEDGMENTS

We thank F. Callaway, N. Daw, P. Ortoleva, R. Dubey, and reviewersfor providing comments on the manuscript. Preliminary analyses ofour dataset were presented at the International Conference onMachine Learning. The analyses presented here are entirely newand go substantially beyond that early work. Funding: This workwas supported by the Future of Life Institute, the OpenPhilanthropy Foundation, the NOMIS Foundation, DARPA(cooperative agreement D17AC00004), and the National ScienceFoundation (grant no. 1718550). Author contributions:Conceptualization: J.C.P., T.L.G., D.D.B., D.R.; Data curation: D.D.B.;Formal analysis: J.C.P., D.D.B., M.A., T.L.G.; Funding acquisition:T.L.G.; Investigation: J.C.P., D.D.B., M.A.; Methodology: J.C.P.,T.L.G.; Project administration: D.R., T.L.G.; Software: J.C.P., D.D.B.;Supervision: T.L.G., D.R.; Visualization: J.C.P., M.A.; Writing –

original draft: J.C.P., M.A.; Writing – review and editing: J.C.P.,T.L.G., D.D.B., M.A., D.R. Competing interests: The authorsdeclare no competing interests. Data and materials availability:All data are available to the public without registration athttps://github.com/jcpeterson/choices13k.

SUPPLEMENTARY MATERIALS

science.sciencemag.org/content/372/6547/1209/suppl/DC1Materials and MethodsFigs. S1 to S11Tables S1 to S7References (43–68)MDAR Reproducibility Checklist

10 August 2020; accepted 6 May 202110.1126/science.abe2629

Peterson et al., Science 372, 1209–1214 (2021) 11 June 2021 6 of 6

RESEARCH | REPORT

Corrected 11 June 2021. See full text. on June 12, 2021

http://science.sciencemag.org/

Dow

nloaded from

decision-makingUsing large-scale experiments and machine learning to discover theories of human

Joshua C. Peterson, David D. Bourgin, Mayank Agrawal, Daniel Reichman and Thomas L. Griffiths

originally published online June 10, 2021DOI: 10.1126/science.abe2629 (6547), 1209-1214.372Science 

, abe2629, this issue p. 1209; see also abi7668, p. 1150Scienceimplications for theory generation in other domains.power, and generate new theories of decision-making (see the Perspective by Bhatia and He). This method has

leverage machine learning to evaluate classical decision theories, increase their predictiveet al.theories. Peterson distinguish from each other and offer limited improvement in accounting for patterns in decision-making over earlier

Theories of human decision-making have proliferated in recent years. However, these theories are often difficult toDiscovering better theories

ARTICLE TOOLS http://science.sciencemag.org/content/372/6547/1209

MATERIALSSUPPLEMENTARY http://science.sciencemag.org/content/suppl/2021/06/09/372.6547.1209.DC1

CONTENTRELATED http://science.sciencemag.org/content/sci/372/6547/1150.full

REFERENCES

http://science.sciencemag.org/content/372/6547/1209#BIBLThis article cites 51 articles, 2 of which you can access for free

PERMISSIONS http://www.sciencemag.org/help/reprints-and-permissions

Terms of ServiceUse of this article is subject to the

is a registered trademark of AAAS.ScienceScience, 1200 New York Avenue NW, Washington, DC 20005. The title (print ISSN 0036-8075; online ISSN 1095-9203) is published by the American Association for the Advancement ofScience

Science. No claim to original U.S. Government WorksCopyright © 2021 The Authors, some rights reserved; exclusive licensee American Association for the Advancement of

on June 12, 2021

http://science.sciencemag.org/

Dow

nloaded from