Dont Trust Your Gut

by Eric Bonabeau

Reprint r0305j

aking high-stakes business decisions has always been hard. But in recent decades,

as the complexities of global commercehave deepened, its become tougher thanever. The choices facing managers andthe data requiring analysis have multi-plied even as the time for analyzingthem has shrunk.

One decision-making tool humanintuitionseems to offer a reliable alter-native to painstaking fact gathering and analysis. Encouraged by scienticresearch on intuition, top managers feelincreasingly condent that, when facedwith complicated choices, they can justtrust their gut. Indeed, a survey that wasconducted in May 2002 by executivesearch rm Christian & Timbers revealsthat fully 45% of corporate executivesnow rely more on instinct than on factsand gures in running their businesses.Decision-making consultant Gary Klein,in his book Intuition at Work, expresses

2 Copyright 2003 by Harvard Business School Publishing Corporation. All rights reserved.

M

F R O N T I E R S

DontTrust Your Gut

by Eric Bonabeau

Intuition plays an important role in decision making,

but it can be dangerously unreliable in complicated

situations. A new set of analytical tools can help you

leverage your instinct without being sabotaged by

its weaknesses.

the common wisdom when he says thatintuition is at the center of the decision-making process and that analysis is, atbest,a supporting tool for making in-tuitive decisions.

The trust in intuition is understand-able. People have always sought to puttheir faith in mystical forces when con-fronted with earthly confusion. But itsalso dangerous. Intuition has its placein decision making you should not ig-nore your instincts any more than youshould ignore your consciencebut any-one who thinks that intuition is a substi-tute for reason is indulging in a riskydelusion. Detached from rigorous analy-sis, intuition is a ckle and undepend-able guideit is as likely to lead to disas-ter as to success. And while some haveargued that intuition becomes morevaluable in highly complex and change-able environments, the opposite is ac-tually true. The more options you haveto evaluate, the more data you have to

ition refers to the brains process of in-terpreting and reaching conclusionsabout phenomena without resorting toconscious thought. And further, its usu-ally assumed that this process draws on the minds vast storehouse of mem-ories. Bruce Henderson, founder of theBoston Consulting Group, may have putit best when, in 1977, he called intuitionthe subconscious integration of all theexperiences, conditioning, and knowl-edge of a lifetime, including the culturaland emotional biases of that lifetime.

Its certainly true that the mind is amarvelous processor of information we would be lost in the world withoutits hidden stream of calculations. Butits also true, as Henderson intimated,that its an imperfect processor. Scholarsof human cognition have shown thatour thinking is subject to all sorts of biases and aws, most of which operateat a subconscious level at the level, inother words, of intuition. We naturallygive more weight to information thatconrms our assumptions and preju-dices, for example, while dismissing information that would call them intoquestion. Were also creatures of the sta-tus quo, drawn to conclusions that jus-tify and perpetuate current conditionsand repelled by anything that would roilthe waters. And were irrationally inu-enced by the rst information we re-ceive on a particular subjectit becomes,as decision researchers put it, the an-chor that determines and distorts howwe process all subsequent data.

The most dangerous of these aws,when it comes to intuition, is our deep-seated need to see patterns. The mindswell-documented facility for patternrecognition seems to lie at the very coreof intuition its how the brain synthe-sizes information from the past and usesit to understand the present and antic-ipate the future. But it can get us intotrouble. Researchers have shown thatour unconscious desire to identify pat-terns is so strong that we routinely per-ceive them where they dont in factexist. When confronted with a new phe-nomenon, our brains try to categorize it based on our previous experiences,to t it into one of the patterns stored

may 2003 3

weigh, and the more unprecedented thechallenges you face, the less you shouldrely on instinct and the more on reasonand analysis.

That brings us back to the essentialconundrum facing todays harried ex-ecutive: How do you analyze more inless time? The answer may lie, it nowappears, in technology. Powerful newdecision-support tools can help execu-tives quickly sort through vast numbersof alternatives and pick the best ones.When combined with the experience,insight, and analytical skills of a goodmanagement team, these tools offercompanies a way to make consistentlysound and rational choices even in theface of bewildering complexitya capa-bility that intuition will never match.

Intuitions Allure

The stories are certainly seductive. FredSmith has an insight into the transportbusiness and, despite widespread skep-ticism,goes on to create Federal Express.Michael Eisner hears a pitch for an off-beat game show and, knowing in hisheart its going to be a blockbuster, im-mediately commits millions to devel-oping Who Wants to Be a Millionaire?George Soros senses in his bones a bigshift in currency markets and, acting onthat hunch, makes a billion-dollar kill-ing. Robert Pittman has a vision of thefuture of on-line media while taking ashower and rushes to lead America On-line in an entirely new direction.

The reason such tales (whether apoc-ryphal or not) have become businesslegends is that we want to believe in thetransformative power of intuition. Forone thing, its romantic. It raises busi-ness above the drab world of spread-sheets and income statements and turnsit into something of an art form. The ex-ecutive ofce becomes a place of inspi-ration and vision rather than planningand number crunching. For another, itsimplies. It says that we neednt worryif we cant decipher complex challengesrationally our subconscious mind willautomatically deliver the right answer.We just need to relax, close our eyes,and let the magic happen.

Finally, it makes us feel special. Any

idiot can run the numbers, but the giftof a good gut thats reserved for thetrue business elite. Two years ago inthese pages, Johnson & Johnson CEORalph Larsen gave voice to this com-mon, if unproven, assumption: Veryoften, people will do a brilliant job upthrough the middle management lev-els, where its very heavily quantitativein terms of the decision-making. Butthen they reach senior management,where the problems get more complexand ambiguous, and we discover thattheir judgment or intuition is not whatit should be.What better way to justifya high status and a huge salary thanto claim the superhuman power of ex-ceptional instinct.

But our desire to believe in the wis-dom of intuition blinds us to the less ro-mantic realities of business decisionmaking. We remember the examples ofhunches that pay off but convenientlyforget all the ones that turn out badly.FedExs Fred Smith also launched Zap-Mail, a proprietary network for faxtransmissions that bombed. Michael Eis-ner was responsible for the debacle ofthe EuroDisney opening, not to men-tion recent box-ofce turkeys The Coun-try Bears and Treasure Planet. GeorgeSoros lost a fortune speculating in Rus-sian securities in the late 1990s and thenpromptly lost another one betting ontech stocks in 2000. And as for AOLsPittman, his instinctive belief that thecompanys future lay in advertisingrather than subscriptions now appearsto be less a brilliant insight than a bril-liant mistake and one of the reasonshes no longer employed at AOL. Theunhappy fact that wed prefer not toadmit to ourselves is this: For every ex-ample of a great gut decision, theresan equal and opposite example of a ter-rible one.

Our Untrustworthy Gut

Critiques of intuition are complicatedby the fact that intuition is such aslippery word. Its denition can bestretched to mean almost anything,from innate instinct to professionaljudgment to plain-old common sense.But people generally agree that intu-

in our memories. The problem is that, inmaking that t, we inevitably lter outthe very things that make the new phe-nomenon new we rush to recycle thereactions and solutions from the past.

That instinct, seemingly hardwiredinto our thinking by evolution, is ex-tremely useful in life-or-death situationswhere ne distinctions are irrelevant.If you were a caveman and had seenstrange animals maul other cavemen inthe past, then it would probably be wisefor you to ee from any strange animalyou happened to come across even ifyoud never seen the beast before. Thebenet of a careful analysis of the situ-ation would be far outweighed by therisk of inaction. But managers are notcavemen. In complex business situa-tions, ne distinctions do matteroften,theyre precisely what separates successfrom failure. If you try to interpret acompetitive threat or market upheavalby simply squeezing it into an old pat-tern, youre likely to miss what makes it differentand take the wrong action.Intuition is a means not of assessingcomplexity but of ignoring it. Thatsvaluable if youre a reghter in a burn-ing building or a soldier on a battleeld.Its not valuable if youre an executivefaced with a pressing decision about in-vesting millions in a new product for arapidly changing market.

The more complex the situation, themore misleading intuition becomes. Ina truly chaotic environment wherecause and effect no longer have a linearrelationship the last thing you want to do is try to apply patterns to it. Theessence of such an environment is thelack of any discernible pattern in its evo-lution. In his McKinsey Quarterly articleOn the Origin of Strategies,consultantEric Beinhocker put it this way: Theproperties of complex adaptive systemspresent particular challenges to the de-velopment of business strategy becausepeople have a natural tendency to look

for patterns. Indeed, the human driveto nd patterns is so strong that theyare often read into perfectly randomdata. Moreover, human beings like toassume that cause directly precedes ef-fect, which makes it difcult to antic-ipate the second-, third-, and fourth-order effects of path dependence.If youmake an intuitive decision that turnsout well in such a situation, its becauseyoure lucky, not gifted. And sooner orlater probably sooner your luck isgoing to run out. Just ask your averageday trader.

The instinctive rush to apply a pat-tern to a phenomenon can also cut offor narrow an individuals or a groupsthinking too quickly. Impatient withambiguity, the mind naturally seeks clo-surethat seems, in fact, to be one of themain functions of intuition but an in-telligent decision-making process oftenrequires the sustained exploration ofmany alternatives. You want to keep theprocess open as long as possible beforeconverging on a nal choice. Thats hardto do when your gut or your bosssgut is giving you The Answer.

Intuition presents another, even moreinsidious problem: It masks me-toothinking. We like to assume that our in-tuition is uniquely our own, a distilla-tion of our particular experiences andinsights. But while that may have beentrue a century ago, when people ledvery different lives depending on wherethey lived and what they did, its nolonger the case. In todays global village,with its instantaneous and unceasingcommunications, human existence hasbecome homogenized we share thesame experiences, the same opinions,even the same thoughts. We live in avast echo chamber, and the voice of intuition we hear inside our heads is increasingly the same voice that speaksto everyone else. If, in making businessdecisions, we blindly follow its counsel,well end up mimicking our competitorsrather than creating strategies that dis-tinguish us and bring us prots.

Expanding the Mind

So, if we cant rely on our intuition buthave neither the time nor the mental ca-

pacity to carefully analyze all the facetsof a complex situation, how in the worldcan we make smart choices? Technol-ogy may hold the key. Sophisticatedcomputer programs are now being de-veloped that can supplement and bol-ster peoples decision-making skills.Many of these new decision-supporttools are still in the early stages of de-velopment and have yet to be applied to strategic business decisions. But theyhold enormous potential for helping executives carry out the two key com-ponents of decision-making or problem-solving exercises: searching for possiblesolutions and evaluating those solutionsin order to choose the best one or ones.The more complex and fast-changingthe situation, the more challenging bothsearch and evaluation become. By ex-panding the analytical as well as the in-tuitive capabilities of the mind, the newprograms allow a much faster, a muchfuller, and a much more rigorous explo-ration of the options. (See the sidebarSearch and Evaluate for an overviewof traditional and emerging decision-support tools.)

Decision Sciences. The traditionaltools of decision sciences system dy-namics, decision trees, real options, port-folio management, and so on con-stitute an important class of rationaldecision-making techniques that can beinvaluable when youre faced with lotsof options. They often lead to muchmore dependable decisions than doesinstinct alone. But they have their lim-its. Their workings are often so mysteri-ous to executives that they can seem likeblack boxes. And in highly complex sit-uations when there are many depen-dencies among possible solutions or noclear way of measuring the solutionsvalue traditional tools become un-wieldy and tend to provide unreliableanswers.

To use decision trees in the pharma-ceutical industry, for example, you haveto assume you know a drugs commer-cial value ten years before it hits themarket. And decision trees and otherdecision-science tools cant adequatelyaccount for emergent phenomena orchance events, such as the discovery that

4 harvard business review

F R O N T I E R S Dont Trust Your Gut

Eric Bonabeau, a frequent contributor to HBR, is the chief scientist of Icosystem,a strategy consulting rm in Cambridge,Massachusetts. Bonabeau can be reachedat eric@icosystem.com.

a drug developed for one disease canbe used to treat another, very differentdisease.

Agent-Based Modeling. Isaac New-ton, after losing his savings in the SouthSea Bubble of 1720, bemoaned the factthat I can calculate the motions of theheavenly bodies, but not the madnessof people. Many managers today arein the same quandary as Newton wasalmost 300 years ago. They have tomake decisions about complex systemswith many interrelated, yet unpredict-able, elements. Global markets, large or-ganizations, supply chains, technologynetworksall can seem impenetrable totraditional forms of analysis.

But agent-based modeling can shedlight on the workings and evolution ofsuch systems. In an agent-based simula-tion,a computer creates thousands,even

millions, of individual actors; each ofthese virtual agents makes decisions,providing an accurate model of a com-plex systems dynamics. Agent-basedmodeling allows you, literally, to do whatNewton couldnt: predict the madnessof crowds. (For more on agent-basedmodeling, see my HBR article Predict-ing the Unpredictable, March 2002.)

Southwest Airlines is using an agent-based model to revamp its rules for han-dling cargo, reaping $2 million in an-nual labor savings in the process. EliLilly is using one to model early-phasedrug development, leading to the crea-tion of organizational forms that prom-ise to boost productivity and enhancespeed. Pacic Gas and Electric is usingan agent-based model to better managethe ow of electrons through its vastpower grid, saving money and avoidingservice disruptions.

In the coming years, agent-basedmodels will no doubt be used to gener-ate scenarios for the evolution of mar-kets and competition, the dynamics ofwhich hinge on the decisions made by

many players. These scenarios can be-come the basis for evaluating a multi-tude of strategic and tactical options and they can be used to put executivesintuitive choices to the test.

Articial Evolution. The best systemever devised for making choices froman almost innite set of alternatives isevolution itself. The basic process of evo-lutiontaking the best-available optionsand then combining and mutating themto create even better onesis now beingincorporated into a type of analyticalsoftware known as articial evolution,or evolutionary computation. This tech-nology uses the computational powerof computers to both search out a vastnumber of solutions and evaluate them.

To see how it works, imagine that yourun a factory and have to determine theproduction schedule that will maximize

the plants output within a given period.You start by randomly generating somealternative schedules their qualitymakes no difference at this point andfeeding them into articial-evolutionsoftware. The software evaluates howwell each schedule performs in maxi-mizing output, picks the few that per-form best, and randomly pairs them formating.The resulting large set of alter-native schedules combines the charac-teristics of the prior generation whileintroducing some random character-istics as mutations. It searches out, inother words, a large new set of possiblesolutions. The software evaluates the so-lutions, and the ones that perform bestin maximizing output are selected foranother round of mating. As more andmore generations go byand computerscan crank through the process in min-utes the resulting schedules becomebetter and better. John Deere alreadyuses this kind of system to help opti-mize its manufacturing operations, andMexican cement producer Cemex uses a similar system to route its trucks.

Interactive Evolution. In the plant-scheduling example, alternatives couldbe judged with an objective measure factory output. As decisions becomemore strategic, however, the criteria forsuccess become more complex and sub-jective. You cant just run the numbers;you have to incorporate the expertise,judgment, and, yes, intuition of sea-soned professionals. You have to bringpeople into the evaluation stage of thedecision-making process. That can be accomplished with interactive evolu-tion, a variation of articial evolution.The basic difference is that a person orgroup of people, rather than a computer,judges each generation of alternatives.

One major automobile manufactureris using interactive evolution to aid innew-car design. That process is highlycomplex because car designers have tosatisfy hundreds of technical constraints,such as wheelbase length, windshieldangle, and engine compartment size,while also being creative in both engi-neering and aesthetics. When designershave to do this without the help of tech-nology, it is extraordinarily time con-suming. They have to test every deci-sion against all sorts of variables, and asa result they can consider only a smallset of options. But interactive-evolutionsoftware can pump out iterations of newdesigns very quickly. The designers ex-amine each set of alternatives and, usingsubjective aesthetic judgments in addi-tion to the computers objective mea-sures, choose the best ones for the nextround of mating.

Other companies, like Procter &Gamble and Pepsi-Cola North America,are using interactive evolution to createnew product and packaging designs but theyre using customers rather thanemployees to pick out the best optionsfrom each generation. One can easilyimagine a similar process for high-levelstrategic decisions that leverages the in-sights of an executive team to continu-ously rene plans.

Open-Ended Search. Articial and interactive evolution are both optimi-zation processes. Alternative designs aregenerated by varying a small set of pa-rameters, and those designs are evalu-

may 2003 5

Dont Trust Your Gut F R O N T I E R S

The best system ever devised for making choices from

an almost innite set of alternatives is evolution itself.

Search

Eval

uat

ion

Simple (Human Processing)

Complex(Computer Processing)

interactive open-ended search

optimization

open-ended articial evolution

gut decisions

Sim

ple

(Hu

man

Pro

cess

ing)

Com

plex

(Com

pute

r P

roce

ssin

g)

interactive evolution by consumers

interactive evolution by experts

data mining

agent-based modeling

decision sciences(trees, real options, etc.)

simulation modeling

mock markets

design

spreadsheet modeling

advocacy

scenario planning

consultants

behavioral observation

6 harvard business review

F R O N T I E R S Dont Trust Your Gut

Search

Eval

uat

ion

Simple Complex

Sim

ple

Com

plex

few options, complex consequences

few options, simple consequences

many options, complex consequences

many options, simple consequences

Search and EvaluateMaking a decision or solving a problem entails two tasks.

First, you have to search for potential solutions (a task that in-

cludes framing the problem and establishing a set of working

assumptions about it). Second, you have to evaluate the solu-

tions and choose one. Each of these tasks is subject to varying

levels of complexity. If, for example, a problem has only a few

solutions but each solution has myriad consequences, the

search will be relatively simple but the evaluation will be ex-

tremely complex. The small gure below provides a simple,

but useful, grid for categorizing problems according to the de-

gree of complexity (for a human being) of the search and the

evaluation tasks.

The more complex the search or the evaluation, the more

difcult it becomes for a person to carry it out the required

computations outstrip the minds processing capabilities. In

such cases, some people will mistakenly rely on their intuition

to simplify their choices; theyll narrow their options or make

a choice based on their gut. But intuition is particularly un-

reliable in complex situations. A much better approach, when

youre faced with a complex search or evaluation, is to sup-

plement the minds analytical and intuitive capabilities with

a computational decision-support tool.

The large gure categorizes both traditional and emerging

decision-support methods and tools in terms of how they

apply to different situations. There are many such tools, rang-

ing from real options to visualization software, in common

use today. Most traditional tools (indicated by white type)

have limited applicability in highly complex situations; theyre

best applied to problems that fall into or near the lower-left

quadrant those requiring relatively simple searches and

evaluations. As we move outward on the complexity scale, we

need to look to new, computer-based computational tools,

such as open-ended search (when there are lots of potential

solutions), agent-based modeling (when the consequences

requiring evaluation are complex) or articial evolution

(when both search and evaluation are highly complex).

ated against a set of criteria objective,subjective, or both. But sometimes youdont know which parameters to use togenerate alternatives, or the number ofparameters is so large that its impossi-ble to reliably sample the entire set ofpossible solutions. In such cases, anothernew computational technique open-ended search, or evolutionary design can be applied to sort through and togenerate options. As its name implies,open-ended search focuses on the initialsearch for options rather than on theirsubsequent evaluation. It has enormouspotential for helping managers makedecisions in highly complex situationsbecause it offers a way to generate op-tions that would be invisible to even themost capacious mind.

Stanford professor John Koza has de-veloped a type of open-ended search,called genetic programming, for use increating electronic circuits. The numberof possible circuits is huge, and its im-possible to characterize all of them withjust a few parameters. Using a smallnumber of parameters (which is all themind can handle) restricts the search toa tiny, predened subset of circuits, pre-cluding truly creative solutions fromemerging.Genetic programming,by con-trast, dis-integrates circuits into theircomponent building blocks diodes,ampliers, resistors, and so forth thenuses a computer to breed alternative cir-cuits by combining and recombining thecomponents.

The process has generated radically

new designs ones that would neverhave been discovered by simply judgingcomplete circuits against traditionalperformance criteria. Koza and his col-leagues at Genetic Programming in LosAltos, California, have recently beenusing the technique to create circuitsthat replicate the functionality of othercircuits without infringing on existingpatents a development that could, forbetter or worse, revolutionize the micro-chip industry.

My rm, Icosystem, has begun help-ing a major petrochemical company useopen-ended search to evaluate pricingstrategies for one of its most importantproducts. The products pricing has totake into account many factors. Theseinclude upstream commodity prices,downstream nished-product prices,demand at various stages in the valuechain, currency uctuations, and com-petitor prices, all of which can changerapidly and unpredictably. As with theelectronic-circuit example, the open-ended design begins with the disaggre-gation of an initial group of pricingstrategies (which the company collectsfrom various pricing experts) into theircomponent parts. In this particular case,the parts take the form of pricing rules,as follows: If volume is > 100, thenprice = x, for instance; or, If winter iscold, price decreases.

To this primordial soup are addedrandom rules some of which directlycontradict the experts rules to addgreater genetic diversity to the mix. Acomputer creates random combinationsof the rules to produce a new set ofstrategies for testing. In this way, thecomputer can quickly explore millionsof combinations, producing innovativestrategies that go well beyond anythingthat might have come out of the con-scious or subconscious minds of eventhe savviest marketers. And, again, itseasy to see how open-ended searchcould be applied to complex strategicchallenges that have many possible so-lutions. Just as with interactive evolu-tion, people can aid in the evaluation ofthe options generated by open-endedsearch. The technique offers a rationalway for managers to approach the most

difcult business problems: those thathave unbounded options with no well-dened criteria for success.

Beyond Intuition

These new decision-support tools donteliminate human intuition; they har-ness its power while remedying its mostpernicious aws. The instincts of smartexecutives and other professionals areincorporated into the process theyreused either to generate initial options orto aid in judging computer-generatedones. But these instincts are subjected tothe rigors of analysis and at the sametime freed from the brains constraintsin imagining possible solutions. Com-puters impose left-brain discipline onright-brain hunchesin a way thats wellbeyond the computational capacity ofthe human mind. Intuition is thus al-lowed to inform decision making with-out short-circuiting or otherwise con-straining it.

But theres more to it than that. Ulti-mately, computers may not just amplifythe minds analytical capabilities; theymay expand its creative potential aswell. And they may allow us to breakthrough the interpretation barrierourdemand that our creations be intelligi-ble to us.

Think about it. When we create de-signs, whether for products or strategies,we are limited by our ability to under-stand those designs their workingsmust be transparent to us. But if we lookat nature, we quickly nd that some ofits greatest creations are opaque theylie beyond our understanding. Thatstrue of the human mind itself, perhapsthe greatest creation of all. We dontknow how it works; we just know that itworks extraordinarily well. Techniqueslike articial evolution and open-endeddesign can also generate designs thatwe cant explain but that produce re-sults beyond even the limits of our imag-inations. They offer, it might be said,the true fulllment of the promise ofhuman intuition.

Reprint r0305j; HBR OnPoint 3604To place an order, call 1-800-988-0886.

For Further Reading

Alden M. Hayashis When to Trust Your Gut (HBR, February 2001, ReprintR0102C) provides a lucid overview of current thinking on how intuition works.

David G. Myers Intuition: Its Powers and Perils (Yale University Press, 2002) offers a lively and thorough review of the pow-ers and pitfalls of gut instinct.

For a good introduction to the uncon-scious biases in our thinking, see John S.Hammond III, Ralph L. Keeney, andHoward Raiffas The Hidden Traps inDecision Making (HBR, SeptemberOctober 1998, Reprint 98505).

may 2003 7

Dont Trust Your Gut F R O N T I E R S