Probable Cause, Probability, and Hindsight...Probable Cause, Probability, and Hindsight jels_ 1230...

Probable Cause, Probability, and Hindsightjels_

1230 72..9872..98

Jeffrey J. Rachlinski, Chris Guthrie, and Andrew J. Wistrich*

When judges assess probable cause, they must do so either in foresight (when determiningwhether to issue a warrant) or in hindsight (when determining whether to allow theadmission of evidence obtained without a search warrant). Although the legal standard forprobable cause is the same, and the facts that might support cause are the same, judges whoassess probable cause in hindsight invariably know whether a search produced incriminatingevidence or not. Research on the hindsight bias suggests that judges will be unable to setaside this knowledge and judge probable cause as if they were working in foresight. In thisarticle, we present three experiments in which we asked 900 state and federal judges tomake judgments of probable cause either in foresight or in hindsight, in hypothetical cases.Surprisingly, we found that that judges make similar rulings on probable cause in foresightand in hindsight. We also found that hindsight appears to cloud judges’ abilities to assess thelikely outcome of the search, but hindsight does not influence their legal judgments.

I. Introduction

The Fourth Amendment’s prohibition against unreasonable searches and seizures createsa decision-making dilemma for judges.1 It requires that judges supervise police investiga-tions. The centerpiece of the Fourth Amendment’s protection of personal privacy lies inthe requirement that the police must obtain permission to conduct such searches byobtaining a warrant from a judge.2 But in the common circumstance in which an exigencyprevents a police officer from obtaining a warrant, the judge must assess probable cause

*Address correspondence to Jeffrey J. Rachlinski, Professor of Law, Cornell Law School, Myron Taylor Hall, Ithaca,NY 14853; email: [email protected]. Guthrie is Dean and John Wade-Kent Syverud Professor of Law, VanderbiltUniversity Law School; Wistrich is Magistrate Judge, U.S. District Court, Central District of California.

1The Fourth Amendment states: “The right of the people to be secure in their persons, houses, papers, and effects,against unreasonable searches and seizures, shall not be violated, and no Warrants shall issue, but upon probablecause, supported by Oath or affirmation, and particularly describing the place to be searched, and the persons orthings to be seized.”

2Chimel v California, 395 U.S. 752, 762 (1969) (“The police must, whenever practical, obtain advance judicialapproval of searches and seizures through the warrant procedure.”); Katz v. United States, 389 U.S. 347, 357 (1967)(“searches conducted outside the judicial process, without prior approval by judge or magistrate, are per se unrea-sonable under the Fourth Amendment”). See William J. Stuntz, Warrants and Fourth Amendment Remedies, 77 Va.L. Rev. 881, 882 (1991).

Journal of Empirical Legal StudiesVolume 8, Issue S1, 72–98, December 2011

72

while knowing that the search was successful.3 Decades of research on judgment in hind-sight suggest that such judgments will be biased, as people cannot suppress the influence ofknown outcomes on their judgments—a phenomenon known as the hindsight bias.4 In thisarticle, we present three experiments in which we asked groups of state and federal judgesto make judgments of probable cause either in foresight or in hindsight, in hypotheticalcases. Surprisingly, we found when making probable cause determinations, judges do notseem to be influenced by the hindsight bias.

Our result is surprising. The nature of probable cause poses a serious cognitivechallenge for judges in implementing their role as the guardians of the Fourth Amend-ment. The cornerstone of reasonableness in searches is the concept of “probable cause.”5

Commonly, a judge will assess probable cause before the search has been conductedbecause the police normally have to obtain a warrant before conducting a search. But withnumerous exceptions to the warrant requirement that arise from the inevitable exigenciesof law enforcement, the police conduct a search without obtaining a warrant.6 When facedwith such an exigency, the police may avoid the need for a warrant, but they must still haveprobable cause for the search.7 The purpose behind allowing an exigency to exempt thepolice from the warrant requirement is a practical one, but is not supposed to allow thepolice to be subject to a more lenient standard of review. When searches conducted withouta warrant produce incriminating evidence to be used against a criminal defendant, thejudge must assess probable cause in full knowledge that the search uncovered incriminatingevidence. Judges must assess the facts just as if they did not know that the search uncoveredincriminating evidence. The standard remains the same, but the perspective from whichjudges review a case inevitably differs in hindsight.

Judging the reasonableness of a police search in hindsight obviously places greatpressure on a judge to side with the police. A judge who refuses to issue a warrant inforesight impedes police investigation somewhat. But concluding that a search was unrea-sonable in hindsight, however, is apt to lead to the suppression of important evidenceagainst a defendant who likely committed a crime. The consequences of concluding that

3See Craig M. Bradley, Two Models of the Fourth Amendment Remedies, 83 Mich. L. Rev. 1468, 1473 (1985).

4See Baruch Fischhoff, Hindsight � Foresight: The Effect of Outcome Knowledge on Judgment Under Uncertainty,1 J. Experimental Psychol. 288 (1975) (first documenting the hindsight bias); Jeffrey J. Rachlinski, A PositivePsychological Theory of Judging in Hindsight, 65 U. Chi. L. Rev. 571, 576–88 (1998) (reviewing the research on thehindsight bias).

5See Wayne R. LaFave, Jerold Israel & Nancy King, Criminal Procedure 144–45 (4th ed. 2004) (arguing that probablecause determinations are central to the Fourth Amendment); but see Andrew E. Taslitz, Margaret L. Paris & LenseeHerbert, Constitutional Criminal Procedure 186–97 (4th ed. 2010) (showing a central role for probable cause, butexpressing concern about numerous exceptions).

6See Bradley, supra note 3, at 1473–74 (cataloging numerous exceptions to the warrant requirement); Donald L. Beci,Fidelity to the Warrant Clause: Using Magistrates, Incentives, and Telecommunications Technology to ReinvigorateFourth Amendment Jurisprudence, 73 Denver U.L. Rev. 293, 295 (1996) (arguing that the exceptions to the warrantrequirement have swallowed the rule).

7LaFave et al., supra note 5, at 144–45.

Probable Cause, Probability, and Hindsight 73

probable cause did not exist for a search when judging in hindsight thus might be muchmore serious than concluding that probable cause does not exist in foresight.

Even if judges are motivated to uphold the Fourth Amendment’s restrictions andmake the difficult choice to suppress incriminating evidence in hindsight, they still facesignificant cognitive impediments to judgment. People who know how events unfoldedcome to believe that the course of events was unavoidable, and even predictable.8 In morethan 150 published studies, when people provide judgments in hindsight, they assigngreater probabilities to the actual outcome than they would have had they been asked topredict how events unfolded without knowing the outcome.9 Hindsight biases judgment.

Most of the research on the hindsight bias puts research participants in much thesame difficult position as trial judges. That is, after people learn how events actually unfold,they want to believe that they and others could have predicted the outcome, just as judgeswould want to think that the outcome of a search was predictable.10 But learning how eventsunfold also teaches people about the circumstances that produced that outcome. Anteced-ent events come to seem relevant in different ways after the course of events comes to beknown—a process that has been called “creeping determinism.”11 Because this process issomewhat transparent, even people who are motivated to ignore the known outcomecannot also ignore what the outcome has taught them about the events that preceded it. Toignore the fact that a search produced incriminating evidence, a judge must be able toignore what come to seem like compelling antecedent circumstances that might supportconducting a search. An innocent or ambiguous observation by a police officer in foresightis apt to seem more sinister once the judge knows that it is actually a sign that incriminatingevidence exists. But to replicate the judgment that a judge would have made in foresight,a judge must attach just as much weight to that observation in hindsight as a judge wouldhave in foresight.

Some scholars have suggested that the hindsight bias likely influences probable causedeterminations.12 Because probable cause itself incorporates an assessment of the likelyoutcome of the search, it seems vulnerable to the influence of the hindsight bias, whichinfluences probability judgments. One paper even suggests that the primary purpose of

8See Fischhoff, supra note 4.

9See Jay J.J. Christensen-Szalanski & Cynthia Fobian Willham, The Hindsight Bias: A Meta-Analysis, 48 OrganizationalBehav. & Hum. Decision Processes 147 (1991) (compiling studies of the bias).

10See Rachlinski, supra note 4, at 582–83 (reviewing motivational accounts of the hindsight bias).

11See Fischhoff, supra note 4, at 288 (referring to the hindsight bias as “creeping determinism”); Scott A. Hawkins &Reid Hastie, Hindsight: Biased Judgments of Past Events After the Outcomes are Known, 107 Psychol. Bull. 311, 322(1990) (explaining “creeping determinism”).

12Peter Henning et al., Mastering Criminal Procedure 46 (2010); Andrew E. Taslitz & James Coleman, Foreword: TheDeath of Probable Cause, 73 Law & Contemp. Prob. i, viii (Summer 2010) (“Ample psychological theory andempirical albeit mostly in other contexts, supports the idea that hindsight bias is at work in the probable-causedetermination.”); Bryan D. Lammon, Note, The Practical Mandates of the Fourth Amendment: A BehavioralArgument for the Exclusionary Rule and Warrant Preference, 85 Wash. U.L. Rev. 1101 (2007).

74 Rachlinski et al.

requiring that the police obtain a warrant before a search occurs is to avoid the influenceof hindsight on the judge’s judgment.13

Even though researchers have demonstrated the influence of the hindsight bias in awide variety of settings with numerous different types of research participants, judgesassessing probable cause might be different. Unlike most people, judges are professionaldecisionmakers. The core element of their job is decision making. Their role also demandsthat they make their best efforts to follow professional norms. Judges are almost invariablylawyers with a great deal of legal training and legal experience. In previous research onjudges, we have found that these factors do not necessarily insulate judges from relying onmisleading cognitive strategies for assessing cases.14 In some contexts, however, we havefound that judges manage to avoid common pitfalls that plague human judgment. In short,it would be a mistake to assume that judges fall prey to the hindsight bias when making legaljudgments without researching judges.

We have studied the influence of the hindsight bias on trial judges in three previousstudies. In one study, in which we asked federal judges to review hypothetical cases, judgesoverstated their ability to predict how an appellate court would rule on an appeal.15 In asecond study using similar methods, state judges based their assessments of whether asecurity guard had improperly apprehended a teen for shoplifting based on the teen’s racelargely on whether the teen was caught with a video game tucked in his coat or not—a factthat the guard could not have known before conducting the search.16

In the third of our previous studies of the hindsight bias in judges we found that thebias did not influence judges’ assessments of probable cause.17 We report the results of thisstudy below, along with that of other efforts to explore this outcome. Although that onestudy suggests that the hindsight bias does not influence judges’ assessments of probablecause, the result is surprising, given the extensive body of research suggesting that thehindsight bias has a broad influence on similar judgments.

In this article, we present a series of studies that provide a more complete investiga-tion of the potential influence of this bias in probable cause judgments. It extends ourprevious work in three important respects. First, we replicate the earlier finding with a

13Lammon, supra note 12.

14See Chris Guthrie, Jeffrey J. Rachlinski & Andrew J. Wistrich, Inside the Judicial Mind, 86 Cornell L. Rev. 777, 779–80(2001) (reporting experimental evidence showing that judges are susceptible to heuristics and biases when makingjudgments); Jeffrey J. Rachlinski, Chris Guthrie & Andrew J. Wistrich, Inside the Bankruptcy Judge’s Mind, 86 B.U.L.Rev. 1227, 1229–30 (2006) (exploring whether specialized bankruptcy judges are similarly susceptible to heuristicsand biases).

15Guthrie et al., supra note 14, at 799–804.

16Chris Guthrie, Jeffrey J. Rachlinski & Andrew J. Wistrich, The “Hidden Judiicary”: An Empirical Examination ofExecutive Branch Justice, 58 Duke L.J. 1477, 1512–16 (2009).

17Andrew J. Wistrich, Chris Guthrie & Jeffrey J. Rachlinski, Can Judges Ignore Inadmissible Information? TheDifficulty of Deliberately Disregarding, 153 U. Pa. L. Rev. 1251, 1258–59 (2005) (reporting experimental evidenceshowing that judges have difficulty deliberately disregarding relevant but inadmissible evidence when making meritsdecisions).


larger sample size so as to rule out the possibility that our earlier failure to find an effect ofhindsight was attributable to a lack of statistical power. Second, we conduct research usingtwo additional scenarios, to ensure that the lack of an influence of probable cause is not anartifact of some unusual aspect of our first scenario. Third, we also ask some of the judgesto make probability judgments, rather than (or in addition to) rulings, so as to get a bettersense of whether the hindsight bias influences any aspect of their judgment.

Our results indicate that our original finding that the bias does not affect judges’assessment of probable cause determinations is a robust one. With 900 different state andfederal judges using three different scenarios, we found that the rate at which judges findprobable cause is nearly identical in foresight and hindsight. We nevertheless found that intwo of our studies, the hindsight bias did influence judges’ assessments of the likelyoutcome of a search. We thus conclude that the hindsight bias affects how judges see a caseand assess the facts, but it does not affect their legal rulings.

II. Methods

To assess whether the hindsight bias influences trial judges’ assessments of probable cause,we presented hypothetical scenarios to judges while they were attending judicial educationconferences. We asked these judges to evaluate several scenarios, one of which involved anassessment of probable cause conducted either in foresight or hindsight. We describe thejudges who participated, the procedure, and the materials in this section.

A. Participants

We recruited a total of 900 state and federal judges in attendance at 10 different judicialeducation conferences to participate in our research on probable cause. All the confer-ences were organized to provide general judicial education, and were not specificallytargeted to psychology or criminal procedure. All were sponsored entirely by a state orfederal entity charged with facilitating judicial education in their jurisdiction. In all cases,we gave our session a title indicating it would discuss “judicial decision making,” withoutspecifically flagging the nature of the presentation as addressing cognitive psychology orcriminal procedure. None of the judges (other than those judges who had helped organizethe presentation and who did not complete the questionnaire) were forewarned that wewould be administering a questionnaire.

Table 1 describes the demographic details of the judges and features of the confer-ence at which the data were collected. Table 1 reports the average years of experience ofthe judges, the percent of the judges that are female, the percent that are Democrats, theyear in which the data were collected, whether the data were collected at a plenary sessionat the conference (at which nearly all judges at the conference would have attended) or ata break-out session in which judges had the option of attending other sessions, and thestudy with which we report the data.

To obtain the political orientation of the judges, we asked all the judges except thefederal district judges the following question: “Which of the two major political parties in


the United States most closely matches your own political beliefs?” For the federal districtjudges we instead asked them to identify the political party of the president who appointedthem. We did not ask judges from the eastern jurisdiction or the first set of federalmagistrate judges about their political orientation.

We also asked the federal district judges, the second group of federal magistratejudges, and both groups of the Florida trial judges whether they had experience as aprosecutor or criminal defense attorney before becoming a judge. Among the federaldistrict court judges, 47 percent had former experience as prosecutors, 51 percent hadformer experience as defense attorneys, and 30 percent had both. Among the federalmagistrate judges, 45 percent had been prosecutors, 52 percent had served as defenseattorneys, and 31 percent had both types of experience. Among the first group of Floridatrial judges, 50 percent had been former prosecutors, 59 percent had been defense attor-neys, and 37 percent had been both. Among the second group of Florida trial judges, 51percent reported having served as former prosecutors, 59 percent as former defenseattorneys, and 32 percent as both.

All the judges are essentially general jurisdiction judges. The federal judges all heara range of civil and criminal cases. Federal magistrate judges cannot preside over criminalfelony cases, but preside over misdemeanor cases and are authorized to grant search

Table 1: Demographic Information About the Judges

Jurisdiction (Sample Size) Average Years of Experience % Female % Democrat Year Plenary? Study

U.S. magistrate judges (26) 14.2 24 n/a 2002 No 1Ohio state judges (242)a 14.2 25 38 2009 Yes 1 & 3b

Eastern U.S. (71)c 9.2 44 n/a 2002 Yes 1Federal district judges (81)d 11.3 22 57 2004 No 2U.S. magistrate judges (43) 10.0 21 15 2004 No 2Florida state judges (101) 14.4 21 56 2004 No 2Florida state judges (295)e 13.1 23 56 2006 Yes 3Arizona state judges (41)f 11.5 39 48 2011 No 3

aThe Ohio state judges included both trial and appellate judges in the sample. Owing to the small number ofappellate judges, we did not analyze the data separately, and kept the data for the appellate judges in the sample. Weasked the Ohio judges about their title to identify them, but 54 of them provided an indeterminate answer, such as“judge” or “retired.” The rest of the judges included 161 trial and 27 appellate judges.bThe judges in Ohio all reviewed the first scenario, as described in Study 1. For roughly half (130) the judges, however,we asked for a ruling on probable cause; we report the results of these judges along with those of the first group ofU.S. magistrate judges and judges from the eastern jurisdiction as Study 1. For the other half of the Ohio judges (112judges), we asked an estimate of the probability that the search would uncover (or would have uncovered) incrimi-nating evidence. We report the results of this version in Study 3.cThese judges are all from an urban jurisdiction in the eastern part of the United States who preferred not to havetheir jurisdiction identified.dThe federal district court judges come from three educational conferences for federal district court judgesorganized by the Federal Judicial Center in Philadelphia (27 judges), Chicago (16 judges), or Seattle (38 federaldistrict judges—and one federal magistrate judge) in 2004.eThe two groups of Florida state trial judges attended the annual meeting of the Florida circuit court judges in Florida.The smaller group attended a break-out session we presented at their conference in 2004 and the larger groupattended a plenary session we presented in 2006. We asked the judges in 2006 whether they had attended ourpresentation in 2004, and 19 percent reported having done so.f The Arizona judges included 30 trial judges, eight appellate judges, and three who identified themselves as retired.


warrants.18 The Ohio judges hear a range of civil and criminal cases. Both the Florida judgesand the judges from the eastern jurisdiction are also general jurisdiction judges who rotatebetween departments, including the criminal department. The judges in these jurisdictionsindicated that they rotate between types of cases, but that a small number of judges mighthave chosen to specialize entirely in family court matters, and would likely not have heardmany criminal matters.

B. Procedure and Materials

We followed the same procedure at all the conferences. Before beginning our formalpresentation, we distributed questionnaires to the judges in person on a random basis. Thequestionnaires contained several scenarios that we asked the judge to read and evaluate,one of which involved a probable cause determination. We asked the judges to read andrespond to each of the questions and to provide the demographic information describedabove.

We did not ask the judges to provide any identification and, in fact, we specificallyinstructed them not to identify themselves. We also gave the judges at each conference theopportunity to limit our use of their results to the conference itself. Most of the judgesallowed us to analyze and report their results, although a small number excluded theirresponses from further research (and we have excluded the responses of these judges fromall analysis). In all cases, we reported the results back to the judges at the conference.Overall, the judges seemed to take the process seriously. The judges worked independ-ently and quietly on the scenarios, taking between five and 20 minutes to complete thequestionnaire.

As we report in greater detail for each of the studies, below, we used three differentprobable cause scenarios. Each scenario presented facts that might support probable cause,although we varied the perspective. Judges evaluated a set of facts that might support arequest for a warrant (foresight perspective) or evaluated the same set of facts plus anassertion that a law enforcement official had conducted a search on the basis of those factsand had uncovered incriminating evidence against a criminal defendant (hindsight per-spective). In the latter case, the evidence obtained might have to be suppressed, as havingbeen the product of a search that was undertaken without probable cause. We either askedjudges to make a ruling on probable cause, to estimate the probability that the search woulduncover (or would have uncovered) incriminating evidence, or both.

Each of the three scenarios involved the search of an automobile. Automobilesrepresent a long-standing exception to the requirement that police obtain a warrant.

Police officers who face “exigent circumstances” may undertake the search evenwithout a warrant, so long as they have probable cause.19 In 1925, the U.S. Supreme Court

18We describe the role of U.S. magistrate judges in detail in an earlier publication. Guthrie et al., supra note 14, at784–86.

19See United States v. Karo, 468 U.S. 705, 718 (1984) (“[I]f truly exigent circumstances exist no warrant is requiredunder general Fourth Amendment principles.”).


determined that the inherent mobility of automobiles presents law enforcement with theexigency that would justify undertaking a search without first obtaining a warrant.20 Theautomobile exception is sufficiently well established that judges should accept the lack of awarrant in the hindsight conditions. Even so, police sometimes obtain warrants for auto-mobiles, and thus the foresight condition in which judges are asked for a warrant is equallyreasonable. Searches of automobiles should thus present a clean test of the influence of theoutcome of the search, rather than the legitimacy of a search conducted without a warrant(which would be a potential issue in many other kinds of cases). The exigency thatautomobiles present does not, however, excuse the necessity that probable cause bepresent. Absent probable cause, a judge should not issue a warrant in foresight or shouldsuppress evidence obtained in hindsight.

In all three scenarios, the key test of the influence of the hindsight bias arises froma comparison of the foresight perspective to the hindsight perspective. In all three sce-narios, the materials in foresight and in hindsight describe identical circumstances thatmight support a determination that probable cause existed for the search. In the foresightperspective, the materials ask judges whether they would grant a warrant or for an estimateof the probability that the search will uncover incriminating evidence (or both). Thematerials creating the hindsight perspective present the same facts, but add that the policeofficer searched the vehicle and uncovered incriminating evidence. The materials for thehindsight perspective ask whether the evidence should be suppressed, or for an estimate ofthe probability that search would have uncovered incriminating evidence as if the judgesdid not know the outcome of the search (or both).

III. The Studies

We report the details of the scenarios and the variations in the dependent measures in thissection. At the outset of the project, we expected that the judges in hindsight would bemore likely to conclude that probable cause was present than judges in foresight. As wereported in previous work, we did not find that. We undertook the variations in theseexperiments as an effort to assess whether that conclusion was robust, and what mightaccount for it.

A. Study 1

Our first effort was simply to determine whether judges’ rulings might be influenced by theoutcome of a potentially defective search. For this, we created and administered a scenariolabeled “Fourth Amendment Issue.” In this scenario, we asked participating judges assigned

20Carrol v. United States, 267 U.S. 132 (1925); United States v. Ross, 456 U.S. 798 (1982). See, e.g., Chambers v.Maroney, 399 U.S. 42, 51 (1970) (holding that police may search automobiles based on probable cause without awarrant); see also United States v. Johns, 469 U.S. 478, 487–88 (1985) (allowing police to search containers withouta warrant when the containers were properly seized from an automobile at an earlier time).


to the foresight condition to determine whether they would grant a warrant under thecircumstances described, and we asked participating judges assigned to the hindsightcondition to rule on the admissibility of evidence collected without a warrant under thesame circumstances. We have reported these results for two of the groups of judges (federalmagistrate judges and judges in the eastern jurisdiction) in a previous publication in whichwe concluded that the hindsight bias did not affect judges.21 We report an extension of thisstudy combined with the original results here.

1. Materials

The scenario describes a police officer on patrol in a parking lot outside a large arenahosting a rock concert. The materials indicate that the officer noticed a well-dressed,nervous-looking man exit a BMW and fiddle with something in the trunk of his car. Theman then met a friend, bought tickets to the event, and entered the arena. Thirty minuteslater, the officer noticed that one of the BMW’s windows was rolled down. Concerned thatthe car might be burglarized, he approached the car to close the window. Upon arriving atthe car, the officer stated that he “smelled something that he believed, based on a demon-stration at a training session several years earlier, to be burnt methamphetamine. He lookedinside the car and didn’t see any drugs, but he did notice some Visine, a local map, and acouple of empty beer cans.”

In the foresight perspective, the materials then stated that the police officerrequested a telephonic warrant to search the trunk of the car. The materials asked simply:“Will you issue the warrant?” Judges could then check one of two statements: “Yes, there isprobable cause to justify granting the warrant” or “No, there is not probable cause to justifygranting the warrant.”

In the hindsight perspective, the materials stated that “[b]ased on these observa-tions,” the police officer searched the trunk of the car. The search produced “10 pounds ofmethamphetamine, other drug paraphernalia, and a gun that had recently been fired.” Thegun turned out to match a weapon used to murder a drug dealer across town earlier thatevening. The driver was then arrested and tried. The materials stated that the defenseattorney moved to suppress the evidence obtained in the search, arguing that the policeofficer lacked probable cause for the search. The materials then asked: “Will you rule tosuppress the evidence?” Judges could then check one of two statements: “Yes, there wasno probable cause to justify the search” or “No, there was probable cause to justify thesearch.”22

21Wistrich et al., supra note 17, at 1314–18.

22There was an error in the phrasing of the two available answers in the version of this question given to the unnamedjurisdiction that makes the answers confusing. Specifically, the judges were given the question “Will you rule tosuppress the evidence?” and two options: “Yes, there was probable cause to justify the search” and “No, there was notprobable cause to justify the search.” Nevertheless, most of the judges reported understanding what the question wasasking, and so we include these results here. Also, as discussed, their results did not differ from those of the federalmagistrate judges later tested in Minneapolis or the Ohio judges, for whom we corrected the error.


2. Results

In foresight, 23 percent (26 out of 113) of the judges concluded that there was probablecause for a search and granted a warrant (six judges did not respond to the question). Inhindsight, 28 percent (30 out of 108) of the judges concluded that there was probable causefor a search and ruled the testimony admissible. Although this trend suggests an influenceof the hindsight bias because more judges in hindsight concluded that the facts supportprobable cause, the trend was not statistically significant.23 With this sample size, we hada 98.9 percent chance of detecting a significant or marginally significant difference if thetrue difference between foresight and hindsight in this study is 20 percentage points orgreater.24

We assessed the demographic variables by running a logistic regression of the judges’decision on the perspective, each demographic variable (jurisdiction, years of experience,gender, and political orientation), and an interaction of the two. Each demographicvariable was thus analyzed separately (an approach we used throughout this article). Usingthis approach, we found no significant differences or interactions involving the jurisdictionor the years of experience.

The gender of the judges played some role in judges’ evaluations. Male judgesexhibited no hindsight bias whatsoever, 29 percent finding probable cause in foresight and28 percent in hindsight. Female judges, however, showed a strong trend, with 12 percentfinding probable cause in foresight and 26 percent in hindsight. Logistic regression of thedecision on condition, gender, and an interaction of these variables, however, showed thatthe interaction between perspective and gender was not significant, although the maineffect of gender was marginally significant.25 Overall, women were less likely to concludethat probable cause was present for the search.

Among the judges in Ohio, the political orientation of the judges did not play asignificant role, but some trends emerged.26 Among Republican judges, 29 percent and 31percent found probable cause in foresight and hindsight, respectively. Among their Demo-cratic counterparts, 18 percent and 27 percent found probable cause in foresight andhindsight, respectively. Even though Republicans seemed somewhat more likely to find

23Fisher’s exact test, p = 0.44.

24This analysis assumes that our estimate of the population percentage in foresight is 23 percent, or 26 out of 113judges, which is what we obtained. With 108 judges in hindsight, 36 or more finding that there was probable causewould produce a marginally significant effect, p = 0.10, using Fisher’s exact test. If the population percentage inhindsight is 43 percent (20 percent higher than in foresight), we would have had a 98.9 percent chance of observing36 or more judges determining that probable cause was present.

25z = 1.80, p = 0.07 for the main effect of gender. Throughout this article, we test the results of any logistic regressionas the z score of the coefficient of the logistic regression equation, with accompanying p value. All statistics wereconducted with Stata.

26We did not ask judges in the eastern jurisdiction or this group of U.S. magistrate judges to identify their politicalorientation.


probable cause, logistic regression of the decision on condition, political orientation, andan interaction of these variables showed no significant main effect or interaction.

3. Discussion

The hindsight bias did not appear to influence the judges’ assessments of probable cause.These results duplicate our earlier findings with a larger sample. Knowing that the searchuncovered incriminating evidence did not have a discernible effect on judges’ decisionmaking. Judges apparently ignored the incriminating evidence and in hindsight produceda pattern of decisions that duplicated that of those judges who were unaware of what thesearch would uncover. This result is consistent with what we reported in our earlier paperdescribing the lack of any influence of the hindsight bias on judges’ assessment of probablecause.

Given the robust findings of the hindsight bias in the psychological literature, theseresults are a surprise. We assumed that judges would be motivated to make decisions thatwould lead to the conviction of a defendant that they knew to be guilty. Even the Ohiojudges, who know they face an electorate that is apt to be less concerned about theconstitutional rights of criminal defendants than most lawyers, were unaffected by theknowledge that their ruling would likely impede the conviction of a guilty defendant. Withthese results, we now show that three different groups of judges, from different parts of thecountry, who attain their positions in different ways, avoid the influence of the hindsightbias on probable cause judgments.

So what accounts for our findings? It is possible that the facts we presented areanomalous, or failed to trigger the bias. Knowing the outcome should have made the policeofficer’s sketchy description of methamphetamine seem far more reasonable. It is possible,however, that these materials are anomalous in some way. The judges’ assessments in thisstudy should depend heavily on the officer’s credibility. Perhaps credibility judgments areless susceptible to the influence of the hindsight bias than other kinds of judgments. Inparticular, judges reviewing the search in hindsight might have worried that the policeofficer was inventing a reason to justify conducting a search that was otherwise based onsketchy evidence of criminal activity more so than the judges in foresight.

B. Study 2

To assess whether the results of our first study were simply an anomaly that arose from anunusual fact pattern we constructed a second hypothetical case and presented it to othergroups of judges. Furthermore, we sought to learn more about the unusual results by alsoasking judges to make a probability assessment in addition to their ruling on probablecause. As noted above, probability is only one aspect of the probable cause determination,and so it is possible that judges’ probability assessments are affected by hindsight, eventhough their determination on probable cause is not.

1. Materials

As with Study 1, the hindsight and foresight perspectives presented similar underlying facts.As before, the key difference was that the foresight perspective asks judges if they would


grant a warrant and the hindsight perspective asks judges if they would admit evidenceobtained without a warrant. Once again, the search involved a parked car that presentedsufficient exigent circumstances that the warrant was unnecessary. Half the judges reviewedthe scenario from a foresight perspective and half from a hindsight perspective.

The materials indicate that a police officer (we refer to “officer” throughout thisdescription, although our materials used “agent” for the federal judges and “officer” for thestate judge) was planning to meet a potential informant at 1:47 am on a Saturday morningwhen he received word that another officer had been attacked nearby. The materials statethat the perpetrator had driven off, but had been wounded in the attack. Fifteen minuteslater, the officer observed a car parked in front of a nearby nightclub. The materials statethat “the driver got out of the car, opened the back door, pulled out a long, curved pieceof metal from the seat, and placed it into the trunk of the car.” After closing the trunk, thedriver entered the club. The officer reported that he observed that the driver had abandaged hand. The officer then walked over to the car, and noticed that “the front left tirewas a small, temporary tire of the type used as a spare,” which “made him realize that themetal object was likely a crowbar.” Upon looking into the back seat, the officer “observed acar jack on the floor and three envelopes on the back seat, two of which appeared to bestuffed with cash.” The officer also “observed a stain, possibly from blood, on the steeringwheel.”

The foresight perspective then indicates that the officer requested a telephonicwarrant. The materials ask simply: “Would you issue the warrant?” Judges can then checkone of two statements: “Yes, there is probable cause for the search; I would issue thewarrant” or “No, there is not probable cause for the search; I would not issue the warrant.”

The hindsight perspective goes on to indicate that the officer searched the trunk ofthe car and “found a bloodied crowbar and a large quantity of white powder that appearedto be cocaine.” The materials also state that the driver was arrested and that “subsequentinvestigative work confirmed that the driver’s fingerprints were on the crowbar” and that“DNA tests also matched the blood on the crowbar with that of the officer who had beenattacked.” As with the materials for Study 1, the defense attorney asked that the evidenceobtained from the trunk be suppressed “on the ground that there was no probable cause toconduct the search.” The materials ask: “Would you allow the evidence to be admitted?”Judges can then check one of two boxes, “Yes, there was probable cause for the search; Iwould admit the evidence” or “No, there was not probable cause for the search; I would notallow the evidence to be admitted.”

In this study, we also varied the severity of the crime. For half the judges, the attackwas said to be serious, but the officer who had been attacked would recover. For the otherhalf, the materials indicated that the officer had been killed in the attack. The underlyingfacts that produced the attack are identical; the only difference is that in one case, the attackwas fatal. Hence, the judges review either a battery or a murder of a police officer (or afederal agent). Thus, the study is a 2 ¥ 2 between-subjects design, with perspective (fore-sight or hindsight) crossed with severity of the crime (battery or murder).

To explore the judges’ reasoning process further, we also asked the judges to providea probability estimate for outcome of the search after they made their ruling on the warrantor admissibility. As an effort to ensure that the judges made their ruling without being


influenced by the prospect of having to provide a probability estimate after the fact, weasked the question on probability at the top of the page after the probable cause scenario.Hence, unless the judges flipped ahead, or read the questionnaires out of order, they wouldhave had no notice of the probability question until after they had made their ruling. Inforesight, we asked the judges: “What is the likelihood that the search, if conducted, woulduncover evidence that would incriminate the driver in the attack on the police officer [oragent]?” In hindsight, we asked: “If [the officer or agent] had requested a telephonicwarrant before conducting the search, what would you have said was the likelihood that thesearch would have uncovered evidence that would incriminate the driver in the attack onthe undercover police officer [or agent]?” Below each question was a blank space followedby a percent symbol.

2. Results

As described, we obtained two decisions from each judge: their ruling on probable causeand their probability estimate.

a. Ruling. As in Study 1, the hindsight bias seemed to play little role in judges’ rulings. Asshown in Table 2, across both crimes, 57 percent of judges (68 out of 119) stated that therewas probable cause to support a warrant, as compared to the hindsight condition, in which46 percent (48 out of 105) concluded that the evidence should be admitted becauseprobable cause was present. (Two judges in hindsight did not provide responses.) Theoverall results across both crimes, in fact, suggest a surprising reversal, although the trendwas not significant.27 Although this trend toward a reversal really only appeared in thebattery condition, a logistic regression of the judges’ decision on the type of crime, theperspective (foresight or hindsight), and an interaction of the two conditions produced nosignificant effects. Thus, while there are some surprising trends, the perspective of foresightand hindsight were statistically indistinguishable.

As with Study 1, we had a sample size that should have been sufficient to detect thehindsight bias. With this sample size, we had a 98.5 percent chance of detecting a significant


Table 2: Percent of Judges (and Sample Size) FindingProbable Cause by Condition

Condition

Crime

TotalBattery Murder

Foresight 56 (61) 59 (58) 57 (119)Hindsight 44 (50) 63 (55) 54 (105)Bias (h/s—f/s) -12 4 -3


or marginally significant difference if the true difference between foresight and hindsightin this study is 20 percentage points or greater.28

As with Study 1, we analyzed each of the demographic variables separately, althoughthe logistic regression included interaction terms for the severity of the crime.29 The threedifferent groups of judges showed little variation. Logistic regression revealed no significantmain effects or interactions of the jurisdiction on the overall rate with which judges foundprobable cause or the interaction terms.

Logistic regression of the choice on perspective, experience, and an interaction termrevealed a marginally significant effect of experience.30 More experienced judges weremore likely to conclude that probable cause was present. The interaction term was notsignificant. Previous experience as prosecutors or defense attorneys had no effect on theoverall rate of concluding that probable cause was present or on the interaction betweenthis type of experience and perspective.

Collapsing across both crimes, a similar percentage of male judges in foresight andhindsight found probable cause (56 percent and 56 percent), while female judges showeda trend toward a reversal, with 59 percent and 46 percent finding probable cause inforesight and hindsight, respectively. Logistic regression of the decision on the perspective,gender, and an interaction term revealed no significant main effects or interactions,however.

Political affiliation had some effect on the judges in this study. Among Republicanjudges, 62 percent found probable cause across all conditions, as compared to 50 percentof Democratic judges. As in Study 1, however, this trend was not significant. Politicalorientation also did not interact significantly with perspective.

b. Probability Estimates. The probability estimates also did not vary with condition. AsTable 3 shows, the judges gave only slightly higher estimates of probable cause in hindsight.An ANOVA of the probability estimates on crime and perspective showed no main effectsor interactions.31 The hindsight question gave the judges problems, as 25 judges in hind-sight declined to respond to this question, as compared to only five in foresight.

Although this result suggests that even probability estimates were not influenced byhindsight because the judges ruled first on probable cause and then provided the prob-ability estimates, it is possible that the ruling influenced the judges’ probability estimates.

28This analysis assumes that our estimate of the population percentage in foresight is 57 percent, or 68 out of 119judges, which is what we obtained. With 105 judges in hindsight, 72 or more finding that there was probable causewould produce a marginally significant effect, or p = 0.10, using Fisher’s exact test. If the population percentage inhindsight is 73 percent (20 percent higher than in foresight), we would have had a 95 percent chance of observing72 or more judges who concluded that there was probable cause.

29In no case did we obtain a significant interaction between a demographic variable and the type of crime, nor wereany of the three-way interactions between the demographic variable, type of crime, and perspective significant.

30z = 1.70, p = 0.09.

31Main effect for crime, F (1, 198) = 1.36, p = 0.25; main effect for perspective, F (1, 198) = 0.83, p = 0.36; interaction,F (1, 198) = 0.05, p = 0.82.


That is, judges who had just stated their belief that probable cause supported the searchmight have then been motivated to provide higher estimates for the success of the searchas a result of their determination. In a realistic setting, judges might be more apt to thinkabout the probability first, and then make a ruling, but we believe the ruling to be morecritical, so we asked for that first.

Assessing the probability estimates by ruling revealed some evidence of the influenceof the hindsight bias. Table 4 presents the average probability estimates that the judgesprovided, broken down by their ruling, type of crime, and perspective. This analysis revealsthat the judges who ruled that probable cause was present gave higher probability estimatesthan those who did not. Averaging across perspective and type of crime, the 115 judges whoconcluded that probable cause was present estimated the likely success of the search as 65percent, while the 86 judges who concluded that probable cause was not present gave anaverage estimate of 46 percent. An ANOVA of the probability estimate on perspective,crime, and the judges’ decisions (probable cause or not) revealed that this difference wassignificant.32 Furthermore, the analysis revealed a marginally significant effect for perspec-tive.33 The interaction between crime and perspective also emerged as marginally signifi-cant in this analysis.34 Table 4 suggests that his interaction resulted from the large hindsight

32F (1, 193) = 27.82, p < 0.001.

33The mean estimates in foresight and hindsight were reported in Table 3. This analysis differs in that it controls forthe judges’ ruling to produce the marginally significant effect of perspective. F (1, 193) = 3.04, p = 0.08.

34F (1, 193) = 3.51, p = 0.06.

Table 3: Mean Likelihood for Successful Search (andSample Size) by Condition

Condition

Crime

TotalBattery Murder

Foresight 54 (58) 56 (47) 55 (105)Hindsight 57 (48) 61 (49) 59 (97)Bias (h/s—f/s) 3 5 4

Table 4: Mean Likelihood for a Successful Search (and Sample Size) by Perspective,Decision, and Crime

Perspective

No Probable Cause Probable Cause

Battery Murder Both Crimes Battery Murder Both Crimes

Foresight 42 (25) 38 (18) 41 (43) 62 (33) 65 (21) 65 (63)Hindsight 49 (26) 57 (17) 52 (43) 68 (30) 64 (31) 65 (52)Bias (h/s—f/s) 7 19 11 6 -1 0


bias effect among judges who determined there was no probable cause while assessing themurder case. No other terms were significant.

Although the order in which we presented the two questions to the judges suggeststhat the ruling on probable cause would likely influence the probability estimate, ratherthan the other way around, we can use these data to estimate the probability that judgeswould think sufficient to support a probable cause estimate. By running a logistic regressionof the judges’ choice on their probability estimates (collapsing across crime and perspec-tive), the cutoff that best distinguishes judges who concluded probable cause was presentfrom those who did not was 50 percent and greater.35 This cut-off rule correctly predictedthe decisions of 66 percent of the judges. Specifically, 97 of the 147 judges (or 66 percent)who provided a probability estimate of 50 percent or more also determined that probablecause was present. Of the 54 judges who ultimately provided probability estimates less than50 percent, 36 (or 67 percent) concluded that probable cause was not present. Thus, thesejudges roughly followed a 50 percent cut-off rule for probable cause.

None of the demographic variables affected probability estimates. A regression of theprobability on crime, perspective, an interaction term of these two, and the demographicvariables (jurisdiction, experience, gender, political affiliation, former experience as pros-ecutor or defense attorney, and court) and the interaction of these terms with the condi-tions, revealed no significant main effects or interactions.

3. Discussion

These results replicate the conclusion from Study 1 that the hindsight bias does notinfluence judges’ assessments of probable cause. The results also revealed a complexrelationship between the hindsight bias and assessments of probable cause, however. Judgeswho determined that probable cause existed gave higher estimates for the likelihood of asuccessful search, suggesting that the likely outcome of the search is related to the judges’assessments of probable cause. Judges’ probability estimates were also higher in hindsightthan foresight, once we accounted for the effect of the judges’ rulings. Even so, hindsightdid not affect their judgments.

These results suggest that judges do not base their rulings of probable cause entirelyon an assessment of the likelihood that the search would uncover (or would have uncov-ered) incriminating evidence. The judges might be assessing police conduct somewhatdifferently. Probable cause has generated a mountain of case law—much of which isfamiliar to the judges we studied and most of which represents a judicial effort to managepolice behavior, not guide criminal investigations. Judges might be comparing the facts ofthe case to precedent and using their analogical reasoning skills. In effect, the hindsightbias might be affecting their assessments of probability, but they might be basing theirrulings on something other than their sense of the likely outcome of the search.

It is also notable that manipulating the severity of the crime had no effect on thejudges. If motivation played an important role in judges’ thinking about probable cause in

35We obtained this cutoff and the statistics in this paragraph using the “lstat” command in Stata, run immediately afterthe logistic regression of judges choice on their probability estimates.


hindsight, we would have expected the more serious crime to induce judges to admit thetestimony. The fact that they showed no such effect suggests that judges are not motivatedin hindsight to rule so as to ensure the conviction of the defendant. To be sure, the judgesknew these materials were hypothetical scenarios, and therefore we cannot be certain thatjudges would behave the same way in a real case, but the result at least suggests that judgesare able to focus on obtaining a correct legal ruling without regard to its consequences.

This pattern of results suggests, overall, that judges might not be relying on prob-ability judgments when assessing probable cause. Even though our results suggest thatprobable cause has a reasonably reliable cutoff of 50 percent, the judges might not be usingtheir sense of the likely outcome to guide their decision. This estimate is consistent withprevious surveys of judges.36 Because we did not ask judges to make probability judg-ments without first making a ruling, however, the results are difficult to interpret as directevidence for this thesis.

C. Study 3

To assess whether the hindsight bias affects judgments of probability in probable causeassessments without the influence of having first made a ruling, we constructed a thirdscenario. We asked judges for an assessment of the likely outcome of the search without firstasking for a ruling. Study 2 suggested that the hindsight bias affects probability judgment,but the judges in that study first made a ruling, which likely influenced their probabilityestimates. To obtain a more direct assessment of probability judgments, we limited ourquestioning to the probability estimate alone. We also wanted to add a third scenario toprovide a final chance to see if hindsight affects probable cause judgments.

1. Materials

As noted in Study 1, we asked half the judges in Ohio to provide only a probability estimate.For these judges, instead of asking for a ruling on whether to grant a warrant or whether toadmit evidence, judges were asked either “What is the probability that the search, ifconducted, would uncover illegal drugs?” in foresight or “If [the officer] had requested atelephonic warrant before conducting the search, what would you have said was the prob-ability that the search would have uncovered illegal drugs?” The scenario otherwiseappeared to the judges exactly as we described in Study 1.

We also presented third scenario to different groups of judges. This scenario tracks thepattern of the first two. It presents the materials from the foresight perspective, in which thejudges do not know the outcome of a search, or the hindsight perspective, in which a searchof a parked car uncovers incriminating evidence. As with the previous studies, the hindsightand foresight perspectives presented the same underlying facts. As before, the key differencewas that the foresight perspective asks judges if they would grant a warrant and the hindsightperspective asks judges if they would admit evidence obtained without a warrant. Once again,

36C.M.A. McCauliff, Burdens of Proof: Degrees of Belief, Quanta of Evidence, or Constitutional Guarantees? 35 Vand.L. Rev. 1293, 1327 (1982).


the search involved a parked car that presented sufficient exigent circumstances that thewarrant was unnecessary. Half the judges saw the version in hindsight, half in foresight. Also,half made only probability assessments and the other half made rulings.

The materials for the third scenario describe a police officer who was “on foot patrolin a high-crime urban area” when he noticed a car parked in front of a fire hydrant “in frontof a bar known to be frequented by drug dealers.” The officer sees the driver “fiddle withsomething in his hand as he sat in the driver’s seat with the door open.” The materials statethat when the officer approaches, “a woman, who had been hanging around the entranceto the bar, looked over at him and yelled out, ‘look out Dan, it’s a cop!’ ” The driver and thewoman run off and get away.

The materials then state that the officer looked into the car and saw “an envelope onthe floor by the driver’s side that had some money in it” along with “a plastic shopping bagfrom a nearby Walgreens that appeared to have three packages of pseudoephedrine in it”along with “some Visine eye drops and a package of baggies on the back seat.” The materialsnote that “pseudoephedrine is often used to manufacture methamphetamine in home labs.”

The foresight condition asking for a ruling then asked “Would you issue thewarrant?” followed by two statements: “Yes, there was probable cause for the search; I wouldissue the warrant” or “No, there was not probable cause for the search; I would not issue thewarrant.” The foresight condition requesting a probability asked if the police officer “hadrequested a telephonic warrant before conducting the search, what would you have said wasthe likelihood that the search would have uncovered illegal drugs?” Unfortunately, amongthe materials given to judges in Florida, the foresight ruling question was also embedded inthe text of the foresight materials in which we asked for judgments of probability. We didnot provide space for a ruling, but many of the judges supplied a ruling in the margins. Thematerials were redrafted for the Arizona judges to eliminate this implicit request.

The hindsight conditions indicate that the officer had searched the car and “founda coffee container filled with marijuana and 20 baggies containing a substance laterdetermined to be methamphetamine” under the driver’s seat. The driver was later foundand arrested. The materials indicated that the defense attorney had moved “to suppress theevidence obtained from the car on the ground that there was no probable cause to conductthe search.” In the condition requesting a ruling, the materials ask “Would you allow theevidence to be admitted?” followed by two statements for the judges to check: “Yes, therewas probable cause for the search; I would allow the evidence to be admitted” and “No,there was not probable cause for the search; I would not allow the evidence to be admitted.”The condition requesting a probability judgment stated that if the officer “had requested atelephonic warrant before conducting the search, what would you have said was the likeli-hood that the search would have uncovered illegal drugs?”

Study 3 thus focuses on comparing the rulings to the probability estimates amongjudges who either judged the probability alone or made only a ruling.

2. Results

We present the results in three parts. First, we describe the results of the judges who issuedrulings in Scenario 3 in our third of three efforts to see if the hindsight bias influences


rulings on probable cause. Second, we describe the analysis of the judges who made theprobabilities estimates alone after reviewing the third scenario. Third, we present theresults from those judges who assessed our first scenario, but were asked to provide onlyprobability estimates.

a. Rulings in Scenario 3. The results showed some trend toward a hindsight bias among theFlorida judges in the third scenario. In foresight, 45.1 percent (33 out of 72) concluded thatprobable cause was present; in hindsight, 57.7 percent (45 out of 77) concluded thatprobable cause was present. This trend, however, was not significant.37 Because we dividedthe judges into two groups, the number of judges who made rulings alone was smaller (149judges). Consequently, the statistical power of this study is weaker than the previous two.With the number of judges who made only a ruling, we had a 83.2 percent chance ofdetecting a significant or marginally significant difference if the true difference betweenforesight and hindsight in this study is 20 percentage points or greater.38

Individual logistic regressions of the judges’ decisions on the perspective with thedemographic variables (years of experience, gender, political orientation, former experi-ence as a prosecutor, and former experience as a defense attorney) and interactionsbetween these parameters and the perspective revealed significant effects only for gender.

The main effect for gender in the regression was significant39 and a marginal effectfor the interaction of gender with perspective also emerged.40 Across both perspectives, 55percent (60 out of 110) of the male judges found probable cause was present as comparedto 37 percent of the female judges (11 out of 30). Among male judges, 48 percent (24 outof 50) found probable cause in foresight, as compared to 53 percent in hindsight. Amongfemale judges, 42 percent (eight out of 19) found probable cause in foresight, as comparedto 81 percent (nine out of 11) in hindsight.

b. Probability Estimates in Scenario 3. The probability estimates showed evidence of theinfluence of the hindsight bias. The 91 judges reviewing the foresight materials (eight didnot respond—all in Florida) gave an average probability estimate of 61.2 percent, ascompared to 73.1 percent among the 82 judges who reviewed the materials in hindsight(two did not respond—both in Florida). This difference was significant.41


38This analysis assumes that our estimate of the population percentage in foresight is 45 percent, or 33 out of 72judges, which is what we obtained. With 77 judges in hindsight, 46 or more finding that there was probable causewould produce a marginally significant effect, or p = 0.10, using Fisher’s exact test. This is only one more than weactually observed. If the population percentage in hindsight is 65 percent (20 percent higher than in foresight), wewould have had a 83.2 percent chance of observing 46 or more judges who concluded that there was probable cause.

39z = 2.01, p = 0.04.

40z = 1.88, p = 0.06.

41t (172) = 3.05, p < 0.005.


Among the Florida judges working in foresight, 52 (out of 77) also provided a ruling,even though we did not explicitly ask for one. Those judges who issued a ruling provideda mean probability estimate of 63.6, as compared to 60.0 among those judges who did notissue a ruling. This difference was not statistically significant.42 Among those judges who didrule, those who concluded there was probable cause gave an estimate of 63.0 as comparedto 64.0 among those who did not. This difference was also not statistically significant.43

The Arizona judges in the foresight perspective provided a slightly lower mean estimate(57 percent) than the Florida judges (63 percent), but this difference was not significant.44

An ANOVA of the probability estimate on perspective and location revealed an effectfor perspective,45 but not for location,46 or the interaction term.47 Taken together, theseresults suggest that although the judges in Florida saw slightly different materials, and manyresponded by providing a ruling, this difference did not affect their assessments of thelikely outcome of the search.

To test for the effect of the demographic variables of experience, gender, party,former experience as a prosecutor (Florida only), and former experience as a defenseattorney (Florida only), we conducted separate ANOVAs of the judges’ probability esti-mates on their perspective, each demographic variable, and an interaction term.48 None ofthese analyses produced significant effects for the demographic variables or interactions.The interaction of gender and perspective came closest.49 Male judges exhibited a some-what greater influence of the hindsight bias, providing a mean probability estimate of 60percent in foresight, as compared to 77 percent in hindsight, while female judges produceda mean estimate of 63 percent in foresight and 66 percent in hindsight.

c. Probability Estimates and Scenario 1. The Ohio judges who were only asked about prob-ability did not show any effect of hindsight on their probability estimates. The 49 judgesassessing in foresight gave an average probability estimate of 47.5 percent (seven judges didnot reply). The 53 (two did not reply) judges assessing in hindsight gave an averageprobability estimate of 39.6 percent. This trend runs in the direction opposite to the

42t (68) = 0.58, p = 0.56.

43t (45) = 0.12, p = 0.91.

44t (89) = 0.91, p = 0.36.

45F (1,169) = 10.83, p = 0.001.

46F (1,169) = 0.02 p = 0.90.

47F (1,169) = 1.77, p = 0.19.

48Experience was treated as continuous, using ANCOVA.

49F (1, 164) = 2.58, p = 0.11.


hindsight bias, but was not statistically significant.50 The statistical power of this study wasrelatively weak, as compared to our other studies. We had only a 64 percent chance ofdetecting a marginally significant effect.51 The effect of perspective in Scenario 1 wassignificantly different than the effect of perspective in Scenario 3.52

To test for effects of the demographic variables of years of experience, gender, andpolitical orientation, we conducted separate analyses for the effect of the perspective,each demographic variable, and the interaction on the probability estimate. Years ofexperience did not significantly affect the probability estimates, but the interaction ofyears of experience with perspective was marginally significant.53 The newer judgesappeared to show a trend toward exhibiting a hindsight bias, while more experiencedjudges showed the reverse, thereby producing the trend toward an interaction. Genderdid not produce a significant main effect or interaction. Political orientation showeda significant main effect,54 with Republicans giving an average estimate of 49 ascompared to 36 among Democrats. The interaction of party with perspective was notsignificant.55

3. Discussion

These results show that the hindsight bias influenced judges’ assessments of probability, butdid not influence their rulings. As happened with the first two scenarios, we detected noeffect of the hindsight bias on the judges’ rulings, but hindsight seemed to affect theirprobability judgments. In the third scenario, judges reviewing the hindsight perspectivegave higher probability estimates for likely success of the search than their colleaguesworking in foresight. Even though the hindsight bias seemed to affect their assessment ofthe scenario, it did not affect their rulings.

To be sure, our effort to replicate this finding using our first scenario was notsuccessful. Judges who reviewed the first scenario showed no differences in probabilityestimates in foresight as compared to hindsight. Not every fact pattern in every hindsightbias study produces an effect, however.56 Our first scenario perhaps depends too heavily on

50t (100) = 1.33, p = 0.18.

51We assumed that the hindsight condition would shift the probability estimates upwards by 11.9 percent—which isthe shift we observed in Scenario 3. To calculate the power, we used the standard deviation we obtained in the Ohiojudges, which was 30.1 (assuming it was uniform across both conditions) and a p value of 0.10.

52An ANOVA of the probability estimates in both scenarios with perspective, scenario, and an interaction termproduced a significant interaction. F (1, 271) = 8.43, p < 0.005.

53F (1, 1,98) = 3.78, p = 0.06.

54F (1,93) = 4.28, p = 0.04.

55F (1, 93) = 0.18, p = 0.67.

56See Fischhoff, supra note 4 (reporting a scenario that did not produce a hindsight bias).


judges’ assessments of the credibility of the police officers, rather than on the tendency ofthe outcome to induce people to reexamine the facts in ways that make the outcome seeminevitable. The third scenario, however, might contain the kind of antecedent facts that canproduce a hindsight bias. For example, it might be difficult for the judges to see the threeboxes of psuedoephedrine as an innocent coincidence after they learn that the search ofthe car produced methamphetamine. In short, maybe the first scenario simply does notproduce any hindsight bias.

These results affirm a paradox, of sorts, in our study. Probability and probablecause determinations seem related. Likelihood that a search will uncover incriminat-ing evidence is a primary factor in making a probable cause assessment.57 We also foundthat judges in our second study who determined that the facts support a finding ofprobable cause gave higher probability estimates than those who did not. And yet, eventhough hindsight influenced probable cause estimates, it does not affect probable causedeterminations. The result resembles the findings of a line of research by Gary Wells,who found that adults acting as jurors were unwilling to find a defendant in a tortcase liable when the only evidence was purely statistical.58 This research shows that evenif people say they are convinced by a probability estimate, they are not always willingto act on that belief. Judges might be undergoing a similar thought process in ourstudy. The outcome might make the prospect of finding incriminating evidence seemmore likely, but they rely on factors other than the probability estimate to make theirdetermination.

D. Summary of Results

Across three scenarios, we observed no significant hindsight bias for judges’ rulings. Sce-nario 3 was the closest, but even with a sizeable sample size, it failed to produce more thana trend toward a bias. Combing across all three studies, we observed roughly the samepercentage of judges in foresight finding probable cause as in hindsight: 41.6 percent (126out of 303) versus 45.4 percent (132 out of 291). This difference was not significant.59 Theresults thus support the conclusion that the hindsight bias does not affect judges’ assess-ments of probable cause.

Nevertheless, two of our three studies show that judges exhibited differences whenasked about probability, rather than ruling (see Table 5). Study 2 shows evidence ofhindsight bias once we accounted for the effect of the judges’ ruling and Study 3 shows

57See Taslitz, supra note 12 (“judges and scholars have long found rough mathematical approximations of the[probable cause] standard useful”).

58Gary L. Wells, Naked Statistical Evidence of Liability: Is Subjective Probability Enough? 62 J. Personality & Soc.Psychol. 739 (1992).

59Fisher’s exact test, p = 0.36. We also regressed (using logistic regression) the ruling on perspective and two dummycoded variables to identify the studies to control for the effect of the different fact patterns, and similarly observed nosignificant effect for the perspective. z = 0.99, p = 0.32.


a sizeable hindsight bias on probability estimates. Scenario 1 shows no such effects,however.60

The demographic background of the judges showed some trends in the data, asTable 6 describes.

Judicial experience had little effect on the judges. More experienced judges weremore likely to find probable cause in the second scenario. In the first scenario, amongjudges who provide probability estimates, we found that less experienced judges exhibitedthe hindsight bias, even though we also observed a reversal among older judges. We foundno similar effects among the judges who made rulings. A logistic regression combiningall three scenarios revealed no main effect for experience or an interaction of experiencewith perspective.61 Overall, these data provide little evidence that new judges are moreprone to the hindsight bias. That said, these studies include a highly experienced group ofjudges, with few inexperienced judges. It is possible that a few years of experience on thebench are all judges need to begin ignoring the influence of hindsight. With few brand-new judges in the sample, we might lack sufficient statistical power to detect any realdifferences between experienced and inexperienced judges.

60We did not conduct a combined analysis for probability estimates because the methods varied across conditions.Notably, all the judges who reviewed the second scenario first made rulings. The hindsight bias we observed inStudy 2 emerged only after we controlled for the judges’ ruling.

61Main effect, z = 1.39, p = 0.16; interaction z = 0.07, p = 0.95. As with the individual studies, we conducted threeseparate logistic regressions of the judges’ ruling on the perspective, each demographic variable, and an interactionterm to identify the effect of the demographic variable, the results of which we report below. In each model, we alsoincluded two dummy codes to control for the differences between the scenarios (although we did not includeinteraction terms with the dummy codes for the scenarios). We did not combine the data for the probability estimates.

Table 5: Summary of Results

Ruling (% Indicating that Probable Cause Was Present)

Scenario (and n)

Perspective

Bias (Hindsight – Foresight)Foresight Hindsight

#1 (221) 23 28 +5#2 (224) 57 54 -3#3 (149) 45 58 +13

Probability Estimates

Scenario (and n)

Perspective

Bias (Hindsight – Foresight)Foresight Hindsight

#1 (102) 48 40 -8#2 (202)* 55 59 +4#3 (173) 61 73 +12

*These judges ruled first on probable cause.


Male and female judges differed slightly in their reactions to the scenarios. InScenarios 1 and 3, female judges were less likely to conclude that probable cause waspresent than were male judges. In Scenario 3, the female judges, unlike their male col-leagues, exhibited some measure of hindsight bias in making their rulings. We alsoobserved the same trend in the first scenario, although it was not significant. We observedno gender differences among judges making probability estimates. A logistic regressioncombining all three scenarios revealed that female judges were less likely than malejudges to find probable cause (34 percent vs. 46 percent, respectively), although theinteraction of perspective and gender was not significant.62 Overall, these results suggestthat female judges in our sample might be less reluctant to grant a warrant or admitevidence. Despite some trends in the data, we cannot draw any conclusions as towhether female judges were more susceptible to the influence of hindsight than weremale judges.

Republican and Democratic judges differed slightly in their reaction to the materials.We found no evidence that they varied in their vulnerability to the hindsight bias, eitherwhen making rulings or probability judgments. Republicans, however, were more likely tofind probable cause in the second scenario, and we observed a similar trend in the firstscenario that was not statistically significant. Republican judges also gave higher probabilityestimates in the first scenario. A logistic regression combining all three scenarios revealedthat Republican judges were more likely than Democratic judges to find probable cause

62Main effect, z = 2.12, p = 0.03; interaction, z = 0.93, p = 0.35.

Table 6: Summary of the Effect of Experience, Gender, and Political Orientation

Variable Scenario Ruling Probability Estimates

Experience #1 No effects New judges show hindsightbias, older show areversal*

#2 Older judges more likely tofind PC*

No effects

#3 No effects No effectsGender #1 Female judges less likely to

find PC*No effects

#2 No effects No effects#3 Female judges less likely to

find PC; female judgesexhibit more hindsightbias*

No effects

Political orientation #1 No effects Republicans gave higherestimates

#2 Republicans more likely tofind PC*

No effects

#3 No effects No effects

*Marginal effect (0.05 < p < 0.10).


(50 percent vs. 43 percent, respectively), although the interaction of perspective andgender was not significant.63

IV. Discussion

Judges resisted the influence of the hindsight bias on probable cause assessments in ourstudy. This result represents a marked departure from most of the research on the hind-sight bias, which reveals robust differences between judgment in foresight and judgment inhindsight.

The results of this study replicate and extend our initial finding on hindsight andprobable cause in three respects. First, we used the same scenario as our earlier work, butcollected more data, thereby ensuring that we had sufficient statistical power to detect aneffect. Second, we replicated the result with two other scenarios, also with sizeable statisticalpower. Third, we also asked judges to make probability judgments so as to explore how itis that judges are able ignore the outcome of the search. We find that judges’ assessmentsof the likely outcome of the search are affected by outcome knowledge, but their judgmentsof probable cause are not.

The combination of results would seem to present a paradox of sorts, but we believethem to be consistent with out “intuition-override” model of judging, which we havedeveloped elsewhere.64 This model argues that judges quickly develop intuitions about howto decide cases, but that they commonly check this intuition with more deliberative rea-soning processes. In some instances, we have observed that the intuitive response never-theless holds sway over judges. For example, irrelevant or misleading numeric anchorsseem to have a powerful influence over how judges think about damage awards in civilcases.65 We have proposed, however, that judges also manage to ignore or suppress mis-leading impressions.

Given the data in this study, the hindsight bias presents a primary case in point forour intuitive-override model of judging. The probability estimates suggest that knowing theoutcome of the search causes judges to think differently about the case. Just as researchershave observed in dozens of other studies of the hindsight bias, we found that judges cannotcompletely suppress knowledge about how events unfolded. But when judges make theirrulings, they do not rely on this intuition. Even though the probable success of a search isa factor in the probable cause determination, their sense of the probable outcome of thesearch does not appear to affect their rulings. The judges in our study judged the facts andapplied the law the same way in hindsight as in foresight—or at least they reached the sameconclusions.

63Main effect, z = 1.83, p = 0.07; interaction, z = 0.35, p = 0.73.

64Chris Guthrie, Jeffrey J Rachlinski & Andrew J. Wistrich, Blinking on the Bench: How Judges Decide Cases, 93Cornell L. Rev. 1 (2007).

65Id. at 19–22.


Exactly how judges manage to suppress their intuition remains uncertain. We canspeculate as to how they might have done this, however. We suspect that judges know thata mountain of case law exists on probable cause on which they can rely. Although we did notprovide them with the kinds of legal briefs and arguments that they would have in a realcase (at least in hindsight), this is a familiar area of law for most of the judges in our study.Judges would be able to remember specific rulings on probable cause cases, and could haveengaged analogical reasoning processes to try to recall cases similar to those that weprovided. If so, and if hindsight does not affect their ability to recall similar cases, then thisprocess might have insulated judges from the influence of the hindsight bias. In effect, it isnot so much their sense of the likely outcome of the search that influenced judges, as theirability to recall a case that would support ruling one way or the other in the case beforethem.

This account distinguishes our two other studies showing hindsight bias in judges.When asking judges to repredict the likely outcome of an appeal of a sanction, we were notasking them for a legal judgment, but for their own sense of the case. Hence there wouldbe no need for them to engage in an effort to recall similar cases. Indeed, the questionasked for their intuitive responses. In our other study, in which we showed that judges areaffected by hindsight bias when assessing a discrimination claim, they could not have reliedon similar cases. The law in that study is clear; if the security guard was motivated by thetarget’s race, then his actions were illegal. That case thus requires a purely factual assess-ment, which would not allow judges to consult their memories for similar cases. By contrast,in making a probable cause determination, a judge who knows that a similar set of factsproduced an appellate opinion on whether probable cause existed or not would be betteroff relying on that case law.

Our research cannot definitely show that judges in real cases actually make the samedeterminations in foresight as in hindsight. We suspect that in actual cases, judges wouldfind it much more difficult to suppress damning evidence against a defendant who theystrongly believe to be guilty than they would in our hypothetical scenarios. Our research isbetter suited to identifying cognitive styles than to measuring judicial motives. Nevertheless,this study itself shows that even hypothetical cases might detect some degree of judicialmotivation. In our study, for example, Republican judges were more likely to side with lawenforcement than were Democratic judges. This result could be attributable to differencein cognitive styles. Republicans might be more prone to believe law enforcement thanDemocrats. But this difference could also reflect a greater desire among Republican judgesto give law enforcement more latitude than Democratic judges—which is more akin to amotivational influence on their judgment than a cognitive one. In effect, we have somelimited evidence that judges are willing to express their political motives in our hypotheticalcases, even though they know nothing is truly at stake in their responses.

The results of our study could also be explained by theories other than a judicialability to neglect the outcome of searches. Judges operating in foresight might be consid-ering completely different factors than judges in hindsight. In all three scenarios, we askedjudges in foresight to rely on an electronic warrant. Even though telephonic warrants areacceptable, a couple of judges noted that they disfavored such procedures. This issue is notpresent in hindsight. In hindsight, judges’ rulings might have reflected their views of the


automobile exception to the warrant requirement, which all three of our scenarios invoked.This exception is long-established, well-founded in the idea of exigent circumstances thatautomobiles usually present (and always did in our scenarios), and in all three scenarios, weidentified the basis for the defense attorney’s objection as the lack of probable cause, ratherthan the inappropriate use of the automobile exception. Nevertheless, judges might haveexpressed their concerns about the lack of a warrant by suppressing the evidence. In effect,although we drafted materials meant to direct the judges to the probable cause determi-nation, they might have worried about other factors in both foresight and hindsight thatcould have affected their rulings. Although judges might have factored these concerns intotheir judgment, we doubt that this explains the data, however. If judges in foresight basedtheir rulings on their reaction to telephonic warrants and judges in hindsight based theirrulings on their reaction to the automobile exception, then it would be hard to see why therate at which they found probable cause would rise and fall with different facts that thethree scenarios presented and yet remain nearly identical in hindsight and foresight.The judges responded to the variations in the facts across the scenarios, and yet drewsimilar conclusions about probable cause.

We believe that our conclusion is good news. Judges seem able to overcome apervasive cognitive bias in judgment on an important aspect of the criminal justice system.Although the requirement that a warrant be obtained is still almost certainly the best way toavoid having the outcome of the search cloud judgment, it is not reasonable to demand thata warrant be present when the police confront exigent circumstances. Even when theyoperate with knowledge of the outcome of the search, judges are able to fulfill theirobligations under the Fourth Amendment to monitor and control police conduct. Judgesare only human beings, but in this study, they show themselves to be highly professionalhuman beings.


Probable Cause, Probability, and Hindsight...Probable Cause, Probability, and Hindsight jels_ 1230...

Documents

Transcript of Probable Cause, Probability, and Hindsight...Probable Cause, Probability, and Hindsight jels_ 1230...