Behavior Research Methods, Instruments, & Computers

1993, 25 (2), 250-256

A picture is worth a thousand p values:On the irrelevance of hypothesis testing

in the microcomputer ageGEOFFREY R. LOFTUS

University of Washington, Seattle, Washington

Hypothesis testing, while by far the most common statistical technique for generating conclu-sions from data, is nonetheless not very informative. It emphasizes a banal and confusing ques-tion (“Is it true that some set of population means are not all identical to one another?”) whoseanswer is, in a mathematical sense, almost inevitably known (“No”). Hypothesis testing, as itis customarily implemented, ignores two issues that are generally much more interesting, im-portant, and relevant: What is the pattern of population means over conditions, and what arethe magnitudes of various variability measures (e.g., standard errors of the mean, estimates ofpopulation standard deviations)?The simple expedient ofplatting relevant sample statistics withassociated variability bars is a substantially better way of conveying the results of an experi-ment. In today’s microcomputer environment, there are many relatively cheap and easily avail-able applications that allow one to do this. I make some brief, informal comments about someof these applications.

I want to make two main points in this article. First,hypothesis testing is overrated, overused, and practicallyuseless as a means of illuminating what the data in someexperiment are trying to tell us. Second, graphical pre-sentation methods are a much betterway to provide suchillumination, particularly given the ease with whichpresent computer technology allows such methods to beimplemented.

THE ENDURING TYRANNYOF HYPOTHESIS TESTING

In 1962, upon ending his editorship of the august Jour-nal ofExperimental Psychology, Arthur Melton wrote aneditorial in which he summarized criteria used by thejour-nal for accepting manuscripts. These criteria revolvedheavily around hypothesis testing. In particular, Meltonnoted that (1) articles in which the null hypothesis wasnot rejectedwere a]most never published and (2) rejectionat the .05 significance level was rarely adequate for ac-ceptance; rather, rejection at the .01 level was typicallyrequired.Melton’s editorial blessed a practice that had already

become widespread within the social sciences: the use ofhypothesis testing as a necessary (and almost sufficient)techniquefor data analysis. Thispractice has not changedmuch in the intervening 30 years; today, hypothesis test-ing is the primary means of inferring conclusions from

data in over90% of the articles in the major psychologyjournals.Hypothesis testing provides the illusion of scientific ob-

jectivity by sanctifying an arbitrary probability (p = .05)of incorrectly rejecting some null hypothesis that almostinevitably is known apriori to be false (see Bakan, 1966;Gigerenzer et al., 1989; Loftus, 1991; Nunnally, 1960,for elucidations of this argument).’ Hypothesis testing,as normally implemented, provides virtually no informa-tion about two critical aspects of an experiment: the degreeof experimental power and the relationship of a set ofpopulation parameters (typically population means)2 toone another. I will argue that the simple expedient of pre-senting a figure that depicts sample means, along with rel-evant error bars (aprocedure to which I will refer forex-positional convenience as the plot-plus-error-bar, or PPE,procedure) provides essentially all the information pro-vided by a hypothesis-testing procedure, plus additionalinformation. Furthermore, the information that is sharedby the hypothesis testing and PPE procedures is gener-ally uninteresting and unimportant, whereas the additionalinformation provided by the PPE procedure is generallyinteresting and important.There are many reasons why hypothesis testing origi-

nally became the default data analysis technique in the so-cial sciences (see discussions by Cohen, 1990; Gigerenzeret al., 1989; Loftus, 1991). Oneofthemisthatithasgener-ally been easy todo it. You plug raw data into a computerprogram and out comes a z or a t or an F value that tellsyou everything you need for writing your article. (AsCohen, 1990, astutely points out, some peddlers of statis-tical software packages have gone so far as to hawk theirwares by claiming, correctly, that you do not have to evenunderstand statistics in order to use the application.)

The writing of this manuscript was supported by an NIMH grant toG. Loftus. Requests for reprints may be addressed to the author at theDepartment of Psychology, University ofWashington, Department ofPsychology, Seattle, WA 98195 (e-mail: gloftus@u.washington.edu).

250 Copyright 1993 Psychonomic Society, Inc.

HYPOTHESIS TESTING 251

In contrast to the relative ease of hypothesis testing,making plots with standard errors had, until recent years,been rather tedious. First you had to buy graph paper,pencils, a pencil sharpener, and many erasers. Then youhad to spend considerable time just to make a rough pre-liminary plot. Then you had to take your rough plot tosome expensivegraphic artist, typically far across campusin the Medical School or somewhere, and wait a weekor so for the final result. If you changed your mind aboutwhat you wanted to plot, you had to cycle through thewhole procedure all over again. There was little in theway of immediate feedback, and the process was not fun.In the past decade, however, things have changed dra-

matically. With the explosion of computer graphics, cut-and-paste procedures, and cheap graphing applications,it is very easy to present data as a plot, or a collectionof plots, rather than as a compendium of F ratios. Thatis what we should be doing.

Two Romans a ClefIn this section, I will tell two stories that are meant to

illustrate the relationship between the hypothesis testing andPPE procedures. In these stories, the names, experiments,and data have been changed in order to deter hurt feel-ings, embarrassment, and general professional acrimony.

1. The Time Course of VisualInformation AcquisitionA couple of years ago, a cognitive psychologist named

Julia Loeb submitted a manuscript to the Journal of Im-portant Results (JIR). Loeb was interested in perceptualencoding of, and memory for, simple dot matrices. Hertask was straightforward: on each of many trials, a sub-ject saw a stimulus consisting of four dots embedded infour randomly selected cells of an n X n matrix. Follow-ing the matrix’s offset, the subject was required toreproduce the dots’ positions.Loeb’s design incorporated three independent variables

(all within subjects). First, the stimulus was shown at oneof eight exposure durations. There were also two levelsof stimulus uncertainty, and two levels of verbal encod-ing/no verbal encoding (forpurposes of today’s discussion,a detaileddescription ofthese variables is not necessary).Loeb ran 10 subjects in her experiment.Loeb had developed a theory that implied the follow-

ing. First, task performance (proportion of correctly lo-cated dots) should be exponentially related to exposureduration. Thus, if d is exposure duration, and p is per-formance, the equation

p = ~ (1)

should describe the relation between them (here c is a con-stant). The second implication of Loeb’s theory was thatboth more uncertain stimuli and lack of verbal encodingshould lead to poorer performance.To examine her results, Loeb planned to plot probabil-

ity correct, p, as a function of exposure duration, d, and

determine the degree to which the resulting curves couldbe fit by Equation I. As she was starting to do so, how-ever, she realized that by expressing performance not interms of raw proportion correct, p, but instead in termsof the transformed score,

P =

the resulting curves relating performance to durationshould be linear rather than exponential. That is, with theuse of P rather than p, Equation 1 becomes,

P = dlc. (2)

Loeb, a very visually-oriented person, decided that linearfunctions of the sort described by Equation 2 are easierto assess, comprehend, and compare than are exponen-tial functions of the sort described by Equation 1. Becauseshe could see no drawbacks associated with expressingperformance in terms ofP rather than p, that is what shedid.Her data, which are reproduced in Figure 1, confirmed

her predictions quite nicely. Each panel shows perfor-mance as a function of exposure duration. The two curvesin each panel represent the two stimulus-uncertaintylevels. The two panels show data for the two encoding-strategy levels. For each curve, the data points representthe condition means along with the relevant standard er-ror bar, and the solid line represents the best-fitting linearfunction. Loeb described a number of other interestingand important aspects of the Figure 1 data having to dowith the exact relationships among the slopes of the fourfunctions, but I will skip a discussion of these aspects,for they are not relevant to today’s story.Hypothesis testing as an alternative to Figure 1.

The fIR reviewers were quite positive about Loeb’s manu-script, and the editor accepted it with minor revisions.However, at the very last stage of the editor’s interactionwith Loeb—aspart of the normally benign correspondencein which is enclosed the green to-be-signed document

8.0

0

4.0C

2.0

O 50 100 150 200 250Exposure Duration (ms)

No Verbal Encoding-o~Low—-o-~ Un~

0 50 100 150 200 250Exposure Duration (ms)

Figure 1. Loeb’s data: Mean memory performance plotted as func-tions of exposure duration. The error bar around each data pointis the standard error of the mean. Solid tines represent best-fittinglinear functions.

Verbal Encoding-o~Low—-o~Hig$,Unoo1a~ty

252 LOFTUS

transferring copyright to the journal—a snag occurred.In his letter, the editor added, “In my final reading ofyour manuscript, I noticed that you didn’tdo any hypoth-esis testing on the Figure 1 data. Please include such tests,along with the relevant F values in the final version ofyour manuscript.” Because this was Loeb’s tenureyear,she didn’t want to make any fuss that might endangerhermanuscript’s publication, so she dutifully added the fol-lowing paragraph to her results section.An 8x2x2 repeated measures ANOVA revealed a maineffect of exposure duration, F(7,63) = 354.49, p < .05,a main effect of stimulusuncertainty, F(1 ,9) = 16.02, p <.05, and a main effect of encoding strategy, F(1 ,9) =121.33, p < .05. The interactions of exposure durationwith uncertainty and encoding strategy were both signifi-cant, Fs(7,63) = 82.23 and 77.90, respectively, bothps <.05. The uncertainty x strategy interactionwas significant,F(l,9) = 24.98,p < .05. The duration X uncertainty xstrategy interaction was significant, F(7,63) = 13.23,p < .05.

The fIR editor was so pleased with this paragraph thathe suggested Figure 1, which he said was now redundant,be removed (thereby following a long tradition of jour-nal editors who, pressured by cost-of-paper-consciouspublishers, are always suggesting that figures be re-moved). This time, however, Loeb stuck to her guns and,in due course, both Figure 1 and the paragraph repro-duced above were published.What’sWrongwith this Story? Let’s step backamo-

ment and look at the big picture. What is important toknow about Loeb’s data? Simply by looking at Figure 1,we can infer quite a bit. First, because the predicted linearfunctions fall within the error bars, we conclude thatlinearity describes the individual curves quite adequately.Second, because the confidence intervals themselves arequite small, we conclude that the data enjoy substantialstatistical power: That is, any deviation of the relevantpopulation means from observed sample means (and thusany departure from linearity on the part of the actual popu-lationcurves) must be small. Third, by comparing the twocurves within each panel, we can conclude that higherun-certainty leads to poorer performance. Finally, by com-paring the curves across panels, we can conclude thatpreventing verbal encoding leads to poorer performance.The last two conclusions are unambiguous, given the largecondition differences relative to the small confidence in-tervals.Figure 1 also allows some utterly banal conclusions.

For instance, we can easily conclude that, within a givencurve, the eight population means corresponding to theeight exposure-durationconditions are not identical to oneanother; if they were, then, given the size of the confi-dence intervals, the sample means could not plausiblydiffer from one another by asmuch as they do. We couldmake analogous conclusions about the other variables. Icharacterize such conclusions as banal because we knowa priori that they must be true. No set of real-valued con-dition population means can be identical to an infinitenumber of decimal places. They must differ. So why is

Lowry StandardTreatment

6(B)

5

43

I

High Power:small standard

errors

2

1Lowry Standard

Treatment

Figure 2. Possible realities corresponding to “no significant dif-ference” with respect to Lowry’s data. Top panel (A) depicts lowexperimental power (largestandard error bars); bottom panel (B)depicts high experimental power (small error bars).

it interesting to conclude that they do? It is not. What isinteresting is not that the population means differ fromone another, but rather what the pattern of populationmeans is. Here, for instance, it is important to be ableto conclude that the relation between population meansand exposure durations is linear.The hypothesis-testing procedure that Loeb described

in the terse, dense, precise, and scientific-sounding para-graph reproduced abovehas nothing do with the interest-ing conclusions. It tells us nothing about what the patternof population means looks like or how confident we canbe about the inferred pattern (i.e., how much statisticalpowerthere is). Rather, it merely confirms the banal con-clusions, telling us again that it is not true that varioussets of population means are identical to one another. Inshort, the information provided by the PPE procedure,embodied in Figure 1, subsumes the standard hypothesis-testing procedure embodied in the quoted paragraph. Ifyou have the paragraph, there is still a need for the fig-ure. But if you have the figure, there is no need for theparagraph. Hypothesis testing is superfluous.

2. Marital Therapy TechniquesDie-hard hypothesis-testing aficionados might argue that

Loeb’s data, just described, are not typical psychologi-cal data. Loeb’s experiment involved complex factorial

7

6

5

4

3

2

1

(A)

Low Power:large standard

errors

C)C0

(I,

C0

C)C

U)U)

C0a,

HYPOTHESIS TESTING 253

designs, a specific hypothesis about the form of obtainedfunctions (linear), a fancy transformation of the depen-dent variable; this is not the kind of bread-and-butterex-perimentation that is so common in our field. What abouta simpler experimental design wherein there are just twogroups, and all you want to know is whether the groupsdiffer fromone another? Surely you do not need a graph.A simple t test will do. Or will it?Two treatments. Let us consider another example

(again fictionalized). A clinical psychologist, JonathanLowry, developed a new marital-therapy treatment (calledthe Lowry Treatment). He did an experiment to test theeffectiveness of the Lowry Treatment, relative to a muchmore time-consuming and expensive treatment then invogue, universally referred to as the StandardTreatment.Forty married couples were randomly assigned one of twogroups. Couples in the first group underwent the Stan-dard Treatment, whereas couples in the second group un-derwent the Lowry Treatment. The outcome measure wasthe rating of marital bliss (on a 1—7 scale) a year afterthe treatment. Lowry’s hope was that the Lowry Treat-ment would be just as good as the Standard Treatment,in which case the Lowry Treatment, being simpler andcheaper, would be preferable.To his delight, Lowry found no difference between the

two treatments. He wrote an article about his experimentwhich he submitted to the premier marital therapy jour-nal, Eternal Togetherness (ET). He expressed his mainfinding this way:

The mean rated bliss of the Standard and Lowry treatmentgroups were 5.05 and 5.03, respectively. The differencebetween the groups was not statistically significant, t(38)= l.06,p > .05.

The ET reviewers thought that the lack of difference be-tween the two treatments had important practical impli-cations, for it meant that the samedegree of marital blisscould now be attained much more easily than had previ-ously been possible, and the ETeditor was thus inclinedto publish Lowry’sarticle. The editor was somewhatner-vous about publishing a conclusion that relied on accept-ing the null hypothesis, because it had beenfirmly drilledinto him during his graduate training that accepting thenull hypothesis is unacceptable. He thought that at leastLowry should do a power analysis. However, althoughthe editor had never admitted it to anyone, he did not ac-tually understand power very well. After pondering theproblem for awhile, he simply accepted the article with-out changes.The meaning of “no significant difference”. When I

read Lowry’s article, I was irritated. What did “no sig-nificant difference” mean? As I have noted earlier, itcould not imply that the population means of the two treat-ment groups are identical. That is a mathematical impos-sibility. However, identity of treatment groups is not reallyan issue in this practical arena. What is important is thisquestion: Are the two treatments sufficiently similar sothat one is justified in opting for the easier-to-use, cheaperLowry Treatment over the Standard Treatment?

The “no significant difference” that Lowry had re-ported could reflect any of many possibilities. To sim-plify, consider two polar alternatives. The first is thatLowry was a sloppy researcher, and that there was somuch variability within his two treatment groups (i.e.,such low experimental po’wer) that the actual populationmean difference between the two groups could plausiblybe just about anything. This possibility is illustrated inFigure 2A, wherein the small solid circles represent thetwo group means and the error bars represent the stan-dard errors. Note that the size of the error bars in Fig-ure 2A (large) provide a direct reflection of the power(low). At the risk of redundancy, I emphasize that stan-dard error bars always provide a direct reflection of ex-perimentalpower: the larger the standard errors, the lowerthe power.The second possibility was that therewas low variabil-

ity within the treatment groups (i.e., high experimentalpower) such that any actual population difference betweenthe two groups would have to be quite small. This possi-bility is illustrated in Figure 2B. The ETeditor’s intuitionwas correct: some kind of power analysis should havebeen done.In addition to knowing about experimental power, it

would be of substantial practical interest to know the stan-darddeviations3 of each of the two treatment groups. Suchknowledge would provide some indication of the rangeof martial bliss that any particular troubled couple mightexpect to achieve given either treatment. For instance,if the standard deviation of the Lowry Treatment groupwere small, any couple administered this treatment couldbe assured of eventual bliss fairly close to the mean ofabout 5; conversely, given a large standarddeviation, theprecise magnitude of any given Lowry Treatment cou-ple’s eventual bliss would be less certain.In short, Lowry’s article provided few clues about any-

thing having to do with the variability of marital bliss.Although information about variability is not directly

accessible in Lowry’ terse description of his results, itis partially computable from the sample sizes, the sam-ple means, and the t value. With this information, I wasable to compute that the standard error of the differencebetween the two population meanswas about 0.14, whichis quite small, given that the entire bliss scale goes from1 to7. It appeared that Lowry’ experiment had relativelyhigh experimental power; that is, in practical terms, anyactual difference between the two treatment populationmeans must be of little consequence. Thus, Lowry’s ac-tual data were more in accord with the Figure 2B exam-ple than with the Figure 2A example.Although I couldn’t compute the individual standard

deviations of the two groups, I could compute the mean4standard deviation of the two groups, which is 0.434. In-sofar as the two groups have similar standard deviations,this tells us that a couple receiving the Lowry Treatment(or the Standard Treatment for that matter) would, withabout 95% probability, end up with marital bliss of withinabout two standard deviations of the mean or, roughlyspeaking, somewhere between 4 and 6. This is important

254 LOFTUS

information for anyone actually considering one of thetreatments.To get more complete information, I e-mailed Lowry,

asking him for his raw data. Later the same day, Lowrye-mailed the data back to me. Electronically cutting thenumbers from Lowry’s e-mail message and pasting theminto a previously prepared Microsoft Excel spreadsheetallowed me to immediately calculate everything I wantedto know. What I discovered was interesting and some-what unexpected: the individual treatmentgroup standarddeviations were 0.608 for the Lowry Treatment and 0.086for the Standard Treatment. Thus, the Standard Treat-ment, although more costly, is more certain in terms ofwhat a given couple’s bliss will actually turn out to be.To generate a graphical representation of all this infor-

mation, I pasted the means, standard deviations, and stan-dard errors from Excel directly into my graphing appli-cation. With a couple of mouse clicks and keystrokes, Igot the graph shown in Figure 3.In this plot, the two black circles represent the two sam-

ple means. Each mean has two error bars associated withit, representing the standard error of the mean (shorter bar)and the standard deviation of the group (longer bar). I as-sert, as I did with the Loeb example, that this plot con-veys the information carried by the standard hypothesis-testingprocedure, plus additional, more interesting infor-mation. The virtual identity of the two means, inconjunc-tion with the sizes of the standard error bars conveys thehypothesis-testing information that the groups are “notsignificantly different.” That the error bars are relativelysmall indicates high power, which, in turn, implies thatthe actual difference between the two population meansmust also be small. The sizes of the standard deviationbars provides information about the range ofwhere a ran-dom couple ineither treatmentwould likely fall given thatthey had one treatment or the other. In short, this simplefigure visually and intuitively conveys all the importantand useful information about Lowry’s results that tookme a couple of paragraphs to convey textually. If Lowry

C)C

a:U).u 4C(a

had substituted something akin to Figure 3 for the APA-approved description of his results that appeared in hisETarticle, his readerswould have had a much easier timebecoming much more informed.

TODAY’S EASY-TO-USECOMPUTER GRAPHICS

I am by no means the first to argue that graphical rep-resentations in general, and the PPE procedure in partic-ular, are useful techniques for understanding and convey-ing information about the data from some experiment (cf.Tufte, 1983, 1990; Tukey, 1977). It is my hope that thepreceding examples, anecdotal though they may be, helpilluminate why this is so - In this final section, I will makesome comments about the nitty-gritty of actually im-plementing the kindsof graphical representations shownin Figures 1—3.

We Shouldn’t Throw Awayour Statistical Packages

There are a multitude of sophisticated and easily ob-tainable computer applications for doingvirtually any con-ceivable kind of statistical analysis. Even given what Ihave been saying, I believe these applications tobe veryuseful. I believe, however, that we should view theirprimary use as summarizing raw data and generating de-scriptive statistics, such as means, standard deviations,standard errors, and mean square errors. From these ap-plications, we can get the raw material used to generateplots of the sort shown in Figures 1 and 3.

Currently Available Graphing ApplicationsI would like to make a couple of points about presently

available graphing applications. First, I will talk aboutsuch applications in general, and then I will describe aninformal survey that I have conducted.

Two General Categoriesof Graphing ApplicationsGenerally speaking, scientists use two different kinds of

graphing applications: those that are associated withotherapplications (e.g, with statistical or spreadsheet applica-tions), and those that are “stand-alone.” With rare excep-tions, I prefer stand-alone applications for several reasons.First, they tend to be more powerful, more flexible, andeasier to use than graphing “features” that are subsidi-aries of something else. Second, the across-application cut-and-paste process has become so simple that it makessense to use each application for its primary function, inconjunction with transferring data from one applicationto another. Recall my descriptions of my interactions withDr. Lowry: I originally cut the raw data from one appli-cation (my communications application) and pasted it intoanother (Excel). Then I cut the Excel results and pastedthem directly into a third application (my graphing appli-cation). All of this took less than a minute.

7

6

Inner bars: Standard Errors of the MeanOuter bars: Standard Deviations

I . I

Lowry StandardTreatment

Figure 3. Lowry’s data: Mean rated marital bliss for two marital-treatment conditions. Shorter error bars aroundeach mean repre-sent standard errors of themean. Longer error bars representesti-mated population standard deviations.

HYROTHESIS TESTING 255

391310

45ISIt

222

225

28324432

202060

525

421742

Who Uses What Applications? Table 1What are the popular graphing applications at present?

In the beginning (that is, starting around 1986) the appli-cation of overwhelming choice was the CricketGraph,which ran only on the Macintosh. Even PC users some-

Frequency and Percentages of Various Mentioned GraphingApplications for Four Eifferent Operating Systems

Application Frequency Percent

Macintcsh (n = 87)how ferreted out their friends’ Macs (usually in the dead CricketGraphof night) and learned enough about them to use Cricket-Graph for their graphing chores.As the years passed, however, CricketGraph fell upon

hard times. It was sold first to Xerox, then to Computer

DeltaGraphKaleidaGraphSigmaPlot

Statistics/spreadsheetAssociates. Neither company appeared eager to updatethe application, and it languished, clearly becomingyesterday’s technology. It was generally incompatible withMacintosh System 7, and even caused some Macs run-fling under System 6.x to crash.5 Within the past year,

DOS — 25—

~°~raphics 8InPIot 1Fig-P 1

an update of CricketGraph (Version ifi) has materialized. Statistics/spreadsheet 8In the intervening time, however, a number of disgruntledusers turned elsewhere. Two applications in particular—KaleidaGraph and DeltaGraph—provedpopular with those

Wind~ws(n = 5)

CricketGraphCharisma

renewing their graphing arsenals. My personal favorite Statistics/spreadsheet 3is KaleidaGraph, which I used to create Figures 1—3.While writing this article (in October, 1992), I became

curious about what my colleagues used for graphing. Ac-cordingly, I carried out a very informal, nonrandom, andunscientific survey in which I first asked people to iden-

Uni, (n = 12)~GnuptotStatistics/spreadsheet

tify themselves as Macintosh, DOS, Windows, OS/2, orUNIX users, and then asked what they used to graph theirdata. I e-mailed this query to all psychologists on my plications. It is of some note that Microsoft Excel wase-mail address list, which included a total of 131 names. the only spreadsheet mentioned.Because one of the “names” was MacPsych, the entry Macintosh applicatiolis. Table 1 indicates an obviousinto a popular Macintosh chitchat network, the survey winner among Macintosh users: the venerable Cricket-recipients were highly biased toward being Macintosh Graph turned up 45% of the time, with DeltaGraph andusers. Nonetheless, the results are instructive. KaleidaGraph trailing quite far behind. A new highly flex-Within a couple of days, 94 people had responded, 93

of whom used some graphics application or another.ible application, Igor, was enthusiastically endorsed bytwo users.

Someof the respondents used more than one application, Other applications. DOS users reported being unhappyand 129 total mentions of applications were tallied. Ta- with the general state of DOS graphing applications. Theble 1 lists the frequencies with which various applications only ones mentioned by more than one person were Sig-were mentioned, subdividedby operating system. As an- maPlot and Harvard Graphics. (Of some interest is thatticipated, given the recipient bias, the large majority (87) only a single Macintosh user mentioned the reasonablyof the applications mentioned were run under Macintosh. respected SigmaPlot, although there exists a MacintoshThe remaining mentioned applications were run under version). Many DOS users reported that they used Macin-DOS (25), Windows (5), and UNIX(12). No one reported toshes to do their graphing.using OS/2. There were surprisingly fewWindows users. Of the fiveIn Table 1, under each operating system is listed the Windows respondents, one used Charisma, and the other

frequency with which various applications were reported used CricketGraph.to be used (many respondents used more than one appli- UNIX users generally favored the AT&T application, S.cation). I havecollapsedmentions of applications that arenot specifically graphing applications under the heading CONCLUSIONSStatistics/Spreadsheet. I counted a statistical or spread-sheet application only if the respondent specifically men- The main argument that I have tried to make in this ar-tioned using the application’s graphing capabilities. Thus, tide is that hypothesis testing is the wave of the past (andfor example, a number of people described doingdata ma- never should havebeen a wave at all). Characterizing con-nipulation in Excel and shipping the results to Cricket- clusions in hypothesis-testing terms requires reducing theGraph for graphing. For such a person, CricketGraph complex, multidimensional information that generallywould be counted, but Excel would not. In all, spread- emerges from an experiment into one or more binary de-sheet and statistical applications were reasonably popu- cisions that are almost always logically predetermined tolar for graphing, constituting 30% of all mentioned ap- begin with.

256 LOFTUS

I have argued that presenting data in the form of oneor more well-designed graphs—particularly graphs that rep-resent the relevant sample means along with various mea-sures of inferred variability—potentially conveys the in-teresting and important information from the experimentin a manner that (1) is immediate and direct and (2) doesnot entail a pseudoprecise attribute (viz., p < .05) that doeslittle but fool naive readers into thinking that some impor-tant conclusion about reality has beenmade. In particular,the size of the standard errors of the meanprovides a directand intuitive visual measure of how precisely the locationsof the relevant population means—and thus the overall pat-tern of population means—can be inferred.Given this strategy, it is important to havepowerful and

easy-to-use tools. There are many such tools in today’smicrocomputer environment. Any of the applications listedin Table 1 would be perfectly adequate for the task, al-though obviously the applications differ along a varietyof dimensions.I believe that the family of PPE techniques, illustrated

in Figures 1-3, have enormous potential for efficientlyconveying information about experimental results. I hopethat members of our discipline, like our natural-sciencebrethren, will begin availing themselves of this potentialmore than is presently the case. In a Memory & Cogni-tion editorial (Loftus, 1993) I pursue this hope further,and provide it with more teeth.

REFERENCES

BAKAN, D. (1966). The test of significance in psychological research.Psychological Bulletin, 66, 423-437.

COHEN, J. (1990). Things I have learned (so far). American Psycholo-gist, 45, 1304-1312.

GIGERENZER, G., SWIJTINK, Z., PORTER, T., DASTON, L., BEATTY, J.,& KRUGER, L. (1989). The Empire ofchance: Howprobability changedscience and everyday life. Cambridge, U.K.: Cambridge UniversityPress.

Lovrus, G. R. (1991). On the tyranny of hypothesis testing in the so-cial sciences. Contemporary Psychology, 36, 102-105.

Lovrus, G. R. (1993). Editorial comment. Memory & Cognition, 21,1-3.

MELTON, A. W. (1962). Editorial. Journal of Ezperimental Psychol-ogy, 64, 553-557.

NUNNALLY, J. (1960). The place of statistics in psychology. Educa-tional & Psychological Measurement, 20, 641-650.

TUFTE, E. R. (1983). The visual display of quantitative information.Cheshire, CT: Graphics Press.

TuFTE, E. R. (1990). Envisioning information. Cheshire, CT: GraphicsPress.

TUKEY, J. W. (1977). Exploratory data analysis. Reading, MA:Addison-Wesley.

NOTES

1. The basic idea is as follows. The null hypothesis typically statesthat somepopulation parameter (e.g, a population mean, the differencebetween two population means, a population correlation) is identicallyequal to some constant. Only rarely could such a hypothesis actuallybe true. Consider, for example, an experiment in which two treatments,A and B, were being compared. The null hypothesis would be “the dif-ference between theTreatment Aoutcome score and the Treatment Boutcome score is zero (to an infinite number ofdecimal places).” Sucha null hypothesis could not literally be true. So the results of a signifi-cance test do not, as advertised, tell us whether or not the null hypothe-sis is actually false (we know a priori that it is false). Rather, the re-sults simply tell us whether there is sufficient experimental power todetect the population mean difference that inevitably exists.2. For illustrative purposes, I assume throughout this article that sample

means are the primary data of interest. All arguments could be equallywell applied to any sample statistic.3. The term “treatment group standard deviation” carries with it some

ambiguity: it could refer either to the group’s actual standard devia-tion, or to the estimate of the relevant population standard deviation(these two statistics differ by a factor of n/In—li). For the purposesof this discussion, I refer to the latter.4. Not the arithmetic mean, actually, but the standard deviation of

the mean of the two individual treatment-group variances.5. Its compatibility with Version 6.x turns out to depend on exactly

what ROM the computer has. We determined this factoid by runningCricketGraph 1.3 on two seemingly identical Mac ilci computers withthe same floppy-based system. Itworked on one and crashed the other.The only difference between the two computers was theROMs they used.6. One respondent, a world-famous visual perception expert, claimed

to still do his graphing by hand.