7/30/2019 McCloskey the Cult of Statistical Significance Article Review

1/3 2009 The Authors. Journal compilation Institute of Economic Affairs 2009. Published by Blackwell Publishing, Oxford

104 b o o k r e v i e w s

291BookReviewbookreviewsXX

T H E P R E D A T O R

S T AT E : H O W

C O N S E R V A T I V E S

A B A N D O N E D T H E

F R E E M A R K E T A N DW H Y L I B E R A L S

S H O U L D T O O

James K. Galbraith

New York: The Free Press, 240pp.,

ISBN: 141 656 683X, 16.99 (hb), 2008

The title ofThe Predator State is partly amisnomer. This short book by economist

James K. Galbraith, son of the celebratedeconomist John K. Galbraith is, in fact,a well-timed critique of what the authorsees as the conventional wisdom ineconomics and a call for an expansion ofthe role of the state. Only its subtitlereveals its true theme:How Conservatives

Abandoned the Free Market and WhyLiberals Should Too. I will deal with thegood things about the book first beforemoving on to discuss my misgivings.

The book is predominantly aimed atAmerican Democrats, but it is of interest

to a wider audience for a number ofreasons. The first is that, though the sondoes not write as elegantly as his fatherdid, the younger Galbraith doesnonetheless succeed in ably slaying anumber of shibboleths. More than a fewof these are held by the left. For instance,he claims that the USA is both moreegalitarian and more efficient than theEuropean Union taken as a whole. Theidea that carbon permits should beglobally distributed on a per capita basis

is rubbished as impractical. Galbraithrightly observes that global trading incarbon emissions is unfeasible becausegovernments in developing countrieswould be unable to enforce their quotas.Proposals linking labour standards totrade deals are similarly given shortshrift.

Secondly, and most interestingly,Galbraiths perspective is close to whatcould be called left-wing public choice.This sometimes makes for more informedand interesting analysis than economistslike Joseph Stiglitz or Paul Krugman canoffer. Stiglitz et al. still inhabit anintellectual universe where it is thoughtuseful to view the policies of government

as those of a benign dictator maximisingsocial utility. Galbraith, on the otherhand, seems to understand thatregulation is often driven by specialinterests rather than by governmentbenevolence.

This brings us to the title of the book.Galbraith rightly notes that despite thefree-market rhetoric espoused byRepublican politicians, the New Dealinstitutions have endured and still shapethe American economy. They survivedReagan quite intact. According toGalbraith, the problem is not that thestate has been captured by ideologuesintent on dismantling it, but that it hasbeen captured by plutocrats intent onexploiting it. The predator state consists

of reactionary businessmen: the worstpolluters, the flagrant monopolists,the technological footdraggers (p. 148).They are unprincipled conservativeswhose reason for being, rather, is tomake money off the state so long as theycontrol it (p. 132).

The question then is: how do we dealwith The Predator State? Here theproblems I have with Galbraiths analysisbecome more pressing. I will mentiononly the most striking.

The book is aimed at the general

reader but economists who read it willspot that Galbraith is that rare beast, along-run Keynesian. Unlike Stiglitz orKrugman, Galbraith holds that economicgrowth is dependent on maintaining highdemand and high investment, and thatinflation is a costpush phenomenon.This viewpoint is deeply unconvincingand it vitiates Galbraiths practicalproposals.

Furthermore, Galbraith appliespublic-choice-style reasoning in one

direction only. The solution to thepredatory state proposed by Galbraith isan alternative coalition of progressivegroups such as trade unions, consumergroups and more reasonablebusinessmen. This is unsatisfying unlesswe believe, as Galbraith appears to, thatindividuals who profess to be progressivecannot be predatory.

Finally, the author makes a numberof outright errors. Adam Smith ismisrepresented as envisioning a vasttrading empire, defended as necessary bymilitary force (p. 67). In fact, Smithdescribed the British empire as a projectwhich has involved immense expense,without being likely to bring any profit

(Wealth of Nations, Bk. v, Ch. iii).Apparently utterly ignorant of thesituation in countries like the UK,Galbraith claims in Chapter 11 that if thestate provided universal healthcare, themarket for private health insurance would

disappear. Galbraiths discussion ofeconomic freedom, derided as thefreedom to shop, is laughablyinsufficient. And at other times he simplyattacks a straw man of perfectlycompetitive markets that is irrelevant tothe task of comparing how differentreal-world comparative economic systemsactually function. This means that thebook is simply too lightweight toconvince anyone not already on the left.The chapter on planning which is central

to Galbraiths alternative economic visionis woefully short on details or analysis.Perhaps Galbraith will offer us a more

detailed and intellectually rigorousmanifesto for planning and regulation inthe future. Until that time, we are leftwith a book which, though lively andprovocative, is ultimately unconvincing.Mark Koyama

DPhil Candidate in Economics

Department of Economics and Wadham

College

University of Oxford

Mark.Koyama@Economics.Oxford.ac.uk291BookReviewbookreviewsXX

T H E C U L T O F

S T A T I S T I C A L

S I G N I F I C A N C E :

H O W T H E S T A N D A R D

E R R O R C O S T S U S

J O B S , J U S T I C E , A N D

L I V E S

Stephen T. Ziliak andDeirdre N. McCloskey

Ann Arbor, MI: University of Michigan Press,

352pp.,

ISBN: 047 205 0079, 18.50 (pb), 2008

The quantitative parts of a science should not

be notable mainly for their lack of common

sense.

(Ziliak and McCloskey, p. 126)

Imagine an Alexandrian disaster in whichall empirical social science researchdisappears immediately. It seems like this

7/30/2019 McCloskey the Cult of Statistical Significance Article Review

2/3

2009 The Authors. Journal compilation Institute of Economic Affairs 2009. Published by Blackwell Publishing, Oxford

iea e c o n o m i c a f f a i r s m a r c h 2 0 0 9 105

would be a disaster of epic magnitude,but Ziliak and McCloskey might arguethat we probably wouldnt have lostmuch. The book under review is theculmination of about two and a halfdecades of inquiry by McCloskey and

about a decade and a half of involvementin the project by Ziliak. It is anundertaking of truly McCloskeyanproportions: Professor McCloskey is anexcellent writer of big-think science, andZiliak has proven to be a more than ablecollaborator. They seek to expose t-testingfor statistical significance as the nakedemperor of empirical research. Theirs is aplea for statistics rightly done. This bookwill some day be assigned in first-yeareconometrics courses. The essence of

their claim is based on the strongstatement they make on p. 9: Fisher-significance is by itself about preciselynothing. According to Ziliak andMcCloskey, statistical testing is precisely-measured sound and fury. And yet itpersists, even at the highest levels of thediscipline.

Questions of statistical significanceare fundamentally questions about theexistence of an effect rather thanmeasurement of an effect, as one can seefrom the rhetoric of empirical social

science. It suggests a McCloskeyan theoryof regression analysis in the spirit of herA-prime, C-prime theory of economictheory. McCloskey argues that for everyset of assumptions A producingconclusions C, there exists a set ofassumptions A arbitrarily close to A thatwill produce conclusions C arbitrarily farfrom C. One can say the same thing aboutregressions. For every set of regressions Rthat produce a conclusion C, there existsa set of regressions R arbitrarily close to

R that produce a set of conclusions Carbitrarily far from C. One finds or

identifies an effect based on a t-ratio.The t-ratio is mute on whether theidentified effect is quantitativelysignificant.

Reading their book is a bit likereading Old Testament prophecy. Aftera moments reflection, the insight isobvious: the standard error of aregression coefficient tells us about theproperties of the sampling distribution,not about the magnitude of the estimatedeffect. The significance of the coefficientcan be altered by changing n.Significance can change radically even ifthe coefficient doesnt budge as long as

one changes the sample size. Addingmore degrees of freedom leads to greatersignificance; reducing the number ofdegrees of freedom reduces it.

But it is more than a jeremiad. Twoobvious questions the authors address

concern how the sciences were brought tosuch a lowly state and how it persisted forso long. I interpret their argument to relyin part on a view of a tragic historicalaccident: Ronald A. Fisher (theCambridge statistician and villain in theirstory) was writing at a time at which theintellectual atmosphere was particularlyconducive to it. It was a complement toneopositivism and falsificationism in thesciences, with Fisher positing that the keyquestion concerns whether one should

reject or not reject various nullhypotheses (p. 149) even thoughfalsificationism and hypothetico-deductivism are illogical, erroneousdeductions (p. 150). One factor they donot discuss is the role of socialism insocial science, in the idea that aneconomy could be successfully andscientifically managed and fine-tuned.

They offer an intellectual history ofstatistical inference which I hope will befleshed out in greater detail in Ziliaksin-progress biography of W. S. Gosset. In

Chapter 18, they discuss Pearsons three-sigma rule, which is basically the ideathat a single statistical test is sufficient inthe neopositivist framework, an inversionof the traditional work of science whichlooked at all the evidence. They offer along and digressive discussion of theintellectual history of significance testing,with Fisher clearly in the role of thevillain and Gosset clearly in the role of thehero.

Significance testing makes for a very

easy, very quick way to assess output:ifp < 0.05, youve found something. Ifp > 0.05, you havent. I would also arguethat the rhetoric of analysis and evaluationin government, education and thebusiness world is partially responsible, aswell. There is nothing wrong withdeveloping measurable standards, but thestandards themselves rarely have a lossfunction. So what is one to do here?According to Ziliak and McCloskey, itappears that there is a simple solution:re-build empirical research from theground up. A key to understanding socialphenomena is to understand exactly whatwe mean when we are carrying outempirical investigations.

Our research is often hard-boiled andhalf-baked, and bad science based purelyon significance testing reduces quality oflife. They discuss, for example, the Vioxxdrug research (pp. 2831) that led toseveral deaths. The key problem was one

first of lying about the data, apparently,but second it was a problem of using thewrong metric statistical significance asa measure of the impact of Vioxx on heartattack probability. This is part of thesocial search process whereby moresuitable ways of doing things emerge. Atthe same time, however, it is tragic thatdeaths (which could have been avoided)occurred.

Numbers are mysterious, and theyseem objective. Significance testing has

the virtue of looking scientific whileexcusing the analyst from takingresponsibility for his or her conclusions.Even in epidemiology (p. 162), Statisticalsignificance came to meanepidemiological significance. Statisticalinsignificance came to mean ignore theresults. They are adamant thatstatistical significance is not the samething as clinical or economic significance,even though it is treated as such byscholars and commentators. What Ziliakand McCloskey would prescribe is a

sophisticated contextual analysis ofscientific results, which requires seriouseconomic thinking. One gets theimpression from reading Ziliak andMcCloskey that the obsession withstatistical significance is a type ofpseudo-intellectual fast talk.

In spite of the claim of objectivity(p. 45), significance testing boils down toan appeal to the authority of Ronald A.Fisher, but Ziliak and McCloskey hope tomove the scientific question beyond do

we have enough data to determinewhether the effect is or isnt zero? Theydo not mince words in their contemptfor the significance-testing paradigm:You might as well use a table of randomnumbers or a late-night phone call to aforeign-language psychic network tomake your scientific decisions (p. 54).For science to proceed we have to thinkabout analysis like economists,comparing the marginal costs andmarginal benefits of different thresholdchoices. In comparison to testimators,Ziliak and McCloskey argue thatScientists make a judgment about athreshold and get on with it (p. 54) withthe goal of being quantitatively

7/30/2019 McCloskey the Cult of Statistical Significance Article Review

3/3

2009 The Authors. Journal compilation Institute of Economic Affairs 2009. Published by Blackwell Publishing, Oxford

106 b o o k r e v i e w s

persuasive, not to be irrelevantlymechanical (p. 55).

The incentives, though, encourage theirrelevant and the mechanistic.Scholarship is a series of short-run goals,and what is in the scholars short-run

interest is not necessarily in the long-runinterest of good science. Finish thedissertation. Get a job. Get tenure. Getpromoted. This is part of the process thatcreates path dependence in scientificmethods. At the same time, however, overthe very long run what is very sexy todayis unlikely to be very sexy tomorrow.Ziliak and McCloskey make this pointspecifically on p. 32, arguing thatsignificance testing brings about variousprofessional niceties but that it is not

science, and it will not last. This providesan intriguing set of prescriptions for theambitious scholar. The cynic, who isperhaps focused on the luxuries that comewith a successful career rather than ondoing good science, sees nothing wrongwith the existing apparatus. The scientist,on the other hand, is presumably moreinterested in truth.

The books penultimate chapter offersa meta-economic and sociological analysisof the factors that help explain thesetrends. There are implications for editors

in that we are in part caught in atransitional gains trap, and we cause acapital loss for people who have builtreputational capital on significancetesting. They refer to the careerist view ofsignificance testing as amused cynicism(p. 59). Again, the question is aboutwhether one is doing science or whetherone just has a job. They argue that thephilosophical atmosphere of 192262 wasperfect for the fastening of Fishers gripon the sizeless sciences (p. 149). A future

jaunt into intellectual history, and achapter I hope we will see in Ziliaksforthcoming book, will relate theFisherian paradigm to the zeal for centralplanning, particularly in the New Deal.

Perhaps there is some overkill in theirsupport for their point. On pp. 5759,they offer a very long bibliography ofstudies on this, suggesting that their pointis not new. This is perhapsunderstandable because Ziliak andMcCloskey have spent about two and ahalf decades as voices crying out in thewilderness. They summarise the findingsof their earlier studies of the use ofstatistical significance in economics

journals; to the shame of the profession,

the statistical analysis in our flagshipjournal theAmerican Economic Review is, in their estimation, anti-scientific.Towards the end of the book they offer amulti-chapter biographical treatment ofKarl Pearson, Gosset and Fisher that

explores the personalities in the debate.I assume these chapters will be fleshed outin greater detail in Ziliaks forthcomingbook.

They offer several explanations for theseeming inefficiency of the intellectualmarketplace, one of which is a Great Mantheory whereby they argue that, withoutFisher, this perhaps would not havehappened. Another possibility, of course,is that the Fisherian programme is best,but they are persuasive in their

demonstration that the Fisherianprogramme is notthe best. Third, theyargue that Path Dependence is at work. Itis hard to switch off the path, and thelarge switching costs suggest thatsignificance testing is okay. It is, quiteliterally, good enough for governmentwork. At the same time, though, pathdependence can sometimes be a just-sopunt for phenomena we do notunderstand fully.

They argue in favour of anexplanation relying on High Modernism,

suggesting that Fisher had the goodfortune to be born just as the prestige ofmechanical methods in all fields, frommathematics to automobilemanufacturing, was coming to a climax(p. 243). They argue further thatsignificance testing is an expression ofwhat Robert Merton has called theBureaucratization of Knowledge (p. 243).In my notes on p. 243, I note that thislooks an awful lot like what F. A. Hayekhas called scientism. Then they invoke

Hayek and scientism on p. 244.An important factor here is that thereare incomplete markets for statisticalanalysis. The feedback mechanism is verynoisy. The FDA cant go out of business,for example. What matters for regressionoutput, according to some scholars, is justthe column ofp-values, and the feedbackmechanism rarely tells researchersotherwise. The authors are pessimistic;they believe that there is little in theinstitutional infrastructure of science thatwill correct these systematic errors.

Ziliak and McCloskey close with adramatic exhortation: The textbooks arewrong. The teaching is wrong. Theseminar you just attended is wrong. The

most prestigious journal in your scientificfield is wrong (p. 250). What they haveproduced is the culmination of a long anddetailed research agenda and proposes anew framework for empirical science. It issomething we would perhaps do well to

take to heart.Art Carden

Assistant Professor

Department of Economics and Business

Rhodes College

Memphis, USA

cardena@rhodes.edu291BookReviewbookreviewsXX

T H E C U L T O F T H E

P R E S I D E N C Y

Gene Healy

Washington, DC: Cato Institute, 264pp.,

ISBN: 193 399 5157, $22.95 (hb), 2008

In Virginia, two weeks before thepresidential election, I found myself inconversation with an older lady. She said,in a disconsolate southern drawl, I do notbelieve we have in America today thesame quality of people we had at theFounding.

But there were rogues even then,I protested. Plenty of them. And thereare plenty of good people today. It is notAmericans who have changed. What haschanged is the office of the presidency,and what we have come to expect fromhim.

Yes, she said with resignation. Hehas become an Emperor.

Gene Healys new book, The Cult of thePresidency, is the story of how we arrivedat this point.

The early Americans understood alltoo well the trouble a country could endup in at the hands of a whimsical andall-powerful ruler. They had a keenappreciation for both human fallibilityand the corrupting effects of power. Theirinsight into human nature, as well as thehistorical antecedents, led the Foundersto want to limit the powers of the newpresident, and particularly to limit hiscapacity to wage war. The Constitutionthey ultimately wrote separated the powerto authorise or initiate war from thepower to direct it and at the same timegave the presidency a modest role.

The presidency that Healy describesin its first one hundred years is