1
“Asking Interesting Questions” Prepared for Sage Handbook of Research Methods in Political Science and International Relations. Robert J. Franzese, Jr. and Luigi Curini, Editors. Forthcoming. ByWilliamRobertsClark1TexasA&MUniversity
Goodresearchisdrivenbyimpatiencewithbadanswerstointerestingquestions.But
wheredointerestingquestionscomefrom?Sincethisistheopeningchapterofahandbookon
researchmethods,itisimperativetopointoutatthestartthatthereisno“method”toasking
researchquestionsinthesenseofacookbookthatyoucanfollowthatwilllead,inexorably,to
scientificdiscovery.Theremaybeascientificmethodforevaluatinganswers,butthereis
certainlynoscientificmethodforaskingquestionsorgeneratinganswers.Andthereis
certainlyroomforalotofcreativityindevelopinginterestingandenlighteningresearch
designs,andseriousshortcomingsto“cookbook”approaches.2KarlPopper,(1962,2003)for
example,arguedthatsciencebeginsafterascientisthasconjecturedananswertoaquestion.
Thescientificmethod,therefore,ismore(perhapsonly)usefulinevaluatinganswersto
questions.Generatingquestionsandanswers,incontrast,isasmuchanartasitisascience.
Butthatisnottosaythattheprocessisrandomorlacksstructure.ThomasKuhn(1962)
saysepisodesofscientificdiscoverybeginwithanindividualwiththe“skill,wit,orgeniusto
recognizethatsomethinghasgonewronginwaysthatmayproveconsequential.”(p.763)But,
hehastenstoadd,“anomaliesdonotemergefromthenormalcourseofscientificresearch1TheauthorwishestothankBranislavSlanchevandLaurieClarkforthoughtfulcommentsandusefulsuggestionsonanearlierdraftofthispaper.2Thefadaround“cleveridentificationstrategies”isbutthemostrecentinstantiationofthisphenomenon.
2
untilbothinstrumentsandconceptshavedevelopedsufficientlytomaketheiremergencelikely
andtomaketheanomalywhichresultsrecognizableasaviolationofexpectation.”(p.763)
Intheparlanceofsocialmedia,scientificdiscoverybeginswitha“WTF”moment.
Scientificdiscoverybeginswhenascholarobservessomethingcontrarytoexpectationsand
recognizesthatthisanomalousobservation“mayproveconsequential.”Notethatthe
motivatingfactmaybeanobservationabouttheworld,butitmayalsobeaboutwhatothers
havesaidabouttheworld.3
Butnotjustanysurprisewilldo.Anyonewhohaseverparentedayoungchildisfamiliar
withthequestions,bornoutofwonder,suchasthosethatourchildrenaskedmypartnerand
me:“whyistheskyblue?,”“wheredoesthesungo(attheendoftheday),”?or,“ifmybrain
controlsmybody,whydoIhavetogotothedoctortofindoutwhat’swrongwithmewhenI
amsick”?Answerstoallofthesequestions(assumingtheyareconsistentwithwhatscientists
currentlybelieve)arediscoveriesfortheinquirerbecausetheychangewhattheyknow,but
theydonotleadtoscientificdiscoveriesunlesstheychangewhatweknow.Thefieldsof
optics,astronomy,andneurosciencehavetheirrespectiveanswerstothequestionsabove
(althoughthelastquestionisprobablylesssettledthantheothertwo).
So,questionsoftenbeginwithsurprise,butgoodresearchquestionsbeginwithwell-
informedsurprise.Ifyoualonearesurprisedbyanobservation,theanswertoyour“WTF
3InthewordsofBranislavSlantchev,(personalcommunication)“theoreticinnovationdoesnothavetobeginwithanempiricalobservationbutwithapotentialflawinthelogic,inconsistencyoftheassumptions,oraninsightaboutageneralclaim(e.g.,theimpossibilityresults)”.
3
moment”islikelytobepersonallyrewarding.Ifmostwell-informedobserversaresurprisedby
anobservation,thenananswerislikelytobesocially,and,therefore,scientificallyvaluable.4
Butsometimes,scienceproceedswhenanindividualrecognizesthattheanswers
embodiedinwhat“weknow”aboutasubjectarenotverygood.Forexample,formillennia
“we”knewthattheanswertothequestion“wheredoesthesungo”tobesomethinglike“the
suncirclesastationaryearth,soatacertainpointeachdayitleavesoursightwhileshinningon
theotherhalfoftheplanetonlytoreturnthenextmorning.”Eventually,however,scientists
with“theskill,wit,orgenius”torecognizethemountinganomaliescreatedbymodelsbasedon
ageocentricviewoftheuniversecametotheconclusionthatabetteranswerwasneeded.At
firstthesebetteranswerscameinattemptstomodifytheheliocentricviewwithelaborate
patchesmeanttoexplainawayanomalousobservations.Inaddition,to“skill,wit,andgenius”
itrequiredagreatdealofcouragetochallengetheexistingviewinamorefundamental
fashion.
So,goodquestionscomefromknowingwhat“we”know.Buttheyalsocomefrom
thinkingdeeplyaboutwhatweknowandbeingsufficientlyunsatisfiedwithbadanswersto
taketheriskofthinkingdifferentlyaboutaproblem.Aswithallthearts,goodscienceseems
tocomefromindividualsandgroupsthatengageinacertainkindofpractice.Iwouldliketo
beginthisessaybycommentingonwhatIseeasacommonstructureofmanygreat
contributionstopoliticalscienceandinternationalrelations.Specifically,Iwillputforwardalist
4Itisfashionableinmanytopgraduatepoliticalscienceprogramsforfacultytosaythat“substantivecourses”areawasteoftimeandenterprisingstudentsshouldhaveanalmostsingle-mindedfocusonmethodstraining.Itisalsocommonplaceforprofessorstocomplainthattheirstudentsarenotadeptatidentifyinginterestingquestions.Isuspectthatthesephenomenaarenotunrelated.
4
offivequestionsthat,whenansweredwell,arelikelytoproduceworkthatasksandanswers
interestingandimportantquestionsandgivesusareasontobeconfidentinthoseanswers.In
thesecondhalfoftheessayIwillruminateonthekindofpracticethatIexpecttoleadtogood
questionaskingandgoodanswergiving.
FiveQuestions
WhenIwasingraduateschool,oneofmyprofessors,D.MichaelShafer,taughtme
howtoread.Hedidsobyencouragingmetoemployatemplatehecreatedsostudentscould
recordthekeypartsofwhattheyread:“Whatisthedependentvariable?”“Whatarethe
independentvariables?“Whatisthelogicthattiesthemtogether?Etc.”Ifoundthis
enormouslyhelpfulingettingthroughtheridiculousamountofreadingrequiredinmy
graduateclasses.WhenIbeganteachingIsharedthislistwithmystudentsandovertheyearsI
haverefineditforvariousreasons.Ihavecometobelievethatthislistofquestionsisuseful
notjustinfocusingourreadingefforts,butalsoinourresearchefforts.Ifyouaskwhatthe
author’sanswertoeachofthefollowingquestionsis,youwillhaveagoodsummaryofmost
articlesorbooksinourdiscipline.5Ifyouaskwhethertheauthorhasagoodanswertoeachof
thequestions,youwillhaveagoodcritiqueofthepaperinquestion.Andifyouareimpatient
withanybadanswersprovidedbytheauthor,anddevelopbetterones,youwillbeonyourway
tomakingyourowncontributiontotheliterature.Consequently,Ihavecometobelievethat
thesequestionscanalsoserveasanexcellentguidewhendesigningaresearchproject.Ifyou
havegoodanswerstothesefivequestions(andatleastoneoftheseanswersisan5Thequestionswouldhavetobeadaptedtoservethispurposeforliteraturereviews,methodspapers,andpurelytheoreticalpapers.
5
improvementoverexistingwork),youwillhaveagoodpaper,dissertation,orbook.These
questionsalsocorrespondtotheorganizationofthemodalpaperinourdiscipline:
“Introduction”,“LiteratureReview”,“Theory”,“ResearchDesign”,and“Findings”.
Itisimportanttoaddthatresearchquestionsneednotbegeneratedbyreading.They
canjustaseasily,andperhapsmoreprofoundly,beprovokedbyourinteractionand
observationofthesocialworld.Wemightobservebehaviorandsay“whydoesthathappen”?
Itisgoodpracticetoofferone’stentativeanswertosuchaquestionunencumberedby“the
literature.”Butitisimprudenttospendverymuchtimeonsuchactivitybeforeevaluating
existinganswerstoyourquestion.
Question1:WhatdoIwishtoexplain?(TheIntroduction)
FollowingKuhn’sdescriptionofscientificrevolutions,mostgoodworkbeginswitha
puzzlingobservation.Beginningwithobservationisimportantbecausegoodreaderswouldlike
tobeconvincedthatthephenomenonyouareexplainingactuallyoccurs(thoughitis
frequentlyfruitfultoengageinthoughtexperimentsaboutthingsthathavenotoccurred).
Thisstepisbynomeanstrivialandconsiderablemethodologicalsophisticationmaybe
necessarytoaccuratelydescribetherealworldeventsor,betterstill-patternsofevents-which
youwishtoexplain.
SamuelHuntington’sclassicPoliticalOrderinChangingSocieties(1968)seekstoexplain
therisingpoliticalinstabilityheobservedaroundtheworld.Asevidenceofthisrising
instability,onpagefourofthis462pagebook,theauthorpresentsU.S.DepartmentofDefense
datashowingthatthenumberofnationsaroundtheworldexperiencingmilitaryconflictsof
6
varioustypesrosealmostmonotonicallyfrom34in1958to57in1965(Table1.1).Thisisa
dramaticincrease:inlessthanadecadethenumberofconflictsnearlydoubled!Theproblem,
however,isthat,asaresultofdecolonization,thenumberofindependentcountriesinthe
worldalsogrewrapidlyduringthisperiod.IfonetakesHuntington’snumbersanddividesthem
bythenumberofindependentcountriesineachyear(asameasureoftheopportunityfor
militaryconflict),therelativefrequencyofmilitaryconflictactuallydeclinedoverthisperiod.
Sincemilitaryconflictwasjustoneproxyforpoliticalinstability,itisentirelypossiblethat
politicalinstabilityactuallyincreasedduringtheobservedperiod.Butifyoubelievethat
therelativefrequencyofconflictisabetterindicatorofpoliticalinstabilitythantheraw
frequency,youwouldbejustifiedinwonderingifthephenomenonexplainedinthesubsequent
fourhundredorsopagesactuallyoccurred.
Thefirstorderofbusiness,therefore,indemonstratingthatsomethingthatmayprove
consequentialhashappened;istodemonstratethatthatthinghashappened.Thiscrucialtask
7
isoftenbestaccomplishedwiththepresentationofclearlypresented,wellthoughtout,
descriptiveevidence.Whilethisoftenrequiresafairamountofmethodologicalskill,
sometimesitsimplyrequiresnumeracy–which,unfortunately,isofteninshortsupply.
Effectivelypresentingevidenceforone’sexplanandumis,perhaps,bestdescribedinthe
breach.Forexample,youcanreadnewspaperheadlinesonalmostadailybasisthatpurportto
capturesomeimportantchangeintheworldthatis,infact,notsupportedbythetextofthe
accompanyingarticle.Wouldthatitwerethecasethatthesemistakeswererareinacademic
work.
Onecommonmistakeistomakeaclaimaboutinter-temporalchangeinavariableby
citingonlycurrentvaluesofthatvariable.“Tenure-trackjobsaredisappearing”readsthetitle
ofanarticle,butthearticlemakesnoreferencetothenumberofsuchjobsthatwereavailable
inthepast.Howdoweknowthatchangehasoccurred?Arelatedissuethatrequiresabit
moremethodologicalskilltoavoidistopointoutadifferencebetweenthevaluesofafew
recentvaluesofavariablefromprecedingvaluesandclaimthattheyareevidenceofanew
trendwithoutcomparingthenewobservationswithalongenoughtrendofdatatodetermine
whethertheyrepresentameaningfuldeviationfromthetrendor,asisoftenthecase,just
typicalvariationwithinthetrend.
Anothercommonerroriswhatmightbecalled“thedenominatorproblem”–thefailure
tochooseadenominatorthatwouldtransformthedataintoavariableappropriateforthe
conceptualcomparisonrelevanttothediscussionathand.Wealreadysawanexampleofthis
whenHuntingtonconfusedatrendintherawfrequencyofavariableforatrendintherelative
frequencyofthedata,whichIarguedwouldhavebeenmoreappropriate.Butitisalsopossible
8
thattherawnumberiswhatmostinterestsus–inwhichcaseweshouldnotbedistractedby
anapparentlyrelatedratio.Toreturntothe“disappearingtenuretrackjobs”problemwe
oftenhearaboutinthepopularpress,intherareinstanceswhereinter-temporaldatais
presentedinanattempttoestablishthistrend,thequantitypresentedistypicallytheratioof
tenuretrackjobstothetotalnumbercollegeteachingjobs.Thisisproblematicbecauseitis
entirelypossiblefortheshareoftenuretrackjobstobedecliningwhenthenumberoftenure
trackjobsisincreasing(ashasbeenthecaseintheUnitedStatesfordecades).Anditis
probablythelatternumberthatisofinteresttomostreaders(forexample,currentdoctoral
studentshopingtoforecastfuturedemandforpeoplewiththecredentialstheyareworking
hardtoobtain).
Question2:Whydoesitneedtobetoexplained?(TheLiteratureReview)
Havingexplainedthatthisthinghasoccurred,itisimportantforauthorstodemonstrate
thata)thisthingviolatesexpectationsinsomeway(i.e.“somethinghasgonewrong”)andb)
thatthisviolationmay“proveconsequential.”Inotherwords,inthewordsofMilesDavis,“so
what?”
Onceagain,itmightbeeasiertosaywhatoneshouldnotdo.Ionceattendedapractice
jobtalkwhereasmart,hard-workingand,subsequently,verysuccessfulscholar,whenpressed
tosaywhathewastryingtoexplain,saidthathewastryingtoexplainwhyaparticularvariable
varies.BeinglesssupportivethanIshouldhavebeen,Iasked,“doyouhaveatheorythatleads
ustoexpectthisvariabletobeaconstant?”Variablesvary.Itisevenintheirname.
9
Observingthatvariation,therefore,hardlyconstitutesasurprise.Soifvariationinavariable
doesnotconstituteaviolationofexpectations,whatdoes?
Asacomparativepoliticsscholar,itpainsmetosaythatIhaveattendedmanyseminar
talksoverthelastfewdecades,mostgivenbysuccessfulandinfluentialseniorscholars,where
theworkinprogressismotivatedbyanassertionthatissomevariantofthefollowing“puzzle”:
theoryQclaimsthathighlevelsofvariableXshouldcauseYtohappen,butincountryiattimet,Xwasveryhigh,andYdidnotoccur.
Theproblemwiththis“puzzle”isthatoncethemisunderstandingitisbasedonisclearedup,it
isnolongerapuzzle.Themisunderstandingisthis:withveryfewexceptions(Icannotthinkof
one)theempiricalimplicationsofsocialscientifictheoriesarebesttreatedasprobabilistic
(Lieberson1991).Whetheronetracesthereasonstotheintrinsicallyprobabilisticnatureofall
humanbehaviorderivingfromhumanagency,thelimitationsofourunderstanding,thefact
thatmost(all?)socialphenomenahavemultiple,context-dependentcauses,orthepossibility
ofclassificationerror(didYoccurordiditnot?wasXreallyhighorlow?andcomparedto
what?)itisbesttothinkofourhypothesesasprobabilistic.WhichmeansthemosttheoryQ
canclaimisthat“highlevelsofvariableXshouldmakeYmorelikelytohappen.”
Consequently,thefactthatYdidnotoccurincountryiattimet,despitethefactthatXwas
veryhighisnot,atleasttomyear,particularlypuzzling.Unlikelyeventsareexpectedtohappen
occasionally.Consequently,onecannotreasonablycallaprobabilisticconjectureintoquestion
withasinglenullcase.Doingsoislikebeingpuzzledaboutone’sunclewholivedtoaripeold
agedespitebeingaheavysmoker.Thisisnotpuzzlingbecausethebestscientificevidenceis
thatsmokingincreasesthelikelihoodofcancer,notthatitalwaysleadstocancer.Incontrast,it
10
wouldbesurprisingtofindanentiresub-sampleofthepopulationthatappearstobeimmune
tothedeleteriouseffectsofsmoking,or,thataftercontrollingforincomeoreducation(orany
otherpotentialconfound),smokersarenotmorepronetocancerthannon-smokers.Insum,
sinceourtheoriestypicallyjustifyexpectationsaboutpatternsofdata,ittakesobservations
aboutpatternsofdata,notdiscreetdatapoints,toviolatethoseexpectations.
Whilerecognizingapatterninthedataisoftennecessaryforgeneratingsurprise,itisbyno
meanssufficient.GoingbacktothemanycomparativepoliticsseminarsIhaveattended,be
waryofthescholarwhoselectsasmallsampleofobservationsanddemonstratesthatawidely
corroboratedempiricalregularity,suchastheincumbencyadvantage,thedemocraticpeace,
Gamson’sLaw,Duverger’sLaw,ortheresourcecurse,“doesn’thold”inthatsub-sample.Why?
Becausesocialbehaviorisprobabilistic,soevenhighlypredictiveempiricalmodelsyield
predictionswithnon-zeroerrors.Asaresult,onecanalwaysfindasubsampleofdatawhere
thebroaderpatterndoesnothold.Takeany“footballshaped”scatterplot,suchasthefamous
scatterplotshowinFigure1.1.6Onecanselectoutasub-sampleofcases,suchasthoseinthe
ellipse,tosuggestthattheregressionlineisflatorevennegativeeventhoughthereisclearlya
positiverelationshipinthesampleonthewhole.
6ThescatterplotisbasedondatafromanexamplefromPearsonandLee’s(1903)examinationofthecorrelationbetweentheadultheightsoffathersandsons.
11
Figure1.1RelationshipBetweentheHeightofFathersandSons(DataSource:Freedman,Pisani,andPurves(2007)addedrandomnoisetodatafromPearsonandLee(1903)whoonlyhaddatatonearestinch,)http://myweb.uiowa.edu/pbreheny/data/pearson.html
12
RecallthatIsaid,“bewary”ofascholarwhomotivatestheirstudywithasub-sampleof
casesthatappeartoruncontrarytoawell-corroboratedsetofexpectations.ButIwouldnot
encourageyoutodismisssuchascholar.Itis,forexample,entirelyappropriatetoshowthat
thereareboundaryconditionsoneventhemostwell-corroboratedempiricalregularities.But
themereexistenceofsuchasub-sampledoesnotconstituteapuzzleuntilonecanconvince
thereaderthatthesub-sampleconstitutesacomprehensiblecategoryandisnotjusttheresult
offelicitous(fromthestandpointoftheauthorseekingsomethingtowriteabout)case
selection.Further,ifonedoestakeastheirprojectthetaskofexplainingwhyawell-
corroboratedregularitydoesnotapplytoaparticularsub-sample,itisincumbentuponthemto
developanexplanationforwhythesub-sampleisdifferentthatyieldspredictionsotherthan
thefactthatthesub-sampleisdifferent.Otherwise,theyareengagedinbothpost-hocandad-
hocreasoning.
Yetanotherproblemcanarisewhenonegeneratestheirresearchprojectbygazingata
scatterplot.ManywilllookatalikeFigure1.1afterestimatingaregressionlineandbe
disturbedthatsomanyobservationsfallfarfromtheregressionline.Itisokaytowantthe
modeltofitthedatawell,butgiventheprobabilistic,multi-causalnatureofourhypotheses,it
isnotpuzzlingthatsomeobservationsfallfarfromtheregressionline.Myfatherwassixfeet
tall,whileI,ahem,amnot.Thatisnotsurprisingbecauseotherfactorsenterintoheightat
adulthoodotherthanmygeneticinheritancefrommyfather–dietandcontributionsfrommy
mother’sgeneticmake-upcometomind.Beingpuzzledinthiswayisaslightlymore
sophisticatedversionofthe“ifXishighincountryiattimet,whydowenotobserveY”
problem.Bothmethodsarefrequentlyusedtojustifytheclaimthat“existingexplanationsare
13
incomplete.”Theproblemisthatanyexplanationtheauthorcomesupwithislikelytobe
susceptibletothesamecriticism.
Iwanttobeclear,thereisnothingwrongwithbeingunsatisfiedwithexplanationsthat
donotfitthedatawell.However,iftheonlyresultofpointingoutobservationsthatfalloffthe
regressionlineisanewmodelthatmarginallyincreasesmeasuresofgoodnessoffit,donotbe
surprisedifreadersfailtoseethisas“consequential.”Ceterisparibus,papersthatare
motivatedbytheidentificationofunclear,misleading,orincorrectunderstandingsinthe
existingliteraturearemoreconsequentialthanthosethatpointtomerely“incomplete”
understandingsbecausetheformercausesustorevise(thatisto“lookatagain”)ratherthan
merelysupplementourcurrentunderstanding.
Sofar,wehavebeenseekingtoidentifyviolationsofexpectationsthatare
consequentialforourunderstandingoftheworld,butonemightalsoplaceapriorityon
consequencesthataremorepractical.Onewayofaskingthe“sowhat”questionistoask,“if
youweresuccessfulinexplainingyouranomalousobservation,howwouldtheworldbe
different?”Unlessoneisentirelynaïve,thisisaverytoughquestiontoanswer.Butsincemost
ofusbecamepoliticalscientistsandinternationalrelationsscholarsbecausewewantedto
maketheworldabetterplace,itisstillworthwhile.Onereasontothinkaboutthe“normative”
implicationsofthequestionsweaskisthatanevenpassingfamiliaritywiththeliteraturein
politicalscienceandinternationalrelationsisenoughtounearthaseeminglyendlesssupplyof
unclear,misleading,orincorrectunderstandings.Inlightofthis,itisnotunreasonabletotryto
tacklefirstthosethataretiedtoissueswecaredeeplyabout.
14
NobellaureateRobertLucassaid“onceyoustartthinkingabouteconomicgrowth,itis
hardtothinkaboutanythingelse.”Isuspectthatisbecauseitisnothardtoseetherealworld,
sticktoyourribs,consequencesofeconomicgrowth.Likewise,immigration,politicalviolence,
economicinequality,governmentcorruption,racialandethnicdiscrimination,financial
instability,authoritarianism,genderbias,illiteracy,failingschools,orahostofotherpolicy
issuesareofinterestbecauseoftheirimpactonmattersofjusticeandhumanwell-being.
Explainingobservationsthatviolateourexpectationscanbequiteconsequentialwhendoingso
shedslightontheseandothersocialproblems.
Marx’slastandmostfamousthesesonFeuerbachisthat“thephilosophershaveonly
interpretedtheworld,invariousways.Thepoint,howeveristochangeit”anditisinteresting
thatitisetchedonhistombdespitehavingneverbeenpublishedwhilehewasalive.8It
capturesthefrustrationofmanyscholarswhowouldliketo“makeadifference.”Itcertainly
capturedmyromanticheartwhenIfirstreaditasayoungman(notmuchyoungerthanMarx
waswhenhewroteit)atthestartofgraduateschool.ButIwasnotingraduateschoollong
beforeIrealizedthecomplexityof“interpreting”theworldandthedangersthatcouldresultif
onesoughttochangetheworldwithouthavinginterpreteditcorrectly.Understandingthe
worldisaprerequisiteforchangingitinaresponsiblemanner.
Whileitisdesirable,perhapsevennoble,bridgingthegapbetweenstudyingtheworld
thewayitisandusingthisinformationtoimprovesocialconditionsisdifficult-particularly
whenpeople,and,therefore,politicsareinvolved.Oneproblemisthatifsocialillshave
politicalrootsevenaccurateexplanationsoftheircausesarelikelytobeinsufficientfor
8Marx(1888).
15
mitigatingthem.Onereasonforthisisthefactthatthehallmarkofpoliticsisconflictingvalues.
Explainingtoprisonersconfrontedwithpleadealsthatrewardthemforincriminatingeach
otherthattheycollectivelybenefitbykeepingmumwillnotsolvetheprisoner’sdilemma
becausetheywillstillhaveindividualincentivestoratontheirco-conspirators.9
So,whileunderstandingtheworldmaybeanecessaryconditionfor(responsibly)
changingit,itisnotlikelytobesufficient.And,conversely,changingtheworldcanmakeitalot
hardertounderstand.Oneofthethingsthatmakessocialsciencedifficultisthattheentities
westudycanreadwhatwewriteandchangetheirbehaviorinwaysthatmakeourmodelsless
predictivelyaccurate.10
SomethinglikethismayhavebeenatworkinthewritingsofMarx.Thephrase
“workersofalllands,unite!”alsoappearsonMarx’stomb.Incontrasttohistheseson
Feuerbach,thisphrasewaspublishedduringhislife-time.Threeyearsafterbemoaningthe
irrelevanceofpriorphilosophersMarxandEnglesclosedoneofthemostinfluentialpolitical
pamphletseverwrittenwithit.11Inan1890appendixtoTheCommunistManifestoEngels
admitsthatfewheededthecallin1848butsuggestsmanyeventuallydidsoovertime,
includingthosewhowereorganizinginsupportoftheeight-hourworkday1890.Itisnot
unreasonabletosuggestthatMarx’sanalysisofaninternallogictocapitalism(thatthe
9Incontrast,iftheonlyproblemisaco-ordinationproblemthenthemeredisseminationofinformationislikelytobesufficient.Butsuchproblemsareaboutaspoliticalasgettingdriverstostayontheirsideoftheroad.10Thoughsometimesthisworksintheoppositiedirection.Forexample,experimentshaveshownthatstudentswhotakeeconomicsclassesbehavemuchlesscooperatively,and,therefore,moreinlinewiththemodelslearnedinthesecourses.11TheCommunistManifesto(MarxandEngels,1996)hadlittleimmediateimpactonembryonicsocialistmovements,butitslongruninfluenceisundeniable.
16
inexorableimmiserationoftheproletariatwouldleadtorevolution)helpedfueltheformation
oflaborunionsandthecreationofsocialprogramsthatimprovedthematerialconditionsof
workers.Butindoingso,thismadethemlessrevolutionary–therebyreducingtheprobability
oftherevolutionhepredicted.
Anotherexampleofhowitishardtohavebothinfluenceintherealworldand
predictiveaccuracycomesfromtherecentliteratureon“thehappinesscurve”–therobust
empiricalregularitythatreportedlifesatisfactiontendstodeclinewhenpeopleareintheir
fortiesandriseconsistentlystartingintheirearlyfifties(Rauch2018).Oneexplanationforthis
empiricalregularityisthatbecausehumanpsychologyisbiasedtowardsoverlyoptimistic
forecasts,youngpeopleover-estimatehowmuchtheirliveswillimproveintheirthirtiesand
forties.Thisresultsindisappointmentduringtheirmiddleyearsevenifindividuals’liveshave
improvedconsiderably,butnotasmuchastheyexpected.Thisdisappointmentalsoleads
peopletoupdatetheirexpectationsandmakegrimforecastsforthefuture.Consequently,
whenlifeintheirfifties,sixties,andbeyondturnsouttobenotasbadasexpected,theyreport
highlevelsoflifesatisfaction.Ifthisprocessistrulyatwork,peoplewhoreadthisliterature
mightbeinclinedtomakemorerealisticpredictionsaboutfuturelifesatisfaction.Iftheydidso
inlargenumbers,the“happinesscurve”coulddisappear.
NoticethattotheextentthatMarxchangedhistoryitmayhavebeeninwaysthat
frustratedbothhispredictiveaccuracyandhissocialdesires(forrevolution)butifhappiness
researchersturnouttohavethesamedegreeofimpactonsocietytheymightbeperfectly
willingtotradepredictiveaccuracyfortangibleimprovementsinpeople’slifesatisfaction.
17
Insum,wewouldliketoanswerquestionsthat,whenanswered,wouldprove
consequential.Theseconsequencescanbeeitherforthewaywethinkabouttheworld,orthe
waypeoplebehave.While,allelseequal,wewouldlikeourresearchtoleadtoimprovements
inhumanwell-being,thestrategicnatureofpoliticsmeansthatevenwhenweprovidegood
answerstoquestionsthatareimportanttousitmaynotleaddirectlytoimprovementsinsocial
outcomes.Thatisnottosuggestweshouldstoptrying.
Question3:Whatistheexplanation?(Theory)
Agoodexplanationwilltakeanobservationthatissufficientlysurprisingthatitjustifies
yourstudy,andturnitintosomethingthat,inretrospect,shouldhavebeenexpectedallalong.
InwhatremainsoneofthefewbooksIknowofthatattemptstoteachpeoplehowtoexplain
things,theauthorsofAnIntroductiontoModelsintheSocialSciences(LaveandMarch,1975)
describeexplanationasaprocessinwhichoneimaginesapriorworldsuchthat,ifitexisted,
thesurprisingfact(s)wouldhavebeenexpected.Technically,anysetofstatementsthat
logicallyimplytheoccurrenceoftheanomalousobservationconstituteanexplanation.But
goodexplanationshaveadditionalattributes,andwewouldliketoproducethebest
explanation.Asatisfyingexplanationwillgivethereaderanunderstandingoftheprocessor
mechanismthatislikelytoproducethepreviouslyanomalousobservation.Readerswantto
knowhowsurprisingeventscameabout,andexplanationsshouldtellthem.Good
explanationsareefficient–theratioofthingstheyexplain(implications)tothingstheyrequire
youtobelieve(assumptions)ishigh.
18
Thereisanoptimaldegreeofnoveltytoanexplanation.Anexplanationshouldbe
interesting,yetsound.By“interesting”Imeanthatanexplanationshouldcauseustoseethe
worldinanewway.By“sound”Imeananexplanationshouldfitinwithotherthingsweknow
abouttheworld.Anexplanationthatcausesustoseeeverythinginanewwayislikelytobe
wrong.Anexplanationthatdoesnotrequireustochangeourmindatallisprobablyjusta
corollaryofthingswealreadyknew(and,byextension,ourmotivatingpuzzlemustnothave
beenmuchofapuzzle).
Finally,explanationsmustbelogicallyconsistent.Ihavehadempiricallyminded
politicalscientistsandinternationalrelationsscholarstellmethatformaltheoryisnot
importantbecausetheyaresophisticatedenoughtolivewiththeoriesthatcontain
contradictions.Thisisnonsense.Itcanbeshownwithelementarylogicthatanythingfollowsa
contradiction.Consequently,ifyourtheorycontainsacontradiction,anythingcanbesaidto
followfromit.Asaresult,acontradictorytheoryrulesnothingoutand,therefore,noamount
ofempiricalinformationwillbesufficienttofalsifyit.Sincepotentialfalsificationisthe
hallmarkofscience,atheorythatcontainsacontradiction,therefore,isnotascientific
theory.12
Onewaytoincreasethelikelihoodthatyourexplanationislogicallyconsistentistotry
tocaptureitwithaformalmodel.Formalmodelsallowustodemonstratethatour
explanation’sconclusionsfollowfromitsassumptions-mostimportantly,thatourpreviously
puzzlingobservationisnotsurprisinginlightoftheworldthatourexplanationposits.Also,by
12InthepossiblyapocryphalwordsoftheoreticalphysicistWolfgangPauli,itis“notevenwrong.”
19
makingtheassumptionsofourexplanationexplicitwearemorelikelytonoticeifthey
contradicteachother.
Whilethesebenefitsofformalizationareundeniable,itdoesnotfollowthatevery
explanationshouldbeformalized.Itypicallyencouragemystudentstofirstarticulatetheir
explanationsasastorythatrevealsaprocessthatproducesthepreviouslyunexpected
observation.Formalizationisonlynecessarywhenonehearssuchastoryandasks,“whywould
peopledothat?”or,equivalently,“Thatdoesn’tsoundlikeanequilibrium,”or,“isn’ttherea
tensionbetweenthispartofthestoryandthatpartofthestory?”Whenoneisconfronted
withsuchquestions,agoodformalmodelcanoftenprovideanswers.Thus,Itellmystudentsto
learnhowtowritedownformalmodelsnotbecausetheywillalwaysneedone,butlikefire
insurance,theyarealwaysatriskofneedingone.
Anotherreasontobeginwithaninformalstatementofone’stheoryistoavoidthetrap
ofthinkingthatagametheoreticmodelwillgenerateatheoryforus.Formalmodelshelpus
interrogatecertainaspectsofourtheory,theydonotproducethetheoryforus.Wemust
beginwithsometheoreticalintuitionaboutwhatexplainsthephenomenoninquestionbefore
wecanbegintomodeltheprocess.
Question4:Iftheexplanationistrue,whatelseshouldweobserve?(ResearchDesign)
Ifyouofferaviewofatheoreticalworldthathasthepreviouslypuzzlingobservationas
oneofitsimplications,youhaveofferedanexplanation.Andwhiletherearevariouswaysto
evaluatethatexplanation,tobescientific,youranswertoyouroriginalquestionmustprovide
ananswertothefollowingquestion:“ifyourexplanationiscorrect,whatelseoughttobe
20
true?”Goodscientificexplanationsprovidelotsofanswerstothisquestion.Ifyour
explanationonlyimpliesthefactsthatyousetouttoexplain,thenthereisnowayto
empiricallyevaluateyouranswer.Youcannotusethefactthatdemocraciesseldomfighteach
other,orthefactthatthereisalotofcorruptioninpresidentialdemocraciestoevaluateyour
explanationofthesethingsbecauseitwasthosefactsthatledyoutodevelopyourexplanation
inthefirstplace.
Thispartoftheresearchprocessisastumblingblockformanyresearcherswhenthey
areattractedtoasubject,ratherthanaquestion.IoncehadastudentwhovisitedBraziland
wasshockedbythelevelofcorruptioninthegovernmentthereanddevelopedanexplanation
thatpointedtoaspectsofthelargedistrictmagnitudeproportionalrepresentationelectoral
systemasacause.ThestudentwassurprisedwhenIsaidIthoughttheargumenthadmerits,
butthatreturningtoBraziltocollectdatawasnotapromisingavenueforevaluatingthe
argument:wealreadynewthatBrazilfittheargument!Perhapsdataoncorruptionlevelsin
countrieswithdifferentelectorallaws(suchastheUnitedStates)wouldbemoreuseful.The
student,however,respondedthathedidnotwanttostudycorruptioninothercountries,after
allhewasinterestedinBrazil!
AsimilarproblemisfoundinaveryfamousbookbyThedaSkocpol,StatesandSocial
Revolutions(1978).Init,theauthorwishestoexplaintheoccurrenceofsocialrevolutionsand
shearguedthathersubjectdictatedherempiricalstrategy.Givenherdefinition,thereareonly
fivehistoricalcasesofsocialrevolution.Shearguedthatasaconsequenceofthisfact,
structuredfocusedcomparison(specifically,Mill’sMethodofAgreement)wastheonlypossible
methodforevaluatingherexplanation.Thatisnottrue.
21
Thechiefproblemhereisthatifanexplanationforasetofrareeventsonlyhas
implicationsaboutthoserareevents,theauthordoesnothaveadataproblem,theyhavea
theoryproblem.Ifanexplanationforglobalwarmingonlypredictsthegeneralriseinthe
temperaturethatmotivatedtheexplanation,thenitisnotaveryusefulexplanation.
Cosmologistshaveofferedexplanationsforthecreationoftheuniverse,buttheydonotchoose
theirmethodologyforevaluatingtheirexplanationsbasedonthefactthattheobjectoftheir
studyonlyhappedonce.Instead,theyask,“ifmyexplanationforthisuniqueeventiscorrect,
whatelseoughttobetrue?”Theythenthinkabouthowbesttocarefullyobservethe
implicationsoftheirargument.
Thegoalofempiricalresearch,therefore,shouldbetoexamineasmanyimplicationsof
one’sexplanationaspossible.Becausemany,manyscholarsrestricttheirattentiontothe
empiricalpuzzlethatmotivatedtheirstudytobeginwith,manyimportantpaperscanbe
writtenbysimplyaskingofexistingexplanations,“ifthisargumentistrue,whatelseoughtwe
observe?”
Onereasonwhyscholarsoftenrestricttheirattentiontothedatathatgeneratedthe
questionisthatitcanoftentakeconsiderablecreativitytothinkabouttheimplicationsofan
explanation.Thereisnocookbook-likeapproachthatcanbeappliedthatwillautomatically
revealtothescholarthatseeminglyunrelatedeventsmightbeinstantiationsofasinglesocial
process.ButonepracticeLaveandMarchrecommendistotrytoseeyouranswertoa
particularquestionasrelatedtoamoregeneralprocess.
Forexample,inhercriticalreviewofSkocpol’sbook,BarbaraGeddes(2003)suggests
thatoneelementofSkocopol’sexplanationofraresocialrevolutionshadimplicationsforthe
22
occurrenceofpeasantrevolts.Geddessuggestthatastatisticalmodelexaminingtheconditions
underwhichpeasantrevoltsdoanddonotoccurwould,therefore,beusefulinevaluatingthe
empiricalrelevanceofSkocpol’sexplanationofsocialrevolutions.
Notice,thatwhenweask“whatelseoughttobetrue”weseparatethequestionof
“whatistheauthorsexplanandum?”from“whatistheauthor’s“dependentvariable?”The
explanandumisastatementofwhattheauthordevelopsatheorytoexplain.The“dependent
variable”istheendogenousvariableinamodeltestingoneormoreoftheimplicationsofthe
author’stheory.Therearetimeswhenthesemightbethesame,butthereisnoreasonto
assumetheywillbe.Infact,whentheyare,weshouldwonderiftheauthorisengagedinpost-
hocreasoning–“havetheyobservedthedependentvariableanditscovariatesandconstructed
acausalstoryafterthefact?”Doingsowouldconstitutea“test”ofthetheoryonlytothe
extentthatthelion’sshareoftheobservationscouldbethoughttohavebeenappreciably
differentfromthosethatwereobservedbeforethetheory’sformulation.Conversely,atheory
thatproducesalotofnovelimplicationshelpsassuagethereader’ssuspicionthattheauthoris
merelyengagedinacurve-fittingexercise.
Insum,itistypicallymorehelpfultothinkofempiricalworkastestingtheimplications
ofatheory,ratherthantestingthetheorydirectly.Onereasonthisistrueisthattestingthe
theorydirectlycaneasilydescendintomoreorlesscomplicatedversionofcurve-fittingand
post-hocreasoning.Instead,spendtimethinkingabouttheimplicationsofyourexplanationfor
observationsotherthanthosethatmotivatedyourquestioninthefirstplace.Themorevaried
thoseimplicationsthebetter,becauseitisonlythoseobservationsthataremadeafterthe
constructionofyourtheorythatruntheriskofbeingfalseandthereforeactuallyconstitutean
23
empiricalcheckonyourexplanation.Andremember:ifyourtheoryonlyhasimplicationsfora
setofeventstoosmalltousestandardinferentialtoolstoevaluate,youdonothaveadata
problem-youhaveatheoryproblem.
Question5:Doweobservetheimplicationsofourexplanation?(Findings)
Determiningifevidenceisconsistentwithone’stheoreticalexpectationsistheprimary
focusofresearchmethodologyand,so,isthecentralfocusoftheremainderofthisvolume.
HereIwillmerelystressthefollowing:many,manystudiespresent,oftenindizzyingdetail,
reamsofinformationthatiseitherirrelevanttoorinconsistentwiththeoreticalexpectations.
Typically,however,itispresentedinamannerthatsuggeststhatthisinformationconfirmsthe
author’sexpectations.Distinguishingwhenthisisthecaseisalargepartofwhatismeantby
learningtoreadcritically.
AsIsaid,allofthecollectivewisdomofresearchmethodologistsisrelevantfor
becomingacriticalreaderandproducerofknowledgebutIwillfocusononeadmonition:
presentclearestimatesofthequantitiesofinterestaswellasastatementaboutthedegreeof
confidenceonehasinthoseestimates.13Thereareafewwaysinwhichthisadmonitionis
frequentlyviolated,andIwouldliketobrieflydrawyourattentiontothem.
AtleastinthesocialscientificpapersIread,explanationstypicallyproduceclaimsabout
theassociationbetweenvariables.Evenwhenoneisengagedinwhatlookslikeadescriptive
exercise,likeHuntingon’sattempttodemonstraterisingpoliticalinstability,oneisengagedin
demonstratingthatvaraiblesarerelatedtoeachotherinaparticularway.Ifonewantsto
13King,Keohane,andVerba(1994).
24
demonstratethataphenomenonischangingovertime,onemustlookattherelationship
betweenthatvariableandtime.Ifonewantstodemonstratethataparticularbehavioror
attitudeismoreprevalentinsomeplacesoramongsomegroups,onemustlookatthe
relationshipbetweenthatvariableandgroupmembershiporspatiallocation.Consequently,
mostofourempiricalclaimsareabouttherelationshipbetweenvariables.Inalinearmodel
wethinkofthisquantityofinterestasaslopecoefficient,soIwillusethatterminologyhere,
thoughtheterm“derivative”mightbeevenmoreappropriate.
Acommonwayinwhichscholarsbecomedistractedfrompresentingthequantityof
interestisbypresentingsomethingotherthananestimateofaslope,whenthatisthequantity
theyareconcernedwith.Forexample,ithasbecomecommonforscholarstoplotthe
predictedprobabilitiesfromalogitmodelonthey-axiswithsomevariableofinterestonthex-
axiswhenthequantitiyofinterestistheassociationbetweenachangeinthatpredicted
probabilityandameaningfulchangeinsomevariableofinterest.Theproblemwithdoingsois
thatitrequiresthereadertoinfertheslopeofthatrelationshipfromthepicture.Whileitis
truethatslopesarenotconstantinnon-linearmodelssuchaslogit,and,thereforethequantity
ofinterestdoesnotreducetoasinglenumber,itwouldbebettertoplotthemarginaleffectof
thevariableofinterestacrossameaningfulsetofvaluesofthatvariableofinterest.14Adding
confidenceintervalsaroundthepredictedprobabilitydoesnothelpbecausethattellsthe
readerifthepredictedprobabilityissignificantlydifferentfromzero,whichistypicallynotthe
hypothesisbeingtested.
14Inthelanguageofcalculus:ifthequantityofinterestisdy/dx,thenplotdy/dxagainstx,notyagainstx.Theformertellsthereaderwhattheyneedtoknow.Thelattermakesthereadertrytoinferwhattheyneedtoknowfromthepicture.
25
Forexample,Hellwig,Ringsmuth,andFreeman(2008)presentthegraphsinFigure1.2
asevidenceinthatthepropensityforcitizenstobelievegovernmentshavelittleroomto
maneuverpolicyinaglobalizedeconomy.Eachpanelplotsthepredictedprobability(and90%
confidenceintervals)thatasurveyrespondentsaidtheydidnotbelievetheU.S.government
retainsthe“roomtomaneuver”policyagainsttherespondent’spartisanship.Theauthors
interprettheapparentdifferencebetweentheslopeoftheplotsinthelefthandpanelfromthe
righthandpanelasevidencethatpartisanshiphasaneffectonrespondentbeliefsamong
respondentswithCollegeDegrees(panela)butnotwithHighSchoolDegreesorLess(panelb)
andamongrespondentsabovetheageoffifty-nine(panelc)butnotbelowtheageofforty
(paneld).Butwhatisthebasisofthisconclusion?Theslopesontherightclearlylooktobe
closetozeroand,incomparison,theslopesontheleftappeartobepositive.Butweare
offeredneitheranestimateoftheslopesforanydegreeofpartisanship,noranestimateofour
uncertaintyaboutthatestimate.Wecantrytocalculatetheslopeatdifferentpointsontheline
byestimatingthe“riseoverrun”andwecankindofcomparethatestimatewiththe
uncertaintyimpliedbytheerrorbars,butwhymakethereaderconstructat-testfromthe
pictureratherthanpresentthatinformationforthereaderbyplottingmarginaleffectswith
theirassociatedconfidenceintervals?Neitherdotheauthorsprovideanyevidencewhether
theslopesintheleft-handpanelsaredifferentfromtheslopesintherighthandpanels.Asa
consequence,thesepictures,andoneslikethemthatappearfrequentlyintheliterature,
providealmostnoquantitativeevidenceaboutthequantityofinterest(underwhatconditions,
ifany,achangeinpartisanshipassociatedwithachangeincitizenbeliefsaboutthe
government’s“roomtomaneuver”).
26
Figure1.2PartisanshipandBeliefsabout‘RoomtoManeuver:TheConditionaleffects
ofKnowledgeandAge.SourceHellwig,Ringsmuth,andFreeman(2008,Figure2,p.875.)
27
Anothercommonwayofobscuringthequantityofinterestisinpresenting“marginal
effects”thatarenotmarginal.Itiscommonplaceforauthorstosaythingslike“togainsome
substantiveunderstandingoftheseresults,InotethataonestandarddeviationchangeinXis
associatedwitha0.056changeinY.”Theproblemwiththisisthatthereisnothingtypicalor
representativeaboutastandarddeviation–indataapproximatinganormaldistributionabout
two-thirdsofallobservationswillbelessthanastandarddeviationawayfromthemean.Asa
consequence,achangeofastandarddeviationinthevariableofinterestisnotaparticularly
meaningfulcounterfactualtoconsider.Thisisparticularlytruewherethispracticeismost
frequentlyfound–wheninterpretingtheresultsofanon-linearmodel.Underthis
circumstance,themarginaleffectofavariableisextremelysensitivetowhereitisbeing
evaluated.Theslopedescribedbya“marginaleffect”thesizeofastandarddeviationislikely
tobeveryfarfromtheslopeofanyestimatedmarginaleffectwithinthisinterval.Another
reasonwhythisisnotaparticularlyusefulcounterfactualcomparisonisthatmarginaleffects
areinterpretedunderaceterisparibusclausewhereotherfactorsareheldconstant–
somethingwhichisnotlikelytobeapproximatedintherealworldwhenthevariableofinterest
experiencesanunusuallylargechangethesizeofastandarddeviation.15
Anothercommonwayscholarspresentinformationthatisnotthequantityofinterestis
whentheyhaveahypothesisthatisconditionalinnatureandeitherpresentresultsfroman
unconditionalmodel,or,equallycommon,estimateaconditionalmodelbutgoontointerpret
someofitsresultsasiftheywereunconditional.16
15SeeKingandZeng(2006)on“TheDangersofExtremeCounterfactuals.”16SeeBrambor,ClarkandGolder(2006)orKamandFranzese(2007)forafullerdiscussion.
28
Summary
Myclaim,uptothispoint,isthatapaper,book,ordissertationthathasgoodanswers
tothefivequestionsabovewillbeausefulpaper,book,ordissertation.Itdoesnotfollowthat
apaper,book,ordissertationmusthaveaninnovativeanswertoallfiveofthosequestions.
Progresscanbemadeaslongasoneoftheanswersisbetterthanexistinganswersandnone
areworse.
Whichquestionsare“mostimportant”and,therefore,whichonesshouldbethefocus
ofyoureffortstoinnovate?Itishardtosay.ThoughIbelievethatitisprobablynotbesttotry
toexplainsomethingthatnoonehasexplainedbefore.Thisisanimportantpoint.Ihavehad
manygraduatestudentsinformmegloomilythatsomeonehasbeatenthemtotheir
“question.”Mystandardreactionistosay,“well,Idoubttheyhavecomeupwiththedefinitive
answer,sowhatareyouworriedabout?”Sinceanyquestionworthaskingislikelytobe
difficulttoanswer,itishighlyunlikelythatanotherscholarislikelyto“beatyoutothepunch”
andstatethe“lastword”onasubject.Indeed,ifyouareaskingaquestionthatnooneelse
hasasked,itshouldgiveyoupause.Maybeitisnotaveryinterestingquestion:ormaybethere
issomethingaboutaskingthequestioninthatwaythatledotherscholarstobelieveproductive
answerswerenotforthcoming.Thatsaid,themerefactthatothersmartpeoplehaveasked
thequestiondoesnotmeanitisagreatideaforyoutotrytoanswerit.
Graduatestudentsaretoldthattheyneedtomakeanoriginalcontributionwhichleads
themtobelievethattheymustaskaquestionthathasneverbeenasked,oratleast,never
beenansweredbefore.Thatisnottrue.Rather,an“originalcontribution”requiresonlythat
thestudentprovideabetteranswertoatleastoneofthequestionsmentionedabove.So,ifa
29
studentattheprospectusstageisgoingtoattempttoofferanovelexplanation,thenpartof
theiranswertoquestion2shouldcontainastatementaboutwhattheybringtothetablethat
mightallowthemtomakeprogresswhereothershavefailed.Whattheoreticalinsight,
methodologicaladvantage,orhistoricalknowledgeputstheauthorinapositionto
simultaneouslyrecognizethat“thingshavegonewrong”withexistingexplanationsandoffera
solutionthatpushesthefieldinapromisingdirection?
Since“theoreticalinnovation”isoftenthoughttobethemostprizedcontributiona
politicalscientistcanmake,scholarsoftenbelievethatagoodpapershouldofferanovel
explanation.Ibelievethiscomes,inpart,fromphysicsenvycombinedwiththenotionthat
theoreticalphysicistshaveahigherstatusthanexperimentalists.Ibelievetheideathatevery
importantcontributionmustcontainatheoreticalinnovationhasgreatlyhamperedthe
progressofourdiscipline.Howistheaccumulationofknowledgepossibleifeverytimea
scholarputspentopapertheyhavetoofferanewexplanation?Givenfrequentlyimperfect
researchdesignsandflawedempiricalmethods,Ioftenthinktheoppositeistrue.Wemightbe
temptedtodeclareamoratoriumonthedevelopmentofnewexplanationsuntilthediscipline
hasreachedconsensusaboutempiricaltestsoftheimplicationsofexistingexplanations.Asmy
critiqueofHuntingtonsuggests,ifwedonotgetatleastsomeoftheempiricsright,howdowe
evenknowifourobservationsviolatecurrenttheoreticalexpectationsenoughtowarrantnew
explanations?Onereasontoresistsuchatemptationisthatnewtheoriesdomorethan
explainanomalies.Foroneexample,theyalsoaddressconceptualandlogicalproblemswith
existingexplanations.
30
Practicesthatencouragegoodquestionasking
FollowingKuhn’slineofreasoningabove,itisworthaskingwhatislikelytopromote
“theskill,wit,andgenius”capableofrecognizingwhenthingshave“gonewronginwaysthat
mayproveconsequential.”OfKuhn’sthreedesiderata,“skill”seemstheleastconstrainedby
naturalabilityand,therefore,mostresponsivetotheenvironmentswecreate.Whileartistic
creationinvolvesmanyaspects,adegreeofcraftsmanshipistypicallyinvolvedand
craftsmanshipisderivedlargelyfrompractice.Extensivetrainingingametheoryandstatistics
isnowcommonplaceinmostgraduate(andsomeundergraduate)programsinpoliticalscience
andinternationalrelationsandthisiswhatistypicallythoughtofwhenscholarsevaluatethe
“skills”ofjobapplicants.Theseskillsareimportantbecausewithoutthem,scholarsmightask
questionsbasedonfaultyreasoningbasedonformalorinformalfallaciessuchastheecological
fallacy,adhominemattacks,hastygeneralization,confusingcorrelationwithcausation,
ignoringstrategyinducedselectioneffects,andfailingtorecognizethepresenceofconfounds.
Butwhilemethodstrainingisextremelyhelpful,itisnotsufficienttoproducescholars
whoaskandanswerinterestingquestions.Theproblemsetstypicallyassignedinquantitative
methodsandformaltheoryclassesdohelpbuildtheskillsnecessarytoexecutesophisticated
research.Justasplayingscalesandarpeggiosbuildthetechniquesnecessarytoexecute
sophisticatedmusic.Butthereismoretotrainingamusicianthanplayingscalesandarpeggios
becauseasimportantasscalesandarpeggiosare,theyarenotmusic.Ihaveheardmusicians
criticizedforhavingsufficienttechniquethatthey“knowhowtosaythingsontheir
instruments,buttheydonotseemtohaveanythingtosay.”Theanalogouscriticismis
frequentlyleveledatnewlytrainedpoliticalscientistsandinternationalrelationsscholars.
31
So,whatistobedone?Toplaygoodmusic,studentshavetolistentogoodmusicand
theyhavetohavealotofexperiencemakinggoodmusic.Mostgraduateprogramsprovide
studentswiththeequivalentoflisteningtomusic.WhenIwasanewlymintedPh.D.Iheard
BruceBuenodeMesquitagivealectureattheHooverSummerPrograminGameTheoryand
InternationalPoliticsatStanfordUniversity.Hebuiltagametheoreticmodelbasedonthe
assumptionsofhegemonicstabilitytheory–seeminglyontheflybasedoncommentsshouted
outbymyclassmates.Ihadanepiphany.Ofcourse,ifdevelopingsocialscientific
explanationsisanart,thenitmustbetaughtastheartsaretaught!Iwaswatchingthemaster
attheeasel–engagedintheverycraftIwastryingtolearn.Itsuddenlyoccurredtomethat
muchofmygraduatetrainingamountedtotheequivalentofsittinginaroomlisteningto
recordingsofmusic,thenwhenitwastimetowritemydissertationitwasasifadoorhadbeen
flungopen,IwashandedaninstrumentIhadneverplayed(Iimaginedacello)andpushedout
ontoastagewhereIwasexpectedtoperform.Mostgraduateprogramsinpoliticalscience
teachpeopletheequivalentofplayingscalesinmethodsclassesandmusichistoryor
appreciationinsubstantiveclassesandarelefttofigureoutontheirownhowtoputthis
togethertomakemusic.
Themissingpieceinmostofourgraduateeducationiswhatmusicianscall“etudes.”
Theseareexercisesdesignedtobemusic-like(sostudentscanbegintothinkabout
interpretationandexpression)butareartificiallydesignedtoallowforadegreeofrepetitionof
particulartechniques(articulation,vibrato,dexterity)thatallowsthoseskillsnecessaryfor
musicalexpressiontoseepintothestudent’smusclememory.Manydoctoralprograms
emphasizethatstudentsshouldwritepublishablepapers,butIbelievethatsuccessisunlikelyif
32
thisisattemptedbeforestudentshaveengagedinmanyrepeatedattemptstoexplainthingsor
thinkaboutwhatobservationsareimpliedbytheirexplanations.Studentsneedtopractice
askingandansweringthefivequestionsoutlinedaboveandwritingasinglepaperineach
seminardoesnotgivethemthe“reps”todevelopmusclememory.Virtuallynoskillworthyof
thenamecanbedevelopedafteradozenorsoattempts.
Consequently,Ihavearguedthatproblemsetsin“substantiveclasses”canhelp
studentsbecomeproficientataskingandansweringthequestionsthatwillmakeforinnovative
research.Ananalogytothevisualartsmightbeuseful.Whenstudentsarelearningtodraw,
theyarenothandedablanksheetofpaperandtoldto“thinkofsomethinginterestingtodraw,
thatnooneelsehasdrawn.”Rather,abowloffruit,orperhapsawoodenmodelofahuman
figureisplacedonatable.Then,everyoneintheclassdrawsthesamethingafterreceiving
instructionfromtheinstructorabouthowtodoso.Incontrast,manypoliticalscience
departmentsdotheequivalentofhandingtheirstudentsablanksheetofpaperandaretoldto
“drawsomethinginteresting.”Problemsetsinsubstantiveclassescanbetheequivalentofa
bowloffruit.Theinstructorcanassignstudentstoaquestionrelatedtoaparticularresearch
area.“ExplainwhyXoccursunderZcircumstances.”“IfPexplainsY,whatelseoughtwe
observe?”“WhyisQaninterestingquestion?”“DoesFigure2countasconfirmingor
disconfirmingevidenceforhypothesis2,andwhy?”
Studentsneedalotofexperience“makingmusic”beforethey“havesomethingtosay.”
Iftheanalogytotheartsdoesnotresonatewithyou,considerthefollowing.Politicalscience
andinternationalrelationscantakealessonfromtheso-called“benchsciences”where
studentsworkonmanyprojectsasmembersoflargeteamsbeforetheyaretaskedwiththe
33
responsibilityofdecidingonthetopicofthegroup’snextproject.Experienceandrepetition
helpsstudentslearnwhatworksandwhatdoesnot.
Whilegraduatepedagogyisimportantforstimulatingcreativequestionaskingand
answering,thebroaderclimateandculturewecreateinourdepartmentsandresearchcenters
isequallyimportant.Inparticular,itisextremelyimportanttocreateanenvironmentwhereit
issafetoplaywithideasandchallengeorthodoxy.Ioncehadacolleaguewho,whilewalking
downthehallreadapassagefromabookthathethoughtwasincorrectandloudlydeclared
theauthoran“idiot.”Creativityandrisktakingisnotencouragedbyaculturethatsuggests
thatonlystupidpeoplesaystupidthings.Instead,itisimportanttocreatetheideathatthe
smartestamongusarecapableoferrorandthatthereisabigdifferencebetweensaying
somethingthatstupidandbeingstupid.Tothatend,Ithinkitisextremelyimportantforsenior
scholarstobetransparentabouttheerrorstheyhavemade.Youngscholarsneedtolearnthat
ifthey’vemadeamistake,theyareinverygoodcompanyandiftherequirementforadmission
wasnevermakingamistakethebuildingwouldbeempty.
Whileacultureofsupportforindividualrisktakingisvitaltoanyscientificorartistic
community,thereisanoptimaldegreeofindividualismbehindscientificdiscovery.Ifyoudon’t
readwhateverybodyelsereadsandfailtotrainlikeeveryoneelsetrains,youwillasknaïve
questionsthattherestofyourcommunityknowstheanswersto.Butifyouonlyreadwhat
everyreads,andonlytrainlikeeveryoneelsetrains,youareunlikelytoexperiencethat
momentwhenyouseesomethingthathasgonewrongthatnooneelsesees.
JazzbassistScottLaFarostartedplayingthebassin1954whenhewas19yearsoldand
inthefewshortyearsbeforehewaskilledinatragiccaraccidentin1961,hecompletely
34
changedtheworld’sconceptionofwhatcouldbeaccomplishedonadoublebassandwhatrole
theinstrumentcouldplayinapianotrio.Priortoplayingthebasshehadplayedtheclarinet
andsaxophoneforyearsandmanyhaveattributedhisphenomenaltechnicalprowesstothe
factthathepracticedthebassbyplayingetudescomposedfortheclarinetbyHyacintheKlosĕ
inthe19thcentury(LaFaro-Fernandez,2009).ThelessonLaFarotaughttheworld,inadditionto
thegeneralbenefitsofinter-disciplinarily,was“ifyouwanttosoundlikeeveryoneelse,
practicelikeeveryoneelse;butifyouwanttosoundlikeno-oneelse,practicelikenoone
else.”17
Justasthereisanoptimaldegreeofindividualitythatislikelytoproducescholarswith
theskill,wit,andgeniustodeterminewhensomethinghasgonewronginwaysthatmayprove
consequential,communitiesthatstriketherightbalancebetweenconformityanddiversityare
likelytoencouragethehabitsthatleadtoscientificbreakthrough.
Ontheonehand,itisimportantforascientificcommunitytoshareacommitmentto
thegrowthanddisseminationofknowledgeandacommonunderstandingofthelogicof
inferenceandthestandardsofevidence.Withoutthissharedunderstanding,criticismislikely
tofallondeafears.Butontheotherhanditisimportantforacommunitytobeasdiverseand
eclecticaspossible.Peoplefromdifferentcultural,class,linguistic,andreligiousbackgrounds
arelikelytoseethesocialworlddifferentlybecausetheyarelikelytohavehaddifferent
experiences.Thesedifferentexperiencesarelikelytoleadtodiversemoral,political,andsocial
intuitionsthatleadthemtoraisequestionsthatamorehomogeneousgroupmightnot(Page,
2007).17Atthesametime,nearlyeveryinnovativejazzmusicianlearnedtheircraftbymemorizingperformancesofmusiciansthatcamebeforethem.
35
Inaddition,diversegroupsarelesslikelytofallpreytowhatIcall“strategic
confirmationbias.”Confirmationbiasoccurswhenanindividualembracesanideauncritically
becauseitconformstotheirpriorbeliefs.Whenconfirmationbiasisatwork,peopleareless
likelytoscrutinizetheresearchpracticesthatproducedtheclaiminquestion.Theyareless
likelytolookforconfounds,toaskaboutthedetailsofdatacollection,ortothinkcritically
abouteitherthemicro-foundationsormoralimplicationsofaclaimbecausetheresultsconfirm
whattheyhavelongsuspectedabouttheworld.
Butstrategicconfirmationbiasoccurswhenanindividualisabletoovercomefirst-order
confirmationbiasandthinkcriticallyabouttheclaimbeingmade,butisdeterredfromvoicing
thecriticismbecausetheybelieveothersarerefrainingfromcriticismasaresultof
confirmationbias.Undersuchcircumstances,criticallyengagingtheclaiminpublicmightsignal
toothersthatthecriticdoesnotsharetheirbeliefsonthematter.
Strategicconfirmationbiasismostlikelytobeaproblemincommunitieswhere
“everybody”sharesparticularbeliefs.Insuchanenvironment,thinkingcriticallyaboutaresult
thatconfirmsthecommunity’sbeliefscouldresultinostracism,orattheveryleast,fewer
dinnerinvitations.Acommunitycomprisedofindividualsfromdiverseeducational,class,
religious,andideologicalbackgroundsislesslikelytoproducethekindofmonolithicviewsthat
encouragestrategicconfirmationbias.Individualsaremorelikelytosaysomethingwhenthey
seesomethingwrongthatmayproveconsequentialbecausethesetoftakenforgranted
sharedbeliefsislikelytobesmaller.Diversityismostlikelytobehelpfulinthisregardwhen
themultipledimensionsofidentityarerelativelyuncorrelated.Ifgender,race,orideologyare
heavilycorrelated,thendissentononedimensioncanbeseenasdefectiononanother.Thus,
36
inidealcircumstancescommunitieswouldhaveasmuchwithingroupdiversityasbetween
groupdiversity.18Ofcourse,diversityhastobesufficientlydevelopedtogiveindividuals
confidencethatspeakingupundersuchcircumstanceswillnotsimplyconfirmthatoneisan
“outsider.”Ifacommunitypromulgatesthenormthatinamultidimensionalspaceweareall,
ononedimensionoranother,outsiders,thecostofrevealingthatone“thinksdifferently”
aboutsomethingislikelytobelesscostly.Thedauntingthingaboutstrategicconfirmationbias
isthatitismostlylikelytooccuraroundissuesscholarsfeelpassionateabout.Asaresult,
thereisadangerthataresearchcommunitywillbeleastscientificaboutthemattersthatit
caresmostdeeplyaboutandmostscientificaboutmattersitsparticipantsviewaslargely
inconsequential.
CONCLUSION
Goodscientistsaskinterestingquestionsandareunsatisfied,evenimpatient,withbad
answers.Ihavearguedthatmostworkinpoliticalscienceandinternationalrelationscanbe
understoodthroughthelensoffivequestionsandthatcontributionscanbemadetothe
literaturebyimprovingonaresearchcommunity’sanswertoanyofthefivequestions.
Sincecomingupwithbetteranswerstoquestionsisasmuchart,asitisscience,Ihave
arguedthatthebestwaytotraingoodsocialscientistsistolearnfromthewayartistsare
trained.Musicalandvisualartistslearntheircraftsthroughstructuredrepetitivepractice.
Theimplicationofthisinsightforthesocialsciencesisthatscholarsshouldbegivenmaterials
toworkwiththatallowthemtoengageinthedailypracticeofaskingandansweringthefive18Theconnectionbetween“intersectionality”andcross-cuttingcleavagesshouldbeexploredfurther.
37
questionsoutlinedinthefirstsectionofthepaper.Ihavesuggestedthatthebestwayto
encouragethisisthroughtheuseofproblemsetsinoursubstantivecourses.Ihavealsohinted
thattherearegreatbenefitstointer-disciplinarity.Bybringhabits,techniques,andinsights
thatarenormalinonedisciplinetoasettingwheretheyarerare,individualsaremorelikelyto
recognizewhensomethinghas“gonewronginwaysthatmayproveconsequential.”Finally,I
havearguedthatdiversecommunitiesaremorelikelytoproducegoodquestionaskers,inpart
becausetheyarelesslikelytofallpreytostrategicconfirmationbias.
38
References
Brambor, Thomas, William Roberts Clark and Matt Golder. 2006. “Understanding Interaction Models:
Improving Empirical Analysis” Political Analysis. 14(1): 63-82.
Cindy D. Kam and Franzese, Robert J. Jr. 2007. Modeling and Interpreting Interactive Hypotheses in
Regression Analysis. (Ann Arbor: University of Michigan Press).
Friedman, David, Robert Pisani and Roger Purves. 2007. (New York: W.W. Norton).
Geddes, Barbara. 2003. Paradigms and Sand Castles: Theory Building and Research Design in
Comparative Politics.
Hellwig, Timoth T, Eve M. Ringsmuth, and John R. Freeman. 2008. “The America Public and the Room
to Maneuver: Responsibility Attributions and Policy Efficacy in an Era of Globalization.” International
Studies Quarterly 52(4).
Huntington, Samuel. 1968. Political Order in Changing Societies. (New Haven: Yale University Press).
King, Gary, Robert O. Keohane, and Sidney Verba. 1994. (Princeton: Princeton University Press).
King, Gary and Langche Zeng. 2005. “The Dangers of Extreme Counterfactuals,” Political Analysis
14:131-159.
Kuhn, Thomas S. 1962. “Historical Structure of Scientific Discovery,” Science 136(3518):760-764.
Lafaro-Fernandez, Helene. 2009. Jade Visions: The Life and Music of Scott LaFaro. (Denton, Tx:
University of North Texas Press)
Lave, Charles A. and James G. March. 1975. (New York: Harper and Row).
Lieberson. Stanley. 1991. “Small N’s and Big Conclusions: An Examination of the Reasoning in
Comparative Studies Based on a Small Number of Cases.” Social Forces 70: 307-320.
39
Marx, Karl. 1888. “Theses on Feuerbach” in Friedrich Engels, editor, Ludwig Feuerbach and the End
of Classical German Philosophy.
Marx, Karl and Friedrich Engels. 1996. The Communist Manifesto. (London: Pluto Press).
Page, Scott. E. 2007. The Difference: How the Power of Diversity Creates Better Groups, Firms, Schools,
and Societies. (Princeton: Princeton University Press).
Pearson, K. and Lee, A. (1903). On the laws of inheritance in man: I. Inheritance of physical characters. Biometika, 2(4), 357-462.
Popper, Sir Karl. 2003 [1959]. The Logic of Scientific Discovery. New York: Routledge.
Popper, Sir Karl. 1962. Conjectures and Refutations: The Growth of Scientific Knowledge. New York:
Basic Books.
Rauch, Jonathan. 2018. The Happiness Curve: Why Life Gets Better After 50. (New York: St. Martin’s
Press).
Skocpol, Theda. 1979. States and Social Revolutions: A Comparative Analysis of France, Russia, &
China. New York: Cambridge University Press.
Top Related