RESEARCH METHODOLOGY IN MANAGEMENT: CURRENT … › diego_prior › sites › gent.uab.cat... ·...

® Academy of Management Journal2000. Vol, 43. No. 6. 1248-1264.

RESEARCH METHODOLOGY IN MANAGEMENT:CURRENT PRACTICES, TRENDS, AND IMPLICATIONS

FOR FUTURE RESEARCH

TERRI A. SCANDURAUniversity of Miami

ETHLYN A. WILLIAMSUniversity of Colorado at Colorado Springs

This study is a comparison of the strategies employed in management research in twoperiods, 1995-97 and 1985-87. Through a content analysis of articles from the Acad-emy of Management Journal, Administrative Science Quarterly, and the Journal ofManagement, we examined the use of research methods for addressing issues of"triangulation" and validity, including internal, external, construct, and statisticalconclusion validity. Results indicate some potentially important trends in reports ofresearch strategies as well as decreases in the internal, external, and construct validity .of studies.

For management research to progress, it is impor-tant for researchers to assess the methods they em-ploy. The impact of management studies dependsupon the appropriateness and rigor of the researchmethods chosen. Design choices about instrumen-tation, data analysis, and construct validation, andmore may affect the types of conclusions that aredrawn (Sackett & Larson, 1990). Given such con-cerns, an examination ofthe use of research meth-ods is needed to investigate possible patternsemerging over tbe last two decades of the 20thcentury. Examination of such patterns may providesome insights into the possible future developmentof research methodology. Although we recognizethat differences in researchers' training may affecttheir research design choices, we present a frame-work within which many areas of management re-search can be discussed. These design issues affectmanagement research in all substantive areas be-cause shifts in the emphasis put on some method-ological principles (for instance, internal validity)and not on others may fundamentally limit theconclusions that can be drawn. It seems clear tbatchoices about the settings of organizational studies,research designs, and analyses have important im-plications for the accumulation of knowledge overtime.

We thank the three anonymous reviewers as well asthe guest editor of the journal for their valuable andconstructive comments on earlier versions of this article.We also thank Ken Doerr for his insightful commentsand Bettina Hamilton for her assistance with the codingprocess.

The purpose of this article was to identify possi-ble trends in the use of various research methodol-ogies in organizational studies. Key issues in re-search methodologies will be discussed. We drewon McGrath's (1982) typology of research strategiesas a means for examining "triangulation" in re-search and referred to Cook and Campbell's (1976) •categories in examining issues relating to internal,external, construct, and statistical conclusion va-lidity. Our analyses show some possible patterns inthe methodologies employed in management stud-ies. Also, examination of the methods reported inmanagement research might provide insights forthose who aspire to have their research publishedin top-tier publication outlets. This awareness isimportant, since researchers should be mindful ofwhat methodological procedures are being re-warded by the top journals. Use of these proceduresmay even become expected, over time, by the bestmanagement journals. It is thus necessary to beginan examination of management journal contentwith respect to research methods. Such data mayinform us of whether methods are becoming morestandardized or more diversified over time.

We compare and contrast research methodolo-gies from the middle 1980s with those from themiddle 1990s to examine patterns in current prac-tices in organizational studies. The predominantpractices from these time periods were examinedthrough a comprehensive review of all articles pub-lished in the Academy of Management Journal(AMJ), Administrative Science Quarterly (ASQ),and the Journal of Management (JOM). These jour-nals are top-tier outlets that publish articles cover-

1248

2000 Scandura and Williams 1249

ing a range of management topics and expressingboth micro and macro perspectives (Aldag &Stearns, 1988; Coe & Weinstock, 1984; Hinkin,1995).

In our theoretical overview, the issues of triangu-lation in methods and the different kinds of valid-ity (internal, external, construct, and statisticalconclusion; Cook & Campbell, 1976) are discussedwithin an overall conceptual scheme presented inFigure 1. These five methodological issues are sum-marized, and the specific techniques subsumed un-der each are shown. Although we acknowledge thatwe cannot present all the methodological applica-tions that pertain to triangulation or to internalvalidity, external validity, construct validity, andstatistical conclusion validity, we provide exam-ples of broad metbodological approaches to illus-trate these issues.

THEORETICAL OVERVIEW

It may well be true, as McGrath (1982) stated, thatit is not possible to do an unflawed study. Anyresearch method chosen will have inherent flaws,and the choice of that method will limit the con-clusions that can be drawn. It is therefore essentialto obtain corroborating evidence from using a vari-,ety of methods. This is the primary manner inwhich McGrath tackled the issue of research trade-

offs, and it is known as triangulation. Triangulationcan be applied to many elements of research meth-ods, including strategies, settings for data collec-tion (these affect external validity), and sources ofdata (single versus multiple). The concept of trian-gulation is not new. Advocates of using multiplemethods to study a problem include Webb, Camp-bell, Schwartz, and Sechrest (1966) and Smith(1975). As Jick noted, the triangulation metaphor istaken from navigation and military strategy, which"use multiple reference points to locate an object'sexact position" (1979: 602). In the social sciences,the use of triangulation dates at least as far back asCampbell and Fiske's (1959] development of themultitrait-multimethod matrix, in which a re-searcher assesses different traits using differentmethods to examine the discriminant and conver-gent validity of measures. Triangulation can alsorefer to the use of different data collection method-ologies. Jick (1979) gave an example of a study ofanxiety and job insecurity during a merger thatemployed data from self-reports (questionnaires),interviews, coworker observations, and companyarchival records that contained newspaper clip-pings and internal memos. Each different source ofdata provided a unique perspective on what hadbeen happening during the merger and how it af-fected employee reports of stress and job insecu-rity. Finally, it is possible to triangulate on the

FIGURE 1Framework for an Investigation of Trends in Management Research Methodology

Triangulation Internal Validity[McGrath, 1982)

Research Strategy

Formal theory/ Time frameliterature reviews Research strategySample surveyLaboratory experimentExperimental simulationField study: Primary dataField study: Secondary dataField experimentJudgment taskComputer simulation

Related Areas

Substantive domainLevel of dependent variables

External Validity Construct Validity

Type of sampleOccupation ofsubjectsResearch strategy

Construct Validity

Confirmatory factor analysisExploratory factor analysisDiscriminant /convergent /predictive validityInterrater reliability

Related Areas

Otber reliability estimatesType of dependent variableNumber of data sources

Statistical ConclusionValidity

Sample sizeNumber of dependentvariablesData analyticapproacbes

1250 Academy of Management Journal December

research strategies themselves. Researchers mightemploy a lahoratory experiment and a field exper-iment to examine the same research question. Anexample of this is work hy Wayne and Ferris (1990)in which they first employed a lahoratory study toexamine the influence tactics used by subordinatesand then conducted a field study to examine theexternal validity of the findings from the laboratorysetting. Thus, triangulation can be employed for thepurposes of measurement, data collection, or re-search strategy.

Research Strategies

Increased triangulation should improve the abil-ity of researchers to draw conclusions from theirstudies. The use of a variety of methods to examinea topic might result in a more robust and general-izable set of findings (higher external validity). Fur-ther, recommendations for managers could hemade with greater clarity and confidence. Thus, theexamination of the degree of triangulation of re-search methods used in management research wasone of the key tasks of this study. We adopted theapproach of McGrath (1982) because he provided acomprehensive template for classifying researchstrategies. He also recognized that research designchoices imply trade-offs, discussing a "three-horned dilemma" to highlight the trade-offs thatresearchers must make. Various research designsresult in more (or less) (1) generalizability to thepopulation that supports the issue of external va-lidity (Sackett & Larson, 1990), (2) precision inmeasurement and control of the behavioral vari-ables, affecting internal and construct validity(Sackett & Larson, 1990), and (3) realism of context.McGrath categorized research strategies into eighttypes: formal theory, sample surveys, laboratoryexperiments, judgment tasks, computer simula-tions, experimental simulations, field studies, andfield experiments. These eight proposed researchstrategies were the foundation of our analysis of thework published in the three top-tier managementjournals.

The eight research strategies described byMcGrath (1982) are presented in Figure 1, with twomodifications. We expanded the category of formaltheory to include literature reviews, and we di-vided field studies into two subgroups, primaryand secondary data collections. These changes re-sulted in our presenting nine research strategies. Asshown in the figure, the research strategies maponto internal and external validity. For the firstresearch strategy, formal theory and literature re-views were combined. In both approaches, re-searchers often summarize the literature in an area

of research in order to conceptualize models forempirical testing. Theory often involves an induc-tive process, as described by Dubin (1976). A gen-eralization that starts from the data points that ob-servations produce represents an inductiveprocess. An example of a theoretical piece is the1996 work of Lei, Hitt, and Bettis on how to de-velop competitive advantage. The purpose of theirresearch was "to build and explain an integrativemodel of the development and outcomes of dy-namic core competencies" (Lei et al., 1996: 550).Literature reviews often employ a deductive pro-cess that generally provides researchers with hy-potheses for empirical testing (Dubin, 1976). How-ever, they may also propose new theories based oninductive conclusions. An example of a literaturereview content-analyzed for this research is Hack-man and Wageman (1995), in which the authorsreviewed the total quality management literatureand identified dilemmas and trends. McGrath(1982) noted that formal theories maximize popu-lation generalizability but are low on realism ofcontext and precision of measurement. In this in-stance, precision is traded for generalizability.

As a research strategy, the sample survey maxi-mizes the representative sampling of the popula-tion units studied. According to McGrath, the "in-vestigator tries to neutralize context by asking for.behaviors that are unrelated to the context withinwhich they are elicited" (1982: 77). An example ofa sample survey appears in Ingram and Simons(1995), in which the National Organizations Study,which gathered data from a national probabilitysample of work establishments, was used to repli-cate and extend work on institutional and resourcedependence determinants of responsiveness towork-family issues. McGrath's discussion of thethree-horned dilemma of research trade-offs sug-gests that surveys maximize population generaliz-ability but are low on realism of context and preci-sion of measurement.

The laboratory experiment brings participantsinto an artificial setting for research purposes(Meltzoff, 1998). An attempt is usually made tocreate a universal setting that will not have a sig-nificant effect on the results. A study by Seers andWoodruff (1997) in which two sets of students per-formed tasks while researchers examined temporalpacing in task forces is an example of a laboratoryexperiment. According to McGrath, this strategymaximizes precision in measurement of behavior.The trade-offs are low generalizability and low re-alism of context. The fourth research strategy, ex-perimental simulation, refers to a situation con-trived by the researcher in which there is anattempt to retain some realism of context through


use of simulated situations or scenarios (McGrath,1982). Atwater and Van Fleet (1997) conducted anexperimental simulation to examine potential pro-female bias and thus investigate whether mencould compete for traditionally female jobs. Thisstrategy is moderately high on precision of mea-surement and realism of context but low on gener-alizability.

The field study investigates behavior in its natu-ral setting. Obtrusive primary data collections in-volve data that are collected by researchers. Hogan(1987), for example, gathered longitudinal data on49 supervisor-subordinate dyads to examine theeffects of prior expectations on performance rat-ings. This strategy maximizes realism of contextsince it is conducted in a field setting, but it can below on precision of measurement and control ofbehavioral variables (there is a lack of experimentalcontrol). It can also be low on generalizability to thepopulation (with the study population not repre-sentative of the target population). Field studiesthat use secondary data (data collected by a person,agency, or organization other than the researchers)are archival studies; the Profit Impact of MarketStrategies (PIMS) database or census data are exam-ples of such data. Archival studies might includemeta-analyses in which aggregation procedures areused to examine existing patterns in data collectedacross investigations and thus to construct or eval-uate theory. These studies have the same limita-tions as field studies using primary data. An exam-ple of a field study using secondary data is anarticle by Reinganum (1985). Data for this studywere Wall Street Journal announcements of corpo-rate events and stock information that allowed theresearcher to examine the effects of executive suc-cession on stockholder wealth. An example of ameta-analysis is McEvoy and Cascio (1987). Theseresearchers conducted a meta-analysis of the rela-tionship between performance and turnover usingdata collected from 24 studies, aggregating correla-tions over the studies to draw conclusions aboutthis body of research.

A field experiment involves collecting data in afield setting but manipulating behavioral variables.For example, Earley (1986) conducted a field ex-periment in which 36 managerial trainees from ei-ther the United States or England participated in astudy assessing different methods of supervision.This research strategy is moderately high on preci-sion of measurement (and control of behavioralvariables) and realism of context but low on gener-alizability. In judgment tasks, participants judge orrate behaviors. Sampling is systematic rather thanrepresentative, and the setting is contrived. For ex-ample, Harrigan (1985) conducted a study in which

judges, who were managers familiar with the targetindustry, were asked to rate relationships betweensampled strategic business units and adjacent busi-ness units. This type of study is moderately high onpopulation generalizability and precision of mea-surement but lower on realism of context.

Finally, computer simulation involves artificialdata creation or simulation of a process. Onemethod used is the Monte Carlo method, a tech-nique in which an estimate of a parameter is ob-tained by random sampling. Researchers often usesuch a technique when it is difficult to obtain ananalytical solution (Nunnally & Bernstein, 1994).An example of work employing this strategy isSapienza and Korsgaard (1996), in which a com-puter simulation was based on the existing litera-ture on entrepreneur-investor relations. The com-puter played the role of entrepreneurs, data weregenerated on company performance, and partici-pants rated their perceptions of the entrepreneurs.This research strategy is moderately high on popu-lation generalizability and realism of context butlower on precision of measurement.

Triangulation within a substantive area of re-search can be examined by looking at the researchstrategies employed over time. For example, workon mentoring began with interviews of mentor-protege pairs (Kram, 1983). Later, as the theorybecame more developed, researchers employedvaried research strategies (such as field studies andlaboratory experiments), examined the impact ofmentoring on dependent variables at different lev-els (individual, group, and organization outcomes),and employed varied time frames (cross-sectionaland longitudinal). Studies of mentoring have alsolooked at different occupations (such as managers,accountants, and educators), employed differenttypes of dependent variables (for instance, perfor-mance, promotions, and attitudes), examined datafrom the perspectives of both mentors and prot6g6s,and employed various analytical procedures (in-cluding analysis of variance techniques and regres-sion analyses). Thus, some degree of triangulationhas occurred over time, yet field experimental de-signs are needed.

In the current investigation, we also looked at thesubstantive areas, or content domains, in manage-ment research. The major areas can be representedunder five general subheadings: (1) policy/strategy,which refers to areas such as business policy, stra-tegic management, strategic planning, and internallabor markets, (2) organization theory, which in-cludes studies of issues such as organization struc-ture, organization development, organization de-sign, management theory, and organizationalculture, (3) organizational behavior, which refers to


areas such as leadership, justice, groups and teams,influence and power, international management,motivation, and employee attitudes, (4) human re-source management, which involves the study ofissues such as performance and performance ap-praisal, unions, careers, gender and diversity inorganizations, and compensation, and (5) researchmethods, which includes studies on construct de-velopment and other methodological issues of mea-surement, design, and analysis. In addition to ex-amining the content areas represented in theresearch in the three top-tier journals coded for thisstudy, we also examined the levels of the depen-dent variables employed to obtain information ontriangulation. Research that examines the impact ofphenomena on outcomes at different levels, for ex-ample, helps to better inform researchers' under-standing of the subject under investigation.

Looking at the pattern of the employment of re-search strategies in terms of the level of the depen-dent variable used addresses triangulation from alevels-of-analysis perspective. Research can be con-ducted at an individual level, at a dyadic or a grouplevel, and at an organizational level of analysis.Triangulation can therefore occur in both qualita-tive and quantitative research at both theoreticaland empirical levels. The involvement of morethan one research strategy or approach indicatestriangulation. For example, qualitative fieldworkand quantitative surveys can be employed withinthe same research domain, just as laboratory exper-iments, field experiments, and field studies arecomplementary. Triangulation can therefore im-prove internal and external validity as the combi-nation of separate research strategies in one studyhelps to counter the trade-offs inherent in others.

Internal and External Validity

Cook and Campbell (1976) proposed four types ofvalidity. These are ranged across the top of Figure1. The first, internal validity, concerns causality. Acause-and-effect relationship can only be assertedif there is true covariation between the variablesunder investigation, the procedures used to gatherthe data demonstrate that the cause preceded theeffect, and alternative explanations have been dis-carded (Sackett & Larson, 1990). External validityrefers to generalizing across times, settings, andindividuals (Cook & Campbell, 1976; Sackett & Lar-son, 1990). Sackett and Larson stated that "externalvalidity is the type of validity closest to our defini-tion of generalizability" (1990: 430). They noted,however, that Cook and Campbell did not apply theterm "external validity" to generalizing across con-ceptually related variables. External validity relies

upon establishing a true representation of the rela-tionship between two constructs and establishingthat the relationship is generalizable to differentpopulations, measures, and circumstances. It isalso important that the person factors held constantin a study do not interact with the causal factors inthe study (Sussmann & Robertson, 1986).

Laboratory experiments, for example, have highpotential internal validity based on their precisionand on the control of behavioral variables, as dolongitudinal studies, in which cause and effect maybe established. These features may be gained at theexpense of generalizability (affecting external va-lidity), which is better addressed by other methods,such as formal theory and sample surveys. Thus, itappears that no one research strategy can ade-quately cover all four aspects of validity, and soresearchers need to adopt different strategies tomaximize the four kinds of validity. This might beachieved by combining different strategies in onestudy. For example, by conducting a laboratory ex-periment and sample survey, a researcher can ad-dress both internal and external validity. The use ofboth qualitative data (interviews) and survey dataimproves the generalizability (affecting external va-lidity) of interview data. Conducting a longitudinalfield experiment might allow a researcher to estab-lish that the cause preceded the effect, and theexperimental nature of the study would allow forcontrol over the behavioral variables and thus al-low demonstrating internal validity. Testing thesame model using a sample survey would allow theresearcher to report more generalizable results (andthus increase external validity). Having a wide setof occupations represented in the sample wouldalso improve external validity. Studies that employsamples from various occupations and/or indus-trial sectors may be triangulating on settings.

Construct Validity

Construct validity concerns how well the mea-sures employed fit the theories for which a test isdesigned (researchers must test for measurementflaws). Measures and manipulations must be faith-ful (or valid) representations of constructs in orderfor valid inferences to be made (Stone-Romero,1994). Sackett and Larson (1990) cited Campbelland Fiske (1959), who noted that "it is valid toassert that a given operation taps a particular con-struct only if it can be shown that that operationproduces results that agree with those achievedwith alternative operationalizations of the sameconstruct" (Sackett & Larson, 1990: 430). This state-ment refers to both convergent and discriminantvalidity. The techniques presented in Figure 1 for


testing construct validity include confirmatory fac-tor analysis (CFA, typically used to confirm thefactors underlying a latent construct), exploratoryfactor analysis (EFA), reports of discriminant, con-vergent, and predictive validity (based on correla-tional techniques, for example), and interrater reli-ability. Other areas related to construct validityreflect the psychometric properties of measures,such as reliability estimates. The type of dependentvariable reflects the nature of the measures em-ployed, and the sources of data indicate the extentto which method variance might be present in mea-sures. Studies with multiple sources of data reflectissues of measurement (see Figure 1) and alsoindicate triangulation (Jick, 1979). The types ofvariables and source of data have unique methodinfluences associated with them. For example, self-report measures (especially those from a singlesource) are a typical example of the measurementof attitudinal variables using Likert-type ratingscales. Precision of measurement using self-reportsmight improve construct validity if multi-itemmeasures are employed. Also, tests for the amountof error in the measurement of a construct, such astests of reliability (internal consistency) requirerfiulti-item measures.

•Statistical Conclusion Validity

Statistical conclusion validity refers to the abilityto draw conclusions on the basis of statistical evi-dence of covariation as well as prediction (Austin,Boyle, & Lualhati, 1998; Cook & Campbell, 1976;Sackett & Larson, 1990). Cook and Campbell (1976)described statistical conclusion validity, the unre-liability of measures, and the use of inappropriatetests as additional threats to internal validity. Cook,Campbell, and Peracchio (1990) noted that covaria-tion is a necessary but not sufficient preconditionfor causation. Other conditions must be met aswell. Cook and his coauthors also noted that statis-tical conclusion validity concerns sources of error(factors that increase variability) and the appropri-ate use of statistical tests for dealing with sucherror, but that concern with bias (factors that sys-tematically affect the values of means) is the do-main of internal validity. Sussmann and Robertson(1986) noted that statistical conclusion validity isbest evaluated in terms of the statistical propertiesof a population parameter. In order to rule outrandom sampling error (the value of a sample sta-tistic is in error as a result of particular randomobservations that happened to be included in thesample), alternative explanations for the covaria-tion between two variables must be eliminated. Themajor issues to be considered are the degree to

which the design permits correction for range re-striction and the power (the probability of correctlyrejecting a false null hypothesis) of the variousdesigns. Although there are correction formulas forrange restriction, they become more useful as thenumber and variety of persons for whom predictortest scores are known rises (Sussmann & Robertson,1986). Sample size, on the other hand, affects thepower a design has to yield statistically significantresults. Figure 1 presents sample size, the numberof dependent variables, and the analytical proce-dures employed as important factors for establish-ing statistical conclusion validity.

The application of the appropriate statistical testor analytical procedure is important for evaluatingstatistical conclusion validity since the underlyingassumptions may limit their applicability. Fieldexperiments, for example, require samples largeenough to address the issue of power. However,issues of statistical conclusion validity are relatedto internal validity since they both apply to asser-tions about the covariation between variables(Sussmann & Robertson, 1986). To exclude alterna-tive explanations for covariation, precision in mea-surement and control over behavioral variables areimportant (features found in laboratory experi-ments). It is also important that statistical conclu-sion validity support external validity, since ran-dom sampling error must be ruled out. Ruling it outcan strengthen a researcher's confidence in theability of results to hold at other times, in othersettings, and/or with other actors.

These methodological issues are important sinceneglecting them may limit the ability to base con-clusions on the research conducted. The trade-offsdiscussed above are the essence of the McCrath(1982) framework, and our purpose was to examinetrends relating to these trade-offs over the past twodecades. We compared research strategies andmethodological approaches over two time periods,the first representing the 1980s (1985-87) and thesecond, the 1990s (1995-97), to examine thesetrends. We employed indexes from the diversityliterature to examine the heterogeneity (the amountof variation occurring) in research within both timeperiods (Blau, 1977). Linear trend analysis was alsoemployed to examine the potential changes overtime. Finally, the implications of these patterns formanagement research in the future are discussed.

METHODS

As noted above, to examine research methods inorganizational studies, we read and coded (content-analyzed) the methods sections of all the articlespublished in the Academy of Management Journal,


Administrative Science Quarterly, and the Journalof Management for the periods 1985-87 and 1995-97. These journals are three ofthe most highly ratedpublication outlets in organizational studies (Aldag& Stearns, 1988; Coe & Weinstock, 1984; Hinkin,1995). Our examination of tbese tbree journals didnot represent an exhaustive suirvey ofthe literature,but we looked at a wide range of empirical workand formal tbeory/literature reviews across broadareas of management and organizational studies.Tbe years 1985 to 1987 were used as a bencbmarkcomparison period against wbich current practicecould be examined. For botb periods, we coded allarticles tbat employed empirical metbods and/orformal tbeory/literature reviews for tbe tbree jour-nals. The extent of differences in research methodsand substantive content between the midpoints ofthe 1980s and the 1990s were examined.

We reviewed 347 articles utilizing formal theory/literature reviews and/or empirical data, omittingcommentaries and book reviews publisbed in tbeperiod 1985-87. Eigbty-three articles employedformal tbeory/literature reviews alone. A total of264 articles were empirical pieces. A number of tbestudies reported in tbese articles employed multi-ple samples, and tbese also often employed sepa-rate designs as a means of eitber supporting priorfindings or exploring further related aspects (Stone-Romero, Weaver, & Glenar, 1995). We treated eachsample from each article with multiple samples asa separate study from a research methodology per-spective (14 articles had multiple samples, 12 hav-ing 2 separate samples and 2 having 3 separatesamples). Thus, a total of 280 (16 additional due tomultiple samples) empirical studies were content-analyzed for the primary research methodologiesemployed for the period 1985 to 1987.

For the period 1995-97, we content-analyzed 385articles utilizing formal theory/literature reviewand/or empirical data. Of these, 77 articles wereformal theory/literature reviews only. A total of 308articles were empirical pieces. Twenty-one articlesconcerned multiple samples (16 having 2 separatesamples and 5 having 3 separate samples). Thus, atotal of 334 (26 additional due to multiple samples)empirical studies were content-analyzed for theprimary research methodology employed for theperiod 1995-97. We simplified the coding by focus-ing on the primary research method employed ineach study. For example, exploratory factor analy-sis was employed in some studies for constructvalidation purposes but was not the primary strat-egy employed for hypothesis testing (see Table 6).Where the primary focus of a study was the devel-opment of a new measure and factor analysis wasthe analytic technique employed to test bypothe-

ses, it was coded as a data analytical approach(Table 8). However, if factor analysis was used forbypotbesis testing, construct validation was codedaccording to reports of discriminant, convergent, orpredictive validity.

Coding

Studies were analyzed and represented by 13coding dimensions (see Figure 1). Tbe categorieswitbin eacb code are presented as a percentage ofthe total number of studies examined and sum to100 percent in Tables 1, 4, 5, 6, 7, and 8. The onlyexception to this rule is the code "other reliabilityestimates" (Table 6), which is presented as a per-centage of the number of empirical studies coded.Reliability estimates are an important supplementto tbe construct validity code since tbey are typi-cally reported in addition to other evidence of con-struct validity. The codes relating to research strat-egies and content domains were applied to all thestudies (363 for the 1980s and 411 for the 1990s).All other codes apply to only the empirical studies(280 for the 1980s and 334 for the 1990s). Thefollowing sections detail how the data were coded.

The procedure employed for the content analysi'sof research methods applied the following princi-ples. We recorded the categories presented in Fig-^ure 1 as they appeared in each study. Since thecategories summed to 100 percent, only one cate-gory within each code was recorded as the primaryresearch method for each study. As noted in thetheoretical overview, the primary research strategywas categorized according to the framework devel-oped by McGrath (1982). We referred to this cate-gory assignment as code 1. These categories wereformal theory/literature reviews, sample surveys,laboratory experiments, experimental simulations,field studies (using primary or secondary data),field experiments, judgment tasks, and computersimulations. Tbe substantive content domains ofthe articles (for instance, leadership, careers, inter-national management, strategic management) werecoded according to divisions of the Academy ofManagement as well as the subtopic areas indicatedin the ABI/INFORM database; tbis was code 2. Aswas described in tbe tbeoretical overview, we col-lapsed subject areas into five general categories tbatgrouped related concepts. Tbe five content areaswere policy/strategy, organization tbeory, organiza-tional behavior, human resource management, andresearch methods. The level of the primary depen-dent variable or variables employed in eacb studywas recorded according to individual, group, ororganizational level; tbis was code 3. Typically, tbefocus of each study was at one level. Where a single


article employed several levels of analysis, it typi-cally did so over multiple samples (thus, in codingarticles with multiple samples, we treated them asseparate studies).

Primary reports of internal, external, construct,and statistical conclusion validity were recordedaccording to the definitions provided by Cook andCampbell (1976). The time frame of a study (code 4)was either cross-sectional or longitudinal. The pri-mary type of sample employed (code 5) was de-fined in terms of the economic sector representedand the type of student subjects (employed or full-time students). Private sector samples were allnongovernmental profit-making entities, and thepublic sector included governmental entities.Not-for-profit samples included governmental andnongovernmental entities. There was also a cate-gory for samples that mixed these entities, a cate-gory for no report of the type of sample, and a notapplicable category (empirical studies using com-puter simulations and meta-analyses, for instance,fell into the latter). The primary occupation of sub-jects was recorded according to industry and pro-fession (code 6). The student category was repeatedwithin this code, since some student samples mightbe composed of full-time students. However, if stu-dents were employed, they were categorized intheir appropriate profession or industry. Occupa-tions included professional work (lawyers, for ex-ample, were in this category); managerial work;manufacturing; health care (doctors and nurses);education (teachers, professors); blue-collar work(custodians and gardeners); white-collar work(clerical employees); and technical work (computerprogrammers). The categories "not reported" and"not applicable" were also included; a field studyusing secondary data and conducted at the organi-zational level would be so categorized.

The nature of the construct validation procedureemployed (if any) was recorded (code 7). Categorieswere confirmatory factor analysis, exploratory fac-tor analysis, discriminant, convergent, or predic-tive validity reports, and interrater reliability.Other categories included "no evidence of con-struct validity reported" and "not applicable" (sim-ulation data would, for instance, receive the lattercode). Other reliability estimates (code 8) were in-cluded as a means of accounting for reports of theinternal consistency (or theoretical validity) ofmeasures (Lord & Novick, 1968). The categories forthe primary type of dependent variable employed(code 9) were supervisory reports of performance,tangible or behavioral outcomes (such as return onassets and turnover), attitudinal outcomes, and per-ceptual outcomes. We included the category "notapplicable" to account for studies such as those

using qualitative data or clustering techniques. Thesource of the data employed (code 10) was catego-rized as single or multiple. The sample size wasrecorded for each study examined (code 11). Thenumber of dependent variables employed (code 12)was recorded; for simplicity in reporting, we col-lapsed the categories into one, two, three, and fouror more dependent variables. Also, "not applica-ble" was included.

Finally, the primary type of data analysis em-ployed was recorded. Techniques were recorded(code 13) according to groupings of similar types ofanalyses. Univariate and multivariate analyses ofvariance and covariance (ANOVA, ANCOVA,MANOVA, and MANCOVA) and ^tests were con-sidered analysis of variance techniques. Linear re-gression analyses included simple, multiple, hier-archical, moderated, and mediated regression.Correlation techniques, meta-analysis, and lineartechniques for categorical dependent variables (forinstance, logistic regression analysis) were re-corded separately. Nonparametric techniques andinterpretative analysis were recorded together (forinstance, use of descriptive statistics, contingencyanalysis, and interpretative analysis). Factor-analytic and clustering techniques included confir-matory factor analysis, multidimensional scaling,and discriminant analysis. Structural equationmodeling and path-analytic techniques were com-bined, and time series analysis was combined withevent history techniques. Multiple-levels-of-analy-sis techniques included hierarchical linear model-ing and within and between analysis. The finalcategory was computer simulation (for instance,Monte Carlo studies).

Each article was examined by one of two coders,and a coding sheet was compiled for all 12 codes(except content domain). Coding was randomly as-signed to coders among journal issues. To assessthe consistency of the coding process, both coderswere randomly assigned approximately 50 percentof the articles to recode. There were varying levelsof agreement across the years of research methodscoded. Rater agreement levels for 1985, 1986, and1987 were 83 percent, 87 percent, and 90 percent,respectively. Agreement levels for 1995, 1996, and1997 were 85 percent, 90 percent, and 89.7 percent,respectively. There were disagreements on allcodes; however, the majority of disagreements oc-curred for sample size, type of sample, subject oc-cupation, and construct validation. Where therewere disagreements, coders discussed the relevantstudy categories until there was complete agree-ment on a judgment. Gerstner and Day (1997) rec-ommended this procedure for reconciling codingdisagreements.


For the substantive content areas, coding wasrepresentative of the divisions of the Academy ofManagement. We also coded secondary contentareas in terms of subjects in the business and man-agement database ABI/INFORM. Two individualscoded all studies for substantive area; one coderspecialized in organizational behavior, and theother specialized in strategy and international man-agement. Where a study was cross-disciplinary,both coders referred to the subject terms providedby ABI/INFORM and discussed the study untilthere was complete agreement (there were very fewdisagreements). For example, research on top man-agement teams was coded as strategy/policy re-search and not as team research in organizationalbehavior.

Tests

Differences in proportions. Tables 1 through 8present results for both periods studied. Each num-ber reported represents the overall percentage ofstudies utilizing the research method for the indi-cated period. To compare the use of research meth-ods and the implications for triangulation, internal,external, construct, and statistical conclusion va-lidity in research across the time periods, we cal-culated tests of the significance of the differencebetween two proportions for each research methodreported (Bruning & Kintz, 1977: 222-227).

Heterogeneity indexes. The literature on diver-sity provides a useful tool for examining the vari-ance within coding dimensions. According to Blau,"the two conditions that deterniine the degree ofheterogeneity in terms of a given nominal parame-ter are the number of groups into which the popu-lation is divided and the distribution of personsamong them" (1977: 78). For our purposes, thepopulation was the number of studies content-analyzed for each code, and the number of groupswas the number of categories within each code. Thepercentages reported for each group within eachcode represent the distribution of the researchmethodologies employed. Using the equation pro-vided by Blau (1977; 78), we calculated heteroge-neity indexes for the codes. Use of this indexhelped explicate the degree to which triangulationwas occvu-ring, allowing us to compare varianceacross the 1980s and 1990s. To standardize theindexes reported for comparative purposes (acrosscodes), we normalized them on the basis of thenumber of categories that each code represented.

Regression analyses. In Tables 1, 5, 6, 7, and 8,the superscript b indicates whether the changesobserved represent some continuity in terms of alinear pattern. To determine the presence of such

linearity, we used linear regression analysis, withthe category of interest (within each dimension)coded as 1 and the remaining categories coded as 0;the dependent variable was "year." A significantresult indicated that the change observed betweenperiods (1985-87 and 1995-97) represents a pat-tern in the use of a research method (reported atp < .05). If a trend was found, we entered thesuperscript b above a "-I-" or a " —," indicatingincreases or decreases.

RESULTS

Triangulation: Research Strategy, SubstantiveArea, and Level of the Dependent Variable

For the period 1985-87, 22.9 percent of the stud-ies employed formal theory/literature reviews asthe primary research strategy, compared with 18.7percent for the period 1995-97 (Table 1). In bothperiods, field study was the primary research strat-egy most often employed. Statistically significantdifferences between the periods occurred for sam-ple surveys, laboratory experiments, and field stud-ies employing secondary data. The heterogeneityindexes for research strategies were .73 and .58 forthe 1980s and 1990s, respectively, values that sup-port the interpretation of a decline in triangulation.Regression analysis revealed that the decreasingproportions represented a pattern of fewer samplesurveys and laboratory experiments from one timeperiod to the next. Field studies using secondarydata rose, however, from the 1980s to the 1990s.

Cross-tabulations were run for the research strat-egy employed by substantive content area for both1985-87 and 1995-97. Chi-square tests revealed

TABLE 1Triangulation: Researcb Strategies"

Research Strategy

Formal theory/literature reviewSample surveyLaboratory experimentExperimental simulationField study

PrimarySecondary

Field experimentJudgment taskComputer simulation

1985-87

22.90%6.90

10.700.60

38.0016.10

3.900.600.30

1995-97

18.70%3.60''-4.90''-1.70

40.9026.60'" •

2.200.201.20

" For the earlier period, n = 363 (347 + 16); for the laterperiod, n = 411 (385 + 26).

^ Time-based regression analyses revealed a pattern of usagein the direction of the change in the proportion. The symbol " + "indicates a significant increase (p < .05). The symbol " - " indi-cates a significant decrease (p < .05).


statistical significance (p < .001 for both the 1980sand the 1990s), indicating a relationship betweenthe substantive area and use of research strategies.As indicated in Table 2, for both periods, strategy/policy and organization theory studies tended toutilize theoretical and field research strategies. Or-ganizational behavior studies employed more the-oretical, laboratory experiment, and field studystrategies; human resource studies used field re-search; and research methods studies relied moreon theory and field study research strategies.

We observed a few notable changes in the con-tent areas represented when comparing studiesfrom 1985 to 1987 with those from 1995 to 1997.For the five major content areas represented inTable 2, no significant changes occurred betweenthe two periods. Heterogeneity indexes werefairly close as well, with values of .92 and .94.However, inspection of the specific domains re-vealed that a pattern of decreases occurred for thestrategic planning area, the careers/turnover area,the motivation/participation area, and the laborunions area. There was a decreasing trend instudies on strategic planning and careers/turn-over. Yet a positive shift occurred in studies onjustice, groups/teams, international management,learning, and perceptions/impression manage-ment for the 1990s.

Table 3 presents the results for the cross-tabula-tion of research strategy by the level of the depen-dent variable. Chi-square tests (p < .001) indicatedthat, for both the 1980s and 1990s, sample surveys,laboratory experiments, field studies employingprimary data, and field experiments tended to use

dependent variables at the individual level. Samplesurveys and field studies employing secondarydata (and, to a lesser degree, field studies employ-ing primary data) appeared to be utilized moreoften at the organizational level. There appears tohave been a rise in the variance for the level of thedependent variable employed, as heterogeneity in-dexes are .86 and .91 for the 1980s and 1990s,respectively. Overall, these trends indicate a reduc-tion in the use of dependent variables at the indi-vidual level and more use of dependent variables atthe organizational level.

Internal and External Validity

There were differences between the tw o periodsin the time frames employed for studies (Table 4),with an significant rise in cross-sectional studies.Heterogeneity indexes fell between the 1980s andthe 1990s, supporting the reduction in variance (.70and .50, respectively). However, regression analy-ses revealed no significant patterns in the use ofeither research approacb.

The highest percentage reported for type of sam-ple was for samples drawn from the private sector,and there was a statistically significant increase inthis type between the 1980s and 1990s (Table 5).The proportions of public sector samples andmixed samples declined. Heterogeneity indexesacross the two periods were .88 and .76, respec-tively. Regression analyses supported this trend intypes of samples employed, with a significantlyhigher representation of people with white-collaroccupations, lower educational levels, and techni-

TABLE 2Triangulation: Research Strategy by Substantive Area

Research Strategy"

Formal theory/literature reviewSample surveyLaboratory experimentExperimental simulationField study

PrimaryField study

SecondaryField experimentJudgment taskComputer simulation

PolicyStrategy

80s

14530

8

20200

gos

12100

20

32000

OrganizationTheory

80s

34300

24

17000

9Ds

30300

26

34000

Suhstantive Area

OrganizationalBehavior

80s

228

282

65

13611

gos

185

172

83

19412

HumanResources

80s

7870

31

7400

90s

8533

34

19200

ResearchMethods

80s

6100

3

3000

90s

9400

5

8002

° The absolute numbers of articles using formal theory/literature reviews and/or empirical data were used for cross-tabulation purposes.For 1985-87, n = 363, x^ = 87.02, p < .001; for 1995-97, n = 411, x" = 120.01, p < .001.


TABLE 3Triangulation: Research Strategy by Dependent Variable"

Research Strategy

Sample surveyLaboratory experimentExperimental simulationField Study

PrimarySecondary

Field experimentJudgment taskComputer simulation

Individual

80s

1233

0

797

1000

90s

715

0

9210

300

80s

040

113200

Level

Group

90s

250

153200

of Dependent Variable

Organization

80s

1000

2837

000

90s

400

3973

000

Not

80s

212

2611

011

Applicable

90s

205

2823

114

" Only empirical articles were used for cross-tabulation purposes. For 1985-87, n = 280,= 127.13, p < .001.

= 105.89, p < .001. For 1995-97, n = 334,

TABLE 4Internal Validity: Time Frame of Studies"

Time Frame 1985-67 1995-97

Cross-sectionalLongitudinal

77.40%22.60

85.60%"'14.40"

" For 1985-87, n = 280; for 1995-97, n = 334. Tbe symbol" + " indicates a significant increase (p < .05). Tbe symbol " - "indicates a significant decrease (p < .05).

cal occupations occurring. The indexes for hetero-geneity were .95 and .88 for the early and the lateperiod, respectively, indicating a reduction in thevariance in the occupations represented in researchfrom the 1980s to the 1990s.

Construct Validation and Otber MeasurementApproaches

Most studies reported no construct validation,and the number that did not report it increasedbetween the two time periods (Table 6). The high-est percentage of reports for construct validationwas for reliability reports. A decline occurred forreports of discriminant, convergent, and predictivevalidity (there were, however, more studies thatemployed confirmatory factor analysis as a primarydata analytic tool when their goal was the develop-ment of a new measure). Heterogeneity indexes [.77for the 1980s and .51 for the 1990s) indicate areduction in the variance of reports. The highestpercentage reported for type of dependent variablewas for tangible or behavioral outcomes as opposedto performance ratings. There were no significantdifferences in proportions, however. There were.

however, changes in the sources from which datawere obtained. The heterogeneity indexes were .77for the 1980s and .39 for the 1990s, a change thatindicated more data were being gathered from sin-gle sources.

Statistical Conclusion Validity: Sample Size,Dependent Variables, and Data Analysis

For 1985 to 1987, the median sample size was129 (values of 67 and 284 for the 25th and 75thpercentiles), and for 1995 to 1997, the median sam-ple size was 173 (values of 70 and 376 for the 25thand 75th percentiles). There was no significant dif-ference between means for the two periods. Thecross-tabulation for sample size (according to per-centile) by substantive area was not statisticallysignificant for either period (Table 7). The highestpercentage reported for number of dependent vari-ables employed was for the category of single de-pendent variable. Regression analysis revealed thatthere was a downward trend in the use of the singledependent variable.

Comparisons between 1985 to 1987 and 1995 to1997 (Table 8) revealed a statistically significantdecline in the use of analysis of variance tech-niques and correlation techniques; exploratory fac-tor analysis also rose. Between these periods, therewas a rise in the use of linear regression techniques,structural equation/path analytic techniques, timeseries/event history techniques, and multiple-levels-of-analysis techniques; there were also morestudies using probit analysis. However, the use ofanalysis of variance techniques fell.


TABLE 5External Validity: Organizational Settings and

Occupations of Subjects*

TABLE 6Construct Validity: Types and Evaluations of

Measurements"

Characteristic

Type of samplePrivate sector, nongovernmentalPublic sector, governmentalNot-for-profit, governmental

and nongovernmentalMixedNot reportedStudent subjectsNot applicable

Occupation of subjectsProfessionalManagerialManufacturingHealth careEducationBlue-collarWhite-collarTechnicalStudents

MixedNot reportedNot applicable

1985-87

37.90%17.702.10

20.600.70

15.605.40

4.3021.403.605.706.802.104.302.80

14.90

14.902.80

16.40

1995-97

54.30%''•^10.40''"0.90

11.90''"0.90

14.906.70

2.1023.303.003.300.90''"3.308.90''^0.30''-

• 10.70

11.900.40''"

31.90''*

Characteristic

Construct validationConfirmatory factor analysisExploratory factory analysisDiscriminant/convergent/

predictive validityInterrater reliabilityNone reportedNot applicable

Other reliability estimates

Type of dependent variableTangible outcomesPerformance ratingPerceptualAttitudinalNot applicable

Number of sources of dataSingleMultiple

1985-87

4.00%9.40

29.90

4.7050.70

1.30

39.40

41.3018.9011.7012.3015.80

73.7026.30

° For the earlier period, n = 280; for the later period.The symbol " + " indicates a significant increase (p <svmhol " - " indicates a sienificant decrease fD < .051.

1995-97

6.60%9.303.30''"

4.8074.80''*1.20

40.00

34.7013.7016.4016.6018.60

88.90''•*•

11.11"

77 = 334.

.05). The

° For the earlier period, n = 280. For the later period, n = 334.The symbol "-t-" indicates a significant increase (p < .05). Thesymbol " - " indicates a significant decrease (p < .05).

'' Time-based regression analyses revealed a pattern of usagein the direction of the change in the proportion.

DISCUSSION

The results of our analysis of articles publishedin three top-tier management journals indicatesome potentially important shifts in what is beingemphasized in research methods. To publish inthese three top-tier general management journals,researchers are increasingly employing researchstrategies and methodological approaches thatcompromise triangulation. The patterns of researchmethods that we found differed across the middle1980s and 1990s and are, in many cases, indicativeof lower internal validity, external validity, andconstruct validity. Sample surveys offer greater ex-ternal validity, yet there was a decreasing trend intheir use. The use of laboratory experiments, whichoffer greater precision in measurement and controlover behavioral variables, also fell (indicating lessinternal validity). Although the use of computersimulations, which offer moderate levels of gener-alizability and realism of context, and the use ofexperimental simulations, offering moderate preci-sion/control and realism of context, increased, theydid so by nonsignificant proportions. There was asignificant rise in the use of field studies. However,although field studies offer high contextual real-

'' Time-based' regression analyses revealed a pattern of usagein the direction of the change in the proportion.

ism, they involve the trade-offs of low generaliz-ability and low precision/control, which impactsboth internal and external validity.

Writing during the period that served as ourbenchmark, Mitchell (1985) noted the serious con-sequences of sacrificing rigor for relevance in man-agement research. Mitchell wrote that the typicalcross-sectional correlational study lacked evidenceon reliability and that construct validity was infre-quently checked. To improve correlational fieldstudies, he suggested more reporting on the re-sponse rate for a sample and on the sample's rep-resentativeness, providing information on the reli-ability and construct validity of measures, andusing hold-out samples and cross-validation. Ouranalysis of trends in research methods over the lasttwo decades indicates that Mitchell's suggestionshave been virtually ignored. The trends we uncov-ered in our analysis suggest that management re-search may be moving even further away fromrigor. Essentially, researchers may be limiting theapplicability of their findings by failing to triangu-late on research strategies in management overtime.

These shifts may have occurred for a number ofreasons. The changes may be due to changes in thepreferences of editors and review boards. Perhapsmore researchers are being trained in business

1260 Academy of Management Journat Decemher

TABLE 7Statistical Conclusion Validity: Substantive Area, Sample Size, and Number of Dependent Variables

(a) Substantive Area ""''icyStrategy

Sample Size byPercentile' 80s

lst-25th 1226th-50th 1851st-75th 1375th-100th 7

(b) Sample Size and Number ofDependent Variables

Sample sizeMeanMedianStandard deviationInterquartile rangeLargest sample

Number of dependent variablesSingleTwoThreeFour or moreNot applicable

90s

107

1026

a

OrganizationTbeory

80s

1216

88

90s

189

1719

OrganizationalBebavior

80s 90s

37 4042 2626 3617 34

1985-87(n = 280)

428.03129.00

1,680.1067-28421,511

38.00%16.5013.3016.4015.80

HumanResources

80s

12142012

90s

13121821

ResearcbMetbods

80s 90s

2 31 10 63 8

1995-97(n = 334)

498.31173.00

1,703.5270-37623,170

31.10%''18.9013.2018.2018.60

" Chi-square statistic not significant for 1985-87 or 1995-97.'' Time-based regression analyses revealed a pattern of usage in the direction of the change in the proportion.

schools that focus more on applied field case stud-ies (hence the drop in laboratory experimentation).Another possibility is that researchers are doingwhat is expedient, in view of external pressures fortenure and merit pay increases. Experimental stud-ies are obtrusive, and it may be more difficult thanit used to be to gain access to sites that allowbehavioral manipulations. Human subjects reviewboards may also be more conservative in approvingexperimentation than in the past. These explana-tions, however, do not justify the trade-offs that arebeing made. If the current trend continues, therewill be almost no high internal validity strategiesbeing employed in management research in tenyears. Their absence will have serious conse-quences for the ability to develop prescriptionsand/or interventions to improve management prac-tice.

Some interesting changes in the substantive con-tent of organizational studies over the two timeperiods are worth noting as well. Fewer studiesexamined strategic planning, careers/turnover, jobdesign/roles/job evaluation, and influence/power/politics in the 1990s. However, there were morestudies on groups/teams, international manage-ment, learning, and perceptions/impression man-agement in the last decade of the 20th century.

Studies of strategic planning fell, but there weremore reports of the use of dependent variables atthe organizational level. This difference seems tobe accounted for by the rising presence of studieson international management, technology, innova-tion, and total quality management. Cross-tabulat-ing the level of the dependent variable by substan-tive content area revealed that more studiesemploying organization-level dependent variableswere reported in organization theory and organiza-tional behavior. Thus, there appears to actually bemore triangulation with respect to some substan-tive areas represented and levels of dependent vari-ables employed; this conclusion was further sup-ported by the heterogeneity indexes, whichindicated higher variance in these areas.

When we crossed substantive area with researchstrategy, it appeared that researchers in a particulararea follow dominant approaches. Yet triangulationof research designs is necessary to avoid the flawsthat are inherent when making trade-offs in re-search (McGrath, 1982], and researchers shouldperhaps consider designs in addition to the onespredominantly used in their fields. Hambrick andMason (1984), for example, developed propositionsfor the study of top management teams that couldbe tested by collecting data from individuals and/or


TABLE 8Statistical Conclusion Validity: Data Analytical

Approaches"

Anal}rtical Procedure

Analysis of variance techniquesLinear regression techniquesCorrelation techniquesMeta-analysisLinear techniques for categorical

dependent variahlesNonparametric/interpretative techniquesFactor-analytic/clustering techniquesStructural equation/path-analytic

techniquesTime series/event history techniquesMultiple-levels-of-analysis techniquesComputer simulation techniques

1985-87

27.80%30.706.101.403.60

15.508.303.60

2.600.000.40

1995-97

13.80%''"42.40*

1.80"0.606.90

11.105.108.70*

7.50*'*1.80*0.30

° For the earlier period, n = 280; for the later period, n = 334.The symbol " + " indicates a significant increase [p < .05). Thesymhol " - " indicates significant decrease (p < .05).

^ Time-hased regression analyses revealed a pattern of usagein the direction of the change in the proportion.

small groups. Researchers in the human resourcesarea could employ more theoretical studies, fieldexperiments, and field studies using secondarydata, and organizational behavior researchersmight begin to employ more sample surveys, fieldexperiments, and field studies using secondarydata.

It seems clear that the field of management is nottriangulating on methods by conducting researchthat is based on all classes of evidence. Many lab-oratory studies are still conducted at universitiesand utilize undergraduate students. Gordon, Slade,and Schmitt (1986) noted that replicating resultsfound in student samples with nonstudents issometimes problematic. Laboratory experimenta-tion is not concerned primarily with behavior thatoccurs in natural social settings. However, experi-ments suggest what might happen in the field.Thus, the low number of field and laboratory ex-periments being conducted is disheartening. Thisis not to say that field studies and rich descriptiveanecdotal research are not useful when they arewell conducted. However, the decreasing trend inthe use of laboratory and other longitudinal designsindicated by our analysis does not bode well for thefuture of management research. This decline oc-curred despite numerous calls for more longitudi-nal research in many areas of study.

Although our analyses reveal that a substantialproportion of the samples examined in the middle1990s were students, almost one-third of these stu-dents were employed, in contrast with fewer than 5percent of the students studied in the middle

1980s. This might suggest that there are more mas-ter of business administration (M.B.A.) studentsbeing used in research than in the past, but for fieldresearch rather than for laboratory experiments. AsMook noted, "A misplaced preoccupation with ex-ternal validity can lead us to dismiss good researchfor which generalization to real life is not intendedor meaningful" (1983: 379). However, the apparentlack of preoccupation with any type of validity(internal, external, construct or statistical conclu-sion) can have serious repercussions. More studiesin the middle 1980s than in the middle 1990s re-ported construct validity information, and reportsof discriminant, convergent and predictive validitydecreased. There was a decrease in the proportionof studies that reported information about con-struct validity as well. Stone (1978) noted that thestate of the science of organizational behavior de-pends on strong reports of validity of all four typesand that past research has devoted insufficient at-tention to this area. The use of single-source datawas also greater in the 1990s, and this trend hasimplications for measurement as well as for trian-gulation. Our findings in this study suggest thatresearch in the field of management needs to em-ploy more triangulation on research strategies andother methodological approaches. The greater thenumber of sources of evidence supporting theory,the stronger the conclusions that can be drawn. It isclear that studies in the middle 1990s containedmore threats to internal, external, and constructvalidity (Cook & Campbell, 1979] than those re-ported in the 1980s.

The news is not all bad, however. Of all the typesof validity that Cook and Campbell emphasized,statistical conclusion validity appeared to be theleast threatened. Both the mean and median samplesizes rose in the 1990s. This increase might suggestthat issues of statistical power are receiving moreattention. This increase in sample size did not ap-pear to be limited to any one area of research. It alsoappears that research in the 1990s was becomingbroader, with more dependent variables and con-tent domains being included. The types of dataanalytical approaches also appeared to be movingtoward the use of more complex techniques, suchas structural equation modeling and event historyanalysis.

Limitations of the Study

The potential for skewed results owing to idio-syncrasies in the articles published during theseyears and in the journals studied exists, thus lim-iting the conclusions that can be drawn. Yet thegoal of comparing research methods over two time


periods appears to have been met by the inclusionof multiple years from two decades. One mightargue tbat something peculiar about the years weselected to review in terms of the studies publishedin these three journals might account for the find-ings reported here. We think not, however, sincethe years were randomly selected. Also, the lineartrend (regression] analyses conducted help avoidthe influence of peculiarities in the data for any oneyear. One might also argue that these journals arenot representing the field well; however, thesethree outlets (AMJ, ASQ, and fOM) are consideredamong the best publishers of research on manage-ment topics. Perhaps inclusion of data from the1960s and 1970s would have indicated an increasein triangulation over time. We acknowledge thatour use of two time periods as data points may nothave been sufficient to establish trends, but theredo appear to have been some interesting changesoccurring between the periods examined in thisstudy. Clearly, this research needs replication instudies using other journals and time periods. De-spite these limitations, we hope that we have sug-gested a conceptual template and data coding andanalysis strategy for future reviews of trends inmanagement research.

The Future of Research in Management Studies

The results presented here may not be character-istic of the three journals we surveyed (especiallygiven that a limited sample of studies was content-analyzed). It is, however, important to note that thefindings on construct validity echo results found ina review of measurement practices for 1991 and1992 for the Academy of Management Journal, theJournal of Management, Strategic ManagementJournal, and the Journal of International BusinessStudies (Schriesheim, Powers, Scandura, Cardiner,& Lankau, 1993). The inclusion of different years inthe 1990s and of a different journal (AdministrativeScience Quarterly) in our review support conclu-sions made by other researchers. We are convincedthat the attention paid to the four types of validitydescribed by Cook and Campbell (1976) needs to beincreased in future management research. The abil-ity to draw well-informed conclusions from data iscentral to sound research. Therefore, more atten-tion to triangulation and to internal, external, con-struct, and statistical conclusion validity is clearlywarranted.

We encourage a similar review for such journalsas the Strategic Management Journal and the Jour-nal of International Business Studies to establishbenchmark study years that can be used in compar-isons with future years, as we have done here. Also,

a wider variety of journals needs to be reviewed toexpand the analysis of patterns reported in thisstudy. The use of trend analysis over longer timeperiods may also be helpful for identifying whereresearch seems to be heading in terms of the trade-offs being made and triangulation on methods. Inview of this review and analysis of differences andtrends from the middle 1980s to the middle 1990s,we expect management research at the beginning ofthe new millennium to continue to emphasize rel-evance over rigor. Researchers must carefully con-sider the implications of reliance on field studies asthe empirical foundation of management research.More attention should be given to issues of inter-nal, external, construct, and statistical conclusionvalidity as well. The ability to draw causal infer-ences and to make recommendations for practicemay be compromised if current practices continue.

Some recent trends may indicate new possibili-ties for research designs that contain both rigor andrelevance. Perhaps the Internet could be used topermit more experimentation while generatingmore representative samples. The rising use of vir-tual reality exercises that can be administered viathe Internet or by specially developed software maylead to an increase in new field and laboratoryexperiments (and a corresponding rise in internalvalidity). Sample surveys can be enhanced withtechnology that would allow respondents to visit aWeb page and input responses (with efficientbranching questions to keep mistakes at a mini-mum). Also, a data file could be generated automat-ically, reducing both the time involved and dataentry errors. These trends in new technologies forresearch can, we hope, be creatively employed tostem some of the disturbing trends indicating a lossof precision, control, and generalizability in man-agement research.

This review highlights some of the trade-offs thatseemed to have been made in research conductedin the last two decades of the 20th century. Trade-offs may be inevitable, but researchers need to bemore aware of their implications for the future ofmanagement research. As McGrath (1982) sug-gested, all research designs have inherent flaws.Use of various designs in a program of research(triangulation) may help counterbalance thestrengths and weaknesses of particular designs.The rigor of the laboratory experiment lacks thecontextual realism of the field setting. Yet fieldresearch often contains confounds that make thesignificance of findings difficult to isolate. Thechoice of research designs must be carefully con-sidered because it has clear implications for astudy's ability to have practical significance. Theresults presented here suggest that some research


design principles that emphasize internal, external,and construct validity as well as triangulation tech-niques are underutilized, such as use of longitudi-nal time frames and of varied settings and occupa-tions. It might he useful for researchers to stepoutside the dominant practices of their suhstantiveareas and try some methods and statistical tech-niques employed in other areas. Although someaspects of research methods are improving (morestudies are using time series/event history analysis,for example), this review may serve as a reminderthat researchers should he aware of how method-ological choices might he affecting the cumulativehody of knowledge in management research. Man-agement researchers need to rethink their apparentpredisposition toward field studies, the most com-mon research strategy now employed. They need toquestion whether studies with amhiguous conclu-sions can say much ahout the settings in whichthey occur. It may he that, without rigor, relevancein management research cannot he claimed.

REFERENCES

Aldag, R, J., & Stearns, T. M, 1988. Issues in researchmethodology./ouma/o/Ma/iagemenf, 14: 253-275.

Atwater, L, E., & Van Fleet, D. D. 1997. Another ceiling?Can males compete for traditionally female jobs?Joumal of Management, 23: 603-626,

Austin, J. T,, Boyle, K. A., & Lualhati, J. C, 1998. Statis-tical conclusion validity for organizational scienceresearchers: A reviev^ . Organizational ResearchMethods, 1: 164-208,

Blau, P. M. 1977. Inequality and heterogeneity: A prim-itive theory of social structure. New York: FreePress,

Bruning, J, L,, & Kintz, B. L. 1977. Computational hand-book of statistics (2nd ed,). Glenview, IL: Scott,Foresman,

Campbell, D, T., & Fiske, D. W. 1959, Convergent anddiscriminant validation by the multitrait-multi-method matrix. Psychological Bulletin, 56: 81-105.

Coe, R., & Weinstock, I. 1984. Evaluating the manage-ment journals: A second look. Academy of Manage-ment Joumal, 27: 660-665,

Cook, T, D,, & Campbell, D. T. 1976, The design and' conduct of quasi-experiments and true experimentsin field settings. In M, D, Dunnette (Ed.), Handbookof industrial and organizational psychology: 223-336. Chicago: Rand McNally,

Cook, T. D., & Campbell, D, T, 1979. Quasi-experimentaldesign: Design and analysis issues for field set-tings. Skokie, IL: Rand McNally,

Cook, T, D., Campbell, D. T,, & Perrachio, L. 1990. Quasiexperimentation. In M, D. Dunnette & L. M. Hough

(Eds,), Handbook of industrial and organizationalpsychology: 491-576. Palo Alto, CA: ConsultingPsychologists Press.

Dubin, R, 1976. Theory building in applied areas. InM. D. Dunnette (Ed,), Handbook of industrial andorganizational psychology: 17-39. Chicago: RandMcNally,

Earley, P. C. 1986, Trust, perceived importance of praiseand criticism, and v rork performance: An examina-tion of feedback in the United States and England,Joumal of Management, 12: 457-473.

Gerstner, C, R., & Day, D. V, 1997, Meta-analytic reviewof leader-member exchange theory: Correlates andconstruct issues, Joumal of Applied Psychology, 82:827-844,

Gordon, M, E,, Slade, L, A., & Schmitt, N. 1986. The"science of the sophomore" revisited: From conjec-ture to empiricism. Academy of Management Re-view, 1: 191-207.

Hackman, J. R., & Wageman, R. 1995. Total quality man-agement: Empirical, conceptual, and practical is-sues. Administrative Science Quarterly, 40: 309-342.

Hambrick, D. C, & Mason, P. A, 1984. Upper echelons:The organization as a reflection of its top managers.Academy of Management Review, 9: 193-206,

Harrigan, K. R. 1985. Exit barriers and vertical integra-tion. Academy of Management Journal, 28: 686-697,

Hinkin, T. R. 1995. A review of scale development prac-tices in the study of organizations. Joumal of Man-agement, 21: 967-988.

Hogan, E. A. 1987. Effects of prior expectations on per-formance ratings: A longitudinal study. Academy ofManagement Joumal, 30: 354-368.

Ingram, P., & Simons, T. 1995, Institutional and resourcedependence determinants of responsiveness towork-family issues. Academy of Management Jour-nal, 38: 1466-1482.

Jick, T. D. 1979. Mixing qualitative and quantitativemethods: Triangulation in action. AdministrativeScience Quarterly, 24: 602-611.

Kram, K. E. 1983, Phases of the mentoring relationship.Academy of Management Joumal, 26: 608-625.

Lei, D., Hitt, M, A., & Bettis, R, 1996. Dynamic corecompetencies through meta-learning and strategiccontext. Joumal of Management, 22: 549-570,

Lord, F, M., & Novick, M, R. 1968, Statistical theories ofmental test scores. Reading, MA: Addison-Wesley,

McEvoy, G, M,, & Cascio, W, F. 1987. Do good or poorperformers leave? A meta-analysis of the relation-ship between performance and turnover. Academyof Management Joumal, 30: 744-762,

McGrath, J. 1982. Dilemmatics: The study of researchchoices and dilemmas. In J. E. McGrath, J. Martin, &


R. A. Kulka (Eds.), Judgment calls in research: 69-102. Newbury Park, CA: Sage.

Meltzoff, J. 1998. Critical thinking about research.Washington, DC: American Psychological Associa-tion.

Mitchell, T. R. 1985. An evaluation of the validity ofcorrelational research conducted in organizations.Academy of Management Review, 2: 192-205.

Mook, D. C. 1983. In defense of external validity. Amer-ican Psychologist, 38: 379-387.

Nunnally, J. C. 1978. Psychometric theory (2nd ed.).New York: McGrav r Hill.

Nunnally, J. C, & Bernstein, I. H. 1994. Psychometrictheory (3rd ed.) Nev f York: McGraw-Hill.

Reinganum, M. R. 1985. The effect of executive succes-sion on stockholder wealth. Administrative ScienceQuarterly, 30: 46-60.

Sackett, P. R., & Larson, J. R., Jr. 1990. Research strategiesand tactics in industrial and organizational psychol-ogy. In M. D. Dunnette & L. M. Hough (Eds.), Hand-book of industrial and organizational psychology:419-489. Palo Alto, CA: Consulting PsychologistsPress.

Sapienza, H. J., & Korsgaard, M. A. 1996. Proceduraljustice in entrepreneur-investor relations. Journal ofManagement, 39: 544-574.

Schriesheim, C. A., Powers, K. J., Scandura, T. A.,Gardiner, G. G. & Lankau, M. J. 1993. Improvingconstruct measurement in management research:Gomments and a quantitative approach for assessingthe theoretical content adequacy of paper-and-pencilsurvey-type instruments. Journal of Management,19: 385-417.

Seers, A., & Woodruff, S. 1997. Temporal pacing intaskforces: Group development or deadline pres-sure?/ourna/o/Managemenf, 23: 169-187.

Smith, H. W. 1975. Strategies of social research: Themethodological imagination. Englewood Gliffs, NJ:Prentice-Hall.

Stone, E. 1978. Research methods in organizationalbehavior. Glenview, IL: Scott, Foresman.

Stone-Romero, E. F. 1994. Gonstruct validity issues inorganizational behavior research. In J. Greenberg(Ed.), Organizational behavior: The state of thescience: 155-179. Hillsdale, NJ: Erlbaum.

Stone-Romero, E. F., Weaver, A. E., & Glenar, J. L. 1995.Trends in research design and data analytic strate-gies in organizational research. Journal of Manage-ment, 21: 141-157.

Sussman, M., & Robertson, D. U. 1986. The validity ofvalidity: An analysis of validation study designs.Journal of Applied Psychology, 71: 461-468.

Wayne, S., & Ferris, G. R. 1990. Influence tactics, affect,and exchange quality in supervisors-subordinate in-teractions: A laboratory experiment and field study.Journal of Applied Psychology, 75: 487-499.

Webb, E. J., Gampbell, D. T., Schwartz, R. D., & Sechrest,L. 1966. Unobtrusive measures: Nonreactive re-search in the social sciences. Ghicago: Rand Mc-Nally.

Terri A. Scandura received her Ph.D. from the Univer-sity of Gincinnati; she is a professor of management andpsychology at the University of Miami. Her current re-search interests are leadership, mentorship, organiza-tional justice, and research methods.

Ethtyn A. Williams earned her Ph.D. at the University ofMiami; she is an assistant professor of management andorganization at the University of Golorado at GoloradoSprings. Her current research interests include mentor-ing, teams, leadership, and research methods.

RESEARCH METHODOLOGY IN MANAGEMENT: CURRENT … › diego_prior › sites › gent.uab.cat... ·...

Documents

Transcript of RESEARCH METHODOLOGY IN MANAGEMENT: CURRENT … › diego_prior › sites › gent.uab.cat... ·...