Symposium on

The Impact of Linguistics Journal Rankings and Citations

Organized by members of the Committee of Editors of Linguistics Journals

Outline

• Brian Joseph (The Ohio State University): The editor’s perspective

• Tim Stowell (University of California, Los Angeles): The administrator’s perspective

• John Cullars (University of Illinois at Chicago): The bibliometrician’s perspective

• Marian Hollingsworth (Thompson Scientific): The industry perspective

• Keren Rice (University of Toronto): Summation• Martha Ratliff (Wayne State University), Discussant

Rossner, M. et al. 2007. Show me the data.

J. of Cell Biology.

… Impact factor data … have a strong influence on the scientific community, affecting decisions on where to publish, whom to promote or hire, the success of grant applications, and even salary bonuses. Yet, members of the community seem to have little understanding of how impact factors are determined … .

What kind of material is there?Bare lists of journals implicitly or explicitly deemed ‘major’: • DeMiller, Anna L. 2000. Linguistics. A guide to the reference

literature (2nd edn.). Englewood: Libraries Unlimited. (on-line)

Citation indexes:• Thomson Scientific, Web of Science, “impact factor”• www.journal-ranking.com (on-line, interactive)

Other efforts to evaluate, rate or rank:• European Science Foundation, “European Reference Index for the

Humanities”, here. [A, B, C assigned to each journal.]• Rolf A. Zwaan & Anton J. Nederhof. 1990. Some aspects of

scholarly communication in linguistics: An empirical study. Language 66.3.553-57. [Survey data from Dutch linguists.]

• Other academic bibliometric work like Georgas and Cullars.

The citations approach

“Citations can be counted, and one way of measuring the quality of a publication to assess whether it appears in a high-citation outlet. … The chain of inference … runs as follows. The more papers in the journal are cited, the more impact that journal has. The more impact a journal has, the more authors will want to publish in that journal. …

Continued …“The more authors who want to publish in the journal, the more demanding will be the selection criteria applied in the refereeing process. The more demanding the selection criteria applied in the refereeing process, the better the average paper will be. The better the average paper in the journal, the more it will be cited. And so a virtuous circle is completed.”

—British Academy Report

An alternative• “The ERIH lists will help identify excellence in

Humanities scholarship and should prove useful for the aggregate benchmarking of national research systems, for example, in determining the international standing of the research activity carried out in a given field in a particular country. As they stand, the lists are not a bibliometric tool. The ERIH Steering Committee and the Expert Panels therefore advise against using the lists as the only basis for assessment.”

Thomson journal-ranking.com

Reactions & meta-evaluation• Wouter Gerritsma, Citation analysis for research evaluation.

[here]• Equivalence of results from two citation analyses: Thomson

ISI’s Citation Index and Google’s Scholar service. Pauly & Stergiou. Inter-Research 2005. (int-res.com)

• Various (negative) editorials in Nature.• Peer Review: the challenges for the humanities and social

sciences. A British Academy Report, Sept. 2007.• Show me the data. Mike Rossner, Heather Van Epps, and

Emma Hill. Journal of Cell Biology 2007.• Chronicle of Higher Education, 10 October 2008: “New

Ratings of Humanities Journals Do More Than Rank — They Rankle” (and earlier articles there.)

Evaluitis — Eine neue KrankheitBruno S. Frey, University of Zurich, 2006

"Evaluitis" - i.e. ex post assessments of organizations and persons - has become a rapidly spreading disease. In addition to the well-known costs imposed on evaluees and evaluators, additional significant costs are commonly disregarded: incentives are distorted, ossification is induced and the decision approach is wrongly conceived. As a result, evaluations are used too often and too intensively.

But really, a good system could …

• encourage rigor, • establish standards, • and set a tone.

More fundamental problems

• Limited or problematic data and measures• Poorly defined standards

Worse:• Results being put to uses that weren’t

intended or aren’t justified?

An example: Rejection rates

No standard and no way to verify, yet numbers are widely quoted. Two dramatically different views:

•#1: Every editorial decision = submission.•#2: Don’t count submissions not sent out for review, or revisions of earlier submissions.

We (ELJ) are working to establish a norm for linguistics.

Time is tight, so let’s get going …

These slides are available at …http://groups.google.com/group/editors-of-linguistics-journals.

We plan to produce and post a white paper based on this session … stay tuned.

Our original questions1. Philosophical• Why is ‘objective’ ranking of journals considered important? • Why have organizations like Thomson ISI and Google Scholar become

important within academia? • What are the arguments for and against rankings and impact factors?

2. Practical• Can measurement of journal impact and rank be done objectively? • Is this being done within linguistics in ways that are valuable? • Is this being done within linguistics in ways that might threaten the vitality

of research and of journals?

3. Professional• What role does the assessment of journal impact and rank actually play in

grantsmanship, promotion, tenure and hiring decisions, library acquisitions, etc.?