Library & Information Science Research 32 (2010) 13–15

Contents lists available at ScienceDirect

Library & Information Science Research

Some thoughts on peer review

Candy Schwartz, Peter HernonGraduate School of Library I information Science, Simmons College, 300 the Fenway, Boston, MA 02115-5898, USA

From time to time, we like to invite the members of our Board ofEditors to comment on a matter of current interest, either related topublishing or to research in library and information science. This timewe took the theme of peer review, the focus of “The manuscriptreviewing process,” by Lutz Bornmann and Hans-Dieter Daniel,published in this issue. We sent a preprint of the article to theBoard, along with a series of questions, and a number of our Boardcolleagues were gracious enough to share their thoughts in responseto a series of questions.

Q. Bornmann and Daniel say “The acceptance of a hoax article bythe Open Information Science Journal in June 2009 after the publishersaid it had been peer-reviewed (Gilbert, 2009) shows that muchmoreresearch on the black box of peer review is needed.”Do you agree, andif so, what kinds of research do you think would be useful/important?

Hazel Hall: Yes, more research is needed, not least becausewithout further research we do not know whether the process isin fact “broken.” It is also a very interesting topic, and theoutcomes of this kind of research may well help authors preparehigh quality submissions.

Jeffrey Pomerantz: Yes, more research on the black box isneeded. This might include

research into what parts of the process can be made moreefficient with the application of the right kind of tools;research into the criteria that reviewers use when writingreviews, which can inform how editors frame the reviewingtask, and maybe speed it up (this is most likely to be field- oreven journal-specific; andresearch into the failed cases—for example, the Open Informa-tion Science Journal case—to identify what went wrong and seehow it can be avoided in future.

Reijo Savolainen: This is anunder-researched topic. To open theblackbox, there is a need for detailed qualitative studies in particular.It would be useful to scrutinize the critical points of the peer-reviewprocess, for example, theways inwhich editors and reviewers applythe decision rules in ambiguous cases. In this regard, comparativestudies across journals in diverse fields would be particularlyinteresting. To my knowledge, the submission of hoax articles isvery rare indeed. Normally, researchers will not risk their academiccareers like this. On the other hand, the “peer review test” conductedby Philip Davis and his colleague revealed gaps of which we shouldbe aware of.

E-mail addresses: candy.schwartz@simmons.edu (C. Schwartz),peter.herenon@simmon.edu (P. Hernon).

0740-8188/$ – see front matter © 2009 Published by Elsevier Inc.doi:10.1016/j.lisr.2009.12.003

Gary Burnett: Acceptance of the hoax article is not necessarilya sign that peer review is broken. In fact, it is pretty obvious thatthe hoax article was not just a hoax; while it was, from the point ofview of pure empirical research, a fake, it also, clearly, captured (ortapped into) a host of other relevant aspects of scholarlyproduction, such as rhetorical stance, crafting of prose, “tone,”etc. So it is not simply a sign of a problem that must be solved, butrather a phenomenon at the nexus of a whole set of importantquestions including the following:

What is the relationship between what we call “data” andrhetoric?To what extent is empirical research a matter of rhetoric,presentation, and socially constructed disciplinary normsrather than something that can be more objectively measured?To what extent is the value of research rooted in objectivelyverifiable standards of “science”?If the goal of scholarly publication is to further the dialog andinvestigation at the core of a discipline, why can't an article thatis a “hoax” be a positive thing, since it can (as happened in thiscase) raise such questions?

I consider myself to be, to some extent, a post-modernist, and Iam quite skeptical in many ways about things like “truth,”“objectivity,” “empiricism,” etc. And I find the phenomenon ofthe “hoax” article to be fascinating in that regard well beyond theissue of whether it's the sign of a problem in the process.

Michelle Kazmer: The idea of the “worth” of the researchbrings up another issue that may be more acute for qualitativeresearch than for other types, but that is a key component of theblack box of peer review: the editors' selection of reviewers foreach manuscript. Every discipline and research tradition hastensions (interpersonal, historical, disciplinary, methodological,etc.) which mean that the selection of specific reviewers for anygivenmanuscript can strongly affect the article's final decision. Theselection of reviewers in particular for qualitative research meansfinding reviewers who can be objective about theories andmethods that may be perceived to be subjective. Selectingreviewers who are hostile to qualitative research in principle ispossible and may be done inadvertently, but can mean legitimatequalitative work does not receive the distribution of which it isworthy. In the end, though, it is clear that editors' selection ofreviewers, while it may be putatively based on each reviewer'sfield of expertise and its match to the topic of the manuscript, is animportant part of the peer review black box in general thatrequires further research exploration.

14 C. Schwartz, P. Hernon / Library & Information Science Research 32 (2010) 13–15

Q. Do you think peer review is broken, and if so, why? Can it befixed?

Pomerantz: I do not think peer review is broken. In fact, I thinkit is one of the few aspects of scholarly publication that is notbroken. There are things about peer review that could beimproved, certainly: it is too slow, it misses things (e.g., the hoaxarticle in Open Information Science Journal), reviewers spend a lotof time for little reward. But on the whole, it accomplishes whateven Bornmann and Daniel say is its objective: “to ensure that thevalid article is accepted, the messy article improved, and theinvalid article rejected.” Maybe there are mechanisms that canaccomplish this better and more efficiently... but many smartpeople have been thinking about this for a long time, so I aminclined to doubt it. I think peer review can be fixed around theedges maybe, but I believe the fundamentals will persist.

Hall: We do not know for certain because not enough researchhas been done. However, from a practical point of view, I do thinkthat something needs to be done to help speed up the reviewprocess. I get the impression that editors have great difficulties insharing out the work of peer review, and it ends up involving somany people that it results in long time delays between thewriting up of research and its dissemination. I am often surprisedat how different journals make different demands of theirreviewers. For example, some expect long reports (like those Iwrite for LISR). Others simply ask for the reviewer to tick boxes in areview form, and the editors are quite surprised to receive a fullwritten report.

Burnett: I do see a continuing value in the peer-reviewprocess, since it at least brings a greater or lesser degree ofstructure and consistency to the process of evaluating research.It is difficult to imagine a truly useful process of “open” peerreview, particularly since one can easily imagine that those whowould opt to participate in such a process might tend to be self-selected. That is, while someone like me can play a part in peer-review processes in either “closed” or “open” settings, in thecurrent structure my participation is mediated by others—someone other than me (e.g., a journal editor) has looked atmy work and decided that I might have something valuable toadd to the process, while in “open” settings, I would simply beable declare myself to be an appropriate reviewer. That is aproblem from both ethical and practical standpoints.

Savolainen: The peer review system still functions relativelywell. The editors perform laudable work in screening the manu-scripts before they go to the review process. Most reviewers havehigh ethical standards and they try their best, although today'sincreasingly busy academic climate may be reflected in the poorerquality of reviews. Many reviewers do not simply have enoughtime to scrutinize the manuscripts in sufficient detail and provideconstructive feedback about how to improve the draft.

Kazmer: Examining the process of peer review necessitatesalso looking at peer review for disciplines outside the sciencesand research traditions other than quantitative. In particular, Ihope to draw our attention to further questions that emergewhen thinking about peer review for empirical qualitativeresearch reports. The purpose of peer review for qualitativeresearch can be less clear than described in the article becausethe data collection and analysis methods are often more difficultto assess using standards of validity and replicability thatreviewers may “borrow” from quantitative traditions. The useof theory in qualitative research—which can vary from a way todesign the research questions to a way to explain the researchfindings, or may involve building theory using grounded theorymethods—further complicates the determination by a peerreviewer of the worth of the research. The use of review formsthat include questions originally based on assessment ofquantitative research also affects the review process for

qualitative research because the bad fit can lead to unclear oreven inaccurate recommendations.

Mike Thelwall: I think that the main issue facing newresearchers is choosing an appropriate venue for their research—one that has effective peer review. Since peer review is nottransparent, new researchers would probably have to rely on thejudgments of experienced researchers about what the goodjournals are. This article would be useful reading for newresearchers to help them to understand the peer review systemand particularly that being asked to review a manuscript carrieswith it a good chance of eventual acceptance. Bentham Science isa real problem, though, as are some conferences that appear tobe fake.Q. Do you envision peer review changing in an increasingly open-

access era, and if so, how?Hall: Possibly, but I suspect this will happen through creep

rather than major changes.Savolainen: Open-access era may encourage researchers to

submit “semi-finished” manuscripts because they believe that OAjournals apply less rigorous criteria in the review process.However, the best OA journals do not differ from the traditionalones with regard to the evaluation criteria applied in the reviewprocess. The example of the two-stage publication adopted byAtmospheric Chemistry and Physics is intriguing. However, I am alittle bit skeptical about the added value obtained from thecomments of “interested members of the scientific community.” Inthe end, would the most qualified members of such communityhave time to assess the working papers submitted on the journal'sWeb site?Q. What do you think of the concept of open peer review, both

from an ethical and practical standpoint?Burnett: I think that the current peer-review process keeps

some interesting work out of reach, simply because such worknever gets published. So I also like the idea of venues in whichinteresting work that does not make it through the peer-reviewprocess can see the light of day and further the dialog of adiscipline. But I don't necessarily see “open” vs. “closed” as aneither/or proposition—I think there might be room for bothapproaches.

Pomerantz: I approve of the idea of open peer review. Iwould like to see peer review open up in two ways, perhaps notuniversally for every journal, but in some places. First, we shouldabandon blindness. Most reviewers can probably guess atauthors more than half the time, making double-blindnessfunctionally only single-blindness. That asymmetry seems unfair.Second, we should introduce public review, perhaps after peerreview. If social media have taught us anything, it is that there isa lot of knowledge lying untapped in the general public and thatsome people will participate in any project. Assuming somemechanism for filtering spam, public review would, I suspect,add value to the review process. Some journals have begun toexperiment with public review, and in an increasingly open-access era, more journals are likely to follow suit.

Hall: There are potential advantages and disadvantages.Advantages are that we would have a wider pool of reviewers,and if these are from the boundaries of our subject domain,there is the potential for new knowledge creation at the point ofredrafting work. One major disadvantage is verifying that peoplein the larger pool are genuine “peers.” If review is to be “blind,” Iprefer the double-blind process; I consider the single-blindapproach (in which the reviewer know who the authors are) tobe rather embarrassing. On a related topic, journals frequentlycite rejection rates as one of several indicators of quality, but thisdoes not acknowledge that authors tend to know the value oftheir work and are therefore self-selecting when it comes tosending in manuscripts.

15C. Schwartz, P. Hernon / Library & Information Science Research 32 (2010) 13–15

Savolainen: Open peer review (as opposed to blind review)may encourage reviewers to provide more constructive feedback.In addition, critique would be based on more detailed argu-ments. On the other hand, reviewers might be more reluctant tosuggest rejection of a manuscript, in fear of “pay back” in thefuture. It might be better to remain with the anonymous blindreview practice.Q. How, if at all, has your experience as a peer reviewer changed

your attitude or behavior as an author?Savolainen: While refining a manuscript I find that I am trying

to anticipate critical questions posed by the reviewers (forexample, the formulation of research questions, the explicationof the methodology, and so on). I ask myself how I would assess amanuscript like this if I were invited to review it.

Burnett: What I've learned from working as a peer reviewer(reviewing actively for at least 5 journals, including one somewhatoutside LIS) has served mostly to clarify my understanding of theprocess involved and to providemewith practical input about howto go about “crafting” my own work in terms of rhetoric andpresentation, if not always in terms of actual content. It has alsogiven me plenty to talk about with our doctoral students, forwhom the process is still completely a “black box”—I consistentlytry to demystify the process for them.

Hall: It has changed quite a lot. Seeing the standard of somematerial that I am asked to review, I have more confidence in

the value of my own work. I am more critical of my own workas it takes form, and less tolerant of co-authors' sloppiness. I paymuch closer attention to the requirements of the journals towhich I submit than previously, even if I disagree with theinstructions. For example, I recently submitted a paper to ajournal that indicates that the methods deployed in anyempirical study should be critically evaluated in a section oftext before the description of study results and their discussion.In my recent paper this looks very strange, yet I have followedthe instruction. I have discovered that I am quite a hard personwhen it comes to evaluating the work of others (some mightsay that my standards are high; others would say that I amunnecessarily fussy). When I have completed a review, it is mypractice to do a final check to be certain that the work has notbeen published elsewhere. In the past 3 years, on two occasions,I have discovered that the time that I have dedicated to thereview process has been wasted since the work has alreadybeen published.Thanks to our colleagues for taking the time to think about these

issues and share their insights. We invite readers to do the same,and if we receive sufficient comments, we will share them in asimilar manner. The article by Bornmann and Daniel will be publiclyavailable on the journal Web site for a year, as will thiscommentary, as part the sample issue.1 Comments may be sent toeither of the co-editors.

1 http://www.elsevier.com/wps/find/journaldescription.cws_home/620211/description.