El i zabeth Langdon-GrayAss i s tan t Provos t f o r Research Deve lopment and P l ann i ng

October 4, 2013

Responsible Conduct of Research Peer Review

Responsible Authorship and Publication

CORNERSTONE OF REVIEW OF SCIENTIFIC PROPOSALS AND PUBLICATIONS

Peer Review

Peer Review – Proposals

Background

NIH peer review mandated by statute: "Scientific Peer Review of Research Grant Applications and Research and Development Contract Projects" (42 CFR Part 52h)

“NIH policy is intended to ensure that grant applications submitted to the NIH are evaluated on the basis of a process that is fair, equitable, timely, and free of bias.” (http://grants.nih.gov/grants/peer_review_process.htm )

NSF “Merit Review” (http://www.nsf.gov/bfa/dias/policy/meritreview/)

Study sections consider merits of proposal + public health needs + goals of the agency

Peer review of proposals vs peer review of articles = evaluation of “competitive position” vs evaluation of “intrinsic soundness”

Lenard, J., “Two facets of peer review and the proper role of study sections”, Accountability in Research, 13:277-283, 2006

Peer Review – Proposals

Criticisms

Enormous administrative burdenStifling of innovation

Complaint that reviewers tend to underscore that which they do not understand = more conservative applications

Exacerbated by more competitive funding environment Increasingly collaborative research landscape not always

matched by expertise of reviewers Bias, conflicts of interest, plagiarism: obligation

of reviewers to maintain integrity of process (those best able to judge are often in direct competition)

National Institutes of Health 2007-2008 Peer Review Self-Study – Final Draft, 2-29-2008, http://enhancing-peer-review.nih.gov/meetings/NIHPeerReviewReportFINALDRAFT.pdf

Alternatives

Ideation Challenge on Diabetes• http://hms.harvard.edu/news/harvard-medicine/

ideation-challenge-diabetes• Griffin Weber, Harvard Catalyst, social

networking analysis

Crowdsourced ideas

Challenged traditional review paradigm• The more expertise, the lower the reviewer will

rate impact.

Potential Impact = Responsibility

Make or break reputationsInfluence public policyEnsure the success or failure of research

programsAffect public health initiatives; environmental

and safety regulationsRESPONSIBILITY to be:

Timely Free from biasConstructive Thorough

Respectful of the need for confidentiality

Peer Review – Publications

History

Pre-peer review, practice of keeping results secret so others could not take credit – undermines common purpose of scientific endeavor

Peer review of journal articles uncommon until the mid-20th century.

Tied to importance of publications: Allows for prioritization of articles (elite journals seen as

indicator of quality of science) Publication major factor in hiring, promotion, reputation,

recognitionCasadevall, A. and Ferric, Fang C., “Is Peer Review Censorship?”, INFECTION AND IMMUNITY, April, 2009, p.1273-1274

Peer Review – Publications

Obligations

Obligation to Review Contribution to science and science as a collaborative endeavor

Obligations of the Reviewer Broad effects of review – on progress of science, careers and reputations of authors, influence on public policy Timeliness Objectivity Sensitivity to potential/perceived conflicts of interest Preservation of confidentiality (e.g. insider trading)

Obligations of the Journal Pick the right reviewers Provide clear guidance Provide recognition (most reviews uncompensated = need for other forms of

recognition)

Groves, T. “Quality and value: How can we get the best out of peer review? A recipe for good peer review”, Nature (2006), doi:10.1038/nature04995

Peer Review – Publications

Background

The review process is widely used in:Journal articlesMulti-authored booksCongress proceedingsBook series with invited articlesScientific meetings (poster/oral presentations)

See: Johannes Schramm, Some Thoughts on the Review Process, World Neurosurgery, Volume 76, Issues 1-2, July-August 2011, Pages 43-44, ISSN 1878-8750, 10.1016/j.wneu.2011.05.038.

Publication and authorship practices vary by field

Peer Review – Publications

Review Criteria

From The British Medical Journal’s “Guidance for Peer Reviewers” (http://resources.bmj.com/bmj/reviewers/peer-reviewers-guidance)

For all articles:Is the article important? Will it help our readers to make better decisions and, if so, how? Will the article add enough to existing knowledge? Does the article read well and make sense? Does it have a clear message?

For research articles please comment on: Originality — does the work add enough to what is already in the published literature? If so, what does it

add? Please cite relevant references to support your comments on originality. Importance of the work to general readers — does this work matter to clinicians, researchers,

policymakers, educators, or patients? Will it help our readers to make better decisions and, if so, how?  Is a general medical journal the right place for it?

Scientific reliability Research question — clearly defined and appropriately answered? Overall design of study — appropriate and adequate to answer the research question? Participants — adequately described, their conditions defined, inclusion and exclusion criteria described? How representative

were they of patients whom this evidence might affect? Methods — adequately described? Main outcome measure clear? For randomised trials, systematic reviews, observational studies,

health economics studies - reported in line with the appropriate reporting statement or checklist (see below)? Was the study ethical (this may go beyond simply whether the study was approved by an ethics committee or IRB)?

Results — answer the research question? Credible? Well presented? Interpretation and conclusions — warranted by and sufficiently derived from/focused on the data? Discussed in the light of

previous evidence? Message clear? References — up to date and relevant? Any glaring omissions? Abstract/summary/key messages/what this paper adds — reflect accurately what the paper says? Documents in the supplemental files eg checklists for these statements CONSORT, PRISMA, MOOSE, STROBE; the BMJ health

economics checklist; and the protocol for an RCT - do these contain information that should be better reported in the manuscript, or raise questions about the work?

Peer Review – Publications

Criticisms and Shortcomings

…“neither Newton or Darwin had to submit to the indignity of peer review!”…

Casadevall, A. and Ferric, Fang C., “Is Peer Review Censorship?”, INFECTION AND IMMUNITY, April, 2009, p.1273-1274

Claims of “ethical transgressions” e.g. Breach of confidentiality Theft of ideas Insider trading Personal attacks Bias / conflicts of interest

Resnik, D.B., Gutierrez-Ford, C., Peddada, S., “Perceptions of Ethical Problems with Scientific Journal Peer Review: An Exploratory Study”, Sci Eng Ethics, 2008 September; 14(3): 305-310.

Failure to detect errors and fraud

Peer review is “an inherently conservative process … [that] … encourages the emergence of self-serving cliques of reviewers”

- UK Parliamentary Office of Science and Technology

…“it’s a lousy system, but it’s the best one we have”…

What could go wrong?

“Blame lies with those journals … that allow authors to nominate their own reviewers and don’t check credentials and contacts.”

“In incidents involving four scientists … journal editors say authors got to critique their own papers by suggesting reviewers with contact emails that actually went to themselves.”

Peer Review – Publications

Open Peer Review – A Panacea?

Reviewers’ names and affiliations made available to authors

Reviewers asked to declare any potential conflicts of interest

Should authors (and readers) see reviewers comments?

Should readers themselves be able to comment?New models being tested at journals

See Nature’s archive of articles on the peer-review debate: http://www.nature.com/nature/peerreview/debate/index.html

Peer Review – Publications

The Role of Technology

Open peer review made easier by internetAllows for dialog between authors, reviewers, editors

and readers; post-publication commentary

Open peer review cousin to:Open access

Databases allow for sharing of raw data Open access to journal articles more prevalent NIH policy “requires scientists to submit final peer-reviewed

journal manuscripts that arise from NIH funds to the digital archive PubMed Central upon acceptance for publication.”

http://publicaccess.nih.gov/

Open access at HU promoted and facilitated by Office for Scholarly Communication (http://osc.hul.harvard.edu/)

Open Access: the end of Peer Review?

PLoS One• Limited Peer Review (Has the

science in this paper been done well enough to warrant it being entered into the scientific literature as a whole?”) +

• Post-publication open comment

ALLOCATING CREDITASSIGNING RESPONSIBILITY

AVOIDING CONFLICT

S E E H A RVA R D W O R K S H O P F O R R E SO U R C E S : H TT P: / / P R O J E C T S. I Q. H A RVA R D. E D U / ATT R I B U T IO N _ W O R K S H O P

Authorship

Authorship

List of authors = apportionment of credit (and accountability)

Credit in publications critical to career advancement, reputation, recognition

Authorship conventions (first, last, second etc.) vary by discipline and research group

Problems: ghost-writing, honorary authors, allocation of credit and responsibility

Proliferation of issues with increased collaboration/multi-site studies

Authorship as Code for Contribution

"If scientists want to convey this information by the way their names are ordered, the method is similar to sending smoke signals, in code, on a dark, windy night." --Drummond Rennie, Deputy Editor, JAMA

Putting Authorship Issues in ContextCassandra Extavour, OEB, Harvard

Putting Authorship Issues in ContextCassandra Extavour, OEB, Harvard

Putting Authorship Issues in ContextCassandra Extavour, OEB, Harvard

Authorship

Case Study 1

2002 J. Hendrik Schön – Physicist at Bell Labs – author or co-author of more than 90 scientific papers

Investigative panel concluded scientific misconduct in 17 of the 24 allegations stemming from examination of 25 papers

“Except for the provision of starting materials by others, all device fabrication, physical measurement and data processing in the work in question were carried out (with minor exceptions) by Hendrik Schön alone, with no participation by any coauthor or other colleague. None of the most significant physical results was witnessed by any coauthor or other colleague.”

“In addition to addressing the question of scientific misconduct, the Committee also addressed the question whether the coauthors of Hendrik Schön exercised appropriate professional responsibility in ensuring the validity of data and physical claims in the papers in question. By virtue of their coauthorship, they implicitly endorse the validity of the work. There is no implication here of scientific misconduct; the issue is one of professional responsibility.”

September, 2002 Report of the Investigative Committee on the Possibility of Scientific Misconduct in the Work of Hendrik Schön and

Coauthors: http://publish.aps.org/reports/lucentrep.pdf

Authorship

Case Study 2

Dr. Gerald Schatten – claimed senior authorship on one of two papers retracted by Woo Suk Hwang in South Korea claiming generation of embryonic stem cells by somatic cell nuclear transfer.

But when work was found to be fraudulent, Schatten “attempted to distance himself from it”

Admitted to have helped write the paper, but not to have participated in the experiments, and not to have interacted with other collaborators/authors

No evidence of misconduct, but Schatten found guilty of “scientific misbehavior” by University of Pittsburgh investigation.

Authorship was acceptable, based on his work on paper, but not senior authorship and co-corresponding author status

Highlights linked notions of credit and responsibility.

“Dr. Schatten’s listing as the last author not only conferred considerable credibility to the paper within the international scientific community, but directly benefitted Dr. Schatten in numerous ways, including enhancement of his scientific reputation, improved opportunities for additional research funding, enhanced positioning for pending patent applications, and considerable personal financial benefit.

“University of Pittsburgh investigators also examined a Nature paper coauthored by Hwang and Schatten that described the cloning of a dog. The so-called “Snuppy paper” was not fraudulent, but the investigating committee questioned Schatten’s assumption of coauthorship since his only contribution was to suggest “that a professional photographer be engaged so that Snuppy would appear with greater visual appeal.”

Strange, K., Authorship: why not just toss a coin? American Journal of Physiology – Cell Physiology 295: C567-C575, 208

Authorship

ICMJE Standards

International Committee of Medical Journal Editors enacted new guidelines at its May, 2000 meeting in Copenhagen: Uniform Requirements for Manuscripts Submitted to Biomedical Journals

“Authorship credit should be based on1. Substantial contributions to conception and design, acquisition

of data, or analysis and interpretation of data;2. Drafting the article or revising it critically for important

intellectual content; and3. Final approval of the version to be published.Authors should meet conditions 1, 2 and 3.”

www.icmje.org/ethical_1author.htmlSome journals require that authors indicate specific contributions

Authorship

Avoiding Problems / Resolving Disagreements

Some suggestions…

Discuss authorship and allocation of credit at the beginning of the project – in writing, if possible

Continue discussion throughout projectAvoid superficial collaborationsCommunicateSeek arbitration / guidance from third party /

institutional committeeStrange, K., Authorship: why not just toss a coin? American Journal of Physiology – Cell Physiology 295:

C567-C575, 208

Framework for Awkward Conversations

Criteria for Authorship: Stephen M. KosslynIdea (250 points)Design (100 points)Implementation (100 points)Conducting the experiment (100 points)Data analysis (200 points)Writing (250 points)

One Taxonomy of Authorship OrderCassandra Extavour, OEB, Harvard University

First Author: Carried out bulk of experimental / analytical work; usually grad student or postdoc

Second Author: Likely did experimental or analytical work; usually a student

Other Middle Author: could have contributed in any way, how is unclear; less experimental / analytical than 1st, 2nd

Last Author: Senior Author = PI; Major contributor of intellectual content and funding