Ensuring excellence in scholarship for the 21 century ... › sites › cochrane.de › files ›...
Transcript of Ensuring excellence in scholarship for the 21 century ... › sites › cochrane.de › files ›...
![Page 1: Ensuring excellence in scholarship for the 21 century ... › sites › cochrane.de › files › ... · Assessing Scientists (RIAS) # Key Principles 1 Contributing to societal needs](https://reader035.fdocuments.us/reader035/viewer/2022070804/5f03569a7e708231d408b885/html5/thumbnails/1.jpg)
Ensuring excellence in scholarship for the 21st century: better research and publication practice
behaviors by a broad spectrum of players
Symposium der Cochrane Deutschland Stiftung
Freiburg, Germany
1st February 2019
David Moher (twitter - @dmoher) Centre for Journalology (http://www.ohri.ca/journalology/)
Ottawa Hospital Research Institute; School of Epidemiology and Public Health, uOttawa;
ORCID: 0000-0003-2434-4206
![Page 2: Ensuring excellence in scholarship for the 21 century ... › sites › cochrane.de › files › ... · Assessing Scientists (RIAS) # Key Principles 1 Contributing to societal needs](https://reader035.fdocuments.us/reader035/viewer/2022070804/5f03569a7e708231d408b885/html5/thumbnails/2.jpg)
Disclosures of interests
• Co-editor-in-chief of Systematic Reviews and receive a small stipend from BioMed Central
• Associate Director of the International Congress on Peer Review and Scientific Publication
• Member of the SRDR Governance Board
• Executive member, REWARD Alliance
• I have no other real or perceived disclosures of interests to declare
![Page 3: Ensuring excellence in scholarship for the 21 century ... › sites › cochrane.de › files › ... · Assessing Scientists (RIAS) # Key Principles 1 Contributing to societal needs](https://reader035.fdocuments.us/reader035/viewer/2022070804/5f03569a7e708231d408b885/html5/thumbnails/3.jpg)
Take home messages
• Publication practices are not optimal yet scientists are using them to advance their careers
• Cochrane (Germany) could act as a strong advocate to improve how scientists are evaluated for career advancement – Bring evidence to the table
• Advocate for meta-research and journalology • Sign DORA • Downplay the Cochrane library’s impact factor • Advocate for stronger publication practices
![Page 4: Ensuring excellence in scholarship for the 21 century ... › sites › cochrane.de › files › ... · Assessing Scientists (RIAS) # Key Principles 1 Contributing to societal needs](https://reader035.fdocuments.us/reader035/viewer/2022070804/5f03569a7e708231d408b885/html5/thumbnails/4.jpg)
![Page 5: Ensuring excellence in scholarship for the 21 century ... › sites › cochrane.de › files › ... · Assessing Scientists (RIAS) # Key Principles 1 Contributing to societal needs](https://reader035.fdocuments.us/reader035/viewer/2022070804/5f03569a7e708231d408b885/html5/thumbnails/5.jpg)
“Our belief is that research funders, scientific societies, school and university teachers, professional medical associations, and scientific publishers (and their editors) can use this Series as an opportunity to examine more forensically why they are doing what they do...and whether they are getting the most value for the time and money invested in science.”
ww
w.re
search
waste
.net
Kleinert S, Horton R. How should medical science change? Lancet 2014; 383(9913):197–8
![Page 6: Ensuring excellence in scholarship for the 21 century ... › sites › cochrane.de › files › ... · Assessing Scientists (RIAS) # Key Principles 1 Contributing to societal needs](https://reader035.fdocuments.us/reader035/viewer/2022070804/5f03569a7e708231d408b885/html5/thumbnails/6.jpg)
Problems with biomedical publications
• There is a reproducibility crises • Publication bias is on the rise
– Statistically negative results have been eviscerated from the literature
• High prevalence of selective outcome reporting bias • High prevalence of spin • Endemic poor quality of reporting of biomedical
research • Researchers are publishing (publicly funded) research
in deceptive journals • Data sharing is uncommon
![Page 7: Ensuring excellence in scholarship for the 21 century ... › sites › cochrane.de › files › ... · Assessing Scientists (RIAS) # Key Principles 1 Contributing to societal needs](https://reader035.fdocuments.us/reader035/viewer/2022070804/5f03569a7e708231d408b885/html5/thumbnails/7.jpg)
University rankings
• Yet researchers have seen their careers advance
• Universities use traditional research metrics to advance their cause (ranking schemes; league tables)
– Advertise their ‘value’
– Recruit faculty and students
– Attract funding
![Page 8: Ensuring excellence in scholarship for the 21 century ... › sites › cochrane.de › files › ... · Assessing Scientists (RIAS) # Key Principles 1 Contributing to societal needs](https://reader035.fdocuments.us/reader035/viewer/2022070804/5f03569a7e708231d408b885/html5/thumbnails/8.jpg)
Current criteria to assess scientists
• Little evidence to inform the discussion
– What are the current criteria?
– Are the current criteria evidence-based?
– Should the criteria be evidence-based?
• I’ve not been able to identify where to examine a curated and annotated list of P&T criteria
– Nothing exists publicly
• Current criteria are not effective – the scientific literature is in a mess
Alperin, J.P. et al. Preprint https://hcommons.org/deposits/objects/hc:21016/datastreams/CONTENT/content
![Page 9: Ensuring excellence in scholarship for the 21 century ... › sites › cochrane.de › files › ... · Assessing Scientists (RIAS) # Key Principles 1 Contributing to societal needs](https://reader035.fdocuments.us/reader035/viewer/2022070804/5f03569a7e708231d408b885/html5/thumbnails/9.jpg)
170 Faculties of Medicine
• Faculties of Medicine or Biomedical Sciences were eligible for inclusion – Leiden Rankings (2017) and took a 20% random
sample (n= 170)
• When available each faculty guideline for assessing scientists for promotion and tenure was reviewed and the presence of – 5 traditional criteria
– 7 progressive criteria
Rice DB, et al. unpublished
![Page 10: Ensuring excellence in scholarship for the 21 century ... › sites › cochrane.de › files › ... · Assessing Scientists (RIAS) # Key Principles 1 Contributing to societal needs](https://reader035.fdocuments.us/reader035/viewer/2022070804/5f03569a7e708231d408b885/html5/thumbnails/10.jpg)
Traditional Incentives
1 Is any quantitative or qualitative mention made about publications required? If quantitative, please specify the requirement.
2 Is any quantitative or qualitative mention made about the specific authorship order in publications? If so, please specify order (e.g. first, senior, single) required.
3 Is any mention made of journal impact factors? If quantitative, what are the minimum thresholds?
4 Is any mention made of grant funding? If quantitative, what are the minimum thresholds (i.e., amount of funding and/or number of grants as principal investigator)?
5 Is any mention made requiring that research is recognized at a national or international level? If so, please specify the requirement.
Progressive Incentives
1 Is any mention made of citations? If quantitative, what are the thresholds of minimum requirement? Are specific citation databases mentioned?
2 Is any mention made of data sharing? If quantitative, what are the minimum thresholds (e.g., percentage of data that is to be made available)?
3 Is any mention made of publishing in open access mediums? If quantitative, what are the minimum thresholds (e.g., percentage of studies to be published in open access journals)?
4 Is any mention made of registration (including preregistration challenge) of studies? If yes, are there thresholds of minimum requirement (e.g., percentage of studies that are to be registered).
5 Is any mention made of adherence to reporting guidelines for publications? If so, are specific guidelines mentioned?
6 Is any mention made of alternative metrics for sharing research (e.g., social media and print media)? If so, are specific metrics mentioned?
7 Is any mention made of accommodations or adjustments to expectations due to extenuating circumstance? If so, please specify the description of accommodations (e.g., an extra year to defer tenure consideration) and the type of eligible circumstances (e.g., parental leave, medical leave)?
![Page 11: Ensuring excellence in scholarship for the 21 century ... › sites › cochrane.de › files › ... · Assessing Scientists (RIAS) # Key Principles 1 Contributing to societal needs](https://reader035.fdocuments.us/reader035/viewer/2022070804/5f03569a7e708231d408b885/html5/thumbnails/11.jpg)
Moher, D et al. Eur J Clin Invest. 2016 May;46(5):383-5.
Assessing scientists for hiring, promotion and tenure
![Page 12: Ensuring excellence in scholarship for the 21 century ... › sites › cochrane.de › files › ... · Assessing Scientists (RIAS) # Key Principles 1 Contributing to societal needs](https://reader035.fdocuments.us/reader035/viewer/2022070804/5f03569a7e708231d408b885/html5/thumbnails/12.jpg)
Declaration of Research Assessment (DORA)
http://www.ascb.org/dora/ Curry S. Nature. 2018 Feb 8;554(7691):147. Moher D, et al. PLOS Biology 2018 Mar 29;16(3):e2004089. doi: 10.1371
The JIF in a given year is the average number of citations to research articles in that journal over the preceding two years
![Page 13: Ensuring excellence in scholarship for the 21 century ... › sites › cochrane.de › files › ... · Assessing Scientists (RIAS) # Key Principles 1 Contributing to societal needs](https://reader035.fdocuments.us/reader035/viewer/2022070804/5f03569a7e708231d408b885/html5/thumbnails/13.jpg)
Data sharing associated with
added value
Piwowar HA, et al.(2007). PLoS ONE 2(3): e308.
Why data sharing
![Page 14: Ensuring excellence in scholarship for the 21 century ... › sites › cochrane.de › files › ... · Assessing Scientists (RIAS) # Key Principles 1 Contributing to societal needs](https://reader035.fdocuments.us/reader035/viewer/2022070804/5f03569a7e708231d408b885/html5/thumbnails/14.jpg)
N Engl J Med 2018;378:2202-11
![Page 15: Ensuring excellence in scholarship for the 21 century ... › sites › cochrane.de › files › ... · Assessing Scientists (RIAS) # Key Principles 1 Contributing to societal needs](https://reader035.fdocuments.us/reader035/viewer/2022070804/5f03569a7e708231d408b885/html5/thumbnails/15.jpg)
Traditional and Progressive
Incentives for Promotion and Tenure
0
1
2
3
4
5
Assistant Professor Associate Professor Professor Tenure
Med
ian N
um
ber
of
Cri
teri
a
Level of Evaluation
Progressive Incentives
Traditional Incentives
Rice DB, et al. unpublished
![Page 16: Ensuring excellence in scholarship for the 21 century ... › sites › cochrane.de › files › ... · Assessing Scientists (RIAS) # Key Principles 1 Contributing to societal needs](https://reader035.fdocuments.us/reader035/viewer/2022070804/5f03569a7e708231d408b885/html5/thumbnails/16.jpg)
Value to the community
![Page 17: Ensuring excellence in scholarship for the 21 century ... › sites › cochrane.de › files › ... · Assessing Scientists (RIAS) # Key Principles 1 Contributing to societal needs](https://reader035.fdocuments.us/reader035/viewer/2022070804/5f03569a7e708231d408b885/html5/thumbnails/17.jpg)
From twitter
• My paper has only been cited 21 times but the software I developed has been downloaded 4300 times
– Value to the community
![Page 18: Ensuring excellence in scholarship for the 21 century ... › sites › cochrane.de › files › ... · Assessing Scientists (RIAS) # Key Principles 1 Contributing to societal needs](https://reader035.fdocuments.us/reader035/viewer/2022070804/5f03569a7e708231d408b885/html5/thumbnails/18.jpg)
University of Ghent
2nd January 2019: http://dariuszgalasinski.com/2019/01/02/ghents-choices/
![Page 19: Ensuring excellence in scholarship for the 21 century ... › sites › cochrane.de › files › ... · Assessing Scientists (RIAS) # Key Principles 1 Contributing to societal needs](https://reader035.fdocuments.us/reader035/viewer/2022070804/5f03569a7e708231d408b885/html5/thumbnails/19.jpg)
Benedictus and Miedema Nature 2016; 538:453-454
![Page 20: Ensuring excellence in scholarship for the 21 century ... › sites › cochrane.de › files › ... · Assessing Scientists (RIAS) # Key Principles 1 Contributing to societal needs](https://reader035.fdocuments.us/reader035/viewer/2022070804/5f03569a7e708231d408b885/html5/thumbnails/20.jpg)
https://www.bihealth.org/en/quest-center/mission-approaches/
![Page 21: Ensuring excellence in scholarship for the 21 century ... › sites › cochrane.de › files › ... · Assessing Scientists (RIAS) # Key Principles 1 Contributing to societal needs](https://reader035.fdocuments.us/reader035/viewer/2022070804/5f03569a7e708231d408b885/html5/thumbnails/21.jpg)
Levers of power in science
• Journals (Cochrane) – peer review and editorial oversight of manuscripts
• Funders (Cochrane) – 14% Canadian institutes of of Health Research – 25%-30% in Germany
• Patients and the public (Cochrane) – Increasingly important
• Academic institutions – The home of current and future generations of
researchers
Sims et al. Heart. 2017 Nov 1. pii: heartjnl-2017-312165 Moher et al. BMC Med. 2017 Sep 11;15(1):167
![Page 22: Ensuring excellence in scholarship for the 21 century ... › sites › cochrane.de › files › ... · Assessing Scientists (RIAS) # Key Principles 1 Contributing to societal needs](https://reader035.fdocuments.us/reader035/viewer/2022070804/5f03569a7e708231d408b885/html5/thumbnails/22.jpg)
Data driven publication practices
• Register all protocols and make results of all research available
– Clinical and preclinical
• Stop switching primary outcomes covertly
• Promote reproducibility
• Share data, materials and code
• Use reporting guidelines
• Promote open science
Moher D, et al. PLOS Biology 2018 Mar 29;16(3):e2004089. doi: 10.1371
![Page 23: Ensuring excellence in scholarship for the 21 century ... › sites › cochrane.de › files › ... · Assessing Scientists (RIAS) # Key Principles 1 Contributing to societal needs](https://reader035.fdocuments.us/reader035/viewer/2022070804/5f03569a7e708231d408b885/html5/thumbnails/23.jpg)
Make the results of research available
Chen R, et al. BMJ. 2016 Feb 17;352:i637
![Page 24: Ensuring excellence in scholarship for the 21 century ... › sites › cochrane.de › files › ... · Assessing Scientists (RIAS) # Key Principles 1 Contributing to societal needs](https://reader035.fdocuments.us/reader035/viewer/2022070804/5f03569a7e708231d408b885/html5/thumbnails/24.jpg)
Open Science Collaboration, Science 349, aac4716 (2015). DOI: 10.1126/science.aac4716
Replicating results
![Page 25: Ensuring excellence in scholarship for the 21 century ... › sites › cochrane.de › files › ... · Assessing Scientists (RIAS) # Key Principles 1 Contributing to societal needs](https://reader035.fdocuments.us/reader035/viewer/2022070804/5f03569a7e708231d408b885/html5/thumbnails/25.jpg)
Stanford University, Hoover Institution
• Systematic review
• Delphi panel
• Face to face consensus meeting (January 2017)
– N = 22
– Multiple stakeholders
![Page 26: Ensuring excellence in scholarship for the 21 century ... › sites › cochrane.de › files › ... · Assessing Scientists (RIAS) # Key Principles 1 Contributing to societal needs](https://reader035.fdocuments.us/reader035/viewer/2022070804/5f03569a7e708231d408b885/html5/thumbnails/26.jpg)
Responsible Indicators for Assessing Scientists (RIAS)
# Key Principles
1 Contributing to societal needs is an important goal of scholarship
2 Assessing faculty should be based on responsible indicators that reflect fully the contribution to the scientific enterprise
3 We should reward publishing and/or reporting all research completely and transparently, regardless of the results
4 The culture of Open Research needs to be rewarded
5 It is important to fund research that can provide an evidence base to inform optimal ways to assess science and faculty
6 Funding out-of-the-box ideas needs to be valued in promotion and tenure decisions
Moher D, et al. (2018) Assessing scientists for hiring, promotion, and tenure. PLoS Biol 16(3): e2004089. https://doi.org/10.1371/
![Page 27: Ensuring excellence in scholarship for the 21 century ... › sites › cochrane.de › files › ... · Assessing Scientists (RIAS) # Key Principles 1 Contributing to societal needs](https://reader035.fdocuments.us/reader035/viewer/2022070804/5f03569a7e708231d408b885/html5/thumbnails/27.jpg)
Principle 2 Action
Assessing faculty should be based on responsible indicators that reflect fully the contribution to the scientific enterprise
Assessing scientists should include indicators that can incentivize best publication practices such as assessing registration (e.g. publishing the protocol, registered reports), reproducible research reporting, contributions to peer review, etc. Such indicators should be measured as objectively and accurately as possible
![Page 28: Ensuring excellence in scholarship for the 21 century ... › sites › cochrane.de › files › ... · Assessing Scientists (RIAS) # Key Principles 1 Contributing to societal needs](https://reader035.fdocuments.us/reader035/viewer/2022070804/5f03569a7e708231d408b885/html5/thumbnails/28.jpg)
Implementation • Academic institutions, funders, and journals
should explicitly endorse efforts to reduce the importance of journal impact factors when assessing faculty (e.g., DORA; Leiden manifesto) and make this known to their constituents. – German Cochrane Foundation could sign DORA
– Substantially reduce the use of Impact Factors in Cochrane library
– An explicit statement about not using these metrics when evaluating career advancement
![Page 29: Ensuring excellence in scholarship for the 21 century ... › sites › cochrane.de › files › ... · Assessing Scientists (RIAS) # Key Principles 1 Contributing to societal needs](https://reader035.fdocuments.us/reader035/viewer/2022070804/5f03569a7e708231d408b885/html5/thumbnails/29.jpg)
Implementation
• Institutions will need to prepare the landscape to ease implementation:
– To facilitate data sharing, it is likely that the FAIR (Findability, Accessibility, Interoperability and Reusability) principles will need to be in place.
– German Cochrane Foundation in collaboration with German universities prepare educational outreach about FAIR
![Page 30: Ensuring excellence in scholarship for the 21 century ... › sites › cochrane.de › files › ... · Assessing Scientists (RIAS) # Key Principles 1 Contributing to societal needs](https://reader035.fdocuments.us/reader035/viewer/2022070804/5f03569a7e708231d408b885/html5/thumbnails/30.jpg)
Implementation
• Data sharing is not without costs
– Cochrane advocate for universities to make funds available to help researchers prepare data for sharing
• Cochrane advocate for universities, particularly promotion and tenure committees, add best publication practices into their assessment criteria
– RIAS
![Page 31: Ensuring excellence in scholarship for the 21 century ... › sites › cochrane.de › files › ... · Assessing Scientists (RIAS) # Key Principles 1 Contributing to societal needs](https://reader035.fdocuments.us/reader035/viewer/2022070804/5f03569a7e708231d408b885/html5/thumbnails/31.jpg)
Experiment with CVs to reward best publication practices
Steven Beyer Roberts http://faculty.washington.edu/sr320
![Page 32: Ensuring excellence in scholarship for the 21 century ... › sites › cochrane.de › files › ... · Assessing Scientists (RIAS) # Key Principles 1 Contributing to societal needs](https://reader035.fdocuments.us/reader035/viewer/2022070804/5f03569a7e708231d408b885/html5/thumbnails/32.jpg)
European Union. Evaluation of Research Careers fully acknowledging Open Science Practices Rewards, incentives and/or recognition for researchers practicing Open Science, 2017. https://bit.ly/2HmMkK8
![Page 33: Ensuring excellence in scholarship for the 21 century ... › sites › cochrane.de › files › ... · Assessing Scientists (RIAS) # Key Principles 1 Contributing to societal needs](https://reader035.fdocuments.us/reader035/viewer/2022070804/5f03569a7e708231d408b885/html5/thumbnails/33.jpg)
Could Cochrane more strongly advocate for:
• Reproducibility
• Data sharing
• Better publication practices
• Using different criteria when assessing scientists for career advancement
• Experimenting with CVs
![Page 34: Ensuring excellence in scholarship for the 21 century ... › sites › cochrane.de › files › ... · Assessing Scientists (RIAS) # Key Principles 1 Contributing to societal needs](https://reader035.fdocuments.us/reader035/viewer/2022070804/5f03569a7e708231d408b885/html5/thumbnails/34.jpg)
Thank you