Enhancing Peer Review - DPCPSI · 11/16/2009 · toni scarpa [email protected] 301-435-1109....
Transcript of Enhancing Peer Review - DPCPSI · 11/16/2009 · toni scarpa [email protected] 301-435-1109....
![Page 1: Enhancing Peer Review - DPCPSI · 11/16/2009 · toni scarpa scarpat@csr.nih.gov 301-435-1109. Council of Councils. Bethesda, MD. November 16, 2009. National Institutes of Health.](https://reader035.fdocuments.us/reader035/viewer/2022071416/6113e390025adf77c5748ae1/html5/thumbnails/1.jpg)
Enhancing Peer Review
toni scarpa [email protected]
Council of CouncilsBethesda, MDNovember 16, 2009
National Institutes of HealthU.S. Department of Health and Human Services
![Page 2: Enhancing Peer Review - DPCPSI · 11/16/2009 · toni scarpa scarpat@csr.nih.gov 301-435-1109. Council of Councils. Bethesda, MD. November 16, 2009. National Institutes of Health.](https://reader035.fdocuments.us/reader035/viewer/2022071416/6113e390025adf77c5748ae1/html5/thumbnails/2.jpg)
NIH Peer Review
Analysis of TR01 Review
Enhancing Peer Review
Review of ARRA Applications
![Page 3: Enhancing Peer Review - DPCPSI · 11/16/2009 · toni scarpa scarpat@csr.nih.gov 301-435-1109. Council of Councils. Bethesda, MD. November 16, 2009. National Institutes of Health.](https://reader035.fdocuments.us/reader035/viewer/2022071416/6113e390025adf77c5748ae1/html5/thumbnails/3.jpg)
Review Highly Transformative Research
• OD Transformative RO1 (T-RO1)Awards once a year, funding for 5 years8-page application740 submitted, 720 reviewed42 funded ($ 32 million)
• Editorial Board ReviewInitial scoring based on innovation and potential science transformation by a small study section of distinguished, broad-science reviewers (the editors)Specific science reviewed by appropriate reviewers (subject experts-the editorial board)Final ranking by the editors
![Page 4: Enhancing Peer Review - DPCPSI · 11/16/2009 · toni scarpa scarpat@csr.nih.gov 301-435-1109. Council of Councils. Bethesda, MD. November 16, 2009. National Institutes of Health.](https://reader035.fdocuments.us/reader035/viewer/2022071416/6113e390025adf77c5748ae1/html5/thumbnails/4.jpg)
Scoring
720 TR01 Applications, 1-9 Scale
![Page 5: Enhancing Peer Review - DPCPSI · 11/16/2009 · toni scarpa scarpat@csr.nih.gov 301-435-1109. Council of Councils. Bethesda, MD. November 16, 2009. National Institutes of Health.](https://reader035.fdocuments.us/reader035/viewer/2022071416/6113e390025adf77c5748ae1/html5/thumbnails/5.jpg)
John Bowers
![Page 6: Enhancing Peer Review - DPCPSI · 11/16/2009 · toni scarpa scarpat@csr.nih.gov 301-435-1109. Council of Councils. Bethesda, MD. November 16, 2009. National Institutes of Health.](https://reader035.fdocuments.us/reader035/viewer/2022071416/6113e390025adf77c5748ae1/html5/thumbnails/6.jpg)
The EditorsPercentage of Stage One reviewers who thought it reasonable to review about 300 applications for Stage One of three stage review process
![Page 7: Enhancing Peer Review - DPCPSI · 11/16/2009 · toni scarpa scarpat@csr.nih.gov 301-435-1109. Council of Councils. Bethesda, MD. November 16, 2009. National Institutes of Health.](https://reader035.fdocuments.us/reader035/viewer/2022071416/6113e390025adf77c5748ae1/html5/thumbnails/7.jpg)
The Editors Of applications reviewed, what percentage understood the goals of the Transformative R01 RFA?
![Page 8: Enhancing Peer Review - DPCPSI · 11/16/2009 · toni scarpa scarpat@csr.nih.gov 301-435-1109. Council of Councils. Bethesda, MD. November 16, 2009. National Institutes of Health.](https://reader035.fdocuments.us/reader035/viewer/2022071416/6113e390025adf77c5748ae1/html5/thumbnails/8.jpg)
The Editors Percentage of applications capable of transforming science - Did we get what we hoped for?
![Page 9: Enhancing Peer Review - DPCPSI · 11/16/2009 · toni scarpa scarpat@csr.nih.gov 301-435-1109. Council of Councils. Bethesda, MD. November 16, 2009. National Institutes of Health.](https://reader035.fdocuments.us/reader035/viewer/2022071416/6113e390025adf77c5748ae1/html5/thumbnails/9.jpg)
The Editors How often did Stage 2 reviewer comments dramatically change reviewer initial assessments?
![Page 10: Enhancing Peer Review - DPCPSI · 11/16/2009 · toni scarpa scarpat@csr.nih.gov 301-435-1109. Council of Councils. Bethesda, MD. November 16, 2009. National Institutes of Health.](https://reader035.fdocuments.us/reader035/viewer/2022071416/6113e390025adf77c5748ae1/html5/thumbnails/10.jpg)
The Editors How often did Stage 2 reviewers appear to have the appropriate expertise to evaluate technical aspects of the applications?
0 1 2 3 4 5 6
Never
Rarely
Sometimes
Often
Always
Number of respondents
![Page 11: Enhancing Peer Review - DPCPSI · 11/16/2009 · toni scarpa scarpat@csr.nih.gov 301-435-1109. Council of Councils. Bethesda, MD. November 16, 2009. National Institutes of Health.](https://reader035.fdocuments.us/reader035/viewer/2022071416/6113e390025adf77c5748ae1/html5/thumbnails/11.jpg)
The Editors Should future announcements have specific areas of science highlighted?
0 1 2 3 4
Strongly agree
Agree
Somewhat agree
Disagree
Strongly disagree
Number of respondents
![Page 12: Enhancing Peer Review - DPCPSI · 11/16/2009 · toni scarpa scarpat@csr.nih.gov 301-435-1109. Council of Councils. Bethesda, MD. November 16, 2009. National Institutes of Health.](https://reader035.fdocuments.us/reader035/viewer/2022071416/6113e390025adf77c5748ae1/html5/thumbnails/12.jpg)
ApplicantsTransformative R01 Applicants by Gender
![Page 13: Enhancing Peer Review - DPCPSI · 11/16/2009 · toni scarpa scarpat@csr.nih.gov 301-435-1109. Council of Councils. Bethesda, MD. November 16, 2009. National Institutes of Health.](https://reader035.fdocuments.us/reader035/viewer/2022071416/6113e390025adf77c5748ae1/html5/thumbnails/13.jpg)
ApplicantsAge Distribution of T-R01 Applicants
0 20 40 60 80 100
80+
75-79
70-74
65-69
60-64
55-59
50-54
45-49
40-44
35-39
30-34
25-29
Under 25
Number of respondents
![Page 14: Enhancing Peer Review - DPCPSI · 11/16/2009 · toni scarpa scarpat@csr.nih.gov 301-435-1109. Council of Councils. Bethesda, MD. November 16, 2009. National Institutes of Health.](https://reader035.fdocuments.us/reader035/viewer/2022071416/6113e390025adf77c5748ae1/html5/thumbnails/14.jpg)
Applicants: Race and Ethnicity
![Page 15: Enhancing Peer Review - DPCPSI · 11/16/2009 · toni scarpa scarpat@csr.nih.gov 301-435-1109. Council of Councils. Bethesda, MD. November 16, 2009. National Institutes of Health.](https://reader035.fdocuments.us/reader035/viewer/2022071416/6113e390025adf77c5748ae1/html5/thumbnails/15.jpg)
ApplicantsPercentage of NIH Experienced TR01 Applicants
![Page 16: Enhancing Peer Review - DPCPSI · 11/16/2009 · toni scarpa scarpat@csr.nih.gov 301-435-1109. Council of Councils. Bethesda, MD. November 16, 2009. National Institutes of Health.](https://reader035.fdocuments.us/reader035/viewer/2022071416/6113e390025adf77c5748ae1/html5/thumbnails/16.jpg)
ApplicantsPercentage of Applicants with ≥ 50% of their Funding (past 5 years) by Source
0 100 200 300 400
Other
For-profits
Other US …
Hospitals, …
Foundations
NIH
Number of respondents
![Page 17: Enhancing Peer Review - DPCPSI · 11/16/2009 · toni scarpa scarpat@csr.nih.gov 301-435-1109. Council of Councils. Bethesda, MD. November 16, 2009. National Institutes of Health.](https://reader035.fdocuments.us/reader035/viewer/2022071416/6113e390025adf77c5748ae1/html5/thumbnails/17.jpg)
ApplicantsPercentage of Applicants Claiming Proposed Project is Significant Departure from Previous Research Direction
82%
18%
Yes
No
![Page 18: Enhancing Peer Review - DPCPSI · 11/16/2009 · toni scarpa scarpat@csr.nih.gov 301-435-1109. Council of Councils. Bethesda, MD. November 16, 2009. National Institutes of Health.](https://reader035.fdocuments.us/reader035/viewer/2022071416/6113e390025adf77c5748ae1/html5/thumbnails/18.jpg)
Applicants
Research Area % of ApplicantsBehavioral and/or Social Science
10.2%
Clinical and/or Translational Research
28.1%
Instrumentation and/or Engineering
5.8%
Molecular, Cellular, and/or Chemical Biology
38.1%
Pathogenesis and/or epidemiology
3.7%
Physiological and/or Integrative Systems
9.1%
Quantitative and/or Mathematical Biology
3.7%
![Page 19: Enhancing Peer Review - DPCPSI · 11/16/2009 · toni scarpa scarpat@csr.nih.gov 301-435-1109. Council of Councils. Bethesda, MD. November 16, 2009. National Institutes of Health.](https://reader035.fdocuments.us/reader035/viewer/2022071416/6113e390025adf77c5748ae1/html5/thumbnails/19.jpg)
ApplicantsCommunication of Concept’s Novelty, Innovation, and Impact
0 100 200 300 400 500
Abstract
Challenge & impact
Approach
Appropriateness
Timeline
Bibliography Unimportant
Of little importance
Moderately important
Important
Very important
![Page 20: Enhancing Peer Review - DPCPSI · 11/16/2009 · toni scarpa scarpat@csr.nih.gov 301-435-1109. Council of Councils. Bethesda, MD. November 16, 2009. National Institutes of Health.](https://reader035.fdocuments.us/reader035/viewer/2022071416/6113e390025adf77c5748ae1/html5/thumbnails/20.jpg)
ApplicantsPossibility of Receiving Funding From other Sources
210
84 24
110
Very unlikely
Somewhat unlikely
Somewhat likely
Very likely
![Page 21: Enhancing Peer Review - DPCPSI · 11/16/2009 · toni scarpa scarpat@csr.nih.gov 301-435-1109. Council of Councils. Bethesda, MD. November 16, 2009. National Institutes of Health.](https://reader035.fdocuments.us/reader035/viewer/2022071416/6113e390025adf77c5748ae1/html5/thumbnails/21.jpg)
Enhancing Peer Review
![Page 22: Enhancing Peer Review - DPCPSI · 11/16/2009 · toni scarpa scarpat@csr.nih.gov 301-435-1109. Council of Councils. Bethesda, MD. November 16, 2009. National Institutes of Health.](https://reader035.fdocuments.us/reader035/viewer/2022071416/6113e390025adf77c5748ae1/html5/thumbnails/22.jpg)
Corporate NIH: Enhancing Peer Review
• The Charge from Dr. Zerhouni:
“Fund the best science, by the best scientists, with the least administrative burden…”
Two advisory committees to the NIH Director
http://enhancing-peer-review.nih.gov
![Page 23: Enhancing Peer Review - DPCPSI · 11/16/2009 · toni scarpa scarpat@csr.nih.gov 301-435-1109. Council of Councils. Bethesda, MD. November 16, 2009. National Institutes of Health.](https://reader035.fdocuments.us/reader035/viewer/2022071416/6113e390025adf77c5748ae1/html5/thumbnails/23.jpg)
The Process
![Page 24: Enhancing Peer Review - DPCPSI · 11/16/2009 · toni scarpa scarpat@csr.nih.gov 301-435-1109. Council of Councils. Bethesda, MD. November 16, 2009. National Institutes of Health.](https://reader035.fdocuments.us/reader035/viewer/2022071416/6113e390025adf77c5748ae1/html5/thumbnails/24.jpg)
Improve Quality and Transparency of the Peer Review Process
Changes occurred at the last meetings (July 2009):• Scores 1-9• Assigned reviewers score each criterion• Shorter summary statement, with boxes for each criterion• Clustering new investigators• Scores of individual criteria given to all applicants• Discussed applications receive additional overall impact score
Changes occurring in 2010:• Shorter application (12 pages for R01) designed to match
scoring criteria
![Page 25: Enhancing Peer Review - DPCPSI · 11/16/2009 · toni scarpa scarpat@csr.nih.gov 301-435-1109. Council of Councils. Bethesda, MD. November 16, 2009. National Institutes of Health.](https://reader035.fdocuments.us/reader035/viewer/2022071416/6113e390025adf77c5748ae1/html5/thumbnails/25.jpg)
A. Enhanced Review Criteria• Overall Impact:
Assessment of the likelihood for the project to exert a sustained, powerful influence on the research field(s)
New Core Criteria Order:SignificanceInvestigator(s)InnovationApproachEnvironment
![Page 26: Enhancing Peer Review - DPCPSI · 11/16/2009 · toni scarpa scarpat@csr.nih.gov 301-435-1109. Council of Councils. Bethesda, MD. November 16, 2009. National Institutes of Health.](https://reader035.fdocuments.us/reader035/viewer/2022071416/6113e390025adf77c5748ae1/html5/thumbnails/26.jpg)
B. Template-Based Critiques
• The objective is to write evaluative statements and to avoid summarizing the application
• Comments should be in the form of bullet points or if necessary short narratives
1. Significance Please limit text to ¼ pageStrengths•••Weaknesses•••
![Page 27: Enhancing Peer Review - DPCPSI · 11/16/2009 · toni scarpa scarpat@csr.nih.gov 301-435-1109. Council of Councils. Bethesda, MD. November 16, 2009. National Institutes of Health.](https://reader035.fdocuments.us/reader035/viewer/2022071416/6113e390025adf77c5748ae1/html5/thumbnails/27.jpg)
C. Scoring
Impact Score Descriptor
High Impact
1 Exceptional
2 Outstanding
3 Excellent
Moderate Impact
4 Very Good
5 Good
6 Satisfactory
Low Impact
7 Fair
8 Marginal
9 Poor
![Page 28: Enhancing Peer Review - DPCPSI · 11/16/2009 · toni scarpa scarpat@csr.nih.gov 301-435-1109. Council of Councils. Bethesda, MD. November 16, 2009. National Institutes of Health.](https://reader035.fdocuments.us/reader035/viewer/2022071416/6113e390025adf77c5748ae1/html5/thumbnails/28.jpg)
C. ScoringPriority Scores of R01 and R21 Reviewed by CSR
![Page 29: Enhancing Peer Review - DPCPSI · 11/16/2009 · toni scarpa scarpat@csr.nih.gov 301-435-1109. Council of Councils. Bethesda, MD. November 16, 2009. National Institutes of Health.](https://reader035.fdocuments.us/reader035/viewer/2022071416/6113e390025adf77c5748ae1/html5/thumbnails/29.jpg)
D. Order of ReviewWhy?•Concern of variation of scores during different times of the meeting.
The original plan was to recalibrate scores at the end of the meeting
Solution:•Recalibrate dynamically by discussing in order of average preliminary scores from assigned reviewers.
Requirement:• Reviewers must participate in entire meeting
![Page 30: Enhancing Peer Review - DPCPSI · 11/16/2009 · toni scarpa scarpat@csr.nih.gov 301-435-1109. Council of Councils. Bethesda, MD. November 16, 2009. National Institutes of Health.](https://reader035.fdocuments.us/reader035/viewer/2022071416/6113e390025adf77c5748ae1/html5/thumbnails/30.jpg)
E. Enhancing Peer Review Training
• CSR and NIH Review Staff6 face to face training sessions, January 20096 face to face training sessions, April 2009Continuous updating
• Chairs27 half a day sections for 400 chairs in 11 cities
• ReviewersTraining material (Power Point, interactive training, frequently asked questions, mock study section video, etc,in April-May 2009Senior CSR staff at the first two meeting using the new system
![Page 31: Enhancing Peer Review - DPCPSI · 11/16/2009 · toni scarpa scarpat@csr.nih.gov 301-435-1109. Council of Councils. Bethesda, MD. November 16, 2009. National Institutes of Health.](https://reader035.fdocuments.us/reader035/viewer/2022071416/6113e390025adf77c5748ae1/html5/thumbnails/31.jpg)
American Recovery and Reinvestment Act
![Page 32: Enhancing Peer Review - DPCPSI · 11/16/2009 · toni scarpa scarpat@csr.nih.gov 301-435-1109. Council of Councils. Bethesda, MD. November 16, 2009. National Institutes of Health.](https://reader035.fdocuments.us/reader035/viewer/2022071416/6113e390025adf77c5748ae1/html5/thumbnails/32.jpg)
CSR Peer Review: 2008
• 77,000 applications received
• 16,000 reviewers
• 1,600 review meetings
• 240 Scientific Review Officers
![Page 33: Enhancing Peer Review - DPCPSI · 11/16/2009 · toni scarpa scarpat@csr.nih.gov 301-435-1109. Council of Councils. Bethesda, MD. November 16, 2009. National Institutes of Health.](https://reader035.fdocuments.us/reader035/viewer/2022071416/6113e390025adf77c5748ae1/html5/thumbnails/33.jpg)
CSR Peer Review: 2009
• 77,000 applications received
• 16,000 reviewers
• 1,600 review meetings
• 240 Scientific Review Officers
![Page 34: Enhancing Peer Review - DPCPSI · 11/16/2009 · toni scarpa scarpat@csr.nih.gov 301-435-1109. Council of Councils. Bethesda, MD. November 16, 2009. National Institutes of Health.](https://reader035.fdocuments.us/reader035/viewer/2022071416/6113e390025adf77c5748ae1/html5/thumbnails/34.jpg)
Number of Applications
![Page 35: Enhancing Peer Review - DPCPSI · 11/16/2009 · toni scarpa scarpat@csr.nih.gov 301-435-1109. Council of Councils. Bethesda, MD. November 16, 2009. National Institutes of Health.](https://reader035.fdocuments.us/reader035/viewer/2022071416/6113e390025adf77c5748ae1/html5/thumbnails/35.jpg)
Applications Received for Review by CSR (May-July 2009)
• Non ARRA Applications (Normal) 18,200
• Challenge Applications 21,000
• Competitive Revisions 1,200
• GO Grants 180
• High End Instrumentation 130
• Small Business 1,500
• TOTAL 42,000
![Page 36: Enhancing Peer Review - DPCPSI · 11/16/2009 · toni scarpa scarpat@csr.nih.gov 301-435-1109. Council of Councils. Bethesda, MD. November 16, 2009. National Institutes of Health.](https://reader035.fdocuments.us/reader035/viewer/2022071416/6113e390025adf77c5748ae1/html5/thumbnails/36.jpg)
Applications Reviewed and Reviewers Used by CSR in May-July 2008 and 2009
![Page 37: Enhancing Peer Review - DPCPSI · 11/16/2009 · toni scarpa scarpat@csr.nih.gov 301-435-1109. Council of Councils. Bethesda, MD. November 16, 2009. National Institutes of Health.](https://reader035.fdocuments.us/reader035/viewer/2022071416/6113e390025adf77c5748ae1/html5/thumbnails/37.jpg)
Applications Reviewed and Reviewers Used by CSR in June 2008 and 2009
![Page 38: Enhancing Peer Review - DPCPSI · 11/16/2009 · toni scarpa scarpat@csr.nih.gov 301-435-1109. Council of Councils. Bethesda, MD. November 16, 2009. National Institutes of Health.](https://reader035.fdocuments.us/reader035/viewer/2022071416/6113e390025adf77c5748ae1/html5/thumbnails/38.jpg)
The Way We Did It
To achieve great things, three things are needed:a plan, great people and not quite enough time
adapted from Leonard Bernstein
![Page 39: Enhancing Peer Review - DPCPSI · 11/16/2009 · toni scarpa scarpat@csr.nih.gov 301-435-1109. Council of Councils. Bethesda, MD. November 16, 2009. National Institutes of Health.](https://reader035.fdocuments.us/reader035/viewer/2022071416/6113e390025adf77c5748ae1/html5/thumbnails/39.jpg)
This is CSR
September 2009
![Page 40: Enhancing Peer Review - DPCPSI · 11/16/2009 · toni scarpa scarpat@csr.nih.gov 301-435-1109. Council of Councils. Bethesda, MD. November 16, 2009. National Institutes of Health.](https://reader035.fdocuments.us/reader035/viewer/2022071416/6113e390025adf77c5748ae1/html5/thumbnails/40.jpg)
Looking at the Future