Trauma Theory Information on slides 2-13 taken from a review by Dr. Toni McNaron.
Center for Scientific Review National Institutes of Health Department of Health and Human Services...
-
Upload
michael-oneil -
Category
Documents
-
view
216 -
download
0
Transcript of Center for Scientific Review National Institutes of Health Department of Health and Human Services...
Center for Scientific ReviewNational Institutes of HealthDepartment of Health and Human Services
Toni Scarpa
NIH Peer Review: Continuity and Change
APS/Cardiovascular Section
Salt Francisco, April 2006
Peer Review: An N.I.H. “Conception”
• Is the heart and soul of NIH• Has produced an effective partnership between the
federal government and research institutions• Has created the best academic medical centers, the best
biomedical/behavioral research and biotechnology• Has made possible the best cures and the best prevention• Has been admired and imitated here and abroad• Has protected NIH against outside influence
Center for Scientific Review
Applications received for all of NIH and applications referred for CSR review, FY 1998-2004
0
20,000
40,000
60,000
80,000
1998 2000 2002 2004Fiscal year
Nu
mb
er o
f ap
pli
cati
on
s
Applications received for all of NIH
Applications assigned for review by CSR
CSR 4 Review Divisions & 23 IRGs
Division of Biologic Basis of Disease
Elliot Postow, Ph.D.
AIDS and RelatedResearch IRG (AARR)
Ranga V. Srinivas, Ph.D.
Endocrinology, Metabolism, Nutrition, and Reproductive
Sciences (EMNR)Sooja Kim, Ph.D.
Immunological SciencesIRG (IMM)
Calbert Laing, Ph.D.
Oncological SciencesIRG (ONC)
Syed Quadri, Ph.D.
Infectious Diseases andMicrobiology IRG (IDM)
Alex Politis, Ph.D.
Division of Physiology and Pathology
Michael Martin, Ph.D.
Cardiovascular SciencesIRG (CVS)
Joyce Gibson, D.Sc.
Integrative, Functionaland Cognitive Neuroscience
IRG (IFCN)Christine Melchior, Ph.D.
Renal and Urological Sciences IRG (RUS)
Daniel McDonald, Ph.D.
HematologyIRG (HEME)
Joyce Gibson, D.Sc.
Digestive Sciences IRG (DIG)
Mushtaq Khan, Ph.D., DVM
Division of Clinical andPopulation-Based Studies
Anita Miller Sostek, Ph.D
Surgery, Radiology, andBioengineering IRG (SRB)
Eileen Bradley, D.Sc.
Health of the Population IRG (HOP)
Robert Weller, Ph.D.
Risk, Prevention, and HealthBehavior IRG (RPHB)Michael Micklin, Ph.D.
Brain Disorders and ClinicalNeuroscience IRG (BDCN)
Dana Plude, Ph.D.
Behavioral & BiobehavioralProcesses IRG (BBBP)Karen Sirocco, Ph.D.
Division of Molecular and Cellular Mechanisms
Donald Schneider, Ph.D.
Bioengineering Sciences and Technologies IRG (BST)
Sally Amero, Ph.D.
Biology of Development and and Aging (BDA)
Sherry Dupere, Ph.D.
Biological Chemistry and Macromolecular Biophysics
IRG (BCMB)John Bowers, Ph.D.
Cell Biology IRG (CB)Marcia Steinberg, Ph.D.
Genes, Genomes, and Genetics IRG (GGG)
Richard Panniers, Ph.D.
Molecular, Cellular and Developmental
Neuroscience IRG (MDCN)Carole Jelsema, Ph.D.
Respiratory Sciences IRG (RES)
Mushtaq Khan, Ph.D., DVM
Musculoskeletal, Oral, and Skin Sciences IRG (MOSS)
Daniel McDonald, Ph.D.
CSR Mission Statement
To see that NIH grant applications receive fair, independent, expert, and timely reviews -- free from inappropriate influences -- so NIH can fund the most promising research.
Time
C
om
ple
xity
an
d I
mp
act
CSR Operations
C
urrent S
ystems
New Systems?
Necessary Changes in CSR Peer Review Operations
• Increase communications between CSR, the ICs, our reviewers and applicants
• Increase uniformity
• Increase efficiency
• Facilitate work of IC program staff
Changes in CSR Operations
Potential of Knowledge Management Tools for Peer Review
Collexis Software or Others
• Knowledge management solutions• Fingerprinting and text retrieving• Disease coding
Benefits for Peer Review
• Assigning applications to Integrated Review Groups or Study Sections
• Selecting reviewers (one application, multiple applications)
Nine pilots are underway to begin to assess these benefits
Study Section Realignment
• Review of one IRG every month• Total review every 2 years
Required Changes in Current Systems
• Shorten the review cycle
This is Not an Ford Assembly Line
Receipt ReferEvaluate Scientific
Merit of Applications
Shortening the NIH Review Cycle, Initial Steps
For most research grants, we are posting summary statements within one month after the study section meeting instead of two to three months after the meeting (effective Oct 05)
We are conducting a pilot study to speed the review process for new investigators so they may revise and resubmit for the very next review cycle 4 months earlier than before (effective Feb 06)
Desirable Changes in CSR Review
• Shorten the review cycle
• Address concern that clinical research is not properly evaluated
• Improve the assessment of innovative, high- risk/high-reward research
“The judging of grants has become a charade.”
The American Society for Cell Biology
“The judging of grants has become a charade. To be funded, the experimental plan has become a litany of experiments already accomplished so that everything is feasible. When grants come back with unfundable scores, new investigators may not have sufficient resources to do the experiments that “show feasibility.”
Zena WerbPresident, ASCB
Newsletter August 2005
Possible Changes in Current Systems
• Shorten the review cycle
• Address concern that clinical research is not properly evaluated
• Improve the assessment of innovative, high- risk/high-reward research
• Do more to recruit and retain more high-quality reviewers
Expanding Peer Review’s Platforms
Electronic Reviews• Telephone Enhanced Discussions• Video Enhanced Discussions• Asynchronous Electronic Discussions
Study Sections
Necessity ● Clinical reviewers Preference ● Physicists, computational biologists
New Opportunities ● Fogarty, International Reviewers
The First NIH Study Section
1945
The Last NIH Study Section
2005
If we didn’t have any peer-review system and we had to design one from scratch, what would it look like?
Possible New Systems
Questions—Applications
• Should we have 3 or 365 deadlines for most applications?
• Should applications (Rs) be shortened? Should appendices be eliminated or reduced in size?
• Is there more value in having 2-3 reviewers reading 25-page applications or 10-15 reviewers reading 5-page applications?
Questions—Study Section Meetings
• What is the ideal number of members to have serve on a study section?
• Is one study section with 50-70 reviewers efficient?
• What is the intellectual contribution of individual reviewers in large study sections?
Questions—Scoring
• Is it proper or desirable to have 50-70 reviewers voting on priority scores for each application referred to their study section?
• Is consensus always good? Or should we focus on score variance?
Applications Received for All of NIH FY 1998-2004
0
20,000
40,000
60,000
80,000
1998 2000 2002 2004Fiscal year
Nu
mb
er o
f ap
pli
cati
on
s
Number of Research Grant Applications/Applicant
1.05
1.1
1.15
1.2
1.25
1.3
1.35
1.4
1.45
1980
1982
1984
1986
1988
1990
1992
1994
1996
1998
2000
2002
2004
CSR Applications Reviewed, Regular and SEP May Council Only
0
2000
4000
6000
8000
10000
12000
14000
16000
1999 2000 2001 2002 2003 2004 2005
Applications Regular
Applications SEP
Study Section Application/Reviewer Ratio October Council Only
0
0.5
1
1.5
2
2.5
3
3.5
4
1995 1996 1997 1998 1999 2000 2001 2002 2003 2004 2005
Regular
SEP
Two groups of challenges/opportunities
• Reviewers– Decrease the number of reviewers and increase the
quality– Increase the number of applications reviewed
without extra workload– Recruit and retain the best reviewers
• SRA– Increase efficiency– Recruit and train
Possible Short Term Approaches for Increasing Efficiency for Reviewers and CSR
• Replace Many SEPs with Smaller Parallel Study Sections
• Enlarge Study Section Membership and Decrease Frequency of Participation
• Pre Meeting Streamlining• Various Review Platforms• Hybrid Review Platforms• Staggering Application Deadlines• 2 instead of 3 reviews• Shorten Applications• More Structured Applications and Reviews
Size of Grant Applications
• RO1 • Will increase number of applications reviewed by
reviewers• Will decrease the number of reviewers in a study section• May be combined with a change in format of the
application, more consonant to review criteria• May be combined with scoring individual criteria (i.e.
relevance, innovation, etc• Strong support by councils and scientific leadership• May result in more innovation
This is CSR
Coronary Heart DiseaseAge-Adjusted Death Rates in U.S.: Actual (blue) vs Expected (yellow)
500
400
300
200
100
01950 1955 1960 1965 1970 1975 1980 1985 1990 1995 2000
Dea
ths
per
100
,000
Year
~ 514,000 Actual Deathsin 2000
~ 1,329,000 Projected Deaths in 2000
815,000 Deaths Prevented in 2000
The True Value of Peer Review