Transcript of CSR Advisory Council Meeting May 19, 2014 Editorial Board Review A Few Good Reviewers Don Schneider,...
- Slide 1
- CSR Advisory Council Meeting May 19, 2014 Editorial Board
Review A Few Good Reviewers Don Schneider, Ph.D.
- Slide 2
- Reviewer pools Past experience Cost considerations Toward
Review by the Best
- Slide 3
- Reviewer StatusNumbers HHMI300 NIH R37700 NAS2,000 R0126,000
CSR needs16,000 Reviewer Pools
- Slide 4
- Format modeled on journal manuscript review First piloted in
2008 with 6 SBIR panels, just in time for TR01s, Challenge grants,
DP1 etc. Two stages First Stage Mail reviewers Second Stage Editors
Past Experience
- Slide 5
- First Stage/Mail Reviewers Subject matter experts Provide depth
in review Focus on scientific and technical merit 2-3 first stage
mail reviewers per application Submit full critiques Give overall
impact and criterion scores Overall impact scores not factored into
final priority score
- Slide 6
- Hold face-to-face meeting Recruit broad experts Provide
perspective in review (assign about 15 applications each) Focus on
impact and significance Assign 3 second stage reviewers per
application Consider first stage critiques in review Write overall
impact paragraph Give overall impact score Final priority score
based on second stage only Second Stage/Editors
- Slide 7
- Provides both depth and breadth in review Optimizes use of the
best reviewers Scales well for large numbers of applications
(second stage discusses a fraction of the applications)
Rationale
- Slide 8
- Perceived Advantages Involves no travel/teleconference for
first stage reviewers Allows small, interactive face-to-face
meetings Promotes better scoring and assessment of impact Lessens
travel and lodging expenses and inconveniences
- Slide 9
- Review# of ApplicationsCost/application Regular R01 F2F$518
DP1/Pioneer EB+I244$280 DP2/New Innov EB593$124 DP5/Early Indep
EB+I84$875 Cost Considerations (Alicia Caffi)
- Slide 10
- Recruitment of large numbers of reviewers Timeline Tight, two
sequential reviews (in the 17 week cycle) More staff time required
(SROs) Some sense of isolation by first stage reviewers
Challenges
- Slide 11
- Each application examined by at least 5 reviewers Interactive,
thoughtful discussions Overall scoring by second stage members
Reviewers and staff like final review products Review Outcomes
- Slide 12
- Survey conducted by A Kopstein of reviewers participating in
SBIR pilots 2008 Outcomes were generally positive Majority willing
to participate in either review stage in future Editorial Board
Review: Increases expert review 3/4 ths of respondents Preferred
for their own applications 2/3 rds of respondents Survey
- Slide 13
- Hopes for a few good reviewers? Discussion