Post on 03-Jan-2016
Richard Nakamura. PhD
CSR Advisory CouncilMay 2014
Strategy for quality measurement
In most production systems, there is a speed, cost, quality
trade-off, but this is not a zero-sum game.
We have just discussed speed and to some extent, cost.
EffectivenessReview quality
For Problem A, we have focused on:
Attracting:…the best reviewers …the best Chairs …the best SROs
After we put together the best review committees, they must work in the best platform/format to determine merit of applications
Best reviewers -…recruit mostly senior scientists …funded awardees of NIH…w/ strong publications…positive review experience …respect from other reviewers…and diversity that aids the SRG
To get the best reviewers, we offer -…continuous submission…new forms of attendance…meetings on West coast
But -…we take away coffee…we make travel less convenient…we issue 1099 forms for all payments
Today, NIH wants to know how to measure quality so that we can systematically improve it.
To the extent that we have a standard, it has been to examine the results of grants ten years after award as evaluated by senior scientists. This has been done for the Pioneer awards (for 7 years) but provides a very slow feedback loop. Can we do better?
We would like to have measures available more immediately than 10 years after award that predict the longer outcome.
How do we get there?To begin with we need to have estimates of the reliability of review scoring (Rene Etcheberrigaray), so we can power our other measures of quality and interventions correctly.
We need evidence that our scoring approaches are not inappropriately biased – Monica Basco.
In measuring quality, CSR takes seriously the opinions of our scientific reviewers:
Surveys – CSR quick feedback (Mary Ann Guadagno)
Committee ranking within IRGS (Problem B and last Council)
Is the application ranking system working? (Amy Rubinstein)
Do our committees favor conformity? (Seymour Garte)
What other approaches could help us evaluate quality? (actively pursued by other groups and not covered today)
Bibliometrics (DPCPSI and CSR):Citations at individual, group levelsRelative citation rating; h-indexConcerns about citation manipulation
Review committee assignments and coverage
Network analysis - Application and publication analysis of applicants and reviewers
Creation of computer assisted assignment
AARRBBBPBCMBBDCNBSTCBCVRSDKUSEMNRETTNGGGHDMIDMIFCNIMMMDCNMOSSOBTOTCPSERPHBSBIBVHOther
A S
yste
m L
evel
Rep
rese
ntat
ion
of
Sci
entif
ic In
tere
sts
Biological Chemistry & Macromolecular Biophysics
Infectious Diseases and Microbiology
Questions? Comments?
CSRDirector@csr.nih.gov