Richard Nakamura. PhD CSR Advisory Council May 2014 Strategy for quality measurement.
-
Upload
peregrine-melvin-fletcher -
Category
Documents
-
view
215 -
download
1
Transcript of Richard Nakamura. PhD CSR Advisory Council May 2014 Strategy for quality measurement.
![Page 1: Richard Nakamura. PhD CSR Advisory Council May 2014 Strategy for quality measurement.](https://reader030.fdocuments.us/reader030/viewer/2022032806/56649f055503460f94c19a46/html5/thumbnails/1.jpg)
Richard Nakamura. PhD
CSR Advisory CouncilMay 2014
Strategy for quality measurement
![Page 2: Richard Nakamura. PhD CSR Advisory Council May 2014 Strategy for quality measurement.](https://reader030.fdocuments.us/reader030/viewer/2022032806/56649f055503460f94c19a46/html5/thumbnails/2.jpg)
In most production systems, there is a speed, cost, quality
trade-off, but this is not a zero-sum game.
We have just discussed speed and to some extent, cost.
![Page 3: Richard Nakamura. PhD CSR Advisory Council May 2014 Strategy for quality measurement.](https://reader030.fdocuments.us/reader030/viewer/2022032806/56649f055503460f94c19a46/html5/thumbnails/3.jpg)
EffectivenessReview quality
![Page 4: Richard Nakamura. PhD CSR Advisory Council May 2014 Strategy for quality measurement.](https://reader030.fdocuments.us/reader030/viewer/2022032806/56649f055503460f94c19a46/html5/thumbnails/4.jpg)
![Page 5: Richard Nakamura. PhD CSR Advisory Council May 2014 Strategy for quality measurement.](https://reader030.fdocuments.us/reader030/viewer/2022032806/56649f055503460f94c19a46/html5/thumbnails/5.jpg)
![Page 6: Richard Nakamura. PhD CSR Advisory Council May 2014 Strategy for quality measurement.](https://reader030.fdocuments.us/reader030/viewer/2022032806/56649f055503460f94c19a46/html5/thumbnails/6.jpg)
For Problem A, we have focused on:
Attracting:…the best reviewers …the best Chairs …the best SROs
![Page 7: Richard Nakamura. PhD CSR Advisory Council May 2014 Strategy for quality measurement.](https://reader030.fdocuments.us/reader030/viewer/2022032806/56649f055503460f94c19a46/html5/thumbnails/7.jpg)
After we put together the best review committees, they must work in the best platform/format to determine merit of applications
![Page 8: Richard Nakamura. PhD CSR Advisory Council May 2014 Strategy for quality measurement.](https://reader030.fdocuments.us/reader030/viewer/2022032806/56649f055503460f94c19a46/html5/thumbnails/8.jpg)
Best reviewers -…recruit mostly senior scientists …funded awardees of NIH…w/ strong publications…positive review experience …respect from other reviewers…and diversity that aids the SRG
![Page 9: Richard Nakamura. PhD CSR Advisory Council May 2014 Strategy for quality measurement.](https://reader030.fdocuments.us/reader030/viewer/2022032806/56649f055503460f94c19a46/html5/thumbnails/9.jpg)
To get the best reviewers, we offer -…continuous submission…new forms of attendance…meetings on West coast
![Page 10: Richard Nakamura. PhD CSR Advisory Council May 2014 Strategy for quality measurement.](https://reader030.fdocuments.us/reader030/viewer/2022032806/56649f055503460f94c19a46/html5/thumbnails/10.jpg)
But -…we take away coffee…we make travel less convenient…we issue 1099 forms for all payments
![Page 11: Richard Nakamura. PhD CSR Advisory Council May 2014 Strategy for quality measurement.](https://reader030.fdocuments.us/reader030/viewer/2022032806/56649f055503460f94c19a46/html5/thumbnails/11.jpg)
Today, NIH wants to know how to measure quality so that we can systematically improve it.
![Page 12: Richard Nakamura. PhD CSR Advisory Council May 2014 Strategy for quality measurement.](https://reader030.fdocuments.us/reader030/viewer/2022032806/56649f055503460f94c19a46/html5/thumbnails/12.jpg)
To the extent that we have a standard, it has been to examine the results of grants ten years after award as evaluated by senior scientists. This has been done for the Pioneer awards (for 7 years) but provides a very slow feedback loop. Can we do better?
![Page 13: Richard Nakamura. PhD CSR Advisory Council May 2014 Strategy for quality measurement.](https://reader030.fdocuments.us/reader030/viewer/2022032806/56649f055503460f94c19a46/html5/thumbnails/13.jpg)
We would like to have measures available more immediately than 10 years after award that predict the longer outcome.
![Page 14: Richard Nakamura. PhD CSR Advisory Council May 2014 Strategy for quality measurement.](https://reader030.fdocuments.us/reader030/viewer/2022032806/56649f055503460f94c19a46/html5/thumbnails/14.jpg)
How do we get there?To begin with we need to have estimates of the reliability of review scoring (Rene Etcheberrigaray), so we can power our other measures of quality and interventions correctly.
![Page 15: Richard Nakamura. PhD CSR Advisory Council May 2014 Strategy for quality measurement.](https://reader030.fdocuments.us/reader030/viewer/2022032806/56649f055503460f94c19a46/html5/thumbnails/15.jpg)
We need evidence that our scoring approaches are not inappropriately biased – Monica Basco.
![Page 16: Richard Nakamura. PhD CSR Advisory Council May 2014 Strategy for quality measurement.](https://reader030.fdocuments.us/reader030/viewer/2022032806/56649f055503460f94c19a46/html5/thumbnails/16.jpg)
In measuring quality, CSR takes seriously the opinions of our scientific reviewers:
Surveys – CSR quick feedback (Mary Ann Guadagno)
Committee ranking within IRGS (Problem B and last Council)
![Page 17: Richard Nakamura. PhD CSR Advisory Council May 2014 Strategy for quality measurement.](https://reader030.fdocuments.us/reader030/viewer/2022032806/56649f055503460f94c19a46/html5/thumbnails/17.jpg)
![Page 18: Richard Nakamura. PhD CSR Advisory Council May 2014 Strategy for quality measurement.](https://reader030.fdocuments.us/reader030/viewer/2022032806/56649f055503460f94c19a46/html5/thumbnails/18.jpg)
Is the application ranking system working? (Amy Rubinstein)
Do our committees favor conformity? (Seymour Garte)
![Page 19: Richard Nakamura. PhD CSR Advisory Council May 2014 Strategy for quality measurement.](https://reader030.fdocuments.us/reader030/viewer/2022032806/56649f055503460f94c19a46/html5/thumbnails/19.jpg)
What other approaches could help us evaluate quality? (actively pursued by other groups and not covered today)
Bibliometrics (DPCPSI and CSR):Citations at individual, group levelsRelative citation rating; h-indexConcerns about citation manipulation
![Page 20: Richard Nakamura. PhD CSR Advisory Council May 2014 Strategy for quality measurement.](https://reader030.fdocuments.us/reader030/viewer/2022032806/56649f055503460f94c19a46/html5/thumbnails/20.jpg)
Review committee assignments and coverage
Network analysis - Application and publication analysis of applicants and reviewers
Creation of computer assisted assignment
![Page 21: Richard Nakamura. PhD CSR Advisory Council May 2014 Strategy for quality measurement.](https://reader030.fdocuments.us/reader030/viewer/2022032806/56649f055503460f94c19a46/html5/thumbnails/21.jpg)
AARRBBBPBCMBBDCNBSTCBCVRSDKUSEMNRETTNGGGHDMIDMIFCNIMMMDCNMOSSOBTOTCPSERPHBSBIBVHOther
A S
yste
m L
evel
Rep
rese
ntat
ion
of
Sci
entif
ic In
tere
sts
![Page 22: Richard Nakamura. PhD CSR Advisory Council May 2014 Strategy for quality measurement.](https://reader030.fdocuments.us/reader030/viewer/2022032806/56649f055503460f94c19a46/html5/thumbnails/22.jpg)
Biological Chemistry & Macromolecular Biophysics
![Page 23: Richard Nakamura. PhD CSR Advisory Council May 2014 Strategy for quality measurement.](https://reader030.fdocuments.us/reader030/viewer/2022032806/56649f055503460f94c19a46/html5/thumbnails/23.jpg)
Infectious Diseases and Microbiology