Michael S. Lauer, MD, FACC, FAHA Director, Division of Cardiovascular Sciences National Heart, Lung,...
-
Upload
rolf-chambers -
Category
Documents
-
view
221 -
download
0
Transcript of Michael S. Lauer, MD, FACC, FAHA Director, Division of Cardiovascular Sciences National Heart, Lung,...
![Page 1: Michael S. Lauer, MD, FACC, FAHA Director, Division of Cardiovascular Sciences National Heart, Lung, and Blood Institute CSR Advisory Council Meeting Pooks.](https://reader036.fdocuments.us/reader036/viewer/2022062308/56649f045503460f94c183dc/html5/thumbnails/1.jpg)
Michael S. Lauer, MD, FACC, FAHADirector, Division of Cardiovascular Sciences
National Heart, Lung, and Blood Institute
CSR Advisory Council MeetingPooks Hill Marriott, Bethesda, MD
October 20, 2014
Perspectives from the NHLBI on Portfolio Prediction and Performance
![Page 2: Michael S. Lauer, MD, FACC, FAHA Director, Division of Cardiovascular Sciences National Heart, Lung, and Blood Institute CSR Advisory Council Meeting Pooks.](https://reader036.fdocuments.us/reader036/viewer/2022062308/56649f045503460f94c183dc/html5/thumbnails/2.jpg)
2 Langer J. Science 2012; 388:171
“The U.S. system for funding research was designed to function well in times of growth. It is failing now because the system as a whole is contracting. What is to be done? The funding agencies must admit that it is not humanly possible to predict, with high accuracy, which research projects ultimately will have the most impact. When there are too many of these, as at present, the agencies must find other ways to decide which to support. That will be hard.”
Predicting Performance? …
![Page 3: Michael S. Lauer, MD, FACC, FAHA Director, Division of Cardiovascular Sciences National Heart, Lung, and Blood Institute CSR Advisory Council Meeting Pooks.](https://reader036.fdocuments.us/reader036/viewer/2022062308/56649f045503460f94c183dc/html5/thumbnails/3.jpg)
Cochrane 2008; DOI: 10.1002/14651858.MR000003.pub2.
“Grant giving relies heavily on peer review for the assessment of the quality of proposals but the evidence of effects of these procedures is scarce.”
We (Largely) Rely on Peer Review
![Page 4: Michael S. Lauer, MD, FACC, FAHA Director, Division of Cardiovascular Sciences National Heart, Lung, and Blood Institute CSR Advisory Council Meeting Pooks.](https://reader036.fdocuments.us/reader036/viewer/2022062308/56649f045503460f94c183dc/html5/thumbnails/4.jpg)
Fang F, Casadevall A. Infect Immun 2009;77:929-32
“The imperfect science of grant reviewing: For a system that determines the fate of scientific proposals, peer review is remarkably unscientific. Analyses have revealed that the NIH peer review system is statistically weak, imprecise, and prone to bias. … Almost no scientific investigation has been performed to examine the predictive accuracy of study section peer review.”
-- Ferric Fang & Arturo Casadevall
Strong Feelings Out There …
![Page 5: Michael S. Lauer, MD, FACC, FAHA Director, Division of Cardiovascular Sciences National Heart, Lung, and Blood Institute CSR Advisory Council Meeting Pooks.](https://reader036.fdocuments.us/reader036/viewer/2022062308/56649f045503460f94c183dc/html5/thumbnails/5.jpg)
Ioannidis JP. Nature 2011;477:529-31
“The imperfections of peer review mean that as many as one-third of current grants are effectively being awarded at random. This situation will only worsen as falling acceptance rates encourage investigators to bombard agencies with proposals, leaving fewer qualified reviewers to judge each one.”
Not Only a Peer Review Issue …
![Page 6: Michael S. Lauer, MD, FACC, FAHA Director, Division of Cardiovascular Sciences National Heart, Lung, and Blood Institute CSR Advisory Council Meeting Pooks.](https://reader036.fdocuments.us/reader036/viewer/2022062308/56649f045503460f94c183dc/html5/thumbnails/6.jpg)
Rand Report (July 2013) MG-1217-AAMC
“Characteristics that are scalable, quantitative, transparent, comparable, free from judgment and suitable for high frequency, longitudinal use.”
Measuring Impact is Hard
![Page 7: Michael S. Lauer, MD, FACC, FAHA Director, Division of Cardiovascular Sciences National Heart, Lung, and Blood Institute CSR Advisory Council Meeting Pooks.](https://reader036.fdocuments.us/reader036/viewer/2022062308/56649f045503460f94c183dc/html5/thumbnails/7.jpg)
Ioannidis JP, Khoury M. JAMA 2014 (June 9 2014 online)
Productivity Number of publications in top tier % of citations
for the scientific field and year* > 1 published reports of the main results*
Quality Need to select standards, limit to top-cited articles
Reproducibility* Sharing of data and other resources Translational influence of research
The PQRST Approach
![Page 8: Michael S. Lauer, MD, FACC, FAHA Director, Division of Cardiovascular Sciences National Heart, Lung, and Blood Institute CSR Advisory Council Meeting Pooks.](https://reader036.fdocuments.us/reader036/viewer/2022062308/56649f045503460f94c183dc/html5/thumbnails/8.jpg)
Danthi N et al. Circ Res 2014;114:600-606; Gordon D et al. N Engl J Med 2013;369:1926-34Kaltman J et al. Circ Res 2014;115:617-624
Three Reports from the (NHLBI) Field
![Page 9: Michael S. Lauer, MD, FACC, FAHA Director, Division of Cardiovascular Sciences National Heart, Lung, and Blood Institute CSR Advisory Council Meeting Pooks.](https://reader036.fdocuments.us/reader036/viewer/2022062308/56649f045503460f94c183dc/html5/thumbnails/9.jpg)
Danthi N et al. Circ Res 2014;114:600-606; Kaltman J et al. Circ Res 2014;115:617-624
Investigator-initiated de novo R01’s that: Were funded by NHLBI between 2001-2008 Were assigned to cardiovascular division(s) Received a percentile ranking Received at least two years of funding
Outcomes Number of publications in top 10 % of citations for
the scientific field and year Normalized citation impact (for field and year)
Percentile Ranking and Citation Impact
![Page 10: Michael S. Lauer, MD, FACC, FAHA Director, Division of Cardiovascular Sciences National Heart, Lung, and Blood Institute CSR Advisory Council Meeting Pooks.](https://reader036.fdocuments.us/reader036/viewer/2022062308/56649f045503460f94c183dc/html5/thumbnails/10.jpg)
Kaltman J et al. Circ Res 2014;115:617-624
No Association with Percentile Ranking
![Page 11: Michael S. Lauer, MD, FACC, FAHA Director, Division of Cardiovascular Sciences National Heart, Lung, and Blood Institute CSR Advisory Council Meeting Pooks.](https://reader036.fdocuments.us/reader036/viewer/2022062308/56649f045503460f94c183dc/html5/thumbnails/11.jpg)
Kaltman J et al. Circ Res 2014;115:617-624
Association with Prior Productivity
![Page 12: Michael S. Lauer, MD, FACC, FAHA Director, Division of Cardiovascular Sciences National Heart, Lung, and Blood Institute CSR Advisory Council Meeting Pooks.](https://reader036.fdocuments.us/reader036/viewer/2022062308/56649f045503460f94c183dc/html5/thumbnails/12.jpg)
No gold standard Citations reflect only one perspective Limited scope
Cardiovascular R01 handled within one division Limited time-span
“Multiple-grant problem” Only analyzed outcomes for funded grants
Limitations
![Page 13: Michael S. Lauer, MD, FACC, FAHA Director, Division of Cardiovascular Sciences National Heart, Lung, and Blood Institute CSR Advisory Council Meeting Pooks.](https://reader036.fdocuments.us/reader036/viewer/2022062308/56649f045503460f94c183dc/html5/thumbnails/13.jpg)
Many thanks to Jamie Doyle, Yancy Bodenheim, and Tom Insel
Replication for NIMH R01 Grants
![Page 14: Michael S. Lauer, MD, FACC, FAHA Director, Division of Cardiovascular Sciences National Heart, Lung, and Blood Institute CSR Advisory Council Meeting Pooks.](https://reader036.fdocuments.us/reader036/viewer/2022062308/56649f045503460f94c183dc/html5/thumbnails/14.jpg)
Kahneman D. Thinking Fast and Slow. New York: Farrar, Strauss, and Giroux, 2011 (220)
“People who attempt to predict the future make many errors; that goes without saying. Errors of prediction are inevitable because the world is unpredictable. High subjective confidence is not to be trusted as an indicator of accuracy. Behavior and achievements can be predicted with fair accuracy from previous behavior and achievements.”
-- Daniel Kahneman
Should We Be Surprised?
![Page 15: Michael S. Lauer, MD, FACC, FAHA Director, Division of Cardiovascular Sciences National Heart, Lung, and Blood Institute CSR Advisory Council Meeting Pooks.](https://reader036.fdocuments.us/reader036/viewer/2022062308/56649f045503460f94c183dc/html5/thumbnails/15.jpg)
15
Taking the Powerball Approach to Funding Medical ResearchWinning a government grant is already a crapshoot. Making it official by running a lottery would be an improvement
By Ferric C. Fang And Arturo CasadevallApril 14, 2014
“And NIH's peer review has been criticized. The latest critique is a landmark study just published by Michael Lauer and colleagues at the NHLBI. Other studies have reported that NIH peer review lacks statistical rigor … We suggest another approach …”
Reactions
![Page 16: Michael S. Lauer, MD, FACC, FAHA Director, Division of Cardiovascular Sciences National Heart, Lung, and Blood Institute CSR Advisory Council Meeting Pooks.](https://reader036.fdocuments.us/reader036/viewer/2022062308/56649f045503460f94c183dc/html5/thumbnails/16.jpg)
Gordon D et al. N Engl J Med 2013;369:1926-34
Different Case …
![Page 17: Michael S. Lauer, MD, FACC, FAHA Director, Division of Cardiovascular Sciences National Heart, Lung, and Blood Institute CSR Advisory Council Meeting Pooks.](https://reader036.fdocuments.us/reader036/viewer/2022062308/56649f045503460f94c183dc/html5/thumbnails/17.jpg)
Gordon D et al. N Engl J Med 2013;369:1926-34
Predictors of time to publication were a focus on clinical events and higher costs up to approximately $5 million. Above $5 million, the cost of the trial was no longer a significant determinant of the time to publication.
What Factors Predict? Endpoints & Investments
![Page 18: Michael S. Lauer, MD, FACC, FAHA Director, Division of Cardiovascular Sciences National Heart, Lung, and Blood Institute CSR Advisory Council Meeting Pooks.](https://reader036.fdocuments.us/reader036/viewer/2022062308/56649f045503460f94c183dc/html5/thumbnails/18.jpg)
Gordon D et al. N Engl J Med 2013;369:1926-34
… And Not So Much What is Often Claimed
![Page 19: Michael S. Lauer, MD, FACC, FAHA Director, Division of Cardiovascular Sciences National Heart, Lung, and Blood Institute CSR Advisory Council Meeting Pooks.](https://reader036.fdocuments.us/reader036/viewer/2022062308/56649f045503460f94c183dc/html5/thumbnails/19.jpg)
Devereaux PJ, Yusuf S. N Engl J Med 2013;369:962-
“Change in priorities of funders, an appreciation from review panels of the importance of [large trials focused on clinical endpoints], and avoidance of inefficient and unimportant aspects of trial design…”
Reactions …
![Page 20: Michael S. Lauer, MD, FACC, FAHA Director, Division of Cardiovascular Sciences National Heart, Lung, and Blood Institute CSR Advisory Council Meeting Pooks.](https://reader036.fdocuments.us/reader036/viewer/2022062308/56649f045503460f94c183dc/html5/thumbnails/20.jpg)
Kaiser J. Science July 25, 2014
NIH Considering Alternate Models
![Page 21: Michael S. Lauer, MD, FACC, FAHA Director, Division of Cardiovascular Sciences National Heart, Lung, and Blood Institute CSR Advisory Council Meeting Pooks.](https://reader036.fdocuments.us/reader036/viewer/2022062308/56649f045503460f94c183dc/html5/thumbnails/21.jpg)
Nature 2010;465:864-6 and Nature 2013;495:437-40
“A second criticism is that the very idea of quantifying scientific impact is misguided. This really will not do. We scientists routinely search out numerical data; we cannot insist that our cogitations are uniquely exempt. The ultimate judge of scientific quality is the scientific community; its judgments are expressed in actions and these actions may be measured.”
Acknowledge Discomfort and Grow
![Page 22: Michael S. Lauer, MD, FACC, FAHA Director, Division of Cardiovascular Sciences National Heart, Lung, and Blood Institute CSR Advisory Council Meeting Pooks.](https://reader036.fdocuments.us/reader036/viewer/2022062308/56649f045503460f94c183dc/html5/thumbnails/22.jpg)
Azoulay P. Nature 2012;483:31-2
“The current system already gambles with scientific careers, just in a haphazard way. … These experiments could be rolled out with minimal disruption. If [we] could test even a small number of hypotheses, the system-wide benefits would dwarf [the] modest investment.”
Should We Consider …
![Page 23: Michael S. Lauer, MD, FACC, FAHA Director, Division of Cardiovascular Sciences National Heart, Lung, and Blood Institute CSR Advisory Council Meeting Pooks.](https://reader036.fdocuments.us/reader036/viewer/2022062308/56649f045503460f94c183dc/html5/thumbnails/23.jpg)
Mervis J. Science 2014;343:596-598
Final Thoughts …
![Page 24: Michael S. Lauer, MD, FACC, FAHA Director, Division of Cardiovascular Sciences National Heart, Lung, and Blood Institute CSR Advisory Council Meeting Pooks.](https://reader036.fdocuments.us/reader036/viewer/2022062308/56649f045503460f94c183dc/html5/thumbnails/24.jpg)
Gary Gibbons Susan Shurin Richard Hodes Nancy Miller Josie Briggs Jim Kiley Keith Hoots Donna DiMichele Sean Coady Susan Pucie Terrie Wheeler Lisa Federer Phil Lee John Ioannidis Mona Puggal Jon Kaltman Monica Shah Josephine Boyington Steve Mockrin
Simhan Danthi Colin Wu Peibei Shi Frank Evans David Gordon Peter Kaufmann Melissa Antman Alice Mascette Wendy Taddei-Peters Ebyan Addou Zorina Galis Gina Wei Caroline Fox Richard Fabsitz Teri Gosselin Gail Pearson Denis Buxton Jodie Black Meena Hiremath
Thank You!
![Page 25: Michael S. Lauer, MD, FACC, FAHA Director, Division of Cardiovascular Sciences National Heart, Lung, and Blood Institute CSR Advisory Council Meeting Pooks.](https://reader036.fdocuments.us/reader036/viewer/2022062308/56649f045503460f94c183dc/html5/thumbnails/25.jpg)