Of˜ce of Portfolio Analysis Symposium - DPCPSI OPA... · 2017-06-16 · ii 2017 Office of...

Wednesday, March 15th

Natcher Conference Center

8:30 am – 4:30 pm

Of�ce of PortfolioAnalysis Symposium

Office of Portfolio Analysis

2017

2017 Office of Portfolio Analysis Symposium i

Table of Contents

Note from the Office of Portfolio Analysis Director ............................................................................ ii

Office of Portfolio Analysis Overview and Services ............................................................................ iii

Meeting Agenda .................................................................................................................................. iv

Posters by Number ................................................................................................................................1

Poster Abstracts .....................................................................................................................................7

Authors by Name ................................................................................................................................63

ii 2017 Office of Portfolio Analysis Symposium

From the Office of Portfolio Analysis Director

Welcome to the 2017 Office of Portfolio Analysis Symposium! We are thrilled to host this event as part of our continued effort to support data-driven decision making at the NIH. We encourage you to take full advantage of this opportunity to share ideas with the NIH portfolio analysis community, learn new analytical approaches, and forge new connections with your colleagues. Indeed, numerous analyses and collaborations began as an informal conversation at previous symposia. If there is anything we can do to help you get the most out of your 2017 Symposium experience, please let us know!

Dr. George Santangelo Director Office of Portfolio Analysis

2017 Office of Portfolio Analysis Symposium iii

Office of Portfolio Analysis Overview and Services

The Office of Portfolio Analysis (OPA) was established in 2011, and is part of the Division of Program Coordination, Planning, and Strategic Initiatives (DPCPSI) within the Office of the NIH Director (OD).

OPA is an interdisciplinary team, led by Dr. George Santangelo, that impacts NIH-supported research by enabling NIH decision makers and research administrators to evaluate and prioritize current and emerging areas of research that will advance the NIH mission. OPA services include training, tool development, and coordination of trans-NIH portfolio analysis activities.

Training and ConsultationsOPA’s training team teaches and supports portfolio analyses across NIH by offering classroom training and workshops. OPA also regularly offers one-on-one consultations to NIH staff seeking guidance on portfolio analysis (see poster #1 for further details).

Tool developmentOPA has a team of talented developers building new specialized data tools to help expand NIH-wide efforts in portfolio analysis. OPA’s currently available tools are:

• iSearch: the OPA next-generation portfolio analysis platform provides easy-to-use access to an extensively linked dataset of grants, publications, patents, clinical trials, and drugs

• iCite: provides a dashboard of bibliometrics for papers associated with a portfolio, calculating Relative Citation Ratio (RCR) values, which estimates the influence of each paper on its field of research

• iTrans: maps published articles along three axes—cellular/molecular research, animal research, and human research—to visualize translation from bench to bedside

• iClean: enables users to rapidly disambiguate a list of names

Coordination of portfolio analysis activities at NIHOPA actively coordinates portfolio analysis activities across NIH and enhances collaboration among all portfolio analysis stakeholders by hosting meetings and providing a variety of venues that encourage NIH staff to share ideas and approaches to portfolio analysis.

iv 2017 Office of Portfolio Analysis Symposium

Meeting Agenda MORNING

TIME SESSION ROOM

8:30-9:00 a.m. Check-in and registration Lower lobby

9:00-9:30 a.m. Welcome and overview

George Santangelo, Ph.D., Director, Office of Portfolio Analysis, DPCPSI

Main auditorium

9:30-10:00 a.m. OPA iTools: iSearch

Paula Fearon, Ph.D., Office of Portfolio Analysis, DPCPSI

Main auditorium

10:00-10:30 a.m. OPA iTools: iCite

B. Ian Hutchins, Ph.D., Office of Portfolio Analysis, DPCPSI

Main auditorium

10:30-11:00 a.m. OPA iTools: iTrans


Main auditorium

11:00-11:15 a.m. Coffee break Lower lobby

11:15 a.m.- 12:15 p.m. Posters – Session 1 (odd numbers) Rooms C, E, and G

12:15-1:00 p.m. Lunch

2017 Office of Portfolio Analysis Symposium v

Meeting Agenda AFTERNOON

TIME SESSION ROOM

1:00-2:00 p.m. Posters – Session 2 (even numbers) Rooms C, E, and G

2:00-2:30 p.m. Featured Talk #1: Shifting demographics among NHLBI Research Project Grant awardees

Marc Charette, Ph.D., Vascular Biology & Hypertension Branch, National Heart, Lung, and Blood Institute (NHLBI)

Main auditorium

2:30-3:00 p.m. Featured Talk #2: Quantifying and predicting knowledge flow in translational science

B. Ian Hutchins, Ph.D., Office of Portfolio Analysis, DPCPSI

Main auditorium

3:00-3:30 p.m. Featured Talk #3: Scoring the impact of NIDA targeted solicitations, 1994-2014

Elena Koustova, Ph.D., Office of Translational Initiatives and Program Innovations, Office of the Director, National Institutes of Drug Abuse (NIDA)

Main auditorium

3:30-4:00 p.m. Featured Talk #4: Systematic analysis of research organisms employed by the biomedical community

Kristine Willis, Ph.D., Genetics Mechanisms Branch, National Institute of General Medical Sciences (NIGMS)

Main auditorium

4:00-4:30 p.m. Summary and closing remarks


Main auditorium

vi 2017 Office of Portfolio Analysis Symposium

Posters by Number

2017 Office of Portfolio Analysis Symposium 1

POSTERS BY NUMBER

All posters will be presented in Natcher Rooms C1/C2, E1/E2, and G1/G2

• Posters with odd numbers will present at Poster Session 1 (11:15 a.m. to 12:15 p.m.)

• Poster with even numbers will present at Poster Session 2 (1:00 p.m. to 2:00 p.m.)

POSTER NO. ABSTRACT TITLE

PRESENTING AUTHOR

ABSTRACT PAGE NO. ROOM

1 The OPA portfolio analysis training program Perkins, Matthew 7 E1/E2

2Using contributions of public dataset as metrics to enhance evaluation of research outcomes – a prototype using the Protein Data Bank (PDB)

Mao, Fenglou 8 E1/E2

3 Using co-author networks to analyze the extent of collaboration among principal investigators

Forcinito, Patricia 9 E1/E2

4Assessing the impact of the knockout mouse phenotyping program (komp2) on the biomedical research enterprise

Jose, Sheethal 10 E1/E2

5 Demographic shift among NIAID RPG awardees Parker, Marie 11 E1/E2

6Using relative citation ratio to assess grant mechanism: solicited grants result in more influential publications

Tiano, Joseph 12 E1/E2

7 The secret of surviving in the niche market Sazonova, Irina 13 E1/E2

8 Towards accurate assessment of the productivity of NIH investments Litovitz, Aviva 14 E1/E2

9 Effects of implementing capping on PI commitment at NIAID Ghosh, Dolan 15 E1/E2

10 Evidence of impact: trace studies and data sources Volkov, Marina 16 E1/E2

11 Shifting demographics among NHLBI research project grant awardees Charette, Marc 17 E1/E2

2 2017 Office of Portfolio Analysis Symposium

POSTERS BY NUMBER


PRESENTING AUTHOR


12 Shared instrumentation grants (S10): impact on NIH-funded research Kalantari, Roya 18 E1/E2

13

Shifting demographics among research project grant awardees at the National Heart, Lung, and Blood Institute (NHLBI): analysis by gender

Maric-Bilkan, Christine 19 E1/E2

14 Estimating the scale of biomedical data generation using text mining

Rosenfeld, Gabriel 20 E1/E2

15A potential solution to establish a stable investigator pipeline: by expanding the size of the pie?

Oh, Young 21 E1/E2

16Evaluating the impact of NIH-funded research on IARC monographs, NCCN guidelines and selected USPSTF recommendations

Dathe, Marina 22 E1/E2

17 Does recommending to resubmit really enhance the quality SBIR/STTR applications?

Prikhodko, Victor 23 E1/E2

18 Scoring the impact of NIDA targeted solicitations, 1994-2014

Koustova, Elena 24 E1/E2

19 A computational approach to study R21-R01 relationship Lin, Dawei 25 E1/E2

20 RCDC categorization of research outputs Baker, Kirk 26 E1/E2

21 Evaluating translational research – beyond bench to bedside

Pettibone, Kristi 27 E1/E2

22 NINDS Office of Pain Policy – interagency pain research portfolio database – 2011-2014

Sankar, Cheryse 28 E1/E2

23 Methods for clustering NIH grants using word2vec Hoppe, Travis 29 E1/E2


POSTERS BY NUMBER


PRESENTING AUTHOR


24Using RCDC categorization data to measure basic and applied expenditures at NIH from 2009-2015

Scholl, Jacob 30 E1/E2

25 Quantifying and predicting knowledge flow in translational science

Hutchins, B. Ian 31 E1/E2

26

Bibliographic tools and approaches for identifying research trends and milestone publications: a demonstration focused on the Framingham Heart Study

Dodson, Sara 32 E1/E2

27Using social network analysis to follow the development of the BPA grantee portfolio publications

Ruben, Elizabeth 33 E1/E2

28Program project grant mechanisms (P01 and P50/SPOREs) outperform R01 research project grants in citation productivity

Kuzmin, Igor 34 E1/E2

29 Mapping the alignment of NIH’s portfolio to burden of disease Bochner, David 35 G1/G2

30 Systematic analysis of research organisms employed by the biomedical community Willis, Kristine 36 G1/G2

31 Identifying prevention research using a machine learning approach Villani, Jennifer 37 G1/G2

32Portfolio analysis of NIH-funded research on 1918 influenza in the context of DURC policies

Finger, Matthew 38 G1/G2

33 A state of the technology assessment for deep brain stimulation Kuszak, Adam 39 G1/G2

34 Fully characterizing the NIH prevention research portfolio

Myles, Ranell L. 40 G1/G2

35 Vulnerable plaque research portfolio analysis Chen, Jue 41 G1/G2


POSTERS BY NUMBER


PRESENTING AUTHOR


36Angiogenesis research: the extramural portfolio supported by the National Heart, Lung, and Blood Institute, 2008-2015

Gao, Yunling 42 G1/G2

37 Lymphatics research supported by NHLBI and its impact Khan, Faiyaz 43 G1/G2

38Portfolio analysis of the current NIH investment in single cell analysis research in human cells and tissues

Smith, Jessica M 44 G1/G2

39 Analysis of NIAID’s type 1 diabetes (T1D) portfolio Zane, Ariel 45 G1/G2

40 Natural products research applications and awards at NIH: a five year analysis

Berzhanskaya, Julia 46 G1/G2

41 Data towards finding research gaps and opportunities in the health of women Coles, Eric 47 G1/G2

42 NCI-funded research in India Prakash, Laura 48 C1/C2

43Portfolio analysis with limited data: a case study on training program effectiveness at a smaller IC

Zarcone, Troy 49 C1/C2

44NCI-funded research in China: a case-study for analyzing NCI international research

Abudu, Rachel 50 C1/C2

45

Portfolio analysis of the Clinical and Translational Science Award (CTSA) program’s institutional mentored career development program (2006-2014)

Baker, Heather L 51 C1/C2

46 A framework for creating regional analyses of NIH-funded international work Haney, Karen 52 C1/C2


POSTERS BY NUMBER


PRESENTING AUTHOR


47R01 and RPG award activity among K-award recipients: a cohort analysis of assistant professors at U.S. medical schools

Doyle, Jamie 53 C1/C2

48 New methods and tools for visualizing and analyzing scientific and budget data

Botero, Sebastian 54 C1/C2

49 Portfolio analysis in the Center for Research Strategy, National Cancer Institute

Antman, Melissa 55 C1/C2

50 An analysis of NIH-funded research and publications in Africa

Harriman, Robert 56 C1/C2

51The picture could be rosier – an in-depth look at NIH RePORTER success rates for NCI training grants

Torres, Sasha 57 C1/C2

52 NIH Internal Research Portfolio Online Reporting Tools (iRePORT)

Danielson, Cindy 58 C1/C2

53 Using propensity score methods to evaluate the early stage investigator policy Walsh, Rachael 59 C1/C2

54 Outcomes Clustering Analysis Tool (OCAT) Zoha, Steven 60 C1/C2

55Using the RCDC portfolio visualization (PVIZ) module to understand trends in NIH data

Lobo, Jessica 61 C1/C2


Poster Abstracts


POSTER ABSTRACTS

Poster 1 ROOM E1/E2

The OPA portfolio analysis training program

Perkins, Matthew; Fearon, Paula; Forcinito, Patricia; Hutchins, B. Ian; Santangelo, GeorgeOffice of Portfolio Analysis, Office of the Director

As part of the mission of Office of Portfolio Analysis (OPA), we offer a variety of portfolio analysis training opportunities for NIH staff. We conduct classroom training in the basics of portfolio analysis (Portfolio Analysis 101), the IN-SPIRE text analysis and clustering software, and OPA-developed portfolio analysis tools, including iSearch. We also hold workshops on bibliometrics and translational science, and we are currently developing a workshop on network analysis. In addition to our more formal classroom and workshop offerings, we provide tips and tutorials on the OPA training website. We plan to add case studies to the training website in the near future. NIH staff seeking guidance on portfolio analysis are welcome to set up a consultation with OPA or drop in at the office hours we hold every other week at various NIH sites. Further detail on all of our training activities is available on the OPA training website: https://opa-trainingportal.od.nih.gov/.


POSTER ABSTRACTS

Poster 2 ROOM E1/E2

Using Contributions of Public Dataset as Metrics to Enhance Evaluation of Research Outcomes – A Prototype using the Protein Data Bank (PDB)

Mao, Fenglou; Lin, DaweiNational Institute of Allergy and Infectious Diseases

Publication- and citation-related metrics are well-established standards for evaluating biomedical research outcomes of individual principal investigators (PI), programs, or portfolios. However, we believe that metrics assessing scientific contribution to public repository datasets may contribute additional value to these evaluations. To prototype the development of such metrics, we chose to examine the Protein Data Bank (PDB). Established in 1971, the PDB is the single worldwide archive of structural data of biological macromolecules, and therefore has a long history with broad impact. We extracted PMID (PubMed Identifier) information found in the associated publications of all 126,081 PDB structures (at the time of downloading) and developed tools to integrate this information with publication data and NIH grant information from SPIRES and PubMed. From this analysis, we were able to generate a variety of metrics to evaluate the contribution of the PDB and the submitting researchers to scientific research. For example, we computed how many 3D protein or DNA structures were funded under each NIH grant. Using the institute information of the author combined with Google Maps, we visualized the contribution of all structures worldwide. We calculated many other metrics from our analysis that include: the number of datasets contributed by individual PIs; the number of datasets contributed by awards; the number of datasets contributed by funding programs; and the number of data.


POSTER ABSTRACTS

Poster 3 ROOM E1/E2

Using co-author networks to analyze the extent of collaboration among principal investigators

Forcinito, Patricia; Meseroll, Rebecca; Fearon, Paula; Palmer, Jason; Harriman, Rob; Santangelo, George Office of Portfolio Analysis, Office of the Director

This analysis examined the extent of collaboration among NIH-funded principal investigators (PIs) in the NIH Pain Consortium Pain portfolio, and compared it to that of four RCDC categories (control groups). We also analyzed PI collaboration for the ten “pain condition” subcategories within the Pain portfolio that had the largest number of awards. We analyzed the extent of collaboration by creating co-author networks for NIH PIs in the Pain portfolio and in each of the control groups, based on the assumption that two PIs who publish together have established some degree of collaboration. The level of collaboration in the Pain portfolio was similar to that of three of the control groups, but considerably less robust than the Alzheimer’s control group. We also observed that PIs in the largest connected components (LCCs) of all co-author networks had higher median Relative Citation Ratio (RCR) values, i.e. published more influential papers, compared to their less-connected or isolated peers. Our analysis of the top ten pain conditions similarly demonstrated that, in general, more influential publications were associated with the conditions that had a higher percentage of PIs in the three Pain network LCCs. However, this increased collaboration surprisingly did not correlate with the fraction of articles cited by at least one CT/CG. We did, however, observe a positive correlation with both actual and predicted translation was a higher rate of publications, or (independently) the fraction of human MeSH terms associated with those publications.


POSTER ABSTRACTS

Poster 4 ROOM E1/E2

Assessing the Impact of the Knockout Mouse Phenotyping Program (KOMP2) on the Biomedical Research Enterprise

Jose, Sheethal1; Lu, Ya-Ling2; Lenzi, Rebecca3; Singh, Bishen4; Mirochnitchenko, Oleg3; Moore, Mark1; Fletcher, Colin1

1National Human Genome Research Institute2NIH Library3Office of the Director4National Institute of Mental Health

The Knockout Mouse Phenotyping Program (KOMP2) is a Common Fund/trans-NIH initiative that aims to generate a resource of mice containing a null (“knockout”) mutation in every gene in the mouse genome, to collect phenotype information for each mouse strain, and to make the materials and data available to the research community. KOMP2 plans to phenotype 6,000 mouse strains using a broad-based, unbiased set of tests. It is anticipated that this data and material resource will enhance the ability of researchers to translate basic genetic research to applications relevant to human health. To study the impact of the KOMP2 program on the scientific community, we sought to identify publications that used materials or data from KOMP. We have identified over 1,100 relevant publications. A bibliometric analysis of these publications was performed to evaluate the productivity and citation impact. In addition, these publications are being matched to the most relevant NIH grants that supported the work, in order to identify which Institutes and Centers have benefited from these resources. Ultimately, we hope to identify research grants and publications that report findings on previously unexamined genes, or present new findings on ‘known’ genes and measure the impact of KOMP2 resources on advancing scientific research.


POSTER ABSTRACTS

Poster 5 ROOM E1/E2

Demographic Shift among NIAID RPG Awardees

Parker, Marie; Ghosh-das, DolanNational Institute of Allergy and Infectious Diseases

NIAID undertook an analysis to determine whether mid-career investigators experience “disproportionate” difficulties in the advancement of their professional careers. This was in response to an analysis done by OER and NHLBI on NHLBI (and NIH) RPG awardees which showed a significant and progressive shift in the demographics of RPG awardees towards an older population. PII data was provided by OER on all NIAID PIs for RPG awards (R, P, and U activities) between FY 1998 and FY 2015. Age was grouped as “Early Stage” (24-40 years), “Mid-Career” (41-55 years) and “Established” (56+ years). Between FY 1998 and FY 2015, overall the number of unique NIAID RPG awardees increased. However, the number of Early Stage PIs declined while the number of Established PIs increased over this time period. When viewed as a proportion of overall RPG awardees, both Early Stage and Mid-Career PIs show a downward trend while Established PIs show a rapid rise. We also found that Established PIs have a disproportionately higher average RPG award amount compared to the Early Stage and Mid-Career PIs. In summary, the NIAID results are similar to the NHLBI and NIH results in that there is a progressive shift in demographics towards an older population of RPG awardees. Established PIs have disproportionate average number of awards and amount of funding, with the trend continuing to grow over time. The net outcome is a selective loss of investigators in the 24-40 age group.


POSTER ABSTRACTS

Poster 6 ROOM E1/E2

Using Relative Citation Ratio to assess grant mechanism: solicited grants result in more influential publications

Tiano, Joseph; Ferland-Beckham, Chantelle; Horsford, JonathanNational Institute of Dental and Craniofacial Research

Program Announcements and Requests for Applications are common Funding Opportunity Announcements NIH uses to solicit grant applications. PAs are primarily used to request investigator initiated applications (unsolicited applications) that are not targeted to specific program priorities. In contrast, RFAs are solicited applications within a more narrowly defined research area identified by NIH program staff. We analyzed R01 and R21 awards and asked does grant mechanism impact the resulting publications influence, and are publications from unsolicited and solicited awards similarly influential? Publication influence was assessed using Relative Citation Ratio (RCR). An analysis of publications from R01s and R21s from 10 diverse NIH ICs showed that, irrespective of solicitation type, R01s and R21s produced similarly influential publications; and regardless of grant mechanism, solicited awards produced more influential publications than unsolicited awards. However, with respect to R21s, NIDCR does not conform to the average of 10 ICs. While solicited R21 publications from 10 ICs had higher RCR values than unsolicited R21s, NIDCR solicited R21s produced publications with lower RCR values than unsolicited R21s. In conclusion, spanning 10 diverse NIH ICs, R01 and R21 grant mechanisms do not appear to impact publication influence. However, the avenue of grant solicitation does appear to impact publication influence; for most ICs solicited grants produce publications with higher RCRs.


POSTER ABSTRACTS

Poster 7 ROOM E1/E2

The Secret of Surviving in the Niche Market

Sazonova, Irina; Prikhodko, Victor; Slipchenko, Tamara; Koustova, Elena National Institute on Drug Abuse

Current pharma/industry commitment to commercialize products and services for substance use disorders (SUD) is woefully inadequate relative to the magnitude of the SUD burden. As a result, NIDA’s SBIR/STTR program is one of the few major funding sources for SUD-related R&D. However, the quantity and the quality of NIDA SBIR/STTR applications were affected by the perceived lack of serviceable market and the stigma associated with the SUD. Historically, the non-discussed rate for the scarce applications to NIDA was as high as 62%. To improve the NIDA small business program, OTIPI introduced 5 new management and technical assistance practices. To determine the impact of the new practices, we conducted a retrospective analysis of 212 SBIR/STTR applications submitted in the pre- (10/2012- 01/2015) and 200 applications submitted in post- (05/2015 -10/2016) “change of practice” periods. All 412 applications were analyzed based on the activity code, technology type, priority score, and funding outcomes. The analysis revealed that the number of SBIR/STTR grant applications has tripled since FY2013. Applicants who received technical assistance had a better priority score (6 points, P=.005), with Research Tool (8 points, P<0.05) and Therapeutic (12 points, P<0.05) -focused applications benefiting the most. The presented analysis provides empirical evidence for the ability of novel management practices to significantly improve the research program.


POSTER ABSTRACTS

Poster 8 ROOM E1/E2

Towards accurate assessment of the productivity of NIH investments

Litovitz, Aviva; Santangelo, GeorgeOffice of Portfolio Analysis, Office of the Director

The Office of Portfolio Analysis (OPA) is developing a variety of metrics that reveal the relationship between NIH investments and productivity. Part of this effort involves formulating a method that can accurately assess NIH investments on a “dollars per publication” basis. Our approach fractionates publications to avoid double counting, and accounts for the temporal relationship between grants and publications, with funding closely preceding publication more relevant than funding many years prior. We used this method to calculate annual funding and productivity for all Research Program Grants (RPGs). Productivity of principal investigators per dollar declines at higher funding levels, providing evidence of diminishing returns. These results are consistent with similar analyses elsewhere at NIH, and complement them by using fractionated credit for publications and RPG dollars to analyze productivity at the level of PIs, portfolios, mechanisms, etc. We plan to add a module to the OPA suite of iTools that employs this method and enables NIH decision-makers to accurately track the productivity of NIH investments at their desktops.


POSTER ABSTRACTS

Poster 9 ROOM E1/E2

Effects of Implementing Capping on PI Commitment at NIAID

Ghosh, Dolan; Parker, MarieNational Institute of Allergy and Infectious Diseases

The NIH Working Group on Policies for Efficient and Stable Funding presented the idea of Research Commitment Index (RCI), which proposes that by limiting the number grants an investigator can hold, NIH could free up additional funding to support investigators who would otherwise go unfunded. We examined the effects of the RCI on NIAID researchers. The RCI is an algorithm that assigns points to grants based on activity codes and whether the grant has multiple PIs. The Working Group proposed a cap for PIs set at 20 points. First, we looked at how many researchers would be affected by this policy. Approximately 12% of NIAID PIs exceed the proposed 20-point threshold. For PIs with only R01s, this percentage is even smaller. If we were to cap those PIs so as not to exceed 20 points, additional awards could be made to other investigators. However, there is a likelihood that these awards could be distributed among investigators who already have an existing award but are below the threshold. Among the researchers who exceed the 20-point threshold are many of our most successful investigators. While funding a higher percentage of investigators through capping, may in principle expand the investigator pool, doing so represents a departure from our current funding practices, which are designed to identify and fund the best research project proposals that meet our programmatic needs.


POSTER ABSTRACTS

Poster 10 ROOM E1/E2

Evidence of Impact: Trace studies and data sources

Volkov, Marina; Dodson, Sara; Bochner, David; Kosub, David; Reczek, Peter; Alexander, Kristine; Baumgart, Joel; Baden, ElizabethOffice of Science Policy, Office of the Director

The OD Office of Science Policy has conducted a series of case studies using a systematic approach to tracing the impact of NIH-funded research. In collaboration with numerous IC and OD offices, four studies were undertaken – on childhood Hib vaccines, neurostimulation technologies, cancer & molecular medicine, and the Framingham Heart Study – tracing the chain of evidence between discoveries to longer-term health, society, and knowledge impacts. The goals of this exercise were two-fold: 1) to develop a systematic approach to producing compelling communication pieces which can tell complex scientific stories of research progress and its ensuing impacts to society, and 2) to map the various data sources which NIH should be drawing from on a regular basis to improve our capacity for assessing research outcomes and impact. In this report, we outline and discuss the diverse data sources utilized in these trace studies, including grants data, research publications, press releases, patents, FDA approvals, clinical guidelines, policy and regulatory decisions, industry reports, economic analyses, medical expenditures, and public health statistics. In piecing these disparate sources together, some clear needs emerged with respect to improving NIH’s capacity to link grants data to structured data from other Federal sources (such as FDA approvals, clinical trials and guidelines, or legal, regulatory, and policy documents), which would greatly facilitate future analyses.


POSTER ABSTRACTS


Shifting Demographics among NHLBI Research Project Grant Awardees

Charette, Marc; Oh, Young; Maric-Bilkan, Christine; Scott, Lindsey; Wu, Charles; Eblen, Mathew; Pearson, Katrina; Tolunay, Eser; Galis, ZorinaNational Heart, Lung, and Blood Institute

In examining the causes and consequences of certain demographic shifts among NHLBI RPG awardees we find that that the RPG system itself contains structural properties that are gradually and steadily creating demographic changes that, in turn, are diminishing the ability of NHLBI and the NIH to maintain a stable pipeline of investigators. One result of these demographic shifts has been a significant and selective reduction in the number of awards to mid-career NHLBI investigators and a corresponding reduction in the proportion of mid-career investigator directed laboratories. This result may decrease the diversity of independent investigators and may decrease the diversity of scientific approaches to many important biomedical problems.


POSTER ABSTRACTS


Shared Instrumentation Grants (S10): Impact on NIH-funded Research

Kalantari, Roya; Levy, Abraham; Horska, Alena; Klosek, MalgorzataOffice of Research Infrastructure Programs, Office of the Director

The assessment of the impact that NIH-funded resources have on enabling research conduct, opening new research opportunities, and facilitating transformative biomedical discoveries is pertinent to programs supported by all ICs. In this poster, we show how we access data and what elements we consider in our annual assessment of a resource program with NIH-wide impact. The Office of Research Infrastructure Programs (ORIP) supports the S10 Shared Instrumentation program, which is unique within the NIH. The S10 program provides funding for state-of-the-art, commercially available instruments which are used on a shared basis, typically with 10-15 users per instrument. Awards, for the purchase of the instruments only, are issued for one year; however, the institutions are expected to provide extended support for the instrument, including infrastructure, technical personnel, and maintenance. This creates a unique challenge in monitoring the impact of the awards beyond the first year. Currently submitted applications include 5 years’ worth of data on previous S10 awards issued to the applicant institutions. In FY2017, this data encompassed 73% of all awarded instruments from 2011-2015 and provides details on the status and number of hours the S10 instruments are used annually, the maintenance agreements on the instruments, and the number of associated publications. We find that instruments are in regular use, well maintained, and enhance research funded by all NIH ICs.


POSTER ABSTRACTS


Shifting Demographics among Research Project Grant Awardees at the National Heart, Lung, and Blood Institute (NHLBI): analysis by gender

Maric-Bilkan, Christine; Charette, Marc; Tolunay, Eser; Galis, Zorina; Scott, Lindsey; Wu, Charles; Pearson, KatrinaNational Heart, Lung, and Blood Institute

Our recent analyses showed that, since 1998, there has been a linear increase in the proportion of NHLBI-funded established investigators (≥56 years of age); while the proportion of mid-career investigators (41-55 years) has been declining and the proportion of early-stage investigators (24-40 years) has been relatively stable. The aim of the present study was to analyze potential gender differences in these observations. Between 1998-2014 at NHLBI, the proportion of early-stage investigators has been constant among both men and women. Among men, there has been a decline in the proportion of mid-career investigators and an increase in established investigators. In women, the proportion of mid-career and established investigators has been constant over this time period. The overall proportion of female investigators was lower compared with male in all ages groups and this does not appear to be, for the most part, due to differences in the success of getting funding, as the award rates were similar between men and women across all age groups. While the number of early-career and established investigators funded by NHLBI has been constant between 2008-2014, the number of mid-career investigators has been declining, most notably among men. Our data demonstrate a demographic shift towards a more established investigator population and a decline in the proportion and number of mid-career investigators funded by NHLBI. This decline is particularly evident in men.


POSTER ABSTRACTS


Estimating the scale of biomedical data generation using text mining

Rosenfeld, Gabriel; Lin, DaweiNational Institute of Allergy and Infectious Diseases

Recent estimates suggest that upwards of ~88% of biomedical data being produced is “invisible” and is not being deposited in data repositories where it is easily accessible to the wider research community. It is incumbent upon the stakeholders of the biomedical data ecosystem to have a better understanding of what data is out there and how much of it there is. Such information is critical to development of data-driven management and sharing of biomedical data to support key goals such as increased rigor and reproducibility. This is a challenging problem and one in which we attempted to address by using advances in word embedding to identify the technical and methodological features of terms used in the free text of articles’ methods sections. We created term usage signatures for five types of biomedical research data, which were used in univariate clustering to correctly identify a large fraction of positive control articles as well as a set of manually annotated articles where generation of particular data types could be confirmed. We estimated the fraction of PLOS articles generating each biomedical data type over time. Out of 129, 918 PLOS articles analyzed, ~7%, 19%, 12%, 18%, and 6% generated flow cytometry, immunoassay, genomic microarray, microscopy, and high-throughput sequencing data respectively. The analysis suggests that in 2016, 40,000 NIH-funded research articles produced ~56,000 datasets consisting of the five data types we studied.


POSTER ABSTRACTS


A Potential Solution to Establish a Stable Investigator Pipeline: By Expanding the Size of the Pie?

Oh, Young; Galis, ZorinaNational Heart, Lung, and Blood Institute

At NIH, a Special Council Review (SCR) policy exists to perform additional consideration of applications from investigators who receive > $1 million (M) in direct costs (DC) from active NIH awards. Different NIH ICs implement this policy variably according to their own policy and strategy. We have examined a model that hypothetically applies a strong funding policy that sets the threshold to all NHLBI investigators. First, we identified a total of 3637 unique Principal Investigators (PIs) who received RPG awards from NHLBI in 2016. Via QVR we found that 335 PIs (9% of total PIs) had received more than $1M DC by NIH RPG awards in 2016. If a strong SCR policy had applied, $254M additional funds would have been available at NHLBI, which could have supported 1016 additional PIs with a modular R01 grant (i.e., $250K DC). When we restricted the RPG awarding Institute only to NHLBI, 152 PIs were identified who received more than $1M DC; and $128M additional funds, equivalent to 512 modular R01s. Next, we analyzed the data limiting to R01 awards only. There were 118 PIs who received more than $1M DC by NIH R01 awards in 2016. If a strong SCR policy were applied, $53M would have been available, equivalent to 212 modular R01s. Restricting the analysis to NHLBI R01 awards only, 59 PIs were identified ($31M additional funds; 124 modular R01s). Therefore, limiting the size of research grant supports per investigator is an effective way of establishing a stable pipeline of investigators.


POSTER ABSTRACTS


Evaluating the impact of NIH-funded research on IARC monographs, NCCN guidelines and selected USPSTF recommendations

Dathe, Marina; Helzlsouer, Kathy; Reid, Britt; Rogers, Scott; National Cancer Institute

Rationale: Representation of research finding in authoritative sources of scientific information is one measure by which to evaluate its translational impact. Goal: To examine the representation and characteristics of articles with evidence of NIH funding in three sources of data: IARC monographs, NCCN guidelines, and cancer-related USPSTF recommendations. Specifically we sought to assess the impact of NIH-funded research in shaping these standards and articles salience using Relative Citation Ratio (RCR) as a proxy for scientific influence. Methods: We used RePARS and a custom-made VBA to link references in IARC monographs, NCCN guidelines and USPSTF recommendations to their unique PubMed Identifier; SPIRES to ascertain underlying NIH funding; and iCite to calculate RCR. Results: 20.0% of IARC monographs references, 21.06% of NCCN guidelines references and 24.6% of USPSTF recommendations references showed evidence of NIH funding. The mean RCR of articles varied substantially across the three datasets (IARC monograph: 2.81 (SD ± 12.25); NCCN guidelines: 6.51 (STD ± 19.25); USPSTF recommendations: 8.72 (STD ± 44.16). NIH funded articles consistently reached a higher mean RCR than articles with no evidence of NIH funding (p<0.001). Discussion: This work sets benchmark measures of translational impact for future evaluations. Differences in articles’ scientific salience based on their funding status suggests NIH funding does have an impact on research quality.


POSTER ABSTRACTS


Does recommending to resubmit really enhance the quality SBIR/STTR applications?

Prikhodko, Victor; Slipchenko, Tamara; Sazonova, Irina; Koustova, ElenaNational Institute on Drug Abuse

National Institute on Drug Abuse allocates approximately $40M to the Small Business Innovation Research (SBIR) and Small Business Technology Transfer (STTR) programs, which operate in a niche market and have historically experienced difficulties in attracting the sufficient number of meritorious applications. Up to 62% of applications are not discussed in review and the rest frequently receive high priority scores. To de-risk the funding decision, program staff often advise resubmission, delaying the funding decision by 9-12 months, the time period in which few small businesses can survive without the capital infusion. To determine the consequences of a resubmission recommendation, we conducted a retrospective analysis of n=513 original (A0) and n=106 resubmitted (A1) SBIR/STTR applications [FY2010-2015]. The analysis revealed that only 16% of non-discussed applications submitted again. For initially discussed applicants that were advised to resubmit, 76% (42/55) of R43 and 56% (18/32) of R44 applicants either did not resubmit or did not improve the priority score upon resubmission. Our result revealed that advice to resubmit merits thorough consideration. Program staff should be cognizant that the probability of not resubmitting, or not improving the priority score is high for small businesses working in NIDA mission space. Overall, the results provide the evidence for the informed risk-benefit assessment of the potential funding.


POSTER ABSTRACTS


Scoring the impact of NIDA targeted solicitations, 1994-2014

Koustova, Elena; Choi, Kristina; Slipchenko, Tamara; Prikhodko, Victor; Sazonova, IrinaNational Institute on Drug Abuse

At NIH, targeted research for well-defined scientific areas is besought through solicitations, RFAs for grants, RFPs for contracts. The projects funded as the result of targeted solicitations are individually evaluated through interim and final reports that include scientific-technical and financial aspects, but do not facilitate the identification of real and potential impacts of the specific RFA or RFP in any amalgamated or standardized fashion. NIDA operates NIH’s largest SBIR contract program, having issued 160 RFPs since the program inception in 1994. We used this unique data set to conduct the in-depth analysis to inform the management of the program, to evaluate and adjust the FOA issuing practices and to test the impact evaluation frameworks. For 146 analyzed RFPs, 24% didn’t generate any interest, 65% led to awards and 31% resulted in creation of identifiable output (product), thus, deemed successful. Three variables contributed to FOA success: research type, solicitation frequency and changes in program management. We also used the Logic and Payback models to score the benefits from targeted research in five domains. Scoring matrix provided a visual depiction for the FOAs having low, medium or high impact. Our analysis productively informs the decision making and provides statistical evidence of the positive effect of new management practices at NIDA. Adapting the existing innovative evaluation methodologies for NIH programs is proven to be possible and beneficial


POSTER ABSTRACTS


A Computational Approach to Study R21-R01 Relationship

Lin, Dawei; Mao, FenglouNational Institute of Allergy and Infectious Diseases

R21 is an NIH grant mechanism that supports short-term exploratory and developmental research. Researchers use it for their high risk and high reward ideas in recent years. In order to understand the impact of R21 mechanism for generating a regular long-term RO1 research project grant, we have developed a text-based computational approach to finding R21-R01 pairs from the same PIs, and then we calculated various scores to measure their relationship, the final goal is to identify if the R21 is a step stone grant of the R01. The manual approach to finding such pairs is challenging. To address the challenge, we use “Research, Condition, and Disease Categorization (RCDC)” signature similarity, or a LIKE-score to serve as a content similarity measure. We introduced a network based neighborhood metrics to further reduce false positives. Using the aforementioned approach, we searched through all 8,975 pairs funded R21 and R01 applications from 2008 to 2014 across NIH based on the same author. We tested 9 cases in DAIT/NIAID verified by the Program Officers who had knowledge of these applications and a test dataset of 140 manually reviewed R21-R01 pairs from NCI. We obtained a precision of ~77% for both tests. Although R21/R01 is the subject of interest for this study, the approach is generic enough to study relationships for other grant mechanisms.


POSTER ABSTRACTS


RCDC Categorization of Research Outputs

Baker, Kirk; Riggie, Judy; Ikeda, Rick, Santangelo, GeorgeOffice of Portfolio Analysis, Office of the Director

As the volume of published scientific information continues to grow, the ability to automatically identify and track research outputs is becoming increasingly important. The Research, Condition, and Disease Categorization (RCDC) Process provides a computerized method the NIH uses to categorize and report on grant funding for each of 233 categories of disease, condition, or research area. We report on an extension of the RCDC process to categorize research outputs including publications and patents. Our analysis focuses on the utility of RCDC categorization for refining acknowledgement of NIH support in publications and patents, and examines the relationship between article-level measures of influence such as Relative Citation Ratio (RCR) and particular RCDC categories.


POSTER ABSTRACTS


Evaluating Translational Research – Beyond Bench to Bedside

Pettibone, Kristi; Balshaw, David; Dilworth, Caroline; Drew, Christie; Heacock, Michelle; Latoni, Alfonso; McAllister, Kimberly; O’Fallon, Liam; Thompson, Claudia; Wright, DemiaNational Institute of Environmental Health Sciences

NIEHS developed a new translational research framework (TRF) for Environmental Health Sciences to provide for a more nuanced understanding of translational research from the basic science phase to health impact. The framework, which builds on previous frameworks and reflects grantee input, changes the graphic representation from linear connections to concentric rings and each ring includes nodes to describe types of activities that might be conducted as part of each translational research phase. Another novel component of the framework is the ability to consider translational research as work that bridges nodes within a ring as well as research that crosses rings. While NIEHS established the TRF to focus on environmental health science research, clinical research is still represented. As a portfolio analysis and evaluation tool, the TRF will enable NIEHS program staff to track research as it moves through the translational research spectrum and to give more “credit” to research that bridges nodes in the basic science ring. This poster will illustrate the TRF, showcase an example of how it can be used to assess the translational research nature of a project and present preliminary ideas for criteria for coding translational research. We have purposely designed the tool to be relevant beyond the environmental health sciences and we seek feedback from symposium participants about its applicability as an analysis tool for other ICs.


POSTER ABSTRACTS


NINDS Office of Pain Policy – Interagency Pain Research Portfolio Database – 2011-2014

Sankar, Cheryse; Pogorzala, Leah; Porter, LindaNational Institute of Neurological Disorders and Stroke

The Interagency Pain Research Coordinating Committee (IPRCC) was established to coordinate pain research efforts within HHS and across other Federal agencies. To support the advancement of pain research, the IPRCC was charged with several tasks, including identification of critical gaps in basic and clinical research on the symptoms and causes of pain, and development of recommendations to ensure that activities of the NIH and other Federal agencies are free of unnecessary duplication of effort. To complete these tasks, the Committee determined that the best approach was a comprehensive analysis of the Federal pain research portfolio. Grants are classified using a multi-tiered system. Tier 1 designates grants as basic, translational, and/or clinical. Tier 2 includes scientific and non-scientific categories within larger themes. Tier 3 categorizes grants by pain condition. A process for data collection, coding, and presentation was adapted for development of an annually updated, searchable database for Federal pain research. In 2015, the NIH library and Office of Pain Policy developed an internal online coding system to enable manual coding of pain grants by designated coders from each NIH institute and federal agency. Using that new system, the database was updated to include 2013-2014 grants. This database has been used extensively to respond to pain-related research initiatives including the Federal Pain Research Strategy, a strategic plan for pain research.


POSTER ABSTRACTS


Methods for clustering NIH grants using word2vec

Hoppe, Travis; Santangelo, GeorgeOffice of Portfolio Analysis, Office of the Director

The abstract and specific aims sections of a grant application are rich sources of semantic, contextual, and categorical information for portfolio analysis. To extract this information from a large number of grants, however, the natural language of the grant must be parsed and encoded in a way a computer can understand. The Office of Portfolio Analysis has constructed an open-source data processing pipeline to analyze and cluster topically-related grants using the “word2vec” embedding scheme. This pipeline handles the pre- and post-processing of data, as well as the clustering, and represents an advance relative to the previous standard methodology (such as keyword analysis, TF-IDF, or IN-SPIRE) by improving recall and model interpretability, and by increasing scalability. As a concrete example, we illustrate the process by clustering competing R01 grant applications over a five-year window.


POSTER ABSTRACTS


Using RCDC Categorization Data to Measure Basic and Applied Expenditures at NIH from 2009-2015

Scholl, Jacob; Riggie, Judy; Parker, MarkDivision of Information Services, Office of the Director

We conducted a feasibility analysis using RCDC categorization data to automate and consistently measure basic and applied research expenditures. In a March 2016, “Open Mike” blog post from Dr. Michael Lauer, NIH Deputy Director for Extramural Research, it was noted that, “Over half of NIH’s research funding supports basic research, as shown in the figure below provided by the NIH Office of Budget.” Our goal was to determine a systematic approach to identify basic and applied research using the official RCDC reporting categories. It would be difficult to gain consensus on a separate basic research reporting category, especially since the combination of existing official reporting categories do not cover the breadth of basic research performed at NIH. Therefore, our method was to first define applied research for Fiscal Years 2009-2015. We utilized the Clinical Research, Translational Research and a separate label for those projects found in “both” categories. These projects are labeled as applied research. The remaining projects across all FYs were labeled as basic research. We further analyzed these basic research projects using metadata to understand if we had found a proper measurement for basic and applied research. Our results align favorably with the Office of Budget figures reported in Dr. Lauer’s blog post, as well as the statements made by NIH senior leadership in the letter to the editor of Science that confirmed the importance of basic research at NIH.


POSTER ABSTRACTS


Quantifying and predicting knowledge flow in translational science

Hutchins, B. Ian; Baker, Kirk; Hoppe, Travis, Santangelo, GeorgeOffice of Portfolio Analysis, Office of the Director

There can be a multi-decade lag between making a basic research discovery and the use of that knowledge in a major clinical study, and the results of that bench-to-bedside translation process are only apparent post-hoc. If earlier markers of translational knowledge transfer were available, it should be possible to accelerate this process. Toward this goal, we have built a machine learning system to detect early signatures that a paper is likely to eventually be cited by a clinical trial or guideline. Despite the noisiness of citation dynamics, we find that as little as two years’ worth of citation information can give accurate predictions about a paper’s eventual citation by a clinical article. By examining the types of papers in an article’s citation network, we find that citation patterns are clustered into common pathways for knowledge flow. These findings capture a fundamental mechanism of knowledge transfer, and demonstrate that translational progress in biomedicine can be assessed and predicted in real time based on the scientific community’s early reaction to a paper.


POSTER ABSTRACTS


Bibliographic tools and approaches for identifying research trends and milestone publications: A demonstration focused on the Framingham Heart Study

Dodson, Sara; Belter, Chris; Alexander, Kristine;Sorlie, Paul;Bochner, David;Baumgart, Joel; Volkov, MarinaOffice of Science Policy, Office of the Director

The OD Office of Science Policy has conducted a series of case studies to pilot various quantitative and qualitative approaches to trace the impact of NIH-funded research. In collaboration NHLBI and the NIH Library, a study of the long-running Framingham Heart Study (FHS) was undertaken – tracing the chain of evidence between “milestone” discoveries to long-term impacts on health, society, and knowledge. The NHLBI-funded investigators at Boston University curate a comprehensive bibliography of FHS-related publications. In order to identify the major research themes and productivity of the study, as well as to isolate the most influential publications and findings, the NIH Library conducted several analyses from the FHS papers using bibliographic data available from Web of Science. In addition to examining citation counts, a bibliographic coupling network was created, which connects papers that share one or more references and sets the strength of the connection to the number of references shared. A community detection algorithm was then applied to identify clusters of topically related papers. Descriptive topics were then manually assigned to each cluster based on the terms that most frequently appeared in the titles of papers in each cluster. Using the bibliographic analyses in tandem with other qualitative determinations of “high impact” research findings, we describe overarching research trends and present a timeline of major milestones from the FHS.


POSTER ABSTRACTS


Using Social Network Analysis to Follow the Development of the BPA Grantee Portfolio Publications

Ruben, Elizabeth; Heindel, Jerold; Shugg, Thaddeus; Pettibone, Kristianna; Drew, ChristinaNational Institute of Environmental Health Sciences

NIEHS is increasingly funding research using a consortium or team science approach. This type of method maximizes the use of resources, delivers data which helps policy makers make informed decisions, and facilitates transdisciplinary, collaborative research that increases our knowledge of the science area. In order to understand the impact of these consortiums on the science, we use social network analyses to assess a group of researchers studying the health effects of Bisphenol-A (BPA) exposures. In 2009, NIEHS funded additional grants in the BPA portfolio through the American Recovery and Reinvestment Act (ARRA) specifically to participate in a large consortium. This poster explores the development of the publication network from 1990-2013 and examines collaborations among authors before and after the consortium funding to understand how collaboration changed over time. Using Social Network Analysis, we can see who collaborates with whom and how the field is changing in terms of both scientist networks and scientific areas.


POSTER ABSTRACTS


Program Project Grant Mechanisms (P01 and P50/SPOREs) Outperform R01 Research Project Grants in Citation Productivity.

Kuzmin, Igor; Hubbard, Leah; Arnold, Julia; Hruszkewycz, Andrew; Nothwehr, Steven; Antman, Melissa; Ujhazy, PeterNational Cancer Institute

Stagnation of biomedical research budgets accentuated the need to support the most productive scientific areas through cost effective funding mechanisms. Bibliometrics has become an important objective component to assess scientific policies and programs. Here, we report a comparative bibliometric analysis of translational cancer research projects supported through R01, P01, or P50/SPORE (Specialized Programs of Research Excellence) grant activities by the NCI in FY09, 10 and 11. The initial dataset included the total cost incurred for each grant activity in FY09, 10, or 11, and the cumulative number of citations attributed to publications supported by a specific activity in a given grant year. For each of the three years, we then calculated the activity-specific citation cost index (CCI) representing the number of citations produced per $100K total cost invested in the targeted group of awards. Among translational award types, the P50/SPOREs featured the highest CCI (mean 295.6, SD=43.0) followed by the P01 program project grants that included at least one translational project (mean 159.6, SD=10.5). Translational R01 grants were the least productive with the mean CCI of 84 (SD=12.6). The CCIs in translational subsets of R01 and P01 activities were lower as compared to those for “all” R01s (mean 113.1, SD=15.9) and “all” P01s (mean 185.5, SD=38.8). The data point to superior cost effectiveness of the SPOREs as determined by the CCI in relation to other translational grant.


POSTER ABSTRACTS

Poster 29 ROOM G1/G2

Mapping the Alignment of NIH’s Portfolio to Burden of Disease

Bochner, David; Dodson, Sara; Haugen, Brian; Lobo, Jessica; Cameron, Laura; Riggie, Judy; Ikeda, RIchard; Volkov, MarinaOffice of Science Policy, Office of the Director

NIH supports research on a wide variety of diseases and conditions, as well as a substantial portfolio of basic research. Many NIH stakeholders, both academic and public, have expressed an interest in understanding how NIH’s portfolio aligns with measurements of the burden of various diseases and conditions, to better understand NIH priorities in the content of public health needs. In response to congressional inquiries, staff in the Office of Science Policy (OSP) in the Office of the NIH Director collaborated with colleagues at the Office of Extramural Research (OER) and the Centers for Disease Control and Prevention (CDC) to match multiple measures of disease burden to the categories reported in NIH’s Research, Condition, and Disease Categories (RCDC). Disease burden data were gathered from the World Health Organization/Gates Foundation/Univ. of Washington Global Burden of Disease dataset, as well as from CDC’s National Vital Statistics System and National Health Interview Survey. OER, OSP, and experts within the National Center for Health Statistics at CDC collaborated to match RCDC categories to existing disease burden data. While there are many caveats and complications involved with attempting to make such matches, both in terms of comparability of disease burden data and in terms of the limitations of classifying NIH-funded biomedical research, results showed that NIH funding positively correlates with several different measures of disease burden.


POSTER ABSTRACTS


Systematic analysis of research organisms employed by the biomedical community

Willis, Kristine; Burgoon, Dylan; Basson, Jake; Dunbar, Lisa; Gorospe, Rafael; Koduri, Sailaja; Sammak, Paul; Zuk, DoritNational Institute of General Medical Sciences

The mission of NIH is to increase knowledge that will ultimately lead to improvements in human health. Notably, the study of non-human species has been the source of several such advances. Examples include telomerase, which was first described in the ciliate Tetrahymena, and CDK2, the master regulator of cell cycle progression that was identified by studies using fission and budding yeasts. Other organisms may offer unique opportunities to elucidate as-yet unknown biological processes that are central to health and disease. We describe here a survey of the research organisms that are currently used by the biomedical community, defined by the PubMed-indexed literature as well as by the number of NIH applications and corresponding award rates. The data suggests that the use of research organisms by the scientific community adapts dynamically in response to theoretical and methodological advances.


POSTER ABSTRACTS


Identifying Prevention Research Using a Machine Learning Approach

Villani, Jennifer; Myles, Ranell; Vargas, Ashley; Lee, Jocelyn; Schully, Sheri; Meyer, Payam; Baker, Kirk; Fearon, Paula; Santangelo, George; Murray, DavidOffice of Disease Prevention, Office of the Director

Introduction: The NIH Office of Disease Prevention (ODP) collaborated with the NIH Office of Portfolio Analysis (OPA) to develop a machine learning framework that can accurately identify NIH-funded prevention research grants. This framework executes multiple algorithms (e.g., Linear and Neural Network classifiers) and combines them into an ensemble classifier that uses a voting system to predict grants as prevention positive or negative. This approach utilizes semi-supervised learning methods to analyze grants’ titles and abstracts, recognize linguistic patterns, and classify grants accordingly. Methods: The ODP coded a set of 1R01 prevention exemplars (positive and negative examples) from FY2010-15. The OPA used these grants to train the classifiers and then used the ensemble classifier to classify the test set of grants as prevention positive or negative. We tested the performance of the ensemble classifier trained using 1, 2, or 3 fiscal years of grant data to predict prevention grants for the following fiscal year. Results: There was no significant difference in the ensemble classifier’s performance using training data from 1 year prior (F1=0.828), 2 years prior (F1=0.837), or 3 years prior (F1=0.836). Conclusions: Adding grants from multiple years to the training set did not improve our ability to identify 1R01 prevention research grants. Next, we plan to use 1R01 data from one year prior to predict prevention grants among the 1R03, 1R21, and 2R01 NIH grant portfolios.


POSTER ABSTRACTS


Portfolio Analysis of NIH-funded Research on 1918 Influenza in the Context of DURC Policies

Finger, Matthew; Kuszak, Adam; Santana, Estevan; Bochner, David; Dodson, Sara; Volkov, MarinaOffice of Science Policy, Office of the Director

In 2005 T.M. Tumpey and colleagues published a groundbreaking paper reporting the complete reconstruction of the 1918 influenza virus (Science; 2005; 310(5745):77-80), a pathogen responsible for upwards of 50 million deaths in the early 20th Century. That same year the virus was added to the HHS list of Select Agents, and in 2012 and 2014 the U.S. Government issued policies for the oversight and management of Dual Use Research of Concern (DURC), which included use of the full 1918 virus, in an effort to mitigate the risk of research being used to harm public health and safety. In a preliminary effort to understand the effects of these policies on influenza research, staff in the Office of Science Policy in the NIH Office of the Director conducted a portfolio analysis and case study on research focused on the 1918 flu. Using OPA’s iTools, as well as data available via PubMed and Web of Science, an analysis of the grant and publication landscapes before and after the 2005 Tumpey publication and the DURC policies was performed. Trends in influenza research, funding, and publications were preliminarily identified, and results reveal a concentrated field of research responsive to world events such as the 2009 H1N1 swine flu outbreak. While results are not comprehensive, they provide a useful picture into a small subset of influenza research and clues as to factors that impact a field’s growth.


POSTER ABSTRACTS


A State of the Technology Assessment for Deep Brain Stimulation

Kuszak, Adam; Bochner, David; Dodson, Sara; Finger, Matthew; Ramos, Khara; White, Samantha; Adams, Amy; Rohrbaugh, Mark; Volkov, MarinaOffice of Science Policy, Office of the Director

As the Brain Research through Advancing Innovative Neurotechnologies® (BRAIN) Initiative supports trans-NIH research aimed at transformative improvements in technology, it is useful to assess the pre-initiative state of a particular neurotechnology as a baseline for assessing future advancements. To this end, staff in the Office of Science Policy (OSP) in the Office of the NIH Director, in collaboration with colleagues at the National Institute of Neurological Diseases and Stroke, performed a pilot analysis focusing on a particular technology: Deep Brain Stimulation, or DBS. The study uses a variety of tools and data sources to characterize the state of DBS technology in 2014, as the BRAIN initiative was just beginning, including NIH grants, other federal funding, publications (both NIH-funded and non NIH-funded), clinical trials, FDA approvals, and commercialization activities (both SBIR/STTR and otherwise). The results suggest several potential areas where data may be available to track future progress, and also identify several promising avenues for deeper investigation. This exploratory analysis can act as a potential template for future assessments of technology (both within the BRAIN Initiative and in other fields), and lessons learned about the characteristics of various data sources can be used to inform future attempts.


POSTER ABSTRACTS


Fully Characterizing the NIH Prevention Research Portfolio

Myles, Ranell L.; Murray, David M.; Villani, Jennifer; Lee, Jocelyn A.; Vargas, Ashley J.; Wu, Jessica Y.; Mabry, Patricia L.; Schully, Sheri D. Office of Disease Prevention, Office of the Director

Introduction: The Research, Condition, and Disease Categorization (RCDC) system has been used to classify NIH grants that are prevention research, but the sensitivity and specificity of RCDC’s ability to accurately classify grants is unknown. Additionally, RCDC does not provide detailed characteristics of prevention research (i.e. research method, population, etc.). The NIH Office of Disease Prevention (ODP) developed a more comprehensive method to fully characterize the prevention research portfolio while also assessing the accuracy of RCDC using a newly developed machine learning algorithm. Methods: All NIH type 1 R01 grant abstracts awarded during FY12-15 that were classified by RCDC as prevention, a 5% random sample of the RCDC non-prevention, and an additional set of abstracts identified through machine learning (N=5,467) were coded by ODP’s contract staff using a team-based consensus approach. The coding identified the study focus, population focus, study design, and type of prevention research. Results: The sensitivity and specificity of the RCDC system for identifying prevention grants is estimated to be 57.3% and 86.7% respectively. For FY12-15, we estimate that 16.9% of awards and 21.2% of dollars invested in type 1 R01s qualified as prevention research. We describe the trends and characteristics of prevention research funded by NIH. Conclusions: ODP can now apply this classification method to other grant mechanisms to reflect all prevention research funded by NIH.


POSTER ABSTRACTS


Vulnerable Plaque Research Portfolio Analysis

Chen, Jue; Olive, Michelle; Rosenberg, Yves; Danthi, Narasimhan; Galis Zorina; Fleg, JeromeNational Heart, Lung, and Blood Institute

Considerable research efforts over the past 15 years have focused on identifying characteristics of high risk atherosclerotic plaques, typically defined as those likely to rupture, causing an acute coronary event or stroke. The current portfolio analysis seeks to identify research activity in this area from 2005-2015 in order to determine research gaps and opportunities. Vulnerable plaque (VP) grants were retrieved via text-string search in NIH RePORT for NIH grants, Federal RePORTER for grants funded by other US government agencies, and in UberResearch database for grants funded by foreign entities. All abstracts were reviewed and only the projects that were clearly focused on VP research were included in the analysis. Of the 180 VP grants identified, about 80% were funded by NIH, 17% by foreign countries, and 3% by other US government agencies. Investigation of trends in NIH-supported VP research using the iTrans program identified a progression from basic to more translational and clinical studies. Using the same text string to search for VP clinical studies registered in clinicaltrials.gov, we identified 154 clinical VP studies. More than half of these clinical studies were supported by foreign entities, and 8% of them were supported by NIH. In conclusion, NIH is the main sponsor of VP global research, while most of the VP-focused clinical trials are sponsored by foreign entities and industry.


POSTER ABSTRACTS


Angiogenesis Research: The Extramural Portfolio Supported by the National Heart, Lung, and Blood Institute, 2008-2015

Gao, Yunling; Stanley, Dennis V.; Robinson, Valerie C.; Reid, Diane M.; and Galis, Zorina S. National Heart, Lung, and Blood Institute

The NHLBI invests in angiogenesis-related extramural research to improve the basic understanding of normal and abnormal angiogenesis and to translate mechanistic findings into therapeutic interventions. This extramural portfolio analysis from fiscal years 2008 through 2015 reports that NHLBI has funded an average of 229 angiogenesis grants investing in average of $116 million per fiscal year. One measure of public impact of each awarded grant was an average of 3.3 publications per year. The angiogenesis research was funded through four major funding mechanisms, mainly through the Research Project Grants, or R01s (64% of the total number of awards). An average of 19% of total awards were issued to new research (Type 1 applications) each year, which may represent pursuit of new ideas or new directions in angiogenesis research. Among the funded portfolio, 74.1% of awards (65% of total dollars) utilized in vivo experimental animal models, 9.7% of awards (5% of total dollars) utilized in vitro basic research, 13.7% of awards (27% of total dollars) involved human subject studies (not including clinical trials), and 2.5% of awards (3% of total dollars) were used for clinical trials. While the NHLBI angiogenesis portfolio supports several mechanistic types of angiogenesis studies, intussusceptive angiogenesis may represent an under-studied area and an opportunity for further understanding its mechanism, regulation, and potential for therapeutic intervention.


POSTER ABSTRACTS


Lymphatics research supported by NHLBI and its impact

Khan, Faiyaz; Stanley, Dennis; Tolunay, EserNational Heart, Lung, and Blood Institute

The lymphatic system serves an essential role in maintenance of fluid homeostasis by returning fluid from extracellular space back to blood circulation, enables the absorption and transport of lipids from the gastrointestinal tract, and participates in immune functions by trafficking of immune cells and antigens to and from lymph nodes. The study of lymphatic drainage of various organs is important in the diagnosis, prognosis and treatment of many diseases. The most recognized disease state of the lymphatic system is lymphedema. Lymphangiogenesis is associated with inflammation in many disease states and plays an important role in both the protective immune response and in pathology of certain immune-mediated diseases. New roles for the lymphatic system in common diseases and conditions such as hypertension, coronary artery disease, inflammatory bowel disease, obesity, metabolic syndrome and diabetes have started to emerge. Research on the lymphatic system is supported by several institutes and centers at NIH. It is a nascent research area that requires regular facilitation by NIH by funding opportunity announcements, workshops and symposia. NHLBI funded 73 competing awards totaling $28 million during FY2007-2016 to understand the interaction and contribution of the lymphatic system to heart, lung and blood health and resilience. We also examined the translational potential and impact of publications from these awards.


POSTER ABSTRACTS


Portfolio Analysis of the Current NIH Investment in Single Cell Analysis Research in Human Cells and Tissues

Smith, Jessica M; Srinivas, Pothur; Galis, ZorinaOffice of Strategic Coordination, Office of the Director

The NIH “Single Cell Analysis in Human Cells or Tissues” portfolio was evaluated by a trans-NIH working group to determine the active NIH investment in this area of research, as well as to identify opportunities and limitations. This portfolio analysis was conducted as part of strategic planning to determine whether the NIH Common Fund should support a program in this area. The term “single cell” was used to identify active projects supported at NIH between 2015-2016 using IMPACII, with further filtering to isolate projects using human cells and tissue. Projects were then manually assessed whether awards matched requirements. This resulted in a validated list of 169 projects totaling $97M in single cell analysis research in human cells and tissues, across 17 ICs. The main human cell and tissue types used are cancer, stem, immune, and neuronal/brain, with many parts of the adult human body not investigated. Primary methods for analyzing these cell and tissue types include: transcriptomic, genomic, and imaging approaches. Most of these projects involve technology or method development and data generation. Based on this analysis and feedback from the community, the relative investment in this area is relatively modest, most importantly, it is fragmented. A Common Fund program built based on common goals and shared scientific principles would have a transformative impact on ability to integrate our current piece-meal understanding of the human body.


POSTER ABSTRACTS


Analysis of NIAID’s Type 1 Diabetes (T1D) Portfolio

Zane, Ariel; Bourcier, Kasia; Ghosh, DolanNational Institute of Allergy and Infectious Diseases

We analyzed T1D NIH grants’ portfolio, as well as a collection of manually curated Web of Science (WoS) publications, to reveal major trends and scientific topic development over time from 2000 to 2016 (2,021 NIH grants; 23,426 WoS publications). We identified top authors in the T1D field, as well as top funding agencies, and applied social network analysis to assess collaborative relationships between co-authors/funding agencies overall, and over time. Text mining methods were applied to Web of Science ISI-keywords and PubMed MeSH terms, to analyze representation of scientific topics, and compare topics present in NIAID’s portfolio with the rest of the T1D field. This process was repeated for NIH-awarded grant abstracts (from NIAID and NIDDK), to compare and contrast topics by IC. Preliminary trends elucidated with these methods suggest that NIAID’s centrality in the T1D field has grown over the time period evaluated. All analyses and visualizations were completed using R (R Foundation for Statistical Computing, Vienna, Austria). Author and funding agency name disambiguation was performed with in-house homebuilt software.


POSTER ABSTRACTS


Natural Products Research Applications and Awards at NIH: A Five Year Analysis

Berzhanskaya, Julia1; Hopp, Craig2

1National Institute on Drug Abuse2National Center for Complementary and Integrative Health

Due to variety of reasons government funding is the only significant source of investment in Natural Products Research at this time. To better understand interest in natural products research at NIH an analysis was undertaken to quantify the number of applications submitted to and awards made by NIH over the last five years (2011–2015). This analysis revealed over 5,000 applications were submitted in this timeframe covering all aspects of natural products research. Over 60% of these applications were submitted to three components of NIH: specifically the National Cancer Institute (NCI), the National Center for Complementary and Integrative Health (NCCIH), and the National Institute of General Medical Sciences (NIGMS). The data suggest natural products related applications are consistently funded at a lower rate than the overall rate for each of the three major natural product funders. There is a significant topic overlap between funded and not funded applications.


POSTER ABSTRACTS


Data Towards Finding Research Gaps and Opportunities in the Health of Women

Coles, EricNational Heart, Lung, and Blood Institute

There are many ways to categorize a ‘portfolio’ for analysis at NIH. One method that is applicable to every IC is by demographic population, such as sex, gender, race, or ethnicity. At the National Heart, Lung, and Blood Institute (NHLBI), an internal working group performed an analysis on our research portfolio in the health of women. Although the synthesis is actively ongoing and still being formulated, we would like to present our data analysis and summarization that were developed to facilitate creating a scientific agenda focused on heart, lung, blood, and sleep (HLBS) research in women’s health. The data includes outputs from a literature and portfolio analysis, as well as health statistics. The literature analysis had two parts. First, it included key reports that were manually coded according to a set categorization schema. Second, there was an automated analysis of the literature landscape around major HLBS conditions that are especially important to women’s health to glean more about the areas. The portfolio analysis included awarded projects that were part of the Institute’s regular public report on research in women (Moyer Report), as well as a smaller manually-coded set of projects that ask specific sex or gender-related inquiries. Similar coding categorizations were used across analyses to allow for cross-sectional comparisons. Finally, health statistics, such as mortality rates and disease prevalence, were included for framing purposes.


POSTER ABSTRACTS

Poster 42 ROOM C1/C2

NCI-funded Research in India

Prakash, LauraNational Cancer Institute

The National Cancer Institute’s (NCI) Center for Global Health (CGH) has partnered with research institutions in India to support their cancer research and cancer control activities since before the Center’s inception in 2011. CGH prepares annual portfolio analyses of NCI’s global activities to understand the landscape of NCI-funded research. This portfolio analysis reflects the CGH and NCI activities in India from 2010-2016. CGH-internal, regional strategic planning documents, were used to identify significant CGH/NCI partnership activities with India from FY12-16. Data were collected from QVR to reflect all NCI-funded extramural grants awarded in FY10-16, with performance or FACTS sites in India. NCI-funded intramural projects with Indian collaborations were obtained through annual CGH data collection from the NCI intramural divisions and centers (DOCs) for FY14-16. The Indian collaborator sites were standardized, and CSO and SITE codes were used to analyze research types and cancer sites of emphasis. These portfolio analyses are practical reports that allow NIH and non-NIH personnel to examine where NCI cancer research activities exist in the global space. They show users where cancer research activities could be enhanced and help them identify gaps where new opportunities might exist. These reports help build partnerships and foster knowledge exchange between investigators, country leadership, and partner organizations dedicated to advancing cancer research and control.


POSTER ABSTRACTS


Portfolio analysis with limited data: A case study on training program effectiveness at a smaller IC

Zarcone, Troy; Powell, Elizabeth; Dunty, Bill; Bechtholt, AnitaNational Institute on Alcohol Abuse and Alcoholism

Individual ICs are confronting difficult funding questions as flattened budgets reduce the number of funded projects. ICs face questions about how best to support researchers, while maintaining a pipeline of investigators at all levels: early, mid and late career. Smaller ICs have limited data (small sample size) from which to make these decisions (i.e., no or low statistical power). This poster describes methods for doing small data portfolio analyses of funded training mechanisms (F, T, K awards) for NIAAA. We compared NIAAA to ICs with similar scientific missions to see if our IC allocated training resources differently. We started with a simple outcome (R01 award) to familiarize ourselves with the data set (IMPAC II) and to begin the process of asking more informed questions. We used off-the-shelf applications (QVR and Excel pivot tables) to describe previous funding activity and address questions about specific outcome measures. We chose three related ICs (NIMH, NIDA, and NIDDK) based on complementary scientific missions and outcomes. While the overall allocations differed for these ICs, proportional estimates of training funding (T, K, F mechanisms) showed all four ICs had slightly decreased training funding (-0.3%; NIAAA, NIDA, NIDDK and -0.7%; NIMH) since 2003. Additional example analyses of R01 outcomes are described. Small data portfolio analyses, designed to answer specific questions, can provide useful information for guiding funding decisions.


POSTER ABSTRACTS


NCI-Funded Research in China: A Case-Study for Analyzing NCI International Research

Abudu, RachelNational Cancer Institute

NCI and cancer research institutions in China are established partners in cancer research, and the NCI Center for Global Health (CGH) supports research and cancer control activities in China. To better understand the landscape of NCI-funded research in China, CGH prepared a portfolio analysis of CGH and NCI activities in China during 2010-2016. All NCI-funded extramural grants awarded in FY10-16, with performance or FACTS sites in China (excluding Hong Kong and Taiwan), and NCI intramural projects from FY14-15 with Chinese collaborations were included in the portfolio. CGH strategic planning documents were used to identify CGH/NCI partnership activities with China from FY12-16. Chinese collaborator sites were standardized, and CSO and SITE codes were used to analyze research types and cancer sites of emphasis. There were 114 type 1, 2, and 3 NCI-funded extramural grants awarded in FY10-16 and 58 NCI intramural projects in FY14-15 with a Chinese collaborator. Extramural research in China spanned 19 activity types and over 20 different cancer sites. Extramural research in China was conducted by 58 US institutions, and partnered with 45 Chinese institutions across 19 cities. Intramural projects focused on 26 cancer sites, and partnered with 39 Chinese institutions across 15 cities. NCI/CGH has participated in 18 partnership activities with China since FY12. This analysis will be used in NCI/CGH strategic planning sessions, and is available for NCI/HHS and external partners.


POSTER ABSTRACTS


Portfolio Analysis of the Clinical and Translational Science Award (CTSA) Program’s Institutional Mentored Career Development Program (2006-2014)

Baker, Heather L; Rosemond, Erica; Purucker, MaryNational Center for Advancing Translational Sciences

The Clinical and Translational Science Award (CTSA) Program supports high quality translational and clinical research locally, regionally, and nationally, and fosters innovation in research methods, training, and career development. As part of the programs’ commitment to developing the translational science workforce, the CTSA Program supports Institutional Mentored Career Development Awards to provide training and education to translational researchers. The goal of this analysis was to assess this program’s outcomes to date, including measures of program completion and the scholar’s receipt of subsequent funding.


POSTER ABSTRACTS


A framework for creating regional analyses of NIH-funded international work

Haney, Karen; Ndumele, AmaraNational Cancer Institute

Background: NCI’s Center for Global Health (CGH) has a special mission to understand the global cancer funding landscape and inform the conversation around international NCI-funded work and activities. CGH uses regional portfolio analyses as resources to promote multi-sectoral cancer research partnerships and build the global cancer research community. Aims: CGH regional analyses provide a comprehensive report capturing the NIH-funded portfolio across cancer sites, activity types, and international research institutions engaged in each region. CGH developed a framework to generate cancer-related regional reports for sub-Saharan Africa, Southeast Asia, and Latin America. Methods: NCI-funded extramural data was obtained from QVR, FY14-16, with performance sites or FACTS sites in each region. NCI-funded intramural data was obtained through annual CGH data collection, FY14-15, to the NCI intramural divisions, offices, and centers. CGH internal reports were used to describe CGH region-wide and country-specific projects during FY14-16. Key regional population and health statistics were drawn from Globocan 2012, The Cancer Atlas, and the World Bank. Conclusion: These user-friendly reports allow NIH and non-NIH personnel to efficiently examine where research activities and partnerships exist or could be enhanced or created, fostering knowledge exchange between investigators, national leadership, and civil societies dedicated to the advancement of cancer research and control.


POSTER ABSTRACTS


R01 and RPG Award Activity among K-award Recipients: A Cohort Analysis of Assistant Professors at US Medical Schools

Doyle, JamieStatistical Analysis & Reporting Branch, Office of the Director

Since 1957, mentored career development awards have provided support to new investigators in fostering their transition to research independence. Existing studies have examined K-awardees as cohorts beginning at the time of award. However, due to differences in the length of postdoctoral and medical fellowship training, there can be heterogeneity in when K-awardees begin their careers in academic medicine. The purpose of this study is to examine subsequent R01 and RPG activity of K-awardees as a cohort when they begin their positions as full-time assistant professors. Data for this analysis comes from the Association of American Medical Colleges (AAMC) Faculty Roster File linked by principal investigator to NIH grant data from IMPAC II. Full-time assistant professors whose faculty appointments started between 2007 and 2008, who have not previously received RPG funding, and who have received a K-award are considered for this analysis. Results show that most this cohort has applied for RPG funding, but not as high of a percentage ever receive funding. Predictive models will also be constructed to examine leading predictors of RPG and R01 funding.


POSTER ABSTRACTS


New Methods and Tools for Visualizing and Analyzing Scientific and Budget Data

Botero, Sebastian; Lee, Linda; Ghosh-das, Dolan; Pool, Virginia; Dogiparthi, Venkata; Footer, Kevin; Smith, Brock; Xu, Shengpin.National Institute of Allergy and Infectious Diseases

A primary function of the Referral and Program Analysis Branch (RPAB), part of the Office of Strategic Planning, Initiative Development and Analysis (OSPIDA), is to assign scientific codes to grants, contracts and intramural projects allowing NIAID categorization and reporting of NIAID’s research activities. These reports are often generated in consultation with NIAID’s Budget Office and used by multiple stakeholders for various analytic activities. Currently, reports are exported as spreadsheets or pdf files, and most data analyses are carried out in Excel. This process can be cumbersome and multiple reports need to be generated to analyze multiple scientific codes at the same time. In addition, when conducting a trend analysis the data must be exported one year at a time, an extremely inefficient process. RPAB is exploring the use of Tableau as a new visualization and analysis tool to expedite portfolio analyses, improve data quality control and reproducibility and provide instant access to all parties involved. We present two dashboards: 1. A dashboard that allows RPAB staff to easily and clearly identify relationships between Office of AIDS Research (OAR) categories across multiple years. 2. A dashboard to be used by RPAB and the Budget Office to quickly perform trend analysis on high priority reports and share comments in real time. This is expected to increase the efficiency of reports generation and allow users to identify significant funding changes from past years.


POSTER ABSTRACTS


Portfolio Analysis in the Center for Research Strategy, National Cancer Institute

Antman, Melissa; Eddie N. Billingslea; James G. Corrigan; Roman Gorelik; Amy E. Kennedy; Grace F. Liou; L. Michelle BennettNational Cancer Institute

The Center for Research Strategy (CRS) was established in December 2015, in the Office of the Director (OD) of the National Cancer Institute (NCI). CRS works collaboratively to develop recommendations for addressing scientific opportunities, to monitor the direction and application of the NCI’s scientific knowledge and resources, and to identify research funding gaps. Portfolio analysis in CRS includes many activities, such as: 1) Analysis conducted at the request of the Institute leadership to inform policy, decision making, and program management; 2) Portfolio analysis support for strategic planning and implementing cross-cutting institute priorities, such as cancer research health disparities, the Beau Biden Cancer MoonshotSM, and the Recalcitrant Cancer Act; 3) Portfolio analysis support for reporting and response to inquiries that are managed by CRS and other NCI OD offices; and 4) Portfolio analysis and evaluation consultation for NCI staff. Additionally, CRS manages the NIH participation in the International Cancer Research Partnership (https://www.icrpartnership.org/), a consortium of cancer funders that produces a searchable database of global cancer funding.


POSTER ABSTRACTS


An analysis of NIH-funded research and publications in Africa

Harriman, Robert; Perkins, Matthew; Santangelo, GeorgeOffice of Portfolio Analysis, Office of the Director

This analysis investigated the changes in NIH grant funding and publication outputs for African institutions from 1980-2016, with a focus on Uganda. Specifically, the analysis examined trends in grants directly received by organizations in Africa as well as grants with at least one performance site at an African organization. Projects were attributed to African countries using available address information associated with each grant. Organization address information collected in the NIH IMPACII database evolved over time, which presented a challenge in accounting for projects by country. For example, incomplete addresses were not uncommon for early foreign grants, and a country name may have been written out, abbreviated, or some combination of both. We developed a strategy for this analysis that allowed accurate accounting of projects by country as data collection standards changed over time. The strategy used a map to ensure proper counting of the variations in country names and abbreviations that appear in the IMPACII database. Finally, we computed the annual NIH-funded proportion of publications authored by Ugandan organizations. Over time, NIH-funded papers represented an increasing proportion of the output from Ugandan organizations.


POSTER ABSTRACTS


The Picture Could be Rosier – An in-depth look at NIH RePORTER success rates for NCI training grants

Torres, Sasha; Jakowlew, Sonia; McGuirl, Michele; Radaev, Sergei; Schmidt, Michael; Wiest, Jonathan; Lei, MingNational Cancer Institute

According to the NIH, success rates are defined as the percentage of reviewed grant applications that receive funding. Success rates are determined by dividing the number of competing applications funded by the sum of the total number of competing applications reviewed and the number of funded carryovers. Applications having one or more submissions for the same project in the same fiscal year are only counted once.1 NIH RePORTER publishes the National Cancer Institute’s (NCI) Cancer Training Branch (CTB)success rates, however, they can differ from what is actually committed. We recently analyzed the NCI-CTB) F32 portfolio and noted that each year about 20% of NCI F32 applicants declined the award in lieu of non-NIH postdoctoral fellowships or because they decided to pursue other career opportunities. Next, we investigated the potential impact on success rates. F32 and K99 postdoc applicants have numerous non-NIH funding options available and usually go with the best financial offer. Awards offered but declined are not adequately captured by the NIH RePORTER, which calculate success rates in the same way for all grant mechanisms. A rosier picture of the training grant portfolio emerges for postdoctoral training when training-specific variations like declined offers are included.


POSTER ABSTRACTS


NIH Internal Research Portfolio Online Reporting Tools (iRePORT)

Danielson, Cindy; Haugen, BrianOffice of Data Analysis Tools & Systems, Office of the Director

NIH’s Internal Research Portfolio Online Reporting Tools (iRePORT) is a portfolio analysis and workflow dashboard for all NIH extramural staff, allowing you to monitor all aspects of multiple research portfolios. iRePORT is aimed at integrating information from our other tools (QVR, SPIRES, RePORTER) into an intuitive dashboard system, focused on ease of use for new users by pushing relevant data without requiring repeated searches. Since its ‘open beta’ release in 2015, a number of new features have been added to make iRePORT more useful to different types of users and portfolios. iRePORT now provides an extensive library of ‘widgets’, which are small charts or tables focused on presenting particular aspects of your portfolio. Users are provided a default portfolio based on IC and application or panel assignments, but new portfolios can be created on the fly or loaded from external sources such as QVR. To provide a one-stop home page, iRePORT widgets provide easy access to QVR saved searches and carts. Other widgets summarize, for each saved portfolio, recent publications, citation patterns, news releases, registered clinical trials, and related clinical guidelines. To facilitate day-to-day portfolio management, the tool provides summaries of unfunded competing applications, projects with progress reports due, and upcoming peer review meetings related to the portfolio. iRePORT is a flexible tool for monitoring the short and long term needs and outputs of NIH portfolios.


POSTER ABSTRACTS


Using Propensity Score Methods to Evaluate the Early Stage Investigator Policy

Walsh, Rachael; Moore, RobertOffice of Extramural Programs, Office of the Director

To assist new scientists in the transition to independent research careers, the National Institutes of Health (NIH) implemented an Early Stage Investigator (ESI) policy in 2009. The ESI designation segregates applications submitted by investigators within 10 years of completing their terminal degree or medical residency from applications submitted by more experienced investigators. The goal of this policy is to give special consideration in the review process to investigators not yet established in their scientific careers to increase their probability of receiving research support. Changing political environments can affect the availability of NIH funding, thus attributing changes in the probability of funding to one policy is challenging. This research used propensity score matching to generate comparable groups, pre- and post-policy implementation, comparing the probability of funding for ESI flagged applications from 2011 to 2015 to applicants with similar demographic and professional characteristics from 2004 to 2008. This paper addresses the statistical necessity for the application of an advanced methodology to evaluate this policy, including model validation, and preliminary results.


POSTER ABSTRACTS


Outcomes Clustering Analysis Tool (OCAT)

Zoha, Steven; Zane, Ariel; Ghosh, DolanNational Institute of Allergy and Infectious Diseases

In conjunction with Altum, Inc., a new application (OCAT V1) was developed to improve efficiency and visualization of portfolio analysis and outcome evaluation. OCAT V1 employs unique data mining, natural language processing, and machine learning algorithms to effectively help researchers gain insight in tracking projects and assessing associated public health impacts. Specifically, it uses sophisticated, streamlined visualization tools with: 1) Citation analysis to explore linkages and highly cited articles; 2) Nearest neighbor analysis to search for publications similar to an exemplar publication; 3) Tagging/classification of articles according to IC support or user defined parameters; 4) Scientific clustering of search results. In its current configuration, it can simultaneously search and connect data from multiple sources (PubMed, USPTO, Clinicaltrials.gov, PARDI (NIH/OER), RCR (NIH/OPA)) through many generations across time to: 1) Connect known public health impacts to foundational research and high profile discoveries; 2) Provide additional objectivity and identify unanticipated collateral benefits for outcomes analysis; 3) Suggest prospective alerts for high profile discoveries. Refinements are being explored for OCAT V2.


POSTER ABSTRACTS


Using the RCDC Portfolio Visualization (PVIZ) Module to Understand Trends in NIH Data

Lobo, Jessica; Scholl, Jacob; Riggie, Judy; Ikeda, RickDivision of Information Services, Office of the Director

As more project-level information is being stored over time across NIH, data discovery tools are becoming increasingly useful to derive meaningful trends. In this demonstration of the Portfolio Visualization (PVIZ) tool, housed within the RCDC system, we will be showing NIH staff how they can use this module to define and investigate project portfolios. This tool provides the user with multiple means of defining a portfolio of interest, including using existing RCDC category filters, project filters (e.g. PCC, RFA numbers, CANs, etc) and QVR carts. Once a portfolio is defined, the user can examine the data using three different interactive visualizations via clustering methodology. Projects in the same cluster are more similar to each other than to projects in other clusters and these cluster names are derived from project indices (RCDC concepts) or extracted project text. The visualizations are fluid, allowing users to further narrow their parameters to export results or to save visualizations for use in presentations. As the system expands, and more functionality is incorporated, users are finding new and unique ways to analyze different types of data. These include analyzing RCDC categories across multiple fiscal years, visualizing programmatic areas, council meetings, etc., finding new or emerging science, and determining what topic areas are absent from a portfolio. RCDC staff continue to work with IC stakeholders to improve and expand functionality of the system.


Authors by Name


AUTHORS BY NAME

AUTHORS (LAST NAME, FIRST NAME), POSTER(S)

Abudu, Rachel; 44

Adams, Amy; 33

Alexander, Kristine; 10; 26

Antman, Melissa; 28; 49

Arnold, Julia; 28

Baden, Elizabeth; 10

Baker, Heather L.; 45

Baker, Kirk; 20; 25; 31

Balshaw, David; 21

Basson, Jake; 30

Baumgart, Joel; 10; 26

Bechtholt, Anita; 43

Belter, Chris; 26

Bennett, L. Michelle; 49

Berzhanskaya, Julia; 40

Billingslea, Eddie N.; 49

Bochner, David; 10; 26; 29; 32; 33

Botero, Sebastian; 48

Bourcier, Kasia; 39

Burgoon, Dylan; 30

Cameron, Laura; 29

Charette, Marc; 11; 13

Chen, Jue; 35

Choi, Kristina; 18

Coles, Eric; 41

Corrigan, James G.; 49

Danielson, Cindy; 52

Danthi, Narasimhan; 35

Dathe, Marina; 16

Dilworth, Caroline; 21

Dodson, Sara; 10; 26; 29; 32; 33

Dogiparthi, Venkata; 48

Doyle, Jamie; 47

Drew, Christie; 21, 27

Dunbar, Lisa; 30

Dunty, Bill; 43

Eblen, Mathew; 11

Fearon, Paula; 1; 3; 31

Ferland-Beckham, Chantelle; 6

Finger, Matthew; 32; 33

Fleg, Jerome; 35

Fletcher, Colin; 4

Footer, Kevin; 48

Forcinito, Patricia; 1; 3

Galis, Zorina; 11; 13; 15; 35; 36; 38

Gao, Yunling; 36

Ghosh-das, Dolan; 5; 9; 39; 48; 54

Gorelik, Roman; 49

Gorospe, Rafael; 30

Haney, Karen; 46

Harriman, Robert; 3; 50

Haugen, Brian; 29; 52

Heacock, Michelle; 21

Heindel, Jerold; 27

Helzlsouer, Kathy; 16

Hopp, Craig; 40


AUTHORS BY NAME

Hoppe, Travis; 23; 25

Horsford, Jonathan; 6

Horska, Alena; 12

Hruszkewycz, Andrew; 28

Hubbard, Leah; 28

Hutchins, B. Ian; 1; 25

Ikeda, Richard; 20; 29; 55

Jakowlew, Sonia; 51

Jose, Sheethal; 4

Kalantari, Roya; 12

Kennedy, AmyE.; 49

Khan, Faiyaz; 37

Klosek, Malgorzata; 12

Koduri, Sailaja; 30

Kosub, David; 10

Koustova, Elena; 7; 17; 18

Kuszak, Adam; 32; 33

Kuzmin, Igor; 28

Latoni, Alfonso; 21

Lee, Jocelyn; 31; 34

Lee, Linda; 48

Lei, Ming; 51

Lenzi, Rebecca; 4

Levy, Abraham; 12

Lin, Dawei; 2; 14; 19

Liou, Grace F.; 49

Litovitz, Aviva; 8

Lobo, Jessica; 29; 55

Lu, Ya-Ling; 4

Mabry, Patricia L.; 34

Mao, Fenglou; 2; 19

Maric-Bilkan, Christine; 11; 13

McAllister, Kimberly; 21

McGuirl, Michele; 51

Meseroll, Rebecca; 3

Meyer, Payam; 31

Mirochnitchenko, Oleg; 4

Moore, Mark; 4

Moore, Robert; 53

Murray, David; 31; 34

Myles, Ranell; 31; 34

Ndumele, Amara; 46

Nothwehr, Steven; 28

O’Fallon, Liam; 21

Oh, Young; 11; 15

Olive, Michelle; 35

Palmer, Jason; 3

Parker, Marie; 5; 9

Parker, Mark; 24

Pearson, Katrina; 11; 13

Perkins, Matthew; 1; 50

Pettibone, Kristi; 21; 27

Pogorzala, Leah; 22

Pool, Virginia; 48

Porter, Linda; 22

Powell, Elizabeth; 43

Prakash, Laura; 42

Prikhodko, Victor; 7; 17; 18

Purucker, Mary; 45

Radaev, Sergei; 51


AUTHORS BY NAME

Ramos, Khara; 33

Reczek, Peter; 10

Reid, Britt; 16

Reid, Diane M.; 36

Riggie, Judy; 20; 24; 29; 55

Robinson, Valerie C.; 36

Rogers, Scott; 16

Rohrbaugh, Mark; 33

Rosemond, Erica; 45

Rosenberg, Yves; 35

Rosenfeld, Gabriel; 14

Ruben, Elizabeth; 27

Sammak, Paul; 30

Sankar, Cheryse; 22

Santana, Estevan; 32

Santangelo, George; 1; 3; 8; 20; 23; 25; 31; 50

Sazonova, Irina; 7; 17; 18

Schmidt, Michael; 51

Scholl, Jacob; 24; 55

Schully, Sheri; 31; 34

Scott, Lindsey; 11; 13

Shugg, Thaddeus; 27

Singh, Bishen; 4

Slipchenko, Tamara; 7; 17; 18

Smith, Brock; 48

Smith, Jessica M; 38

Sorlie, Paul; 26

Srinivas, Pothur; 38

Stanley, Dennis; 36; 37

Thompson, Claudia; 21

Tiano, Joseph; 6

Tolunay, Eser; 11; 13; 37

Torres, Sasha; 51

Ujhazy, Peter; 28

Vargas, Ashley; 31; 34

Villani, Jennifer; 31; 34

Volkov, Marina; 10; 26; 29; 32; 33

Walsh, Rachael; 53

White, Samantha; 33

Wiest, Jonathan; 51

Willis, Kristine; 30

Wright, Demia; 21

Wu, Charles; 11; 13

Wu, Jessica Y.; 34

Xu, Shengpin; 48

Zane, Ariel; 39; 54

Zarcone, Troy; 43

Zoha, Steven; 54

Zuk, Dorit; 30


Notes

Of˜ce of Portfolio Analysis Symposium - DPCPSI OPA... · 2017-06-16 · ii 2017 Office of...

Documents

Transcript of Of˜ce of Portfolio Analysis Symposium - DPCPSI OPA... · 2017-06-16 · ii 2017 Office of...