NIH LRP Evaluation - National Institutes of Health
Transcript of NIH LRP Evaluation - National Institutes of Health
NIH Extramural Loan Repayment Program Evaluation i
National Institutes of Health Loan Repayment Program
Evaluation Working Group
Milton Hernandez, Chair
National Institute of Allergy and
Infectious Disease
Steve Boehlert
Division of Loan Repayment
Frank Calvo
National Institute of Diabetes and
Digestive and Kidney Diseases
James Corrigan
National Cancer Institute
Paul Johnson
National Institute of Child Health and
Human Development
Steve Korn
National Institute of Neurological
Disorders and Stroke
Matthew Lockhart
Division of Loan Repayment
Sherry Mills
Office of Extramural Research
Richard Phillips
Division of Loan Repayment
Peggy Reed
Division of Loan Repayment
Carl Roth
National Heart, Lung, and Blood
Institute
Mercedes Rubio
National Institute of Mental Health
Jennifer Sutton
Office of Extramural Research
Study Director
Laurel Haak
Discovery Logic
Analysts
Beth Masimore, Rajan Munshi, Matt Probus, Yvette Seger
Discovery Logic
NIH Extramural Loan Repayment Program Evaluation ii
TABLE OF CONTENTS
EXECUTIVE SUMMARY 1
Findings 1
Who is Applying? 1 Do the LRPs Retain Participants? 2 How do the LRPs Compare to Other NIH Mentored Career Awards Programs? 2
Conclusions and Recommendations 2
HISTORY OF THE NIH EXTRAMURAL LRP 3
EVALUATION OBJECTIVES 5
RESEARCH DESIGN 5
Scope 5
Data Sources 6
Statistics 6
Applicant Pool 6
Impact of Participation 6
Comparison with Other NIH Mentored Career Awards 6
QUESTION 1: WHO IS APPLYING? 7
1.1 Goals 7
1.2 Methods 7
1.3 New Applicants 7
1.3.1 New applications and awards, by program and IC 7
1.3.2 New applications and awards, by gender 9
1.3.3 New applications and awards, by race and ethnicity 10
1.3.3.1 Distribution of ethnicity 10
1.3.3.2 Distribution of race 11
1.3.3.3 Distribution of ethnicity / race and gender 13
1.3.3.4 Distribution of ethnicity / race by IC 14
1.3.4 New applications and awards, by degree type 15
1.3.4.1 Distribution of degree types 15
1.3.4.2 Distribution of degree types and gender 15
1.3.4.2.1 Degree by gender 16
1.3.4.2.2 Gender by degree 16
1.3.4.3 Distribution of degree types by IC 16
1.3.5 New applications and awards, by LRP 17
NIH Extramural Loan Repayment Program Evaluation iii
1.3.5.1 Distribution of IC and LRP 17
1.3.5.2 Distribution of gender and LRP 17
1.3.5.2.1 LRP by gender 18
1.3.5.2.2 Gender by program 18
1.3.5.3 Distribution of degree types and LRP 18
1.3.6 New applications and awards, by years since degree 19
1.3.6.1 Distribution of years since degree and degree type 19
1.3.6.2 Distribution of years since degree and gender 21
1.3.7 New applications and awards, by prior research support 22
1.3.8 New applications and awards, by educational debt level 23
1.3.8.1 Distribution of educational debt level 23
1.3.8.2 Distribution of educational debt level and degree type 24
Summary of Findings: FY03-FY07 Extramural New Applicant Pool 26
1.4 Renewal Applicants 27
1.4.1 Renewal applications and awards 27
1.4.2 Renewal applications and awards, by degree type 29
1.4.3 Renewal applications and awards, by educational debt level 29
Summary of Findings: FY03-FY07 Extramural Renewal Applicant Pool 30
QUESTION 2: DOES THE LRP RETAIN PARTICIPANTS IN THE NIH EXTRAMURAL WORKFORCE? 31
2.1 Goals 31
2.2 Methods 31
2.3 Analysis of the Study Cohort 32
2.3.1 Comparison of the study cohort and the total new applicant pool 32
2.3.2 Comparison of funded and not funded applicants in the study cohort 33
2.4 Subsequent Participation in NIH Grant Programs 33
2.4.1 Composite Analysis 33
2.4.2 Gender 35
2.4.3 Degree Type 35
2.4.4 Program 36
Summary of Findings: Retention in the NIH Workforce 38
2.5. Publications and Citations 39
Summary of Findings: Publications and Citations 40
2.6. Outcomes Analysis for Applicants without Subsequent NIH Records 41
2.6.1 Current Affiliation 41
2.6.2 Current Position 42
Summary of Findings: Applicants without Subsequent Records 42
NIH Extramural Loan Repayment Program Evaluation iv
QUESTION 3: HOW DOES THE LRP COMPARE WITH MENTORED K AWARDS? 43
3.1 Goals 43
3.2 Methods 43
3.3 Demographic Distribution 43
3.3.1 Gender 45
3.3.2 Degree type 45
3.3.3 Subsequent Participation in NIH Grant Programs 46
3.3.4 Composite Analysis 46
3.3.5 Gender 47
3.3.6 Degree Type 48
3.4 Publications and Citations 49
Summary of Findings: Comparison of LRP with Mentored K Awards—Top 10% 51
APPENDIX 1: DETAILS BY IC 52
A1.1. New and Renewal Applications and Awards, by IC 52
A1.2. New Applications and Awards, by IC and LRP 53
A1.3. New Awards, by IC and Degree Category 54
APPENDIX 2: DEGREE CLASSIFICATIONS 56
APPENDIX 3: DETAILED OUTCOMES METHODS 58
A3.1. Construction of the Funding Bubble 58
A3.2. Determination of Subsequent NIH Activity 60
A3.3. Matching Applicants to MEDLINE Publication Records 60
A3.4. Manual Error Checking 61
FIGURES
Figure 1. Distribution of new LRP applications, FY03-FY07 7
Figure 2. Gender distribution of new male and female LRP applicants, and success rates in each
fiscal year, FY03-FY07 9
Figure 3. Percent of new extramural Hispanic or Latino LRP applicants and awardees in each
fiscal year between FY03-FY07 11
Figure 4. Percent of new extramural LRP applicants in the shown racial categories in each fiscal
year from FY03-FY07 12
Figure 5. Success rates of new LRP applicants in the listed racial categories in each fiscal year,
FY03-FY07 13
Figure 6. Gender distribution of ethnicity and race of new LRP applicants from FY03-FY07 14
Figure 7. IC distribution of ethnicity and race of new LRP applicants from FY03-FY07 14
Figure 8. Gender distribution of new LRP applicants, by degree type, FY03-FY07 16
Figure 9. Gender distribution of new extramural applicants by LRP, FY03-FY07 17
NIH Extramural Loan Repayment Program Evaluation v
Figure 10. Distribution of degree type of new LRP applicants in each of the five LRPs, FY03-FY07
18
Figure 11. Distribution of years since degree of new LRP applicants, FY03-FY07 19
Figure 12. Years since degree distribution of new LRP applicants, FY03-FY07 20
Figure 13. Interval distribution of years since degree of new LRP applicants, FY03-FY07 21
Figure 14. Gender by years since degree of new LRP applicants, FY03-FY07 21
Figure 15. Prior research support of new extramural LRP applicants and awardees, FY03-FY07
22
Figure 16. Percent of new extramural LRP awardees FY03-FY07 by prior support source 22
Figure 17. Correlation between applicant reported and DLR verified educational debt level of
new LRP applicants, FY03-FY07 23
Figure 18. Reported educational debt level of new extramural LRP applicants, FY03-FY07 24
Figure 19. Distribution of reported educational debt level of new extramural LRP applicants with
different degree types, FY03-FY07 25
Figure 20. Distribution of renewal extramural LRP applications, FY03-FY07 27
Figure 21. Number of renewal LRP applications and awards in each fiscal year, FY03-FY07 28
Figure 22. Identifying applicants on the “Funding Bubble” 31
Figure 23. Composite analysis of outcomes for the funded and not funded groups in the study
cohort 34
Figure 24. Analysis of outcomes, by gender, of subsequent NIH grant participation 35
Figure 25. Analysis of outcomes, by degree type, of subsequent NIH grant participation 36
Figure 26. Analysis of outcomes, by degree type, of subsequent NIH grant participation 37
Figure 27. Distribution of the number of articles published by applicants in the study cohort 39
Figure 28. Distribution of the number of articles published by applicants in the study cohort for
FY03-FY04 39
Figure 29. Distribution of the number of citations for articles published by applicants in the
study cohort for FY03-FY08 40
Figure 30. Current affiliation of applicants without subsequent NIH records 41
Figure 31. Current position of applicants without subsequent NIH records 42
Figure 32. Definitions of each comparison cohort and the number of awardees in each group
43
Figure 33. Gender distribution of LRP, K, and LRP+K Top-10% awardee cohorts 45
Figure 34. Distribution of degree type of the LRP, K, and LRP+K Top-10% awardee cohorts 46
Figure 35. Composite analysis of outcomes for the LRP, K, and LRP+K Top-10% awardee cohorts
47
Figure 36. Analysis of outcomes, by gender, of subsequent NIH grant participation of the LRP
and the K Top-10% awardee cohort 48
Figure 37. Analysis of outcomes, by degree type, of subsequent NIH grant participation of the
LRP and the K Top-10% awardee cohort 49
Figure 38. Distribution of the number of articles published by the LRP, K, and LRP+K Top-10%
awardee cohorts 50
NIH Extramural Loan Repayment Program Evaluation vi
Figure 39. Distribution of the number of citations for articles published by the LRP, K, and LRP+K
Top-10% awardee cohorts 50
Figure A3.1. Schematic outline of the steps used for the generation of the FY-IC Bubble 59
TABLES
Table 1. Number of new LRP applications and awards, FY03-FY07 8
Table 2. Supporting ICs for each LRP 8
Table 3. Degree distribution of new LRP applicants and awardees, FY03-FY07 15
Table 4. Proportion of new extramural LRP awards to physician researchers at ICs that state a
funding preference for physician researchers, FY03-FY07 17
Table 5. Number of renewal extramural LRP applications and awards, FY03-FY07 28
Table 6. Distribution of degree types for renewal extramural LRP applicants and awardees, FY03-
FY07 29
Table 7. Distribution of renewal awards by initial educational debt for extramural LRP awardees,
FY03-FY07 29
Table 8. Demographic distribution of funded and not funded applicants, and comparison to total
new applicant pool 33
Table 9. Available information on career outcomes for applicants without subsequent NIH
records 41
Table 10. Demographic distribution of total LRP, K, and LRP+K Top-10% awardee cohorts 45
Table 11. Demographic distribution of the 0-6 YSD LRP, K, and LRP+K Top-10% awardee cohorts
45
Table A1.1. Distribution (number and percent of total) of new extramural LRP applications and
awards at each IC, FY03-FY07 52
Table A1.2. Distribution (number and percent of total) of renewal extramural LRP applications
and awards at each IC, FY03-FY07 53
Table A1.3. Success rate of new Clinical Research LRP applications by IC, FY03-FY07, 54
Table A1.4. Success rate of new Pediatric Research LRP applications by IC, FY03-FY07, 54
Table A1.5. Percent of new extramural LRP awards to physician doctorates and academic
doctorates by IC, FY03-FY07 55
Table A2.1. Classification of LRP applicant degrees 56
Table A2.2. Degree definitions and classifications 57
NIH Extramural Loan Repayment Program Evaluation 1
EXECUTIVE SUMMARY
The National Institutes of Health (NIH) Extramural Loan Repayment Program (LRP) is a set of programs
that have as a mandate to attract and retain early career health professionals in biomedical and
behavioral research careers. A barrier to entering these careers, particularly for physicians, is believed to
be the educational debt burden from student loans. The LRP offers qualified applicants repayment of up
to $35,000 per year of eligible educational debt in exchange for a 2-year commitment to conduct
qualified research. Awardees may apply for additional one- or two-year renewal contracts.
The extramural NIH Loan Repayment Program is managed by the NIH Office of the Director, Office of
Extramural Research, Office of Extramural Programs, Division of Loan Repayment. Since its inception in
2001, NIH has made approximately 7,500 awards totaling more than $347 million in loan repayment
funds.
To determine whether the extramural programs are meeting their programmatic goals, DLR carried out
a quantitative program evaluation for new extramural LRP applicants from FY2003 to FY2007. The
evaluation addressed three key areas:
(1) Are the extramural LRPs attracting early-career clinician researchers with high debt?
(2) Do the LRPs positively affect career outcomes for awardees?
(3) How do the LRPs compare to other NIH career development award programs?
Findings
Who is Applying?
The number of new applications has remained fairly constant at about 1900 per year since 2003.
The Clinical Research and Pediatric Research LRPs attract and fund over 80% of new applicants.
About 46% of applicants receive awards; the success rate decreased beginning in FY2004 as NIH
began funding LRP renewal contracts.
The LRPs attract more women than expected based on the demographics of recent PhD and MD
classes. Near-equal numbers of men and women applicants have physician doctoral degrees;
however, 2.5 times more women applicants have academic doctorates. There is no gender
difference in success rate.
The LRPs are attracting more African Americans or Blacks in comparison to the demographics of
recent MD and PhD graduating classes. This is largely due to programs funded by the National
Center on Minority Health and Health Disparities (NCMHD). There are no racial or ethnic differences
in success rate.
The LRPs are attracting biomedical researchers with high debt compared to recent MD and PhD
graduating classes.
NIH Extramural Loan Repayment Program Evaluation 2
The LRPs are attracting early career researchers: 72% of new applicants apply within 6 years of
receiving their qualifying degree. Most academic doctorates apply within 1-3 years of receiving their
degree, while Physician Doctorates tend to apply near the completion of their residency, about 6
years after receiving their degree.
Do the LRPs Retain Participants?
The LRPs are increasing the retention of physician doctorates in the NIH extramural workforce, both
through increased grant applications and participation in research activities.
The LRPs are selectively losing women awardees.
At this point in the LRPs history, there is no significant difference between funded and non-funded
applicants in research productivity. This may be because the current study examined outcome data
for 1-4 years since award, which may be too soon a time point to use to measure productivity for
LRP applicants. This may be further accentuated by the fact that LRP participants tend to be early in
their career.
How does the LRP Compare to Other NIH Mentored Career Awards Programs?
The LRPs are attracting applicants that are earlier in their research career than the K program.
Overall, the top 10% of LRP awardees are more likely to be Academic Doctorates and to be within 3
years of degree than either K or LRP+K top 10% awardees.
The K awardees were more likely to apply for NIH Research Project Grant (RPG) support 1-4 years
after their award than the LRP awardees. This follows from the previous finding that the LRP
awardees are in a less developed career stage than K awardees and may need time to produce
research results that can support a RPG application.
Gender distribution among the top 10% of awardees was similar for LRP and K awardees. However,
K awardee women were more likely than their LRP counterparts to apply for subsequent NIH grant
support and to be retained in the NIH extramural workforce 1-4 years after award.
Similarly, even though the LRPs have a clear retention effect on physician doctorates, K awardee
Physician Doctorates were more likely to apply for and receive subsequent funding, and to be
retained in the NIH extramural workforce 1-4 years after award.
Conclusions and Recommendations
These findings suggest that the LRP is attaining its goals and may be a model program for recruiting
and retaining early career scientists who otherwise would have a high tendency to leave the
biomedical research workforce after earning a doctoral degree.
The LRP appears to be effective in recruiting and retaining its target population, physician
doctorates, but is not effective in retaining its largest applicant pool, women PhDs. It would be
appropriate to examine program design and retention in more detail to determine how the LRP can
better serve women PhDs.
The LRP is attracting early-career researchers, who appear to be in an earlier stage of their career
maturation than K awardees. The NIH may consider helping to prepare LRP awardees for
NIH Extramural Loan Repayment Program Evaluation 3
subsequent grant submissions, for example, to the K-awards process, to improve retention in the
NIH extramural workforce.
This evaluation focused on researchers 2-5 years after their receipt of the LRP award. While this
provides an early indication of whether the program is meeting its goals, it will be important to
reassess LRP awardee research productivity in subsequent studies.
HISTORY OF THE NIH EXTRAMURAL LRP
The National Institutes of Health (NIH) Extramural Loan Repayment Program (LRP) is a set of programs
that have a mandate to attract and retain early career health professionals in biomedical and behavioral
research careers. A barrier to entering these careers, particularly for physicians, is believed to be the
high educational debt burden from student loans. The LRP encourages qualified applicants an
opportunity pursue research careers b y repaying their eligible educational debt in exchange for a 2-year
commitment to conduct qualified research.1 Awardees may apply for additional one- or two-year
renewal contracts.
NIH administers several intramural and extramural LRPs. The intramural LRPs have been offered since
fiscal year 1989 and are available to NIH intramural researchers only. The extramural Clinical and
Pediatric Loan Repayment Programs (LRP) were established by Congress in 2000.2 The extramural
Clinical Research LRP for Individuals from Disadvantaged Background and the LRP for Health
Disparities Research were established under the Minority Health and Health Disparities Research and
Education Act of 2000 (Public Law 106-525).3 The extramural Loan Repayment Program for
Contraception and Infertility Researchers awards are made pursuant to the NIH Revitalization Act of
1993 (Pub. L. 103-43), adding section 487B of the Public Health Service (PHS) Act, Loan Repayment
Program for Research with Respect to Contraception and Infertility, 42 U.S.C. 288-2.4 The Lenfant
Committee set NIH policy and procedures for the extramural program.5
1 The LRP provides for the repayment of up to $35,000 of the principal and interest of the extant educational loans of such
health professionals for each year of obligated service. Loan repayment benefits represent taxable income. Payments equal to
39 percent of total loan repayments are issued to the Internal Revenue Service on behalf of program participants to offset
Federal tax liabilities incurred. 2 The Loan Repayment Program for Clinical Researchers was authorized under the Clinical Research Enhancement Act of the
Public Health Improvement Act of 2000 (PL 106-505). The NIH define clinical research as “patient-oriented research that is
conducted with human subjects..., or ...involving material of human origin for which an investigator or colleague directly
interacts with human subjects...or epidemiologic or behavioral studies, outcomes research or health services research…”. The
Loan Repayment Program for Pediatric Research was established under the Children's Health Act (PL 106-310). The Act defines
Pediatric Research as research that is “directly related to diseases, disorders, and other conditions in children”. 3 http://grants.nih.gov/grants/guide/notice-files/NOT-OD-08-089.html; http://grants.nih.gov/grants/guide/notice-files/NOT-
OD-08-086.html 4 http://grants.nih.gov/grants/guide/notice-files/NOT-OD-08-087.html
5 See, e.g., http://www.nih.gov/icd/od/foia/icdirminutes/icdir013102.htm; http://deainfo.nci.nih.gov/ADVISORY/ncab/122_0602/mins11Jun02.pdf.
NIH Extramural Loan Repayment Program Evaluation 4
The extramural LRPs are administered by the NIH Office of Extramural Research (OER), Office of
Extramural Programs, Division of Loan Repayment (DLR). Twenty-four NIH Institutes and Centers
participate in the LRP; their program funding level was set by NIH in FY2003 using a formula basis
related to the size of their clinical research portfolio. The five extramural programs are:
(1) The Clinical Research LRP supports individuals who are conducting patient-oriented research
with human subjects or research on the causes and consequences of disease in human
populations involving material of human origin (such as tissue specimens and cognitive
phenomena) for which an investigator or colleague that requires direct interaction with human
subjects in an outpatient or inpatient setting;
(2) The Pediatric Research LRP supports individuals who are conducting studies of diseases,
disorders, and other conditions that affect children;
(3) The Health Disparities Research LRP aims to recruit and retain highly qualified health
professionals to research careers in basic research, clinical research, or behavioral research
directly related to health disparity populations and the medically underserved;
(4) The Clinical Research LRP for Individuals from Disadvantaged Backgrounds offers an
opportunity in clinical research to individuals from a family with an annual income below low-
income thresholds published by the U.S. Bureau of the Census;
(5) Finally, the objective of the Contraception and Infertility Research LRP is two-fold: to evaluate,
treat or improve conditions which result in the failure of couples to either conceive or bear
young, and to provide new or improved methods of preventing pregnancy.
To qualify, LRP applicants must be US citizens or permanent residents, their educational debt must be
greater than 20 percent of their annual salary and compensation, they must have a qualifying doctoral
degree (except for CIR), and they must be in a position to commit at least 50 percent time (at least 20
hours per week) to research in a qualifying area. An applicant’s research must be funded by a domestic
nonprofit or U.S. Government (federal, state, or local) entity, and the institution at which the applicant is
engaged in research must provide assurance of research support through the end of the two-year
contract. The actual loan repayment amount is determined by a formula that includes the awardee’s
educational debt level and annual income.
There is only one application receipt date per year, in early December. Applications are assigned to and
reviewed by each participating NIH Institute or Center (IC). Each IC conducts peer review of the
applications using a panel composed of extramural scientists. Review criteria emphasize the potential to
become a contributing scientist in the area of the LRP. Standard NIH scoring (100 – 500) is used.
However, this score is only one part of the funding priority.
The extramural Loan Repayment Program has made approximately 7,500 awards from over $347 million
in loan repayment funds from FY2001 through FY2007. To determine whether the extramural programs
are meeting their programmatic goals, in FY 2008 DLR carried out a quantitative program evaluation of
new extramural LRP applicants that submitted an application to the LRP during the FY2003 through
FY2007 cycles.
NIH Extramural Loan Repayment Program Evaluation 5
EVALUATION OBJECTIVES
The overall objective of the LRP program evaluation is to determine whether LRP awards are effective in
their broad purpose of recruiting and retaining early-career health professionals in biomedical and
behavioral research careers. The evaluation addressed the following questions:
1. Who is applying to the extramural Loan Repayment Programs? Is the program attracting the
target applicant pool of early-career researchers with high educational debt?
a. How are applications distributed between the five programs?
b. What is the racial, ethnic, and gender distribution of the applicant pool?
c. What is the educational degree distribution of the applicant pool? How soon after
earning their qualifying degree are researchers applying to the LRP?
d. What is the educational debt level of LRP applicants?
e. What proportion of LRP awardees are submitting renewal applications?
2. Does participation in the extramural Loan Repayment Program lead to increased retention in
the extramural NIH-funded workforce?
a. Are LRP award recipients more likely, less likely, or equally likely than matched non-
recipients to apply for and receive subsequent NIH awards? To participate as a non-PI
on a NIH research grant?
b. Are there differences in retention by gender or degree?
c. Are LRP award recipients more likely, less likely, or equally likely than non-recipients to
publish research articles? How do the citation rates of the two groups compare?
3. How does the LRP compare to other NIH mentored career-development programs for early-
career investigators?
a. Are the applicant pools similar?
b. Are retention and publication rates similar between NIH awards programs? Is there an
effect of gender or degree on retention?
RESEARCH DESIGN
Scope. The extramural LRP initiated in fiscal year 2001 with the Health Disparities and Clinical Research
LRP for Individuals from Disadvantaged Backgrounds LRPs. Then in 2002, NIH received the first
applications for the Clinical and Pediatric LRPs. In FY 2003, NIH started to receive and fund renewal LRP
applications. Between 2003 and 2007, the extramural program received 9,820 new applications, of
which 4,565 were awarded. 4,092 renewal applications were also submitted, of which 2,937 were
awarded.6 The evaluation focused on those years after the renewal program had been established.
6 Note that this evaluation collected application counts as of 3/31 and awards as of 10/1. This differs in some years
from the LRP Data Book, which collected application counts on 1/31/2003, 12/31/2003, 3/23/2004, 12/1/2005, 3/31/2006, and 3/31/2007.
NIH Extramural Loan Repayment Program Evaluation 6
Data Sources. The LRP Online Application System (OAS) database was the primary data source used for
LRP applicant information. OAS contains records of personal, demographical, institutional, educational,
funding, loan, and research project information for all LRP applicants. This information is submitted by
applicants and their colleagues via a web interface during the LRP application process. Other activities
related to the submission, including eligibility verification, financial vetting, peer-review, and funding
decisions are also recorded in this database. Data can be aggregated into fiscal years/cohorts, is
independent of prior years, and represents a snapshot of the applications following funding decisions,
after which no more activities are recorded/updated.
We used the NIH grants database, IMPAC II, to obtain data on current and subsequent NIH grant
applications for LRP and career development awardees. We gathered information on publications from
the National Library of Medicine MEDLINE database and information on citations from the Thomson-
Reuters ISI Web of Science.
Statistics. We used a Chi-square 2x2 contingency table to assess difference between groups. The p-
value was calculated using a Fischer’s exact test.
Applicant Pool. To understand who is applying to the extramural LRP we used data from OAS to analyze
the following parameters for new LRP applicants from FY2003 to FY2007: (i) total applications by year;
(ii) applications by LRP; (iii) applications by funding NIH Institute or Center (IC); (iv) gender; (v) race and
ethnicity (vi) degree type; (vii) years since qualifying degree; (viii) prior research support; and (ix)
educational debt level.
Impact of Participation. We used applicant data from OAS and priority score data from IMPAC II to
develop matched cohorts of funded and not funded applicants. The cohort for this analysis was defined
as unique new LRP applicants from FY2003 to FY2007. Using novel methodology (detailed in Appendix
3), we identified applicants and awardees on the “funding bubble” – the part of the distribution where
applicants have an equal chance of being funded or not funded. We used a regression discontinuity
design to examine the impact of receiving an LRP on subsequent involvement in the extramural NIH-
funded workforce for funded and not funded LRP applicants. We used grant data from IMPAC II and Key
Personnel data from the IMPAC II Enumeration table to determine subsequent Research Project Grant
(RPG) and Fellowship and Training Grant application and awards for all applicants in the bubble. We
determined if outcomes for participants varied by gender or degree. We also used a manually-verified
automated process to match publications to applicant publications to determine whether productivity
differed between the funded and not funded applicants.
Comparison with Other NIH Mentored Career Awards. We used LRP applicant data from OAS and grant
data from IMPAC II to select the top 10% of awardees by priority score for LRP as well as the top 10% of
mentored K awards, namely K01, K08, and K23 awards. We then performed the same grant outcome
and publication productivity analyses as was conducted for the funding bubble group.
NIH Extramural Loan Repayment Program Evaluation 7
QUESTION 1 : WHO IS APPLYING?
1.1 Goals
Investigate the characteristics of the extramural applicant pool to understand who is applying to the
program and whether the programs are attracting their target audience of early-career clinical
researchers.
1.2 Methods
We defined a set of nine parameters to evaluate the applicant and awardee pool of the program. The
parameters examined were: (i) total applications by year; (ii) applications by LRP; (iii) applications by
funding NIH Institute or Center (IC); (iv) gender; (v) race and ethnicity (vi) degree type; (vii) years since
qualifying degree; (viii) prior research support; and (ix) educational debt level. Single parameter analysis
provided valuable information regarding the applicant pool, for example, the number of male and
female applicants over the five-year evaluation period. However, this analysis was limited in its ability to
evaluate conditional dependencies, for example, the number of male and female applicants with a
certain degree type. Thus, to better understand parameter co-dependencies, we performed cross-
parameter analyses.
1.3 New Applicants
1.3.1 New applications and awards, by program and IC
From FY2003 to FY2007, LRP received a total of 9,820 new applications and made 4,565 new awards, a
success rate of 46%. The Clinical Research LRP received the majority of new applications (57%).
Additionally, 26% were submitted to the Pediatric Research LRP, 13% to the Health Disparities Research
LRP, and 2% each to the Clinical Research LRP for Individuals from Disadvantaged Backgrounds LRP and
to the Contraception and Infertility Research LRP (Figure 1). A similar distribution was observed for new
awards made by each LRP. These data and the success rates for each LRP are presented in Table 1.
Figure 1. Distribution of new LRP applications, FY03-FY07.
PediatricResearch
(26%)
Clinical
Research(57%)
Health DisparitiesResearch
(13%)
Contraception
and InfertilityResearch
(2%)
Clinical Researchfor Individuals
from DisadvantagedBackgrounds
(2%)
The Clinical and Pediatric Research LRPs account for the majority
of new applications.
NIH Extramural Loan Repayment Program Evaluation 8
Table 1. Number of new LRP applications and awards, FY03-FY07.
LRP New
Applications
Percent of New
Applications New Awards
Percent of New Awards
Success Rate
Clinical Research 5,646 57% 2,631 58% 47%
Pediatric Research
2,515 26% 1,102 24% 44%
Health Disparities Research
1,314 13% 674 15% 51%
Clinical Research for Individuals from Disadvantaged Backgrounds
171 2% 83 2% 49%
Contraception and Infertility Research
174 2% 75 2% 43%
Total 9,820 4,565 46%
The National Center on Minority Health and Health Disparities (NCMHD) receives all applications
submitted to the Health Disparities Research LRP and the Clinical Research LRP for Individuals from
Disadvantaged Backgrounds. All applications submitted to the Contraception and Infertility Research LRP
are sent to the National Institute of Child Health and Human Development (NICHD). With the exception
of NCMHD, the Clinical Research LRP and the Pediatric Research LRP are supported by all participating
NIH ICs (Table 2).
Table 2. Supporting ICs for each LRP.
Of the total applications received between FY2003-FY2007, 15% of new applications were referred to
NCMHD for review. The National Cancer Institute (NCI) received 13%, the National Heart, Lung, and
Blood Institute (NHLBI) and the National Institute of Mental Health (NIMH) each received 11%, and
NICHD received 10% of new applications. The complete distribution of new applications and awards for
each IC is presented in Appendix 1.
LRP Supporting IC
Clinical Research ICs except NCMHD
Pediatric Research ICs except NCMHD
Health Disparities Research NCMHD
Clinical Research for Individuals from Disadvantaged Backgrounds
NCMHD
Contraception and Infertility Research
NICHD
NIH Extramural Loan Repayment Program Evaluation 9
1.3.2. New applications and awards, by gender
The National Science Foundation reports that in recent years, men received 52.5% of biological sciences
Ph.D. degrees while women received 47.5%.7 The American Association of Medical Colleges reports that
55.8% of M.D. degrees were earned by men and 44.2% by women.8 Among biomedical postdoctoral
researchers, 60.8% were male and 39.2% were female.9
To analyze gender distribution in the loan repayment program, we determined the number of men and
women who applied for a new LRP award from FY2003 to FY2007. In contrast to what one may expect
based on the potential pool of applicants among recent Ph.D. and M.D. graduates and postdoctorates,
more new applications were submitted by women (53%) than men (44%).10 This distribution and success
rate is shown in Figure 2. The higher proportion of women LRP applicants is observed throughout the
reporting period. The success rates for men and women are not significantly different.
Figure 2. Gender distribution of new male and female LRP applicants, and success rates in each
fiscal year, FY03-FY07.
7 National Science Foundation (2008). Women, Minorities, and Persons with Disabilities in Science and Engineering, Table F-11. National Science Foundation: Arlington, VA. http://www.nsf.gov/statistics/wmpd/figf-1.htm. We chose the 2001-2005 timeframe to calculate gender distribution as most PhDs apply for the LRP within one year of degree. 8 Association for American Medical Colleges (2008). AAMC Data Book: Medical Schools and Teaching Hospitals by the Numbers, Table B10. AAMC: Washington DC. We chose the 1997-2001 timeframe as most MDs apply for the LRP 4-6 years after degree. 9 National Science Foundation (2008). Women, Minorities, and Persons with Disabilities in Science and Engineering, Table G-2. National Science Foundation: Arlington, VA. http://www.nsf.gov/statistics/wmpd/postdoc.htm. We calculated the average gender distribution from 2001-2005. 10 3% of applicants did not specify gender.
0%
10%
20%
30%
40%
50%
60%
70%
0%
10%
20%
30%
40%
50%
60%
70%
FY03 FY04 FY05 FY06 FY07
Succ
ess
Rat
e
Per
cen
t o
f N
ew L
RP
Ap
plic
atio
ns
Female Applicants Male Applicants
Success Rate (Female) Success Rate (Male)
More new applications
were submitted by
women. Success rates
between genders were
not significantly different.
NIH Extramural Loan Repayment Program Evaluation 10
1.3.3. New applications and awards, by race and ethnicity
Race and ethnicity data are self-reported on the LRP application form. The 1997 Office of Management
and Budget (OMB) Directive 15 classifies race and ethnicity separately. This standard includes collecting
and reporting data on two ethnic categories (Hispanic or Latino11, and Not Hispanic or Latino) and five
racial categories (American Indian or Alaska Native12, Asian13, Black or African American14, Native
Hawaiian or Other Pacific Islander15, and White16). Until FY 2008, the application form did not separate
the ethnicity designation from the race designation; however applicants were allowed to select one or
more, thus allowing them to select ethnicity and race. Analysis of race and ethnicity for this report is in
accordance with OMB Directive 15, as described below.
The National Science Foundation reports that Hispanics received 4.5% of biological sciences Ph.D.
degrees, Blacks 3.2%, Asians 12.7%, Native Americans 0.3%, and Whites 76%.17 The American
Association of Medical Colleges reports that 6.4% of M.D. degrees were earned by Hispanics, 7.1% by
Blacks, 19.8% by Asians, 0.8% by Native Americans, and 64.1% by Whites.18 Detailed data on the
race/ethnicity of postdoctoral researchers are not readily available.
1.3.3.1 Distribution of ethnicity
From FY2003-FY2007, the number of new LRP applicants who self-identified as Hispanic or Latino during
each fiscal year is shown in Figure 3. Between 5-7% of new LRP applicants identified themselves as
belonging to this ethnic category; similar proportions were observed for new awardees as well (5-8%).
The success rate is not significantly different from the overall applicant pool.
11 Hispanic or Latino: A person of Cuban, Mexican, Puerto Rican, South or Central American, or other Spanish
culture or origin, regardless of race. 12 American Indian or Alaska Native: A person having origins in any of the original peoples of North, Central, or
South America, and who maintains tribal affiliations or community attachment. 13 Asian: A person having origins in any of the original peoples of the Far East, Southeast Asia, or the Indian
subcontinent including, for example, Cambodia, China, India, Japan, Korea, Malaysia, Pakistan, the Philippine Islands, Thailand, and Vietnam.
14 Black or African American: A person having origins in any of the black racial groups of Africa. 15 Native Hawaiian or Other Pacific Islander: A person having origins in any of the original peoples of Hawaii, Guam,
Samoa, or other Pacific Islands. 16 White: A person having origins in any of the original peoples of Europe, the Middle East, or North Africa. 17 National Science Foundation (2008). Women, Minorities, and Persons with Disabilities in Science and Engineering, Table F-11. National Science Foundation: Arlington, VA. http://www.nsf.gov/statistics/wmpd/figf-1.htm. We chose the 2001-2005 timeframe to calculate gender distribution as most PhDs apply for the LRP within one year of degree. 18
Association for American Medical Colleges (2008). AAMC Data Book: Medical Schools and Teaching Hospitals by the Numbers, Table B8. AAMC: Washington DC. We chose the 1997-2001 timeframe as most MDs apply for the LRP 4-6 years after degree.
NIH Extramural Loan Repayment Program Evaluation 11
Figure 3. Percent of new extramural Hispanic or Latino LRP applicants and awardees in each fiscal
year between FY03-FY07. Also shown is the success rate for the Hispanic or Latino and the Not
Hispanic or Latino ethnic groups
1.3.3.2 Distribution of race
Data on race were analyzed by placing an applicant in one category only. Applicants who self-identified
themselves in more than one category were assigned to a distinct multi-racial category. Those applicants
who declined to respond to the question were classified in another distinct category, “Other/No
Response”. The distribution of race of applicants is shown in Figure 4.
0%
10%
20%
30%
40%
50%
60%
70%
0%
2%
4%
6%
8%
10%
12%
14%
FY03 FY04 FY05 FY06 FY07
Succ
ess
Rat
e
Per
cen
t o
f N
ew L
RP
Ap
plic
atio
ns
and
Aw
ard
s
Applicants
Awardees
Success Rate (Hispanic or Latino)
Success Rate (Not Hispanic or Latino)
Success rate for Hispanic
or Latino applicants is not
different from the overall
success rate.
NIH Extramural Loan Repayment Program Evaluation 12
Figure 4. Percent of new extramural LRP applicants in the shown racial categories in each fiscal
year from FY03-FY07.19
Asians represented ~10% of new applicants in each fiscal year between FY2003-FY2007, proportionately
less than in recent PhD and MD classes. Blacks or African Americans represented between 9-12% of the
new applicant pool during this reporting period, more than in recent MD and PhD classes. American
Indians or Native Alaskans represented <1% of the new LRP applicants, as did Native Hawaiians or Other
Pacific Islanders (<1%). Over the reporting period, Whites constituted an average of 64% of the new
applicant pool, while multi-racial applicants comprised ~2% of applicants. On average, 9% of applicants
declined to self-identify at the time of application to the LRP. The success rate for each racial category,
shown in Figure 5, did not differ from each other or from the average overall success rate.
19
To avoid compression on the y-axis because of the difference between the proportion of White applicants and the other racial categories, this racial category is not represented in Figure 4. The distribution of Whites for each fiscal year is: 67% (FY03), 66% (FY04), 63% (FY05), 62% (FY06), and 63% (FY07).
0%
2%
4%
6%
8%
10%
12%
14%
FY03 FY04 FY05 FY06 FY07
Per
cen
t o
f N
ew L
RP
Ap
plic
ants
American Indian or Alaska NativeAsianBlack or African AmericanNative Hawaiian or Other Pacific IslanderMore Than One Race
Asians represented proportionally
fewer new applicants than in
recent PhD and MD classes,
whereas Blacks and African
Americans represented more of the
applicant pool.
NIH Extramural Loan Repayment Program Evaluation 13
Figure 5. Success rates of new LRP applicants in the listed racial categories in each fiscal year,
FY03-FY07. Due to the small sample size of the American Indian or Alaska Natives and the Native
Hawaiian or Other Pacific Islanders, the success rate for these racial groups is not shown.
1.3.3.3 Distribution of ethnicity / race and gender
Figure 6 shows the distribution of ethnicity and race of new male and female applicants between
FY2003-FY2007. For most categories, significantly more women applied than men.20
20 We were not able to measure significance for Hawaiian and Pacific Islanders due to small sample size.
0%
10%
20%
30%
40%
50%
60%
70%
80%
FY03 FY04 FY05 FY06 FY07
Succ
ess
Rat
e,N
ew L
RP
Ap
plic
ants
Asian Black or African American
White More than One Race
There was no significant
difference in success rates
among racial categories or in
comparison to the average
success rate of 46%.
NIH Extramural Loan Repayment Program Evaluation 14
Figure 6. Gender distribution of ethnicity and race of new LRP applicants from FY03-FY07.
1.3.3.4 Distribution of ethnicity / race by IC
Figure 7 shows the distribution of ethnicity and race of new applicants by IC between FY2003-FY2007.
The NCMHD LRPs account for the majority of applications by Blacks and Hispanics.
Figure 7. IC distribution of ethnicity and race of new LRP applicants from FY03-FY07.
0%
10%
20%
30%
40%
50%
60%
70%
Am
eric
an
Ind
ian
or
Ala
ska
Nat
ive
Asi
an
Bla
ck o
r A
fric
an
Am
eric
an
His
pan
ic
or
Lati
no
Nat
ive
H
awai
ian
o
r O
ther
P
acif
ic …
Wh
ite
Mo
re
Than
On
e R
ace
Per
cen
t o
f N
ew L
RP
Ap
plic
atio
ns
All IC All IC except NCMHD NCMHD only
Across races and
ethnicities,
significantly more
women applied to
LRPs than men.
The majority of
applications by
Blacks and Hispanics
were made to the
NCMHD LRPs.
NIH Extramural Loan Repayment Program Evaluation 15
1.3.4. New applications and awards, by degree type
To determine how successful the LRP has been in attracting early career health professionals in clinical
research careers, we examined degree type and how long after earning a doctoral degree a person
applied to the program.21 Details about degree classification, including degree abbreviations, are
included in Appendix 2.
1.3.4.1 Distribution of degree types
Table 3 shows the distribution of degree types for new applicants and awardees between FY2003-
FY2007. Applicants with physician doctorates and with academic doctorates are represented at similar
proportions (47% and 42%, respectively), with 6% of applicants having a dual physician
doctorate/academic doctorate. Five percent of applicants had a professional clinical degree, while <1%
were allied health professionals.
Table 3. Degree distribution of new LRP applicants and awardees, FY03-FY07.
Degree Type New LRP
Applications % New LRP
Applications New LRP Awards
% New LRP Awards
Success Rate
Physician Doctorate 4,663 47% 2,134 47% 46%
Professional Clinical Doctorate
442 5% 135 3% 31%
Academic Doctorate 4,130 42% 1,915 42% 46%
Physician Doctorate / Academic Doctorate
551 6% 368 8% 67%
Allied Health Professional
34 <1% 13 <1% 38%
Total 9,820
4,565
1.3.4.2 Distribution of degree types and gender
We examined the differences in the number of men and women applicants by gender and by degree.
This cross-parameter analysis revealed gender differences, shown in Figure 8.
21
While a doctoral-level degree is a prerequisite for submitting an LRP application, the Contraception and Infertility Research LRP accepts applications from non-doctoral-level allied health professional degree holders, defined as those individuals with a bachelor’s and/or master’s degree in a relevant discipline.
NIH Extramural Loan Repayment Program Evaluation 16
Figure 8. Gender distribution of new LRP applicants, by degree type, FY03-FY07
1.3.4.2.1 Degree by gender: Of all new male applicants (blue bars, Figure 8), a majority (54%) were
physician doctorates and 33% were academic doctorates. In comparison, of all new women applicants
(green bars, Figure 8), 42% were physician doctorates and 50% were academic doctorates.
1.3.4.2.2 Gender by degree: When comparing the distribution of men and women applicants across
degree categories (i.e., comparing blue to green bars in Figure 8), there are significant gender
differences in the proportion of applications from physician doctorates, academic doctorates, and dual
physician doctorate/academic doctorates.
1.3.4.3 Distribution of degree types by IC
Another way to examine whether the LRP is reaching its intended audience is to test whether the
applicant pool matches IC funding preferences. Of the 24 funding ICs, five have a publicly-stated
preference for the type of doctoral degree an LRP applicant holds: the National Institute of Allergy and
Infectious Diseases (NIAID), the National Institute of Biomedical Imaging and Bioengineering (NIBIB),
NICHD, the National Institute of General Medical Sciences (NIGMS), and the National Institute of
Neurological Disorders and Stroke (NINDS).22 Table 4 shows the distribution of degree types of the
awardees at these ICs.
22 Based on the preference statements provided by each IC, posted on the LRP Web site at www.lrp.nih.gov.
Male applicants typically held
physician doctorates, whereas
the majority of female applicants
held academic doctorates.
NIH Extramural Loan Repayment Program Evaluation 17
Table 4. Proportion of new extramural LRP awards to physician researchers at ICs that state a
funding preference for physician researchers, FY03-FY07.
IC Percent of new LRP awards
to physician doctorates, FY03-FY07
NIAID 77%
NIBIB 44%
NICHD 53%
NIGMS 79%
NINDS 72%
The other 19 participating ICs did not state a preference for applicant degree type. Details about the
degrees of LRP recipients by IC are presented in Appendix 1.
1.3.5. New applications and awards, by LRP
1.3.5.1 Distribution of IC and LRP
The Clinical Research LRP and the Pediatric Research LRP were supported by all ICs except NCMHD,
which does not participate in either program. Details about the participation of various ICs in the
different LRPs are presented in Appendix 1.
1.3.5.2 Distribution of gender and LRP
To determine if there were any differences in the number of LRP applicants based on gender, a cross-
parameter analysis was done of men and women applicants in each of the five LRPs, shown in Figure 9.
Figure 9. Gender distribution of new extramural applicants by LRP, FY03-FY07.
A greater proportion of
women applied to the
Health Disparities Research
LRP compared to men.
NIH Extramural Loan Repayment Program Evaluation 18
1.3.5.2.1 LRP by gender: Of all new male applicants, (blue bars, Figure 9), a majority (64%) applied to the
Clinical Research LRP and 24% applied to the Pediatric Research LRP. This distribution was also seen
among women applicants (green bars, Figure 9): a majority (51%) applied to the Clinical Research LRP
and 27% applied to the Pediatric Research LRP. It is interesting to note that a greater proportion of
women applications (17%, green bar) applied to the Health Disparities Research LRP than male
applicants (9%, blue bar).
1.3.5.2.2 Gender by program: When comparing the distribution of men and women applicants in each of
the five degree type categories (i.e., comparing blue vs. green bars in Figure 9), for all programs, there
was a significant gender difference in the proportion of men and women applying. Proportionately
more men applied to the Clinical Research program, and more women applied to the other four LRPs.
1.3.5.3 Distribution of degree types and LRP
We performed a cross-analysis of degree and LRP to determine what proportion of degree types
submitted applications to each of the LRPs. As expected by the preferences of each IC and the
differential representation of IC per LRP, the five degree types are not equally represented in each of the
LRPs (Figure 10). This parameter set interrelates with the differential gender representation in the
various programs: one would expect and we in fact did see more women in programs that have more
academic doctorates. Health Disparities Research was the only LRP that attracted many more academic
doctorates than physician doctorates: 74% of new applicants to this LRP were academic doctorates.
Figure 10. Distribution of degree type of new LRP applicants in each of the five LRPs, FY03-FY07.
0%
10%
20%
30%
40%
50%
60%
70%
80%
90%
100%
Clinical Research
Pediatric Research
Health Disparities Research
Clinical Research for
Individuals from Disadvantaged Backgrounds
Contraception and Infertility
Research
Per
cen
t o
f N
ew L
RP
Ap
plic
ants
Allied Health Professional Degree
Physician Doctorate / Academic Doctorate
Academic Doctorate
Professional Clinical Doctorate
Physician Doctorate
Physician doctorates
represented a majority of
applicants for most of the
LRPs except Health Disparities
Research, which had a
majority of academic
doctorates.
NIH Extramural Loan Repayment Program Evaluation 19
1.3.6. New applications and awards, by years since degree
1.3.6.1 Distribution of years since degree and degree type
To determine if the program is attracting early career researchers, we analyzed years since qualifying
degree for new applicants for the entire reporting period (FY2003-FY2007). Time since degree is
dependent on a number of factors, including postdoctoral training, medical residency and internship,
and available research support and resources. Figure 11 shows the distribution of number of
applications and awards plotted against the years since receiving a qualifying degree. Seventy-two
percent of new applicants apply within six years of receiving their degree. Ninety percent of new
applicants apply within nine years of receiving their degree. Of those applicants who applied after 10
years of receiving their qualifying degree 58% were Physician Doctorates, 19% were Academic
Doctorates, 18% were dual Physician Doctorates/Academic Doctorates, 56% were men, and 41% were
women—a very different distribution than that of the overall applicant pool.
Figure 11. Distribution of years since degree of new LRP applicants, FY03-FY07.
The peaks observed at one and five years since degree suggest a bimodal distribution. To determine the
underlying basis for this distribution, we performed a cross-parameter analysis with applicant degree
type. As illustrated in Figure 12A, physician doctorates and academic doctorates apply to LRP at
different times following their degree. A majority of academic doctorates (67%) apply within three
years of receiving their qualifying degree, while only 9% of physician doctorates and 8% of dual degree
physician doctorates/academic doctorates apply in the same time period. Conversely, 53% of physician
doctorates (and 34% of dual doctorates) apply between four to six years of receiving their qualifying
degree, and only 21% of academic doctorates submit applications at this time of their career (Figure
12B).
0
200
400
600
800
1000
1200
1400
1600
<1 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 15+
Nu
mb
er o
f N
ew L
RP
Ap
plic
ants
an
d A
war
dee
s
Years Since Degree
Applicants Awardees
The peaks of applications
and awards at 1 and 5
years following degree
conferral suggest two
distinct applicant groups.
NIH Extramural Loan Repayment Program Evaluation 20
Figure 12. Years since degree distribution of new LRP applicants, FY03-FY07. A (top), Distribution of different degree types. B (bottom), Proportion of degree types with respect to time interval categories.
0
200
400
600
800
1000
1200
1400
<1 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 15+
Nu
mb
er o
f N
ew L
RP
Ap
plic
ants
Years Since Degree
Physician Doctorate
Professional Clinical Doctorate
Academic Doctorate
Physician Doctorate / Academic Doctorate
0%
10%
20%
30%
40%
50%
60%
70%
80%
90%
100%
Physician Doctorate
Professional Clinical
Doctorate
Academic Doctorate
Physician Doctorate / Academic Doctorate
Allied Health Professional
Degree
Up to 3 Years 4 - 6 Years 7 - 9 Years 10+ Years
Most new LRP applications
are filed within 3 years of
degree conferral by
academic doctorates and
within 4-6 years by physician
doctorates.
NIH Extramural Loan Repayment Program Evaluation 21
Based on this distribution, for cross-correlation analysis we defined four time intervals: (i) Up to 3 years;
(ii) 4-6 years; (iii) 7-9 years; and (iv) 10+ years. Our analysis shows that a majority of new LRP applicants
are in early stages of their career: 72% of applicants completed their qualifying degree within the six
years prior to applying to the LRP, and 90% of new applicants completed their degree within nine years
of applying to the program (Figure 13).
Figure 13. Interval distribution of years since degree of new LRP applicants, FY03-FY07.
1.3.6.2 Distribution of years since degree and gender
To determine if men and women applied to LRP at different times after completing their qualifying
degree, we analyzed gender distribution in the four intervals defined above. More women (62%) apply
for a new LRP award within three years of receiving their qualifying degree than men (35%) (Figure 14);
this finding flows with the association between gender and degree. The proportion of new men and
women applicants in other time intervals is relatively similar.
Figure 14. Gender by years since degree of new LRP applicants, FY03-FY07.
4-6 Years
(37%)
Up to 3 Years
(35%)
7-9 Years(18%)
10+ Years
(10%)
0
500
1000
1500
2000
2500
Up to 3 Years4-6 Years7-9 Years
10+ Years
Nu
mb
er o
f N
ew L
RP
Ap
plic
ants
Years Since Degree
Male
Female
More women than
men apply to LRPs
within 3 years of
receiving their degree
NIH Extramural Loan Repayment Program Evaluation 22
1.3.7. New applications and awards, by prior research support
For the new applicant and awardee pools, we analyzed the distribution of prior research support from
three sources: the NIH alone, both NIH and non-profit organizations, and non-profit organizations alone.
While a near-equal proportion of new applicants had prior research support from each of these three
sources, 15% more awardees had prior research support from the NIH alone than from non-profit
organizations alone (39% vs. 24%) (Figure 15). This observation extends to each fiscal year in the
reporting period; individuals with prior research support from the NIH alone had a greater success rate
than individuals with prior research support from non-profit organizations alone (Figure 16).
Figure 15. Prior research support of new extramural LRP applicants and awardees, FY03-FY07.
The overall success rate of each group is shown (encircled numbers).
Figure 16. Percent of new extramural LRP awardees FY03-FY07 by prior support source.
3560(36%)
3135(32%)
3125(32%)
1791(39%)
1665(36%)
1109(24%)
0
500
1000
1500
2000
2500
3000
3500
4000
NIH Only NIH and Non-Profit Non-Profit Only
Nu
mb
er o
f N
ew L
RP
Ap
plic
atio
ns
and
Aw
ard
s
Applications Awards
0%
5%
10%
15%
20%
25%
30%
35%
40%
45%
FY03 FY04 FY05 FY06 FY07
Per
cen
t o
f N
ew E
xtra
mu
ral
LRP
Aw
ard
s
NIH Only NIH and Non-Profit Non-Profit Only
Although a comparable
number had prior research
support from either the NIH
or non-profit organizations
alone, new applicants with
NIH funding were more likely
to receive LRP awards.
NIH Extramural Loan Repayment Program Evaluation 23
1.3.8. New applications and awards, by educational debt level
The goal of the loan repayment programs is to attract healthcare professionals into research careers by
offering qualified researchers an opportunity to pay down educational debt. The programs make
payments on all qualified educational debt, provided that the educational debt is at least 20% of the
applicant’s annual base salary. New LRP applicants are required to report their current educational debt
level as part of their application to the program. Prior to awards being made, the Division of Loan
Repayment verifies the legitimacy of the loans and the current balances.
We compared reported debt for LRP applicants to that reported by NSF for the pool recent graduates.
Among biological sciences academic doctorates, 50% reported no educational debt at the time of
graduation. Of those with debt, 41% reported cumulative debt under $50,000. The average debt was
about $20,000.23 Among medical school graduates, 17.2% reported no debt, 14.2% reported debt under
$50,000, and 28.8% reported debt under $100,000. The average debt upon graduation was $97,000.24
1.3.8.1 Distribution of educational debt level
Debt levels of a majority of non-awardees are usually not verified. We determined the correlation
between reported and verified debt levels by comparing these amounts for a subset of applicants for
which both sets of amounts were known, namely, the 4,565 new LRP awardees. As shown in Figure 17,
these amounts correlate strongly, with an R2 = 0.907 (inset, Figure 17). We therefore used applicant
reported (but unverified) debt level for subsequent analysis.
Figure 17. Correlation between applicant reported and DLR verified educational debt level of new
LRP applicants, FY03-FY07.
23 National Science Foundation (2008). Science and Engineering Indicators, Appx. Table 2-12. NSF: Arlington, VA. Note that educational debt for psychology PhDs, who make up over 40% of the LRP applicants, is higher. 70% of psychology PhDs graduate with some debt (vs. 50% for biology), and 29% have < $50K in debt (vs. 8% for biology). 24 Association of American Medical Colleges (2008). AAMC Data Book: Medical Schools and Teaching Hospitals by the Numbers, Table E4. AAMC: Washington, DC. We used student-reported responses from 2001.
0
200
400
600
800
1000
Nu
mb
er
of
Ne
w L
RP
Ap
plic
ants
an
d A
war
de
es
Educational Debt Level(In thousands)
Reported by Applicant
Verified by DLR
There is a strong
correlation between
reported and verified
debt levels of new LRP
applicants.
NIH Extramural Loan Repayment Program Evaluation 24
We next analyzed the distribution of educational debt level of all 9,820 new LRP applicants (Figure 18).
Sixty three percent reported an educational debt level up to $100,000; 31% reported an educational
debt level between $100,000 and $200,000, and 6% reported an educational debt level greater than
$200,000.
Figure 18. Reported educational debt level of new extramural LRP applicants, FY03-FY07.
1.3.8.2 Distribution of educational debt level and degree type
The distribution of debt level between applicants with different degree types shows striking differences
(Figure 19). The proportion of academic doctorates in a debt range decreases with greater educational
debt, with a corresponding increase in the proportion of physician doctorates. Seventy-nine percent of
applicants with an educational debt less than $20,000 have academic doctoral degrees, while only 13%
have physician doctorates. Similarly, of applicants with between $20,000 and $40,000 of educational
debt, 66% have academic doctorates compared to 24% with physician doctorates. At higher educational
debt levels ($180,000-$200,000), the difference is reversed and is even more striking: 83% of applicants
with high debt have a physician doctorate while only 9% have an academic doctorate. The average debt
reported by academic doctorates was $62,165 and by physician doctorates was $114,042, both
substantially higher than the average debt of recent graduates (see above), an indicator that LRPs are
attracting researchers with high debt.
0
200
400
600
800
1000
1200
1400
1600
1800
Nu
mb
er o
f N
ew L
RP
Ap
plic
ants
Educational Debt Level(reported by applicants, in thousands)
The majority of new
LRP applicants reported
an educational debt of
up to $100,000.
NIH Extramural Loan Repayment Program Evaluation 25
Figure 19. Distribution of reported educational debt level of new extramural LRP applicants with
different degree types, FY03-FY07.
0
200
400
600
800
1000
1200
Nu
mb
er o
f N
ew L
RP
Ap
plic
ants
Educational Debt Level(reported by applicants, in thousands)
Physician Doctorate
Academic Doctorate
Physician Doctorate /Academic Doctorate
Professional ClinicalDoctorate
Allied HealthProfessional
Average Debt for Academic Doctorates:$61,165
Physician Doctorate
Academic Doctorate
Physician Doctorate /Academic Doctorate
Professional ClinicalDoctorate
Allied HealthProfessional
Average Debt for Physician
Doctorates: $114,042
The LRP attracts
researchers with
higher than average educational debt
NIH Extramural Loan Repayment Program Evaluation 26
Summary of Findings
FY03 – FY07 Extramural LRP New Applicant Pool
New Applications to the Extramural LRPs
Since 2003, there have been approximately 1,900 new applications per year to the extramural LRPs. Of the total applications in FY03-FY07, 15% were assigned to NCMHD, 13% were assigned to NCI, 11%
were assigned to both NHLBI and NIMH, and 10% to NICHD. The Clinical Research and Pediatric Research LRPs received the majority of new applications. Overall success rate of new applications across the five LRPs is 46%.
Gender and Race and Ethnicity Identification of New Applicants
More new applications were submitted by women (53%) than men (44%). There was not a significant
difference in success rates between genders. More Blacks and fewer Asians applied in comparison to recent MD and PhD graduating classes. Success
rates did not differ by race or ethnicity.
The majority of applications by African Americans or Blacks and Hispanics were submitted to the
NCMHD LRPs.
Educational Background of New Applicants
The applicant pool was 47% physician doctorates and 42% academic doctorates. There are significant gender differences by degree type. Most male applicants (54%) held physician
doctorates. In contrast, most female applicants (50%) held academic doctorates. Prior NIH research support increased an applicant’s chance of receiving an LRP award.
Career Phase of New Applicants
The LRPs attract early career researchers. 72% of new applicants apply within six years after receiving
their degree, and 90% apply within nine years. Physician doctorates and academic doctorates apply to the LRPs at different stages of their careers.
The majority of academic doctorates (67%) apply within 3 years of receiving their degree, while the
majority of physician doctorates (53%) apply between 4-6 years of receiving their degree. More women applicants (62%) are within 3 years of receiving their qualifying degree than men (35%).
Debt Levels of New Applicants to the Extramural LRP
Among new LRP applicants, the majority (63%) reported an educational debt less than $100,000, 31%
reported a debt level between $100,000 and $200,000, and 6% reported a debt level greater than
$200,000. Debt level differs with the degree type of the applicant. The majority of applicants in the debt ranges
of $0-$20,000 and $20,000-$40,000 have academic doctoral degrees (79% and 66%, respectively). At
the higher debt levels (over $100,000), the majority of applicants have physician doctoral degrees. Academic doctorates report an average debt of $62,165 compared to the average debt of $114,042
reported by physician doctorates. On average, LRP applicants have a higher debt load than recent MD and PhD graduates.
NIH Extramural Loan Repayment Program Evaluation 27
1.4. Renewal Applicants
LRP awardees are allowed to apply for one- or two-year renewals. The same eligibility criteria for new
applicants apply to renewal applicants with the exception of the 20% debt-to-income ratio. During the
FY2003 through FY2007 application cycles, renewals comprised 29% of all LRP applications, and 39% of
all awards. We asked three questions:
1. What is the historical trend of renewal LRP applicants and awardees from FY2003-FY2007?
2. Does degree correlate with the likelihood of applying for a renewal award?
3. Does educational debt level correlate with the likelihood of obtaining a renewal award?
1.4.1. Renewal applications and awards
From FY2003-FY2007, LRP received a total of 4,092 renewal applications and made 2,937 renewal
awards, an overall success rate of 72%. Similar to new LRP applications, the Clinical Research LRP
received a majority of renewal applications (58%). Twenty-four percent of applications were submitted
to the Pediatric Research LRP, 14% to the Health Disparities LRP, 4% to the Clinical Research for
Individuals from Disadvantaged Backgrounds LRP, and 1% to the Contraception and Infertility Research
LRP. A similar distribution was observed for renewal awards made by each LRP (Figure 20). These data
and the success rates for each LRP are tabulated in Table 5.
Figure 20. Distribution of renewal extramural LRP applications, FY03-FY07.
PediatricResearch
(24%)
ClinicalResearch
(58%)
Health Disparities
Research(14%)
Contraceptionand Infertility
Research(1%)
Clinical Research
for Individualsfrom Disadvantaged
Backgrounds
(4%)
The majority of renewal
applications are submitted
to the Clinical Research
and Pediatric Research
LRPs.
NIH Extramural Loan Repayment Program Evaluation 28
Table 5. Number of renewal extramural LRP applications and awards, FY03-FY07.
LRP Renewal
Applications
Percent of Renewal
Applications
Renewal Awards
Percent of Renewal Awards
Success Rate
Clinical Research 2,379 58% 1,730 59% 73%
Clinical Research for Individuals from Disadvantaged Backgrounds
146 4% 106 4% 73%
Contraception and Infertility Research
34 1% 21 1% 62%
Health Disparities Research
569 14% 390 13% 69%
Pediatric Research
964 24% 690 23% 72%
Total 4,092 2,937 72%
The number of renewal applications received and renewal awards made in each fiscal year from FY2003-
FY2007 is shown in Figure 21.
Figure 21. Number of renewal LRP applications and awards in each fiscal year, FY03-FY07.
Of all renewal applications received between FY2003-FY2007, 17% were submitted to NCMHD. NCI
received 11%, NHLBI received 10%, and NIMH and NIAID each received 9% of renewal applications. The
complete distribution of renewal applications and awards for each IC for the combined reporting period
(FY2003-FY2007) is presented in Appendix 1.
35
460
1147
1284
1166
30
390
782
893842
0
200
400
600
800
1000
1200
1400
FY03 FY04 FY05 FY06 FY07
Nu
mb
er o
f R
enew
al L
RP
Ap
plic
atio
ns
and
Aw
ard
s
Applications Awards
LRP renewal applications
stabilized in FY05, with a
current success rate of
72% across all ICs.
NIH Extramural Loan Repayment Program Evaluation 29
1.4.2. Renewal applications and awards, by degree type
Table 6 shows the distribution of degree types for renewal applicants and awardees between FY2003-
FY2007. Similar to new applications (Table 3), 9% percent of renewal applicants had a dual physician
doctorate/academic doctorate; 3% had a professional clinical degree, while <1% were allied health
professionals.
Table 6. Distribution of degree types for renewal extramural LRP applicants and awardees, FY03-FY07.
Degree Type Renewal LRP Applications
% Renewal LRP Applications
Renewal LRP Awards
% Renewal LRP
Awards Success Rate
Physician Doctorate 1927 47% 1429 49% 74%
Professional Clinical Doctorate
116 3% 77 3% 66%
Academic Doctorate 1669 41% 1137 39% 68%
Physician Doctorate / Academic Doctorate
376 9% 293 10% 78%
Allied Health Professional
4 <1% 1 <1% 25%
Total 4,092 2,937
1.4.3. Renewal applications and awards, by educational debt level
We next asked whether educational debt level correlates with the likelihood of obtaining a renewal
award. Table 7 shows the distribution of renewals by debt level. A quarter of initial awardees did not
have a renewal award; 45% had one renewal award, 21% had two renewal awards, and 9.5% had three
or more renewal awards. As debt increases, the likelihood of obtaining multiple renewals increases.
While this seems intuitive, it also indicates that the LRP is enabling awardees with high debt to continue
performing research during their debt repayment.
Table 7. Distribution of renewal awards by initial educational debt for extramural LRP awardees, FY03-
FY07.
Debt Level Number of Initial LRP Awardees
Total Number of
Renewal Awards
% Awardees
with no Renewals
% Awardees
with only 1 Renewal
% Awardees with ≥ 2
Renewals
% Awardees with ≥ 3
Renewals
Less than $10K 22 12 45.5% 54.5% 0% 0%
$10K-$50K 1590 1658 26.6% 48.5% 24.9% 5.1%
$50K-$100K 1455 1790 23.4% 42.8% 33.7% 10.2%
$100K-$150K 982 1173 25.1% 45.2% 29.6% 10.1%
$150K-$200K 380 505 23.4% 40.0% 36.5% 13.9%
Over $200K 209 255 28.2% 37.3% 34.4% 13.9%
NIH Extramural Loan Repayment Program Evaluation 30
Summary of Findings:
FY03 – FY07 Extramural LRP Renewal Applicant Pool
Renewal Applications to the Extramural LRPs
Between FY2003 – FY2007, LRP renewals comprised 29% of extramural LRP applications and 39% of
all awards. Of the total renewal applications submitted in FY2003-FY2007, 17% were submitted to NCMHD, 11%
were submitted to NCI, 10% were submitted to NHLBI, and NIMH and NIAID each received 9%. The Clinical Research and Pediatric Research LRPs received the majority of renewal applications. Overall success rate of renewal applications across LRPs is 72%.
Educational Background of Renewal Applicants
Renewal applicants and awardees were similar to new applicants in terms of degree type, with 47%
physician doctorates, 41% academic doctorates, and 9 % dual physician/academic doctorates.
Debt Levels of Renewal Applicants to the Extramural LRP
A majority of LRP awardees obtain one or two renewal awards. LRP applicants with a higher debt load are more likely to obtain multiple renewal awards.
NIH Extramural Loan Repayment Program Evaluation 31
QUESTION 2 : DOES THE LRP RETAIN PARTICIPANTS IN THE NIH EXTRAMURAL
WORKFORCE?
2.1 Goals
Investigate the effect of participation in the extramural Loan Repayment Programs on the retention of
researchers in the extramural NIH-funded workforce.
2.2 Methods
Using priority scores – an NIH-wide quantitative metric – we identified two nearly-identical applicant
groups for comparison: LRP applicants with similar priority scores, of which approximately half were
funded and half were not funded. We employed a regression discontinuity design to examine the impact
of receiving an LRP on subsequent involvement in the extramural NIH-funded workforce for funded and
not funded LRP applicants.
These two cohorts were identified by selecting those applicants in the “funding bubble”, i.e., applicants
with similar priority scores who were equally likely to be funded as not funded. This concept is
illustrated in Figure 22, showing the applicants that were selected for this analysis.
Figure 22. Identifying applicants on the “Funding Bubble”. Applicants were identified for each FY
between 2003-2006 for each IC. A total of 513 applicants were identified in this manner.
The Priority Score “bubble”
represents a group of applicants
with similar priority scores that
are equally likely to be funded.
NIH Extramural Loan Repayment Program Evaluation 32
We identified the funding bubble for each IC between fiscal years 2003 and 2006. These bubbles were
combined to form funded and not funded cohorts, then we applied regression discontinuity analysis to
examine differences between these matched cohorts. Additional details and methodology concerning
the construction of this matched sample are presented in Appendix 3.
Subsequent involvement in the extramural NIH research community was assessed by linking applicants
in the study cohort to person records in the NIH grants database, Information for Management,
Planning, Analysis, and Coordination (IMPAC II).25 Details about the linkages between LRP applicants and
IMPAC II profile records are provided in Appendix 3. For each applicant in the study cohort, IMPAC II
was queried for Research Project Grant (RPG) applications and awards, Fellowship and Training Grant
applications and awards, and participation in a research project in a role other than PI.26 Subsequent
participation is defined as at least one fiscal year following receipt of an LRP award.
Publication counts and citation rates provide an additional way to assess ongoing involvement in
scientific research. The complete set of IMPAC II records for each applicant was used to construct a list
of all known email addresses and name aliases for each applicant. This information was then used to
retrieve publication records from MEDLINE by matching MEDLINE authors to IMPAC II profile(s) for each
applicant. Additional information about the links between IMPAC II and MEDLINE is presented in
Appendix 3.
2.3. Analysis of the Study Cohort
The study cohort included 513 individuals. The cohort was composed of two nearly-identical applicant
groups, about half of whom were funded (n = 259) and half of whom were not funded (n = 254). We first
determined if the study cohort and the full applicant population have similar composition with regard to
gender, degree type, and years since qualifying degree. We also analyzed these demographics to
determine if the funded and not funded cohorts were comparable.
2.3.1. Comparison of the study cohort and the total new applicant pool
To determine if applicants in the study cohort are representative of the total applicant pool, we
analyzed three demographic parameters: gender, degree type, and years since qualifying degree (Table
8). The results show that (i) the degree distribution characteristics of the bubble are similar to the
overall applicant pool, (ii) the distribution of men and women on the bubble is significantly different
from the distribution of all new applicants (p=0.0044), and (iii) there are significantly more individuals in
the bubble who completed their qualifying degree within three years prior to application (p<0.0001).
25 http://era.nih.gov/impacii/index.cfm. 26
RPGs include activity codes starting with R, P, M, S, U (except UC6), G12, D42, DP1, and DP2. K awards were included in this group. All Type 9 grants were excluded. Fellowship and Training grants include activity codes starting with T, F, and G (except G12).
NIH Extramural Loan Repayment Program Evaluation 33
2.3.2. Comparison of funded and not funded applicants in the study cohort
We analyzed the same three parameters defined above to determine the similarity between the funded
and not funded groups within the FY-IC Bubble. As shown in Table 8, these two groups are not
significantly different in all categories listed and are valid comparison groups for all analysis of
subsequent NIH grant participation and publication activity.
Table 8. Demographic distribution of funded and not funded applicants, and comparison to total
new applicant pool.
Parameter Category %
Funded IC-FY Bubble
(n=259)
% Not Funded IC-FY Bubble
(n=254)
% IC-FY Bubble
(n=513)
% All New
Applicants, FY03-FY07 (n=9820)
Gender Male 41% 36% 38% 44% Female 57% 62% 60% 53%
Degree Type
Physician Doctorate 48% 46% 47% 47% Professional Clinical Doctorate 2% 6% 4% 5%
Academic Doctorate 44% 45% 45% 42% Physician Doctorate / Academic Doctorate 5% 2% 3% 6%
Allied Health Professional 1% 1% 1% <1%
Years Since Degree
Up to 3 Years 52% 57% 54% 35% 4-6 Years 34% 35% 35% 37% 7-10 Years 11% 6% 8% 18% 10+ Years 2% 3% 3% 10%
2.4. Subsequent Participation in NIH Grant Programs
Outcome analysis for the study cohort was carried out by determining subsequent participation in NIH
grant programs. The analysis included a comparison of the funded and not funded groups and a more
granular comparison of different demographic parameters.
2.4.1. Composite analysis
To determine if there was an overall difference between the funded and not funded groups, we
performed a composite analysis as shown in Figure 23. The figure indicates the percentage of funded or
not funded applicants in the study cohort who: applied and received a later NIH award (green); applied
NIH Extramural Loan Repayment Program Evaluation 34
but did not receive a later NIH award (orange); had another involvement in an NIH award (blue) 27; or
had an NIH award prior to an LRP award (olive). For those applicants with no other information in IMPAC
II, we checked whether the applicant had served as a key person on an NIH research grant between
FY2005-FY2007 (aqua).28 The figure also shows the percentage of the group for whom no additional
information was available in IMPAC II (grey). It should be noted that each category is mutually exclusive;
if an applicant is accounted for in one category, that applicant is excluded from further consideration in
any other category below it. Mutual exclusivity progresses in an order of decreasing importance of
outcomes, from the top of the list to the bottom.
In comparing funded vs. not funded applicants, we found that funded LRP applicants were more likely to
receive later NIH grant support than non-funded applicants (green bars, p=0.0102).
Figure 23. Composite analysis of outcomes for the funded and not funded groups in the study
cohort.
27 “Other Involvement” means the applicant had a role on an awarded Research project in a fiscal year after their initial LRP
application fiscal year, and one of the following two criteria was met: (1) They were not the PI (generally, this would mean they
had a TA role), OR (2) the project for which they were PI was not a Type “1” application (which suggests they replaced a prior PI
or were named PI on a continuation). Note that we excluded from this category those individuals who met criteria that
qualified them for other outcome categories (applied and received later award or applied but no award). 28 The Enumeration database was used for this purpose; for details on this database see: http://report.nih.gov/UploadDocs/Enumeration_DataReport_20081219.pdf.
Funded LRP applicants were
more likely to receive later NIH
grant support than applicants
who were Not Funded (Green
Bars, 10% versus 4%)
NIH Extramural Loan Repayment Program Evaluation 35
2.4.2. Gender
We refined the analysis of subsequent NIH activity by considering gender as a categorical variable.
Results of this analysis are presented in Figure 24. The analysis of the bubble applicants showed that
women LRP awardees were less likely than men to have applied for an NIH award (p=0.0147)or to have
a subsequent observable role in the NIH extramural program (p=0.474).
Figure 24. Analysis of outcomes, by gender, of subsequent NIH grant participation.
2.4.3. Degree Type
We next analyzed the impact of the applicant’s qualifying degree on subsequent grant involvement. As
shown in Table 8, academic doctorates comprise 45% of the total applicant pool, while physician
doctorates comprise 50% of applicants. These degree types are similarly distributed between funded
and not funded groups within the Bubble. Analysis of subsequent NIH grant involvement is shown in
Figure 25.
Female LRP awardees were
less likely than male awardees
to apply for subsequent NIH
award or hold any role in the
extramural program.
NIH Extramural Loan Repayment Program Evaluation 36
Figure 25. Analysis of outcomes, by degree type, of subsequent NIH grant participation.
Outcome analysis of subsequent NIH grant participation for academic doctorates and physician
doctorates (which include dual physician doctorates- academic doctorates) indicates that significant
differences occur by applicant degree. Participation in the LRP program increased the likelihood that a
physician doctorate applied for a subsequent NIH award (p=0.0106) and was retained in the NIH
extramural workforce (p=0.0470). While funded and non-funded academic doctorate were equally likely
to apply for an NIH grant, participation in the LRP program increased the likelihood that an academic
doctorate was awarded an NIH grant (p=0.0324).
2.4.4. Program
To determine if any difference exists between funded and not funded applicants in different LRPs, we
compared applicants in the Clinical Research LRP and the Pediatric Research LRP. These two programs
account for 78% of the applicants in the FY-IC Bubble (Table 8), and are roughly evenly balanced by
degree type (Figure 10). Outcome analysis of subsequent NIH grant participation is shown in Figure 26.
Physician doctorate LRP awardees
are more likely to apply for a
subsequent NIH award than
academic doctorate LRP awardees.
NIH Extramural Loan Repayment Program Evaluation 37
Figure 26. Analysis of outcomes, by degree type, of subsequent NIH grant participation.
As shown by the difference between the green area at the top of the Clinical Research bars, funded
applicants to the Clinical Research LRP were significantly more likely to be awarded a subsequent NIH
award (p=0.0236). Funded Clinical Research LRP applicants were also more likely to be retained in the
extramural NIH workforce, as shown by the grey bars (p=0.0117). This analysis revealed no significant
differences between funded and not funded Pediatric Research LRP applicants. This may be related to
the difference in gender representation in the two programs (see Figure 9); the small sample size also
affects statistical tests.
Clinical Research LRP awardees were more
likely to receive a later NIH award and to be
retained in the NIH workforce. No outcome
differences were seen for the Pediatric
Research LRP.
NIH Extramural Loan Repayment Program Evaluation 38
Summary of Findings
Analysis of the “Bubble” Cohort: Retention in the NIH Workforce
Comparison of the study cohort to the Total New Applicant Pool
There are significantly more women in the study cohort (60%) than in the total new applicant pool
(53%). The distribution of degree types is similar between the study cohort and the total new applicant
pool. Significantly more individuals in the study cohort completed their qualifying degree within 3 years
prior to applying to the LRP.
Comparison of Funded and Not Funded Applicants
Within the study cohort, there was no significant difference in the distribution of gender, degree
type, and years since degree between Funded and Not Funded applicants.
Subsequent Participation of Applicants in NIH Grant Programs
Funded LRP applicants were more likely to receive later NIH grant support than those who were not
funded. The LRPs are selectively losing women awardees.
The LRPs are increasing the retention of physician doctorates in the NIH extramural workforce, both
through increased grant applications and participation in research activities.
Applicants funded in the Clinical Research LRP were significantly more likely to be awarded a
subsequent NIH award than applicants funded in the Pediatric Research LRP.
NIH Extramural Loan Repayment Program Evaluation 39
2.5. Publications and Citations
Number of publications in scholarly journals is a standard quantitative measure of research productivity.
We analyzed the number of articles published by study cohort applicants starting one fiscal year
following their LRP application. Publications were matched to applicants based on the criteria described
in Appendix 3. Figure 27 shows the distribution of publications by funded and not funded applicants.
Both groups show a similar pattern, with ~70% of the applicants publishing at least one article in a peer-
reviewed journal.
Figure 27. Distribution of the number of articles published by applicants in the study cohort.
One concern with considering publication count as a measure of research output is the significant time
needed to publish results. By considering the publication record for only those LRP applicants in the
study cohort during the FY2003-FY2004 funding cycles, we analyzed the publication output for those
applicants with the longest amount of time to accumulate publications, shown in Figure 28.
Figure 28. Distribution of the number of articles published by applicants in the study cohort for FY03-FY04.
0
20
40
60
80
100
120
140
160
0 1-5 6-10 11-15 16-20 21-25 26-30
Nu
mb
er o
f New
LR
P A
pp
lican
ts in
th
e FY
-IC
Bu
bb
le
Number of Publications
Funded Not Funded
0%
10%
20%
30%
40%
50%
60%
0 1-5 6-10 11-15 16-20 21-25 26-30
Per
cen
t o
f New
LR
P A
pp
lican
ts in
th
e FY
-IC
Bu
bb
le
Number of Publications
FY03-FY04 Funded
FY03-FY04 Not Funded
The majority of both Funded and Not Funded applicants published at least one peer reviewed
journal article.
NIH Extramural Loan Repayment Program Evaluation 40
These analyses suggest a trend toward differences in publication productivity between funded and not
funded applicants. Nineteen-percent of funded LRP applicants had no publications compared to 30% of
not funded applicants. Thirty-one percent of funded applicants had six or more papers, compared to
19% of not funded LRP applicants.
We also analyzed citation counts between the funded and not funded groups. As shown in Figure 29,
there was no significant difference between these groups, either in the number of applicants with no
subsequent citations or the number with five or more citations.
Figure 29. Distribution of the number of citations for articles published by applicants in the study cohort for FY03-FY08.
It would be worthwhile to repeat the publication and citation analyses after more time has elapsed to
determine whether significant differences emerge between the groups.
2.6. Outcomes Analysis for Applicants without Subsequent NIH Records
0
10
20
30
40
50
60
Nu
mb
er o
f N
ew
LR
P A
pp
lica
nts
in
the
FY
-IC
Bu
bb
le
Citations per Publication
Funded Not Funded
No significant difference
between the citation
counts of Funded and Not
Funded applicants was
detected.
Summary of Findings
Analysis of the “Bubble” Cohort: Publications and Citations
Comparison of Publication Productivity Between Funded and Not Funded Applicants
The majority of both the Funded and Not Funded applicants published at least one article in a peer-
reviewed journal. Preliminary evidence indicates that there may be differences in publication productivity between
applicants who are funded or not funded, however the lack of a significant difference between the
two groups may be accounted for by the timing of this evaluation. No significant difference in the number of citations per publication was detected between funded
and not funded applicants. Allowing more time to elapse is necessary to test whether the timeframe of 1-4 years post-award is
too short a time point to use as a measurement of productivity for LRP applicants.
NIH Extramural Loan Repayment Program Evaluation 41
2.6. Outcomes Analysis for Applicants without Subsequent NIH Records
The Bubble analysis showed that 44% of funded LRP applicants and 49% of those not funded had no
post-LRP grant information available in IMPAC II. To learn more about the career outcomes of these
individuals, Internet searches were conducted. Initial searches were conducted using the applicant
name as stored in IMPAC II as the Google search engine query string. Depending on the number of hits
returned, the middle name or initial and degree(s) were added or removed. In some cases, it was
necessary to add the last known institution and/or academic department or field to the search terms, or
to search PubMed for most recent affiliation information. The NIH Employee Directory (NED) and the
professional networking site LinkedIn were also queried.
Confidence in the quality of information retrieved from the Web searches was established by the ability
to find an individual on at least two recently updated websites. Individuals for whom this could be
achieved were scored as a “High Confidence Match.” Individuals for whom information could be
confirmed through only one website were scored as a “Low Confidence Match.” Individuals for whom
information was out-of-date or could not be found were scored as “No Information” (Table 9).
Table 9. Available information on career outcomes for applicants without subsequent NIH Records.
High Confidence Match Low Confidence Match No Information
Funded (n=113) 71.7% 18.6% 9.7% Not Funded (n=124) 77.4% 8.9% 13.7%
2.6.1. Current Affiliation: Using the self-reported information from Web searches, the funded and not
funded groups were compared to the information provided at the time of application for current
affiliation and position. Information regarding additional funding sources was also recorded, if available.
As shown in Figure 30, in both groups, over 50% were most likely to be at the same institution as when
they applied to the LRP.
Figure 30. Current affiliation of applicants without subsequent NIH records.
61.1%29.2%
9.7%
Funded(n = 113)
50.8%33.9%
15.3%
Not Funded(n = 124)
Same Institution
Different Institution
Unknown
The majority of
applicants are at the
same institution as
time of application.
NIH Extramural Loan Repayment Program Evaluation 42
2.6.2. Current Position: Professional titles for both the Funded and Not Funded groups were compared (Figure 31). Due to the breadth of titles, information was categorized as follows:
Tenure Track: Assistant, Associate, or Full Professor
Practitioner: Attending Physician, Section Chief, Private Practice, or Clinical Instructor
Training: Graduate Student, Post-Doctoral Fellow, Resident, or Other Medical Fellow
Other: Administrative University Staff or Non-University Positions
Unknown: Information not available or reported
In both the Funded and Not Funded groups, approximately one-third held Tenure Track positions,
and another one-third held Practitioner or Training positions.
Figure 31. Current position of applicants without subsequent NIH records.
37.2%
22.1%15.0%
11.5%
14.2%
Funded(n = 113)
33.1%
26.6%
8.1%
11.3%21.0%
Not Funded(n = 124)
Tenure Track
Practitioner
Training
Other
Unknown
Approximately 2/3 of
both Funded and Not
Funded Grey-Bar
applicants remained in
the biomedical
workforce.
Summary of Findings
Analysis of the “Bubble” Cohort: Applicants without Subsequent NIH Records Bubble cohort analysis showed that 44% of Funded and 49% of Not Funded applicants had no post-
LRP grant information in the NIH IMPAC II database. Current information for approximately 75% of these applicants was located through Internet
searches. The majority of both the Funded and Not Funded applicants remain at the same institution as at
the time of LRP application. In both the Funded and Not Funded Grey-Bar applicant groups, approximately one-third held
tenure-track positions while another one-third held practitioner or training positions. Thus the
majority of applicants without Subsequent NIH Records appear to have continued medical or
biomedical careers.
NIH Extramural Loan Repayment Program Evaluation 43
QUESTION 3 : HOW DOES THE LRP COMPARE WITH MENTORED K AWARDS?
3.1 Goals
Investigate the similarities and differences between the LRP and other NIH mentored career-
development programs for early-career investigators, with regards to features of the application pool,
retention rate in the NIH workforce, and publication rates.
3.2 Methods
The LRP was evaluated in comparison to a group of mentored career development awards, specifically
the K01, K08, and K23 awards. Application priority scores were used to create mutually-exclusive
comparison cohorts of the top-rated applicants: the top 10% of LRP awardees, the top 10% of K
awardees, and the top 10% of LRP awardees who also received a K award. We derived these cohorts by
identifying awardees with priority scores in the top 10% for each IC and each fiscal year from 2003 to
2006. Figure 32 shows the definitions of the cohorts and the number of awardees in each group.
Because of differences in the typical career stage of LRP and K awardees (see Table 9 below), each
cohort was restricted to awardees who earned their qualifying degree within six years prior to award
receipt.
Figure 32. Definitions of each comparison cohort and the number of awardees in each group.
For each of the top 10% cohorts, we examined subsequent NIH grant participation and publication
productivity as described for the funding bubble cohorts. The only difference for this analysis was that K
awards were not counted as subsequent grants for any of the three groups.
3.3 Demographic Distribution
We analyzed gender and degree type distributions of each cohort to determine if the cohorts
were similar. Demographics for the top 10% group are summarized in Table 10. While the
LRP Cohort
LRP Top-10%n = 157
Funded Type I awardees, FY03-FY06Top 10% priority score, each FY/IC
Within 6 years of degreeL awards only
K Top-10%n = 101
K CohortK01, K08, K23
Funded Type I awardees, FY03-FY06Top 10% priority score, each FY/IC
Within 6 years of degreeK awards only
LRP+K Cohort
LRP+K Top-10%n = 39
Funded Type I awardees, FY03-FY06Top 10% priority score, each FY/IC
Within 6 years of degreeL and K awards
NIH Extramural Loan Repayment Program Evaluation 44
gender distribution is about equal between and across groups, there were substantial
differences in degree distribution and years since degree between the LRP, K, and K+LRP
cohorts. The top 10% LRP awardee group is evenly split between Academic and Physician
Doctorates (including dual degree holders), while the K and K+LRP awardee groups are both 2/3
Physician Doctorates (p<0.01) (these differences are highlighted in bold text in theTable 10).
The top 10% LRP awardee group is fewer years from degree than either the K or K+LRP group.
To increase similarity between groups, we narrowed our examination of outcomes to awardees
within six years of receiving their qualifying degree (Table 11).
Table 10. Demographic distribution of total LRP, K, and LRP+K Top-10% awardee cohorts.
Parameter Category LRP Top 10% (n=215)
K Top 10% (n=218)
LRP+K Top 10% (n=96)
Gender Male 44% 51% 46% Female 51% 49% 47%
Degree Type
Physician Doctorate 31% 32% 50%
Professional Clinical Doctorate 0% 0% 0%
Academic Doctorate 52% 36% 33%
Physician Doctorate / Academic Doctorate 16% 32% 15%
Allied Health Professional 0% 0% 0%
Years Since Degree
Up to 3 Years 34% 15% 24%
4-6 Years 39% 32% 17%
7-10 Years 17% 28% 43%
10+ Years 10% 20% 17%
NIH Extramural Loan Repayment Program Evaluation 45
Table 11. Demographic distribution of the 0-6 YSD LRP, K, and LRP+K Top-10% awardee cohorts
Parameter Category
LRP Top-10%,
0-6 Years Since
Degree
K Top-10%,
0-6 Years Since
Degree
LRP+K Top-
10%,
0-6 Years Since
Degree
Gender Female 50% 51% 72%
Male 45% 49% 26%
Degree Type
Physician Doctorate 26% 16% 15%
Professional Clinical Doctorate
1% 3% 5%
Academic Doctorate 60% 52% 67%
Physician Doctorate / Academic Doctorate
12% 29% 13%
Allied Health Professional
1% 0% 0%
3.3.1. Gender
Figure 33 shows that the proportion of men and women in the LRP Top-10% and K Top-10% awardee
cohorts are similar. The gender distribution in the LRP+K Top-10 awardee cohort is different, with 70%
of this group being women.
Figure 33. Gender distribution of LRP, K, and LRP+K Top-10% awardee cohorts.
3.3.2. Degree type
The distribution of academic doctorates and physician doctorates (which includes dual physician
doctorates-academic doctorates) is shown in Figure 34. Degree distribution is not significantly different
across the cohorts of awardees.
0%
10%
20%
30%
40%
50%
60%
70%
80%
0-6 YSD LRP Top-10% Awardee Cohort
0-6 YSD K Top-10% Awardee Cohort
0-6 YSD LRP+K Top-10% Awardee Cohort
Per
cen
t o
f Aw
ard
ee C
oho
rt
Female Male
The gender distribution
among LRP Top-10% and K
Top-10% was similar, but
more women (70%) were
in the LRP + K group.
NIH Extramural Loan Repayment Program Evaluation 46
Figure 34. Distribution of degree type of the LRP, K, and LRP+K Top-10% awardee cohorts.
3.4. Subsequent Participation in NIH Grant Programs
Outcome analysis of the Top-10% comparison cohorts was carried out by determining subsequent
participation on NIH grants, similar to the analysis for the Bubble applicants. The analysis included a
comparison of each cohort, and a more granular comparison of demographic parameters (gender and
degree type).
3.4.1. Composite Analysis
To determine if there was an overall difference between the LRP, the K, and the LRP+K Top-10%
awardee cohorts, we performed a composite analysis as shown in Figure 35. The figure indicates the
percentage of awardees that applied and received a later NIH award (green); applied but did not receive
a later NIH award (orange); had another involvement in an NIH award (blue); or had an NIH award prior
to an LRP award (olive).29 For those applicants with no other information in IMPAC II, we checked
whether the applicant had served as a key person on an NIH research grant between FY2005-FY2007
(aqua).30 The figure also shows the percentage of the group for whom no additional information was
available in IMPAC II (grey). Each category is mutually exclusive; an applicant accounted for in one
category is excluded from further consideration in any other category below it. Mutual exclusivity
progresses in an order of decreasing importance of outcomes, from the top of the list to the bottom.
29
Note that for the Top 10% cohort analysis, unlike the Bubble, “later NIH award” does not include K awards. 30 The Enumeration database was used for this purpose; for details on this database see: http://report.nih.gov/UploadDocs/Enumeration_DataReport_20081219.pdf.
0%
10%
20%
30%
40%
50%
60%
70%
0-6 YSD LRP Top-10% Awardee Cohort
0-6 YSD K Top-10% Awardee Cohort
0-6 YSD LRP+K Top-10% Awardee Cohort
Pe
rce
nt
of A
war
de
e C
oh
ort
Physician Doctorate Academic Doctorate
There was no
significant difference
in the distribution of
degree types across
the Top 10% awardee
cohorts.
NIH Extramural Loan Repayment Program Evaluation 47
Figure 35. Composite analysis of outcomes for the LRP, K, and LRP+K Top-10% awardee cohorts.
While the LRP has a significant effect on the likelihood of applying for NIH funding (see Figure 23), when
compared to other NIH career development awards, LRP recipients are less likely to apply for
subsequent NIH funding. This is shown by the significant difference in the total green and orange area
between the LRP and K awardees (p = 0.0003) and between the LRP and LRP+K awardees (p = 0.0176).
LRP awardees are also less likely than K awardees to have a subsequent role on NIH awards (difference
in size of grey bars between LRP and K awardees; p = 0.0003). It should be noted that, while many K
awardees hold faculty positions, LRP recipients may or may not be in a position where they can apply for
NIH research project grants.
3.4.2. Gender
We refined the analysis of subsequent involvement by considering the gender of awardees in the LRP
and the K Top-10% awardee cohorts (Figure 36). The LRP+K Top-10% awardee cohort was excluded from
this analysis because of the small sample size.
LRP awardees are less
likely than K awardees to
apply for subsequent NIH
funding or have a role on
NIH awards.
NIH Extramural Loan Repayment Program Evaluation 48
Figure 36. Analysis of outcomes, by gender, of subsequent NIH grant participation of the LRP and
the K Top-10% awardee cohort.
Unlike the Bubble cohort, there were no significant differences in outcome measures between top 10%
LRP cohort men and women. When compared to top 10% K awardees, top 10% women LRP awardees
were less likely to apply for subsequent NIH grant support within 1-4 years after award, as indicated by
the difference in the green and orange areas at the top of the female bars (p = 0.0003). They were also
less likely than the women K awardees to have a subsequent role on an NIH award, shown by the
difference in the size of the grey areas at the bottom of the female bars (p = 0.0013).
3.4.3 Degree type
We analyzed the impact of the awardee’s qualifying degree on subsequent grant involvement. As
above, the LRP+K Top-10% awardee cohort was excluded from this analysis because of the small sample
size. Results of this analysis are shown in Figure 37.
Female LRP awardees were
less likely to apply for
subsequent NIH grant support
or have a subsequent role on
an NIH award.
NIH Extramural Loan Repayment Program Evaluation 49
Figure 37. Analysis of outcomes, by degree type, of subsequent NIH grant participation of the LRP
and the K Top-10% awardee cohort.
While participation in LRPs has a strong retention effect for physician doctorates (see Figure 25), this
effect is even stronger for K awardees. Among physician doctorate awardees, LRP awardees were less
likely than K awardees to apply for or receive subsequent NIH funding. The difference in grant awards is
indicated by the difference in the green areas at the top of the physician doctorate bars (p=0.0141), and
the difference in grant applications is indicated by the difference in orange areas in the physician
doctorate bars (p = 0.0001). Among both academic doctorates and physician doctorates, LRP awardees
were less likely than K awardees to have subsequent involvement on NIH grants (indicated by the size of
the grey bars; physician doctorate, p = 0.0004; academic doctorate p = 0.0018).
3.5. Publications and Citations
We analyzed the number of articles published by awardees in each of the three Top-10% comparison
cohorts, shown in Figure 38. Publications appearing in peer-reviewed journals one fiscal year following
their LRP award were identified and were matched to applicants based on the criteria described in
Appendix 3. All awardee cohorts have been contributing to biomedical research, with about 90% of
awardees in each cohort publishing at least one article in peer-reviewed journals. During the timeframe
under study, more K awardees published six or more papers than LRP awardees (p = 0.0001).
Physician doctorate K
awardees were more
likely than LRP awardees
to apply for or receive
subsequent NIH funding.
NIH Extramural Loan Repayment Program Evaluation 50
Figure 38. Distribution of the number of articles published by the LRP, K, and LRP+K Top-10% awardee cohorts.
An analysis of citations (Figure 39) indicates that in the K group, more awardees have more citations per
publication than in the LRP group (p < 0.01). Using the citation/publication measure normalizes the
results and allows for a comparison between groups with unequal numbers of publications.
Figure 39. Distribution of the number of citations for articles published by the LRP, K, and LRP+K Top-10% awardee
cohorts.
0%
10%
20%
30%
40%
50%
60%
0 1-5 6-10 11-15 16-20 21-25 26-30 31+Per
cen
t o
f To
p-1
0% A
war
dee
Co
ho
rt,
0-6
Yea
rs S
ince
Deg
ree
Number of Publications
LRP K LRP+K
0%
5%
10%
15%
20%
25%
30%
35%
40%
Per
cen
t o
f To
p-1
0% A
war
dee
Co
ho
rt
Citations per Publication
LRP K LRP + K
A greater proportion
of K awardees
published 6 or more
papers than LRP
awardees.
The K awardee group
has more citations per
publication than the
LRP awardee group.
NIH Extramural Loan Repayment Program Evaluation 51
Summary of Findings
Comparison of LRP with Mentored K Awards: Top 10% Cohort Analysis
Similarities and Differences of Outcomes among LRP and the K Award Recipients
Overall, the top 10% of LRP awardees are more likely to be Academic Doctorates and to be within
3 years of degree conferral than either K or LRP+K top 10% awardees. This strongly suggests that
the LRP is attracting applicants that are earlier in their research careers than the K awards
program. The K awardees were more likely to apply for grant support within 1-4 years after award than the
LRP awardees. This follows from the previous finding that the LRP awardees are in a less mature
career stage than K awardees and may need additional time to produce research results capable
of supporting an RPG application. Gender distribution among the top 10% of awardees was similar for LRP and K awardees.
However, female K awardees were more likely than their LRP counterparts to apply for
subsequent NIH grant support and to be retained in the NIH extramural workforce 1-4 years after
award. Similarly, even though the LRP has a clear retention effect on physician doctorates, K awardee
physician doctorates were more likely to apply for and receive subsequent funding, and to be
retained in the NIH extramural workforce 1-4 years after award.
NIH Extramural Loan Repayment Program Evaluation 52
APPENDIX 1 : DETAILS BY IC
A1.1. New and Renewal Applications and Awards, by IC
Twenty four Institutes and Centers (IC) sponsored LRP awards between FY2003 and FY2007. The
complete distribution of new applications and awards at each IC is presented in Table A1.1.
Table A1.1. Distribution (number and percent of total) of new extramural LRP applications and
awards at each IC, FY03-FY07
IC New LRP
Applications
% New LRP
Applications
New LRP Awards
% New LRP Awards
FIC 2 <1% 2 <1%
NCCAM 96 1% 20 <1%
NCI 1,231 13% 508 11%
NCMHD 1,485 15% 757 17%
NCRR 193 2% 137 3%
NEI 124 1% 106 2%
NHGRI 14 <1% 12 <1%
NHLBI 1,103 11% 543 12%
NIA 314 3% 171 4%
NIAAA 159 2% 131 3%
NIAID 652 7% 340 7%
NIAMS 257 3% 117 3%
NIBIB 26 <1% 9 <1%
NICHD 1,016 10% 395 9%
NIDA 338 3% 241 5%
NIDCD 134 1% 112 2%
NIDCR 85 1% 46 1%
NIDDK 829 8% 258 6%
NIEHS 63 1% 38 1%
NIGMS 91 1% 39 1%
NIMH 1,060 11% 396 9%
NINDS 436 4% 155 3%
NINR 65 1% 9 <1%
NLM 47 <1% 23 1%
Total 9,820
4,565
NIH Extramural Loan Repayment Program Evaluation 53
All ICs that sponsored new awards during the FY2003-FY2007 period also sponsored renewal awards.
The complete distribution of renewal applications and awards at each IC is presented in Table A1.2.
Table A1.2. Distribution (number and percent of total) of renewal extramural LRP applications
and awards at each IC, FY03-FY07
IC Renewal
LRP Applications
% Renewal LRP
Applications
Renewal LRP
Awards
% Renewal
LRP Awards
FIC 2 <1% 2 <1%
NCCAM 26 1% 23 1%
NCI 461 11% 307 10%
NCMHD 715 17% 496 17%
NCRR 160 4% 135 5%
NEI 72 2% 64 2%
NHGRI 18 <1% 15 1%
NHLBI 421 10% 325 11%
NIA 166 4% 121 4%
NIAAA 116 3% 98 3%
NIAID 380 9% 292 10%
NIAMS 84 2% 66 2%
NIBIB 4 <1% 2 <1%
NICHD 267 7% 175 6%
NIDA 278 7% 207 7%
NIDCD 54 1% 48 2%
NIDCR 33 1% 25 1%
NIDDK 208 5% 110 4%
NIEHS 38 1% 33 1%
NIGMS 21 1% 16 1%
NIMH 386 9% 260 9%
NINDS 167 4% 108 4%
NINR 6 <1% 5 <1%
NLM 9 <1% 4 <1%
Total 4,092
2,937
A1.2. New Applications and Awards, by IC and LRP
The National Center on Minority Health and Health Disparities (NCMHD) receives all applications
submitted to the Health Disparities Research LRP and the Clinical Research LRP for Individuals from
NIH Extramural Loan Repayment Program Evaluation 54
Disadvantaged Backgrounds. All applications submitted to the Contraception and Infertility Research LRP
are received by the National Institute of Child Health and Human Development (NICHD). Twenty three
ICs (all listed in the above table except NCMHD) support the Clinical Research LRP and the Pediatric
Research LRP. Table A1.3 lists the success rate of new Clinical Research LRP applications for all ICs during
the evaluation period (FY2003 – FY2007). Table A1.4 presents the same information for the Pediatric
Research LRP.
Table A1.3. Success rate of new Clinical Research LRP applications by IC, FY03-FY07.
Success rate of new Clinical Research LRP applications
(Number of ICs) IC (% of new Clinical Research LRP awards)
Up to 50% (4)
NICHD (35%)31, FIC (50%)32, NIEHS (50%)
51%-74% (10)
NIDDK (53%), NINR (56%), NIGMS (59%), NIDCR (63%), NCRR (65%), NIAID (65%), NHLBI (66%), NINDS (66%), NIAMS (71%), NIDCD (71%)
Greater than 75% (10)
NHGRI (75%), NEI (75%), NIMH (79%), NCI (81%), NIDA (85%), NIAAA (85%), NIBIB (89%), NLM (96%), NIA (99%), NCCAM (100%)
Table A1.4. Success rate of new Pediatric Research LRP applications by IC, FY03-FY07.
Success rate of new Pediatric Research LRP applications
(Number of ICs) IC (% of new Pediatric Research LRP awards)
Up to 20% (8)
NCCAM (0%), NIA (1%), NLM (4%), NIBIB (11%), NIAAA (15%), NIDA (15%), NCI (19%)
21%-40% (10)
NIMH (21%), NEI (25%), NHGRI (25%), NIDCD (29%), NIAMS (29%), NINDS (34%), NHLBI (34%), NIAID (35%), NCRR (35%), NIDCR (37%)
41%-50% (6)
NIGMS (41%), NINR (44%), NICHD (46%), NIDDK (47%), FIC (50%), NIEHS (50%)
A1.3. New Awards, by IC and Degree Category
Nineteen ICs do not state a preference for a degree category in awarding LRP contracts. To analyze the
relationship between stated preference and actual awards, we categorized the ICs into three groups
based on the proportion of physician doctorates and academic doctorates receiving new awards (Table
A1.5).
31 NICHD is the only IC that funds the Contraception and Infertility Research LRP. 32 Only two new awards made during the reporting period: one new award in FY04, and one new award in FY05.
NIH Extramural Loan Repayment Program Evaluation 55
Table A1.5. Percent of new extramural LRP awards to physician doctorates and academic
doctorates by IC, FY03-FY07.
IC Category (Number of ICs)
IC (% of new LRP awards with certain degree)
ICs where physician doctorates received >65% of new LRP awards
(6)
NIAMS (71%), NLM (78%), NCI (79%), NCRR (83%), NHLBI (84%), NIDDK (87%)
ICs where academic doctorates received >65% of new LRP awards
(7)
NIMH (68%), NCMHD (69%), NIDCD (72%), NHGRI (75%), NIDA (78%), NIAAA (82%), NINR (100%)
ICs where a single degree type did not predominate the new LRP
awardee pool (6)
FIC, NCCAM, NEI, NIA, NIDCR, NIEHS
This analysis shows that of the 19 ICs that did not state a preference for a certain degree category, six
(FIC, NCCAM, NEI, NIA, NIDCR, and NIEHS) did not have a single degree category predominate the
awardee pool. Seven showed a preference for awarding academic doctorates and 6 showed a
preference for awarding physician doctorates.
NIH Extramural Loan Repayment Program Evaluation 56
APPENDIX 2 : DEGREE CLASSIFICATIONS
The extramural Loan Repayment Program attracts applicants with a wide variety of degree types.33 We
classified these degrees into five categories (Table A2.1) to facilitate cross-parameter analysis of degree
type with gender, program, IC, years since degree and educational debt level.
Table A2.1. Classification of LRP applicant degrees.
Degree Category Degrees Included
1 Physician Doctorate M.D., D.O.
2 Professional Clinical
Doctorate D.D.S., D.M.D., D.P.M., D.P.T., D.V.M., N.D., O.D.,
Pharm.D., Psy.D., V.M.D.
3 Academic Doctorate* Ph.D., D.C., D.N.Sc., D.P.H., Dr.P.H., D.Sc., D.S.W., Ed.D.,
Eng.D., J.D., Sc.D.
4 Physician Doctorate / Academic Doctorate
Any combination of (1) and (3), e.g., M.D./Ph.D., D.O./Ph.D., M.D./Ed.D.
5 Allied Health Professional
A.B., B.A., B.S., B.S.N., M.A., M.B.A., M.L.S., M.P.A., M.P.H., M.S., M.S.N., R.N.
*Includes dual doctorate combinations from (2) and (3), e.g., D.M.D./Ph.D., Pharm.D./Sc.D.
Physician doctorates were distinguished from other clinical professionals; the former group comprised
M.D. and D.O. (and equivalent) degrees, while the latter group included related medical degrees (e.g.,
naturopathy, podiatric medicine, optometry, and psychology), dental, veterinary, and pharmacy
degrees. Academic degrees included doctorates in chiropractic medicine, nursing, public health, social
work, education, engineering, law, science, and the doctorate in philosophy.
Table A2.2 presents the full degree names for those degree types classified in Table A2.1.
33 While a doctoral-level degree is a prerequisite for submitting an LRP application, the Contraception and Infertility Research LRP accepts applications from non-doctoral-level allied health professional degree holders, defined as those individuals with a bachelor’s and/or master’s degree in a relevant discipline.
NIH Extramural Loan Repayment Program Evaluation 57
Table A2.2. Degree definitions and classifications.
Degree Category
Degree Code
Degree
Physician Doctorate
D.O. Doctor of Osteopathy
M.D. Doctor of Medicine
Professional Clinical Doctorate
D.D.S. Doctor of Dental Surgery
D.M.D. Doctor of Medical Dentistry
D.P.M. Doctor of Podiatric Medicine
D.P.T. Doctor of Physical Therapy
D.V.M. Doctor of Veterinary Medicine
N.D. Doctor of Naturopathy
O.D. Doctor of Optometry
Pharm.D. Doctor of Pharmacy
Psy.D. Doctor of Psychology
V.M.D. Doctor of Veterinary Medicine
Academic Doctorate
D.C. Doctor of Chiropractic
D.N.Sc. Doctor of Nursing Science
D.P.H. Doctor of Public Health
Dr.P.H. Doctor of Public Health
D.Sc. Doctor of Science
D.S.W. Doctor of Social Work
Ed.D. Doctor of Education
Eng.D. Doctor of Engineering
J.D. Juris Doctor
Ph.D. Doctor of Philosophy
Sc.D. Doctor of Science
Allied Health Professional
Degree
A.B. Bachelor of Arts
B.A. Bachelor of Arts
B.S. Bachelor of Science
B.S.N. Bachelor of Science In Nursing
M.A. Master of Arts
M.B.A. Master of Business Administration
M.L.S. Master of Library Science
M.P.A. Master of Public Administration
M.P.H. Master of Public Health
M.S. Master of Science
M.S.N. Master of Science In Nursing
R.N. Registered Nurse
NIH Extramural Loan Repayment Program Evaluation 58
APPENDIX 3 : DETAILED OUTCOMES METHODS
Section 2 presented outcome analyses for matched cohorts of funded and not funded LRP applicants.
These analyses required us to identify appropriate applicants, match these applicants with records in
IMPAC II, and identify publications authored by the applicants. All matches were manually reviewed for
accuracy. Similar methods were used for the analyses presented in Section 3.
A3.1. Construction of the Funding Bubble
We developed a method to identify candidates for these cohorts based on the priority score of the
candidate’s application. The two cohorts were identified by selecting those applicants in the “funding
bubble”, which are those applicants with similar priority scores who were equally likely to be funded or
not funded. Since this priority score range depends on fiscal year and IC, we identified candidate
applicants for each FY and IC. The candidates identified were combined into one bubble comprised of
funded and not funded cohorts (513 applicants; 259 funded, 254 unfunded).
The priority score range that defined the funding bubble was determined for each IC (24), for each fiscal
year between FY2003 and FY2006 (4 years). For each IC and each fiscal year, the priority score range
(100 - 500) was divided into equal sized bins and each bin was populated with the number of funded and
not funded applicants, respectively. An ideal bubble bin would contain equal numbers of funded and not
funded applicants; one would expect bins at the low end of the priority score range to be dominantly
populated by funded applicants and bins at the high end of the priority score range to contain mostly
not funded applicants. Since the optimal bin width is dependent on a number of factors, including the
total number of applicants in a particular FY-IC and the particular use of priority score to determine
funding (e.g., gradual vs. a sharp threshold) the process was repeated with different bin widths. The
minimum, and preferred, bin width was a score range of 5 (80 bins over the 100-500 range) and the
maximum allowed bin width was a score range of 30 (13 bins). Six different bin widths were considered
for each FY-IC. After the nearly 200 bin occupancies were determined for a particular FY-IC, the optimal
bubble bin was selected using the following scoring function:
𝐴𝑝𝑝𝑙𝑖𝑐𝑎𝑡𝑖𝑜𝑛𝑠 𝑖𝑛 𝐵𝑖𝑛
𝑅𝑎𝑡𝑖𝑜 × 1 − 0.15 × 𝑆𝑐𝑜𝑟𝑒 𝑅𝑎𝑛𝑔𝑒 − 1
where Applications in Bin is the total number of the funded and not funded applicants in the bin, Ratio is
the ratio of funded and not funded applicants in the bin, and Score Range is the score range of the bin
(ranges from 5 - 30). This function is designed to preferentially select bins (by returning a high score)
that contain a large number of applicants, contain a 1:1 ratio of funded to not funded applicants, and
have a narrow score range.
NIH Extramural Loan Repayment Program Evaluation 59
Bins were not available for every IC and every fiscal year due to the lack of a score range and/or the lack
of appropriate candidate bins. We determined that a minimum of four applicants (two funded and two
not funded) was necessary for a functional funding bubble; bins that did not meet this criterion were
discarded. Of the remainder, the bin with the highest score for each IC in each fiscal year was selected to
represent the funding bubble for that IC in that fiscal year. Individual funding bubbles were combined to
form a composite FY-IC Bubble consisting of 513 applicants. Of these, 259 applicants were funded and
254 applicants were not funded. This process is shown in Figure A3.1.
Figure A3.1. Schematic outline of the steps used for the generation of the FY-IC Bubble.
Create scoring “bins”for all priority scores for each IC and each FY
Score bins
Select top-scoring bin foreach available FY-IC combination
F8
NF8
NIAID, FY05
F11
NF14
NICHD, FY04
F259
NF254
IC-FY Bubble513
Combineindividual bubbles
Priority Scorerange,
e.g. 190-214
F8
NF9
NIMH, FY03
Funded
NotFunded
Total bins: 44,460
NIH Extramural Loan Repayment Program Evaluation 60
A3.2. Determination of Subsequent NIH Activity
For each applicant in the IC-FY Bubble, the NIH grants database, IMPAC II, was queried for subsequent
Research Project Grant (RPG) and Fellowship and Training Grant applications, awards, or non-PI
participation. Subsequent participation is defined as activity at least one fiscal year following receipt of
an LRP award.34
An attempt was made to identify all IMPAC II profile records for each applicant. Although the IMPAC II
architecture is intended to use one record per person, there are a number of ways by which multiple
profiles are created. Therefore, one or more IMPAC II profile records were linked to each LRP applicant
using the following criteria, all of which had to be satisfied:
1. Last Names: The last names in different profiles were matched. If more than one last name
was present in the profile, then one last name had to be a prefix of the other and the
applicant’s social security number or date of birth had to match. Alternatively, both the
applicant’s social security number and date of birth had to match.
2. First Name: There could be no conflict in the first names; either the first names matched
exactly, one was a prefix of the other, only matching initials were provided for both, one of
the first names was missing, or the first name could be found in the middle or last name of
the candidate match. (The last criteria accounts for possible first/middle/last name swaps).
3. Middle Name: There could be no conflict in any provided middle names, using the same
rules as for first name matching.
4. Date of Birth: There could be no significant difference in birth dates; if available, they had to
be within one month of each other.
5. Social Security Number (SSN): There could be no strong difference between applicant social
security numbers; wherever possible, the last four digits had to match exactly.
6. Gender: The gender of the applicant, if available, had to match exactly.
Finally, the last names in the LRP applicant record and the IMPAC II profile record had to match exactly.
Additionally, both records had either an SSN or date of birth available for comparison, or at least one of
the email addresses, phone numbers, fax numbers, or organizations linked to the IMPAC II profile record
had to match that of the applicant.
A3.3. Matching Applicants to MEDLINE Publication Records
The complete set of IMPAC II records for each applicant was used to construct a list of all known email
addresses and name aliases for each applicant. This information was then used to retrieve publication
records from MEDLINE by matching MEDLINE authors to IMPAC II profile(s) for each applicant based on:
1. An exact match between last names or email addresses (or both).
34
RPGs include activity codes starting with R, P, M, S, U (except UC6), G12, D42, DP1, and DP2. K awards were included in this group. All Type 9 grants were excluded. Fellowship and Training grants include activity codes starting with T, F, and G (except G12).
NIH Extramural Loan Repayment Program Evaluation 61
2. The author’s first and middle initials could not be in conflict with that of the IMPAC II profile.
3. The author’s first name was parsed from a MEDLINE field that combines all portions of a
name that are not considered to be the last name. This representation, which is sometimes
limited to an initial, could not conflict with the first name from the IMPAC II profile record.
An initial that matched the first letter of a first name was considered acceptable.
4. The year of the publication had to be greater than that of the fiscal year of the applicant’s
LRP application or award year.
A3.4. Manual Error Checking
All results were manually checked to further exclude false positive hits retrieved by the processes
described above. We used additional information, such as the applicant’s institutional affiliation,
scientific research area, and recurring co-authors to definitively verify the applicant as the author listed
on a publication.