Slide 1
Brian Zuckerman
Presented to
COSEPUP Evaluation of the National Science
Foundation's Experimental Program to Stimulate
Competitive Research (EPSCoR) and Similar Programs
in Other Federal AgenciesDecember 17th, 2012
Results of the Assessment of theDefense Experimental Program to Stimulate
Competitive Research (DEPSCoR)
Slide 2
Study Origin and Timeline
• FY2008 Defense Authorization Act (Public Law 110-181), Section 241, instructed the Secretary of Defense to utilize a defense Federally Funded Research and Development Center (FFRDC) to carry out an assessment of the DEPSCoR program.
• Institute for Defense Analyses was tasked to conduct study February 2008
• Results briefed to Senate and House Armed Services Committee staff (SASC/HASC) November 2008
• Study results cleared for public distribution January 2009
Slide 3
Legislative Mandate for Study
1. Tangible results and progress toward the objectives of the programa. Applications used by, or supportive of, operational usersb. Expanded national research infrastructure
2. Activities consistent with statute
3. Assessment of program elements
4. Assessment of activities of state committees
5. Advantages and disadvantages of institution & state based formulas
6. Mechanisms for improving the management and implementation of the program
Slide 4
Descriptive Statistics
• Between 1993 and 2008, 729 total DEPSCoR awards– 546 individual Principal Investigators (PIs)
» 121 PIs with multiple awards» 42 with three or more» 1 PI with eight awards
– 22% of PIs have won 42% of awards
• 1993-2008 funding of $243 million– Decline after 2000 peak partially reversed in 2008 competition
• 27 states and territories (states) have been eligible for at least one year since program authorized in current form in 1995, plus Missouri (eligible in 1993)
– All eligible states except for the Virgin Islands have won awards
• 19% (5) of eligible states have won 35% of awards– Montana, Alabama, Oklahoma, Nebraska, South Carolina
• 7% (5) of institutions have won 28% of awards– Montana State, U. Nebraska-Lincoln, U. Wyoming, West Virginia U., U.
Arkansas
Slide 5
DEPSCoR Program Objectives
1. To enhance the capabilities of institutions of higher education in eligible states to develop, plan, and execute science and engineering [S&E] research that is competitive under the peer-review systems used for awarding federal research assistance
2. To increase the probability of long-term growth in the competitively awarded financial assistance that institutions of higher education in eligible states receive from the federal government for science and engineering research
Slide 6
The DEPSCoR State Share of DOD S&E Increased
Source: National Science Foundation (NSF) Survey of Federal Science and Engineering Support to Universities, Colleges, and Nonprofit Institutions and the DEPSCoR program database of funding totals by state and year
Note: Graph includes all states ever involved in DEPSCoR program 1995-2008
Note: The dotted lines represent linear regression models applied to the data
6%
8%
10%
12%
14%
16%
18%
1992 1993 1994 1995 1996 1997 1998 1999 2000 2001 2002 2003 2004 2005
Year
Sh
are
of
Do
D U
niv
ersi
ty F
un
din
g t
o D
EP
SC
oR
Sta
tes,
A
dju
stin
g f
or
DE
PS
Co
R P
rog
ram
mat
ic F
un
din
g
R&D (Actual) R&D (Linear Model) Research (Actual) Research (Linear Model)
Slide 7
DEPSCoR Funding As a Percentage of DOD Funding in DEPSCoR States
• DEPSCoR has declined in importance as a source of funding for eligible states since 2000
Sources: NSF Survey of Federal Science and Engineering Support to Universities, Colleges, and Nonprofit Institutions and the DEPSCoR program database of funding totals by state and year
0%
5%
10%
15%
20%
25%
30%
35%
40%
45%
1993 1994 1995 1996 1997 1998 1999 2000 2001 2002 2003 2004 2005
Year
Per
cen
tag
e o
f D
oD
Fu
nd
ing
in
DE
PS
Co
R-E
lig
ible
S
tate
s A
cco
un
ted
fo
r b
y D
EP
SC
oR
Pro
gra
m F
un
ds
Percentage of DoD Total University S&E R&D Percentage of DoD Total University S&E Research
Slide 8
“Success” Varied Among States
0.0%
2.0%
4.0%
6.0%
8.0%
10.0%
12.0%
14.0%
16.0%
18.0%
20.0%
1992 1993 1994 1995 1996 1997 1998 1999 2000 2001 2002 2003 2004 2005
Year
Sh
are
of
Do
D U
niv
ersi
ty S
&E
R&
D
Near or Above Threshold Rising Fast Middle Lagging
• DEPSCoR-eligible states fell into four groups:– 6 states Near or above threshold (AL, HI, LA, MS, NM, SC)– 9 states Rising fast (AK, ID, KY, ME, MT, NE, NV, ND, SD)– 6 states Middle (AR, DE, KS, OK, RI, TN)– 4 states + 2 territories Lagging (NH, PR, VT, VI, WV, WY)
Sources: NSF Survey of Federal Science and Engineering Support to Universities, Colleges, and Nonprofit Institutions, the IDA DEPSCoR database, and the DEPSCoR program database of funding totals by state and year
Slide 9
• DEPSCoR share of university S&E R&D funding varies substantially by state
• More than 60% for VT, WY
• “Rising fast” states’ (AK, ID, KY, ME, MT, NE, NV, ND, SD) DEPSCoR shares decline 2001-2005 compared with 1993-2000
0
AZ
0CA
0
CO
0
CT
0DC
0
FL
0
GA
0
GU
0IA
0IL
0
IN
0MA
0
MD
0
MI
0
MN
0
NC
0
NJ
0
NY
0OH
0
OR
0
PA
0
TX
0
UT0
VA
0VI
0
WA
0WI
1HI
2
RI
4
NM
5
DE
5TN
6
AK
6
MO
6
NH
12
SD
18
PR
26ME
28
LA
28ND
31
VT
33
ID
35
MS
36NV
37KY
38
AR
38
KS
38WV
41WY
47SC
49
NE
52OK
53AL
54MT
0 DEPSCoRs1-10 DEPSCoRs11-20 DEPSCoRs21-30 DEPSCoRs31+ DEPSCoRs
Sources: NSF Survey of Federal Science and Engineering Support to Universities, Colleges, and Nonprofit Institutions, the IDA DEPSCoR database, and the DEPSCoR program database of funding totals by state and year
Note: Average calculated as total DEPSCoR funds during eligible years divided by total DOD funds during eligible years
Average DEPSCoR Funding as a Fraction of DOD Funding by State
Slide 10
Legislative Mandate for Study
1. Tangible results and progress toward the objectives of the programa. Applications used by, or supportive of, operational usersb. Expanded national research infrastructure
2. Activities consistent with statute
3. Assessment of program elements
4. Assessment of activities of state committees
5. Advantages and disadvantages of institution & state based formulas
6. Mechanisms for improving the management and implementation of the program
Slide 11
Expanded National Research Infrastructure (1)
• Involving new investigators:– Recent (2006-2008) cohorts had about 60% new PIs
– Most Army-funded DEPSCoR awardees (82%) had not previously received funding from Army Research Office (ARO)
– 56% of PIs had been funded by the NSF either previous to or within the same year of their first DEPSCoR award
• Training graduate students and postdoctoral fellows:– ARO and Office of Naval Research (ONR) data suggest that
awards fund about 1 PhD, 1 Master’s degree, 2 postdocs
• Building physical infrastructure: – DEPSCoR awards have supported purchase and maintenance
of equipment but data not collected systematically by services
Slide 12
Expanded National Research Infrastructure (2)
• Leveraging new funding for defense-related research is limited:
– 8% of non-DEPSCoR ARO awardees in DEPSCoR states received a DEPSCoR award before (or in the same year as) their first non-DEPSCoR ARO award
– 4 DEPSCoR awardees (less than 1%) won a DOD Multidisciplinary University Research Initiative (MURI) award after or in the same year as their first DEPSCoR award
» 2 of these investigators received non-DEPSCoR DOD funding before their first DEPSCoR award
• Leveraging other funding:– 63 DEPSCoR awardees (12%) received their first NSF funding
subsequent to their first DEPSCoR funding
Slide 13
Legislative Mandate for Study
1. Tangible results and progress toward the objectives of the programa. Applications used by, or supportive of, operational usersb. Expand national research infrastructure
2. Activities consistent with statute• Activities were found to be consistent with statute
3. Assessment of program elements• Available data on DEPSCoR program activities and outcomes are
insufficient for monitoring and evaluation purposes
4. Assessment of activities of state committees• State committees prioritized proposals that met state infrastructure
development goals and reflected the mission/research needs of DOD• Committee processes varied widely from state to state and limited and
variable data prevented detailed assessment
5. Advantages and disadvantages of institution & state based formulas
6. Mechanisms for improving the management and implementation of the program
Slide 14
Legislative Mandate for Study
1. Tangible results and progress toward the objectives of the programa. Applications used by, or supportive of, operational usersb. Expanded national research infrastructure
2. Activities consistent with statute
3. Assessment of program elements
4. Assessment of activities of state committees
5. Advantages and disadvantages of institution & state based formulas
6. Mechanisms for improving the management and implementation of the program
Slide 15
Approach Taken
• Assessment focused on comparing the current state-based formula with an institution-based criterion based on a maximum threshold for DOD S&E research funding
• As directed in the study legislative mandate, particular emphasis was given to supporting defense missions and expanding the nation's defense research infrastructure
– Since the legislative charge for the assessment does not specify how “expanding the nation’s defense research infrastructure” should be interpreted, the assessment considered advantages and disadvantages using a variety of possible interpretations
• Assessment also considered alternative criteria:– Indicators of state-level S&T capacity (SEI, Milken Institute)– Normalization by state population (Census)
• State-based and institution-based criteria can be combined– PI-level criteria are also possible
Slide 16
Effect of Institution Based Funding Threshold
• 77 institutions in 2008 DEPSCoR-eligible states (360 total) received nonzero research funding from DOD in 2005
– 38 Carnegie “Very High” or “High” research institutions in DEPSCoR states
• $5 million threshold would make 269 institutions eligible– Twelve of the 77 institutions in currently eligible jurisdictions (e.g., University of
Delaware, University of Nevada, Brown, Clemson, Vanderbilt, University of Nebraska) would become ineligible
• Considering only Carnegie “Very High” or “High” research universities, shift would increase number of eligible universities from 38 to 121
Maximum DOD Funding Threshold
All Universities With Non-Zero DOD Research
Funding in 2005
Eligible for the 2008 Competition
Potentially Eligible for the 2009 Competition
No limit 360 77 114
$10 million 316 75 109
$5 million 269 65 90
$3 million 231 55 76
$1 million 157 36 52
Slide 17
Definition of “Expanding National Research Infrastructure”
• If interpreted as increasing equity in funding among states or achieving state-level infrastructure goals, a state-based formula would be advantageous
– Current state-based formula for eligibility harnesses the state EPSCoR committees to coordinate infrastructure and capacity-building at the state level
– If intent is to increase the competitiveness of historically-underrepresented states, eligibility can easily be determined at a state level
• If interpreted as involving new investigators or institutions in defense-related research, an institution-based formula would be advantageous
– Allows targeting of programmatic resources toward investigators at institutions that have not historically built relationships with DOD
– Approach taken by late 1980s/early 1990s DOD Research Initiation Program
– While a state-based approach includes the flexibility to channel DEPSCoR proposals toward historically underrepresented universities or new investigators within an eligible state, the institution-based approach allows greater flexibility to target underrepresented universities and investigators throughout the entire country
• Could not be determined whether state-based or institution-based approach would elicit more qualified applications to support defense missions
– Larger number of eligible institutions implies more proposals, but quality indeterminate
Slide 18
Effect of State Population Normalization
• Green states were eligible in 2008, red states are “graduates”, blue states were never eligible
• Comparing top chart with bottom shows dramatic difference in order
• DEPSCoR graduates among highest per capita recipients
• Several DEPSCoR-eligible states (e.g., AK, DE, ND, RI, MT, SD) above average in funding per capita
$0
$50,000
$100,000
$150,000
$200,000
$250,000
$300,000
$350,000
State
FY
20
05
Do
D U
niv
ers
ity
S&
E R
&D
Fu
nd
ing
(K
$)
$0.00
$10.00
$20.00
$30.00
$40.00
$50.00
$60.00
$70.00
State
FY
20
05
Do
D U
niv
ers
ity
S&
E R
&D
Fu
nd
ing
Pe
r C
ap
ita
’05 DOD S&E R&D Funding, by State
’05 Per capita DOD S&E R&D Funding, by State
Slide 19
Other Capacity Indicators
• Some DEPSCoR-eligible states (e.g., NH, RI, DE, KS) above average in Milken Institute state S&T index
– Some “graduated” states below average
• Some DEPSCoR-eligible states (e.g., DE, KS, NE, NH, RI, VT) in the top or second quartiles for more than half of Science and Engineering Indicators’ seven R&D output measures
0
10
20
30
40
50
60
70
80
90
Milk
en In
dex,
200
8 Sc
ore
Slide 20
Legislative Mandate for Study
1. Tangible results and progress toward the objectives of the programa. Applications used by, or supportive of, operational usersb. Expanded national research infrastructure
2. Activities consistent with statute
3. Assessment of program elements
4. Assessment of activities of state committees
5. Advantages and disadvantages of institution & state based formulas
6. Mechanisms for improving the management and implementation of the program
Slide 21
IDA Recommendations
• DOD should change the current process for review of proposals to focus more heavily on investigators’ future potential to conduct research rather than on their current research capabilities
• DOD program managers should be formally encouraged to serve as mentors and facilitators for DEPSCoR investigators seeking to engage in further defense-related research
• DOD should create data systems that will allow systematic tracking of DEPSCoR activities and outcomes
• Congress should re-examine and consider clarifying ambiguities in the DEPSCoR legislative mandate
• Once the DEPSCoR objectives have been clarified, redesign the program with a strategy for enhancing competitiveness at relevant level (e.g., individual, institution, state)
Slide 22
Some DEPSCOR Legislative Language Is Inconsistent
• Objectives specify that the research institution is the level at which competitiveness is to be enhanced but authorizing legislation also specifies that eligibility for DEPSCoR be determined at the state level
– 2008 change to eligibility criteria leaves ambiguous whether Congress intends DEPSCoR to target competitiveness at the institution or state level
• Objectives specify that goal is to increase probability of long-term growth in the competitively awarded financial assistance that institutions of higher education in eligible states receive from the federal government but eligibility determined based on DOD funding
– Statute does not discuss whether Congress intends DEPSCoR should focus narrowly on [6.1] research that is relevant to DOD missions and priorities although this is how program is run
Slide 23
Recommendations for Legislative Clarification
• To ensure that the program is implemented in a manner that meets current legislative priorities, Congress should clarify whether
– The program is intended to increase competitiveness for federal research funding in general or for particular types of research funding
– The program is intended to fund primarily basic research, primarily applied research, or a combination
– The primary unit at which competitiveness should be enhanced is the institution, state, or other (e.g., individual investigator)
• Eligibility criteria, funded activities, and other program elements should be structured in accordance with the program’s objectives
Slide 24
Competitiveness can be Enhanced at Different Levels
• DEPSCoR supports individual or small-group research projects and can therefore be understood to primarily target capacity-building at the level of the individual
– Though supporting training and purchase of equipment target capacity-building at level of the institution as well
• DEPSCoR also operates at state level (e.g., involvement of EPSCoR committees, state-based eligibility criteria)
• While it might be argued that institutional competitiveness depends on individual competitiveness and state competitiveness depends on institutional competitiveness, these dependencies are neither straightforward nor self-evident
Slide 25
Structure Program to Enhance at the Desired Organizational Level
• Assessment identified other programs that might serve as potential models for DEPSCoR
– State-level: NSF EPSCoR/Research Infrastructure Improvement (RII) Awards
» One per state; integrated into state S&T plan; EPSCoR committee involvement; capacity building rather than research
– Centers: NIH IDeA/Centers of Biomedical Research Excellence (COBRE)
» Large-team research, equipment, mentoring; milestones for transition to support by standard NIH mechanisms
– Investigator-level: DOD Research Initiation Program (RIP) or NIH/IDeA Academic Research Enhancement Award (AREA) Program
» Institution-based eligibility criteria; small research awards to individual investigators; build relationships/expertise to allow for transition to support by standard research mechanisms
Slide 26
Backup Slides
Slide 27
State Eligibility over Time
Jurisdiction 95 96 97 98 99 00 01 02 03 04 05 06 07 08
AL
AK
AR
DE
HI
ID
KS
KY
LA
ME
MS
MT
NE
NV
NH
NM
ND
OK
RI
SC
SD
TN
VT
WV
WY
PR
VI
Fiscal Year
Source: IDA analysis of DEPSCoR BAAs and DOD DEPSCoR press releases
Note: Red cells denote years in which jurisdictions were not eligible, and blue cells denote years in which states were eligible.
Note: Missouri, which was eligible only in the 1993-4 competition, was not included in the table.
Slide 28
No Obvious Correlation
0.0%
2.0%
4.0%
6.0%
8.0%
10.0%
12.0%
14.0%
16.0%
18.0%
20.0%
1992 1993 1994 1995 1996 1997 1998 1999 2000 2001 2002 2003 2004 2005
Year
Sh
are
of
Do
D U
niv
ersi
ty F
un
din
g t
o D
EP
SC
oR
Sta
tes,
Ad
just
ing
fo
r D
EP
SC
oR
Pro
gra
mm
atic
Fu
nd
ing
R&D (Actual) R&D (Linear Model) Research (Actual) Research (Linear Model)$0
$5
$10
$15
$20
$25
1993-4 1995 1996 1997 1998 1999 2000 2001 2002 2003 2004 2005 2006 2007 2008
Competition
Fu
nd
ing
(M
$)
0
10
20
30
40
50
60
70
80
90
Nu
mb
er o
f aw
ard
s
Funding (M$) Number of Awards
Number of DEPSCoR awards and program funding
Linear regression model of DEPSCoR state share of DOD university funding
Slide 29
DEPSCoR Emphasis is on Basic Research
• Research oriented towards developing operationally useful devices or components is not a legislatively-mandated program goal
• DEPSCoR projects mostly fund basic research– Not a statutory requirement– Administered by basic research organizations within Air Force
and Army, plus Office of Naval Research (ONR)» Interviews with DEPSCoR program officers suggest that more
applied research does not perform well in review– DEPSCoR program officers tend to track transitions to 6.2 or
6.3 research, but not further
• Two transitions to operational use were found
Slide 30
Two Transitions to Operational Use Were Found
• Ronald DeVore, University of South Carolina: Wavelet mathematics for image compression for tactical applications
– DeVore and colleagues collaborate with program managers at Naval Air Warfare Center NAWC at China Lake to deliver wavelet-based image processing platform
– Charles Creussere of NAWC implements wavelet-based image processing system for navigation in the Tomahawk Block II program
• Michael Pursley, Clemson University: wireless, mobile, distributed, multimedia communication networks
– Pursley and colleagues working since 1970s with ITT on tactical radio development
– Group used DEPScoR funding to support research that provided better anti-jam communications and greater multiple-access capability
– Research led to the Soldier Level Integrated Communications Environment (SLICE) wide band networking waveforms that have been integrated into the SINCGARS radio
Top Related