Measuring Interdisciplinarity: A Unique Comparison Between the Researcher and Research Proposal Asha...
-
Upload
thomasina-clementine-shelton -
Category
Documents
-
view
222 -
download
7
Transcript of Measuring Interdisciplinarity: A Unique Comparison Between the Researcher and Research Proposal Asha...
Measuring Interdisciplinarity: A Unique Comparison Between the Researcher and
Research Proposal
Asha BalakrishnanVanessa Peña
Bhavya Lal ([email protected]), Task Leader
November 5, 2011American Evaluation Association, 2011
Relevant Task Goal• Federal agency wished to know if their set-
aside potential transformative research (PTR) program was funding interdisciplinary research (IDR)
11/5/2011
Associated Study Questions
• How interdisciplinary is the set-aside program’s research [as compared with traditional programs]? – How interdisciplinary is the body of knowledge on
which the awarded proposal draws?• How interdisciplinary is the funded PI [as
compared with PIs of traditional programs]?– How interdisciplinary is the PI’s prior publication
history?
11/5/2011
Definition of Interdisciplinarity
• Interdisciplinary approaches integrate separate disciplinary data, methods, tools, concepts, and theories in order to create a holistic view or common understanding of a complex issue, question, or problem.
• The critical indicators of interdisciplinarity in research include evidence that the integrative synthesis is different from, and greater than, the sum of its parts.
• Process of knowledge integration is important11/5/2011
C.S. Wagner et al. / Journal of Informetrics 165 (2011) 14–26
Operationalizing Interdisciplinarity – Concept of Diversity Score
• What is the metric?– The “I-score” measures the number of disparate scientific
fields connected by a researcher’s body of work. (Porter, 2007)• How is it measured?
– By using the subject categories of the cited references. A co-citation analysis of all journals has been measured empirically.
• What does it intend to represent?– The I-score intends to measure the interdisciplinarity of the
researcher’s work. – Low I-scores indicate that the researcher draws primarily from
a specialized body of work.
11/5/2011
Evaluative Approach
I-Score metric was employed to test if differences existed between test program awards and comparison awards to determine if:a) Test program is attracting more
interdisciplinary PIsb) Test program is soliciting more
interdisciplinary proposals
11/5/2011
Comparison Group Selection
Selection of Comparison Awards:To assess the difference between the proposals funded by test program and proposals funded through traditional mechanisms, we considered many options and then selected a set of funded proposals from programs as the comparison group.
Selection Methodology:
Identified a set of comparison awards from funded active awards that were:• On the order of total funding of test
program• On the order of total duration of test
program
Randomly selected 44 “comparison” awards
44 “test program” awards
11/5/2011
PROPOSAL INTERDISCIPLINARITY
11/5/2011
Measuring Interdisciplinarity of Proposals
• Activity:– Gathered the cited references from the Treatment
and Comparison Group proposals– Extracted journal titles from references and
matched to Thomson ISI subject categories– Used techniques and tools described in Porter
et.al., 2010* to compute integration scores
*Porter, et al. Practical research proposal and publication profiling. Research Evaluation 19(1): 29-44 (2010).
11/5/2011
Caveats
• We assume that the references cited in the awarded proposal represent the interdisciplinarity of the proposed ideas
• Not all references cited in a proposal were indexed in ISI’s Web of Knowledge database (ranged from 25% to 90% coverage)
• Subject category assignment of journals is a proxy for “discipline”
11/5/2011
Treatment Proposal Integration Scores are Higher than the Comparison Group Proposal Integration Scores
Less than 0.35 0.351 to .45 0.451 to 0.55 0.551 to 0.65 0.651 to 0.75 0.751 to 0.85 Above 0.85
Test 1 1 5 12 21 4 0
Comparison 2 6 14 15 4 3 0
2.5
7.5
12.5
17.5
22.5
Coun
t of P
ropo
sals
11/5/2011
Visual Representation of Average Test and Comparison Group Proposal Integration
ScoreTest Program FY2007-FY2009
Average I-Score = 0.64Comparison Group FY2007-FY2009
Average I-Score = 0.55
Env Sci & Tech
Bus & Mngt
Econ. Polit. & Geo Comp Sci
Eng
Clin Med
Biomed
Cog Sci
Matl Sci
Chem
Inf Dis
Geo
Agri Sci
Hlth & Soc
Ecol Sci
Clin Med
Biomed
Inf Dis
Matl Sci
Geo
Bus & Mngt
Econ. Polit. & Geo Comp Sci
Eng
PhysPhys
Psych
Ecol Sci
Psych
11/5/2011
Distribution of Disciplines – Project Level
I-Score = 0.64
I-Score = 0.68
I-Score = 0.6511/5/2011
RESEARCHER INTERDISCIPLINARITY
11/5/2011
Measuring Principal Investigator Interdisciplinarity
• Activity:– Searched and obtain each of the Treatment and
Comparison Group Principal Investigator’s publication histories (from Jan 1980 to June 2010) in the ISI Web of KnowledgeSM database
– Employed integration score technique based on Porter et.al* to determine integration score of each PI (44 test PIs and 44 Comparison group PIs)
– For PIs with a low number of publications, sensitivity analyses were conducted to ensure conclusions still held
*Porter, et al. Measuring researcher interdisciplinarity. Scientometrics 72(1): 117-147 (2007).11/5/2011
Caveats
• Only journal publications available in ISI’s Web of Knowledge database included
• Subject category assignment of journals is a proxy for “discipline”
• Only measured the PI’s interdisciplinarity, not those of the Co-PI’s or the team
11/5/2011
No Difference Between Treatment PI Integration Scores and Comparison Group PI Integration Scores
Less than 0.35
0.351 to .45 0.451 to 0.55
0.551 to 0.65
0.651 to 0.75
Above 0.75
Test 4 16 11 8 5 0
Compar-ison
4 11 11 13 5 0
1
3
5
7
9
11
13
15
17
Coun
t of P
ropo
sals
11/5/2011
Average Test and Comparison Group PI Integration Score
Comparison Group FY2007-FY2009Average PI I-Score = 0.48
Test Program FY2007-FY2009Average PI I-Score = 0.50
Env Sci & Tech
Bus & Mngt
Econ. Polit. & Geo Comp Sci
Eng
Clin Med
Biomed
Cog Sci
Matl Sci
Comp Sci
Eng
Clin Med
Biomed
Cog Sci
Hlth & Soc
Inf Dis
Phys
Chem
Geo
Env Sci & Tech
Matl Sci
Chem
Econ. Polit. & Geo
Inf Dis
Geo
11/5/2011
Summary of Findings
• I-scores for test group PIs and comparison group PIs are not different
• I-scores from test groups proposal cited references are higher than I-scores from comparison group proposal cited references
Set-aside PTR program is not attracting PIs with a more interdisciplinary background than other programs. However, program is successful attracting researchers to submit more interdisciplinary ideas!
11/5/2011
Methodological Finding: I-score does not tell the complete story
ExamplePI Maps
ExampleProposal Maps
I-Score = 0.41I-Score = 0.64 I-Score = 0.73
I-Score = 0.42I-Score = 0.72 I-Score = 0.54
11/5/2011
Caveats: Study Based on Many Assumptions
• I-score is a good measure of interdisciplinarity– Little is known about how subject categories are
generated and how static they are over time– Journal coverage in databases is incomplete
• Proposal cited references reflect knowledge integration– Rhetorical vs. reward citations
11/5/2011
Future Questions of Interest
• How interdisciplinary was the team (integrate all PI publications into one set and determine i-score)– Are younger PIs more interdisciplinary?
• Testing on larger population of awards – What is the relationship between PI and proposal I-scores?
• Exploratory look found none in the test group, and significantly positive for comparison group
– What is the relationship between Proposal Integration Score and PTR Score?• Exploratory look found none
11/5/2011