[ Resampled Range of Witty Titles]
description
Transcript of [ Resampled Range of Witty Titles]
[RESAMPLED RANGE OF WITTY TITLES]Understanding and Using the NRC Assessment of Doctorate Programs
Lydia Snover, Greg Harris & Scott BargeOffice of the Provost, Institutional Research
Massachusetts Institute of Technology • 2 Feb 2010
Overview1. Background & Context2. Approaches to Ranking3. The NRC Model: A Modified Hybrid4. Presenting & Using the Results
OV
ERV
IEW
2*NB: All figures/data in this presentation are used for illustrative purposes only and do not represent a known institution.
Background & Context
A. History of NRC RankingsB. MIT Data Collection Process
INTR
OD
UC
TION
3
Participating MIT ProgramsAeronautics and Astronautics Computer Science
Applied Biosciences Economics
Applied Mathematics Electrical and Computer EngineeringAstrophysics and Astronomy and Planetary Science Geology and Geochemistry and Geophysics
Atmospheres, Oceans & Climate including MIT/WHOI Oceanography History, Theory and Criticism
Biological Engineering, Health Science and Technology Linguistics
Biology/Biochemistry and Biophysics Material Sciences and Engineering
Biology/Cell and Developmental Mathematics
Biology/Genetics and Genomics Mechanical Engineering
Chemical Engineering Neuroscience
Chemistry Operations Research
Civil and Environmental Engineering Philosophy
Cognitive Science Physics
Computer Engineering Political Science 4
INTR
OD
UC
TION
Section 2
2. Approaches to Ranking
APPR
OA
CH
ES TO R
AN
KIN
G
5
How do we measure program quality?• Use INDICATORS (“countable” information)
to compute a rating– Number of publications– Funded research per faculty member– Etc.,
• Try to quantify more subjective measures through an overall PERCEPTION-BASED RATING– Reputation– “Creative blending of interdisciplinary perspectives”
APPR
OA
CH
ES TO R
AN
KIN
GS
6
Section 3
3. The NRC Approach
THE N
RC
APPR
OA
CH
7
So how does NRC blend the two?The NRC used a modified hybrid of the two basic approaches:•In total, a 4-step process, indicator based, by field•Process results in 2 sets of indicator weights developed through faculty surveys:
– “Bottom up” –importance of indicators– “Top-down” – perception-based ratings of a
sample of programs•Multiple iterations (re-sampling) to model “the variability in ratings by peer raters.” *
THE N
RC
APPR
OA
CH
8*For more information on the rationale for re-sampling, see pp. 14-15 of the NRC Methodology Report
So how does NRC blend the two?STEP 1: Gather raw data from institutions, faculty & external sources on programs. Random University (RU) submitted data for its participating doctoral programs.
THE N
RC
APPR
OA
CH
9
RU PhysicsIndicator Value# publications/fac 1.07
# citations/article 1.17
Median GRE 746
Gender diversity 44%
Time to degree 5.67
…
RU ChemEIndicator Value# publications/fac 1.07
# citations/article 1.17
Median GRE 746
Gender diversity 44%
Time to degree 5.67
…
RU EconomicsIndicator Value# publications/fac 1.07
# citations/article 1.17
Median GRE 746
Gender diversity 44%
Time to degree 5.67
…
NRC
So how does NRC blend the two?STEP 2: Use faculty input to develop weights:– Method 1: Direct prioritization of indicators--
“What characteristics (indicators) are important to program quality in your field?”
THE N
RC
APPR
OA
CH
10
Program Faculty Quality Most Impt Indicator (Mark 4)
Top 2 Indicators
Number of publications per faculty member
Number of citations per publication
Racial/ethnic diversity of the student population
Avg. # of Ph.D.s granted over last 4 years
Gender diversity of program faculty
… … …
Direct WeightsIndicator 1= 0.2
Indicator 2= 0.0
Indicator 3= 0.1
Indicator 4= 0.1
Indicator 5= 0.2
…
Calculations
So how does NRC blend the two?STEP 2: Use faculty input to develop weights:– Method 2: A sample of faculty each rate a sample of 15
programs from which indicator weights are derived.
THE N
RC
APPR
OA
CH
11
Program #2: Yale University Economics
Some Facts About the Program:# of Ph.D.s 2001-2006: _____ Faculty % Female: _____Median Time to Degree: _____ Current Faculty List, etc.
Program #2: Yale University EconomicsProgram #2: Ivy University Economics
Program #1: Land Grant University Economics
Some Facts About the Program:# of Ph.D.s 2001-2006: XX Faculty % Female: YY%Median Time to Degree: Z.Z Current Faculty List, etc.
On a scale from 1 to 3, indicate your familiarity with this program?___ 1 (Little or none)___ 2 (Some)___ 3 (Considerable)
On a scale from 1 to 6, how would you rate this program?___ 1 (Not adequate for doc educ.)___ 2 (Marginal)___ 3 (Adequate___ 4 (Good)___ 5 (Strong)___ 6 (Distinguished)___ 9 (Don’t know well enough)
Regression-based
WeightsInd. 1= 0.3
Ind. 2= 0.04
Ind. 3= 0.2
Ind. 4= 0.15
…
PrincipleComponents& Regression
So how does NRC blend the two?STEP 3: Combine both sets of indicator weights and apply them to the raw data:
THE N
RC
APPR
OA
CH
12
Direct Weights
Ind. 1= 0.3
…
Regression-based
WeightsInd. 1= 0.2
…
Combined Weights
Ind. 1= 0.25
…
DATAIndicatorValue# publications/fac 1.07
# citations/article 1.17
Median GRE 746
Gender diversity 44%
Time to degree 5.67
…
X= Rating
RANKEDLIST
1. Ivy Univ (98)
2. Random Univ (94)
3. Private Univ (91)
4. Land Grant U (88)
5. Univ of State (87)
…
So how does NRC blend the two?STEP 4: Repeat steps 500 times for each field
THE N
RC
APPR
OA
CH
13
A) Randomly draw ½ of faculty
“important characteristics”
surveys
C) Randomly draw ½ of faculty
program rating surveys
B) Calculate “direct” weights
D) Compute “regression-
based” weights
E) Combine weights
F) Repeat (A) – (E) 500 times to develop 500 sets of weights for each field
G) Randomly perturb institutions’ program data 500 times*
H) Use each pair of iterations (1 perturbation of data (G) + 1 set of weights (F)) to rate programs and
prepare 500 ranked lists
I) Toss out the lowest 125 and highest 125 rankings for each
program and present the remaining range of rankings
*For more information on the perturbation of program data, see pp. 50-1 in the NRC Methodology Report
Section 4
4. Presenting & Using the Results
RESU
LTS
14
What are the indicators?Program Faculty Quality Student Characteristics Program Characteristics# of publications per faculty member
Median GRE of entering students
Avg. # Ph.D.s granted in past 5 years
# of citations per faculty member
% students receiving full financial support
% entering students who complete
Receipt of extramural grants for research
% students with portable fellowships
Time to degree
Involvement in interdisciplinary work
Racial/ethnic diversity of student population
Placement of students after grad
Racial/ethnic diversity of the program faculty
Gender diversity of student population
% students with individual work space
Gender diversity of the program faculty
High % of international students
% of health insurance premiums covered
Reception of peers of a faculty member’s work as measured by honors/awards
# of student support activities provided
15
RESU
LTS
What will the results look like?• TABLE 1: Program values for each indicator plus
overall summary statistics for the field
RESU
LTS
16
RU Econ All Economics Programs (n=117)Indicator Value Min 25th
%tile75th
%tileMax Std.
Dev.# publications/fac 1.07 .049 .369 .655 1.257 .246
# citations/article 1.17 .153 .684 1.771 5.485 1.002
Median GRE 746 353 740 790 800 55
% female students 44% 0% 28.6% 42.9% 76.9% 12%
% female faculty 12.5% 0% 10.5% 21.1% 66.7% 9.9%
Time to degree 5.67 3 5 6 8 .8
…
What will the results look like?• TABLE 2: Indicators and indicator weights – one
standard deviation above and below the mean of the 500 weights produced for each indicator through the iterative process (and a locally calculated mean)
RESU
LTS
17
Indicator Minus 1 SD Plus 1 SD Calculated Mean
# publications/fac 0.130 0.134 0.132# citations/article 0.294 0.267 0.2805Median GRE 0.091 0.089 0.09% female students -0.029 -0.043 -0.036% female faculty n.s.* n.s.*
Time to degree -0.026 -0.031 -0.0285…
*n.s. in a cell means the coefficient was not significantly different from 0 at the p=.05 level.
What will the results look like?• TABLE 3: Range of rankings for RU’s Economics
program alongside other programs, overall and dimensional rankings
RESU
LTS
18
Institution Overall Research Activity
Diversity of Acad Environ.
Student Supp/Outcomes
25th %tile
75th %tile
25th 75th 25th 75th 25th 75th
Ivy Univ 30 36 31 32 37 41 28 31
Univ of State 45 54 40 42 42 50 45 46
Random Univ 45 56 38 42 47 51 43 47
Private Univ 48 57 41 42 40 47 45 49
Land Grant U 55 63 59 64 48 50 54 61
Total # of ranked programs = 117
What will the results look like?• TABLE 4: Range of rankings for all RU’s programs
RESU
LTS
19
Program Overall Research Activity
Diversity of Acad Environ.
Student Supp/Out
comes
1995 NRC
Ranking
2009 USNWR Ranking
25th 75th 25th 75th 25th 75th 25th 75th
Linguistics 45 56 … 40 38Material Sciences and Engineering
25 26 24 24
Mathematics 21 23 23 25Mechanical Engineering
32 36 33 33
Neuroscience 34 35 34 35Operations Research
54 56 56 53
Philosophy 43 44 … 44 43
Q&AQ
&A
20
For more information…• The full NRC Methodology Reporthttp://www.nap.edu/catalog.php?record_id=12676
• Helpful NRC Frequently Asked Questions Pagehttp://sites.nationalacademies.org/pga/Resdoc/PGA_051962
RESO
UR
CES
21