The Convergence of University Rankings and System Benchmarking An Apparent Paradox of...
-
Upload
imani-crutcher -
Category
Documents
-
view
216 -
download
1
Transcript of The Convergence of University Rankings and System Benchmarking An Apparent Paradox of...
The Convergence of University Rankings and System Benchmarking
An Apparent Paradoxof “Rankology”
IREG - Warsaw, 16-17 May 2013 2
Questions
Two approaches: University RankingsSystem Benchmarking
Are they: Complementary? Competing? Consistent?
IREG - Warsaw, 16-17 May 2013 3
Outline
(1) Background: from ranking to benchmarking
(2) Method of investigation
(3) Results
(4) Interpretation and conclusion
IREG - Warsaw, 16-17 May 2013 4
(1) University Rankings
IREG - Warsaw, 16-17 May 2013 5
U Rankings: a Polarizing Exercise
U Rankings: hated/loved, criticized/commended,threatening/stimulating
but proliferating (“here to stay”)
Ph. Albatch’s advice [“Don’t take too much notice of rankings” (UWN, March 23, 2013)]: unlikely to be widely followed
More pitfalls discovered, uncovered, elucidated more attempts to improve methods
IREG - Warsaw, 16-17 May 2013 6
U Rankings: the DiseaseMethodological caveats
Biases: Research, English, STEMComposite indicators: Weighting => ElitismSubjective (reputation) /non transparent
Dangerous use (“misuses”, “abuses”)Universities: (1) Focus on competition with others instead of own
improvement / Affect strategic planning(2) Focus on biased criteria (research)
Policy makers: Focus on a few WCUs instead of whole system
Students: Impact on university selection Overall: Impact on financing
Commercialization (crowded) market
IREG - Warsaw, 16-17 May 2013 7
From Ranking to Benchmarking
“If Ranking is the Disease, Is Benchmarking the Cure?”
(Jamil Salmi, Sunita Kosaraju. Evaluation in Higher Education, Vol. 5 no.1, June 2011)
“Rankings: Neither a Disease nor a Cure”(Ph. Albatch, UWN, 2013)
IREG - Warsaw, 16-17 May 2013 8
(2) System Benchmarking
Resources
Access
Equity
TE SYSTEM
Governance
Quality control
Private ProvidersEconomic, Social
& Technological Environment
IREG - Warsaw, 16-17 May 2013 9
Benchmarking: Objective & Criteria
Objective: assess strength, health and performance of countries' tertiary education systems
Criteria: resources, inputs, governance, outputs and outcomes of the system (access, equity, quality, relevance)
IREG - Warsaw, 16-17 May 2013 10
Benchmarking: Main Initiatives
• SABER: System Approach for Better Education Results (World Bank) Still under construction
• U21 (Universitas 21/ University of Melbourne) Most recent, comprehensive available case See below
• Benchmarking University Governance (World Bank – MENA): Hybrid
• AHELO: Assessment of Higher Education Learning Outcomes (OECD) Still under experimentation
IREG - Warsaw, 16-17 May 2013 11
Hypothesis
Benchmarking developed in reaction to Rankings
Objectives, level of observation and criteria of Benchmarking and Ranking are quite different
==Shouldn’t they yield different results?
IREG - Warsaw, 16-17 May 2013 12
Method (1)
1/ Select 4 of the more popular university rankings: ARWU, THE, QS, WEBOmetrics
2/ Pick the most recent system benchmarking: U21
3/ Compare their results
IREG - Warsaw, 16-17 May 2013 13
Method (2)
Issue: How to compare U and Systems?
Solution: Translate U rankings into Country Rankings
Method: From: number of top universities
to: number of tertiary aged youths in one country potentially served by top universities in that country
(e.g. supply of top universities)
NB: no correlation between the 2 measures
IREG - Warsaw, 16-17 May 2013 14
NB: Number of Top 400 U and Supply of Top 400 U (THE) : Rank)
0
5
10
15
20
25
30
35
40
45
21
13
19
11
16
22
26
10
23
5
China
3
14
24 25
12
8
2
27
6 7
4
31
9
30
India
1820
37
32
36
17
34
29
15
28
Iceland
38 39
3335
Supply (density)
Nbr of top 400 Uni
IREG - Warsaw, 16-17 May 2013 15
Method (3)
Quick look at the 4 leagues selectedThe “sample”: Top 400 universities
THE ARWU QS WEBO
Nbr of countries with at least one top 400 university in each league 41 38 45 41
Nbr of countries with at least one top 400 university found in all 4 leagues (Overlap) 34 34 34 34
Nbr of top 400 universities in the countries with at least one top 400 university found in all 4 leagues (Overlap)
389 394 378 387
The 34Australia JapanAustria MexicoBelgium Netherlands
Brazil New ZealandCanada NorwayChina Poland
Czech Republic PortugalDenmark Russian FederationFinland SingaporeFrance South Africa
Germany South KoreaGreece Spain
Hong Kong SwedenIndia Switzerland
Ireland TaiwanIsrael United KingdomItaly United States
IREG - Warsaw, 16-17 May 2013 17
Comparing the results of the 4 Rankings (1)
Correlation between results of the 4 leagues: (Number of top universities in each country)
United Kingdom
Germany
Canada
Australia
FranceJapan
ChinaIta
ly
Netherland
Sweden
South KoreaSpain
Switzerla
nd
BelgiumIsr
aelBrazil
Hong Kong
Taiwan
Denmark
Austria
Finland
Ireland
Norway
Greece
New Zealand
Poland
Singapore
South Africa
Czech
India
Mexico
Portugal
Russia
0
10
20
30
40
50
60
Webo
THE
ARWU
QS
IREG - Warsaw, 16-17 May 2013 18
Comparing the results of the 4 Rankings (2)
Correlation between results of the 4 leagues:(1) number of top universities in each country
Nbr of Top 400 Universities: R2
THE QS ARWU WEBOTHE 0.98 0.98 0.96QS 0.95 0.93ARWU 0.98WEBO
IREG - Warsaw, 16-17 May 2013 19
Comparing the results of the 4 Rankings (3)
Correlation between results of the 4 leagues:(2) Supply of top universities
Density: R2
THE QS ARWU WEBO
THE 0.96 0.87 0.78
QS 0.83 0.72
ARWU 0.86
WEBO
Supply: Nbr of top U/ TE aged population
The first five countries QS ARWU THE WEBO
1 Finland 16.1 6.9 11.5 9.22 New Zealand 14.5 4.8 14.5 2.43 Switzerland 13.4 11.8 13.4 11.84 Ireland 13.3 8.0 13.3 5.35 Denmark 11.5 9.2 11.5 9.2
The last five countries QS ARWU THE WEBO
30 Poland 0.3 0.5 0.5 0.831 Mexico 0.1 0.1 0.1 0.132 Brazil 0.1 0.2 0.1 0.433 China 0.1 0.1 0.1 0.234 India 0.04 0.01 0.02 0.01
IREG - Warsaw, 16-17 May 2013 21
Benchmarking: “U 21”Method (1)
1/ A priori selection of 48 countries ( +2)
2/ Assessment of countries’ performance based on one overall indicator and 4 “measures”:
(1) Resources (2) Environment(3)Connectivity (4)Output
IREG - Warsaw, 16-17 May 2013 22
Benchmarking: Method (2)
(1) Resources (25%): 5 indicators on expenditures
(2) Environment (25%): 2 indicators on gender balance, 1 indicator on data quality, 3 indicators on policy and regulatory
environment,
1 homegrown index on internal governance
IREG - Warsaw, 16-17 May 2013 23
Benchmarking: Method (3)
(3) Connectivity (10%):
2 indicators on degree of internationalization (students & research)
(4) Output (40%):5 indicators on research,1 indicator on Probability of a person to attend a top 500 university
(*) based on ARWU…1 indicator on enrollment1 indicator on tertiary educated population1 indicator on unemployment among tertiary educated population
IREG - Warsaw, 16-17 May 2013 24
Benchmarking: Links between the 5 measures
OverallResources
(25%)Outputs
(40%)Environment
(25%)Connectivity
(25%)
Overall 0.88 0.93 0.63 0.59
Resources 0.75 0.39 0.43
Outputs 0.50 0.38
Environment 0.40
Connectivity
IREG - Warsaw, 16-17 May 2013 25
Comparing Results of Rankings and Benchmarking (1a)
Countries Overlap between UR and SB:
U21 & THE: 37 common countriesU21 & QS: 40 common countriesU21 & ARWU: 37 common countriesU21 & WEBO: 41 common countries
Essentially same pool of countries
IREG - Warsaw, 16-17 May 2013 26
Comparing Results of Rankings and Benchmarking (1b)
Not in U21Not in one (or more) Ranking
Colombia ArgentinaEstonia BulgariaIceland ChileLebanon CroatiaOman HungaryPhilippines IndonesiaSaudi Arabia IranUAE Malaysia
RomaniaSlovakiaSloveniaThailandTurkeyUkraine
IREG - Warsaw, 16-17 May 2013 27
Comparing Results of Rankings and Benchmarking (2)
U21 THE QS ARWU WEBO
Overall 0.74 0.73 0.76 0.76Ressources 0.69 0.69 0.78 0.77Outputs 0.55 0.57 0.63 0.62Environmment 0.50 0.49 0.45 0.38Connectivity 0.73 0.65 0.63 0.58
Correlation between U21 Indicators and Rankings (Supply): R2
IREG - Warsaw, 16-17 May 2013 28
Comparing Results of Rankings and Benchmarking (3)
U21 (Overall) and THE Rankings (R2= 0.74)
United States
Sweden
Canada
Finland
Denmark
Switzerla
nd
Norway
Australia
Netherland
United Kingdom
Singapore
Austria
Belgium
New Zealand
France
Ireland
Germany
Hong KongIsr
aelJapan
Taiwan
South Korea
Portugal
Spain
Czech
Rep
Poland
Greece Italy
RussiaChina
Brazil
Thailand
Iran
Mexico
Turkey
South Africa
India0
20
40
60
80
100
120
140
160
USA
Sweden
Switzerland New Zealand
France
Ireland
Hong KongU21
Supply
Logarithmic (Supply)
IREG - Warsaw, 16-17 May 2013 29
Comparing Results of Rankings and Benchmarking (4)
U21 (Resources) & ARWU (Supply): R2 = 0.78
U21 (Resources) ARWU (Supply)
U21 (Resources) ARWU (Supply)
Canada 100 60 South Korea 60 16Denmark 97 92 New Zealand 59 48Sweden 94 136 Portugal 58 13USA 92 46 Spain 58 23Norway 92 74 Iran 57 1Finland 89 69 UK 56 60Switzerland 87 118 Japan 53 19Singapore 82 48 Poland 49 5Netherland 80 90 Italy 47 31Austria 75 46 Czech Rep 47 12Ireland 72 80 Russia 43 1Belgium 69 72 Brazil 42 2France 67 32 Mexico 40 1Hong Kong 64 86 Hungary 40 12Israel 64 80 Argentina 39 2Germany 64 46 South Africa 35 3Taiwan 63 16 China 33 1Australia 63 82 India 23 0.1Greece 63 24
IREG - Warsaw, 16-17 May 2013 30
Conclusions /Interpretation
1/ Hypothesis not confirmed:a/ same set of countriesb/ similar results
2/ Two types of explanations:a/ methodologicalb/ structural
IREG - Warsaw, 16-17 May 2013 31
Epilogue
• System Benchmarking ends up ranking countries
• Boundaries between UR and SB are blurred• SB suffers common symptoms with UR • Convergence of the two streams of
“Rankology” not surprising• Benchmarking needs to expand its pool of
countries to become more relevant
IREG - Warsaw, 16-17 May 2013 32
Take Away
IREG - Warsaw, 16-17 May 2013 33
Thank You
SB
UR