league tables as policy league tables as policy instruments: instruments:

the political economy of the political economy of accountability in tertiary accountability in tertiary

educationeducation

Jamil Salmi and Jamil Salmi and Alenoush SaroyanAlenoush Saroyan

CIEP, 18-20 June 2006CIEP, 18-20 June 2006

Lexus-Nexus index on Lexus-Nexus index on rankingsrankings

1981-85

1986-90

1991-95

1996-00

2001-06

Asia/Pacific: 0 0 6 18 27

Middle East/Africa: 0 0 0 1 9

Europe: 0 0 2 24 68

N. & S. America: 0 3 17 23 68

The rankings businessA ranking of league tables

September 10, 2005

outline of the outline of the presentationpresentation

• typology of rankingstypology of rankings

• a world of controversiesa world of controversies

• do rankings measure do rankings measure quality?quality?

• policy implicationspolicy implications

typology of typology of rankings:rankings:

how is it done?how is it done?

• statistical indicators– produced by institutions

– publicly available

• survey of “stakeholders”– employers

– professors

– students

• combination of both

typology of typology of rankings:rankings:

what does it apply what does it apply to?to?

•entire institution or specific program

•gives a global score or measures several dimensions separately

• research or teaching / learning

cluster of indicators cluster of indicators in league tables as in league tables as

measures of qualitymeasures of quality

• beginning characteristicsbeginning characteristics

• learning inputs- stafflearning inputs- staff

• learning inputs- resourceslearning inputs- resources

• learning outputslearning outputs

• final outcomesfinal outcomes

• researchresearch

• reputationreputation

who prepares the who prepares the ranking?ranking?

• A = government agency (Ministry of Higher A = government agency (Ministry of Higher Education, Higher Education Commission, Education, Higher Education Commission, University Grants Council, etc.)University Grants Council, etc.)

• B = independent organization / professional B = independent organization / professional association / universityassociation / university

• C = newspaper / magazine / mediaC = newspaper / magazine / media

• D = accreditation agencyD = accreditation agency

• I = International ranking (IA, IB, IC and I = International ranking (IA, IB, IC and ID linking the international dimension ID linking the international dimension to the type of institution conducting the to the type of institution conducting the ranking)ranking)

ranking systems in 2006

RegionRegionNational and International Ranking National and International Ranking SystemSystem

Eastern Europe and Central Eastern Europe and Central AsiaAsia

Poland (C), Slovakia (B), Russia (B), Ukraine (B)Poland (C), Slovakia (B), Russia (B), Ukraine (B)

East Asia and PacificEast Asia and PacificAustralia (B), China (B, IB), Hong Kong (C), Australia (B), China (B, IB), Hong Kong (C), Japan (C), New Zealand (A), Thailand (A)Japan (C), New Zealand (A), Thailand (A)

Latin America and the Latin America and the CaribbeanCaribbean

Argentina (D)Argentina (D)

Middle East and North Middle East and North AfricaAfrica

North AmericaNorth America Canada (C), United States (C)Canada (C), United States (C)

SSouth Asiaouth Asia India (D), Pakistan (A)India (D), Pakistan (A)

Sub-Saharan AfricaSub-Saharan Africa Nigeria (A)Nigeria (A)

Western EuropeWestern EuropeGermany (B/C), Italy (C), Netherlands (A), Germany (B/C), Italy (C), Netherlands (A), Spain (B), United Kingdom (A, B, IC)Spain (B), United Kingdom (A, B, IC)

outline of the outline of the presentationpresentation

• typology of rankingstypology of rankings

• a world of controversiesa world of controversies

a thin line between a thin line between love and hatelove and hate

a thin line between a thin line between love and hatelove and hate

•disagreement with principle (“Anglo-Saxon” exercise)

a thin line between a thin line between love and hatelove and hate

•disagreement with principle (“Anglo-Saxon” exercise)

•criticism of methodology

a thin line between a thin line between love and hatelove and hate

•disagreement with principle (“Anglo-Saxon” exercise)

•criticism of methodology

•boycotts

boycottsboycotts

boycottsboycotts

•Asiaweek

•US News and World Report

•McLeans

a thin line between a thin line between love and hatelove and hate

•disagreement with principle (“Anglo-Saxon” exercise)

•criticism of methodology

•boycotts (Asiaweek, USA)

•court actions (New Zealand, Holland)

outline of the outline of the presentationpresentation

• typology of rankingstypology of rankings

• a world of controversiesa world of controversies

• do rankings measure do rankings measure quality?quality?

But do they But do they measure quality?measure quality?

– quality a moving target; quality a moving target; Illusive definitionIllusive definition

– mutlidimensional construct; mutlidimensional construct; unidimensional score unidimensional score (subjective weights to (subjective weights to indicators)indicators)

– theoretical justification of theoretical justification of measures and methodologymeasures and methodology

– empirical support for empirical support for indicators indicators

other other shortcomingsshortcomings

• methodological flawsmethodological flaws

– lesser emphasis on outcome lesser emphasis on outcome indicatorsindicators

• few meaningful indicators to few meaningful indicators to assess teaching qualityassess teaching quality

– one size fits all: general one size fits all: general disregard for non-research disregard for non-research universities and non-universities and non-university institutionsuniversity institutions

encourages universities encourages universities to adjust method of data to adjust method of data reportingreporting

and the winner is …and the winner is …

the Anglo-Saxon the Anglo-Saxon factorfactor

the “English” factor in the 2005 rankingsthe “English” factor in the 2005 rankings

THES

60 out of top 100

51 31 3

12 1

3 2 2

SJTU

68 of top 100

53

11

4

USUKCanada

AustraliaN.Z.

HKSingaporeIndia

outline of the outline of the presentationpresentation

• typology of rankingstypology of rankings

• a world of controversiesa world of controversies

• do rankings measure do rankings measure quality?quality?

• policy implicationspolicy implications

usefulness of usefulness of rankings?rankings?

• for the Government?for the Government?

• for the institutions?for the institutions?

• for the public?for the public?

government use of government use of rankingsrankings

• Pakistan casePakistan case

government use of government use of rankingsrankings

• Pakistan casePakistan case

– promoting a culture of promoting a culture of accurate and transparent accurate and transparent informationinformation

government use of government use of rankingsrankings

• Pakistan casePakistan case

– promoting a culture of promoting a culture of accurate and transparent accurate and transparent informationinformation

– promoting a culture of promoting a culture of qualityquality

from the from the viewpoint of viewpoint of institutionsinstitutions

• sensitive to factors that sensitive to factors that affect their rankings affect their rankings (benchmarking)(benchmarking)

• goal setting for strategic goal setting for strategic planning purposesplanning purposes

• forming strategic alliancesforming strategic alliances

applying public applying public pressurepressure

• ProvãoProvão

applying public applying public pressurepressure

• ProvãoProvão

• FranceFrance

applying public applying public pressurepressure

• ProvãoProvão

• FranceFrance

• ColombiaColombia

conclusion:conclusion: divisive or helpful? divisive or helpful?

conclusion:conclusion: divisive or helpful? divisive or helpful?

conclusion:conclusion: divisive or helpful? divisive or helpful?

• rankings are here to stay

• useful for prospective students

• useful in the absence of an established evaluation and/or accreditation system

• useful for benchmarking, goal-setting and self-improvement purposes

• useful to conduct a healthy debate on issues and challenges

• useful to promote a culture of accountability

principles of an principles of an appropriateappropriate

ranking instrument ranking instrument

• compare similar institutionscompare similar institutions

• better to focus on program better to focus on program than on entire institutionthan on entire institution

• better to rank by indicator than better to rank by indicator than wholesale (Germany – wholesale (Germany – Pakistan)Pakistan)

• better to focus on results better to focus on results rather than inputs (labor rather than inputs (labor market outcomes, publications, market outcomes, publications, patents)patents)

• better if used for self-better if used for self-improvement purposesimprovement purposes

• better to advertise results better to advertise results publicly than to keep them publicly than to keep them secretsecret