Mauro Degli Esposti Towards a new model for evaluating the merit of Italian academic scientists and...
-
Upload
michelle-hart -
Category
Documents
-
view
217 -
download
1
Transcript of Mauro Degli Esposti Towards a new model for evaluating the merit of Italian academic scientists and...
Faculty ofLife SciencesFaculty ofLife Sciences
Mauro Degli Esposti
Towards a new model for evaluating the merit of Italianacademic scientists and their institutions
Ferrara, Jan 2011
A new analysis of Italian universities – the start
From The Times October 9, 2008
Whatever happened to the University of Bologna?
The University of Bologna, which received its charter from Frederick I Barbarossa in 1158, was probably the first university founded in the Western world. It became a renowned home to scholars of great distinction in both the sciences and the humanities. The Times Higher Education-QS World University Rankings survey was published this week. It rates the performance of universities according to the quality of their research, the employment rates of graduates and their international profile.
The University of Bologna does not feature in the top 100!
Manchester = 29 – now 26
Harvard = 1, Cambdrige = 3, Oxford = 4, UCL = 7...
M Degli Esposti, presented at the first conference of Via-academy, 21 September 2009 http://via-academy.org/Relazione_Prima_Conferenza_della_VIA_a_Manchester.aspx
MERIT MERIT – – a new approach to evaluation a new approach to evaluation
M Degli Esposti, presented at the first conference of Via-academy, 21 September 2009 http://via-academy.org/Relazione_Prima_Conferenza_della_VIA_a_Manchester.aspx
The members of the Via-academy network who work in UK have been selectedon the basis of merit, but educated in Italy.
The number of academic links of Via-members with Italian institution will be proportional to the relative teaching capacity of these institutions – i.e. they will be an output measure of their quality and relative standing.
Diritto di cronaca
12/10/2009
Un ateneo per scappare
di FLAVIA AMABILE
Italian University Times QS Leiden 250 VIA members
Bologna 1 3 3
Roma 1 ‘La Sapienza’ 2 2 2
Padova 4 4 4
Pisa 5 7 6
Firenze 6 8 8
Napoli ‘Federico II’ 7 5 5
Milano 15 1 1
Kendall tau correlation coefficient
0.31with Leiden
1With Leiden
0.79**with Leidenby Marco Geraci
M Degli Esposti, presented at the first conference of Via-academy, 21 September 2009
by Marco Geraci
Italian University VIA25 Gelmini virtuose (27+)
Padova 1 14
Bologna 2 16
Milano 3 11
Roma 1 ‘La Sapienza’ 4 (42)
Torino 5 8
Pisa 6 22
Firenze 7 27 (later)
Pavia 8 25
Trento 9 1
Napoli ‘Federico II’ 10 (38)
Kendall tau correlation coefficient
0.63**
with Leiden
0.09
with Leiden
VIA25 - cumulative ranking from 10 separate rankings
http://rpc264.cs.man.ac.uk/VIA/index.php/VIA25
October 2009
parere sull'iniziativa 'top italian scientists'
0
10
20
30
40
50
60
70
80
positivo neutro negativo ISI negativo
nu
me
ro
parere sull'iniziativa 'top italian scientists'
0
10
20
30
40
50
60
70
80
positivo neutro negativo ISI negativo
nu
me
ro
M Degli Esposti, presented at the second conference of Via-academy, 18 September 2010
From ‘top Italian scientists’: ranking of Italian institutes
La Stampa 30/8/10 Rank Italian Institution Sum of H-Index N. of TIS
1 Bologna 3019 712 CNR, all 2524 633 Milano 2113 464 Padova 2060 465 Roma 1713 416 Torino 1619 387 INAF, all 1420 388 San Raffaele Milano 1333 289 Firenze 1205 29
10 Ist. Naz. Tumori Milano 1106 2511 Pisa 1049 2312 Ferrara 998 2613 Napoli 959 2314 Mario Negri Miano 931 2015 Genova 891 2016 Roma 2 880 2117 Perugia 804 1918 Pavia 712 1519 Brescia 610 1520 Trieste 585 1321 Modena 489 1222 Normale Pisa 478 1123 Milano Bicocca 436 824 Parma 433 1025 IEO Milano 427 926 Sanita' Roma 423 1027 IFOM Milano 399 828 Bari 388 929 Verona 386 930 SISSA Trieste 374 1031 Siena 374 932 Trento 373 933 Humanitas Milano 336 634 IIT Genova 326 835 Roma 3 326 836 Udine 297 737 Cagliari 250 538 Cattolica Roma 244 639 Bocconi Milano 222 540 Ancona 217 641 Gaslini Genova 186 342 Palermo 182 5
M Degli Esposti, December 31, 2010 – retrieved from: http://www.topitalianscientists.org/Top_italian_scientists_VIA-Academy.aspx
400 TIS
1230 TIS La Repubblica 5/1/2011
Italian biochemists and their h-index
NoteValori di h-index sonostati calcolati cosi’:
X, con Google Scholaradd-standard, con iniziale,ristretto alle aree ‘chemistry’,‘biology’ and ‘medicine’
Y, con ISI Web of Science,In genere ‘all years’ nei settori di bio-medicina (non clinici) e chimica(e.g. medicinal chemistry)
Correlation h-index for biochemists, 54 names
y = 0.97
R2 = 0.973
0
10
20
30
40
50
60
70
80
90
100
0 20 40 60 80 100
h-index Google Scholar add-on
h-in
dex
ISI W
oS
Mauro Degli Esposti La Voce.it, 14-9-10
http://www.topitalianscientists.org/downloads/Manualetto_del_Ricercatore_Provetto_ver1.2.pdf
mean for full professors = 19
Italian scientists: home and abroad
Abroad (50)Home - in Italy (50)
>70 >70
over 70
under 50
female
prize winners
cell biologists/cancer
physics
chemistry & biochemistry
computer sciences
economics
over 70
under 50
female
prize winners
cell biologists/cancer
physics
chemistry & biochemistry
computer sciences
economics
average h-index computer scientitsts
0
10
20
30
40
50
60
70
abroad Italy
aver
age
h-i
nd
ex
TIS Reports 001http://tisreports.com/fullreport/1-Top_100_-_home___abroad.aspx
1118 names
all top updated
ID h index Area1 125 cancer - medicine2 119 cancer - medicine3 109 immunology - medicine4 109 immunology - medicine5 102 immunology - medicine6 99 nephrology - medicine7 98 immunology - medicine8 98 computer, neurosciences, social sciences
9 95 immunology - medicine
ultimo aggiornamento: 09/11/2010
Top Italian Scientists List
Academic Nation Italian Institution noteCarlo Croce USANapoleone Ferrara USA Lasker award 2010Alberto Mantovani Italy Humanitas e MilanoGiorgio Trinchieri France/USAEttore Appella USAGiuseppe Remuzzi Italy Mario NegriAlex Sette USA youngest topTomaso Poggio USA most eclectic
Lorenzo Moretta Italy Gaslini GE
Top Italian Scientists – towards the saturation of names?
83% of the top100 work inBio-medicine!Bio-medicine!
M Degli Esposti, December 31, 2010 – retrieved from: http://www.topitalianscientists.org/Top_italian_scientists_VIA-Academy.aspx
Evolution TIS List
0
200
400
600
800
1000
1200
1400
10-Aug 30-Aug 19-Sep 09-Oct 29-Oct 18-Nov 08-Dec 28-Dec
time (date, 2010)
nu
mb
er
of
na
me
s
Top Italian scientists abroad(total = 363)
0
50
100
150
200
USAUK
Swizerla
nd
Franc
e
Germ
any
Sweden
Spain
Austri
a
Vatica
n
Canad
a
Belgium
Other
nat
ions
nu
mb
er
% total Top Italians
0
20
40
60
80
100
Ital
y
abro
ad
doub
leaf
filia
tion
M Degli Esposti (2010), retrievedJanuary 3, 2011 from: http://www.topitalianscientists.org/Top_italian_scientists_VIA-Academy.aspx
0
0.5
1
1.5
2
2.5
3
3.5
4
haem
atolo
gy
aging
endo
crino
logy
canc
er re
searc
h
neph
rolog
y
haep
atolo
gy
onco
logy
neur
olog
y
inte
rnal m
edici
ne
imm
unolo
gy/a
llerg
y
neur
oscie
nces
micr
obiolog
y
biolo
gy
cell b
iolog
y
bioc
hemist
ry
phar
maco
logy
phys
ics-a
stro
physic
s
nucle
ar p
hysic
s
mat
hem
atica
l phy
sics
mat
hem
atics
econ
omy
chem
istry
vete
rinar
y
no
rmal
ized
ran
kin
g v
s. m
edic
ine
haematology
immunologybiochemistry
cancer research
0
0.5
1
1.5
2
2.5
3
3.5
4
haem
atolo
gy
aging
endo
crino
logy
canc
er re
searc
h
neph
rolog
y
haep
atolo
gy
onco
logy
neur
olog
y
inte
rnal m
edici
ne
imm
unolo
gy/a
llerg
y
neur
oscie
nces
micr
obiolog
y
biolo
gy
cell b
iolog
y
bioc
hemist
ry
phar
maco
logy
phys
ics-a
stro
physic
s
nucle
ar p
hysic
s
mat
hem
atica
l phy
sics
mat
hem
atics
econ
omy
chem
istry
vete
rinar
y
no
rmal
ized
ran
kin
g v
s. m
edic
ine
haematology
immunologybiochemistry
cancer research
Areas of relative excellence for Italy
SCImago. (2007). SJR — SCImago Journal & Country Rank.Retrieved November 10, 2010, from http://www.scimagojr.com
top scienziati per discipline
050
100150200250300350400450500
med
icina
biolog
iafis
ica
astro
fisica
chim
ica
infor
mat
ica
farm
acolo
gia
econ
omia
mat
emat
ica
tech
nolog
y
nu
me
ro d
i s
cie
nzi
ati
M Degli Esposti, December 31, 2010 – retrieved from: http://www.topitalianscientists.org/Top_italian_scientists_VIA-Academy.aspx
Ranking di universita’ statali e non – problems
L’università italiana ha finito anche la fruttadi Andrea AparoIL Fatto Quotidiano, 12 Novembre 2010
Ricercatori italiani uguale fannulloni?di Pietro GrecoScienza in Rete (http://www.scienzainrete.it)04/05/2009
Ranking for Italian universities – various and different
Kendall corr. coefficientVIA25_2 VIA_membtimes_09 iarwu_09 webo_09 leid_orangecampus campus_supersole_09 censis_09 censis_10 social_cap_gategelmini_09 udu_09 MIUR_09
VIA25_1 stampa 0.88** (25) 0.53** (16) 0.42* (19) 0.65** (18) 0.68** (19) 0.63** (18) 0.39* (18) 0.54** (21) -0.12 (17) -0.19 (14) 0.08 (13) 0.65** (22) -0.04 (17) 0.57** (20) 0.59** (20)VIA25_2 0.51** (16) 0.46* (19) 0.7** (18) 0.73** (19) 0.63** (18) 0.42* (18) 0.52** (21) -0.12 (17) -0.25 (14) -0.1 (13) 0.69** (22) -0.03 (17) 0.59** (20) 0.6** (20)VIA_memb 0.43 (12) 0.66** (12) 0.49* (13) 0.79** (13) 0.29 (11) 0.46* (15) -0.36 (10) -0.03 (9) -0.6 (6) 0.49* (13) -0.02 (10) 0.64** (13) 0.71** (14)times_09 0.24 (14) 0.49* (16) 0.31 (16) 0.08 (14) 0.25 (17) 0.02 (12) -0.02 (11) -0.24 (10) 0.46* (18) -0.1 (13) 0.3 (17) 0.35 (16)iarwu_09 0.56** (15) 0.64** (15) 0.33 (13) 0.45* (15) -6.16 -0.28 (9) -0.57 (8) 0.53** (17) -0.24 (11) 0.57** (17) 0.57** (18)webo_09 0.49* (14) 0.12 (13) 0.31 (15) -0.35 (12) -0.22 (9) -0.11 (8) 0.81** (19) -0.4 (13) 0.48** (17) 0.49** (16)leid_orange 0.08 (13) 0.42* (16) -0.64** (12)-0.06 (9) 0.07 (8) 0.53** (16) 0.09 (12) 0.67** (16) 0.75** (17)campus 0.54** (14) 0.21 (14) 0 (12) -0.24 (10) 0.22 (15) 0.28 (15) -0.01 (14) 0.03 (14)campus_super 0.05 (14) 0.03 (13) -0.05 (11) 0.35* (18) -0.05 (13) 0.53** (17) 0.41* (17)sole_09 0.26 (13) 0.13 (11) -0.25 (14) 0.01 (15) -0.3 (12) -0.42 (12)censis_09 0.73** (10) -0.38 (11) 0.09 (12) -0.24 (10) -0.07 (10)censis_10 -0.09 (11) 0.06 (9) -0.2 (10) -0.06 (9)social_cap_gate -0.16 (14) 0.49** (19) 0.53** (18)gelmini_09 -0.15 (12) -0.12 (12)udu_09 0.75** (19)
VIA25-1 s VIA25_2 VIA_membtimes_09 iarwu_09 webo_09 leid_orangecampus campus_supersole_09 censis_09 censis_10 social_cap_gategelmini_09 udu_09 MIUR_09
Marco Geraci e Mauro Degli Esposti, unpublished results, October 2010
Ranking From source nature
Times QS - THE UK QS Agency and Times Mix of 6 parameters
ARWU China Jiao Tong Shanghai University
Mix of 6 parameters (different)
Leiden top 250 Holland CSTS Leiden University Bibliometric (normalized)
Webometrics Spain Cybermetrics lab, CISC Based on data of web visibility
Vision UK - Italy Vision think-tank Mix of 5 parameters (2009)
Campus super Italy Campus magazine Average of 8 rankings
Sole 24 Ore Italy Sole 24 Ore newspaper Mix of national parameters
Repubblica Censis Italy Censis Institute Mix of national parameters
MIUR Italy CIRV, VTR2001-2003 Bibliometric (selected)
Marco Geraci e Mauro Degli Esposti, November 2010, in preparation
In depth analysis of all ranking – recent advances
GRAND VIA - a new meta-ranking combining 5 international and 4 national ones
From co-variance patterns – principal components analysis, PCA (Pearson, 1901)
Ranking of Italian universities: 2 principal components
Principal component analysis of different rankings of Italian universities
International impact and visibility
Na
tio
na
l p
erf
orm
an
ce
-0.5 0.0 0.5 1.0
-0.5
0.0
0.5
1.0
Rankings map
PC1 (39.7%)
PC
2 (
25
.9%
)
Times-QS
iARWU
Leiden OrangeVIA-Member
Webometrics
Campus
Sole
Censis-Repubblica
MIUR
Grand VIA
VIA-TopH
Campus Super
UDU
Gelmini
Top Italian Scientists
Vision
Marco Geraci e Mauro Degli Esposti, November 2010 in preparation
Grand VIA
Ranking from ‘top Italian scientists’ – better than others!
Marco Geraci e Mauro Degli Esposti, November 2010
cumulative correlation with rankings
0
0.5
1
1.5
2
2.5
3
3.5
Times QS ARWU Grand VIA Top HItalians
cu
mu
lati
ve
co
rre
lati
on
Computed over 4 different international rankings
Vision
Evaluation landscape of state Italian universities
Marco Geraci e Mauro Degli Esposti, submitted
Individuals factor map
International impact and visibility (PC1) +-
Nat
ion
al p
erf
orm
ance
(P
C2)
+-
International impact and visibility (PC1) +-
Nat
ion
al p
erf
orm
ance
(P
C2)
+-
Padova
Bologna
Roma 1
Torino
Milano
Politecnico MI
Napoli
Pavia
Trento
Perugia
Parma
average
average
Roma 2
Pisa
Firenze
Landscape of state Italian universities - all
Marco Geraci e Mauro Degli Esposti 2010, submitted…
Ferrara
Ferrara university – ranking among top institutes in Italy: before 15-12
+ 11 new TIS = 26Total TIS = 15
and after systematic analysis 31-12
Rank Italian Institution Sum of H-Index N. of TIS
1 Bologna 3019 712 CNR - global 2524 633 Milano 2113 464 Padova 2060 465 Roma 1713 416 Torino 1619 387 INAF - global 1420 388 San Raffaele Milano 1333 289 Firenze 1205 29
10 Ist Naz Tumori Milano 1106 2511 Pisa 1049 2312 Napoli 959 2313 Mario Negri Milano 931 2014 Genova 891 2015 Roma 2 880 2116 Perugia 804 1917 Pavia 752 1618 Brescia 610 1519 Ferrara 589 1520 Trieste 585 1321 Modena 489 1222 Normale Pisa 478 11
Rank Italian Institution Sum of H-Index N. of TIS
1 Bologna 3019 712 CNR, all 2524 633 Milano 2113 464 Padova 2060 465 Roma 1713 416 Torino 1619 387 INAF, all 1420 388 San Raffaele Milano 1333 289 Firenze 1205 29
10 Istit. Naz. Tumori Milano 1106 2511 Pisa 1049 2312 Ferrara 998 2613 Napoli 959 2314 Mario Negri Miano 931 2015 Genova 891 2016 Roma 2 880 2117 Perugia 804 1918 Pavia 712 1519 Brescia 610 1520 Trieste 585 1321 Modena 489 1222 Normale Pisa 478 11
Ferrara university – ranking among state universities: before
Total TIS = 15
Ranking = 15
M Degli Esposti, December 15, 2010 – retrieved from: http://www.topitalianscientists.org/Top_italian_scientists_VIA-Academy.aspx
+ 12 new TIS = 27
and after systematic analysis
Ranking = 10
M Degli Esposti, January 5, 2011 - retrieved from: http://www.topitalianscientists.org/Top_italian_scientists_VIA-Academy.aspx
Different Area – different impact(publications and citations)
•Normalization? By which criteria? •Which database to use for reference?
M Degli Esposti, September, 2010 – retrieved from: Manueletto del Ricercatore Provetto
NOT ISI WoS!
Scientometrics 73 (2007)
The normalization factorfollows a ‘stretched exponential’;it becomes bigger by increasing the number of published articles
For scientists with less than 50 published articles, the field-specific differences are small
NoteValori di h-index sonostati calcolati cosi’:
X, con Google Scholaradd-standard, con iniziale,ristretto all’area ‘egineering and mathematics’
Y, con ISI Web of Science,all years, ristretto ai settori matematici
Matematici con nomiCompositi (e.g. De Giorgi)sono esclusi - ISI non li trova!
Mauro Degli Esposti – Via-academy.org, 6-9-2010
Correlation h-index mathematicians, 24 names
y = 0.599
R2 = 0.83
0
10
20
30
40
0 10 20 30 40 50 60
h-index Google Scholar add-on
h-i
nd
ex
ISI W
oS
h-index - discrepancies between different databases
NOTA: usando Google Scholar si normalizza gia’ le citazioni dei matematicirispetto a quelle che si computano con ISI WoS.Questo risulta in una normalizzazione automatica di un fattore = 1.67,Simile al fattore di nromalizzatione proposto da Iglesias (2007) ca. = 1.83
Discrepancies between different databases for h-index: the problem with chemistry
Add-on
PorP
ISI WoS
Av
era
ge
h-i
nd
ex
Google Scholar
0
5
10
15
20
25
database
ISI
All chemists of Ferrara university
Correlation h-index for chemists
y = 1.379
R2 = 0.802n = 38
0
10
20
30
40
50
60
70
80
0 20 40 60
PorPIS
I Wo
S
Mauro Degli Esposti, 21-12-2010Correlation factor = 1.38
top areas in Ferrara
0
10
20
30
40
50
60
70
Cardiol
ogia
Patolo
gia
Anato
mia
Uman
a
Farmaco
logia
Istolo
gia ed
Em
briol
ogia
Med
icina
Inter
na
Micr
obiol
ogia
Scienze
Far
mace
utiche
Emato
logia
Anato
mia,
Isto
logia,
Cito
logia
Fisica
Chimica
Biolog
ia Evo
lutiv
a
Fisiolo
gi Um
ana
Scienze
dell
a Ter
ra
Mat
emat
ica
Info
rmat
ica
Average h-index Total citations Number of TIS
Mauro Degli Esposti, 5-1-2011
average h-index Ferrara departments
y = 0.9086xR
2 = 0.9325
0
5
10
15
20
25
30
35
40
0 5 10 15 20 25 30 35 40
PorP (initial)
Ad
d-O
n (
full
na
me
)
Biomedicine: area variation of publication rateP
ub
lica
tion
s p
er
yea
r x
10
-3
Relative disease burden (Log %)
Correlation, RCorrelation, R2 2 = 0.72= 0.72 Neoplasms
6
4
2
%articles that arepatent-associated[
biomedical and clinical scientists in Ferrara
0
5
10
15
20
25
30
Anato
mia
Com
para
ta
Anato
mia
Um
ana
Anato
mia
, Isto
logia
, Cito
logia
Biochi
mica
Biochi
mica
Ese
rcizi
o Fisi
co
Biolog
ia A
mbi
enta
le
Biolog
ia C
el/Gen
etica
Mol
Biolog
ia E
volut
iva
Cardio
logia
Chirur
gia G
ener
ale
Clinica
Chir
urgic
a
Clinica
Der
mat
ologi
ca
Clinica
Med
ica
Clinica
Oste
tricia
Emat
olog
ia
Endoc
rinol
ogia
Farm
acol
ogia
Fisiolo
gia
e Biof
isica
Fisiolo
gia
Gener
ale
Fisiolo
gia
Uman
a
Genet
ica M
edica
Igie
ne
Istolo
gia
ed E
mbr
iolog
ia
Mal
attie
Infe
ttive
Med
icina
Inte
rna
Med
icina
Leg
ale
Med
icina
Nuc
leare
Micr
obiol
ogia
Nutriz
ione
clinic
a
Patolo
gia
Pedia
tria
average h-index
total citations x 10-3
5-1-2011
Ordine alfabetico
Evaluate university institutions and individuals:
The VIA-Ferrara model
A 3-step new evaluation model
Mauro Degli Esposti, 6 December 2010
TIME FRAME: end of 2010, beginning of 2011
Evaluate university institutions and individuals:
The Manchester model
Mauro Degli Esposti, 6 December 2010
Total weightPublications = 40% of RQI
Evaluate university institutions and individuals:
The Manchester model
Mauro Degli Esposti, 6 December 2010
Before RAE 2009:Category - weight Esteems 5Publications 75Environment* 25
PROBLEMS:1. Evaluation of Publication quality byInternal Committee – prone to politics!2. No consideration of international standing and individual’s scientific impact, e.g. h-index
*including research income
Research Quality Indicator = RQI
Evaluate university institutions and individuals:
The VIA-Ferrara model
A 3-step new evaluation model
1. Via-academy •Quantitative evaluation of the scientific impact of all Ferrara scientists based on the systematic analysis of h-index.•Area-specific analysis of h-index using average values of top Italian scientists as reference
2. Ferrara All scientists to provide by e-mail:• a list with their top 4 papers in
the last 2 years;• the number of Ph.D. and Master
students supervised;• the total grant income obtained
from external sources.
TIME FRAME: end of 2010
3. Combine the analysis 1 with data in 2 to define the basic format (like in Manchester) and quantify individual profiles - with 1 at least counting for 50%
Mauro Degli Esposti, 6 December 2010
Questions?
Simple model for auto-evaluation in specific disciplines within a universitySimple model for auto-evaluation in specific disciplines within a university
Set a threshold of intrinsic (local) excellence by applying this formula:
Threshold = average + SD + 1 (in absolute value of h-index)
For example, Ferrara chemists:Threshold = 15.6 + 7.5 + 1 = 24
Italian scientists: home and away
0
10
20
30
40
50
60
70
rettori edirettori
biologi astrofisici economisti
h-i
nd
ex
me
dio
Nota: la barra corrisponde alla deviazione standard.
Media dei valori di h-index per gruppi di scienziati ed accademici italiani che lavorano in Italia. I valori sono simili a quelli degli Italiani che lavorano all’estero.
Mauro Degli Esposti – La Voce.it, 14-9-2010
Valore sogliaper entrare in lista
Top Italian scientists per regione Lombardia
Emilia-Romagna
Lazio
Toscana
Veneto
Piemonte
Friuli
Campania
Liguria
Umbria
Sicilia
Trentino-Alto Adige
Puglia
Marche
Sardegna
Calabria
Abruzzi
Su un totale di 1188 scienziati nella TIS al 15-12-2010,Di cui 834 in Italia – escluso CNR ed INAF
M Degli Esposti, December 15, 2010 – retrieved from: http://www.topitalianscientists.org/Top_italian_scientists_VIA-Academy.aspx