Multiparametric characterization of scientometric …Multiparametric characterization of...

28
Multiparametric characterization of scientometric performance profiles assisted by neural networks: a study of Mexican higher education institutions Elio Ateno ´genes Villasen ˜or 1 Ricardo Arencibia-Jorge 2 Humberto Carrillo-Calvet 3 Received: 13 April 2016 / Published online: 11 November 2016 Ó Akade ´miai Kiado ´, Budapest, Hungary 2016 Abstract Development of accurate systems to assess academic research performance is an essential topic in national science agendas around the world. Providing quantitative ele- ments such as scientometric rankings and indicators have contributed to measure prestige and excellence of universities, but more sophisticated computational tools are seldom exploited. We compare the evolution of Mexican scientific production in Scopus and the Web of Science as well as Mexico’s scientific productivity in relation to the growth of the National Researchers System of Mexico is analyzed. As a main analysis tool we introduce an artificial intelligence procedure based on self-organizing neural networks. The neural network technique proves to be a worthy scientometric data mining and visualization tool which automatically carries out multiparametric scientometric characterizations of the production profiles of the 50 most productive Mexican Higher Education Institutions (in Scopus database). With this procedure we automatically identify and visually depict clusters of institutions that share similar bibliometric profiles in bidimensional maps. Four perspectives were represented in scientometric maps: productivity, impact, expected vis- ibility and excellence. Since each cluster of institutions represents a bibliometric pattern of institutional performance, the neural network helps locate various bibliometric profiles of academic production, and the identification of groups of institutions which have similar patterns of performance. Also, scientometric maps allow for the identification of atypical & Humberto Carrillo-Calvet [email protected] Elio Ateno ´genes Villasen ˜or [email protected] Ricardo Arencibia-Jorge [email protected] 1 Center of Research and Innovation in Information and Communication Technologies INFOTEC, Circuito Tecnopolo Sur 112, Mexico, Aguascalientes, Mexico 2 Empresa de Tecnologı ´as de la Informacio ´n, La Habana, Cuba 3 Laboratory of Nonlinear Dynamics, Faculty of Sciences and Center of Complexity Sciences, National Autonomous University of Mexico, Mexico City, D.F., Mexico 123 Scientometrics (2017) 110:77–104 DOI 10.1007/s11192-016-2166-0

Transcript of Multiparametric characterization of scientometric …Multiparametric characterization of...

Page 1: Multiparametric characterization of scientometric …Multiparametric characterization of scientometric performance profiles assisted by neural networks: a study of Mexican higher

Multiparametric characterization of scientometricperformance profiles assisted by neural networks: a studyof Mexican higher education institutions

Elio Atenogenes Villasenor1 • Ricardo Arencibia-Jorge2 •

Humberto Carrillo-Calvet3

Received: 13 April 2016 / Published online: 11 November 2016� Akademiai Kiado, Budapest, Hungary 2016

Abstract Development of accurate systems to assess academic research performance is an

essential topic in national science agendas around the world. Providing quantitative ele-

ments such as scientometric rankings and indicators have contributed to measure prestige

and excellence of universities, but more sophisticated computational tools are seldom

exploited. We compare the evolution of Mexican scientific production in Scopus and the

Web of Science as well as Mexico’s scientific productivity in relation to the growth of the

National Researchers System of Mexico is analyzed. As a main analysis tool we introduce

an artificial intelligence procedure based on self-organizing neural networks. The neural

network technique proves to be a worthy scientometric data mining and visualization tool

which automatically carries out multiparametric scientometric characterizations of the

production profiles of the 50 most productive Mexican Higher Education Institutions (in

Scopus database). With this procedure we automatically identify and visually depict

clusters of institutions that share similar bibliometric profiles in bidimensional maps. Four

perspectives were represented in scientometric maps: productivity, impact, expected vis-

ibility and excellence. Since each cluster of institutions represents a bibliometric pattern of

institutional performance, the neural network helps locate various bibliometric profiles of

academic production, and the identification of groups of institutions which have similar

patterns of performance. Also, scientometric maps allow for the identification of atypical

& Humberto [email protected]

Elio Atenogenes [email protected]

Ricardo [email protected]

1 Center of Research and Innovation in Information and Communication Technologies INFOTEC,Circuito Tecnopolo Sur 112, Mexico, Aguascalientes, Mexico

2 Empresa de Tecnologıas de la Informacion, La Habana, Cuba

3 Laboratory of Nonlinear Dynamics, Faculty of Sciences and Center of Complexity Sciences,National Autonomous University of Mexico, Mexico City, D.F., Mexico

123

Scientometrics (2017) 110:77–104DOI 10.1007/s11192-016-2166-0

Page 2: Multiparametric characterization of scientometric …Multiparametric characterization of scientometric performance profiles assisted by neural networks: a study of Mexican higher

behaviors (outliers) which are difficult to identify with classical tools, since they outstand

not because of a disparate value in just one variable, but due to an uncommon combination

of a set of indicators values.

Keywords Bibliometric rankings � Higher education � Institutional academic assessment �Scientometric indicators � Self-organized neural networks � Scientometric data mining �Mexico

Mathematics Subject Classification 68T10 � 62H30 � 91C20

JEL C630 � I230

Introduction

Higher education plays an important role in the domestic systems of science, technology

and innovation. According to the theoretical framework of the Triple Helix model (Ley-

desdorff and Meyer 2007), the articulated work of academy, industry and government is

fundamental for the development of knowledge-based economies. The global generation of

new institutional and social structures for the production, transfer and application of

knowledge is considered a strategic action in agendas of national science policies around

the world. Measures for these kind of linkages are still in growth, but pressures on research

systems to achieve such goal have been presented in the evaluation initiatives developed by

industrialized and developing nations (Krishna et al. 2002; Kurzydlowski 2003; Statzner

and Resh 2010; Weingart 2005; Zell 2005).

The adoption of policies to improve the efficiency and effectiveness of national higher

education systems and the implementation of more accurate systems for academic research

performance evaluation are essential steps for the development of a functional triple helix.

However, there are other aspects that affect collaboration between universities, industries

and government such as research budget, university localization, radicalness of research,

degree of risk-taking culture and researcher’s publication strategies (Belkhodja and Landry

2007). In this competitive environment, the bibliometric based (and other types of)

rankings of universities developed during the last years emerge as important tools for

benchmarking Higher Education Institutions (HEIs), with the aim of identifying ‘‘excel-

lence’’ in universities and among researchers (Bengoetxea and Buela-Casal 2013).

The introduction of a full scale of bibliometric rankings has become a ‘‘hot topic’’ in the

field of quantitative studies of science and technology, with literature in favor and against. In

the midst of it all, it has been recognized that the politicized use of rankings is an unavoidable

competitive tool, urgently demanded by science administrators (Weingart 2005).

Various authors have deeply criticized methodologies that use bibliometric tools,

because of misuse of their results for purposes they were not designed for, or for proving

incapable to cover essential missions of universities such as teaching quality, knowledge

transfer, international orientation and regional engagement (Billaut et al. 2010; Ioannidis

et al. 2007; Zhao et al. 2009). On the other hand, some authors remark the effectiveness of

the bibliometric approach (Abramo et al. 2013; Aguillo et al. 2010; Belter 2013; Huang

2012), and the correlation of excellence measured by peer-reviewed versus bibliometric

methods (Allen and Heath 2013; Mryglod et al. 2013). Other authors analyze the

78 Scientometrics (2017) 110:77–104

123

Page 3: Multiparametric characterization of scientometric …Multiparametric characterization of scientometric performance profiles assisted by neural networks: a study of Mexican higher

possibility to obtain more accurate rankings, based on improved indicators (Bengoetxea

and Buela-Casal 2013; Benito and Romera 2011; Torres-Salinas et al. 2011).

In this context, the appearance of new tools to measure academic research performance

is unstoppable. Every day, global initiatives using advanced bibliometric methods are

launched. Projects like SciVal (USA), the CTWS Leiden Ranking (The Netherlands), and

the SCImago Institutions Rankings (Spain), offer some of the most relevant rankings, due

to their valuable set of advanced citation-based indicators. However, some countries have

developed their own individual efforts to characterize and evaluate universities and

research institutions (Allen and Heath 2013; Vanclay and Bornmann 2012).

In Latin America, isolated efforts are focused on analyzing productivity and impact in

their scholarly systems. Experiences in Brazil, Argentina, Chile, Cuba and Venezuela have

been reported (Arencibia-Jorge and de Moya-Anegon 2010; Caputo et al. 2012; Helene and

Ribeiro 2011; Krauskopf et al. 2007; Miguel et al. 2010), but there is still a lack of research

on the effectiveness of science systems in Latin America and their contribution to the

research and development process in higher education institutions.

Mexico is not the exception. Some papers have analyzed quality of the Mexican pro-

duction in different research fields such as Physics and Astronomy, Agriculture, Nanos-

ciences and Health Sciences, among others (Collazo-Reyes et al. 2008, 2010; Lena 1997;

Licea de Arenas et al. 2002; Macias-Chapula et al. 2007; Sierra-Flores and Barnard 2009;

Sierra-Flores et al. 2009). Other papers have developed approaches on the Mexican higher

education system (Arvanitis et al. 1996; Licea de Arenas et al. 2000; Luna-Morales 2012).

The National University and some centralized federal agencies have developed biblio-

metric research performance evaluations at macro level. Three of the most prominent

initiatives include: the project Atlas of Mexican Science developed by the National Council

of Science and Technology (CONACyT 2012); the Comparative Study of Mexican

Universities developed by the National Autonomous University of Mexico (UNAM 2012);

and a bibliometric study developed by the SCImago Research Group for the Foro Con-

sultivo Cientıfico y Tecnologico of Mexico (FCCyT 2011).

For the Latin American context (including the Mexican case), three global projects of

rankings offer useful analytical tools: the Web Ranking of World Universities by the

Cybermetric Lab (Spain), which measures web visibility (Aguillo et al. 2005), the SCI-

mago Journal & Country Rank (SJCR), and the SCImago Institutions Rankings (SIR).

These analytical products provide a novel set of indicators that combine the traditional

output-impact approach with indicators of specialization, leadership and research excel-

lence (Gomez-Nunez et al. 2011). The Ibero American Ranking of Universities, one of the

information products of SIR, offers a scientometric view of Latin American institutions

with at least one paper published in a journal indexed by Scopus (SCImago Research

Group 2013). Decision makers and researchers can refer to each edition of this ranking,

published since 2009, to analyze the scientific production of Ibero American higher edu-

cation institutions.

The current work complements previous studies and provides the following new elements

for the analysis of Mexican scientific production: (1) the increase of members in the National

Researchers System (NRS, SNI in Spanish) and its relation to the rise of Mexican output

indexed by WoS and Scopus; (2) the evolution of national and institutional productivity and

the visibility of published papers; (3) the multiparametric characterization of universities

bibliometric performance through the application of Self-Organizing Maps based on neural

networks (Kohonen 2001) and advanced scientometric indicators. The aim is to analyse the

NRS contribution to the scientific performance of Higher Education, with emphasis in the

study of productivity and impact of universities, from a scientometric perspective.

Scientometrics (2017) 110:77–104 79

123

Page 4: Multiparametric characterization of scientometric …Multiparametric characterization of scientometric performance profiles assisted by neural networks: a study of Mexican higher

The paper is organized as follows. In the section ‘‘Materials and Methods’’ we present the

battery of indicators that will be used, and show a diagram depicting the devised procedure.

In section ‘‘Results and Discussion’’, we begin analyzing the evolution of the national

production and its impact, both in WoS and Scopus; and then present a bi-parametric

analysis of the Mexican HEIs’ performance indicators at a mezzo level. The rest of the

section is devoted to application and discussion of the results of the multiparametric char-

acterization of the HEIs’ performance profiles, assisted by SOM neural networks. We end

with a ‘‘Conclusions’’ section, where we summarize the results and contribution of our study.

Materials and methods

Sources and time windows

Our study focuses in two time slices. First, we identified and compared the total scientific

production of Mexico, in Scopus and the Web of Science (WoS), during a sixteen-year

period from 1996 to 2011. In the second part of our study, we analyzed the scientific

production in Scopus of the Mexican HEIs, during a five-year period from 2007 to 2011.

With the aid of artificial intelligence technology, we carried out a multiparametric bib-

liometric performance comparison of the 50 most productive Mexican HEIs, and were able

to represent their various production profiles in topographical clusters. To carry out the

analysis, we developed various scientometric indicators (described below), and used two

scientometric tools developed for Scopus by the SCImago Research Group (Spain): the

SCImago Journal and Country Rank (SJCR), and the SCImago Institutions Rankings (SIR).

All data were retrieved in May, 2013.

Production indicators

The annual number of documents produced in the country (ANdoc) is a standard pro-

duction indicator. However, we differentiate this indicator taking into account the pro-

duction of documents that are indexed in Scopus and Web of Science (Table 1, section a)

and define ANdoc Scopus and ANdoc WoS.

National research system indicators

The NRS constitutes the main resource of scientific production in Mexico. Hence, the

output volume of the nation is expected to grow in direct proportion to the number of NRS

scientists. Thus we define the indicator Nnrs as the number of NRS members. We eval-

uated Nnrs annually, and for each HEI, we define the indicator INnrs as the Institution’s

number of NRS scientists had per year, on average, during the period 2007–2011.

Nnrs was obtained from the project Atlas of Mexican Science, developed by CONACyT

(2012). INnrs values were obtained from the portal of the National Council of Science and

Technology (CONACyT 2013).

Country’s productivity indicators

Besides considering production indicators, we analyze the evolution from 1996 to 2011 of

the average number of scientific articles produced in the country each year, per each NRS

80 Scientometrics (2017) 110:77–104

123

Page 5: Multiparametric characterization of scientometric …Multiparametric characterization of scientometric performance profiles assisted by neural networks: a study of Mexican higher

member. This perspective leads to the concept, and indicators, of the scientific production

efficiency of the country, that we call the National Scientific Productivity: NSP WoS and NSP

Scopus (Table 1, section b). These indicators calculate the productivity restricted to each of

these databases, relative to the NRS. However, it is important to remark that these numbers

represent only estimations of the real quantities, because there is a residual number of sci-

entific articles produced in Mexico by researchers that do not belong to the NRS.

Performance profiles of the Mexican HEIs

Descending at the meso level, we identify and compare, with the neural network, the

scientometric performance profiles of the most productive Mexican HEIs bibliometric,

during the period 2007–2011. For this we consider 50 HEIs with production rates greater

than or equal to 30 articles per year (150 articles during this period), in journals indexed by

Scopus.

The analysis of each HEI was carried out considering four perspectives of scientific

production: output (volume), productivity (i.e., efficiency in the production), expected

Table 1 The summary of indicators used in this study

(a) Production indicators

ANdoc Annual number of documents produced by the country

ANdoc WoS Annual number of documents produced in journals that are indexed in the WoS

ANdocScopus

Annual number of documents produced in journals that are indexed in Scopus

Nnrs Number of National Research System (NRS) members

(b) Country’s productivity indicators

NSP National Scientific Productivity = ANdoc/Nnrs

NSP WoS National Scientific Productivity in WoS = ANdoc WoS/Nnrs

NSP Scopus National Scientific Productivity in Scopus = ANdoc Scopus/Nnrs

(c) Indicators of institutional productivity

AINdoc Annual Institutional Production = Average number of Scopus documents the institutionproduced per year, during the period 2007–2011

INnrs Institution’s number of NRS scientists had per year, on average, during the period2007–2011

ISP Institutional Scientific Productivity = AINdoc/INnrs

IPR Institutional Productivity Rate = [(AINdoc/Average ANdoc Scopus, during 2007–2011/(INnrs/Average Nnrs of Mexico during 2007–2011)]

(d) Indicators of expected and real visibility

%Q1 Measured by the percentage of the total amount of documents published in the first quartileof any Scopus’ subject category, according to the SCImago Journal Rank indicator

%Exc Excellence, measured by the percentage of the total number of documents (produced byeither the country or an institution) included in the 10% of the most cited articles of eachscientific field

RI Relative Impact = (Average number of citations that articles of a country, published in theyear (y), and indexed in Scopus, have accumulated during the time interval T)/(Averagenumber of citations that articles, published by the whole world in the year (y) and indexedin Scopus, have accumulated during T)

NI Normalized impact = Average value during the period 2007–2011 of the item oriented fieldnormalized citation score average

Scientometrics (2017) 110:77–104 81

123

Page 6: Multiparametric characterization of scientometric …Multiparametric characterization of scientometric performance profiles assisted by neural networks: a study of Mexican higher

visibility (estimated from the SCImago Journal Rank indicator), research impact (in terms

of the citation rates), and excellence (accounting highly cited documents). We define these

indicators in the following paragraphs and summarize them in Table 1, sections c and d.

Indicators of institutional productivity

To estimate annual institutional productivity of each HEI, relative to the NRS and Scopus,

during the period 2007–2011, we introduce concept of Institutional Scientific Productivity

(ISP). It represents the number of papers produced by each HEI per NRS member, in the

Scopus database. For instance, if ISP = 1 for an institution, that means that the institution

produces one research article per each national researcher on average. To calculate the ISP

indicator, we used information provided by both, the SCImago’s SIR and the CONACyT

of Mexico.

Equivalently, we could also assess the institutional scientific productivity from a dif-

ferent perspective, introducing the concept of Institutional Productivity Rate as the fol-

lowing percentages ratio: IPR = (Percentage of the National production, produced by the

institution)/(Percentage of the Nnrs of the country, belonging to the institution).

Thus, the IPR indicates the degree of productivity of an institution, relative to the

general productivity degree of the country to which it belongs. If an institution has an

IPR = 2, it means that the institution has a productivity level which doubles that of the

country.

There is a strong correlation between the two institutional productivity indicators (ISP

and IPR) that we have just defined. In fact they are linearly dependent, since for each

institution:

IPR ¼ ISP/NSP�;

where NSP* = (Average ANdoc Scopus, during 2007–2011)/(Average Nnrs during

2007–2011) = 0.86.

One might interpret NSP* as the ‘‘Average value of NSP, during 2007–2011’’, but this

interpretation would not be correct in general: we must notice that the constant NSP*,

being a ratio of averages (RoA), is not equal to the average National Productivity Rate

during the period 2007–2011, because the latter is not a RoA but an average of ratios

(AoR). However, these two quantities are strongly related: it is well known that a RoA can

be seen as a weighed AoR (Waltman et al. 2011); furthermore, it has been proved that the

regression line of a cloud of points of the form (AoRi, RoAi) is the identity line (Egghe

2012).

Since ISP and IPR have different values and different interpretations, they provide two

complementary views of data. For instance, if we consider the reference value (1) of the

two indicators as a threshold for ‘‘acceptable productivity efficiency’’, it results that only

36 of the 50 most productive HEIs of Mexico during 2007–2011 have an ISP larger than 1.

However, during this period 43 HEIs have an IPR over 1.

Indicators of expected and real visibility

As an indicator of expected visibility we consider %Q1, measuring the percentage of the

total amount of documents published in the first quartile of any Scopus’ subject category,

according to the SCImago Journal Rank indicator (Miguel et al. 2011).

82 Scientometrics (2017) 110:77–104

123

Page 7: Multiparametric characterization of scientometric …Multiparametric characterization of scientometric performance profiles assisted by neural networks: a study of Mexican higher

To estimate the actual visibility, we consider other three indicators of the SJCR: RI, NI

and %Exc.

For a given time interval T (in our case 1996–2011), and for each year (y) included in

the time interval T, the Scopus-based relative impact, RI, was calculated relativizing the

number of citation accumulated by the country by the number of citations accumulated by

the whole world. Authors used various names for this indicator in bibliometric literature:

‘‘impact normalized by world average’’ (Noyons 2005), or ‘‘impact relative to world’’, in

the glossary of inCitesTM (Thomsom Reuters 2015). For simplicity, we just call it ‘‘rel-

ative impact.’’

From the SJCR we obtained data to calculate and analyze the evolution of RI for

Mexico (Fig. 3) and, to analyze the RI of institutions, we used another impact indicator

(IN), obtained from SIR. This measure is based in the concept of ‘‘item oriented field

normalized citation score average’’ proposed by the Karolinska Institutet: ‘‘…it is calcu-

lated by normalizing each individual publication’s citation rate against an average citation

rate for articles in the same subject area, the same type and of the same age, and finally the

average of all the normalized citation values is calculated’’ (Rehn and Kronman 2008).

This indicator is useful to quantitatively compare the citation volumes in different areas of

knowledge.

As an indicator of excellence we use %Exc, measured by the percentage of the total

number of documents (produced by either the country or an institution) included in the

10% of the most cited articles of each scientific field (Bornmann et al. 2012).

Artificial intelligence method

In this paper we assume that Mexican HEIs have different performance profiles that can be

characterized from a scientometric perspective. This assumption places the hard problem

of automatically carrying out a multiparemetric characterization of scientometric perfor-

mance profiles of the 50 most productive HEIs of Mexico (in terms of the battery of

scientometric indicators described above). To deal with this problem we devised a data

mining method based on the SOM family of neural networks (Teuvo Kohonen 2013). The

method constitutes a novel tool for computational multiparametric analysis, presented here

as a scientometric data mining application graphically described in Fig. 1.

There have been other successful applications of SOM neural networks to scientometric

studies. Pioneer applications of SOM in scientometrics studies include the analysis of

keywords co-occurrence matrix in biomedical domains (Sotolongo-Aguilar et al.

2001, 2002), and for science and technology mapping (Polanco et al. 2001; Moya-Anegon

et al. 2006). More recent applications include the analysis of a biomedical domain evo-

lution (Guzman et al. 2010), and the improvement of SOM based mapping of science

(Skupin et al. 2013). In general, the problem of science mapping has been the main

application of the SOM neural network in the scientometric context. As far as we know,

there have been no applications of SOM based multiparametric analysis and characteri-

zation of the academic institutions performance profiles using scientometric indicators.

Our method was implemented in a software system called LabSOM, developed by the

Laboratory of Nonlinear Dynamics at the Faculty of Sciences of the National Autonomous

University of Mexico (UNAM) and the Company ‘‘Tecnologıas Inteligentes y Modelacion

de Sistemas’’. With this tool, the analysis and interpretation of the multidimensional data is

automated and facilitated through friendly visualization.

Typically, the SOM neural network is modelled as a two dimensional hexagonal grid.

Each hexagon represents an artificial neuron and, at the same time, a location where data

Scientometrics (2017) 110:77–104 83

123

Page 8: Multiparametric characterization of scientometric …Multiparametric characterization of scientometric performance profiles assisted by neural networks: a study of Mexican higher

points can be mapped. The application of SOM neural networks develops a nonlinear

projection of data into the neural network. The final (self-organized) map is the result of

the neural network iterative training process, by which the network learns to project similar

patterns into close locations (hexagons) in the map. The projection provides a represen-

tation of the distribution of the data set in the multidimensional space in a 2D cartography

that allows for visualization of the data. Thus, two HEIs can be searched in the resulting

map, and their distance can be analyzed as distance in the multidimensional space. The

main assumption to make inferences from this application is that the similarity between

HEIs’ performances can be estimated by calculating the ‘‘scientometric distance’’ among

their multidimensional representations.

One of the main issues to be faced in this investigation is the search for atypical

performance profiles. We call the institutions whose set of scientometric indicators exhibit

an atypical pattern outliers. It is very simple to identify atypical values of a single indi-

cator. A more difficult task is to identify atypical combinations of multiple indicators. We

show that the neural method is suited to automatically carry out this complex task.

To identify outliers, it is convenient to use a neural network with more neurons than

data points, which is contrary to what is recommended in a typical application of SOM

neural networks. To get the SOM maps exposed in this paper we used a neural network

with 2100 (30 9 70) neurons, trained after 500,000 computational iterations. The ratio

between length and width of this rectangular neural net is approximately the ratio between

the two main eigenvalues of the covariance matrix of the data set. This proportion has also

been successfully used in other contexts by other authors (Ultsch and Morchen 2005).

The clusters map obtained represents the set of performance patterns: it visually exhibits

the outliers and groups of institutions that share the various scientometric performance

styles identified by the neural network. The optimal number of clusters was calculated by

the software system LabSOM, with an algorithm that uses the Dunn index (Dunn 1973).

In order to obtain satisfactory visualizations which reveal non-trivial information about

the different patterns of performance, by means of the so called ‘‘Component Planes’’

Fig. 1 This diagram graphically explains the devised data mining procedure and the use of the SOMnetwork to process ISP, NI, %Q1 and %Exc data

84 Scientometrics (2017) 110:77–104

123

Page 9: Multiparametric characterization of scientometric …Multiparametric characterization of scientometric performance profiles assisted by neural networks: a study of Mexican higher

(Vesanto 1999), we had to apply an original normalization technique for data, based on the

logarithmic transformation of an appropriate piecewise linear scaling. The explanation of

this scaling, and other technical aspects of this methodology, will be detailed in further

publications.

Results and discussion

From 1996 to 2011 Mexico’s scientific production has increased in WoS and Scopus, with

average rates of change of 560 and 695 articles per year, respectively. Thus, the 2011

production indicator in Scopus tripled with respect to 1996, and in WoS, although in

smaller scale, it had a similar behavior.

Consequently, during the five-year period 2007–2011, in Elsevier database the country

had 67,116 registers, representing 46.28% of the whole output during the sixteen-year

period 1996–2011 (144,997). The output in Thomson Reuters’ databases during the same

period was 141,067 records, of which 62,087 (44%) were produced during 2007–2011.

The national researchers system

The growth of the scientific production in Scopus correlates to the growth of the total

amount of NRS members (Nnrs) which also tripled from 1996 to 2011 (Fig. 2), going from

5969 in 1996, to 17,568 in 2011, with an average annual growth rate of 7.5%. This is five

times greater than the average annual growth rate of the Mexican population (1.4%) during

the same period, passing from 97.2 million of people in 1996 to 119.4 in 2011. Therefore,

the number of NRS members per inhabitant grew from 61.4 Nnrs per million of people in

1996 to 147.1 in 2011. This significant growth in the number of Mexican scientists was due

to a sustained human resources development program of the national council of science

and technology (CONACYT). As a result of this effort, the annual scientific production of

the country in WoS has doubled, passing from 4.8 scientific articles published per million

Fig. 2 Parallel growth of the total number of NRS members and the Mexican scientific output in Web ofScience and Scopus, 1996–2011. The lower curve plots the difference Ndoc Scopus—Ndoc WoS

Scientometrics (2017) 110:77–104 85

123

Page 10: Multiparametric characterization of scientometric …Multiparametric characterization of scientometric performance profiles assisted by neural networks: a study of Mexican higher

people in 1996 to 11 in 2011. Outstandingly, the output in Scopus tripled during this

period, passing from 4 to 12 articles per million of people. However, there is still a long

road to go for Mexico if we compare the value of this indicator in the year 2011 with Brazil

(27), China (28), USA (185) or Finland (300).

Rates of growth

During the eleven years that go from 1996 to 2006, the average growth rate of NRS

members was 7.3%, and it grew to 7.7% during the five-year period 2007–2011. However,

growth rates of scientific production processes have a different behavior. In Scopus

database, from 1996 to 2006 there was a production average growth rate of 10.17%;

meanwhile during 2007–2011, the average went down to 5.34%, which means a decrement

of 47.40% among these two time windows. Similarly, from 1996 to 2006, we observe an

average growth rate of 8.80% in the WoS database, while in the period 2007–2011 the

average is 5.48%, signifying a decrement of 32.20%.

This situation should not be worrisome. In this scenario one should not expect rates of

growth to be increasing or constant, since in the latter case this would imply growth at an

exponential rate, and in the first case it would imply that production would grow

unbounded in finite time. Furthermore, it is well known that production rates of growth

might decrease in inverse proportion to the production indicator (number of scientific

articles being published per year), while this production indicator grows at a linear rate

(with a constant slope).

Parallel growth of Mexican scientific productivity and the nationalresearchers system

The number of Mexican papers produced per year in both databases is close to the number

of NRS members of the country. However, in Fig. 2 the curve of the number of NRS

members is above Scopus and WoS curves during the whole period, which can be assumed

as evidence that the country published less than one document per each NRS scientist in

high visible journals. The validity of this statement is accentuated by the fact that there are

contributions to the total national scientific production coming from a minority of Mexican

researchers that do not belong to the NRS. But we have to carefully interpret this data for

assessment purposes. In the first place, scientific papers are not the only valid product for

national researchers, since books, patents, utility models, etc. are also an important part of

their production. Secondly, we can statistically conclude that each NRS member con-

tributes to the national production with less than one document per year during the studied

period, but we cannot conclude that each NRS member produces less than one scientific

paper per year, because due to scientific collaboration, a considerable amount of docu-

ments are coauthored by more than one NRS scientist.

Quality versus quantity

In Fig. 2 we see that the production of scientific articles and the total number of NRS

members have both increased constantly during the period 1996–2011. However, different

trends in their rates of change have negatively impacted the national scientific productivity

indicators (NSP Scopus and NSP WoS) in the latest part of this period (Fig. 3). The

86 Scientometrics (2017) 110:77–104

123

Page 11: Multiparametric characterization of scientometric …Multiparametric characterization of scientometric performance profiles assisted by neural networks: a study of Mexican higher

decrement of values for these indicators might be interpreted as a loss of efficiency in

scientific production.

The NSP in WoS oscillated around a mean value of 0.85 during 1996–2007, but from

thereon, it has decreased to a minimum value of 0.74. During this period, the NSP in

Scopus increased from its minimum value (0.74) to a peak value of 0.95 in 2006, and

thereon oscillated around a mean value of 0.86 during the last five years. The evolution of

the two national scientific productivity indicators (NSP Scopus and NSP WoS) is compared

in Fig. 3 with the evolution of the relative impact (RI) of publications during 2007–2011.

We recall that, providing a comparison among the average number of citations per paper of

Mexico and the world, a relative impact of value 1 means that the number of citations that

Mexico has received—on average per article—is the same than the average number of

citations that the whole world has received per article in Scopus; larger values of the

relative impact would mean that Mexico receives more citations per paper, in Scopus, than

the world average.

In Fig. 3 the NSP indicators show a growing trend during the first part of the 1996–2011

period, but this increasing trend stops in the later part of the period. In fact, the NSP WoS

indicator decreases from 2003, reaching its minimum value in 2011.

Contrasting with this decrement of NSP WoS, the relative impact of the Mexican

scientific production exhibited an average growth trend of 5.53% during the period

2003–2011 (Fig. 4), approaching the value 1.07 from below with an evolution pattern

similar to that of Japan from 2003 on, and having higher growth rates during the period

2010–2011 than the five most productive countries: Mexico 10.3%, UK 6.2%, Germany

5.2%, Japan 3.0%, USA 2.0% and China 1.6%.

The noticeable increment that the relative impact of the Mexican production has had

since the year 2006 on, must be interpreted with care because the values of the relative

impact are a function of time, and recent values of relative impact could undergo large

variations (Bornmann and Leydesdorff 2013). Therefore, it would be expected that the

Fig. 3 Behavior of national scientific productivity indicators and the relative impact of Mexican scientificproduction in Scopus (1996–2011)

Scientometrics (2017) 110:77–104 87

123

Page 12: Multiparametric characterization of scientometric …Multiparametric characterization of scientometric performance profiles assisted by neural networks: a study of Mexican higher

strong rate of growth trend (18.88%) of the relative impact of Mexico during the interval

2010–2011 may be subject of adjustments in the years to come.

The 50 most productive higher education institutions of Mexico

Since most of the Mexican productive scientists are affiliated to a HEI, the Higher Edu-

cation System is the most important research sector of Mexico; from a total of 357 HEIs

registered in the SIR in 2011, the 50 most productive have the 80.21% of the NRS

members.

In what follows, descending at the meso level, and using the battery of previously

defined scientometric indicators, we carry out a comparative analysis of the dynamics of

scientific production in the 50 Mexican HEIs with greatest output in Scopus in the period

2007–2011. For this comparison, we first display scatter plots of various indicator pairs and

finally use a methodology based on artificial neural networks, that we develop to auto-

matically identify the main scientometric performance profiles within this group of HEIs,

taking into account all the indicators ensemble.

According to the Ibero American Ranking of Universities 2013 (SCImago Research

Group 2013) all of the 50 selected HEIs have produced more than 150 documents, covered

by Scopus, during this five-year period (Table 2).

Institutional productivity rates

The Universidad Nacional Autonoma de Mexico (UNAM) outstands as having the largest

number of NRS members, and the largest volume of scientific production during the

studied period, producing 19,349 articles, which represent the 28.83% of the national

output, and concentrating 21.78% of the Mexican NRS members to score an Institutional

Productivity Rate of 1.32. Its annual average production in this period was 3870 articles.

Being an outlier by its production score, we exclude it from some of the graphs of this

Fig. 4 Comparison of Mexico‘s relative impact evolution with the top 5 world’s most productive countries(SCImago Country Ranking)

88 Scientometrics (2017) 110:77–104

123

Page 13: Multiparametric characterization of scientometric …Multiparametric characterization of scientometric performance profiles assisted by neural networks: a study of Mexican higher

Table

2T

he

mo

stp

rod

uct

ive

Mex

ican

HE

Isin

Sco

pu

sd

uri

ng

the

per

iod

20

07

–2

01

1

Hig

her

educa

tion

inst

ituti

on

Ndoc

Ndoc%

Nnrs

Nnrs

%A

Ndoc

ISP

IPR

NI

Exc%

Q1

%

Un

iv.

Nac

.A

uto

n.

de

Mex

ico

(UN

AM

)19,349

28.83

3320

21.78

3869.8

1.1

71

.32

0.79

7.31

45.46

Cen

t.d

eIn

ves

t.y

de

Est

.A

van

z.(C

INV

ES

TA

V)

7072

10.54

621.6

4.08

1414.4

2.28

2.58

1.03

9.68

42.05

Inst

.P

oli

tec.

Nac

.(I

PN

)5581

8.32

668.2

4.38

1116.2

1.67

1.9

0.6

35

.27

30.69

Un

iv.

Au

ton

.M

etro

po

lita

na

(UA

M)

3934

5.86

816.6

5.36

786.8

0.9

61

.09

0.6

86.54

34.88

Un

iv.

de

Gu

adal

ajar

a(U

dG

)2097

3.12

579

3.8

419.4

0.7

20

.82

0.5

33

.56

32.81

Un

iv.

Au

ton

.d

eN

uev

oL

eon

(UA

NL

)1884

2.81

348

2.28

376.8

1.0

81

.23

0.6

25

.62

29

.62

Ben

em.

Un

iv.

Au

ton

.d

eP

ueb

la(B

UA

P)

1876

2.8

331.8

2.18

375.2

1.1

31

.29

1.07

8.54

34.38

Inst

.T

ecn

ol.

yd

eE

st.

Su

p.

de

Mo

nte

rrey

(IT

ES

M)

1649

2.46

236.4

1.55

329.8

1.4

1.5

90.82

8.36

29

.23

Un

iv.

de

Gu

anaj

uat

o(U

G)

1579

2.35

217.8

1.43

315.8

1.45

1.64

0.77

7.78

37.68

Un

iv.

Mic

h.

de

San

Nic

ola

sd

eH

idal

go

(UM

SN

H)

1488

2.22

262.8

1.72

297.6

1.1

31

.29

0.88

7.46

33.74

Un

iv.

Au

ton

.d

eS

anL

uis

Po

tosi

(UA

SL

P)

1482

2.21

21

3.6

1.4

296.4

1.3

91

.58

1.09

9.64

42.91

Un

iv.

Au

ton

.d

elE

stad

od

eM

ore

los

(UA

EM

)1

30

21

.94

20

7.2

1.3

62

60

.41

.26

1.4

30.79

6.38

42.63

Un

iv.

Au

ton

.d

eB

aja

Cal

ifo

rnia

(UA

BC

)1

19

81

.78

19

1.4

1.2

62

39

.61

.25

1.4

20

.59

4.8

92

5.3

8

Cole

gio

de

Post

gra

duad

os

(CO

LP

OS

T)

1130

1.6

8226.4

1.49

22

61

1.1

30

.34

2.2

316.81

Un

iv.

Au

ton

.d

elE

stad

od

eM

exic

o(U

AE

ME

X)

10

75

1.6

238.8

1.57

21

50

.91

.02

0.5

64

.89

27

.44

Univ

.V

erac

ruza

na

(UV

ER

)880

1.3

1233.4

1.53

17

60

.75

0.8

60

.56

4.6

134.32

Un

iv.

Au

ton

.d

eY

uca

tan

(UA

Y)

82

11

.22

12

8.2

0.8

41

64

.21

.28

1.4

50.78

8.16

34.47

Un

iv.

de

So

no

ra(U

SO

N)

80

11

.19

17

9.2

1.1

81

60

.20

.89

1.0

10

.53

3.9

636.2

Un

iv.

Au

ton

.d

elE

stad

od

eH

idal

go

(UA

EH

)7

47

1.1

11

51

.40

.99

14

9.4

0.9

91

.12

0.4

62

.79

27

.44

Un

iv.

Au

ton

.d

eQ

uer

etar

o(U

AQ

)6

76

1.0

11

18

.80

.78

13

5.2

1.1

41

.30.71

6.41

33.73

Univ

.Ib

eroam

eric

ana

(UIB

ER

)578

0.8

694.6

0.6

2115.6

1.2

21.3

92.42

24.4

59.69

Un

iv.

de

Co

lim

a(U

CO

L)

53

80

.81

08

.20

.71

10

7.6

0.9

91

.13

0.6

34

.64

40.89

Un

iv.

Au

ton

.C

hap

ing

o(U

AC

HA

P)

52

10

.78

10

8.2

0.7

11

04

.20

.96

1.1

0.3

62

.04

15.93

Un

iv.

de

las

Am

eric

asP

ueb

la(U

AM

ER

P)

52

10

.78

54

.80

.36

10

4.2

1.9

2.17

0.78

7.95

27

.45

Scientometrics (2017) 110:77–104 89

123

Page 14: Multiparametric characterization of scientometric …Multiparametric characterization of scientometric performance profiles assisted by neural networks: a study of Mexican higher

Table

2co

nti

nued

Hig

her

educa

tion

inst

ituti

on

Ndoc

Ndoc%

Nnrs

Nnrs

%A

Ndoc

ISP

IPR

NI

Exc%

Q1

%

Un

iv.

Au

ton

.d

eZ

acat

ecas

(UA

ZA

C)

44

20

.66

11

0.4

0.7

28

8.4

0.8

0.9

10

.56

4.5

29

.41

Un

iv.

Au

ton

.d

eS

inal

oa

(UA

SIN

)4

29

0.6

41

12

.20

.74

85

.80

.76

0.8

71.53

12.5

40.79

Univ

.A

uto

n.

de

Tam

auli

pas

(UA

TA

M)

391

0.5

862.8

0.4

178.2

1.2

51.4

10.5

55.7

23.02

Inst

.T

ecn

ol.

de

Tij

uan

a(I

TT

IJ)

37

90

.56

16

.80

.11

75

.84.51

5.08

0.74

9.73

12.66

Un

iv.

Au

ton

.d

eC

iud

adJu

arez

(UA

CJ)

33

80

.56

6.6

0.4

46

7.6

1.0

21

.14

0.6

96.64

28

.7

Univ

.A

uto

n.

de

Aguas

cali

ente

s(U

AA

GU

)318

0.4

751.2

0.3

463.6

1.2

41.4

0.4

3.3

523.58

Inst

.T

ecn

ol.

de

Cel

aya

(IT

CE

L)

31

70

.47

35

0.2

36

3.4

1.81

2.05

0.9

10.3

35.33

Un

iv.

Au

ton

.d

eC

hih

uah

ua

(UA

CH

IHU

)2

91

0.4

34

60

.35

8.2

1.2

71

.43

0.5

25

.93

26

.12

Un

iv.

Juar

ezA

uto

n.

de

Tab

asco

(UJA

TA

B)

27

90

.42

47

.40

.31

55

.81

.18

1.3

50

.38

1.5

720.79

Un

iv.

Au

ton

.A

gra

ria

An

ton

ioN

arro

(UA

AA

N)

27

80

.41

44

0.2

95

5.6

1.2

61

.42

0.4

4.4

322.3

Un

iv.

Juar

ezd

elE

stad

od

eD

ura

ng

o(U

JED

)2

53

0.3

83

10

.25

0.6

1.63

1.87

0.6

74

.96

23.72

Inst

.T

ecn

ol.

Au

ton

.d

eM

exic

o(I

TA

M)

25

00

.37

65

.60

.43

50

0.7

60

.86

0.6

54

.83

34.4

Cen

t.N

ac.

de

Inv

est.

yD

esar

r.T

ecn

ol.

(CN

IDT

)2

44

0.3

62

1.6

0.1

44

8.8

2.26

2.54

0.77

5.9

617.62

Univ

.A

uto

n.

de

Tla

xca

la(U

TL

AX

)243

0.3

647.2

0.3

148.6

1.0

31.1

61.11

10.5

39.09

Un

iv.

Au

ton

.d

eC

oah

uil

a(U

AC

OA

H)

21

70

.32

47

0.3

14

3.4

0.9

21

.04

0.6

65

.21

28

.57

Univ

.A

uto

n.

de

Cam

pec

he

(UA

CA

MP

)214

0.3

233.2

0.2

242.8

1.2

91.4

70.6

77.73

37.85

Un

iv.

Au

ton

.d

eB

aja

Cal

ifo

rnia

Su

r(U

AB

CS

)2

05

0.3

12

80

.18

41

1.46

1.69

0.5

72

.56

31.71

Univ

.A

uto

n.

de

Guer

rero

(UA

GU

E)

205

0.3

132.4

0.2

141

1.2

71.4

61.14

13.7

31.71

Un

iv.

Au

ton

.d

ela

Ciu

dad

de

Mex

ico

(UA

CM

)1

99

0.3

59

.60

.39

39

.80

.67

0.7

70

.59

3.1

441.71

Un

iv.

Pan

amer

ican

a(U

PA

N)

19

80

.33

70

.24

39

.61

.07

1.2

40

.47

4.9

421.72

Inst

.T

ecn

ol.

de

Mo

reli

a(I

TM

OR

)1

83

0.2

71

40

.09

36

.62.61

2.94

0.4

21

.08

18

.03

Un

iv.

Tec

no

l.d

ela

Mix

teca

(UT

MIX

)1

80

0.2

71

9.6

0.1

33

61.84

2.1

0.6

95

.52

13

.89

Inst

.T

ecn

ol.

de

To

luca

(IT

TO

L)

16

60

.25

6.6

0.0

43

3.2

5.03

5.77

0.3

92

.78

22

.29

Inst

.T

ecn

ol.

de

Ver

acru

z(I

TV

ER

)1

59

0.2

41

80

.12

31

.81.77

2.03

0.4

11

.31

15

.72

90 Scientometrics (2017) 110:77–104

123

Page 15: Multiparametric characterization of scientometric …Multiparametric characterization of scientometric performance profiles assisted by neural networks: a study of Mexican higher

Table

2co

nti

nued

Hig

her

educa

tion

inst

ituti

on

Ndoc

Ndoc%

Nnrs

Nnrs

%A

Ndoc

ISP

IPR

NI

Exc%

Q1

%

Univ

.A

uto

n.

de

Nay

arit

(UA

NA

Y)

159

0.2

420.6

0.1

431.8

1.54

1.78

0.6

32

.88

29

.56

Un

iv.

Au

ton

.d

eC

hia

pas

(UA

CH

IA)

15

40

.23

37

.60

.25

30

.80

.82

0.9

30

.38

2.7

42

6.6

2

Av

erag

es1

30

91

.951

21

5.4

1.4

13

26

1.9

1.4

1.5

90

.70

6.1

53

0.2

5

Nnrs

and

AN

doc

can

be

frac

tional

num

ber

sbec

ause

thes

ear

eav

erag

eval

ues

duri

ng

the

per

iod

2007–2011

Bold

val

ues

indic

ate

the

val

ues

above

the

aver

age

Scientometrics (2017) 110:77–104 91

123

Page 16: Multiparametric characterization of scientometric …Multiparametric characterization of scientometric performance profiles assisted by neural networks: a study of Mexican higher

paper (e.g., Fig. 5) in order to gain a better picture of the behavior of the other 49

institutions.

There are only two other institutions that had an annual average production greater than

1000 articles: the Centro de Investigaciones y Estudios Avanzados (CINVESTAV; 1414

articles), and the Instituto Politecnico Nacional (IPN; 1116 articles). These institutions

concentrated 10.5 and 8.3% of the Mexican scientific production, with 4.08 and 4.38% of

the total national researchers, obtaining institutional productivity rates of 2.28 and 1.67,

respectively. With an IPR of 1.09, the Universidad Autonoma Metropolitana (UAM) is the

second institution with more NRS members (5.16%), and the last one with more than 5%

of the country’s scientific output.

Institutional scientific productivity

The NSP indicator has an average value of 0.86 during the period 2007–2011, and the top

50 most productive HEIs average an ISP value of 1.4. So these 50 institutions have a

relevant contribution to the national scientific productivity. There were 14 HEIs that had

ISP values higher than 1.4, among them: Instituto Tecnologico de Veracruz (ITVER; 1.77),

Instituto Politecnico Nacional (IPN; 1.67), Universidad Juarez del Estado de Durango

(UJED; 1.63), Universidad Autonoma de Nayarit (UANAY; 1.54), Universidad Autonoma

de Baja California Sur (UABCS; 1.46), and Universidad de Guanajuato (UG; 1.45).

Thirteen HEIs in the selected sample score values of ISP smaller than one, and 37

produced more articles per year (visible in Scopus) than their number of national

researchers. Seven institutions have an ISP below the NSP indicator value (0.86):

Universidad Autonoma de Chiapas (UACHIA; 0.82), Universidad Autonoma de la Ciudad

de Mexico (UACM; 0.67), Instituto Tecnologico Autonomo de Mexico (ITAM; 0.76),

Universidad Autonoma de Sinaloa (UASIN; 0.76), Universidad Autonoma de Zacatecas

(UAZAC; 0.8), Universidad Veracruzana (UV; 0.75), and Universidad de Guadalajara

(UdG; 0.72).

Fig. 5 Productivity of the most prolific Mexican universities during the period 2007–2011. UNAM is notdepicted in the picture because it’s an outlier with 3320 National Researchers System members(ISP = 1.17)

92 Scientometrics (2017) 110:77–104

123

Page 17: Multiparametric characterization of scientometric …Multiparametric characterization of scientometric performance profiles assisted by neural networks: a study of Mexican higher

On the other hand, some institutions stand out by producing more than two papers per

NRS researcher annually (in Scopus journals), therefore obtaining an ISP[ 2. The most

conspicuous outliers with respect to this indicator are Instituto Tecnologico de Toluca

(ITTOL; 5.03) and Instituto Tecnologico de Tijuana (ITTIJ; 4.51). We will exclude them

from some of the graphics (e.g., Fig. 7) to gain better understanding of the institutions that

have more typical ISP values. There are other HEIs with large ISP values: Instituto Tec-

nologico de Morelia (ITMOR; 2.61), Centro de Investigacion y Estudios Avanzados

(CINVESTAV: 2.28), and Centro Nacional de Investigaciones y Desarrollo Tecnologico

(CNIDT; 2.26).

Other institutions with high productivity levels (ISP[ 1.8) are: Universidad de las

Americas de Puebla (UAMERP; 1.9), the Universidad Tecnologica de la Mixteca

(UTMIX; 1.84) and the Instituto Tecnologico de Celaya (UTCEL; 1.81).

The lowest values of the ISP indicator were obtained by Universidad Autonoma de la

Ciudad de Mexico (UACM; 0.67), Universidad de Guadalajara (UdG; 0.72), Universidad

Veracruzana (UVER; 0.75), Instituto Tecnologico Autonomo de Mexico (ITAM; 0.76), and

Universidad Autonoma de Sinaloa (UASIN; 0.76). In Fig. 5 we can see that, with the

exception of CINVESTAV, IPN, ITESM and UG, the highest values of ISP were obtained

by HEIs with a small number of national researchers (Nnrs\ 100).

Visibility, impact and excellence of Mexican Universities

Considering the inverse relation of productivity and impact observed at the macro level, we

would expect that the production efficiency of each institution (number of papers per NRS

member) might not be positively correlated with the production’s efficacy (production’s

impact, estimated in terms of the number of citations). In what follows, to enrich our

analysis, and to focus in this type of questions, we will consider a set of citation based

indicators proposed by the SIR.

For the most productive Mexican institutions, Table 2 shows the normalized impact

(NI) of its scientific production (compared to the world‘s) as well as %Q1, the percentage

of articles published in the most visible Scopus journals (Scopus quartile 1, according to

the SJCR), and %Exc, the percentage of papers that belong to the set of the 10% most cited

articles of each subject category of Scopus (the Productions Excellence Core). The other

indicators in Table 2 are: Total output (Ndoc), percentage of the total amount of docu-

ments published by the country (Ndoc%), average annual number of NRS members (Nnrs),

percentage of the total amount of national researchers (Nnrs%), institutional scientific

productivity (ISP) and institutional productivity rate (IPR).

In Table 2 we see that seven Mexican HEIs in the 50 institutions sample had a higher

normalized impact than the world during the period 2007–2011, and nine HEIs have %Q1

larger than the sample’s average (30.25%). It is noticeable that all the 50 institutions of the

sample have positive values of the excellence indicator (%Exc), meaning that they had at

least one paper in the Scopus Excellence Core.

The Universidad Iberoamericana (UIBER) does not stand out in this group of insti-

tutions for its productivity: with an ISP = 1.22 it is below the mean value of the 50 HEIs

(ISP = 1.4). However, it is the most outstanding outlier in this group of HEIs, achieving

extremely atypical values in the three citations-based indicators: UIBER published 59.7%

of their articles in highly visible journals, obtaining a normalized impact of 2.42 (its papers

obtained 142% more cites than the world’s average). Remarkably, 24.4% of its scientific

output was located in the ‘‘Excellence core’’, which is one of the best scores in Latin

America, comparable to the scores of some prestigious universities of USA and UK

Scientometrics (2017) 110:77–104 93

123

Page 18: Multiparametric characterization of scientometric …Multiparametric characterization of scientometric performance profiles assisted by neural networks: a study of Mexican higher

(Harvard University 28.69%, Stanford University 27.29%, and University of Cambridge

24.56%). The number of authors in UIBER publications also has very extreme values: 45%

of its articles have more than 500 authors and accumulate 80% of the total amount of

citations of the period 2007–2011. These data might indicate that this institution has an

Fig. 6 Normalized impact versus Excellence in Mexican HEIs (2007–2011). Universidad Iberoamericanais not depicted in the picture because it’s an outlier with the highest value (24.41%) of its production in the

10% most cited at the world level, and a NI value of 2.42. For these two indicators R2 ¼ 0:9:

Fig. 7 Excellence versus Institutional Scientific Productivity in Mexican HEIs (2007–2011). TheUniversidad Iberoamericana is not depicted in the picture because it’s an outlier with an extreme valueof %Exc = 24.41% (ISP = 1.22)

94 Scientometrics (2017) 110:77–104

123

Page 19: Multiparametric characterization of scientometric …Multiparametric characterization of scientometric performance profiles assisted by neural networks: a study of Mexican higher

important level of participation in large international research projects. As we did with

other outliers, we excluded the UIBER from some of the graphs (Figs. 5, 6, 7).

The mean value of the normalized impact was around 0.7 in the studied sample. There

are six HEIs besides UIBER, that have attained a NI[ 1: Universidad Autonoma de

Sinaloa (UASIN; 1.53), Universidad Autonoma de Guerrero (UAGUE; 1.14), Universidad

Autonoma de Tlaxcala (UTLAX; 1.11), Universidad Autonoma de San Luis Potosı

(UASLP; 1.09), Benemerita Universidad Autonoma de Puebla (BUAP; 1.07), and Centro

de Investigacion y de Estudios Avanzados (CINVESTAV; 1.03). The institutions Colegio

de Postgraduados (COLPOST; 0.34) and Universidad Autonoma de Chapingo (UACHAP;

0.36) showed the lowest values. For worldwide comparison purposes, it is convenient to

consider the institutions that belong to the 2011 top normalized impact ranking: Mas-

sachusetts Institute of Technology (29.86), The Rockefeller University (29.18), Harvard

University (28.28), and London Business School (27.01).

The excellence indicator (%Exc) showed a mean value of 6.2% in the 50 HEIs sample,

and 20 institutions had values of this indicator above the mean value. UAGUE (13.7%),

UASIN (12.5%) and UTLAX (10.5%) were in the leaders group. The Instituto Tecnologico

de Celaya (ITCEL; 10.3%) was the last institution with more than 10% of their papers in

the excellence core. All the 50 elements of the sample have at least one article in the

excellence core and the lowest values of the excellence indicator correspond to the Instituto

Tecnologico de Morelia (ITMOR; 1.1%) and the Instituto Tecnologico de Veracruz

(ITVER; 1.3%). At the world level in the top ranking of the excellence indicator we have

the following values: London Business School (70.33), The Rockefeller University

(61.64), Massachusetts Institute of Technology (57.87), and Harvard University 56.84.

With 45.5% of its papers published in 1st quartile-journals of Scopus, UNAM was

second in publishing in the most visible journals—after UIBER (60%). The Universidad

Autonoma de San Luis Potosı (UASLP; 42.9%), Universidad Autonoma del Estado de

Morelos (UAEM; 42.6%), CINVESTAV (42%) and Universidad Autonoma de la Ciudad

de Mexico (UACM; 41.7%), were also among the leaders of this indicator which has a

mean value of 30.2%. However, some institutions registered values under 20%.

Remarkably, the ITTIJ, which has the fifth place of excellence in the country, with

%Q1 = 12.7% has the lowest percentage of the 50 studied universities.

Biparametric analysis

In Figs. 5 and 6 we display data of these 50 institutions from two different perspectives:

first, we detect institutions which have the most efficacious bibliometric performance by

plotting the normalized impact and excellence (Fig. 6); secondly, we compare efficiency

versus effectivity, contrasting productivity with excellence (Fig. 7). Figure 8 complements

the biparametric analysis comparing excellence (%Exc) versus expected visibility (%Q1).

In all these figures we exclude UIBER in order to obtain a better picture of the most typical

behavior. This analysis covers the period 2007–2011.

Figure 6 exhibits the group of Mexican institutions with best bibliometric performance

from an efficacy-based perspective: it plots indicators which do not take into account

volume or efficiency of production but its impact, estimated with publication’s visibility

revealed by the citation process. The fact that the institutions arrange around a straight line

in the graph (with correlation coefficient of 0.9), proves that there is a considerable degree

of correlation among these two indicators: the values of NI increase in direct proportion to

the values of %Exc. Five institutions stand out with values of the NI above threshold

(UASIN, UTLAX, UASLP, BUAP and CINVESTAV), and 16 institutions lie in the graph

Scientometrics (2017) 110:77–104 95

123

Page 20: Multiparametric characterization of scientometric …Multiparametric characterization of scientometric performance profiles assisted by neural networks: a study of Mexican higher

above the mean values of these two indicators: UIBER, UASIN, UAGUE, UTLAX,

UASLP, UNAM, BUAP, CINVESTAV, ITCEL, ITTIJ, ITESM, UAMERP, UMSNH,

UAEM, UAQ y UAY.

A different situation is observed in Fig. 7, where the relation of Excellence (%Exc) and

ISP indicators is exhibited. There is no positive correlation among these indicators (cor-

relation coefficient R2 ¼ 0:0002Þ and the institutions distribute widely in the Cartesian

space. The highest values of %Exc were attained by UIBER, UAGUE, UASIN, UATLAX;

and all of them have relatively low productivity values (below the mean). The ITTIJ is a

notable exception. Above the mean values, and with an important degree of balance among

these indicators we find CINVESTAV, ITCEL, UAMERP and UG. On the other hand,

although with relatively small score in excellence, ITTOL and ITMOR outstand with very

high values of the productivity indicator. It looks as if less quantity would be the price of

quality, and vice versa.

Contrary to what one might expect, the scatter plot of %Exc versus %Q1 in Fig. 8

exhibits very low degree of correlation among the indicators (correlation coefficient R2 ¼0:3755Þ: However, an important number of institutions (15) score above the mean values

of both indicators. ITTIJ shows the most atypical behavior: the lowest score in %Q1 (12.6)

with high score in %Exc (9.7). Nevertheless, for the rest of the institutions, the emptiness

of the lower right quadrant (below %Q1 = 30.25 and to the right of %Exc = 6.15) sug-

gests that the achievement of excellence is strongly linked to the publication in high

visibility journals. Differently, the upper left quadrant populated by 8 institutions suggests

that, with significant probability, a poor achievement of excellence can occur in spite of

publishing in high impact journals or equivalently: ‘‘Low excellence is not the result of the

publication in low impact journals’’.

Some of the issues revealed by the biparametric analysis are better understood from a

multiparametric perspective. In the next section an artificial intelligence approach, based

on self-organized mapping techniques, will serve to obtain a multiparametric characteri-

zation of the most productive Mexican universities.

Fig. 8 Excellence versus expected visibility in Mexican HEIs (2007–2011). For these two indicators

R2 ¼ 0:3755:

96 Scientometrics (2017) 110:77–104

123

Page 21: Multiparametric characterization of scientometric …Multiparametric characterization of scientometric performance profiles assisted by neural networks: a study of Mexican higher

Scientometric performance profiles of the Mexican Universities

Now we proceed to carry out the multiparametric scientometric performance analysis of

the 50 most productive Mexican HEIs, according to four indicators: ISP, %Q1, %Exc and

NI. Instead of obtaining four different rankings for each of these indicators, or analyzing

the dataset from the perspective of two indicators by means of scatter plots, here we face

the problem of comparing the performance of these institutions taking into account the four

scientometric indicators simultaneously.

For this purpose, we first represent each university as a point in a Euclidean four-

dimensional space, whose coordinates are given by values of the four indicators. We then

use a neural network to deal automatically with the comparison and visualization problem.

Our visualization technique throws five maps (cartographies) which exhibit the perfor-

mance comparison and clustering done by the neural network. These maps are drawn over

a hexagonal grid, where each hexagon represents one neuron. In this case we used a

70 9 20 neurons grid (Fig. 9).

Once the neural network is trained by means of an adaptive and unsupervised proce-

dure, each institution is plotted in the map applying a nonlinear projection from the 4D

space into the neural grid. Since we are using a similarity measure that considers the four

scientometric indicators, institutions projected close to each other over the grid share

similar bibliometric characteristics. The projection clusters the institutions on the map in

such a way that those having the most similar profiles are plotted inside the same cluster.

Since each cluster of institutions represents a bibliometric pattern of institutional

Fig. 9 Self-organized clusters map of Mexican universities obtained by the multiparametric analysis

Scientometrics (2017) 110:77–104 97

123

Page 22: Multiparametric characterization of scientometric …Multiparametric characterization of scientometric performance profiles assisted by neural networks: a study of Mexican higher

performance, this technology allows us to visually locate various bibliometric profiles of

academic production, and to identify groups of institutions that have similar patterns of

performance.

In Fig. 9 we observe five maps, the first one displays the clusters map of the 50

institutions. This map exhibits various regions (clusters) distinguished by colors, and the

group of universities that belong to each region. Universities in each cluster have similar

scientometric performance patterns, according to the four selected indicators. Atypical

behaviors (outliers) are represented by regions to which only one university belongs. Such

are the cases of UIBER, UASIN, UA GUE, ITTIJ, ITTOL, ITMOR, and UACM. The rest

of the universities share their production profile with other institutions of the group.

The other four maps in Fig. 9 display another type of visualization provided by the

neural network. They are called component maps. There is one map for each of the four

indicators that compose the quantitative bibliometric information considered for each

university. These maps are fundamental for the results interpretation. By means of a

chromatic scale they show the map’s distribution of indicator’s values. In each component

map the maximum value of the indicator corresponds to the darker red, the mean value is

associated to the yellow tone, and the minimum value to the lighter green. Intense yellow is

associated with values close to the mean of each indicator. Thus, component maps allow us

to read the meaning of the data distribution on the neural grid.

In the cluster visualization in Fig. 9, we can see that the neural grid is divided in thirteen

regions (clusters). As we have pointed out before, seven of these regions (O1,…,O7) are

characterized by the fact that there is only one institution (in this group of 50 universities)

that is mapped into them; the other six regions (C1,…,C6) contain more than one institution.

For the results interpretation, one must observe that the upper zone of the map labeled

%Q1 in Fig. 9, corresponds to the highest values (darker red) of the %Q1 indicator and, the

values of this indicator continuously decrease as we go down in the map. For example, the

institutions that belong to the regions O1, O2, O3, O7, C1, C2, and the upper part of the

region C3, have the highest percentages of articles published in first quartile journals. We

call this zone the High Expected Visibility Zone.

Similarly, in the map labeled %Exc, the upper-left zone corresponds to the highest

values of the %Exc indicator, and the same is true for the map labeled with the NI

indicator—a fact that was expected given the high degree of correlation among these two

indicators. We will call this zone the Efficacy Zone. The institutions that have the highest

scores in %Exc and NI are located in this zone of the map. This zone intersects with the

High Expected Visibility Zone and %Exc and %NI values decrease continuously as we

advance diagonally towards the lower-right corner of the map.

The lower-right corner of all the component maps is a relatively low value zone (below

average values) for all four indicators (ISP, NI %Exc, and %Q1). Most of the institutions in

Cluster C5 are characterized by low efficiency, low expected visibility, and low effectivity

profile.

Efficacy zone

Institutions that belong to this zone might have a good deal of high impact publications or

might be participating in some large international research projects that produce articles

(with hundreds of authors) which accumulate an extraordinary amount of citations. The

elite strip of the Efficacy Zone is composed by the institutions UIBER, UASIN, UAGUE

(O1, O2, O3), and the group of institutions that compose the cluster C1 = {UASLP,

CINVESTAV, UTLAX, BUAP, ITCEL}. Also in the Efficacy Zone, but with lower scores

98 Scientometrics (2017) 110:77–104

123

Page 23: Multiparametric characterization of scientometric …Multiparametric characterization of scientometric performance profiles assisted by neural networks: a study of Mexican higher

of NI and %Exc, are eight institutions belonging to cluster C2 = {UNAM, UAEM, UG,

UCOL, UMSNH, UAM, UAQ}. All the institutions in the Efficacy Zone are characterized

by high efficacy (impact of their outputs), but efficiency (ISP and IPR) mostly under the

mean values of the 50 most productive Mexican HEIs (this is observed in Scatter Plots

Figs. 5, 6). CINVESTAV has the most balanced profile within the 50 institutions sample;

and with scores well above averages in the four indicators considered, it is an exemplary

exception to this rule (observe the isolated red spot over CINVESTAV in the ISP Com-

ponent Map). ITCEL is the other institution of the group that has a well balanced profile.

Institutions that belong to the region C2 are characterized by their high percentage of

papers in first quartile (%Q1), efficacy parameters close to the mean value from above, and

efficiency scores (ISP, IPR) below the average.

As if efficiency were the price of efficacy, the three outliers (O1,O2,O3) of the Efficacy

Zone, having the highest %Exc and NI scores, do not outstand in their ISP scores. In fact,

UIBER, which conspicuously stands out in this group because of its extreme values of

%Exc and NI, does not have high efficiency indicator values (ISP and IPR). Also UASIN

and UAGUE share this high efficacy-low efficiency profile. UASIN is extreme in this

respect, because its ISP score is very close to the minimum value for all the universities,

but it is within the top three in %Exc and NI. The neural net differentiates these two

outliers, UASIN and UAGUE, in two clusters (O2, O3) due to their %Exc and NI disparity

(Fig. 6: NI larger for UASIN and %Exc larger for UAGUE).

Efficiency zone

In the ISP component map, we observe that the lower-left corner, the Efficiency Zone, is

where institutions with the highest ISP scores are located. Three outliers O4, O5, O6, part

of the region C6 (CNIDT), and the region where CINVESTAV is located, constitute the

elite group of the Efficacy Zone. ITTOL and ITTIJ are the leaders in this indicator for the

whole group of 50 institutions, followed by ITMOR and CNIDT. After them, we find

UTMIX and ITVER, with lower efficiency level but still above average. It is important to

remark that the average ISP value (1.4) for the 50 institutions is high with respect to the

same value for the rest of the country: the NSP in Scopus for the 2007–2011 period is

approximately 0.86. So ISP = 1, might be considered low with respect to the average

value (1.4), but it is relatively high with respect to the whole country.

High expected visibility zone

The upper part of the %Q1 component map concentrates institutions that have highest

percentage of publications in first quartile journals. As might be expected, the Efficacy

Zone is a subset of the Expected Visibility Zone, meaning that institutions with high

impact scores also have high scores of %Q1, but this is not a rule without exception. An

interesting exceptional case is that of ITTIJ (O4): while having the lower %Q1 value

among the 50 HEI, it scores high in %Exc, and has the second place in productivity (ISP).

Because of this unique profile ITTIJ does not fit in another cluster.

High expected visibility and low efficacy and low efficiency zone

There is a triangular zone in the upper-right part of the map that contains various insti-

tutions with relatively high values of %Q1 but low or just below average value of ISP,

Scientometrics (2017) 110:77–104 99

123

Page 24: Multiparametric characterization of scientometric …Multiparametric characterization of scientometric performance profiles assisted by neural networks: a study of Mexican higher

%Exc and NI indicators. This triangle contains the lower part of cluster C2, the upper part

of cluster C3, and cluster O7. Among the institutions with this bibliometric profile we find:

UACM, UAM, UAQ, USON, ITAM, and UVER. We remark that the ISP values for these

institutions are low, with respect to its average value (1.4), but for some of them this value

is still high relatively to the NSP.

The existence of this High Expected Visibility & Low Efficacy Zone confirms the well

known fact that publication in high impact journals does not guarantee high impact.

Furthermore, there is an uncommon situation in which high impact is reached without

publishing in high impact journals. Such counterexample is shown in the %Exc component

map. In the lower-left part of this map, we can see a region (red) with high %Exc score that

corresponds to the exceptional ITTIJ, for which the %Q1 indicator attains its minimum

value.

Conclusions

The artificial intelligence procedure that we devised has been successfully applied here for

scientometric data mining. Particularly, it was useful to carry out the multifactorial analysis

and scientometric characterization of the most productive Mexican universities. It allowed

us to automatically identify various institutional scientometric performance profiles, the

universities that fit in them, as well as bibliometric outliers that stand out with peculiar

profiles. Furthermore, it provided clear and useful data visualization resources that resulted

to be an excellent complement to the traditional scatterplot and correlation analysis. From

all of this we conclude that it is worth to consider the use of this analysis and visualization

procedure for science and technology data mining.

The non-linear projection capabilities offered by the SOM neural network and the novel

data normalization technique based on the logarithmic transformation of an appropriate

piecewise linear scaling, not only cluster the data but also offer different visual repre-

sentations which allow for further inferences about the data-set. As illustrated in the paper,

the traditional scatterplot and correlation analysis is useful only to observe global corre-

lations—outliers are identified only in the case where there is an extreme value. The

visualizations obtained are powerful tools to discover non-trivial outliers that otherwise

would be very difficult to discover.

It follows from our analysis that there is not a consistent correlation among productivity

(efficiency of the production) and effectivity (understood as the production impact) for the

Mexican HEIs (e.g., it is not the case that the more productive institutions are producing

the highest impact research). This phenomenon would have to be investigated in greater

depth in order to relate quantity and quality of the Mexican HEIs scientific production. A

negative correlation among quality and quantity was also observed at the country level: in

the more recent period a decrement in the number of scientific articles per national

researcher coexisted with a growing trend of the impact, estimated with the RI indicator.

However, this conclusion must be taken with caution, given the intrinsic volatility of this

indicator.

It was very important that the battery of selected scientometric indicators not only

considered production volume, but also provided an efficiency and quality estimation of

HEIs research output. Since there are great differences in size among the 50 most pro-

ductive Mexican HEIs, we introduced new size independent indicators that try to balance

these differences. For this, the use of NRS data was fundamental.

100 Scientometrics (2017) 110:77–104

123

Page 25: Multiparametric characterization of scientometric …Multiparametric characterization of scientometric performance profiles assisted by neural networks: a study of Mexican higher

Our approach has shown the prominence, not only of the larger institutions of the

country, but also some intermediate and, remarkably, small ones with outstanding scores in

scientometric indicators. This is a positive indication of the existence of excellent

researchers in these institutions, which is commendable and noteworthy. Notwithstanding,

for assessment purposes, or for the comparison of the larger with the smaller universities,

this fact has to be interpreted with care: statistical analysis, like the one we have applied

here, takes in account averages, and it is not the same to average the work of thousands of

researchers (e.g., UNAM case) than to average over just a few dozens of them. In big

universities the large numbers law minimizes the possibility of statistical irregularities.

Also, large universities, unlike relatively smaller technological institutes, tend to be the-

matically generalistic with large groups of researchers in areas of social science, arts and

humanities, which do not have the same publication patterns of engineers, mathematicians

or natural science researchers.

Scientometric characterizations are useful, but further studies are necessary for the

assessment of institution’s performance, to take into account other aspects of scientific

production. For instance, our current research remarks the importance of taking the number

of authors in publications into consideration, which should be analyzed in future studies. It

is noteworthy that most of the highest scores in the %Exc and NI indicators do not

correspond to the larger Mexican HEIs, but to relatively small ones, and a similar situation

is present in relation with the ISP indicator. Indicator %Q1 also displays unexpected

behavior. We have found that, with some exceptions, this indicator does not correlate

positively with the productivity (ISP indicator) nor with the impact indicators.

We close our conclusions pointing out, from a country perspective, that a comparative

analysis during the period 1996–2011 of the Mexican Scientific production in WoS and

Scopus reveals a sustained growth in both databases, which is kept with the growth of the

National Research System of Mexico. The evolution of the Mexican scientific production

in WoS and Scopus is very similar from 1996 to 2005, but from there on Scopus production

exhibits a higher level of growth.

Acknowledgements This research was partially supported by the Proyecto CITMA-CONACyT (B330.166)and the Empresa de Tecnologıas Inteligentes y Modelacion de Sistemas S.A. de C.V. The authorsacknowledge the collaboration of Jose Luis Jimenez Andrade (UNAM, Mexico), and of Dr. Felix de MoyaAnegon (CSIC, Spain) for the data support given from SCImago Institutions Rankings.

References

Abramo, G., Cicero, T., & D’Angelo, C. A. (2013). National peer-review research assessment exercises forthe hard sciences can be a complete waste of money: The Italian case. Scientometrics, 95(1), 311–324.

Aguillo, I. F., Bar-Ilan, J., Levene, M., & Ortega, J. L. (2010). Comparing university rankings. Sciento-metrics, 85(1), 243–256.

Aguillo, I. F., Granadino, B., & Llamas, G. (2005). Web positioning of the university system in LatinAmerica. Interciencia, 30(12), 735–738.

Allen, N., & Heath, O. (2013). Reputations and research quality in british political science: The importanceof journal and publisher rankings in the 2008 RAE. British Journal of Politics & InternationalRelations, 15(1), 147–162.

Arencibia-Jorge, R., & de Moya-Anegon, F. (2010). Challenges in the study of Cuban scientific output.Scientometrics, 83(3), 723–737.

Arvanitis, R., Russell, J. M., & Rosas, A. M. (1996). Experiences with the national citation reports databasefor measuring national performance: The case of Mexico. Scientometrics, 35(2), 247–255.

Scientometrics (2017) 110:77–104 101

123

Page 26: Multiparametric characterization of scientometric …Multiparametric characterization of scientometric performance profiles assisted by neural networks: a study of Mexican higher

Belkhodja, O., & Landry, R. (2007). The Triple-helix collaboration: Why do researchers collaborate withindustry and the government? What are the factors that influence the perceived barriers? Sciento-metrics, 70(2), 301–332.

Belter, C. W. (2013). A bibliometric analysis of NOAA’s Office of Ocean Exploration and Research.Scientometrics, 95(2), 629–644.

Bengoetxea, E., & Buela-Casal, G. (2013). The new multidimensional and user-driven higher educationranking concept of the European Union. International Journal of Clinical and Health Psychology,13(1), 67–73.

Benito, M., & Romera, R. (2011). Improving quality assessment of composite indicators in universityrankings: A case study of French and German universities of excellence. Scientometrics, 89(1),153–176.

Billaut, J. C., Bouyssou, D., & Vincke, P. (2010). Should you believe in the Shanghai ranking? Sciento-metrics, 84(1), 237–263.

Bornmann, L., & Leydesdorff, L. (2013). Macro-indicators of citation impacts of six prolific countries:InCites data and the statistical significance of trends. PLoS ONE, 8(2), e56768.

Bornmann, L., Moya-Anegon, F., & Leydesdorff, L. (2012). The new excellence indicator in the WorldReport of the SCImago Institutions Rankings 2011. Journal of Informetrics, 6(2), 333–335.

Caputo, C., Requena, J., & Vargas, D. (2012). Life sciences research in Venezuela. Scientometrics, 90(3),781–805.

Collazo-Reyes, F., Luna-Morales, M. E., Russell, J. M., & Perez-Angon, M. A. (2008). Publication andcitation patterns of Latin American and Caribbean journals in the SCI and SSCI from 1995 to 2004.Scientometrics, 75(1), 145–161.

Collazo-Reyes, F., Luna-Morales, M. E., Russell, J. M., & Perez-Angon, M. A. (2010). Enriching knowl-edge production patterns of Mexican physics in particles and fields. Scientometrics, 85(3), 791–802.

CONACyT. (2012). Atlas de la Ciencia Mexicana. Mexico D.F: Academia Mexicana de Ciencias.CONACyT. (2013). Resultados Anteriores del Sistema Nacionales de Investigadores (SNI). http://conacyt.

gob.mx/index.php/resultados-anteriores.Dunn, J. C. (1973). A fuzzy relative of the ISODATA process and its use in detecting compact well-

separated clusters. Journal of Cybernetics, 3(3), 32–57.Egghe, L. (2012). Averages of ratios compared to ratios of averages: Mathematical results. Journal of

Informetrics, 6(2), 307–317.Foro Consultivo Cientıfico y Tecnologico. (2011). Ranking de Produccion Cientıfica Mexicana. http://www.

foroconsultivo.org.mx/libros_editados/ranking_por_institucion_2011.pdf.Gomez-Nunez, A. J., Vargas-Quesada, B., de Moya-Anegon, F., & Glanzel, W. (2011). Improving SCImago

Journal and Country Rank (SJR) subject classification through reference analysis. Scientometrics,89(3), 741–758.

Guzman, M. V., Carrillo, H., Jimenez, J. L., & Villasenor, E. A. (2010). Bioinformetric studies on TBvaccines research. In: N. M. Nor, A. Acosta & M. E. Sarmiento (eds.), The art and science ofTuberculosis vaccine development (pp. 425–441). Oxford: Oxford University Press.

Helene, A. F., & Ribeiro, P. L. (2011). Brazilian scientific production, financial support, establishedinvestigators and doctoral graduates. Scientometrics, 89(2), 677–686.

Huang, M. H. (2012). Exploring the h-index at the institutional level: A practical application in worlduniversity rankings. Online Information Review, 36(4), 534–547.

Ioannidis, J. P., Patsopoulos, N. A., Kavvoura, F. K., Tatsioni, A., Evangelou, E., Kouri, I., et al. (2007).International ranking systems for universities and institutions: A critical appraisal. BMC Medicine,5(30), 1.

Kohonen, T. (2001). Self-organizing maps (3rd ed.). Berlin: Springer.Kohonen, T. (2013). Essentials of the self-organizing map. Neural Networks, 37, 52–65.Krauskopf, M., Krauskopf, E., & Mendez, B. (2007). Low awareness of the link between science and

innovation affects public policies in developing countries: The Chilean case. Scientometrics, 72(1),93–103.

Krishna, D., Mohan, S. R., Murthy, B. S. N., & Rao, A. R. (2002). Performance evaluation of publicresearch institutes using principal component analysis. Journal of Scientific & Industrial Research,61(11), 940–947.

Kurzydlowski, K. J. (2003). Materials research in Poland. In W. Lojkowski & J. R. Blizzard (Eds.),Interfacial effects and novel properties of nanomaterials (pp. 349–356). Zurich-Uetikon: Trans TechPublications Ltd.

Lena, M. (1997). Scientific productivity in environmental psychology in Mexico—A bibliometric analysis.Environment and Behavior, 29(2), 169–197.

102 Scientometrics (2017) 110:77–104

123

Page 27: Multiparametric characterization of scientometric …Multiparametric characterization of scientometric performance profiles assisted by neural networks: a study of Mexican higher

Leydesdorff, L., & Meyer, M. (2007). The scientometrics of a Triple Helix of university–industry–gov-ernment relations (introduction to the topical issue). Scientometrics, 70(2), 207–222.

Licea de Arenas, J., Castanos-Lomnitz, H., & Arenas-Licea, J. (2002). Significant Mexican research in thehealth sciences: A bibliometric analysis. Scientometrics, 53(1), 39–48.

Licea de Arenas, J., Valles, J., & Arenas, M. (2000). Educational research in Mexico: Socio-demographicand visibility issues. Educational Research, 42(1), 85–90.

Luna-Morales, M. E. (2012). International scientific collaboration and recognition of Mexican science from1980 to 2004. Investigacion Bibliotecologica, 26(57), 103–129.

Macias-Chapula, C. A., Mendoza-Guerrero, J. A., Rodea-Castro, I. P., & Gutierrez-Carrasco, A. (2007).Institutional health research collaboration in Mexico: A Bibliometric study. In: Proceedings of the 11thinternational conference of the international society for scientometrics and informetrics (pp. 894–895).Madrid: ISSI.

Miguel, S., Chinchilla-Rodrıguez, Z., & Moya-Anegon, F. (2011). Open access and Scopus: A newapproach to scientific from the standpoint of access. Journal of the American Society for InformationScience and Technology, 62(6), 1130–1145.

Miguel, S., Moya-Anegon, F., & Herrero-Solana, V. (2010). The impact of the socio-economic crisis of2001 on the scientific system of Argentina from the scientometric perspective. Scientometrics, 85(2),495–507.

Moya-Anegon, F., Herrero-Solana, V., & Jimenez-Contreras, E. (2006). A connectionist and multivariateapproach to science maps: The SOM, clustering and MDS applied to library and information scienceresearch. Journal of Information Science, 32(1), 63–77.

Mryglod, O., Kenna, R., Holovatch, Y., & Berche, B. (2013). Absolute and specific measures of researchgroup excellence. Scientometrics, 95(1), 115–127.

Noyons, C. M. (2005). Science maps within a science policy context. In: H. F. Moed, W. Glanzel & U.Schmoch (eds.), Handbook of quantitative science and technology research (pp. 237–255). Springer:Netherlands.

Polanco, X., Francois, C., & Lamirel, J. C. (2001). Using artificial neural networks for mapping of scienceand technology: A multi-self-organizing-maps approach. Scientometrics, 51(1), 267–292.

Rehn, C., & Kronman, U. (2008). Bibliometric handbook for Karolinska Institutet. Estocolmo: KarolinskaInstitutet University Library.

SCImago Research Group. (2013). SIR Iber 2013. http://www.scimagoir.com/pdf/SIR%20Iber%202013.pdf.

Sierra-Flores, M. M., & Barnard, J. M. R. (2009). The most productive research groups of the NationalAutonomous University of Mexico (UNAM) in the area of physics, 1990–1999. Investigacion Bib-liotecologica, 23(48), 127–155.

Sierra-Flores, M. M., Guzman, M. V., Raga, A. C., & Perez, I. (2009). The productivity of Mexicanastronomers in the field of outflows from young stars. Scientometrics, 81(3), 765–777.

Skupin, A., Biberstine, J. R., & Borner, K. (2013). Visualizing the topical structure of the medical sciences:A self-organizing map approach. PLoS ONE, 8(3), e58779.

Sotolongo-Aguilar, G., Guzman-Sanchez, M. V., & Carrillo-Calvet, H. (2002). ViBlioSOM: Visualizacionde informacion bibliometrica mediante el mapeo autoorganizado. Revista Espanola de DocumentacionCientıfica, 25(4), 477–484.

Sotolongo-Aguilar, G., Guzman-Sanchez, M. V., Saavedra-Fernandez, O., Carrillo-Calvet, H. A. (2001).Mining informetric data with self-organizing maps. In Proceedings of the 8th international society forscientometrics and informetrics (pp. 665–673). Sydney: BIRG.

Statzner, B., & Resh, V. H. (2010). Negative changes in the scientific publication process in ecology:potential causes and consequences. Freshwater Biology, 55(12), 2639–2653.

Thomson Reuters. (2015). http://incites.isiknowledge.com/common/help/h_glossary.html.Torres-Salinas, D., Moreno-Torres, J. G., Delgado-Lopez-Cozar, E., & Herrera, F. (2011). A methodology

for institution-field ranking based on a bidimensional analysis: The IFQ(2)A index. Scientometrics,88(3), 771–786.

Ultsch, A., & Morchen, F. (2005). ESOM-maps: Tools for clustering, visualization, and classification withemergent SOM. Technical report. 46. Department of Mathematics and Computer Science, Universityof Marburg, Germany.

UNAM. (2012). Estudio Comparativo de Universidades Mexicanas. http://www.ecum.unam.mx/.Vanclay, J. K., & Bornmann, L. (2012). Metrics to evaluate research performance in academic institutions:

A critique of ERA 2010 as applied in forestry and the indirect H-2 index as a possible alternative.Scientometrics, 91(3), 751–771.

Vesanto, J. (1999). SOM-based data visualization methods. Intelligent Data Analysis, 3(2), 111–126.

Scientometrics (2017) 110:77–104 103

123

Page 28: Multiparametric characterization of scientometric …Multiparametric characterization of scientometric performance profiles assisted by neural networks: a study of Mexican higher

Waltman, L., van Eck, N. J., van Leeuwen, T. N., Visser, M. S., & van Raan, A. F. (2011). Towards a newcrown indicator: An empirical analysis. Scientometrics, 87(3), 467–481.

Weingart, P. (2005). Impact of bibliometrics upon the science system: Inadvertent consequences? Scien-tometrics, 62(1), 117–131.

Zell, D. (2005). Pressure for relevancy at top-tier business schools. Journal of Management Inquiry, 14(3),271–274.

Zhao, S. X., Zheng, L. C., & Cai, Z. C. (2009). Evaluation of science and technology innovation ability ofcolleges and universities in China. In International conference on management of tech-nology,pp. 220–225.

104 Scientometrics (2017) 110:77–104

123