Measuring and ranking R&D performance at the country level
Transcript of Measuring and ranking R&D performance at the country level
100 Marianela Carrillo ISSN 2071-789X
RECENT ISSUES IN ECONOMIC DEVELOPMENT
Economics & Sociology, Vol. 12, No. 1, 2019
MEASURING AND RANKING R&D
PERFORMANCE AT THE COUNTRY
LEVEL Marianela Carrillo, Universidad de La Laguna (ULL) Tenerife, Spain E-mail: [email protected] Received: July, 2018 1st Revision: October, 2018 Accepted: February, 2019
DOI: 10.14254/2071-789X.2019/12-1/5
ABSTRACT. In the context of generally growing interest in
R&D (Research and Development) efficiency, the main objective of this work is evaluation and ranking of the R&D performance on a set of selected countries with the highest worldwide level of engagement in R&D activities. To that aim, R&D efficiency of the sample countries is assessed with Data Envelopment Analysis, then the overall performance score is obtained with the cross-efficiency method and the considered countries are listed in the order according to their R&D performance. The findings of this study point at Switzerland, the United Kingdom and the Netherlands as the three leading countries as far as R&D performance is concerned, while the countries that make important investment efforts in terms of their GDP, such as Japan or Israel, do not seem to obtain the desired results and need to implement targeted policy actions to
encourage R&D outputs.
JEL Classification: C44, H50, O30, O57.
Keywords: R&D performance, Data Envelopment Analysis, Cross-efficiency, Ranking, OECD countries.
Introduction
Economic growth has been the central issue for economists and policy-makers for a
long time, and many research efforts have been devoted to understanding the most important
factors behind growth. Neoclassical considerations about the effect of technological progress
on economic growth were followed in the last decades of the past century by new theories of
endogenous growth emphasizing the role of Research and Development (R&D) activities as
important drivers of economic growth (Barro & Sala-i-Martín, 1995).
Subsequent theoretical as well as empirical research on this topic has been undertaken
and a general agreement has been reached on the links between innovations, R&D and
economic growth (Horvath, 2011; Inekwe, 2015; Pessoa, 2010). Technological developments
are claimed to help increase productivity within a context of limited resources (Grossman &
Helpman, 1994) and in that sense, innovation becomes crucial for sustainable growth, social
welfare and quality of life (Akcali & Sismanoglu, 2015). It is also known that R&D
investments represent a major source in fostering knowledge creation and innovation,
therefore, policies based on raising R&D expenditures are expected to have a positive effect
on productivity and competitiveness of nations on the global scale. This idea has encouraged
governments of many countries to significantly increase the level of R&D expenditures in an
effort to enhance their capacity for innovation (Sokolov-Mladenović et al., 2016).
Carrillo, M. (2019). Measuring and ranking R&D performance at the country level. Economics and Sociology, 12(1), 100-114. doi:10.14254/2071-789X.2019/12-1/5
101 Marianela Carrillo ISSN 2071-789X
RECENT ISSUES IN ECONOMIC DEVELOPMENT
Economics & Sociology, Vol. 12, No. 1, 2019
R&D and innovation have also become the key policy components in the EU's agenda
for growth in the current decade. The Europe 2020 strategy established a target of investing
on average 3% of GDP in R&D, as it is understood that greater capacity for R&D in
combination with increased resource efficiency will strengthen the whole economy by
enhancing employment, tertiary educational attainment and innovation, which ultimately
contributes to energy efficiency improvements and sustainability (European
Commission, 2010).
However, a great dispersion in R&D activity is observed across nations. According to
the data provided by the Main Science and Technology Indicators (MSTI) of the OECD
(Organisation for Economic Co-operation and Development), in 2015 the top investors in
R&D were the United States, accounting for 25% of the total R&D investment worldwide,
closely followed by China, a country that has been steadily increasing its volume of R&D
spending during the last decade, and was responsible in 2015 for 21% of global R&D
expenditures. Another indicator that gives a more precise idea of the real efforts that countries
devote to innovation activities is known as R&D intensity and refers to the R&D spending as
percentage of GDP. R&D intensity figures for 2015 show that Israel and Republic of Korea
are indisputable leaders with more than 4% of their GDP being invested in R&D, closely
followed by Japan, Switzerland and Sweden with an R&D spending near 3.3% of their GDP.
To further illustrate the differences across countries, Figure 1 shows R&D expenditures per
capita in relation to GDP per capita for the selected countries that exhibit notable R&D
intensity.
Figure 1. R&D expenditures per capita and GDP per capita for top R&D investors
Source: OECD MSTI
Regardless the amount of the investment effort made in R&D, its main objectives of
stimulating technological progress and fostering economic growth might not be achieved if
R&D resources are used inefficiently (Wang and Huang, 2007). Given the severe financial
constraints that currently affect both public and private sectors, the evaluation of how
efficiently R&D processes are performing becomes of key importance for the allocation of
resources. For this reason, recent years have seen a growing number of studies that attempt to
assess and compare R&D efficiency at different scales of analysis (Aristovnik, 2012; Cullman
& Zloczysti, 2014; Han et al., 2017; Lee & Lee, 2015).
102 Marianela Carrillo ISSN 2071-789X
RECENT ISSUES IN ECONOMIC DEVELOPMENT
Economics & Sociology, Vol. 12, No. 1, 2019
In the measurement of the efficiency of R&D activities, observation units (whether
countries, regions, research institutes or firms) are regarded as entities operating a production
process where a number of inputs, mainly capital and manpower, are transformed to produce
a number of R&D outputs (Wang, 2007), and in this context Data Envelopment Analysis
(DEA) is increasingly being used in the empirical literature (Lee & Shin, 2014).
Efficiency evaluation in DEA relies on a self-appraisal scheme, where units are
allowed to exploit their strengths as to receive the most favorable assessment (Cooper et al.,
2011). As a consequence, poor discrimination is often observed in empirical applications,
meaning that several units are evaluated as efficient performers but being otherwise
indistinguishable. This is particularly inconvenient when an unambiguous ordering of the
observed units according to their performance is needed, a requirement that is becoming more
and more common with the pervasiveness of ranking lists in our society..
Admittedly, rankings can sometimes be controversial, but they have important effects
on decision making and policy making. Particularly when studying the performance of R&D
programs or research institutes, a ranked list is needed for making funding-related decisions.
At a country level, competitiveness demands comparative information for benchmarking
performance against partners or competitors. Besides, given that industries support worldwide
R&D to a great extent, a ranked list of countries according to their R&D performance would
be valuable for managers and stakeholders in defining their strategic goals. However, while
existent studies concerning the efficiency of R&D investments succeed in identifying whether
the countries use their R&D resources efficiently or not, they are unable to produce country
ranking outcomes. This work aims at filling this gap by approaching the evaluation of R&D
efficiency at the country level with the additional objective of obtaining a ranked list of the
countries considered. The sample of our study will be comprised of the countries with a
significant involvement in R&D activities, showing R&D intensity figures above 1%. Using
DEA the technically efficient performers will be identified and then, using cross-efficiency
evaluation, an overall R&D performance score will be obtained that provides a country
ranking.
The remainder of the paper is organized as follows. Next section presents a review of
the existing literature on the evaluation of R&D performance with DEA that evidences a lack
of ranking studies. Section 2 describes the methodology used in this study as well as the
empirical data used. The results of our study are presented in Section 3 and some concluding
remarks are provided in the final section.
1. Literature review
Throughout the past decades, many efforts have been devoted to study the impacts of
R&D and innovation on raising productivity and competitiveness of nations, regions,
industries and firms, and the role of R&D as a driver of economic growth is now sufficiently
well established (Blanco et al. 2016; Goto and Suzuki, 1989; Guloglu and Tekin, 2012;
Hovarth, 2011; Kaur and Singh, 2016) with little attention being generally paid to how
efficiently R&D resources were being used. But whatever the scope of the analysis, the ability
to operate efficiently and the identification of possible inefficiencies become essential for an
optimal allocation of resources as well as for the formulation of strategies for improving R&D
performance (Wang, 2007).
Efficiency analysis studies evaluate the performance of a certain number of production
units in terms of their ability to operate close to or on the boundary of their production set.
Production frontier approaches, which are based on comparing the actual performance of
individual units against the best-practice frontier, are widely used for this purpose, and both
parametric and nonparametric techniques have been adopted in the R&D-related empirical
103 Marianela Carrillo ISSN 2071-789X
RECENT ISSUES IN ECONOMIC DEVELOPMENT
Economics & Sociology, Vol. 12, No. 1, 2019
literature. Parametric methods, such as Stochastic Frontier Analysis (SFA), apply econometric
techniques that require the specification of a functional form for the production frontier, and
inefficiency is modeled as a stochastic term. Nonparametric methods like Data Envelopment
Analysis (DEA) are frequently the preferred option when no functional form is known for the
production function. A major advantage of DEA is that it can successfully handle multiple
inputs and multiple outputs, and it also can deal with situations where prior information on
preferences about the variables involved does not exist, which fits particularly well the
problem of R&D performance evaluation where no universal agreement on the importance of
inputs and outputs exist (Lee, Park and Choi, 2009). Moreover, DEA has been claimed to
have attractive features for the analysis of public sector activities (Zabala-Iturriagagoitia et al.,
2007). Its usefulness in assessing efficiency in Science and Technology has also been
highlighted (Bonaccorsi and Daraio, 2004) and in recent years it has been more and more
adopted for measuring R&D performance (Lee and Shin, 2014). Therefore, although some
interesting cross-country studies based on a parametric R&D efficiency analysis have been
performed (see for example Cullman and Zloczysti, 2014; Fu and Yang, 2009; Wang, 2007) ,
the present research will be focused on DEA.
The original DEA model was developed by Charnes et al. (1978) under the assumption
that production exhibits constant returns to scale (CRS) and it later was extended by Banker et
al. (1984) for its application in a variable returns to scale (VRS) framework. DEA models are
also distinguished by their objective, either minimize inputs given a fixed level of outputs
(input-oriented) or maximize outputs at the current level of inputs (output-oriented).
One of the pioneering applications of DEA in R&D performance analysis can be
found in Rousseau and Rousseau (1998), which elaborates on an earlier contribution by the
same authors. In that work the efficiency of R&D investments in 18 developed countries is
assessed using 1993 data with GDP, active population and R&D expenditures as inputs and
publications and patents as outputs. Their results identified eight efficient countries under the
CRS input-oriented framework, and when additional restrictions on the contribution of inputs
and outputs to the efficiency index were imposed, Switzerland and Netherlands were found to
be the most efficient countries. The authors hinted that using data from the European Patent
Office could negatively affect the results for non-European countries like USA, Canada or
Japan. They also suggested that a time lag between input data and output data should be taken
into account, given that outputs result from inputs occurred some years earlier.
Using an output-oriented formulation, twenty seven nations, most of them OECD
member countries, were analyzed in Lee and Park (2005) under the CRS assumption,
although they suggested that there is no enough evidence on that case and the VRS
framework should be taken into account. They found that six countries, namely Austria,
Finland, Germany, Hungary, New Zealand and the United Kingdom, were performing
efficiently, whereas China, Republic of Korea and Taiwan were lagging in R&D efficiency
and required the implementation of policies to enhance R&D performance to a satisfactory
level.
The analysis performed in Wang and Huang (2007) was based on an input-oriented
VRS DEA formulation. The authors assessed R&D efficiency of 30 developed countries
using R&D expenditures, researchers and technicians as inputs and patents and publications
as output variables and found that about one third of the countries considered achieved
maximum efficiency. They also investigated the effect of environmental factors on R&D
efficiency and concluded that an increase in the national English proficiency indicator and the
rate of higher education enrollment can help improve the R&D performance of the countries.
With a similar model in terms of structure and variables employed, Sharma and
Thomas (2008) also studied the efficiency of the R&D process across 22 nations that included
developed as well as developing economies. According to their results, six countries were
104 Marianela Carrillo ISSN 2071-789X
RECENT ISSUES IN ECONOMIC DEVELOPMENT
Economics & Sociology, Vol. 12, No. 1, 2019
classified as efficient performers, including Japan, Republic of Korea, China, India, Slovenia
and Hungary, while some critical inefficiencies in the use of R&D resources were highlighted
in some developed countries.
Aristovnik (2012) evaluated the R&D performance of a group of 32 selected European
and OECD countries. Efficiency scores computed with an output-oriented DEA formulation
with two inputs and three outputs under the VRS assumption showed that six European
countries were performing efficiently, including Switzerland, Netherlands, Iceland, Hungary,
Cyprus and Turkey, while a relatively high inefficiency level was observed among non-
efficient countries in their sample. In particular they concluded that countries that had recently
been incorporated as EU member states and some of the less-developed OECD economies
showed generally low R&D performance, and therefore the improvement of the sector's
efficiency should be a key priority for those countries.
Further contributions of DEA to the evaluation of R&D efficiency differ in the scope
and focus of the analysis implemented. Kocher et al. (2006) performed a cross-country
analysis focused on the productivity of economic research, while selected European regions
became the target of the analysis in Roman (2010) and Zabala-Iturriagagoitia et al. (2007).
Han et al. (2017) evaluated R&D efficiency across China's high-tech industrial sectors, and
universities or other research institutes were evaluated in Abramo et al. (2011), Lee and Lee
(2015), Meng et al. (2008), Liu and Lu (2010), to name a few. Both CRS and VRS DEA
formulations have been used in applications related to R&D performance, either with an input
or an output orientation, but the output-oriented VRS model has recently become more and
more common (Lee and Lee, 2015).
As can be seen from the above review, a typical feature of all DEA applications is that
very often multiple production units are classified as efficient performers, achieving the most
favorable efficiency score but not allowing further discrimination or ranking among them.
This lack of discrimination in DEA is well documented, particularly when the number of
inputs and outputs is too high in relation to the number of observed units, and some
recommendations have been made to limit the number of variables included in the model and
avoid too many efficient units. Several strategies have been proposed to increase the
discriminative power of DEA (see for example Adler, Friedman and Sinuany-Stern, 2002). In
the context of R&D performance analysis, some attempts have been made to overcome the
discrimination issues of DEA, but either the procedures involved are too intricate for a
practical use (Liu and Lu, 2010) or they fail to achieve complete discrimination (Meng et al.,
2008).
In this work the lack of discrimination in DEA will be handled by means of a cross-
efficiency analysis, which represents an interesting option for deriving a realistic ranking of
the observed units according to their performance. Noteworthy, this kind of analysis has not
been explored before in the R&D literature, to this author's knowledge.
2. Methodology and Data
2.1. DEA and Cross-Efficiency
DEA is a non-parametric technique for measuring the relative efficiency of a
homogeneous set of decision making units (DMUs) that operate in a production system where
multiple inputs are consumed to produce multiple outputs. After four decades of development,
DEA has proved to be a valuable tool for performance evaluation in many different contexts,
including R&D performance. Practical applications of DEA are countless in banking,
healthcare, agriculture, energy, education or finance, among others (Lo Storto & Goncharuk,
2017; Liu, et al., 2013).
105 Marianela Carrillo ISSN 2071-789X
RECENT ISSUES IN ECONOMIC DEVELOPMENT
Economics & Sociology, Vol. 12, No. 1, 2019
Various types of DEA models can be formulated according to the assumptions about
returns to scale (CRS or VRS) and orientation (input or output oriented). Input-oriented
formulations are focused on proportional reduction of inputs without changing the output
levels, while output-oriented efficiency measures study how much outputs can be
proportionally increased with the current level of inputs. In practice, which of these two
measures is more appropriate depends on whether input reduction is more important than
output augmentation (Daraio & Simar, 2007). Since the main objective of R&D can be said to
be increasing outputs, rather than decreasing inputs, we will choose an output orientation. As
for the R&D returns to scale, there is no evidence of CRS in the production of knowledge,
and it has been found that R&D activity can exhibit increasing or decreasing returns to scale
as well as constant returns to scale (Lee, Park & Choi, 2009). Therefore, the VRS DEA
formulation will be used in this study.
DEA builds an efficient frontier using a self-evaluation scheme that allows DMUs to
choose their own weights on inputs and outputs that yield the most favorable efficiency score.
Specifically, assuming that there are n production units or DMUs, each of them being
evaluated in terms of r inputs and s outputs, where xij and ykj are nonnegative values denoting
respectively the amount of input i consumed and the amount of output k produced by the j-th
DMU (i = 1,...r, k = 1,...s, j = 1,...n), the output-oriented VRS model in multiplier form can be
expressed as follows (Lim & Zhu, 2015)
𝑚𝑖𝑛 ∑ 𝑣𝑖𝑞𝑥𝑖𝑞
𝑟
𝑖=1
+ 𝑤𝑞
𝑠. 𝑡. ∑ 𝑢𝑘𝑞𝑦𝑘𝑞
𝑠
𝑘=1
= 1
∑ 𝑣𝑖𝑞𝑥𝑖𝑗 −
𝑟
𝑖=1
∑ 𝑢𝑘𝑞𝑦𝑘𝑗
𝑠
𝑘=1
+ 𝑤𝑞 ≥ 0 𝑗 = 1, . . 𝑛
𝑢𝑘𝑞 , 𝑣𝑖𝑞 ≥ 0, ∀𝑘, 𝑖
(1)
When the above model is solved, an output-oriented (self-evaluated) efficiency score
of DMU q is obtained as the ratio
𝐸𝑞𝑞 =∑ 𝑣𝑖𝑞
∗ 𝑥𝑖𝑞𝑟𝑖=1 + 𝑤𝑞
∗
∑ 𝑢𝑘𝑞∗ 𝑦𝑘𝑞
𝑠𝑘=1
where 𝑢𝑘𝑞∗ , 𝑣𝑖𝑞
∗ , 𝑤𝑞∗ denote the optimal weight values chosen by DMU q in (1). By solving the
above model n times, this process easily distinguishes between efficient units, which obtain a
unitary efficiency value, and inefficient units, which lie away from the efficient frontier.
However, this evaluation scheme may often lead to unreasonable weighting profiles that tend
to overestimate the efficiency scores (Cooper et al., 2011), thus producing many efficient
units that cannot be further discriminated by this means. To overcome this drawback, a cross-
efficiency evaluation can be undertaken.
Cross-efficiency evaluation, originally proposed by Sexton et al. (1986) and further
developed by Doyle and Green (1994), introduced the idea of peer-appraisal in combination
with the traditional DEA self-appraisal mode, in such a way that DMUs are repeatedly
assessed using the range of optimal weights selected by all the peer units instead of their own
set of weights only, and therefore a more thorough evaluation of the performance of the units
is attained. In this way, when the most favorable weights for DMU q are used to evaluate the
other DMUs in a peer-evaluation mode, we obtain output-oriented cross-efficiency scores as:
𝐸𝑗𝑞 =∑ 𝑣𝑖𝑞
∗ 𝑥𝑖𝑗𝑟𝑖=1 + 𝑤𝑞
∗
∑ 𝑢𝑘𝑞∗ 𝑦𝑘𝑗
𝑠𝑘=1
106 Marianela Carrillo ISSN 2071-789X
RECENT ISSUES IN ECONOMIC DEVELOPMENT
Economics & Sociology, Vol. 12, No. 1, 2019
which measures the performance of DMU j when assessed under the perspective of DMU q.
Note that the second constraint in (1) guarantees that 𝐸𝑗𝑞 ≥ 1 for all j, q. Then, by averaging
the self-rated and the peer-rated efficiency scores that each DMU receives, the Average
Cross-Efficiency (ACE) score is obtained that represents an overall performance measure of
the jth DMU and takes into account the preferences over the input and output variables
expressed by all DMUs:
𝐴𝐶𝐸𝑗 =1
𝑛∑ 𝐸𝑗𝑞
𝑛
𝑞=1
This evaluation framework has proved to exhibit strong discrimination properties and
therefore a ranking of the DMUs according to their performance can be obtained for decision
purposes, with the smallest value representing the best performance. Besides, an appealing
feature of the cross-evaluation approach relates to the democratic connotation it enjoys (Doyle
& Green, 1994, p. 570), as all the DMUs play an evaluating role and all the assessments they
make are considered when computing the average cross-efficiency score, which can thus be
considered as a consensual performance index. These advantages have made cross-efficiency
evaluation approaches being profusely used in a vast number of applications (see Ruiz &
Sirvent, 2016 and references therein).
A major limitation in the cross-evaluation approach is the possible existence of
multiple optimal solutions to problem (1), which can lead to non-uniqueness of cross-
efficiency scores 𝐸𝑗𝑞depending on the particular optimal weight pattern used by DMU q for
the assessments. The use of a secondary goal that uniquely determines an optimal bundle of
weights for each DMU was proposed to overcome this drawback (Doyle and Green 1994;
Sexton et al., 1984). These secondary goals could be established either with a benevolent or
aggressive motivation, depending on whether they are intended to (benevolently) improve or
(aggressively) deteriorate the other DMUs' scores while maintaining self-efficiency. Each of
these approaches add slightly different connotations to the cross-evaluation process, although
benevolent formulations can be considered more consistent with the classical DEA principle
based on being as favorable as possible to each DMU (Lim, 2012). Therefore, in the present
work we will compute benevolent average cross-efficiency scores.
2.2. Empirical data
This study is focused on measuring R&D and innovation performance at the country
level. In order to have a comparable group of countries for the analysis, only the countries
with a moderate or high involvement in R&D activities have been included in our sample.
Given the last available Gross Domestic Expenditure on R&D (GERD) figures corresponding
to 2015, all countries exhibiting R&D expenditure above 1% of their Gross Domestic Product
(GDP) in 2015 have been considered. A total of 36 countries met this requirement, although 3
of them (Lithuania, Malaysia and Taiwan) had to be disregarded due to incomplete or
unavailable data. As a consequence, our final sample consists of 33 countries, 30 of them are
members of the OECD and 20 of them are European Union (EU) member states.
Table 1 shows the complete list of the countries considered and the evolution of their
R&D intensity (defined as the ratio of GERD to GDP) over the 2004-2015 time frame
covered in this study. Except for a few countries, most economies show a growing
commitment to R&D investment in the time frame considered. As seen, Israel and Republic
of Korea are the countries devoting the greatest efforts to R&D activities, followed by Japan
and some European countries such as Switzerland, Sweden, Austria and Denmark. Other
countries, despite revealing lower levels of engagement in R&D, have been experiencing
107 Marianela Carrillo ISSN 2071-789X
RECENT ISSUES IN ECONOMIC DEVELOPMENT
Economics & Sociology, Vol. 12, No. 1, 2019
important increments in their R&D intensity figures in the last decade, such as China, Slovak
Republic, Poland or Estonia.
The process of innovation production is complex and multiple factors are involved. In
order to appropriately approach innovation performance measurement, our selection of input
and output variables for this study has been based on the previous literature, which shows a
general agreement in relation to the relevant factors. The inputs to innovation activities
considered in previous studies are mainly physical resources and human capital, which have
been proxied here by GERD, expressed in purchasing power parity (PPP) million $, and Total
Researchers, expressed in full-time equivalent units, respectively. The corresponding data
were collected from the OCDE Main Scientific Technology Indicators (MSTI) database.
Table 1. R&D intensity for the sample considered
R&D intensity
Country 2004 2015
Australia 1,7 1,9
Austria 2,2 3,0
Belgium 1,8 2,5
Canada 2,0 1,6
China 1,2 2,1
Czech Republic 1,1 1,9
Denmark 2,4 3,0
Estonia 0,9 1,5
Finland 3,3 2,9
France 2,1 2,3
Germany 2,4 2,9
Hungary 0,9 1,4
Iceland 2,7 2,2
Ireland 1,2 1,2
Israel 3,9 4,3
Italy 1,1 1,3
Japan 3,0 3,3
Republic of Korea 2,5 4,2
Luxembourg 1,6 1,3
Netherlands 1,8 2,0
New Zealand 1,1 1,3
Norway 1,5 1,9
Poland 0,6 1,0
Portugal 0,7 1,2
Russia 1,1 1,1
Singapore 2,1 2,2
Slovak Republic 0,5 1,2
Slovenia 1,4 2,2
Spain 1,0 1,2
Sweden 3,4 3,3
Switzerland 2,7 3,4
United Kingdom 1,5 1,7
United States 2,5 2,7
Source: OECD MSTI
As far as the outputs is concerned, patent number is the most widely adopted measure
of innovative output, and academic publications are also frequently used to evaluate
108 Marianela Carrillo ISSN 2071-789X
RECENT ISSUES IN ECONOMIC DEVELOPMENT
Economics & Sociology, Vol. 12, No. 1, 2019
performance of researchers and as an indicator of scientific knowledge (Lee et al., 2009).
Moreover, as high-tech products are defined as products involving a high intensity of R&D,
the volume of high-tech exports can be also considered as an output of the R&D process that
captures the economic benefits of the results of innovation (Matei & Aldea, 2012). Therefore,
the outputs used in this study comprise the number of patents granted to residents, the number
of scientific publications and level of high-tech exports (including pharmaceutical, aerospace,
computer, electronic and optical industries), expressed in PPP million $, which have been
collected from the World Intellectual Property Organization (WIPO), SCImago and MSTI,
respectively.
Lastly, it is known that a certain length of time is required for the inputs of the R&D
process to be transformed into outputs, and the importance of taking this factor into account
when measuring R&D efficiency has been previously acknowledged (Sharma and Thomas,
2008). Preliminary tests have shown that a 3-year lag is appropriate in the study of relative
efficiency of R&D processes at the country level based on aggregate data (Wang and Huang,
2007). According to this, the input variables in our study will be lagged by 3 years as
compared to the outputs. Our analysis will be then carried out for the periods 2004-2007 and
2012-2015 to gain a perspective of where the countries stand on the overall R&D picture and
how the situation has changed during the last decade.
3. Results
Using the methodology described above, the R&D efficiency of our selected sample of
33 countries has been evaluated in both time periods considered. Columns 2 and 3 in Table 2
report the efficiency scores obtained with an output-oriented VRS DEA model based on the
input and output variables selected in the preceding section. Figures in parentheses indicate
the rank order of each country according to the computed efficiency score, where all efficient
countries tie for first ranking position. As seen, 16 out of the 33 countries studied are found to
perform efficiently in 2015 and up to 18 countries are considered efficient performers in
2007, a result that confirms the low discrimination power of this method.
Among the countries that are evaluated as efficient throughout the whole time frame
we find ten European countries (Germany, Iceland, Ireland, Italy, Luxembourg, Netherlands,
Poland, Slovakia, Switzerland and United Kingdom) and four Asian countries (China, Japan,
Republic of Korea, and Singapore) with New Zealand and United States ending the list.
Within that group, the United States, China, Japan, Germany and Republic of Korea are the
top 5 world investors in R&D, which together are responsible for 65% of the global R&D
expenditure.
Australia, Czech Republic and Portugal can be said to perform near efficiency in the
2012-2015 period, while the remaining 14 countries still have some room for improvement
and a higher level of outputs can still be obtained with the quantities of inputs being currently
employed, suggesting the need to implement specific policies for encouraging R&D
outcomes. Special attention deserves the score obtained for Israel, which shows the worst
efficiency score in the table despite being the country with the world’s largest share of GDP
devoted to R&D, implying that there is much more to R&D performance than spending
levels. In this case, DEA results suggest that Israel should be able to increase its outputs by
83% with the actual input levels.
With nearly half the countries in the sample being classified as efficient, it is difficult
to further compare their performance. In order to obtain a complete ranked list of countries,
we compute the Average Cross-Efficiency (ACE) scores as described in the preceding section,
which are reported in columns 4 and 5 of Table 2 with the corresponding ranking position in
parenthesis. According to these results, United Kingdom, Switzerland and Netherlands are the
109 Marianela Carrillo ISSN 2071-789X
RECENT ISSUES IN ECONOMIC DEVELOPMENT
Economics & Sociology, Vol. 12, No. 1, 2019
3-top performers in the time period 2012-2015, with similar ranking positions being earned in
the earlier period 2004-2007. On the contrary, Luxembourg, Estonia and Iceland are the three
worst countries as regards R&D performance in both time periods considered, a result that
should call the attention of policy-makers.
Table 2. VRS DEA scores and ACE scores, with rankings in parenthesis
VRS DEA scores ACE scores
Country 2004-2007 2012-2015 2004-2007 2012-2015
Australia 1,10 (21) 1,04 (18) 1,44 (7) 1,34 (5)
Austria 1,60 (29) 1,72 (31) 2,16 (20) 2,04 (22)
Belgium 1,19 (23) 1,23 (22) 1,57 (10) 1,52 (11)
Canada 1,29 (26) 1,24 (23) 1,51 (8) 1,36 (7)
China 1,00 (1) 1,00 (1) 1,74 (14) 1,81 (17)
Czech Republic 1,02 (19) 1,09 (19) 1,81 (15) 1,66 (14)
Denmark 1,48 (28) 1,33 (29) 2,21 (21) 1,72 (16)
Estonia 1,00 (1) 1,26 (24) 10,9 (31) 6,96 (32)
Finland 1,76 (31) 1,45 (30) 2,54 (25) 2,11 (24)
France 1,27 (25) 1,22 (21) 1,72 (13) 1,64 (12)
Germany 1,00 (1) 1,00 (1) 1,61 (11) 1,67 (15)
Hungary 1,02 (20) 1,32 (28) 2,08 (19) 2,45 (26)
Iceland 1,00 (1) 1,00 (1) 20,7 (32) 13,31 (33)
Ireland 1,00 (1) 1,00 (1) 1,92 (17) 1,65 (13)
Italy 1,00 (1) 1,00 (1) 1,15 (3) 1,13 (4)
Israel 1,85 (33) 1,83 (33) 2,38 (22) 2,38 (25)
Japan 1,00 (1) 1,00 (1) 2,63 (27) 2,51 (27)
Republic of Korea 1,00 (1) 1,00 (1) 1,52 (9) 1,93 (18)
Luxembourg 1,00 (1) 1,00 (1) 21,75 (33) 5,58 (31)
Netherlands 1,00 (1) 1,00 (1) 1,14 (1) 1,11 (3)
New Zealand 1,00 (1) 1,00 (1) 2,38 (23) 2,09 (23)
Norway 1,41 (27) 1,28 (25) 2,46 (24) 1,97 (20)
Poland 1,00 (1) 1,00 (1) 1,70 (12) 1,39 (8)
Portugal 1,22 (24) 1,03 (17) 2,56 (26) 2,02 (21)
Russia 1,77 (32) 1,79 (32) 5,24 (30) 3,25 (29)
Singapore 1,00 (1) 1,00 (1) 1,30 (5) 1,43 (9)
Slovak Republic 1,00 (1) 1,00 (1) 3,77 (28) 2,79 (28)
Slovenia 1,00 (1) 1,29 (26) 4,43 (29) 3,74 (30)
Spain 1,11 (22) 1,11 (20) 1,44 (6) 1,34 (6)
Sweden 1,60 (30) 1,30 (27) 1,91 (16) 1,49 (10)
Switzerland 1,00 (1) 1,00 (1) 1,16 (4) 1,06 (2)
United Kingdom 1,00 (1) 1,00 (1) 1,14 (2) 1,05 (1)
United States 1,00 (1) 1,00 (1) 2,04 (18) 1,96 (19)
Source: own computation
Also, the fact that Iceland and Luxembourg rank at the very last positions of the
ordered list despite being classified as efficient with the VRS DEA model draws particular
110 Marianela Carrillo ISSN 2071-789X
RECENT ISSUES IN ECONOMIC DEVELOPMENT
Economics & Sociology, Vol. 12, No. 1, 2019
attention. Other DEA-efficient countries such as Japan, Slovak Republic or New Zealand also
appear in the last quartile of the ranked list obtained in terms of ACE score, while the inverse
situation is observed for Israel, a country that improved its position from 33rd place when
using DEA score up to the 25th spot in the ordering derived from ACE score. These are not
uncommon phenomena when shifting from a self-assessment framework, where DMUs may
appear efficient in ways that are difficult to justify (Cooper et al., 2011), to a peer-assessment
framework where multiple evaluations are averaged into an agreed summary performance
score.
Certainly, it is well-established that since DEA scores are based on an endogenous and
quite flexible weighting system, some DMUs can achieve efficiency by selecting highly
specialized input and output weights that enhance their strengths and ignore their weaknesses,
and the existence of many efficient DMUs is not necessarily a result of uniformly outstanding
performance but of some units using an unusual mix of inputs and outputs (Wang & Huang,
2007; Ruiz & Sirvent, 2016). Cross-efficiency evaluation suitably overcomes this issue by
relying on a peer-evaluation scheme, and since all the DMUs are assessed with a range of
weight profiles, units that achieved efficiency with highly specialized set of weights will
generally obtain poor rates from their peers. In that sense, cross-efficiency scores can be said
to approximate actual performance of countries more realistically (Martić & Savić, 2001)
A comparative analysis of the ranking positions attained by the countries in the time
periods considered can provide useful insights for benchmarking developments against best-
performer economies and defining future courses of action. In this sense, it can be seen that
Sweden underwent the most notable improvement in the 2004-2015 time frame, moving up
six positions in the ranking list to the 10thplace and outperforming some world R&D leaders
such as Germany, which lost two ranking positions in this period, or Republic of Korea,
which shows the most notable backward shift losing 7 positions from the 9thdown to
18thplace. This suggests that some countries are more successful than others in the strategic
management of their R&D resources. Other EU countries also improved their position in the
list, such as Denmark and Portugal, which moved up 5 ranking places, or Poland and Ireland,
who climbed up 4 spots. Although except Poland, they still rank out of the top 10 positions,
their advancement shows their efforts in raising the efficiency of their R&D processes. The
same tendency is not observed, however, in all EU countries, and we can find that Hungary
and Germany have dropped seven and four places, respectively, revealing the need to revise
their R&D policies.
Other economies have not experienced significant shifts in their ranking positions
along the studied period. Besides the top-3 performers, countries such as Australia, Canada,
Italy, Singapore and Spain have been able to remain roughly stable in the top ten, suggesting
that a certain degree of stability has been reached in their R&D processes. On the other hand,
Japan, Slovak Republic, Slovenia, Russia, Estonia, Iceland and Luxembourg systematically
take the last seven ranking places and therefore policy initiatives for an efficient management
of R&D processes are key for these countries. In the middle of the table, the picture has
remained practically unaltered during the time span studied for Czech Republic, Finland,
France, New Zealand and United States.
Another remarkable observation is that the most important contributors to global R&D
expenditure are not necessarily ranked as world-top performers, with some lower R&D-
intensive economies, such as Poland, Spain or Italy ranking higher than some of the largest
world’s investors such as the United States, China or Republic of Korea. This might be
explained by the increased degree of coordination and bureaucratic control that larger-scale
R&D processes require, making management activities monopolize a great deal of resources
(Zabala-Iturriagagoitia et al., 2017).
111 Marianela Carrillo ISSN 2071-789X
RECENT ISSUES IN ECONOMIC DEVELOPMENT
Economics & Sociology, Vol. 12, No. 1, 2019
Conclusion
Given the importance of R&D activities as a driver for growth and competitiveness,
and with acute financial constraints affecting both the public and private sectors, there is an
increasing interest in measuring how efficiently R&D resources are being used. Following the
path undertaken in previous studies, this work evaluates and compares the R&D performance
of 33 countries over the period 2004-2015.
Using an output-oriented Data Envelopment Analysis model, we find that 16 out of the
33 countries considered were operating in the production frontier. Since DEA does not allow
further discrimination among the efficient units, a cross-efficiency evaluation approach is
carried out to obtain an overall score that enables comparison and ranking of the countries
with regard to their R&D performance. Cross-efficiency scores can be said to represent a
quite realistic performance measure due to its peer-assessment foundation, as opposed to the
self-assessment scheme used to define DEA scores that produces biased, overestimated
values.
As to this author's knowledge, the application of a cross-efficiency approach for the
evaluation of country R&D performance is unknown in the literature, with prior studies
focused on the classification of countries (or firms) into efficient and non-efficient
performers. However, a ranked list of countries provides useful managerial insights for the
different agents involved in R&D processes. On one side, governments constantly demand
information about how their own country performs in different areas, including R&D, and
how their performance compares to that of their partners or competitors. Rankings are thus a
useful tool in this setting to monitor performance over time and to benchmark developments
against outstanding economies. On the other side, funding organizations also make use of this
kind of ordered lists to estimate the risk of their investments and make informed decisions.
Therefore, the study proposed here deals with a relevant issue for R&D management.
In summary, the results obtained with the cross-efficiency approach indicate that
Switzerland, United Kingdom and Netherlands are the three leading countries in the
considered period as far as R&D performance is concerned, and therefore they are not
expected to be able to significantly expand their R&D outputs with their current level of
inputs. Some EU countries such as Sweden, Denmark and Poland have experienced important
improvements in their performance relative to the other countries, suggesting that they are
successfully implementing strategies for raising their R&D efficiency. On the contrary, a
notable shift to the lower-half positions of the list is experienced by Republic of Korea, in
spite of being the country with the second highest R&D intensity in 2015. This indicates that
the increasing efforts of Republic of Korea in R&D are not having the expected results and
there is still room for improving the effectiveness of R&D policies in that country.
Another remarkable observation is that countries that lead the worldwide contribution
to GERD like United States, China or Republic of Korea do not always earn as high a ranking
position as expected with the ACE performance score, a fact that could be explained by the
increasing resources that must be devoted to managing such large-scale R&D activities.
Therefore, the key implication of this analysis is that strategic management of R&D
resources is critical for achieving the desired results. By monitoring country performance over
time, policy makers should identify strengths and weaknesses in their R&D processes,
formulate specific strategies for fostering innovation and implement targeted policies when
required.
A limitation of this study relates to data availability. On the one hand, an important
R&D investor such as Taiwan has been excluded from our analysis due to a lack of
information regarded the variables considered. On the other hand, although the quantitative
side of R&D outputs has been covered, qualitative issues have not been taken into account,
112 Marianela Carrillo ISSN 2071-789X
RECENT ISSUES IN ECONOMIC DEVELOPMENT
Economics & Sociology, Vol. 12, No. 1, 2019
despite the fact that the quality of patents and publications are acknowledged as essential for
technological progress and knowledge creation. Since it is known that the inputs and outputs
included in a DEA model may have significant effects on the results, this aspect will have to
be considered in future research.
References
Abramo,G., Cicero, T., & D’Angelo, C:A. (2011). A field-standardized application of DEA to
national-scale research assessment of universities. Journal of Informetrics, 5(4), 618-
628. doi: 10.1016/j.joi.2011.06.001
Adler, N., Friedman, L., & Sinuany-Stern, Z. (2002). Review of ranking methods in the data
envelopment analysis context. European Journal of Operational Research, 140(2), 249-
265. doi: 10.1016/S0377-2217(02)00068-1
Akcali, B.Y., & Sismanoglu, E. (2015). Innovation and the effect of Research and
Development (R&D) expenditure in growth in some developing and developed
countries. Procedia – Social and Behavioral Sciences, 195(3), 768-775. doi:
10.1016/j.sbspro.2015.06.474
Aristovnik, A. (2012). The relative efficiency of education and R&D expenditures in the new
EU member states. Journal of Business Economics and Management, 13(5), 832-848.
doi:10.3846/16111699.2011.620167
Banker, R. D., Charnes, A., & Cooper, W. W. (1984). Some models for estimating technical
and scale inefficiencies in data envelopment analysis. Management Science, 30(9),
1078–1092.
Barro, R.J., & Sala-i-Martín, X. (1995). Economic growth. McGraw-Hill, New York.
Bonaccorsi, A. and Daraio, C. (2004). Econometric approaches to the analysis of productivity
of R&D systems. In Moed, H.F., Glänzel, W. & Schmoch, U. (Eds), Handbook of
quantitative Science and Technology Research. (pp. 51-74). Kluwer Academic
Publishers, New York.
Charnes, A., Cooper, W. W., & Rhodes, E. (1978). Measuring the efficiency of decision
making units. European Journal of Operational Research, 2(6), 429-444. doi:
10.1016/0377-2217(78)90138-8
Cooper, W.W., Ruiz, J.L., & Sirvent, I. (2011). Choices and uses of DEA weights. In Cooper,
W.W, Seiford, L.M. & Zhu, J. (Eds), Handbook of Data Envelopment Analysis,
International Series in Operations Research and Management Science, 164, (pp. 93-
126). Springer, New York.
Cullman, A.,& Zloczysti, P. (2014). R&D efficiency and heterogeneity-a latent class
application for the OECD. Applied Economics, 46(30), 3750-3762. doi:
10.1080/00036846.2014.939410
Daraio, C., & Simar, L. (2007). Advanced Robust and Nonparametric Methods in Efficiency
Analysis. New York, NY: Springer.
Doyle, J.,& Green, R (1994). Efficiency and Cross-Efficiency in DEA: Derivations, meaning
and uses. Journal of the Operational Research Society, 45(5), 567-578.
European Commission (2010). Europe 2020. A strategy for smart, sustainable and inclusive
growth. Retreived June 19, 2018, from http://eur-
lex.europa.eu/LexUriServ/LexUriServ.do?uri= COM:2010:2020:FIN:EN:PDF.
Fu, X.,& Yang, Q.G. (2009). Exploring the cross-country gap in patenting: A Stochastic
Frontier Approach. Research Policy, 38(7), 1203-1213. doi:
10.1016/j.respol.2009.05.005
113 Marianela Carrillo ISSN 2071-789X
RECENT ISSUES IN ECONOMIC DEVELOPMENT
Economics & Sociology, Vol. 12, No. 1, 2019
Goto, A., & Suzuki, K. (1989). R&D Capital, Rate of Return on R&D Investment and
Spillover of R&D in Japanese Manufacturing Industries. The Review of Economic and
Statistics, 71(4), 555-564.
Grossman, G.M., & and Helpman, E. (1994). Endogenous innovation in the theory of growth.
Journal of Economic Perspectives, 8(1), 23-44.Guloglu, B., & Tekin, R.B. (2012). A
Panel Causality Analysis of the Relationship among Research and Development,
Innovation, and Economic Growth in High-Income OECD Countries. Eurasian
Economic Review, 2(1), 32-47. doi: 10.14208/BF03353831
Han, C., Thomas, S.R., Yang, M., Ieromonachou, P.,& Zhang, H. (2017). Evaluating R&D
investment efficiency in China’s high-tech industry. Journal of High Technology
Management Research, 28(1), 93-109. doi: 10.1016/j.hitech.2017.04.007
Horvath, R. (2011). Research & development and growth: a bayesian model averaging
analysis. Economic Modelling, 28(6), 2669-2673. doi: 10.1016/j.econmod.2011.08.007
Inekwe, J.N. (2015). The contribution of R&D expenditure to economic growth in developing
economies. Social Indicators Research, 124(3), 727-745. doi: 10.1007/s11205-014-
0807-3
Kaur, M., & Singh, L. (2016). R&D expenditure and economic growth: An empirical
analysis. International Journal of Technology Management & Sustainable
Development, 15(3), 195-213. doi: 10.1386/tmsd.15.3.195_1
Kocher, M. G., Luptacik, M., & Sutter, M. (2006). Measuring productivity of research in
economics: A cross-country study using DEA. Socio-Economic Planning Sciences,
40(4), 314-332. doi: 10.1016/j.seps.2005.04.001
Lee, S., & Lee, H. (2015). Measuring and comparing the R&D performance of government
research institutes: A bottom-up data envelopment analysis approach. Journal of
Informetrics, 9(4), 942-953. doi: 10.1016/j.joi.2015.10.001
Lee, H., & Park, Y. (2005). An international comparison of R&D efficiency: DEA approach.
Asian Journal of Technology Innovation, 13(2), 207-222. doi:
10.1080/19761597.2005.9668614
Lee, H., Park, Y., & Choi, H. (2009). Comparative evaluation of performance of national
R&D programs with heterogeneous objectives: A DEA approach. European Journal of
Operational Research, 196(3), 847-855. doi: 10.1016/j.ejor.2008.06.016
Lee, H.,& Shin, J. (2014). Measuring journal performance for multidisciplinary research: An
efficiency perspective. Journal of Informetrics, 8(1), 77-88. doi:
10.1016/j.joi.2013.10.004
Lim, S. (2012). Minimax and maximin formulations of cross-efficiency in DEA. Computers
& Industrial Engineering, 62(3), 726–731. doi:10.1016/j.cie.2011.11.010 Lim, S., & Zhu, J. (2015). DEA Cross Efficiency under variable returns to scale. In Zhu, J.
(Ed), Data Envelopment Analysis. A handbook of models and methods, International
Series in Operations Research and Management Science, 221, (pp.45- 66). Springer,
New York.
Liu, J.S.,& Lu, W.M. (2010). DEA and ranking with the network-based approach: a case of
R&D performance. Omega, 38(6), 453-464. doi:10.1016/j.omega.2009.12.002
Liu, J.S., Lu, L.Y.Y., Lu, W.M., & Lin, B.J.Y. (2013). A survey of DEA applications.
Omega, 41(5), 893-902. doi: 10.1016/j.omega.2012.11.004
Lo Storto, C., & Goncharuk, A.G. (2017). Efficiency vs Effectiveness: a Benchmarking Study
on European Healthcare Systems. Economics and Sociology, 10(3), 102-115.
doi:10.14254/2071-789X.2017/10-3/8
Martić, M., &Savić, G. (2001). An application of DEA for comparative analysis and ranking
of regions in Serbia with regards to social-economic development. European Journal of
the Operational Research, 132(2), 343-356. doi: 10.1016/S0377-2217(00)00156-9
114 Marianela Carrillo ISSN 2071-789X
RECENT ISSUES IN ECONOMIC DEVELOPMENT
Economics & Sociology, Vol. 12, No. 1, 2019
Matei, M. M., & Aldea, A. (2012). Ranking National Innovation Systems according to their
technical efficiency. Procedia-Social and Behavioral Sciences, 62, 968-974. doi:
10.1016/j.sbspro.2012.09.165
Meng, W., Zhang, D., Qi, L., & Liu, W. (2008). Two-level DEA approaches in research
evaluation. Omega, 36(6), 950-957. doi: 10.1016/j.omega.2007.12.005
Pessoa, A. (2010). R&D and economic growth: how strong is the link? Economics letters,
107(2), 152-154. doi:10.1016/j.econlet.2010.01.010
Roman, M. (2010). Regional efficiency of knowledge economy in the new EU countries: The
Romanian and Bulgarian case. MPRA Paper No. 23083. Retrieved June, 7, 2018, from
https://mpra.ub.uni-muenchen.de/23083/
Rousseau, S., & Rousseau, R. (1998). The scientific wealth of European nations: Taking
effectiveness into account. Scientometrics, 42(1), 75-87.
Ruiz J.L., & Sirvent, I. (2016). Ranking Decision Making Units: The Cross-Efficiency
Evaluation. In Hwang, S.N., Lee, H.S. and Zhu, J. (Eds), Handbook of Operations
Analytics Using Data Envelopment Analysis, International Series in Operations
Research & Management Science, 239 (pp. 1-29). Springer, New York.
Sexton, T.R., Silkman, R.H., & Hogan, A.J. (1986). Data Envelopment Analysis: Critique and
Extensions. New Directions for Program Evaluation, Volume 1986, issue 32, 73-105.
doi:10.1002/ev.1441
Sharma, S., & Thomas, V.J. (2008). Inter-country R&D efficiency analysis: an application of
data envelopment analysis. Scientometrics, 76(3) 483-501. doi: 10.1007/s11192-007-
1896-4
Sokolov-Mladenović, S., Cvetanović, S., & Mladenović, I. (2016). R&D expenditure and
economic growth: EU 28 evidence for the period 2002-2012. Economic Research-
Ekonomska Istrazivanja, 29(1), 1005-1020. doi: 10.1080/1331677X.2016.1211948
Wang, E.C. (2007). R&D efficiency and economic performance: A cross-country analysis
using the stochastic frontier approach. Journal of Policy Modeling, 29(2), 345-360. doi:
10.1016/j.jpolmod.2006.12.005
Wang, E.C., & Huang, W. (2007). Relative efficiency of R&D activities: A cross-country
study accounting for environmental factors in the DEA approach. Research Policy,
36(2), 260-273. doi: 10.1016/j.respol.2006.11.004
Zabala-Iturriagagoitia J.M., Voigt, P., Gutiérrez-Gracia, A., & Jiménez-Sáez, F. (2007).
Regional Innovation Systems: How to Assess Performance, Regional Studies, 41(5),
661-672. doi: 10.1080/00343400601120270