1

Understanding Evaluation of STI Policy in a Multi-level Governance Context

Karlsruhe, 13th June 2012

2

• STI Policy itself has increased in complexity due to:1. Broadening of the innovation concept2. Evolution from linear to systemic views of innovation (Lundvall,

1992; Nelson, 1993)3. Inclusion of the policy mix concept in innovation field (Flanagan et

al., 2011)• Regions can be considered policy spaces, in which policies

administered at different levels are being felt (Uyarra and Flanagan, 2010)

• In this context, multi-level governance of STI Policy constitutes a key issue and evaluation an important stage for policy-learning purposes.

Motivation

3

• Policy evaluation can be used for understanding this complexity through the concept of additionality

• We can broadly distinguish between 3 different types of additionalities: 1. input 2. output3. behavioural

• Most existing STI policy evaluation doesn’t capture all of the policy complexity.

• Therefore regions or any other policy space might be defining STI policies without taking into account potential effects derived from interactions among rationales, domains and instruments from different administrative levels.

Policy evaluation and additionality concepts

4

• The Basque region is a very interesting case study and a good example of complexity

• Analysis of a specific regional R&D programme with impact on firms, versus analysis of the impacts of the whole funding system on Basque firms

• Measurement of input, output and behavioural additionality

Basque region as policy space

5

• Triangulation of methods: Matching approach and semi-structured interviewso Matching approach: Comparison of two groups: firms that have

received an R&D subsidy with innovative firms that haven’t received any.

o Semi-structured interviews with some beneficiaries as a complementary technique

• Focus of the analysis: Firms that have received any subsidy between 2001 and 2004 and their results from 2007-onwards

Methodology

6

Methodology

• Use of propensity score matching to find similar firms in both the treated group (subsidized firms) and the control group (innovative but non-subsidized firms according to a set of individual characteristics) (Rosenbaum and Rubin, 1983; Dehejia and Wahba, 2002)

• Use the Nearest Neighbour Matching to match similar units in both groups according to their propensity score

• Calculate the ATT (Average Effect of Treatment on the Treated) through the difference between the average of the dependent variable in both groups.

• Given this, it is possible to calculate the causal effect of the treatment:

}{}1{)( XDEXDPXp

CJ

TjjT

Ti

TiT

NNM YwN

YN

11

)1()1()( 01 iiii TYETYEE

We have run the model twice: for analysing the impact of the regional programme in isolation and the impact of the whole funding system (for the same amount of subsidies received)

7

Results for the whole population of firms

Controlling for other R&D programmes (national & local) Without controlling for other R&D programmes

Variable(log)

Expected coefficient sign

Expected mean from the matched sample(SD. in parentheses) (* significant at 95%)

Differences in averages (firms subsidized by Intek- control firms) Mean (SD. in parentheses)(*significant at 95%)

Expected coefficient sign

Expected mean from the matched sample (SD. in parentheses) (* significant at 95%)

Differences in averages (firms subsidized by Intek- control firms) Mean (SD. in parentheses) (*significant at 95%)

Total R&D expenditure +

+ 2.9087(*)(1.0350)

-0.03290 (0.06281) +

+1.7288 (*)(0.3669)

0.73215 (*)(0.07422)

R&D intensity ++ 0.20326(0.13079)

-0.07085(*)(0.01616) +

+ 0.10687(0.10195)

0.01288(0.01317)

R&D employees in the Basque Country (FTE) +

+0.63063 (0.3625)

0.0382706 (*)(0.0198618) +

+ 0.235450(*)(0.142805)

0.3270 (*)(0.2818)

Number of PhDs in the whole firm +

+ 0.06199(0.04394)

-0.006315(0.008215) +

+0.02914(0.02984)

0.024913 (*)(0.006307)

Input additionality:

Output additionality: Controlling for other R+D programmes (national & local) Without controlling for other R&D programmes

Variable

Expected coefficient sign

Expected mean from the matched sample(SD. in parentheses) (* significant at 95%)

Differences in averages (firms subsidized by Intek- control firms) Mean (SD. in parentheses)(*significant at 95%)

Expected coefficient sign

Expected mean from the matched sample (SD. in parentheses) (* significant at 95%)

Differences in averages (firms subsidized by Intek- control firms) Mean (SD. in parentheses) (*significant at 95%)

Productivity (log) ++ 2.0645 (*)(0.5403)

0.18070(*)(0.09763) +

+1.8083(*)(0.3835)

0.38977 (*)(0.06988)

Patents (num) ++0.03315 (0.04978)

0.15022 (*)(0.01774) +

+ 0.02910(0.04392)

0.16183(*)(0.01293)

Patentes (probability of patent) +

+0.01970(0.02960)

0.057807 (*)(0.009587) +

+0.01679(0.02623)

0.064494(*)(0.007204)

8

Controlling for other R+D programmes (national & local) Without controlling for other R&D programmes

Variable

Expected coefficient sign

Expected mean from the matched sample(SD. in parentheses) (* significant at 95%)

Differences in averages (firms subsidized by Intek- control firms) Mean (SD. in parentheses)(*significant at 95%)

Expected coefficient sign

Expected mean from the matched sample (SD. in parentheses) (* significant at 95%)

Differences in averages (firms subsidized by Intek- control firms) Mean (SD. in parentheses) (*significant at 95%)

R&D systematization +

+ 0.217078(0.112531)

-0.04127(0.03443) +

+ 0.32557(0.09402)

-0.13087(*)(0.02579)

EU(log€) ++ 0.2342(0.1815)

0.01766(0.02538) +

+ 0.10522(0.15460)

0.12508 (*)(0.01978)

PhDs/total employment +

+ 0.009044(0.015052)

0.012230 (*)(0.002006) +

+ 0.006391(0.016156)

0.012120(*)(0.001552)

Results for the whole population of firms

Behavioural additionality:

9

Results: Additionalities achieved in collaborative firms

Controlling for other R+D programmes (national & local) Without controlling for other R&D programmes

VariableExpected coefficient sign

Expected mean from the matched sample(SD. in parentheses) (* significant at 95%)

Differences in averages (firms subsidized by collaborative Intek projects- control firms) Mean (SD. in parentheses)(*significant at 95%)

Expected coefficient sign

Expected mean from the matched sample (SD. in parentheses) (* significant at 95%)

Differences in averages (firms subsidized by collaborative Intek projects- control firms) Mean (SD. in parentheses) (*significant at 95%)

Total R&D expenditure +

+ 2.6135788(*)(1.3729163)

0.03029(0.09158) +

+2.0158 (0.9482)

0.34444(0.18365)

R&D employees in the Basque Country (FTE) +

+ 0.6456(0.4442)

-0.009584(0.038343) +

+0.4512(0.2838)

0.08376(0.06673)

Number of PhDs in the whole firm +

+0.08702(0.09255)

-0.02632 (0.02301) +

+ 0.07107(0.09030)

-0.01768(0.02353)

R&D intensity ++ 0.1257(0.1386)

0.047739(0.028338) +

+ 0.1153(0.1281)

0.052973(0.029321)

Productivity (log) +

+ 1.9810 (*)(0.8843)

0.1507(0.1494) +

+1.7711(*)(0.7021)

0.25348(0.16148)

Patents (num) +

+ 0.08432(0.28174)

0.04181(0.09125) +

+0.05109(0.26250)

0.07954(0.08584)

Patents (prob to patent) +

+0.04795(0.13152)

0.01061(0.03214) +

+ 0.03859(0.12307)

0.01997(0.03187)

R&D systematization +

+0.3611(0.2933)

-0.18094(0.05677) +

+0.4066(0.2530)

-0.21743 (*)(0.05459)

EU(log€) ++ 0.03757(0.11801)

0.32363 (*)(0.03853) +

+ 0.01254(0.10281)

0.29160 (*)(0.03799)

PhDs/ total employment +

+ 0.01431(0.03404)

0.015709(0.008898) +

+ 0.01150(0.03090)

1.708e-02(8.937e-03)

Higher effect of the regional programme in isolation

10

Synthesis of Results

Regional Policy Multi-level Policies

All Firms • Low input additionality or crowding out effects• Output additionality• Low Behavioural additionality

• Multiplier effects on input, output and behavioural additionalities

Firms that collaborate

• No input and output additionality• Behavioural additionality

• No input and output additionality• Low behavioural additionality or crowding out effects

11

Conclusions and implications for public policy

• Evaluation needs to evolve in order to capture complexity at policy level perspective

• Interaction of policies administered at different levels have a multiplier effect on regions so it is appropriate to consider regions as policy spaces.

• STI policy has different effects depending on the administrative level it has been designed: • Input and output additionality could be better achieved by other STI policies

different to regional ones (mainly national policies)• Behavioural additionality are achieved by STI regional policies, which maximise the

value of proximity for innovation

• In consequence, multi-governance of STI policies is necessary to take the best from each level

• Regions have to develop capacities to adequately design and implement regional STI policies to avoid risks of considering policy effects in isolation and not from a systemic perspective

12

Thank you

edurne.magro@orkestra.deusto.es

www.orkestra.deusto.es