Maastricht Economic and Social Research and Training Centre on Innovation and Technology (UNU-MERIT)...

Maastricht Economic and Social Research and Training Centre on Innovation and Technology (UNU-MERIT)

and FH Aachen, University of Applied Sciences

Science, Technology and Innovation (STI) Indicators

for Evaluation

Professor Norbert Janz

DEIP Montevideo30 March 2009

DEIP Montevideo 2009 - Indicators 30 March 2009 - 2

STI Indicators

Measuring Innovation: Manuals upon Manuals

Typology of Indicators

Innovation Surveys

Use of Innovation Data


Measuring Innovation:Manuals upon Manuals

OECD Manuals Frascati manual How to measure R&D Oslo manual How to measure innovation Canberra manual How to use human

resource data to measure innovation

Patent Statistics How to use patent data to manual measure innovation

Non-OECD Manuals Bogota manual How to measure innovation in

Latin American countries Santiago manual Internationalisation of R&D … NEPAD study How to measure innovation in

African countries


Frascati Manual

History of the manualFirst edition 1963, last revision 2002

Scope of the manualR&D statistics

Sampling approachBusiness enterprises, government, non-profit,

higher education, hospitals/health careSurvey of all enterprises known or assumed to

perform R&DRecently, samples of service industries


Frascati Manual: Main Indicators

R&D personnel Data on head counts, full time equivalents (FTE)Classified by occupation, qualificationBreakdown by sector

(Intramural) R&D expenditureCurrent costs, capital expenditureBreakdown by sources of fundsDomestic: GERD, National: GNERD

Other indicatorsGBOARD (Governmental Budget)

Remark: National, regional, sectoral aggregates


Frascati Manual:GERD by Region in Europe


Frascati R&D Definition

Basic ResearchNew knowledge without application or use in view

Applied ResearchNew knowledge with practical aim or objective

Experimental DevelopmentExisting knowledge directed to new applications

or improving applications substantially


Oslo Manual


Scope of the manual Innovation statistics

Sampling approachBusiness enterprises (at least 10 employees)Stratified random sample (some census)Repeated cross-section, some panel surveys


Oslo Manual: Main Indicators

Innovative / innovating firmsTypes of innovative activities Intramural and extramural R&DAcquisition of machinery etc.Other Preparations by type of activity

Innovation expenditureImpact of innovationSales with product innovation

Objectives of and Obstacles to innovationsLinkages in innovation Information sourcesCo-operation

Remark: Shares of firms using sampling weights


Oslo Manual: Protection Methods by TypeS

ou

rce:

Eu

rost

at,

Sta

tist

ics

in F

ocu

s 9

1/2

00

7


Oslo Innovation Definition

Product innovationNSI characteristics or intended uses

Process innovationNSI production / delivery methodsTechniques, equipment, software

Marketing innovation (recently)NSI design, packaging, placement, promotion,

pricingOrganisational innovation (recently)NSI business practice, workplace organisations or

external relations


Oslo Innovation Definition 2

Period under reviewSampling period: often more than one year

What is new?New to the firm: Innovation or diffusion?New to the market: Regional, national, global?New to the world: Who knows that?

Again, what is new?New: How new to be new?Significantly impr.: What is significantly?


Canberra Manual

History of the manualFirst edition 1995

Scope of the manualHuman Resources in Science and Technology

(HRST)Sampling approachPerson (individual) in household surveys,

population censuses and administrative records


Canberra Manual:HRST in Europe


Patent Statistics Manual


Scope of the manualPatent indicators

Sampling approachPatent derived data mainly

using patent databases


EU Member States EPO Patent Applications 2003


Patent Statistics Manual: Main Indicators

Number of patents (patent counts)Patents filed (applications)Patents granted or registeredClassified by country, region, industry,

institutions, inventors, technology fieldCitation based indicators (weighted patent counts)Backward and forward citationsCurrent Impact Index:

Patents of the last 5 years cited this yearCitation Performance Index:

Number of patents in the most highly citedPatent Values


EU Member States EPO Patent Applications 2003


Bogota Manual

History of the manualRegional manual, first edition 2001

Scope of the manual Innovation statistics for Latin America and the

Caribbean CountriesSampling approachSimilar to Oslo recommendationsMore detailed industry strata (ISIC 3 or 4)


Bogota Manual: Main Indicators

Innovation focus Innovation process instead of result Innovation system approach more pronounced

Innovation effortsEmbodied and disembodied technology

Innovation results and innovation goalsInnovation funding (sources of funds)Innovation linkages Frequencies by type, agent, institutionDegree of satisfaction

Innovation policy assessmentKnowledge of institutions and programsAssessment of programs


NEPAD Innovation Survey Design

History of the NEPAD studyFirst publication, 2004 Intergovernmental committee, 2007

Scope of the studyPolicy relevant innovation surveys Implementation in AfricaTraining Module

Additional ore more detailed innovation indicatorsLearning process Innovation-related policies: importance, impact


What Is an Indicator?

“Indicator Indicator” or “Greater Honeyguide”


What Was an Indicator again?

IndicatorLatin “indicare”: to indicate, to show, to be a

sign of, to give notice ofTool indicating facts/information in general

not directly measurable

Economic indicatorAn economic indicator is a statistic about the

economy allowing analyses of economic performance and predictions of future performance

Leading, lagging and coincident indicators


Typology of Indicators

Traditional vs. new innovation indicatorsR&D survey/Patent statistics based indicators Innovation survey based indicators etc.

Input, Throughput and Output IndicatorsLinear view of innovation Input Process Output

Simple, Complex and Composite IndicatorsNumber of indicators involved and how they are

combined


Traditional and New Indicators

Traditional Innovation Indicators Innovation indicators related to R&D:

Number of R&D employees Amount of R&D expenditure

Innovation indicators related to patents: Number of patents granted Number of patent applications

New Innovation Indicators Innovation indicators related to broader concepts

of innovationLiterature based innovation indicators


Inputoriented Indicators

R&D based input indicatorsR&D personnel: share in total personnelR&D expenditure: share in GDP

(Broader) innovation based input indicators Innovation expenditure: share in GDP ICT expenditure: share in GDP

Patent indicatorsPatent applications, grants, stock:

per population


R&D Expenditure

Source: DST South Africa (2007)


R&D Personnel

Source: DST South Africa (2007)


Patent Applications

Source: OECD Patent Statistics Manual, 2009


Output Indicators

Direct output indicators Scientific publications:

(weighted) number of articles, pages Product, process, organizational, marketing

innovations: share of firms

Innovative sales: share in total sales

Indirect output indicators Changes in profits, costs, productivity,

employment, market shares


Scientific PublicationsS

ou

rce:

OE

CD

, S

TI S

core

board

20

07


Product InnovationsS

ou

rce:

OE

CD

, S

TI S

core

board

20

07


Innovative SalesS

ou

rce:

OE

CD

, S

TI S

core

board

20

07


And another Typology of Innovation Indicators

Simple IndicatorsBuilding mostly on a single variablee.g. share of innovative firms in an industry etc.

Complex IndicatorsCombining variables, but not aggregatinge.g. share of firms with linkages to universities in

innovative firms etc.

Composite IndicatorsAttempt to aggregate variablese.g. degree of openness Most prominent: IQ (Intelligence Quotient)Mostly forgotten: Exam grade


Complex Innovation Indicator: Effects of InnovationS

ou

rce:

Eu

rost

at,

Sta

tist

ics

in F

ocu

s 1

13

/20

07


Complex Innovation Indicator: Sources of InformationS

ou

rce:

Eu

rost

at,

Sta

tist

ics

in F

ocu

s 8

1/2

00

7


Composite Indicators

Definition of composite indicator Individual indicators compiled into single index Ideally based on a modelMeasuring multidimensional concepts which

cannot be measured with a single indicatorE.g. competitiveness

OECD Handbook of Constructing Composite Indicators, 2008


Composite Indicators: Pros and Cons (OECD Handbook)


EIS 2008: Innovation Performance

0.000

0.100

0.200

0.300

0.400

0.500

0.600

0.700

TR BG LV RO HR LT P L SK HU MT IT GR P T ES NO CZ SI EE IS CY EU NL FR BE LU IE AT UK DK DE FI SE CH

ModerateModerateinnovatorsinnovators

InnovationInnovationfollowersfollowers

InnovationInnovationleadersleaders

Catching-upCatching-upcountriescountries


Dimensions of Composite Innovation Indicators EIS 2008


Comparing Countries with Composite Indicators: EIS 2006

0.16 0.

20

0.20

0.21

0.21 0.23

0.24 0.

26 0.27 0.28

0.28 0.30

0.31 0.32

0.32 0.

36 0.40 0.42 0.

44 0.45

0.46 0.

48

0.48 0.50 0.51

0.58 0.

60

0.68 0.

71

0.72

0.00

0.10

0.20

0.30

0.40

0.50

0.60

0.70

0.80

RO MT LV EL SK PL BG CZ LT PT CY ES HU SI EE IT NO IE LU IS FR UK NL BE AT DE DK FI CH SE

Innovation drivers Know ledge creation Innovation & entrepr. Applications Intellectual property


Comparing Estonian and Slovenian Sub-Indexes

0.00

0.05

0.10

0.15Innovation drivers

Knowledge creation

Innovation & entrepr.Applications

Intellectual property

Slovenia

Estonia

Examples for Hugo Hollanders, 2009. Thanks Hugo!


Composite Indicators Revisited

“Everything should be made as simple as possible,

but no simpler.”

(Albert Einstein, 1934)


Characteristics of a Good Survey

Target and frame populationTarget population is well definedFrame population has good coverage:

only minor undercoverage and overcoverage

Stratification and SamplingStratification criteria are observable

in frame and target populationRepresentative sampling: random sampleUnbalanced sampling is well motivated



QuestionnaireQuestions are based on research questionsQuestions are not suggestiveQuestions uses diction of the correspondentDefinitions are short and near to the questionsComplicated questions are accompanied by a list

of good examplesAs few open questions as possibleScales are balanced Information asked is availableQuestions should be answerable by one personQuestionnaire is pretested



Data cleansingData consistency has been checked

Non-responseKey questions are asked to non-respondents in

an additional surveyNon-response bias is tested for

ExpansionsWeighting factors are based on sampling

probabilitiesWeighting factors are adjusted for non-responseWeighting factors are possibly adjusted for non-

response biasMissing values / items are properly imputed


Use of Innovation Survey Data

Users of Innovation Survey Data

Political Use: Innovation Reporting

Academic Use: Innovation Analyses

Requirements for Innovation Surveys


Users of the Innovation Surveys: Case of Germany

Main external users: German federal government (esp. STE)European Commission, OECDParticipating Firms

Other external users:Federal state governments (larger states) Industrial associationsAcademic research (universities, non-profit

research institutes, incl. PhDs)

Internal users:Contract researchAcademic research (incl. PhDs)


Aspects of German Innovation Survey

Annual innovation surveyBi-annually extended questionnaire with focus

on a special topic (8-16 pages)Bi-annually short questionnaire on core

indicators (4 pages)

Panel innovation surveySampling the same set of firms every yearBi-annually adjustment of the sample

Cutting sample for firm failure etc.Extending sample with newly formed firms

Data links at micro levelPatent databases (German and European)Database of R&D subsidies


Innovation Reports for Policy

Innovation reporting Indicator reports (annually)Background reports (biannually)Sectoral reports (annually)

Expert Commission Research and InnovationExpert Report on Research, Innovation and

Technological PerformanceStudies on the German Innovation System

European reportingEuropean Innovation Scoreboard (EIS)and much more


Indicator Reports

UsersGerman federal government (esp. BMBF)Published printed and online

Reporting strategyShort report (16-20 pages)Highlighting main developments

Main contents:Development of innovation indicators over time

(CIS-type core indicators)Projections for the current and following year

(innovation expenditure, innovation intensity)


Cost Reduction through Process Innovations


Sectoral Reports

UsersParticipating firms: printed versionOthers (time delay): internet version

21 sectoral reports (only in German)12 for manufacturing industries8 for service sector industries1 for mining, energy, water supply

Brief contents (4 pages):Ranking of industriesDevelopment of innovation indicatorsBenchmarking of best-practice firms

(biannually)


Sectoral Reports:Example of Automotive Industry


Expert Commission

Annual Expert Report Immediate and intermediate need for actionCore topics this yearStructure and Trends: Education, R&D,

Innovation, SMEs, Formations, Patents, Publications

Studies on the Innovation SystemR&D and Knowledge intensive sectors Innovation behaviour and finance International R&D of German firmsand 10 more


Expert Commission Report: Example


Expert Commission Report: Example 2


Academic Research: Questions

Determinants of innovation behaviourTechnological opportunities depending on

absorptive capacity

Determinants of innovation successContinuous R&D activity or R&D departmentCo-operation esp. with customers

Innovation and employmentProduct innovation creatingProcess innovation reducing, but less clearSkill bias of technological change, esp. for the

service sector


Academic Research: Examples

Internationalisation of R&D-co-opetition

An empirical analysis of the effects of patents and secrecy on knowledge spillovers

Persistence of innovation

Capital control, debt financing and innovation activity

Employment effects of different innovation activities

... could be completed with 75-100 other topics


Scientific Use File

Micro-aggregation

Factual anonymisation

Multiplicative errors: e.g. turnover, employees Intensities, rates: e.g. innovation expenditureTruncation: i.e. upper limits for extreme values,

e.g. innovation intensity to 35%Grouping: i.e. range (ordinal) instead of value,

e.g. innovative sales between 5 and 10 %Aggregation: i.e. less detailed classification,

e.g. location of innovation partnersWithholding information:

e.g. some variables for banks, insurances


Scientific Use File (2)

Availability of dataAll waves with 3 years time delayPossibility to build a panelContract specifying

Non-profit academic researchResearch topicNames of researcher

Access to original micro data at surveying institution or statistical office


Education Use File

Complete anonymisationArtificial data set Generated by statistical re-samplingNo large firms contained

AvailabilitySingle cross-sections 1999 and 2000Contract specifying useDownload from ftp-server

Teaching purposeEconometric courses on micro dataCourses on empirical economics of innovation

Maastricht Economic and Social Research and Training Centre on Innovation and Technology (UNU-MERIT)

and FH Aachen, University of Applied Sciences

How to summarize this?

Maastricht Economic and Social Research and Training Centre on Innovation and Technology (UNU-MERIT)...

Documents

Transcript of Maastricht Economic and Social Research and Training Centre on Innovation and Technology (UNU-MERIT)...