Evaluation of Science, Technology and Innovation Policy in Brazil: high complexity and new methodological frontiers

Tatiana de P. A. MaranhãoS &T Analyst

Ministry of Science, Technology and Innovation

Montevideo, 24 de setembro de 2013.

Situation of Science, Technology and Innovation in Brazil

Prior = focused mainly on human resources formation (post-graduation, master and PhD degrees) and international indicators Present = technological development for innovation, related to health, defense, education and infrastructure

More innovation driven efforts than pure and applied research

Fact 1 = BR government invests more in ST&I than private sector

Dispêndio nacional em ciência e tecnologia (C&T) em relação ao produto interno bruto (PIB) por setor, 2000-

2011

Fonte: Coordenação-Geral de Indicadores (CGIN) - ASCAV/SEXEC - Ministério da Ciência, Tecnologia e Inovação (MCTI).

Situation of Science, Technology and Innovation in Brazil

New policies:-Intensified stimuli for industry R&D associated with national industry strategies;-Funding industry R&D activities, including venture capital - Science without Borders = internationalization of human resources formation

Fact 2: Highly complex environment (concentrated investments)

Fact 3 = Total amount invested is still low compared to other countries

Fonte: Coordenação-Geral de Indicadores (CGIN) - ASCAV/SEXEC - Ministério da Ciência, Tecnologia e Inovação (MCTI).

8.1.1 Dispêndios nacionais em pesquisa e desenvolvimento (P&D) de países selecionados, 2000-2011

Fonte(s):

Organisation for Economic Co-operation and Development (OECD), Main Science and Technology Indicators, 2013/1 e Brasil: Coordenação-Geral de Indicadores (CGIN) - ASCAV/SEXEC - Ministério da Ciência, Tecnologia e Inovação (MCTI).

Elaboração

:

Coordenação-Geral de Indicadores (CGIN) - ASCAV/SEXEC - Ministério da Ciência, Tecnologia e Inovação (MCTI).

Nota(s):

1) PPC - Paridade do poder de compra.

Atualizada em:

28/08/2013

Fact 4 = We have more doctors, researchers and research groups ( 137% more doctors in 2010 than 2002)

Data available: www.cnpq.br and www.capes.gov.br

Mapa de investimentosPlataforma Carlos ChagasCNPq

Mapa das bolsas - GeoCAPESCAPES

Scholarships (Ministry of ST&I and Ministry of Education)

Fact 5 = Researches are publishing more articles than before 2007

• The National System of Science , Technology and Innovation is highly fragmented, with many stakeholders and different levels of political action;

• Fragmented evaluation experiences within MCTI

Research Infrastructure (MCTI)

Coordenação-Geralde Indicadores

Assessoria de Acompanhamento

e Avaliação

SecretariaExecutiva

Ministério da Ciência, Tecnologia e Inovação

Information and Actor Network

Fontes nacionais deinformações primárias

IBGECENSO, PNAD,PINTEC, Contas

Nacionais eCONCLA

CNPqFomento, bolsase Diretório dos

Grupos dePesquisa

CAPESDados da

pós-graduaçãoe bolsas

INEPCenso do ensino

superior

SETEC/MCTLei do BemFINEP/MCT

Fundos setoriais

SESU / MECBolsas PET

RFB / MFRenúncia fiscal

STN / MFSIAFI, balanços

dos estados

INPIPatentes e

balançotecnológico

Banco Central Balanço

tecnológico

Secretariasestaduais

de C&TInvestimentos

em C&T

Ministério doTrabalho

RAIS

FAPsBolsas efomento

MCT

Rede de Articulação Nacional de Indicadores de C&T

CGINSecretarias

estaduais deC&T e FAPsRIEC&T

IBGE

USPTOPatentes

Thomson Reuters

SCImagoBibliometria

EPOPatentes

Fontes internacionaisde informações primáriasCCT

EUROSTATComparaçõesinternacionais

OCDE ComparaçõesInternacionais

e manuais

Banco MundialDados

econômicos(PPC)

UNESCOComparaçõesInternacionais

e manuais

RICyTComparaçõesinternacionais

Disseminação • Comunidade científica e tecnológica

• Mídia

• Setor governamental

• Setor empresarial

Usuários

Sites e publicações:• MCT• RICyT• UNESCO• OCDE

Assessoramento Técnico

• Consultores• Comissão Permanente de Indicadores

Ministérios eestatais federaisCGEE

Previous efforts•Ministry of Social Development – specific Secretary focused on monitoring and evaluation (M&E) Bolsa Família;•Ministry of Education – system of evaluation of basic education (SAEB) (alumni grades); ENEM and Prova Brasil; evaluation of post-graduate courses•Ministry of ST&I – PACTI (2007-2010), ENCTI (2012-2015) and Monitoring and Evaluation Policy (Portaria nº

PACTI (2007-2010)

Present effort•Ministry of ST&I – Monitoring and Evaluation Policy, coordinated by Permanent Committee on M&E;•PAMA 2013 = 7 program evaluations and 12 monitoring activities;•Currently, there are 3.772 researchers focused on evaluation and 27 groups (Lattes CNPq);

Formulation and planning

Implementation

Results/Impacts

ex-ante evaluation

process evaluation

ex-post evaluation

Monitoring and info systems

Results and impact evaluations

“Base line” studies (map)

EVALUATION OF ST&I POLICIES

Results Results • How many patents and innovations have been generated by projects /

institutions supported and in which areas?• How many publications have been produced by scholars or supported

projects?ImpactImpact

• Policies encouraging innovation has improved the innovative performance of firms and sectors benefited?

• The scholarship policy has contributed to improve the training of professionals in strategic areas for the development of the country?

• What is the impact of scientific and technological research projects fostered by MCTI?

Need for methodologies able to establish causality and to isolate other factors that influence the observed results.

Examples•PROANTAR = Bibliometric study of scientific production – quality (citation index), cooperation networks, scientific areas; interview with more and less productive and evaluation of program management;

Examples•Impact Assessment Subvention (“Lei do Bem”) over private investment in R & D = econometric study with use of control groups: companies that use the LB tend to spend more on R & D because of the use, or just replace a font by another?

What has been achieved in the impact assessment?

Employing the control group, just as in experiments with drugs, it was observed that companies that make use of the Lei do Bem tend to invest in R & D at least 7% more than companies that do not use this legislation.

Examples•Evaluation of the Brazilian Space Program Study = qualitative and quantitative impact on technological development of domestic industry, its direct and indirect impacts, and development of international indicators;•Evaluation of programs and actions available on International Cooperation CNPq;•Evaluation of projects supported by the Sectorial Funds since 2000.

New frontiers ST&I

• Quasi-experimental;

• Econometric studies with group control;

• Qualitative methods such as discourse analysis, with webcrawler instruments;

• In depth interview related to previous data, creating new categories;

• Rethink planning and implementation problems based on the

incorporation of social demands.

Our challenges..... how to overcome?

• Fragmentation of ST&I public policies (many stakeholders, low coordination at national level);

• Concentration of resources and investments in the Southeast region;

• High quality evaluation needs time, resources (human, systems and $) and lots of political support!

Sites• www.mct.gov.br• http://lattes.cnpq.br/• www.capes.gov.br• www.embrapa.gov.br• http://www.ibge.gov.br/biblioteca/visualizacao/

periodicos/2/bn_2013_v21.pdf• www.cgee.org.br• www.ipea.gov.br• www.embrapa.br • www.fapesp.br

Thank you!Tatiana de P. Albuquerque Maranhãotmaranhao@mct.gov.br