Improving Measures of Science, Technology, and Innovation: Interim Report (2012)

Committee on National Statistics Division of Behavioral and Social Sciences and EducationROBERT E. LITAN, ANDREW W. WYCKOFF, AND KAYE HUSBANDS FEALING, EDITORS;

PANEL ON DEVELOPING SCIENCE, TECHNOLOGY, AND INNOVATION INDICATORS FOR THE FUTURE; NATIONAL RESEARCH COUNCIL OF THE NATIONAL ACADEMY OF SCIENCES

Kathleen Padova, December 1, 2014

Chapters 6 & 7

What indicators do we want How can we get them

What Indicators Do We Want?

Current Indicators: Current State & Issues

Current Indicators on National or State level; often not granular enough

Statistics at geographic levels finer than states often not published

Previously, physical and political boundaries strongly influenced the location and transfer of innovation activities; the knowledge economy observes no limitations

Can’t track technology transfer between multi-state companies or multi-university collaborations

Economic growth, worker displacement, etc. occurring at a subnational level, data at a subnational level needed to support policy decisions at this level

• state, county, and metropolitan tables of data from the Business R&D and Innovation Survey

• degrees granted in science, technology, engineering, and mathematics (STEM) (production and migration)

• academic R&D expenditures

• federal R&D expenditures

• total R&D

• STEM jobs

• STEM workforce migration

• patent applications, grants, and citations

• STI equity investments

• STEM occupational projections

• STEM occupation classification

• STEM graduate and workforce migration

• firm innovation processes

• propensity to innovate ratings

• mappings of entrepreneurial density

• industry support for R&D in universities

• firm births, mergers and acquisitions, deaths

• venture capital investments

• state and federal grants and loans

• initial public offerings

• new products

• drug and other approvals

• data on dealmakers and entrepreneurs, including number of connections among dealmakers and entrepreneurs)

• data on emerging industries, based on universities, government laboratories, firms, value chains, key occupations, and individuals

Potential Indicators: Challenges

Very difficult or expensive to collect

Data might not maintain same type of meaning at disaggregate level

Place matters; but is “leaky:

not necessarily clear what the geographical span of impact is for a university or a firm in a given locale

multiplant firms that span more than one state might have difficulty allocating activities accordingly

These potential problems would affect data quality and reliability at finer geographical levels.

Politics!

Panel Recommendations

“The National Center for Science and Engineering Statistics should host

working groups in the near future to further develop subnational

science, technology, and innovation indicators. Participants in the

working groups should be both users and providers of the data. A

main focus of the discussion should be on data reliability, particularly

at fine geographical scales. Potential indicators should include

subnational research and development statistics, and subnational

science, technology, engineering, and mathematics workforce

statistics.”

How Can We Collect Desired Indicators?

Collection: Current State & Issues

Surveys, lots of surveys

Declining survey response rate

Processing time lag between collection and publication

Data sources, standards, and taxonomies are disparate, expanding, and evolving

Users’ expectations increasing

Potential Collection Sources/Methods

Web scraping - semistructured data from a public web page and register it into a structured database

Can occur continuously

No need to cooperate

Partnerships/Negotiating with sources

Data is already structured

Data feed from sources (e.g. XML feed)

Combination of traditional surveys and alternate methods

Contests!

Facebook, Google+: number of students at a university, how many major in which fields

LinkedIn, Monster.com, Zerply: the composition of the labor force, geographic breakdown, skill sets, etc.

Mendeley, Academia.edu, CiteULike: how many researchers are active in which fields, how many collaborations, who collaborates with whom, how useful is a given piece of research

Potential Collection: Challenges

Ownership and legality

Willingness of data sources to cooperate

Scope, consistency, and quality of data

Dependence on external source for updating and maintaining data

Determining appropriate statistics for alternate data sources

Data integration

Resources

Panel Recommendations

“The National Center for Science and Engineering Statistics should fund exploratory activities on frontier data extraction and development methods. These activities should include:

research funding or prize competitions to harness the computing power of data specialists with a view to (a) analyzing existing public databases to develop better indicators of science, technology, and innovation activities and (b) analyzing the huge and growing amount of information on the Internet for similar purposes

pilot programs or experiments to produce a subset of indicators using web tools

convening a workshop of experts on multimodal data development, to explore the new territory of developing metrics and indicators from surveys, administrative records, and scientometric sources.”

Final Conclusions

Recommended Actions & Future Work

Interim Report (2012) – Recommended actions

Develop new and revised indicators based on existing survey data

Develop new methods for obtaining data

Develop subnational STI indicators, more comprehensive measures of trade in research and development services

Develop a conceptual framework for measuring innovation activities

Final Report (2014)

National Research Council. Capturing Change in Science, Technology, and Innovation: Improving Indicators to Inform Policy. Washington, DC: The National Academies Press, 2014.

http://www.nap.edu/catalog/18606/capturing-change-in-science-technology-and-innovation-improving-indicators-to

Discussion

How reliable do you suppose some of the considered external data sources are for developing indicators to be used for policy making?

Given the purpose of collecting and reporting these indicators for other to respond to them, how “agile” do agencies need to be?