Improving Measures of Science, Technology, and Innovation: Interim Report (2012)
Committee on National Statistics Division of Behavioral and Social Sciences and EducationROBERT E. LITAN, ANDREW W. WYCKOFF, AND KAYE HUSBANDS FEALING, EDITORS;
PANEL ON DEVELOPING SCIENCE, TECHNOLOGY, AND INNOVATION INDICATORS FOR THE FUTURE; NATIONAL RESEARCH COUNCIL OF THE NATIONAL ACADEMY OF SCIENCES
Kathleen Padova, December 1, 2014
Chapters 6 & 7
What indicators do we want How can we get them
What Indicators Do We Want?
Current Indicators: Current State & Issues
Current Indicators on National or State level; often not granular enough
Statistics at geographic levels finer than states often not published
Previously, physical and political boundaries strongly influenced the location and transfer of innovation activities; the knowledge economy observes no limitations
Can’t track technology transfer between multi-state companies or multi-university collaborations
Economic growth, worker displacement, etc. occurring at a subnational level, data at a subnational level needed to support policy decisions at this level
• state, county, and metropolitan tables of data from the Business R&D and Innovation Survey
• degrees granted in science, technology, engineering, and mathematics (STEM) (production and migration)
• academic R&D expenditures
• federal R&D expenditures
• total R&D
• STEM jobs
• STEM workforce migration
• patent applications, grants, and citations
• STI equity investments
• STEM occupational projections
• STEM occupation classification
• STEM graduate and workforce migration
• firm innovation processes
• propensity to innovate ratings
• mappings of entrepreneurial density
• industry support for R&D in universities
• firm births, mergers and acquisitions, deaths
• venture capital investments
• state and federal grants and loans
• initial public offerings
• new products
• drug and other approvals
• data on dealmakers and entrepreneurs, including number of connections among dealmakers and entrepreneurs)
• data on emerging industries, based on universities, government laboratories, firms, value chains, key occupations, and individuals
Potential Indicators: Challenges
Very difficult or expensive to collect
Data might not maintain same type of meaning at disaggregate level
Place matters; but is “leaky:
not necessarily clear what the geographical span of impact is for a university or a firm in a given locale
multiplant firms that span more than one state might have difficulty allocating activities accordingly
These potential problems would affect data quality and reliability at finer geographical levels.
Politics!
Panel Recommendations
“The National Center for Science and Engineering Statistics should host
working groups in the near future to further develop subnational
science, technology, and innovation indicators. Participants in the
working groups should be both users and providers of the data. A
main focus of the discussion should be on data reliability, particularly
at fine geographical scales. Potential indicators should include
subnational research and development statistics, and subnational
science, technology, engineering, and mathematics workforce
statistics.”
How Can We Collect Desired Indicators?
Collection: Current State & Issues
Surveys, lots of surveys
Declining survey response rate
Processing time lag between collection and publication
Data sources, standards, and taxonomies are disparate, expanding, and evolving
Users’ expectations increasing
Potential Collection Sources/Methods
Web scraping - semistructured data from a public web page and register it into a structured database
Can occur continuously
No need to cooperate
Partnerships/Negotiating with sources
Data is already structured
Data feed from sources (e.g. XML feed)
Combination of traditional surveys and alternate methods
Contests!
Facebook, Google+: number of students at a university, how many major in which fields
LinkedIn, Monster.com, Zerply: the composition of the labor force, geographic breakdown, skill sets, etc.
Mendeley, Academia.edu, CiteULike: how many researchers are active in which fields, how many collaborations, who collaborates with whom, how useful is a given piece of research
Potential Collection: Challenges
Ownership and legality
Willingness of data sources to cooperate
Scope, consistency, and quality of data
Dependence on external source for updating and maintaining data
Determining appropriate statistics for alternate data sources
Data integration
Resources
Panel Recommendations
“The National Center for Science and Engineering Statistics should fund exploratory activities on frontier data extraction and development methods. These activities should include:
research funding or prize competitions to harness the computing power of data specialists with a view to (a) analyzing existing public databases to develop better indicators of science, technology, and innovation activities and (b) analyzing the huge and growing amount of information on the Internet for similar purposes
pilot programs or experiments to produce a subset of indicators using web tools
convening a workshop of experts on multimodal data development, to explore the new territory of developing metrics and indicators from surveys, administrative records, and scientometric sources.”
Final Conclusions
Recommended Actions & Future Work
Interim Report (2012) – Recommended actions
Develop new and revised indicators based on existing survey data
Develop new methods for obtaining data
Develop subnational STI indicators, more comprehensive measures of trade in research and development services
Develop a conceptual framework for measuring innovation activities
Final Report (2014)
National Research Council. Capturing Change in Science, Technology, and Innovation: Improving Indicators to Inform Policy. Washington, DC: The National Academies Press, 2014.
http://www.nap.edu/catalog/18606/capturing-change-in-science-technology-and-innovation-improving-indicators-to
Discussion
How reliable do you suppose some of the considered external data sources are for developing indicators to be used for policy making?
Given the purpose of collecting and reporting these indicators for other to respond to them, how “agile” do agencies need to be?
Top Related