What Might a Theory-Based Roadmap for Prospective Evaluation and Developing Innovation Policy Look...

What Might a Theory-Based Roadmap for Prospective Evaluation

and Developing Innovation Policy Look Like?

Presented atAmerican Evaluation Association Conference

November 2009 Gretchen Jordan, Sandia National Laboratories

[email protected]

Portions of the work presented here were completed for the U.S. DOE Office of Science by Sandia National Laboratories, Albuquerque, New Mexico, USA under Contract DE-AC04-94AL8500. Sandia is operated by Sandia Corporation, a subsidiary of Lockheed Martin Corporation. Opinions expressed are solely those of the author.

SAND Number: 2009-7359C

G. Jordan AEA November 2009 2

• Prospective evaluation in context

• National interest (SoSP, SciSIP)

• Models of what is known about innovation, what we need to know

• Theories and an example

• Conclusions

Outline


Evaluation in the Policy Cycle

Foresight Technology Roadmapping

TechnologyAssessment

Wolfgang Polt30-10-2007


National Interest: SoSP and SciSIP

• The science of science policy (SoSP) is an emerging field of interdisciplinary research, the goal of which is to provide a scientifically rigorous, quantitative basis from which policy makers and researchers can assess the impacts of the Nation’s scientific and engineering enterprise, improve their understanding of its dynamics, and assess the likely outcomes.

• A National Science and Technology Council (NSTC) Interagency Task Group (ITG)• The Science of Science & Innovation Policy (SciSIP) program was established at NSF in 2005.

John MarburgerApril 2005


SoSP Workshop in December 2008

Primary Conclusion of SoSP Roadmap: “Expert judgment” remains the best available

decision support tool for science policy makers, but

a nascent community of practice is emerging in the science policy arena that holds enormous potential to provide rigorous and quantitative decision support tools in the near future. ”

The White House SoSP Interagency Task Group should take the lead to set the Federal agency research agenda.


White House S&T Priorities for the FY 2011 Budget

Agencies should describe in their budget submission how they are• prioritizing activities toward four challenges and strengthening four

cross-cutting areas (which include productivity of research institutions)

• Expecting outcomes of research in above areas, providing quantitative metrics where possible

• Building capacity to rigorously evaluate programs, and how assessments have been used to eliminate or reduce programs

• Operating in the open innovation model and supporting long term high-risk, high payoff research

Agencies will:• Develop outcome oriented goals for S&T, target investment toward

high performers, develop ‘science of science policy” tools that can improve management and assessment of impact

-Peter Orszag, John Holdren, August 4, 2009


The SoSP Roadmap 10 Science Questions

1. What Are The Behavioral Foundations Of Innovation?

2. What Explains Technology Development, Adoption And Diffusion?

3. How And Why Do Communities Of Science And Innovation Form And Evolve?

4. What Is The Value Of The Nation’s Public Investment In Science?

5. Is It Possible To “Predict Discovery”?

6. Is It Possible To Describe The Impact Of Discovery On Innovation?7. What Are The Determinants Of Investment Effectiveness?8. What Impact Does Science Have On Innovation And Competitiveness?9. How Competitive Is The U.S. Scientific Workforce?10. What Is The Relative Importance Of Different Policy Instruments In Science Policy?

Theme 1: Understanding Science

and Innovation

Theme 2: Investing in Science

and Innovation

Theme 3: Using the Science of Science Policy to

Address National Priorities

The National Imperative

Science Questions

Findings

Recommendations

Source: J. Lane, April 2009


http://www.cs.unibo.it/schools/AC2005/docs/Bertinoro.ppt#266,11,The Blind Men and the Elephant

Parts are studied and understood better than the whole!

Source: Bhavya Lal, STPI, at AEA 2006


A Science of Science and Innovation Policy must build a theory that connects levels

ResearchTeam

ResearchOrganization

The Sector’sIdea Innovation

Network

The Sector’sNational and

Global Context

micro meso macro


Anticipate effects of Scientific discovery

Anticipate effects of

science on R&D

Understand technology

development & diffusion

Understand behavioral

foundations

Understand network

behaviors

Science Workforce

competitiveness

Impacts on competitiveness,

etc.

Assess real time value of new knowledge

Relative importance of

policy instruments

Determinants of investment effectiveness

7

2

31

810

9

65 4

SoSP Roadmap Questions Rearranged into a Three Level Logic Model

Draft by G. Jordan 12/12/2008

Investment, incentives, Use

People & organizational inputs & incentives

The (non-linear) S&T, R&D process

Understanding a multi-level eco-system

Macro

Micro

Meso


The science must explain relationships among institutions

Demand

Consumers (final demand)Producers (intermediate demand)

Industrial system Education and research system

Political system

Government

Governance

RTD Policies

Professional educationand training

Higher educationand research

Public sectorresearch

Large companies

Mature small/ mediumenterprises ( SMEs)

New, technology -based firms

Infrastructure

Intermediaries

ResearchinstitutesBrokers

Banking, venture capital

IPR and information

Innovation andbusiness support

Standardsand norms

Framework conditionsFinancial environment; taxation and incentives; propensity to innovation

and entrepreneurship; mobility

A National Innovation System Model

The potential reachof public policies …

Demand

Consumers (final demand)Producers (intermediate demand)

Industrial system Education and research system

Political system

Government

Governance

RTD Policies

Professional educationand training

Higher educationand research

Public sectorresearch

Large companies

Mature small/ mediumenterprises ( SMEs)

New, technology -based firms

Infrastructure

Intermediaries

ResearchinstitutesBrokers

Banking, venture capital

IPR and information

Innovation andbusiness support

Standardsand norms

Framework conditionsFinancial environment; taxation and incentives; propensity to innovation

and entrepreneurship; mobility

Source: Arnold and Kuhlman, 2001

A National Innovation System Model

The potential reachof public policies …


Source: G. Jordan, 2007. Modified from R. Cooper/ Exxon’s Stage Gate, Hage & Hollingsworth’s Idea Innovation Network

Marketing R&D, Quality R&D

Diffusion and use

Engineering & manufacturing R&D

7

8

6

Connectivity and Throughput

Production, Refinement

Micro, meso, macro

impacts9

10

The science must explain connections among arenas of research and development


ConfirmationAwareness Persuasion Decision Implementation

Feedback

Continued adoptionLater adoption

DiscontinuanceContinued rejection

Adoption

Rejection

Product Characteristics

• Relative advantage

• Compatibility

• Complexity

• Trialability

• Observability

Characteristics of the decision-making unit

• Adopter type

• Personality type

• Communication behavior

• Socio-economic status

Socio-cultural/market environment

• Market structure

• Market segments

• Prior practice

• Culture and norms

• Innovativeness

Communication field• Broadcast

• Contagion

Source: Everett Rogers 1994 as modified by Innovologie, LLC. 2005

The science must understand Diffusion and relate it to R&D


All this information is useful to predict where and how policy makers can intervene to achieve desired goals


Diffusion and use


78

6



Micro, meso, macro

impacts910


Diffusion and use


78

6



Micro, meso, macro

impacts910



• Market segments

• Prior practice


• Innovativeness



• Market segments

• Prior practice


• Innovativeness


Feedback



Adoption

Rejection



• Compatibility

• Complexity

• Trialability

• Observability


• Adopter type





• Contagion


Feedback



Adoption

Rejection



• Compatibility

• Complexity

• Trialability

• Observability


• Adopter type





• Contagion

Interventions at micro, meso, and/or macro

levels?


Theories that could be integrated to understand how we can drive innovation

Research Team– Management of innovation literature, learning theory

Research Organization– Organizational innovation theories– Research Profiles theory

Science/technological Sector– Idea Innovation Network on S&T/R&D process– Network theories– Diffusion theory– Sector economic models

National and global context– Modes of coordination theories – Institutional and institutional change theory– Policy decision making –theories of


One possible decision tool to identify bottlenecks, policy objectives & effectiveness

Socio economic outcomes

Technical progress

Network connectedness

Organizational profiles – do

attributes match the profile?

RTD arenas – are there sufficient funds

Portfolios -need more/ less radical, large scope?

Modes of coordination –

effective?

Capabilities –Level, mix, availability

High riskcapital –

available where

Basic research

Manufacturingresearch

Applied research

Development research

Quality research

Commercialization research

Macro- Institutional Rules as they affect the sector

Micro - funds allocation by arena and profile

INNOVATION

Meso - Performance byTech sector and arena

Policy Objectives-Structural-Technical

Source: Jordan, Hage, and Mote, 2006, 2007, 2008


Conclusion

• Innovation occurs within a multi-level, complex, dynamic eco-system

• Prospective evaluation predicts

• Prediction requires understanding, characterization, theory

• There are theories that can be used now

• Synthesis of existing theories and building new theories are needed going forward.


Selected ReferencesArnold, E. (2004). Evaluating research and innovation policy: A systems world needs systems evaluations. Research Evaluation, 13(1), 3-17

Hage, Jerry, G.B. Jordan and J. Mote (2007). A Theories-Based Innovation Systems Framework for Evaluating Diverse Portfolios of Research: Part Two - Macro Indicators and Policy Interventions. Science and Public Policy, 34(10): 731-741.

Jordan, G. B., Hage, J., & Mote, J. 2008. A theories-based systemic framework for evaluating diverse portfolios of scientific work, part 1: Micro and meso indicators. In C.L.S. Coryn & Michael Scriven (Eds.), Reforming the evaluation of research. New Directions for Evaluation, 118, 7–24.

Jordan, G.B. 2006. Factors Influencing Advances in Basic and Applied Research: Variation Due to Diversity in Research Profiles. In Innovation, Science, and Institutional Change: A Handbook of Research, J. Hage and M. Meeus (eds). Oxford University Press: Oxford, 173-195.

Mote, J., Y. Whitestone, G. Jordan and J. Hage. 2008. Innovation, Networks and the Research Environment: Examining the Linkages. International Journal of Foresight and Innovation Policy 4(3): 246-264.

Reed, John H, G. Jordan, Using Systems Theory and Logic Models to Define Integrated Outcomes and Performance Measures in Multi-program Settings, in Research Evaluation, Volume 16 Number 3 September 2007.

What Might a Theory-Based Roadmap for Prospective Evaluation and Developing Innovation Policy Look...

Documents

Transcript of What Might a Theory-Based Roadmap for Prospective Evaluation and Developing Innovation Policy Look...