Presentation
-
Upload
macey-nunez -
Category
Documents
-
view
15 -
download
0
description
Transcript of Presentation
Version 9.2.2003
VTT TECHNOLOGY STUDIES
Evaluating the societal impacts ofPublic research organisations:
A (belated) paradigm shift in the making
Kaisa Lähteenmäki-Smith and Kirsi HyytinenVTT Technology Studies
Version 9.2.2003
VTT TECHNOLOGY STUDIES
Presentation
• The project on societal impact evaluation: research questions, methods, participating organisations and their differentiated working environments, steering processes and R&D ambitions
• Evaluation model for societal impacts of publicly funded R&D organisations
• Some key findings and conclusions and their implications for the evaluation model
Version 9.2.2003
VTT TECHNOLOGY STUDIES
Description of the project: questions
• How are publicly funded R&D organisations adjusting their evaluation practices to the emerging ‘management by results’-based accountability demands set by the government?
• How can different evaluation practices and methods be used for this purpose in assessing the societal impacts of R&D activities?
• What are the similarities and differences, as well as lessons to be drawn between policy sectors (best practices, challenges, bottlenecks…)?
• What are the implications of impact assessment on administrative practices, including steering processes, in respect of governmental actors and public research organisations?
Version 9.2.2003
VTT TECHNOLOGY STUDIES
Description of the project: organisations
• MULTI-SECTOR TECHNOLOGY R&D INSTITUTE (VTT) is national research institute in technology working in close interface with both the business sector and policy-making.
• DEFENCE SECTOR R&D INSTITUTE: The Finnish Defence Forces Technical Research Centre
• RESEARCH INSTITUTE IN THE AGRICULTURE AND FOOD SECTOR, MTT Agrifood Research Finland
• TWO POLYTECHNICS – one WITH ITS SUBSTANTIVE FOCUS IN THE AREA OF BUSINESS MANAGEMENT, ANOTHER MULTIDISCIPLINARY POLYTECHNIC
Version 9.2.2003
VTT TECHNOLOGY STUDIES
Motivations of impact assessment
Internal and external motives :
Change in management and administrative cultureDemand for the accountability and result-driven planningSpread of a service-oriented culture – customer orientationDemand for transparency and accountability
Compatibility and integration of policy objectivesSelection of sectoral objectives and consolidation of sector objectives with inter-connected sectors Selection and compatibility of horizontal goals
Demand for organisational developmentIdentification of demands and multiple development challengesSupporting strategic planning – contributes to steering and priority-setting, allows for operational freedom in developing core competences and activities Prioritizing the activities
Version 9.2.2003
VTT TECHNOLOGY STUDIES
National approach to accountability
• National and international dimensions of effectiveness• Customer-orientation: are the needs of the clients met?• Future orientation: are the visions and strategic targets set
meeting the needs of the changing nature of society and of the policy environment?
• Cost effectiveness• Socio-economic and societal dimension: Have the main
external costs and effects been taken into consideration?
Version 9.2.2003
VTT TECHNOLOGY STUDIES
Key concepts 1/3- output, outcome, impact,
effectiveness
The preservation of healthy oat grain compounds in the baking process
The healthy food product
Lower cholesterol levels
Improved national health longer life-expectancy economic effects
Output
Outcome
Impact
Effectiveness
Version 9.2.2003
VTT TECHNOLOGY STUDIES
Kohdealueenulkopuolella
T&k-vaikutukset
Key concepts 2/3- impact typology
IMPACTS
ANTICIPATED
In target area In target area
Useful
Detrimental
Useful
Detrimental
UNANTICIPATED
Outside target area Outside target area
Other impacts
R&Dimpacts
Otherimpacts
R&Dimpacts
( Mickwitz 2004)
Version 9.2.2003
VTT TECHNOLOGY STUDIES
Key concepts 3/3-impacts through time
IMMEDIATE•networking• improved R&D
efficiency• patent applications• publications• prototypes
INTERMEDIATE•partnership-based cooperation• new/ improved products, services and processes•company growth•improved company competitiveness•cost savings•higher employment•strengthened expertise•technology transfer•standards, norms•support for decision making•public discourse
ULTIMATE• improved industry
competitiveness• higher investment• better safety• improved quality of life• promotion of regional
development• improved awareness
-1 0 1 2 4 6 8 10 15 YEARS, OR MORE
IMPACTS
Benefits to economy and
society
Benefits to project participants
(Tassey 2003)
Version 9.2.2003
VTT TECHNOLOGY STUDIES
Dimensions of impacts and final beneficiaries
1. Economy, technology and commercialisation
2. Knowledge, expertise and human capital
3. Networking and social capital
4. Decision making and public discourse
5. Social and physical environment
Own organisation
Customer
Society
Public R&D-organisation
Version 9.2.2003
VTT TECHNOLOGY STUDIES
Evaluation model: some criteria
• Relevance• Quantifiable nature• Reliability• Easy availability• Dynamics of change and trends• Strategic feedback loop
Version 9.2.2003
VTT TECHNOLOGY STUDIES
How to get information about impacts and effectiveness
through evaluation
Ex ante evaluationPreparation phase
Execution phase
Concluding phase
Ex post evaluation
Stakeholder and client ’inputs’ into evaluation cycle•Follow-up studies
•Client satisfaction surveys
•Interactive discussion fora
Mid-term evaluation
Version 9.2.2003
VTT TECHNOLOGY STUDIES
Impact assessment as a learning and steering tool
Rationales for public support
MOTIVESMarket failure
Transaction costsRisks and uncertainty
Policy statements
OBJECTIVESPublic safety
New enterprises and workplaces
Welfare services
Policy implementation
ORGANIZATIONAL CAPACITY
as the means(for doing
’the rights things’)
Transparency, permanence, flexibility, predictability, legitimacy, justness
Using the knowledge about changes amongst
the stakeholders
ADDED VALUE OF R&D
To the organisation To the customer
For society
Definition of the new rationales
Allocation of the resources
Prioritizing
Definition on the strategy
Feedback