Metronome - trimis.ec.europa.eu · Distribution list European Commission: [email protected]...

MetronomeMetronome

A METhodology foR evaluatiON of prOject iMpacts in the fiEld of Transport

Grant No. 213546

D5.1 Methodology for Evaluation of FP 5 and FP6 project impacts on

Community and Public Policies

Project: METRONOME 213546

Document Number and Title: D.5.1

Work-Package: WP5

Deliverable Type: Report

Contractual Date of Delivery:

Actual Date of Delivery: June 2009

Author/Responsible(s): A. Sitov

Contributors: J. van der Waard, H. Flikkema, A. Binsted,C. Ferris, A. Tuominen

Approval of this report: A. Tuominen, VTT

Keyword List: project evaluation, community and publicpolicies, research project impacts, method-ology

Dissemination level: Public (PU)

Grant No. 213546

2/65

MetronomeMetronome

Change History

Version Date Status Author (Partner) Description

1 31.7. 2009 Draft A. Sitov

2 20.8. 2009 Final A. Sitov

Distribution list

European Commission:

[email protected]

METRONOME partners:

[email protected]

[email protected]

[email protected]

[email protected]

[email protected]

[email protected]

[email protected]

[email protected]

METRONOME Steering Group:[email protected]

[email protected]

[email protected]

[email protected]

[email protected]

[email protected]

mailto:[email protected]















Grant No. 213546

3/65

MetronomeMetronome

Table of contents

1 Introduction ............................................................................................. 41.1 WP5 Objectives and Scope ...........................................................................5

2 Conceptual framework for C&PP ................................................................ 72.1 Background.................................................................................................72.2 Specific problems related to evaluation of transport research projects’ impacts .82.3 Literature review .........................................................................................8

3 Methodology for evaluation of project impacts on C&PP ............................ 103.1 WP5 methodological approach ....................................................................103.2 The data ...................................................................................................11

3.2.1 FP5 structure and objectives ................................................................143.2.2 FP6 Structure and Objectives ...............................................................15

3.3 European Strategic C&PP objectives and related indicators............................163.4 Selection of FP5 and FP6 projects and data collection ...................................173.5 Analyses of Final Reports - Evaluation matrix ...............................................17

3.5.1 Research objectives.............................................................................173.5.2 Impact indicators ................................................................................18

3.6 Analyses of dissemination activities – Dissemination quality matrix ................203.7 Surveys ....................................................................................................21

3.7.1 Co-ordinator survey.............................................................................213.7.2 Lead user questionnaire and interviews.................................................21

4 Methodology testing................................................................................ 224.1 Selection of FP5 and FP6 projects and data collection ...................................224.2 Results from Matrix Evaluation – objectives and impacts ...............................234.3 Results from Matrix Evaluation – dissemination activities...............................274.4 Results from the co-ordinator survey...........................................................284.5 Results from lead user interviews................................................................36

4.5.1 Methodology applied ...........................................................................364.5.2 Main findings ......................................................................................36

5 Discussion and conclusions...................................................................... 385.1 Methodological Development ......................................................................385.2 Results .....................................................................................................41

6 Recommendations for facilitation of future FP evaluation........................... 44Annex 1........................................................................................................ 46Annex 2........................................................................................................ 49Annex 3........................................................................................................ 54Annex 4........................................................................................................ 60Annex 5........................................................................................................ 65

Grant No. 213546

4/65

MetronomeMetronome

1 IntroductionEvaluation is a critical issue for research programmes’ and projects’ usefulness and practi-cality. Different types of evaluation focus on the development and management of effectiveevaluation plan and involve different methodological tools: analyses, surveys, panels, impactstudies. Evaluation must cover the whole process /all stages of project/ programme lifecy-cle/ from the development of the programme/formulation of project tasks/ up to the exploi-tation of results.

Many European policy documents and studies stress the importance of improvement of re-search project and programme evaluation, need for consolidation of existing evaluationmethods and preparation a robust and effective methodology for FP evaluation and assess-ment. It is impossible to achieve Community and Public Policies (C&PP) objectives withoutcardinal change of existing evaluation culture, development of methodological approachesand tools based on new experiences and requirements.

This Deliverable was prepared based on effective analysis of relevant literature and currentevaluation practice at European and national levels. Review of Monitoring and EvaluationMethods offers an insight into methods used for monitoring and evaluation of research pro-jects.

A special feature of Work Package 5 activities was that a large share of them was the se-quential ones. WP5 started some of its activities using WP2, WP3 and WP4 results: con-cepts, findings, conclusions, recommendations. This deliverable contains a structured reporton the WP5 results. WP5 deliverable provides not only a summary of works done, but alsogives a basis and recommendations for further adjustments and improvement of proposedmethodological approach and tools for FP5 and FP6 transport project evaluation as well asrecommendations for FPs evaluation improvement in future.

The first chapter of the Deliverable 5.1 provides a description of WP5 objectives and scope.Chapter 2 presents the Conceptual Methodological Framework. The preliminary Frameworkwas prepared in WP2 and further specified and applied in WP5. The objective of the Frame-work was to establish a sound and appropriate basis for the development/compilation ofmethodology for evaluation of FP5 and FP6 transport research projects contribution toC&PP. Taking into account data availability and other limitations for comprehensive and ro-bust evaluation of FP5 and FP6 impacts on C&PP, the evaluation design was prepared.

In chapter three, the C&PP evaluation methodology is presented. Based on the knowledgegathered and inputs from WP2, 3 and 4 and based on Evaluation Design the stepwise ap-proach and evaluation/estimation toolkit were proposed.

Chapter 4 presents the results of the methodology testing. A small, but representative sam-ple of FP5 and FP6 projects was selected for testing and adjustments of proposed methodo-logical approach and tools. The results of final report’s analyses, surveys and interviewswere processed. In addition, the collected information on dissemination and exploitation ofselected project results was analysed. WP5 approach and recommended methodologicaltoolkit present the first versions of evaluation and impact assessment of FP5 and FP6 pro-jects in the field of C&PP related to transport. Further adjustments and improvement stepsare possible.

Grant No. 213546

5/65

MetronomeMetronome

Finally, sections five and six present the discussion of the results and recommendations. Atthe end of this deliverable five annexes present detailed results that were produced in thework progress of Work Package 5:

1. List of relevant literature

2. Questionnaires

3. Summarised analysis of Matrix evaluations

4. Summarised analysis of Dissemination quality evaluations

5. Results of Lead User interviews

1.1 WP5 Objectives and Scope

The main objective of METRONOME WP 5 was to develop of a methodological approach forevaluation and assessment of different types of transport projects in terms of their impactson Community and Public Policies (C&PP).

The methodology in WP5 adopted a quite similar stepwise approach as in WP 3 and WP 4.

First, programme/ project information was analysed in respect to data availability and qual-ity. WP5 used the following data collection methods: project coordinator's questionnairesurvey, lead user interviews, project final report reviews, and dissemination quality, with aview to end user's engagement and addressing.

Second, the gathered data was analysed. Third, the WP5 evaluation strategy was proposed.Fourth, projects selected in WPs 2, 3 and 4 were re-screened and classified for the specialpurposes of WP5 and sample of selected projects was assessed according to the developedcriteria (proposed methodology for evaluation). Finally, a synopsis report on the evaluationtesting results was produced.

In general, WP5 included the following main tasks:

Task 5.1: Review and Analysis of Existing Methods of Transport Project Assessment andEvaluation

The first task of the WP 5 was to create a Review of existing approaches and methods relat-ing to C&PP for assessment of projects in the field of transport. On the basis of analysis ofexisting approaches and methods the team of experts proposed an approach and methodo-logical framework for evaluation and assessment of FP 5 and FP 6 transport projects fo-cused on innovative knowledge/ impact potential for improving Community and Public poli-cies. Also, results of the FP6 project TRANSFORUM were used as a source of information.

Task 5.2: Development of C&PP Methodology

The aim of the WP5 participants was to further develop the METRONOME-methodology, sothat it could be applied for ex-post evaluation of the FP results in the field of C&PP in thetransport sector. On the basis of the common methodological framework created by WP2,and using WP3 and WP 4 preliminary results, the WP5 proposed a methodology for detailselection, assessment and evaluation of projects in the field of transport by scrutinizing andanalyzing the project portfolio.

Specific approaches, procedures and criteria for each category of projects with regard to thenature, objectives and type of deliverables of the selected projects were developed. Defin-ing the potential impact of transport projects into EU policy documents played an important

Grant No. 213546

6/65

MetronomeMetronome

role here. Impact is defined as positive or/and negative changes produced by a project, di-rectly or indirectly.

Task 5.3: Methodology implementation and result validation

The final activity of the WP 5 applied developed approaches and methods for an evaluationand assessment of a representative sample of selected projects.

Task 5.4: Presenting the results

This task is focused on the definition of new research policy objectives and intermediateperformance targets for FP7.

Grant No. 213546

7/65

MetronomeMetronome

2 Conceptual framework for C&PP2.1 Background

The main goal of WP5 was a development of methodology for FP5 and FP6 projects in thefield of transport evaluation from the view of their contribution to an improvement of Com-munity and Public policies (C&PP).

There is no commonly accepted definition of Community and Public policies. They include allsectoral policies: social, employment, tourism, trade, health, culture and so on. According toWP2 and consortium partners’ recommendations, WP5 reduced its scope and was focusedfirst on the FP5 and FP6 transport projects’ contribution to transport policy.

Contribution to C&PP can be considered in broad sense and in narrow sense. Currently, Sus-tainability and Competitiveness are the backbone principles and main objectives of Euro-pean Transport Policy. Transport Policy (and other related policies) is aimed at promotingsustainability and industrial competitiveness. Hence, evaluation of FP5 and FP6 projects con-tribution to sustainability and competitiveness means the evaluation of research pro-ject/programme impact on transport policy.

WP5 used the results of ex post evaluation and impact assessment created by WP3 andWP4 for summarising the impact from this point of view.

Another interpretation for C&PP is to understand it as a means of policy creation at differentstages of policy cycle (Figure 1).

Identification of constraints

Formulation of problems

Definition of policy targets, measures and indicators

Implementation of measures

Monitoring and evaluation of implemented measures

MoIdentification and analysis of outcomes

Policy restatement

Figure 1. Policy Cycle.

A well-known risk of partial methodological approach /based on surveys, panels, impactstudies/ is that the results may be hard to generalize because of a small number of selectedprojects or control group of users. WP5 suggests minimizing this risk by using procedures

Grant No. 213546

8/65

MetronomeMetronome

containing a number of steps aimed at enhancing the quality, reliability and generality ofevaluation findings.

The first step was the selection of a representative set of FP5 and FP6 projects aimed at in-fluencing the transport policy.

WP5 selected projects with considerable policy implications which were capable of having asignificant direct impact on the transport policy and on the policy process.

The most logical and valid technique of research impact evaluation is a comparison betweenthe state "before" and "after" research project/programme results implementa-tion/exploitation. There are 2 main possibilities: theoretical/analytical/qualitative estimationof potential effects, and experimental - panels with core users, questionnaire surveys andinterviews. WP5 decided to apply both analytical and experimental techniques for estimationof FP5 and FP6 transport project's contribution to C&PP.

In addition to final report reviews/ using analytical techniques/, WP5 have conducted ques-tionnaire surveys and carried out interviews with partners and lead users of FP5 and FP6transport projects. The results have been generalized with the help of an analysis (qualita-tively, as well as statistically). Finally, the results have been interpreted drawing upon in-sights from the broader literature on impact assessment. Based on WP5 experience/resultsof applying tools conclusions and recommendations for FP5 and FP6 transport projectsevaluation were developed.

2.2 Specific problems related to evaluation of transport research projects’impacts

Transport is a unique sector with many interrelations and potential impacts on differentother sectors, and that is one the main difficulties of detection of research project impactson Community and Public policies.

Impact Evaluation (IE) consists in detection and measuring the changes that were causedby project/programme results exploitation by beneficiaries. It typically is conducted by com-paring states of impact domains, beneficiaries /control group of target users/, before andafter a project/programme results have been implemented.

Effective, rigorous and accurate evaluation is possible only using methods, which are able tofind the real causal impact of a project/programme by observing the beneficiaries’ outcomesbefore and after the use of project results.

2.3 Literature review

WP5 surveys confirmed the Special Report1 findings and conclusions concerning Meth-odological difficulties and limitations for FPs evaluation.

There has been no systematic attempt to collect data from participants after the completionof the research project, except for specific purposes. The use of WP5 surveys has been hin-dered by the fact that coordinators, in contrast to national RTD programmes. For example,Germany, the Czech Republic and some other countries are not obliged to provide dataabout monitoring and exploitation of research results.

1 Source: Court of Auditors Special Report No 9/2007 concerning „Evaluating of the EU Research and Technological De-velopment (RTD) framework programmes – could the Commission’s approach be improved?

Grant No. 213546

9/65

MetronomeMetronome

The Court of Auditors Special Report No 9/2007 “Evaluating the EU Research and Techno-logical Development (RTD) framework programmes – could the Commission’s approach beimproved?” states that: “In depth evaluation of research programmes requires the use ofappropriate methodologies and techniques. “ The Court checked the extent to which specificguidelines for the evaluation of the FPs existed, whether information needs were properlyanalysed on a timely basis and whether relevant data were made available to evaluators.

This means that before evaluation of FP5 and FP6 it is necessary to spend a lot of time forcollecting the data necessary for reliable evaluation of project/programme.

In our case, there were two options:

1. To develop evaluation strategy only on the basis of existing data /which werevery poor – FP5 and FP6 ex ante evaluation was not conducted/ without collec-tion of additional data and information.

2. To attempt to collect data and information with help of surveys and using varioussources of information and try to carry out more detailed and reliable methodo-logical approach for project/programme evaluation.

The second option was chosen in WP5.

Grant No. 213546

10/65

MetronomeMetronome

3 Methodology for evaluation of project impacts onC&PP

3.1 WP5 methodological approach

WP5 chose to proceed in a step-wise fashion: first, conceptual models and modules weredeveloped. Second, these conceptual models were implemented and tested. Stepwise ap-proach is sufficiently flexible allowing modifications for different types of projects, key ac-tions and thematic programmes and the estimation of possibilities for the Work packagemonitoring and evaluation. In general, the following components were included in themethodology for evaluating the programme -level and project -level impacts.

Analytical:

- Project final report estimations and evaluations (evaluation matrix)

- Analysis of dissemination activities (dissemination quality matrix)

Surveying:

- Coordinator – oriented questionnaires

- Lead-user (research project result end-user) interviews

The most important type of survey used to collect information on programme/project im-pacts were lead-user surveys, conducted among potential and target users of FP5 and FP6projects. Please note that the selection of potential lead-users was based on potential re-spondents’ characteristics and not on their potential interest for the specific projects in thesample. This means that the end-users views do not reflect only their opinion on the sampleprojects, but on a wider sample of FP5 and FP6 projects.

Combination of theoretical/analytical and surveying methods was considered as the mostappropriate since

A qualitative scale should be used in most cases, in order to avoid too much ef-fort in data collection (e.g. low relevance to high relevance of difficulties in ob-taining input data)Analyses of dissemination/ and exploitation/ plans are very important for the de-tection of impact domains and target/focused groups of users identification.Surveying methods/questionnaire and interview/ could be the most appropriatedata collection methods to obtain additional data on the projects for interpreta-tionStatistics with the results of the questionnaires may be used to resume the re-sults

Figure 2 presents the main characteristics of WP5 methodological approach.

Grant No. 213546

11/65

MetronomeMetronome

Figure 2. Main characteristics of WP5 methodological approach.

3.2 The data

The first two steps of the WP5 approach consider the data availability. The main datasources screened were as follows:

1. TRKC database (http://www.transport–research.info/ )

The TRKC website provides summaries and final reports of individual projects, as well asinformation on analyses and publications.

Transport Research Knowledge Centre database was added on the basis of input from coor-dinators, but not all data for finished projects are up-to-date.

2. CORDIS website

Analysis of data availability

Data collection and quality analysis

Screening and selection of sample of FP 5 and FP6 projects

Methodology fine-tuning

Analyses of dissemination and exploitation plans/dissemination quality matrix/

Final results based on matrix evaluations and surveys

Identification of survey subjects and survey design/coordinator survey//lead user survey/

Evaluation design

Analyses of final reports /evaluation matrix/

http://www.transport

Grant No. 213546

12/65

MetronomeMetronome

RTD - Projects Database on the Community Research and Development Information Service(CORDIS) (http://cordis.europa.eu/ )

3. Individual project websites

4. Czech National Information Centre for European Research Database (www.tc.cz)

This database includes information and evaluation of national participation in FPs. Not allthese data sources are tailored for impact assessment. Significant part of available data:coordinator’s name and contact information, webpage and so on - is not updated. Table 1presents the problems in data availability.

Table 1. Problems with data availability.

Availability aspects Comments

Availability Not all data needed for evaluation are available. Missing data:ex ante evaluation, mid – term evaluation, ex post evaluationof FP5 and FP6. Data are a bottleneck for robust pro-ject/programme evaluation

Accessibility Not all data known are accessible. Dissemination and Exploi-tation plans often are confidential.

Relevancy Data available and accessible are not relevant

Data quality evaluation and assessment based on Table 2 criteria, in order to determineif the data available are usable and reliable for WP5 intended purposes.

Table 2. Data quality.

Data Quality Comments

Timeliness Data are not actual/current

Comprehensiveness For evaluation purposes, data are not comprehensive

Precision Data with not enough precision and accuracy

Comparability Data sets are often defined differently. There is no datastandard or format for final reports.

Data analysis on C&PP confirmed the following conclusions of Court of Auditors SpecialReport No 9/20072:

Insufficient analysis of information needsInsufficient attention has been paid by the ‘research DGs’ to understanding what data is re-quired for evaluation and monitoring purposes. Rather than carrying out a systematic andcomprehensive analysis of information needs, the Commission has specified data require-ments late in the programming period and without due regard to overall evaluation needs.

2 Source: Court of Auditors Special Report No 9/2007. Evaluating the EU Research and Technological Development (RTD)framework programmes – could the Commission’s approach be improved?

http://cordis.europa.eu/

http://www.tc.cz

Grant No. 213546

13/65

MetronomeMetronome

The FP6 socioeconomic questionnaire was the only common attempt among DGs to agreeon such data. However, this questionnaire was finalised in 2004 (three years after the startof the programming period) and primarily covers aspects such as gender participation andenvironmental impact.

Some of the main methodological difficulties in measuring the impact and results of publicintervention in RTD are:

Attribution problemsAttribution problems are manifest in evaluations and performance measurement. They areclosely linked to the ‘additionality’ question, which asks whether changes would not havehappened anyhow. The difficulty of attributing results to a specific public intervention is alsoinfluenced by a large number of external factors (such as the effects of other policies or pro-grammes, changes in the legal framework, societal changes, etc.). Furthermore, in research,not achieving the expected results can also be a success. A further complication is the ap-parent difficulty of clearly attributing research outputs, not to mention outcomes and im-pacts, to specific aspects of the RTD process, or even to individual entities involved in a re-search activity.

Measurement problems

Data availabilityGenerally speaking, most data available, in particular the data collected through monitoringsystems, is on the inputs and outputs of RTD activities, rather than on outcomes or impacts.

Understanding the dynamicsGiven the apparent difficulty to model the causal relations between inputs and outputs, theinner workings of the research process are unclear; and in the absence of explicit interven-tion logic, the process itself is mostly considered as a ‘black box’.

Comparability of resultsThe extent to which a comparison of results between different scientific fields can be madeis limited, so that these generally should be interpreted in the context of specific pro-grammes and sub-programmes. In addition, the evaluation of results is more difficult for ba-sic (or fundamental) research, than for applied research or technological development. Themain reason for this is the fact that basic research generally consists of unique, non-replicable procedures. Even in the field of applied research, results vary considerably andcannot therefore be compared easily.

AggregationConcluding from evaluation results in terms of outputs, outcomes and impacts attained atthe lower level for the next higher level poses problems (for example when aggregating pro-ject results, first for clusters of projects, then for domains within a programme and finally atthe level of the programme).

Adequacy of indicatorsAnother question is whether evaluators are measuring the right thing. Metrics must be easyto measure, difficult to manipulate and drive the right behaviour. However, a ‘measurable’indicator for output does not necessarily permit conclusions on outcomes or impacts to bedrawn.

Timing problemsA further obstacle is the considerable time lag from the research activity to the generation ofoutputs and outcomes, so that impact can be assessed. Data needs to be collected over thelong term so that meaningful and robust conclusions can be drawn. Such time scales for

Grant No. 213546

14/65

MetronomeMetronome

evaluation are often not consistent with the expectations and needs of policy makers, and atrade-off must be made between early delivery for policy making and allowing the adequatetime for the effects to emerge.

Project fallacyEvaluation of outcomes and impacts of individual projects within a programme needs to takeaccount of the dilemma that the technological and scientific results have usually been ob-tained using previous work. Similarly, before the publication and exploitation of the scientificresults takes place, the project results may be used for other work by the participant. There-fore, a project in itself is not necessarily the appropriate assessment framework, in particularfor longer-term outcomes and impacts.

3.2.1 FP5 structure and objectives

In the next step, the framework programmes’ structure is analysed.

The FP5 budget was allocated to 21 key actions, which were socio-economic priorities thegoals of which were to be achieved precisely by way of the FP5 projects. The key actionsconcentrated on the following five thematic programmes:

Quality of Life and Management of Living Resources (QoL)

Information Society Technologies (IST)

Competitiveness and Sustainable Development (GROWTH)

Sustainable Development and the Environment (ENVI)

Energy (ENERGY)

In addition to the thematic programmes, FP5 also encompassed these three horizontal pro-grammes:

Confirming the International Role of Research in the European Community(INCO)

Promoting Innovation and the Participation of Small and Medium Enterprises(INNO-SME)

Improving Human Research Potential and the Socio-economic Knowledge Base(IHP)

The goals of the individual key actions were outlined only generally and they were specifiedin detail for the given period in the relevant work programmes. All programmes had theirown programme management structures, the most important body of which was always therespective programme committee comprising representatives of all the participating states.A major innovation of FP5 was the concept of “Key Actions”.

FP5 included a very rich spectrum of various project types. The project type was given bythe activities of the contractors' consortium. Each project type had a specifically determinedgoal and the project type significantly affected the percentage of the financial contributionfrom the EC to the total cost of the project. For example, the EC participated with up to50% in the costs of research activities; in case of demonstration activities its share was30%; in so-called co-operative research this was 50%; costs of various types of fellowshipswere usually fully covered by the EC (paid to the individuals and institutions that hosted thevisit); miscellaneous 'accompanying measures' were again subsidised up to 100%; the EC

Grant No. 213546

15/65

MetronomeMetronome

contributed a fixed amount to conferences; it granted awards from a budget established inadvance; etc.

3.2.2 FP6 Structure and Objectives

The main objective of FP6 was to contribute to the creation of “European Research Area”(ERA). FP6 was composed of three Blocks of activities and included various project types.FP6 funding instruments were as follows:

Traditional instrumentsSpecific targeted research projects - STREP

The aim of this type is a generation of new knowledge – research outputsCoordination actions - CA

The aim of CA is a support to the coordination, cooperation and networking of re-searchers. For the most part of coordination actions no research was performed.

Specific support actions – SSAThe aim of SSA is a support to the implementation of the Framework Programme(support to policies). This type of projects is not a research – oriented one.

Specific projects for SMESpecific actions to promote research infrastructures

Three new instrumentsIntegrated projects –IP

This type of projects aimed at increasing Europe's competitiveness and major societalneeds. The aim of this type is a generation of new knowledge.

Networks of Excellence – NoEThis type of "instrument" aimed at strengthening excellence in European research.The aim of this type is the integration of partners for future joint research. No re-search performed.

Article 169 for the joint implementation of national programmesThis instrument requires co-operation at the level of national governments.

According to WP5 coordinator survey (Figure 3), FP5 coordinators very much agree thatfunding instrument was satisfied for a project. In FP6, the satisfaction to funding instru-ment is lower.

Grant No. 213546

16/65

MetronomeMetronome

0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

FP5 FP6

agree strongly agree neither/ nor disagree disagree strongly n.a.

Figure 3. Funding instrument satisfaction.

3.3 European Strategic C&PP objectives and related indicators

The following objectives form the basis of the methodological development in the MET-RONOME project. The objectives were derived from policy documents and FP5 and FP5Work Programmes in the course of METRONOME WP2.

Figure 4. European Strategic objectives relating to C&PP in the filed of transport.

Based on the objectives, WP5 has developed the following preliminary common indica-tors to describe the effects/ contributions to policy, i.e. the project/programme impact toC&PP. These indicators have been used as one of the building blocks for the matrix andsurvey approaches described in detail in the following sections.

European Community and Public Policies Objectives

Improvements in the relationship between research and policy at all levels in the EU, timelyand effective scientific inputs for wide field of policies

Improvements in the efficiency and transparency of decision making

Improved policy coherence

Expanded and improved investment in human capital

Development of the European Research Area and scientific support for EU enlargement

Better implementation of existing EU environment policies and legislation at the local level

Greater coherence between macroeconomic and structural policies

Good governance

Co-operation between ministries

Quantitative tools for decision making, reflecting socio-economic objectives

Grant No. 213546

17/65

MetronomeMetronome

1. Identification of constraints

2. Political decisions adopted at European or national level

3. Legislative acts prepared with the use of research results

4. Use of results in other projects

5. Amount of resources for project continuation

6. Utilization of project results for creating new policy objectives

3.4 Selection of FP5 and FP6 projects and data collection

In this fourth step of the evaluation, the final list of projects to be evaluated in WP5 is de-fined. The preliminary list of projects was produced in WP2. This fine-tuning of the list in-cludes the verification of their suitability for the C&PP approach, the availability of researchreports in the corresponding internet sources, and the verification of their complete finalisa-tion, necessary for the implementation of the METRONOME methodology.

3.5 Analyses of Final Reports - Evaluation matrix

3.5.1 Research objectives

The first stage of the process was the identification of the research objectives relevant toeach of the projects that were selected for evaluation. This involved a review of the FP5and FP6 research and commissioning structures. The review identified that three levels ofobjectives are relevant to many of the transport research projects commissioned underthese Framework Programmes. These are:

Specific ‘Work Programme’ or ‘Thematic Area’ objectives;

Key Action (or equivalent) objectives of the Work Programme that the projectwas commissioned under; and

Strategic project objectives.

In the case of projects commissioned under the FP6 IST programme an additional level ofobjectives was identified, as this Priority Thematic Area is structured slightly differently tothose that other projects reviewed were commissioned under.

The first two levels of objectives were identified from the Transport Research KnowledgeCentre (TRKC) online resource (http://www.transport-research.info/web/), which is commis-sioned by DG-TREN. There are a range of official resources that detail the objectives ofeach Work Programme, although for consistency all of the objectives contained within thematrices were sourced from the TRKC, which contains information about transport researchprogrammes from across the European Research Area.

A number of the Work Programmes and Key Actions (and their equivalents) are directly re-lated to the improvement of Community and Public Policies, and so all of their objectives arerelevant to the review of FP5 and FP6 projects for their contribution to these policies. Anumber of the objectives that research projects have been commissioned under are not,however, directly related to Community and Public Policies, and owing to the scope of themethodology, these objectives were excluded from the matrices. While some of the objec-tives that were excluded could be considered to have an indirect impact upon the SD of theCommunity, full or partial fulfilment of these objectives was not thought to give a good indi-

http://www.transport-research.info/web/

Grant No. 213546

18/65

MetronomeMetronome

cation of the extent to which a transport research project contributes to the improvement ofCommunity and Public Policies.

The objectives of all of the relevant FP5 and FP6 Work Programmes, Key Actions and theirequivalents were tabulated and a systematic review of these objectives was conducted toidentify which should be included in the evaluation matrices. The decisions made were thenvalidated by an expert within the Centre for Sustainability (C4S) at TRL.

The strategic project objectives were identified using the RTD - Projects Database on theCommunity Research and Development Information Service (CORDIS) (http://cordis. eu-ropa.eu/). The completeness and quality of the project information contained within thisproject database, which contains details of RTD projects financed wholly or partly by the EC,meant that in some instances objectives, were instead obtained from the official projectwebsites of the selected projects. The source of the strategic project objectives (eitherfrom CORDIS or project websites) is specified on each evaluation matrix.

The evaluation matrix for each project also contains wider European EU Community andPublic Policy objectives. These objectives have been identified in WP2 during the projectselection phase and are wide-ranging in their scope. The projects were also reviewedagainst three of the four indicator groups identified by WP2 (see below).

3.5.2 Impact indicators

WP2 reviewed existing evaluation methodologies. This review was used as a basis for iden-tifying indicators that projects could be reviewed against to determine their impacts. Theindicator groups developed by WP2 are as follows:

Scientific impact indicators (these indicators explore the quality and validity of re-search projects);

Customer/end-user impact indicators (these reflect the benefit of the research re-sults to end-users);

Societal impact indicators (these reflect the impact of the research to society atlarge); and

Management and co-ordination impact indicators (referred to as co-ordination in-dicators within the evaluation matrices these ‘enabling factors’ complement theimpacts measured in the above three categories).

A number of these impact indicators were included within the evaluation matrix. Thesewere added to the matrix to provide a more general overview of the impacts of the researchprojects, and to indicate the specific ways in which transport projects have contributed tothe improvement of Community and Public Policies in Europe.

The first category of impact indicators identified by WP2 (scientific) was not included in theproject evaluation matrices. This is because for the purpose of the evaluations it was con-sidered that an evaluation of the extent to which research projects had fulfilled the specificobjectives that they had been commissioned under (as detailed previously) would give asufficient indication of a project’s scientific impact upon improvement of Community andPublic Policies.

The research objectives and impact indicators form the basis of the evaluation matrices, forwhich a number of different approaches were trialled. Two of the approaches developedwere considered to be fit for purpose, and so an FP6 project was selected to trial the two

Grant No. 213546

19/65

MetronomeMetronome

approaches against. The pilot demonstrated that one of the approaches was considerablymore effective than the other. A skeleton template for the approach adopted is shown inEvaluation matrix below.

Fully/yes Partially Indirectly Not at all/no

Wor

kp

rogr

amm

eob

ject

ives

Soci

etal

imp

act

indi

cato

rs

Co-

ord

inat

ion

indi

cato

rs

Extent to which the project met/contributedto objectives

Cust

om

er/e

nd-

use

r im

pact

indi

cato

rs

Key

act

ion

obje

ctiv

es

Stra

teg

icp

roje

ctob

ject

ives

Euro

pea

nS

D p

olic

yob

ject

ives

Figure 5. Evaluation matrix.

This matrix supports a qualitative evaluation of the extent to which FP5 and FP6 researchprojects have contributed to the improvement of Community and Public Policies. It enablesreviewers to specify whether each of the objectives have been fulfilled fully, partially, indi-rectly or not at all. The actual assessment was performed based on the analyses of selectedproject final reports.

This will result in a more comprehensive evaluation in which evaluators can make the valu-able distinction between whether projects that have contributed to Community and PublicPolicy related objectives have done so fully, partially or indirectly. In practice the perform-ance of research projects in relation to impact indicators is likely to be more clear cut, withimpacts either having been realised or not.

It was decided that each completed evaluation matrix should be accompanied by a textualsummary. The summary should supplement the matrix by detailing other relevant and spe-cific information about projects and/or their outcomes.

Com

mun

ity &

Publ

ic P

olic

yob

ject

ives

Grant No. 213546

20/65

MetronomeMetronome

3.6 Analyses of dissemination activities – Dissemination quality matrix

To fully evaluate the impact of FP projects on Community and Public Policies there is a needto examine the dissemination activities that have been conducted within each project.

FP projects typically result in the publication of a wide range of deliverables and outputs,both formal and informal. For the purpose of this exercise, the dissemination reports werereviewed to provide evidence of the scope and nature of dissemination activities conducted.In projects where no dissemination report had been produced final project reports and pro-ject websites (where these had been produced and where they were still active) were re-viewed. Where dissemination reports had been produced at the beginning of project finalreports or website links were also referred to for actual outcomes.

To ensure that the impact of dissemination activities was given an appropriate emphasis aseparate matrix was developed in which to detail specific dissemination activities. Severalsets of indicators were trialled, and the set that was used is detailed in Figure 6 below.These indicators were selected on the basis that they were comprehensive whilst also feasi-ble to be answered based upon written documents in the public domain.

Figure 6. Dissemination quality matrix.

The structure of the matrix enabled a qualitative analysis of the potential impact domain,period of dissemination, user's groups and nature of their engagement.

List of dissemination activities

Purpose of dissemination activities

Were potential end-users and beneficiaries of the results directly involved or consulted in thedifferent stages of the project?

Did dissemination occur throughout the project or only after project completion?

Was there a piloting of results with potential end-users before finalisation?

Were plans made to update the results based upon the outcome of the dissemination activi-ties?

Ove

rvie

w

Was there a specific dissemination work package

Type of event(s) organised

Aspects of the project that were disseminated

Type of audience at event(s) (if gov. local, national or European?)

Size of audience at event(s)

Timing of event in relation to project lifecycle

Even

ts

Location of event(s)

Publications

Aspects of the project that were disseminated

Nature of publications

Intended readership/audiencePubl

icat

ions

Timing of publication in relation to project lifecycle

Oth

er

Non-formal dissemination (i.e. discussions with stakeholders)

Grant No. 213546

21/65

MetronomeMetronome

3.7 Surveys

3.7.1 Co-ordinator survey

In order to identify project coordinators’ views on FP5 and Fp6 project impacts on C&PP, aquestionnaire was prepared. The questionnaire consists of four sections of questions relat-ing to (A) scientific impacts, (B) societal and end-user impacts (C), management, co-ordination and dissemination impacts and (D) finance. The questionnaire uses the followingsix-point scale for answers:

1. agree strongly 2. agree 3. neither /nor 4. disagree 5. disagree strongly 6. Don’t know

The main goals of this questionnaire were:1. Collection of necessary information for further evaluation,2. Coordinator‘s (self) evaluation of projects’ impacts

A template of the questionnaire is presented in Annex 2 of this Deliverable.

3.7.2 Lead user questionnaire and interviews

Identification of the target or at least potential users (lead users) and their key concerns isessential for the development of an appropriate evaluation plan. Without determining theusers who are interested in the results of the evaluation and without understanding how theevaluation results will be used it is very difficult to cover all aspects of the evaluation and tofocus it. Feedback from potential users is very important for evaluation strategy definition,specification the priorities and the total approach for evaluation.

METRONOME lead user questionnaire was used to assess lead user opinions on the impactand performance of FP5 and FP6 research projects.

METRONOME partners approached self-selected lead users (defined as persons (civil ser-vants, consultants, scientists, policy makers, etc.) really using the knowledge gained fromEU-research). In order to maintain consistency between the results of interview by differentpartners, a uniform questionnaire format was developed and applied. The partners werefree to use either telephone or face-to-face interviews or distribute the questionnaire byemail to pre-selected respondents.

The questions in the questionnaire related to the perceived impact of FP5 and FP6 researchin general, the results of specific projects in which the respondents had been involved, thebenefits for the respondent and his/her organisation and what did and did not work in FP5and FP6 projects. In the course of the METRONOME WP5 activities, the evaluation of a sam-ple of FP5 & FP6 projects has been undertaken, using the METRONOME methodology fo-cused on the assessment of project impacts on improving Community and Public Policies.

These assessments were discussed between the two persons and based on the two assess-ments the final evaluation-format was filled in. A template of the lead user questionnaire ispresented in Annex 2 of this Deliverable.

Grant No. 213546

22/65

MetronomeMetronome

4 Methodology testing

4.1 Selection of FP5 and FP6 projects and data collection

On the basis of screening, the following projects were selected for WP5 methodologytesting.

Table 3. Selected FP5 and FP6 projects.

FP5 FP6

BESTUFS CAESAR

ETIS-BASE HEATCO

GUIDEMAPS MOTOS

IASON PEPPER

IMMORTAL REFIT

MOST REORIENT

PROTRANS SMILE

ROSEBUD TRANS - TOOLS

SAMNET

SULOGTRA

THEMIS

THINK -UP

TRANS –TALK

The total sample size for WP5 methodology testing is 21 projects: 13 from FP5 and 8 fromFP6. This number was received from the estimation of time and resources for WP5 method-ology application and validation. Selected projects covered a wide range of various types ofprojects. The projects were selected through a multi-stage screening process on the basis ofproposed criteria.

The sample was allocated by FPs based upon anticipated research project impacts takinginto account that there were more information about FP5 projects dissemination and exploi-tation.

Figure 7 presents the partner distribution in the selected P5 and FP6 projects.

Grant No. 213546

23/65

MetronomeMetronome

Partners of selected FP5 a FP6 projects

52%

28%

20%

EU Member States (15)New Member StatesOther States

Figure 7. Partners of selected FP5 and FP6 projects.

4.2 Results from Matrix Evaluation – objectives and impacts

In the Work Package 5, projects were reviewed against two sets of criteria:

1. their contribution to research objectives

2. their impacts, based on impact indicators identified in WP2

The project evaluations (reviews) were carried by the WP5 members whose institutionswere not participating in the evaluated projects. Final report of each project was reviewedby one independent evaluator.

The primary review concentrated on share of assessments expressed in positive responseson achieving the relevant objectives fully, partially or at least indirectly. The evaluation crite-ria enable to mark several levels (in case of some projects both “fully”, “partially” and “indi-rect” might have been marked). This first phase review results showed that the goals weremet in almost all projects at least to a certain extent.

In the second phase, the results were analysed in details; the results varied significantly ac-cording to the type of projects, type of funding and project goals on one side and the qual-ity of the final reports on the other one.

The comparison of findings of these two types of analyses highlights a need of setting andselection of the appropriate criteria relevant to the relevant type of research activities andtheir expected impact.

Altogether 20 final reports were analysed. Annex 3 of this deliverable contains the full list ofobjectives and impact indicators that FP5 and FP6 projects were reviewed against to esti-mate their potential contribution to Community and Public Policies. The only objectives notdetailed within the Table are the Strategic Project objectives, which have been withheld topreserve the anonymity of the projects reviewed.

Grant No. 213546

24/65

MetronomeMetronome

Objectives

The set of objectives that were the best met were the Strategic Project objectives. The ma-jority (90%) of these for FP5 projects were considered to have been ‘fully met’. For FP6 pro-jects the percentage is lower (51%), but in both cases this set of objectives has been metbetter than others have. This is not a surprising finding as the Strategic Project objectivesare those with the most direct relevance to the project. The fulfilment of higher-level objec-tives, such as Work Programme, Key Action and Programme Subdivision objectives, insteadindicates whether the project is likely to have broader impacts upon CP&P, which is why it isto be expected that these will not be as fully contributed to as those that the project wascommissioned under. It should also be noted that it is not possible to evaluate projectsbased on the extent to which they have contributed to higher-level objectives. This islargely because the purpose and scope of projects commissioned varies considerably andthat the magnitude and nature of the impacts and contributions will necessarily vary be-tween projects. A review against contribution to higher-level objectives does, however,provide an indication of depth and breadth of contribution.

The low sample size of projects reviewed limits the reliability of findings, but from the pro-jects reviewed it is surprising that both the FP5 and FP6 projects were considered to havecontributed more ‘fully’ to Work Programme objectives than to the Key Action objectives(and in the case of FP6 projects also Programme Subdivision objectives). Projects con-ducted in both FP5 and FP6 were considered to have not contributed at all to a greaternumber of Key Action and Subdivision objectives than Work Programme objectives. This issurprising because the lower level objectives (Key Action and Programme Subdivision) couldbe considered more directly applicable to the projects commissioned. It is, however, a find-ing also shared with the projects that were reviewed by WP4.

Based upon the reviews conducted the potential contribution and impact of FP5 and FP6projects to Work Programme objectives is almost identical (see Annex 4). The analyses re-vealed that a large percentage of work programme objectives have been achieved. Contri-bution to Key Action objectives is significantly higher for FP5. The low sample size alsomeans that it cannot be ascertained whether the ‘Programme Subdivision’ objectives thatare set in FP6 but not in FP5 have had any impact upon this variation in results.

Contribution to Community and Public Policies is very similar between both FP5 and FP6projects, and although there can be seen to be some differentiation between the two FPsthis could be owing to the different sample sizes as much as to differing contributions toCommunity and Public Policies.

On an individual project level, the results indicate fairly clearly that there is not a close rela-tionship between Strategic Project objectives and wider Community and Public Policy relatedobjectives, either on an EU or a WP level. This is again not a surprising finding, as the de-gree of comparison will almost inevitably vary based upon the purpose and nature of eachproject. The contribution of projects that have met all of their Strategic Project objectivesto other relevant sets of objectives was too variable to support any meaningful conclusionsin this respect.

Impact indicators

The potential impact of projects in terms of the customer/end-user, societal and co-ordination indicators used was consistently positive across the range of projects reviewed.Management and co-ordination impacts received the most positive responses in both FP5

Grant No. 213546

25/65

MetronomeMetronome

and FP6. As shown in Annex 4, there is evidence that either as part of a conscious effort oras part of standard FP processes the potential impacts may be suggested in almost all pro-jects. In the case of FP5 and FP6 projects where potential customer/end-user impacts werenot thought to have been considered this was not the result of any considerations havingbeen overlooked by the project consortiums, rather that certain indicators, notably whetherthe projects resulted in the definition of new regulations, standards or policy measures,were not applicable to the programmes of work being undertaken.

The following figures present the main findings of the Matrix evaluation. The analysis re-vealed that C&PP objectives presented in Work Programmes were achieved little better inFP5 than in FP6, but in both cases, they were achieved to a very large degree. In addition,projects were considered to generate positive impacts within all of the impact groups.

OBJECTIVES - FP5 PROJECTSExtent to which projects met the objectives

0%

20%

40%

60%

80%

100%

WorkProgramme

Key Action StrategicProject

Community andPublic Policies


Figure 8. Objectives of FP5 projects - peer reviews.

IMPACT INDICATORS - FP5 PROJECTS

0%

20%

40%

60%

80%

100%

Customer/end-user Societal impact Co-ordination


Figure 9. Impact indicators of FP5 projects - peer reviews.

Grant No. 213546

26/65

MetronomeMetronome

OBJECTIVES - FP6 PROJECTSExtent to which projects met the objectives

0%

20%

40%

60%

80%

100%

WorkProgramme

Key Action ProgrammeSubdivision

StrategicProject

Communityand PublicPolicies


Figure 10. Objectives of FP6 projects - peer reviews.

IMPACT INDICATORS - FP6 PROJECTS

0%

20%

40%

60%

80%

100%

Customer/end-user Societal impact Co-ordination


Figure 11. Impact indicators of FP6 projects - peer reviews.

Recommendations

The matrix used within the project impact evaluation can be a useful tool. Nevertheless,when applied in whole programme assessment, it could be rather problematic. The mainanticipation is a quantifiable format of the used matrices. By combining and weighing thescores for the various items (work programme objectives, key action objectives, etc., etc.) itmay help to avoid the difficult analysis of each separate question.

The main drawback of the matrix approach is that it is very theoretical, and just becauseproject results can be seen to directly contribute to a range of results this does not meanthat this contribution has been realised in practice. To avoid subjectivity, we can make rec-ommendations on the quality of reporting in order to avoid vague interpretations of finalreports.

Grant No. 213546

27/65

MetronomeMetronome

Another weakness can be seen in the very small sample size of informants and missing an-swers from end-user group.

For the end-user interviews to provide meaningful results it might be useful for the evalua-tor to create a list of the full range of end-user types/categories specific to each project be-ing evaluated and for a selection of people to be interviewed within each of these catego-ries. This would help the full range of impacts to be identified, although the data generatedfrom the interviews could not be considered generalisable/fully representative.

4.3 Results from Matrix Evaluation – dissemination activities

The structure of Dissemination evaluation matrix (see section 3.6) enabled a qualitativeanalysis of the potential impact domain, timing of dissemination, beneficiary/user groupsand nature of their engagement. Dissemination evaluation was carried out for 13 FP5 and 8FP6 projects.

FP5

Main dissemination channels and tools in FP5 projects were conferences, workshops, semi-nars, project websites, user group surveys, user/beneficiary panels, and training courses.Dissemination concerned all results and all aspects of the projects. Dissemination eventswere organised at European level rather than national or local levels. Typical audiences forevents included EC officials, scientists, researchers, policy makers, decision makers, experts,trans-port operators, logistics operators.

Only 15% of the evaluated projects indicated that a target or control user/beneficiary groupfor analysing the needs of future users, finding the best way to address them, and ensuringthe effective feedback was defined.

It is possible to state that for the most part a direct involvement of potential users in FP5research projects was not planned and performed. A website was the only channel for indi-rect involvement of persons concerned and potential users of project results in FP5 projects.Analyses of dissemination activities show that in the majority of cases dissemination activi-ties fell on the final stage of project lifecycle. About 85% of projects indicated efforts fo-cused on piloting of results before finalisation of results.

Based on findings it may be concluded that for the majority of evaluated FP5 projects dis-semination activities were used only for presentation of project results, rather than for con-sultations and as an opportunity to test and improve project results.

WP5 investigations show that the adjustment/modification of results indicated only thoseprojects, which from the beginning have defined a user group/user panel and consulted pre-liminary results with this audience. Taking into account that many projects planned theirdissemination activities after project completion, there was no opportunity/and time/to mod-ify results with due regard to dissemination results. In 46% of projects, a special WP fordissemination of project results was established.

The publications produced within FP5 projects were articles, handbooks, databases, recom-mendations, best practices handbooks, toolkits, position papers, press releases, case stud-ies. Intended readership/audiences included EC officials, scientists, policy makers, experts,

Grant No. 213546

28/65

MetronomeMetronome

policy advisors, consultants to the government, transport operators, government officials.About 80% of projects indicated publications after the project completion.

FP6

Main dissemination channels and tools in FP6 were conferences, workshops, seminars, sur-veys, websites, potential user group surveys, and user panels/user forums, special teamsfor dissemination and exploitation, handbooks, and telephone interviews. Disseminationconcerned all results and all aspects of the projects. Dissemination events were organised atEuropean level rather than national or local levels. Typical audiences for events includedscientists, researchers, policymakers, experts, operators, transport planners.

Only 25% of evaluated projects indicated a definition of target user group for analysing theneeds of future users, finding the best way to address them, and ensuring the effectivefeedback. This is, however, higher rate than for FP5 projects.

Majority of projects used a website for presentation of project results. Analyses of final re-ports show that project consortium’s efforts were concentrated essentially on the achieve-ment of project objectives, rather than on consultations with users. Findings show that 75%of projects presented consortium efforts focused on piloting of results before finalisation ofresults. Dissemination manager work is a commendable example of piloting.

As regards updating the results based upon the outcome of the dissemination activities, theresults are similar to FP5 results. 50% of evaluated FP6 projects have established a specialWP for dissemination, which is also comparable to FO5 results. 75% of project final reportsindicated that dissemination has started at the end or after reaching final project results.

The publications produced within FP6 projects were articles, handbooks, databases, recom-mendations, best practices handbooks, toolkits, position papers, press releases, leaflets,case studies. Intended readership/audiences included EC officials, national government offi-cials, scientists, policy makers, decision makers, experts, policy advisors, consultants to thegovernment, transport operators, logistics operators. About 80% of projects indicated publi-cations after the project completion.

In general, we can conclude that analyses of dissemination activities agree with conclusionsfrom the survey for coordinators that there is a need for dissemination improvement. A de-tailed analysis of the Dissemination quality matrix results is presented in Annex 4 of this De-liverable.

4.4 Results from the co-ordinator survey

A questionnaire was submitted in January 2009 to 21 coordinators of FP5 and FP6 projectsin the field of transport completed during the period 1998 - 2008. We received 15 responses(response rate - 71, 4 %) resulting in 8 completed questionnaires for FP5 – response rate –61,5 % and 7 – for FP6 (response rate – 87,5 %). RR for FP5 is lower than expected. FP5projects were finished many years ago and contact details of coordinators were changed.

The respondents who did not complete the questionnaires indicated various reasons, suchas the coordinator was not available due change of job, retirement, change of address andso on.

Grant No. 213546

29/65

MetronomeMetronome

The primary evaluation is based on positive responses to the questionnaire (percentage ofrespondents-project coordinators stating “agree strongly” or “agree”). The comparison be-tween FP5 and FP6 projects shows that projects completed relatively recently are assessedmore critically and the results vary more significantly. Despite certain subjectivity of the re-spondents´ view these results indicate that the evaluation process should be carried out assoon as possible. This is particularly import with respect to the “self-evaluating” questions,where the respondents tend to score highly the activities in their direct responsibility. WP5experience showed that coordinator’s suggestions and comments submitted in Notes part ofquestionnaire are very valuable for project and programme evaluation.

The survey revealed that project co-ordinators considered impacts of FP5 projects in all fourimpact groups ((A) scientific impacts, (B) societal and end-user impacts (C), management,co-ordination and dissemination impacts and (D) finance) to be higher than of FP6 projects.

The most positive answers (performance successes) in both FPs were identified to thequestions regarding- the scientific competence of consortium- contribution to the development of decision making tools- involvement of civil servants and policy makers in the projects- publishing project results in scientific journals or books (FP5)- improved networking between researchers and public/private organisations (FP6)- financial issues (FP5)

Dissemination of project results received somewhat conflicting results. 80% of the FP5project co-ordinators agreed that project results had been adequately disseminated toend-users, but the rest 20 % disagreed. Regarding FP6, about 50% of the project co-ordinators agreed that project results had been adequately disseminated to end-users,the rest neither agreed nor disagreed.

The most negative results were received regarding the following questions/statements- project results have been transferred into policy recommendations and regulations- project raised new unsolved research questions

The following figures present the main findings of the co-ordinator questionnaire surveyin detail.

Grant No. 213546

30/65

MetronomeMetronome

The survey showed that FP5 scientific impact is for questions A2, A3, A5 and A6 higherthan FP6. For A1 and A4 it is the same (see the detailed questions on following pages).

Percentage of positive responsesA. Scientific impact

0%

20%

40%

60%

80%

100%

A1 A2 A3 A4 A5 A6

FP5 FP6

Results of the survey indicated that societal impact for FP5 is also higher than FP6.

Percentage of positive responsesB. Societal and user impact

0%

20%

40%

60%

80%

100%

B1 B2 B3 B4 B5 B6 B7

FP5 FP6

The same findings are true for responses to management/coordination questions

Percentage of positive responsesC. Management & Coordination

0%

20%

40%

60%

80%

100%

C1 C2 C3 C4 C5 C6 C7 C8 C9

FP5 FP6

Grant No. 213546

31/65

MetronomeMetronome

Satisfaction with project financing is considerably higher for FP5 projects

Percentage of positive responsesD. Finance

0%

20%

40%

60%

80%

100%

D1 D2

FP5 FP6

Grant No. 213546

32/65

MetronomeMetronome

A. Scientific Impact

A1: The goals (research objectives) set to the project were all metA2: The research goals required specific elaboration at the start of the projectA3: There was a good fit between the research framework and data availableA4: The scientific competence of the consortium was sufficient for the taskA5: Project results have been published in scientific journals and/or booksA6: The project raised new unsolved research questions in the field of Community and Public Policy

0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

A1 A2 A3 A4 A5 A6

FP5 - SCIENTIFIC IMPACT


0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

A1 A2 A3 A4 A5 A6

FP6 - SCIENTIFIC IMPACT


0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

A1 A2 A3 A4 A5 A6

FP5+FP6 SCIENTIFIC IMPACT


Grant No. 213546

33/65

MetronomeMetronome

B. Societal and End-user ImpactB1: The project has contributed to priority setting, e.g. future research or policy goalsB2: The project has contributed to new transport policy developmentB3: The project results have been transferred into policy recommendations and/or regulationsB4: The project has contributed to development of decision making tools (guidance, models)B5: The project has contributed to decision making processes of public or private organisationsB6: The project has led to new product or service development in the transport sectorB7: The project results have been used by public sector, industry or other societal actors

0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

B1 B2 B3 B4 B5 B6 B7

FP5 - SOCIETAL & END USER IMPACT


0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

B1 B2 B3 B4 B5 B6 B7

FP6 - SOCIETAL & END USER IMPACT


0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

B1 B2 B3 B4 B5 B6 B7

FP5+FP6 SOCIETAL & END USER IMPACT


Grant No. 213546

34/65

MetronomeMetronome

C. Management, Coordination and DisseminationC1: Needs and views of end users were taken into considerationC2: Civil servants and/or policy makers were involved in the projectC3: Transport operators or service sector were involved in the projectC4: Transport industry sector was involved in the projectC5: The project webpage was user-friendly and updated regularlyC6: The project webpage has frequently been visited by non-project-membersC7: The project results have been adequately disseminated to end users: civil servants and policy makers,public, industry etc.C8: The project (consortium) has improved networking between researchers and public/private organisations(e.g. initiated new research projects)C9: The project built or strengthened networks with global/EU/national research partners

0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

C1 C2 C3 C4 C5 C6 C7 C8 C9

FP5 - MANAGEMENT, COORDINATION & DISSEMINATION


0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

C1 C2 C3 C4 C5 C6 C7 C8 C9

FP6 - MANAGEMENT, COORDINATION & DISSEMINATION


0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

C1 C2 C3 C4 C5 C6 C7 C8 C9

FP5+FP6 - MANAGEMENT, COORDINATION & DISSEMINATION


Grant No. 213546

35/65

MetronomeMetronome

D. Finance

D1: The research budget (project size) was sufficientD2: The financial instrument was adequate for the project

0%10%20%30%40%50%60%70%80%90%

100%

D1 D2

FP5 - FINANCE


0%10%20%30%40%50%60%70%80%90%

100%

D1 D2

FP6 - FINANCE


0%10%20%30%40%50%60%70%80%90%

100%

D1 D2

FP5+FP6 - FINANCE


Figure 12. Findings of co-ordinator (questionnaire) survey.

Grant No. 213546

36/65

MetronomeMetronome

4.5 Results from lead user interviews

4.5.1 Methodology applied

The interviews took place in two ‘waves’. The first wave was performed in the first monthsof 2009. In this ‘wave’ the METRONOME partners approached a number of lead users (de-fined as persons (civil servants, consultants, scientists, policy makers, etc.) really using theknowledge gained from EU-research) in various EU members states. In order to maintainconsistency between the results of interview by different partners, a uniform questionnaireformat was developed and applied. The partners were free to use either telephone or face-to-face interviews or distribute the questionnaire by email to pre-selected respondents.

The questions in the questionnaire related to the perceived impact of FP5 and FP6 researchin general, the results of specific projects in which the respondents had been involved, thebenefits for the respondent and his/her organisation and what did and did not work in FP5and FP6 projects.

After analyses of the first response wave in March 2009, it was decided to enlarge the re-sponse by adding a second ‘wave’. The format was slightly changed to better accommodatethe use of email and besides lead-users pre-selected by the METRONOME partners, thepeople registered as potential participants in the (cancelled) 2nd METRONOME workshopwere asked by email to fill in and return a questionnaire. This second wave was performedin April and May of 2009.

A total of 23 usable questionnaires became available for analysis. The respondents werefrom the Czech Republic, Lithuania, Sweden, Great Britain, the Netherlands, Germany,Finland, Spain and Belgium. 22% of the respondents had not (in any way) been involved inFP5/FP6 projects and could therefore not answer any questions on specific project results.From the 18 remaining respondents six formulated their answers based on involvement inan FP5 project and 12 based on involvement in an FP6 project. This involvement differedfrom project partner to participant in project events (workshops, etc.). The distribution ofthe 23 respondents by background of the respondents is presented in Annex 5.

4.5.2 Main findings

The lead user interviews present complementary findings to the co-ordinator interviews,which indicate that the results can be considered realistic and plausible. Some, namely six,of the respondents were researchers, but this cannot be seen to affect the main direction ofthe results.

The most positive answers (performance successes) were identified to the questions regard-ing- high level of expertise within the project participants- improved networking between researchers and public/private organisations (especiallyFP6) and strengthened networks between international parties- taking the needs and views of end users were into consideration in the projects- usage of project results have been used by the public sector or other societal actor- dissemination of project results (only FP6, not FP5)- development of decision making tools

Grant No. 213546

37/65

MetronomeMetronome

Contribution to policy goals or policy development received slightly positive answers.The most negative results were received regarding the following questions/statements

- project results have been transferred into policy recommendations and regu-lations (complementary result to co-ordinator survey)

- project was cost effective (only FP6, not FP5)

Figure 13 presents the overall findings of the lead user interviews. The detailed results andfigures of the interviews are presented in Annex 5 of this Deliverable in the form of PowerPoint presentation.

0%

20%

40%

60%

80%

100%

users

involve

ment

pilotin

g resu

lts

disse

minatio

n WP

specif

ic audien

ce

FP 5 PROJECTS

YES NO N/A

0%

20%

40%

60%

80%

100%

users

invo

lvemen

t

pilotin

g res

ults

dissem

ination W

P

spec

ific au

dienc

e

FP 6 PROJECTS

YES NO N/A

Figure 13. Findings of lead user survey.

Grant No. 213546

38/65

MetronomeMetronome

5 Discussion and conclusions5.1 Methodological Development

Based on our experience of piloting some methodological tools, and in the context of theissues detailed in the table below, we recommend taking a stepwise approach to evaluatethe contribution of FPs to Community and Public Policies. The proposed approach is holisticwith each step providing a valuable insight into a different nature and aspect of impact. Theresults can therefore be combined to provide a comprehensive and multi-faceted indicationof the contribution of a specific FP to Community and Public Policies.

In addition, evaluation of the contribution of an FP to Community and Public Policies shouldinclude both theoretical and survey methodological tools. We have summarised the appro-priateness of the evaluation methods we tested in the following table.

Table 4. The appropriateness of the tested evaluation methods.

Purpose Strengths Weaknesses Feasibility of tool Recommendedas a tool?

Final reportanalysis

To evaluatethe extent towhich projectresults havecontributed toCommunityand PublicPolicy objec-tives.

Uses a base toprovide an indica-tion of the resultsof each researchproject evaluated.

Provides a holisticsummary of theresearch projectfindings, contribu-tion to objectives,estimation of im-pact areas andtypes.

Final reports of differ-ent types of projectshave significantly dif-ferent formats andcontents.

The output is theoreti-cally based with po-tential/expected im-pacts being impliedfrom scientific resultsand associated dis-semination and en-gagement activitiesrather than deter-mined from evidenceof impact ‘on theground’ or from anend-user perspective.

Overall: Yes.

Final reports are inalmost all instancesreadily available and,in many cases, con-tain comprehensiveand accurate detailsof project results.

The approach is timeconsuming /3 inde-pendent experts/ butinvestment of time isnecessary if a thor-ough understandingof the project, itsbackground and whatit has achieved is tobe gained. This is aprerequisite forevaluating the impactof projects.

The matrix is easy-to –apply.

Yes, but inconjunctionwith othertools for de-tection ofpotential im-pacts.

Review ofdisseminationactivities

To provide anoverview ofdisseminationactivitiesconducted byprojects to beused to de-velop a pic-ture of theactual likelyimpact ofprojects ‘onthe ground’(i.e. on end-users, audi-ence, do-mains).

It supports the finalreport analysis byproviding a moredetailed indicationof the likely im-pacts of researchprojects.

It is highly dependentupon the availability ofplan and report, levelof detail and quality ofreports on dissemina-tion activities andengagement. Evenwhere disseminationactivities are detaileddetails of other formsof engagement,and/or the stake-holders involved, canbe limited.

It does not demon-strate whether thestakeholders who wereengaged with actually

Overall: Yes.

The review does nothave to be conductedby an expert as thematrix is straightfor-ward and easy tounderstand.

Yes, particu-larly whencombined withother tools todemonstrateapplication ofthe projectfindings dis-seminated.

Grant No. 213546

39/65

MetronomeMetronome

applied the projectresults and thereforewhether it had anyactual impact.

Co-ordinatorquestionnairesurvey

To get theperspective ofthe co-ordinator onthe societal,end-user andscientificimpacts of theresearch andon issuesrelating tomanagement,dissemination,co-ordinationand finance.

Provides a self-evaluation andinsight into thecontribution ofprojects to Com-munity and PublicPolicy objectives,and into the poten-tial and actual im-pacts of the pro-ject.

The results can beused to reinforcethe validity of find-ings from othermethodological toolapplication.

There are numerousbarriers to effectiveresults being obtainedfrom self-evaluations,notably respondentbias. This could mani-fest itself in the desireto cast the project(and institution) in afavourable light. Co-ordinators could beunable to make anobjective assessmentof the results andimpacts of their pro-ject owing to theirhigh level of connec-tion with it.

Relies upon co-ordinators being will-ing to take the time tofully respond.

Overall: Yes.

The actual contactdetails of co-ordinators are readilyavailable.

Minimal time andresource needs to beused by the evalua-tor.

The survey is conciseand comprises ofstraightforward toanswer questions. Itis not likely to takelong to complete andco-ordinators arelikely to be willing toprovide feedback onprojects.

The approach is lessfeasible for projectsfinalised in the past.If co-ordinators moveroles or organisa-tions, they can bedifficult to contact;and experience indi-cates that the moretime that elapsesbetween the projectcompletion and beingissued the survey theless willing they areto respond.

Not in its cur-rent format.It could beused to rein-force findingsfrom otherapproachesbut muchcaution isrequired wheninterpretingself-assessmentsand the sur-vey adds littleto informationthat can begleaned fromother arguablymore objectivesources (nota-bly the finalreport reviewand dissemi-nation re-view). Tomake it morevaluable addi-tional informa-tion should bepursued.

End-userinterviews

To provide anindication ofthe degree towhich projectresults areactually beingused andapplied on theground, andthereforewhether theyare ultimatelyhaving animpact uponCommunityand PublicPolicies.

Gives a direct in-sight into the ex-tent to which re-search results arebeing applied inpractice.

Provides insightinto reasons whythe project resultsare both beingused and not beingused.

Participant bias couldhave a direct impactupon the results. Re-spondents might, forexample, wish to tellthe researcher whatthey want to hear andtherefore over-emphasise the applica-tion of FP projects.

It is difficult to get arealistic indication ofthe actual scale andpurpose for whichproject findings arebeing applied. It is notlikely to be possible toget an accurate indica-tion of the actualrange of actors whoare making use of theproject results, nor todetermine the actualway in which they arebeing, or have been,used.

Overall: Not in itspresent form. Ad-justments are neces-sary.

Minimal time andresource needs to beinvested by theevaluator but keystakeholders can beunwilling to dedicateenough time to theinterview to producemeaningful results.Experience showedthat the best insightswere achieved whenend-users were sentthe interview ques-tions in advance ofthe interview andgave their answersthought before theinterview. The timerequired to do so wasoccasionally prohibi-tive. Some interviewsalso took up to an

Yes, but onlyif the ques-tions askedare reducedand the sam-ple size islarge andrepresentativeenough tosupport mean-ingful conclu-sions.

Grant No. 213546

40/65

MetronomeMetronome

Relies upon end-usersbeing willing to takethe time to fully re-spond.

hour to conduct.Non-response cancompromise the va-lidity and generalisednature of the find-ings.

Matrix evaluations

A project assessment (test against project goals and objectives and analysis of dissemina-tion activities) took between one and two men days, depending on the availability of mate-rial. We feel this is an acceptable length of time for a project evaluation.

1. We have experienced from working with four persons looking independently at theavailable information that the application of the methodology can lead to quite differentresults. Scoring the different elements in categories like fully, partially, indirectly, etc.are of course subjective. This subjective element sets certain requirements towards theability of the persons performing the evaluation. People that are quite familiar with FP-research could have a rather biased view on certain projects; on the other hand, know-ing a bit about the context of the framework program can speed up the search andevaluation process. It could be useful to apply a more strict separation betweendata/information gathering and evaluation (performed by different individuals), althoughwe found a lot of extra info during the evaluation process, because of references in thereports to events or specific activities.

2. The methodology is very sensitive to the availability of project information. In general,we found that this availability is rather poor. Final reports can be obtained from theTransport Research Knowledge Centre, but other types of reports are scarcely available.Usually they can be found on project websites, but for many of the evaluated FP5-projects the sites are no longer ‘on line’. An improvement of the METRONOME method-ology could be to apply some ‘enforcement’ in the process of the information gatheringprocess, for instance by an enclosed letter from DG-TREN.

3. The dependence of the evaluation method on specific project information implies that aformal evaluation cannot be performed when information is lacking, while the fact thatinformation is lacking can in itself be considered a negative evaluation criteria. This phe-nomenon could be seen as a minor inconsistency in the methodology.

4. For recently finished projects the material is often not be publicly available when theCommission has not yet approved the final report. This means that it will be difficult toapply the METRONOME methodology to ongoing programs.

5. In web-searches for available information, the heavy use of acronyms is not always veryhelpful. Many other research projects (national and international) seem to exist with thesame acronyms as EU research projects.

Dissemination qualityThe most part of coordinator’s suggestions and recommendations dealt with better dissemi-nation of FP results. The answers showed that there is a need for a better dissemination ofFramework Programme results. The main obstacle is a lack of sufficient dissemination ofreached results not only to other target or to potential users, but within the organizations,which were partners of evaluated project.

Grant No. 213546

41/65

MetronomeMetronome

There is a need to determine the extent to which projects within the FPs have sought to es-tablish and build links with end-users and key stakeholders. The analysis should be basedon a representative sample of projects. The analysis could provide an indication of how pro-active the projects that comprise the FP were in seeking to ensure that their results had adirect and positive impact upon wider European Community and Public Policy with implica-tions for its ultimate contribution.

End User InterviewsA methodology to evaluate the impact of an FP and the projects within it on wider EuropeanCommunity and Public Policy should involve consultation with end users. There will be chal-lenges in achieving responses from a broadly representative sample of end-users, and in-deed in identifying what constitutes a representative sample for each FP and project, butthis is a necessary process to determine the extent to which dissemination activities and thewider European research network have led to the effective dissemination of project resultsand awareness of research activities.

Data availability and qualityWP5 work confirmed the former conclusions concerning insufficient analysis of informationneeds. Insufficient attention has been paid to understanding what data are required forevaluation and monitoring purposes. There are data and various statistics, but there are noinformation appropriate results for evaluation purposes.

WP5 initial conceptual idea and proposed approach were in principle correct. WP5 approachas a whole and all proposed methodological tools must arise from data availability and qual-ity.

The WP5 investigations have revealed that collection of all necessary data for comprehen-sive evaluation of FP5 projects in the field of transport is hardly practicable. The same con-clusion applies to FP6 projects.

Evaluation design and the strategy, evaluation tools depend on data availability and quality.Not only final reports and other deliverables, but also information on post project monitoringresults provided a basis for all evaluation activities.

WP5 had only data, which enable us to evaluate – to prepare project management appraisal/efficiency and effectiveness for the first part of project lifecycle – stages before implemen-tation and exploitation of project results. Data for research project results exploitation/useafter project completion are not available.

5.2 Results

Drawing sound conclusions and generalising based on WP5 sample of projects is not statis-tically representative. The empirical material was limited in the scope and its representa-tiveness is not known. The sample of projects was representative for single project method-ology testing, but it was small for generalizing at the programme level. With this caveat inmind the following could, however, be said about the FP5 and FP6 performance in the fieldof C&PP.

Based on the Evaluation Matrix results it is possible to conclude the following:

The set of research objectives that were the best met were the Strategic Project objectives.The majority (90%) of these for FP5 projects were considered to have been ‘fully met’. ForFP6 projects the percentage is lower (51%), but in both cases this set of objectives hasbeen met better than others have. The potential contribution and impact of FP5 and FP6

Grant No. 213546

42/65

MetronomeMetronome

projects to Work Programme objectives is almost identical. A large percentage of work pro-gramme objectives have been achieved. Surprising finding is that both the FP5 and FP6 pro-jects were considered to have contributed more ‘fully’ to ‘higher level’ Work Programme ob-jectives than to the ‘lower level’ Key Action objectives.

Contribution to Community and Public Policies in general is very similar between both FP5and FP6 projects, and although there can be seen to be some differentiation between thetwo FPs this could be owing to the different sample sizes as much as to differing contribu-tions to Community and Public Policies. The potential impact of projects in terms of the cus-tomer/end-user, societal and co-ordination indicators used was consistently positive acrossthe range of projects reviewed. Management and co-ordination impacts received the mostpositive responses in both FP5 and FP6.

Based on the Dissemination Quality evaluation it is possible to conclude the following:

The main dissemination channels for both FP5 and FP6 projects were such as seminars,conferences, workshops, web pages and different publications. The most typical audiencesincluded EC officials, researchers, civil servants, experts and transport operators.

Surprising finding is that in FP5, only 15% and in FP6 only 25% of the evaluated projectshad identified the end user(s) of project results in the course of the project and involvedthem in the project proceedings. The dissemination activities focused in both FPs mostly onpresenting the results on the final stages of the projects, rather than consultation opportuni-ties to improve project results.

Based on the Coordinator Survey it is possible to conclude the following:

The project co-ordinators considered impacts of FP5 projects in all four impact groups to behigher than of FP6 projects. The most positive answers regarding both FP5 and FP6 projectsconcerned the scientific competence of consortium, contribution to the development of deci-sion making tools and involvement of civil servants and policy makers in the projects. In ad-dition, publishing project results in scientific journals or books and financial issues wererated highly in FP5 and improved networking between researchers and public/private or-ganisations in FP6. The most negative results were received to questions/statements in bothFPs regarding transferring the project results transferred into policy recommendations andregulations and project raising new unsolved research questions.

The following list presents some additional comments by the project coordinators.

- There has to be more freedom to involve and integrate non-profit organisations as e.g.promotion centres into the project. Full partner or subcontracting roles are not ade-quate.

- Interrelation with other research projects could be better.

- Most important is the issue of dissemination after the project completion. At the end ofthe project there are no resources any more to do any updating or inquiries or helpdeskor whatever. Therefore it needs to be assured that an effort is made that project resultsare taken up by some sort of international platform or network, which takes care aboutthis after the end of the project.

- The responsible person from the Commission (usually the project officer, but sometimesalso reviewers) should put a focus on the above, but should also make a clear decision

Grant No. 213546

43/65

MetronomeMetronome

that if the results are not taken up by such networks, no resources should be wasted onany continuation.

- It should be checked thoroughly whether new research projects really take into accountthe results of former projects.

- In order that projects are not forced to use results that are not practical, the results of aproject should be independently evaluated in some way, and only projects with "good"or "excellent" results should be flagged as such that should be used by new projects.

- Those involved in the execution of the project may exaggerate the positive impacts tojustify their current work and promote the funding of similar future projects.

- It may take several years (or even longer) before impacts of a policy-oriented projectcan be seen.

- Policy framework EC not clear at the time of the project.

- Support projects fill the role as disseminators of results, but we have yet to see the re-sults of that.

Based on Lead User Survey it is possible to conclude the following:

The most positive answers regarding both FP5 and FP6 projects concerned the followingissues: high level of expertise within the project participants; improved networking betweenresearchers and public/private organisations (especially in FP6) and strengthened networksbetween international parties; taking the needs and views of end users were into considera-tion in the projects; usage of project results by the public sector or other societal actor; dis-semination of project results (only FP6, not FP5) and development of decision making tools.

Contribution to policy goals or policy development received slightly positive answers regard-ing both FPs. The most negative results were received regarding transfer of project resultsinto policy recommendations and regulations. Also cost effectiveness of FP5 projects re-ceived negative response, but FP6 projects positive.

The results of coordinator and lead user surveys present astoundingly similar results, whichallows us to claim that the strengths and weaknesses identified for the FP5 and FP6 projectsabove can be considered as relevant and tenable.

Grant No. 213546

44/65

MetronomeMetronome

6 Recommendations for facilitation of future FP evalu-ation

We propose the following general recommendations to be considered in the context of fu-ture FP evaluations in the field of contributing to transport related Community and PublicPolicies.

Increase awareness throughout the European research community of the value of feed-back and evaluation. A central EU facility (i.e. web portal) or contact (specific memberof personnel) could be promoted and made visible for supporting.

In the FP development phase, greater emphasis should be paid to the level of consis-tency between the different levels of the objectives set for the projects (namely project,key action, work programme and European policy levels)

Comparative and inter-disciplinary evaluation framework for CPP methodology is neces-sary. Comparative is important because it places recommendations in context. Policyanalysis is not possible (or useful) without interdisciplinarity.

It should be remembered that there is a difference between research – e.g. supported/coordination actions, networks of excellence/ and research /STREP, IP/ projects.

The evaluation of final reports (either all commissioned under an FP or a selection ofthem) could be built into the official EU project closure process. This could, for exam-ple, be made a component of a project audit, or could be conducted as part of the peerreview process.

In questionnaire surveys, the list of questions that need to be posed to end-users to ob-tain valuable and meaningful insights into the impact of specific projects and wider FPsshould be as small as possible. This could increase the response rate from end-users. Pi-loting the interviews made it apparent that a similar level of detail could be obtainedfrom a more focused and concise list of questions.

Establish and maintain a comprehensive database of FP5 and 6 projects and their deliv-erables. This would support dissemination and increase awareness of FP projects andtheir results. The TRKC currently fulfils a useful role, particularly through the concise andinformative descriptions that it provides for each project, but it is incomplete – it doesnot contain all transport projects commissioned under FP5 and 6, and it also does notcontain the deliverables for all of the projects.

Next to the research, it is necessary to pay greater emphasis to the definition of targetusers/specified focused end user groups and future exploitation areas of research re-sults.

Many things needs to be improved in knowledge management and sharing. The transferof knowledge coming from the project findings to other researchers and projects is es-sential for long-term exploitation of results.

Recommendations concerning improvement of data availability and quality:

Well-written reports are an important means to get results widespread. There is a needto define mandatory elements of final reports.

Grant No. 213546

45/65

MetronomeMetronome

A detailed description of dissemination activities and list of target users could be a partof each final report.

The Commission could organise that databases and models are continuously updatedand facilitated; actions are now taken through JRC-IPTS, but the experts who havemade the data and tools could be more directly involved to improve quality. If tools anddata are made for the Commission, they could start or before the project think aboutthe phase after the project.

The evaluation of the FPs needs to be supported by appropriate organisational arrange-ments, which should follow from the Commission’s strategic orientation for the evalua-tion of the FPs. The organisational structure of this unit should ensure that the evalua-tion system was appropriately coordinated and adequately resourced.

Recommendations concerning improvement of dissemination:

A matrix similar to the ‘dissemination review’ matrix produced in this project could beintegrated into the official EU dissemination report template. It would provide a concisesummary of dissemination activities conducted by the project so would be of direct rele-vance to the content of the report whilst also supporting evaluation procedures.

All projects should be required to have a project website to support dissemination andact as a long-term and consistent point of reference for information about the project.

A METRONOME recommendation could be to demand 5 -10 year period of project web-site availability; this would even be useful without updating info. An alternative could bea more centralized administration of ALL project info (newsletters, reports, lists ofevents, participants in events, etc.).

Project websites should include web tools: counter of visitors, guestbook or brief ques-tionnaires to obtain information necessary for evaluation. All project websites shouldhave a very short voluntary form on the front page asking visitors to enter:

1. the organisation that they are a member of;2. the purpose for visiting the site;3. where they heard about the project from; and4. whether they would be like to be contacted to be told more about the project.

This would support the identification of end-users and provide useful feedback about thelikely contribution of projects to Community and Public Policies. This would also providevaluable information about how often the project is being referred to and help to deter-mine the level of awareness about the existence of the project website and the effec-tiveness of websites as a tool for dissemination.

Organization of scientifically robust and high-level workshops (ideal size 30-50) afterproject finalisation could enhance project result dissemination. Such workshops shouldbe well organized and involve serious paper-givers. Participants should be a mix of re-search, policy and industry.

Recommendations concerning improvement of monitoring:

Quality monitoring system should be established. Make sure that a list of target usersand exploitation plan are available at the beginning of the project in which the variousrights and exploitation opportunities are communicated.

Grant No. 213546

46/65

MetronomeMetronome

Annex 1Literature Review

EU documents:

COMMISSION OF THE EUROPEAN COMMUNITIES COM (2008) 533 final Ex-post evaluation of the‘Information Society Technologies (IST)’ Thematic Priority in the Sixth Framework Programme forResearch, Technological Development and Demonstration (RTD)

IMPACT ASSESSMENT GUIDELINESEuropean Commission SEC (2005) 79115th June 2005

IMPACT ASSESSMENT GUIDELINESEuropean Commission SEC (2009) 9215th January 2009

EX ANTE EVALUATIONA Practical Guide for Preparing Proposals for Expenditure ProgrammesEuropean Commission DG BudgetDecember 10th 2001

EX-POST EVALUATION OF EXTRA (1998-2001) AND MID-TERM EVALUATION OF EXTRAWEB (2002-2006)European Commission DG Energy and TransportFinal Report – 4 May 2006

THIRD ANNUAL THEMATIC RESEARCH SUMMARY – DECISION-SUPPORT TOOLSEuropean Commission DG Energy and Transport

NOTICES FROM EUROPEAN UNION INSTITUTIONS AND BODIESCOURT OF AUDITORS SPECIAL REPORT No 9/2007Evaluating the EU Research and Technological Development (RTD) framework programmes —could the Commission’s approach be improved? together with the Commission’s replies

OVERVIEW OF EVALUATION GUIDES IN THE COMMISSIONEuropean Commission DG BudgetEvaluation Unit, October 2004

Grant No. 213546

47/65

MetronomeMetronome

Research projects:

RTD EVALUATION TOOLBOX - ASSESSING THE SOCIO-ECONOMIC IMPACT OF RTD-POLICIESStrata Project HPV 1 CT 1999 - 00005, August 2002Institute for prospective technological studies

VA DISABILITY COMPENSATION PROGRAMLiterature Review, Dec. 2004Economic Systems Inc.VA Office of Policy, Planning, and Preparedness

MEASURING THE IMPACT OF RESEARCHMichelle Duryea, Mark Hochman and Andrew Parfitt discussCOMMUNICATION TO THE COMMISSION FROM MS GRYBAUSKAITÉ INAGREEMENT WITH THE PRESIDENTResponding to Strategic Needs: Reinforcing the use of evaluation

MAESTRO DELIVERABLE 3Action Plan for RTD Programme Evaluation and the Results of the Validation ofthe MAESTRO MethodologyMAESTRO Consortium, TTR et al (1999)

USABILITY TESTING OF THE MAESTRO GUIDELINES, MAESTRO DELIVERABLE 4Results of the Validation Process of the MAESTRO Methodology and of the MAESTRO Consortium,TTR et al (1999)

MAESTRO DELIVERABLE 6The MAESTRO GuidelinesMAESTRO Consortium. TTR et al (2000)

DANISH EVALUATION OF THE IMPACT ASSESSMENTS CARRIED OUT BY THE COMMISSION IN2006DANISH COMMERCE AND COMPANIES AGENCYKampmannsgade 1; DK-1780 København V; Denmark

EU EVALUATION PRACTICE AND TOOLKIT APPLICABILITY FOR JAPANStudy on behalf of Mitsubishi Research Institute, Japan

EVALUATION AND IMPACT ASSESSMENT OF EUROPEAN FP FOR R&D:Points of views from businessDr. Leif Kjaergaard

GOOD PRACTICE STANDARDS FOR EVALUATION OF MDBSUPPORTED PUBLIC SECTOR OPERATIONSMultilateral Development Bank (MDB) Evaluation Cooperation Group (ECG)Working Group on Evaluation Criteria and Ratings for Public Sector Evaluation (WGEC)

Grant No. 213546

48/65

MetronomeMetronome

Workshops:

Presentations from European Forum on Research and Development Impact AssessmentPrague Congress Centre, 24th – 25th February 2009 www.eufordia2009.eu.List of presentations:

LESSONS FROM AND FOR RESEARCH POLICY IMPACT EVALUATION Stefan Kuhlmann, University of Twente, NL

ADVANCING THE SCIENCE OF R&D PROGRAM EVALUATION: THE US APPROACH Julia Lane, National Science Foundation, USA

FP6 EX-POST EVALUATION: METHODOLOGY CHOSEN BY THE EC Peter Fisch, DG RTD, European Commission

FP6 EX-POST EVALUATION: REPORT ON FINDINGS OF THE EXPERT EVALUATIONGROUP

Ernst Th. Rietschel, Leibniz Gemeinschaft, DEDG INFO ’S ICT RESEARCH INTERVENTION AND THE EX-POST EVALUATION OF ICT RTDIN FP6 AND IT’S FOLLOW UP

Frank Cunningham, DG INFSO, European CommissionSWEDISH FP6 IMPACT ASSESSMENT STUDY (WITH PARTICULAR FOCUS ON INDUSTRYCOMPETITIVENESS)

Lennart Norgren, VINNOVA, SEAN INDICATOR-BASED REVIEW OF THE OUTCOME AND IMPACT OF THE DIRECT AC-TIONS IN THE FP (JRC)

Pieter Van Nes, JRC, European CommissionHOW META-EVALUATION HELPS US UNDERSTAND IN THE FPs

Erik Arnold, TechnopolisQUANTITATIVE INDICATORS FOR EVALUATION AND IMPACT ASSESSMENT OF RE-SEARCH PROGRAMMES

Benedetto Lepori, University of Lugano, CHUsing INTERVENTION LOGIC FOR the decision of PUBLIC R&D policies and programmes -what could be done in the EU context?

Martin Weber, European Court of Auditors

Evaluation and Monitoring of European Research Framework Programmes, CZELO/CLORA/STOAWorkshop, 3 December 2008, www.czelo.czList of presentations:

FP 6 EX-POST EVALUATION FOR IST ACTIVITIES (Peter Fisch) EXAMPLES OF EXPERTISE AND EXPERIENCE IN RESEARCH AND PUBLIC POLICYEVALUATION (Luc Chalsege)FP6 EX-POST EVALUATION FOR IST ACTIVITIES (Nicoletta Stame)NETWORKS OF EXCELLENCE: THE REPORT OF THE EXPERT GROUP ON THE FUTURE OFNOES (Manfred Horvat)THE EX-POST FP6 EVALUATION OF THE DIRECT RESEARCH ACTIONS OF THE JRC(Pieter Van Nes)EVALUATION AND IMPACT ASSESSMENT OF EUROPEAN FP FOR R&D: POINTS OF VIEWSFORM BUSINESS (Leif Kjaergaard)WHAT DO RESEARCH ORGANIZATIONS LOOK FOR IN AND CONTRIBUTE TO FP 7 (AchimBachem)IMPACTS OF FP3-6 IN SWEDEN (Goran Marklund)COURT OF AUDITORS SPECIAL REPORT NO. 9/2007: EVALUATING THE EU RESEARCHAND TECHNOLOGICAL DEVELOPMENT (RTD) FRAMEWORK PROGRAMMES - COULD THECOMMISSION'S APPROACH BE IMPROVED (Martin Weber)EX-POST EVALUATION OF IST RESEARCH IN FP6 (Frank Cunningham)

http://www.eufordia2009.eu.

http://www.czelo.cz

Grant No. 213546

49/65

MetronomeMetronome

Annex 2METRONOME questionnaires

WP5 Questionnaire for co-ordinators of FP5 and FP6 funded projectsPlease specify your answer regarding questions marked * at the bottom

Mark X to the best suitable alternative agreestrongly

agree neither/nor

disagree disagreestrongly

N.A.

A. Scientific impactsA1 The goals (research objectives) set to the

project were all met

A2 The research goals required specificelaboration at the start of the project

A3 There was a good fit between the re-search framework and data available

A4 The scientific competence of the consor-tium was sufficient for the task

A5 Project results have been published inscientific journals and/or books *

A6The project raised new unsolved researchquestions in the field of Community andPublic Policy *

B. Societal and end user impactsB1 The project has contributed to priority set-

ting, e.g. future research or policy goals *

B2 The project has contributed to new trans-port policy development *

B3The project results have been transferredinto policy recommendations and/or regu-lations *

B4The project has contributed to develop-ment of decision making tools (guidance,models) *

B5The project has contributed to decisionmaking processes of public or private or-ganisations *

B6 The project has led to new product or ser-vice development in the transport sector *

B7The project results have been used bypublic sector, industry or other societalactors *

C. Management, coordination and disseminationC1 Needs and views of end users were taken

into consideration *

C2 Civil servants and/or policy makers wereinvolved in the project *

C3 Transport operators or service sectorwere involved in the project *

C4 Transport industry sector was involved inthe project *

C5 The project webpage was user-friendlyand updated regularly

C6 The project webpage has frequently beenvisited by non-project-members

C7The project results have been adequatelydisseminated to end users: civil servantsand policy makers, public, industry etc. *

C8

The project (consortium) has improvednetworking between researchers and pub-lic/private organisations (e.g. initiated newresearch projects)

Grant No. 213546

50/65

MetronomeMetronome

C9 The project built or strengthened networkswith global/EU/national research partners

D. FinanceD1 The research budget (project size) was

sufficient

D2 The financial instrument was adequate forthe project

Please, write your specific answers hereA5 Number of articles/publications?

about 10A6 Which research questions?

B1 On what?

B2 Which policy?Decision-making in the transport sector und in-volvement and participation of different stake-holders.

B3 Which recommendations?

B4 Which tools?Two specific handbooks about decision-makingprocesses and suitable tools to involve citizens,stakeholders etc.

B5 Which organisations and on what?Planning authorities, grass-root organisations ondecision making for transport projects, public in-volvement and participation

B6 Which products or services?

B7 Which actors? Planning authorities, public transport organisa-tions, grass-root organisations, politicians andother stakeholders

C1 In which way?

C2 In which way?Experience from several case-studies

C3 In which way?in demonstration projects

C4 In which way?

C7 In which way?The results were widely disseminated in Europe:hardcopy-handbooks, CD, download from CIVI-TAS-website etc.

Could you in the light of your experiences come up with suggestions for better promotionand exploitation of project results

1

2

Grant No. 213546

51/65

MetronomeMetronome

3

4

Notes:1

2

3

Grant No. 213546

52/65

MetronomeMetronome

Questions for lead users of EC FP5 and FP6 funded transport projects contributing to C&PPolicies

FP5 and FP6 project impacts in general

B1 FP5 or FP6 projects have contributed to priority setting, e.g. policygoals

B2 FP5 or FP6 projects have contributed to new transport policy devel-opment

B3 FP5 or FP6 project results have been transferred into policy recom-mendations and/or regulations

B4 FP5 or FP6 projects have contributed to development of decisionmaking tools (guidance, models)

B5 FP5 or FP6 projects have contributed to decision making processesof public or private organisations

B6 FP5 or FP6 projects have led to new product or service developmentin the transport sector

B7 FP5 or FP6 project results have been used by public sector, industryor other societal actors

Specific FP5 or FP6 project impacts YES NO Don'tknow

C1 I have been involved in the FP5 or FP6 projects?

If YES,

C2 Needs and views of end users were taken into consideration in theproject

C3 Also transport operators or service sector were involved in the pro-ject

C4 Also transport industry sector was involved in the project

C5 The project results were adequately disseminated to end users: civilservants and policy makers, public, industry etc.

C6 The project improved networking between researchers and pub-lic/private organisations

C7 The project built or strengthened networks with global/EU/nationalparties

C8 The project was cost effective in terms of money or resources spent

C9 The project participants had high level of expertise of the field ofthe project

C10 What did work well in the project, and what didn't?

C11 The main benefit of the FP5 and FP6 projects to me/my organisa-tion?

Please, write your specific answers here

Grant No. 213546

53/65

MetronomeMetronome

C1 Which projects?

In which way?

C2 Which projects?

In which way?

C3 Which projects?

In which way?

C4 Which projects?

In which way?

C5 Which projects?

In which way?

C6 Which projects?

In which way?

C7 Which projects?

In which way?

C8 Which projects?

In which way?

C9 Which projects?

In which way?

C10 Which projects?

What worked?

What didn’t?

C11 Which projects particularly?

The main benefits:

Grant No. 213546

54/65

MetronomeMetronome

Annex 3Summarised analysis of the Evaluation matrix results

The evaluation matrix

Table below contains the full list of objectives and impact indicators that FP5 and FP6projects were reviewed against to estimate their potential contribution to Community andPublic Policies. The only objectives not detailed within the Table are the Strategic Pro-ject objectives, which have been withheld to preserve the anonymity of the projects re-viewed.

Table A3.1. Objectives and impact indicators used for the review of FP5 andFP6 projects.

Objectives/impact indicatorsNumber ofprojectsreviewed

FP5 GROWTH• To produce, disseminate and use the knowledge and technologies needed todesign and develop processes and produce high quality, environment- andconsumer-friendly products which will be competitive on tomorrow's market.• To support the development and implementation of Community policies thatenable competitive and sustainable development.• Development of related services (including transport) which are economic,safe and protective of the environment and quality of life.• The optimal use of specific research infrastructures.

8

FP6 POLICIES• To underpin the formulation and implementation of Community policies byproviding scientific contributions to policies that are targeted precisely onneeds.• To ensure timely and effective scientific inputs, covering a wide field of poli-cies and with the prospect of improved information, exploitation and uptake ofresults, at national and EU level.• To ensure a coherent research base that reflects the increasing integration ofCommunity policies and of the science that underpins them.• Systematic improvements in the relationship between research and policy atall levels in the EU.• To ensure development of the European Research Area by encouraging asingle playing field in relation to policy-related research.

1

Wor

k pr

ogra

mm

e ob

ject

ives

FP6 SUSTDEV• To implement a sustainable development model in the short and in the longterm integrating its social, economic and environmental dimensions.• To make a significant contribution to international efforts to mitigate or evento reverse current adverse trends.• To preserve the equilibrium of ecosystems.

3

Grant No. 213546

55/65

MetronomeMetronome

FP5 GROWTH• Development of strategies and tools for managing the impact of economic,social, political, demographic and technological developments on mobility de-mand and transport policies.• To deliver the building blocks for a European strategic decision support andinformation system in the field of transport for policy-makers, authorities, in-dustry and operators. The three major building blocks are quantitative tools,knowledge of today’s and tomorrow’s driving forces in transport and effectivepolicies.• Based on the above to provide the keys to further refine and operationalisethe concept of sustainable mobility to further develop integrated transport sys-tems in the specific European context.• To provide the foundation for harmonised pan-European safety regulations.

8

Key

actio

n ob

ject

ives

FP6 SUSTDEV• New technologies and concepts for all surface transport modes (road, railand waterborne).• Transport specific advanced design and production techniques.• Rebalancing and integrating different transport modes.• Increasing road, rail and waterborne safety and avoiding traffic congestion.

3

Prog

ram

me

subd

ivis

ion

obje

ctiv

es

FP6 POLICIES• Underpinning European integration, sustainable development, competitive-ness and trade policies (including improved means to assess economic devel-opment and cohesion).• The development of tools, indicators and operational parameters for assess-ing sustainable transport and energy systems performance (economic, envi-ronmental and social).• Global security analysis and validation systems for transport and researchrelating to accident risks and safety in mobility systems.• Forecasting and developing innovative policies for sustainability in the me-dium and long term.• Information Society issues (such as management and protection of digitalassets, and inclusive access to the information society).• The protection of cultural heritage and associated conservation strategies.• Improved quality, accessibility and dissemination of European statistics.

1

Stra

tegi

c pr

o-je

ct o

bjec

tives Not stated here to preserve anonymity.

12

Improvements in the relationship between research and policy at all levels inthe EU, timely and effective scientific inputs for wide field of policies. 12

Improvements in the efficiency and transparency of decision making. 12

Improved policy coherence. 12

Expanded and improved investment in human capital. 12

Euro

pean

Com

mun

ityan

d Pu

blic

Pol

icie

s Po

licy

Obj

ectiv

es

Better implementation of existing EU environment policies and legislation atthe local level. 12

Grant No. 213546

56/65

MetronomeMetronome

Development of the European Research Area and scientific support for EUenlargement. 12

Greater coherence between macroeconomic and structural policies. 12

Good governance. 12

Co-operation between ministries. 12

Quantitative tools for decision making, reflecting socio-economic objectives. 12Has the project resulted in the definition of new regulations or regulatory stan-dards? 12

Has the project developed in new policy measures? 12

Has the project developed new products/initiatives for the transport sector? 12

Have the findings been translated into policy recommendations? 12

12

12

Cust

omer

/end

-use

r im

pact

indi

-ca

tors

Have the project results been disseminated to:Policy makersCivil servantsIndustry

12

Have events been organised for the dissemination of project results? 12

Have the project results been translated into future research needs? 12

Soci

etal

impa

ct in

di-

cato

rs

Has society participated in the research? 12

Has the project strengthened or created new research networks? 12

Have there been joint publications of project results? 12

Have civil servants and/or policy makers participated in the project? 12

Co-o

rdin

atio

n in

di-

cato

rs

Has the project improved networking between researchers and/or pub-lic/private organisations? 12

Realisation of FP5 and FP6 objectives and potential impactsObjectives

The set of objectives that were the best met were the Strategic Project objectives. Thevast majority of these were considered to have been ‘fully met,’ with the others (two outof 61) ‘partially met’ (see Table below). This is not a surprising finding as the StrategicProject objectives are those with the most direct relevance to the project. The fulfilmentof higher level objectives, such as Work Programme, Key Action and Programme Subdivi-sion objectives, instead indicates whether the project is likely to have broader impactsupon community and public policies in the EU, which is why it is to be expected thatthese will not be as fully contributed to as those that the project was commissioned un-der. It should also be noted that it is not possible to evaluate projects based on the ex-tent to which they have contributed to higher level objectives. This is largely becausethe purpose and scope of projects commissioned varies considerably and that the magni-

Grant No. 213546

57/65

MetronomeMetronome

tude and nature of the impacts and contributions will also necessarily vary between pro-jects. A review against contribution to higher level objectives does, however, provide anindication of breadth of contribution.

The low sample size (see Table below) of projects reviewed limits the reliability of find-ings, but from the projects reviewed it is surprising that both the FP5 and FP6 projectswere considered to have contributed more ‘fully’ to Work Programme objectives than tothe Key Action objectives (and in the case of FP6 projects also Programme Subdivisionobjectives). Projects conducted in both FP5 and FP6 were also considered to have notcontributed at all to a greater number of Key Action and Subdivision objectives thanWork Programme objectives. This is surprising because the lower level objectives (KeyAction and Programme Subdivision) could be considered to be more directly applicable tothe projects commissioned. It is, however, a finding also shared with the projects thatwere reviewed by WP4.

Table A3.2. The extent to which FP5 and FP6 projects reviewed contributed toobjectives and impacts relating to community and public policies.

Based upon the reviews conducted the potential contribution and impact of FP5 and FP6projects to both Work Programme and Strategic Project objectives is almost identical(see the results of the reviews in Tables 3 and 4 below). Contribution to Key Action orSubdivision objectives is also very similar, with the only distinction being that a greaterproportion of FP6 projects were considered to have not contributed to these objectiveswhilst more FP5 projects were thought to have ‘fully’ contributed to this set of objectives.In terms of actual numbers, however, there is very little variation which means thatthere can be no meaningful distinction made between performance of FP5 and FP6 pro-jects in this respect. The low sample size also means that it cannot be ascertainedwhether the ‘Programme Subdivision’ objectives that are set in FP6 but not FP5 have hadany impact upon this variation in results.

Contribution to Community and Public Policies is also very similar between both FP5 andFP6 projects, and although there can be seen to be some differentiation between thetwo FPs this could be owing to the different sample sizes as much as to differing contri-butions to Community and Public Policies (see Tables 3 and 4 below).

Extent to which the project met/contributedto objectives

Fully/yes Partially IndirectlyNot atall/no

Samplesize

Work Programme 47% 29% 21% 3% 38Key Action 14% 46% 22% 19% 37Programme Subdivision 14% 14% 14% 57% 7Strategic Project 97% 3% - - 61

Obj

ectiv

es

Community and Public Policies 13% 36% 29% 22% 100Customer/end-user 66% 1% 1% 31% 67

Societal 97% - - 3% 29

Impa

ct in

-di

cato

rs

Co-ordination 100% - - - 35

Grant No. 213546

58/65

MetronomeMetronome

Table A3.3. The extent to which FP5 projects reviewed contributed to objec-tives and impacts relating to community and public policies.

Extent to which the project met/contributed toobjectives


Work Programme 12 7 5 -Key Action 5 10 5 4Strategic Project 29 2 - -

Obj

ectiv

es

Community and Public Policies 7 19 21 13Customer/end-user 28 - 1 13

Societal impact 18 - - -

Impa

ct in

-di

cato

rs

Co-ordination 22 - - -

Table A3.4. The extent to which FP6 projects reviewed contributed to objec-tives and impacts relating to community and public policies.

Extent to which the project met/contributed toobjectives


Work Programme 6 4 3 1Key Action - 7 3 3Programme Subdivision 1 1 1 4Strategic Project 30 - - -

Obj

ectiv

es

Community and Public Policies 6 17 8 9Customer/end-user 17 - - 8

Societal 10 - - 1

Impa

ct in

di-

cato

rs

Co-ordination 13 - - -

On an individual project level the results indicate fairly clearly that there is not a closerelationship between Strategic Project objectives and wider Community and Public Policyrelated objectives, either on an EU or a WP level. This is again not a surprising findingas the degree of comparison will almost inevitably vary based upon the purpose and na-ture of each project. The contribution of projects that have met all of their Strategic Pro-ject objectives to other relevant sets of objectives was too variable to support any mean-ingful conclusions in this respect. Only five of the 10 projects reviewed met their Strate-gic Project objectives as well as the relevant Key Action (or equivalent) objective thatthey were commissioned under as well as one or more of the relevant Work Programmeobjectives. A higher number of projects in this category could have indicated greaterconsistency between the different levels of objectives set.

Grant No. 213546

59/65

MetronomeMetronome

Impact indicators

The potential impact of projects in terms of the customer/end-user, societal and co-ordination indicators used was consistently positive across the range of projects re-viewed. As shown in Tables 3 and 4 above there is evidence that either as part of aconscious effort or as part of standard FP processes the positive impacts were realised inalmost all projects. In the 14 instances for FP5 projects and the eight for FP6 where po-tential customer/end-user impacts were not thought to have been realised this was notthe result of any considerations having been overlooked by the project consortiums,rather that certain indicators, notably whether the projects resulted in the definition ofnew regulations, standards or policy measures, were not applicable to the programmesof work being undertaken.

Grant No. 213546

60/65

MetronomeMetronome

Annex 4Summarised analysis of the Dissemination quality matrix results

Table A4.1. Dissemination aspects of FP5 projects.

DISSEMINATION ASPECTSFP5

Yes No

List of dissemination activities 11 (85%) 2 (15%)

Were potential end-users and beneficiariesof the results directly involved or consultedin the different stages of the project?

12 (92%) 1 (8%)

Did dissemination occur throughout theproject?

12 (92%) 1 (8%)

Was there a piloting of results with poten-tial end-users before finalisation?

11 (85%) 2 (15%)

Were plans made to update the resultsbased upon the outcome of the dissemina-tion activities?

3 (23%) 10 (77%)

Ove

rvie

w

Was there a specific dissemination work-package?

6 (46%) 7 (54%)

Comments to table A4.1.


Main dissemination channels and tools were: conferences, workshops, seminars, projectwebsites, user group surveys, user/beneficiary panels, training courses.Only 15% of evaluated projects indicated that a target or control user/beneficiary groupfor analysing the needs of future users, finding the best way to address them, and en-suring the effective feedback was defined.

Potential end-users and beneficiaries of the results direct involvement in a project:

It is possible to state that for the most part a direct involvement of potential users in re-search projects was not planned and performed. A website was the only channel for indi-rect involvement of persons concerned and potential users of project results in FP5 pro-jects.

Grant No. 213546

61/65

MetronomeMetronome

Analyses of dissemination activities show that in the majority of cases dissemination ac-tivities fallen on the final stage of project lifecycle.

Piloting of results with potential end-users before finalisation:

WP5 analyses and peer reviews show that about 85% of projects indicated efforts fo-cused on piloting of results before finalisation of results.

Updating the results based upon the outcome of the dissemination activities:

Based on findings it may be concluded that for the majority of evaluated projects dis-semination activities were used only for presentation of project results, rather than forconsultations and as an opportunity to test and improve project results.WP5 investigations show that the adjustment/modification of results indicated onlythose projects, which from the beginning have defined a user group/user panel and con-sulted preliminary results with this audience. Taking into account that a lot of projectswere planned their dissemination activities after project completion, there was no oppor-tunity/and time/to modify results with due regard to dissemination results.

Specific dissemination workpackage establishment:

46% of projects have established a special WP for dissemination of project results.

DISSEMINATION EVENTS

Type of event(s): the same as in a list of dissemination activities: conferences, work-shops, seminars, and so on.

Disseminated results: all results and all aspects of project.

Level, type and size of audience:European, national, local – the most part of events were at European level rather thannational or local levels.Type: EC officials, scientists, researchers, policy makers, decision makers, experts, trans-port operators, logistics operators.Size: from 9 to 70 participantsLocation of conferences: most of the projects, which indicated the events location, hadat least one of the main conferences in Brussels. Venue in Brussels attracts the EU pol-icy- and/or decision-makers interest and significantly contributes to a broader successfuldissemination at the European level.

Timing of event in relation to project lifecycle:31% of projects indicated that dissemination has started at the final stage of project fora presentation of final project results.

PUBLICATIONS

Disseminated aspects: all aspects

Grant No. 213546

62/65

MetronomeMetronome

Nature of publications: articles, handbooks, databases, recommendations, best practiceshandbooks, toolkits, position papers, press releases, case studies.

Intended readership/audience: EC officials, scientists, policy makers, experts, policy advi-sors, consultants to the government, transport operators, government officials.

Timing of publication in relation to project lifecycle:Based on analyses it may be deduced that key publications were issued after projectcompletion.

About 80% of projects indicated publications after the project completion.In the most cases the leaflets, progress reports and newsletters presented via a projectwebsite were used during a project lifecycle.

OTHER DISSEMINATION METHODS

Non-formal dissemination (i.e. consultations/discussions with stakeholders)

There is so little information and ambiguity that it is impossible to draw reliable conclu-sions.Lack of information gives no way of drawing conclusions. Local government surveys wereindicated for one project.

Table A4.2. Dissemination aspects of FP6 projects.

DISSEMINATION ASPECTSFP6

Yes No

List of dissemination activities 8 (100%) -

Were potential end-users and beneficiaries of the results directly involved orconsulted in the different stages of the project?

5 (62,5%) 3 (37,5%)

Did dissemination occur throughout the project? 6 (75%) 2 (25%)

Was there a piloting of results with potential end-users before finalisation? 6 (75%) 2 (25%)

Ove

rvie

w

Were plans made to update the results based upon the outcome of the dissemi-nation activities?

5 (62,5%) 3 (37,5%)

Grant No. 213546

63/65

MetronomeMetronome

Was there a specific dissemination workpackage 4 (50%) 4 (50%)

In the comparison of FP5 and FP6 results, consideration must be given to the differenceof FPs sample sizes.

Comments to Table A4.2.


Main dissemination channels and tools were: conferences, workshops, seminars, surveys,websites, potential user group surveys, user panels/user forums, special teams for dis-semination and exploitation, handbooks, telephone interviews.Only 25% of evaluated projects indicated a definition of target user group for analysingthe needs of future users, finding the best way to address them, and ensuring the effec-tive feedback.

Potential end-users and beneficiaries of the results direct involvement in aproject:

It is possible to state that the majority of projects used a website for presentation of pro-ject results.Analyses of final reports show that project consortium’s efforts were concentrated essen-tially on the achievement of project objectives, rather than on consultations with users.Considering findings of peer reviews it is possible to conclude that mainly a project web-site is the only channel for dissemination of preliminary results. It is important to point out that there is no information on website tools used.

Piloting of results with potential end-users before finalisation:

A large majority of findings show that 75% of projects presented consortium efforts fo-cused on piloting of results before finalisation of results. Dissemination manager work isa commendable example of piloting.

Updating the results based upon the outcome of the dissemination activities:

On close examination of results it is possible to conclude that the adjust-ments/modification of research results indicated only those projects, which from the be-ginning have defined a target user group/user panel and consulted preliminary resultswith this audience. A lot of projects were planned their dissemination activities for finalstages of project. Because of this, there was no opportunity/and time/ to take into ac-count comments and to update results.

Specific dissemination workpackage establishment:

50% of evaluated projects have established a special WP for dissemination.

Grant No. 213546

64/65

MetronomeMetronome

DISSEMINATION EVENTS

Type of event(s): identical with those specified in a list of dissemination activities: con-ferences, workshops, seminars, final conferences and so on.

Disseminated results: all aspects of project.

Level, type and size of audience:European, national, local – most part of events were at European level, rather than atnational or local levels.Type: scientists, researchers, policymakers, experts, operators, transport plannersSize: from 10 to 100 participantsLocation of conferences: most of the projects, which indicated the events location, hadat least one of the main conferences in Brussels. Venue in Brussels attracts the EU pol-icy- and/or decision-makers interest and significantly contributes to a broader successfuldissemination at the European level.

Timing of event in relation to project lifecycle:75% of project final reports indicated that dissemination has started at the end or afterreaching final project results.

PUBLICATIONS

Disseminated aspects: all aspects, depending on the project type and objectives.

Nature of publications: articles, handbooks, databases, recommendations, best practiceshandbooks, toolkits, position papers, press releases, leaflets, case studies.

Intended readership/audience: EC officials, national government officials, scientists, pol-icy makers, decision makers, experts, policy advisors, consultants to the government,transport operators, logistics operators.

Timing of publication in relation to project lifecycle:31% of projects indicated that a substantial part of publications were issued at the finalstage of project for a presentation of final project results.

OTHER DISSEMINATION METHODS

Non-formal dissemination (i.e. consultations/discussions with stakeholders)For the most part of analyzed projects it was found that information on this type of dis-semination is not available. The only interesting example of non – formal disseminationwas arrangement of regular telephone conferences.Lack of information gives no way of drawing general conclusions.

Analyses of dissemination activities agree with conclusions from the survey for coordina-tors that there is a need for dissemination improvement.

Grant No. 213546

65/65

MetronomeMetronome

Annex 5Analyses of lead user’s interviews (Power Point presentation)

Metronome - trimis.ec.europa.eu · Distribution list European Commission: [email protected]...

Documents

Transcript of Metronome - trimis.ec.europa.eu · Distribution list European Commission: [email protected]...