Draft Evaluation Methodology

Futurreg

4th Steering Committee

Logroño, 28th November 2006

Evaluation:

“The evaluation of public intervention consist of judging its value in relation to explicit criteria, and on the basis of information that has been specially gathered and analysed.”

Intrinsic features:

• Description

• Regulatory dimension

• Instrumental dimension

Others features:

• Scope of public policy evaluations. Controlling public expenditure and improving administration.

• Evaluation should be more than just an instrument of control to measure complex features.

• Key actors involved in the process.

¿Why evaluate?

• Verify programme effectiveness and efficiency

• Learn

• Aid reflection

• To see what is happening?

• Aid transparency

+ Futurreg

• Foresight as a technique for decision making in organisations.

• Evaluation strengthens the stated purpose of the project to provide a learning tool for European regions.

• Evaluate to ensure transfer is as good as possible.

• Foresight must be evaluated in context.

• Evaluate whether the results and conclusions are used and/or taken into account in the region.

¿Which kind of evaluation is required?

• On going supplements the monitoring system and broadens the capacity to manoeuvre.

• Ex post attempts to measure project effectiveness and efficiency.

Evaluation structure

• Context

• Methodology

• Data collection

• Data analysis

• Global analysis and selection of cases for study

• Summaries of results and/or impacts

• Communication of results and recommendations

Evaluation Methodology

• Analysis of the project in two dimensions:

• Analysis of the project as a whole

• Analysis of foresight applications performed in Futurreg. Applications grouped into different categories.

• Indicators

Indicators concerning the project as a whole, its objectives, toolkit and foresight methods.

• Three questionnaires for different actors involved in the project

Partners (7), Application managers (14-15) and Participants in foresight applications (case of studies)

Analize the project as a whole (I)

•As Futurreg is part of the INTERREG IIIC programme, its contribution to the programme objectives through this project needs to be evaluated.

•Importance to evaluate aspects like cooperation and project partner inter-communication, partner experience exchange, the toolkit, etc.

•Project as learning experience for participating regions and as a methodological guide for future experiences in other European regions in the futures and foresight field

Analize the project as a whole (II)

At this level, the main questions for evaluation are:

•Were the objectives well defined?

•Have they been achieved?

•Have the expected results been obtained?

•Was the workplan carried out?

•Have the deadlines and planned costs of the project been respected?

• Have the results been transferred from the project to other regions?

Analyze the project as a whole (III)

Indicators:

•Project activity and working plan indicators (objective indicators)

•Result indicators

•Others indicators

Indicators (I) AREA INDICATORS OBJECTIVE INDICATOR

General

Project activity and working plan indicators (objective indicators)

Organization and

coordination

Steering Committee meetings

Partner participation in Steering Committee meetings

Recruitment of local auditors

Partner refund deadlines

Activities performed according to initial project agenda deadlines

Good working atmosphere created and consensus promoted

Report sent on time

Activity and financial reports

Monitoring reports

Evaluation reports

Project development in alignment with planned objectives.

Project development in alignment with costs.

6

100%

6

2 weeks

100%

5

Every six months

Ongoing / ex post

Benchmarking

and evaluation

Number of regional and interregional stakeholders informed about Futurreg

Number of regions informed about their benchmarking position as per

regional indicators

Foresight tool report

Regional benchmarking and evaluation report

Opportunity matrix for regional toolkit applications

Creation of a basis and evaluation criteria for project applications and

implementation.

Ongoing evaluation report.

Ex post evaluation report.

Evaluation of interregional workshops.

70, 10 per region

5

7

1

Definition of

foresight

processes

Participants in interregional workshops

Type of participants in interregional workshops

Participant satisfaction with interregional workshops

Production of regional futures toolkit with guide.

Production of materials for up-skilling workshops

Interregional Workshops

Participants in each workshop

Production of plans for regional applications

Interregional matrix comparing application-related issues and features

30

85%

2 sets

2

30 in each workshop

14

Indicators (II) AREA INDICATORS OBJECTIVE INDICATOR

General

Project activity and working plan indicators (objective indicators)

Implementing

foresight

Number of regional stakeholders used in futures toolkit application

Satisfaction of regional organizations with accepted applications.

Number of futures-toolkit-related plans and future collaborations

Applications of futures toolkit

Regional application monitoring reports

Summary report on steps to ensure sustainability of resulting operations.

28 stakeholders, 4 per region

14

14

starting from 2006, 1

biannual for region. In all, 21

Diffusion Diffusion level for “champions” in public sector organizations (number of

encounters held and number of partners used)

Visits to web page

Publication of Futurreg project publicity brochure

Encounters with 105 stakeholders to diffuse toolkit and project

Interregional conference

Project web page, with information of interest on project and applications.

Project web page in partners’ web pages.

150 colleagues used by the

champions, 5 per champion

5,000 visits

3000 copies

2

100 participants

Other project indicators

Meetings held

Press notes released after meetings

Targets hit

Applications developed

Number of intra-regional and inter-regional networks created and participants

(public administrations, universities, technology centres, companies, etc)

Number of foresight experiences generated

Objective indicators

Objective 1 Number of agents involved (and interested?) in regional foresight practice.

Objective 2 Number of experiences at regional level in the use of foresight techniques.

Number of foresight exercises practised

Number of users requesting application in their own region. (Number of users

demanding foresight in their own region).

Objective 3 Number of integrated sources of information.

Indicators (III) Other Indicators

AREA INDICATORS

General Progress on creating foresight culture

Beneficiaries taking foresight results on board.

Regional public administrations’ satisfaction with, and interest in, foresight.

Objective 1 Transferability. Toolkit transferability level to other European regions. (Number of

applications from other regions interested in the application)

Extension of the interregional network that maintains foresight tools: number of partners,

class of partners, type of channels and communication flows, etc.

Toolkit adaptability and flexibility for other regions and studies. User friendliness, success

of project, etc.

Objective 2 Applicability. Implementation or use status of foresight results in regional economic

policies.

Regional public administration and business interest in toolkit.

Regional agents’ integrated use of foresight.

Sectors where it is applied (sector type).

Future sustainability of application in the region

Objective 3 Interest for the region of area/sector benefiting from use of foresight.

Impact on regional economy.

Quality of sources

Evaluating applications (I)

Critical areas(1) were used in classifying applications:

• Social, factors influencing human beings, society and lifestyle

• Technological, factors stimulated by science and its applications

• Economic, factors affecting industry and wealth creation

• Environment, factors impacting on the physical world we living

• Political, factors relating to government and administration

[1] Preliminary version of the UPGRADE blueprint has been produced for the dissemination conference “Building the future on knowledge”. Brussels, September 23, 2004.

Social Technological Economical Environment Political

Aristotle University of ThessalonikiMain theme: Innovation, new product development.Type of application: Strategy development towards technology and knowledge transfer..

Galway-Mayo Institute of TechnologyMain theme: Innovation in SMEType of application: Product Development and Innovation Planning.

Institute of Technology, SligoMain theme: Higher educationType of application: Develop strategy

Municipality of ThermiMain theme: Digital Research Centre: Digital Cities, e-goverment.Type of application: regional policies, organizational restructure.

ADER, INESCOP and AICCORMain theme: Innovation, research and technology.Type of application: Identification of Future Technological trends in the footwear Sector.

Malta enterpriseMain theme: Innovation.Type of application: Development of a regional innovation strategy.

University of MaltaMain theme: Knowledge TransferType of application: development of knowledge transfer strategy within the University´s ten year vision.

Loimaa Region Development CentreMain theme: Economic development (entrepreneurship)Type of application: Create networks and star up new joint projects among entrepreneurs.

Regional Council of SatakuntaMain theme: manages the responsability of the long term regional development, defines objectives of the long term development and provides guidelines for the member comunities of land uses.Type of application: preparation of a new Regional strategic plan

Countryside Council of WalesMain theme:Conserving tha natural enviroment and helping people to enjoy it and earn from it.Type of application:Developing the organisation´s strategic direction in medium term.

Menter a BusnesMain theme: Economic value of the Welsh Language.Type of application: Company development and strategic contribution to policy framework and programme for action.

Fondation Rurale de WallonieMain theme: Support for rural development operations and on going education..Type of application: diagnosis on future needs of rural territories of Wallonia in order to define the new strategy of the organisation FRW. Support existing working groups of the organisation.

Tr@me SCRLMain theme: Territorial development, tourism and patrimony, agriculture and sustainable energies.Type of application: improve the use of foresight in rural areas of Wallonia.

Athone Institute of TechnologyMain theme: Innovation/Sustainable development.Type of application: Development of a strategy/Organizational development.

Evaluating applications (II)

Case study selection criteria:

• Operational level and situation of exercise.

• Design of foresight exercise.

• Relevance of issues on which the foresight exercise is carried out.

• Territorial balance according to types and regions involved in the project.

• Novelty.

Evaluating applications (III)

The main questions (1) the evaluation considers at this level are:

1. Introduction of foresight in the region and complementarity to other regional actions.

Are the defined objectives consistent with the needs of the environment? Are they defined in detail? Has foresight implementation design taken into account regional characteristics and problems? Is foresight integrated and complemented by other types of regional actions, with which it shares common objectives?

2. Target hits

Has the project fulfilled its general objectives? Were results as expected?

(1) The questions are a result of the collective work of the partners and of the conclusions propoused in the last up skilling workshop

Evaluating applications (IV)

3. Foresight as a useful tool

Has foresight proved its use as a tool for partners and regional organizations? What type of method was used? Were the methods used appropriate for the targets set? To what extent was the foresight guide useful in the development of applications?

4. Foresight as a participatory process

What is the role of private and public agents in the development of the applications? Was civil society involved? Was the foresight practice carried out as an open, participatory process?

5. The role of experts, stakeholders and external consultants

What role did foresight experts, stakeholders and external consultants play in the performance of the project and applications?

Evaluating applications (V)

6. Application of results

Have the results of the foresight applications been applied in the regions?

7. Diffusion of regional foresight results

Have other European regions been informed of the programme results? Do regional agents know about regional foresight applications and their results?

8. Extension and sustainability of the foresight culture

In view of the foresight experience in the region, has the foresight culture developed in the region? Are regional organizations showing interest in maintaining and using this tool?

Evaluating applications (VI)

Indicators:

• Indicators on Futures toolkit

• Indicators on foresight methods

Indicators (I) Indicators on Futures toolkit

Number of foresight methods used

Availability of data and sources. (Number and quality of sources).

Duration of foresight exercise with regard to period planned.

Cost of foresight exercise in comparison to originally estimated cost.

Staff deployed in foresight process.

Number of experiences in which it is being applied.

Number of agents involved.

Indicators (II) Indicators on foresight methods

Scenarios Months in operation.

Cost of application.

Staff deployed and taking part.

Number of environment-related variables.

Number of scenarios considered and degree of difference between them.

Trend analysis Available data.

Time used.

Cost of application.

Staff deployed and taking part.

Number of variables selected.

Number of trend conditioning factors detected.

Delphi Time used.

Staff deployed and taking part.

Number of experts.

Level of knowledge in research field.

Quantification of questions.

Number of consultations.

Degree of consensus

Visionary Management Number of participants (working groups or key people).

Communication tools used.

Time used.

Cost of application.

Staff deployed and taking part.

Horizon Scanning Number of studies identified.

Number of studies selected.

Available data.

Number of trend keys detected.

Time used.

Cost of application.

Staff deployed and taking part.

Indicators (III) Indicators on foresight methods

Panel of experts Selecting the experts: profile, qualification, etc.

Range and variety of agents taking part (origin or sector to which they belong)

Indicators on work done by panel of experts: meetings, working documentation, documents

generated, deadlines reached, motivation of participants, etc.

Time used.

Cost of application.

Staff deployed and taking part.

Diffusion of panel’s results

Level of consensus achieved

Ideas generated

Futures Workshop Indicators on the management of this method: cooperation between agents, transparency

levels and fairness of process, number and length of workshops held, intervals between them,

etc.

Study horizon.

Time used.

Cost of application.

Staff deployed and taking part.

Ideas generated

Indicators (IV)

Other indicators

AREA INDICATORS

Toolkit Degree of complexity in handling

Degree of satisfaction in handling

Questionnaries

Presented below are three kind of questionnaires for agents taking part in the Futurreg project:

• Questionnaire for partners

The questionnaire is designed to obtain information on how the working plan developed and the measures worked, on the information generated, what type of difficulties were founded, etc.

• Questionnaire to application agentsSent to the agents of all regional applications. It will provide outstanding information on the management of applications, their design, how they worked, the methods used, the measures developed once foresight had concluded, the role of the experts, etc.

• Questionnaire to participants in applications This questionnaire will only be sent to participants in regional applications selected as case studies.

Evaluation methodology:

Presentation of Draft. 28 November 2006.

Deadline for comments. End of December 2006.

Methodology evaluation document. 15 of January.

On-going evaluation:

Evaluation criterion. 28 November 2006.

Indicators. 28 November 2006.

Evaluation of applications (selection of a specific number of applications). 30 November 2006.

Questionnaire 28 November 2006.

Data processing. 31 January 2006.

Draft ongoing evaluation report. 31 May 2007.

Ongoing evaluation report. 29 June 2007.

Evaluation Ex-post evaluation:

Evaluation criterion. 31 August 2007.

Indicators. 31 August 2007.

Evaluation of applications (selection of a specific number of applications). 28 September 2007.

Questionnaire. 28 September 2007.

Data processing. 31 October 2007.

Draft ex post evaluation report. 30 November 2007.

Ex post evaluation report. 28 December 2007.

Ch

ron

og

ram