Monitoring & Evaluation (mid-term/ex-post) of CRC's

Monitoring & Evaluation (mid-term/ex-post) of CRC's

Work in progress

Goal & Content & Approach• Goal

• Collect data about current monitoring and mid/end term evaluation practices

• Identify common basis• Try to distill ‘good practice’ approach

• Approach followed• Desk work• Based on documents from partners (> 1.500 pages)• Completed with additional web searches

• Content• Austria K-plus and K-ind/net• Germany Kompetenzenetze• Valencia• RCN Norway CRI• Flanders Competentiepolen• Sweden VINN Excellence Center• Estonia Competence centre program

AustriaK-plus & K ind/net

K-plus center evaluation (mid term evaluation - 4YE) 1 source: Ein Evaluierungskonzept für das

Kompetenzzentrenprogramm K plus (Nov 2001) Mixed group of reviewers

• 2 sets of experts Standing group ( 3 experts science-industry relations),

same group for evaluation of ALL centers Peer group (3 foreign scientific peers, 1 of them involved

in ex-ante evaluation) Support from small number of non-voting Austrian

experts 2 main steps

• Review of written information (Core document provided by center)

Ex-post part:outputs, outcomes, performance Future vision: research plan for years 5-7

• Second step (6-7 weeks later) 1.5 day site visit

Result: • Written evaluation report (incl. go/no go)

K-plus center evaluation (mid term evaluation - 4YE) 2 A Center has been working successfully if ...

it has managed to build up working structures (including boards, management ... according to K plus standards) and start regular business after an initial phase;

it has dealt with it’s own budgets and working plans in a responsible way – without ignoring new chances and challenges;

it has started to produce a sufficient number of scientific-technological outputs, like patents, publications or PhD’s;

it has succeeded to install a working transfer of results towards it’s industrial partners (without becoming a short- term RTD service provider) and if it could build up a knowledge base of it’s own;

horizontal issues like educational programmes and strategic research projects could be established successfully;

it has succeeded to formulate a long term strategy.

List of indicators for evaluation of K-Plus centers Outputs:

Performance of planning and management Publications Patents Successful tendering for RTD funds Invitations, scientific prestige PhD's, young scientists Conferences, workshops, visiting scientists

See also: Evaluation for Kplus Competence Centres , General Rules and Information Base, 1999

List of indicators for evaluation of K-Plus centers Effects:

Scientific relevance Long term cooperation Strengthening of quantitative and qualitative RTD

efforts in Centers' industrial partner structure Use of results on partner firm level New cooperation partners of the Center Cooperations between Center partners (firms!) Transfer of Personnel General Management and cultural issues

Austria Kplus and K ind/net : Program evaluation

Source: Assessment "Zukunft der Kompetenz-zentrenprogramma (K Plus un K ind/net) un Zukunft der Kompetenzzentren", Jan 04

Method used:• External evaluation by Fraunhofer & KMU

Forschung Austria• Qualitative analysis based on expert interviews (42

face to face) and primary documents• Quantitative based on structural data from existing

documents and survey conducted in the centres• Discussion of results with stakeholders, feedback

integrated in final analysis

Main Assessment criteria

Take structural changes/additionality into account, not only output!

Assessment criteria• Input additionality (more private R&D)• Behavior add.

More networking and more interdisciplinarity More risk taking

• Critical mass in Research domaine• Increase of interregional and international

cooperation

Targets (K ind/net)

Stimulation and increase of private R&D expend. Increased valorization and combination of existing

knowledge Bundling of resources Increased knowledge of and participation in Internation

funding programs More high tech spin-offs Improved local/regional "scientific conditions"

K-plus targets

Better use and production of knowledge with high additionality for science and economy

Attractiveness of Austria as a High-tech location and protection of Austrian knowledge capacity

Critical mass in industrial research Higher participation level of Austrian partners in

international funding programs Public awareness and Higher acceptance level for long

term strategic research in the (public) research debate

Criteria for monitoring of a complex program

Take needs from 3 parties/levels into account:• Political level • Program manager level (agency)• Project level (self evaluation)

Dangers of monitoring system• Political influence• Increased Overhead Should bring benefits for the

project organization• Numbers don't tell the whole story

Recommended Content of monitoring system 1

Financial data• % funding• % self financing (and sources)• Overhead costs

Organization issues (and changers hereto)• Project Partners (segmentation : Industry/Univ/…,

size, sectors, Austrian/Non-Austrian) • Non project partners (idem)• Cooperation in (inter)national funding programs


Scientific output and performance• External scientific reports• No and Q of publications• No of Masters and PhDs• No of Awards• No of Patents

Technolgy transfer activities• Spin-offs from CRC• Personal exchange with industry• No of prototypes• No of implementations of use in production

processes• Communication activities

Seminars with company involvement Joint publications with industry


Human Capital• No and profile of CRC scientific staff• No Liasons with industry• No Joint participation in scientific fora and kind• No of Expert activities• Internal training• No of doctoral/master/ … works

Q?: Match targets & monitoring system ?• Most items covered, some missing or less explicit

Increase private R&D-spending Critical mass only indirect Contribution to improvement of regional innovation

context Contribution to public debate about Strategic

Research

Germany

German Kompetenznetze current status

No information yet about monitoring & evaluation procedures (probably due to complex funding structure)

Further information requested at www.kompetenznetze.de

Benchmarking process (see presentation of Jan Wessels Compera Workshop 19.5.2009)

http://www.kompetenznetze.de/

Valencia

Valencia current status

Limited information available• Reporting guidelines• Program application structure

RCN NorwayNorwegian centres for research-based innovation

RCI Norway

Source : The Norwegian centres for research-based innovation ; Requirements and Guidelines; march 15 2007

Monitoring & follow up:• Annual report• Half yearly progress report• Site visits at expedient intervals• Foreign experts can be invited

Mid term (after 3.5 years)• Uniform evaluation scheme• Success criteria are defined (next slide)• In particular evaluation will assess the scientific results

and whether these lead to innovation, value creation and long term research

• Assessment of final 3 year plan and administrative conditions

Result: go: for overall 8 y, no go: terminate after 5 y

RCI Norway Success criteria for CRI

The research• The centre engages in long-term industrial research of

a high international calibre in the field outlined in the project description, and demonstrates its high quality through its production of doctorates, scientific publications, papers for presentation at recognised international conferences and other forms of scientific merit.

• The centre has a distinct research profile and has been successful at the international level (e.g. when researchers win prizes or are invited to be keynote speakers at international conferences).

• Researchers from the host institution and partners participate actively in the centre's research.

• The centre's user partners have increased their research activities both through participation in the centre's activities and their own R&D activities on topics of relevance to the centre.

RCI Norway Success criteria for CRI 2

Innovation and value creation • The centre's research has engendered or is expected to

engender possibilities for innovation and enhanced competitiveness among user partners and expectations about social ramifications over and above the partners' direct participation in the centre's activities.

• The centre has achieved mutual mobility of personnel between the centre and the user partners. Researchers from partners work at the centre and research fellows and researchers from the host institution are seconded to the user partners for periods of time.

• The centre has conducted projects to ensure that the competence and results achieved by the research are effectively transferred to and utilised by the partners.

• The CRI paves the way for results that fall outside user partners' core areas to be commercialised by other means, e.g. through establishing new research-based enterprises.


Internationalisation• The centre is successful in international research

cooperation, e.g. as a player in the EU's framework programme.

• The centre engages in active collaboration with international research groups and has also in other ways contributed to the internationalisation of Norwegian research and industry.

• The centre attracts outstanding foreign researchers, including research fellows and senior staff as visiting foreign researchers.


Researcher training and recruitment• The centre attends to researcher training

effectively, and helps to train highly skilled personnel in the centre's special fields.

• The centre is actively engaged in education, especially at the masters level, and helps improve recruitment to the centre's subject areas with particular emphasis on increased recruitment of women.


Partners and funding• The centre receives long-term funding from the

host institution and partners, and they increase their funding to exceed the minimum requirements.

• Active efforts are made to attract new partners and the centre's partners also include small and medium-sized enterprises with a high technology and innovation profile.

• The centre has been successful in securing other external funding.


Organisation• The centre has good visibility and a strong identity

and has successful collaboration with the partners. • The centre is organised in a manner that allows it

to fit into the host institution's organisation. • The centre has a Board of Directors and

management that help ensure that the intentions and the plan underlying the establishment of the centre are followed up.

• The centre has a common administration with a high degree of professional and administrative autonomy.

FlandersCompetence Centres

Mid term evaluation (2y) and End evaluation (4y)

Mid term• Based on

Self evaluation report (acc. specific structure) Annual reports Update Business plan next 2y

• IWT Staff -> IWT board ->Minister• Go/no GO: in most cases conditional go

End evaluation• Based on

same info sources Mid term New business plan New project plan

• External experts & IWT staff -> IWT board -> Minister -> Flemish Government

• Most cases continuation Remark: IWT is represented in board of CRC!

Mid Term Reporting 1

Chapter 1: Description of the Competence Pool– Mission & objectives– Activities (summary of headlines)– Organization (structure, number of employees, key

strategic processes)

Chapter 2: Logical Framework Analysis

Chapter 3: Input– Financial data from year to year– Detailed overview of activities– Expenditure per activity

Mid Term Reporting 2

Chapter 4: Output– Outputs per activity (e.g. consultancy, projects realized,

number of visitors) and discussing these (linked to output indicators)

Chapter 5: Results and effects– Results and effects by activity and for the overall

mission and objectives

Chapter 6: summary of performance indicators– Reports on agreed list of indicators

Chapter 7: Findings– Conclusions & options for improvement, suggested

changes

Monitoring system 1 Services Activity reporting (since 2003 in e-tool)

• 4 - 6 monthly e-reporting• Std set of activity indicators (with target values at start)

No of company visit No of publications No of seminars No of innovation audits No of innovation plans No of innovation projects No of Tech Transfer No of partnermatching No of networking activities …

• Effort reporting• Success stories• Std set of direct effect indicators (target values at start), also

other allowed• Set of indirect effect indicators (idem), idem

Monitoring system 2

Research projects (e-reporting not yet implemented)• Follow-up of deliverables• Project progress (task completion)• Follow up of valorization status (dissemination and

use of results)• Efforts (mm planned vs actual)• Follow up of special contractual conditions• Follow up of user groups

Finland : Strategic CentresInformation requested at Tekes but not yet received

VINN Excellence CenterSweden

Based on case study 'Wisenet' (mid term evaluation)

Source: IWT colleague participated as observer in evaluation 3-5 march, 2009

Documents by VINNOVA (very well documented procedure!)

• Guidelines for the evaluation of VINN Excellence Centres and Berzelii Centres (june 2007, februari 2008)

Vinnova evaluates 19 centres (running for 1.5 to 2 years) in 4 blocks, 5 centres/week

2 generalist evaluators lead ALL centres evaluation (2 university professors, non swedish, with high level of routine in this matter)

• Increases efficiency of process• Have understanding of 'reference'-level

2 foreign external scientific experts, academics with sound knowlegde of domain of centre

Based on case study 'Wisenet' (mid term evaluation) 2 Focus of this first mid term evaluation

• General evaluation• Organisational issues• Mgt• Financial issues• Contact with companies• Cooperation among scientific partners

Scientific experts involved to make the acquainted with the centre and evaluation procedure for evalution in later stages

Centre has to prepare report based on strict guidelines (40p)• Mission, vision• Partners• Research program• Financial report• Organization and management• Personnel • Prospects

Based on case study 'Wisenet' (mid term evaluation) 3

Day 1: evening• Get together experts & Vinnova • Briefing• Exchange of impressions based on report

Day 2: • Site visit

short presentations Interviews

• Report by external experts (6 pages) with recommendations (ranging from 2 to above 20) and comments

Report is send to centre that gets a 2 week period to correct factual mistakes

Teleconference feedback from VINNOVA to Centre Most cases continuation for 2 y (conditional) Vinnova publishes summary of all evaluation reports

Competence Centre ProgrammaEstonia

Estonian Competence Centres mid term evaluation

Source: Selection process of Estonian Competence Centres 2009; Mid-Term Evaluation of the Competence Centre Programme (2008); Competence Centre Programme Feasibility Study (2002); Regulation of the minister (June 2008)

Programme ‘design’ and first mid term evaluation of the programme by external consultancy organisation

2 External scientific experts involved with a sound track record in CRC

Goal of mid term evaluation: to assess the rationale, appropriateness and objectives of the Competence Centre programme and to make recommendations for the next funding round

Remark: designer (Technolpolis) of the programme is also the first evaluator …

Estonian Competence Centres mid term evaluation 2• 5 CRC were involved (peer reviewed by 2 scientific experts also

involved in the selection phase)• Elements of the evaluation

• Team, management• Partnerships in R&D-projects• Balance Fundamental, basic, applied research, synergy between

projects• Scientific output

Publications Patents Post graduates produced FP funding

• Long term sustainability, and related policy• Potential industrial application• General SWOT

• Quotes to reflect upon:• “The strategy was overly driven by company needs, and

therefore lacked research focus”• “SME’s can not provide long term commitment”

Estonian Competence Centres mid term evaluation 3

• Also stakeholders were interviewed• Universities• Companies• The researchers

Some recommendations from the Estonian Excercise

• Use matching and fully qualified scientific experts• Provide a clear and quantified scoring system to the

experts• Provide distinctive questions to the experts, treat

different aspects in separate questions

• “A set of criteria that is bot more differentiated and that uses a mix of words and numbers would in the future provide a clearer picture of peers’ review”

Some general conclusions on Monitoring and Mid term / End evaluation of CRC’s

Work in progress

Monitoring of CRC’s

• A large variety of (mainly science based output) indicators available

• Service indicators less used/described• Few (direct) effect indicators• Little information on how to collect and use the

monitoring data in an efficient way

Monitoring not really used as a management tool

Mid term & End evaluation of CRC 1

• When?• Mid term early in development of CRC, 2 years

after start up• “End” term at the end 4 to 7 years after start up

• How?• By 2 groups of external experts

Generalist, – Familiar with the evaluation process– Same for all CRC’s under evaluation

Scientific peers– Focus on scientific quality of the work

• Based on Written report by CRC (more or less fixed format) Complemented with site visit and interviews

Mid term & End evaluation of CRC 2

• What?• Mid term focus on:

Managerial issues Scientific performance Future plans

• End term (less clear) Scientific output

• Results?• Mid term: most cases continuation and new plan

for remaining time• End term: ?

Monitoring & Mid term & End evaluation of CRCSome recommendations

• LFA at selection phase • Can help to get coherent and SMART objectives• Helps to identify monitoring indicators and set clear criteria

for mid term evaluation• More emphasis on self monitoring and self steering to increase

agility of centre• Monitoring indicators = Crucial management info• Also monitor direct effects• Early alerts on defaulting CRCs• Common reporting/monitoring tools for all CRCs required

• More stakeholder involvement• Improves demand driven character of R&D• Improves technology transfer

• Involvement of ‘Company/economic’ experts besides technology/scientific experts

• Stronger involvement of funding agency in CRC• Active participation in mid term evaluation process can have

added value (eg. bench mark reference level)• Representation in the board (as observer) to increase cross

CRC cooperation, indentify common needs of CRC, …

Work in progress : What’s next?

• Collect missing information and non-partner information

• Finland • Germany• France

• Complete analysis• Formulate a ‘good practice’ approach and

recommendations• Report next workshop (November 09)

BUT : What do you expect from this exercise?

Monitoring & Evaluation (mid-term/ex-post) of CRC's

Documents

Transcript of Monitoring & Evaluation (mid-term/ex-post) of CRC's