What can be learnt from thematic evaluations · 2015-03-09 · What can be learnt from thematic...

What can be learnt from thematic evaluations ?

Alasdair Reid - ADE S.A. 1


Draft Paper presented at the Fourth Conference on the Evaluation of the Structural Funds

Edinburgh 18-19 September 2000

Alasdair Reid, ADE S.A. i [email protected]

Abstract This paper seeks to draw lessons from a methodological viewpoint for thematic type evaluations from three major evaluations carried out during the period 1998-99 by ADE: • External evaluation of the Community Support Frameworks for Objective 3 and 4 in Belgium for the

evaluation unit of DG EMPL (final report April 2000); • an Evaluation of Phare financed Agricultural Reform Programmes in the Candidate Member States for the

evaluation unit of the SCR (final report November 1999). • the Thematic Evaluation of research, technological development and innovation related actions financed

under the Structural Funds (Objective 2 regions) for the evaluation unit of DG REGIO (final report December 1998);

Carrying out sectoral or thematic evaluation requires that a vast amount of information be summarised in a concise and ordered manner before any in-depth analysis can be undertaken. To varying degrees, the analytical approach was based on the logical framework analysis approach adapted in order to apply the technique to programmes with multiple objectives. In two of the evaluations a classification by type of measure or actions was used to group like measures across programmes in an inventory. The development of such an inventory in the absence of well designed programmes and adequate monitoring systems can consume an excessive part of the evaluation study resources yet is at the same time crucial to understanding the underlying logic of the interventions and the weight given to certain objectives. The case of the Belgian Objective 3 and 4 CSF evaluation (covering 10 Operational Programmes) illustrates that even within one (federalised) country variations in programme monitoring and evaluation methods can seriously constrain the possibility of carrying out a sectoral review in terms of comparing baseline indicators and results. The Phare Agricultural evaluation shows that even when evaluators have access to a database of financial information and a multi-country monitoring system, the development of a baseline inventory permitting further in-depth research is not an easy task. The Thematic Evaluation of RTDI in Objective 2 zones highlights the methodological difficulties in undertaking synthesis or thematic evaluations across a large number of programmes and countries when monitoring and evaluation data and systems vary widely. In-depth case studies become the only, partial, means of assessing effectiveness of different types of instruments.



1. Introduction and conceptual considerations This paper sets out to answer a simple but at the same time relatively ambitious question: what can be learnt from thematic evaluation? More precisely, given the objective of such evaluations, what is the impact on policy design for subsequent rounds of programming? The term thematic evaluation has come to the fore in recent years at European levelii. To date, the main client for such work has been the European Commission and notably the DG Regio which financed a series of thematic evaluations in 1998-99 (SME, Partnership, Research Technological Development and Innovation, Environment, Transport). In fields other than regional development, the Common Service for External Relations (SCR) of the Commission and global organisations such as the World Bank iii have also commissioned such studies. Although, it is not the purpose of this paper to engage in a semantic debate about the definition or the role of thematic evaluations, it is worth clarifying the concept at the outset. Indeed, it is sometimes used loosely to encompass evaluations that include aspects of a meta-evaluation (evaluation of an evaluation) or synthesis approach. On other occasions the term thematic is replaced by sectoral (this is the term preferred by the World Bank and the SCR for instance). In the glossary of terms proposed by the MEANS collection, the term ‘Thematic Evaluation’ is defined as an :

Evaluation which analyses in a horizontal manner a specific topic (a theme) in the framework of several interventions within one programme, or in several programmes implemented in a number of different countries or regions.

From the point of view of the MEANS approach, the theme should correspond to an expected impact (e.g. increasing competitiveness of SMEs) or a distinct field of intervention (e.g. research and innovation). A thematic evaluation should provide the occasion for both an in-depth analysis of a specific evaluation question (e.g. the impact of support for rural diversification on employment in rural economies) with a broader comparative vision of effectiveness or good practice at a multi- regional or European level. With respect to the question addressed by this paper, the contribution of such evaluations to the policy design process depends on the extent to which a comparative multi-programme analysis can shed light on the relative effectiveness of certain types of intervention and identify “best practice”. The factors determining the effective contribution of a thematic evaluation to the policy design process include their timing in the programming cycle, the comparability of the baseline data and monitoring information, the quality and consistency of on-going evaluation reports, etc. The question of timing is a classic problem for the clients of evaluations of all types. Launching a thematic evaluation too early is likely to lead to difficulties in terms of identifying best practice or reporting on comparative effectiveness of different types of measures. Starting too late will mean that the results can not be integrated into the orientations and guidelines for the next programming round. Thematic evaluations are often contingent on the availability of on-going or mid-term evaluation reports complicating again the decision on timing the launch of the former. Here there are no hard and fast rules but it does suggest the need to establish very early in the programming cycle a timetable of evaluations to facilitate the carrying out of thematic evaluations during the second half of the programming cycle.



The availability of a reasonably consistent level of baseline data is pre-condition for a successful and good quality thematic evaluation. The MEANS volume on thematic evaluation approaches sets out clearly the strengths and weaknesses of various information sources that can be drawn on by thematic evaluation teams (monitoring systems, statistical surveys, studies and evaluations). The limitations of existing data (for example, the heterogeneity of indicators) and information are important since the budgets for thematic evaluations constrain the amount of ex-post data collection that can be conducted. Moreover, attempting to construct ex-post a series of comparable indicators for measuring effectiveness, etc. is fraught with difficulties while case study analysis always runs the risk of being considered anecdotal and subjective. Hence, a fair baseline hypothesis is that thematic evaluations can only be as good as the monitoring systems of the programmes analysed. An additional factor influencing the quality of the results is clearly the appropriateness of the methods and tools used by the thematic evaluation teams. The approach to the evaluation and the tools used clearly is a function of the initial terms of reference, the remit given to the evaluators, and the budget available iv. This paper draws on three recent evaluations carried out by ADE S.A. for the Commission services in order to examine how such factors influence the usefulness of thematic evaluation. All three have a common element of stocktaking at a late stage in the programming cycle with a view to proposing recommendations for next round of funding. Hence, their output is essentially targeted at policy makers and at improving the effectiveness of future rounds of programming. The three evaluation considered correspond to distinct types of thematic evaluation : • a single country / multi-programme evaluation. The external evaluation of European

Social Fund (ESF) Objective 3 and 4 programmes in Belgium. A synthesis approach concerning 10 individual Operation Programmes for two Community Support Frameworks for the period 1994-99;

• A multi-country / multi project evaluation of all Phare financed interventions in support of reform in the agricultural sector in ten Central and Eastern European Countries. The evaluation covered some 773 projects implemented between 1989-98;

• A multi-country, multi-programme evaluation of research, technological development and innovation type interventions supported under Objective 2 of the Structural Funds. The evaluation covered 89 Objective 2 programmes in 11 Member states with a focus on the period 1994-99.

The paper is structured around a discussion of four main points for each of the evaluations considered: • context of the evaluation study - the objective, role and timing in the

programming/evaluation cycle; • method and tools applied to carry out the evaluation; • the limits of such analysis in the specific context of each type of initiative; • the lessons that can be learned in terms of evaluation methods and policy design. A concluding section aims to provide some suggestions on setting up and carrying out thematic evaluations in the period 2000-2006



2. European Social Fund programmes in Belgium – evaluating in a Federal context Context Although labelled an "External Evaluation" the study carried out by ADE during the twelve months beginning in May 1999 was essentially focused on providing a synthetic analysis of the implementation of the Community Support Frameworks (CSF) for Objectives 3 v and 4vi in Belgium. Given the highly federalised nature of the Belgian state, evaluations covering the entire territory are extremely rare events and demand a good understanding of the complex system of governance. Each CSF comprised a separate operational programme (OP) managed by each of the central or sub-national governments with competence over employment and/or training policies : Ø 1 OP for the Federal Ministry of Employment; Ø 1 OP for the Flemish administration; Ø 1 joint OP for the French Community administration, the Walloon administration and the

Brussels regional administration (COCOF); Ø 1 OP for the Brussels regional administration; Ø 1 OP for the German-speaking community. To a non-Belgian reader it is important to underline that the territories and/or topics for interventions of these programmes were not mutually exclusive. The most flagrant example of the underlying complexity of the programming structure is the case of Brussels-Capital. Here not one but four separate programmes (those of the Brussels region, the French and Flemish-speaking authorities and the Federal Ministry) intervened in favour of essentially similar goals with complementary (or indeed overlapping) tools. At the very least the interventions required co-ordination to ensure an efficient and effective outcome. Indeed, the CSFs for Objective 3 and 4 were adopted in 1994 essentially due to the insistence of the Commission services that the various programmes proposed contribute in a coherent manner to meeting a certain number of objectives across the Belgian territory. The CSF was an attempt to fix certain verifiable objectives for the intervention of the ESF, succinctly these were : • for Objective 3 : the effective application of the principle of “pathways to integration” to

all actions funded at the latest by 1997; • for Objective 4 : the focus was on anticipation and the inter-sectoral nature of the actions

with priority given to improving training and advisory systems and to increased participation of the private sector in training vii.

The terms of reference defined the objective of the external evaluation as “the drawing up of a synthetic appraisal of the implementation of ESF interventions in the framework of Objectives 3 and 4 at the level of the CSF”. An emphasis was placed on the need for both a comparative approach taking account of the different contexts, notably institutional, of the implementation of the programmes; and a synthetic approach making available an overall vision of the efficiency and effectiveness of the ESF actions across Belgium. The terms of reference set out a series of thematic questions to be taken into account with respect to both Objective 3 and 4 such as the methods for implementing and monitoring the pathways to integration



approach for the former; and the way in which concepts such as anticipation of labour market needs and industrial change had been interpreted at the operational level for the latter. The evaluation took place in 1999 which was essentially the end of the programming period. However, in reality, and as for all Structural Fund programmes, the majority of measures of and actions will run until 2001. The situation is further complicated by the fact that a number of the OPs had faced difficulties in launching actions (notably the Objective 4 OPs) making the assessment of effectiveness well nigh impossible. During the evaluation, the various authorities had already begun preparation for new Objective 3 SPDs for 2000-2006. Methodological approach and tools; The evaluation was conducted in essentially two phases : • A thematic analysis based on existing documentation with a view to drawing up the

synthetic appraisal which was presented in an interim report. • A second participatory analysis phase via a series of group interviews or round tables

with the programme managers and on-going evaluators for each of the OP. The aim here was to validate and nuance the documentary analysis, identify good practice and discuss initial conclusions and possible future orientations.

The analysis of the first phase was essentially based on a logical framework type structuring of a series of research questions (socio-economic and institutional context, strategic objectives of the OP, internal and external coherence, types of activities, financial absorption and activity completion rates, resultants and impacts). For each OP, the available evaluation reports, monitoring committee minutes, annual or multi-annual reports, etc; were analysed and summarised with respect to these research questions in a standard template. In order to take into account the multi-objective nature of the CSFs, a logical impact diagram was also prepared for both CSFs. This diagram was constructed taking into account the various types of activities/measures in the five OP which were grouped under a series of standard measures. The example of the Objective 3 diagram is given below (in French). In many respects this was an ex-post reconstruction of the intervention logic that did not necessarily coincide with any ex-ante strategic vision. Such a vision was largely lacking in any event due to the weakness of the OPs as programming documents (no clearly stated results or impacts indicators, etc.) and the composite nature of the CSFs.



Following the interim report, and in order to focus the activitie s of the participatory phase, the analysis was restructured around five key evaluation questions : • Do the strategies (priorities and target public) adopted respond to the identified and

recognised socio-economic needs ? (relevance) • Is there are logical relationship between the measures implemented by the OP and the

objectives of the CSFs ? (Coherence) • Are the first results and impacts contributing to attaining the objectives of the CSFs ?

(effectiveness) • Has the programme management and the way in which measures have been implemented

contributed to optimising the results ? (efficiency) • Has the intervention of the ESF led to a real added-value with respect to existing policies

? (taken as a proxy for impact) A final aspect of the approach was the attempt to undertake an international comparison with the ESF experience in the UK. In the end this exercise was limited largely to contributing ideas on alternative approaches or methods for programme implementation (for instance in terms of selection systems) although at the outset it had been hoped that it would be possible to make some comparison between the effectiveness of measures.

Diagramme Logique d’impact “générique” - CCA Objectif 3

Aides à laformation deformateurs / tutoraten entreprise

La personneest inséréedurablement

La personneest insérée

La personneest insérable

La personneentre en contactavec un/desemployeur(s)

La personneest accompagnée sur son lieu de travail

La personnereçoit un conseilou une médiation

La personneacquiert unequalificationdemandée

La personneacquiert uneattitude pro-fessionnelle

La personneacquiert uneexpériencede travail

La personnesurmonte desobstaclesspécifiques

La personneest motivée

La personneacquiert unequalification

La personnesuit uneformation

en alternance

L ’opérateur crée ouadapte la formation àla demande

Les opérateurs s ’organisentafin de repondre aux besoinsdu public cible et des firmes

La personneest conseillée

dans le cadred ’un suivipersonnalisé

Aides auxcrèches

Action spé-cifiques pourpersonneshandicapées

Soutien auxstructuresdepartenariat

Aides à laformation

Aides auxstructuresd ’accueilet de suivi

Aides auxstructures deformation

Le bénéficiaire avecenfants suit de façonoptimale son parcours

Aides àl’embauche

Le handicapé recoit uneformation / experience detravail spécifique



Limits to the analysis The initial terms of reference set a limitation on the scope of the evaluation by excluding additional field-work such as surveys or interview with beneficiaries or project promoters. This in part reflected the budget available but also the limited time-scale. Indeed, the evaluation teams responsible for the on-going/mid-term evaluations of the Ops were supposed to carry out surveys and in-depth work. With respect to the five questions posed the main area where it proved difficult to arrive at robust conclusions was effectiveness (comparable or otherwise) of the interventions. Although it was possible to propose a series of general conclusions, these were based essentially on a qualitative analysis with little quantified evidence or in-depth analysis to substantiate them. Corroboration was possible with respect to the absence of impact of Objective 4 measures in favour of studies in terms of leading to new training actions at least in part thanks to a relatively good analysis for Wallonia of the outcome of the studies. As far the Objective 3 actions are concerned, the quasi-absence of data on the effects of training-guidance-job placement measures meant that the effects on the beneficiaries could not be assessed. The reasons for the absence of data were both attributable to the weaknesses of the monitoring system for beneficiaries and the limited data available in the form of longitudinal surveys of participants carried out by the on-going evaluation teams (surveys were only carried out by two of the five OP evaluation teams). In terms of the monitoring systems, available indicators were limited to financial data and activity figures (number of participants to measures) and no harmonisation or coordination of methods had been attempted between the five authorities. The Flemish region had developed a relatively sophisticated dual database system (one for following financial data, the other for tracking beneficiaries); while, the German Community (due to its small size) had carried out an exhaustive survey and put in place a centralised system of collect of information on the beneficiaries. The other three authorities either had taken no action to assure a follow-up of the impact of measures on beneficiaries (Federal level) or left the follow-up to individual project promoters. In the Walloon region, there was a simple refusal, partly on grounds of protection of individual rights, to centralise data in any form of database. What is remarkable is that the OP evaluations teams themselves generally based their analysis on such incomplete “second hand” data and material with little in-depth analysis or case study work. In conclusion, the overall quality of the thematic evaluation certainly suffered from the lack of case studies which in the absence of good quality data on the effectiveness of measures could have served to shed more light on the impact of the programmes. Lessons learned As has been underlined above, Belgian wide evaluations are rare occurrences due to the lack of a hierarchy of power between the institutions. This meant that the Commission had to play a strong role in launching a process of “inter-governmental” evaluation in a context in which it would not normally have occurredviii. The creation of a steering committee bringing together the main partners of the CSF but presided by the Commission as a “neutral” arbitrator, allowed an interesting and at time remarkably candid exchange of views between the Belgian authorities to develop. Clearly the expectations of the various partners were different and the satisfaction with the outcome or the utility of the exercise variable. The



Flemish authorities, whose programme and procedures were the most advanced, were less willing to take on board recommendations about improvement of monitoring or selection procedures (based on, for example, the UK comparison). Although there is obviously regular contacts between the authorities, exchanges of information seem limited to formal communication on specific topics. Hence the external evaluation, despite its limitations, not only provided both the Commission with a more complete view of the effects of ESF interventions, but would seem to have equally contributed to an exchange of ideas between the various authorities on programming procedures and on what types of approaches/measures are effective. The final report of the evaluation was the subject of a diffusion seminar in May 2000 to which the members of the CSF Monitoring Committee (including the social partners) as well as the OP evaluation teams were invited. By this time, all five authorities had already submitted their new Objective 3 SPDs and had largely completed the negotiation procedure with the Commission. From this point of view, one could assume that the policy lessons aspect of the evaluation was essentially a failure since the results and conclusions were only made available and debated with a wider audience at this time. However, the conclusions of the evaluation had been already largely discussed and appropriated by the various parties to the evaluation steering committee so that the Commission was in a position to draw on the report when negotiating with the Belgian authorities.

ESF Evaluation Belgium - Main conclusions and policy recommendations Conclusions. • the Operational Programmes were largely insufficient as a framework for implementing a

coherent intervention strategy. The documents were inadequate in terms of weighting of financial priorities with respect to needs and in terms of coherence between programmes ix.

• The CSF Objective 3 permitted the launching of a process of partnership at local level with an opening of the training-job guidance system to specialised third parties (non-profit organisations) better placed to deal with the needs of specific groups of targeted personsx;

• The assessment of the extent to which the aims of the CSF Objective 4 had been attained was globally negative . In particular, there was little or no evidence of the creation of mechanisms permitting a recurrent process of anticipation of training needs and very limited private sector involvement in the programmes. Isolated cases of good practice were identified.

• Project selection procedures and criteria are largely insufficient and there is a marked tendency towards funding the existing “training-employment system” rather than a selection of projects based on quality and coherence with strategic objectives. Political intervention in the selection process is a characteristic of the Belgian system.

• Very limited progress in improving the quality and effectiveness of monitoring systems . The conclusion here is less valid for Flanders and the German speaking community.

Recommendations Strategic level: • increase the capacity of local and sectoral groups to undertake needs analysis and input to

employment policy. This should be done through reinforcing the coordinating role of employment observatories;

• increase the financial resources available for technical assistance to programme managers notably with a view to ensuring a more harmonised monitoring system linked to the National Action Plan objectives; and increasing the exchange of experience between the different programmes.

Appropriateness of actions (effectiveness, needs) • reinforce the resources made available to the last stage of the pathway to integration approach – job

search and placement - and make the reduction of time -lags between the different stages a key criteria for project selection;



• discontinue funding of measures aimed at subsidising employers for hiring unemployed persons (hiring bonuses) due to deadweight effects and difficulties in coordinating with other actions;

• concentrate resources related to the anticipation of training needs in SMEs on direct support measures to SMEs such as the extension of financial support for “training plan” schemes. Avoid funding general studies on training needs by academics/training organisations in favour of integrated actions (training needs-design and delivery of training) involving beneficiary firms with a specialised organisation.

Programme management and implementation procedures • adjust funding criteria (based on number of beneficiaries) to incite project promoters to choose

between volume (high coverage rate) and intensity (limited number of participants with specific needs);

• Reinforce substantially selection procedure with the introduction or extension scoring systems and external or partnership (peer review) based methods of selection;

• Improve the monitoring and evaluation procedures for the future SPDs notably through a more rapid launch of on-going evaluations – increase the independence of the evaluation process by making a clearer distinction between technical assistance and evaluation;

• Place a stronger obligation, with eventually corresponding additional funding, on project promoters to provide a limited number of statistics on the results of training activities on beneficiaries;

• With a view to imp roving the understanding of impact of measures, increase the regularity of longitudinal surveys of ESF beneficiaries and harmonise data collected in order to provide comparable data for the NAP. Sub-contract such survey work to a single specialised organisation.

3. Phare agriculture evaluation Context The Evaluation of Pharexi financed Agricultural Reform Programmes in the Candidate Member States (CMS) was part of a series of sectoral evaluations under the 1998-1999 Phare Evaluation Programme, implemented by the Evaluation Unit of the Common Service for External Relations (SCR) of the European Commission. The evaluation covered all Phare-financed programmes in support of reforms in the agricultural sector in the ten CMS in Central and Eastern Europe (Estonia, Latvia, Lithuania, Poland, the Czech Republic, the Slovak Republic, Hungary, Slovenia, Romania and Bulgaria), from their inception until the end of 1998. The total budget allocated to these programmes amounts to € 480 millionxii or some 6 per cent of the cumulative Phare budget over that period. The purpose of the evaluation was not only to provide a historical assessment of Phare contributions to agricultural reform in the CMS. It also aimed to offer guidance for possible improvement in the design and set-up of future Phare programmes in this domain, and thereby contribute to facilitation of the accession process. Another aspect, given the large number of project appraised, was the contribution of the study to increasing the transparency and accountability of the EU’s expenditure under the Phare programme. The evaluation carried out by FAI Ltd (UK) and ADE S.A. for the Evaluation Unit of the SCR began with a launch workshop in November 1998 and was completed in December 1999xiii.



Methodological approach and tools; Whilst the Phare agricultural programmes are designed for individual countries, the evaluation took a horizontal approach across countries and considered projects grouped under a number of themes, which represented the priority areas for Phare intervention. The evaluation was conducted in two main phases (plus a final report phase): Phase 1 involved compiling an inventory of all agricultural projects undertaken in each country since the beginning of the Phare programme to the end of 1998. This provided details on the overall scope of Phare assistance to the agriculture sector. A ‘scoping mission’ was conducted in each country to work on the project inventory and to hold preliminary consultations with key actors responsible for the Phare programme. On the basis of the inventory, a random sample of projects was selected for in-depth evaluation on the basis of a number of stratification factors these being the themes, the size xiv, time of implementation and country/regionxv. Given the evaluation resources it was possible to consider a sample of 25 projects over six selected themesxvi : Ø Land reform Ø Credit market Ø Marketing and market information Ø Extension, research & development, education & training Ø Food safety, animal & plant disease control Ø Policy & legislation, agricultural statistics

The evaluation process for the PHARE agricultural evaluation DESIREE Database

(completed with financial data)

Programming and monitoring documents

(National Indicative Programmes,

OMASxvii Reports, etc.)

Scoping missions in the countries

Detailed Inventory of Projects

773 distinguishable projects € 480 mln

Sampling Frame 239 evaluable projects

€ 303 mln

Statistical analysis of the total project population (detailed inventory of

projects) Sample of projects to evaluate

25 evaluable projects € 92 mln

Phase 2 consisted of the preparation of reviews for each selected theme (developments, outstanding problems and current strategy) drawing on published data and discussions with those working in the area to progress. The main fieldwork stage was the evaluation of the selected sample of projects using the logical framework approach. Individual evaluation reports were prepared for each project visited



The in-depth evaluation methodology was based on the logical framework approach. In particular "Generic logframes" were constructed for each of the six themes. The objective of these generic logframes was to serve as evaluation grids for projects in the same theme, ensuring methodological consistency and comparability for the evaluation of projects within and across themes.

Model for Generic Logical Frameworks

Evaluation criteria

Level of analysis

Questions to investigate

Relevance Programme to which the project under review belongs, and project

• How does the programme respond to needs and how does it fit into a reform strategy?

• How does the project contribute to the programme? • Is the overall composition of the programme appropriate:

(complementarity with other Phare projects, other donors intervention, Government actions)

• Justification for using Phare grant funds

Adequacy of inputs Project level, types of inputs

What inputs were provided? How?

Output efficiency Project level, types of outputs

What outputs were delivered?

Qualitative aspects of the delivered output: did they meet the need? Did they correspond to the ToRs?

Effectiveness Project level, use of output

• Advice, studies, legal proposals, etc.: have they been adopted, approved, discussed, considered, used as a basis for action? If not, what were the reasons and the main obstacles?

• Training: are trainees employed in the capacity they have been trained for? Is the training contributing to the ability of the trainee to do his/her current job?

Supplies/equipment: is it in place, maintained, operational for the intended purpose?

Impact Project level, aggregating all outputs

Analyse the results against the wider objectives of project which are usually some distance from project output.

Sustainability Project level, aggregating all outputs

Has level and quality of target activity instigated by the project been sustained since the end of the project?

In addition, the evaluators and specialists were asked to provide a score for each of the different evaluation criteria separately. Scores were on the scale of 1-5, with each score having the following approximate meaning: 1: Zero, negligible or extremely poor outcome, 2: poor, or much less than satisfactory outcome, 3: a just acceptable outcome, 4: good outcome; 5 extremely good outcome. An overall score was provided both for relevance combining the three components and for the efficiency, effectiveness, impact and sustainability criteria. The relevance score was not combined with the other scores given the difference in the level of evaluation (the one mostly at programme level and set against the theme review) and the other at project level.



Limits to the analysis; The main technical difficulties in conducting the evaluation centred around the creation of an inventory of projects. Within the Commission there is no centralised systematic archive or comprehensive database of Phare assistance to Eastern Europe and it was therefore necessary to devote a substantial share of the evaluation resources to obtain a clear picture of the population of projects. The starting point was the Desiree database which is the main Commission information source on budgets and contracts. It uses the following conceptsxviii: Ø Programme: defined for a particular year (the programming year) and country and

regroups a set of interventions in a particular domain. There is no consistent list of these domains, which are generally the main categories of assistance identified in a country operational programme. So that in some cases, a country operational programme can group agricultural projects under one or several agricultural programmes, while for another country, or for the same country in another year, there may be no agricultural programme, but agricultural projects categorised under programmes such as “support to private sector development” (for instance, farm privatisation projects) or under “human resources development” (e.g.: institutional strengthening of a Ministry of Agriculture).

Ø Project: a project is one or several sub-projects belonging to a programme. Ø Sub-project: a sub-project is an activity belonging to a larger intervention. Ø Contract: a contract is the formal document by which the Commission allocates the

resources to a contracting firm or individual in order to implement the specified activities. This classification, apart from the concept of contract that is strictly defined but does not permit to identify a consistent intervention since it can be related to a minor component of it, was considered inappropriate to select projects for evaluation. For this reason the evaluation worked with two concepts: • Distinguishable projects are identifiable interventions, from the Desiree database or

from other sources, which are conducted as part of agricultural programmes or relate to the agricultural sector. In most cases distinguishable projects correspond to sub-projects in Desiree; they may regroup several contracts when these are obviously linked to the same activity. In general distinguishable projects correspond to an administrative concept and do not constitute an appropriate basis for evaluation since they relate only to part of an intervention.

• Evaluable projects group together distinguishable projects that are complementary to each other or continuations of previous activities. For example, two distinguishable projects could be the provision of material for drawing geodesic information and the technical assistance associated with the use of this material. Such projects cannot be evaluated independently from each other. Neither is it sensible to consider just one part of an activity because it happens to have been financed under two programmes and therefore appears as two separate projects. Accordingly, for the purposes of the evaluation these projects and sub-projects were grouped together to form “evaluable projects” comprising a coherent set of activitiesxix.



The detailed inventory comprises 773 distinguishable projects having a total value of approximately € 480 million. However, it is not exhaustive. Many early programmes (before the programming year 1992) are missing or not identifiable and there are projects for which no written information could be found. Although it was not possible to ascertain the exact magnitude of the missing information, the prevailing opinion among Commission and country Programme Management Unit (PMU) staff familiar with Phare data is that it can be considered as marginal and mostly related to early projects. In addition to being a list of Phare projects, not available until now, the detailed inventory fulfilled two functions described in the two boxes of the lower part of the diagram above. It permits a sampling frame to be generated and it provided the material for the statistical analysis of the total population of Phare projects. Lessons learned The SCR evaluation web site notes that in terms of "feedback" the report has been widely distributed including publication of results on the Internet. Beyond this diffusion of results, which is now an official policy of the Commissionxx, it is probably too early to judge the impact of the evaluation on both on-going Phare interventions and future SAPARD type programmes. The absence of rural development as a theme for the in-depth evaluation is obviously a weakness in terms of drawing policy lessons for SAPARD but this merely reflects the priorities of previous Phare interventions rather than a methodological or timing error. Phare agriculture projects appear to score well in terms of relevance suggesting a good level of prior analysis, which may reflect the use of internationally qualified experts in the prior analysis and establishment of objectives. However, the project were often rather poorly designed (a feature they share with many Structural Fund measures) in terms of, for instance, an incoherence between the problems and objectives identified and the activities undertaken so that the application of the logical framework approach at the project design stage does not seem to have significant effects on consistency of projects. The size of the projects funded and evaluated are clearly smaller than average Structural Fund Objective 2 (rural) type programmes. However, the Phare projects are often at least as important in terms of funds as many of the measures in Objective 5b/Objective 2 rural programmes with which they can be more appropriately compared. In the selection of case studies, a certain preference was given to larger project which reflect the likely size of future SAPARD projects. In conclusion, it would appear that the existence of a multi-country monitoring programme (OMAS) and a Commission database of projects does not seem to have alleviated the need for a considerable amount of work on inventorying of the projects before the evaluation could move on to consider relevance, effectiveness etc. in a more in-depth manner. This suggests that even relatively sophisticated monitoring systems have their limits in terms of the policy usefulness of the data produced.



Phare Agriculture Evaluation - Main conclusions and policy recommendations

Conclusions: The report finds that, in general, Phare-financed agricultural programmes were relevant to the needs of the countries concerned and Phare financing was justifiable; but the appropriateness of design was less than acceptable, especially for rural credit programmes. While performance at the project management and implementation stage was of acceptable quality, overall performance was found to be less than acceptable mainly due to low effectiveness, impact and sustainability of project outputs. Rural credit schemes often became vehicles for providing subsidies to farmers, inhibiting the development of credit markets. Marketing projects had a limited impact on increasing market transparency; existing policies sometimes discouraged improvements. Extension projects show a mixed picture. For food safety and disease control projects it was too early to judge impact. Policy projects had little discernible impact on policy. Recommendations : The recommendations apply both to future Phare and Sapard programmes. They include the following: Ø Phare should give a high priority to improved market efficiency and completion of transition and to

adoption of acquis in all areas where this adoption facilitates market efficiency. Ø The right legal, administrative and economic framework should be established before embarking on

highly specific land reform activities and greater consideration should be given to harnessing market forces in tackling problems of fragmentation of holdings;

Ø Phare should not support credit projects where the use of interest subsidies inhibits the development of the market. Project design should address the roots of the problem of availability and accessibility to credit rather than its symptoms;

Ø More emphasis should be given to (a) information dissemination within extension and marketing projects to maximise the potential for replication and (b) the development of systems which reduce reliance on national budgetary support;

Ø Food safety, animal & plant disease control programmes should support the further privatisation of institutions, or some separation of functions from Ministries in favour of more efficient organisational arrangements. Accreditation of food quality control laboratories in the CEECs needs to be speeded up;

Ø Policy-type projects should only be established in Ministries where there is a will to utilise the outputs and to allocate resources to training and retention of local staff;

Ø Grants should not be used to fund activities which the market could support; Ø Phare should ensure that more expertise goes into the design stage of projects. Smaller projects

should not be rejected merely because of higher administrative costs. They tend to be better focused, easier to manage and more successful.



4. Thematic evaluation of RTDI in the Structural Funds (Objective 2 zones) Context and objectives The thematic evaluation of RTDI in the Structural Funds was commissioned by DG Regio evaluation unit in December 1997 and completed in December 1998xxi. The evaluation was conducted in two lots : for the Objective 1 and 6 countriesxxii and for the Objective 2 zones. The discussion below draws on the evaluation carried out for the latter type of zone by a consortium composed of Zenit Gmbh (Germany), ADE S.A. and Enterprise Plc (UK). According to the terms of reference, the purpose of this study was to provide a thorough and systematic analysis of the contribution and impact of the Structural Funds RTD and innovation actions in the Member States and regions from an economic development perspective. The study was also expected to contribute to establishing guidelines for future investments by Structural Funds in this field in the assisted regions, based on the experience of the past and current intervention. Hence there was a clear orientation towards drawing lessons on comparative effectiveness of measures and of providing policy lessons for the future 2000-2006 programming period. The evaluation cons isted of a comparative analysis carried out across 11 Member States and 86 Objective 2 zones eligible during the 1994-99 period. The five key aspects of the comparative analysis are summarised in the table below.

Five Key Aspects of the Comparative Analysis

Theme Focus of analysis RTDI funding Amount and proportion of funds

allocated to RTDI The importance assigned to different RTDI related actions

Breakdown of expenditure on basis of categorisation of types of projects

Quality of RTDI organisation and structures

Evaluation and management structures

The effectiveness of instruments and mechanisms

Appraisal of key types of instruments through case studies

The impact on human resources and training

Analysis of ESF measures in certain case study regions

A series of issues related to the policy impact and the effectiveness of the Structural Fund were also highlighted in terms of the degree to which RTDI capabilities have been promoted; the extent to which industry-university links have been strengthened; competitiveness and economic development enhanced, etc. These were used as sub-themes when selecting good practice examples. Methodological approach and tools; The evaluation took place in two stages: the first essentially desk-research based focused on the funding trends, the importance in the types of actions funded, synthesising available information on the outputs and results, etc.. This analysis was carried out on the basis of the programming documents and available programme level evaluations and other policy or statistical documents. A key tool of the comparative analysis was a categorisation of the



RTDI actions into three groups in terms of operational objectives (enhancement of science; industrial technological development; innovation promotion). In a second phase, the evaluation team carried out case studies in 15 Objective 2 zones in order to offer a more in-depth and qualitative view on the issues related to the quality of programme management and partnerships; the impact (through a survey of SMEs in a number of the case study regions) and the identification of good practice examples. Due to the relative lack of data available at programme level on impact and effectiveness, the latter were used as a proxy for a quantifiable assessment of the relative effectiveness of RTDI actions. The evaluation process for the RTDI Thematic Evaluation A series of methods and tools were developed in order to gather and analyse the data and qualitative information at each stage of the evaluation: • a categorisation of RTDI measures with a view to facilitating the analysis of funding

trends; • a logical framework type table which allowed the structuring of key information on RTDI

measures in each of the SPDs; • a template for the country reports with synthetic financial and analytical tables including

regional innovation profiles; • an SME impact survey questionnaire; • good practice templates used to summarise measures and projects providing a novel

approach to supporting the innovation activities of regional firms or with an identifiable impact on regional competitiveness.

• an interview guide and questionnaire for the case studies. A key aspect of the approach was the categorisation. In the Terms of Reference for the evaluation, the Commission suggested three broad categories of measures: scientific potential; technological potential and support to innovation. In order to establish a more precise definition of the parameters and limits of the analysis, these categories were broken down into a series of "standard RTDI measures" with the aim of:

Jan. 98 Jan. 99

Horizontal theme : synergies with RTD FP /Article 10 ERDF / Community Initiatives

Task

Output

Primary analysisof SPDs/OPs

Discussion withdesk officers

•Characterisestrategies ofOP/SPDS•initial input/output data•refine selectionof case studies

July 98

Field work / surveys•policy makers,•intermediaries,•beneficiary firms

Country Reports

InterimReport

Comparative analysis - thematic approach

FinalSynthesisReport

Trends analysis by category of measures;Identification / analysis of good practiceEstimation of impact by objectiveAnalysis of programme management

Regional case studies



• allowing an adequate level of demarcation between different policy options, instruments

and delivery mechanisms; and • permitting an inter-regional and across country comparison with respect to funding flows. Due to the increasingly all encompassing definition of the term innovation, such a categorisation is complex since there is inevitably overlap with traditional policy domains, notably business support. Moreover, each Member State tends, for historical or institutional reasons, to define the scope and remit of RTDI measures/actions in a different manner (e.g. in Germany, standard investment grants are considered to promote "innovation" in the recipient companies through the acquisition of new equipment, etc.). A further sub-division of measures was operated with respect to the type of investment funded : infrastructure (e.g. a technology centre building); equipment and actions. Limits to the analysis Three main problems were encountered in undertaking the evaluation : • the composite nature of many SPD measures made it difficult to arrive at a precise

estimation of funding levels for RTDI in Objective 2; • the quality and availability of baseline data from monitoring systems and mid-term or on-

going evaluation reports varied widely and in most cases was of poor quality; • it proved extremely difficult to implement the impact survey of SME which led to more

emphasis being placed on "good practice" examples as a proxy for effectiveness. Assessing the level of funding of RTDI measures was a key aspect but one that proved extremely difficult. Problems similar to those encountered in constructing the Phare Agricultural evaluation inventory were encountered when actually matching measures or sub-measures with the categorisation of measures proposed. The categorisation proved difficult to apply due to two main reasons: • firstly, in many SPDs, RTDI actions are included in measures of a generic nature

supporting, in particular, SME development (see boxed example below); • secondly, even where a specific measure contains only RTDI type actions the content is

often defined only at a very aggregate level and during implementation diverse projects may be funded altering considerably the real content and (more critically) the objectives of the measure.

Difficulties in Identifying RTDI content

Measure 1.1 "Support for business initiatives" in the 1994-96 SPD for Alsace gathers under a single measure accounting for 15% of the SPD, actions as different as: • Aids for capital investment in SMEs (which often have a very limited technology and innovation content) • Aids for intangible investment and access to counselling (same remark as above) • Support for exports actions (not relevant for analysis) • Support for joint business initiatives (which might have an innovation or technology transfer content) • Support for acquisition of technologies by SMEs (relevant to analysis) • Support for innovation projects in firms (relevant to analysis) • Recruitment of technicians in SMEs (probably relevant to analysis).



It would have been preferable for the purposes of the classification and the inventory to have been able to separate out specific measures and build up "evaluable measures" but this proved extremely difficult or impossible in many countries. Given the above comments, it was impossible to provide a clear-cut quantitative answer to the question of the share of RTDI within SPDs as well as to attempt to draw inter-country or inter-regional comparisons in terms of the share of RTDI in SPD budgets. This would have required an in-depth analysis at the level of the projects approved in each of the 86 zones. Such an analysis was clearly beyond the remit and budget of the evaluation. On the basis of the data available, the evaluation team finally chose to present a lower (taking into account only measures comprising with a 100% RTDI focus) and an upper range (including measures which had non-RTDI sub-measures and where it was impossible on the basis of the information to separate out actual funding between sub-measures).

RTDI funding levels in Objective 2 programmes The RTDI evaluation concluded that for the 1994-96 programming period, Objective 2 regions planned on average to devote around 12% of their financial means to measures which were exclusively funding RTDI related actionsxxiii. If funding devoted to measures including non-RTDI sub-measures is added the relative weight of RTDI funding in the SPD budgets jumps considerably in certain Member states (a doubling of the overall share of funding devoted to RTDI to 24% across the 11 Member States). This upper end of the range is clearly not realistic either since few informed commentators would accept the argument that a quarter of Objective 2 funding is devoted to RTDI. A weighted average of the amounts allocated to each of the types of measures (pure RTDI, mixed, etc.) produced a figure of 16-17% of Objective 2 funds devoted to RTDI during the 1994-96 programming period.

Despite these caveats the categorisation enabled the evaluation team to ana lyse the evolution of RTDI content using the broad categories. This first attempt at a categorisation of RTDI measures in Europe allowed a significant body of evidence to be constituted concerning planned funding allocations across the 86 Objective 2 zones. This data allowed the testing of a number of postulates such as the existence of a trend over the periods 1989-93, 94-96 and 97-99 from a concentration of resources on the first two categories (science and technology) towards innovation support measures (see table below). Evolution of the share of S,T,I categories in planned RTDI resources (in % of RTDI resources) 1994-96 1997-99 Change

Country S T I S T I S T I Belgium-Luxembourg 12 51 37 17 42 41 Ú Ø Ú Denmark 6 22 72 0 20 80 Ø Ø Ú France - - - - - - - - - Germany 0 43 57 0 32 68 Ù Ø Ú Italy 6 67 27 2 74 24 Ø Ú Ù Netherlands 4 33 63 0 29 71 Ø Ø Ú Spain 6 63 31 8 40 52 Ú Ø Ú UK 4 51 45 3 49 48 Ø Ù Ú

Total for these countries 4 52 44 3 45 52 Ø Ø Ú N.B. Ú=increasing, Ø= decreasing, Ù= stable. The figures can slightly differ from table 4.4. in order to take the same basis for the two periods. For Spain, this table presents only figures for Pais Vasco and La Rioja regions. Austria, Finland and Sweden became Member States in 1995, and are not included. The second difficulty faced by the evaluation team was that across the fifteen case study regions and 11 Member States, the situation in terms of monitoring and evaluation can be, at best, summed up as unsatisfactory. The case studies in particular noted that:



• there is a generalised lack of what can be termed an "evaluation culture" with regions failing to integrate evaluation and monitoring within the programme management cycle;

• monitoring activities concentrate on providing the minimum financial and output indicators required by national and EU authorities;

• RTDI measures in SPDs are, with few exceptions, evaluated within programme wide evaluations;

• very limited impact assessments are made with funding for measures being continued from one programming period to another in spite of lack of verifiable results; and even in spite of recommendations to change orientations in certain cases.

As a general rule, the monitoring systems and annual reports of the Objective 2 programmes are oriented towards the collection of financial and operational information which, for instance in the UK, serves mainly to establish auditing trailsxxiv. There is little or no monitoring of whether RTDI actions implemented are meeting their targets. Moreover, in terms of indicators, output, result and impact targets are often confused. In many cases, impact indicators are simply missing. In others, they are very vague and difficult to measure (“increased competitiveness of SMEs”) and in most cases they do not capture the innovation dimension but reflect rather a job creation objective. In the case study regions, for example, there is very little data available regarding the effects of contracts signed between research laboratories and regional SMEs, although this is often a clear objective in the SPD. This meant that it was extremely hazardous to attempt to draw firm conclusions on effectiveness of actions from a structured analysis of available indicators and in-depth studies. In the case of the SME impact survey, the decision on which measures to assess was for the most part driven by the issue of data availability and access to the firms. Hence, the survey was finally restricted to four Member States (Austria, Denmark, Germany and the UK). In the other Member States either the data situation was too poor to implement the survey (case of France) or the local experts found it too early to assess the impact since most of the projects were still on-going by the time the current study was carried out (typically the case for the Scandinavian countries Finland and Sweden). In Italy the RTDI measures identified as being appropriate for the impact survey could not be used for implementing the survey since in one case the firms were the subject of a questionnaire survey being carried out by the beneficiary and in another case the programme managers indicated the firms were already over-surveyed by academics so that a high level of resistance of the beneficiaries to answer the questionnaire was expected in both cases. The impact surveys carried out in Austria, Denmark, Germany and in the UK covered some 210 firms in six regions. In total, eleven funding schemes are integrated in the survey. In Austria and Denmark only one support scheme was evaluated, whereas in Germany three and in the UK six schemes were covered. Although some conclusions could be drawn from the results, the size of the sample and the selectivity of the measures rather undermines their robustness.



Lessons learned Judging the policy impact of a single evaluation on the programming process of the Structural Funds is a relatively hazardous task. All the more so when the field in question is relatively diffuse and evolving as is the case with innovation. However, according to the 1999 Commission Activity Report on Evaluations (DG Budget), the Commission made use of the results through “the incorporation of key findings into the Structural Fund guidance on priorities for 2000-2006” and for the “preparation of ‘negotiating mandates’ for Objective 2 programmes”. There is some evidence for this in terms of orientations in the guidelines or direct citations in new policy documents such as the Draft Communication from the Commission on "The Regions in the New Economy (Guidelines for innovative measures under the ERDF in the period 2000-2006)". One of the key messages of the thematic evaluation was the weakness of policy design and monitoring/evaluation tools for RTDI measures. The July 1999 Guidelines for programmes in the period 2000-2006xxv stress the importance, alongside three priority actions areas (promoting innovation, networking and industrial co-operation and developing human capabilities), of “consolidating RTD and innovation actions through effective policy management”. The need for improved policy management in terms of the introduction of performance-oriented schemes, the improvement of statistical methods and the use of scoring systems all reflect recommendations made by the evaluation. What is perhaps disappointing is that the same guidelines place RTDI under a broad heading of “Creating the basic conditions for regional competitiveness” (alongside transport infrastructure, energy networks, etc.) rather than under the second heading of “Competitive enterprises for employment creation” where business support services are a priority. The key message of the evaluation that there is a need to coordinate more effectively business support priorities with innovation measures risks to be ignored.

RTDI evaluation Objective 2 zones – Five main messages Improve understanding of the systematic practice of business innovation • Structural Fund partnerships should allocate resources to support creation of “regional innovation

observatories”; • Commission should fund training and assistance to regions on drawing up minimum baseline

indicators; • Member States should act to disaggregate existing statistics on RTDI; • Specific ex-ante appraisals of RTDI priorities to be obligatory - focus on whether actions match

business needs. RTDI Strategic partnerships - a pre-condition for increased funding • Support for RIS type partnerships for all Objective 2 regions - take-up of projects developed

through RIS facilitated by system of regular calls or at mid -term review stage; • Non-territorial nature of innovation should be recognised explicitly - RTDI infrastructure projects

must take account of resources beyond boundaries of Objective 2 zones; • focus on value added of Objective 2 funding with respect to national initiatives - ‘seed-capital’ for

policy innovation. Increase synergies between RTDI, SME and human resources measures • Pertinence of single “Business Innovation” priority should be examined for new programmes; • Complete over-haul of ESF approach required - extend practice of integrated ERDF/ESF measures

- shift content away from generic training; • “slim-down”/focus measures on clear and quantifiable objectives. An increased emphasis on business as the beneficiary • Project development needs to be driven by companies - requires pro-active role of programme

managers / RIS as neutral brokers;



• special delivery mechanisms to make actions more accessible to “non-innovative” SMEs; • focus of selection criteria needs to switch from “eligibility” to compatibility and pertinence with

measure objectives; • room should be made in programming cycle for experimental actions which can be scaled up if

successful. Enhance the capacity to chart the economic effect of RTDI actions • strengthen project level monitoring - and impose periodic impact reviews on “non-stop” projects; • Commission should fund training/help desk service on monitoring and evaluation of RTDI; • Increase EU wide “benchmarking” of project results - mid-term reviews by international experts

should be mandatory; • reinforce publicity requirements and oblige final beneficiaries to disclose minimum level of

information.

5. Conclusions This paper has illustrated through three different examples, the difficulties in undertaking thematic evaluations both at a multi-country level and even within one countryxxvi. These difficulties arise essentially from five main factors: • A lack of forward planning of evaluations that results in a relatively chaotic timetable of

mid-term evaluations, etc. on which thematic evaluations need to draw on; • A lack of “harmonisation” in terms of a core set of indicators for specific themes and

types of measures; • Monitoring systems which track essentially financial data and which do not place an

emphasis on integrating performance targets for measures; • Little or no thematic analysis or in-depth analysis at the level of programmes – specific

topics being subsumed into overall programme evaluations; • A persistent weakness of evaluation culture in many Member states which means that

“national” measures co-financed by the Structural funds are rarely evaluated; These types of conclusions and findings are corroborated by a number of reports and recent communications such as the “Special Report N°15/98 on the assessment of Structural Fund interventions” by the Court of Auditors or the Communication to the Commission from Commissioner Schreyer of July 2000 on “Strengthening Evaluation of Commission activities”. Practical solutions for improving the baseline data and analysis available for thematic evaluations need to be found without increasing the “burden” on programme managers in terms of reporting. On the basis of the cases examined here, investing in large multi-country monitoring databases or systems does not necessarily seem to reduce the need for considerable work on developing an inventory of activities in the first stage of thematic evaluations. A more cost-effective approach would appear to be the development and application of a limited number of common indicators for themes/types of measures allied to a series of standard research questions to be dealt with by all mid-term evaluations. DG Agriculture has adopted the latter approach with respect to the current ex-post evaluations of Objective 5b programmes while the new guidelines for ex-ante evaluations (ERDF, ESF Interreg) place a much greater stress on identifying a limited number of core indicators which can be effectively monitored. While methods and specific tools will vary across programmes this



approach at least offers the perspective to thematic evaluators of finding a more homogenous set of data and analysis. Another proposal would be to extend and reinforce thematic “knowledge networks” bringing together practitioners and evaluators at EU level (an example from the previous period was IQ-NET for a certain number of Objective 2 regions). The work of such networks, eventually supplemented by an extra-net type tool, could be orientated towards developing tools and methods for in-depth thematic analysis in member regions and bringing together practitioners in “benchmarking” seminars to facilitate integration of lessons into decision-making on an on-going basis. The integration of programme managers from applicant countries into such a network would also allow a transfer of “good practice” for the ISPA and SAPARD type instruments. Independent thematic evaluations launched in the second half of the programming period would thus have a much broader basis for establishing robust conclusions on comparative effectiveness and drawing policy lessons. NOTES i Aide à la Décision Economique (ADE) S.A. provides consulting services in the field of evaluation, strategic policy initiatives, programme design and technical assistance. ADE operates at a global level for clients including the European Commission, the World Bank, national overseas development agencies, regional governments and private organisations. Further information on our capacities and activities can be found at : www.ade.be ii See for instance: the Annual Evaluation Review 1999 prepared by DG Budget available at http://europa.eu.int/comm/budget/evaluation/en/bilan/evaluationreview1999.html which lists evaluation reports completed or on-going during 1999. Aside from the clear thematic nature of six of the eight DG REGIO evaluations listed, a series of other evaluations appear to be of the thematic type for instance the evaluation of projects in the fie ld of social exclusion for DG EMPL; or the Evaluation of Phare financed SME support projects by the SCR. iii See for instance http://www.worldbank.org/html/oed/evaluation/html/sector_evaluations.html. A good example is the sector Study No.19061: Nongovernmental Organizations in World Bank-Supported Projects: A Review. iv Chapter 4 of Volume 5 of the MEANS collection suggests a number of approaches such as a scoring chart, case studies, surveys, sampling, etc. v Objective 3 of the Structural Fund concerns “Combating long-term unemployment and facilitating the integration into working life of young people and of persons exposed to exclusion from the labour market; promoting equality of opportunity. A total of 396 million EUR of ESF funding was made available to Objective 3 for the period 1994-99. This amounted to 55% of the total ESF allocation to Belgium. The breakdown by OP was 9.1% for the Federal Ministry; 3.1%for Brussels region, 39.5% for the French speaking community /Wallonia; 46.9% for the Flemish government and 1.4% for the German speaking community. vi Objective 4 concerns actions “Facilitating the adaptation of workers to changes in production systems”. The original CSF for 1994-96 was extended in 1996 until 1999 with a total ESF contribution of EUR 69 million or 9.6% of the total ESF allocation. The breakdown by OP was 8.7% for the Federal Ministry, 3.1% for Brussels, 25.3% for the French-speaking community/Wallonia; 62.2% for the Flemish government and 0.7% for the German speaking community. vii Belgian has one of the lowest rates of adults participating in life-long learning in Europe and a very low rate of investment by firms in training. viii A similar role is attributed by Barbier to the Commission in the case of the French Objective 3 evaluation. See “Inter-governmental Evaluation : Balancing Stakeholders’ Expectations with Enlightenment Objectives”. Barbier Jean-Claud. Evaluation Volume 5 N°4 October 1999. ix A good example is a roughly proportional split of the budget between the first three priorities of the Objective 3 OP for Brussels which in no way reflected the priority needs in terms of focusing attention on those receiving the “minimex” (minimum subsistence payment). External coherence between OPs intervening in the same territory (the three “regional” OPs for Brussels, the Federal OP vis -à-vis the regional Ops) was also extremely



poor with a serious lack of coordination between actions funded under different programmes that together constituted complementary phases of the pathway to integration process. x This conclusion was less strong for Wallonia where little progress was made in the first three years of the OP. Moreover sustainability of this process was relatively fragile with a risk that the public training and employment agencies will become dominant partners at the end of ESF funding in the local consortia created in Flanders. xi Phare was established in 1989 and originally stood for “Poland Hungary Aid for the Reconstruction of the Economy”. However, it was quickly extended, in terms of both countries and budget, and by 1997, 13 CEECs had become eligible for Phare support: Poland, Bulgaria, Hungary, Lithuania, Latvia, Estonia, Czech Republic, Romania, Slovakia and Slovenia, Albania, Bosnia and the former Yugoslav Republic of Macedonia. At the end of 1998, cumulative commitments amounted to some € 8,7 billion, all in the form of non-reimbursable grants, financed from the E.U. budget. From 1998 onwards, Phare programmes are based on Accession Partnerships, which indicate the areas of the ‘Acquis Communautaire’ (the set of E.U. legislation and regulations) where Candiate Member States need to make further progress in order to pave the way for full membership. Phare provides both technical assistance and investment support to help them implement their National Programme for Adoption of the Acquis. From the year 2000 onwards, Phare will be complemented with an opening up of the EU’s Cohesion Fund (ISPA, € 1.5 bln/year) and Agricultural Fund (SAPARD, € 0.5 bln/year) to the candidate member states. The Phare countries that are not candidate member states (Albania, Bosnia and FYROM) will continue to receive the more ‘traditional’ types of Phare assistance, with emphasis on economic and political transition. xii While this is a large amount of assistance in absolute terms, it should be kept in perspective. It is only € 5 per year per worker/farmer in agriculture in the Phare countries, and less than € 1 per year per hectare of agricultural land. By contrast, the annual budgetary cost of the EU’s CAP is over € 4,500 per worker/farmer in agriculture and over € 250 per hectare. The two figures are not directly comparable, of course. One is assisting a sector to become (generally) more efficient and better adapted to a competitive market economy; the other - several hundred times greater - is providing income support and (generally) hindering adaptation to a more competitive environment. The comparison merely shows that the amount involved is very small compared with the EU's annual budget for its own agricultural sector (€ 44,946.1 mln in 1998). It is also small compared with the proposed annual budget of € 520 mln under the Special Accession Progra mme for Agriculture & Rural Development (SAPARD). xiii The full report is available on the SCR site : http://europa.eu.int/comm/scr/evaluation/evinfo/phare/951493_ev.html xiv The average value of the projects in the sampling frame was just over € 1.25 million with a range from € 10 000 to € 23 million. xv These six themes constituted one stratification factor. Two other stratification factors were used. One was the period in which the project was first programmed. Projects in the second half of the nineties are more likely to deal with current problems than the ones at the beginning of the decade when the major problem was assisting the change from central planning. The other stratification factor was the project size in order to ensure that some major projects were included in the sample. The overall effect of stratification is to improve the accuracy of the results compared with a simple random sample. Finally, although no attempt was made to obtain a representative sample by country, it was regarded as necessary to have at least one project for each Phare country (except for the small programme in Slovenia) because Phare assistance is conducted under national programmes and therefore country specific factors influence relevance and results. xvi Projects in the inventory were grouped into nine different 'themes'. The sampling frame excluded the following themes: • the privatisation of non-farm assets because it had been of minor importance under the agricultural

programme (3 per cent) and was likely to become even less important. Phare support for privatisation was usually covered by dedicated Phare privatisation programmes, taking a horizontal rather than sectoral approach to privatisation.

• Rural development. Though it will become extremely important in the future under SAPARD, there were very few projects on this theme (2 per cent of the total expenditure) and most were simply farm diversification projects.

• Critical aid and local project management. These were mostly import programmes to provide urgently needed essential inputs for the agri-food sector in the early years of transition.

xvii OMAS is a Commission funded monitoring project covering all Phare programmes. xviii In the Desiree database the delimitation between the last three categories is very loose. A sub-project corresponds generally to one contract or to the different contracts with the same company relating to the same activity. The grouping of sub-projects into one project does not respond to any identifiable logic but seems rather to depend on administrative criteria.



xix The rules applied were that distinguishable projects would form an evaluable project: (i) when they belonged to a common theme under a given project heading in a given programme (often when subprojects or contracts were complementary to each other in terms of provision of equipment and training), and/or (ii) when they belonged to a common theme with a continuation of an activity between programmes (extension or second or third phase projects). In the case of Poland, where regional PMUs organised their own programmes, some distinguishable projects have been grouped into evaluable projects on the basis of geographical distribution as well as common themes (for example, the extension projects). xx See Summary of Good Practice Guidelines for the Management of the Evaluation Function (SEC(2000)254/4) adopted by the Commission on 23 February 2000. Principle D : As a general rule, evaluation results should be made publicly available. xxi The full report is available at : http://www.inforegio.cec.eu.int/wbdoc/docgener/evaluation/rafon_en.htm xxii A paper by Lena Tsipouri entitled “Lessons from RTDI Enhancement in Less-Favoured Regions” in The IPTS Report n°40 December 1999 provides a summary of the Thematic evaluation of RTDI actions in Objective 1 and 6 regions. The evaluation was carried out by CIRCA (Ireland), University of Athens and PriceWaterhouse-Coopers (the Netherlands). xxiii This average figure is broadly in line with other calculations notably the estimation of DGXII reported in the Second European Report on Science and Technology Indicators (1997) which attributed to RTDI related activities a share of 14.25% of Objective 2 funding during the period 1994-99 (up from 11.5% in 1989-93) and 18.45 % in 1997-99 xxiv Although even the financial monitoring was weak. In France, for instance, " it was often difficult for an external observer to understand the relationships between the various programming documents and the financial tables produced in the regions, and the case was not infrequent that seemingly identical reports contained completely different data for reasons that remained obscure"'. xxv See : http://www.inforegio.cec.eu.int/wbdoc/docoffic/coordfon/coord_en.htm xxvi The Belgium case is probably an extreme but the trend of increasing “federalisation” of power in many Member States is likely to lead to similar situations in other countries (for instance, the almost complete autonomy in terms of economic development power of Scotland within the UK).

What can be learnt from thematic evaluations · 2015-03-09 · What can be learnt from thematic...

Documents

Transcript of What can be learnt from thematic evaluations · 2015-03-09 · What can be learnt from thematic...