EN EN

EUROPEAN COMMISSION

Brussels, 27.1.2021

SWD(2021) 10 final

COMMISSION STAFF WORKING DOCUMENT

EVALUATION

of the 2014-2018 ESF support to employment and labour mobility, social inclusion and

education and training

{SWD(2021) 11 final}

1

Contents

1 INTRODUCTION ................................................................................................................................ 6

2 BACKGROUND TO THE INTERVENTION ..................................................................................... 6

2.1 Policy context .................................................................................................................. 6

2.1.1 ESF objectives and functioning ................................................................................ 8

2.1.2 ESF support to employment and labour mobility (thematic objective 8) ............... 9

2.1.3 ESF support to social inclusion (thematic objective 9) .......................................... 10

2.1.4 ESF support to education and training (thematic objective 10) ........................... 10

2.2 Baseline and points of comparison ............................................................................... 10

2.2.1 Employment .......................................................................................................... 11

2.2.2 Labour mobility ..................................................................................................... 13

2.2.3 Social inclusion ...................................................................................................... 14

2.2.4 Education and training .......................................................................................... 16

3 IMPLEMENTATION / STATE OF PLAY ........................................................................................ 17

3.1 Programming and budget ............................................................................................. 17

3.2 Employment and labour mobility .................................................................................. 18

3.3 Social inclusion .............................................................................................................. 19

3.4 Education and training .................................................................................................. 20

3.5 Typology of Operations funded ..................................................................................... 22

3.5.1 ESF support to employment and labour mobility ................................................. 22

3.5.2 Social inclusion ...................................................................................................... 23

3.5.3 Education and training .......................................................................................... 24

3.6 Financial and operational implementation ................................................................... 26

4 METHOD............................................................................................................................................ 27

4.1 Short description of methodology ................................................................................ 27

4.2 Limitations and robustness of findings ......................................................................... 27

5 ANALYSIS AND ANSWERS TO THE EVALUATION QUESTIONS .......................................... 28

5.1 Effectiveness .................................................................................................................. 28

5.1.1 Financial and output achievements ...................................................................... 28

5.1.2 Findings on results and contribution to overarching objectives ........................... 32

5.1.3 Effects on participants ........................................................................................... 36

5.1.4 Factors hindering or promoting effectiveness ...................................................... 38

5.2 Efficiency ....................................................................................................................... 40

2

5.2.1 Assessment of efficiency ....................................................................................... 40

5.2.2 Factors hindering or promoting efficiency ............................................................ 41

5.2.3 Visibility of ESF interventions ................................................................................ 42

5.3 Relevance ...................................................................................................................... 43

5.3.1 Alignment with needs ........................................................................................... 43

5.3.2 Observed factors that improve relevance ............................................................. 47

5.3.3 Flexibility in a changing context ............................................................................ 49

5.4 Coherence ..................................................................................................................... 50

5.4.1 ESF Internal coherence .......................................................................................... 50

5.4.2 ESF external coherence ......................................................................................... 53

5.5 EU added value .............................................................................................................. 58

5.5.1 Volume effects ...................................................................................................... 58

5.5.2 Scope effects ......................................................................................................... 60

5.5.3 Role effects ............................................................................................................ 62

5.5.4 Process effects including sustainability ................................................................. 63

6 CONCLUSIONS ................................................................................................................................. 65

6.1 Lessons learned ............................................................................................................. 71

ANNEX 1: PROCEDURAL INFORMATION ............................................................................................ 73

ANNEX 2: STAKEHOLDER CONSULTATION ....................................................................................... 76

ANNEX 3: METHODS AND ANALYTICAL MODELS ........................................................................ 105

ANNEX 4: ESF INTERVENTION LOGIC ............................................................................................... 114

List of figures

Figure 1. Allocations to thematic objectives 8, 9, and 10 (ESI Funds plus national co-financing) -

EUR millions ..................................................................................................................................... 7

Figure 2. Europe 2020 - Distance to targets for headline indicators ............................................ 11

Figure 3. Employment rate as function of age in 2014 and in 2018 ............................................. 12

Figure 4. Proportion (in %) of the EU population at risk of poverty or social exclusion ............... 15

Figure 5. Total ESF and YEI budget per thematic objective - EUR millions ................................... 18

Figure 6. TO8 - Indicative allocations per investment priority - in EUR millions ........................... 18

Figure 7. TO9 - Indicative allocation by investment priority - in EUR millions .............................. 20

Figure 8. TO10 - Indicative allocations by investment priority - in EUR millions .......................... 21

Figure 9. Distribution of thematic objective 10 eligible costs by area of operation ..................... 25

Figure 10. Number of participations by selected characteristic and thematic objective ............. 31

Figure 11. How would you qualify the following administrative arrangements for the

implementation of operations? .................................................................................................... 39

3

LIST OF TABLES

Table 1.. Types of operations per investment priority – TO8 ....................................................... 23

Table 2. Types of operations per investment priority - TO9 ......................................................... 24

Table 3. Project selection rates ..................................................................................................... 29

Table 4. Absorption rates .............................................................................................................. 30

Table 5. Participant profiles .......................................................................................................... 30

Table 6. Achievement rates for specific output indicators ........................................................... 32

Table 7. Reported results (common result indicators only; upon leaving and after six months) . 33

Table 8. Cost per participation ...................................................................................................... 41

LIST OF BOXES

Box 1. Identifying the target group needs for social inclusion operations ................................... 44

Box 2. Ensuring the relevance of early leavers from education and training operations through

risk assessment ............................................................................................................................. 45

Box 3. Integrated approaches for ensuring relevance of support to employment ...................... 48

Box 4. Examples of flexibility ......................................................................................................... 49

Box 5. Preliminary reflections on the ESF’s possible role in addressing COVID 19 crisis .............. 50

Box 6. Types of ESF added value ................................................................................................... 58

Box 7. Significant volume effects of ESF support to education and training ................................ 60

Box 8. Social inclusion scope effects: broadening actions to more groups .................................. 61

Box 9. Examples of TO8 role effects .............................................................................................. 62

4

Glossary

Term or acronym Meaning or definition

AIR Annual Implementation Report

Beneficiary Public or private body responsible for initiating or both initiating

and implementing operations

CPR Common Provisions Regulation

CSR Country-specific recommendations

DG EMPL Directorate-General Employment, Social Affairs & Inclusion

DG REGIO Directorate-General Regional and Urban Policy

EAFRD European Agricultural Fund for Rural Development

ELET Early leavers from education and training

EMFF European Maritime and Fisheries Fund

ERDF European Regional Development Fund

ESF European Social Fund

FEAD Fund for European Aid to the Most Deprived

GDP Gross domestic product

ISCED International Standard Classification of Education

JRC Joint Research Centre

MS Member State

IP Investment priority

NEET Person not in employment, education or training

NGO Non-Governmental Organisation

OP Person not in employment, education or training

Operation Operation means a project, contract, action or group of projects

selected by the managing authorities of the programmes

concerned, or under their responsibility, that contributes to the

objectives of a priority or priorities

Participant A person benefiting directly from an ESF intervention who can be

identified and asked for their characteristics, and for whom

specific expenditure is earmarked

5

Term or acronym Meaning or definition

SFC System for Fund Management in the European Union

SWD Staff working document

TO Thematic objective

VET Vocational education and training

YEI Youth Employment Initiative

6

1 INTRODUCTION

This staff working document (SWD) presents results of the evaluation of the European Social

Fund (ESF) support to policies in the areas of employment and labour mobility (thematic

objective 8), social inclusion (thematic objective 9), and education and training (thematic

objective 10).

The conclusions of the evaluation are relevant for the implementation of the final stages of the

2014-2020 ESF investments and the preparation of the 2021-2027 programming period of

European Social Fund Plus (ESF+) by providing lessons, notably as regards cost-effectiveness,

outreach and target groups most in need. The evaluation can support the negotiation of the

ESF+ programmes and the design of operations by the Member States in the respective areas. In

addition, it paves the way for the ex-post evaluation of the ESF.

The evaluation assesses the effectiveness, efficiency, relevance, coherence and the EU added-

value including the sustainability of activities funded by the ESF in the period 2014-2018. It also

addresses the complementarity and coherence with other relevant initiatives during the period

concerned. It covers all Member States during that period (i.e. including the United Kingdom). It

builds on the results of three separate studies, one for each thematic objective. The ESF and

Youth Employment Initiative (YEI) support to youth employment, (part of thematic objective 8),

was the subject of another thematic evaluation1. ESF support to Institutional capacity building

(thematic objective 11) has been covered by a study2.

The onset of the COVID-19 pandemic happened when the field work for the supporting studies

was completed or about to be completed. Also the time frame covered, 2014-2018, excludes

the pandemic and the subsequent crisis. Nevertheless, a number of aspects of the lessons

learned take on a particular relevance (see Box 5).

2 BACKGROUND TO THE INTERVENTION

2.1 Policy context

The ESF is one of the five European Structural and Investment (ESI) Funds. These are EU funding

instruments providing structural and investment financing across the EU3. Their total budget in

2014-2020, to be invested in 11 areas called ‘thematic objectives’ (TO) as defined in the

1 The staff working document is available at:

https://ec.europa.eu/social/main.jsp?langId=en&catId=89&newsId=9793&furtherNews=yes

2 The report is available at: https://op.europa.eu/en/publication-detail/-/publication/c7c56421-9fc4-

11ea-9d2d-01aa75ed71a1

3 The other funds are the European Regional Development Fund (ERDF), the Cohesion Fund (CF), the

European Agricultural Rural Development Fund (EARDF) and the European Maritime and Fisheries

Fund (EMFF).

7

Common Provisions Regulation (CPR)4, is EUR 461 billion. The funds allocated for thematic

objectives 8, 9 and 10 amount to EUR 173 billion (Figure 1).

Figure 1. Allocations to thematic objectives 8, 9, and 10 (ESI Funds plus national co-financing) - EUR millions

Source: https://cohesiondata.ec.europa.eu

While the five funds address different stakeholders and sectors, they share two important

features:

• They support growth and job creation, are aligned with the Europe 2020 strategy and contribute to the implementation of relevant country specific recommendations (CSR) 5;

• They are delivered under shared management6.

The Europe 2020 strategy7 emphasises smart, sustainable and inclusive growth as a way to

overcome the structural weaknesses in Europe’s economy, improve its competitiveness and

productivity and underpin a sustainable social market economy. The ESF should contribute to

4 Regulation (EU) 1303/2013 of the European Parliament and Council laying down common provisions

on the ERDF, ESF, the Cohesion Fund, the EAFRD and the EMFF or European Structural and

Investment Funds (ESIF Funds). https://eur-lex.europa.eu/legal-

content/EN/TXT/?uri=celex%3A32013R1303

5 More information on the European Semester is available at: https://ec.europa.eu/info/business-

economy-euro/economic-and-fiscal-policy-coordination/eu-economic-governance-monitoring-

prevention-correction/european-semester/framework/eus-economic-governance-explained_en

6 For an explanation of shared management, see the section ESF objectives and functioning.

7 https://ec.europa.eu/eurostat/statistics-explained/index.php?title=Glossary:EU_2020_Strategy

8

achieving the headline targets of the Europe 2020 strategy on employment, education and

poverty reduction8.

The Communication ‘Investing in jobs and growth – maximising the contribution of European

Structural and Investment Funds’9 provides an overview of what the ESI Funds are expected to

achieve in 2014-2020.

In November 2017, the Commission, the European Parliament and the Council proclaimed the

European Pillar of Social Rights10 which is about delivering new and more effective rights for

citizens. It builds upon 20 key principles, structured around three chapters:

• Equal opportunities and access to the labour market; • Fair working conditions; • Social protection and inclusion.

The Pillar relies on previous policy guidance available at EU level, which oriented the use of

resources provided by ESI Funds (ESIF).

2.1.1 ESF objectives and functioning

The ESF contributes to four of the 11 thematic objectives defined by the Common Provision

Regulation11:

• Thematic objective 8: Promoting sustainable and quality employment and supporting labour mobility;

• Thematic objective 9: Promoting social inclusion, combating poverty and any discrimination;

• Thematic objective 10: Investing in education and training and vocational training for skills and lifelong learning;

• Thematic objective 11: Enhancing institutional capacity of public authorities and stakeholders and efficient public administration.12

In each Member State, the partnership agreement (PA)13 describes the investment framework

for the ESI Funds in that Member State. The partnership agreements are aligned with Europe

8 Article 4.1. of the ESF Regulation: (Regulation (EU) No 1304/2013 of the European Parliament and of

the Council of 17 December 2013 on the European Social Fund and repealing Council Regulation

(EC) No 1081/2006, OJ L 347, 20.12.2013, https://eur-lex.europa.eu/legal-

content/en/TXT/?uri=CELEX%3A32013R1304

9 https://eur-lex.europa.eu/legal-content/EN/TXT/?uri=CELEX%3A52015DC0639

10 https://ec.europa.eu/commission/priorities/deeper-and-fairer-economic-and-monetary-union/european-

pillar-social-rights/european-pillar-social-rights-20-principles_en

11 The scope of ESF assistance for the 2014-2020 programming period is defined by Article 3 of the ESF

Regulation No 1304/2013

12 Not covered in this evaluation, as it was recently the subject of a separate study. See:

https://op.europa.eu/en/publication-detail/-/publication/c7c56421-9fc4-11ea-9d2d-

01aa75ed71a1/language-en

13 http://ec.europa.eu/contracts_grants/agreements/index_en.htm

9

2020, the EU strategy for smart, sustainable and inclusive growth, taking into account where

appropriate the relevant country specific recommendations14.

The actual implementation is framed by programmes agreed between the Commission and the

Member States, known as operational programmes for cohesion policy funds15. Besides the EU

funding, operational programmes also receive national funding, public or private. The total

budget for ESF operations for 2014-2020 is EUR 121 billion, with an EU contribution of EUR 84

billion16. National co-funding ranges from about 20% to over 50%, and depends on the

development of regions, with less developed regions expected to contribute less than more

developed ones.

Operational programmes are structured by priority axes, each composed of one or more

investment priorities setting at least one specific objective. Priority axes may combine one or

more complementary investment priorities from the ESF, the European Regional Development

Fund (ERDF) and the Cohesion Fund. They may have a national or regional dimension. In total,

187 operational programmes received ESF funding in the 2014-2020 programming period.

Ninety-two operational programmes are multi-fund meaning that they also receive ERDF or

Cohesion Fund funding.

Under shared management the actual day-to-day management of operational programmes is

entrusted to national bodies, working in accordance with rules established at Union level. The

Commission supervises implementation and retains ultimate responsibility for the execution of

the EU budget. The operational programmes fund projects or operations run by a range of

public and private organisations called beneficiaries. These projects benefit their participants –

for the ESF, these are usually individuals, but they can also be companies or organisations. The

selection of projects, their monitoring, their control and the reporting are done by the national

bodies.

The overall intervention logic of the ESF is given in Annex 4.

2.1.2 ESF support to employment and labour mobility (thematic objective 8)

Thematic objective 8 promotes sustainable, quality employment by boosting job creation,

helping people into employment, and supporting labour mobility. The investment priorities

supported within TO8 can be separated into three broad categories:

• investment priorities directly supporting access to employment

14

https://ec.europa.eu/info/publications/partnership-agreements-european-structural-and-investment-

funds_en

15 Besides the ESF, the cohesion policy funds include the European Regional Development Fund

(ERDF) and the Cohesion Fund (CF). The 2014 - 2020 7-year European budget set aside EUR352

billion for cohesion policy. More than half of this – EUR182.2 billion – was set aside for less

developed regions. These are regions with a per capita GDP less than 75 % of the EU-27 average.

EUR35 billion was allocated to transition regions, which are those with a per capita GDP between 75

% and 90 % of the EU average, and EUR54 billion to more developed regions which are those with a

per capita GDP of more than 90 % of the EU average.

16 https://ec.europa.eu/regional_policy/en/policy/how/is-my-region-covered/

10

o 8i Access to employment for job-seekers and inactive people, including the long-term unemployed and people far from the labour market, also through local employment initiatives and support for labour mobility;

o 8iv Equality between men and women in all areas including in access to employment, career progression, reconciliation of work and private life and promotion of equal pay for equal work;

o 8vi Active and healthy ageing

It also includes 8ii (sustainable integration into the labour market of young people)

which has been covered by another evaluation exercise17.

• investment priorities supporting businesses o 8iii Self-employment, entrepreneurship and business creation; o 8v Adaptation of workers, enterprises and entrepreneurs to change.

• investment priority supporting institutions o 8vii Modernisation of labour market institutions.

2.1.3 ESF support to social inclusion (thematic objective 9)

Thematic objective 9 promotes social inclusion and combats poverty and any discrimination.

The six investment priorities supported under TO9 tend to be characterised by a focus on a

specific target group (marginalised communities - 9ii) or an issue (discrimination and equal

opportunities - 9iii, access to services - 9iv and the social economy - 9v). Investment priority 9vi

is about the instrument chosen to deliver policy (community-led local development strategies).

Investment priority 9i (active inclusion) is very broadly defined. In a Member State, a minimum

of 20% of the ESF allocation has to be allocated to TO9 (social inclusion). This is the only

thematic objective with such earmarking.

2.1.4 ESF support to education and training (thematic objective 10)

Thematic objective 10 focuses on ‘investing in education, training and vocational training for

skills and lifelong learning’, delivered through four interlinked investment priorities, which cover

all stages of education and target a wide range of potential groups:

• 10i Reducing and preventing early school leaving and promoting equal access to education, including formal, non-formal and informal learning pathways for reintegrating into education and training;

• 10ii Improving the quality and efficiency of, and access to, tertiary and equivalent education and training;

• 10iii Enhancing equal access to lifelong learning for all age groups in formal, non-formal and informal settings; and

• 10iv Improving the labour market relevance of education and training systems.

2.2 Baseline and points of comparison

Four of the headline targets of the Europe 2020 strategy for jobs and smart, sustainable and

inclusive growth18, are directly relevant to the ESF scope of action. They represent the labour

17

https://ec.europa.eu/social/main.jsp?langId=en&catId=89&newsId=9793&furtherNews=yes

18 For more information on the Europe 2020 strategy please see:

https://ec.europa.eu/eurostat/web/europe-2020-indicators

11

market, social inclusion and education and training dimensions of the strategy. Figure 2 below

shows the distance to the EU target for these four headline indicators. By the end of 2018, the

EU seemed to be on track for the targets on the employment, early leavers from education and

training and tertiary educational attainment. However, the target of lifting 20 million people out

of poverty or social exclusion seemed out of reach. There are however, large differences

between and even within Member States. The COVID 19 pandemic is already having an impact

on these indicators.

Figure 2. Europe 2020 - Distance to targets for headline indicators

Source: Eurostat: https://ec.europa.eu/eurostat/web/europe-2020-indicators/europe-2020-

strategy/main-tables

Abbreviations: lhs: left hand scale; rhs: right hand scale.

2.2.1 Employment

One of the headline targets of the Europe 2020 strategy adopted in 2010 by the European

Council is to increase the employment rate of the population aged 20-64 to at least 75% by

2020. In addition, Member States set national targets for the employment rate in 2020.

Between 2014 and 2018, the EU employment rate grew by 4.0 percentage points to 73.2%,

surpassing the pre-crisis 2008 rate of 70.2% in 2015. However, large differences in employment

rate remained between Member States and also between gender and age groups.

Employment rates (in 2018) are below 70% in Belgium, Greece, Spain, Croatia, Italy and

Romania but close to 80% or above in Czechia, Germany, Estonia, the Netherlands and Sweden.

Employment rates, in Greece (in particular), Cyprus and Spain (marginally) were still below pre-

crisis levels.

The employment rate gap between men and women is almost 12 percentage points. The main

reason behind the employment gap between women and men is the high share of women in

12

inactivity (in 2018 27.6% vs. 15.7% of men). In 2014, nearly a third of inactive women aged 20-

64 (29.4%) were not seeking employment due to family and caring responsibilities.

The employment rate is correlated with age. The highest rate is noted for the 40-44 age group

(83.1% in 2018). The employment rate drops to 71.8% for the 55-59 age group and to only

44.4% for the 60-64 age group. Nevertheless, the employment rate of the older workers has

been increasing more rapidly than the overall employment rate by 7.8% for the 60-64 age group

compared to a 4.0% overall increase from 2014 to 2018 (see Figure 3). Active ageing, i.e

increasing the proportion of older workers still in employment remains key to achieving the

overall employment target.

Figure 3. Employment rate as function of age in 2014 and in 2018

Source: Eurostat: lfsa_ergan

In 2013 when the unemployment rate in the EU reached its peak (10.8%), about 20.5 million

people in the 25-64 age group were unemployed. In 2018, though the unemployment rate

decreased to 6.8% (below the pre-crisis level of 7.0%), there were still 16.9 million people

unemployed. At the same time, the share of long-term unemployed among unemployed,

though considerably down from its peak in 2014 (38.5%) was still above the pre-crisis level

(35.0%; 33.4% in 2008).19

Between 2014 and 2018, as the economic situation improved, youth unemployment rates fell in

all Member States and by an average of 7 percentage points at EU level. However, not all

Member States saw the same degree of improvement. Over that period, the rate of young

people not in employment, education or training fell from 12.5% in 2014 to 10.5% in 2018.

More importantly, it fell in all countries but Germany and Denmark, where it only marginally

19

Eurostat: lfsa_upgan

13

increased. The improvement derives entirely from the fall in the number of unemployed young

people (down 3 percentage points) as the number of inactive young people slightly increased

(up 1.2 percentage points). The composition of the target population is thus shifting, with a

higher share of inactive young people (58% in 2018). For young migrants (15-24 years), the

unemployment rate fell from 29.2% in 2014 to 20.9% in 2018.

In 2014, 29.9 million people aged 25-64 were self-employed which corresponds to 15.3% of

those employed. By 2018 the share of self-employed dropped by 1 percentage point to 14.4%

which is 29.6 million people aged 25-64 years. Thus, at EU level the number of self-employed

people has hardly changed since 2014, but since the total number of employed has risen

significantly the share of self-employment has gone down.

It is broadly acknowledged that the success of EU businesses and industry, coupled with a fair

and balanced socio-economic legal framework, is vital to increasing employment and creating

new and better jobs. In 2014, there were 23.3 million SMEs in the EU-28’s non-financial

business economy accounting for more than 99% of all enterprises. The overwhelming majority

of these (92.5% of all enterprises) were enterprises employing fewer than 10 people. By 2016

(the most recent year for which there are complete available data for the EU as whole), the

number of SMEs increased to 24.7 million (still accounting for more than 99% of all enterprises

in the EU’s non-financial business economy) employing 94.9 million people.

2.2.2 Labour mobility

Labour mobility has both a geographical (place of employment) and an occupational dimension

(type of work performed). Free movement of workers (geographical mobility) is one of the

central tenets of EU policies. Labour mobility in the European Union is increasing. One measure

of geographical mobility is the share of non-nationals in total employment20.

Between 2014 and 2018, the share of non-nationals in total employment increased from 7.1%

to 8.3%21. There are very big differences between Member States. In Luxembourg, over half

the employed are non-nationals while in Bulgaria, Croatia, Lithuania, Poland, Slovakia and

Romania, less than 1% of the employed persons are non-nationals. Other aspects of

geographical mobility include cross-border work and commuting between regions. Occupational

mobility is more difficult to define and measure but also seems to be increasing. Job tenure

tends to become shorter while the age at which people leave the labour market is increasing. In

the EU28, the share of long job tenures (60 months or more) declined from 62.9% in 2014 to

60.8% in 201822. The EU28 average duration of the working life has increased by nearly one full

year between 2014 (35.3 years) and 2018 (36.2 years)23.

20

For more statistics on mobility see:

https://ec.europa.eu/eurostat/cache/digpub/eumove/index.html?lang=en

21 Eurostat: lfsa_egan

22 Eurostat: lfsa_egad

23 Eurostat: lfsi_dwl_a

14

2.2.3 Social inclusion

The EU generally saw an improvement in living standards in 2014-2018. This improvement is

attributed in part to increases in real median income and household incomes, as well as

improvements in economic activity and the labour market. Gross disposable household income

has increased in real terms since 2012-2013 in nearly all Member States, although in some this

has not yet returned to pre-crisis levels (notably some southern Member States)24.

Despite these positive developments, as well as the Europe 2020 target of lifting 20 million

people out of poverty, over one fifth of the EU population remains at risk of poverty or social

exclusion. The Europe 2020 target was set before the 2008 economic and financial crisis, which

had a detrimental impact on the EU’s ability to reach this target. The highest rates are found in

the southern and eastern regions of the EU. The proportion of people suffering from severe

material and social deprivation declined between 2014 and 2018 from 8.9% to 5.9% in the EU-

2825. The proportion of people in the EU-28 experiencing severe housing deprivation slightly

declined from 5% to 4% in the same period. Those who were disproportionately affected by this

condition included persons at risk of poverty, tenants, households with dependent children,

people in rural areas and Roma.

24

Employment and Social Developments in Europe – Annual Review 2018. Available at:

https://ec.europa.eu/social/main.jsp?catId=738&langId=en&pubId=8110

25 Eurostat: ilc_mdsd07

15

Figure 4. Proportion (in %) of the EU population at risk of poverty or social exclusion

Source: Eurostat, People at risk of poverty or social exclusion by age and sex [ilc_peps01.

Notes: The middle line is the EU28 average; the outer lines represent the maximum and minimum

observed at the level of the Member States.

On access to services, the proportion of people in the EU-28 self-reporting unmet needs for

medical examination declined from 6.7% to 3.6% between 2014 and 2018. However, strong

inequalities persist across certain groups of the population, with people in lower income groups,

the unemployed, people with low education and people living in rural areas more likely to

report unmet needs for medical examination. Across the EU-28, the reason most often cited is

that it is too expensive, followed by long waiting lists. Although in 2016, 86.3% of EU children

from 3 up to the minimum compulsory school age received formal early childhood education

and care, this varied widely between countries, from 51.3% in Croatia to 98.6% in Belgium26.

Although employment is generally seen as a route out of poverty, in 2018 the EU average in-

work poverty rate was 9.5%, unchanged from 2014. Groups more at risk of in-work poverty are

people in households with low work intensity, single parents with dependent children, people

with low education, migrants, people with a disability and young people. Regional disparities in

unemployment and the prevalence of in-work poverty persisted. Although unemployment

rates declined between 2014 and 2018, strong regional disparities persisted, notably in Belgium,

Bulgaria, France and Italy.

26

Eurostat: ilc_caindformal

16

2.2.4 Education and training

The rate of early leavers from education and training (ELET), one of the Europe 2020 headline

targets, fell from 11.9% in 2013 to 10.6% in 201827 although it has still not reached the EU

2020 target (below 10% of 18-24 year olds) and rates of ELET vary significantly across Member

States. A total of 17 Member States were below the overall EU target of 10% in 2018. 16 of

them had already achieved it in 2013, but reductions can be observed in almost all Member

States. However, despite some improvement in the Member States that had a particularly

unsatisfactory performance at the start of the period (Spain, Italy, Malta, Portugal and United

Kingdom), there are still Member States that are not reaching the EU target. These include

those mentioned above as well as Bulgaria, Denmark, Germany, Estonia, Hungary and Romania.

Tertiary education attainment, another Europe 2020 headline target, experienced steady

growth during the programming period, from 37.1% in 2013 to 40.7% in 2018. Approximately

two-thirds of the Member States (18 of 28) had already achieved the EU target by 2018 (40% of

30-34 year olds have completed tertiary or equivalent education), up from 16 Member States

that had achieved it in 2013. All countries made significant progress, most notably some Eastern

European Member States. There are still several countries where further efforts are needed to

achieve their national targets, however, including Bulgaria, Germany, Croatia, Hungary, Malta,

Portugal and Romania. Moreover, significant gender imbalances remain in tertiary education

enrolment in specific subjects; women are underrepresented in STEM (science, technology,

engineering and mathematics) disciplines (e.g. in ICT, female enrolment is 18% v. 82% for men,

and in engineering, manufacturing and construction it is 26.1% vs 73.9%)28.

Regional disparities in ELET and tertiary education attainment remain significant across the

EU. Overall, 60 EU regions of 281 considered in the analysis and for which data are available,

have improved their relative performance in ELET during the current programming period. 170

EU regions show relative stability while ELET performance deteriorated over the period in 51.

The challenge of reducing and preventing early leavers from education and training is still

particularly substantial in several Mediterranean and Eastern European regions. As regards

tertiary education attainment, 40 regions improved their relative performance in 2013 – 2018,

184 regions remained relatively stable and the performance of 52 regions decreased. There are

significant remaining challenges to improving the quality and efficiency of, and access to,

tertiary education in several central and eastern European regions, in Italy’s south and in several

Eastern German regions.

The participation of adults in education and training increased over time, reaching 10.1% for

men and 12.1% for women in 2018. The participation of employed, unemployed and inactive

persons in education and training recovered after the economic crisis and increased during the

current programming period but, in this case too, there are significant disparities between

countries and more than half of the Member States have a participation rate which is below the

EU average.

27

The share of early school leavers is systematically higher among males and this gender disparity is not

showing any signs of reduction over time.

28 The main driver being possibly the persistence of man/women stereotypes in education and socio-

cultural environment.

17

Overall, the number of Vocational Education and Training (VET) students has been declining in

the EU over the decade, with some exceptions such as an increase of upper secondary level

students in Estonia, Cyprus29, Luxembourg, Malta, Slovenia and the United Kingdom; post-

secondary level students in Czechia and France and short-cycle tertiary level students in

Sweden.

The share of students enrolled in vocational education are highest in post-secondary (91.5% in

2017) and short-cycle tertiary levels (86.4%). In lower secondary education, the share of VET

students is low (7.4%) while in upper secondary education there is a substantial equilibrium

between students in general and vocational education.

Educational outcomes on basic skills (reading, mathematics and science), according to the

OECD Programme for International Student Assessment (PISA), also vary significantly across

countries and regions, and these disparities persist, with some Member States recording a

performance that is consistently below the average. Furthermore, no improvement can be

observed in PISA 2018, in the sense that the EU has not made any progress towards the ET

202030 benchmark on the share of low-achievers.

Finally, the employment rate of recent graduates increased during the 2014-2018 evaluation

period, although has not fully recovered from the economic crisis. The challenges include a

persistent gender gap in the employment rate of recent graduates, which has not changed

substantially during the current programming period, driven by the advantage that men

continue to have in entering the labour market. In 2018, for upper secondary and post-

secondary non-tertiary education graduates, the employment rate for men was 76.7% vs 71.4%

for women. The gender gap was therefore 5.3 percentage points. For recent tertiary education

graduates, the employment rate was 83.9% for women and 87.4% for men (gap of 3.5

percentage points in 2018).

3 IMPLEMENTATION / STATE OF PLAY

3.1 Programming and budget

About EUR 130 billion are available for ESF and YEI operations in the 2014 – 2020 programming

period. The EU contribution corresponds to 71% of total planned expenditure. National public

and private contributions make up the balance31. The YEI is implemented through investment

priority 8ii under thematic objective 8. Figure 5, below, shows the budgets per thematic

objective.

29

Although from a very low baseline rate.

30 The strategic framework for European cooperation in education and training (ET 2020).

https://ec.europa.eu/education/policies/european-policy-cooperation/et2020-framework_en

31 Source: Open data portal: https://cohesiondata.ec.europa.eu/

18

Figure 5. Total ESF and YEI budget per thematic objective - EUR millions

Source: https://cohesiondata.ec.europa.eu

3.2 Employment and labour mobility

ESF support to employment and labour mobility is supported specifically (excluding youth

employment support) through 151 operational programmes, in all Member States. The total

allocation of these programmes, excluding 8ii32, is EUR 32.1 billion, which is the equivalent of

26% of the total ESF budget for 2014-2020. This underlines that Member States consider

support for employment and labour mobility as a central objective within the ESF. Figure 6

shows the indicative allocations per investment priority33.

Figure 6. TO8 - Indicative allocations per investment priority - in EUR millions

32

Investment priority 8ii corresponds to youth employment. The ESF support to Youth Employment

staff working document can be found at

https://ec.europa.eu/social/main.jsp?langId=en&catId=89&newsId=9793&furtherNews=yes

33 While mobility is mentioned explicitly in the title of TO8, there is no specific investment priority

dedicated to labour mobility. In theory, Member States can programme operations towards mobility

under any investment priority and it can mean different things as explained in the text above.

19

Source: https://cohesiondata.ec.europa.eu

Access to employment (8i) is the dominant investment priority, absorbing more than half of the

entire budget allocated to TO8 (53%) and programmed by all Member States except Denmark34.

Adaptability to change (8v) is another substantial priority, corresponding to EUR 7.3 billion

(23%). It is particularly important in Germany (EUR 2.2 billion) and France (EUR 1.3 billion),

where it represents over half of the total TO8 investments.

The ESF TO8 actions in transition regions is comparatively less focused on access to employment

(44%; EU average 53%). These regions spend more on entrepreneurship (17%) and adaptability

(33%). Programmes in less developed regions concentrate comparatively less on adaptability

(16%; 23%), to favour access to employment.

3.3 Social inclusion

In the current programming period TO9 operations are financed by 145 operational programs

financing covering all Member States and regions. They account for a total planned expenditure

(including EU and national co-financing) of approximately EUR 31.3 billion (of which EUR 21.4

billion EU funding). This corresponds to about one quarter of total ESF funding which is more

than the required earmarking. The absolute amounts planned vary considerably across Member

States, from EUR 8 million in Luxembourg to EUR 4.0 billion in Germany. The total amounts

planned for TO9 as percentage of total ESF does range from 20% in Finland to over 70% in the

Netherlands. These budgets have remained relatively stable in 2014-2018, with a slight overall

increase of less than 1%. The biggest increases in absolute terms were in Greece. Figure 7Figure

34

Note that youth employment (formally also addressed by thematic objective 8) is excluded from these

figures, since this is the subject of a separate evaluation.

20

7. TO9 - Indicative allocation by investment priority - in EUR millions shows the indicative

allocation per investment priority.

Figure 7. TO9 - Indicative allocation by investment priority - in EUR millions

Source: https://cohesiondata.ec.europa.eu

Similar amounts are allocated to less developed regions (43%) and to more developed regions

(42%). The balance (15%) goes to transition regions. Looking at investment priorities (Figure 7),

around 73% (EUR 22.1 billion) is allocated to 9i (active inclusion). The next most important

investment priority in terms of indicative allocation is 9iv (access to services) at 17% (EUR 4.8

billion). The remaining investment priorities represent between 1% and 6% of total funding

allocated to TO9. A similar picture emerges at MS level. Generally, the bulk of the allocations is

to 9i (active inclusion). In eleven MS (Belgium, Denmark, Germany, Ireland, France,

Luxembourg, Austria, Finland, Sweden and United Kingdom), 9i even amounts to 90% or more

of the total TO9. Only eight MS allocate the biggest share of the total to a priority other than 9i

(9ii – Hungary; 9iii – Cyprus; 9iv – Bulgaria, Estonia, Croatia, Latvia and Romania).

3.4 Education and training

Total planned expenditure on TO10 in the current programming period is approximately EUR

39.2 billion (including EU and national co-financing). This corresponds to nearly one third of

total ESF (32%). The amounts planned vary considerably across Member States, from EUR 5

472.8 million in Italy to EUR 8 million in Luxembourg and zero in the Netherlands. The total

amounts planned on TO10 as a percentage of total ESF range from a maximum of 53% in

Portugal, to a minimum of 15% in Cyprus, ignoring the Netherlands.

Planned expenditure on TO10 relating to public spending on education is significant in several

EU countries. For example, total TO10 is 7.1% of general government expenditure on

21

education35 in Portugal, 4.2% in Romania and 4.1% in Poland. TO10 is also important in the

Baltic countries, Bulgaria, Greece, Croatia, Hungary and, Slovakia.

More than half of the total TO10 funds, 51.4%, goes to less developed regions, 34.8% to more

developed regions and 13.7% to transition regions. Looking at investment priorities (see Figure

8), around 29% (EUR 711.24billion) is planned on 10i (early leavers from education and training),

15% (6 billion) on 10ii (tertiary education), 30% (11.8 billion) on 10iii (lifelong learning) and 26%

(10.19 billion) on 10iv (labour market relevance).

Figure 8. TO10 - Indicative allocations by investment priority - in EUR millions

Source: https://cohesiondata.ec.europa.eu

Several Member States have selected all investment priorities, but only a few (such as Portugal)

have a fairly equal distribution of funds across all. Many Member States decided to concentrate

on a limited number. For instance, France focuses on 10iii and 10i, the United Kingdom on 10iii

and 10iv, Belgium, Slovenia and Sweden on 10iii, and Czechia on 10i and 10ii.

Between 2016 and 2018 there were changes in ESF resources planned under TO10. Overall, in

the EU, there was a 0.1% increase in total planned resources (EU and national co-financing).

Looking at the ESF component only, the positive change between 2016 and 2018 was

EUR +262.6 million (+1%). However, considering that the total net increase (EU and national co-

financing) was much smaller and amounted approximately to EUR 55 million between 2016 and

2018, the observed positive change in planned ESF (i.e. EUR 262.2 million) was clearly

compensated by a reduction in national resources.

35

Annual average for 2014-2018.

22

In absolute values, the biggest increases (considering both EU and national co-financing) took

place in Spain (EUR +300.7 million) and Greece (EUR +182.3 million) while the largest reductions

were recorded in the United Kingdom (EUR -322 million) and Italy (EUR -116.4 million) where,

however, there was a 32.6 million increase in EU resources planned under TO1036.

3.5 Typology of Operations funded

3.5.1 ESF support to employment and labour mobility

The supporting study conducted a mapping of the types of operations. ESF TO8 operations

combining types of operations are the most common and represent 18% of the costs. Support

for entrepreneurs is also around 18%, closely followed by guidance and support (12%),

adaptability (14%), and financial incentives (10%).

Of investments in the area of entrepreneurship (8iii), 99% consist of operations that provide

some sort of support for entrepreneurs. Likewise, adaptability operations are basically only

found within the adaptability investment priority (8v), where these operations represent half of

the reported eligible costs. Operations in support of active ageing dominate the active ageing

investment priority (66% of that investment priority’s costs), while operations that seek to

strengthen institutional capacity represent 84% of the costs reported for the investment

priority on labour market institutions (8vii). For access to employment, a variety of types of

operation can be found, such as guidance and support (20%), financial incentives (17%),

education and training (8%), and a considerable share of operations that combine multiple

types (30%).

36

In percentage terms, the most significant changes which can be observed between 2016 and 2018 are

those concerning Cyprus, where there was a 34.6% reduction, and to a lesser extent Denmark (-8.9%)

and the UK (-8.6%). Conversely, in Greece there was a 14.9% increase and in Spain a 10.5% increase.

Overall, the distribution of funds across IPs did not vary considerably.

23

Table 1. Types of operations per investment priority – TO8

Type of operation

Programmed under

Total TO8

(excl. 8ii)

Access to

employ-

ment (8i)

Entre-

preneurship

(8iii)

Gender

equality

(8iv)

Adaptability

(8v)

Active

ageing

(8vi)

LM insti-

tutions (8vii)

Work-based learning 4% 5% 7%

Education & training 7% 8% 10% 9% 8%

Guidance & support 12% 20% 7%

Financial incentives 10% 17% 3% 2%

Support Entrepreneur 18% 3% 99% 3% 7%

Institutional capacity 4% 4% 1% 84%

Women in employment 5% 1% 80% 1%

Adaptability 14% 1% 55% 1%

Active ageing 2% 2% 66%

Integrated pathways 6% 10% 2% 3% 3%

Other

Combined 18% 30% 2% 5% 33% 5%

Total 100% 100% 100%37

100% 100% 100% 100%

Source: mapping by authors, based on qualitative screening of the annual implementation reports 2015-

2018.

Mapping of all operations under TO8 shows that in practice almost all mobility operations can

be found under the ‘Access to employment’ (8i), and to a lesser extent in support to labour

market institutions (8vii). Investments targeting labour market institutions are relatively small,

with an overall investment of EUR 1.3 billion. These investments include support for labour

market institutions to improve their provision of labour mobility services (e.g. through EURES).

3.5.2 Social inclusion

Social Inclusion operations are complex, highly diverse and delivered to a range of vulnerable

groups as well as entities. There is significant overlap between investment priorities under

TO9. The definition of 9i is broad and actions may more easily fit within it than under other

more narrowly defined investment priorities such as 9v. As a result, most operational

programmes covered 9i.

The supporting study on social inclusion38 identifies six types of operations on the basis of the

objective actually pursued and the target groups aimed at. Four types of operations encompass

actions directed to people, while the two remaining types comprise actions directed to people

and to organisations or society at large (e.g. capacity building of social services, support to social

enterprises, information campaigns). Table 2 below presents the six types of operations and

their prevalence across the operational programmes, including TO9 actions.

37

All other types of operations that support entrepreneurship amount cumulatively to 1%.

38 https://ec.europa.eu/social/main.jsp?catId=738&langId=en&pubId=8350&furtherPubs=yes

24

Table 2. Types of operations per investment priority - TO9

Type of operation Main target

groups

Number of

operational

programmes

programmed under

9i I9ii 9iii 9iv 9v 9vi

1 Employment-

focused actions

people with a disability

LTU39

136 95% 12% 6% 1% 9%

2 Enhance basic skills

LTU

people with a

disability

Low skilled

29 69% 17% 14%

3 Basic school

education

people with a

disability

LTU

Recipients of minimum

income

25 32% 48% 20%

4 Access to services people with a

disability

LTU

97 54% 22% 21% 68% 1% 2%

5 Social

entrepreneurship

people with a disability

SMEs, social enterprises

75 25% 67% 12%

6

Actions influencing

attitudes and

systems

79 47% 19% 16% 41% 25% 9%

Source: mapping by authors, based on qualitative screening of the annual implementation reports 2015-

2018.

This pattern of prevalence of investment priorities and types of operations across the

operational programmes is also reflected in terms of finances. Investment priority 9i accounts

for 71% of indicative TO9 allocations, 73% of the planned allocation and 80% of the declared

expenditure. This concentration on one investment priority is less marked at the level of the

types of operations. The operations aiming to move participants in employment (type 1) are the

most important. They represent 45% of the planned allocations and 48% of the declared

expenditure.

Investment priority 9vi (community-led local development, CLLD) is a very specific case. CLLD is

an instrument to involve local private and public actors and to propose integrated, local

solutions. In total, 14 Member States plan the use of CLLD, combining ESF and other ESI Funds

resources, mostly EARDF40. Investment priority 9vi accounts for 2.4% of the indicative TO9

allocations, 1% of the planned allocations and only 0.2% of the declared expenditure.

3.5.3 Education and training

The analysis of the operations financed by ESF under TO10 highlighted that secondary and post-

secondary education and training is the most important area of intervention in financial terms.

Overall, 28.3% of total eligible costs are related to operations supporting vocational secondary

education and training, and a similar share, 27.4% of the total supports general secondary

education and training.

39

Long-term unemployed

40 See paper presented at the 2019 Leader conference.

https://leaderconference2019.minhaterra.pt/rwst/files/I119-CLLDX231219XSERVILLOXKAH.PDF

25

As regards the other areas of intervention, 21.2% is allocated to non-formal job-related

education and training and 4.3% to non-formal non job-related education and training; 15.4%

of the total went to tertiary education and 3.5% is allocated, altogether, to pre-primary and

primary education (see Figure 9 below).

The distribution of funds by area of operation varies considerably between Member States. For

example, more than 50% of the funds are allocated to vocational secondary and post-secondary

education and training in Cyprus, Austria and Portugal while the focus is on general (lower and

upper) secondary education and training in Bulgaria, Estonia, Spain, Italy, Latvia, Slovenia and

Romania. Non-formal job-related education and training prevail in Belgium, Denmark, Ireland,

Slovenia, Finland and the United Kingdom, while non-formal non job-related education has an

important role in Ireland. Tertiary education is central in Croatia, Lithuania, Malta and Poland,

while pre-primary and primary education are significant in Czechia.

Figure 9. Distribution of thematic objective 10 eligible costs by area of operation

Source: mapping by authors, based on qualitative screening of the annual implementation reports 2015-

2018.

The distribution of funds by area of operation varies also across investment priorities. Under

10i, 68.2% of total resources are allocated to support general secondary and post-secondary

education and training. 20.1% of total allocations go to vocational secondary and post-

secondary education and training and 8.6% to primary education. Under 10ii, most resources,

86.7%, go to tertiary education operations, followed by non-formal job-related education and

training (9.1% of the total).

Under 10iii, funds are concentrated on non-formal job-related education and training (53.8%),

while 31% go altogether to vocational, general secondary, post-secondary education and

training, and 13% to non-formal non job-related education and training. As regards 10iv, funds

are mostly used for vocational secondary and post-secondary education and training (72.1% of

total). 14.6% is used for non-formal job-related education and training, and 4.9% for tertiary

education.

26

3.6 Financial and operational implementation

The performance review foreseen by the Common Provisions Regulation was carried out in

2019. It examined the achievement of the milestones for financial and output indicators as

reported in the 2018 annual implementation reports submitted by Member States in 201941.

Across all thematic objectives, ESF performing priorities represented 83% of the performance

reserve for the ESF (i.e. EUR 4.2 billion) that is definitively allocated on the basis of Commission

decisions on the performance review. However, there are important differences between

Member States and programmes. Nine Member States managed to achieve all their ESF

milestones. On the contrary, in some Member States, most priority axes have not achieved their

milestones. This is the case in Romania, for example.

There were also large differences between thematic objectives. In absolute terms, the highest

amount to reallocate concerned TO9 social inclusion, TO10 education and training and multi-TO

priorities42. TO8 employment performed well both in absolute and relative terms with an

achievement rate of around 90%.

At the end of 2018, the number of non-performing priority axes was high, about one fifth, and

far beyond what was expected in light of the programme amendments adopted in previous

years. It demonstrates, inter-alia, that the setting of milestones and the selection of the

indicators are very challenging. The European Court of Auditors found that for around one

quarter of the indicators included in the performance framework of the examined operational

programmes (28 %) it was not possible to draw conclusions about the accuracy of the target

value for the milestone, either because the calculation methodology could not be verified or

because there was no detailed explanation of the unit costs or the assumptions made. It

considered that around 18 % of the remaining milestones and target values are not realistic and

achievable or insufficiently ambitious. Taken together this means that in around 46 % doubts

remain as to the milestones specified for releasing the performance reserve43.

Financial implementation as measured by the ratio of interim payments to the Member States

in relation to the total allocation is running slower than at a comparable time in the previous

programming period. In May 2013, this ratio was approaching 50%. It is just over one third in

May 2020. The main reasons for this delay in implementation include the late adoption of the

Regulations, and a more relaxed automatic de-commitment rule44 (a move from a combination

of N+2 and N+3 in 2007-2013, to a general N+3 for all Member States in 2014-2020).

41

All figures in this section cover the whole ESF and the YEI.

42 However, in relative terms, compared to the total allocation by TO, the most significant short-coming

concerns TO11 institutional capacity. This thematic objective is not covered by this staff working

document.

43 ECA special report No 15. Available at:

https://www.eca.europa.eu/Lists/ECADocuments/SR17_15/SR_PARTNERSHIP_EN.pdf

44 The funds allocated have to be spent within 2 or 3 years of their allocation, failing that they are not

available anymore,

27

4 METHOD

4.1 Short description of methodology

Three separate supporting studies were contracted out (one corresponding to each thematic

objective)45. The evaluation questions specific to each supporting study, derived from the better

regulation evaluation criteria, were broken down into a number of sub-questions for each

thematic objective and agreed with the corresponding inter-service groups. The studies applied

an appropriate mix of evaluation methods. This includes desk research, public consultations,

case studies, modelling, interviews with stakeholders and Member States’ officials and focus

groups. They are explained in further detail in Annex C to this staff working document. The

results of the three twelve-week public consultations can be found in annex 2.

4.2 Limitations and robustness of findings

The findings and conclusions are reasonably robust, since they are based on sufficient

triangulation of sources and tools, such as the data reported by the monitoring systems,

stakeholder consultation, case studies…. However, the evaluation also faced several limitations,

notably:

• Data quality and timeliness. The figures in this SWD are based on the situation as reported by the Member States in the annual implementation reports for 2018. The cut-off date for all data is thus 31 December 2018. During the process of examination and acceptance of the annual implementation reports, corrections to the data may be made. This explains small differences in figures in the supporting studies as the data were extracted from the electronic data exchange platform at different times.

• Detailed programme information is usually not available. The ESF is funding thousands of projects or operations, sometimes even very small ones, each having their own logic, implementation partners and procedures.

• Demarcation of operations focused on employment and labour mobility from those focused on youth employment (covered by a separate SWD46). Although youth employment operations are mainly funded under 8ii, they can be found across other investment priorities.

• Public consultations which by nature cannot be considered as representative by definition, although some mitigation measures have been applied, notably triangulation with other data sources.

45

The final reports of the three studies are available at:

TO8 – Employment and labour mobility:

https://ec.europa.eu/social/main.jsp?catId=738&langId=en&pubId=8330&furtherPubs=yes

TO9 – Social Inclusion:

https://ec.europa.eu/social/main.jsp?catId=738&langId=en&pubId=8350&furtherPubs=yes

TO10 –Education and training:

https://ec.europa.eu/social/main.jsp?catId=738&langId=en&pubId=8346&furtherPubs=yes 46

See Youth Employment staff working document:

https://ec.europa.eu/social/main.jsp?langId=en&catId=89&newsId=9793&furtherNews=yes

28

• Towards the end of the field work, holding focus groups in the Member States and at EU level became impossible due to the COVID-19 pandemic. Instead EU level Delphi surveys47 were held instead.

• Data for comparative analysis of the socio-economic context, notably SILC data, were sometimes only available till 2017.

• Cost effectiveness analyses, including the attempts to conduct cost benefit analyses, are affected by timeliness and completeness of data, due to delayed and non-simultaneous reporting on outputs, results and costs.

• Counterfactual impact evaluations48 were used only for a small number of operations49. The main limitation is that the results cannot be extrapolated to the whole of the ESF. They also inform about the effect of treatment single intervention, not about the ESF as such.

• Rhomolo analysis, like all general equilibrium models, need to be based on a number of simplifying assumptions, although it is a high-quality and mature model very well-suited for estimating macroeconomic impacts.

• COVID-19 pandemic. The fieldwork for the studies supporting this SWD was carried out before the COVID-19 pandemic arrived in Europe. This is not a limitation to assess the 2014-2018 period as such. It does present challenges to assess the relevance of ESF support in the current context.

Annex 3 to this SWD explains in more detail how these limitations were dealt with.

5 ANALYSIS AND ANSWERS TO THE EVALUATION QUESTIONS

5.1 Effectiveness

5.1.1 Financial and output achievements

At the end of 2018, financial execution50 was 7% lower than at a comparable moment in the

previous programming period (20% versus 27% at the end of 2011). However, this was due to

expenditure declarations (from final beneficiaries to Member States authorities and from

Member States to the Commission) rather than commitments or project selection on the

ground, where levels were comparable to those in 2007-201351. The delay in terms of claims

submitted by MS is only 4.24% (accumulated between 2014 and 2017). In fact, proportionally

47

The Delphi survey is a technique that relies on successive rounds of interactions with a panel of

experts. The experts answer questionnaires in two or more rounds. After each round, a facilitator or

change agent provides an anonymised summary of the experts' reactions from the previous round as

well as the reasons they provided for their judgments. The objective is to build consensus on a number

of issues. The method was originally developed to produce forecasts.

48 These methods estimate the net impact of a treatment by comparing the situation for the participants of

an intervention with that of non-participants.

49 Pilot and feasibility study on the sustainability and effectiveness of results for European Social Fund

participants using counterfactual impact evaluations. Available at: https://op.europa.eu/en/publication-

detail/-/publication/84cc9eb9-b33d-11e9-9d01-01aa75ed71a1

50 Financial execution is the ratio of interim payments made by the Commission to Member States to the

total amount allocated.

51 By July 2020 (latest data available) the overall ESF/YEI project selection rate was 92% of total

programmes allocation.

29

more claims were received in 2018 (13.2%) for the 2014-20 period than in 2011 (12.92%) for the

2007-13 period.

Financial implementation was, on average, comparable across the three thematic objectives.

There was however, considerable variation between Member States, regions, operational

programmes, investment priorities and actions types. Other things being equal, ESF

implementation tended to be slower in less developed regions, a notable exception being the

TO10 (education and training) operations in Portugal.

Project selection rates52 up to 2018 were similar, 72% for TO8 (employment and mobility), 71%

for TO9 (social inclusion) and 69% for TO10 (education and training) (see Table 3).

Table 3. Project selection rates

Project selection rate

(in %)

TO8 (employment and

labour mobility)

TO9 (social inclusion) TO10 (education and

training)

EU average 72 71 69

Range per investment

priority

38-77 26-77 64-70

Range per Member

State

58-111 48-114 26-144

Source: https://cohesiondata.ec.europa.eu

These similar average rates hide considerable differences across Member States and investment

priorities. Globally, the differences between thematic objectives (TOs) within Members States

were limited. The major exceptions were Bulgaria, Slovakia and Romania. Six Member States

(Ireland, Cyprus, Luxembourg, Hungary, Malta and the Netherlands) had basically already

engaged the full allocated amounts by the end of 2018 across all thematic objectives. In the

other Member States, there was a clear difference between thematic objectives. For instance,

Bulgaria and Slovakia had fully allocated the planned amounts for TO8 but only a third of the

amount for TO10, with TO9 being in between. The project selection rates were generally higher

for TO8. In Italy, Poland, Portugal and Finland, the highest selection rates were for TO10. The

highest projection selection rates in Germany, Austria and Romania were for TO9.

The absorption rates, defined as the proportion of expenditure claimed by beneficiaries to

allocated resources, were still relatively low (see Table 4). In three Member States, the

absorption was very low, less than 10 % for TO9 (Ireland) and TO8 (Romania), a TO10 (Slovakia,

Romania).

52

This is the ratio of funds allocated by the managing authority to specific projects or operations to the

total amount available (by TO) or indicatively planned (by IP).

30

Table 4. Absorption rates

Declared expenditure

rate

(in %)

TO8 (employment and

mobility)

TO9 (social inclusion) TO10 (education and

training)

EU average 33 28 27

Range per investment

priority

16-41 3-32 21-32

Range per Member

State

7-53 5-68 6-45

Source: https://cohesiondata.ec.europa.eu

The beginning of the programming period coincided with the aftermaths of the economic and

financial crisis. Across all thematic objectives, the focus of implementation has been on tackling

urgent challenges related to the employment crisis, partly at the expense of more structural

longer-term policies. The econometric analysis carried out for TO8 (employment and mobility)

indicates that the reduction of unemployment rates in certain countries/regions was associated

with lower financial progress, further stressing the fact that in several Member States the

improvement of labour market conditions resulted in a reduction of the pace of financial

implementation.

A person can participate more in than one in ESF project. Therefore the number reported is of

participations, not participants. In total, there were 22.2 million participations over the 2014-

2018 period (see Table 5). The differences in participant profiles across thematic objectives is

largely in line with what can be expected. TO10 (education and training) had a higher proportion

of inactive young persons. TO8 (employment and mobility) focused on the unemployed. The

highest proportions of people with a disability, people with a migrant background or minorities

and people with other disadvantages were recorded for TO9 (social inclusion).

Table 5. Participant profiles

53

The differences can be attributed to IP8ii (youth employment) and TO 11 (institutional capacity)

which are not covered by this staff working document.

TO8 (IP8ii

excluded) TO9 TO10

Total

ESF/YEI53

Participations (in millions) 6,9 6,2 9,1 26,3

of which

Employed 25% 15% 25% 20%

Unemployed 68% 53% 10% 44%

Long term unemployed 23% 24% 3% 16%

Inactive 7% 31% 66% 36%

Women 54% 52% 51% 52%

Young people (16-24 years) 15% 22% 69% 43%

Older workers (55-64 years) 11% 11% 4% 7%

Low skilled (ISCED 1-2) 33% 54% 61% 48%

31

Source: https://cohesiondata.ec.europa.eu

The collection of some types of data has proven problematic. The data marked ** in Table 5 are

in special categories defined under Article 8 of Directive 95/46/EC. Data on sensitive

characteristics were not collected consistently in some Member States (for instance Portugal

collected data on disability only in operations targeted to disabled people). The data on

homelessness and rural areas were estimated based on sampling and reported only once.

Figure 10. Number of participations by selected characteristic and thematic objective

Source: https://cohesiondata.ec.europa.eu

Access to employment operations (investment priority 8i) significantly over performed other

investment priorities under thematic objective 8 (employment and mobility). The employment-

focused operations under thematic objective 9 (social inclusion) received the bulk of the

available financing and had the highest success rates (calculated as the proportion of

participants for whom positive results were recorded). Based on progress measured against

common output indicators, activities under investment priority 10i (early leavers from

education and training and access to learning pathways) and 10iii (lifelong learning) have

Migrants, people with a foreign background,

minorities** 13% 28% 9% 15%

People with disabilities** 7% 16% 3% 7%

Other disadvantages** 11% 37% 11% 17%

Homeless or affected by housing exclusion 1% 4% 0% 1%

From rural areas 23% 16% 23% 21%

32

generally performed better than activities under the other thematic objective 10 (education and

training) investment priorities.

Regarding TO8, progress towards result targets set to be achieved by 2023 was relatively in line

with the progress towards output targets. Operations related to access to employment made

more progress towards their result targets than other investment priorities. By the end of 2018,

an average of 55% of all result targets have been achieved at EU level (42% if weighted by actual

size of the targets). Progress in transition regions (38%) was considerably lower than in less

developed (56%) and more developed regions (61%). Progress in Cyprus and Sweden was

somewhat of an outlier, due to possibly incorrect target settings. Progress towards the result

targets was relatively modest in Bulgaria, Ireland, Croatia and Malta, and has not yet been

reported for Romania. Overall, adaptability investments showed a similar progress towards the

targets, though it was also influenced by a number of outliers (Bulgaria, Czechia, Hungary).

Progress towards the result targets for investment in entrepreneurship, gender equality, active

ageing and labour market institutions was slightly lower, varying between 30% and 40%. While

slightly lower than for other investment priorities, these figures are within the acceptable limits.

For TO9 (social inclusion), most targets were set against programme specific indicators. The

average achievement rate for specific output indicators54 was 99%, meaning that on average,

the targets set for the end of the programming period (2023) were already achieved by the end

of 2018 (see Table 6). The achievement rates for specific output indicators under TO10 were a

bit lower. There was also less variation between investment priorities and Member States

indicating more even implementation.

Table 6. Achievement rates for specific output indicators

SOI achievement rates

(in % of 2023 targets)

TO8 (employment and

labour mobility)

excluding IP8ii

TO9 (social inclusion) TO10 (education and

training)

EU average 59 99 79

Range per investment

priority

24%-90 19 -118 44-87

Range per Member

State

2%-279 1 -540 3-108

Source: https://cohesiondata.ec.europa.eu

However, progress towards set targets, including performance framework milestones, may not

be a very reliable indication of implementation performance. Targets were often set far too low,

making it easy to overshoot them sometimes even by multiples. On the other hand some

achievement rates were very low. The most likely explanations are that: (i) targets were not

realistic, (ii) implementation had barely started when progress was measured or (iii) reporting

was not reliable.

5.1.2 Findings on results and contribution to overarching objectives

There has been good progress on results achieved from participation in ESF funded

operations, with a total of 9.5 million positive results by December 2018 (this includes

54

This is the ratio of achievements to targets for specific output indicators with targets.

33

participants who engaged in job searching or found a job, were in employment including self-

employment; entered education/training, or gained a qualification (see Table 7).

Table 7. Reported results (common result indicators only; upon leaving and after six months)

(Figures in thousands) TO8 (IP8ii

excluded) TO9 TO10

Total

(ESF & YEI)

Results upon leaving

Inactive participants engaged in job search 31 309 155 862

Participants in education or training 185 289 792 1 695

Participants gaining a qualification 697 365 2 222 3 949

Participants in employment 1 389 697 210 3 153

Disabled participants with positive result 611 1 202 630 2 950

Total immediate common result indicators 2 303 1 659 3 380 9 660

Results at six months - - - -

Participants in employment 1 595 1 118 338 4 105

Participants with improved labour market

situation 269 167 208 734

Older workers in employment 106 62 10 213

Disadvantaged in employment, in education or

training or gaining a qualification 316 722 84 1 489

Total results at six months 1 864 1 284 546 4 839

Source: https://cohesiondata.ec.europa.eu

It is impossible to disentangle the impact of ESF funding from other factors. However, evidence

shows that the actual contribution of ESF operations towards the EU 2020 targets is clearly

positive, both from a micro and a macro perspective. This assessment is based on results

achievements, qualitative evidence from stakeholder views and micro and macro econometric

modelling.

As regards employment and labour mobility support (TO8) the most outstanding result was the

1.6 million participants in employment after 6 months, contributing to the labour market

integration of the unemployed. In terms of investment priorities, access to employment (8i)

operations made more progress towards their result targets than others, also according to

stakeholders, followed by support to self-employment/entrepreneurship (8iii). This confirms

that in the first years of ESF implementation, most of the focus was on fighting high

unemployment levels and less attention was paid to more structural long-term policies.

Evidence of the effects of ESF support shows that these have been mostly positive, though

typically stronger for people furthest from the labour market and for women. These

quantitative findings are underpinned qualitatively by stakeholders’ views, including on soft

outcomes (e.g. increased self-confidence, motivation, active engagement and cooperation). In

addition, effects are generally more positive and stronger in regions that: (i) are net receivers of

EU support, (ii) have a larger labour supply and (iii) have a stronger export orientation. Many of

these are in Belgium, Spain, Italy, Poland, Portugal, Slovenia and Slovakia, as well as, to some

extent, the United Kingdom.

34

ESF support to social inclusion has contributed to progress made towards achieving of Europe

2020 target of lifting 20 million people out of poverty55. Although the extent to which TO9

operations contributed to this progress cannot be directly assessed, evidence on the scale and

type of results generated by ESF support to TO9 suggests that the contribution was positive. In

total, more than 3 million positive results were reported in terms of engagement in job search,

participation in education and training and accessing employment including self-employment. In

terms of engagement in employment, the most successful operations were Type 1

(employment-focused actions). In addition to reflecting a high degree of effectiveness, these

achievements may also indicate some degree of 'creaming' effects whereby participants in

these operations were very close to the labour market from the start.

On the other hand, although the proportion of recorded results for enhanced basic skills and

basic school (Types 2 and 3 operations) was low, their perceived effectiveness is high. A high

proportion of respondents to the public consultation noted that basic skills training (90%) and

training and education (89%) were mostly useful or very useful in the promotion of social

inclusion and in combating poverty and discrimination. Respondents also perceived the

effectiveness of support to overcoming barriers to job search actions as high (85%)56.

ESF support to education and training resulted in notable contributions to progress towards

achieving the Europe 2020 education and training targets. Based on the available data, over

4.34 million participants (46% of total) achieved positive results from participation in ESF

support to education and training (50% of participants with such results are women). The

majority of positive results related to gaining a qualification upon leaving (3.3 million) or

engaging in education and training. Stronger results success is seen for early leavers from

education and training (10i) and lifelong learning (10iii), in line with the finding that Member

States have been effective in their use of ESF funds to support national and regional strategic

priorities in these areas. However, many Member States have faced challenges related to

achieving positive results under ESF tertiary education (10ii) operations. The country reviews

suggested that only Bulgaria, Hungary, Ireland and Portugal were on course to achieve their

output and result targets under this investment priority.57 Particular achievements are

highlighted in the country reviews and case studies (see them in the TO10 supporting study).

Further evidence is provided by stakeholder views, which consider that ESF operations have

played a key role in helping the EU reach its headline targets in education, contributing

particularly to the drop in the proportion of early leavers from education and training and the

rise in participation in adult learning. For instance, managing authorities intermediate bodies

and public respondents to the public consultation considered that the ESF’s contribution was

55

The number of people at risk of poverty or social exclusion (AROPE) in the EU decreased from about

122 million in 2014 to 110 million in 2018 – a decrease of about 12 million.

56 The total number of responses to these questions was 486. There was little difference between

stakeholder types.

57 Since 2014 there has been a general improvement in the proportion of the population completing

tertiary education (especially in some eastern European countries such as Czechia and Slovakia).

However there is limited evidence from the evaluation that links general improvements in tertiary

education participation with the role of ESF.

35

either ‘very effective’ or ‘mostly effective’ in supporting the development of work-based

learning systems and improving access to lifelong learning opportunities (69.2%; N=104). The

responses to the open-ended public consultation questions also suggested that the focus on

prevention and early intervention was beneficial in reducing school drop-out rates.58

Moreover, there are also some examples of positive impacts on early leavers from education

and training and lifelong learning based on counterfactual evidence to show what would have

happened without intervention (for example in Spain and the United Kingdom)59. However

limited impact studies have been completed to date.

At this stage of the programming period, the ability of quantitative analysis to show the overall

effects of ESF operations on reaching Europe 2020 targets is limited. However, it is already clear

that ESF support for the development of system level changes and sustainable innovative

measures can have a significant long-term impact. Qualitative evidence (country-based

analyses including case studies) suggests that ESF support to social inclusion also has broader

level impacts including improved access to public services, deinstitutionalisation, and cross-

sectoral collaboration to promote innovative approaches. The ESF’s TO10 operations on

education contribute to reach the education and training targets by supporting systemic

change. This is highlighted by examples of operations that have led to systemic changes in

government policy approaches to education and training, showing greater potential under 10i

(early leavers from education and training) and 10iii (lifelong learning)60.

Finally, the modelling of ESF impacts based on RHOMOLO simulations61 estimates that

operations under the three thematic objectives generate positive macroeconomic impacts on

the economy as a whole, on top of the direct results for participants in ESF operations62. For

TO8, investment in employment and mobility up to 2018 would translate into the creation of 47

000 jobs in the long term and an increase in GDP of 0.06% compared to the baseline (2014). This

would be due to final macroeconomic impacts generated by human capital investments and

the improvement on labour productivity, as well as the structural changes involved. ESF

support to social inclusion is expected to increase the EU GDP by 0.037% (which amounts to

58

A minority of stakeholders replied in the public consultation that the ESF support have not been

effective at all in reducing early school leaving: 1% of managing authority respondents (1 of 104), 2%

of beneficiary organisation respondents (5 of 227) and 5% of certifying, coordination and monitoring

body respondents (2 of 38). 59

See in TO10 supporting study the counterfactual positive results in both cases: contribution to reduce

early school leavers in Galicia (Spain) operational programme and lifelong learning in West Wales and

the Valleys operational programme (UK) – Traineeship programme. 60

See section “5.5.5 Process effects including sustainability” for a more detailed discussion of systemic

change.

61 RHOMOLO is the Commision’s spatial computable general equilibrium model, developed by the Joint

Research Centre. See for details: https://ec.europa.eu/jrc/en/rhomolo

62 These results describe only the macroeconomic potential impacts of ESF as it has evolved up to 2018

and are subject to change as the programmes continue. Therefore, in order to ensure the robustness of

results presented in this exercise, an accurate, systematic analysis with respect to the evolution of the

ESF is required for future ex-post evaluation. The results included do not cover the indirect impacts

that the programme has had throughout society (an analysis of these would probably be nearly

impossible anyway, due to its complexity). However, it is clear that the positive effects of the ESF

operations extend beyond what is known and can be quantified.

36

EUR 4 billion) by 2023 and create an additional 127 000 jobs, whereas education and training

operations are expected to add 0.16% to GDP (EUR 18 billion of euro) by 2023 compared to the

baseline, and create around 170 000 additional jobs. All increases are expected to be long-term

(until 2033), as GDP and employment are expected to still be higher relative to the baseline. At

the same time, labour supply, i.e. the number of individuals either working or looking for a job,

should increase as well (moving from inactivity to activity), in particular as a result of ESF

investments in employment and social inclusion (TO8 and TO9).

Moreover, at the regional level, a considerable number of regions seem to benefit significantly

from ESF support, in particular less developed regions and those with less favourable conditions

in terms of employment, social development and human capital and education. In some

regions, these positive effects can be quite significant, given the size of the policy.

In the context of the recent COVID-19 pandemic all results should be considered with due

caution. The model assumes that there are no other adverse or positive shocks affecting the

economy other than the ESF. However, COVID-19 is already having a very strong impact on

observed GDP and employment rates. Poverty and inequality are likely to rise, with the elderly,

women and young people with low skills likely to be affected the most. On the other hand,

RHOMOLO estimations show however that the expected post-COVID-19 macroeconomic

outcomes would be worse in the absence of ESF support63.

5.1.3 Effects on participants

There is evidence that ESF operations have a sustainable positive impact on participants. These

impacts primarily relate to gains in terms of employment, education and qualifications, both in

the immediate and longer-term (see Table 7). Although a wide range of 'soft' outcomes (such as

soft skills or increased self-confidence) was also identified notably for social inclusion

operations, these were recorded in only a few instances, as they are not part of the common

indicators.

The figures in Table 7 should be interpreted with considerable caution as there is significant

under reporting, as explained in chapter 3 - Implementation / State of play.

Evidence from the ESF implementation of employment related operations under TO8

(employment and labour mobility) and TO9 (social inclusion), is in line with meta-analyses of

active labour market policies64, responses received from the public consultations, and in the

case of TO8 (employment and mobility) of counterfactual evaluations conducted.

63

For the purpose of this exercise, note that Rhomolo simulation results focus more on the potential

long-run impact of ESF support. The policy’s impact is estimated based on investments that took

place up to 2018, as is detailed in the econometric analysis enclosed with this staff working document.

64 For example, Butschek, S. & Walter, T., 2014. What active labour market programmes work for

immigrants in Europe? A meta-analysis of the evaluation literature. IZA Journal of Migration (2014)

3:48. doi 10.1186/s40176-014-0023-6; Card, D., Kluve, J., Weber, A., 2015. What Works? A meta-

analysis of recent active labour market program evaluations. IZA Discussion Paper No. 9236; Kluve,

J., 2006. The effectiveness of European active labour market policy. IZA Discussion paper no. 2018;

Vooren et al, 2019. The effectiveness of active labour market policies: a meta-analysis. Journal of

Economic Surveys (2019) Vol. 33, No. 1, pp. 125–149. doi: 10.1111/joes.12269.

37

In general, in the short term, net employment effects seem to be positive for service/sanction

type programmes (job counselling, guidance, orientation), whereas for traineeships and job

incentives they were stronger in the medium- to long-term (from one to two year onwards).

Public work typically showed lower net employment effects. The effects of vocational and

educational training are positive as long as this training is personalised, adjusted to labour

market needs and possibly coupled with work experience. This is made easier if training offered

to employees is on very specific problems faced by companies and also addresses change

management.

The effect of the different forms of support tend to also vary based on target group

characteristics. For instance, job guidance and counselling can be very effective for low skilled

people and migrants, both in the short- and medium- to long-term, but is less useful to

graduates. Self-employment support seems to have also a positive effect especially for the low

skilled. Work-based learning seems to generate homogeneous effects on the probability of

employment across all skill groups in the medium- to long-term. Lighter forms of support such

as job counselling can however be ineffective for those at a greater distance from the labour

market, such as the people who are inactive, have been unemployed for more than two years or

have multiple disadvantages. They also appear to show lower than average benefits for the high

skilled.

There is evidence that T010 (education and training) operations have contributed to reducing

ELET, supporting the achievement of school leaving qualifications, reducing the number of

young people at risk of becoming NEET and helping young people to achieve qualifications that

enable them to gain access to the labour market. Operations focusing on adults with low levels

of qualifications target both those in employment and those who are unemployed or inactive.

Case studies have highlighted successful examples of ESF operations providing second-chance

education opportunities for adults with low levels of qualifications.

In the case of TO9 (social inclusion) the success rate was used as a first indication of

effectiveness. This is the ratio of recorded successes (as measured by result indicators) to the

corresponding reference group (as measured by output indicators). The observed range is from

zero to 29%. This relatively low rate can be explained by low reporting on results and the fact

that common result indicators do not necessarily correspond to the desired results, (for

instance employment outcomes for actions aimed at parents of school going children).

Moreover, the bulk of the operations took place under investment priority 9i (active inclusion)

hiding, a large variety of actions with labour market, education and training or service delivery

aims. The estimated success rates are higher when calculated for the most relevant indicator for

each type of operation (e.g. the success rate for employment focused actions is 89% for the

result indicator referring to being in employment or self-employment). While the differences

across investment priorities and types of actions are in line with expectations, any analysis of

result indicators should be treated with great caution.

However, the success of ESF actions should not be analysed only in terms of employment,

education and qualifications outcomes. Inputs to an EU level Delphi survey on thematic

objective 9 (social inclusion) identified a number of areas including individual behaviour and

social roles where ESF actions may have induce change. The soft outcomes most commonly

identified based on country evidence and national evaluations (e.g. in Bulgaria, Spain, Finland,

38

France, Italy, the Netherlands, Slovakia, and the United Kingdom) are soft skills, increased self-

esteem, and improved labour market prospects. More than half of the organisations involved in

the delivery of ESF support who responded (N=295) to the public consultation on TO9 agreed

that participation in ESF supported actions generated soft-skills (57%) and increased self-

confidence (54%). Such soft outcomes are particularly important to TO9but not limited to it. A

high proportion (41% out of 141 participants) of respondents to the public consultation on TO10

(education and training) also felt that their self-confidence increased as a result of participation.

The evaluation of TO 8 (employment and mobility) notes that soft-outcomes are quite

widespread but underreported.

5.1.4 Factors hindering or promoting effectiveness

The various supporting studies identified a range of factors which affect the effectiveness of ESF

operations as regards the delivery of support. There are broadly three groups of factors:

(1) The environment in which the operations take place. This is made up of external factors that are not directly influenced by the entities involved in the delivery of these operations;

(2) Factors internal to the management entities. They relate to institutional capacity, regulatory requirements, governance and partnerships;

(3) The target groups themselves.

A supportive environment is crucial. This refers both to how the operations fit into a wider

national or regional context and to how they are perceived by the local communities in which

they take place. It is particularly important for TO9 (social inclusion) and TO10 (education and

training) operations. Shared management entails that the actual day-to-day management of

operational programmes is entrusted to national (or regional) bodies. Operational programme

strategies should be integrated and embedded into national (or regional) strategies if they are

to be implemented successfully. Performance improves when there is a strong strategic

approach that informs the prioritisation and targeting of resources to address specific needs. On

the other hand, there are several documented cases where the lack of local support or buy-in

has slowed down the implementation of TO9 (social inclusion) measures. For instance, a

housing programme in Italy was challenged when owners remained reluctant to rent their

property to vulnerable groups (homeless people and members of the LGBT community) even

with a guarantee of payment. Such examples show that there might be a need to envisage

actions that build support for the proposed operations.

The various supporting studies have also shown that institutional capacity, at all levels of

implementation, and regulatory requirements, are linked and cross-cutting factors influencing

effectiveness. Managing authorities have to lead strong coordination processes involving local

and regional authorities and key partners such as NGOs, civil society organisations and social

partners. In turn, beneficiaries must have the capacity to deliver ESF operations at the scale

required. They may need to rely on the managing authorities to build the capacity of their staff

on how to access, implement and monitor the funds. Regulatory issues include those related to

the setting up and operation of the monitoring systems, the definition of simplified cost options,

duplications of controls and compliance with state aid regulations. Responses to the public

consultation tend to confirm that these regulatory issues can place an important burden on

beneficiaries and managing authorities in the implementation of ESF supported actions.

39

For example, respondents to the TO8 public consultation on employment and labour mobility

were asked about factors that may hinder the implementation of actions. Approximately three

quarters of them (76% of 238) agreed that the administrative burden on beneficiaries and

managing authorities hampers progress. In addition, external factors, such as structural

problems (i.e. lack of employment opportunities and low education and skills level) also play an

important role in respondents’ views (for 71.3% of 238 respondents). Some organisations

specified the obstacles types they had encountered in a related open question. Out of 80

responses, 33 indicated cumbersome bureaucracy and lengthy procedures as the most

important factors hampering the effectiveness of ESF procedures, which is aligned with the

responses given to the closed question.

The public consultation for TO9 (social inclusion) asked the organisations involved in the

management and delivery of ESF support to qualify the administrative arrangements for

implementing operations. The results suggest that the most effective administrative

arrangement of all is communication, with 71% of respondents rating it as appropriate.

Opinions on the management and control system were, however, mixed, with 45% of

respondents believing that it is effective and 41% considering it burdensome.

Figure 11. How would you qualify the following administrative arrangements for the implementation of operations?

N= 295 Source: TO9 supporting study report annex 7 -

https://ec.europa.eu/social/main.jsp?catId=738&langId=en&pubId=8350&furtherPubs=yes

Partnerships between managing authorities, beneficiary organisations and other actors

improve effectiveness. A high level of cooperation is especially important in the delivery of

personalised operations or integrated support. This success factor was highlighted in the public

40

consultations as well as the country-based analyses of several Member States across all

supporting studies. Local actors, civil society organisations and NGOs, often play a crucial role in

engaging with harder to reach target groups.

The correct alignment of operations with the needs of the target group is also a success factor.

An econometric analysis of TO8 (employment and mobility) data showed that the socio-

economic context and distance of the target groups from the labour market might affect the

‘gross’ results of the operations with an employment objective. Combined with a focus on

targets or results, this may lead to ‘creaming’. This finding also applies to TO9 and TO10

operations. However, it is the form and suitability of the support offered to participants that

determines its net effects, irrespective of where it is implemented and for whom. The

overshooting of targets and high success rates across all ESF operations indicates that creaming

does indeed occur.

The precise identification of a target group makes it easier to take into account the additional

efforts required to reach it, as opposed to those people for whom a result can be generated

more easily. The econometric analysis of monitoring data for TO8 (employment and mobility)

suggests that operations targeting needs that are comparatively widespread and constant

among the population tend to proceed at a faster pace, especially where there are no obstacles

to participation such as disabilities or multiple disadvantages.

Similarly, the evaluation of ESF support to TO10 showed that all Member States identified target

groups at the design stage of operations. This is largely thanks to the programming process of

the European Structural and Investment Funds, which ensured that needs assessments were

conducted by all Member States in the partnership agreement and reflected in the operational

programmes.

At a general level, 143 of the 145 operational programmes that planned for ESF TO9 operations

were found to be consistent with the country specific recommendations (CSRs). The CSRs from

2014 to 2019 were reviewed more closely to identify relevant recommendations for ESF support

to social inclusion.

In their replies to the public consultation, beneficiary organisations and managing authorities

also underlined the importance of knowing and understanding the specific circumstances and

needs of the target group to successfully deliver ESF support. However, the precise alignment of

operations to the needs of the target group does not need to take place in the operational

programme itself. The broad definition of the target group in some operational programmes

has allowed for flexibility in implementation, for instance in reacting to the refugee crisis (e.g.

Belgium, Austria).

5.2 Efficiency

5.2.1 Assessment of efficiency

At this stage of implementation, efficiency can only be assessed based on the cost per

participation. Cost per result would be a more appropriate measure, but it cannot be used yet

because of delayed and non-simultaneous reporting on results and costs.

41

Table 8. Cost per participation

Cost per participation

(in EUR)

TO8 Employment and

mobility

TO9 Social

inclusion

TO10 education and

training

average 1,309 1,441 1,137

Range across investment

priority 142 - 1,935 581 – 3,048 729 - 2,754

Range across Member

States 440 - 8,129 326 – 7,493 343 – 6,791

2007 – 2013 Ex-post

update* 1,215 1,685 739

*The 2007-2013 estimate was calculated ex post at the end of the period. The current estimates for 2014-

2020 are based on end-of-2018 data and are affected by delays in reporting.

The cost per participation is the ratio of declared expenditure to the reported number of

participations. The observed cost per participation (see Table 8) varies widely, reflecting the

variety of operation types implemented, the different needs of target groups and the range of

contexts in which operations are implemented. The estimates for TO8 (employment and

mobility) and TO9 (social inclusion) are roughly in line with the unit costs reported in the

supporting study the update of data reported in the 2007-2013 ESF ex-post evaluation65. The

estimate for TO10 (education and training) is 50% more. This is essentially thought to reflect: (i)

a bigger emphasis on more costly operations, indicating a wider use of individual grants, and (ii)

high quality and longer schemes supporting higher education and training. Moreover, in 2007-

2013, the reporting of indirect participants was more widespread (e.g. where a school or

curriculum was directly targeted, the number of pupils was also reported).

Overall cost-benefit analyses carried out for TO9 (social inclusion)66 and TO10 (education and

training)67 suggest positive returns particularly under 9i (active inclusion), and significant

private and public returns for 10i (early leavers from education and training) and 10ii (tertiary

education). However a note of caution should be attached to the results as a number of broad

assumptions needed to be applied, given the limitations of the monitoring data and the limited

impact evidence. A detailed cost benefit analysis identified positive returns for four out of five

projects investigated under TO 9 (social inclusion). The net benefits of TO9 are even higher if all

outcomes, in particular including 'soft outcomes' are considered.

5.2.2 Factors hindering or promoting efficiency

As regards TO8 operations, actual and perceived administrative burden has hampered effective

implementation, through delays, and has also impacted cost-effectiveness. New monitoring

systems and databases were developed at Member State level to comply with regulatory

changes since the 2007-2013 programming period. The most significant factor is insufficient

administrative capacity, which in turn affects implementation. Furthermore, additional costs

65

https://ec.europa.eu/social/main.jsp?catId=738&langId=en&pubId=8330&furtherPubs=yes

66 https://ec.europa.eu/social/main.jsp?catId=738&langId=en&pubId=8350&furtherPubs=yes

67 https://ec.europa.eu/social/main.jsp?catId=738&langId=en&pubId=8346&furtherPubs=yes

42

have been incurred for recruiting and delivering ESF to harder-to-reach groups, especially those

in remote/rural areas.

As regards TO9, there were mixed views on whether the application of simplified cost options

reduced administrative burden or not. The implementation of mono-fund operational

programmes appeared to be more efficient than multi-fund operational programmes. Gold

plating was identified in 13 Member States68.

TO10 operations, gold plating was also frequently highlighted by national stakeholders as

creating additional administrative barriers for ESF project managers and potential beneficiaries.

The particular challenges in managing ESF TO10 operations highlighted in the case studies

included the following:

• high number of stakeholders in implementing ESF operations which requires a strong coordination capacity to ensure efficient delivery;

• lack of cooperation between national governmental ministries in the delivery of operations under the same investment priority;

• insufficient communication and information exchange between project managers in the implementing organisations.

• lack of mutual coordination in aligning individual education policy actions; • lack of consistency in implementing operations (for example in cases some activities

are carried out in parallel and should take place in successive stages).

Overall, most organisations that responded to the TO10 public consultation and most

stakeholders believe that, on the whole, the ESF administrative and regulatory requirements are

proportionate to the benefits achieved; most managing authorities and intermediary bodies

that responded (62.3%, i.e. 65 out of 104) judged the listed arrangements to be appropriate.

However, on average, more than 20% (23 out of 104) of respondents rated the arrangements to

be burdensome, while around 8% (8 of 104) judged them as insufficient. Most respondents

from beneficiary organisations (59.5%, i.e. 135 out of 227) judged the listed arrangements as

appropriate. However, more than 22% (50 out of 227) of beneficiary organisations rated them

to be burdensome, while 6% (13 out of 227) perceived them as insufficient. Moreover,

interviewed stakeholders from countries with smaller ESF allocations felt that the requirements

would be too onerous. The evaluation research suggests that administrative requirements can

discourage smaller beneficiaries such as NGOs, which are crucial for reaching vulnerable groups,

from accessing ESF funds.

5.2.3 Visibility of ESF interventions

Regarding the visibility of ESF operations, there is room for improvement. In particular,

evidence of the visibility of TO8 operations remains thin, even if there are several good

examples of effective communication. Out of 541 respondents to the TO8 public consultation,

70.4% seem to consider social media campaigns as the best channel of information, followed at

a distance by information and awareness raising events (44.5%). The least popular channel

seem to be the distribution of flyers and placement of advertisements in newspapers.

68

Gold-plating describes additional rules and regulatory obligations that go beyond the European

Structural and Investment Funds (ESIF) requirements set out at EU level, and that make the

implementation of ESIF more costly and burdensome for programme bodies and beneficiaries.

43

Interestingly, the channels of information actually encountered by surveyed ESF participants are

slightly different: most respondents say that they learned about the support measures they

benefited from through employment services and employment information centres (43.4%),

and less than a fifth through social media.

, a significant proportion of stakeholders and potential participants were not aware of or

familiar with the ESF at all. Although many Member States invested in raising awareness of the

ESF among beneficiaries, target groups and the general population, the level of awareness

remains mixed. The picture is somewhat better for TO10, where there was positive evidence of

the visibility of ESF funded operations: responses to the public consultation suggested a good

level of the ESF’s visibility among organisations without a direct role in managing and delivering

ESF programmes (only 12.9%, i.e 8 out of 62- organisations reported never hearing of the ESF

before the survey). Programme managers were familiar with the programme’s visibility rules

and claimed to be applying them. Beneficiaries are aware of visibility rules as this is part of their

contractual obligations and intermediary organisations have played an important role as

multipliers in maximising the visibility of TO10 operations among beneficiary organisations69.

5.3 Relevance

5.3.1 Alignment with needs

This evaluation criterion examines the extent to which ESF operations were relevant for

addressing policy needs and the needs of target groups requiring support, at the programming

and implementation levels. Overall, evidence from the three supporting studies shows that

there is a high degree of alignment of planned ESF operations under the three thematic

objectives with employment, social inclusion and education and training needs. The evidence

also shows that ESF operations have in general been relevant for the needs of target groups

though there is still room for improvement in this area. This could be achieved by better

involving different actors in both the design and monitoring stages, stakeholder involvement is

key to meeting target group needs – particularly those of the most disadvantaged groups.

A) At the planning stage, evidence shows that the most important needs of the main target

groups of each ESF thematic objective were taken on board to a great extent. This is largely part

thanks to the programming process of the European Structural and Investment Funds which

ensured that needs assessments were conducted by all Member States in the partnership

agreement and reflected in the operational programmes. The broad scope in the design of the

investment priorities, covering a wide range of policy, areas also contributed to ensuring

relevance at the strategic level.

For instance, the evaluation findings show that TO8 operations on employment and labour

mobility, contributed to helping unemployed people join the labour market. In response to the

high level of unemployment across the EU at the start of the programming period most

attention was given to (immediate) employment measures, such as providing individual

guidance to job seekers, integrated approaches, hiring incentives for employers, and supporting

apprenticeship models. Less attention was dedicated to more structural employment measures,

69

As detailed in the TO10 supporting study.

https://ec.europa.eu/social/main.jsp?catId=738&langId=en&pubId=8346&furtherPubs=yes

44

such as gender equality, active ageing or labour market institutions. Moreover, analysis of

regional allocation of expenditure suggests that TO8 funding was typically directed to where it

was most needed. This means that regions with the highest unemployment rates received not

only the bulk of funding but also the largest share of expenditure with respect to their GDP.

Box 1. Identifying the target group needs for social inclusion operations

Social inclusion (TO9) operations were highly relevant given that target groups were typically identified at the planning stage, through needs assessments and existing evaluations and consultative processes, which promoted buy-in and helped ensure that objectives were realistic and adapted to the socio-economic context. For instance in Austria, target groups were identified through consultations between the Managing Authority, the intermediate bodies at the federal and regional level, the social partners, NGOs, and other stakeholders. In Lithuania, the specific needs of the ageing population were reviewed during the planning stage through workshops with local stakeholders; TO9 actions were tailored accordingly. Delphi survey respondents also underscored the importance of identifying the target groups in the early stages of the project to promote relevance.

As regards education and training, all Member States identified the target groups at the design

stage of operations. The design of investment priorities covered a wide range of education and

training objectives and types of intervention across all education levels throughout the life-

cycle, allowing Member States to tackle diverse education and training needs in different

Member States or regions. This programming flexibility has been widely praised by national and

EU stakeholders alike for increasing the relevance of planned operations to the needs of target

groups.

B) During the implementation stage, ESF operations succeeded in reaching the most important

target groups of each thematic objective and addressed their needs. This was demonstrated by

participation indicators and confirmed by stakeholder views and other evaluation evidence,

although some challenges were identified for specific target groups.

The objectives and operations funded by ESF support to employment and labour mobility

(TO8) were mostly aligned with the needs of the target groups addressed. Participation figures

confirm the dominance of operations targeting the unemployed. Until December 2018, the

number of participations of unemployed people across all programmes amounted to some 4.6

million compared to 0.5 million of inactive persons, and 1.7 million of employed people.

According to organisations that responded to the public consultation, job-seekers, inactive

people and the long-term unemployed were the most successfully supported groups. In-depth

analysis of operational programmes and country case studies provide evidence that Member

States have developed operations that are relevant to the needs of the unemployed,

entrepreneurs, women, employees, older workers and labour market institutions.

As regards social inclusion, most of the participations recorded under TO9 involved people who

were unemployed (53%) and had a low education level - primary or lower secondary - (54%). A

large proportion of participations were from people with a foreign background or from minority

groups including Roma (28%) and people with a disability (16%). The reported proportion of

people living in rural areas (16%) is far below the almost one third of the EU28 population living

45

in rural areas - see Table 5 and Figure 8. Several social inclusion operational programmes carried

out gender-based targeting, (e.g. women not engaged in the labour market were targeted

Ireland, lone immigrant women were targeted in Finland and Roma women were targeted in

Spain). Moreover, TO9 funds were mostly allocated to less economically favourable regions,

thus addressing those geographic areas in which most people in need are living; about 68% of

recorded participations were in less economically favourable regions.

Education and training operations succeeded in reaching and addressing the needs of their

target groups, particularly of young people, a key target group for TO10: approximately 6.26

million participations were by people under 25 years old (66% of the total TO10

participations)70. Young people were the main focus of operations under investments priorities

10i (early leavers from education and training and access to learning pathways) and 10ii

(tertiary education). Evidence shows that ESF TO10 operations made good progress in

addressing the needs of young people71. This contributes to reducing the share of early leavers

from education and training, supporting the achievement of school qualifications, reducing the

number of young people at risk of becoming NEET and helping young people to get

qualifications that enable them to gain access to the labour market.

Box 2. Ensuring the relevance of early leavers from education and training operations through risk assessment

Planning risk assessments at the start of operations that target disadvantaged student groups, has allowed for more tailored support that directly addresses people’s needs. For example, in Latvia, a risk assessment methodology was developed as part of the programme supporting students at risk of early school leaving in Latvia. This made it possible to identify people’s specific needs and their circumstances from the start, which is important as these vary even within the same target group. Some at-risk students, in particular those with learning difficulties, needed extra classes, while others, such as those from low-income families, needed transport and accommodation compensation instead. The risk assessment ensured that the programme to be tailored to these specific needs, increasing its relevance.

C) However the evidence shows that ESF operations did not always fully succeed on reaching

and addressing the needs of some specific target groups, particularly the most disadvantaged

ones. This is discussed in more details below

TO8 employment operations have managed to reach participants in high need of support,

including inactive people, the long-term unemployed and migrants. Despite some reported

issues with eligibility criteria including as a result of gold-plating (e.g. for migrants) one in eight

participant has a migrant or minority background. Over one in four participations registered

comes from a rural background. This is less than the proportion of the rural population in the

EU28 but there is considerable evidence of underreporting in this area. However, there are still

some challenges to address, as disadvantaged groups are particularly targeted in more

developed regions, and in regions in which labour market conditions are improving and which

70

People in employment and unemployment account for a relatively small proportion of the overall

number of participations under ESF TO10 (most participants are classified as inactive).

71 As highlighted in the effectiveness section

46

have progressively increased their focus on individuals at a greater distance from the labour

market. Less developed areas seldom report positive results for disadvantaged. Organisations

that responded to the public consultation seem to moderately agree on the ESF’s ability to

support migrants (49.2%) and individuals with a foreign background or people in remote areas

(48.9%) and highly disagree about TO8 ESF support to people affected by poverty (39.1%) and

homeless people (17.4%). Moreover, operations have been less effective in addressing the

needs of older workers (participants aged over 54) and gender equality, with limited focus on

measures for active ageing and those that address strongly embedded gender gaps.

Compared with ESF support under TO8 and TO10, support for social inclusion (TO9) reached a

substantially higher proportion of people with a disability, migrants, people with a foreign

background and minorities, as well as people with low education72 - see Table 5. However,

evidence on whether ESF support for social inclusion reached the most vulnerable populations

with the greatest needs is mixed. In the replies to the public consultation, 39% of respondents

considered that the target groups that should be prioritised were in effect being reached while

35% believed the opposite. Respondents to the EU-level Delphi survey noted that reaching

target groups outside the reach of social services, employment services, health care and

education is a key challenge. Among other factors (see below), the assessment identified the

risk of ‘creaming’ in TO9 operations, i.e. targeting less vulnerable people with less complex

needs who can get better results. For example in Poland, the managing authority of the national

ESF operational programme believed that the focus on monitoring employment results created

a tendency to recruit participants who were more likely to become employed rather than the

people furthest away from the labour market.

Education and training operations have faced the biggest challenges in reaching

disadvantaged groups, registering lower participation rates for disadvantaged people

compared with other ESF operations (TO8 and TO9): Overall, the proportion of disadvantaged

participants in TO10 is 21%, below the proportion of disadvantaged participants in all ESF

operations (31%). Multiple stakeholders at the EU and national level identified refugees and

migrants, Roma, the long-term unemployed, older workers and people with disabilities as the

most challenging groups to engage across all investment priorities. The responses to the public

consultation (see Annex 2) provide evidence of TO10’s overall success in supporting specific

target groups but confirm that Roma and other minorities were often not seen to be

successfully supported through the ESF.

Moreover, improving access to higher education for vulnerable groups is a big challenge.

Disadvantaged groups have been hard to reach for ESF TO10 tertiary sector operations (10ii)

because of difficulties in developing collaborative relationships between the tertiary sector and

schools, hampering the identification of suitable groups that would widen the diversity of the

student body. Reaching and engaging NEETs is another challenge for ESF operations supporting

education, as institutions have limited resources to implement outreach activities. There are

some notable exceptions, as some operational programmes highlight NEETs as a priority target

group, e.g. West Wales and Valleys in the United Kingdom, where TO10 operations aim to

72

However, ESF support for social inclusion reached a lower proportion of people in rural areas than

TO8 and TO10.

47

reduce the number of pupils at risk of becoming NEET, and the Portuguese national operational

programme. In fact, identifying marginalised young people was often seen as less of a challenge

than getting them to take part in operations.

Particular reasons why reaching disadvantaged groups is a challenge vary by country,

investment priority and type of operation. Common challenges include:

1) eligibility criteria set for participation (e.g. in certain countries individuals must be

registered as unemployed before being eligible for ESF operations, making operations

inaccessible to the inactive; a similar obstacle was mentioned in relation to Roma

individuals, who may not have identity papers);

2) administrative requirements for beneficiaries discourage smaller NGOs with stronger

links to harder-to-reach groups;

3) a need for more integrated and follow-up actions;

4) additional funding required to target hard-to-reach groups;

5) higher risk of targeting operations to vulnerable groups in terms of achievement of

targets (leading to ‘creaming’, as explained above).

5.3.2 Observed factors that improve relevance

Individual-centred approaches based on specific target groups needs were considered

especially relevant for all thematic objectives. For instance, for TO8, a German operation in the

federal operational programme, focused on difficult-to-reach target groups and, offered

counselling - under the heading of gender equality measures - based on a tailor-made package

of measures. Similarly, in Finland, personal guidance and a needs assessment were the starting

points, which ensured that actions met local beneficiary needs. For TO10, the education

stakeholders interviewed felt that interventions focusing on NEETs require more integrated

operations and follow-up activities to improve their effectiveness and sustainability.

Adequate and effective involvement of a wide spectrum of key partners throughout ESF

programming and implementation is key to aligning operations with the specific needs of

target groups and local context, improve buy-in and ensure that the objectives are realistic.

Whilst the partnership principle73 was thoroughly implemented in all Member States at

planning level on paper (i.e. in the drafting of the partnership agreements and operational

programmes), most stakeholders consulted expressed concerns about the insufficient

involvement and diversification of stakeholders. For instance, in education ESF operations,

public authorities (national and regional levels), are generally overrepresented at the expense of

education and training institutions, local authorities, civil society and social and economic

73

Article 5 of the Common Provision Regulation (CPR) requires Member State to organise for each

ESIF fund programme a partnership with the competent regional, local authorities and other relevant

stakeholders, at all programming and implementation stages and at all levels, in line with its

institutional and legal framework.

48

partners74. Case studies in ESF support to employment showed that the cooperation between

new partners at regional level made it possible to develop innovative ideas in response to

specific topics and changing needs. For social inclusion operations, overall, the different

partners were highly involved in the programming phase and to some extent during

implementation; for multi-fund operational programmes, however, the implementation of the

partnerships and mobilisation of partners was identified as a challenge, possibly due to the

programme’s complexity. Delphi survey respondents also underlined the importance of

identifying the target groups in the early stages of a project to improve relevance. Finally, the

capacity of key ESF stakeholders needs to be increased, to allow them to better contribute to

the design, implementation and monitoring of ESF operations and thus improve relevance to

target group needs.

The use of an integrated approach, combining several operations at the same time (depending

on needs) is particularly relevant for operations related employment, social inclusion and early

leavers from education and training.

Box 3. Integrated approaches for ensuring relevance of support to employment

In most support to employment operations, particularly in investment priority 8i, different activities for individuals are combined in an integrated way (guidance, training, accreditation of prior learning, job searching and matching etc.). This ensures that the ESF does not support isolated operations, but those that feed into an individual, tailor-made plan for participants. In some cases, the supply side (increasing the qualifications and skills of job seekers) and demand side (incentives to hire job seekers or provide work experience places) measures are combined. The supporting studies provide many examples: • In the employment-related operations of the Campania operational programme in Italy,

needs were addressed by pursuing different types of measures related to recipients’ profiles and characteristics (pathways combining guidance, counselling, upskilling and reskilling training and, company incentives for the long term unemployed).

• In Germany, the national operational programme combined operations targeted at SMEs, individuals and intermediaries (social partners), and strengthened regional structures.

• In France, operations integrated several areas to remove obstacles hindering integration in the labour market (acquiring relevant skills, guidance for elaborating a career plan, etc.).

Integrated approaches are particularly key to addressing the needs of disadvantaged groups; in

social exclusion operations, actions were tailored to target groups by taking a holistic approach

and adapting to their multiple needs. For example in the Italian National operational

programme on social inclusion, actions to tackle extreme social exclusion amongst adults sought

to address the needs of people experiencing housing exclusion. However, this integrated

approach was missing from some operations, for example in the Romanian human capital

operational programme, and Italy’s Puglia operational programme, where the operation was

weakened because of a too narrow focus on training.

74

Non-public partners such as training institutions/vocational schools, employers and entrepreneurs,

NGOs, civil society appear to not have been appropriately consulted, and when they were consulted, it

was not in a timely manner, prohibiting meaningful contribution.

49

5.3.3 Flexibility in a changing context

As regarding TO8 operations, the operational programmes have proved to be flexible enough to

allow changes throughout the programming period, which ensured that they were able to

respond to changing needs.

For most programmes, objectives, operations types and intended target groups were defined

relatively broadly in. This provided the flexibility to respond to changes in the implementation

context without additional procedures. Formal amendments of operational programmes for

formal changes in allocated budgets and revisions to indicators, were also common. These

adjustments were tracked and show different patterns across regions, in line with the changing

situation of employment and labour mobility (see Box 4).

The case studies gives ample evidence of how programmes were able to flexibly respond to

changing socio-economic contexts.

Box 4. Examples of flexibility

In the German national operational programme, for instance, the increasing importance of digital skills

for employment received additional attention in the last calls for proposals, within the broader

programmed framework of skills for employment. This involved a slight modification of calls for

proposals, which continued to fit within the broader framework set by the operational programme.

Another example is Denmark, where the improved economic situation reduced the possibilities for

firms to allow staff to participate in long-term skill upgrading that was initially planned. With lower

demand for long term operations than expected, additional operations were set up to support briefer

training sessions focusing on more specialised skills.

In Spain, particularly in regions where the unemployment rate improved substantially, existing

employment operations were reformulated to benefit not only the unemployed, but also people with a

precarious or temporary job situation.

In response to the rapidly improving youth employment rates since 2014, Czechia was able to re-

balance its operational programmes to focus more specifically on older unemployed workers, whose

unemployment rates were not improving at the same rate.

Managing authorities considered that TO9 operations allowed for sufficient flexibility to adapt

to changes in context, including the migration crisis that started around 2015. Changes in the

operation types that were implemented compared with what had been planned occurred in at

least 10 Member States. Shifts in target groups reached by TO9 operations were identified in 13

Member States.

Most Member States already identified priority target groups at the planning stage, but others

started with a less precise target group definition, which was further focused and disaggregated

during implementation. This flexibility may have helped TO9 operations to reach more target

groups than originally planned.

Similarly, operations planned under TO10 remained relevant throughout the programming

period, despite changes in the socio-economic context. Evidence shows that where changes

were needed to ensure continued relevance - for example when national/regional needs and/or

political priorities changed, operational programmes were able to adequately adapt. Notable

50

examples of operational programmes that were successfully reprogrammed to respond to

emerging needs and adapt to new realities:

• In Ireland, it became harder to engage unemployed people in training programmes due to the declining unemployment rate. The operational programme was adapted to address this, ESF funds for training for the unemployed (10ii) were reallocated to operations to promote adult literacy under 10iii.

• In Portugal, the human capital operational programme was reprogrammed in 2018 to allocate more funding for operations targeting low-skilled adults, which had been less of a priority in initial programming given the strong focus on measures for the unemployed at the start of the period.

There were very few examples of the ESF being unable to adapt to a significantly altered

implementation or political context. One such example comes from the United Kingdom, where

the adult participation rate in lifelong learning fell slowly but consistently between 2013 and

2018 with the 2018 rate higher than the EU average, yet ESF TO10 actions were not adjusted to

address this.

Box 5. Preliminary reflections on the ESF’s possible role in addressing COVID 19 crisis

When the COVID-19 pandemic started, the field work for the supporting studies was completed or about to be completed. The time frame covered, 2014-2018, excludes the pandemic and the subsequent crisis. Nevertheless, a number of lessons learned show that ESF operations are still highly relevant in the new COVID-19 context. As in any crisis, Europe’s more vulnerable groups suffer disproportionally. This is because they are less well connected to internet, their jobs tend not to lend themselves to tele-working, online learning is not accessible to their children, etc. Vulnerable groups have complex needs and very often the hardest to reach. Strong cooperation and coordination is therefore especially important when deciding, designing and delivering targeted operations or support that are known to be more efficient in addressing these needs. This, in turn, requires high levels of inclusive stakeholder involvement and partnerships. Unemployment levels are already rising as a consequence of the COVID-19 pandemic. This is only likely to worsen as the economic impact of the crisis unfolds. Changes to the economy may prove to be long terms. In this context, lifelong learning and training will be crucial in preparing people to re-enter the labour market and adapt to new working arrangements and the economy’s changing. Issues related to the provision of guidance and counselling also plays an important role.

5.4 Coherence

5.4.1 ESF Internal coherence

ESF operations show high levels of complementarity and coherence with each other across

different investment priorities, both within thematic objective and across the different thematic

objectives 8, 9 and 10.

51

5.4.1.1 Internal coherence within thematic objectives

ESF support to employment operations is complementary and coherent across investment

priorities, with operations either indirectly or directly contributing to one another’s objectives

of and/or complementing each other. The definition of the scope of each investment priority

maximises complementarity and synergy among all actions contributing towards the general

objective of improving employment. At the same time, some flexibility is allowed as regard the

operations that can be funded. For instance, there is complementarity between investment

priority 8iv for activities that promote equality between men and women in all areas and

investment priority 8i on access to employment.

However, there is room to improvement coherence to avoid overlap and maximise synergies,

particularly: a) when operations cover horizontal themes such as social innovation, and b)

when national and regional operational programmes operate on the same territories. As

regards point a), given the relatively flexible boundaries demarcating TO8 investment priorities,

it is possible for operations funded under different programmes and strands to focus on the

same objectives and target groups (e.g. social innovation operations, the analysis of Germany’s

operations and Spain’ show). These operations are cross-sectoral by their very nature and thus

an additional effort is required to fine-tune them so that they meet the needs and objectives of

thematic support. As regards point b) overlap can happen when national and regional

operational programmes contribute to similar objectives, such as in the case of Italy, France and

Spain. For instance, in Italy, although the scope of the national operational programme is more

systemic, similar operations are being implemented at national and regional level, for example

to support the public employment services or provide employment incentives.

As regards social inclusion, by addressing the multiple drivers of social exclusion and

discrimination, ESF operations complement each other in helping vulnerable groups towards

social inclusion. ESF support takes holistic approach by combining different intervention types.

Operation types 1-4 focus on groups of individuals facing social exclusion and discrimination.

Types 5 and 6 focus on the demand side (e.g. the capacity of organisations to adequately

support socially excluded populations). In sum, about two thirds of operational programmes

working on social inclusion combine both operation types, targeting both individuals and

entities75. For example, in Sweden, ESF support to social inclusion included operation types 1, 4

and 6. In Spain (under the Madrid operational programme), ESF support to social inclusion

included operations types 1, 4, 5 and 6).

Overall, coherence is very strong between investment priorities within ESF operations

focussing on education and training (TO10), as well as between national and regional

operational programmes. Most consulted stakeholders and case study research emphasised

the fact that the sectoral focus of the different investment priorities helped ensure clarity of

scope and coherence between the different levels and types of education and training: pre-

school and school education (10i), higher education (10ii), lifelong learning (10iii) and vocational

education and training (10iv). There is some slight blurring of the distinction between 10iii

(lifelong learning) and 10iv (labour market relevance) in different Member States, however. In

particular, VET-focused priorities and operations could be found under both, especially for

75

66% of planned funds, 67% of allocated funds and 67% of declared expenditures.

52

relation to adult learners and workers. However, specific operational programmes generally

used a clear intervention logic for each (this varied according to the needs of the Member State

or region), and there is no evidence of overlaps. Still, challenges related to ensuring coherence

were occasionally raised, in particular in terms of potential duplications between national and

regional operational programmes. One example is Portugal, where joint work was required to

avoid overlaps between the national programme for success at school76 and the local action

plans to tackle school failure77 which are supported under regional operational programmes.

5.4.1.2 Coherence across thematic objectives

Overall, internal coherence between operations funded across different ESF thematic

objectives (TOs 8, 9 and 10) is good, with many examples of coherence and complementarity

between operations. There have been no overlaps in practice. However, there were some

challenges related to implementation (e.g. cases of over-separation of management

responsibilities and weak inter-sectoral cooperation) which may have limited the synergies

between operations under different thematic objectives in certain Member States. Key factors

contributing to internal coherence (confirmed in the Policy Delphi validation) included: (i)

rigorous planning at programming stage, (ii) ensuring alignment with CSRs and national

strategies and (iii) good coordination and communication between authorities managing

different TOs/investment priorities/operational programmes.

At the planning stage there is high potential for overlaps between TO9 and the other two TOs,

given that there are strong similarities between most types of social inclusion operations and

with operations under other TOs. Indeed, some actions that could have been programmed

under other TOs may have been programmed under TO9 to meet the requirement that Member

States allocate at least 20% of ESF funds to TO9. The potential for overlap is particularly high

between some types of TO9 and TO8 operations. For example, Type 1 (employed-focused

actions) and Type 2 (enhance basic skills) have potential overlaps with TO8 - Employment

Objective78. The overlap with TO10 was small, mainly because of the limited presence of Type 3

(basic school education) operations under social inclusion.

Although they often have a similar employment goal, operations supporting social inclusion

focus on the more disadvantaged groups and take a more holistic approach than other

thematic objectives, which helps avoid potential overlaps. Social inclusion operations often

focused on the most vulnerable groups and people who were furthest from the labour market,

which ensured that they were different - as well as complementary - to TO8 and TO10

operations, which covered a much broader target group. There are several examples where TO9

operations were distinguishable from TO8 operations, primarily regarding Type 1 and 2

operations. In Spain, the Acceder project provided services tailored to a specific group in a

vulnerable situation (which would not be provided by regular public employment services). In

Italy participants in TO9 operations were recruited from social services or mental health

76

PNPSE - Programa nacional de promoção do sucesso escolar.

77 PiiCiE - Plano integrado e inovador de combate ao insucesso escolar.

78 More than half of expenditures declared by beneficiaries to managing authorities (55%) and recorded

participations (55%) for TO9 had potential overlaps with TO8 – Employment objective.

53

services, while participants in operations under TO8 and TO10 were recruited through public

employment services or training providers.

Overall, there has been a good degree of coherence across ESF thematic objective operations,

as well between TO10 and TO8, particularly in relation to youth employment and vocational

training For example, several operations relating to training for young people also refer to YEI79

or TO8 funding: (i) the Irish national evaluation of TO1080 underlines the coherence with YEI; (ii)

in the regional operational programme for Lombardy, the ESF contribution to vocational

education in TO10 is coherent with TO8; (iii) a Slovenian case study also identifies

complementarity with TO8.

Some consulted stakeholders did, however, suggest that coherence could be improved, by

involving stakeholders more in design and implementation phases.. Issues were raised in this

area due to difficulties in relationships with stakeholders involved in implementation. Some

targeted consultations also underlined that, synergies between the different TOs were

sometimes limited, as the actors involved in TO8 and TO10 came from ‘different worlds’. Finally,

evidence from case studies highlights a number of some challenges for ensuring coherence, for

example in Romania, where despite good coherence between thematic objectives on paper, a

lack of inter-sectoral cooperation and multi-sectoral policy programming meant that tackling

the phenomenon of early school leaving, or adapting education systems to the needs of the

labour market, are not addressed in a sufficiently coherent way.

5.4.2 ESF external coherence

5.4.2.1 Coherence with EU policies

Overall, ESF operations are well aligned with EU policies in the fields of employment, social

inclusion and education and training. Moreover, evidence gathered during the evaluation shows

that there is strong alignment and a generally high level of coherence between ESF operations

and the priorities, analyses and country-specific recommendations (CSRs)81 of the European

Semester82.

ESF support to employment operations are fully aligned with EU employment policy and

objectives, as set in the EU2020 strategy and in the European Semester. In general, the country

specific recommendations are addressed by the ESF TO8 investments (see point 5.4.4 below).

In all Member States, ESF support to social inclusion was aligned with the European policy

framework on social inclusion83, at both the design and the planning stage, and remained so

79

Youth Employment Initiative

80 POBAL (2017), Kickboxing, Kindness and Going the Extra Mile.

81 https://ec.europa.eu/info/business-economy-euro/economic-and-fiscal-policy-coordination/eu-

economic-governance-monitoring-prevention-correction/european-semester/european-semester-

timeline/eu-country-specific-recommendations_en

82 https://ec.europa.eu/info/business-economy-euro/economic-and-fiscal-policy-coordination/eu-

economic-governance-monitoring-prevention-correction/european-semester_en

83 The European policy framework on social protection and social inclusion encompasses a range of

policies that are specific to and cut across sectors and target groups.

54

after reprogramming and OP modifications. The EU 2020 strategy84 and the European Pillar of

Social Rights85, introduced in 2017, were policies the most often cited in planning documents –

the former was cited in the planning documents in 16 Member States86 while the latter was

cited in the planning documents in seven Member States87. Planning documents were updated

during the programming period to reflect the introduction of the European Pillar of Social Rights

in 2017. The social investment package (SIP)88 was only referenced in the planning documents

for TO9 operations in Romania89. Few references to EU policies for specific target groups (e.g.

Roma and persons with a disability) were made, although in practice these target groups were

frequently addressed by social inclusion operations. The EU framework for national Roma

integration strategies was cited by three countries (Bulgaria, Germany, and Romania), the

European disability strategy 2010-2020 was only cited by Lithuania. In contrast, the review of

annual implementation reports found that operations in 22 Member States aimed to help

specifically Roma people and other minorities and operations in 25 Member States aimed to

help people with disabilities. Respondents to the EU-level Delphi survey also highlighted the

value of considering the bigger picture and linking social inclusion to innovation - environment

and digital policies, among others90 - (only one Member State (Romania) referenced the digital

single market Strategy).

Overall, ESF TO10 operations are well aligned with other EU policies in the field of education

and training, including the ET 2020 strategic framework. They are also in general well aligned

with the EU’s new policies and policy priorities which have emerged since 2014, such as the

digital education action plan91, the Council Recommendation on high-quality early childhood

education and care systems92 and the education priorities of the action plan for the

84

European 2020 – A European strategy for smart, sustainable and inclusive growth:

https://ec.europa.eu/eu2020/pdf/COMPLET%20EN%20BARROSO%20%20%20007%20-

%20Europe%202020%20-%20EN%20version.pdf

85 European Pillar of Social Rights – Building a more inclusive and fairer European Union:

https://ec.europa.eu/commission/priorities/deeper-and-fairer-economic-and-monetary-union/european-

pillar-social-rights_en

86 Germany, Estonia, Ireland, Greece, Croatia, Italy, Latvia, Luxembourg, Lithuania, the Netherlands,

Austria, Poland, Portugal, Slovenia, Finland and the UK.

87 Belgium, Ireland, Spain, Croatia, Italy, Slovenia and Romania.

88 European Commission, 2015. Policy roadmap for the implementation of the social investment

package:

https://ec.europa.eu/social/main.jsp?langId=en&catId=1044&newsId=1807&furtherNews=yes

89 Communication from the Commission 2013. Towards social investment for growth and cohesion

including implementing the European Social Fund 2014 – 2020. staff working document(2013) 39

final.

90 Annex 6.1 – EU-level Delphi survey findings.

91 https://ec.europa.eu/education/education-in-the-eu/digital-education-action-plan_en

92 https://eur-lex.europa.eu/legal-content/en/ALL/?uri=CELEX%3A52018DC0271

55

integration of third-country nationals93, which was launched in response to the 2015 migrant

crisis. Moreover, evidence, from ESF-funded operations shows that the ESF has supported the

implementation of these policies on the ground. Investments made under TO10 also contribute

to the Skills Agenda for Europe94, particular via interventions funded under 10iii and 10iv.

However, due principally to ESF’s national focus, few of the operations funded by ESF TO10 use

EU policy tools (e.g. EQF95, ECVET96, EQAVET97).

There is also evidence that the ex-ante conditionalities led to greater coherence with EU and

national policies, fostering structural reforms in some of the Member States/regions that had

to fulfil them. The 2017 Commission staff working document on the added value of ex-ante

conditionalities98 highlights some examples: in Czechia, a detailed action plan for inclusive

education was implemented to fulfil the ex-ante conditionalities on early leavers from

education and training, addressing the repeated country specific recommendation on the need

to include disadvantaged children (including Roma) in mainstream schools.

5.4.2.2 Coherence with other EU instruments

ESF TO8 operations also complement actions aiming to improve the competitiveness of SMEs

(TO3) and research and innovation (TO1) mostly supported by the ERDF. However, the

integration between the ESF and the ERDF is not always straightforward and strengthened

coordination could further encourage integrated projects. Only 8.9% of the respondents to the

public consultation considered that they are not aligned.

Another important ESF TO8 synergy of is with the EURES (mobility) axis of the EaSI

programme99; in some countries ESF directly supports the EURES network. In general, there is a

good level of complementarity with other structural funds and EU programmes, although there

are not many instances in which this complementarity gives life to multi-fund projects. As

93

https://ec.europa.eu/home-affairs/sites/homeaffairs/files/what-we-do/policies/european-agenda-

migration/proposal-implementation-

package/docs/20160607/communication_action_plan_integration_third-country_nationals_en.pdf

94 https://eur-lex.europa.eu/legal-content/EN/TXT/?uri=CELEX:52016DC0381

95 Council Recommendation of 22 May 2017 on the European Qualifications Framework for lifelong

learning and repealing the Recommendation of the European Parliament and of the Council of 23 April

2008 on the establishment of the European Qualifications Framework for lifelong learning,

http://data.consilium.europa.eu/doc/document/ST-9620-2017-INIT/en/pdf

96 Recommendation of the European Parliament and of the Council of 18 June 2009 on the establishment

of a European credit system for vocational education and training (ECVET), https://eur-

lex.europa.eu/legal-content/EN/TXT/PDF/?uri=CELEX:32009H0708(02)&from=EN

97 Recommendation of the European Parliament and of the Council of 18 June 2009 on the establishment

of a European quality assurance reference framework for vocational education and training, https://eur-

lex.europa.eu/legal-content/EN/TXT/PDF/?uri=CELEX:32009H0708(01)&from=EN

98 European Commission (2017), The value added of ex-ante conditionalities in the European Structural

and Investment Funds, staff working document(2017) 127 final.

99 https://ec.europa.eu/social/main.jsp?langId=en&catId=1081

56

regards support to labour mobility, more specific issues like complementarity with Interreg and

facilitation of life of frontier workers and employers could be explored.

There is evidence to suggest that ERDF funds for infrastructure such as housing and social

structures were complementary to ESF support for TO9. There is some evidence that goods

purchased with FEAD funds were used to support the provision of more comprehensive

measures funded through ESF support for TO9. Though little was used under ESF, community

led local development (CLLD) has provided a way to channel resources from different funds

(notably the EARDF) to support integrated, local solutions promoted by local private and public

actors.

The coordinated use of the Asylum, Migration and Integration Fund (AMIF) with ESF support

was less common, but at least in Italy this combination has made it possible to offer more

support to migrants following the 2015 crisis. Limited coordination across funds, as well as the

complexities linked to drawing from different funds, were identified as reasons for not (further)

pursuing the integrated use of funds.

As regards ESF TO10 operations, there are many examples of ESF coherence with the

European Regional Development Fund (ERDF) and Erasmus+ both at operational programme

and operation level. However, examples for the other European Structural and Investment

Funds are more limited (notably the EARDF and other funding instruments such as the Asylum,

Migration and Integration Fund (AMIF)100 and the Marie Skłodowska-Curie actions (MSCA).

There are synergies between ESF TO10 and ERDF funding, and they provide very valuable and

complementary support to major reforms and structural change in the field of education and

training. Examples include major reforms in early childhood education and care or in digital

education, for instance in Croatia or Slovenia. The level of coherence and complementarity is

also high between ESF TO10 and Erasmus+, and there are several good examples of synergies in

practice. One such example is ESF funding being used to provide top-up support for

disadvantaged students and support their mobility under Erasmus+, or upscale Erasmus+

projects.

As regards coherence with the EARDF, some examples of good synergies are presented in the

2019 Evaluation of the impact of the CAP on generational renewal, local development and jobs

in rural areas101. The evidence from this evaluation shows that, coherence between EU funds

has improved in the current programming period compared with previous ones. There is

coherence between the EARDF and the ESF, particularly in those Member States where the

different policies are delivered together by sub-regional delivery bodies or similar arrangements

(e.g. local development companies in Ireland delivering social inclusion programmes, local

integrated approaches in some regions of Italy, and chambers of agriculture in France mobilising

ESF support to train candidates receiving EARDF support).

100

https://ec.europa.eu/home-affairs/financing/fundings/migration-asylum-borders/asylum-migration-

integration-fund/union-actions_en

101 https://op.europa.eu/en/publication-detail/-/publication/4bd0b0a2-0503-11ea-8c1f-01aa75ed71a1

57

Examples of potential synergies with other EU funding instruments such as AMIF or the Marie

Skłodowska-Curie actions (Horizon 2020) being used, are limited, despite high levels of

coherence in the legal texts. Although alignment with the objectives of EU policy tools such as

ECVET, EQF and EQAVET exists on paper, examples of using of these tools in ESF TO10

operations or programming are rare, especially compared to Erasmus+ projects. This is

potentially due to the fact that transnational cooperation is not a priority under ESF, which

limits the need to align activities with EU frameworks.

5.4.2.3 Coherence with national policies

ESF TO8 investments are generally underpinned by country specific recommendations and thus

contribute to addressing the challenges identified. This is also attributable to the negotiation

process that has taken place at the programming stage between the Commission and the

Member States; the negotiation process continues during implementation as Member States

negotiate with the amendments and changes to the operational programmes. Operations

funded under TO8 are generally coherent with other policies and strategies supporting

employment and mobility at the national and regional levels.

This coherence is confirmed by positive evidence from evaluations and the case study analyses.

This is the case for instance for evaluations in the Veneto, Hamburg, Central Macedonia,

Lombardia, Umbria, Thüringen and Lubuskie operational programmes. In Veneto, an evaluation

found that that the operational programme is coherent with the regional comprehensive

strategy for 2014-2020. In Hamburg, the ESF’s coherence with European and national strategies

is reported as high and the link between the ESF and these strategies increased in the last two

years. The evaluation also showed that the EU framework for national Roma integration

strategies up to 2020 provides a useful complement to the policy framework. Operations

implemented by the Croatian employment service and financed by the European Social Fund

are complementary to the guidelines for the implementation of an active employment policy in

the Republic of Croatia for the 2015-2017 and 2018-2020 periods.

In France, the TO8 focus on entrepreneurship is coherent and complementary with the national

pact on growth, competitiveness and employment. Together these actions: (i) support

companies’ development and investments, by introducing entrepreneurship in the educational

programmes of secondary schools and higher education, and (ii) to make it easier to start a

business by simplifying processes and adapting support.

Overall ESF support for TO9 was found to be coherent with national and regional policies and

programmes. For example, in Greece, ESF support to social inclusion complemented and

reinforced a number of means-tested policies aiming to help people in vulnerable situations

(minimum income guarantee, child benefits, rent subsidy, heating benefits, ad hoc transfers to

the poorest households and ad hoc measures to facilitate access of the most vulnerable to

public health care services). In Lithuania, the alignment of ESF actions under TO9 with national

and regional policy measures is ensured through the requirement of having a direct link

between planned measures with EU and national funds and national/regional strategic

documents.

As regards TO10 operations, there is strong coherence between investments under made TO10

and other activities supporting education and training at national and regional level. This close

58

coherence was further confirmed by most case studies, targeted consultations and, the results

of the public consultation. Even greater coherence could be achieved through wider

involvement of national/regional stakeholders (including labour market actors and target

groups), as well as through better coordination of policies and actions at national/regional level.

Target groups themselves (or their representatives) should also be involved in order to better

identify and monitor evolving needs.

Overall, Member States have been relatively effective in their use of ESF funds to support

national and regional strategic priorities under the various TOs.

5.5 EU added value

The assessment of the EU added value of ESF operations has involved the identification of key

dimensions and types of effect (See Box 6 for a description), and the gathering of evidence to

determine the extent to which ESF operations have had an effect at the national or regional

level that would not have happened without EU interventions. The supporting evaluation

studies in general conclude that for all three ESF thematic objectives there is a good degree of

EU added value across all areas, with substantial evidence pointing to volume, scope, role and

process effects.

Box 6. Types of ESF added value

• Volume effects: ESF action adds to existing action, either by supporting national action in general (‘mirroring’) or in specific areas of national policy (‘boosting’);

• Scope effects: ESF action broadens existing action by supporting groups or policy areas that would not otherwise receive support;

• Role effects: ESF action supports local/regional innovations that are taken up at national level or national innovative actions that are then mainstreamed;

• Process effects: ESF action influences Member-State administrations and organisations involved in the programmes.

5.5.1 Volume effects

Evidence shows that ESF support to employment has considerably added value in terms of

volume effects, as it made it possible to support more people to be supported than would be

the case if only national programmes were available. According to a recent study on the impact

assessment of human capital investments102, the ESF had a significant volume effect, especially

through investment priority 8i (Access to employment), which received the largest share of the

ESF budget within TO8.

It is difficult to assess the precise extent of the effect but, based on existing evidence, a high

proportion of people supported by the ESF would otherwise have probably simply benefited

from passive unemployment support. This is particularly true in Czechia, where most active

labour market policies are co-funded by the ESF. Without ESF support, many disadvantaged

people (especially those aged over 55, people with disabilities, the long-term unemployed and

the low-skilled) would simply receive passive unemployment support.

102

FGB, Study supporting the impact assessment of human capital investments, Final report, May 2018.

59

In France, ESF-funded operations are recognised to have a powerful leverage effect. The

programming strategy is clearly oriented towards a volume effect: ESF funding is mostly used to

reinforce existing employment policy (without adding new operations). In the Rhône-Alpes

operational programme, for instance, the ‘Mode d’emploi’ operation (which provides intensive

individualised support to participants) complements the support provided by the public

Employment service. In Spain, the ESF operations funded under TO8 allowed intermediate

bodies to increase their impact by reaching a wider audience which would not have been

possible without the ESF. In Romania, operations under investment priorities 8i and 8iii added

to the measures financed by the State budget. In Denmark, TO8 operations have added value

by increasing the number of people who benefited from entrepreneurial training and education

and, entrepreneurial support and consultancy, as well as the number of companies involved in

skills upgrading. In the Campania operational programme – Italy, the volume effect was clear

for both the number of actions and the number of participants involved for all investment

priorities under TO8; this is due to the integration between ESF funding and national/regional

funding for the same types of actions and groups.

There is also evidence that ESF support to social inclusion played a significant role in funding

measures fighting social exclusion and poverty, complementing national policies. A volume

effect was identified in 22 Member States. This effect was primarily observed in terms of

complementarity with national efforts (17 Member States) and also in terms of boosting

funding for social inclusion (5 Member States). The analysis103 provided examples for each type

of volume effect:

• Complementary to national efforts: In Spain the Special Employment Centres (Centros Especiales de Empleo) focus on labour market integration of workers with disabilities, which is a widespread national policy co-funded by several ESF operational programmes -it provides a partial subsidy towards the wage costs of workers with disabilities in Special Employment Centres. In Lithuania, the ESF-supported activation measures for elderly people, complement and boost national policies. In Italy, ESF support to social inclusion boosted national measures implementing the national anti-poverty strategy.

• Primary role in social inclusion funding: In Hungary, most measures aiming to support the social inclusion of people in vulnerable situations are funded by ESF. Similarly in Latvia, the creation and provision of community-based services helping people with mental disorders lead independent lives and, social care and rehabilitation services for disabled children and their family members, would likely have not been possible at all without ESF support. Moreover, in Poland, ESF support to social inclusion promoted the development of the social economy. However, the primary role of ESF support raises some concerns about the longer-term sustainability of these interventions.

The results of the public consultations show that the vast majority of ESF participants, the

general public, organisations managing/delivering ESF actions and other organisations all have a

positive perception of the EU added value of ESF support in terms of volume effects. For

instance, for education and training operations, 87.2% out of 370 respondents from

organisations directly involved in the management or delivery of ESF actions considered that

103

https://ec.europa.eu/social/main.jsp?catId=738&langId=en&pubId=8350&furtherPubs=yes

60

‘more can be done than with national resources alone’. Over 80% out of 46 respondents from

organisations familiar with the ESF but not directly involved also held this view, as did 73.9% out

of 173 members of the general public with an awareness of ESF. For employment support (TO8

public consultation), 48% of all 541 respondents considered that without the ESF there would

not be enough money to pay for the supported programmes. The importance of the volume

effect of ESF-funded operations was also highlighted during the focus group discussions, in the

Policy Delphi survey and in the various case studies.

Only a minority of respondents to the public consultations answered that the EU should not be

involved. In the case of social inclusion for instance, 14% (7 of 50 respondents not involved in

ESF delivery and not aware of the ESF) felt there was no role for the EU in the promotion of

social inclusion, combating poverty and any discrimination.

Box 7. Significant volume effects of ESF support to education and training

Many of volume effects of education and training operations were identified in the supporting study104, including: (i) an increase in the number of participants (teachers/trainers or learners) and learning establishments that can benefit from education, training or school/college improvement programmes, which increases the geographical scope of implementation of education and training initiatives; and (ii) implementation of innovative education and training programmes at a larger scale than would be possible without ESF support (e.g. inclusive education, adult learning, programmes to help early leavers from education and training). Although volume effects were most visible – and most often cited – for large-scale interventions across schools (or pre-schools) under 10i, the supporting study noted volume effects under all investment priorities, with particularly strong volume effects relating to adult learners (10iii). There are many examples of volume effects across the different Member States. These include: (i) the ‘Time for Leaders’105 ESF operation in Lithuania which has already involved over 20 000 educational staff (from pre-school institutions to vocational education centres), and (ii) the PiiCiE operation in Portugal (Plano integrado e inovador de combate ao insucesso escolar), which targets primary and secondary education students and develops a wide range of activities aiming to reduce school failure. In some Member States whose national education and training budgets were particularly hard hit by the economic crisis, ESF support allowed the education and training system to continue developing despite the grave impact of the crisis on all levels of education. This important EU added value was highlighted, for instance, in the case study on the Pomorskie region in Poland, where both the ESF and ERDF helped to plug the gap left by significant cuts in the education and training budget allocated to regions by the central government.

5.5.2 Scope effects

Several sources suggest106 that in some cases, ESF support to employment operations covered

target groups which are not covered by nationally funded operations, therefore reflecting the

104

https://ec.europa.eu/social/main.jsp?catId=738&langId=en&pubId=8350&furtherPubs=yes

105 www.lyderiulaikas.smm.lt

106 Impact Assessment study, the Labour Market Policies database, focus group and case studies.

61

scope effect of ESF co-funded operations. Such target groups include young people,

entrepreneurs/self-employed, women and older workers.

According to the labour market policies database107, this is the case for employment incentives

in Estonia, Spain, Cyprus, Latvia, Lithuania, Hungary and Slovenia, where the long-term

unemployed are specifically targeted only by ESF co-funded operations. The supporting study

provides many relevant examples where ESF operations broadened the target groups. One

example is Estonia, where the supported measures were extended to new target groups (low-

skilled, young people, older people, and people with disabilities). Similarly, in France, by

targeting groups such as women and older people, ESF-funded operations have a leverage

effect. In Italy, ESF employment operations co-financed under the national system for active

employment policies operational programme also broadened the target populations (e.g. to

include the unemployed in incentive projects).

The ESF has also contributed to the broadening of existing operations including the adoption

of individualised operations and targeted approaches to focus on the needs of specific groups.

Perhaps its main contribution was to increasing awareness of gender equality and the

introduction of specific policies, which were previously noticeably absent in some Member

States. In Germany, ESF operations play a role in plugging the gap left by national actions

related to specific target groups that are hard to reach and specific topics related to

employment. These include, for instance, ESF funding for 8iv Equality between men and

women; without the ESF, these type of operations would simply not exist. Another example is

the ‘Unternehmens Wert: Mensch’ operation, under which counselling of companies could not

be proposed without the ESF support. Finally, there is also some evidence (e.g. Croatia) where

the ESF has prioritised investments in hard-to-reach areas that had previously not been priority.

Box 8. Social inclusion scope effects: broadening actions to more groups

A scope effect of ESF support to social inclusion was identified in 17 Member States108. The ESF provided support for specific target groups that were not covered or that received less coverage in nationally funded interventions. For example, in Bulgaria, ESF support to social inclusion covered Roma children and children with special educational needs, in Denmark it covered people with ‘limited attachment’ to the labour market, and in Croatia operations supported people with disabilities. The Netherlands used ESF to support refugees and Italy used it to help LGBT people facing housing exclusion. In Austria the 'Youth College Vienna' project provided education and counselling to young refugees who would have not received support from other sources and who are less likely to have the skills needed to enter training or the labour market.

For education and training operations, there is strong evidence of scope effects both in terms

of broadening actions to new target groups and to new policy areas. In terms of broadening

actions to include new target groups, evidence clearly shows that ESF funding has given

107

https://webgate.ec.europa.eu/empl/redisstat/databrowser/explore/all/lmp?lang=en&display=card&

sort=category

108 Belgium, Bulgaria, Denmark, Germany, Ireland, Greece, Croatia, Italy, Lithuania, Luxemburg, Malta,

the Netherlands, Austria, Slovenia, Slovakia, Sweden and Romania.

62

education and training providers an opportunity to address more disadvantaged and harder-to-

reach groups. This is true in all Member States and across all education sectors and all

investment priorities. Such groups include young NEETs, early school leavers, children with low

socio-economic status, low-skilled or low-qualified adults, migrants, Roma, people with special

educational needs (SEN), people living in remote areas, as older workers and workers with

obsolescent skills. The additional cost of reaching out to these groups and providing the sort of

individualised and cost-intensive support which they require often means that they are de facto

excluded from standard education and training provision. Examples of good practices in this

area include the Bulgarian ‘Your Class’ ESF operation109 and the German ‘Voluntary 10th school

year’ operation110.

Moreover, ESF funding has allowed education and training providers to address wider policy

areas such as inclusive education, adult/lifelong learning, non-formal education, digitisation,

civic education, developing vocational training in wider/future-oriented sectors and developing

soft/transversal competences. Examples include the ‘PuMPuRs’111 operation targeting early

leavers from education and training in Latvia and the ‘digital skills bridge’112 ESF operation in

Luxembourg, which aims to support the development of digital skills for workers affected by

digital transformations.

5.5.3 Role effects

Evidence on role effect of ESF TO8 operations is mixed. There is little information on the extent

to which the operations supported by the ESF supported innovation and the transfer of ideas,

although there are some examples of good practice with lessons that can be learned.

Box 9. Examples of TO8 role effects

• Limited role effects: The case studies provide some insight into the ESF’s role effect of the ESF, highlighting that, in Spain and Romania, TO8 operations provided little or no support to innovation. In Spain for instance, innovation was included in ESF actions only to a limited extent due to uncertainty regarding its eligibility. Most supported operations complemented already existing actions, without necessarily introducing new approaches to tackling the problems. A lack of innovative measures was also identified in Romania, particularly for ESF operations focusing on the most disadvantaged groups, such as Roma or low educated people.

• Substantial role effects: In Austria, it is reported that the ESF supported innovative operations to promote gender equality (8iv) and support active and healthy ageing (8vi). Similarly, in Germany, the role effects of ESF-funded operations in the field of employment and mobility were recognised at the level of federal and regional operational programmes. Examples include two innovative federal operations supported under investment priority 8v: ‘UnternehmensWert: Mensch’ (which aims to improve working conditions) and ‘IQ-Qualifizierungsprogramm’ (which supports people who need help getting recognition of their certificates). Some elements of the latter

109

See box A5-1 in Annex 5 of the TO10 supporting study.

110 See Box 8 in the TO10 supporting study.

111 http://www.pumpurs.lv/

112 https://www.skillsbridge.lu/

63

project were subsequently transferred into actions under regular national funding.

The focus group also highlighted two examples where the ESF created opportunities to test

innovative approaches and mainstream theme. In Estonia, peer counselling is proposed for

people with disabilities (for instance a disabled worker provides advice to disabled people who

are looking for a job or who just started to work and need support). In Malta, in order to

provide people with mental disabilities with employment opportunities, work activities are

divided into sub-tasks to allow individuals to perform single tasks, which are eventually

combined.

In general, the role effect of the ESF TO8 operations was well acknowledged. In Austria, it is

reported that the ESF-supported innovative operations to promote gender equality (8iv) and

support active and healthy ageing (8vi). Similarly, in Germany, the role effects of ESF funded

operations in the field of employment and mobility were recognised both at the level of the

federal and regional operational programmes.

ESF TO10 investments have contributed to the transfer of ideas by funding the piloting or wider

implementation of ideas from other Member States or from regional to national level. There is

also evidence that ESF actions supported innovation in education and training including various

ways of modernisation of practices and policies (e.g. digitisation, inclusive education), targeting

skills development in new or evolving sectors of the economy (e.g. sustainable development,

high-tech industries), testing new or alternative pedagogies (e.g. inclusive educational methods,

use of theatre) and developing innovative training offers.

Lastly, ESF TO10 has also led to structural changes in education and training systems including

the roll-out or decentralisation of new approaches and policies. Examples of TO10 support for

structural changes exist under all investment priorities; one such example is the establishment

of the network of regional centres of VET competence in Croatia.

5.5.4 Process effects including sustainability

The TO8 case studies show that there are various benefits for administrations and organisations

involved in ESF operations. In Czechia for instance, cooperation between the Managing

authority, the Ministry of Labour and Social Affairs, other state institutions, social partners and

NGOs (in particular for actions supporting the capacity of labour offices, the social partners and

NGOs) was highlighted as a process effect of ESF operations.

The process effects of ESF supported operations were also largely recognised during the focus

group discussions. ES operations increased inter-institutional cooperation, for example in

Bulgaria, where social partners and social services now work together, as a result of the

cooperation needed to implement individual centred approaches to support people into (or

closer to) employment.

As regards the continuity of EU-supported operations after funding ends, there is increasing

evidence of them being mainstreamed into national policies especially in socio-economic

contexts that is more favourable compared to the financial crisis. Thus, the systemic changes

achieved through the implementation of ESF TO8 support are likely to remain even after ESF

funding ends.

64

Managing authorities consider that it is still too early to assess the sustainability of TO9 support.

ESF-funded TO9 operations generated process effects by improving the administrative for

capacity and knowledge of the design and delivery of services.

Similarly, ESF TO10 support generated positive effects which are likely to continue after the end

of funding. There is a strong likelihood of ongoing benefits at all three levels: individual,

institutional and policy.

However, the COVID19 pandemic may affect the sustainability of the positive effects of

operations run under the ESF’s three ESF thematic objectives.

At individual level, ESF TO10 operations had an immediate positive impact on many people

across the EU. The public consultation confirmed this positive impact with a large proportion of

ESF participants reporting that the support had made a positive difference to their situation:

over half said that, without EU support, there would been fewer or no opportunities for people

in their situation. Key outcomes mentioned by individual respondents included: learning new

skills and/or getting a qualification, feeling more confident, starting a new education or training

course, and improving employment conditions, such as increasing salary or receiving a

promotion.

These immediate results of participation in TO10 ESF-funded operations are likely to have

lasting impacts on the lives and life chances of most people involved.

The findings also identify to sustainable impacts of ESF TO10 support at institutional level. Many

of the effects on institutions are likely to have a long-lasting impact. For example:

• Improvements to the competences of teachers and trainers are likely to contribute to long-term improvements of the institutions in which they work, for example through improved teaching skills and methods, increased awareness of how to deal with diversity, better organisation and more innovation. Once new approaches and methods are introduced, evidence suggests that it is unlikely that institutions will ‘slip back’ to old ways and instead will continue a positive trajectory of ongoing improvements.

• The links established with other organisations and the wider community (families, communities, NGOs, employers, other education and training providers, etc.) will also remain in place, and continue to provide a more effective and integrated context for educational and training activities.

• The investments in new teaching curricula (e.g. in Croatia), resources and methods will benefit institutions for many years beyond the end of ESF funding.

ESF TO10 funding also had many policy impacts that are highly likely to have sustainable

positive effects. Examples highlighted in the supporting study include ESF support for: the

establishment of the Maltese systematic approach to addressing early leavers from education

and training; the testing and introduction of the dual learning system in Belgium (Flanders); the

new start programme to ensure that VET schools in Greece better adapt to the needs of

students from more vulnerable backgrounds; and the introduction of a major digital education

reform in Croatia..

65

6 CONCLUSIONS

The evidence from the three thematic evaluation-supporting studies underpin the following

overarching conclusions on 2014-2018 ESF operations.

The assessment of effectiveness has shown that there has been progress in the financial

implementation of ESF programmes and in the achievement of participation and result

targets. However there are large differences in performance across countries, operational

programmes and investment priorities. There were initial delays in implementing ESF operations

and financial execution is lagging 7% behind the comparable stage of the previous programming

period. Nevertheless, by the end of 2018, implementation rates were generally on track to meet

the targets by the end of the programming period; although, administrative capacity to deliver

operations remains a constraint in some countries and (especially) within less developed

regions.

Overall, across the EU, 27% of planned resources were spent by the end of 2018, and the

project selection rates113 were around 70%. As regards output targets they show good progress,

as 22.2 million participations were reported covering the specific target groups of each ESF

thematic objective (with higher shares of unemployed participants in employment operations,

of disadvantaged groups in social inclusion ones and of inactive and young people in education

and training).

Although implementation is still ongoing and comprehensive results data are still to be

reported, the evaluation has nevertheless highlighted some significant achievements. There has

been good progress in achieving results from participation in ESF-funded operations, as 9.5

million positive results were reported in total by December 2018 (this includes participants

engaged in job searching or who found a job; including self-employment; entered

education/training or gained a qualification).

There is evidence that ESF operations have a sustainable positive impact on participants. In

addition to the reported impacts in terms of gains of employment and education and

qualifications, both in the immediate and longer-term, a wide range of 'soft' outcomes were

identified notably for employment and social inclusion operations. Such outcomes are quite

widespread but underreported, as they are not part of the common indicators. The most

common soft outcomes are soft skills, increased self-esteem and improved labour market

prospects.

The three evaluation-supporting studies have identified some notable achievements at the level

of investment priorities and types of operations:

Within ESF support to employment, the most successful operations have been access to

employment for the unemployed. The assessment confirms varying but positive effects in

terms of employment chances for unemployed participants due to the support they received. In

addition, exploratory research on overall macroeconomic impacts tends to confirm that support

is beneficial and sustainable for several regions with high unemployment levels. Also, ESF

113

This is the ratio of funds allocated by the Managing Authority to specific projects or operations over

the total amount available (by TO) or indicatively planned (by IP).

66

operations have contributed to the development of a more adaptable workforce and have

helped improve the employability of the participants. ESF support has been most effective – in

terms of supporting employment - for individuals who are at a certain distance from the labour

market, i.e. neither close (such as well qualified individuals) nor too far away from it (e.g. those

with multiple disadvantages).

More than half of results generated for ESF support to social inclusion were for Type 1

operations (employment-focused actions). An additional 35% of results were generated for

Type 6 operations (action influencing attitudes and systems). The result-level achievement rate

was more moderate than that for outputs, which is in line with the fact that results take more

time to materialise, especially for disadvantaged groups. Moreover, social inclusion operations

helped reduce discrimination, improved integration of marginalised communities, changed

attitudes towards education and increased soft skills.

As regards education and training, early leavers from education and training and lifelong

learning operations have been more successful in terms of both direct results and the potential

for longer-term systemic change. The relative success of these priorities appears to relate to

their stronger alignment with national and regional strategic priorities. Many successful types of

operations have been identified - ranging from support to vulnerable learners and the teaching

staff that work with them, to digital innovations in the classrooms and local action plans to

address early leavers from education and training. However, many Member States have faced

challenges preventing from achieving positive results under ESF tertiary education operations.

On top of the positive results for participants in ESF operations, the modelling of ESF impacts

based on RHOMOLO114 simulations suggests that the programme will have final positive

macroeconomic impacts (measured as additional EU GDP and jobs compared to baseline) as a

result of the ESF human capital investments, which enhance the productivity of the labour force

and increase participation in the labour market, among other effects. The final estimated impact

would add 0.26% to European GDP by 2023, and 316 000 jobs.

Therefore, although it is not possible to fully disentangle the impact of ESF funding from other

context factors, there is clear evidence on the positive contribution of ESF operations to the

overarching objectives of ESF support to employment, social inclusion and education towards

the EU 2020 targets, both from a micro and a macro perspective (based on results

achievements, qualitative evidence from stakeholder views and micro and macro econometric

modelling).

Finally, significant potential for long-term impacts was identified through ESF support for the

development of system level changes and sustainable innovative measures. Qualitative

evidence (country-based analyses including case studies) suggest that ESF support to social

inclusion also had broader level impacts including enhanced access to public services,

deinstitutionalisation, and cross-sectoral collaborations to promote innovative approaches.

Education and training operations greatly support systemic change, enhancing changes in

114

RHOMOLO is the spatial computable general equilibrium model of the European Commission, see for

details: https://ec.europa.eu/jrc/en/rhomolo

67

government policy approaches to education, particularly visible for early leavers from education

and training and lifelong learning operations.

The analysis of efficiency has shown that there was a high level of variation across investment

priorities and Member States in the cost-effectiveness of the different types of operations. The

large variance in the cost per participation reflects the wide range of types of operation and

delivery mechanisms, as well as the different costs levels in the Member States.

Overall cost-benefit analyses attempted for social inclusion115 and education and training116

suggest positive returns particularly under active inclusion, early leavers from education and

training and tertiary education operations, however a note of caution should be attached to the

results as a number of broad assumptions needed to be applied, given the limitations of the

monitoring data and the evaluation evidence.

Actual and perceived administrative burdens have hampered effective implementation, through

delays, and also have impacted cost-effectiveness. New monitoring systems and databases have

been developed at Member State level to comply with regulatory changes since the 2007-2013

programming period, but a more significant factor is insufficient administrative capacity, which

in turn impacts implementation.

At this stage of the implementation, efficiency can only be approached through cost per

participation. The large variance reflects the wide range of types of operations and delivery

mechanisms, as well as the different costs levels in the Member States. The cost-benefit

analyses carried out for social inclusion and education and training support suggest positive

returns particularly under active inclusion, early leavers from education and training and tertiary

education operations. ESF administrative requirements are considered to be proportionate to

the benefits achieved. Nonetheless, there is recognition that these requirements can be

complex and time-consuming, making them onerous for smaller beneficiary organisations, for

example those working with specific target groups or schools.

In support to employment operations, there have been additional costs for recruiting and

delivering ESF to harder-to-reach groups, especially those in remote/rural areas. As regards

social inclusion, other key factors that limit the efficiency of ESF operations include gold plating

and drawing on other EU funds (e.g. ERDF, CF) to support the operational programme. The

complexity of administering an operational programme with multiple funds may have led to

delayed implementation and generation of results. For education and training support, there

was also evidence that having many partners involved in implementation can result in delays.

Qualitative analysis has shown that factors that can foster efficiency include the use of

intermediary bodies, wider partnership arrangement, including specialist sectoral

organisations, and simplified cost options (SCOs). The introduction of SCOs may lead to an

initial increase in administrative burden for some beneficiaries that need to adjust procedures

115

See TO9 supporting study:

https://ec.europa.eu/social/main.jsp?catId=738&langId=en&pubId=8350&furtherPubs=yes

116 See TO10 supporting study:

https://ec.europa.eu/social/main.jsp?catId=738&langId=en&pubId=8346&furtherPubs=yes

68

and train their staff. Over time, however, the use of SCOs can promote the take-up of ESF and

lower the administrative burden.

Overall, there was a high degree of relevance of ESF to employment, social inclusion and

education policy needs and to the needs of target groups requiring support, at the

programming and implementation levels. Programming remained relevant throughout the

programming period to date, as the operational programmes have proved to be flexible enough

to adapt to socio-economic and policy changes. The flexibility of OPs allowed Member States to

deal with unexpected shifts in the socio-economic context, such as the 2015 refugee crisis.

In terms of target groups reached, the evidence shows that ESF operations were generally

aligned to their needs but that challenges remain in reaching and addressing the needs of

some groups, particularly the most disadvantaged ones. Support to employment managed to

reach participants in high need of support, including the inactive, the long-term unemployed

and migrants. The low budget available, lengthy implementation periods and a need for more

holistic and informed approaches resulted in issues linked to active ageing and embedded

gender stereotypes being little or hardly addressed. The supporting study shows that ESF

support was most relevant when combined with other measures that support the participants

(including health, housing, etc.), especially those furthest away from the labour market.

ESF support for social inclusion reached a substantially higher share of people with a disability,

migrants, people with a foreign background and minorities, as well as those with low education

than did support to employment and labour mobility and education operations, which is fully

consistent with the focus and target groups of social inclusion. However, evidence as to whether

ESF support to social inclusion reached the most vulnerable populations with the greatest needs

is mixed due to the challenges in identifying these groups with the existing common indicators,

as these are necessarily broad groups relevant at the EU level. The assessment also identified

risks of ‘creaming’, i.e. targeting less vulnerable people with less complex needs who can get

better results.

The biggest challenges faced by education and training operations have been to reach some

disadvantaged groups, and they have registered lower participation shares for these groups as

compared to other ESF operations. Refugees and migrants, Roma, the long-term unemployed,

older workers and people with disabilities were identified as the most challenging groups to

engage. In particular, improving access to higher education for vulnerable population groups is a

big challenge, as disadvantaged groups have been hard to reach for ESF tertiary education

operations because of difficulties in developing collaborative relationships between the tertiary

sector and schools.

Ensuring relevance to needs was fostered by a number of factors including the needs analyses

in partnership agreements, a flexible approach to programming, allowing adaptation of

operational programmes and planned actions to any changing needs of Member States/regions,

and close alignment of ESF priorities with the European Semester and its country-specific

recommendations. The assessment found that the European Semester is a strong framework for

ensuring that ESF operations are relevant to target groups. Improvements to relevance could,

however be made through enhanced consultation with actors closer to the needs of target

groups, such as NGOs, social partners and, training providers. While these actors were involved

69

and consulted to some extent – in part thanks to the partnership principle enshrined in the ESI

Funds – there is room for more meaningful consultation and more engagement with a broader

range of actors in the needs assessment, programming and implementation stages.

The assessment also showed that ESF operations have a good overall degree of internal and

external coherence. They are well aligned with other EU policies and initiatives in the field of

employment, social inclusion and education and have supported their concrete implementation

on the ground. There is also a good degree of coherence with the European Semester CSRs

which are well reflected overall in the situation analyses of partnership agreements, helping

address the challenges identified, thanks to the negotiation process between Commission and

Member States. at the programming stage. Moreover, coherence was improved thanks to the

fulfilment of ESF ex-ante conditionalities, which lead to structural reforms, although this also

caused some delays in ESF implementation (e.g. in Romania).

ESF employment, social inclusion and education and training operations are internally coherent

(between different thematic objectives) and between investment priorities within each

thematic objective. ESF support to social inclusion has a high potential overlap with some

support to employment operations (e. g. promoting sustainable and quality employment and

supporting labour mobility), and actions that could have been programmed under other

thematic objectives may have been programmed under social inclusion to meet the

requirement that Member States allocate at least 20% of ESF funds to this thematic objective.

However, actual overlaps are low due to the different target groups addressed and the more

holistic approach taken under social inclusion.

Coherence with other EU funding instruments in related fields is more mixed: while there is

often good coherence in the legal texts, evidence of synergies in implementation varies, and a

variety of challenges in combining different funds were identified. ESF employment operations

show a good level of complementarity with ERDF thematic objective 3 (competitiveness of

SMEs). However, integration with ESF can be improved. Employment operations also show

significant synergies with other EU-funded programmes, namely EaSI, the European

Globalisation Fund and the Asylum Migration and Integration Fund.

ESF social inclusion support was also combined with other EU funds, the most common EU fund

being the ERDF. While strong complementarities were found among examples of OPs that draw

on ESF and other EU funds, the complexities involved in the implementation of multi-fund OPs

imply delays in implementation.

In education and training operations, good examples of coherence in implementation were

nonetheless found in particular with ERDF and Erasmus+, however only to a very limited degree

with other funding instruments (e.g. Marie Skłodowska-Curie actions).

ESF operations are well aligned with national and regional policies in the Member States. Such

complementarity can take the form of ESF having a supporting role for local policies or filling

policy gaps. However, alignment and complementarity with national policies could be improved

concerning operations supporting horizontal themes such as social innovation or regarding

tertiary education ESF operations.

70

ESF operations have generated a good degree of EU added value across all dimensions, with

substantial evidence pointing to volume, scope, role and process effect., There were important

volume effects in terms of the increased participation of target groups (e.g. teachers and

learners in education operations), the number of actions, and widening the geographical scope

and scale of implementation of key programmes. Therefore ESF support is playing a pivotal role

in funding employment, social inclusion and education policies that complement national

efforts.

Scope effects were also clear, particularly for broadening actions to wider target groups that

would have not been supported otherwise, including many disadvantaged groups (e.g. social

inclusion operations supported Roma children and children with special educational needs,

people with disabilities, people facing housing exclusion and refugees, whereas education and

training ESF support covered Roma, older workers and low-skilled adults). There were also

scope effects in addressing policy areas which were not previously high priorities on the agenda

(e.g. inclusive education, adult learning and, non-formal education).

ESF operations also played an important role in supporting the transfer of ideas and the

introduction of innovations and structural reforms. In employment operations the ESF has

fostered innovation especially in the support of harder-to-reach groups, including those with

disabilities, and those from remote rural areas and there is evidence of mainstreaming, with

innovative approaches being adopted by Member States as part of their labour market

programmes (e.g. the ESF has helped to raise the profile of gender issues and promoted social

innovation). In social inclusion innovation has focused on the use of individualised and targeted

approaches, especially toward migrants, older workers and women. The ESF also supported the

improvement of existing national frameworks (e.g. through the establishment of monitoring

and coordination mechanisms, design and implementation of integrated approaches to

combating social exclusion), and allowed for the testing of new partnerships. In education and

training, significant innovation effects were recorded, with the use of digitisation and new

pedagogies, as well as structural reforms such as the development of local or regional education

strategies or the rolling-out of new educational approaches and strategies nationwide. Role

effects could however potentially be further enhanced through a greater focus on peer learning,

transnational cooperation, visibility of ESF support and, dissemination of results of ESF-funded

operations.

Many process effects from ESF operations are also in evidence. Crucially, the ESF has helped

build effective delivery capacity in Member States in terms of programme and cost

management and monitoring systems, and awareness and knowledge of target groups.

Relevant examples include improved administrative capacity and knowledge in the design and

delivery of services promoting social inclusion, improvements to governance and organisation of

the offer of education and training, and establishing closer links and cooperation between

schools and training providers with other stakeholders at local/regional or national level.

71

6.1 Lessons learned

The supporting studies provided evidence on those factors that seem to lead to better use of th

ESF funding and on the main points for improvement117.

Operational programme strategies should be integrated and embedded into national (or

regional) strategies. Shared management entails that the actual day-to-day management of

operational programmes is entrusted to official national (or regional) bodies. Performance is

enhanced by a strong strategic approach which informs the prioritisation and targeting of

resources to address specific needs. This is clearly the case when the ESF is used to increase the

volume of support offered. It is also the case when innovative approaches are tested before, if

successful, being mainstreamed. All thematic objectives included actions intended to impact

structures and systems. Success requires the active support from the national (or regional and

local) actors concerned.

Institutional capacity, at all levels of implementation, and regulatory requirements, are linked

and strongly influence effectiveness. Managing authorities have to lead strong coordination

processes involving local and regional authorities and key partners such as NGOs, civil society

organisations and social partners. Beneficiaries in turn must have the capacity to deliver ESF

operations on the scale required. They may need to rely on the managing authorities to build

the capacity of their staff on how to access, implement and monitor the ESF. The issues most

commonly cited as placing a high demand on the capacity are the setting up and operation of

the monitoring and control systems, the definition and roll-out of simplified cost options, and

the application of state aid regulations.

Strong partnerships are required. Greater cross-sectoral cooperation between education,

health, labour market, social services and services dealing with territorial planning and

development should be fostered at all levels (EU, national, regional) to address the complex

needs of target groups. This is especially important in the delivery of personalised operations or

integrated support of interventions that need to follow up on each other and cannot be

conducted in parallel. Local actors, civil society organisations and NGOs, often play a crucial role

in engaging with harder to reach target groups. Yet the partnership principle is not always fully

implemented in practice, with stakeholders pointing out some examples of superficial

consultation processes that hindered meaningful engagement of partners across all areas of

programme design, implementation and monitoring. Moreover, having a high number of

partners makes more demands on the institutional capacity of the managing authorities and

may therefore come to be a constraint on efficient delivery. Managing authorities may be

sometimes avoid these local actors, civil society organisations and NGOs, precisely because of

the perceived demand on their institutional capacity. Diversifying the stakeholder groups that

are consulted remains an important area for improvement, as public authorities, especially

from the national and regional levels, are often overrepresented.

The correct alignment of operations to the needs of the target group is another success factor.

It is the form and suitability of the support offered to participants that determines its net

117

The supporting studies provide additional lessons learned and points for improvement specific for each

ESF thematic objective.

72

effects, irrespective of where it is implemented and where the money comes from, ESF or non-

ESF. Overall, there is strong evidence that the most important needs of the main target groups

of each ESF thematic objective were taken on board. Nevertheless, ESF operations did not

always fully succeed in reaching and addressing the needs of some specific target groups,

particularly the most disadvantaged ones. It is the precise identification of a target group that

allows account to be taken of the additional efforts required to reach it, rather than reaching

those for whom a result can be generated more easily. The alignment of operations to the

needs of the target group does not need to take place in the operational programme stage

itself. Some operational programmes attributed their flexibility to a broad definition of target

groups, at the planning stage, narrowed down during implementation.

ESF-supported actions bring about changes in number of areas including individual behaviour

and/or social roles well beyond employment, labour mobility and education outcomes but

these are poorly documented. Outcomes such as qualifications can be captured and

adequately analysed by using the existing set of indicators. However, these soft-outcomes are

quite widespread among participants, difficult to capture in quantitative terms and thus

underreported. Efforts could be made to improve the systems to this end, including through

experimenting with “distance travelled” approaches.

More should be done to promote the visibility of ESF support and mutual learning. Adopting a

greater focus on dissemination of ESF results and good practices, with a robust consideration of

success factors, challenges and transferability – at EU, national, regional and operation level –

has potential to inspire other potential beneficiaries, and ensure that lessons learned in

addressing key challenges and reaching specific target groups are better shared among the

community of practitioners. A focus on dissemination would also ensure better visibility of the

contributions of ESF across the EU, and could foster greater absorption and improved funding

applications. As seen in the supporting studies, there was evidence of non-awareness of the

opportunities offered by ESF support.

Finally, in order to provide robust evidence on the impacts of the ESF-financed interventions,

more timely availability of data is crucial. No proper cost effectiveness analysis could be done

at this stage due to incompleteness of data, as reporting on outputs, results and costs is not

synchronised and operations are not completed. Moreover, the counterfactual impact

evaluations require detailed data not only on the participants but also on the non-participants,

and thus data collection should be planned well in advance.

73

ANNEX 1: PROCEDURAL INFORMATION

Lead DG, Decide Planning/CWP references

This evaluation was carried out by DG EMPL, as a result of three different initiatives related to

the ESF support to policies in the areas of:

• Employment and labour mobility (thematic objective 8) under the Decide number PLAN/2018/3178. The initiative was published in July 2018.

• Social inclusion (thematic objective 9) under the Decide number PLAN/2018/3177. The initiative was published in December 2018.

• Education and training (thematic objective 10) under the Decide number PLAN/2018/3147. The initiative was published on December 2018.

Organisation and timing

2.1. Thematic objective 8:

An Inter-Service Steering Group (ISSG) was created in July 2018. The invitation launched on 17

July 2018 was addressed to the following DGs: AGRI, BUDG, EAC, GROW, HOME, JRC, JUST,

MARE, REGIO, RTD, SG and SJ.

The timing of the evaluation was as follows:

20 July 2018 - 17 August 2018

Publication of the roadmap and feedback period

8 August 2018 - 28 September 2018

Two rounds of consultation of ISSG via written procedure

19 October 2019 Request for services for the external study supporting the evaluation

4 March 2019 Signature of the contract for the supporting study with the consortium led by Fondazione Giacomo Brodolini

19 March 2019 1st meeting of the ISSG: kick-off meeting for the supporting study

17 April 2019 2nd meeting of the ISSG: draft inception report of the supporting study

11 September 2019 3rd meeting of the ISSG: draft interim report of the supporting study

13 January 2020 Final deliverable of the Rhomolo modelling

12 February 2020 Draft final report of the study

18 February 2020 4th meeting of the ISSG: draft final report of the supporting study

September 2020 Written consultation on the draft staff working document

74

2.2. Thematic objective 9:

An Inter-Service Steering Group (ISSG) was created in December 2018. The invitation was

addressed to the following DGs: AGRI, BUDG, EAC, GROW, HOME, JRC, JUST, REGIO and SG.

The timing of the evaluation was as follows:

19 December 2018 - 16 January 2019

Publication of the roadmap and feedback period

17 December 2018 1st meeting of the ISSG

d1 February 2019 Request for services for the external study supporting the evaluation

16 May 2019 Signature of the contract for the supporting study with the consortium ICF-

17 May 2019 2nd meeting of the ISSG: kick-off meeting for the supporting study

26 June 2019 3rd meeting of the ISSG: draft inception report of the supporting study

14 November 2019 4th meeting of the ISSG: draft interim report of the supporting study

28 February 2020 Draft final report of the supporting study

5 March 2020 Final deliverable of the Rhomolo modelling

September 2020 Written consultation on the draft staff working document

2.3. Thematic objective 10:

An Inter-Service Steering Group (ISSG) was created in January 2019, which included the

following DGs: EMPL, BUDG, EAC, HOME, JRC, REGIO and SG.

The timing of the evaluation was as follows:

21 December 2018 - 18 January 2019

Publication of the roadmap and feedback period

25 January2019 1st meeting of the ISSG

28 February 2019 Request for services for the external study supporting the evaluation

6 June 2019 Signature of the contract for the supporting study with the consortium ECORYS-Ismeri

12 June 2019 2nd meeting of the ISSG: kick-off meeting for the supporting study

12 July 2019 3rd meeting of the ISSG: draft inception report of the supporting study

20 November 2019 4th meeting of the ISSG: draft interim report of the supporting study

6 April 2020 Draft final report of the supporting study

2 June 2020 Final deliverable of the Rhomolo modelling

September 2020 Written consultation on the draft staff working document

75

Exceptions to the better regulation guidelines

N/A

Consultation of the RSB (if applicable)

N/A

Evidence, sources and quality

Each of the three initiatives were based on three main instruments:

• An EU-wide public consultation, which was carried out and analysed with the assistance of the external team of experts.

• A supporting study carried out by an external team of experts, through a contract (VC/2019/0089 for thematic objective 8, VC/2019/0268 for thematic objective 9 and VC/2019/0351 for thematic objective 10)118, through DG EMPL Multiple Framework Contract VT/2016-027 for the provision of services related to the implementation of the better regulation guidelines. The supporting study combined the results of the two other tools together with additional work, such as desk research, public consultations, case studies, interviews with stakeholders, focus groups and Delphi surveys (see Annex 2 for more details).

• An estimate of macroeconomic effects of ESF supported operations using the Dynamic Computable General Equilibrium RHOMOLO of the Joint Research Centre, through an Administrative agreement between DG EMPL and the JRC-Seville.

All three Inter-Service Steering Groups assessed the quality of the final report of the external

contractors as satisfactory.

The conclusions and findings of the evaluation are considered as robust, within the limitations

and the mitigating measures described in section 4 and annex 3 of this staff working document.

118

The links to the reports are:

For TO8: https://ec.europa.eu/social/main.jsp?catId=738&langId=en&pubId=8330&furtherPubs=yes

For TO9: https://ec.europa.eu/social/main.jsp?catId=738&langId=en&pubId=8350&furtherPubs=yes

For TO10: https://ec.europa.eu/social/main.jsp?catId=738&langId=en&pubId=8346&furtherPubs=yes

76

ANNEX 2: STAKEHOLDER CONSULTATION

1. INTRODUCTION

This synopsis report outlines the consultation activities that were organised as part of the

evaluation of ESF support to (1) employment and labour mobility (excluding youth

employment), (2) social inclusion and (3) education and training for the 2014-2020

programming period and presents the main findings.

2. CONSULTATION STRATEGIES

2.1. Objectives

To ensure transparency and involve stakeholders as much as possible, the consultations were

organised in line with the standards and methods set out in the better regulation guidelines. As

set out in their respective roadmaps and strategies, all consultation activities aimed to collect

views and evidence from stakeholders on the visibility, outreach, usefulness, relevance and

effectiveness of the ESF support.

2.2. Consultation stakeholders

When identifying the stakeholders to consult, consideration was given to the impact of ESF

support (direct, indirect and potential) on individuals and organisations, and their degree of

involvement. The stakeholders targeted by the consultation were organisations or individuals

that:

• had or might have participated in the operations; or • had or might have run or been involved in running the operations; or • had an interest in social inclusion operations funded under the ESF; or • had expertise in the subject.

Stakeholders involved in managing the funds were consulted regularly. Member State

representatives involved in ESF monitoring and evaluation were informed of the progress of the

supporting study, the interim findings as well as the final findings and conclusions at the

Evaluation Partnership meetings held between February 2019 and June 2020.

The draft reports and individual country factsheets (both in the original English version and in

the national languages) were circulated to the evaluation partnership members and desk

officers working for the Commission’s Directorate-General for Employment, for comments. The

comments received were mostly to flag inconsistencies in data related to individual Member

States, which were subsequently corrected. All the supporting studies’ deliverables of (reports,

country factsheet and related annexes for example on methodology and mapping) were

uploaded on ‘CIRCABC’, the ESF Evaluation Partnership’s online workspace 119.

119

https://circabc.europa.eu/ui/group/58263cb1-f430-4a10-88c7-eaf436177c89/library/ba675037-3cd1-

402d-8170-57c4a269ad39?p=1&n=10&sort=modified_DESC

77

3. THEMATIC OBJECTIVE 8 - EMPLOYMENT AND LABOUR MOBILITY

3.1. The consultation strategy

3.1.1. Roadmap

The evaluation roadmap was published on the Commission’s ‘Have your Say’ portal and made

available for public feedback between 20 July and 17 August 2018120. No contributions were

received in this context.

3.1.2. Consultation stakeholders

The following groups were consulted:

(1) Participants: individuals supported by the European Social Fund operations were consulted to obtain a direct insight into: (i) the extent to which the operations met the “integration in the labour market and mobility” objective, and (ii) success factors that cannot be directly identified through the monitoring mechanisms.

(2) Organisations involved in the management of operational programmes such as: Member states, managing authorities at national and/or regional level, intermediate bodies and, social partners represented in the monitoring committee;

(3) Organisations involved in the delivery of ESF funded operations such as beneficiaries or project partners including public bodies, NGOs and local authorities. Their feedback helped to identify efficiency and delivery issues in the implementation of ESF operations.

(4) Organisations not directly involved in the operations but that, representing the people targeted by the ESF - more specifically people at risk of social exclusion - and can therefore provide a broader view of the needs and of issues faced by recipients and potential recipients.

(5) Academic and research bodies involved the field of employment and mobility, that could provide insight as to the relevance, effectiveness, efficiency, coherence or sustainability of measures;

(6) Organisations representing employers, that can give feedback on the factors that play a role in the successful integration of end recipients, and young people in general, in the labour market;

(7) General public: any individual or organisation that could provide useful views on how the EU can support employment and labour mobility and the other aims of thematic objective 8 (TO8).

3.1.3. Consultation methods and tools

To adequately reach these stakeholders, a range of consultation activities and methods were

used namely, a public consultation, over 60 targeted interviews (including interviews with EU

and national level stakeholders, 10 case studies121 which also included consultations with key

stakeholders in the form of in-depth interviews), and a validation process on the evaluation’s

120

https://ec.europa.eu/info/law/better-regulation/have-your-say/initiatives/1862-Evaluation-of-the-

support-to-employment-and-mobility-by-the-European-Social-Fund/public-consultation

121 Belgium, Germany, Greece, Spain, France, Malta, Portugal, Italy, Poland and Slovakia.

78

draft findings involving experts working together in focus groups in four Member States (Spain,

Croatia, Luxembourg & Romania) and one at EU level.

Table 1: Overview of the stakeholders reached through each consultation tool/method

Type of stakeholder Targeted

interviews

Public

consultation

Validation focus

groups

Participants x

Organisations involved in the management of

operational programmes x x x

Organisations involved in the delivery of the

operational programmes x x x

Organisations not directly involved in the

implementation of the operational programmes x x x

Academic and research bodies involved the field

of employment and mobility x x x

Organisations representing employers x x x

General public x

3.2. Specific consultations/activities

3.2.1. The public consultation

3.2.1.1. Description of the activity

The online public consultation ran from 14 October 2019 to 6 January 2020 and took the form of

a survey. The questionnaire comprised four strands of questions (excluding profiling and closing

sections), structured around the main evaluation questions and including both open-ended and

closed questions. Each strand was customised to one specific group of respondents.

3.2.1.2. Stakeholders

The public consultation had a separate section for each of the following groups of respondents:

• Group A: EU citizens who have benefited or are benefiting from ESF support (‘‘ESF participants’’);

• Group B: organisations that are or were involved in ESF and/or received support (i.e. managing authorities, intermediate bodies, beneficiaries, and social partners) that supposedly have a direct and detailed knowledge of ESF, although to varying degrees, depending on their specific function/role in the ESF (‘organisations involved’);

• Group C: EU citizens who did not benefit from ESF support (‘other citizens’); • Group D: organisations not involved in ESF and organisations that did not declare

their status or responded ‘I don’t know’ to the question ‘Have you been involved in or have you benefited from actions promoting employment and labour mobility supported by the European Social Fund?’ (‘organisations not involved’).

79

Figure 1: Breakdown of respondents by MS and group – TO8

Source: Final report of the public consultation on the evaluation of the EU support to employment and

labour mobility

The public consultation received 541 responses. There was an unbalanced geographical

distribution with four countries (Bulgaria, Croatia, Germany and Italy) responsible for around

60% of responses received. Almost 60% of the 541 respondents were women and around 40%

men. Of the 541 respondents, 53 (9.8%) were from Group A (ESF participants); 238 (43.9%)

were from Group B (organisations involved); 140 (25.9%) were from Group C (other citizens)

and 110 (20.4%) were from Group D (organisations not involved). Of this Group D, 79

organisations declared they were not involved in ESF. The remaining 31 either did not answer

the relevant question or said they did not know. Therefore, almost half of the public

consultation respondents were organisations that are or were involved in ESF and/or received

support. However, relatively few ESF participants responded to the public consultation. Figure 1

above summarises the respondents by country of reference and group.

80

3.2.2. Member State level interviews and focus groups

The fieldwork involved semi-structured interviews in all Member States and case studies in 10

Member States, covering 20 operational programmes and all TO8 investment priorities (except

youth employment).

Four focus groups were held in addition to the EU-level focus group. The types of stakeholders

consulted were managing authorities, beneficiaries, socioeconomic partners and

evaluators/researchers. The interviews and case studies were used as the main source of

information to answer the evaluation questions and feed into case study reports.

3.2.2.1. EU level validation focus group

The EU-level focus group took place on 3 February 2020 in Brussels. It aimed to discuss key

issues in some of the evaluation questions with a group of experts involved in European-level

policy debates on employment related matters and labour mobility, as well as with

representatives of managing authorities. Nineteen (19) representatives participated.

Discussions were facilitated by five consultants supplied by the contractor.

3.3. Overview of results of consultation activities

3.3.1. Effectiveness

For respondents that received ESF support, the most common benefit related to improvement

of the quality of the labour market position and help in looking for a job (approximately one in

five mentioned each of these), whereas very few said it helped with achieving a better work-life

balance and entering or going back to education and training. Respondents were also asked

about their current and past labour market situation. Before ESF support, 41.5% of 53

responding ESF participants were employed and after ESF support the employment rate rose to

84.9%. Likewise, 20.8% were unemployed before support and 7.5% after support.

Overall, most participating organisations agreed that ESF operations were effective in achieving

TO8 objectives, in particular: i) in promoting access to employment for jobseekers and inactive

people (89.9% of the 238 respondents agreed); ii) promoting self-employment,

entrepreneurship and business creation (81.1%); and promoting equality between men and

women (77.3%).

The objectives that were the least mentioned as being effective were the promoting work-life

balance and active and healthy ageing.

For helping jobseekers and inactive people enter the labour market, most respondents

mentioned training and support in finding a job as the most effective actions. On support to

self-employment, entrepreneurship and business creation, organisations mostly indicated

financial support/microcredit. To support gender equality in the labour market, organisations

considered flexible work arrangements and access to quality childcare to be the most

successful actions, whereas less than half of them considered training, gender mainstreaming

and awareness campaigns to be successful. To promoting active and healthy ageing, flexible

work arrangements and health and safety at the workplace were identified as effective,

although the number of respondents not knowing/not answering is particularly high.

81

Respondents consider career guidance and vocational training of individuals as most successful

measure to help workers and companies adapt to change, whereas awareness raising

campaigns were mentioned by less than a quarter of them. On the promotion of employment

support systems, strengthening cooperation between labour market institutions and

stakeholders, improvement of public employment services, and training of staff were

considered to be the most successful measures by the highest proportion of respondents,

whereas performance management and transnational partnership were mentioned least.

Finally, on the promotion of geographic and occupational mobility of workers, organisations

considered language training, recognition of qualifications and validation of competences and

acquired skills as successful.

On effective support to target groups, the majority of responding organisations believe that ESF

is successful in supporting job seekers and inactive people (77.6% of the 348 respondents

agreed or strongly agreed), and long-term unemployed (64.7% agreed). More than half also

believe that ESF effectively supports people leaving education without a qualification (58%), and

people at risk of social exclusion (56.5%). They seem to be more sceptical about ESF ability to

support migrants or individuals with a foreign background (less than half of respondents believe

ESF was successful in supporting them: 49.2% out of 348 respondents) or people in remote

areas (48.9%). Regarding people affected by poverty and homeless people, a number of

respondents said they did not know (33.4% and 46.1% respectively). Considering this, the

assessment seems to be particularly negative about the ESF’s effectiveness in reaching

homeless people with 36.6% saying it did not reach them vs 17.4% saying it did). Respondents

referring to Italy and Croatia tended to give a less positive assessment of the success of ESF in

addressing different target groups compared to the other countries.

Overall, most respondents seem to consider social media campaigns as the best channel of

information (70.4% of 541 the respondents), followed by information and awareness raising

events (44.5%). The least popular channels seem to be the distribution of flyers and

advertisements in newspapers. Most respondents said they learnt about the support measures

they benefited from through employment services and employment info centres (43.4%), and

less than a fifth through social media.

Regarding potential hindering factors for individuals, information and ability to reach out to

potential participants seemed to be the main factors. In fact, individual respondents who did

not take part in ESF support mostly said that they were not aware of it, followed by

respondents saying they did not know where to ask for information.

The focus groups discussions highlighted the fact that the ESF may not be very visible to the

public, as it is hard to ‘show’ the contribution of ESF given that it has an essentially human

dimension, as opposed to, the ERDF and the Cohesion Fund for instance, which deal with

infrastructure.

Organisations were also asked about factors that may hinder the implementation of actions and

approximately three quarters of them pointed to administrative burden for beneficiaries (76%

of the 238 respondents) and managing authorities. External factors, such as structural

problems (i.e. lack employment opportunities and low education and skills level of participants)

are also considered to play an important role (by 71.3%). Some organisations specified the types

82

of obstacles they have encountered in a related open question. A total of 33 responses

indicated cumbersome bureaucracy and lengthy procedures as the significant factors hampering

the effectiveness of ESF procedures, which supports the responses given in the closed question.

In addition, nine respondents mentioned that targets and measures are not carefully tailored to

needs, especially when it comes to people facing multiple barriers and discrimination. In the EU-

level focus groups, it was highlighted that both managing authorities and beneficiaries continue

to face a high administrative burden (because of both the ESF and CPR Regulations and the so

called ‘gold plating’ of rules at national level). This is particularly limiting for smaller

beneficiaries (e.g. NGOs representing vulnerable individuals) and discourages them to access to

ESF funds. Issues with inter-institutional cooperation were also highlighted especially in the case

of integrated pathways.

Regarding success factors, more than nine in ten responding organisations agreed that

personalised services and opportunities help promote participation (95.3% out the 238

respondents), followed by services that are in line with labour market needs (92.7%), flexibility

of duration (92.1%), flexibility in the way actions are implemented (91.5%) and financial benefits

for participants (89%).

In a related open question, 12 of the 40 that responded mentioned the importance of creating

partnerships, believing that they help bringing together different experiences to help design

meaningful solutions that address different needs, as well as tailored and holistic approaches

to support end users. Ten respondents also mentioned the quality of staff and dedication of

grant recipients and delivery partners.

The focus groups participants considered that the net effect of different measures, should not

to be judged by the seemingly low employment rates after the end of support, as this might not

reflect the strong net effects for individuals who are further away from the labour market. A

discussion on target groups highlighted the importance of tailored approaches, focusing too

much on target groups may cause certain individuals to be overlooked, notably those that do

not fit under pre-defined categories but are nonetheless in need of support.

Regarding employers, it was mentioned that it could be difficult to get employers to use

financial incentives unless they are aligned to the actual labour market needs.

It was also mentioned that ‘standard’ employment measures pose fewer obstacles to

implementation and that ‘newer’ types of measures might take more time to implement.

The importance of integrating the support process between regional and national level public

employment services was mentioned as an important ingredient. Regional and national level

public employment services sometimes serve the same clients, but refrain from sharing detailed

information to avoid the risk of duplication.

3.3.2. Efficiency

Most respondents consider training and education to be a cost-efficient measure (77.4% of the

541 respondents), followed by basic skills training and incentives for employers (73.6%). Open

suggestions to improve efficiency point include reducing bureaucracy and simplifying

procedures to reduce the cost of operations, increasing the flexibility of budget and

83

procedures, and involvement of final beneficiaries earlier and more deeply in the design of the

measures.

Administrative arrangements can have a bearing on the efficiency of operations and

organisations were asked their opinion on them. Approximately two-thirds considered the

communication arrangements appropriate. A smaller share (50%-60%), considered the

evaluation and project selection procedures, the application of simplified cost options, project

implementation and the reporting and monitoring arrangements to be appropriate. Some

respondents declined to answer ranged (12.9% on reporting and monitoring and 24.6% on

audit) on reporting and monitoring in particular, there seems to be room for improvement since

33.8% of the 348 respondents said they were burdensome or insufficient.

Opinions on the audit and management and control system arrangements were even more

negative. Less than half of the respondents considered them to be appropriate. However,

organisations involved in ESF – which arguable have a more informed opinion – were more

likely to consider the arrangements appropriate. Among these organisations, the highest share

of respondents saying that arrangements were burdensome were referring to for the

management and control system and reporting and monitoring (38.7% and 31.1% of the 238

respondents respectively). Among the four countries with the higher response rates,

respondents referring to Germany and Italy seem to be relatively more negative about the

management and control system compared to those in Croatia and Bulgaria and in general the

rest of respondents.

3.3.3. Relevance

Overall, respondents rated the usefulness of the various actions positively. In particular, almost

90% considered that actions to place a person in a job to be useful (88.9% out of the 541

respondents). Actions aimed at increasing the human capital strengthen the workforce

(improving the quality and welfare of human resources in companies and organisations, and

helping a person perform better in an existing job) were deemed useful, by approximately 80%

of the respondents. Actions promoting organisational change and increasing equality between

men and women received a less favourable assessment, 17% and 19.2% respectively said that

they have little or no usefulness. Organisations involved in ESF and ESF participants seem to

have a slightly more positive opinion on the usefulness of these actions. In addition, all actions

were generally considered to be useful by most respondents in the different countries. When

asked about additional useful actions, respondents mentioned further enhancing the VET

systems and improving training modules, developing better targeting mechanisms, and, a more

coherent EU-wide qualification scheme.

Approximately 70% of the 348 responding organisations believe that the ESF was able to adapt

to the evolving socio-economic conditions and participants’ needs. A higher proportion of

positive responses was received for Italy and Bulgaria and a lower proportion for Croatia and

Germany. A few respondents indicated that the ESF was flexible in terms of adjusting targets

and adapting to the changing needs of the target groups and less flexible in terms of excessive

procedures and requirements and cumbersome bureaucracy.

84

According to focus groups discussions, three types of support (not necessarily limited to ESF

support) can be useful for labour mobility: namely support for:

• The system and the network, including cross-cutting activities at EU-level, technical systems for information sharing, etc.

• Service providers, to support activities such as recruitment, outreach activities, information to employers, matching CVs with jobs, etc.

• Individuals and employers, for example regarding relocation, recognition of competences, language trainings or the creation of a ‘welcome package’.

Focus group members also made the point that the ESF’s support to labour mobility through

ESF seems limited. Although some countries use ESF funds for the EURES122 network, the

network is mainly supported through EaSI123 funds. The fact that ESF has traditionally had a

national focus also means that a somewhat lower level of attention has been paid to cross-

border activities.

It was suggested that it could be useful to encourage ‘return mobility’ and get workers to move

back to their country of origin with the new skills they acquired abroad. It was noted that third

country nationals have the same needs (in terms of language for example), so the same services

could be provided to all. However, funding streams are often separate for EU and third country

nationals, and more synergies could be exploited

3.3.4. Coherence

In this section, respondents often answered that they did not know or did not have an opinion.

For example when asked about the ESF’s coherence with existing national and regional

programmes, 40.8% and 72.7% of the 541 respondents respectively did not have an opinion).

Respondents were slightly more likely to have an opinion on the European Regional

Development Fund (ERDF), but less than half of them (45.1%) considered the ERDF to be

complementary with the ESF, with 41.3% saying they did not know and only 8.9% saying that

they were not aligned.

Focus groups participants considered that the coherence and complementarity of ESF’s TO8

with other funds had improved, particularly with the Youth Employment Initiative, the

European Globalisation Adjustment Fund (EGF) (where the ESF plays a ‘preventive’ action and

the EGF a ‘corrective’ one), and the AMIF and also vis à vis labour mobility. They also

mentioned high complementarity between the thematic objectives and synergy with the

European semester cycle of economic policy coordination and particularly the country-specific

recommendations. However, they also reported difficulties in the concrete integration of funds.

3.3.5 EU added value

Approximately half (47.7% out of the 541 respondents) believed that funding provided by the

ESF would not otherwise have been available (the volume effect). A further 28.5% said that

122

For more info on EURES check the home page at: https://ec.europa.eu/eures/public/en/homepage

123 EU Programme for Employment and Social Innovation (EaSI). More information is available at:

https://ec.europa.eu/eip/ageing/funding/EaSI_en

85

without EU support there would be less attention paid to unemployed people (the scope

effect). Finally 10.2% stated that ESF support does not make a difference (to national

programmes) and the rest (11.3%) said they didn’t know.

Similarly, most respondents believe that there is an ‘EU value added’ aspect to ESF TO8 support

to labour mobility. Just above 40 per cent (42.9% of the 541 respondents) believe that is

because there would be not enough money to pay for such actions, and the rest (27.7%) said

that without EU support there would be less or no attention paid to unemployed people. Finally

11.6% stated that ESF support does not make a difference and 16.3% did not know.

Unsurprisingly, in both cases, organisations and individuals that do not have first-hand

experience with the ESF were more likely to respond that they did not know or that ESF did not

make a difference.

Participants in the EU-level focus groups agreed that TO8 support has brought about clear EU

added value in terms of:

• providing additional resources; • covering new target groups; • creating opportunities to test innovative projects and upscale them (provided

there’s sufficient innovation readiness); • spreading good ‘policies’ across countries and steering attention to key challenges

also through country specific recommendations; • strengthening partnerships, including private public partnerships; • increased attention to social innovation and social entrepreneurship: ESF helps to

tailor the approaches, and test mechanisms and roll out successful methods; • increased inter-institutional cooperation.

4. THEMATIC OBJECTIVE TO9 – SOCIAL INCLUSION

4.1. The consultation strategy

4.1.1. Roadmap

The evaluation roadmap was published on the Commission’s ‘Have your Say’ portal and made

available for public feedback between 19 December 2018 and 16 January 2019124. Six

contributions were received. Two were anonymous and four came from organisations (ENSIE,

FEANTSA, EuroHealthNet, and the European Microfinance Network).They expressed support for

the evaluation but did not suggest changes to the roadmap.

4.1.2. Consultation stakeholders

The following individuals and groups were consulted:

(1) Participants: individuals supported by the European Social Fund operations were consulted to obtain a direct insight into: (i) the extent to which their own objectives

124

https://ec.europa.eu/info/law/better-regulation/have-your-say/initiatives/2030-Evaluation-of-the-

support-to-social-inclusion-by-the-European-Social-Fund

86

for participation have been achieved; and (ii) success factors that could not be directly identified through the monitoring mechanisms.

(2) Potential participants: individuals that did not received ESF support but who according to the ESF regulation might have qualified were consulted to obtain a direct insight into the reasons for their non-participation.

(3) Beneficiaries are the ultimate recipients of funds. They cover organisations actually delivering ESF support to participants and organisations using ESF support to enhance access to affordable, sustainable and high quality services of general interest, in particular in the fields of health care, employment and training services, services for the homeless, out of school care, child care and long-term care services. They were consulted to obtain a direct insight into: (i) the extent to which their own objectives for participation have been achieved; and (ii) success factors and barriers that could not be directly identified through the monitoring mechanisms.

(4) Potential beneficiaries are organisations that could have benefited from ESF support but did not. They were consulted to obtain a direct insight into the reasons for their non-participation.

(5) Organisations or institutions involved in the management of the operational programmes such as: Member States, managing authorities, intermediate bodies, organisations or institutions sitting on the monitoring committees.

(6) Organisations not directly involved but that represent the people targeted by the ESF - more specifically those at risk of social exclusion, poverty and discrimination - who can therefore provide a broader view of the needs and issues at stake with recipients and potential recipients.

(7) Social partners who could give feedback on important success factors;. (8) Academic and research bodies involved in the area of social exclusion, poverty and

discrimination - including the access to affordable, sustainable and high quality services of general interest – that could provide insights into the relevance, effectiveness, efficiency, coherence or sustainability of measures.

(9) General public: any individual or organisation that could provide useful views on how the ESF can social inclusion, and combat poverty and any discrimination (thematic objective TO9).

4.1.3. Consultation methods and tools

To reach adequately these stakeholders, a range of consultation activities and methods were

used, namely: a public consultation, targeted consultations (including interviews with EU and

national level stakeholders, case studies of 19 Operational Programmes in 11 countries125 which

also included in-depth interviews with key stakeholders, focus groups in each case study

Member State, and a Delphi survey to validate the evaluation’s draft findings with selected

experts.

125

Austria, Bulgaria, Cyprus, Spain, Hungary, Italy, Latvia, the Netherlands, Poland, Portugal and

Sweden

87

Table 2: Overview of the stakeholders reached through each consultation tool/method

Type of stakeholder

Targeted

interviews

(EU and

national

level)

Targeted

interviews

(case studies)

Public

consultation

Focus

groups

Delphi

Survey

Participants X

Potential participants X

Beneficiaries X X X X X

Potential beneficiaries X X

Managing authorities X X X X X

Organisations or institutions

involved in the management of

the operational programmes

X X X X

Organisations not directly

involved in the implementation

of the operations

X X X X

Social partners X X

Academic and research bodies X X

General public X

The consultation methods were implemented, in line with the agreed consultation strategy. The

only major deviation was that the focus group planned in Italy could not be held because it

coincided with the start of the COVID-19 pandemic. Instead an EU-level Delphi survey with

selected experts was organised to validate the draft findings and to gather their views on two

issues that came out from the field work, (i) the documentation of “soft-outcomes” and (ii) the

access to EU-funds for small, local organisations.

4.2. Specific consultations/activities

4.2.1. The public consultation

4.2.1.1. Description of the activity

The online public consultation ran from 26 September 2019 to 19 December 2019 and took

form of a survey. The questionnaire comprised six strands of questions (excluding profiling and

closing sections), structured around the main evaluation questions and including both open-

ended and closed questions. Each strand was customised to one specific group of respondents.

4.2.1.2. Stakeholders

A total of 574 individuals and groups responded. Bulgaria was the Member State with the

highest number of responses (123 representing 21.4% of the total), followed Hungary (82

responses- 14.3% of the total), and Croatia (79 responses= 13.8% of the total) from.

Importantly, responses were received from all 28 EU Member States. Five of the 574

respondents came from outside the EU.

88

Figure 2: Number of respondents to the public consultation for ESF support to social inclusion

by Member State.

Source: TO9 supporting study report Annex 7

The public consultation had a separate section for each of the following six groups of

respondents:

• Strand I: organisations with no knowledge of or involvement in ESF TO9; • Strand II: organisations with knowledge of ESF TO9 but not involved in its delivery:

e.g. NGOs and think-tanks; • Strand III: organisations involved in the delivery of ESF TO9: e.g. Managing

Authorities, Intermediate Bodies, beneficiary organisations or project promoters; • Strand IV: individuals who have received ESF TO9 support; • Strand V: individuals who have not received ESF TO9 support but who have

knowledge of it: e.g. academic experts; • Strand VI: individuals with no knowledge of or involvement in ESF TO9.

Both individual respondents and those responding on behalf of organisations were directed to

specific strands of questions corresponding to their level of knowledge or involvement with

ESF. The number of questions varied by strand. See Table 3 below for the number and

proportion oof respondents by group.

Table 3: Number of respondents by group

Strand No. of

responses %

Strand I: Organisations not aware of ESF 10 1.7%

Strand II: Organisations aware of ESF but not playing a direct role in the 59 10.3%

89

Strand No. of

responses %

delivery

Strand III: Organisations directly involved in the delivery of ESF 295 51.4%

Strand IV: Individuals having received ESF support 38 6.6%

Strand V: Individuals aware of the ESF but not having received support 132 23.0%

Strand VI: Individuals not aware of the ESF and not having received support 40 7.0%

Total 574 100%

Looking at the type of respondent, the 574 responses were split unevenly between

organisations (364 responses or 63% of the total) and citizens (210 responses or 37%). Of the

organisations, the most responses came from public authorities (168), followed by NGOs (70).

All respondents, except two, identified as EU citizens. More than 60% identified as female and

more than 80% were aged between 25 and 54 years old.

Of the 574 respondents, more than half responded on behalf of organisations that are directly

involved in ESF (295 or 51.4%). Of the individual respondents, the biggest group was those

aware of ESF but not having received ESF support (132 or 23%).

More than a third (136 out of 364) of those responding on behalf of organisations work in large

organisations (with 250 employees or more) that predominantly specialise in the labour market

and social inclusion area, of which just over 40% were involved in the management of EU funds.

On their involvement in ESF delivery, 41.9% (152 out of 364) identified as beneficiaries, having

received ESF funding for projects. A total of 110 responses (30.3%) were submitted on behalf of

Managing Authorities or Intermediate Bodies. Only a few of the 364 respondents (30 or 8.3%) in

the ‘organisations’ strand indicated that they did not play a no role in ESF delivery.

Turning to individual respondents’ experience or involvement with the ESF, nearly three

quarters of 210 respondents said that they had never received ESF support; i.e. had never taken

part in an ESF activity. 33 said they had recently received ESF support and only 5 said that they

are currently receiving ESF support. These results appear to coincide with the answers on socio-

economic status in that nearly three-quarters of individuals (155 out of 210 or 73.8%) say they

are employed full-time, and ESF social inclusion activities tend to target the unemployed or the

irregularly in employed.

A total of 25 respondents also attached a position paper.

4.2.2. Targeted interviews with stakeholders

A total of 141 interviews were carried out with EU and national level stakeholders, covering all

Member States. The types of stakeholders consulted included managing authorities,

beneficiaries, socioeconomic partners, evaluators/researchers and representatives of

participants.

4.2.3. Case studies

Twenty (20) operational programmes (OPs) in 11 Member States were selected for case study.

Within these 20 OPs, country experts - in consultation with the Managing Authorities -

90

identified a total of 62 projects that could be appropriate for an in-depth analysis. However, one

of the OPs was dropped from the list as an interview could not be organised with the Managing

Authority. Focus groups were held in each of these Member States except Italy due to the onset

of the Covid-19 pandemic.

OPs and projects were selected on the basis of criteria agreed with Commission during the

inception phase as well as consultations with the Commission and the Managing Authorities.

The interviews and case studies were the main source of information to answer the evaluation

questions. They also fed into case study reports.

4.2.4. Policy Delphi validation process

Instead of an eleventh focus group, a Delphi survey was conducted between 9 April 2020 and

13 May 2020. It aimed to gather a range of perspectives from experts involved in EU-level policy

debates on social inclusion, poverty and discrimination as well as from managing authorities not

consulted in the previous stages, one auditor and social partners. Invitations were sent to

representatives of 15 organisations of which a total of 10 participated in at least one round.

These were:

• Solidar • A Belgian ESF managing authority • National Audit Union of Social Cooperatives – Poland • European Roma Grassroots Organisations Network • Inclusion Europe • European Disability Forum (EDF) • EuroHealthNet • ENISE • ESN • European Network on Independent Living

4.3. Overview of results of consultation activities

4.3.1. Effectiveness

Overall, the vast majority of respondents from Strands II, III and V (80% of the 486) believe that

all ESF actions are either very useful or mostly useful. The most useful actions (>80%) are:

• basic skills training (e.g. social skills, IT, language) (90% or 438 respondents); • training and education (including vocational training) (89% or 433); • support to people with disabilities (e.g. promotion of community-based care) (86%

or 419); • support to overcome barriers to job search actions (e.g. transport or childcare) (85%

or, 413); • information, guidance, tutoring in the search for a job (85% or 412).

The organisations involved in the actual delivery (295 respondents) deemed that the most

successful actions related to the following target groups:

• Persons having a disability (61% or 180); • Unemployed for 12 months or more (60% or 178);

91

• Unemployed for less than 12 months (59% or 174); and • Low-skilled (57% or 168).

Most respondents said that these actions brought about job-related changes for participants,

improved their soft skills and self-confidence, and increased awareness of social inclusion

poverty and discrimination. Partnerships between managing authorities and NGOs and targeted

or individualised support are some of the main factors identified as contributing to the success

of ESF actions. Nearly all the respondents that had received ESF support said that their

expectations from participating in ESF activities had been partially or fully met. The most

frequently expectations were obtaining better job conditions, finding a job or being actively

included in society.

4.3.2. Efficiency

The efficiency question was only put to the Strand II and III respondents. Most (48-80% of the

354 answers) said that the various ESF activities are cost-effective (the resources invested are

proportionate to the results achieved), in particular: basic skills training; training and education;

information, guidance and tutoring in the search for a job; and internships and traineeships to

learn a trade. The only action deemed inefficient by a majority of respondents was ‘studies and

evaluations of institutions’.

Most organisations involved in the delivery of the ESF indicate direct, practical and

individualised support as the most cost-effective, followed by flexible and multidimensional

support, and support for integration to employment. The factors identified as contributing to

inefficiency, included the so-called ‘creaming’ effects (i.e. projects focusing on achieving set

targets rather than on real social change), and the standardisation and rigidity of ESF delivery

structures and timeframes.

4.3.3. Relevance

Most respondents agreed that the EU should be involved in promoting social inclusion and in

combating poverty and all kinds of discrimination.

92

Figure 3: What kind of support is provided with ESF funding to promote social inclusion,

combat poverty and any discrimination? Answers by strand III respondents.

N= 295 Source: TO9 supporting study report Annex 7

ESF actions to get people into employment were deemed relevant by the highest proportion of

respondents, regardless of their knowledge of or experience with ESF. Two-thirds of those that

had received ESF support mentioned having participated in ESF training and education actions

or had received information and guidance when searching for a job. Of those who answered on

behalf of organisations directly involved in ESF delivery, most (51% of the 295) said that their

support is directed at people who have been unemployed for 12 months or more. Many, (42%

of the 295) said it was also directed at people with a disability.

The review of position papers suggests that the ESF’s Thematic Objective 9 is still generally

considered to be appropriate, as it supports actions related to social inclusion and anti-

discrimination, which are as relevant today as they were in 2014.

4.3.4. Coherence

Overall, the ESF actions that promote social inclusion, combat poverty or discrimination were

considered to be coherent with other schemes, in particular with national, regional or local

programmes.

93

Figure 4: In your opinion to what extent are ESF actions promoting social inclusion, combating

poverty and any discrimination coherent with other schemes? Respondent strands II, III and V

N=486 Source: TO9 supporting study report Annex 7

Regarding EU schemes, Erasmus+ was most frequently identified as complementing and

reinforcing ESF actions promoting social inclusion or combating poverty and discrimination. A

trend noticeable in the review of position papers is that there will be scope under ESF+ to

further strengthen coherence with the European Pillar of Social Rights (on access to education

and sustainable employment) and with Interreg (on social innovation).

4.3.5. EU added value

An overwhelming majority of respondents indicated that one of the advantages of EU

interventions to promote social inclusion, combat poverty and discrimination is that they can

achieve more could be done than national or local resources only. More than half of the

respondents also thought that ESF measures allow new social issues to be covered and new

social services to be tested. Very few respondents thought that ESF or even EU-supported

interventions make no difference in promoting social inclusion or combating poverty and

discrimination.

68%

60%

52%

52%

34%

1%

2%

1%

4%

4%

6%

5%

3%

10%

27%

34%

43%

46%

58%

European Solidarity Corps

FEAD

ERDF/CF

Erasmus+

National, regional or local programmes

They complement or reinforce each other They do the same

They are contradictory / They hinder each other I do not know / I do not wish to answer

94

Table 4: In your view, what would be the advantage of having European Union interventions?

(broken down by Strand III respondents’ role in ESF delivery)

Beneficiary org.

receiving ESF

funding

(N=152)

MA / IB

(N=109)

Member of an

ESF Monitoring

Committee

(N=18)

EU funds

Coordinating

body

(N=12)

Certifying or

Audit

Authority

(N=4)

More could be done

than with national

or local resources

only

88% 84% 94% 50% 100%

New issues could

be covered 55% 62% 78% 58% 75%

New ways of

delivering services

could be tested

57% 58% 61% 33% 50%

None. I do not think

it would really

make a difference

2% 1% 0% 0% 0%

Other 3% 2% 6% 0% 0%

I do not wish to

answer / I do not

know

2% 6% 0% 17% 0%

N=295. Source: PC results; multiple answers possible.

4.3.6. Overall conclusion

The public consultation suggests that ESF support for TO9 is rated positively value by a wide

range of stakeholders – both individuals and organisations that are not fully aware of the ESF

and those who actively involved in its delivery. ESF actions are generally considered to be

effective and to offer value for money that could not be achieved with national or local

resources only. However, some replies highlight areas of concern and of potential

improvements, including 'creaming' effects that hinder the efficiency of actions and the scope

for greater coherence with the European Pillar of Social Rights.

5. THEMATIC OBJECTIVE TO10 – EDUCATION AND TRAINING

5.1. The consultation strategy

5.1.1. Roadmap

The evaluation roadmap was published on the Commission’s ‘Have your Say’ portal and made

available for public feedback between 21 December 2018 and 18 January 2019. Six replies

were received of which 3 were anonymous. The other three were from the organisations: ENSIE

(European Network of Social Integration Enterprises), Masaryk University (Czechia) and EAPN

Nederland (Non-governmental organisation). They expressed support for the evaluation but did

not suggest changes to the roadmap.

5.1.2. Consultation stakeholders

The following groups were consulted:

95

(1) Participants: individuals supported by the European Social Fund operations were consulted to obtain a direct insight into: (i) the extent to which their own objectives for participation have been achieved; and (ii) success factors that could not be directly identified through the monitoring mechanisms.

(2) Potential participants: individuals who did not received ESF support but who according to the ESF regulation might have qualified were consulted to obtain a direct insight into the reasons for their non-participation;

(3) Beneficiaries are the ultimate; and (ii) the success factors and barriers that could not be directly identified through the monitoring mechanisms.;

(4) Potential beneficiaries are organisations that could have benefited from ESF support but did not. They were consulted to obtain a direct insight into the reasons for their non-participation;

(5) Organisations or institutions involved in the management of the operational programmes such as: Member States, managing authorities, intermediate bodies, organisations or institutions sitting on the monitoring committees (see Table 1, rows 5 to 9)

(6) Organisations not directly involved in the implementation of the operations but that represent the people targeted by the ESF - more specifically those in education and training - and could therefore provide a broader view of the needs and issues faced by the recipients and potential recipients.

(7) Social partners who could give feedback regarding important success factors. (8) Academic and research bodies involved in the area of education and training that

could provide insights as to the relevance, effectiveness, efficiency, coherence or sustainability of measures.

(9) General public: any individual or organisation that could provide useful views on how the ESF can support education and training.

5.1.3. Consultation methods and tools

To adequately reach these stakeholders, a range of consultation methods were used, namely: a

public consultation, targeted consultations (including interviews with EU and national level

stakeholders, case studies of 20 Operational Programmes in 12 countries126 which also included

in-depth interviews with key stakeholders ), and a policy Delphi validation process on the

evaluation’s draft findings. The supporting study made use of a meta-analysis of six pre-existing

national ESF participant surveys127.

126

Poland, Portugal, Germany, Spain, Italy, United Kingdom, France, Czechia, Romania, Lithuania,

Ireland and Slovenia

127 Six specific ESF participant and beneficiary surveys on TO10 from six Member States (Spain, France,

Italy, Croatia, Lithuania and the UK).

96

Table 5: Overview of the stakeholders reached through each consultation tool/method

Type of stakeholder

Targeted

interviews

(EU and

national

level)

Targeted

interviews

(case studies)

Public

consultation

Policy Delphi

validation

process

Participants X

Potential participants X

Beneficiaries X X X X

Potential beneficiaries X X

Managing authorities X X X X

National and regional education

ministries and VET institutions X X X X

National and regional labour and

employment ministries and institutions X X X

European institutions involved in ESF or

related education and training policy

development

X X

Other relevant national, regional and

European bodies responsible for

management / implementation or match

funding of education, training or

employment funds

X X X X

Organisations not directly involved in

implementation/ associations representing

those in education or training

X X X

Social and economic partners at national

and EU level X X X

Research and academic organisations X X X

General public X

The consultation methods were implemented in line with the agreed consultation strategy.

There were some challenges in securing all the interviews for the targeted consultation as some

EU-level stakeholders declined to participate due to limited or no knowledge on ESF or were

non-responsive despite reminders. The impact of this on the consultation strategy was minimal,

however, as the targeted consultations allowed the in-depth views of key stakeholders to be

gathered.

97

5.2. Specific consultations/activities

5.2.1. Public Consultation

5.2.1.1. Description of the activity

The public consultation ran for 14 weeks and took the form of an online survey. The

questionnaire included both closed and open questions, addressing the key evaluation criteria,

and was tailored to the different stakeholder groups using a routing system.

5.2.1.2. Stakeholders

A total of 817 individuals and groups responded to the public consultation. The geographical

distribution was somewhat unbalanced with most responses (153) coming from Portugal,

followed by Spain (105), Poland (102), Italy (86) and Croatia (68). All other countries had fewer

than 60 respondents as shown in Figure 5.

Figure 5: Number of respondents by country of origin – TO10

The public consultation had a separate section for each of the following groups of respondents:

• Group A: ESF participants: individual EU and non-EU citizens receiving ESF support now or since 2014;

• Group B: Individual EU and non-EU citizens who did not take part in the ESF: ‘the general public’, reporting either having an awareness of the ESF or no direct knowledge of the ESF measures;

• Group C.1: Managing Authorities and Intermediate Bodies, organisations involved in the management of the ESF that supposedly have a direct and detailed knowledge of the ESF support;

98

• Group C.2: Beneficiaries: organisations or entities receiving ESF funding for the implementation of a project;

• Group C.3: EU Funds Coordinating bodies, Certifying/Audit authorities, Members of a Monitoring Committee directly involved in the delivery of ESF.

• Group D: Other organisations: organisations with no direct role in the management, monitoring and implementation of the ESF including civil society.

Of the 817 responses, the highest number (370 or 45.3% of the total) was from the group

representing organisations managing/delivering the ESF, followed by the general public (244

respondents or 29.9% of the total), ESF participants (141 - 17.3%) and other organisations (62 -

7.6%). Of the organisations managing/delivering the ESF, ESF beneficiaries provided the most

responses (227 or 61.4% of the total) followed by managing authorities and intermediate bodies

(104 responses or 28.1%), and EU funds coordinating bodies, certifying or audit authorities and

members of monitoring committees (39 responses or 10.5%). Figure 6 presents the breakdown

of respondents by stakeholder/profile group128.

Figure 6: Breakdown of respondents by country of origin and group – TO10

5.2.2. Targeted interviews with EU and national level stakeholders

Targeted interviews were conducted with: (i) EU=level stakeholders directly and indirectly

linked to the themes of education and training and (ii) national stakeholders involved in ESF

operations, primarily managing authorities and representatives of Education/Labour Ministries;

and (iii) other stakeholders where relevant. A total of 25 interviews were conducted between

October 2019 to March 2020 (See Table 6 below).

Table 6: Targeted interviews completed

Level Organisation

EU DG EMPL Unit F1 ESF Coordination

DG EAC group interview with:

Unit A2 Policy Strategy and Evaluation - Country analysis

Unit C2 Marie Skłodowska Curie actions

128

The figure only includes countries with at least 20 responses. The countries with fewer than 20

respondents were grouped together under ‘Other’.

99

Level Organisation

Unit B4 Erasmus+ Coordination

DG EMPL Unit E3 Vocational education, apprenticeships and adult learning

EU Lifelong Learning Platform

European University Association

Marie Curie Alumni Association (MCAA)

European Federation of Education Employers

European Parliament: EMPL Committee

European Student's Union

European Association of Institutions in Higher Education

European Trade Union Committee for Education

Business Europe

SME United

European Forum of Technical and Vocational Education and Training

European Vocational Training Association

Eurochild

Austria Ministry of Education Science and Research

Luxembourg ESF Managing Authority; Ministry of Education Childhood and Youth

Denmark Danish Business Authority (ESF MA)

Malta ESF Managing Authority Ministry for European Affairs and Equality

Greece ESF Managing Authority of OP Human Resource Management, Education and Lifelong Learning

Hungary ESF Managing Authority

Latvia Ministry of Education and Science

Sweden ESF Managing Authority

5.2.3. Case studies

Case studies of 20 Operational Programmes in 12 Member States were conducted as part of the

evaluation research.

An average of seven interviews were conducted per case study with stakeholders from the

following categories:

• officials in charge of the specific OP; • authority responsible for TO10 operations; • major beneficiaries of ESF TO10 operations; and

• other stakeholders, including evaluators of ESF T10 measures, vocational training organisations, agencies dealing with illiteracy and early leavers from education and

training, universities, etc.

The case study interviews took place between October 2019 and April 2020.

5.2.4. Policy Delphi validation process

A validation process involving workshops was planned in order to validate the findings of the

evaluation. Due to the COVID-19 restrictions, these workshops were replaced by online policy

Delphi consultations, one for each investment priority, which took place between 27 April and

27 May 2020. The consultees included thematic experts, policy experts from relevant

100

Commission departments, ESF managing authorities, ESF TO10 beneficiaries, social partners,

and EU-level NGOs.

During this process, 22 responses were received from a range of stakeholder groups as detailed

in Table 7.

Table 7: Respondents to the policy Delphi validation

Type of stakeholder Number of respondents

NGOs 6

Research and academic experts 4

National and regional education ministries and VET institutions 4

European institutions involved in education and training policy development 3

Social and economic partners at national and EU level 3

Beneficiaries 1

Managing authorities 1

5.3. Overview of results of consultation activities

5.3.1. Effectiveness

Overall, there was consensus among stakeholders that progress has been made in ESF TO10

operations, but differences in performance were noted between countries, OPs and IPs. ESF

participants’ views were positive: of the 114 who responded to the public consultation

effectiveness question 79 (or 56%) stated that ESF activities had fully met their initial

expectations and 25 (or 17.7%) said that they had surpassed them. Of the 141 respondents, 74

(52.5%) reported that without EU support there would be less or no opportunities for people in

their situation and 62 (44%) said that there would not be enough money to pay for such actions

without the ESF. Only 3.5% (5 of the 141 respondents) considered that the ESF does not make a

real difference. However, a notable minority of ESF participants felt that the support only

partially met expectations (26 of the 141 respondents or 18.4%), while 2 respondents (1.4%)

stated that the support did not meet their expectations.

Of the ESF participant surveys analysed, five out of six support this generally positive conclusion

on the impact of the ESF. EU-level and national stakeholders consulted in the policy Delphi

process also shared the view that the ESF’s TO10 has contributed to structural changes and

reforms of education and training systems at all levels. However, they also agreed that

significant improvements could be made upscaling ESF initiatives.

There is some discrepancy in views on the extent to which the ESF was effective in reaching all

target groups. Respondents replying to the public consultation on behalf of organisations felt

that, overall, ESF operations were ‘very’ or ‘mostly’ successful in providing support to a large

number of target groups. Of the 104 managing authorities/intermediate bodies that

participated, 69 (or 66.3%) considered that the most effectively reached groups were:

individuals with no or low qualifications and skills had been effectively reached; , 62 (or 59.6%)

said the same about people with disabilities and 60 (or 57.7% - 60 out of 104) said the same

about older workers. Of the 227 replying on behalf of beneficiary organisations, 133 (or 58.6%)

considered teachers and trainers at all levels to have been reached effectively, while 126

101

(55.5%) said the same about students (, 118 (52%) about individuals with no or low

qualifications or skills (, and 115 (50.7%) individuals with disabilities.

A small proportion of respondents who felt that ESF support was ‘mostly’ unsuccessful was low

(10%or less) for all target groups listed. However, EU-level and national stakeholders consulted

in the targeted interviews and in the case studies, highlighted several challenges in reaching

some of these target groups, particularly the individuals who are most disadvantaged. Older

workers, the low-skilled, Roma, refugees and migrants, the long-term unemployed and the

disabled were identified by EU-level and national stakeholders as not having been sufficiently

reached, both in the interviews and the policy Delphi responses. In the latter, respondents said

that measures risked focusing on ‘low hanging fruit’, supporting people who need only minimal

support anyway and leaving aside the more problematic cases.

By contrast, the factors increasing the effectiveness of ESF TO10 operations were generally

agreed upon by all stakeholders responding to the public consultation’s qualitative open

questions and being interviewed. They mentioned in particular: (i) operations that bring about

systemic change and that are strategically planned; (ii) strong coordination and the involvement

of local and regional stakeholders and non-public partners; and (iii) operations that focus on

individuals and their specific support needs. Reducing the administrative burden and simplifying

the governance structure of the fund was also frequently mentioned by stakeholders in the

open questions and the targeted interviews. Respondents to the policy Delphi and the targeted

interviews also strongly agreed that vulnerable groups could be reached more effectively by

increasing the participation of civil society organisations, NGOs and local stakeholders.

5.3.2. Relevance

There was strong consensus among stakeholders that TO10 investment priorities and planned

operations have been and remain relevant to education and training needs. Of the 71

respondents that answered the relevant question 58 (82%) agreed that the ESF should be

involved in improving skills and supporting the development of education and training. Only 5

(or 7%) disagreed and would prefer to support education and training on the national or local

level (the remaining 11% did not know or did not wish to answer). Survey respondents also

pointed to the relevance of ESF operations. This is supported by EU and national stakeholders

who all considered that ESF TO10 operations were relevant to the education and training needs

in Member States. EU-level stakeholders said that the breadth of the investment priorities (IP)

contributed to this relevance, as it allowed a range of operations for different education levels

and groups in need to be programmed. EU-level and national and stakeholders also cited this as

a reason for the continued relevance of ESF TO10 operations throughout the programming

period.

According to the public consultation, the most relevant actions included: (i) support for the

unemployed to learn new skills to help find work; (ii), support to link tertiary education and

training institutions with businesses and with local communities; and (iii) support for students

with fewer opportunities to complete higher education scholarship studies. Those consultees in

the case studies largely agreed, highlighting in particular the relevance of upskilling and

reskilling measures. EU stakeholders stressed that all investment priorities, but in particular

IP10.iii (lifelong learning) and IP10.iv (labour market relevance) are increasingly relevant given

102

the need to ensure the whole population is equipped with the right skills for a changing world of

work.

However, stakeholders involved in IP10.iii and IP10.iv operations pointed out in the policy

Delphi a decrease in partnership involvement, especially in VET, during the programming

period, citing this as an obstacle to the relevance of operations. The need for employers, trades

unions and NGOs to be more involved was stressed, and also emerged strongly in the targeted

interviews, particularly from EU level stakeholders. There was consensus across a range of

stakeholders that this was necessary to increase relevance of ESF TO10 to the needs of

vulnerable groups, in particular.

5.3.3. Efficiency

There was a strong overall consensus among stakeholders that most of the ESF activities are

cost-effective. Of 17 activities presented in the public consultation survey, 13 were considered

cost-effective (more than 50% of respondents agreed or strongly agreed was the case). In all

subgroups that were asked about the administrative burden, most respondents said the

arrangements were appropriate: 62.3% of managing authority respondents (65 of 104), 59.5%

of beneficiary organisation respondents (135 of 227) and 56.1% of certifying, coordination and

monitoring body respondents (22 of 39).

However, the management and control system and reporting and monitoring were judged as

the most burdensome arrangements by more than a third of respondents to the question How

would you qualify the following administrative arrangements for the implementation of

operations?’ (370 total respondents). This is reflected in the views of EU-level and national

stakeholders consulted. They all mentioned ESF regulatory requirements – with most

highlighting the heavy reporting requirements – as posing challenges to efficient

implementation. There is strong consensus that these heavy requirements are particularly

burdensome for smaller beneficiaries. Stakeholders consulted in the targeted interviews, and

through the case study interviews shared the view that voluntary organisations and NGOs

involved in social inclusion activities find it more difficult to deal with the funding and data

collection requirements. On the other end of the scale, communication arrangements were the

most commonly deemed appropriate across respondents (by 264 of 370 or 71% of

respondents).

The vast majority of national stakeholders interviewed for the targeted consultations and the

case studies stated that simplified cost options (SCOs) help improve efficiency. Respondents to

the policy Delphi validation agreed. However, these stakeholders also identified challenges in

implementing them. Several mentioned that developing SCOs still involves a heavy

administrative burden, and one that takes time to implement. Stakeholders spoke about delays

from the European Commission in approving SCOs proposed by Member States. Public

consultation respondents considered the application of SCOs to be the most inefficient

administrative arrangement although several national respondents said that SCOs help simplify

project preparation.

Several respondents to the public consultation suggested to focus more on qualitative

indicators rather than quantitative measures to assess the results of the ESF measures. This was

103

also expressed by some national stakeholders, who suggested that results indicators should be

more relevant to the IP goals.

5.3.4. Coherence

Overall stakeholders expressed the view that the ESF’s TO10 operations are generally coherent

with other EU funding programmes, though views vary significantly per fund. There is consensus

that Erasmus+ and ERDF are the funds that are most coherent with ESF. Of all funds, Erasmus+

was rated as the most coherent with TO10 by the highest proportion of respondents, 62.3% of

the general public (107 of 173); 63.8% of organisations managing ESF (236 of 370); and 74% of

other organisations (34 of 46), followed by the ERDF. These two funds were also the most

frequently highlighted in the targeted interviews.

However, many of the stakeholders interviewed felt that coherence was a challenge during

implementation. This was confirmed by respondents to the policy Delphi validation. There was

agreement on the complexity of having to apply under both funds for a joint project, as well as

on the factors influencing coherence, namely: (i) the lack of cooperation between institutions

responsible for each fund on the national level; (ii) different implementation and delivery

modes (direct vs shared management); (iii) the difference in target groups of the funds; and (iv)

insufficient communication to potential beneficiaries on the possibility of combining funds.

Views on coherence with other EU funding instruments are more mixed. Of the members of

the public that replied to the public consultation, many did not know or did not wish to provide

an answer on whether ESF activities are coherent with a number of other schemes (i.e. COSME-

112 of 173 (65%), EURES- 110 of 173 (64%), AMIF- 102 of 173 (59%), other ESI Funds and

Horizon 2020/MSCA 100 of 173 (58%)). The EU-level and national stakeholders interviewed

were also less able to comment on the degree of coherence of ESF with the Marie Skłodowska

Curie Actions and Horizon 2020. However, the views expressed though they appear to agree on

the limited coherence with Horizon 2020 (MSCA). Stakeholder views from targeted interviews,

case study interviews and the policy Delphi validation showed that coherence between ESF and

MSCA was challenging in practice and only 29% from the general public and organisations

managing/delivering ESF felt that there was coherence between the two funds.

5.3.5. EU added value

The perception of EU added value resulting from ESF TO10 operations was positive overall

among the range of stakeholders consulted. The vast majority of ESF participants responding

to the public consultation agreed that having ESF support made a difference, though

organisational respondents tended to be more positive. 323 out of the 370 organisation

respondents (or 87.2%) indicated that more can be done than with national or local resources

only. 73.9% of respondents from the general public (128 of 173) agreed. Other positive aspects

highlighted were that new issues can be covered 41% of general public (71 of 173) and 61.8% of

organisations (229 of 370) and that the ESF enables experimenting with new ways of delivering

services (60 of 173 (39.3%) of general public and 212 of 370 (57.2%) of organisations)

A majority of stakeholders interviewed shared this view, highlighting in particular that ESF TO10

supported disadvantaged groups that would have otherwise not been targeted. Both EU-level

and national stakeholders confirm that the ESF TO10 operations produced effects at national

104

and/or regional level that would not otherwise have taken place. However, a minority of

stakeholders interviewed, coming particularly EU-level stakeholders, expressed the view that

the ESF TO10 support to innovation in education and training has been limited. This is also

reflected in the public consultation responses where ‘ESF enables experimenting with new ways

of delivering services’ was the least selected advantage of ESF TO10 operations by the general

public and individuals aware of the ESF but not receiving support (68 out of 173 respondents or

39.3%). Respondents to the policy Delphi agreed, highlighting that where innovation is achieved

through the ESF it needs to be better embedded into national legislation to ensure spill over

effects of the good practice.

105

ANNEX 3: METHODS AND ANALYTICAL MODELS

1. OVERALL APPROACH TO THE EVALUATION WORK

In line with the Better Regulation guidelines, DG EMPL decided to base this evaluation on the

work carried out by external evaluators and took the following approach:

• collect and analyse the relevant evidence; • provide answers to all evaluation questions; • present evidence-based conclusions.

COVID-19 pandemic: It shall be noted that the fieldwork for these evaluations was carried out before the COVID-19 pandemic outbreak reached Europe. The Corona Response Investment Initiative (CRII) will affect the support to Employment for the remainder of the current implementation period and the proposals for the next programming period will also aim at mitigating the consequences of this pandemic. This COVID-19 pandemic is a major negative shock to the global and European economy. Already at the end of March of 2020, a substantial negative economic impact on Europe has materialised, at least for the first half of 2020 and possibly longer if the pandemic is not contained rapidly. For the future, the degree of the negative outlook will depend on a number of parameters such as the lack of supply of critical materials, the effectiveness of containment measures, the downtime in manufacturing in the EU work days lost in companies and public administrations, and demand effects (e.g. mobility restrictions, travel cancellations).

2. RATIONALE OF THE EVALUATION

The main objective of this evaluation is to assess the effectiveness, efficiency, coherence and EU

value added of ESF support to the thematic objective 8, 9 and 10. The evaluation includes both

an individual assessment of each country and a cross-cutting and comparative assessment.

External contractors collected and examined data covering 2014-2018 and prepared three

studies129.

3. EVALUATION QUESTIONS AND STRUCTURE OF THE REPORT

The evaluations were based on the following overarching evaluation questions:

• Effectiveness: How effective has the ESF been in achieving the objectives of thematic objective 8/9/10?

• Efficiency: How efficient has the ESF been in the achievement of the objectives of thematic objective 8/9/10?

• Relevance: How relevant have the operations funded by the thematic objective 8 been? How relevant are the ESF operations under thematic objective 9/10?

129

Supporting studies:

TO8: https://ec.europa.eu/social/main.jsp?catId=738&langId=en&pubId=8330&furtherPubs=yes; TO9:

https://ec.europa.eu/social/main.jsp?catId=738&langId=en&pubId=8350&furtherPubs=yes; TO10:

https://ec.europa.eu/social/main.jsp?catId=738&langId=en&pubId=8346&furtherPubs=yes

106

• Coherence: How coherent have the operations funded by thematic objective 8/9/10 been among themselves and with other actions in the same field?

• EU Value added: What is the EU added value of the ESF-funded operations in the field of employment and mobility / of social inclusion, combating poverty and anti-discrimination / of education?

• Sustainability: How sustainable the ESF-funded operations under TO8 have been?

Each evaluation question was further detailed in sub-questions presented in Table 9 below. The

evaluation work and the report were structured around the conclusions reached on each of the

questions.

4. METHODOLOGY AND DATA SOURCES

The evaluation was based on a complex methodology aimed at collecting solid evidence and

providing well-informed answers to the evaluation questions. This included:

• Desk research: o analysis of the monitoring data in the SFC2014 database (System for Fund

Management in the European Union); o in-depth analysis of the operational programmes (OPs) and Annual

Implementation Reports, and specifically the 2018 Annual Implementation Reports that include reported data as of 31st December 2018;

o The mapping of investment priorities, target populations and types of operations, based on the SFC2014 database, operational programmes, Annual Implementation Reports and related documentation, including the Commission’s Country-specific Recommendations.

o econometric analysis and provision of data to the Joint Research Centre (JRC) for the RHOMOLO simulations;

o literature review (policy documents, regulations, national and EU-wide evaluations, ESF websites and publications);

o Synthesis of evaluation reports by member states using the database of evaluations compiled by the Evaluation Helpdesk, Annual Implementation Reports and additional desk research.

• Public consultation: assisting the Commission in developing the questionnaire, and analysing the results of the public consultation.

• Cost-effectiveness analysis using a combination of SFC2014 programme data (for unit costs), relevant counterfactual impact evaluations, feedback from the stakeholders both through the public consultation as well as EU level focus group, case studies and a study of micro-data where available. In addition, first attempts were made to conduct cost benefit analyses for TO9 (Social inclusion) and TO10 (education and training).

• Case studies: addressing the research questions at national and regional level, based on desk research, interviews and focus groups.

• Interviews with desk officers, Managing Authorities and other stakeholders, in particular for the case studies.

• Holding focus groups in the Member States and at the EU level or Delphi surveys when the EU level focus group has not been possible due to the Covid-19.

5. OVERALL ASSESSMENT OF STRENGTHS AND WEAKNESSES OF THE METHODOLOGY AND DATA

The various sources were triangulated, and where the evidence was insufficient or inconclusive,

the approaches were combined: data-based, documentary and perception-based sources as

107

well as quantitative and qualitative techniques, depending on the nature of the evaluation

question and the strengths of the relevant data and approaches.

The evaluation drew on a number of sources providing opinions and perceptions, including

surveys, interviews and the public consultation. Data on opinions and perceptions are important

evidence where evaluation questions elicit the views of stakeholders. In some cases,

respondents or interviewees may be the only source of knowledge or witness accounts of

events when no other sources are available.

Under the Better Regulation guidelines, the public consultation is an important tool for

collecting stakeholder input and views on EU policy initiatives. It cannot be expected to provide

a fully representative view of EU public opinion, but it does offer a channel for the people who

care about a given issue to voice their opinion. As expected, respondents’ knowledge and

involvement in running employment operations differed, but the design of the public

consultation made it possible to distinguish between the respondents who were well-informed

and others. This distinction fed into the analysis of the responses, and triangulated with other

sources of evidence.

In conclusion, while acknowledging that there are some data gaps and methodical limitations,

the evaluation presents well-informed, evidence-based and reliable answers to the questions,

to the extent possible.

108

Table 1: Evaluation questions and sub-questions for each specific supporting study

TO8 – Employment and labour mobility TO9 – Social inclusion TO10 – Education and training Section in

the staff

working

document

Data

and info

sources

How effective has the ESF been in achieving the

objectives of thematic objective 8?

How effective has the ESF been in

achieving the objectives of thematic

objective 9?

10 How effective has the ESF been in achieving

the objectives of thematic objective 10?

6.1 (1) (2)

(3) (4)

(5)

1.1. To what extent have the financial implementation

and the achievement of the expected outputs

progressed according to the targets set in the

programmes? What are the main factors involved

(delays in implementation, ESF absorption…?

1.1. To what extent have the financial

implementation and the achievement of the

expected outputs progressed according to

the targets set in the programmes? What

were the main factors involved (delays in

implementation, ESF absorption…)?

1.1. To what extent have the financial

implementation and the achievement of the

expected outputs progressed according to the output

targets set in the programmes? What are the main

factors involved (delays in implementation, ESF

absorption…)? In particular, what is the progress

made by ESF on target indicators in higher

education and secondary education among all

countries?

6.1.1 (1) (2)

(3) (5)

1.2. How and to what extent does ESF contribute to the

achievement of the general objectives thematic

objective 8? In particular, to what extent have the ESF

operations, contributed to the positive evolution of the

employment situation, the labour mobility, the self-

employment and the other objectives of thematic

objective 8 up to now? How did it contribute to

addressing problems faced by target groups?

1.7. To what extent has the ESF contributed to

structural changes in national education systems,

vocational training systems, public employment

systems or employment policies?

1.8. What was the concrete contribution of the TO8

operations to the promotion of gender equality and

active ageing which are also objectives of TO8?

1.2. How and to what extent does ESF

contribute to the promotion of social

inclusion, combating poverty and

discrimination and the social inclusion

target of Europe 2020?

1.2. How and to what extent does ESF contribute to

the investment in education, training and vocational

training for skills and lifelong learning and

education targets of Europe 2020?

6.1.2 (1) (2)

(3) (4)

(5)

1.3. To what extent were the target groups reached by 1.3. How were relevant national strategies 1.3. How were relevant national strategies and 6.3 (1) (2)

109

TO8 – Employment and labour mobility TO9 – Social inclusion TO10 – Education and training Section in

the staff

working

document

Data

and info

sources

the operations, including disadvantaged persons,

especially those from marginalized communities and

those leaving education without qualifications? To

what extent higher education institutions/universities

were beneficiaries of TO8 investments (in terms of

numbers and size projects)? Have any Member States

invested in developing graduate tracking systems or

graduate tracer studies or similar measures for

measuring outcomes of graduates on the labour

market?

and policy contexts and challenges

translated into operations?

policy contexts and challenges translated into

operations?

(3) (5)

1.4. What was the quality and timeliness of

employment obtained?

1.5. Which types of operations were the most effective

and most sustainable, for which groups and in which

contexts (e.g. more developed, less developed and

transition regions, urban and rural areas etc.)?

1.4. Which changes (intended and

unintended) did the ESF support bring to

the target groups? How were these

changes, notably soft outcomes, assessed

and documented? Which types of

operations are or were the most effective

and most sustainable, for which groups and

in which contexts?

1.4. Which changes (intended and unintended) did

the ESF support bring to each specific target group,

e.g. young people in initial education and training,

NEETs, low qualified adults, people in

employment, unemployed, inactive? Which types of

operations are or were the most effective and most

sustainable, for which groups and in which

contexts? In particular, did the ESF support impact

equally on groups with advantaged and

disadvantaged backgrounds? To what extent was

access to the different sectors of education (e.g.

higher education) for underrepresented groups

facilitated thanks to ESF? What kind of efficiency-

enhancing measures did ESF finance in higher

education as part of investment priority TO10(ii)?

To what extent have non-formal pathways been

used to prevent and address early school leaving

and support re-integration in education and

training?

6.1.3 (1) (3)

(5)

1.6. What main factors (geographical, socioeconomic,

organisational…) had a bigger impact on the

effectiveness of ESF operations under thematic

objective 8, by type of operation?

1.5. Which factors facilitate or hinder the

effectiveness of ESF operations under

thematic objective 9, by type of operation?

1.5. Which factors facilitate or hinder the

effectiveness of ESF operations under thematic

objective 10, by type of operation and by main

target group?

6.1.4 (1) (3)

(5)

110

TO8 – Employment and labour mobility TO9 – Social inclusion TO10 – Education and training Section in

the staff

working

document

Data

and info

sources

How efficient has the ESF been in the achievement

of the objectives of thematic objective 8?

How efficient has the ESF been in the

achievement of the objectives of

thematic objective 9?

How efficient has the ESF been in the

achievement of the objectives of thematic

objective 10?

6.2 (1) (2)

(3) (5)

2.1. To what extent were operations cost-effective?

What types of operations were more and less cost-

effective? In what contexts? What were the

determining factors?

2.2. Are there significant cost differences between

Member States/Regions in the implementation of the

operations? What are these differences related to?

2.1. To what extent were operations cost-

effective? What types of operations were

more and less cost-effective? In what

contexts? What were the determining

factors?

2.2. Are there significant cost differences

between Member States/Regions in the

implementation of the operations? What

are these differences related to?

2.1. To what extent were operations cost-effective?

To what extent are the costs of the intervention

proportionate, given the changes/effects it has

achieved? What types of operations were more and

less cost-effective for which target group? In what

contexts? What were the determining factors?

2.2. Are there significant cost differences between

Member States/regions in the implementation of

similar operations? What are these differences

related to?

6.2.1 (1) (2)

(3) (5)

2.3. To what extent were the organisational

arrangements, including management and control

systems at all levels conducive to the effectiveness of

operations? Was there administrative burden, in

particular gold plating involved?

2.4. In particular, how timely and cost-efficient were

the procedures for reporting and monitoring?

2.3. How do organisational arrangements

influence service delivery by beneficiaries

or, eventually, lead to non-take up by

potential beneficiaries? To what extent is

non-take-up a choice or due to non-

awareness of the instrument?

2.4. To what extent were the organisational

arrangements, including management and

control systems at all levels, conducive to

the effectiveness of operations? Is there

gold plating? Were the procedures for

reporting and monitoring timely and

efficient?

2.3. How do organisational arrangements influence

service delivery by beneficiaries or, eventually, lead

to non-take up by potential beneficiaries? To what

extent is non-take-up a choice or due to non-

awareness of the instrument?

2.4. To what extent were the organisational

arrangements, including management and control

systems at all levels, conducive to the effectiveness

of operations? Is there gold plating16? Are there

opportunities to simplify the national legislation or

reduce unnecessary regulatory costs without

undermining the intended objectives of the

intervention? Were the procedures for reporting and

monitoring timely and efficient?

6.2.2 (1) (3)

(5)

2.5. How visible were TO8 funded operations? 2.5. How visible are ESF funded operations

under TO9?

2.5. How visible are ESF funded operations under

TO10

6.2.3 (1) (3)

(5)

How relevant have the operations funded by the

thematic objective 8 been?

How relevant are the ESF operations

under thematic objective 9?

How relevant are the ESF operations under

thematic objective 10?

6.3 (1) (3)

(4) (5)

3.1. To what extent were the objectives and the

operations funded by the ESF relevant to the needs of

3.1. Were the objectives and the operations

funded by the ESF relevant to the needs of

3.1. Were the objectives and the operations funded

by the ESF relevant to the needs of target groups?

6.3.1 (1) (3)

(4) (5)

111

TO8 – Employment and labour mobility TO9 – Social inclusion TO10 – Education and training Section in

the staff

working

document

Data

and info

sources

target groups?

3.3. Were the most relevant groups, in the different

socio-economic contexts (e.g. more developed, less

developed and transition groups; urban and rural areas

etc.), targeted starting from the design stage? Were the

most important needs of these groups addressed?

3.4. Were the TO8 operations undertaken suitable to

support active ageing and gender equality? How the

TO8 operations implemented fall within the broader

context of TO8?

target groups? How were the different

target groups prioritised and the actions

tailored to their specific needs? Were the

most important needs of these groups

addressed?

3.2. Were the most relevant groups, in the

different socio-economic contexts targeted

starting from the design stage? How was

the partnership and multi-level governance

implemented?

To what extent are they still relevant? How were

the different target groups prioritised and the

actions tailored to their specific needs? Were the

most important needs of these groups addressed?

3.2. Were the most relevant groups, in the different

education and socio-economic contexts targeted

starting from the design stage? How was the

partnership and multi-level governance

implemented?

3.2. To what extent were OPs flexible and able to adapt

to changes in the implementation context, notably the

evolution in the situation of employment and mobility?

3.3. To what extent were OPs flexible and

able to adapt to changes in the

implementation context or political

priorities?

3.3. To what extent were OPs flexible and able to

adapt to changes in the implementation context or

political priorities?

6.3.3 (1) (3)

(5)

How coherent have the operations funded by

thematic objective 8 been among themselves and

with other (actions in the same field?

How coherent are the operations funded

by thematic objective 9 among

themselves and with other actions in the

same field?

How coherent are the operations funded by

thematic objective 10 among themselves and

with other actions in the same field?

6.4 (1) (3)

(5)

4.1. In which manner were the ESF operations of TO8

complementary with each other? What were the main

factors in this regard?

4.2. Were the ESF operations of TO9

complementary with each other and with

interventions under other thematic

objectives? What were the main factors in

this regard?

4.2. Were the ESF operations of TO10

complementary with each other and with

interventions under other thematic objectives? What

were the main factors fostering and/or hindering

complementarity?

6.4.1 (1) (3)

(5)

4.2. To what extent were they complementary and

coherent with other thematic objectives and with other

programmes (e.g. EURES, EaSI, Erasmus+, ERDF,

EAFRD, EMFF, EGF…) oriented to employment and

mobility at the EU level?

4.3. To which extent are the investments under TO8

contributing and how consistent these are with the

National Reform Programmes and, the Country

Specific Recommendations in the framework of the

4.1. Were ESF interventions in line with

EU policies on social inclusion?

4.3. To what extent were they

complementary and coherent with other

EU funding instruments such as ERDF,

EARDF, EaSi, FEAD or AMIF?

4.4. To which extent are the investments

under TO9 consistent with the analyses and

priorities identified in the context of the

4.1. Were ESF interventions in line with EU

policies on education and training? To which extent

are the investments under TO10 contributing to

other EU initiatives such as the New Skills Agenda

for Europe? To what extent did they take advantage

of or were aligned to EU policy tools, such as the

Europass documents, the European Qualifications

Framework (EQF), the European Quality Assurance

Framework in Vocational Education and Training

6.4.2 (1) (3)

(5)

112

TO8 – Employment and labour mobility TO9 – Social inclusion TO10 – Education and training Section in

the staff

working

document

Data

and info

sources

European Semester?

4.4. To what extent were they complementary and

coherent with other activities supporting employment

and mobility at national/regional level?

European Semester notably in the Country

Reports, the National Reform Programmes

and, the Country Specific

Recommendations?

4.5. To what extent were they

complementary and coherent with other

activities supporting social inclusion and

combating poverty and discrimination at

national/regional level?

(EQAVET) and the European Credit transfer

system for Vocational Education and Training

(ECVET)?4.3. To what extent were they

complementary and coherent with other EU funding

instruments such as ERDF, EARDF, ERASMUS+

or AMIF?

4.4. To which extent are the investments under

TO10 consistent with the analyses and priorities

identified in the context of the European Semester

notably in the Country Reports, the National

Reform Programmes and the Country Specific

Recommendations?

To which extent fulfilling ex ante conditionalities in

the different sectors led to structural reforms or

legislative changes in those countries/regions that

had to work to fulfil them?

4.5. To what extent were they complementary and

coherent with other activities supporting education,

training and vocational training at national/regional

level?

What is the EU added value of the ESF-funded

operations in the field of employment and mobility?

What is the EU added value of the ESF-

funded operations in the field of social

inclusion, combating poverty and anti-

discrimination?

What is the EU added value of the ESF-funded

operations in the field of education?

6.5 (1) (3)

(5)

5.1. To what extent did the ESF-funded operations

produce effects at the national and regional level that

would not have taken place without the EU

intervention? This question shall be addressed from the

following perspectives:

5.1. To what extent did the ESF-funded

operations produce effects at the national

and regional level that would otherwise not

have taken place without the EU

intervention? This question shall be

addressed from the following perspectives:

5.1 To what extent did the ESF-funded operations

produce effects at the national and regional level

that would otherwise not have taken place without

the EU intervention? This question shall be

addressed from the following perspectives:

6.5 (1) (3)

(5)

Volume effect: Have the operations added to existing

actions or directly produced beneficial effects that can

be measured in terms of volume?

Volume effect: Have the operations added

to existing actions or directly produced

beneficial effects that can be measured in

Volume effect: Have the operations added to

existing actions or directly produced beneficial

effects that can be measured in terms of volume?

6.5.1 (1) (3)

(5)

113

TO8 – Employment and labour mobility TO9 – Social inclusion TO10 – Education and training Section in

the staff

working

document

Data

and info

sources

terms of volume?

Scope effect: Have the operations broadened existing

actions by addressing groups or policy areas that would

otherwise not have been addressed?

Scope effect: Have the operations

broadened existing actions by addressing

groups or policy areas that would otherwise

not have been addressed?

Scope effect: Have the operations broadened

existing actions by addressing groups or policy

areas that would otherwise not have been

addressed?

6.5.2 (1) (3)

(5)

Role effect: Have the operations supported innovation

and the transfer of ideas that have been subsequently

rolled out in different contexts?

Role effect: Have the operations supported

innovation and the transfer of ideas that

have been subsequently rolled out in

different contexts?

Role effect: Have the operations supported

innovation and the transfer of ideas that have been

subsequently rolled out in different contexts?

6.5.3 (1) (3)

(5)

Process effect: Have Member State administrations

and participating organisations derived benefits from

being involved in the operations? (Sustainability

question)

Process effect: Have Member States

administrations and participating

organisations derived benefits from being

involved in the operations? To what extent

the effects of ESF support under TO10 are

likely to continue after the end of the

funding, both at individual, institutional

and policy level?

Process effect: Have Member States

administrations and participating organisations

derived benefits from being involved in the

operations? To what extent the effects of ESF

support under TO10 are likely to continue after the

end of the funding, both at individual, institutional

and policy level?

6.5.4 (1) (3)

(5)

How sustainable the ESF-funded operations under

TO8 have been?

6.5.4 (1) (3)

(5)

6.1. To what extent the effects of ESF support under

TO8 are likely to continue after the end of the funding,

both at individual and policy level?

6.5.4 (1) (3)

(5)

Data and info sources to reply evaluation questions: (1) – external study; (2) – SFC2014; (3) – national evaluations; (4) – EUROSTAT data; (5) public consultations

114

ANNEX 4: ESF INTERVENTION LOGIC

Overall objectives

Specific objectives

Inputs

Outputs

Results

Impacts

ESF, CPR

Regulations

Treaties objectives:

Strengthening economic and

social cohesion

Improve employment

opportunities

TO 8 ‘Employment

and labour mobility’

Access to

employment*

Support to

businesses

Modernisation of

labour markets

EU: 37 billion € MS: 14 billion €

Workers, enterprises, including actors in

the social economy, and entrepreneurs, supported, as well as systems and

structures with a view to facilitating their

adaptation to new challenges including reducing skill mismatches and promoting

good governance, social progress.

People benefited, including disadvantaged people such as the long-term unemployed,

people with disabilities, low skilled, migrants, ethnic minorities, marginalised

communities and people of all ages facing

poverty and social exclusion.

Implementation of reforms, in particular

in the fields of employment, education, training and social policies.

Employment, enhanced

qualifications,

other positive results such as

increased self-

confidence,

improved

employability,

Labour market institutions

modernised and strengthened

Better performing

education and

training systems

Improved

administrative

capacity

Increased levels of

employment

Improved access to the

labour market

Increased geographical

and occupational

mobility of workers

Enhanced adaptability

to change

Increased levels of

education and training

Transition from

education to employment facilitated

Contribution to

reduced poverty

Enhanced social

inclusion

Gender equality, non-

discrimination and

equal opportunities

promoted

TO9 ‘Social

Inclusion’

Support for

specific groups

Community led

development strategies

EU: 21 billion €

MS: 10 billion €

Europe 2020 strategy

Integrated guidelines

Country Specific

Recommendations

National Reform

Programmes

European Pillar of

Social Rights

TO 10 ‘Education

and Training’

ELET and access to education and

training

Tertiary education

Lifelong learning

Labour market

relevance of E&T systems

EU: 27 billion €

MS: 14 billion €

TO 11 ‘Institutional capacity’*

EU: 4

billion €*

MS: 1

billion €*

115