Evaluation of the Intermediate Labour Market (ILM) · PDF fileEvaluation of the Intermediate...

Evaluation of the Intermediate Labour Market (ILM)

Programme

Final report

A report by Wavehill for Wales Council for Voluntary Action (WCVA)

August 2012

Acknowledgements

We would like to thank everybody who contributed to this report and the research upon which it is

based. In particular we would like to thank the ILM programme staff, panel members and those

organisations and participants that have been part of the ILM process who took the time to speak to

the Wavehill team. This evaluation would not have been possible without each of those

contributions.

Report prepared by:

Llyr Roberts

Sarah L. Bulloch

Simon Hartwell

Endaf Griffiths

Any queries or questions about this report should be referred in the first instance to Endaf Griffiths

at Wavehill: t: 01545 571711 | e: [email protected]

Client contact:

Mike Richards, Communications Officer, WCVA

t: 029 2043 1754| e: [email protected]

Report version: FINAL – 23rd

August 2012

Cover photographs courtesy of WCVA

Contents

Executive summary ....................................................................................................... 1

1. Introduction .................................................................................................................. 8

1.1. The ILM Programme ........................................................................................................... 8

1.2. The evaluation ................................................................................................................. 11

1.3. Methodology ................................................................................................................... 12

1.4. Structure of the report ..................................................................................................... 12

2. Strategic and delivery review ...................................................................................... 13

2.1. The economic climate ...................................................................................................... 14

2.2. Strategic position of the programme ................................................................................ 17

2.3. Membership and the role of the Economic Inactivity Panel .............................................. 20

2.4. The procurement process................................................................................................. 22

2.5. Contracts to deliver a service ........................................................................................... 23

2.6. WCVA’s support structure ................................................................................................ 24

2.7. Building on the experience ............................................................................................... 25

3. Organisational level evidence ..................................................................................... 26

3.1. Engaging with ILM contractors ......................................................................................... 27

3.2. Delivering the ILM ............................................................................................................ 32

3.3. Benefits of the ILM programme ........................................................................................ 35

4. Participant level evidence ........................................................................................... 41

4.1. Aspirations ....................................................................................................................... 42

4.2. Soft outcomes .................................................................................................................. 42

4.3. Job outcomes ................................................................................................................... 50

4.4. The support structure ...................................................................................................... 53

5. Programme management evidence ............................................................................ 57

5.1. Outputs and results .......................................................................................................... 58

5.2. Outcomes and impacts ..................................................................................................... 64

6. Conclusion and recommendations .............................................................................. 75

6.1. Conclusion ....................................................................................................................... 75

6.2. Recommendations ........................................................................................................... 75

Appendix 1: Soft skills questionnaire ................................................................................. 78

Appendix 2: Wordle ............................................................................................................ 88

Appendix 3: Participant questionnaire ............................................................................... 89

Appendix 4: Analysis of ILM programme management data............................................ 100

Evaluation of the Intermediate Labour Market (ILM) Programme: Final report

©Copyright Wavehill Ltd, 2012 Page | 1

Executive summary

This is final report of the independent evaluation of the Intermediate Labour Market (ILM)

programme managed and delivered by WCVA. The evaluation was undertaken over a three year

period alongside the delivery of the programme by social research and evaluation specialists

Wavehill.

The programme

The ILM programme’s objective was to reduce economic inactivity by funding organisations to

provide paid work experience and training for over 2,500 economically inactive people facing

barriers to employment. The ILM model aims to do that by developing the skills, confidence, job

readiness and basic training for people who have never been in the job market or have been out of

employment. The model recognises that employers tend to create employment opportunities for

those in work and those who can demonstrate work experience by creating funded employment

opportunities.

The four-year programme (2008 to 2012) is made up of two separate projects funded by the Welsh

Government and European funding programmes in Wales; the Convergence programme in West

Wales and the Valleys and the Regional Competitiveness and Employment (RCE) areas of East

Wales1.

In the Convergence area, a total of 45 contracts averaging 14.5 months were commissioned to

deliver ILM projects. These were valued at a total of nearly £8.5 million with a match-funding

(provided by the contractors) of £2.3 million. Contractors in these areas worked with between 20

and 75 participants giving an average of 40 ILM participants per organisation. A total of 22 contracts

averaging 15 months each were commissioned in the RCE areas valued at just over £3 million with

match-funding of just shy of £1 million. Organisations in the RCE area worked with between 4 and 88

participants, at an average of 36 participants per organisation.

Key findings: management and delivery of the programme

• The ILM programme built upon a similar programme delivered by WCVA during the previous

round of European funding in Wales (2000 to 2006), the North Wales Labour Market

Intermediate (LMI) programme. The evaluation found that this was beneficial in a number of

different ways including being able to ‘hit the ground running’ in those areas where the previous

programme had been active. The opposite seems to have been the case in those parts of Wales

which where the previous programme had not been active (the programme took longer than

anticipated to become fully operational) which contributed to a slower than anticipated start for

the programme as a whole.

• Due to the recession, the state of the labour market has changed considerably since the business

plan for the ILM programme was prepared. Specifically, the labour market has become more

competitive as levels of unemployment have increased. This has meant that achieving

employment outcomes for participants has been particularly challenging.

1 Further information about European funding programmes in Wales is available here: http://wefo.wales.gov.uk



• Some stakeholders (those involved in the management and delivery of the programme) were

concerned that the number and range of welfare-to-work programmes operating alongside the

ILM could discourage cooperation amongst providers, disrupt the participant referral process or

even lead to ‘competition’ between providers. Whilst the risk is clear, especially given the

emphasis on meeting performance related targets, there is little evidence that this has occurred.

• The introduction of the Department for Work and Pension’s new Work Programme in February

2011 caused some disruption within the ILM programme especially when issues relating to the

impact of the introduction of the programme on participant’s eligibility for ILM support were

unclear. However, once those issues were clarified, the introduction of the Work Programme

had little consequence on the delivery of the ILM.

• The management structure within WCVA for the ILM was designed with a specific focus on

establishing a model that could function within the restrictions of a competitive procurement

process whereby organisations tendered to WCVA to be awarded contracts to deliver ILM

projects. Two separate teams were created within WCVA; one to manage the procurement

process and one to monitor and provide support to organisations once they had been awarded

ILM contracts. Whilst this structure was robust from a procurement perspective, the fact that

there are two separate teams managing the programme has been challenging from a

management perspective. Most importantly, communication between the teams needed to be

excellent something which the evaluation has found, whilst challenging, has generally been the

case.

• WCVA’s Economic Inactivity Panel oversees the ILM programme receiving regular reports from

the ILM team. The panel is made up of WCVA Board Members and representatives from other

organisations invited onto the panel. The evaluation has found that the role of the panel has

changed as the programme has progressed from one focused on looking in some detail at the

setting-up of programme management structures, to a more strategic role focused on

overseeing the programme. This is something that needs to be managed as the knowledge and

expertise within the panel may need to change as the role undertaken by the panel changes.

There has also been a concern amongst some stakeholders during the lifetime of the

programme, that the panel has been ‘dominated’ by WCVA Board Members and that it has been

difficult to effectively ‘attract’ representatives from other organisations to become members of

the panel and attend meetings on a regular basis. It is however important to recognise that

consistent attendance of meetings by WCVA Board Members has maintained continuity within

the panel over the lifetime of the programme and that, as one stakeholder put it, “without the

WCVA Board Members there wouldn’t be a panel”.

• Evidence suggests that organisations ‘applying’ for an ILM had a mix of experience in terms of

submitting pre-qualifying questionnaires (PQQs) and tendering for contracts. From the 29

organisations responding to a survey of organisations that participated in the procurement

process undertaken for the evaluation in 2010, only six considered themselves to be ‘very

experienced’. Almost half (14/29) had little or no experience of the procurement process

suggesting that the ILM had succeeded in terms of its stated objective of seeking to encourage

new organisations to participant in the process.



• There was some concern amongst stakeholders that the procurement approach being used to

manage the ILM programme was having a detrimental effect on the number of organisations

participating and there is some anecdotal evidence of this. For example, one organisation in

North Wales (that had previously successfully delivered an LMI project funded under the

previous programme) was offered funding for an ILM project but withdrew due to concerns

about the contract and specifically the potential for the clawing back of funds should targets

specified within the contract not be achieved. Despite submitting successful PQQs a number of

the organisations that responded to the survey undertaken as part of the evaluation in 2010

decided not to proceed in submitting a full tender to WCVA. A number of different reasons were

identified including the timing of the contract, the prevailing economic conditions and a

perceived lack of capacity within the organisation to deliver an ILM contract. There is no clear

evidence therefore that organisations were withdrawing from the process due to concerns

about the procurement process itself.

• Setting up the procurement process took a significant amount of time and resources. Given the

amount of money involved (and hence risk to WCVA) the focus within WCVA during the early

part of the ILM programme on getting the procurement process right is clearly understandable.

But, the move away from a procurement based approach to a competitive grant system for the

final year of the programme could be interpreted as an indication that the procurement system

had not been effective. This was not the case; the change of approach was due to the

understandable desire on the part of WCVA to manage the ILM and another European funded

projects delivered by WCVA (Engagement Gateway) in the same way and it was problems within

the other programme, not ILM, which triggered the move to competitive grants. In the case of

the ILM, despite the concerns of some stakeholders, the procurement and contract based

approach would not seem to have been generally detrimental to the programme. It fact,

organisations involved would seem to have benefited by developing new skills and policies,

especially if they have little or no prior involvement.

• The feedback from the 32 organisations contracted to deliver ILM projects interviewed about

the support they received from WCVA’s ILM team was generally very positive. A small number of

those interviewed did however refer to the “overwhelming” amount of paperwork involved

although it was not considered to be an issue by the majority of those interviewed. The principal

potential improvements to the programme identified by those contracted to deliver an ILM

project were: (a) increase the length of the job placements for participants, (b) increase the

length of the contract given to providers, and (c) reduce the bureaucracy involved.

• The feedback from participants on the support that they had received via the specific ILM

project that supported them was generally very positive. The survey of participants in 2012

found that:

o 90% (346/384) believed the ILM where they worked understood their needs;

o 81% (310/384) believed they received the right sort of training at the ILM;

o 89% (340/383) believed they received enough support at their ILM job; and

o 83% (318/383) believed the ILM matched their needs.

• The improvement most commonly indentified by participants was an increase in the duration of

the job placement that could be provided via the programme.



• As shown in the table below, a review of the characteristics of participants in the programme

shows that the ILM has supported more males than females in both the Convergence and RCE

areas. It is unclear why this is the case.

Participant group Convergence area RCE area

Gender Male – 66% (1333/2,035)

Female – 34% (702/2,035)

Male – 60% (607/1,009)

Female – 40% (402/1,009)

Age

16-24 – 45% (919/2,035)

25-54 – 48% (967/2,035)

55-64 – 7% (146/2,035)

65+ - 0% (3/2,035)

16-24 – 54% (550/1,009)

25-54 – 41% (409/1,009)

55-64 –5% (50/1,009)

65+ - 0% (0/1,009)

Source: WCVA

• The programme (in both the Convergence and RCE areas) has found it more difficult than

anticipated to engage with some groups of participants, in particular older participants and

those with work limiting health conditions and / or a disability. One potential explanation for this

is that perspective contractors recognised the challenges of working with and achieving ILM

targets with these groups.

Key findings: benefit to participants

• The ILM programme has achieved a number of positive results for participants. At the time of

writing this report, the key results of the ILM programme in the Competitiveness and RCE areas

were:

Results Convergence area:

Achieved to date

RCE area:

Achieved to

date

Participants entering employment

(Full-time and part-time) 760 316

Participants gaining qualifications 1,261 698

Participants entering further learning 48 27

Participants gaining other positive outcomes

(e.g. voluntary work or self-employment) 121 88

Source: WCVA

• Whilst collecting the data has been challenging, analysis of data collected by WCVA’s soft skills

questionnaire (1,000+ respondents) demonstrates a significant increase in participants’ levels of

soft skills between the start and the end of the intervention especially in terms of ‘confidence’

and ‘knowledge of opportunities’.

• Eighty-two percent (314/383) of participants interviewed by the evaluation team in 2012

believed they had developed their ‘job skills’ during their time with the ILM programme. The

main job skills developed, according to the participants, were their ‘communication skills’ (40%;

123/310), ‘team working abilities’ (11%; 35/310), and ‘interview skills’ (9%; 27/310).



• Three-quarters of participants (288/381) perceived an increase in their ‘confidence’. A further,

three-quarters of respondents (286/382) stated that they perceived that they were more likely

to get a job as a result of their involvement with the ILM.

• Participants believed that the lack of work in the area (39%; 132/340), lack of experience on

their part (14%; 49/340), and lack of qualifications (8%; 27/340) were the main reasons why they

were finding it difficult to find a job (pre ILM support) all of which are issues being tackled by the

ILM.

• Fifty-five percent (175/317) of participants interviewed who had left the ILM programme had

secured a job after leaving. This is clearly positive although this figure reduced to 46% (146/317)

who were in employment at the time of the interview suggesting that at least a proportion of

those who move into employment immediately after being supported will leave or lose that job.

• Eighty-three percent (145/174) of the respondents who were employed after exiting the ILM

programme felt it was the sort of job they wanted and 74% (111/150) said that they believed

they were likely to stay in the job over the next 12 months.

• At least 95% (360/377) of respondents would recommend the ILM programme to others in their

position. This is a positive indication of the perceived benefit of the support provided.

Key findings: other outcomes

• Generally, organisations who had been involved with the procurement process by submitting a

tender and/or PQQ to WCVA perceive that they had benefited from doing so. It would however

seem clear that smaller organisations with little experience of the process gained the most from

being involved. Those contracted to deliver an ILM project also referred to the new skills

developed by their staff as a result of running an ILM.

• Organisations delivering ILM projects also discussed the ‘knock-on’ community benefits of the

programme via the services being provided by the ILM participants. For example, reference was

made to the charity work which has taken place, the increase in the number of local volunteers

and the services provided to local communities via the ILM. Specific reference was made to

gardening activities provided via an ILM project and the selling of produce at a discount to low

income families.

Conclusion and recommendations

The key conclusion of this evaluation is that the ILM programme in both the Convergence and RCE

areas has generated significant benefits for those it has supported and has been successful in terms

of helping participants progress and move into employment. The fact that the programme has been

operating in difficult economic conditions has clearly been challenging. But, the programme has still

generated positive employment outcomes as well as a range of other outcomes that should improve

the employment prospects of participants. They key is however how long-term and sustainable

those outcomes are. Will those who have not moved into employment yet ultimately find a job?

Will those who have found a job keep those jobs? The answers to these questions are unclear at this

time but it is likely that the economic conditions over the coming months and years will be a

significant influence.



In terms of the management of the programme, the evaluation has found that both the

organisations providing the support and participants are generally very positive about the

programme. Within WCVA, the procurement process has however clearly been a dominating factor

especially in the first half of the lifetime of the programme with a significant amount of time and

resource being spent on it. The WCVA team has also clearly been on a very steep learning curve

when it comes to working via a procurement process and has to paraphrase discussions with

stakeholders ‘learnt a lot’.

This could of course be interpreted as another positive outcome. However, it could also be argued

that the focus on procurement has also been an unwelcome distraction for the main purpose of the

programme; to facilitate the provision of support to participants via ILM projects.

Looking to the future, the following recommendations are made for consideration if and when a

successor to the current programme is being developed:

1. The recession and its impact on the labour market had a significant influence on the delivery of

the ILM programme especially in terms of achieving employment results. Whilst it would clearly

have been impossible to anticipate the recession within the business plan for the ILM, some

element of flexibility should be built into any future business plan in order to allow the

programme to adapt to any unforeseen changes in the prevailing economic conditions. There

should also be flexibility within any future business plan to deal with other potential and

unforeseen changes such as the introduction of new welfare to work schemes that may impact

upon the eligibility of participants.

2. The ILM programme benefited significantly from being able to build on the previous North

Wales LMI programme. Any future programme should therefore seek to build upon the success

of the current ILM programme and the infrastructure (and capacity) that is now in place across

Wales to manage and deliver ILM projects.

3. A key lesson for the ILM programme during its early phases was the need to account for the time

it takes to move from a position of establishing a programme to a position where that

programme is delivering against outputs. The ILM programme found it challenging to engage

with delivery organisations during the early phases, largely it would seem due to the need to

establish a new procurement based approach. Other factors including the need to recruit new

members of staff to the ILM team along with the challenges of establishing the programme in a

new area were also prominent. The potential for a slow start should therefore be built into the

planning process for any future programmes or projects.

4. The two-team management structure put in place for the ILM within WCVA was effective but

presented a number of challenges which need to be taken into account if a similar structure is to

be used in the future. Specifically, processes need to be in place to ensure that there is effective

communication between the two teams and to ensure that there are no ‘mixed-messages’.

5. Further consideration should be given to what the role of any panel overseeing the management

and delivery of any future programme should be. This should include further discussion on how

non-WCVA representatives should be engaged. The role of the panel changes as a programme

develops and this needs to be taken into account when ‘membership’ of the panel is being

discussed. Potential options for discussion include:



a) Setting up sub-groups (or ‘task and finish’ groups) to deal in detail with specific issues and

then report back to the main panel; such as, for example, the setting up of a new

procurement management system.

b) Inviting external organisations and/ or individuals to become members of those sub-groups

rather than ‘full’ members of the panel so that their role is clear.

c) Inviting external organisations and/ or individuals to ‘one-off’ meetings of the panel when

specific issues are being discussed, such as welfare-to-work reform rather than inviting them

to become full members of the panel.

6. A wide range of organisations have been involved in the ILM programme including some with

significant experience of tendering competitively for contracts and a number with very little if

any experience of doing so. Any support that is provided during the tendering (or application)

process (for example, PQQ briefing sessions) needs to take this into account in order to ensure

that the advice being provided is applicable and as useful as possible to the organisations in

question. Potentially, this should include differentiating between advice for beginners and

advanced support for the more experienced.

7. The ILM programme has not been successful in engaging with some target groups including

older people and those with learning disabilities. Consideration should therefore be given to

whether these and other target groups require ‘specialist’ ILM projects designed to take into

account the challenges of working with those groups. This may include setting different targets

for projects working with those who are considered to be the furthest away from the labour

market in order to ensure that potential providers are not discouraged from developing ILM

projects due to what they perceive to be unrealistic employment targets.

8. In light of feedback from participants and organisations delivering ILM projects, the potential to

extend the period of work experience that ILM projects in the future could offer should be

considered. Future evaluations should however also explore whether the outcomes generated

are enhanced by a longer work experience period.

9. Whilst recognising that a certain level of bureaucracy is inevitable with any publically funded

project, the potential to streamline the paperwork and the process of collecting paperwork

associated with future programmes should be explored by, for example, establishing a working

group to consider the matter with those engaged to deliver projects within the current ILM

programme.

10. The ‘soft outcomes’ data captured by WCVA for the ILM programme has been an important

monitoring and evaluation tool. Collecting the evidence can however be difficult and future

programmes should ensure that adequate processes are in place so that the evidence for ‘soft

outcomes’ is as comprehensive as possible. There may also be a need to emphasise the

importance of this evidence to contractors and participants.

11. Whilst the ILM programme ultimately reverted to a competitive grant process, it is important to

note the benefits generated by the procurement based approach employed for the majority of

the lifetime of the programme; these benefits can potentially aid organisations as they tender

for contracts from the public sector in the future. Whilst recognising the fact that a procurement

based approach does restrict the support that can be provided to organisations that can

potentially deliver effective ILM projects, the use of a procurement based approach in the future

should not be completely discounted.



1. Introduction

In November 2009, Wales Council for Voluntary Action (WCVA) commissioned research and

evaluation specialists Wavehill to undertake an ongoing evaluation of the Intermediate Labour

Market programme (hereinafter referred to as the ILM programme). This is the final report of that

evaluation which builds on two previous reports produced in 2010 and 2011. The evaluation has

been undertaken alongside the delivery of the programme which spans from October 2008 to

September 2012.

1.1. The ILM Programme

The ILM programme aims to reduce economic inactivity in the Convergence areas of North Wales

and Regional Competitiveness and Employment (RCE) areas of East Wales by investing £18 million of

European funds in organisations to provide paid work experience and training for people facing

barriers to employment.

The ILM programme offers contracts (latterly grants) up to a maximum of £300,000 to organisations

successful during a competitive procurement process. The intention is to ensure that organisations

are eligible to run an ILM contract through a fair and transparent process. Contracted organisations

are expected to deliver personal development outcomes through a period of supported employment

that results in a proportion of individuals consequently gaining non-supported work and / or national

recognised qualifications. It is a programme which aims to build individuals’ confidence and skills to

move them towards sustainable employment. It is then hoped that this will lead to greater social

cohesion and sustainable development in the most disadvantaged communities.

The ILM invests in organisations that are able to support and develop people who have become

dependent on social benefits and those who face barriers to work. The main target groups for the

programme include:

• People in receipt of incapacity benefit

• Women who want to return to work

• Black, Asian and minority ethnic people

• People with work limiting health conditions or disability

• People over 50 years of age

Other target groups include ex-offenders, people with a history of substance misuse, lone parents,

young people not in employment, education or training (NEET), and the economically inactive.

The targets set were revised part way through the lifetime of the programme as a result of a re-

profiling exercise undertaken to take account for a later than anticipated start date and changing

economic circumstances (discussed later).



The revised (or re-profiled) headline targets of the programme in both the Convergence and RCE

areas are as follows:

Table 1(a): ILM programme targets (revised)

Headline targets Convergence areas RCE areas

Economically inactive participants to be engaged

Participants to move into sustainable employment

Participants achieving a recognised qualification

Participants entering further education

Gaining other positive outcomes

Employers supported

1,995

1,117 (56%)

1,596 (70%)

236 (12%)

1,995 (100%)

750

1,000

560 (56%)

700 (70%)

120 (12%)

1,000 (100%)

200 Source: WCVA

The original targets were as follows:

Table 1(b): ILM programme targets (original)

Headline targets Convergence areas RCE areas

Economically inactive participants to be engaged

Participants to move into sustainable employment

Participants achieving a recognised qualification

Participants entering further education

Gaining other positive outcomes

Employers supported

1,800

705 (66%)

1,404 (78%)

180 (10%)

1,764 (98%)

750

750

420 (66%)

585 (78%)

45 (6%)

735 (98%)

200 Source: WCVA

The programme is managed by Wales Council for Voluntary Action (WCVA), the umbrella body for

the third sector in Wales. The programme was launched in the Convergence areas of North Wales in

November 2008 and across all RCE areas in June 2009 and has been operational in almost all parts of

Wales with the exception of Carmarthenshire and Pembrokeshire. The programme was jointly

launched with another programme managed by WCVA in the Convergence areas of Wales; the

Engagement Gateway programme invests in organisations that target people who may be out of

touch with mainstream services or at high risk of long term benefit dependency in both the RCE and

Convergence areas of Wales.

1.1.1. Programme funding

ILM was originally a £29 million programme funded by the Welsh Government and European

Structural Funds with almost £21 million being funded at a grant rate of 55% in the Convergence

areas of North Wales and almost £8 million at a rate of 40% in the RCE areas2. As a result of the re-

profiling exercise noted above, the total value of the programme was however later reduced to

£18.93 million.

2 Figures quoted in Competitiveness and North West Wales ILM Programme Business Plan.



Contract

notice

Expression

of interestPQQ

Approved

supplier list

Invitation

to tenderTender

Contract

award

ILM operates under Priority 2 Theme 1 of the Convergence and RCE programmes in Wales. Theme 1

aims to increase employment and tackle economic inactivity by bringing more people into

employment through:

• Implementing active labour policies and supporting people to overcome a wide range of barriers

to sustainable employment.

• Helping to address specific barriers faced by disadvantaged groups.

The Convergence Programme is the successor to the Objective 1 Programme 2000–2006, and

geographically will cover 15 local authority areas in West Wales and the Valleys area. East Wales has

been designated as a Regional Competitiveness and Employment (RCE) area with an approximate

funding allocation of £97 million (€136 million) for 2007-2013, covering seven local authority areas.3

1.1.2. Building on previous provision

The ILM programme succeeds a previous grant scheme programme (the North Wales LMI) which

funded supported employment opportunities in North West Wales as part of the previous round of

European programmes (2000-2006, Objective 1 and 3).

In addition to the greater geographical coverage of the current ILM programme, the other significant

difference to the old model is that the North Wales LMI programme was a grant funded project. The

new programme, until recently, operated through a restricted procurement process4. Latterly, as

discussed in the report, the programme has however reverted to a competitive grant based system.

The restricted procurement process was a two-part procedure which begins with organisations

stating their expression of interest to WCVA and requesting a Pre-Qualification Questionnaire (PQQ).

If successful during the PQQ stage, WCVA invited organisations on its approved supplier list to

tender for an ILM contract. Following a successful tender, organisations were invited to deliver an

ILM project which required them to become contracted to WCVA. These steps are outlined in

diagram 1 below.

Diagram 1: A restricted procurement model

The contracts (and latterly grants) were for a period of up to 18 months.

1.1.3. Programme management structure

WCVA is contracted to the Welsh European Funding Office (WEFO) as the lead sponsor for the ILM

programme and therefore has ultimate responsibility for the delivery of the programme.

3

Welsh Assembly Government (2009) European Structural Funds 2007-2013 Strategic Framework. [Online]. December 2009. Available

from: http://wales.gov.uk/docs/wefo/publications/strategicframeworks/100202employmentinactivityframeworken.pdf [Accessed: 6th

July 2011]. 4

Since May 2011 the ILM Programme has reverted to a competitive grant which formed part of the North Wales LMI model.



The ILM management and delivery team consists of 15 core members of staff led by the Director of

Enterprise & Regeneration. The Economic Inactivity Initiatives Manager has day-to-day responsibility

for the programme and has three managers dealing with separate departments, including:

Development & Support, Contracts & Procurement, and ILM Organiser. These departments have a

team of Development Support Advisors (x3), Procurement & Contracts Assessors (x3) and

Administrators (x4)

A key characteristic of the team is the divisions which have been created in order to provide

specified support. As a result, separate members of the team are responsible for dealing with

procurement related issues leaving other members of the team to help contracted organisations

with facilitation and monitoring support. WCVA also draws on its corporate team to support the

delivery of the ILM programme. This involves recruitment and development of staff and the

management of budgets and payments.

With representation from the public, private and voluntary sectors, WCVA’s Economic Inactivity

Panel provides strategic direction to the ILM programme receiving regular reports from the ILM

team.

1.2. The evaluation

1.2.1. Aims and objectives

The evaluation considered the following key questions:

• To what extent is the project delivering on its aims and objectives in relation to targets set down

in funding contracts with WEFO and the Welsh Government?

• Is the project successfully reaching individuals furthest from the labour market and engaging

effectively with them?

• What has been the impact on the participants who took part in the project in terms of the

development of skills and employability?

• What has been the impact on the organisations invested in, including linkages with local

employers?

• What has been the impact on the communities in which the projects were delivered?

• A review of the challenges in setting up an ILM in RCE areas.

• Investigation of how the ILM links with other projects aimed at tackling inactivity and

unemployment.

• How effective are the support mechanisms and networks offered to organisations by WCVA?

• How can WCVA improve its procurement administration based on a review of its current

processes?



1.3. Methodology

The methodology used to undertake the evaluation investigated the following:

• Inputs – the resources taken in and used by the project.

• Process – the way in which the project has been managed and delivered.

• Outputs – what is measurable and attributable to the project?

• Outcomes – the impact or outcomes the project has had on participants.

Our approach to the evaluation is therefore based on the need to take into account a number of

different perspectives. With this in mind, the evaluation involved the following fieldwork:

• 32 interviews with strategic and delivery level stakeholders;

• 51 interviews with ILM contractors;

• 7 interviews with non-contracted organisations;

• A survey with 29 organisations involved with the ILM procurement process;

• Interviews with 26 organisations employing ILM participants;

• Surveys of 169 ILM participants in 2011 and 384 in 2012; and

• Observation of activities and interviews with participants and staff through 16 case study visits.

In addition to the above primary sources of evidence, secondary evidence including the

programme’s monitoring data and soft skills data have also been analysed. The evaluation team

have also attended a number of ILM team and panel meetings since the evaluation began in

November 2009.

1.4. Structure of the report

The remainder of this report is structured as follows:

• Chapter 2 identifies and discusses the key issues raised during strategic and delivery level

interviews;

• Chapter 3 considers the views of organisations that have been part of the ILM;

• Chapter 4 reflects on the views of participants;

• Chapter 5 analyses the outputs, results, impacts and outcomes to date; and

• Chapter 6 draws together the key conclusions of the evaluations and makes a number of

recommendations.



2. Strategic and delivery review

This section discusses the findings of the interviews with strategic and delivery stakeholders over the

lifetime of the evaluation and is focused on the management and delivery of the programme at both

a strategic and delivery level.

Key findings

• Due to the recession, the state of the labour market has changed considerably since the business

plan for the ILM programme was prepared. Most pertinent is the impact the recession has had

on the labour market, specifically levels of unemployment.

• There is a perception that the increasing number of welfare-to-work programmes active

alongside the ILM could be leading to competition between providers. This issue led the ILM

management team to combine the ILM with the Future Jobs Fund project also being delivered by

WCVA.

• Whilst the Welfare Reform Bill and the changes introduced by the Department for Work and

Pension’s new Work Programme have been a concern to the ILM team since its launch in

February 2011, the changes have actually had little consequence on the delivery of the ILM

programme.

• During the lifetime of the programme, the role of the WCVA’s Economic Inactivity Panel in

respects of the ILM has changed from focusing on processes and more specifically, procurement,

to concentrating to a greater extent on strategic issues.

• Despite being very resource intensive and time-consuming during in the early stages, the ILM

team grew accustomed with the procurement process and steps were taken to “speed things

up” by, for example, introducing a rolling PQQ and allowing decision-making to happen

electronically.

• The “robust delivery model” developed – specifically dealing with procurement and delivery

aspects on the ILM programme separately - has been effective. There is however an ongoing

need to ensure that communication lines between these two parts are clear thus reducing the

“distance” between the procurement and delivery teams.



2.1. The economic climate

The first year of evaluation raised the issue of the fragile economic climate and the discussions with

stakeholders during the second year indicated that this developed into more of an issue as the

programme began to focus on delivery.

“Some organisations have been delivering for a while but work is difficult to find [for

participants].”

“The economic climate is an issue. There is no work at all in some areas.”

The economic situation has changed considerably since the business plan for the ILM programme

was drafted. The collapse of Lehman Brothers bank in September 2008 incurred massive losses on

financial institutions throughout the world with significant government intervention demanded to

protect a full-scale banking collapse taking place. This led to massive restrictions on lending designed

to help begin to address the huge debt bubble that had emerged with the shock of the impact

leading to a huge reduction in consumer confidence. In addition, significant reductions in public

sector spending were made to address the public sector debt that had been built throughout the

Labour government’s tenure on the assumption of sustained economic growth. These factors

(amongst many others) combined to lead to an economic downturn and subsequent recession of the

scale not seen in the UK since the great depression of the 1930s. Economic growth in the UK and

Wales remains anaemic (even falling at times) and, synonymous with all other recessions, the legacy

of higher unemployment, post-recession, remains.

The most pertinent impact of the recession is the impact on the labour market, specifically

unemployment. There are two standard measures for unemployment used in official statistics in the

UK: the ILO unemployment measure and the claimant count. These are continually updated by the

Office for National Statistics (ONS)5.

The International Labour Organisation (ILO) unemployment measure is a count of those who are out

of work and want a job, have actively sought work in the last four weeks and are available to start

work in the next two weeks; plus those who are out of work, have found a job and are waiting to

start in the next two weeks. Unemployment rates (based on the ILO definition) are calculated in all

EU countries leading to the calculation of EU harmonised unemployment rates. These rates are

expressed as a percentage of the economically active population. Graph 1 over illustrates the

increase in the ILO unemployment rate in Wales since the beginning of 2008.

5 http://www.ons.gov.uk/ons/index.html



Graph 1: ILO Unemployment Rate in Wales (% of pop. 16 and over, quarterly)

Source: StatsWales

Since late 2007, there has been a sharp rise in the quarterly unemployment rate in Wales from 4.7%

of the population aged 16 or over to 8.9% (in the quarter ending February 2012, the last for which

data is available). The rate grew most rapidly during the end of 2008 - middle of 2009 before tailing

off slightly prior to another sharp rise in mid 2011. The unemployment rate reached a high of 9.3%

during July 2011 to September 2011, nearly double the rate three years previous. Over the same

period, the unemployment rate in the UK as a whole rose from 5.2% to 8.3%, a smaller rise than in

Wales but still significant. The UK rate peaked slightly after Wales during September 2011 to January

2012, reaching 8.4% for three quarters in a row.

The claimant count is a count of all persons claiming unemployment-related benefits, taken from

Jobcentre Plus administrative systems. As such it is not subject to sampling variability and can

therefore be disaggregated to very high levels of detail. However, it excludes those who are

unemployed who are not eligible to claim (for example those out of work but whose partner works),

and those who do not wish to claim.

4

5

6

7

8

9

10

No

v 2

00

7 t

o J

an

20

08

Jan

20

08

to

Ma

r 2

00

8

Ma

r 2

00

8 t

o M

ay 2

00

8

Ma

y 2

00

8 t

o J

ul 2

00

8

Jul 2

00

8 t

o S

ep

20

08

Se

p 2

00

8 t

o N

ov 2

00

8

No

v 2

00

8 t

o J

an

20

09

Jan

20

09

to

Ma

r 2

00

9

Ma

r 2

00

9 t

o M

ay 2

00

9

Ma

y 2

00

9 t

o J

ul 2

00

9

Jul 2

00

9 t

o S

ep

20

09

Se

p 2

00

9 t

o N

ov 2

00

9

No

v 2

00

9 t

o J

an

20

10

Jan

20

10

to

Ma

r 2

01

0

Ma

r 2

01

0 t

o M

ay 2

01

0

Ma

y 2

01

0 t

o J

ul 2

01

0

Jul 2

01

0 t

o S

ep

20

10

Se

p 2

01

0 t

o N

ov 2

01

0

No

v 2

01

0 t

o J

an

20

11

Jan

20

11

to

Ma

r 2

01

1

Ma

r 2

01

1 t

o M

ay 2

01

1

Ma

y 2

01

1 t

o J

ul 2

01

1

Jul 2

01

1 t

o S

ep

20

11

Se

p 2

01

1 t

o N

ov 2

01

1

No

v 2

01

1 t

o J

an

20

12

% of population



Graph 2: Claimant Count Wales, January 2008 – March 2012

Source: StatsWales

Claimant counts have taken a similar path to unemployment in Wales between the start of 2008 and

early 2012, rising sharply during 2008 and early 2009, tailoring off toward the end of 2009, but rising

again during the second half of 2011 and the start of 2012. At its peak in October 2009, there were

80,700 claimants in Wales, a rate of 5.5%.6 The rate has remained higher in Wales than the UK for

this whole period, increasing from a difference of 0.2% in January 2008 to 0.7% in March 2012.

Taken together, the unemployment rate and claimant count suggest that the labour market in Wales

has suffered to an even greater extent than the UK as a whole in recent years. It has also been

suggested that the rise in unemployment is lower than expected given that it has followed the

equivalent drop in GVA in other recessions.7 This suggests that there has been some relative success

in cushioning the impact of the recession in terms of unemployment, but that this may not be

sustained once the effects of public sector cuts are felt (especially welfare) in the coming years. This

could be particularly significant for Wales, with a greater proportion of people employed in the

public sector than the UK average.

Whilst Wales has clearly been affected economically and whilst interviews with six stakeholders

suggest that this has contributed to the challenges of finding opportunities for participants, another

stakeholder expressed a different view outlining:

“In areas such as Anglesey and North Gwynedd where unemployment rates are high,

organisations are good at finding job opportunities.”

6 Workplace-based rates which are calculated by expressing the number of claimants who are resident in each area as a percentage of

workforce jobs plus the claimant count. Workforce jobs are the sum of: employee jobs; self-employment jobs; Her Majesty`s Forces;

government-supported trainees. The largest component of the workforce is employee jobs which comes from a survey of employers. As a

result the rate is a workplace-based measure, which expresses the demand for jobs in an area. 7 Drinkwater S, Blackaby D, Murphy P (2011) The Welsh Labour Market Following the Great Recession, WISERD/PBS/002.

30000

40000

50000

60000

70000

80000

90000

Jan

ua

ry 2

00

8

Ma

rch

20

08

Ma

y 2

00

8

July

20

08

Se

pte

mb

er

20

08

No

ve

mb

er

20

08

Jan

ua

ry 2

00

9

Ma

rch

20

09

Ma

y 2

00

9

July

20

09

Se

pte

mb

er

20

09

No

ve

mb

er

20

09

Jan

ua

ry 2

01

0

Ma

rch

20

10

Ma

y 2

01

0

July

20

10

Se

pte

mb

er

20

10

No

ve

mb

er

20

10

Jan

ua

ry 2

01

1

Ma

rch

20

11

Ma

y 2

01

1

July

20

11

Se

pte

mb

er

20

11

No

ve

mb

er

20

11

Jan

ua

ry 2

01

2

Ma

rch

20

12

No.



As the comment above suggests, the economic condition varies from area to area. The case studies

highlight the unemployment rates in different local authority areas. Whilst the views of stakeholders

differ, the management team have clearly considered different ways of providing support to

organisations managing ILM contracts due to the prevailing economic climate. For example, the

project team undertook some analysis and consultation with contracted organisations during the

latter part of 2011 to see how they engaged with potential employers.

2.2. Strategic position of the programme

Four stakeholders reaffirmed the views expressed during the first year evaluation that the overlap

with other programmes and schemes on a regional and local level could have a negative impact on

the ILM; others did not feel this to be a major concern. The following comments reflect the

differences of opinion between individuals.

“I’m not sure if projects are competing against each other. There are certainly no shortages

of people for the ILM.”

“The competition between projects is a problem.”

The comments suggest that there are ample numbers of potential participants for the programme

and labour market statistics suggested that this was indeed the case when the argument was raised

with around 56,200 (from a total of 122,000 in Wales) unemployed people in the areas where the

ILM programme is run.8 Having said this, there is a feeling that a high number of welfare-to-work

programmes, schemes and projects can potentially lead to competition between providers. Due to

their involvement with other programmes therefore, some organisations may be less likely to deliver

an ILM contract / project.

The issue of competition has certainly been a consideration for the ILM management team and, as a

result, efforts have been made to integrate the Future Jobs Fund9 programme delivered by WCVA

and the ILM. According to one stakeholder this reduces the amount of match funding required whilst

also extending the length of placement from 16 weeks to nine months to participants. On the other

hand, the same individual highlights the challenges involved in raising awareness of the ILM to

providers. This is interesting, especially as the 2010 evaluation report highlights that the joint launch

of the ILM and WCVA’s Engagement Gateway programme10

contributed to the lack of understanding

surrounding the aims and objectives of the ILM.

In addition to issues surrounding competition from other programmes or schemes, uncertainty has

emerged amongst stakeholders around the Welfare Reform and the Department for Work and

Pension’s (DWP) new Work Programme.

8 Nomis (2011) Official labour market statistics. [Online]. Office for National Statistics. Available from: http://www.nomisweb.co.uk/

[Accessed: 10th

August 2011]. 9 The Future Jobs Fund was a scheme to help mainly young unemployed people back into work.

http://webarchive.nationalarchives.gov.uk/+/www.direct.gov.uk/en/Employment/Jobseekers/LookingForWork/DG_184167 10

Engagement Gateway is European funded project delivered by WCVA in the Convergence and Competitiveness areas which aims to

reduce economic inactivity in Wales by improving the employability of people who face barriers to employment and are furthest from the

labour market. http://www.wcva.org.uk/europe/index.cfm?sub=4&display_sitetextid=226



The proposals put forward by the Coalition Government in the Welfare Reform Bill 2011 were

perhaps the most radical that post war Britain has seen. The bill which was released on 16th

February 2011 aims to make the benefit and tax systems simpler and fairer through “creating the

right incentives to get more people into work by ensuring work always pays, protecting the most

vulnerable in our society and delivering fairness to those claiming benefit and to the taxpayer.”11

These changes look to have major impacts on the quality of life and the living standards of both

individuals and households; in particular these impacts will be seen amongst the low income

households who find themselves dependent on benefit payments. There will also be a geographical

impact as Hamnett (2010) explains:

“With some of the greatest effects being felt in the poorest and most deprived areas of Britain,

where economic activity levels are lowest and unemployment and the proportion of the

population who are dependent on social security benefits is highest.”12

The Welsh Government and leading welfare groups expressed concern about the impact these

reforms will have in Wales, particularly on the most vulnerable groups and individuals.13

Indeed, the

Welsh Government’s Ministerial Task and Finish Group for Welfare Reform commissioned a

programme of work to assess the impact of the UK Government’s welfare reforms in Wales. Findings

from the first stage of this programme of work were released in early 2012. The key finding of the

report was that Wales will be hit disproportionately compared to the rest of the UK by the welfare

reforms and more specifically:

• The impact of the tax and benefit changes to be implemented by 2014-15 will mean that on

average households in Wales can expect to lose 4.1% of their income (or £1,110 per year).

• The changes will be regressive in nature at the UK level (i.e. they will take more proportionately

from lower-income groups than from higher-income groups except the very richest income

group), and are likely to be even more regressive in Wales.

• The poorest households with children are estimated to lose the largest proportion of their

income as a result of the tax and benefit changes. In particular, non-working lone parents and

workless couples with children are expected to suffer a disproportionate financial hit. Families

with children aged under five and families with more than two children will also be particularly

badly affected.

• Cuts to benefits and more stringent benefit entitlement rules may push people out of

unemployment and into crime activities.

The report also notes that the effects of the welfare reforms will depend on the strength of the

wider economy and the extent to which people change their behaviour in response to benefit

reductions.

11

Department for Work and Pensions (2012) Welfare Reform Bill 2011. [Online]. Available from: http://www.dwp.gov.uk/policy/welfare-

reform/legislation-and-key-documents/welfare-reform-bill-2011/ [Accessed: 5th

June 2012]. 12

Hamnett C. (2010) The Re-shaping of the British Welfare System and its Implications for Geography and Geographers Progress in Human

Geography Sage: London. 13

Welsh Government (2012) Welsh Ministers to discuss concerns over Welfare Reforms. [Online] Available from:

http://wales.gov.uk/newsroom/educationandskills/2012/120319welfarereforms/?lang=en [Accessed: 5th

June 2012].



Central to the reform of welfare is the new Work Programme which sits alongside changes to

welfare benefits. The Work Programme replaces all previous mainstream (government) support

activity, and is, in theory a ‘black box approach’ open to all, although disabled people may get

support from the Work Choice programme.14

The new provision aims to offer individualised support

underpinned by an element of benefit compulsion where people not undertaking required activity

can be subject to benefit sanctions and possible mandatory activity. People coming out of their

Work Programme without a job may be put onto Mandatory Community Activity and could cycle

between the two if they do not gain employment.

The programme is delivered by specialist public, private and voluntary sector providers designing

flexible support based on customer need, and these providers will be rewarded for keeping people

in work and for assisting harder to help customers. These rewards are in the form of three fee

payments: an attachment fee when an individual starts the programme, a job outcome fee and a

sustainment fee for sustained employment.

The DWP has appointed Working Links and Rehab Jobfit to deliver the programme in Wales; as

‘prime providers’ (with contracts lasting for seven years) they will be expected to work with a range

of sub-contractors to deliver the personalised and localised services enshrined in the programme,

although in their original bids to be prime contractors Working Links proposed to sub-contract 5.3%

of their total contract value to the voluntary sector, whilst Rehab Jobfit proposed 13% voluntary

sector sub-contracting.

Importantly, ESF support (such as that provided through the ILM) cannot be used for people eligible

for mandatory entry to the Work Programme as it would substitute expenditure which the Member

State would make from its own resources. WEFO guidance states that the following groups are

eligible for ESF support:

• Jobseekers Allowance (JSA) and Employment Support Allowance claimants with mandatory

access to the Work Programme prior to any referral onto the Work Programme (however, ESF

projects must be confident that they can achieve an outcome with these individuals before the

point of referral);

• JSA/ ESA claimants with voluntary access to the Work Programme who are not accessing it;

• Income support recipients (e.g. single parents of children aged under 7 and other carers);

• Incapacity Benefit (IB) recipients15

;

• People out of work who are not claiming benefits;

• People being made redundant;

• 16-17 year olds (unless they are claiming JSA and on the Work Programme);

• People in receipt of state pension; • Unemployed or economically inactive people in receipt of carers allowance (if not claiming

Jobseekers Allowance (JSA) or Employment Support Allowance (ESA)).

14

http://www.direct.gov.uk/en/DisabledPeople/Employmentsupport/WorkSchemesAndProgrammes/DG_187696 15

IB is to be gradually phased out and replaced by ESA and Universal Credit over the next 3 years, with Job Centre Plus approximate

figures indicating that the c116,000 IB recipients due to be migrated will be moved onto ESA Work Related Activity Group (39%), ESA

(29%), and 31% onto JSA or off benefit.



The introduction of the Work Programme presented two specific problems for ESF projects in

demonstrating added value. The intention to deliver to all categories of JSA and ESA recipients will

cut across the cohorts which many of the ESF projects have drawn on to date. Beyond this,

implementation of a ‘black box’ approach makes it difficult to identify any clearly defined ‘additional’

interventions available for ESF support. With the Welsh Government underlining the importance

that Government support does not duplicate or substitute the Work Programme and WEFO also

committed to ensuring the ESF is not used to substitute expenditure which the Member State would

otherwise make from its own resources, there is some uncertainty surrounding the future of ESF

funded programmes. It is likely that ESF projects need to re-focus their activities. The changes to

eligible groups could lead to formal changes to the Convergence and Competitiveness ESF

programmes for Wales in terms of participants, targets and outputs.

At the time of writing this report, the picture has become somewhat clearer. WCVA have been

involved in lobbying WEFO around the eligibility criteria and have sought clarity around issues

including delivery models, contractors and processes to ensure that there is compliance between

European and Central Government funding. Following a meeting with WEFO in June 2011 the ILM

team have held events and consulted with Project Sponsors to look at the impact on client groups

that the Work Programme could potentially have. In response to the guidance received by WEFO

and having discussed the issue further with partners, WCVA remained confident that the impact

would be minimal and there would not be a need to submit new or adjusted business plans.

Therefore, despite being a concern for many stakeholders over the past year or so, it is safe to say

that these changes have had little impact on the ILM programme itself.

2.3. Membership and the role of the Economic Inactivity Panel

The role of WCVA’s Economic Inactivity Panel during the early stages of the ILM involved ensuring

that adequate processes were in place and that the programme had a strong basis to develop from.

Some members of the panel were however of the view that there had been an over-focus on the

procurement process during those early stages and that this was a very time-consuming process.

This issue was perhaps complicated by the fact that it took time for some panel members to become

familiar with the new ILM and move away from the ‘old’ (grant based) LMI model.

“At the beginning, the panel took it the same way as the grant process. However, they are

growing familiar with their role now and do not look at all tenders now.”

The delivery team also believed that this over focus on processes contributed to slow decision

making, especially when deciding whether a contract should be awarded to an organisation, or not.

Since the 2010 evaluation report the role of the panel has moved away from focusing on processes

and more specifically, procurement, to concentrating on delivery issues more generally.

“There’s a strategic role for the panel... We can back away from looking at PQQ criteria now.”

Having familiarised with the programme and, as the ILM has moved into the ‘implementation

phase’, panel meetings have been held less frequently and communication has since been taking

place over e-mail. There is a suggestion that this development has contributed to “speeding up” the

decision-making process.



Two of the stakeholders interviewed were however concerned that some Panel Members were

“uncertain of what was happening.” Their view was that the way information is transferred to Panel

Members is an issue and one suggested that there should be an effort to “present rather than

summarise on paper.” There also seemed to be some uncertainty as to whether the new method of

communication led to representatives absorbing the necessary information, in detail.

The Panel was set up to include members from a wide range of external organisations as well as a

number of WCVA Board Members involved due to the financial value (and hence risk to WCVA) of

the programmes being overseen which included Engagement Gateway as well as the ILM.

The need to ensure sufficient representation from various organisations at a panel level was

discussed in the 2010 evaluation report.

“There is not enough broadness in the Panel. There are too many WCVA members.”

“WCVA Board do not have the right makeup. There must be around six members of WCVA

Board on the Panel.”

As illustrated by the comments above, one of the concerns was that the Panel was ‘dominated’ by

WCVA Board Members and that this went against the intention to draw in a wide range of different

organisations. However, it is also important to note that the presence of WCVA Board Members on

the panel has provided the necessary continuity over the lifetime of the ILM programme. As one

member put it when this matter was discussed at the July 2012 meeting of the Panel:

“Without the WCVA Board Members there wouldn’t be a Panel.”

As shown in the graph below, the number of representatives attending Panel meetings has

fluctuated since the inception of the ILM programme.

Graph 3: Number of attendees against absentees for ILM panel meetings

Source: WCVA

12

8

6

10

6

98

10 10

7

10

89

7

4 4

68

6 5

65

4

1

7

9

5

9

57

5 5

7

5

76

7

11 11

76

8 7

56

7

Attendees Absentees



Whilst it is important to note that the graph does not include the ‘e-meetings’ and email

communication which have taken place, the fluctuation is clear. A number of factors have affected

attendance including location of meetings which rotates between Aberystwyth and Rhyl and

sometimes included the opportunity to link to the meeting by video conference. It is also apparent

that it has been difficult to engage external stakeholders as members of the panel and, perhaps

more importantly, maintain that engagement over the longer term. External members would seem

to have tended to come and go. However, perhaps the key issue is that within this fluctuation, the

attendance of WCVA Board Members has remained relatively constant which has provided the Panel

with the necessary continuity to effectively oversee the management and delivery of the ILM and

the other programmes for which it has responsibility.

Twenty-one organisations have, at some point, been involved in Panel meetings. However,

interviews suggest that the “make-up of the Panel” continues to be an issue; the Panel has struggled

to find the balance between the need for continuity (provided by WCVA Board Members) and the

desire to draw in expertise from outside WCVA in order to be able to have an informed discussion

about key (strategic) issues such as the changes within the welfare-to-work sector previously

discussed. It is therefore an issue that needs further debate and discussion if and when the Panel

takes a similar role in the future. Potential options that need to be considered include:

• Setting up sub-groups (or ‘task and finish’ groups) to deal in detail with specific issues and then

report back to the main panel; such as, for example, the setting up of a new procurement

management system.

• Inviting external organisations and/ or individuals to become members of those sub-groups

rather than full members of the Panel so that their role is clear.

• Inviting external organisations and/ or individuals to ‘one-off’ meetings of the Panel when

specific issues are being discussed, such as welfare-to-work reform.

2.4. The procurement process

The procurement process had a significant impact on the early stages of the ILM programme

according to stakeholders. It was a new experience to members of WCVA team and it took time to

fully understand the process. The team’s lack of experience may have also contributed to a very

cautious approach when establishing the process.

“We got our heads around procurement during the first year. We were over-cautious initially

but then loosened our belts.”

Despite being a resource intensive and time-consuming process during the early days of the ILM

programme, the team grew accustomed with the procurement process and took steps to “speed

things up” by, for example, introducing a rolling PQQ and allowing decision-making to happen

electronically.

Despite some initial concerns on the part of the ILM team in the initial stages there is little evidence

that the procurement process had a significant negative impact on the delivery of the programme.

As discussed further later in the report, the evidence collected by the evaluation suggests that only a

small number of organisations were actually deterred from engaging with the programme due to the

procurement process. The fact that the programme budget was fully allocated also suggests that

there was adequate demand for the funding being made available.



However, for the final year of the programme, there has been a move away from a procurement

based approach to a ‘competitive grant’ process. This change was introduced due to issues relating

to the management and delivery of WCVA’s Engagement Gateway programme (which was finding

the procurement based approach overly restrictive) and a desire to maintain an element of

continuity in the way the two programmes were being managed. The change was not triggered by

any issues relating to the ILM programme.

In reality, WCVA staff will argue that there is little difference between a procurement based

approach and a competitive grant approach at what may be described as a ‘delivery level’. Both lead

to what is effectively a contract between WCVA as the programme manager and the recipient of the

contract / grant to deliver a specified service and to meet a specified set of targets. The key

difference is in the support and feedback that WCVA can provide to organisations during what is an

application process (in a competitive grant based approach) or tendering process (in a procurement

based approach) which is far more restricted. And, it was due to issues caused by these restrictions

within the Engagement Gateway programme that the decision to move away from a procurement

based approach was taken.

Overall, stakeholders had mixed feelings on the issue of procurement with one stating that they

were “not massively disappointed that we’ll be moving away from procurement” and another stating

that they “should get rid of procurement.” Others, however, favoured the procurement based

approach outlining that it provides a “level playing field” for organisations. Another stakeholder

outlined that “sticking with procurement is important as it’s better than a grant.” Despite feeling

mixed about the process the evidence suggests that the move away from procurement should not

be considered as an indication that the process failed.

2.5. Contracts to deliver a service

At an early stage of the evaluation, there was a suggestion that potential delivery organisations are

discouraged by the fact that ILM involves a contract to deliver a service rather than a grant to deliver

a project. It was argued that the shift towards a target-driven approach has deterred several

organisations from applying for a contract. And there is some evidence that this is indeed the case;

two organisations were deterred from signing a contract offered to them following the successful

tender stage due to concerns about the ‘contract’ that would be in place and specifically the

potential for the clawing back of funds if specified targets were not achieved.

“People panic about the contract, especially the job outcome figure.”

“The 56% job outcome rate has precluded some organisations from tendering.”

Although the job outcome figure seems to be the biggest concern according to stakeholders, the

length of contracts offered to suppliers is also an issue with 12 to 18 months being relatively short.

“It’s difficult to recruit someone for 12 months. The bulk of work is done between months two

and ten.”

Others questioned whether the programme included unnecessary contracts - “Some organisations

have withdrawn from the ILM. This is not a good result and is a concern. Are we building in

unnecessary contracts?” One stakeholder noted that the benefit to the delivery organisation needs

to be recognised and recorded.



It was also a concern amongst some members of the ILM facilitation team that having a contract to

deliver a service has made it increasingly difficult (i.e. against procurement rules) to assist

organisations during the procurement stage. In the past, the facilitation team were able to provide

extensive support during the application / procurement stage.

Other stakeholders argued that more consideration should be given to the needs of the third sector

during the procurement process. For example, some members argued that not all voluntary

organisations are computer literate whilst others stressed that contracts are ‘new’ to this audience

despite the fact that they had been accessing grant support in the past.

Whilst the concerns noted above need to be acknowledged, the fact that the ILM has succeeded to

allocate funds suggests that demand for such contracts was still high.

2.6. WCVA’s support structure

Seven stakeholders made reference to the contribution the WCVA team had made when discussing

the strengths of the ILM programme during the fieldwork undertaken for the second annual

evaluation.

“The team is well established and organised.”

The 2010 evaluation report highlighted that the programme has benefited significantly from the

experience of the facilitation team some of whom had an involvement with the previous LMI

programme. Having increased their knowledge and experience of the procurement process, the PQQ

briefing sessions organised by the WCVA team also developed significantly. It was also highlighted

that a strong team within WCVA has played a key role in establishing and facilitating the programme.

However, it is important to note that the programme faced difficulties during the early stages when

recruiting members of staff. Indeed, this contributed towards the early delay in delivery.

Initial discussions with members of the ILM team also suggested that it took longer to “get off the

ground” in the RCE areas in contrast to the familiar Convergence areas of North Wales where the

team had operated in the past.

“It takes a lot of one-to-one support to gather interest especially in Powys because they

haven’t operated an ILM.”

A topic which dominated discussions surrounding the support structure was the need for and

importance of having a “robust delivery model.” Indeed, stakeholders were quick to highlight the

importance of dealing with procurement and delivery separately which was considered the most

appropriate way of ensuring that issues can be dealt with fairly, without ruining the ongoing

relationships the facilitation team have developed with some delivery organisations over the years.

This is an indication of how the move to a procurement based approach dictated the management

structures that were put in place.

“The team are from different backgrounds which is good.”

“A lot of time is going into changing the attitudes of organisations. Many have been put off [by

the procurement process] but a follow-up has meant that some third sector organisations are

returning.”



Two stakeholders however expressed some concern about the model with one in particular stating

that there was some ‘distance’ between the procurement and delivery teams. There have been

instances, for example, where the delivery team have not been informed of issues relating to

contracted organisations which have led to these organisations repeating or duplicating work. With

both teams based in separate rooms within WCVA premises it was stressed that individuals need to

“talk to each other more.” One of the advantages of the shift away from procurement to a grant

based approach is however that there has been less emphasis on the need for the teams to be seen

to be separate. As a result, they have been able to have a closer working relationship during the

latter stages of the programme.

2.7. Building on the experience

One major development since the 2010 report has been the addition of a new pilot ILM project

which aims to help young people into work in the Communities First areas of Conwy, Gwynedd,

Ceredigion, Swansea, Bridgend and the Heads of the Valleys16

. The pilot project, which was available

throughout the Convergence area, funded by the Communities First programme, is part of the Welsh

Government’s Youth Engagement and Employment Action Plan and aims to create 180 employment

opportunities for 16-17 year olds. The pilot was launched in January 2011 and WCVA have been

awarding contracts to organisations through a procurement process. Whilst it is important to

acknowledge this development in this report, the pilot was not part of this evaluation and therefore

not discussed further.

16

Communities First is the Welsh Government's flagship programme to improve the living conditions and prospects for people in the most

disadvantaged communities across Wales.

http://wales.gov.uk/topics/housingandcommunity/regeneration/communitiesfirst/?lang=en



3. Organisational level evidence

This section discusses the findings of the consultations with organisations that have been involved

with the ILM in a range of different capacities including:

• 51 interviews were held with 32 different contractors delivering ILM projects;

• 7 interviews with non-contracted organisations;

• A survey with 29 organisations involved with the programme’s procurement process; and

• Interviews with 26 organisations employing ILM participants.

The section reflects on three key themes: (1) engagement with the programme; (2) experience of

delivering the ILM; and (3) benefits of the ILM programme.

Key findings

• Generally, those involved with the procurement process perceive that their skills in that area

developed as a result of their involvement.

• Contractors referred to the increased level of partnership working with other organisations and

the ILM programme’s contribution in developing the skills of employees when discussing the

benefits of the ILM to their organisations.

• Increasing confidence and self-esteem is the main impact on participants of the ILM according to

contractors.

• The state of the local labour market due to the recession and the challenges for participants to

travel to their jobs within the rural areas were key issues affecting the delivery and / or the

impact of the ILM according to contractors.

• Twenty-three of the 32 contractors interviewed did not have an issue with the ILM programme

monitoring arrangements although nine respondents referred to the “overwhelming” amount of

paperwork. Contractors rated the support provided by WCVA’s team highly.

• Whilst realising the challenges of organising information sharing events, the evidence suggests

that a more tailored approach could be adopted depending on the past experiences of those

organisations attending.

• All 32 contractors were hopeful of delivering similar contracts to the ILM in future and all

contractors also stated that they would deliver the ILM again if possible.



3.1. Engaging with ILM contractors

Due to the concerns expressed by strategic and delivery stakeholders surrounding the possible

negative impact of the procurement process on the ILM programme, a survey was undertaken for

the 2010 evaluation report to test the attitudes of those individuals and organisations who had been

involved with the process. A total of 29 organisations responded to the survey.

3.1.1. The experience of applicants

Evidence suggests that organisations ‘applying’ for an ILM had a mix of experience in terms of

submitting PQQs and tendering for contracts. From the 29 organisations responding to the survey

only six considered themselves to be very experienced. The majority, (a total of nine organisations)

considered themselves to be relatively experienced whilst eight organisations have tendered for

what they described as ‘a few’ contracts. The procurement process has also gained interest from

those new to the process with six stating that they had never tendered for contracts or submitted

PQQs before. Although almost half the organisations (14 organisations from a total of 29)

responding to the survey had little or no experience of the procurement process, 15 respondents

considered their organisations to be experienced in delivering services similar to the ILM.

Not surprisingly, a number of successful organisations (i.e. those who’s tender had been successful)

responding to the survey in 2010 had existing experience of the procurement process. From the nine

organisations that tendered for an ILM contract and were successful, five organisations considered

themselves to be ‘relatively experienced’ or ‘experienced’ with the process. It is interesting to note

that three successful organisations had no experience of the process whilst one had only a little.

The level of experience is an interesting issue especially as the ILM programme actively seeks to

encourage inexperienced organisations to submit tenders for contracts. It was concluded that

supporting this group will, in effect, ensure that the programme is having a significant impact on the

skills of individuals and organisations. Having said that, the importance in continuing to engage with

experienced organisations was also highlighted as those organisations help ensure that the ILM

programme meets its objectives.

3.1.2. Engagement with the ILM programme

The evidence shows that WCVA E-briefings (14 organisations) and the Sell2Wales adverts (12

organisations) were effective methods of raising awareness of the ILM programme amongst

surveyed organisations. Eight organisations had also received direct contact from the ILM team.

Survey results suggest that organisations have a number of reasons for becoming involved with the

ILM programme. The majority state that the social benefits the project could generate (20

organisations) and increasing the capacity and resources of the organisation to provide support (20

organisations) are however the main reasons for being involved. Eighteen organisations also felt that

it would be an opportunity to develop a portfolio of projects being delivered whilst 14 were

motivated due to the financial benefit to their organisation.

It is interesting to note that all eight of the organisations that had received direct contact from the

ILM team stated that they have become involved due to the social benefits the project could

generate. This suggests that the ILM team had emphasised this point to them during their

discussions essentially ‘selling’ the ILM on the grounds of the social benefits that the programme

could generate.



3.1.3. The PQQ briefing session

The PQQ briefing sessions provided an opportunity for interested organisations to develop a clear

understanding of the aims and objectives of the ILM programme before submitting their PQQ. The

main aim at this stage however was to inform organisations on the requirements of the PQQ and to

take individuals through the process step by step.

Twenty-five organisations from the 29 responding to the survey attended a PQQ briefing session. Of

this number, 12 organisations expected the briefing session to provide advice on how to complete a

PQQ and to cover the requirements involved. One individual hoped to receive “a clear description of

the process and of the bidding documents. I hoped to learn in-depth detail of what would be

expected of bidders.” Another four organisations noted that their expectations involved further

information on what was expected from a tender and how to bid, whilst another two expected the

session to give them more detail or information on the process. It is not a surprise to see a

difference in expectations between respondents, especially as survey results suggest that people

with different levels of experience attended the PQQ briefing session.

Despite having mixed expectations, 17 out of 25 organisations responding felt that the PQQ briefing

session met their expectations; only three respondents felt that this was not the case. An analysis of

the qualitative data gathered implies that individuals received sufficient information and guidance

during this stage of the process. A total of 13 organisations made reference to the clarity of

information and guidance received with the following remark typifying the comments made.

“It helped point out key areas where WCVA would be looking to award points. It also clarified

the detail required in some of the answers.”

The positive feedback hints that the slight changes made by the ILM team following early feedback

from attendees worked well. It must be noted however that four people responding to the survey

left the briefing session unsure about certain information / details.

“We came away with questions around the form filling, which we sought guidance with as we

went through the process. For instance, we needed clarity on the amount of detail required in

places we thought were duplicating already answered questions.”

It is positive to see that the briefing session has contributed to the understanding of the process of

submitting a PQQ and the information that would be required after the briefing session.

Although 12 organisations ‘knew quite a bit’ or ‘fully understood’ the PQQ process prior to the

session, the evidence shows that the number of organisations stating ‘I knew quite a bit’ or ‘I fully

understood’ has increased to 22 following the support. The positive response overall perhaps

reflects the perception that organisations preparing a PQQ for an ILM contract should be required to

attend a briefing session. A total of 22 organisations from the 25 agreed that this would be

beneficial.

Despite reflecting positively on the briefing session, two of the contracted organisations

participating in interviews highlighted possible future improvements to the session.

“From my point of view, it answered my questions. There were a lot of small organisations

there that perhaps didn't get as much out of the session. They were perhaps expecting more

hands on advice on how to answer questions as opposed to a question and answer session."



Another organisation suggested that further guidance on Annex 2, which is the financial section of

the PQQ, would have contributed to the individual’s understanding of WCVA’s needs. This builds on

the feedback received from the survey suggesting that organisations would have appreciated further

information through the knowledge of presenters (two organisations) or on the “do’s and don’ts” of

the process (two organisations). Two organisations also stated that an example PQQ would have

been worthwhile.

3.1.4. The tender

From the sample of 19 organisations successful during the PQQ stage a total of six organisations

decided not to tender for ILM contracts. The main reasons outlined in the survey include:

• The timing of the contract

• Economic reasons

• Uncertainty whether the ILM contract was appropriate to the organisation

• Time restrictions during the procurement process

• Lack of capacity to deliver the ILM contract

Discussions with non-contracted organisations reaffirm the concern surrounding the timing of the

contract with three sharing the general belief that “the process is too long and with all the delays

that have occurred it is bound to have an impact on delivery.” Although organisations did not relate

to the issue of having a choice between similar projects when interviewed, it seems that one non-

contracted organisation had already engaged with another project.

“We have run a number of projects with the Future Jobs Fund which is similar to ILM and it is

linked with WCVA so we had our PQQ accepted there and as a result of the Future Jobs fund

we have taken on approximately 15 new people from within the ILM area.”

This may relate to the concerns expressed by panel members of crowded provision for the ILM’s

target groups although there is no firm evidence to suggest that this is the case. Organisations after

all did not state that delivering other projects would prevent them from applying for an ILM. In fact,

when asked through the survey to note what would need to be different in order to tender for

future contracts, the only issue raised was by an organisation stating the need for projects to be

more financially viable.

A total of 10 organisations from the 29 surveyed tendered for an ILM contract with only one being

unsuccessful with their application. Further research with the 10 organisations suggests that

knowledge of the process of submitting for an ILM contract and the information that would be

required before submitting the tender varies between organisations. One organisation for example

stated that they ‘fully understood’ the process whilst a further five felt they ‘knew quite a bit.’ Four

respondents ‘knew very little’ however.

The feedback shows that the main challenges experienced by tendering organisations can broadly be

grouped into three main categories:

• Mistakes due to lack of experience

• Keeping to the word limit

• Reaching the deadline



Whilst these seem to be basic challenges, it is important to remember that these organisations vary

in term of their experiences of the process. However, these seem to be challenges that can be easily

addressed with further guidance and / or experience.

3.1.5. Reaction to the procurement process

As previously discussed, the shift from awarding grants to awarding contracts and the consequent

impact this may have on the reaction of third sector organisations was a key area of concern

amongst Panel Members and WCVA staff during the early stages of the ILM. It was therefore

interesting to find that respondents did not suggest that there had been a negative reaction to the

procurement process. Twenty-one organisations from the 29 responding to the survey noted that

they were planning to submit PQQs and / or tenders for other future contracts although eight

respondents remained unsure.

The response to the survey shows that six organisations stated that their plans for the future had

changed as a result of their involvement with the ILM procurement process although only two

organisations related to the negative impact of the procurement process when considering how

their plans would change.

“We are not so interested in becoming involved with the ILM scheme in its current

administration. In consideration of our previous ILM successes and the fact that we can’t even

pass the PQQ stage now makes it a bit off-putting to get further involved.”

Whilst it is important to note that only two organisations gave negative feedback on this matter, the

comments reflect some of the concerns raised early on by stakeholders that the process was very

demanding for many organisations. It is important to remember however that 19 organisations

would not change their plans whilst four were unsure.

The response from contracted organisations during interviews portrays a message that those

familiar and unfamiliar with the procurement process accept and understand the reasons behind

WCVA’s approach with some even praising the process.

“I thought it was really good – it was a very clear and logical process with good information.”

"It was relatively well thought through and planned in advance. Guidance and briefing events

in advance helped to inform."

“I didn’t find it too bad – no worse than the old application form.”

Despite understanding the reasons behind the process a small number of organisations found it to

be a long and inflexible process.

“It was shorter than some but more torturous than others (because of ESF).”

“It was very inflexible and strange.”



Two organisations questioned the clarity of information provided during the preparation stage of the

process. Both referred to the confusion surrounding the issue of target groups and how the guidance

received differed between the support team when phoning for assistance and when receiving

further support from the facilitation team. Although the ‘two team’ model seems to be an effective

way of delivering the programme this underlines the need for good communication to ensure that

consistent messages are transferred to participating organisations.

One contracted organisation also expressed their concern regarding the scoring criteria used to

judge tenders. The same concern was also expressed by non-contracted organisations. Overall, these

organisations did not have a favourable view of the procurement process, relating to words such as

“complex” and “bureaucratic.”

“Since WCVA started to do contract stuff they have become much more inflexible in their

approach and once you submit an application then there seems to be no going back.”

The views were shared by an individual representing a contracted organisation who stated that the

process was introducing “pressure on core activities” for smaller organisations with limited

administrative support. This particular organisation found that they were placing a number of staff

on ‘how to tender’ courses as a result. One organisation highlighted what they considered to be the

bad timing of the process (as it was at that time) because the submission date for the round in

question clashed with the end of the financial year. The ILM management team have since taken

steps to address this and other issues relating to the timing of submissions by introduce a rolling

PQQ which has helped to address this particular issue. This is an example of action being taken by

the management team to address issues as and when they have been identified.

For those contracted organisations able to compare, the procurement process relates positively to

other similar processes being used by other programmes or organisations. Although a small number

of individuals feel that they have experienced more flexibility in the past, overall, “it is not

dramatically different to other processes.” It is also positive that the process has not discouraged any

of the contracted organisations from tendering and this is reflected by the fact that many have

applied to the ILM on more than one occasion. Match funding and the capacity of a small number of

successful organisations were the main issues under consideration in 2010, not the procurement

process. Having said this, it seems that operating a procurement process may well have slowed the

process of engaging with organisations to deliver contracts.

It is interesting to note that the views of the contracted organisations mirror the views of non-

contracted organisations, although it must be stressed that their views are more mixed in

comparison. One non-contracted organisation for instance portrayed a process which was “less

friendly than others.” Others clearly feel frustrated at missing out due to what they perceive to be

minor issues.

Looking forward, three non-contracted organisations said that they would apply again for an ILM

contract. Only one non-contracted organisation did not want to apply in future. Whilst disappointing

to see that one organisation has been discouraged, the fact that other organisations are re-

submitting applications suggests that they are eager to deliver the service and that the process has

not been such that they would not consider re-submitting a PQQ or tender.



3.1.6. Perceptions of WCVA and managing contracts

Another concern raised during strategic stakeholder interviews was how the change in WCVA’s

support structure may have contributed to the way people perceived WCVA. This was a topic raised

during interviews with contracted and non-contracted organisations.

Despite not being able to comment on whether their perceptions of WCVA had changed with time,

non-contracted organisations did feel that grants are “friendlier” than a process of tendering for

contracts. For contracted organisations however, it seems that the perceptions of WCVA have not

changed dramatically. One organisation however felt that there has been a shift from helping

organisations to funding contracts. Another organisation also mentioned that their perceptions have

changed saying; “my perceptions have changed... They [WCVA] seem to be getting bigger.”

The comment above was made in relation to the individual’s perception of WCVA being seen as a

body authorising contracts, which perhaps highlights the difficulty the ILM team have had in dealing

with delicate procurement issues on the one hand and managing and facilitating contracts on the

other. To the contrary however, most organisations were either unsure of how to respond to this

question or felt that their perceptions have improved.

“I find them the most personable organisation that we have worked with... they are very

supportive.”

In contrast to non-contracted organisations, some contracted organisations feel there is a difference

between managing a grant and a contract with preference to the latter “although it takes longer to

get there.” There was a feeling that contracts “keep you on your toes” although one negative

highlighted by an organisation was the wait to receive finance.

3.2. Delivering the ILM

The interviews with 32 contractors provided an opportunity for respondents to reflect on their

experiences of delivering an ILM project. Individuals were asked to rate WCVA’s support and

monitoring arrangements and discussions allowed some insight into good practice examples of

delivering the support.

3.2.1. Support arrangements

A large majority of contractors rated the support provided by the ILM team highly. Twenty-five

individuals for instance made use of positive phrases such as “very good,” “excellent,” “supportive”

and “brilliant.”

“It’s brilliant. They’ve been really good. We know we can pick up the phone and they’re very

quick to come out and give advice and guidance.”

“They’re brilliant – very, very good. They’re on top of paperwork and want to help out. They

have quite a lot of experience which is important, especially as this is our first ILM contract.”

One individual compared the programme to others they ran and highlighted that “it has been

brilliant – a breath of fresh air compared to the Rural Development Plan. All the focus was there and

it wasn’t heavy handed.” Only four individuals suggested improvements to the support, which

included:



• A need to arrange events between organisations and employers;

• The “frustrating” means of dealing with paperwork over fax;

• The amount of paperwork involved; and

• A lack of communication between the ILM and FJF team leading to “doubling the amount of

paperwork.”

Overall, it is reassuring to see that contractors are positive in their views of WCVA’s support

arrangements. The evidence also strengthens the views of strategic and delivery stakeholders as

previously discussed.

3.2.2. Monitoring arrangements

Twenty-three of the 32 contractors did not have an issue with the monitoring arrangements of the

programme with people’s response ranging from “OK” or “fine” to “very good” or “great.” The

following comments highlight some of the positive views expressed by individuals:

“The monitoring is great – it’s regular and supportive. The level of documentation just has

to be there.”

“It’s incredible – there are no other contracts like it. The support is up front.”

However, nine of the contractors expressed less positive views during discussions on the matter.

Seven of these referred to the “overwhelming” amount of paperwork although one of these

contractors was sensitive to the fact.

“I have never worked in a job with so much paperwork before. Having said that, I can’t see a

way around it.”

Four contractors also highlighted the following points when discussing the programme’s monitoring

arrangements:

• The difficulty in collecting soft skills evidence from participants;

• Monitoring visits from WCVA can be long; and

• It is difficult to gather evidence of those participants involved with external placements.

It seems that some of the contractors have been on a very steep learning curve in terms of the

process involved with the delivery of European funded projects; something which not all contractors

had experience of undertaking. Those individuals involved may have been very experienced within

their field of work but not necessarily experienced in terms of how a publically funded project -

which inevitably have a certain level of bureaucracy within them - should, or must, be delivered.

3.2.3. Good practice examples

Seven of the 32 contractors discussed the importance of networking when reflecting on good

practice examples.

“Going into places and holding face-to-face meetings has been the best way of getting

placements for participants.”

Three of the contractors specifically mentioned the importance of working with local businesses and

external providers which helped bring opportunities to participants.



Five contractors referred to the way in which they managed activities. Three of these contractors

went on to discuss their experiences of working with specific target groups whilst others went on to

discuss the importance of “treating all members of staff the same” and ensuring that staff are

“versatile” to participants’ needs.

Four contractors highlighted the importance of holding one-to-one meetings on a regular basis with

participants. There is a suggestion that this is especially important with some groups, particularly

those with learning difficulties. One of the contractors working with lone parents in the RCE area

however did not feel that this was making best use of time and, instead, decided to set up a weekly

job club. A further four contractors discussed the importance of providing training to participants.

These are interesting points which highlight the need to consider the target groups the organisations

are dealing with along with adding value to the programme by drawing on external expertise.

In addition to the themes raised already, other interesting examples were also raised by individual

contractors. These included:

• Focusing participants on preparing for one specific event;

• Being transparent with participants;

• Providing external placements; and

• Raising awareness of the programme by providing leaflets to potential participants and

businesses.

3.2.4. Sharing best practice

It became clear that 11 of the 32 contractors interviewed in 2011 had attended a networking event

with other ILM contractors. Eight of the 11 contractors viewed this opportunity positively with one

commenting that “it was useful to look at other people’s ideas as it helped to solve some issues. For

instance, it was useful to look at how people measured soft outcomes; how they succeeded to get

people into work; and how to sell the ILM to job centres.” Other contractors raised similar issues with

two of these specifically noting the benefits of holding the session early in their project.

Two of the more experienced contractors however were less positive with their views as they felt

the session was tailored for those who had never delivered an ILM contract before.

“It was of little benefit really. People who attended were not delivering... It discussed setting

up the contracts rather than establishing. It was also a structured meeting and there wasn’t

a great deal of opportunity to network with others. It would have been useful to have this

earlier.”

A further two contractors also touched upon the fact that the session was structured with little time

to discuss and network with others in a similar position. Whilst realising the challenges of organising

different types of events, the evidence suggests that a more tailored approach could be adopted

depending on the past experiences of representatives from the contracted organisations.

3.2.5. Other factors impacting delivery

Contractors were asked whether there had been any external factors which have affected the

delivery and / or the impact of the ILM. Twelve contractors raised the state of the local labour

market during discussions, as highlighted by the comments below:

“The labour market is a problem. We’re lucky that Tesco is here.”



“In Newport there are around 17 people going for one job.”

“It is difficult to find placements [for the participant]. Existing organisations are finding it

difficult to be seen taking people on after making people redundant.”

Half of these contractors were operating in the RCE areas (including Powys, Vale of Glamorgan,

Newport, Flintshire, Wrexham, Cardiff and Monmouthshire) with half in the Convergence areas

(including Denbighshire, Anglesey, Conwy and Gwynedd). One of the contractors in the RCE area

expressed concerns with the hard target outputs attached to the programme and argued “there are

just not enough placements around here.” In contrast, a respondent working in the Cardiff area felt

that the weak position of the local labour market actually made it easier to find jobs for participants.

It was argued that employers are now seeking ‘value-for-money’ and ‘short-term solutions’ rather

than employing people on longer-term contracts.

Four contractors discussed the challenges for participants who need to travel to their jobs within the

rural areas of Gwynedd, Monmouthshire and Powys whilst two contractors raised the difficulty in

finding jobs for participants. According to one contractor, a number of other providers also want to

find jobs for their participants, reaffirming the concerns expressed by strategic and delivery level

stakeholders regarding competition amongst schemes and programmes.

A number of other issues were also raised by individual organisations including:

• Public holidays taking the continuity out of delivery;

• The changes to tax credit legislation reducing the amount paid towards childcare leading to

some participants questioning the advantages of being in work;

• A higher percentage of the younger generation (18-24) have no work ethic according to one

contractor who has delivered this type of support over the years;

• The FJF coming to an end will lead to a reduced period of support; and

• Less money available for training.

3.3. Benefits of the ILM programme

The interviews with 32 ILM contractors along with the interviews with 26 organisations employing

ILM participants (which included 12 who had originally been contracted to WCVA as ILM contractors)

in 2011 provided insight into their thoughts on the benefits and positive impact of being involved

with the ILM.

3.3.1. Benefits to organisations

Eleven contractors stated that the programme has contributed to an increase in the level of

partnership working with other organisations.

“It’s still early days but getting our name out there and enabling us to engage has been the

main impact from the organisation’s point of view.”

It appears that links have developed between private sector organisations, colleges and other local

employment projects. Three contractors discussed the importance of developing networks in new

and unfamiliar areas whilst one contractor based in North West Wales specifically mentioned the

importance of establishing networks in a rural area.



Fourteen contractors also described how the programme has played a part in developing the skills of

their employees. Respondents made reference to the following skills and experiences when

elaborating on the matter:

• Increased knowledge of employment issues;

• Experience of awarding qualifications;

• Developing teaching and tutoring skills and experience;

• An increased understanding of the support available to individuals;

• Developed knowledge and experience of the tendering process;

• Developed knowledge surrounding self-employment; and

• Experience of working with different people.

The issue of capacity building within organisations was also explored when undertaking a review of

the procurement process with contractors and non-contractors in 2010. Of the 14 organisations

stating that they had learnt new skills, seven referred to the skills of completing PQQs and tenders,

which also involves seeking contract notices from Sell2Wales. One individual even mentioned that

they have been involved with helping other previously unsuccessful bidders to complete and pass

their ILM PQQs. Other skills identified as having been developed during the procurement process

included:

• Sharing experience and skills;

• Developing policies and accreditations, especially health and safety;

• How to write precisely;

• Co-ordination between internal departments;

• Transferable skills; and

• Financial skills.

In-depth interviews with contracted and non-contracted organisations provided an opportunity to

delve deeper into the issue of capacity building and skills development. During discussions with

contracted organisations it became evident that the process has had a positive impact on the bid

writing skills and confidence of individuals. Although a mixture of organisations related to the impact

on their organisations, it was clear from interviews that smaller organisations with little experience

of the process gained the most from being involved. It is also important to understand that in such

organisations the impact was on individuals within an organisation rather than on the organisation

as a whole, as summarised in the following comment.

"In this sector you do not have teams of people doing things so the effect is on me and my

finance guy."

It must be stressed however that having an impact on the skills of individuals in small organisations

can have a direct impact on the activities of these particular organisations. Another two contracted

organisations had clearly been working with other colleagues and found that the individual’s

experience coupled with the direct experience of being involved with the process had an impact on

the skills of those new to the process. It was also interesting to see that the process has prompted

some organisations to apply and update their internal policies and procedures. This is an important

step, especially considering that many third sector organisations will need to develop in such a way

in order to compete for future contracts issued using a competitive procurement process.



During interviews in 2011, a further four contractors made reference to the role of the programme

in building the capacity of their employees. One contractor also believed “It has [the ILM

programme] also introduced new skills. For example, we provide more training now and the team

have been able to work with different people in different areas.” This comment suggests that the

programme has allowed some organisations to venture into new areas of work.

This is something which was later raised by a couple of contractors. One contractor, for instance,

discussed how the programme has allowed the organisation to “build” the not-for-profit side of the

business - “it allows it [the not-for-profit side] to expand which then provides the opportunity to take

more of them [ILM participants] on.” Another contractor new to this field of work discussed how the

experience has been “a stepping stone into the training sector.” This particular organisation also

discussed how it has been a good way of coming into contact with WCVA. A longer established

contractor discussed how the experience could “attract further funding due to the development of

the portfolio.”

Considering the nature of some of the organisations involved with the programme it is no wonder

that a total of nine respondents stated that it has helped them “raise” or “maintain” their profile

within the local community.

“It has enabled us to engage with more people from the area and to reach out to more

young people in the area, not just the homeless, but others who may not be homeless but

still have real issues and who are out of work. It has improved the profile of our organisation

within the local community.”

Three of the contractors stressed the importance of being proactive in attempts to engage with local

people and local providers in order to raise awareness of their services.

The discussions also demonstrate the benefit of being able to acquire additional employees without

losing out financially. A total of six contractors mentioned that the ILM has provided an opportunity

to recruit more people. One of the respondents mentioned that this “has helped keep it [the

business] going” whilst another has been able to expand. Interestingly, one contractor believed the

opportunity has led them to analyse the internal processes with the aim to “fine tune and improve

delivery.” This organisation has identified the need to address “gaps in HR” and to start thinking of a

succession plan for the organisation.

3.3.2. Benefits to participants

Although seven contractors stated that the employment outcomes are an important achievement –

“the figures tell it all – finding job opportunities is crucial” – the view of those interviewed was that

the soft outcomes of increasing confidence and self-esteem is the main impact on participants. One

of the contractors referred to the evidence within the WCVA soft skills questionnaire to strengthen

the argument whilst another referred to the discussions held between participants and support

staff.

“When they start they are shy and have no confidence, no work. We gradually build them to

a place where they are more confident and have self-esteem. They have all benefited

because none of them like the job centre because they just don't feel it helps them.”



Although it is clear that contractors feel that the support has made participants more confident

about their job prospects, it appears that they perceive that participants’ personal and social

confidence has also generally increased. A further 11 contractors discussed how the experience had

improved participants social skills. These contractors made reference to participants “opening up”

and “making conversation with other members of staff.” Contractors also referred to “improved

time-keeping,” “self worth,” “team work” and “empowerment.” One contactor also noticed a change

in some of the participants as they now “ask the leaders where they need to be” rather than the

leaders chasing them.

Seven contractors highlighted the importance of providing a flavour of work to participants as it

provides “an insight into the world of work” and a “chance to get a footing within the industry” or

“show that they can do the job.” This is an important step for many participants, especially those

who have never worked before or who have been unemployed for a long period of time. Five

contractors mentioned how the ILM job has helped people’s routines by being “a motivator to get

out of bed in the morning.” One of these respondents added that this has contributed to increasing

the fitness levels of their participants. Additionally, three contractors referred to the improvements

in people’s personal life, home life and mental health which again demonstrates the benefits being

involved with the programme can have on participants.

The evidence from 16 contractors also suggests that the training has helped develop the skills of

participants which, combined with qualifications, has contributed greatly to participants’ CVs. One

respondent discussed how this type of support contributed to the employability of one participant.

“One participant had a medical condition which held her back from going for jobs. She has

since gained experience and a lot of accreditations. She's now in full-time employment and

volunteers in her spare time.”

It is worth remembering that a large proportion of participants have been out of employment for a

long period of time and so the need to develop their CV and have references are crucial steps in

their attempt to find employment. Having said that, one contractor suggested that, in some

instances, the programme engages with those who may be relatively close to the labour market.

"We recruit more employable people, not those at the lower end [of the labour market].

But, it's not because of lack of trying. We sort of cherry pick without consciously doing so.

We pick people from the Engagement Gateway project for instance. The organisation might

have had a speculative letter from the person but there might not be any funds available.

Here, the ILM pays for it. Most of them would and probably could walk into the job

themselves."

This demonstrates that it is important to be aware that the needs of individual participants differ

and that the impact is likely to vary accordingly.

In addition to gaining the views of ILM contractors, the evaluation sought to capture the thoughts

and opinions of organisations that have employed ILM participants following their period with the

programme. Participants have been employed into different types of roles although the most

common type of work involves manual, administrative or office work. The feedback suggests that

the key traits of inter-personal skills, reliability, trust, communication, good character, empathy and

the correct attitude and application are more important than qualifications to those that have gone

on to employ past ILM participants. The following comment highlights the needs of one employer:



“Good attendance and time keeping is important. We needed someone who adapts quickly

and someone who gets on with the work.”

Having discovered the needs of employers at the time of recruiting it was important to explore

the progress of participants who have left the programme.

Twenty-three of the 26 employers discussed the progress of employees positively.

“Excellent! They have all progressed a long way. They have developed their confidence,

time-keeping and skills and now have a routine.”

“There has been excellent progress - from being young and unemployed to being in work

with confidence.”

Twenty-four of the 26 employers had also witnessed a positive change in the people who have been

placed with them.

Once again, the feedback suggests that improving people’s confidence has been the main impact.

“When they joined they were low on confidence and doubted their own skills of helping

others. There has been a big change since being with us.”

“Confidence has been the main development. There are definite changes – people are more

confident and more outgoing. They also enjoy being part of a team and work is now on top

of their agenda.”

The findings suggest that, overall, participants have settled well into their jobs. There is a suggestion

from two of the 12 contractors who have gone on to employ participants that they have adapted to

the environment because of their familiarity with the organisations.

“They have settled in easier than others because they are familiar with the organisation.”

Two organisations had however experienced some difficulties with their employees. Although one

stated that there “had been a few hiccups which are all sorted now,” the other organisation had to

take more serious action. This small business had employed a past ILM participant as an electrician

to carry out PAT testing. Although the individual had the necessary skills to do the work the

employer was unimpressed with the attitude shown towards customers and, unfortunately, the

individual’s employment was terminated. Both of the organisations that had experienced challenges

had not been delivering ILM contracts in the past and the latter business had not heard of the ILM

until the time of interview. This raises the question of whether individuals are more likely to develop

in an organisation familiar and supportive of some of the issues participants may be seeking to

overcome.



3.3.3. Wider benefits

In addition to the benefits for organisations and participants, contractors were asked whether there

had been any wider benefits of being involved with the ILM programme. A total of 14 contractors,

discussed how the support has led to community benefits. Contractors, for example, made reference

to the charity work which has taken place, the increase in the number of local volunteers and the

services provided to local people. With regards to the latter point, respondents discussed the

gardening activities, the selling of produce at a discount to low income families and the story of how

participants helped bring food in on the train to Amlwch, Anglesey during the snow over the

Christmas period in 2010.

Six contractors also discussed the importance of helping people to become less dependent on

benefits. Whilst helping individuals in their quest for employment it was also mentioned that this

contributes to current ‘national efforts’ by the Government to reform the welfare-to-work services.

Three contractors went on to discuss how this would also have a positive effect on young people. It

was argued that having local role models would help tackle the culture of unemployment and that

friends and family, with time, would realise the advantages of being in work.

Interestingly, an organisation working with ex-offenders noticed that participants have not been re-

offending since being involved with the programme. This is encouraging and demonstrates the

wider, positive impacts the programme can have. Another contractor mentioned how the

organisation now “thinks outside the box” and has committed to employing local people.

3.3.4. Reflecting on the ILM

Contractors identified a wide-range of strengths when reflecting on their involvement with the ILM

programme. The feedback can however be broadly grouped into five main themes:

• Impact on individuals;

• The work experience offered to participants;

• The training, skills and qualifications on offer;

• The ILM delivery model; and

• Benefits to contractors.

Twenty-eight contractors commented on the weaknesses of the programme. The following

highlights the main themes identified by contractors:

• Length of job placements; and

• The bureaucracy involved with the programme.

All 32 contractors were hopeful of delivering similar contracts to the ILM in future and all

contractors also stated that they would deliver the ILM again if possible. The following comments

express the opinions of two contractors:

“Yes we would like it to continue. We certainly plan to if the schemes continue.”

“It’s one of the best tools to progress people into employment.”

Despite this, three contractors did raise some concerns. Two contractors, for example, mentioned

that they would need to consider the financial aspect with one concerned with the short timescale

involved with delivering contracts.



4. Participant level evidence

This section of the report discusses the findings of 384 participants in the ILM programme

undertaken in 2012 and 169 participants interviewed in 201117

. Fifty-nine of those interviews in

2012 were second interviews with participants originally interviewed in 2011 in order to explore the

longer term outcomes of the support provided. Reference is also made to the monitoring data

collected by WCVA. The evaluation team also undertook 16 visits with 10 different organisations.

The background of these organisations and the impact on participants are summarised in the form

of a case study available in a separate Case Study Report.

Key findings

• Gaining employment was the main aspiration for 61% (234/382) of participants. Participants

were also eager to gain experience (27%; 102/382) and qualifications (10%; 39/382). According

to 75% (283/375) of respondents, the ILM programme has contributed to achieving these goals.

• Eighty-two percent (314/383) of respondents believed they had developed their job skills during

their time with the ILM programme. The main job skills developed according to participants

included communication (40%; 123/310), team working (11%; 35/310), and interview skills (9%;

27/310).

• Three-quarters of participants (75%; 288/381) perceived an increase in their confidence.

Further, three-quarters of respondents (75%; 286/382) stated that they perceived that they

were more likely to get a job as a result of their involvement with the ILM.

• Participants believed that the lack of work in the area (39%; 132/340), lack of experience on

their part (14%; 49/340), and lack of qualifications (8%; 27/340) were the main reasons why they

were finding it difficult to find a job; all of which are issues being tackled by the ILM.

• Although 55% (175/317) of participants interviewed who had left the programme had secured a

job after leaving the ILM, only 46% (146/317) stated that they were in employment at the time

of this research. A further 46% (147/317) were not in employment but seeking work.

• Eighty-three percent (145/174) of the respondents who were employed felt it was the sort of job

they wanted and 74% (111/150) said that they believed they were likely to stay in the job over

the next 12 months.

• At least 95% (360/377) of respondents would recommend the ILM programme to others in their

position with 83% (314/377) stating that they would ‘definitely’ do so.

17

It is important to note that the frequency of respondents noted within the analysis varies from question to question. This is because

some respondents did not answer all the questions and ‘non-response’ has been removed from the analysis.



4.1. Aspirations

Participants were asked to explain what they hoped to achieve before joining the ILM. The evidence

shows that gaining employment was the main aspiration amongst 61% (234/382) of participants.

Participants were also eager to gain experience (27%; 102/382) and qualifications (10%; 39/382).

According to 75% (283/375) of respondents, the ILM programme has contributed to achieving these

goals.

Table 2: Did the ILM help you achieve those goals?

Response Frequency Percent

Yes, a lot 198 53%

Yes, a little 85 23%

No 72 19%

Not sure 20 5%

Total 375 100%

Source: Wavehill telephone interviews

Interestingly, there has been a percentage decrease in those who feel the support has helped them

achieve these goals; in 2011, 92% (143/156) of participants believed the ILM has helped achieve

their goals. It is important to remember that most respondents were still involved with the ILM in

2011 whilst most of those responding to the 2012 survey have now left the programme. This may

suggest that views change once participants leave the programme; eighty-eight percent (63/72) of

those who said ‘no’ to the above question declared that getting a job was the main aspiration. It

seems that not succeeding with this goal has had an impact on people’s response which reaffirms

the belief that the slight decrease in positive attitude since 2011 has occurred due to changes in

people’s circumstances.

4.2. Soft outcomes

Eighty-two percent (314/383) of respondents believed they had developed their job skills during

their time with the ILM programme.

Table 3: Have you developed your job skills during your time with the ILM programme?


Yes, a lot 205 53%


No 62 16%

Not sure 7 2%

Total 383 100%


The main job skills developed according to participants include:

• Communication (40%; 123/310);

• Team working (11%; 35/310); and

• Interview skills (9%; 27/310).



A further 13% (41/310) raised the issue of confidence whilst other key skills developed included

manual skills (8%; 24/310) and IT skills (9%; 27/310).

The increase in confidence amongst participants was also prominent in the 2011 sample and this

was once again an opportunity to further investigate the issue. As highlighted below, three-quarters

of participants had perceived an increase in their confidence as a result of the support they had

received.

Table 4: Has your confidence changed as a result of the support you have received from the ILM

programme?


Yes, more confident 288 75%

Yes, less confident 2 1%

No 77 20%

Not sure 14 4%

Total 381 100%


The main reasons offered by those who perceived a positive change were:

• Speaking / socialising with other people (17%; 57/339);

• A general boost in confidence due to the type of experience gained (13%; 45/339); and

• Gained experience of working (9%; 32/339).

Whilst a large proportion of respondents perceived an increase in their confidence, 11% (36/339)

said that they were already confident and 6% (22/339) did not identify a change. Interestingly, the

ILM led to a decrease in confidence amongst 1% (5/339) of respondents whilst 1% (2/339) explained

that it had knocked their confidence. Whilst these figures are low, it is a reminder that offering the

right sort of support and to assess need on an individual basis is crucial to the success of such

programmes.

A comparison of the responses from 59 participants who completed the survey in both 2011 and

2012 suggests a small decrease in confidence over time.

Table 5: Confidence of participants in 2011 and 2012

Response 2011 response 2012 response Percentage

change Frequency Percent Frequency Percent

Yes, more

confident

54 92% 50 85% -7%

No 2 3% 7 12% +9%

Not sure 3 5% 2 3% -2%

Total 59 100% 59 100% -


One of the possible reasons for this is that the circumstances of the participants in question have

changed since the interview in 2011 was undertaken; in particular they are no longer being

supported by the ILM. It is also important to highlight that a change in an individual’s confidence

could frequently take place and this due to a number of factors including personal (e.g. family or

employment) or other reasons (e.g. the state of the labour market).



Despite this, discussions with participants once again highlighted the positive impact on their

confidence (45%; 138/307). Ten percent (31/307) believed the support also improved their self-

esteem with the following comments providing a flavour of the responses:

“[The ILM has] given me confidence and made me happier. Now that I’m working, I feel better

about myself.”

“Apart from an increased confidence in myself and my abilities I have become a lot happier in

everything I do which is an unusual thing to describe; but I love the job.”

“I became more sociable and able to talk to people again. When I was not very well I spent a

lot of time on my own. It was good to get into the work place and meet people and make

friends.”

In addition, other personal effects involved participants becoming more experienced (10%; 30/307),

becoming more positive (6%; 19/307), motivated (6%; 17/307) and happier (5%; 14/307). Other

wider effects such as making friends (25%; 36/142) were also identified.

Although having an impact on the skills and attitudes of participants are key aims of the ILM, the

ultimate goal is to influence the job prospect of participants and help them into employment. The

evaluation therefore asked participants to think of their situation before and after their involvement

with the programme. In the first instance, participants were asked to consider the likelihood of

getting a job before their involvement with the ILM. They were later asked to consider the likelihood

of getting a job after their involvement with the ILM.

Table 6: How likely do you think you were/are to get a job before and after your involvement in the

ILM?

Response Before After Percentage

change Frequency Percent Frequency Percent

Very likely 13 4% 137 46% +42%

Likely 84 25% 113 38% +13%

Unlikely 145 44% 31 10% -114%

Very unlikely 91 27% 17 6% -74%

Total 333 100% 298 100% -


The data suggests that respondents feel much more confident about their job prospects and the

success of 32% (119/373) in finding a job or work experience (18%; 67/373) has certainly contributed

to this change. The development of skills and qualifications (6%; 21/373) was also given as a reason.

In contrast, however, 14% (29/373) were concerned about the lack of jobs in the local area.

The fact that 75% (286/382) of respondents perceived that they were more likely to get a job as a

result of their involvement with the ILM supports the conclusion that the ILM programme has had a

positive impact in respects of perceived job prospects. It is however interesting to note that 61%

(44/71) of those who stated that they were likely to find a job before their involvement with the ILM

had actually managed to do so.



Graph 4: Are you more likely to get a job as a result of your involvement with the ILM?


Base=382

When asked to explain their response, 17% (55/315) of respondents referred to the fact that they

had indeed found a job. Others referred to the experience (37%; 116/315), improved CV (10%;

33/315), confidence (10%; 31/315) and skills (7%; 23/315) they had gained as a result of the support.

Once more, ‘skills’ and ‘qualifications’ appear to be key factors. Responses to the survey suggest that

the most prominent qualifications amongst respondents prior to their involvement with the ILM

were:

• GCSE (63%; 241/384);

• Driving licence (31%; 120/384);

• A-levels (22%; 84/384); and

• NVQ (18%; 70/384).

According to the programme’s monitoring data, a total of 1,959 participants have gained a

qualification since being involved with the ILM and this could well support the reasons given by

respondents in explaining their positive attitude towards future employment.

One method of testing whether the views of participants have changed over a period of time was by

studying how they described their job prospects before and after they had been supported by ILM.

The words most prominently used are illustrated in the diagram below.

Yes, definitely,

48%

Yes, a little,

27%

No, 10%

Not sure, 15%



Diagram 2: What three words would you use to describe your job prospects BEFORE and AFTER

joining the ILM?


In addition to the figure above, Appendix 2 demonstrates this change in the form of a Wordle where

the most common terms used to describe their job prospects appear in large font and the less

common words appear in smaller font. The results show that the views of participants have radically

changed from being negative prior to accessing the support to being positive after joining the ILM.

A similar trend was apparent in the 2011 survey and the 2011 evaluation report highlighted that this

change had happened over a short period of time as the majority of participants, at the time, stated

that they had been in their ILM job for less than four weeks (16%; 27/169) or for more than four

weeks but had more than four weeks remaining (64%; 109/169).

Reassuringly, views have remained positive over a period of time. Eighty-three percent (317/383) of

participants responding to the 2012 survey had exited the programme and finished their ILM jobs.

This may be further tested by honing in on the 59 respondents who completed the survey in 2011

and 2012.

Diagram 3: What three words would you use to describe your job prospects BEFORE and AFTER

joining the ILM?


BEFORE

1. Poor (18%; 71/384)

2. Hopeless (8%; 31/384)

3. Difficult (8%; 30/384)

AFTER

1. Better (28%; 97/351)

2. Improved (18%; 64/351)

3. Positive (12%; 42/351)

BEFORE

1. Poor (15%; 8/55)

2. Unlikely (13%; 7/55)

3. Hopeless (11%; 6/55)

2011

1. Confident (42%; 25/59)

2. Better (25%; 15/59)

3. Good (17%; 10/59)

2012

1. Better (20%; 12/59)

2. Confident (14%; 8/59)

3. Improved (14%; 8/59)



As shown above, there has been little change in people’s descriptions between 2011 and 2012 and

this despite all of the participants completing their period of support with the ILM. There is, in

contrast, a significant change prior to these individuals accessing the support. It may well be that this

change in attitude occurred due to the difficulties respondents have had in finding jobs before

accessing the support, as highlighted in the table below.

Table 7: Did you find it difficult to find a job before being involved with the ILM?


Yes, definitely 244 64%


Not sure 8 2%

No 30 8%

Total 384 100%


Some of the issues which have already been outlined were once again raised when discussing why

respondents were finding it difficult to find a job. The majority, for example, referred to the

following issues:

• Lack of work in the area (39%; 132/340);

• Lack of experience on their behalf (14%; 49/340); and

• Lack of qualifications (8%; 27/340).

The feedback from participants strengthens the belief that the ILM are addressing the key issues in

tackling unemployment. Although 40% (139/346) of respondents who found it difficult to find a job

before being involved with the ILM had been out of work for less than six months, 45% (120/346) of

respondents had been out of work for over a year. Sixty-three percent (19/30) of the respondents

who did not find it difficult had been out of work for less than six months.

Studying the response of participants to a series of statements also provided an idea of the progress

they have made since becoming involved with the ILM.



Table 8: To what extent do you agree or disagree with the following statements?

Statement Strongly Agree Agree

Neither Agree

nor Disagree Disagree

Strongly

disagree

Frequency Percent Frequency Percent Frequency Percent Frequency Percent Frequency Percent

I feel that

I made

good

progress

whilst at

the ILM

130 34% 222 58% 12 3% 20 5% 0 0

I now

have a

better

idea of

what job I

want to

do in the

future

95 25% 180 47% 69 18% 39 10% 1 0%

The ILM

job has

prepared

me well

for future

work

95 25% 219 57% 30 8% 36 9% 2 1%

I feel

work

ready as a

result of

the

support

received

from the

ILM

100 26% 211 56% 41 11% 28 7% 1 0%


The table above once more highlights the positive strides individuals have made as a result of the

support. Considering that 83% (317/383) of individuals responding to the survey had already finished

their ILM job, this was a good opportunity to reflect on the support they had received.

Ninety-two percent (352/384) of respondents for example felt they have made good progress whilst

at the ILM. It becomes clear from studying the figures above that the ILM has helped provide

direction for the majority who have been involved. Seventy-two percent (275/384) of respondents

stated that they now have a better idea of what job they want to do in the future.



Importantly, the responses also provide an indication of the role the programme has played in

helping to prepare participants for future work. As shown in the table above, 82% (314/382) of

respondents felt the ILM has prepared them well for future work whilst 81% (311/381) feel work

ready as a result of the support. Seven percent (29/380) of respondents on the other hand did not

feel work ready as a result of the support. This is an interesting finding, especially as 93% (27/29) of

these individuals had already finished their ILM job. This may suggest that the length of contract was

not sufficient or that they did not gain the experiences they sought.

When asked what respondents did immediately after leaving the ILM, 23% (73/316) found

themselves in a position where they were applying for jobs or where they returned to job seekers

allowance (17%; 54/316) - “I signed back on the dole.” In addition, whilst 69% (219/317) of those

who had completed their ILM job had been applying for positions it is a concern that 31% (98/317)

have not yet done so. A further concern is that those who have been applying for positions have

found themselves applying for an incredible amount without any luck. Discussion with some

respondents for example helped to paint a picture of their situation with around 8% (14/176)

applying for over 100 jobs. The majority however (38%; 67/176) applied for less than 10 jobs. The

jobs that respondents applied for varied a great deal although the most common involved:

• Retail work (22%; 49/218);

• Administrative work (21%; 45/218);

• Factory work (10%; 21/218); and

• Bar work / waitressing (7%; 15/218).

A total of 60% (132/219) of respondents had managed to reach the interview stage. In light of the

feedback from those who had not secured a job following their involvement with the ILM, it may be

argued that further support is needed in order to help them into employment. Whilst all individual

cases vary it is likely that the unstable economic environment has contributed to increasing the

challenge for some respondents.



4.3. Job outcomes

Within the 2012 sample, 55% (175/317) of respondents who had left the programme had secured a

job after leaving the ILM. Fifty-eight percent (102/175) of those that had secured a job had done so

immediately after leaving the programme.

Table 9: How soon afterwards did you get the job?

Time period Frequency Percent

Before the end of the ILM 8 5%

Immediately afterwards 102 59%

Less than 1 month afterwards 28 16%

1-6 months afterwards 32 18%

6-12 months afterwards 1 1%

A year later 2 1%

Total 173 100%


It is interesting to note that proportionately more beneficiaries from the old North Wales LMI

programme were in employment at the time of a telephone survey undertaken (as part of an

evaluation of that programme also undertaken by Wavehill) than had achieved job outcomes

immediately following their LMI employment. Whilst it was, at the time, possible to speculate that

many LMI employees may therefore go on to employment at a later date as a result of the support

of the LMI, the evidence suggests that the opposite may be true with the current ILM programme

possibly due to the different prevailing economic conditions. This is however difficult to assess with

any confidence.

Further analysis of the 2012 survey indicates that 51% (89/175) of respondents managing to find a

job had been out of employment for less than six months prior to joining the ILM. The figures

suggest that, the less amount of time a participant has been out of work, the more likely they are to

find employment opportunities.



Graph 5: The length of time those who managed to find employment were out of employment before

joining the ILM


Base=175

As demonstrated above however, a key strength of the ILM is the opportunity it offers for those who

have been out of work for a longer period of time. The analysis highlights that all of those who had

found a job had completed their ILM job. This perhaps suggests that participants are more likely to

improve their situation if they manage to stay with the ILM for the duration. And, 88% (153/174) of

respondents believed their time at the ILM helped them to get a job.

Graph 6: Do you think your time at the ILM helped you to get this job?


Base=174

51%

21%17%

11%

Less than 6 months Between 7 and 12

months

Between 1 and 2

years

Over 2 years

Yes, definitely,

74%

Yes, a little, 14%

No, 11%

Not sure, 1%



Interestingly, only 21% (36/175) of those who found employment had previously undertaken any

work-placements or any job-related training schemes. The perception amongst stakeholders that

combining the Future Jobs Fund and ILM will lead to greater impact in terms of job creation cannot

therefore be proven.

However, it seems that respondents’ involvement with contracted organisations (a period extended

by Future Jobs Fund) was the main reason they managed to find a job; 32% (53/164) of respondents

stated that they managed to get a job through their ILM placement. Whilst this goes some way in

explaining why the majority of participants found a job immediately after leaving the ILM, it also

highlights the importance contracted organisations have had in enabling the ILM programme to

move participants into employment. Other respondents referred to the boost in confidence from

being on the ILM (21%; 35/164) and through training received (9%; 15/164).

Eighty-three percent (145/174) of respondents felt it was the sort of job they wanted, as highlighted

below.

Table 12: Was this the sort of job you wanted?


Yes, definitely 109 62%


No 19 11%

Not sure 10 6%

Total 174 100%


Whilst recognising the difficulty for participants to answer whether they are likely to stay in a job

over the next 12 months, the response is interesting. Seventy-four percent (111/150) of respondents

were likely to stay in the job over the next 12 months with 61% (91/150) stating that they were ‘very

likely’ and a further 13% (20/150) stating ‘likely.’

“It is a permanent job and I enjoy it.”

“I’m in for the long run.”

“I want to advance and stay in this company.”

Twenty-six percent (39/150) of respondents believed they were unlikely to stay in the job over the

next 12 months (with 21%; 31/150 stating ‘very unlikely’ and 5%; 8/150 stating ‘unlikely’).

The 2012 survey with those who had completed their ILM jobs was an opportunity to identify

whether they had accessed training since leaving. Twenty-six percent (83/314) of respondents

received training after leaving the ILM with first aid (11%; 9/82) and health and safety (10%; 8/82)

featuring prominently.



Although 55% (175/317) of respondents who had left the programme had secured a job after leaving

the ILM, only 46% (146/317) stated that they were in employment at the time of the interview. As

demonstrated below, a further 46% (147/317) were not in employment and seeking work.

Table 13: Which of the following statements best describes your current situation?

Statement Frequency Percent

I am currently in employment 146 46%

I am currently in full-time education / going into full-time

education 11 4%

I am currently undertaking part-time education / training 3 1%

I am not currently in employment and am not seeking work 10 3%

I am not currently in employment, but am seeking work 147 46%

Total 317 100%


4.4. The support structure

In addition to providing an insight into the impact experienced by respondents, the survey provided

an opportunity to consider the nature and relevance of the support received. Participants were

asked to indicate to what extent they agreed or disagreed with a series of statements about the

support they received.



Table 14: To what extent do you agree or disagree with the following statements?

Statement Strongly Agree Agree

Neither Agree

nor Disagree Disagree

Strongly

disagree

Frequency Percent Frequency Percent Frequency Percent Frequency Percent Frequency Percent

The ILM

where I

worked

understood

my needs

100 26% 246 64% 15 4% 20 5% 3 1%

I received

enough

training at

my ILM job

83 22% 226 58% 25 7% 46 12% 4 1%

I received

the right

sort of

training at

my ILM job

79 21% 231 60% 34 9% 36 9% 4 1%

I received

enough

support at

my ILM job

124 32% 216 56% 19 5% 19 5% 5 2%

The ILM

job

matched

my needs

89 23% 229 59% 30 8% 29 8% 6 2%


Overall, the figures indicate that respondents are satisfied with the support and the following

conclusions can be drawn as a result of the feedback:

• 90% (346/384) believed the ILM where they worked understood their needs;

• 80% (309/384) believed they received enough training at their ILM job;

• 81% (310/384) believed they received the right sort of training at the ILM;

• 89% (340/383) believed they received enough support at their ILM job; and

• 83% (318/383) believed the ILM matched their needs.

It is important to consider that 56% (213/383) of participants responding to the survey were located

within an external organisation delivering on behalf of ILM organisations whilst 44% (170/383) were

located with organisations managing the ILM. There is no evidence to suggest that one model fits

better than the other when reflecting on how satisfied respondents are.



Participants were asked to identify the best thing about the ILM. The most common response was

made around the experience on offer (38%; 130/345), as demonstrated by the following comments:

“It gives people a chance to get experience in different jobs.”

“Just getting some hands-on experience.”

Others referred to the confidence (16%; 54/345) and qualifications (13%; 44/345) they gained.

Interestingly, despite being a key factor, the wage (11%; 39/345) and employment opportunities

(10%; 35/345) were not amongst the most common responses offered.

A look back at the 2011 survey results shows that there has been a slight shift in terms of what were

considered as main strengths. The most common response in 2011 included comments around the

benefits associated with training (30%; 50/165), being in a job (17%; 27/165) and gaining experience

(16%; 26/165). Considering that 89% (317/383) of individuals responding to the survey had already

finished their job or had four weeks remaining compared to 17% (28/269) in 2011, it is not

unreasonable to think that their change in circumstances have led to a slight change in the way they

view the benefits of the ILM.

Having said this, further analysis of the responses given by those who responded to the survey twice

shows there has been little change with time. The best thing about the ILM according to the majority

of respondents in 2011 included:

• Paid work experience (34%; 20/59);

• Improve skills (22%; 13/59);

• Experience (19%; 11/59); and

• Confidence (14%; 8/59).

A similar explanation was offered in 2012 although more emphasis was provided on the benefits of

increasing confidence, as highlighted below.

• Paid work experience (23%; 13/57);

• Improve skills (19%; 11/57);

• Confidence (19%; 11/57); and

• Experience (11%; 6/57).

The findings therefore suggest that a number of positives were outlined by respondents and the

number of different respondents answering the survey contributed to the slight shift between the

most common responses offered in 2011 and 2012.

The views of participants are crucial in order to identify possible improvements to the programme.

Whilst a number of comments were made around a range of different issues including the need for

more tailored training (5%; 11/218) and the need to receive training certificates earlier (3%; 7/218),

the majority made comments around the following:

• The need for a longer job placement (25%; 54/218);

• Greater / better organisation (8%; 18/218); and

• Managing the expectations of participants better and not mislead them about job availability at

the end (5%; 11/218).



When asked if there was anything else they would like to add about their experiences or the support

they had received the comments were mainly positive. Twenty percent (41/203) referred to the

“brilliant” support whilst 17% (35/203) believed the ILM was great. Others mentioned how much

they had enjoyed (8%; 16/203) and how positive the experience had been (7%; 14/203). The

comments varied considerably although the following provide a flavour:

“I am glad I was on the ILM as it gave me the confidence and experience in getting employed.”

“I think going on the ILM saved me and my family from a lot of stress and heartache as I had

no job and the experience I gained through the ILM helped me personally and financially and

finally helped me better myself.”

“I was very happy with the support I received.”

“I'd tell anyone to do it, young or old.”

“It's been astonishing. Thank you to WCVA for funding it!”

Indeed, when asked whether they would recommend the programme to others, it became clear that

participants are in favour of the support overall.

Graph 7: Would you recommend the ILM programme to others in your position?


Base=377

At least 95% (360/377) of respondents would recommend the ILM programme to others in their

position with 83% (314/377) stating that they would ‘definitely’ do so.

Yes, definitely,

83%

Yes, probably,

12%

No, 3%Not sure, 2%



5. Programme management evidence

A number of quantitative targets were set for the ILM programme as part of the approval of the

projects by WEFO. This section will firstly review the outputs and results of the programme before

reflecting on the soft outcomes data captured by WCVA as part of their monitoring data.

Key findings

• Monitoring data demonstrates that the programme has excelled in delivering against some of

the outputs (in both the Convergence and RCE areas). For example, the programme has engaged

effectively with NEET, female, BME and lone parent participants.

• Interestingly, the programme seems to have struggled with particular target groups and these

are prevalent across both areas. The target groups in question include older participants and

participants with work limiting health conditions and / or disability.

• It seems that the programme has not engaged as successfully with the economically inactive

cohort in the RCE areas in comparison to the Convergence areas.

• A key aim of the ILM programme is to help participants into full-time or part-time employment.

At the time of writing this report, the percentage achieved stood at 75% for both the

Convergence and RCE areas although the programme has succeeded in helping 760 (from a

target of 1,008) participants into employment in the Convergence areas and 316 (from a target

of 420) in the RCE areas.

• An analysis of participants’ location in Convergence areas suggests that the programme has

engaged relatively evenly across different Local Authority areas. However, an analysis of

participants’ location within RCE areas shows a higher uptake of the support in the Wrexham

area. One possible explanation for this is the location of the core ILM team in Rhyl, North Wales.

• Overall, WCVA monitoring data demonstrates a significant increase in participants’ levels of soft

skills between the start and the end of the intervention.



5.1. Outputs and results

This section reviews the achievements of the ILM programme against target outputs and results

before analysing the characteristics of those who have been involved to date.

5.1.1. Achievements to date

Table 15: ILM Programme outputs and results18

Convergence areas RCE areas

Outputs Target Achieved

to date

% of target

achieved Target

Achieved

to date

% of target

achieved

Economically inactive 630 661 105% 525 251 48%

Unemployed 27 1,373 5,085% 225 758 337%

NEET participants 72 891 1,238% 15 528 3,520%

Female participants 380 702 185% 317 402 127%

BME participants 18 31 172% 15 43 287%

Older participants (aged

50+) 315 242 77% 263 100 38%

Lone parents 72 212 294% 60 164 273%

Participants with work

limiting health conditions

and/or disability

300 124 41% 250 72 28%

Employers Assisted 750 - - 200 - -

Results Target Achieved

to date

% of target

achieved Target

Achieved

to date

% of target

achieved

Participants gaining

qualifications 1,404 1,261 89% 585 698 119%

Participants entering

employment (Full-time and

part-time) 1,008 760 75% 420 316 75%

Sustained employment (in

work for 52 weeks) 705 - - 277 - -

Participants entering further

learning 180 48 27% 45 27 60%

Participants gaining other

positive outcomes (e.g.

voluntary work or self-

employment)

1,764 121 7% 735 88 12%

Employers adopting or

improving equality and

diversity strategies

113 - - 30 - -

Source: WCVA

18

It should be noted that the targets noted here are those within the original programme business plan.



The table above sets out the targets established for the ILM programme in terms of outputs and

results. It also notes the numbers achieved to date and what percentage of the target that

represents. All actual output figures cited below are true as of the end of May 2012 and are based

on the ILM team’s own dataset. It should be highlighted that the figures below illustrate the total

number of people engaged with the programme, including those who did not complete their

placement period. At the time of this report, 207 participants had left the programme early in the

Convergence areas and 155 in the RCE areas.

As highlighted below, there has been a significant increase in the number of participants engaged

with the programme since the figures quoted in the 2010 and 2011 reports (September 2010 and

June 2011).

Graph 8: Number of participants in the Convergence and RCE areas during the three stages of the

evaluation

Source: WCVA

Importantly, there has been a significant increase in the number of participants over the past 11

months. This increase is reassuring and is evidence that the programme has effectively engaged with

participants despite the early delays. Overall, the figures suggest that the team have successfully

moved from a position of establishing processes to delivering against their targets.

Table 15 demonstrates that the programme has excelled in delivering against some of the outputs in

particular. In both the Convergence and RCE areas, for example, the programme has engaged

effectively with NEET, female, BME and lone parent participants. Interestingly, the programme

seems to have struggled with particular target groups and these are prevalent across both areas.

These target groups include older participants and participants with work limiting health conditions

and / or disability although it seems that the team have not engaged as successfully in the RCE areas

with the economically inactive in comparison to the Convergence areas.

The economically inactive are those people who are neither in employment nor unemployment and

may include, for example, those who are looking after the family and / or home or retired. This may

suggest that the ILM team’s links with referral agencies and possible contractors are not as

established within the RCE areas. This is perhaps strengthened when considering the higher

engagement figure with this particular target group in the Convergence areas of Wales; the areas in

which the team have greater experience of delivering an ILM / LMI programme.



There has also been an increase in the results generated by the programme since June 2011,

especially with regards to participants gaining qualifications and participants entering further

employment. Those gaining qualifications, for example, have increased from 357 to 1,261 in the

Convergence areas and from 312 to 698 in the RCE areas over the past 11 months. However, as the

figures suggest, the programme has some way to go at the time of writing this report if the targets

for sustained employment, further learning and other positive outcomes are reached before the end

of the programme. It should be noted however that the ILM team have began the process of

attempting to capture such evidence from those participants who have been involved with the

programme in the past. However, it is highly unlikely that targets for results such as sustained

employment will be achieved before the end of the programme in three months time.

Although the figures in Table 15 highlight the number of participants engaged it is necessary to

understand that not all participants will complete their placement period. The management data for

instance shows that there have been 207 ‘early leavers’ in the Convergence areas and 155 in the RCE

areas.

Table 16: Status of ILM participants as of the end of May 2012

Status

Convergence areas RCE areas

No. % No. %

Completer 1,367 67% 655 65%

Early lever 207 10% 155 15%

Not specified / determined 461 23% 199 20%

Total 2,035 100% 1,009 100%

Source: WCVA

The management data also provides insight into the reasons for leaving the programme early. The

main reasons in both the Convergence and RCE areas are:

• Misconduct issues (26%; 54/207 in the Convergence areas and 22%; 34/155 in the RCE areas);

and

• Lack of attendance / failure to return to work issues (19%; 40/207 in the Convergence areas and

20%; 31/207 in the RCE areas).

It also emerged that participants succeeded to gain employment in the Convergence areas (16%;

34/207) whilst medical / health issues were prevalent in both areas (10%; 21/207 in the

Convergence areas and 14%; 21/155 in the RCE areas).

5.1.2. The characteristics of participants

The following table outlines the key characteristics of participants in each area. The characteristics of

participants are summarised in a graph format in Appendix 4 of this report.



Table 17: Characteristics of participants

Participant group Convergence RCE

Gender Male – 66% (1333/2,035)

Female – 34% (702/2,035)

Male – 60% (607/1,009)

Female – 40% (402/1,009)

Age

16-24 – 45% (919/2,035)

25-54 – 48% (967/2,035)

55-64 – 7% (146/2,035)

65+ - 0% (3/2,035)

16-24 – 54% (550/1,009)

25-54 – 41% (409/1,009)

55-64 –5% (50/1,009)

NEET 44% (891/2,035) 52% (528/1,009)

Lone parents 10% (212/2,035) 16% (164/1,009)

Work limiting health condition 6% (124/2,035) 7% (72/1,009)

BME 2% (31/2,035) 4% (43/1,009)

Source: WCVA

Reflecting on the table above and the outputs to date suggest that the programme has had more

success in engaging with some target groups compared to others. As already highlighted, the

involvement of older participants and participants with work limiting health conditions and / or a

disability could have been higher. Having said this, the ILM programme has engaged with a number

of target groups and this despite not being registered as headline targets. A review of the

management data shows that the programme has engaged with other key groups.

Table 18: Other target groups

Participant group Convergence RCE

Those based in rural areas 47% (964/2,035) 24% (243/1,009)

Homeless 2% (45/2,035) 1% (12/1,009)

Participants without any

qualifications 24% (498/2,035) 26% (259/1,009)

Ex forces 2% (46/2,035) 2% (17/1,009)

Ex Offenders 18% (376/2,035) 9% (92/1,009)

Second language English

speakers 7% (150/2,035) 3% (29/1,009)

Source: WCVA



An analysis of participants’ location in Convergence areas suggests that the programme has engaged

relatively evenly across different Local Authority areas. An analysis of participants’ location within

RCE areas shows a higher uptake of the support in the Wrexham area. One of the reasons for this

may be the location of the core ILM team in Rhyl and the links to the previous North Wales LMI

programme. Analysis of the management data also shows that four organisations have been

contracted to deliver the programme in the Wrexham area and all four were previously involved

with the old LMI programme.

A total of 22 contracts averaging 15 months each were commissioned in the RCE areas with these

valued at just over £3 million with a match of just shy of £1 million and a Future Jobs Fund match of

£776,000. Organisations worked with between four to eighty-eight participants, at an average of 36

participants per organisation.

In the Convergence areas, a total of 45 contracts averaging 14.5 months were commissioned. These

were valued at a total of nearly £8.5 million with a match element of £2.3 million and a Future Jobs

Fund element of nearly £580,000. Contractors in these areas worked with between 20 and 75

participants giving an average of 40 ILM participants per organisation.

As illustrated in the map below, these contracts led to the programme working with individuals from

the most deprived areas of Wales.



Map 1: Location of ILM participants

Source data: WCVA

Map produced for Wavehill by Map Analysis



5.2. Outcomes and impacts

This section reviews the outcomes and impact of the ILM programme by drawing on the evidence

gathered through the programme’s soft skills questionnaire (see Appendix 1 for further

information).

5.2.1. Employment outcomes

A key aim of the ILM programme is to help participants into full-time or part-time employment. The

percentage target achieved to date stands at 75% for both the Convergence and RCE areas although

the programme has succeeded to help 760 (from a target of 1,008) participants into employment in

the Convergence areas whilst helping 316 (from a target of 420) in the RCE areas. Issues such as the

weak economic climate may have contributed to this challenge although it is likely that these figures

will increase as the programme nears its final date and as WCVA continues to collect updated

evidence from past participants. Discussions with the management team however suggest that it has

been a challenge to gather the necessary evidence from participants after they have left the

programme.

5.2.2. Soft outcomes and distance travelled by participants

As previously discussed, the ILM programme provides opportunities for participants to work on a set

of skills, all of which are deemed to be important facilitators to entering and remaining in

employment. Time in the programme is expected to lead to improvements in the following

outcomes:

• Communication Skills (reading, writing, comprehension and computer skills)

• Self Presentation (timeliness, physical presentation)

• Confidence (self-image, ability to ask questions, enjoyment meeting people)

• Motivation (seeing tasks through, concentration, perception of being trusted, independent

working, need for encouragement, liking challenges)

• Overcoming setbacks (ability to take criticism, prioritising, feeling positive about new things,

multi-tasking)

• Feelings (current happiness, optimism for future, managing negative feelings, managing violent

behaviour)

• Influencing Others (expression of opinions, ability to say ‘no’, defensiveness, ease around

others, empathy, perception of team working, willingness to follow regulations)

• Use of Numbers (confidence with addition, subtraction, multiplication and division, ability to use

a calculator)

• Knowledge of Opportunities (readiness to seek employment, understanding of strengths,

weaknesses and need for training, understanding of job search process, presentation and

interview process)



The soft skills questionnaire, designed to be filled out by participants at three stages of their

engagement with the ILM initiative (upon their entry into the programme, part-way through, and on

exiting the programme) asks participants to report about each of the above skills. In this way, the

soft skills questionnaire complements the other forms of data collected, by providing more detailed

evidence of the ways in which participants benefit from the programme. Importantly, administering

the questionnaires repeatedly during the initiative helps to capture dynamic changes that result

from the initiative, allowing measurement of progress, or ‘distance travelled’ by each participant.

Whilst typically only half of those participants who filled in the initial questionnaire also filled in the

final questionnaire (approximately 1,200 out of the initial 2,400), the final numbers available for this

report are large enough to allow for a more thorough examination of the change on soft skills than

was seen in the 2011 report. Interestingly, there were fewest responses to the mid-initiative

questionnaires. Indeed, it seems the ILM management are aware of this issue and decided, following

concerns expressed in the 2011 evaluation report about the number of forms being completed, not

to ask participants to complete a questionnaire mid-way through the support. For these reasons this

time point is left out in the following analyses, with the focus being on answers to the entry and exit

questionnaires.

As mentioned above, a number of the participants enter the programme with specific barriers to

employment, such as homelessness, alcohol and drug abuse. The time in the programme is also

expected to help participants address these issues and therefore changes in the following factors are

recorded over the course of the initiative:

• Accommodation (availability of accommodation, security of accommodation, use of a

telephone, space for writing of job applications)

• Alcohol (consumption of alcohol, feelings about alcohol consumption, volume of consumption)

• Drugs (consumption of recreational drugs and extent thereof, proportion of income spent on

drugs, feelings about drug consumption)

The analysis presented here uses the responses to the soft skills questionnaires to examine patterns

of progress, or ‘distance travelled’ by respondents throughout the ILM initiatives. The overall picture

of progress is broken down by the types of outcomes, as presented above, to provide a more

detailed picture of where the ILM programme is particularly effective, and where there is room for

improvement. In addition, the analyses will examine whether there are any notable differences in

soft skills development between the RCE and Convergence areas and between men and women. It

will also look at particular target groups separately, namely individuals over 50 years old, BME and

lone parent groups.

Average distance travelled

Table 19 examines all the valid questionnaire responses at the point of entry and exit of the ILM

initiative, reporting average scores on each of the soft skills outcomes, as well as an average of all of

these scores combined. Each outcome is reported on a scale of 1 through 4, where 4 represents the

most positive outcome. Due to the higher numbers of responses available, it was possible to use

inferential statistics to test whether the distance travelled on each of the soft skills dimensions was

statistically significant.



On average, participants enter the ILM programme with most positive scores on self presentation,

and feelings, and lowest scores on overcoming setbacks and knowledge of opportunities. Those

participants that completed the final questionnaire, and therefore completed the ILM programme,

exited with the same skill set as strongest: self presentation and feelings. However, their weakest

soft skills were slightly different than on entry, with most trouble experienced in relation to

influencing others and overcoming setbacks.

Table 19: Changes in soft skills scores across intervention for those who completed the exit

questionnaire

Soft skill outcome (max

score =4)

Average entry

score

Average exit

score

Average 'distance

travelled'

Test of

distance

travelled

Communication Skills 3.21 3.31 .10 ***

Self Presentation 3.63 3.63 .00

Confidence 3.14 3.31 .17 ***

Motivation 3.32 3.39 .07 ***

Overcoming Setbacks 3.02 3.15 .13 ***

Feelings 3.49 3.50 .01

Influencing Others 3.26 3.09 -.16 ***

Use of Numbers 3.29 3.40 .11 ***

Knowledge of

Opportunities 3.10 3.27 .17 ***

Overall 3.27 3.34 0.07 ***

*=p<0.1; **=p<0.05; ***=p<0.01

Sample size between 1211 and 1420, depending on the soft skill domain.

Source: Analysis of data collated by WCVA

Examining the final two columns in Table 19 shows average distance travelled on each soft skill

dimension, as well as overall (in the final row). Overall, the evidence suggests an increase in the soft

skills of participations between the start and the end of the intervention. The paired samples t-test19

suggests that this increase is statistically significant. Looking at the various soft skills dimensions

separately, the only significant decrease is seen with respect to respondent’s perceptions of their

being able to influence others. This is mirrored in the rest of the analyses presented here.

Otherwise, progress is particularly positive with regards to participant’s increasing knowledge of

opportunities, as well as their increased confidence. In addition to this, on average, knowledge of

opportunities is one of the weakest skills participants have on entry. The fact that this skill set is

increased more than most others indicates that this has been one of the strengths of the ILM

programme.

19

The Paired Samples T Test compares the means of two variables. It calculates the difference between the two variables

for each case, and tests to see if the average difference is significantly different.



Average change in barriers faced

As mentioned above, the soft skills questionnaire also collects data on participant’s handling of

barriers to employment, namely instability of accommodation arrangements, alcohol consumption

and drug consumption. Reports of these issues at the entry and exit from the ILM programme allow

us to reflect not only upon the ways in which the programme promotes enablers of employment,

but also how it helps to reduce barriers to employment.

Table 20 provides an overtime review of participants’ handling of these barriers. The table only

focuses on those participants who provided answers to the exit soft skills questionnaire, and thus

those who completed the intervention. It is important to note that these results are based on

particular filter questions. Those participants who answered the questions on accommodation

stability were in accommodation at the time of the questionnaire, and this was the vast majority of

those answering the soft skills questionnaire.

Only 1.5% (37/2512) of the total sample of participants on the ILM database were registered as

homeless. The aim of examining accommodation in the soft skills questionnaire, therefore, was to

examine stability of accommodation, rather than whether or not the individuals were in

accommodation at all. The participants who answered the questions on alcohol consumption had

consumed alcohol in the month prior to the questionnaire. Again, the majority of all the ILM

participants (67.9%; 1709/2512) fell into this category, and again the aim of the questions on alcohol

consumption was to grade the level of consumption and dependence. Those participants who

answered the questions on recreational drug consumption had used recreational drugs in the month

prior to the questionnaire. This group is, of course, much smaller than the other two groups

examined. The overall ILM database registers 10.2% (257/2512) of the participants as having either a

drug or alcohol consumption problem. As a result of these different filter questions, the sample sizes

for the different barrier questions vary enormously. Table 20 provides the valid sample size on which

each average score is based.

Table 20: Average change in barriers faced

Soft skill outcome (max score =4)

Average entry

score

Average exit

score

Average 'distance

travelled'

Accommodation stability (n=1190) 3.39 3.41 0.02

Control of alcohol consumption

(n=808) 2.89 2.88 -0.01

Control of drug consumption (67) 3.19 3.17 -0.02

Overall 3.16 3.16 0.00

Source: Analysis of data collated by WCVA

Table 20 indicates that there is much less change in terms of barriers over the course of the

intervention than what was seen with changes in soft skills of the participants. The final column

shows hardly any distance travelled on these dimensions. A paired samples t-test indicates that none

of these very slight changes are statistically significant. It is notable that control of alcohol

consumption at the point of entry and exit of the initiative is the lowest of the three barriers.

Evidence suggests that the comparatively low level of control of alcohol consumption is also not

addressed by the ILM programme.



The above sections charted progress made by all participants who answered the soft skills

questionnaires. The following sections break this progress down further, recognising that groups of

participants may differ in terms of their entry level soft skills and their soft skill needs, depending,

for example on their age and gender. Thanks to the higher response rate to the soft skills survey, as

compared to the response rate in 2011, it is now possible to specifically focus on the experience and

learning progress of groups such as ethnic minorities, single parents and older people. The rest of

this section examines any differences in distance travelled by funding area, by gender, by age, by

parent status (lone parents vs. non-lone parents) and by ethnic minority.

Distance travelled: Convergence vs. RCE areas

As detailed earlier in the report, the funding of the ILM programme differs between the

Convergence area and the RCE area in Wales. There is an unequal number of participants from the

two areas; almost two-thirds of the respondents to the soft skills questionnaires were from the

Convergence area (66%; 1648/2512). It is however relevant to examine whether there are any

differences in outcomes of the ILM programme across these two areas in order to learn lessons

about best practice in both areas. The data examined and interpreted here do not suggest reasons

for any differences found between the two areas. Potential explanations of any differences include

variations in funding structure, differences in the demographic makeup of the population, variations

in service provision in the areas, to name a few. We therefore ask the reader not to jump to any

conclusions about the root of any differences in outcomes found across the two areas.

Table 21 provides the average distance travelled by participants in both areas who completed the

final exit questionnaire, thereby completing the ILM programme. It also tests whether the distance

travelled is significantly different by funding area, as indicated by the asterisks against the final

column.

In comparison to the 2011 evaluation report, it is notable that with fuller data due to more

respondents from the two areas, the difference between the two funding areas is reduced. Looking

at the final row, which indicates the overall average distance travelled by participants, the data

indicates that there is no significant difference between outcomes across the funding areas.

Participants enter the programme with similar soft skill scores in both areas. Both areas see

participants in their ILM programmes exit with significantly higher soft skills than they entered.

Examining each of the soft skills dimensions separately confirms that there is little difference

between distance travelled on each of the dimensions by funding area. The only negative outcome –

the reduction in participant’s levels of influencing others – is a finding that is consistent in both

funding areas, although it is higher in the RCE area, as compared to the Convergence area.



Table 21: Average distance by funding area


Average

entry score

Average exit

score

Average 'distance

travelled'

Test of area

difference

Communication Skills Con 3.21 3.31 0.09

RCE 3.20 3.30 0.10

Self Presentation Con 3.63 3.63 0.00

RCE 3.63 3.63 0.00

Confidence Con 3.15 3.31 0.16

RCE 3.12 3.31 0.19

Motivation Con 3.33 3.40 0.08

RCE 3.31 3.38 0.07

Overcoming Setbacks Con 3.04 3.15 0.12

RCE 2.98 3.13 0.15

Feelings Con 3.51 3.51 0.00

RCE 3.45 3.48 0.03

Influencing Others Con 3.26 3.12 -0.14

RCE 3.25 3.04 -0.20 ***

Use of Numbers Con 3.31 3.42 0.11

RCE 3.24 3.36 0.12

Knowledge of

Opportunities

Con 3.09 3.26 0.17

RCE 3.11 3.29 0.19

Overall

Con 3.28 3.35 0.06

RCE 3.25 3.32 0.07

*=p<0.1; **=p<0.05; ***=p<0.01 Source: Analysis of data collated by WCVA



Gender and distance travelled

Almost two-thirds of the individuals participating in the ILM programme are male (1606/2512). The

report produced in 2011, on the basis of fewer participants going through the ILM programme,

indicated that there was very little overall difference between men and women in terms of distance

travelled on the soft skills dimensions. Table 22 reports the more recent findings.

Table 22: Distance travelled and gender


Average

entry score

Average exit

score

Average 'distance

travelled'

Test of

gender

difference

Communication Skills Women 3.24 3.34 0.10

Men 3.19 3.29 0.10

Self Presentation Women 3.69 3.70 0.02

Men 3.60 3.59 -0.01

Confidence Women 3.07 3.26 0.18

Men 3.18 3.33 0.16

Motivation Women 3.33 3.41 0.08

Men 3.31 3.38 0.07

Overcoming Setbacks Women 3.02 3.12 0.10

Men 3.02 3.16 0.14

Feelings Women 3.49 3.53 0.05

Men 3.49 3.48 -0.01 ***

Influencing Others Women 3.26 3.07 -0.19

Men 3.26 3.11 -0.15 **

Use of Numbers Women 3.21 3.33 0.12

Men 3.33 3.44 0.11

Knowledge of

Opportunities

Women 3.11 3.29 0.19

Men 3.09 3.26 0.17

Overall

Women 3.27 3.34 0.07

Men 3.27 3.34 0.06

*=p<0.1; **=p<0.05; ***=p<0.01 Source: Analysis of data collated by WCVA

Similarly to the 2011 report, there are very few differences between men and women’s distance

travelled. These differences lie in the feelings and influencing others dimensions. Women make

more progress than men during the course of the ILM programme in terms of managing their

negative feelings, and feeling more positively towards the future, as tested using an independent

samples t-test. However, women make more negative progress than men in relation to influencing

others. On aggregate across the different soft skills dimensions, however, there is no significant

difference in terms of distance travelled by men and women.



Age and distance travelled

In the 2011 report, the evaluation team focused on the difference between those participants who

were under 25 and those who were over 25. It has since been recognised that the over 50’s age

group are a priority group in relation to the ILM programme, and therefore we focus on the

distinction between under 50’s and over 50’s in this report. The majority of ILM participants in the

soft skills questionnaire are under 50 years old, but 10.5% are over 50. This group face particular

challenges in terms of employment opportunities and employment stability.

Table 23: Age and distance travelled

Soft skill outcome (max score =4) Average entry

score

Average exit

score

Average 'distance

travelled'

Communication Skills Under 50 3.23 3.32 0.09

50+ 3.01 3.18 0.18

Self Presentation Under 50 3.64 3.63 0.00

50+ 3.59 3.62 0.03

Confidence Under 50 3.14 3.30 0.16

50+ 3.15 3.39 0.24

Motivation Under 50 3.32 3.39 0.06

50+ 3.31 3.47 0.16

Overcoming Setbacks Under 50 3.02 3.14 0.13

50+ 3.01 3.15 0.14

Feelings Under 50 3.49 3.50 0.01

50+ 3.41 3.48 0.06

Influencing Others Under 50 3.26 3.09 -0.17

50+ 3.27 3.13 -0.14

Use of Numbers Under 50 3.29 3.40 0.11

50+ 3.25 3.39 0.14

Knowledge of Opportunities Under 50 3.10 3.28 0.18

50+ 3.06 3.21 0.16

Overall

Under 50 3.28 3.34 0.06

50+ 3.23 3.34 0.11 Source: Analysis of data collated by WCVA

Examining the average distance travelled over the course of the intervention, the evidence suggests

that despite being a target group due to the challenges the group faces in terms of employment,

those over 50 years old are actually doing no worse than those under 50 years old. It rather seems to

be the case that the ILM programme is equipping this target group particularly well in relation to soft

skills.

Over 50 year olds, overall, travel a greater distance between the entry and exit questionnaire. At the

entry point, the over 50’s score lower on overall soft skills than those under 50 years old. However,

by the time they exit the programme, their soft skills are at the same level as those of under 50 year

olds. The programme is particularly good at improving motivation, presentation skills and

communication skills of over 50’s, relative to under 50’s. The age effect was not tested using

inferential statistics due to the fact that the two groups that are being compared are very uneven in

size (with over 50’s representing only 10% of the sample).



However, the descriptive evidence provided here suggests that the ILM programme is doing well to

overcome the particular challenges faced by the over 50’s group in relation to soft skills.

Lone parents and distance travelled

In the ILM soft skills data, 11.9% (300/2,511) of respondents are lone parents, with higher

proportions of the RCE sample being lone parents than in the Convergence area (17% as compared

to 9.3%). Similarly to those over 50 years old, lone parents have been identified as facing particular

challenges in terms of gaining and retaining employment. Table 24 examines how lone parents fared

in comparison to the rest of the sample in terms of distance travelled on the various dimensions of

soft skills.

Table 24: Lone parenthood and distance travelled

Soft skill outcome (max score =4) Average

entry score

Average exit

score

Average 'distance

travelled'

Communication Skills Not lone parent 3.21 3.31 0.10

Lone parent 3.20 3.24 0.04

Self Presentation Not lone parent 3.63 3.63 0.00

Lone parent 3.64 3.67 0.03

Confidence Not lone parent 3.15 3.31 0.17

Lone parent 3.07 3.25 0.17

Motivation Not lone parent 3.31 3.40 0.08

Lone parent 3.36 3.38 0.02

Overcoming Setbacks Not lone parent 3.03 3.16 0.14

Lone parent 2.95 3.03 0.07

Feelings Not lone parent 3.49 3.50 0.01

Lone parent 3.46 3.47 0.00

Influencing Others Not lone parent 3.26 3.11 -0.16

Lone parent 3.22 2.99 -0.23

Use of Numbers Not lone parent 3.30 3.42 0.12

Lone parent 3.20 3.25 0.04

Knowledge of Opportunities Not lone parent 3.10 3.28 0.17

Lone parent 3.04 3.22 0.18

Overall

Not lone parent 3.28 3.35 0.07

Lone parent 3.24 3.28 0.04 Source: Analysis of data collated by WCVA

Lone parents enter the programme with slightly lower aggregate levels of soft skills than not-lone

parents. They have lower knowledge of opportunities upon entry, lower use of numbers, lower

confidence and lower ability to overcome setbacks. It is worth noting, however, that they enter with

higher levels of motivation than not-lone parents.

Looking at the aggregate distance travelled for lone parents, this is slightly lower than that of those

who are not lone parents. Lone parents started out with slightly lower levels of soft skills, and upon

exit of the programme, they had progressed to the level of skills that the not-lone parent group was

at before the intervention.



Again, due to unequal group sizes, we do not test the difference between the two groups using

inferential statistics. However, the descriptive statistics presented here do indicate that there are

challenges faced by lone parents that they do not over-come as effectively during the programme

than do those who are not lone parents. In particular, communication skills, motivation, overcoming

setbacks and use of numbers are soft skills on which the distance travelled by lone parents is

noticeably smaller than that of not-lone parent groups. The evidence suggests, therefore, that lone

parents still require more attention and that the intervention as it stands, may need to develop an

approach that is better targeted to the specific needs of this priority group.

Ethnicity and distance travelled

Black and ethnic minority groups have been identified as another priority group for the ILM

programme. For the purposes of this analysis, we group respondents who self-identify with any one

of the following groupings: Asian or Asian British Indian; Asian or Asian British Pakistani; Black or

Black British African; Black or Black British Caribbean; Black or Black British Other; Chinese; Mixed -

White & Asian; Mixed - White & Black African; Mixed - White & Black Caribbean. These groups, here

referred to as ‘BME’ groups, are contrasted with participants who identify as White British, White

Irish, White Welsh or White Other. Under this grouping, 3.5% (87/2,504) of the ILM soft skills

respondents identify as BME. Table 25 examines their soft skills distance travelled, as compared to

non-BME groups.

Table 25: Ethnicity and distance travelled

Soft skill outcome (max score =4) Average

entry score

Average exit

score

Average 'distance

travelled'

Communication Skills Non-BME 3.21 3.31 0.10

BME 3.18 3.25 0.07

Self Presentation Non-BME 3.64 3.63 0.00

BME 3.49 3.53 0.03

Confidence Non-BME 3.14 3.30 0.17

BME 3.23 3.41 0.18

Motivation Non-BME 3.32 3.40 0.07

BME 3.25 3.35 0.10

Overcoming Setbacks Non-BME 3.01 3.14 0.13

BME 3.13 3.19 0.05

Feelings Non-BME 3.49 3.50 0.01

BME 3.39 3.53 0.14

Influencing Others Non-BME 3.26 3.09 -0.16

BME 3.29 3.12 -0.17

Use of Numbers Non-BME 3.29 3.41 0.12

BME 3.23 3.25 0.02

Knowledge of Opportunities Non-BME 3.09 3.27 0.18

BME 3.17 3.28 0.12

Overall

Non-BME 3.27 3.34 0.07

BME 3.26 3.32 0.06 Source: Analysis of data collated by WCVA



Table 25 indicates that BME groups by no means begin the ILM process with lower scores on all soft

skills than non-BME groups. BME groups have notably higher initial levels of confidence, the ability

to overcome setbacks and knowledge of opportunities. They score lower, initially, than non-BME

groups on presentation skills, motivation, positive feelings and use of numbers. When examined on

aggregate, there is very little difference between the distance travelled by BME and non-BME groups

in terms of soft skills during the course of the ILM programme. However, there are particular

dimensions where there are differences between the groups. BME groups made noticeably more

progress than non-BME groups on feelings. However, BME groups made noticeably less progress

than non-BME groups on overcoming setbacks, the use of numbers and knowledge of opportunities.

This last finding needs to be contextualised by noting that BME groups actually started the process

having higher scores on ability to overcome setbacks.



6. Conclusion and recommendations

This final section will draw together the main findings along with the lessons learnt during the

evaluation period. These are grouped into a number of key themes which emerged during the

evaluation.

6.1. Conclusion

The key conclusion of this evaluation is that the ILM programme in both the Convergence and RCE

areas has generated significant benefits for those it has supported and has been successful in terms

of helping participants progress and move into employment. The fact that the programme has been

operating in difficult economic conditions has clearly been challenging. But, the programme has still

generated positive employment outcomes as well as a range of other outcomes that should improve

the employment prospects of participants. The key is however how long-term and sustainable those

outcomes are. Will those who have not moved into employment yet ultimately find a job? Will

those who have found a job keep those jobs? The answers to these questions are unclear at this

time but it is likely that the economic conditions over the coming months and years will be a

significant influence.

In terms of the management of the programme, the evaluation has found that both the

organisations providing the support and participants are generally very positive about the

programme. Within WCVA, the procurement process has clearly been a dominating factor especially

in the first half of the lifetime of the programme with a significant amount of time and resource

being spent on it. The WCVA team has also clearly been on a very steep learning curve when it

comes to working via a procurement process and has, to paraphrase discussions with stakeholders,

‘learnt a lot’. This could of course be interpreted as another positive outcome. However, it could also

be argued that the focus on procurement has also been an unwelcome distraction from the main

purpose of the programme; to facilitate the provision of support to participants via ILM projects.

6.2. Recommendations

1. The recession and its impact on the labour market had a significant influence on the delivery of

the ILM programme especially in terms of achieving employment results. Whilst it would clearly

have been impossible to anticipate the recession within the business plan for the ILM, some

element of flexibility should be built into any future business plan in order to allow the

programme to adapt to any unforeseen changes in the prevailing economic conditions. There

should also be flexibility within any future business plan to deal with other potential and

unforeseen changes such as the introduction of new Welfare to Work schemes that may impact

upon the eligibility of participants.

2. The ILM programme benefited significantly from being able to build on the previous North

Wales LMI programme. Any future programme should therefore seek to build upon the success

of the current ILM programme and the infrastructure (and capacity) that is now in place across

Wales to manage and deliver ILM projects.



3. A key lesson for the ILM programme during its early phases was the need to account for the time

it takes to move from a position of establishing a programme to a position where that

programme is delivering against outputs. The ILM programme found it challenging to engage

with delivery organisations during the early phases, largely it would seem due to the need to

establish a new procurement based approach. Other factors including the need to recruit new

members of staff to the ILM team along with the challenges of establishing the programme in a

new area were also however prominent. The potential for a slow start should therefore be built

into the planning process for any future programmes or projects.

4. The two-team management structure put in place for the ILM within WCVA was effective but

presented a number of challenges which need to be taken into account if a similar structure is to

be used in the future. Specifically, processes need to be in place to ensure that there is effective

communication between the two teams and to ensure that there are no ‘mixed-messages’.

5. Further consideration should be given to what the role of any panel overseeing the management

and delivery of any future programme should be. This should include further discussion on how

non-WCVA representatives should be engaged. The role of the panel changes as a programme

develops and this needs to be taken into account when ‘membership’ of the panel is being

discussed. Potential options for discussion include:

d) Setting up sub-groups (or ‘task and finish’ groups) to deal in detail with specific issues and

then report back to the main panel; such as, for example, the setting up of a new

procurement management system.

e) Inviting external organisations and/ or individuals to become members of those sub-groups

rather than ‘full’ members of the Panel so that their role is clear.

f) Inviting external organisations and/ or individuals to ‘one-off’ meetings of the panel when

specific issues are being discussed, such as welfare-to-work reform rather than inviting them

to become full members of the Panel.

6. A wide range of organisations have been involved in the ILM programme including some with

significant experience of tendering competitively for contracts and a number with very little if

any experience of doing so. Any support that is provided during the tendering (or application)

process (for example, PQQ briefing sessions) needs to take this into account in order to ensure

that the advice that is being provided is applicable and as useful as possible to the organisations

in question. Potentially, this should include differentiating between advice for beginners and

advanced support for the more experienced.

7. The ILM programme has not been successful in engaging with some target groups including

older people and those with learning disabilities. Consideration should therefore be given to

whether these and other target groups require ‘specialist’ ILM projects designed to take into

account the challenges of working with those groups. This may include setting different targets

for projects working with those who are considered to be the furthest away from the labour

market in order to ensure that potential providers are not discouraged from developing ILM

projects due to what they perceive to be unrealistic employment targets.

8. In light of feedback from participants and organisations delivering ILM projects, the potential to

extend the period of work experience that ILM projects in the future could offer should be

considered. Future evaluations should however also explore whether the outcomes generated

are enhanced by a longer work experience period.



9. Whilst recognising that a certain level of bureaucracy is inevitable with any publically funded

project, the potential to streamline the paperwork and the process of collecting paperwork

associated with future programmes should be explored by, for example, establishing a working

group to consider the matter with those engaged to deliver projects within the current ILM

programme.

10. The ‘soft outcomes’ data captured by WCVA for the ILM programme has been an important

monitoring and evaluation tool. Collecting the evidence can however be difficult and future

programmes should ensure that adequate processes are in place so that the evidence for ‘soft

outcomes’ is as comprehensive as possible. There may also be a need to emphasise the

importance of this evidence to contractors and participants.

11. Whilst the ILM programme ultimately reverted to a competitive grant process, it is important to

note the benefits generated by the procurement based approach employed for the majority of

the lifetime of the programme; these benefits can potentially aid organisations as they tender

for contracts from the public sector in the future. Whilst recognising the fact that a procurement

based approach does restrict the support that can be provided to organisations that can

potentially deliver effective ILM projects, the use of a procurement based approach in the future

should not be completely discounted.



Appendix 1: Soft skills questionnaire

Soft Skills Questionnaire

Name ___________________ ID Number ___________________ Contracted Supplier ___________________ Tender Number ___________________

Start Programme* End Programme* *delete as applicable

Date ___________________

Intermediate

Labour

Market

Ref: External D 1 F



This is an individual assessment and aims to reflect the entrance level of skills and help you and the organisation identify the areas you want support in. All sections in this assessment are weighted equally. Once you have completed the assessment you will discuss it with a supervisor and draw up an action plan. A Communication Skills Strongly

Agree Agree Disagree Strongly

Disagree Move To PTP

1. I find completing forms easy Strongly



2. I sometimes need help with reading instructions

Strongly



3. I often worry about my handwriting and spelling

Strongly



4. When I need to check change I do it in my head

Strongly



5. I can use a computer Strongly



6. I often have difficulty understanding when people speak English



B Presentation Strongly



1. I have somewhere to wash Strongly



2. I have somewhere to wash my clothes

Strongly



3. I am always punctual e.g: work/the course/the appointment

Strongly



4. I always telephone with a reason if I know I am going to be late

Strongly



5. I often have to leave early because of my situation e.g: working/the course/the appointment

C Confidence Strongly



1. I am as good as the next person

Strongly



2. I enjoy meeting new people Strongly



3. I am nervous about asking questions in a group

Strongly



4. My mind goes blank when I am asked a question



D Motivation Strongly



1. I find it difficult to motivate myself

Strongly



2. I like to keep working on something until it is completed

Strongly



3. I find it hard to concentrate for long

Strongly



4. People trust me to get a job done

Strongly



5. I need a lot of encouragement Strongly



6. I feel capable of being independent

Strongly



7. I like challenges E Overcoming Setbacks Strongly



1. I am easily hurt by criticism Strongly



2. I learn from things that go wrong



Strongly



3. When doing things, I decide what is more important and what is less important

Strongly



4. I get nervous about learning new things

Strongly



5. I can’t always cope with doing several different things at once

F Feelings Strongly



1. I have felt unhappy all week Strongly



2. I feel positive about my future Strongly



3. I feel despairing or hopeless Strongly



4. Talking to people is too much for me

Strongly



5. I feel my problems are too much for me

Strongly



6. Over the last week I have been physically violent towards others

Strongly



7. Over the last week I have felt panic or terror



G Influencing Others Strongly



1. I have difficulty expressing my views or feelings

Strongly



2. I find it difficult to make eye contact with people

Strongly



3. I have the right to say “no” to others without feeling guilty

Strongly



4. If others are offensive, I am offensive back

Strongly



5. I feel uncomfortable when someone compliments me

Strongly



6. I am happy to stand up for myself

Strongly



7. I am willing to understand the needs of others

Strongly



8. Working together, people get more done

Strongly



9. I understand the need for rules and regulations

Strongly



10. Nobody has the right to tell me what to do



Strongly



11. People in authority intimidate me

Strongly



12. If I don’t understand the instructions, I ask questions

H Use of Number Strongly



1. I am familiar with using a calculator

Strongly



2. I can pay for an item and accurately count my change

Strongly



3. I am happy with using percentages such as VAT

Strongly



4. I can calculate an area using a tape measure or ruler

I Knowledge of Opportunities I feel that I am ready to look for a job Yes No If Answer is No do not answer the following questions – you have completed the assessment If Answer is Yes complete the following questions Strongly



1. I understand my strengths and weaknesses

Strongly



2. I know what kind of job I am looking for



Strongly



3. I need further training Strongly



4. I can think of several different ways of finding out about job vacancies

Strongly



5. I think I could present myself well during an interview

Strongly



6. My CV does not make the best of my experience

Strongly



7. I have thought about the kind of questions I might be asked during an interview

Strongly



8. I have practised my interviewing skills

Strongly



9. I would ask questions during an interview

Strongly



10. I have the clothes to go to an interview

Strongly



11. I would be prepared to change how I present myself for work



J Accommodation (if applicable) I have accommodation at the moment Yes No If answer is Yes complete the following sections Strongly



1. I don’t know where I will be staying in a month’s time

Strongly



2. I do not have use of a telephone Strongly



3. I never get my telephone messages

Strongly



4. I could afford my rent if I got a job

Strongly



5. I would have less money if I got a job than on benefits

Strongly



6. I have space to write job applications or study

K Alcohol (if applicable) Have you drunk alcohol over the last month? Yes No If Answer is Yes complete the following questions Strongly



1. Drinking makes me more confident

Strongly



2. It irritates me when people refer to my drinking



Strongly



3. I drink more than is healthy for me

Strongly



4. I feel guilty when I am drinking Strongly



5. I often have a drink in the morning

L Drugs (if applicable) Have you used recreational drugs in the last month? Yes No If Answer is Yes complete the following questions Strongly



1. My drug-taking changes my coordination and judgement

Strongly



2. My drug use is increasing

Strongly



3. I spend more than half my income on drugs

Strongly



4. When I use drugs I have difficulty in getting up in the morning

Strongly



5. Using drugs makes me more confident

Signed by Participant ___________________ Date _________ Signed by Supervisor ___________________ Date __________



Appendix 2: Wordle

The following highlights the words used by participants to describe their job prospects. The most

common words appear in large font with the less common words in smaller font.

What three words would you use to describe your job prospects BEFORE joining the ILM?

What three words would you use to describe your job prospects AFTER joining the ILM?



Appendix 3: Participant questionnaire

Hello

We would like to invite you to participate in an evaluation of the ILM Programme that you are part

of. We want to know if and how the programme has helped people. By taking part in the evaluation

you can help us to find out. The evaluation is being undertaken by the research company Wavehill

(www.wavehill.com) for the Wales Council for Voluntary Action (WCVA) who are managing the

programme that is supporting you. It looks at how the project is supporting people and what impact

it is having on people’s lives.

The interview should take no more than 10 minutes to complete and your name will be entered into

a prize draw for a chance to win £50 worth of high street vouchers.

You do not have to answer the questions if you do not want to; it is up to you whether or not you

take part in the evaluation. If you cannot answer a question just leave it, and go on to the next

question.

Anything that you say will be completely confidential and your name will not appear in any

reports. Your contact details will not be passed to anyone else – they will only be used for this

evaluation.

If you have any questions about the questionnaire or the evaluation, please ask a project worker or

you can call Llŷr at Wavehill on 01545 571711 or email: [email protected]

Thank you!

The Wavehill Team



SECTION A - BEFORE JOINING THE ILM PROGRAMME

This section looks at your situation before receiving support from the ILM Programme.

A1a. Before your involvement with the ILM what qualifications did you have?

Certificate of Education / GCEs GNVQs

GCSE or O-levels A-levels

NVQs HNC

Degree or higher degree HND

Driving license HGV license

Health and Safety Manual Handling

First Aid Food preparation

None Other qualification (proceed to

A1b)

A1b. If other qualification, please specify

A2. How long were you out of employment before you joined the ILM?

Less than 6 months Between 1 and 2 years

Between 7 and 12 months Over 2 years

A3. What other jobs (if any) had you done before working at the ILM?

Interviewer note: If the interviewee has had a number of jobs, relate the question to the most

recent job. Please get the interviewee to briefly describe the job rather than just provide a job

title. We need to understand the kind of job they held in the past.

A4a. How likely do you think you were to get a job before your involvement with the ILM?

Very unlikely Unlikely Likely Very likely Don’t know

A4b. Please explain your answer

A5. What three words would you use to describe your job prospects before joining the ILM?

1.

2.

3.



A6a. Did you find it difficult to find a job before being involved with the ILM?

Yes – definitely (proceed to A6b)

Yes – a little (proceed to A6b)

No (proceed to A7a)

Not sure (proceed to A7a)

A6b. If you were finding it difficult to get a job, why do you think that was?

A7a. Have you previously undertaken any work-placements or any job-related training schemes

such as the Future Jobs Fund?

Yes (Proceed to A7b)

No (Proceed to Section B)

A7b. If yes, what was this?

Interviewer: Probe for information on the type of work-placements and/or training and when

they undertook that work placement; how long ago?



SECTION B – YOUR TIME WITH THE ILM PROGRAMME

This section considers your time with the ILM Programme to date and the benefits of being involved.

Your answers will help us understand the impact of the ILM Programme and possible ways of

improving the support.

B1. Which of the following best describes your current position?

I have been in my ILM job for less than 4 weeks (complete all sections apart from Section

D)

I have been in my ILM job for more than 4 weeks but have more than 4 weeks left

(complete all sections apart from Section D)

I have 4 weeks left in my ILM job (complete all sections apart from Section D)

I have finished my ILM job (complete all sections)

B2. For which ILM organisation do/did you work?

B3a. Was your ILM job

Interviewer: please consult with the database prior to the interview to confirm which ILM the

interviewee was supported by and whether it was an internal or external placement model.

Use this question to then confirm.

Located with an organisation managing the ILM? (proceed to B4)

Located with an external organisation delivering on behalf of the ILM organisation?

(proceed to B3b)

B3b. If external, where did you undertake your job?

B4. Can you briefly describe your ILM job?

B5a. What training have you done as part of the ILM?

Health and Safety Manual Handling

First Aid Food Preparation

Other (proceed to B5b)

B5b. If other, please state the name or type of training



B6. What is the best thing about the ILM?

Interviewer: Prompt on issues such as training and mentoring. In your notes, please

differentiate between responses volunteered by the respondent and those made in response

to a prompt.

Non prompted response:

Prompted response:

B7. What (if anything) could be improved?

Interviewer: Prompt on issues such as the level of support, training or mentoring.


Prompted response:

B8. Please indicate to what extent do you agree or disagree with the following statements?

a) The ILM where I worked understood my

needs

Strongly

Disagree

Disagree Neither

Agree or

Disagree

Agree Strongly

Agree

b) I received enough training at my ILM job Strongly

Disagree

Disagree Neither

Agree or

Disagree

Agree Strongly

Agree

c) I received the right sort of training at my

ILM job

Strongly

Disagree

Disagree Neither

Agree or

Disagree

Agree Strongly

Agree

d) I received enough support at my ILM job Strongly

Disagree

Disagree Neither

Agree or

Disagree

Agree Strongly

Agree

e) The ILM job matched my needs Strongly

Disagree

Disagree Neither

Agree or

Disagree

Agree Strongly

Agree

f) I feel that I made good progress whilst at

the ILM

Strongly

Disagree

Disagree Neither

Agree or

Disagree

Agree Strongly

Agree

g) I now have a better idea of what job I

want to do in the future

Strongly

Disagree

Disagree Neither

Agree or

Disagree

Agree Strongly

Agree

h) The ILM job has prepared me well for

future work

Strongly

Disagree

Disagree Neither

Agree or

Disagree

Agree Strongly

Agree

i) I feel work ready as a result of the

support received from the ILM

Strongly

Disagree

Disagree Neither

Agree or

Disagree

Agree Strongly

Agree



SECTION C – PERCEIVED IMPACT

This section considers the possible impact of the support on you as an individual.

C1a. What did you hope to achieve before joining the ILM?

C1b. Did the ILM help you achieve those goals?

Yes, a lot

Yes, a little

No

Not sure

C2. What three words would you use to describe your job prospects after joining the ILM?

1.

2.

3.

C3a. Have you developed your job skills during your time with the ILM programme?

Yes, a lot (proceed to C3b)

Yes, a little (proceed to C3b)

No (proceed to C4a)

Not sure (proceed to C4a)

C3b. If yes, what job skills have you developed?

Interviewer: Please collect as much information as possible in response to this question.

C4a. Has your confidence changed as a result of the support you have received from the ILM

programme?

Yes – more confident

Yes – less confident

No

Not sure

C4b. Please explain your answer

C5a. How likely do you think you are to get a job after your involvement with the ILM?





C6a. Are you more likely to get a job as a result of your involvement with the ILM?

Interviewer: please get the interviewee to compare how likely they believed they were to get

a job before and after (or during) their involvement with the ILM.

Yes - definitely

Yes - a little

No

Not sure


Interviewer: Please collect as much information as possible from the interviewee.

C7. What effect has working at the ILM have on you personally?

Interviewer: Probe for comments on confidence, motivation, lifestyle, future prospects and

optimism.


Prompted response:

C8. Has the support led to any wider effects?

Interviewer: Probe for comments on impact on family life, change in lifestyle, attitudes etc.


Prompted response:



SECTION D – PARTICIPANTS WHO HAVE COMPLETED THEIR ILM PLACEMENTS

D1. What did you do immediately after leaving the ILM?

Interviewer: Please check the database to see when interviewees leave the ILM. Probe for

comments on employment, education, volunteering etc and check when developments took

place.

D2a. Have you been applying for jobs since leaving the ILM?

Yes (proceed to D2b – D2d)

No (proceed to D3a)

D2b. How many jobs have you applied for?

D2c. What type of jobs were they?

D2d. Have you had an interview for a job?

D3a. Did you get a job after leaving the ILM?

Yes (proceed to D3b – D3f)

No (proceed to D4a)

D3b. How soon afterwards did you get the job?

D3c. Do you think your time at the ILM helped you to get this job?

Yes - definitely

Yes - a little

No

Not sure

D3d. Please explain your answer



D3e. Was this the sort of job you wanted?

Yes - definitely

Yes - a little

No

Not sure

D3f. Please explain your answer

D3g. In your opinion, how likely are you to stay in the job over the next 12 months?


C3h. Please explain your answer

D4a. Have you received any training since leaving the ILM?

Yes (proceed to D4b)

No (proceed to D5)

D4b. If yes, what was this training?

D5. Which of the following statements best describes your current situation?

I am currently in employment

I am not currently in employment, but am seeking work

I am not currently in employment and am not seeking work

I am currently in full-time education / going into full-time education

(have been accepted onto a course but haven’t started yet)

I am currently undertaking part-time education / training



SECTION E – AND FINALLY

E1. Would you recommend the ILM programme to others in your position?

Yes - definitely

Yes – probably

No

Not sure

E2. What is the way forward for you now as regards work, career and employment over the next

12 months?

E3. Is there anything you would like to add about your experience or the support you have

received?



SECTION F - ADMINISTRATIVE QUESTIONS

This section includes some administrative questions.

Please note that this interview is completely confidential. We only need your name so that we know

who’s taking part in the evaluation.

F1. Your name

F2a. We would speak to some of the people that have completed this questionnaire again in the

future to see whether their views have changed. Would you be happy for somebody to

contact you in the future to do this?

Yes (proceed to F2b)

No (End interview)

F2b. If yes, can you please provide your contact details? The information that you provide will be

protected under the Data Protection Act and will not be used for any purpose other than to

the survey.

Telephone number:

Email:

THANK YOU / DIOLCH YN FAWR

For admin purposes:

Language interview was undertaken

in:

Welsh English

Interviewer code

Date of interview

Location of interview



Appendix 4: Analysis of ILM programme

management data

The following analysis is based on the project management data for the ILM programme as at the

end of May 2012.

Figure 1: Gender of participants in Convergence Figure 2: Gender of participants in RCE areas

areas

N=2,035 N=1,009

Figure 3: Age group of participants in Convergence Figure 4: Age groups of participants in RCE

areas areas

N=2,035 N=1,009

Male,

66%

Female,

34%

Male,

60%

Female,

40%

16-24,

45%25-54,

48%

55-64,

7%65+, 0%

16-24,

54%

25-54,

41%

55-64,

5%



Figure 5: NEET participants in Convergence areas Figure 6: NEET participants in RCE areas

N=2,035 N=1,009

Figure 7: Lone parent participants in Convergence Figure 8: Lone parent participants in RCE

areas – yes or no areas – yes or no

N=2,035 N=1,009

NEET,

40%

Not

NEET,

60%

NEET,

52%

Not

NEET,

48%

, Yes,

10%

No, 90%

Yes, 16%

No, 84%



Figure 9: Work limiting health and/or disability Figure 10: Work limiting health and/or

participants in Convergence areas – yes or no disability participants in RCE areas – yes or no

N=2,035 N=1,009

Figure 11: Participants from BME background Figure 12: Participants from BME background

in Convergence areas in RCE areas

N=2,035 N=1,009

Yes, 6%

No,

94%

Yes, 7%

No, 93%

BME,

2%

Other,

98%

BME,

4%

Other,

96%

Evaluation of the Intermediate Labour Market (ILM) · PDF fileEvaluation of the Intermediate...

Documents

Transcript of Evaluation of the Intermediate Labour Market (ILM) · PDF fileEvaluation of the Intermediate...