Construction Futures Wales External Evaluation... · Construction Futures Wales External Evaluation...

27th June 2018

Construction Futures Wales

External Evaluation

Final Report

Wavehill Ltd.

• Wales office: 21 Alban Square, Aberaeron, Ceredigion, SA46 0DB (registered office)

• West England office: 217 Paintworks, Arnos Vale, Bristol, BS4 3AH

Contact details: Tel: 01545 571711 Email: [email protected] Twitter: @wavehilltweets More information:

www.wavehill.com https://twitter.com/wavehilltweets © Wavehill Ltd. This report is subject to copyright. The authors of the report (Wavehill Ltd.) should be acknowledged in any reference that is made to its contents.

mailto:[email protected]

http://www.wavehill.com/

https://twitter.com/wavehilltweets

Authors: Eva Trier Oliver Allies Anna Burgess Any questions in relation to this report should be directed in the first instance to Oliver Allies (t: 01545 571711 | e: [email protected]) Date of document: 27th June 2018 Version: 1.0 Client contact: Mr Cei Davies, Business Development Manager, Construction Sector Team [email protected]



Contents 1 Introduction ....................................................................................................................... 1

1.1 The Construction Futures Wales programme ............................................................. 1

1.2 The evaluation ............................................................................................................. 1

1.3 The structure of the report ......................................................................................... 1

2 Evaluation approach and methodology ............................................................................. 3

3 Construction Futures Wales Theory of Change ................................................................. 5

4 Governance, delivery mechanism and activities ............................................................... 9

4.1 CFW model .................................................................................................................. 9

4.2 Governance ............................................................................................................... 10

4.3 Strategic engagement ............................................................................................... 11

4.4 CFW programme management ................................................................................. 11

4.5 CFW structure and integration.................................................................................. 12

4.6 CFW offer and activities ............................................................................................ 13

4.7 CFW CRM system ...................................................................................................... 15

5 Results achieved by CFW ................................................................................................. 16

5.1 CFW outputs .............................................................................................................. 17

5.2 Survey coverage ........................................................................................................ 18

5.3 Accessing CFW support and overall customer satisfaction ...................................... 21

5.4 Beneficiary experience of specific CFW supports ..................................................... 28

5.5 Outcomes for CFW beneficiaries............................................................................... 38

5.6 Tangible results secured by CFW beneficiaries ......................................................... 41

5.7 Use of other supports ............................................................................................... 48

5.8 Need for additional support ...................................................................................... 48

6 Conclusions and recommendations ................................................................................. 50

6.1 Conclusions................................................................................................................ 50

6.2 Recommendations .................................................................................................... 51

Construction Futures Wales External Evaluation Final Report

1

1 Introduction This report presents the findings of the end of programme evaluation undertaken of the Construction Futures Wales (CFW) programme.

1.1 The Construction Futures Wales programme

The Construction Futures Wales Programme is the key delivery mechanism to provide support to Welsh Construction Sector companies and their supply chains. CFW is a business & skills development programme and has been developed jointly by the Welsh Government and CITB Wales to ensure a co-ordinated approach to support the construction sector in Wales. The programme is jointly funded by the Welsh Government and the CITB with an overall programme budget of £3m. The aims of the programme are to provide practical assistance to construction companies to improve their performance in terms of Quality, Cost & Delivery (QCD) in a safer working environment, and to enhance and recognise the skills of individuals in the construction sector companies. It is designed to equip the industry to better understand the challenges it faces moving forward and equip the workforce to better meet those challenges. Construction Futures Wales uses proven methodologies of awareness raising seminars; construction supply chain network development workshops; direct company interventions; benchmarking diagnostics; skills development.

1.2 The evaluation

The evaluation had the following aims:

• To provide quantitative reports on CFW’s achievement of deliverables against targets

• To provide qualitative reports on the strengths and weaknesses of the service delivery and the value of the service delivered to its client base

• To provide recommendations to ensure that support and opportunities to generate wealth and employment opportunities and provide positive contribution to global well-being continue.

1.3 The structure of the report

The report is structured in six main sections. Section 2 provides an overview of the evaluation methodology before Section 3 sets out the Theory of Change for the programme, that underpins the overall evaluation framework.


2

Section 4 considers the governance approach, delivery mechanism and mix of activities offered by CFW drawing mainly on the stakeholder interviews, but also taking in relevant results from the beneficiary survey. This is followed by a detailed analysis of the specific results achieved at different levels in Section 5 before Section 6 offers conclusions and recommendations.


3

2 Evaluation approach and

methodology The evaluation was undertaken in two main phases: 1. Inception phase: This focused on the development of an overarching evaluation

framework and the design of all key research tools. Drawing on an extensive documentary review stage and a review of the services delivered, an initial review of management and monitoring information and a series of scoping interviews with key programme stakeholders, a detailed Theory of Change was developed to underpin all detailed research tasks. This created the foundation for:

• The definition of a suite of indicators

• The sampling methodology for a beneficiary survey

• The survey questionnaire

• The identification of key stakeholders and development of a semi-structured qualitative interview guide.

2. End of programme summary evaluation: The tools developed during the Inception phase

were deployed in primary research undertaken with CFW beneficiaries and stakeholders. The following table sets out the key considerations regarding the sampling methodology and provides a breakdown of the final sample structure. Table 2.1: Sample structure

Sampling criteria No. of unique

businesses1 Target sample

size Ultimate sample

size

L & M Attendance 81 30 37

Benchmark completed 62 20 17

One-to-one support 141 45 43

L & M Completions 39 20 22

Attended an event

thematic 1425 40

58

MTB 68

A random sampling approach was adopted while monitoring coverage of other relevant criteria such as the size, sub-sector and location of businesses.

1 These figures were compiled on the basis of the December 2017 CRM data, which is when the sampling was undertaken. They therefore don’t necessarily match the final KPIs as reported at the end of March 2018.


4

Many businesses had taken advantage of several CFW elements of support. The survey questionnaire was therefore structured to cater for two overarching cohorts:

• Light touch: those businesses that had only attended one or more events.

• Intensive support: those businesses that had used either one-to-one support or participated in the L&M course or both (most of whom had also attended CFW events).

The beneficiary survey was undertaken through scheduled telephone interviews of between 10 and 30 minutes duration. In addition, a total of 14 stakeholder interviews were undertaken with a cross-section of stakeholders representing different perspectives and expectations towards the CFW programme. The perspectives covered included:

• Representative organisations and trade/ professional bodies in the construction sector

• Local authority construction framework owners and end clients of large construction projects

• Other public-sector organisations with a specific offer for the construction sector and

• Tier 1 contractors and some of the larger Welsh construction companies with a role in structuring supply chains.


5

3 Construction Futures Wales Theory of Change The following diagram sets out the Theory of Change for the Construction Futures Wales programme. It was developed combining evidence from the documentary review and the scoping interviews with key programme stakeholders. It captures the intervention logic that underpins programme delivery.


6

The partnership between Welsh Government and the CITB behind the programme is reflected in the governance arrangements, with the Joint Investment Partnership Project Board having equal representation from both organisations. Drawing on its strong profile in the sector, the CITB was charged with day-to-day management of the programme with a dedicated project manager overseeing delivery. Delivery of the different CFW elements was procured from external providers and delivered in four lots as follows:

Lot 1 - Development of a company Diagnostic tool, Research and Horizon Scanning

Contractor: Cardiff Metropolitan University

Service provided: This Lot was of a ‘task and finish’ nature and involved the development of a bespoke diagnostic tool to underpin the activities of the CFW programme. The tool is the primary mechanism by which companies are assessed to determine what type of support is needed. The idea behind the diagnostic tool was also for this to provide intelligence for the programme on an ongoing basis. In addition, the contractor was required to undertake research and horizon scanning reports on behalf of the programme to inform its future development.

Lot 2 - Client Relationship Management System (CRM) Events & Marketing

Contractor: BE Group

Service provided: Lot 2 was tasked with delivering a robust Client Relationship Management System that could be operated at differing levels by all contractors and the programme management team itself. A bespoke tool for CFW was created from scratch. In addition, BE Group had responsibility for delivering all marketing activity on behalf of CFW, including the events, publication materials and delivery of a functional website to promote the support available through the programme. This included regular evaluation activities of all events delivered.


7

Lot 3 - Skills Development

Contractor: Cardiff Metropolitan University

Service provided: Lot 3 is targeted at developing individual skills within the industry, particularly those of future Managers and Leaders. Cardiff Metropolitan University developed a bespoke construction sector M&L programme for CFW. The taught modules and assignments lead to accreditations to:

• level 3-4 NVQ for site managers and

• level 5-7NVQ for organisation leaders. The course was developed by building on Cardiff Metropolitan University’s existing 20Twenty Leadership Programme. The courses consisted of several one or two day-sessions over a period of months with a piece of written work at the end (a strategic growth plan for the higher-level course). The courses are entirely free to construction companies. The diagnostic tool was used to support participating companies in identifying the most appropriate development trajectories for their business.

Lot 4 - Sourcing / Meet the Buyer and Supply Chain Development opportunities

Contractor: Enterprise Consulting Limited

Service provided: Lot 4 was targeted at providing sourcing support as well as providing individual 1-to-1 company support through the provision of process and systems improvements within the company. Working with Tier 1 companies, local authority construction framework owners and end clients, this CFW service was designed to help SMEs identify and make the most of opportunities to strengthen their role in Welsh supply chains. More in-depth supply chain development support was available for groups of companies wishing to work together to strengthen the effectiveness of collaborative delivery of construction projects. This Lot also used the diagnostic tool to define support projects for individual SMEs.


8

The CFW website, which was hosted on the main Business Wales portal, acted as the main entry point to the programme. Success of the CFW programme depended on buy-in from both supply chain leads involved in tendering for and delivering large construction projects in Wales – mainly Tier 1 contractors - and SMEs with an existing involvement in Welsh construction supply chains or an interest in enhancing their capabilities to win new work as suppliers for such projects. The idea was for CFW to play a role not only in raising awareness of the technical requirements to allow SMEs to play their part in supply chains, but also in identifying key challenges for the industry as a whole, particularly those that have the potential to fundamentally change the way of working and require a collective response from different parts of the construction supply chain. As outlined in the introduction, the Theory of Change provided the structure and research questions for the evaluation as follows:

• The stakeholder interviews focused primarily on the strategic fit (as reflected in the rationale and objectives as well as the delivery mechanism and nature of activities).

• The data analysis focused on confirming whether the output targets have been achieved.

• The beneficiary survey focused on ascertaining to what extent the outputs and outcomes had been achieved in interactions with individual construction sector SMEs in Wales.

• The impact analysis of the survey results provides insights regarding the extent to which the CFW programme has contributed towards the higher-level impacts for the construction sector in Wales.


9

4 Governance, delivery mechanism

and activities

4.1 CFW model

CFW as a joint vehicle between the Welsh Government and the CITB, two organisations with very different requirements, depended for its success on clarity regarding specific outcomes that would together contribute to growth in the construction sector in Wales. The interviews with CFW governance, management and delivery stakeholders suggest that the programme design and delivery mechanisms were therefore developed on the basis of initially setting out a construction excellence model. This considered key elements of business models in the construction sector; the role of innovation, leadership and lean practices; the requirements for data and information gathering in strengthening productivity in construction industry businesses and an awareness of the nature and importance of different kinds of supply chain linkages and in particular the role of community benefits in guiding supply chain activities. On the basis of the initial research into the dynamics shaping supply chain opportunities in the construction sector in Wales, a decision was also taken to adopt an approach of reaching into adjacent sub-sectors, such as manufacturing companies involved in off-site pre-construction works. The aim was to ensure that the full potential of construction supply chain opportunities would be made accessible to businesses in Wales. The basic health check (a set of questions for businesses making initial contact with the programme) and business diagnostic (a subsequent stage of more detailed interactions with businesses designed to capture key information about their activities and capabilities) were of central importance for programme delivery. This offered a way to not only engage businesses in an initial interaction about their specific needs and objectives, but also ensured that each intervention was clearly structured and focused on specifics. At the same time, its function as an important eligibility filter and tool to channel businesses into the right support was essential in ensuring that the programme could reach growth companies in the sector, those that would contribute to the shared WG – CITB objectives and go beyond existing grant support (from Welsh Government) and the existing skills development offer (from the CITB). Finally, the CFW model had a strong demand-led element, working directly with Tier 1 contractors and end clients/ public sector construction framework owners in order to ensure that SMEs would benefit from business development that was directly linked to tangible market opportunities.


10

4.2 Governance

The stakeholder interviews suggest that Construction Futures Wales, through the governance arrangements and regular consideration of how CFW could best complement other policy and delivery activities, offered a valuable opportunity to create a strong regular link between the Welsh Government and the CITB. At the same time, stakeholder comments suggest that a degree of uncertainty in terms of the CITB’s role and precise offer as well as changes in the WG policy direction and support infrastructure created a difficult backdrop and challenged ongoing integration of the CFW offer and other provision for businesses in the construction sector. In particular the CITB Consensus process, grant reform and changes in the CITB staff base in Wales meant that CFW could not necessarily rely on the strength of strategic direction that would have been anticipated at the start. Similarly, the evolution of the Welsh Government’s sector teams against the backdrop of a new Welsh Government strategy and Economic Action Plan being developed meant that WG strategic objectives evolved over the CFW implementation period whilst construction industry hopes that the use of Building Information Modelling (BIM) to focus on the whole life cycle costs of a building would become part of Welsh Government policy for the construction sector have not come to fruition. Operational aspects, such as close working relationships with the CITB Area Delivery Managers and continually feeding intelligence about sectoral developments into the ongoing alignment of CFW provision with developments in the sector were also challenged by the changing organisational environment. Creating an effective mechanism of aligning associated governance decisions with operational choices was difficult. This is, in particular, important so that CFW can continually be adjusted to ensure that the focus on specific market failures and policy objectives is reflected in delivery at all times. Some stakeholders queried whether sourcing support for large Tier 1 contractors, for instance, was still an appropriate role for a publicly-funded support intervention like CFW. The view was expressed that following an extended period of providing help in identifying SME partners who can strengthen the delivery community benefits in public sector construction projects, it might now be time to for industry to take this on themselves. This contrasts however, with calls from elsewhere for more support to ensure that SMEs tap into the supply chain opportunities expected to arise from large investments such as the forthcoming Transport for Wales infrastructure projects. Other examples as highlighted by stakeholders of where continuing adjustments are necessary include the need to not only focus on SME development needs, but also policy objectives such as reducing the whole life costs of construction projects or maintaining a challenge to SMEs to invest in collaborative working practices rather than relying on Tier 1 contractors as integrators. As one stakeholder put it, ‘the culture of construction industry companies using what they’ve always used needs to be challenged in order to improve the whole life costs of construction projects’.


11

Ensuring such adjustments are made on an ongoing basis requires an effective governance mechanism that ties delivery and operational choices into the wider policy and market environment.

4.3 Strategic engagement

The stakeholder interviews and insights gained from the survey feedback suggest that CFW was able to secure credibility and general buy-in on the strength of its link to the Welsh Government and the CITB. Moreover, there is evidence to suggest that CFW was valued by strategic partners in the sector as the only dedicated construction sector vehicle in Wales with the potential to reach into different sub-sectors and go beyond the scope of the levy in terms of the sub sectors it was able to work with. Seeing that many UK national organisations don’t have a strong focus on Wales and even where they do their work is strictly demarcated by the particular trade or profession they represent, CFW was seen to fill a gap in that respect. At the same time, in-depth buy-in from potential partner organisations in a competitive industry with an institutionally dense environment also created challenges for CFW. With each professional body or trade association catering for a particular target group, identifying specific opportunities in the integrated package of CFW supports that would warrant collaboration on specific aspects was not always straight forward. The risk of duplication was high in relation to initiatives like Construction Excellence Wales (and could not always be avoided according to some stakeholders) and complementarity of services with, for instance, the Supply Chain Sustainability School or Business Wales itself required ongoing attention.

4.4 CFW programme management

The approach to day-to-day management of CFW depended on a strong focus on programme and lot management. Based on strategic decision-making by the Project Board, the programme manager was responsible for budget management and the authorisation of specific interventions per Lot. As a result of the procurement of CFW delivery in lots delivered by external providers and the challenges involved in aggregating intelligence from a range of different programme interventions and across several layers of governance and management responsibilities, tweaking budgets and detailed delivery decisions to ensure maximum results in terms of anticipated outcomes was challenging at times. In particular, understanding the precise combination of the different CFW levers in contributing to business growth was complicated by the distributed delivery infrastructure involved in raising awareness of key issues (through the thematic events), support for particular aspects of business development, the personal development and leadership capacity of individual members of staff (through the L&M programme) and the identification of specific opportunities (through the sourcing support and Meet the Buyer events).


12

The stakeholder interviews suggest that CFW successfully identified a number of synergies with other activities and programmes. This ranged from a collaboration with Public Health Wales’ Healthy Working Wales programme to reach the predominantly male employees in the construction sector, a ‘hard-to-reach’ group for health messages, to tapping into Cardiff University’s expertise in levering High Performance Computational Informatics for BIM applications, which culminated in a Knowledge Economy Skills Scholarship between Cardiff University and Enterprise Consulting aimed at further strengthening access to this expertise for SMEs in the construction sector in Wales.

4.5 CFW structure and integration

The review of CFW activities and feedback from the beneficiary survey suggest that the four lots broadly delivered the anticipated activities and achieved a degree of integration. A number of examples illustrate a degree of divergence from the original project plan as set out at the procurement stage. The diagnostic, which had been meant to provide a red thread and regular intelligence for the programme as a whole, was used slightly differently between the two main lots that it formed part of, the L&M course under Lot 3 and the one-to-one interventions under Lot 4. There is a clear rationale for this degree of adaption in that for Lot 3, the purpose of using the diagnostic was to stimulate thinking about strategic development needs to strengthen productivity, whereas the focus of Lot 4 was on defining a support intervention. The latter clearly needed to have a more narrative focus. Nevertheless, it would have been desirable to find a way of (re-)aligning the format and use of the diagnostic to ensure consistency across the CFW programme. Referrals between different delivery elements and between CFW and the wider support environment were a further key building block of the CFW delivery mechanism. The analysis of the monitoring data together with the survey results confirm that once businesses’ basic eligibility had been determined and they had accessed individual support elements, a considerable number of them were able to take advantage of other support elements too. Internal referral processes therefore appear to have worked well in creating a responsive client journey. Considerable upfront investment was made in clarifying how CFW could best interact with Business Wales (BW). With the CFW website hosted on the BW portal, initial integration was created at the point of entry for businesses. In addition, detailed discussions with the BW hotline team helped clarify how referrals should be managed. The fact that both BW and CFW involve a number of external delivery partners creates constraints in terms of maintaining full product knowledge of different WG-backed supports for SMEs on the part of all delivery partners involved. An ongoing effort is therefore required to maintain appropriate integration and referrals.


13

There is some evidence of CFW brand recognition, but stakeholders also highlighted a remaining degree of confusion in the industry. One stakeholder suggested that there is ‘no string tying the different initiatives in Wales together’ with a range of organisations from Constructing Excellence Wales to Working Well Together Wales all the way to Job Centre Plus having a role to play, for instance in relation to delivering community benefits support. Concerns over the clarity of the offer and ease of finding the right support were also reflected in some beneficiary survey responses from businesses that had not taken advantage of the more in-depth support elements (i.e. one-to-one support or L&M course). One respondent suggested that ‘it would be useful to have an area manager in North and South Wales to have a full view of what's going on’ and another explained that ‘there needs to be more help specifically targeted at SMEs. CFW needs to raise awareness of the support that is available and to simplify the process so that SME's can go to one place and get all the advice, funding and training that they need’.

4.6 CFW offer and activities

The CFW offer was set out in the original project plan and the procurement specification. As outlined in section 3, it was delivered under four different lots of specific activity. Taken together, the CFW offer included a good mix of different interventions for businesses to choose from. Ranging from awareness-raising to more focused thematic events, from individual Meet the Buyer events to more hands-on sourcing or even one-to-one support interventions and between different qualification levels for L&M course, CFW sought to cater for different needs and intensities of engagement. The evidence suggests that the CFW focus on business development as the distinguishing feature for the programme (e.g. as compared to CITB’s own skills development focus) was clear in the market place. With regard to marketing, some stakeholders queried, whether the marketing of CFW activities could at times have been better targeted, with events catering for specific sub-sections of the sector. The one-to-one support catering for specific construction sector requirements was seen as essential by stakeholders. While some stakeholders queried the relevance and usefulness of the L&M provision as potentially ‘too academic’, the survey results and specific feedback from those who had a more direct exposure to the course content and interactions suggest that the personal development focus was highly appropriate in allowing SMEs to introduce a clear strategic management dimension into their business. This was caveated by comments about the practicalities of SMEs taking time out for the course, particularly where businesses are based outside of the South East Wales area and by concerns over the extent to which businesses are indeed ready more generally to manage change. Instances were described, where middle managers returned to the business with ideas for change that proved difficult to accommodate without a clear focus on succession planning and the risk of senior staff moving on or being poached was identified as potentially limiting take-up.


14

Feedback was collected regularly (e.g. L&M evaluation of each workshop/ speaker with good results; event happy sheets) and fed into a process of responsive programme development and a dedicated CFW customer survey was undertaken to understand specific SME needs. This resulted in a strong CFW performance in terms of identifying specific opportunities for thematic events to follow up on expressed SME needs. However, it also led to a slightly transactional approach instead of a more strategic focus on creating genuine momentum towards innovation and a focus on future needs in the construction sector in Wales more widely. The feedback from stakeholders regarding the CFW reach into the sector and targeted marketing of events tends to be positive. However, the actual reach into the sector beyond the ‘usual suspects’ was queried by some stakeholders. Geographical differences in terms of provision between the different Welsh regions was seen to be broadly in line with the sectoral dynamics in different parts of Wales. Securing take-up for events in North Wales, for instance, proved to be more difficult than in South East Wales, which reflects industry dynamics of North Wales based businesses often targeting bigger markets in North West England. CFW responded to such challenges by, for instance, continuing to build up database contacts, extending the L&M offer into North Wales and persevering with events, even where take-up wasn’t initially at desirable levels. With regard to the programme of events, the original project plan had anticipated regular interaction between a programme of research (covered under Lot 1) and the identification of relevant topics for thematic events. In the event, it proved challenging to secure industry engagement with horizon scanning and more forward-looking topics in terms of innovation trajectories (e.g. off-site manufacturing, new materials). The focus therefore shifted towards areas of more direct interest for SMEs in the construction sector (e.g. health and safety, quality assurance, NEC contracts). While this is appropriate in that many SMEs in the construction sector lack an understanding of basic supply chain requirements (e.g. the need for accreditations and the kinds of capabilities needed to work with Tier 1 companies and contractors) and the challenge of securing engagement beyond this is recognised, it would be desirable to find ways of making such topics accessible to a wider range of businesses in the construction sector. The challenge involved is summed up succinctly in a quote from a survey respondent who suggested that ‘there is a gap between the urgency small businesses feel over paying wages and covering their outgoings each month, and the "blue sky" of the funding bodies who seem to forget about the day-to-day struggles that we face. There's a lack of reality in how they approach us, thinking we have more time or mental space than we actually do’.


15

One area of activity that proved particularly hard to establish as a CFW delivery component was the development of Supply Chain Networks. Feedback provided by stakeholders and beneficiaries as part of this evaluation suggests that two issues arose; firstly, there are still many instances where interactions between large, Tier 1 contractors and SMEs in their supply chain, are not characterised by the degree of trust that would be required for successful development collaboration. Secondly, the role of the CFW team in supporting supply chain development activities was less clear cut than for the other support offers. For a Supply Chain Network to deliver on the specific requirements of a particular lead contractor, activities need to be highly bespoke requiring considerable expertise and direct input from the Network lead. The precise nature and level of input from CFW therefore proved to be hard to define. An issue that was raised in the stakeholder interviews in particular, relates to the level of construction sector expertise held by the CFW team themselves. As a result of changes in the position of the construction sector in the context of Welsh Government economic development policy priorities in recent years, the expertise that had been built up in the construction team couldn’t be maintained to the full extent. In sourcing external expertise, CFW needed to find an appropriate combination of experience of providing development support to SMEs, relevant process knowledge (which in some instances was transferred from other settings, e.g. lean methodologies from a manufacturing environment) and specific construction expertise at affordable rates. This proved a challenging equation and meant that cutting edge construction sector expertise wasn’t always the first priority in delivering support.

4.7 CFW CRM system

Finally, the CFW CRM system was an essential building block of providing an integrated delivery mechanism. Part of Lot 2, the aim was to build up a database of construction sector companies and their profiles and interests to allow the provision of targeted support. The evidence collected for this evaluation suggests that it has indeed been possible to build up a valuable database. This was highly appreciated by a number of external partners, from public sector representatives with a role in ensuring appropriate SME involvement in public sector construction-related procurement and Tier 1 contractors with an interest in identifying new local suppliers in Wales to other public-sector agencies with a need to reach construction sector SMEs. The CRM system was also required to support management of the complex CFW workflow and track diverse client journeys. It was this latter element that proved more challenging. The complexity of data that was required to support delivery interactions for the individual lots (e.g. details of sessions attended, assignments submitted etc. in L&M) meant that the central CRM system was complemented by delivery contractors’ own additional data collection. As a result, the CRM system did provide an effective mechanism for the collection of data on CFW KPIs and as a marketing tool but had more limited scope to provide qualitative insights about the interactions and results achieved with individual beneficiaries.


16

5 Results achieved by CFW Building on the review of the CFW offer and governance and delivery mechanism, a key area of interest for the evaluation was to establish what tangible results had been secured by the programme. This was the main focus of a beneficiary survey that was undertaken as part of the evaluation research. The results of the survey – presented in the subsequent sections – complement the monitoring data which captures the outputs secured.


17

5.1 CFW outputs

CFW operations were accompanied by an extensive monitoring effort designed to provide real time intelligence about the performance of different strands of the programme. Now, at the end of the programming period the aggregated data provides a useful first snapshot of CFW’s performance. Table 5.1: CFW performance against targets

2014/15 2014/15 2015/16 2015/16 2016/17 2016/17 2017/18 2017/18 Total Total target Total

differential

KPI Target Actuals Target Actuals Target Actuals Target Actuals Total

Companies Assisted 50 44 110 130 80 215 10 233 622 250 +372

Individuals Achieving Qualifications

30 0 40 60 30 0 0 55 115 100 +15

New Jobs Created 40 0 110 41 90 76 10 92 258 250 +8

Jobs Safeguarded 80 0 180 159 120 431 20 231 1083 400 +683

Jobs Assisted 150 0 150 3 150 325 150 151 673 600 +83

Deals £1,500,000 £0 £5,000,000 £5,736,834 £3,000,000 £17,449,696 £500,000 £16,209,382 £45,754,421 £10,000,000 £35,754,421

As illustrated in table 5.1, CFW has exceeded all targets. Results have been particularly high in relation to jobs safeguarded, with 271% of the target achieved, companies assisted with 249% of the target achieved and deals, with 458% of the target achieved. For the other three indicators, the target has also been exceeded, albeit by more modest amounts (103% for new jobs created, 112% for jobs assisted and 115% for individuals achieving qualifications.


18

5.2 Survey coverage

Building on the sampling methodology as set out in section 2 above, the following diagrams describe the profile of the survey sample reached by the telephone interviews in more detail. Figure 5.1: Size distribution of survey respondents

(n=129) As figure 5.1 above shows, the survey sample includes a useful distribution of beneficiary company sizes. At 44% of all survey respondents, the largest sub sample relates to companies in the 11 to 49 size bracket, a key target group for the CFW programme. Alongside this, smaller and larger companies are also represented in the sub sample, which provides a useful counter balance of perspectives. A further dimension that is important in understanding the survey results relates to the intensity of interactions that survey respondents have had with the CFW programme. Figure 5.2: Intensity of support received by survey respondents

(n=131)


19

In this respect, figure 5.2 above shows that the survey is balanced in terms of perspectives from beneficiaries who had received only light-touch support (i.e. attendance at thematic and/ or Meet the Buyer events only) and those who had more intensive interactions with the CFW offer (i.e. one-to-one support, participation in L&M course plus potentially event attendance). The survey also needed to provide insights from different sub-sectors in the construction industry. Figure 5.3: Survey sample structure in terms of construction sector sub-sectors

(n=131) Figure 5.3 shows that a good balance has been achieved for the survey between the four main sub sectors of construction of buildings (32%), civil engineering (27%), specialised construction engineering (22%) and other professional scientific and technical activities (18%). Finally, it is worth considering the distribution of geographical target markets in the survey sample.


20

Figure 5.4: Distribution of geographical target markets in survey sample

(n=128) Figure 5.4 above illustrates that a majority of businesses in all four sectoral survey cohorts trade UK-wide while a smaller proportion trades in Wales only, but a remaining sizeable minority also trades internationally.


21

5.3 Accessing CFW support and overall customer

satisfaction

Understanding how beneficiary businesses found out about the support available from CFW is important in assessing programme success. Figure 5.5: How did you find out about CFW?

(n=131) The largest proportion of respondents were referred into the CFW programme by Business Wales (which includes the Business Wales hotline). Adding up the respondents who had found out about CFW through recommendations from various sources makes up a further 15% of respondents, a sizeable minority, while seven per cent had identified the CFW offer through own research. It is also worthwhile, to consider what additional sources make up the ‘other’ category.

34%

27%

8%

8%

4%

4%

3%

3%

1%

0% 5% 10% 15% 20% 25% 30% 35% 40%

Referral from Business Wales

Other

Can't remember

Recommendation/ referral from other businessadvisor

Own research

Recommendation from supplier

Recommendation from other businesses thathad used CFW

Internet search

Recommendation from customer

Overall


22

Figure 5.6: How did you find out about CFW? – ‘other’

(n=41) This suggests that a further 23 respondents, which corresponds to an additional 28% of all respondents, attributed their contacting CFW to a recommendation of some kind and nine were contacted directly by CFW. In other words, a total of 33% of survey respondents received a recommendation to seek out CFW support. These results point to CFW successfully having built up strong brand recognition and a good reputation for its support offer for businesses in the construction sector. Both other support providers and people in construction sector companies appear to recognise the value that CFW offers sufficiently to pass on recommendations. This analysis also suggests that a diversity of access routes were available to, and used by, businesses. A majority of respondents (81%) also reported that it was clear to them who was eligible for CFW support, further indicating that CFW’s marketing messages seem to have been clear to the market. Building on this, survey respondents were asked how important different aspects had been for their decision to engage with CFW.


23

Figure 5.7: Importance of different aspects in engaging with CFW support

(n=118) As figure 5.7 highlights, business development aspirations were the overriding reason for businesses to seek out CFW support. Additional respondent comments shed some further lights on the motivations for using CFW.

‘Although none of our customers were engaged with CFW, our competitors were, and this was influential in our decision to access CFW support.’ ‘Mainly for the contacts and connections.’ ‘Looking to expand into Wales and offer engineers more work within Wales region. Want to employ local labour in Wales and get Welsh apprentices there - looking after the next generation of our workforce. ‘Themed events: to give an insight into those topics for my staff. - MTB: networking opportunity - get our name out there.’

In other words, CFW is perceived as a key platform for networking and a source of general information about topics of relevance to construction sector SMEs in Wales. This reputational background and reasons for engaging with CFW can usefully be compared and contrasted with the satisfaction levels reported by CFW beneficiaries themselves. Survey respondents were asked to consider their satisfaction with different elements of the support separately. To begin with, respondents were asked to rate their overall CFW experience on a scale of 1 to 5.


24

Figure 5.8: Satisfaction with interaction with CFW – by intensity of engagement

(n=118) As figure 5.8 above illustrates, overall satisfaction levels were fairly high for respondents regardless of the intensity of engagement with CFW. Examples of positive comments in response to the question of what respondents found particularly useful include the following:

‘Nothing could be improved I was very happy with the support I received.’ ‘It's an excellent package overall.’ ‘Best thing about the CFW support was the variety of topics covered and the relevance of the course content to the challenges facing the construction industry.’ ‘Making links with businesses and networking were the most useful aspects.’ ‘I certainly find the events useful, the access to external consultants is useful because as a small practice it would cost us about £600 a day to have someone come in so it's great to have that support to give us access to people without having to pay.’ ‘The most useful aspect of the support was that it identified a new market for us in terms of the nuclear sector.’ ‘Finding out what standards and accreditations we'd need. It was interesting to see what projects were going to be coming up in the area.’ ‘Opportunity to talk to buyers face-to-face and explain our work. Finding out what was happening in Wales and if there's any specific Welsh criteria to enable us to work there.’


25

Alongside this positive feedback, it’s important to consider some less complimentary comments too. These included a few outliers like ‘I wasn't impressed at all, nothing helpful came out of it’ or ‘for them to follow up with the support they had promised’. They also included calls for more detailed attention to individual businesses’ needs, however, one respondent suggested that ‘they need to sit down with smaller businesses and genuinely ask them what they want to get out of it, where they want to be in five years’ time and how they can help. No one actually asked me any of that. I think we were the first or second group in the system and I don't think at that point, they'd actually grasped what the construction sector was about and how diverse it was. It hasn't changed since the 1980s’. Similarly, an effort to create more specific opportunities was identified by one respondent as a possible improvement: ‘I would like a forum for specialist sub-contractors to be able to meet. Sometimes the events, and support, seems to be aimed at contractors, main contractors, rather than sub-contractors. I would like to see more for sub-contractors.’ Another thought that ‘CFW support could be improved by having one strong personal point of contact for a company as the support is not personalised enough at the moment’. A similar sentiment is echoed by another respondent who thought that ‘better communication between CFW staff’ would be needed, ‘and then integrating the different types of support better’. It is worth considering other patterns in terms of levels of satisfaction too. Figure 5.9: Satisfaction with interaction with CFW – by size of business

(n=87) The group of businesses with between 2 and 10 employees has the lowest satisfaction levels. This is reflected in respondent comments too.


26

Several comments suggest that in particular smaller businesses found it hard on occasions to gain benefits from their CFW interaction, with one, for instance, commenting that ‘it was alright on the day, the set up of it was OK but with the way it was presented, it wasn't targeting small businesses like mine, it was more for the bigger businesses’. Indeed, there are calls in the comments for CFW to focus more on the smaller end of SMEs as illustrated by one respondent commenting that ‘they need to start supporting grass roots businesses’ and suggesting that ‘it seems they are just going straight to the top and skipping us. Small businesses need more support with education, health and safety regulations, processes etc. but CFW seem to be bypassing us to get to the big companies at the top’. The corresponding analysis of responses by businesses’ target markets does not reveal any patterns, with satisfaction levels ranging from 3.9 to 4.1. Finally, a number of comments suggest that many businesses would like to be kept in the loop more proactively, both with regards to CFW’s own portfolio of activities and in terms of other supports, including grants; ‘Letting us know what else is available through CFW and what other resources we can tap into.’ Considering the picture by sub-sector suggests that there were marked differences in overall satisfaction levels between sub sectors. Figure 5.10: Satisfaction with interaction with CFW – by sub-sector

(n=117) Clearly, the core construction sub-sectors with a direct link into large contractor supply chains - and frequently public-sector end clients - were most positive about their overall interaction with CFW while more specialist businesses rated their overall satisfaction lower.


27

Some specific comments from respondents in these sub-sectors are illustrative of some of the issues. One specialised construction engineering business that had had a light touch engagement with CFW suggested, for instance, that they ‘had to ask a few people within CFW - although we're construction related, we're not actually building anything.’ In other words, the offer to them wasn’t quite clear. Another suggested that ‘they wanted plasterers and labourers - we're too specialist’. For some, there were clearly specific issues that go to the heart of the difficulty of creating opportunities for a longer supply chain. One respondent commented that ‘for our business we need to be in the loop much earlier - part of the process when engineers are spec-ing out the land. By the time MTB events happen, the contract has been specified and it's too late for us to be a part of a project’. Another echoed a similar point, commenting that ‘it may be different for other manufacturers, but in my experience these events are too late. Design and planning are already underway. It seems that the events are more for main contractors. It is too late for sub-contractors, or SMEs, to be able to gain anything.’ A further respondent was disappointed, because they ‘couldn't get any traction in the construction industry, ‘Our IT product was really applicable - there was already a similar product in the market and we thought ours would be accepted straight away. But it was too much of a change for the companies.’ Such comments provide a clear indication that where a more diverse and longer supply chain is targeted, formats of engagement also need to be more varied and bespoke. This sense is echoed in another comment relating to the health check which was seen by one respondent as ‘too generic and not specific enough for different sectors’. A third comment confirms the difficulty of accommodating diverse expectations by saying that ‘everybody has what they want to get out of it so it's difficult to get a course where everyone gets a satisfaction out of it.’ However, with these comments illustrating some of the tweaks that may be necessary to raise satisfaction levels in those sub sectors further too, and in spite of the slightly lower rating of overall satisfaction with the CFW interaction, positive comments about both the light touch and more intensive interventions prevailed in this group too. Finally, it is worthwhile to consider suggestions for improvements for the CFW support offer overall.


28

Figure 5.11: What could be improved?

(n=119) Finally, in answer to the question of whether respondents thought that CFW has made a difference for the construction sector in Wales as a whole, 92% of respondents thought that this was the case. The specific comments describe CFW as a valued vehicle that enhance the capacity of businesses in the sector and provides networking opportunities and greater coherence for the sector in Wales. The following sections that focus on the specific support elements will shed some additional light on the detail behind respondents’ experience of CFW.

5.4 Beneficiary experience of specific CFW supports

Beyond questions about the overall satisfaction with their CFW experience, survey respondents were also asked for their feedback about the individual CFW building blocks. This is presented in the following sub sections.

5.4.1 CFW events

To begin with, survey respondents were asked about their satisfaction with different aspects of CFW events, both thematic and Meet the Buyer. A clear majority of respondents across the light touch and intensive support cohorts confirmed that they found attending the events useful.


29

Figure 5.12: Usefulness of events

(n=89) Building on this overall assessment, respondents were asked about their satisfaction with specific aspects of the CFW events. Figure 5.13: Satisfaction levels with the quality of CFW events

(n=51) Figure 5.13 above shows that satisfaction levels ranged from 3.5 out of 5 for the uniqueness of the topic in the context of other construction sector events to 4.2 out of 5 for the relevance of the topic.


30

Specific thoughts ranged from a number of comments suggesting that ‘the events from CFW are always well run and managed’, and comments that highlighted the value of the specific insights gained, such as for instance one respondent who confirmed that ‘the events attended, specifically the NEC were invaluable because of the knowledge I gained from the speakers and by talking to other people in the industry, it was very helpful’. They also extended to specific ideas for tweaks and minor improvements, for instance included things like havening ‘information available about which companies are present and what they do so that I can target who I talk with on the day’ or more consistency in follow-up actions, like ‘sending out further information, on the topics that were discussed’. Clearly, the fact that the topics covered weren’t unique in the context of other construction sector events was seen as an issue by many respondents, since support for the relevance of the event topics is high. Significantly, a number of comments suggested that CFW was seen as the main route for some into this kind of thematic events, e.g. ‘I don't know what other events are happening’. Specific comments about the Meet the Buyer events were much more mixed. Some participants clearly got a lot of value out of them as illustrated in the following quotes:

‘The Meet the Buyer event got us access to 14 key construction companies and local authorities which normally we would struggle to get a meeting with. By meeting with them we could highlight the strengths of our business and also learn what they were looking in terms of supplier requirements.’

‘The meet the buyer event was most useful because it gave me access to presentations and decision makers that I was unlikely to get elsewhere.’ ‘I find the meet the buyer events very useful for Information gathering. I can quickly find out about current and upcoming projects.’ ‘The opportunity to network and get an introduction to the buyer. Making a contact is not the easiest thing for us - we don't have a sales force so business is built on word of mouth and reputation. Event also was time out from office - good to get some headspace and have time to reflect on it.’

At the same time, the Meet the Buyer events were the CFW element that came in for most criticism. Complaints related specifically to the extent to which they resulted in genuine opportunities or indeed contracts secured, e.g. ‘The supply chain events although good in terms of networking rarely result in actual contracts. The larger companies tend to dominate too much.’


31

Comments also related to potential levers available to CFW itself to enhance the experience, including the following:

‘They need to have more understanding of what Meet the Buyer events require. They should vet who will be there, in other words, who will attend. It is a waste of time. I have been to ten of these events. They have been a waste of time. That is ten days wasted for me.’ ‘It would be good to actually get the opportunity to price for the buyers at MTB events - to make sure they give people a definite chance.’ ‘More business support - in terms of presenting ourselves to the customer, and exactly what they'd be looking for.’ ‘It would be useful to have some feedback from the hosting company as well, or be told what they need from you. I still had to chase them myself, when I'd already completed their forms at the event.’ ‘The meet the buyer events need to be targeted at the companies who have a realistic chance of getting the tender or who are already been accepted so that the links are developed for that contract.’

The dilemma that CFW faces in having to work within existing market structures and relationships in the construction sector while trying to create opportunities for Welsh SMEs was summed up well by two respondents, with one suggesting that ‘the Meet the Buyer opportunities have to be genuine. There has to be pressure on the contractors to offer contracts to attendees and not just go through the motions. I have attended numerous meet the buy events and they have resulted in no contract.’ and a second respondent commenting that ‘encouraging collaboration between companies is a sticky issue, because some collaboration is successful and other times it can be a disaster. I think they need to be careful about who and what they're promoting collaboration with’. At the opposite end of the spectrum, however, the MTB events are seen as a CFW-created opportunity for SMEs to challenge large contractors, with one respondent, for instance, suggesting that ‘CFW can't do any more - if the buyers QSs don't want you to do the work, that's it. It was a useful chance to say to the buyers, "Why aren't you giving me the work when I'm already approved on your list?"’


32

5.4.2 One-to-one support

Respondents were asked about the extent to which they agreed with a number of statements about the one-to-one support they had received. Figure 5.14: To what extent do you agree with the following statements about the one-to-one support you received?

(n=47) Figure 5.14 above shows that satisfaction levels with the one-to-one support received were consistently high, with a large majority confirming that they agreed or strongly agreed with statements regarding the quality of different aspects of the support received. A sample of quotes will again illustrate the particular aspects that were valued by survey respondents:

‘We were delighted at the time - it helped with health and safety and being able to analyse where we're losing money and then stop that happening.’ ‘The advice given was always practical and something that we could do quite easily introduce and result in more efficient work practices.’ ‘It answered our questions and enabled us to take things further ourselves, it helped us to better understand issues and find out what we needed to do.’ ‘I was very happy with the support offered as the one-to-one support meant that we could tap into someone with a great deal of experience and knowledge.’


33

‘The one-to-one support was very useful as it was bespoke support tailored to what we needed. The advisor was knowledgeable and provided practical solutions to problems.’ ‘Help with the tender in one-to-ones - our first big tender and now we've built on that and used that model again. It set us on the right path.’ ‘We needed support with a range of certifications. The guy who came to see us was very helpful and knowledgeable. CFW effectively helped us create an Integrated Management System and get certified for 9014, 14001 and 18001 and also the EN 1090.’

Suggestions for improvements mostly consist in calls for more support as illustrated by the following quotes:

‘More general support, the guy we spoke to did as much as he could in helping us identify our business' needs in the construction industry, but we could have possibly done with a bit more general guidance overall. The business had only started up at the time so some of the advice and specifics discussed with the mentor would have been more appropriate for a longer standing business.’ ‘More follow-up from the CFW advisor after the one-to-one support. Businesses of this size are so focused on day-to-day tasks just to keep the business running - so more pressure from the advisor to implement their advice would help stop it "dropping off the list" and make sure that the changes happen.’ ‘To have a follow up after the one-to-one support. We had a really useful report given to us but we didn't know what to do with it and we didn't have further contact after that to help us with it and to make the most out of it.’

One specific comment also related to the quality of the advisors who had worked with the business, suggesting that ‘they need to be very picky over the expertise of their consultants. It was a bit woolly, the consultants vary immensely so some just get you and have a wealth of knowledge and others don't really get it and don't tell you anything you don't already know’. This was echoed in another negative comment suggesting that ‘during the initial meetings with the one-to-one advisor we didn't feel that he was experienced enough in delivering the support we required so we ended the support’.


34

5.4.3 Benchmarking

Respondents were asked about the extent to which they agreed with a number of statements about the benchmarking service they had received. Figure 5.15: To what extent do you agree with the following statements about the benchmarking service you received?

(n=16) Overall satisfaction levels are similarly high as for the one-to-one support. Specific comments in relation to the benchmarking service were limited, but shed some light on what businesses valued, including:

‘Benchmarking gave us peace of mind.’ ‘It flagged up to higher management the need to look at our long-term plans, look at the future and past the two-year contracts we have with some of our clients to establish what needs to be done later on.’ ‘It gave us a good insight in terms of where we were and where we needed to be.’ ‘More benchmarking would be valuable, because it would be useful to see how businesses have improved in the longer term.’

One critical view queried the extent to which the analysis was useful in leading to specific insights that could be actioned: ‘It was based on figures and financials (e.g. staff numbers and turnover), which wasn't what we were looking to increase immediately, we wanted to increase other things, like productivity. They took the figures away, and we got no real explanation of them, it wasn't clear.’ Another respondent suggested that ‘the benchmarking wasn’t integrated well with one-to-ones, there wasn’t much overlap’.


35

5.4.4 Leadership & Management training

Respondents were further asked about the extent to which they agreed with a number of statements about the L&M training they had participated in. Figure 5.16: To what extent do you agree with the following statements about the L&M training you participated in?

(n=36) As figure 5.16 above illustrates, the L&M course stands out as the CFW element that received the greatest support from those who had taken advantage of it, particularly in terms of both the context being relevant to businesses in the construction sector and with regard to the value of the qualifications gained. This can again be illustrated by a few quotes from survey respondents as follows:

‘The best thing about the CFW support was the variety of topics covered and the relevance of the course content to the challenges facing the construction industry.’ ‘The person leading the course was brilliant and very helpful.’ ‘The quality and level of the qualification available was excellent, and the fact that it is funded.’ ‘Some speakers were really good, and the collaboration workshops were useful, it was good to network with other businesses.’


36

‘The guest speakers at the L&M training were great, they talked about innovation which gave us some food for thought and made us think about things we previously hadn't thought of. We've put some of the things they discussed into practice, specifically how to improve some of our processes to try and make us a better company overall.’ ‘Attending the leadership course was really useful as it explained the different approaches to management and leadership and their strengths and weaknesses. The course helped develop my soft skills e.g. people skills and importance of body language.’ ‘The course raised my confidence in terms of leadership and management. The coaching on the leadership course was excellent.’ ‘It was the early stuff on the L&M course, looking at the leadership styles which were prevalent throughout the company, looking at what management systems we were using, our leadership styles and how to improve on those.’ ‘Overall it was useful because it gave me the ability to think about the business, to look at it from the outside, from a buyer’s perspective. I’d normally just get on and do the job so that was useful.’ ‘The course was challenging and hard work especially when maintaining the balance between work, leisure and the course but it was rewarding and the results and benefits from the course made it worth it.’

The module focusing on financial analysis was highlighted as particularly useful by several respondents:

‘For me personally, the finance section was very useful as I gained a much better understanding of the business' accounts so I'm able to look at them on a monthly or quarterly basis and understand what's going on. Also, the planning section of the training, looking at the three to five-year action plan was useful.’ ‘The financial element was useful. It showed us a different way of extracting meaningful data from our financial statements, how to understand them better and what it means for our cash-flow. It allowed us to get a bigger picture of our financial situation rather than just hearing it from the bank. It gave me the confidence to challenge the statements which I wouldn't have done before.’

Similarly, participating in the L&M course was described by a number of respondents as a very useful way of networking with other businesses in the construction sector.


37

While support for the content and taught elements appears to be consistently high, it is the focused written work that appears to most divide opinion, with higher proportions disagreeing with the usefulness of the three-year action plan and the assignment striking the right balance between being challenging and doable for busy business people. While one respondent, for instance, suggested that ‘the number of assignments for the leadership and management course was a little high’, another found precisely the assignments useful, ‘because I based them on what I wanted to learn more about, it helped me to look at things and learn about things I hadn't thought of before’. Criticisms included the suggestion that some elements of the course were not seen to be relevant to SMEs, e.g. ’the BIM element was unrealistic for a small businesses to undertake’ and that the time frame was difficult for SMEs to accommodate, with the course duration of two days a week over eight weeks being seen as ‘not great’ and ‘cramming in a lot of information in a short period of time’, suggesting that ‘the programme would be better delivered over a longer period’ and one respondent describing the course as ‘a little too intense and advanced, the speed a bit too quick and the workload too heavy’.

5.4.5 Engagement with a specific Tier 1 contractor or major client

Finally, respondents were asked about the extent to which they agreed with a number of statements about CFW support in engaging with a specific Tier 1 contractor or major client. Figure 5.17: To what extent do you agree with the following statements about the support in engaging with contractors/ clients?

(n=62)


38

If the Leadership & Management provision was the CFW element with the highest approval ratings, this is the one with the lowest approval ratings, particularly in relation to CFW support in identifying areas for improvement. The strongest approval ratings are recorded for the appropriateness and relevance of the contacts made.

5.5 Outcomes for CFW beneficiaries

An initial focus in considering what specific benefits the CFW support has generated in beneficiary businesses related to qualitative outcomes in terms of their individual or business capacity and capability. Both cohorts, those receiving light touch support and those benefiting from more intensive support in the form of one-to-one interventions or participation in the L&M course were asked about (i) any improvements in their understanding of requirements for stronger business performance and (ii) their awareness of future challenges. The results are highlighted in the figures below. Figure 5.18: Improvements in understanding of requirements for stronger business performance - light touch cohort

(n=60) Figure 5.19: Improvements in understanding of requirements for stronger business performance – intensive support cohort

(n=68)


39

For the light touch cohort, levels of agreement hover around the 50% mark for all aspects with the exception of collaborative working practices, where 60% of businesses in this cohort strongly agree or agree with the statement. For the intensive support cohort, solid majorities between 62% and 74% report that they strongly agree of agree with all statements regarding their understanding of requirements for stronger business performance. A comparative analysis of these results suggests that, unsurprisingly, across the board those who had received intensive support reported a stronger impact on their understanding of requirements for stronger business performance. The difference is most pronounced in relation to quality assurance systems and processes – with 27% more respondents from the intensive support group strongly agreeing or agreeing with the respective statement, a finding that may well indicate that this would have been a focus area in many of the more intensive support elements and interventions. The differential is lowest in relation to collaborative working practices, where only seven per cent more of the intensive support cohort strongly agree or agree that the CFW support has made a difference to their understanding of these issues. High overall proportions of both cohorts also confirm that the CFW support has strengthened their understanding in this respect (i.e. a total of 60% for the light touch cohort and 67% for the intensive support cohort). Considering the difference made by the CFW support in enhancing beneficiaries’ awareness of future challenges, the differences are more pronounced. Figure 5.20: Awareness of future challenges - light touch cohort

(n=60)


40

Figure 5.21: Awareness of future challenges – intensive support cohort

(n=68) As figures 5.20 and 5.21 show, a considerably higher proportion of respondents in the intensive support cohort confirm that the CFW support has given them a stronger awareness of future challenges, with over 70% either strongly agreeing or agreeing with this for sustainability concerns and changing construction practices and around 60% for technological developments and the impact of ICTs – compared to levels between 38% for technological developments, 43% for the impact of ICTs, 53% for sustainability concerns and a maximum of 58% for changing construction practices. The differentials between the two cohorts are highest with regard to technological developments and sustainability concerns, with 25% more respondents either strongly agreeing or agreeing in the intensive support cohort than the light touch cohort. Levels of disagreement are lowest in relation to the awareness of changing construction practices, where only two per cent more of the light touch cohort disagree with the statement that CFW has strengthened their awareness of such issues.


41

5.6 Tangible results secured by CFW beneficiaries

A final section of the survey focused on a consideration of the tangible quantitative results secured by CFW beneficiaries through the support received.

5.6.1 Gross results secured by survey respondents

A first set of questions related to any tendering activities that might have been stimulated and/ or strengthened through the CFW support. Figure 5.22: Tendering activity that wouldn’t have occurred without the support – by intensity of support

(n=110) As figure 5.22 above shows, a majority of respondents had not tendered for any contracts that they wouldn’t have tendered for without the CFW support. In a second stage, however, those that had submitted additional tenders as a result of the support were asked about the success rate for any additional tenders. Figure 5.23: Success rate in additional tendering activity stimulated by CFW support – by intensity of support

(n=45) Clearly, here the picture is much stronger, 76% of respondents reporting that they had been successful with the additional tenders they had submitted. A stark difference between the intensive support cohort at 86% and the light touch cohort at 59% suggests that this is not so much down to pure brokerage of contacts at Meet the Buyer events but does actually require more intensive investment in construction sector SMEs capabilities.


42

Figures 5.24 to 5.25 look at this distribution in terms of sub-sectors. Figure 5.24: Tendering activity that wouldn’t have occurred without the support – by sub-sector

(n=107) Figure 5.25: Success rate in additional tendering activity stimulated by CFW support – by sub-sector

(n=107)


43

Numbers in the different sub samples for company size are low, but it is worth considering overall patterns in that respect as depicted in figures 5.26 to 5.27 below. Figure 5.26: Tendering activity that wouldn’t have occurred without the support – by size

(n=82) Figure 5. 27: Success rate in additional tendering activity stimulated by CFW support – by size

(n=82) Building on this analysis, survey respondents were also asked, to what extent the CFW support had made a difference to their profitability.


44

Figure 5.28: Influence of CFW support on beneficiary profitability – by intensity of support

(n=104) As figure 5.28 illustrates, this is where the starkest differences occur between light touch and intensive support. While 58% of respondents from the high intensity cohort report that their profitability had increased to some extent through the support received, this was true for only 32% of businesses who had received light touch support. In addition, a sizeable minority of 13% of businesses in the high intensity cohort indicate that the support had impacted a great deal on their profitability whereas no light touch businesses reported a more sizeable effect. A total of 68% of the light touch cohort suggested that the support had had no effect on their profitability. Clearly, this kind of longer term impact on the fabric of a business cannot be achieved through awareness raising and Meet the Buyer events alone. Considering the pattern for different sub-sectors also reveals some clear differences. Figure 5.29: Influence of CFW support on beneficiary profitability – by sub-sector

(n=101)


45

As figure 5.29 shows, other professional, scientific and technical activities were most able to convert the CFW support received into increased profitability. Respondent businesses were also asked if they had created any jobs in their organisation since receiving support through the CFW programme. Figure 5.30 below illustrates that over half of all businesses surveyed, and over two thirds of businesses intensively assisted, had created jobs in their businesses since receiving support. Figure 5.30: Have you created any additional jobs since you started receiving the support?

Table 5.2 below illustrates that 422 FTE jobs were created across surveyed businesses, with 144 created amongst businesses in receipt of light touch support and 278 created amongst those intensively assisted. When asked to estimate the proportion of jobs created that respondent businesses attributed to CFW support, the mean average rate of attribution stood at 5.5% of jobs created amongst businesses in receipt of light touch support but almost five times that figure (26.2%) amongst businesses intensively assisted. By dividing the number of businesses surveyed against each intervention category against the number of FTE jobs created and attributing these to the support provided, it is possible to create an estimate of the number of jobs created per business supported across the CFW population as a whole. This illustrates that whilst amongst light touch businesses just 0.1 FTE job is created per business supported, amongst those intensively assisted, the impact attributed to CFW is 10 times that figure, at one additional job per business supported.


46

Table 5.2: Attributing Jobs Created to CFW Provision

Overall Light-touch Intensive

Total jobs created 422 144 278

Total jobs created attributed to support 81 8 73

Proportional Attribution 19.1% 5.5% 26.2%

Number of businesses interviewed 131 60 71

Jobs created per business attributed to support 0.62 0.13 1.03

These figures can be grossed up to provide an estimate of the total jobs created amongst all businesses supported through CFW and broken down by intensity of engagement, which can then be compared to the reported job creation figure in the monitoring data. Our analysis of CFW monitoring data identified a population of 1,657 unique businesses supported through the programme. A total of 1,405 of these businesses received ‘light touch support’ (including referrals to, but not completions of, leadership and management qualifications and 1-2-3 support). The remaining 252 businesses were in receipt of intensive support. Table 5.3 below provides further analysis on job creation. It grosses up the figures obtained through the business survey to the total number of unique businesses as identified through the research team’s analysis of the CFW database (based on data accessed in January 2018). The table shows that, if the rate of job creation and attribution identified through the survey was replicated across the entire population of businesses supported through CFW, it would equate to 443 jobs across all businesses and 259 across those businesses that had been intensively assisted. Only seven (12%) of the survey respondents in receipt of light touch support attributed job creation to the support received compared to 25 (35%) of intensively assisted businesses. Of those seven in receipt of light touch support that attribute job creation to the CFW programme, all attended a meet the buyer event and described that as the most useful element of their support. Our analysis of management information suggests that 571 businesses participated in a meet the buyer event (amongst other support) consequently at least 850 other businesses with light touch support did not. The meet the buyer event would appear to have played a critical role in job creation for some businesses but the majority of businesses classed as being in receipt of light touch support did not attend these events, therefore, erring on the side of caution, we have not included the job creation figures from light touch support in the economic impact model with only those intensively assisted being included.


47

Table 5.3: Estimated Direct Job Creation Effects of CFW


Jobs created per business attributed to support 0.62 0.13 1.03

Total businesses supported by CFW (to Jan 18) 1,657 1,405 252

Estimated jobs created 443 184 259

5.6.2 Net additional Impact of Intensive Support

To assess the net additional impact of the intensive support through CFW on participant businesses for Wales, a number of considerations need to be accounted for, namely:

• Deadweight – what would have happened in the absence of the support (this has already been calculated based on the self-reported attribution of jobs created to the support received)

• Displacement – how supporting one business to grow, displaces levels of activity within that market for another business

• Leakage – the proportion of impact of the support that leaks out of the target area

• Multiplier effects – the further economic activity (jobs, expenditure or income) stimulated by the direct benefits of an intervention.

In the absence of specific evidence in relation to the scale of these effects in the context of CFW, recognised and approved benchmarks have been used as ready reckoners for the impact model.2 Through the application of these formulae and the use of turnover per employee ratios for the construction sector in Wales (and GVA: turnover ratios)3 it is possible to generate an estimate in terms of net additional sales and net additional GVA. Importantly, turnover (sales), GVA and Jobs Created are different ways to express the net additional impact of CFW, the figures should not be combined. Table 5.4: Calculating the Net Additional Impact of CFW


Est. Jobs created per business 443 184 259

Deadweight Accounted for

Displacement 0.293 (29.3%)

Leakage 0.115 (11.5%)

Multiplier Effects 1.5

Net Additional FTEs n/a n/a 243

Net Additional Turnover (sales) n/a n/a 25,164,700

Net Additional GVA n/a n/a 10,141,374

2 See https://assets.publishing.service.gov.uk/government/uploads/system/uploads/attachment_data/file/191512/Research_to_improve_the_assessment_of_additionality.pdf 3 Derived from regional data within the Annual Business Survey: https://www.ons.gov.uk/businessindustryandtrade/business/businessservices/datasets/uknonfinancialbusinesseconomyannualbusinesssurveyregionalresultssectionsas

https://assets.publishing.service.gov.uk/government/uploads/system/uploads/attachment_data/file/191512/Research_to_improve_the_assessment_of_additionality.pdf

https://assets.publishing.service.gov.uk/government/uploads/system/uploads/attachment_data/file/191512/Research_to_improve_the_assessment_of_additionality.pdf

https://www.ons.gov.uk/businessindustryandtrade/business/businessservices/datasets/uknonfinancialbusinesseconomyannualbusinesssurveyregionalresultssectionsas

https://www.ons.gov.uk/businessindustryandtrade/business/businessservices/datasets/uknonfinancialbusinesseconomyannualbusinesssurveyregionalresultssectionsas


48

As the analysis in table 5.4 above shows, the intensive support is can be associated with the generation of £25m additional sales amongst the recipient businesses and net additional GVA of £10m.

5.7 Use of other supports

In considering CFW’s work with construction sector businesses, it is important to understand the extent to which CFW beneficiaries had also used other supports. Figure 5.31: CFW beneficiary use of other supports

(n=104) The answers to this question suggest that a majority of businesses had used other supports, in fact, only 28 businesses had not used other supports. Business Wales is clearly the most prevalent other support mechanism that CFW beneficiaries had taken advantage of, followed by CITB skills development support.

5.8 Need for additional support

Respondents were also asked to comment on whether any other supports would help strengthen the construction sector in Wales as a whole.


49

Figure 5.32: Need for additional support

(n=57) As illustrated in figure 5.32 above, 57 survey respondents made specific suggestions for additional supports. These included calls to promote the Welsh construction sector, to do more to attract school leavers into the industry and the call for a stronger focus on small businesses that has been running through free text comments as a red thread. By far the most frequent mention, however, were calls for more and specific training. Specific topics identified ranged from more on BIM, health & safety and tendering topics to a focus on collaborative practices, staff retention and further management skills as well as future proofing (e.g. ecological concerns, new technologies). Alongside this, several respondents highlighted the need for policy work in relation to changing the market dynamics and the dominant role of large contractors.


50

6 Conclusions and recommendations

6.1 Conclusions

In conclusion, the research undertaken for this evaluation paints a picture of strong Construction Futures Wales performance. This is true for most of the delivery activities and KPIs and for the value attributed to the support received by beneficiary businesses and key stakeholders. While the survey results suggest that the support has not worked for all SMEs, a majority offer positive feedback about their experience of using the CFW supports and the results achieved. This is true for both the light touch interventions of thematic and Meet the Buyer events and for the more intensive support elements, namely the one-to-one support and the Leadership & Management training. The delivery structure adopted for CFW has provided agility in responding to expressed SME needs, but relying on external delivery partners in delivering such agility has created challenges for the governance of the programme. There is therefore scope to further strengthen the internal integration of different elements of support and to further hone interactions with partners in the wider support infrastructure – both generic, such as Business Wales, and specialist construction sector providers, such as trade bodies and representative organisations that can play a role in enhancing the targeted marketing of the CFW offer. 76% of respondents overall and 86% of businesses in the intensive support cohort report that they had been successful with the additional tenders they had submitted. While 58% of respondents from the high intensity cohort report that their profitability had increased to some extent – compared to 32% of businesses in the light touch cohort - and a sizeable minority of 13% of businesses a great deal through the CFW support. A total of 68% of the light touch cohort suggested that the support had had no effect on their profitability. An economic impact analysis undertaken on the basis of the survey results suggests that the intensive support can be associated with the generation of £25m additional sales amongst the recipient businesses and net additional GVA of £10m. There appear to be instances where the CFW portfolio of supports does not fully meet businesses’ needs. The differences between different cohorts in terms of business size and sub-sector will allow CFW to further hone its offer. It is the Meet They Buyer events that come in for the greatest criticism, with comments in particular highlighting a degree of cynicism on the part of SMEs regarding the extent to which large contractors offer genuine supply chain opportunities for SMEs in Wales. At the same time, Tier 1 contractors themselves highly value the role of Construction Futures Wales in offering opportunities to identify Welsh SMEs that can help them meet the Community Benefit requirements expected by the Welsh Government and public-sector clients in Wales.


51

Similarly, Construction Futures Wales is seen as a valuable platform for other public-sector programmes and providers seeking to work with construction sector SMEs. CFW is seen to have built up a valuable database of Welsh businesses enabling the programme to have unrivalled reach into the construction sector in Wales and is valued as a useful platform that can complement individual professional and trade-based organisations by going beyond levy boundaries and working across a wide range of construction sub sectors. The focus on business development is recognised as appropriate by stakeholders and distinguishes CFW from other construction sector infrastructure organisations. CFW has actively engaged with and sought to respond to wider operational and strategic constraints in the construction sector in Wales and the associated challenges and tensions for SMEs and the programme itself. The fact that the Supply Chain Network activity, anticipated to create fora for closer working relationships between large contractors and SMEs, is the one CFW element that has not taken off as planned, is indicative of fundamental imbalances in the relationship between supply chain partners in the construction sector more generally. The dominant role of large contractors means that SMEs have limited incentives and capacity to look beyond immediate needs and to more actively consider future challenges. CFW has sought to address this by delivering on expressed needs while incorporating a degree of awareness raising of wider challenges. More fundamental improvements in Welsh construction sector SMEs’ competitive position, however, will to a large extent depend on policy interventions – many of which will be beyond the Welsh Government’s reach - aimed at establishing a more level playing field to allow SMEs to secure greater independence from Tier 1 contractors.

6.2 Recommendations

The recommendations that can be derived from the analysis of the evaluation evidence as presented in this report fall into two main groups, namely those relating to the specific building blocks of the CFW support offer and those relating to governance and management arrangements. These are presented separately in the following sections.

6.2.1 CFW governance and management

Recommendation 1: Further refine the logic model for Construction Futures Wales by clearly identifying the precise market failures the programme is designed to address and the policy objectives it can make a contribution towards. Recommendation 2: Refine and strengthen the link between Board level strategic decision-making and responsive day-to-day management of CFW operations by making more effective use of real time intelligence as derived from direct interactions with SMEs. Recommendation 3: Strengthen engagement with organisations representing different sub sectors, most notably trade and professional associations, to ensure strategic integration and maximise their potential contribution to the targeted marketing of the CFW offer.


52

6.2.2 CFW support offer

Recommendation 4: Continue to challenge and refine the thematic focus and coverage of the CFW offer in order to ensure that immediate SME needs can be met while future challenges and policy objectives are addressed. This should include the identification of innovation opportunities as well as additional support products such as support for stronger SME collaboration. Recommendation 5: Further refine the marketing approach by combining targeted opportunities for specific sub sectors and SME needs with an offer that makes the most of CFW’s value as a cross-sectoral platform. Geographical dynamics and the scope for CFW to play a role in creating momentum in different parts of Wales should also be taken into account as should appropriate linkages and referral mechanisms with other support providers. Recommendation 6: Consider the precise nature of the market failure in relation to SMEs’ role in supply chains for public sector construction projects and adjust and target the corresponding sourcing support to ensure that an appropriate balance of costs and rewards is achieved between public sector organisations in Wales, large contractors and SMEs.


53

Construction Futures Wales External Evaluation... · Construction Futures Wales External Evaluation...

Documents

Transcript of Construction Futures Wales External Evaluation... · Construction Futures Wales External Evaluation...