Project Evaluation under the Project Evaluation under the 2007-2013 Convergence and 2007-2013 Convergence and

Competitiveness Competitiveness ProgrammesProgrammes

Richard GauntRichard GauntMark BeynonMark Beynon

CRG Research Ltd.CRG Research Ltd.

OverviewOverview

About CRGAbout CRG WEFO RequirementsWEFO Requirements Experiences and Perceptions of EvaluationExperiences and Perceptions of Evaluation Case StudiesCase Studies Evaluation – Principles and PracticeEvaluation – Principles and Practice Do’s and Don’tsDo’s and Don’ts Q & AQ & A

About CRGAbout CRG Leading provider of research, evaluation,

consultancy and policy development services

Completed over 400 projects for a wide range of clients in both the public and private sectors

15 dedicated researchers from a range of disciplines, capable of utilising a full range of qualitative and quantitative research techniques

High level of expertise in different areas of Welsh policy and funding, including EU policy and Structural Fund Programmes

Excellent data collection, entry and analysis services

Full bilingual (English/Welsh) service provision

ISO 9001 accredited procedures and quality assurance.

WEFO RequirementsWEFO Requirements 2007-2013 Programmes – all sponsors required to undertake or 2007-2013 Programmes – all sponsors required to undertake or

commission evaluation of projectscommission evaluation of projects All projects awarded £2million + in grant (ERDF/ESF) are All projects awarded £2million + in grant (ERDF/ESF) are

required to have their projects evaluated by an external required to have their projects evaluated by an external independent contractor (+ projects involved in implementing independent contractor (+ projects involved in implementing ERDF innovative or experimental actions and innovative ERDF innovative or experimental actions and innovative projects under Article 7 ESF)projects under Article 7 ESF)

Projects below this threshold may still wish to appoint external Projects below this threshold may still wish to appoint external contractor because of independent perspective which can be contractor because of independent perspective which can be providedprovided

The size of the evaluation should be proportionate to the size The size of the evaluation should be proportionate to the size and complexity of the projectand complexity of the project

Costs associated with undertaking evaluation are eligible for Costs associated with undertaking evaluation are eligible for Structural Fund assistanceStructural Fund assistance

RationaleRationaleWhy EvaluateWhy Evaluate

Assess whether projects achieve their objectives How efficiently were outputs and results achieved Wider consideration of outcomes and impacts What would have happened without the intervention

Evaluation is wider in scope than monitoringEvaluation is wider in scope than monitoring Whilst monitoring data can help asses performance against objectives –

wont produce rounded judgement of success Evaluation – considers quality of achievements and contextual factors

which have helped/hindered success Example: Monitoring data will tell you 100 jobs created, but evaluation

will tell you the quality of those jobs, the impact on beneficiaries, and considers whether a proportion of the jobs would have been created in any case

Share good practice, make recommendations for future action.

Evaluation PlanningEvaluation PlanningSetting Aims and ObjectivesSetting Aims and Objectives

Evaluation Plan to be submitted when developing your project Outline evaluation activity during the life of your project State when evaluation will be undertaken, by whom, and which

evaluation methods will be used. Plan will identify key evaluation questions, and the scope and size of

the evaluation Size of evaluation proportionate to size and risk of project – agreed with

the WEFO Project Development Officer at development stage Guidance on evaluation questions and suggested methodologies

available on WEFO Website This is an effectively an overview document – doesn’t necessarily have

all the detail you need to design and implement the evaluation and ensure it is integrated into your project

Who should evaluate?Who should evaluate? Size of project: > £2million = External, < £2Million = option to run it Size of project: > £2million = External, < £2Million = option to run it

internallyinternally If your project falls below threshold you may still wish to go external. If your project falls below threshold you may still wish to go external.

Issues to consider include:Issues to consider include: Do you have the skills internallyDo you have the skills internally Do you have enough staff resource to carry out the workDo you have enough staff resource to carry out the work Independent PerspectiveIndependent Perspective External Evaluation CostExternal Evaluation Cost Contract Management TimeContract Management Time

The other factor to consider is risk. High Risk/ High Profile projects will The other factor to consider is risk. High Risk/ High Profile projects will require a thorough evaluation which may well be resource intensive. require a thorough evaluation which may well be resource intensive. Project which are innovative or pilot in nature or where there are learning Project which are innovative or pilot in nature or where there are learning or participatory elements will need more resource intensive evaluations.or participatory elements will need more resource intensive evaluations.

Key Factors: Cost, Complexity, Resources, External PerspectiveKey Factors: Cost, Complexity, Resources, External Perspective

Objectives of Project EvaluationObjectives of Project Evaluation

Demonstrating the need for your project - feasibility

Evaluating Project Processes

Evaluating Project Outcomes

Reporting against Impact Indicators

Issues to ConsiderIssues to Consider Formative vs. Summative – important decision that will shape the research Formative vs. Summative – important decision that will shape the research

methodology and impact upon the way the project makes use of evaluation data. Big methodology and impact upon the way the project makes use of evaluation data. Big advantage of formative is that it provides an ongoing assessment of progress against advantage of formative is that it provides an ongoing assessment of progress against objectives, and allows opportunities/ corrective actions to be identified at every objectives, and allows opportunities/ corrective actions to be identified at every stage. stage.

Methods for collecting Data – Covers primary and secondary data, and methods Methods for collecting Data – Covers primary and secondary data, and methods should be appropriate to answer the key research questions. Selection of method has should be appropriate to answer the key research questions. Selection of method has resource implications. Weigh up the pro’s and con’s of different approachesresource implications. Weigh up the pro’s and con’s of different approaches

Methods for analysing data – should be considered when deciding the methods, size Methods for analysing data – should be considered when deciding the methods, size and scope of your evaluation. The method of data collection will have a clear impact and scope of your evaluation. The method of data collection will have a clear impact on the method of data analysis. The two main methods are:on the method of data analysis. The two main methods are:

Quantitative – statistics used to provide characteristics of a sample, and if there is sufficient Quantitative – statistics used to provide characteristics of a sample, and if there is sufficient data, to derive conclusions about performance of the projectdata, to derive conclusions about performance of the project

Qualitative – patterns and themes emerging from interview transcripts, data or recordings Qualitative – patterns and themes emerging from interview transcripts, data or recordings can be categorized to provide an analytical description of a sample and the key can be categorized to provide an analytical description of a sample and the key themes/issues emerging from a projectsthemes/issues emerging from a projects

Reporting & Dissemination – how do you intend to use evaluation data? Is it just to Reporting & Dissemination – how do you intend to use evaluation data? Is it just to satisfy WEFO requirements or will it actively be used as part of the project satisfy WEFO requirements or will it actively be used as part of the project management process.management process.

Research SpecificationResearch SpecificationThe Specification will set the parameters for the The Specification will set the parameters for the evaluation. It will need to include:evaluation. It will need to include:

IntroductionIntroduction Background/ Requirements Background/ Requirements Method/Budget – 2 options: detail method or detail budgetMethod/Budget – 2 options: detail method or detail budget Reporting/ DeliverablesReporting/ Deliverables DisseminationDissemination TimetableTimetable Data Protection IssuesData Protection Issues

Contract Award CriteriaContract Award Criteria Contract Management ArrangementsContract Management Arrangements CostCost

Internal/External?Internal/External?

Commissioning EvaluationCommissioning EvaluationIf external evaluators are to be used, need to If external evaluators are to be used, need to commission effectively. Issues to consider:commission effectively. Issues to consider:

Detailed Research Specification and Invitation to Tender Detailed Research Specification and Invitation to Tender Evaluation should meet project evaluation needs and WEFO requirementsEvaluation should meet project evaluation needs and WEFO requirements Need to comply with your own procurement guidelinesNeed to comply with your own procurement guidelines WEFO will advise on approach, method and budgetWEFO will advise on approach, method and budget PDO’s don’t appear to be paying much attention to evaluation pre-PDO’s don’t appear to be paying much attention to evaluation pre-

approval. Uncertainty over to what degree WEFO will involve themselves approval. Uncertainty over to what degree WEFO will involve themselves in the commissioning process at later stagesin the commissioning process at later stages

Evaluation costs are eligible for Structural Fund Support, so the Evaluation costs are eligible for Structural Fund Support, so the expenditure will be subject to approval, audit, monitoring, etc.expenditure will be subject to approval, audit, monitoring, etc.

Possibility of Supplier Frameworks being used to provide details of Possibility of Supplier Frameworks being used to provide details of approved ‘reputable’ companiesapproved ‘reputable’ companies

Decide on approach early and work with WEFO to ensure they are aware, Decide on approach early and work with WEFO to ensure they are aware, and happy with the way the project is being evaluated.and happy with the way the project is being evaluated.

The Evaluation ExperienceThe Evaluation Experience

Evaluation can be a powerful project management and review tool. Can measure progress against targets, identify what works well/ less well, provides the opportunity for corrective action to minimise risks and take advantage of unforeseen opportunities.

It also identifies best practice, and explores outputs, outcomes and impacts. Recommendations can inform both existing and future delivery.

If badly designed, poorly resourced, or ineffectively delivered they can be a burden to project managers, partners and beneficiaries.

Keen to hear your experiences?? Any good news/ horror stories? Any concerns about evaluation?

Case Study 1 Case Study 1 Research Skills TrainingResearch Skills Training

Objective 1 ESF funded collaborative project between Objective 1 ESF funded collaborative project between Bangor, Aberystwyth and Swansea UniversitiesBangor, Aberystwyth and Swansea Universities

Funded 67 PhD Studentship and ran collaborative Funded 67 PhD Studentship and ran collaborative research projects with regional SMEsresearch projects with regional SMEs

““to provide research training to enable individuals to to provide research training to enable individuals to develop the skills to contribute to research as develop the skills to contribute to research as professionals. The project will also increase the research professionals. The project will also increase the research capacity of SMEs, encourage them to undertake research capacity of SMEs, encourage them to undertake research and recruit researchers. It will also support the and recruit researchers. It will also support the development of key technology clusters in the region”.development of key technology clusters in the region”.

Case Study 1Case Study 1Research Skills TrainingResearch Skills Training

Summative EvaluationSummative Evaluation

Research Methodology:Research Methodology: Desk researchDesk research Face to Face Interviews with academic supervisorsFace to Face Interviews with academic supervisors Case Studies and Interviews with company supervisorsCase Studies and Interviews with company supervisors Interviews with key management and administrative staffInterviews with key management and administrative staff Focus Groups and telephone interviews with beneficiary Focus Groups and telephone interviews with beneficiary

PhD StudentsPhD Students Regular updates, draft final and final reports to clientRegular updates, draft final and final reports to client

Case Study 1 Case Study 1 Research Skills TrainingResearch Skills Training

Outcomes for the client:Outcomes for the client: Thorough evaluation of the project model, internal processes, outputs Thorough evaluation of the project model, internal processes, outputs

and outcomes for all parties – University, companies and most and outcomes for all parties – University, companies and most importantly the studentsimportantly the students

Highlighted successes, and explored why some collaborations failed to Highlighted successes, and explored why some collaborations failed to achieve anticipated outcomes – unintended outcomesachieve anticipated outcomes – unintended outcomes

Identified a number of learning points for future collaborative projects – Identified a number of learning points for future collaborative projects – client able to refine the model for new funding bidclient able to refine the model for new funding bid

Final Report completed in time for client to use as part of discussions Final Report completed in time for client to use as part of discussions with funder for future phases – powerful tool, evidence success and with funder for future phases – powerful tool, evidence success and demonstrate commitment to continual improvementdemonstrate commitment to continual improvement

Proactive approach to evaluationProactive approach to evaluation

Case Study 2Case Study 2DCELLS Skills in the WorkplaceDCELLS Skills in the Workplace

Skills in the Workplace, launched in June 2005, was a Welsh Skills in the Workplace, launched in June 2005, was a Welsh Assembly Government initiative designed to Assembly Government initiative designed to “raise skill levels of “raise skill levels of employees and create an ethos of training within SMEs in North employees and create an ethos of training within SMEs in North Wales”Wales”..

The project attracted in excess of £4 million of European Social The project attracted in excess of £4 million of European Social Fund (ESF) support, and ran until July 2008.Fund (ESF) support, and ran until July 2008.

FE Colleges and Private Training providers offering ‘bite size’ FE Colleges and Private Training providers offering ‘bite size’ chunks of training to SMEschunks of training to SMEs

Case Study 2Case Study 2DCELLS Skills in the WorkplaceDCELLS Skills in the Workplace

Comprehensive Formative Evaluation, run over 3 stages – Comprehensive Formative Evaluation, run over 3 stages – Baseline, Mid-point, FinalBaseline, Mid-point, Final

4 Thematic Reports – confusion in the marketplace, size of 4 Thematic Reports – confusion in the marketplace, size of company, sectoral analysis, modes of deliverycompany, sectoral analysis, modes of delivery

Research Methodology:Research Methodology: KI Interviews KI Interviews Employer SurveyEmployer Survey Company Case StudiesCompany Case Studies Desk Research/ MI AnalysisDesk Research/ MI Analysis Regular attendance at steering group meetings Regular attendance at steering group meetings Dissemination/Stakeholder eventsDissemination/Stakeholder events Comprehensive reporting Comprehensive reporting

Case Study 2Case Study 2DCELLS Skills in the WorkplaceDCELLS Skills in the Workplace

Outcomes for the client:Outcomes for the client:

Formative evaluation model, measuring distance travelled and Formative evaluation model, measuring distance travelled and allowing partners to identify issues/opportunities at every allowing partners to identify issues/opportunities at every stage of delivery – integrated approachstage of delivery – integrated approach

Thematic reports offered in-depth analysis of key themes Thematic reports offered in-depth analysis of key themes which impacted upon or resulted from the projectwhich impacted upon or resulted from the project

Conclusions and Recommendations used to inform and Conclusions and Recommendations used to inform and evidence decisions for future business support modelsevidence decisions for future business support models

Evaluation was at the heart of dissemination efforts which Evaluation was at the heart of dissemination efforts which celebrated successes and reflected upon lessons learntcelebrated successes and reflected upon lessons learnt

Evaluation PrinciplesEvaluation Principles

In practice evaluation is seldom In practice evaluation is seldom “pure”“pure”

It reflects real life situationsIt reflects real life situations It’s done for a mix of purposesIt’s done for a mix of purposes Combines a number of types and design optionsCombines a number of types and design options Measures at a number of levelsMeasures at a number of levels Utilises a mixture of techniques (triangulation)Utilises a mixture of techniques (triangulation) It allows judgements to be madeIt allows judgements to be made

Evaluation is not an exact scienceEvaluation is not an exact science

Why Evaluate?Why Evaluate?

TO LEARNTO LEARN Demonstrate/plan/improve utilisation of resourcesDemonstrate/plan/improve utilisation of resources Provide evidence to fundersProvide evidence to funders Develop provisionDevelop provision Inform policy developmentInform policy development AccountabilityAccountability Because we have to?Because we have to?

When do you evaluate?When do you evaluate?

Taken from: The Green Book: Appraisal and Evaluation in Central Government. Treasury Guidance

What do we really What do we really want to know?want to know?

Long-term impact or Long-term impact or short term outcomesshort term outcomes

Areas for Areas for improvement improvement

‘‘Next time’Next time’ Policy reviewPolicy review

Ex-Ante (feasibility, what if)Ex-Ante (feasibility, what if) Formative (action research/ could it be better/different?)Formative (action research/ could it be better/different?) Summative (what we did/achieved)Summative (what we did/achieved) Process (how was it implemented)Process (how was it implemented) Output (what was delivered)Output (what was delivered) Outcome or Impact (what difference it made)Outcome or Impact (what difference it made) Goal free (unintended consequences)Goal free (unintended consequences) Economic Appraisal Economic Appraisal (Builds on, covers more than monitoring)(Builds on, covers more than monitoring)

Evaluation OptionsEvaluation Options

Design OptionsDesign Options

Desk Research (what’s already known)Desk Research (what’s already known) Data Reanalysis (what’s available already?)Data Reanalysis (what’s available already?) Cross-sectional (cost-effective sampling)Cross-sectional (cost-effective sampling) Comparative (between groups/contexts)Comparative (between groups/contexts) Longitudinal (over time)Longitudinal (over time) Some/ all of the aboveSome/ all of the above

Unit of AnalysisUnit of Analysis

Small/ larger areasSmall/ larger areas Stakeholder groupStakeholder group Individual/cohort (farmer or group of farmers, offenders)Individual/cohort (farmer or group of farmers, offenders) Sector (farming, all farmers, prisons or prisoners)Sector (farming, all farmers, prisons or prisoners) The economy (all economic outputs)The economy (all economic outputs)

Qualitative TechniquesQualitative TechniquesSemi-structured data collectionSemi-structured data collection

Focus groups, telephone and face-to-face interviewsFocus groups, telephone and face-to-face interviews ObservationsObservations WorkshopsWorkshops DiariesDiaries Data collection – topic guides vs. questionnairesData collection – topic guides vs. questionnaires AnalysisAnalysis

ThematicThematic Content AnalysisContent Analysis EthnographicEthnographic

Quantitative TechniquesQuantitative TechniquesStatistical analysisStatistical analysis

Surveys – CAPI, telephone, face-to-face, postal, Surveys – CAPI, telephone, face-to-face, postal, internet/emailinternet/email

A set of clear questions to poseA set of clear questions to pose Hard dataHard data

Census – fact basedCensus – fact based Statistical returns/ MISStatistical returns/ MIS Trends/statistical reanalysisTrends/statistical reanalysis Enhance primary dataEnhance primary data

Data Collection (1)Data Collection (1)AdvantagesAdvantages

Face to face interviews:Face to face interviews: Good for open-ended issuesGood for open-ended issues Good (80%) response ratesGood (80%) response rates

Group interviews:Group interviews: Excellent for open-ended Excellent for open-ended

issuesissues May be developmental in May be developmental in

themselves (staff)themselves (staff)

Focus groups/community panels:Focus groups/community panels: Good for exploring issuesGood for exploring issues Possible to standardisePossible to standardise Possible to preserve anonymityPossible to preserve anonymity

DisadvantagesDisadvantages

Difficult to analyseDifficult to analyse Resource heavy (for researcher & Resource heavy (for researcher &

interviewee)interviewee)

Difficult to organise and analyseDifficult to organise and analyse Need trained interviewersNeed trained interviewers Useless for “competitors”Useless for “competitors”

Needs trained facilitatorsNeeds trained facilitators Difficult to analyse accuratelyDifficult to analyse accurately May be to balance the groupMay be to balance the group

Data Collection (2)Data Collection (2)

AdvantagesAdvantages

Telephone interviews:Telephone interviews: Good is the topics can be Good is the topics can be

defineddefined Quick (if you can get through)Quick (if you can get through)

Postal questionnaire:Postal questionnaire: Easy to administerEasy to administer Ensures privacyEnsures privacy Good for standardised dataGood for standardised data

DisadvantagesDisadvantages

Poor for qualitative datePoor for qualitative date Resistance to SUGOMResistance to SUGOM

Need to define questions very Need to define questions very tightlytightly

Low response rateLow response rate Unable to control who fills in the Unable to control who fills in the

formform

ReportingReporting

To whom?To whom? For what purpose?For what purpose?

Brief report to a meeting?Brief report to a meeting? Feedback to participants/clients/more widely?Feedback to participants/clients/more widely? Part of an on-going process?Part of an on-going process? Need for standardisation?Need for standardisation? Multiple audiencesMultiple audiences

SummarySummary Main reportMain report Extended analysisExtended analysis

Feedback to respondentsFeedback to respondents

EvaluationEvaluationDo’s & Don’tsDo’s & Don’ts

Do…Do… Establish key research questions from the outset – what do you really Establish key research questions from the outset – what do you really

want to know?want to know? Collect good baseline dataCollect good baseline data Clear specification will ensure better quality of tendersClear specification will ensure better quality of tenders Consider the pro’s and con’s of internal/external evaluationConsider the pro’s and con’s of internal/external evaluation How do you want the evaluation to work – is it to satisfy the funder, or How do you want the evaluation to work – is it to satisfy the funder, or

do you want it work as a project management tool do you want it work as a project management tool Make evaluation work for youMake evaluation work for you Make sure you set aside adequate time and resources to implement and Make sure you set aside adequate time and resources to implement and

manage the evaluationmanage the evaluation Involve stakeholders in the evaluationInvolve stakeholders in the evaluation Disseminate and act on findingsDisseminate and act on findings

Don’t…Don’t…

Think about evaluation too late - unlikely to be effectiveThink about evaluation too late - unlikely to be effective Send out ITT with ill-defined ideas and requirements – difficult for Send out ITT with ill-defined ideas and requirements – difficult for

tenders to respond to effectively; difficult to compare tenderstenders to respond to effectively; difficult to compare tenders Leave evaluators working in isolation – manage and learn from Leave evaluators working in isolation – manage and learn from

the processthe process Cast the net too wide when commissioning evaluators – wastes Cast the net too wide when commissioning evaluators – wastes

time and resources for all time and resources for all Lose sight of why you are evaluatingLose sight of why you are evaluating Treat evaluation as a bolt-onTreat evaluation as a bolt-on Ignore findings – act on themIgnore findings – act on them

Thanks for your time Thanks for your time

Happy to receive comments or queriesHappy to receive comments or queries

Mark BeynonMark BeynonCRG ResearchCRG Research

25 Cathedral Road25 Cathedral RoadCardiffCardiff

CF11 9TZCF11 9TZ

029 2022 3218029 2022 3218mark@crgresearch.co.ukmark@crgresearch.co.uk