EVALUATION OF SME AND ENTREPRENEURSHIP PROGRAMMESJonathan Potter and Stuart Thompson

Trento13th November 2015

Structure of the presentation

1. Role and importance of evaluation2. Approaches to evaluation – measures and

techniques3. Evaluation examples4. Planning and organising evaluation5. Some conclusions

Part 1

Role and importance of evaluation

OECD Istanbul Ministerial Declaration

Recognised“The need to develop a strong evaluation culture in ministries and agencies responsible for SME policies and programmes” “Evaluation provides a means of ensuring that SME programmes remain cost-effective and adapt to changing conditions in a dynamic world”

What do we mean by evaluation?

Papaconstantinou and Polt (1997): “Evaluation refers to a process that seeks to determine as systematically and objectively as possible the relevance, efficiency and effectiveness of an activity in terms of its objectives” Purpose is to provide decision makers with the best information to answer crucial questions, e.g.• Does my programme address the key

issues? • What is working and not working? • What can I do better next time?

Why evaluate?

• Learning from experience• To establish the impact of policies and

programmes against their objectives• To make informed decisions about the

allocation of funds across a portfolio• To show the taxpayer whether the

programme is a cost-effective use of public funds

• To stimulate informed debate• To achieve continued improvements in the

design and administration of programmes

Some evaluation questions

• Is the activity still relevant? • Is it achieving its objectives? • Is it cost effective and proportional to

what it seeks to achieve? • Should the activity be continued or

terminated? • If continued, how can it be improved?• Can it be used in different contexts?• What issues will occur when scaling up a

pilot project?

Part 2

Approaches to evaluation – measures and techniques

Inputs          

Resources required for the programme (finance, people etc)

  Often defined in advance, needs to be sufficient

  ↓                

  Activities        

Actual actions taken (training courses, advice services etc)

  Determined from technical considerations and experience

    ↓              

    Outputs      

Immediate measure of activities (number of trainees, people advised)

  Determined from previous experience and evaluations

      ↓            

      Outcomes    

Change in behaviour of target (number of start-ups etc)

  Requires detailed analysis and research

        ↓          

        Impacts  

Overall effect (noting what would have happened anyway etc)

  Often only calculable well after the programme is complete

Measures at different levels

Measure Definition Typical QuestionsRelevance The extent to which the activity is

suited to the priorities and policies of the target group, recipient and government.

To what extent are the objectives of the programme still valid?Are the activities and outputs of the programme consistent with the overall goal and the attainment of its objectives?

Effectiveness The extent to which the intervention's objectives were achieved, or are expected to be achieved, taking into account their relative importance.

To what extent were the objectives achieved / are likely to be achieved?What were the major factors influencing the achievement or non-achievement of the objectives?

Efficiency The outputs in relation to the inputs. It signifies that the intervention uses the least costly resources possible in order to achieve the desired results.

Were activities cost-efficient?Were objectives achieved on time?Was the programme or project implemented in the most efficient way compared to alternatives?

Impact 

The positive and negative changes produced by an intervention, directly or indirectly, intended or unintended on the local social, economic, environmental and other development indicators.

What has happened as a result of the programme or project?What real difference has the activity made to the beneficiaries?How many people have been affected?

Sustainability Sustainability is concerned with measuring whether the benefits of an activity are likely to continue after funding has been withdrawn. Projects need to be environmentally as well as financially sustainable.

To what extent did the benefits of a programme or project continue after funding ceased?What were the major factors which influenced the achievement or non-achievement of sustainability of the programme or project?

Typical evaluation measures

Monitoring Step I Take up of program Step II Recipients’ opinions Step III Recipients’ views of the differences

made by the program Evaluation Step IV Comparison of the performance of

‘assisted’ with ‘typical’ ventures Step V Comparison with ‘matched’ ventures Step VI Taking account of selection bias

Approach - six steps

The ideal approach – Randomised control group

Example - GATE

• Open to all, but targeted at the unemployed

• Applicants randomly given assistance or not

• No barriers to using other sources of advice

• Result showed robust evidence of an impact

Impaq International (2009), Growing America Through Entrepreneurship: Final Evaluation of Project GATE

Regression model

N

NN N

N

N N

N

NN

N

N

NN

N N

N

NN

N

B

B

BB B

BBB

B

BB

B

BB

B

B

B

B

Out

put v

aria

ble

(res

ult o

f a v

arie

ty o

f fa

ctor

s af

fect

ing

firm

cha

ract

eris

tics)

Firm characteristics (control variables and value of policy support)

Beneficiaries

Non-beneficiaries

Matching (propensity scoring)

Differences in differences

Regression discontinuity

Part 3

Evaluation examples

Example – Prince’s Trust Youth Business Initiative

Greene, Francis J (2005), Evaluating Youth Entrepreneurship: the Case of the Prince’s Trust

• The Prince’s Trust is one of the longest running programmes to support youth entrepreneurship in the United Kingdom. It uses a mix of soft (mentoring) and hard support (loans and grants) and has maintained the same approach over a long period.

• The programme’s specific aims are:• Supporting individuals who would be unlikely

to start up an enterprise• Deepening the human capital of participants• Improving survival rates

Prince’s Trust - methodology

• Survey of recipients• Survey of non-recipients• Survey of non-survivors• Matched survey with sample

selection• 3600 respondents• Survival rates assessed over 3 years

Prince’s Trust - results

• Only mentoring appears to have a positive effect

• This is at variance with simpler evaluations which only looked at the views of participants, who were very positive

• No evidence that entrepreneurship support assisted employability

Example – Start-up grants to unemployed immigrants in Germany

• Germany has had long term policies of providing support and “welfare bridges” to the unemployed i.e. continued subsidies even when they are self-employed. Exact levels of benefits have varied over time.

• Immigrants have different characteristics from the general population and may respond to policies differently.

• There have been a number of positive evaluations of the scheme as a whole, but these do not mean that it works in the same way for all groups of the unemployed, for example immigrants.Robinson, Eric (2011), Start-Up Grants to Unemployed Immigrants in

Germany: A Means to Reach Employment Parity?

Methodology

• Use of long term socio-economic database, the German Socio Economic Panel – including more than 140,000 individuals tracked between 1983 and 1993.

• Regression techniques, controlling for sex, age, marital status, education, work experience, sector, type of self-employment

Results

• Unlike natives, foreigners are more likely to lapse back into unemployment when start-up benefits are high. This difference is statistically significant.

• Consideration of skill levels suggests that this is because high benefits give an incentive for ill-prepared individuals to start up business.

Part 4

Planning and managing evaluations

Evaluation and the policy cycle

ProgrammingWhat is the objective?

IdentificationWhat are the barriers?

FormulationHow can we address it?

ResourcingCan we mobilise resources?

ImplementationIs it on track?

Ex Post EvaluationDid it produce the results?

Ex Ante EvaluationOngoing/Interim

Evaluation

Meta evaluationsBenchmarking

Terminal Evaluation

• A timetable for evaluations• Responsibility for commissioning or

performing evaluations• Quality assurance• Mobilisation of resources

Evaluation Framework - management

• Identifying stakeholder needs• Deciding what to evaluate• Key Performance Indicators• Timely and accurate information• Qualitative information• Information regarding assumptions• Baselines• Counterfactual

Evaluation framework - information

Getting the information?Cost

Complexity

Administrativereports

Managementreports

Availablestatistics

Adapted availablestatistics

Interviews withbeneficiaries

Specialisedsurveys

Dealing with the results

• Is there logic in the report?• Is it peer-reviewed? Quality assured?• Have things changed?• What needs to be public?• What is politically necessary?

Part 5

Conclusions

Conclusions

• It is important to evaluate consistently and robustly across programmes

• Effective and efficient evaluation requires planning from the beginning of implementation

• Evaluation often involves deciding on proportionality – the amount of effort involved in gathering data against the usefulness of the result

• Managing authorities need to develop capacity in commissioning evaluations rather than in actually doing them

• Full impact evaluations can be difficult

Questions