Evidence Review Business Advice - whatworksgrowth.org
Transcript of Evidence Review Business Advice - whatworksgrowth.org
Evidence Review
Business Advice
May 2014
Contents
1. Preface 2
2. Executive summary 4
3. Introduction 8
4. Impact evaluation 10
5. Methodology 13
6. Definition 16
7. Findings 17
8. Detailed findings 23
9. Summary of findings 32
10. How to use this review 34
11. References 36
Appendix A: Evidence Reviews 38
Appendix B: Search Terms and Sources 40
Appendix C: Findings for RCT studies 45
Appendix D: Case Studies 47
00
Evidence Review: Business Advice - May 2014 3
Preface
Thisreportpresentsfindingsfromasystematicreviewofevaluationsofbusinessinformation,adviceandmentoringprogrammes(‘businessadvice’)aimedatimprovingbusinessgrowthandotheroutcomes.
ItisthesecondofaseriesofreviewsthatwillbeproducedbytheWhat Works Centre for Local Economic Growth.TheWhatWorksCentreisacollaborationbetweentheLondon School of Economics and Political Science,Centre for CitiesandArupandisfundedbytheEconomic & Social Research Council,The Department for Communities and Local GovernmentandThe Department for Business Innovation & Skills.
Thesereviewsconsideraspecifictypeofevidence–impact evaluation–thatseekstounderstandthecausaleffectofpolicyinterventionsandtoestablishtheircost-effectiveness.Toputitanotherwaytheyask‘didthepolicywork’and‘diditrepresentgoodvalueformoney’?Bylookingatthedetailsofthepoliciesevaluatedwecanalsostarttoanswerquestionsaboutdeliveryissues–forexample,whetherbusinessadvicepolicieswhichusemanagedbrokerage(i.e.a‘hands-on’approach)performbetterthanprogrammesthatarelighttouch(i.e.involvelittleornoengagementwithclients).
Evidenceonimpactandeffectivenessisclearlyacrucialinputtogoodpolicymaking.Processevaluation–lookingindetailathowprogrammesoperatedaytoday–providesavaluablecomplementtoimpactevaluation,butwedeliberatelydonotfocusonthis.Werecognisethatmaysometimescausefrustrationforpractitionersanddecision-makerswhoareresponsibleforthedeliveryofpolicy.However, we see these impact-focused reviews as an essential part of more effective policy making.Weoftensimplydonotknowtheanswerstomanyofthequestionsthatmightreasonablybeaskedwhenimplementinganewpolicy–notleast,doesitwork?Figuringoutwhatwedoknowallowsustobetterdesignpoliciesandundertakefurtherevaluationstostartfillingthegapsinourknowledge.This also helps us to have more informed discussions about process and delivery issues and to improve policy making.
Thesereviewsthereforerepresentafirststepinimprovingourunderstandingofwhatworksforlocaleconomicgrowth.Inthemonthsahead,wewillbeworkingwithlocaldecision-makersandpractitioners,usingthesefindingstohelpthemgeneratebetterpolicy.
Henry OvermanDirector,WhatWorksCentreforLocalEconomicGrowth
01
Evidence Review: Business Advice - May 2014 4
Executive Summary
Thisreportpresentsfindingsfromasystematicreviewofevaluationsofbusinessinformation,adviceandmentoringprogrammes(‘businessadvice’)aimedatimprovingbusinessgrowthandotheroutcomes.ItisthesecondofaseriesofreviewsthatwillbeproducedbytheWhatWorksCentreforLocalEconomicGrowth.
Thereviewconsideredalmost700policyevaluationsandevidencereviewsfromtheUKandotherOECDcountries.
Itfound23impactevaluationsthatmettheCentre’sminimumstandards.Thisisasmallerevidencebasethanforourfirstreview(onemploymenttraining)althoughthismaystillbelargerthantheevidencebaseformanyotherlocaleconomicgrowthpolicies.Itisaverysmallbaserelativetothatavailableforsomeotherpolicyareas(e.g.medicine,aspectsofinternationaldevelopment,educationandsocialpolicy).
02
Figure 1: Methodology
Evidence Review: Business Advice - May 2014 5
Overall, of the 23 evaluations reviewed, 17 found positive programme impacts on at least one business outcome. Four evaluations found that business advice didn’t work (had no statistically significant effects) and two evaluations found that business advice might be harmful.
ApproachThisreviewconsiderstheeffectivenessofbusinessadviceinimprovingfirmperformance(intermsofproductivity,employmentandotherperformancemeasures).Figure2providesasummaryofthenumberofevaluationsthatlookatdifferentprogrammefeatures.
Findings
What the evidence shows:
• Businessadvicehadapositiveimpactonatleastonebusinessoutcomein17outof23evaluations.
• Programmesthatusedahands-on,‘managedbrokerage’approachmayperformbetterthanthoseusingalighttouchdeliverymodelsuchasprovidingadvicethroughawebsite. Note,however,thatthisconclusionisbasedononlyonedirectcomparisonstudyandignoresthequestionofcosteffectiveness.
• Businessadviceprogrammesshowconsistentlybetterresultsforproductivityandoutputthantheydoforemployment.Resultsforsales,profitsandexportsaremixed.
Figure 2: Number of shortlisted evaluations by programme feature and context
0 5 10 15 20 25
Employment
Exports
Hybrid
National
Local
Both
Public
Private
Hybrid/Both
Not stated
Light touch
Managed brokerage
Pipeline forcing
Export promotion
Other
Comparison betweentypes of support
Sector specific
General
Prog
ram
me
obje
ctive
Nati
onal
vs
loca
lPu
blic
vs p
rivat
e De
liver
y m
odel
Sect
orsp
ecifi
city
Breakdown of studies by variable
Evidence Review: Business Advice - May 2014 6
Where the evidence is inconclusive:
• Inmostcases,programmeshadvagueormultipleobjectives,whichmakesmeasuringsuccessdifficult.
• Wefindnostrongdifferencesinresultsbetweenprogrammeswithmultipleobjectivesandprogrammeswithmorefocusedobjectives.
• Wefoundnoevidencethatwouldsuggestonelevelofdelivery–nationalorlocal–ismoreeffectivethananother.
• Itisdifficulttoreachanyconclusionsabouttheeffectivenessofpublic-ledvs.private-leddelivery.
• Overall,itisdifficulttoreachanystrongconclusionsonthelinkbetweenspecificprogrammefeaturesandbetterfirmoutcomes.
Where there is a lack of evidence:
• Thereisinsufficientevidencetoestablishtheeffectivenessofsectorspecificprogrammescomparedtomoregeneralprogrammes.
• Wefoundnohighqualityimpactevaluationsthatexplicitlylookattheoutcomesforfemale-headedorBMEbusinesses.
• Wefoundtwohigh-qualityevaluationsofprogrammesaimedatincubatingstart-ups.Bothprogrammesweretargetedatunemployedpeopleandshowmixedresultsoverall.However,thereisalackofimpactevaluationforDragons’Den-typeacceleratorprogrammesthataimtolaunchhigh-growthbusinessesandinvolvecompetitiveentry.
How to use these reviewsTheCentre’sreviewsconsideraspecifictypeofevidence,impactevaluation,whichseekstounderstandthecausaleffectofpolicyinterventions,andtoestablishtheircost-effectiveness.Inthelongerterm,theCentrewillproducearangeofevidencereviewsthatwillhelplocaldecision-makersdecidethebroadpolicyareasonwhichtospendlimitedresources.Figure3illustrateshowthereviewsrelatetotheotherworkstreamsoftheCentre.
Evidence reviews
Demonstrationprojects
You are here
Capacitybuilding
Understanding what works
More effective policy
Capacitybuilding
Capacitybuilding
Figure 3: What Works Centre work programme
Evidence Review: Business Advice - May 2014 7
Supporting and complementing local knowledgeTheevidencereviewsetsoutanumberof‘BestBets’–approachestobusinessadvicethathaveperformedmoststronglybasedonthebestavailableimpactevaluations.
However,the‘BestBets’donotaddressthespecificsof‘whatworkswhere’or‘whatwillworkforaparticularfirm’.Detailedlocalknowledgeandcontextremaincrucial.
‘BestBets’alsoraiseanoteofcautionforpolicymakersiftheydecidetointroduceaprogrammewhichhasnotworkedwellelsewhere.
Specific recommendationsThe23evaluationsofferarichsourceofmaterialforpolicymakerstouseindesigningspecificbusinessadvicepolicies.Inparticular,theevaluationswillbeofusetopolicymakersattwokeystagesinthepolicydesignprocess:determiningthepolicyoptions,andthenselectingthepreferredoption.
• Ifwewanttoknowwhatworksintheareaofbusinessadviceweneedtoimproveprogrammedesignandevaluation.Whendesigningaprogramme,localpolicymakersshouldidentify one or two clear programme objectives,andthenidentifyoutcomemeasuresthatarebothclearlyrelatedtotheprogrammeobjectives,andfeasibletomeasure.
• Businessadviceprogrammestendtobemoresuccessfulinincreasingfirms’productivitythaninincreasingtheiremployment.
• Onecomparativestudysuggeststhatsmaller, better-resourced programmes are more likely to achieve successandthanlarger‘hands-off’policies.Butitisunclearwhichoftheseapproachesismorecost-effective.
Filling the Evidence GapsThisreviewhasnotfoundanswerstosomeofthequestionsthatwillbeforemostinpolicymakers’minds.
Thesegapshighlighttheneedforimprovedevaluationandgreaterexperimentation,specificallyexperimentsthatfocuson:
• identifyinghowdifferentelementsofbusinessadviceprogrammedesigncontributetobetterorworseoutcomes;and,
• thevalueformoneyofdifferentapproaches.Only5ofthe23shortlistedstudiesincludedcost-benefitanalysis,andnotalloftheseusedmeasuresthatarecomparableacrossstudies.Thereisaclearneedformore,consistentanalysisofcost-effectivenessinbusinessadviceimpactevaluations.
Thisrequiresevaluationtobeembeddedinpolicydesign,andthinkingdifferentlyaboutthepolicycycleasawhole.
Evidence Review: Business Advice - May 2014 8
Introduction
Theprovisionofpubliclyfundedadvice,mentoringandsupporttobusinesses,andparticularlytoentrepreneursandsmallbusinesses,isubiquitousinOECDcountries.1Take-upofpublicsectorsupport,however,isgenerallyassumedtobelow.IntheUK,forexample,itisestimatedthatabout40%ofbusinesseshavereceivedformalexternaladvice,butonly20%receivedadvicefromapublicsectorprovider.2
Thisreviewfocusesonprogrammesthatarefundedbygovernmentandthatprovideinformation,structuredadviceorlongertermmentoringtofirms(hence‘businessadvice’).3Suchinterventionstypicallyaimtoincreaseratesoffirmcreation,toimprovebusinesssurvival,andtopromotebusinessproductivityandemploymentgrowth.
Whyarepolicymakerssointerestedinthiskindofbusinesssupport?Somegovernmentsmaywanttopromote‘enterpriseculture’onitsownmerits.Othersstartfromthefactthatsmallandmedium-sizeenterprises(SMEs)formthevastmajorityofbusinessesintheUKandotherdevelopedeconomies;4andthatsmall,newfirmsaccountforthemajorityofjobcreation.5
Intheory,publiclysupportedadvisoryservicescanbejustifiedontwogrounds–informationfailuresandwidereconomicimpacts.Inthefirstcase,wheninformationishardtoaccessorofvariablequality,firmsmayunder-investinservicesthatcouldsupporttheirbusinesses.Economistsrefertothesechallengesas‘informationasymmetries’.6Suchmarketfailuresmayresultwhenbusinessownersare:
• Unawareofinformationandadvicethatwouldbevaluabletothem;
• Unclearabouthowtoaccesssuchresources;
1 OECD(2002)OECDSmallandMediumEnterpriseOutlook,OECD,Paris2 www.gov.uk/government/uploads/system/uploads/attachment_data/file/32250/11-1288-research-barriers-to-use- of-business-support.pdf3 Asdistinctfromprogrammestoa)improveaccesstofinance;b)improvethegeneralbusinessenvironment andc)reduceburdens,regulationsandcosts.4 Curran(2000),Bennett(2008).5 SeeHaltiwangeretal(2010)fortheUS,andBravo-Bioscaetal(2011)fortheUK.6 TwooftheclassicstudiesoninformationasymmetriesareAkerlof(1970),Stiglitz(1979)andGrossmanandStiglitz (1980).Forageneraloverviewofinformationeconomics,seeStiglitz(2002).
03
Evidence Review: Business Advice - May 2014 9
• Concernedaboutthequalityofadviceoffered;
• Facingfinancialortimeconstraintsonaccessingadvicewhichexceedtheperceivedbenefits;and/or
• Worriedthatconfidentialinformationcouldendupinthehandsofcompetitors.
Inprinciple,publicpolicycansolvetheseproblemsandhelpbusinessestogrowbyprovidingimpartial,freeorsubsidisedadviceandmentoring.
BusinesssupportinterventionsmayalsobejustifiedbecauseSMEsareimportantforeconomicdevelopment.Ifinformation,mentoringandadvicecanhelpindividualfirmstogrow,thiscouldhavespillovereffects–or‘externalities’–fortheeconomyasawhole.Theseincludethecreationofmorejobs,moreinnovation,orlowerpricestoconsumers.
Whilethereisatheoreticalcaseforgovernmentintervention,inpractice,itisnotstraightforwardforgovernmenttoprovideeffectivebusinessadvice,andtherearedangersofpolicyfailureindoingso.Buttheliteraturealsohighlightsrealmarketfailuresformanystart-upsandearlystagefirms;thatsomeprogrammeshaveafarhigherimpactthanothers;andthattherearesignificantdifferencesinusertake-up(forexample,betweenmaleandfemale-headedbusinesses).Thisimpliesthatwell-designedinterventionscouldhavepositiveimpacts.
Consistentwithallofthis,inourreview,wefindanumberofeffectiveprogrammesbutwealsofindexamplesofpolicyfailure.Thissuggeststhatabetterunderstandingofwhatworkswouldaddsignificantvaluetothepolicymakingprocessinthisarea.
Evidence Review: Business Advice - May 2014 10
Impact evaluation
Governmentsaroundtheworldincreasinglyhavestrongsystemstomonitorpolicyinputs(suchasspendingonabusinesssupportprogramme)andoutputs(suchasthenumberoffirmswhohavegonethroughtheprogramme).However,theyarelesssuccessfulatidentifyingpolicyoutcomes(suchastheeffectofabusinessadviceprogrammeonfirmemployment).Inparticular,manygovernment-sponsoredevaluationsthatlookatoutcomesdonotusecrediblestrategiestoassessthecausal impactofpolicyinterventions.
Evaluationofcausalimpactsfocusonmeasuringthedifferencethatcanbeexpectedbetweentheoutcomeforfirms‘treated’inaprogramme,andtheaverageoutcometheywouldhaveexperiencedwithoutit.Pinningdowncausalityisacruciallyimportantpartofimpactevaluation.Estimates of the benefits of a programme are of limited use to policymakers unless those benefits can be attributed, with a reasonable degree of certainty, to that programme.
Thecredibilitywithwhichevaluationsestablishcausalityisthecriteriononwhichthisreviewassessestheliterature.
Using CounterfactualsEstablishing causality requires the construction of a valid counterfactual–i.e.whatwouldhavehappenedtoprogrammeparticipantshadtheynotbeentreatedundertheprogramme.Thatoutcomeisfundamentallyunobservable,soresearchersspendagreatdealoftimetryingtorebuildit.Thewayinwhichthiscounterfactualis(re)constructedisthekeyelementofimpactevaluationdesign.
A standard approach is to create a counterfactual group of similar individuals not participating in the programme being evaluated. Changesinoutcomescanthenbecomparedbetweenthe‘treatmentgroup’(thoseaffectedbythepolicy)andthe‘controlgroup’(similarindividualsnotexposedtothepolicy).
A key issue in creating the counterfactual group is dealing with the ‘selection into treatment’ problem.Selectionintotreatmentoccurswhenparticipantsintheprogrammedifferfromthosewhodonotparticipateintheprogramme.
04
Evidence Review: Business Advice - May 2014 11
Anexampleofthisprobleminbusinessadviceprogrammeswouldbewhenmoreambitiousfirmsapplyforadvice.Ifthishappens,estimatesofpolicyimpactmaybebiasedupwardsbecauseweincorrectlyattributebetterfirmoutcomestothepolicy,ratherthantothefactthatthemoreambitiousparticipantswouldhavedonebetterevenwithouttheprogramme.
Selectionproblemsmayalsoleadtodownwardbias.Forexample,firmsthatapplyforadvicemightbeexperiencingproblemsandsuchfirmsmaybelesslikelytogroworsucceedindependentofanyadvicetheyreceive.Thesefactorsareoftenunobservabletoresearchers.
So the challenge for good programme evaluation is to deal with these issues, and to demonstrate that the control group is plausible.Iftheconstructionofplausiblecounterfactualsiscentraltogoodpolicyevaluation,thenthecrucialquestionbecomes:how do we design counterfactuals? Box1providessomeexamples.
Box 1: Impact evaluation techniques
Onewaytoidentifycausalimpactsofaprogrammeistorandomlyassignparticipantstotreatmentandcontrolgroups.Forresearchers,suchRandomisedControlTrials(RCTs)areoftenconsideredthe‘goldstandard’ofevaluation.Properlyimplemented,randomisationensuresthattreatmentandcontrolgroupsarecomparablebothintermsofobservedandunobservedattributes,thusidentifyingthecausalimpactofpolicy.However,implementationofthese‘realworld’experimentsischallengingandcanbeproblematic.RCTsmaynotalwaysbefeasibleforlocaleconomicgrowthpolicies–forexample,policymakersmaybeunwillingtorandomise.7Andsmall-scaletrialsmayhavelimitedwiderapplicability.
Whererandomisedcontroltrialsarenotanoption,‘quasi-experimental’approachesofrandomisationcanhelp.Thesestrategiescandealwithselectiononunobservables,by(say)exploitinginstitutionalrulesandprocessesthatresultinsomefirmsquasi-randomlyreceivingtreatment.
Evenusingthesestrategies,though,thetreatmentandcontrolgroupsmaynotbefullycomparableintermsofobservables.StatisticaltechniquessuchasOrdinaryLeastSquares(OLS)andmatchingcanbeusedtoaddressthisproblem.
Notethathigherqualityimpactevaluationfirstusesidentificationstrategiestoconstructacontrolgroupanddealwithselectiononunobservables.Thenittriestocontrolforremainingdifferencesinobservablecharacteristics.Itisthecombinationthatisparticularlypowerful:OLSormatchingaloneraiseconcernsabouttheextenttowhichunobservablecharacteristicsdeterminebothtreatmentandoutcomesandthusbiastheevaluation.
7 Gibbons,NathanandOverman(2014).
Evidence Review: Business Advice - May 2014 12
Evidence included in the review We include any evaluation that compares outcomes for firms receiving treatment (the treated group) after an intervention with outcomes in the treated group before the intervention, relative to a comparison group used to provide a counterfactual of what would have happened to these outcomes in the absence of treatment.
Thismeanswelookatevaluationsthatdoareasonablejobofestimatingtheimpactoftreatmentusingeitherrandomisedcontroltrials,quasi-randomvariationorstatisticaltechniques(suchasOLSandmatching)thathelpmaketreatmentandcontrolgroupscomparable.Weviewtheseevaluationsasprovidingcredibleimpactevaluationinthesensethattheyidentifyeffectswhichcanbeattributed,withareasonabledegreeofcertainty,totheimplementationoftheprogrammeinquestion.AfulllistofshortlistedstudiesisgiveninAppendixA.
Evidence excluded from the reviewWeexcludeevaluationsthatprovideasimplebeforeandaftercomparisononlyforthosereceivingthetreatmentbecausewecannotreasonablyassumethatchangesforthetreatedgroupcanbeattributedtotheeffectoftheprogramme.
Wealsoexcludecasestudiesorevaluationsthatfocusonprocess(howthepolicyisimplemented)ratherthanimpact(whatwastheeffectofthepolicy).Suchstudieshavearoletoplayinhelpingformulatebetterpolicybuttheyarenotthefocusofourevidencereviews.
Evidence Review: Business Advice - May 2014 13
Methodology
Toidentifyrobustevaluationevidenceonthecausalimpactofbusinessadviceprogrammes,weconductedasystematicreviewoftheevidencefromtheUKandacrosstheworld.Ourreviewsfollowedafive-stageprocess:scope,search,sift,scoreandsynthesise.
Stage 1: Scope of Review WorkingwithourUserPanelandamemberofourAcademicPanel,weagreedthereviewquestion,keytermsandinclusioncriteria.Wealsousedexistingliteraturereviewsandmeta-analysestoinformourthinking.
05
Figure 1: Methodology
Evidence Review: Business Advice - May 2014 14
Stage 2: Searching for EvaluationsWesearchedforevaluationevidenceacrossawiderangeofsources,frompeer-reviewedacademicresearch,togovernmentevaluationsandthinktankreports.Specifically,welookedatacademicdatabases(suchasEconLit,WebofScienceandGoogleScholar),specialistresearchinstitutes(suchasCEPRandIZA),UKcentralandlocalgovernmentdepartments,andworkdonebythinktanks(suchastheOECD,ILO,IPPRandPolicyExchange).Wealsoissuedacallforevidenceviaourmailinglistandsocialmedia.Thissearchfoundcloseto700books,articlesandreports.AppendixBprovidesafulllistofsourcesandsearchterms.
Stage 3: Sifting EvaluationsWescreenedourlong-listonrelevance,geography,languageandmethods,keepingimpactevaluationsfromtheUKandotherOECDcountries,withnotimerestrictionsonwhentheevaluationwasdone.WefocussedonEnglish-languagestudies,butwouldconsiderkeyevidenceifitwasinotherlanguages.Wethenscreenedtheremainingevaluationsontherobustnessoftheirresearchmethods,keepingonlythemorerobustimpactevaluations.WeusedtheMarylandScientificMethodsScale(SMS)todothis.8TheSMSisafive-pointscalerangingfrom1,forevaluationsbasedonsimplecrosssectionalcorrelations,to5forrandomisedcontroltrials(seeBox2).Weshortlistedallthoseimpactevaluationsthatcouldpotentiallyscore3oraboveontheSMS.9Inthiscasewefoundnoevaluationsscoring4.Forexamplesofbusinessadviceevaluationsthatscore3and5ontheSMSscale,seeAppendixD.
Stage 4: Scoring EvaluationsWeconductedafullappraisalofeachevaluationontheshortlist,collectingkeyresultsandusingtheSMStogiveafinalscoreforevaluationsthatreflectedboththequalityofmethodschosenandqualityofimplementation(whichcanbelowerthanclaimedbysomeauthors).Scoringandshortlistingdecisionswerecross-checkedwiththeacademicpanelmemberandthecoreteamatLSE.Thefinallistofincludedstudiesandtheirreferencenumbers(usedintherestofthisreport)canbefoundinAppendixA.
Stage 5: Synthesising EvaluationsWedrewtogetherourfindings,combiningmaterialfromourevaluationsandtheexistingliterature.
8 Sherman,Gottfredson,MacKenzie,Eck,Reuter,andBushway(1998).9 Shermanetal.(1998)alsosuggestthatlevel3istheminimumlevelrequiredforareasonableaccuracyofresults.
Evidence Review: Business Advice - May 2014 15
Box 2: The Scientific Maryland Scale
Level 1:Correlation of outcomes with presence or intensity of treatment, cross-sectional comparisons of treated groups with untreated groups, or other cross-sectional methods in which there is no attempt to establish a counterfactual.Nouseofcontrolvariablesinstatisticalanalysistoadjustfordifferencesbetweentreatedanduntreatedgroups.
Level 2:Comparison of outcomes in treated group after an intervention, with outcomes in the treated group before the intervention (‘before and after’ study). Nocomparisongroupusedtoprovideacounterfactual,oracomparatorgroupisusedbutthisisnotchosentobesimilartothetreatmentgroup,nordemonstratedtobesimilar(e.g.nationalaveragesusedascomparisonforpolicyinterventioninaspecificarea).No,orinappropriate,controlvariablesusedinstatisticalanalysistoadjustfordifferencesbetweentreatedanduntreatedgroups.
Level 3:Comparison of outcomes in treated group after an intervention, with outcomes in the treated group before the intervention, and a comparison group used to provide a counterfactual (e.g. difference in difference). Somejustificationgiventochoiceofcomparatorgroupthatispotentiallysimilartothetreatmentgroup.Evidencepresentedoncomparabilityoftreatmentandcontrolgroupsbutthesegroupsarepoorlybalancedonpre-treatmentcharacteristics.Controlvariablesmaybeusedtoadjustfordifferencebetweentreatedanduntreatedgroups,buttherearelikelytobeimportantuncontrolleddifferencesremaining.
Level 4:Comparison of outcomes in treated group after an intervention, with outcomes in the treated group before the intervention, and a comparison group used to provide a counterfactual (i.e. difference in difference).Careful and credible justification provided for choice of a comparator group that is closely matched to the treatment group.Treatmentandcontrolgroupsarebalancedonpre-treatmentcharacteristicsandextensiveevidencepresentedonthiscomparability,withonlyminororirrelevantdifferencesremaining.Controlvariables(e.g.OLSormatching)orotherstatisticaltechniques(e.g.instrumentalvariables,IV)maybeusedtoadjustforpotentialdifferencesbetweentreatedanduntreatedgroups.Problemsofattritionfromsampleandimplicationsdiscussedbutnotnecessarilycorrected.
Level 5:Reserved for research designs that involve randomisation into treatment and control groups.Randomisedcontroltrialsprovidethedefinitiveexample,althoughother‘naturalexperiment’researchdesignsthatexploitplausiblyrandomvariationintreatmentmayfallinthiscategory.Extensiveevidenceprovidedoncomparabilityoftreatmentandcontrolgroups,showingnosignificantdifferencesintermsoflevelsortrends.Controlvariablesmaybeusedtoadjustfortreatmentandcontrolgroupdifferences,butthisadjustmentshouldnothavealargeimpactonthemainresults.Attentionpaidtoproblemsofselectiveattritionfromrandomlyassignedgroups,whichisshowntobeofnegligibleimportance.
Evidence Review: Business Advice - May 2014 16
Definition
Weincludedinourdefinitionofbusinessadviceandmentoringgovernmentfundedprogrammesthatfocusedon:
• Supportingindividualstosetuptheirownbusinesses
• Supportingexistingbusinessestogrow,wheregrowthmaybebroadlydefinedtoinclude:
• Improvedproductivity(intermsofsales/turnoverperemployee10orvalueaddedperemployee11)
• Growthinemployment
• Growthinturnover
• Growthinprofits
• Expansionintonewmarkets(particularlyoverseas).
Weexcluded:
• Financialsupportandaccesstofinanceschemes–wewilladdressthistopicindependentlyinafuturereview
• Incubatorprogrammes–duetoalackofevidenceontheirimpact.
10 Studies165,166,167and284.11 Studies169and170.
06
Evidence Review: Business Advice - May 2014 17
Findings
Thissectionsetsoutthereview’sfindings.Webeginwithadiscussionoftheevidencebase,andthenexploretheoverallpatternofpositiveandnegativeresults.Afterthisweconsiderspecificprogrammefeaturesinmoredetail.
Quantity and quality of the evidence baseFromaninitiallonglistof690studies,23evaluationsmetourminimumstandards.12Thisisasmallerevidencebasethanforourfirstreview(onemploymenttraining),althoughthismaystillbelargerthantheevidencebaseformanyotherlocaleconomicgrowthpolicies.Table1showsthedistributionofthestudiesrankedaccordingtotheSMS.
Table 1: Ranking studies by quality of implementation
SMS Score Number by implementation5 4
4 0
3 19
Total 23
Therearefourrandomisedcontroltrials,allofwhichscored5ontheSMS.13Wefoundnostudiesthatusedcrediblequasi-randomsourcesofvariation(i.e.scored4ontheSMS)toidentifypolicyimpacts.
19studiesscored3ontheSMS,andusevariationsonOLSormatchingtechniques.Thetechniquesappliedinthesestudiesmeanthatwecanbereasonablyconfidentthattheevaluationhasdoneagoodjobofcontrollingforallobservablecharacteristicsoffirmsorindividuals(forexample:firmage;size;sector),whichmightexplaindifferencesinfirmoutcomes.However,forthesestudies,itislikelythatunobservablecharacteristicssuchasentrepreneurialtalentorfirms’desiretogrowmaystillbeaffectingtheresults.Thisraisesconcernsthattheevaluationincorrectlyattributesbeneficialoutcomes
12 Manyofthestudiesnotincludedprovidedcasestudiesorprocessevaluationswhichareoftenvaluable,butare notthefocusofourreview.Seemethodologysectionforfurtherdiscussion.13 Studies162,163,282,287.
07
Evidence Review: Business Advice - May 2014 18
totheprogrammeratherthantothesefirmcharacteristics.WecanonlybefullyconfidentthatselectiononunobservableshasbeeneradicatedwithanRCTmethodology,whereparticipantsarerandomlyassignedtotreatmentorcontrolgroups.
Foreightofthese19studieswehaveconcernsoverthebaselineyearused(eitherpost-treatmentorunknown)soweneedtobecarefulininterpretingtheresults.Ifpositiveprogrammeeffectsarefeltimmediately,thenthesestudiesmayunderestimatetheimpact.Conversely,ifparticipationintheprogrammeinitiallyworsensperformance(e.g.becauseanentrepreneurisspendingtimeimplementingadviceorre-organisingthefirmratherthanfocusingonproduction)thenthesestudiesmayoverestimateimpact.
Type and focus of business adviceBroadlyspeakingtherearetwoseparatetypesofinterventionthatfallunderthebannerof‘businessadviceandmentoring’,thoseinwhichthefocusisonsupportingindividualstosetuptheirownbusinessandthoseinwhichthefocusisonsupportingexistingbusinessestogrow(where‘growth’maybedefinedinvariouswaysincludinggrowthinturnover,employment,profits,expansionintonewmarkets,etc).
Fiveofthe23evaluationslookedatsupporttoindividualstoestablishnewbusinesses,typicallymicro-enterpriseswith1-10employees.14Ofthose,fourwerefromtheUSA(ofwhichtwolookedatthesameprogramme–ProjectGATE-butfoundcontrastingresults).15OnewasfromGermany.16Inallcases,thefocusisonsupportingunemployedindividualstosetupeitherasown-accountworkersorasmicroenterprises,andonimprovingthesuccessrateofthosestart-ups.Theseprogrammesgenerallyinvolvetraininginbasicbusinessskillsandconceptsandsupportinsuchendeavoursaswritingabusinessplan.
Oftheremaining18evaluations:
• Eightcoverfourprogrammes,orgroupsofprogrammes,whichprovidegeneralbusinessadviceandaimtosupportfirmgrowth.ForBusinessLinkwehaveincludedboththeoriginalevaluationcommissionedbyBIS(study284)andthreeacademicfollowupstudiesthatexploreaspectsoftheprogrammeinmoredetail;17
• ThreecovervariousprogrammesrunbyUKTrade&Industryfocusedonpromotingexportsorhelpingfirmstoaccessforeignmarkets;18
• Threelookedatprogrammesprovidingsubsidiestoallowfirmstoaccessmarket-providedbusinesssupportservices;19
• TwolookedatthePLATOprogramme(Belgium),whichfocusesonpeerlearningandnetworksasaroutetobusinessgrowthandimprovement;20
• OnelookedatNewZealand’sTrade&Enterpriseprogrammetosupportfirmswithhighgrowthpotential;21
14 Studies163,217,276,282,287.15 Studies163,276,282,287.16 Study217.17 Studies166,167,177,284and285coverBusinessLink(UK);162coversCompetitiveProductivityorIPCC (Mexico);168coversthePueblaInstituteforCompetitiveProductivity;and172coversvariousservices(Germany).18 Studies165,169,183,184whichcoverOverseasMarketIntroductionService(OMIS),UK;PassporttoExport,UK; UKTIBusinessSupportservices(various,includingPassport),UK.19 Studies162,170,182.20 Studies174,286.21 Study269.
Evidence Review: Business Advice - May 2014 19
• Onedidnotlookataspecificprogramme,butanalysedtheimpactofvariousdifferentformsofexternaladvice(includingpubliclyprovidedbusinesssupport)onfirmperformance.22
Overall effects on business growth
Business support and advice had a positive effect on at least one business outcome in 17 out of 23 evaluations.
Ofthe23evaluationsreviewed,17foundpositiveprogrammeimpactsonatleastonefirmoutcome.Fourevaluationsfoundthatbusinessadvicedidn’twork(i.e.theyfoundnostatisticallysignificantevidenceofpositiveimpactsonfirmoutcomes)andtwoevaluationsfoundthatbusinessadvicemightbeharmful(i.e.statisticallysignificantnegativeimpactsonfirmsoutcomeswithnooffsettingpositiveeffects).23Inoneofthesetwocases,theauthorsattributethenegativeresultstotwofactors:thepossibilitythatbusinessadviceenablesweakfounderstoformmarginalbusinesses;andtheover-provisionofnon-intensivesupportinthisparticularprogramme.24
Table 2: Summary of effects of business advice & mentoring programmes
Finding No of studies Evaluation reference numbers
Supportmayhelp(positivecoefficientsonatleastonefirmoutcome)
17 162,163,165,166,167,168,169,174,177,183,269,182,184,195,284,285,
286,287
Supportdoesn’twork(nostatisticallysignificantfindings)
4 170,195,276,282
Supportmaybeharmful(nopositive,somenegativecoefficients)
2 172,217
Total 23
Table3summariseskeycharacteristicsoftheprogrammesintheshortlistedevaluations.Itoutlineswhichcharacteristicswereaddressedbywhichevaluations,andalsowheredirectcomparisonsweremade.
22 Study195.23 Study165foundanegativeeffectonexports,butapositiveeffectonthreeotherfirmsoutcomes.Wediscussthis furtherbelow.24 Study172.
Evidence Review: Business Advice - May 2014 20
Table 3: Overview of programme features
No. of studies
Findings (on balance of evidence)
Evaluation Reference Numbers
Overall
Programme objectiveEmployment 2 Mixed
Mostprogrammestargetmultipleorvagueobjectives.Onlythreeevaluationsfoundaclearlinkfromprogrammeobjectivestoimprovedfirmoutcomesforthatobjective.
-Helps(1) 287
-Harmful(1) 217
Exports 3 Mixed
-Helps(2) 169,184
-Harmful(1) 165
Hybrid 18 Mixed
-Helps(13) 162,163,166,167,168,174,177,183,183,269,284,285,
286
-Zero(4) 170,195,276,282
-Harmful(1) 172
National vs local deliveryNational 8 Mixed
Thereisnoevidencethatonelevelofdeliveryismoreeffectivethanothers
-Helps(5) 169,182,183,184,269
-Zero(2) 170,195
-Harmful(1) 165
Local 6 Mixed
-Helps(4) 162,168,174,286
-Zero(1) 276
-Harms(1) 172
Both 9 Mixed
-Helps(7) 163,166,167,177,284,285,
287
-Zero(1) 282
-Harmful(1) 217
Evidence Review: Business Advice - May 2014 21
No. of studies
Findings (on balance of evidence)
Evaluation Reference Numbers
Overall
Public vs private led deliveryPublic 3 Mixed
Thereisnoevidencethatonedeliverymodelismoreeffectivethanothers.
-Helps(2) 183,184
-Harmful(1) 165
Private 4 Mixed
-Helps(3) 168,182,286
-Zero(1) 170
Hybrid/Both 14 Mixed
-Helps(10) 162,163,166,167,174,177,269,284,285,
287
-Zero(2) 282,276
-Harmful(2) 172,217
Notstated 2 169,195
Delivery model/ techniqueLighttouch 1 Helps 168
Programmesthatusedahands-on,‘managedbrokerage’approachmayperformbetterthanthoseusinga‘lighttouch’approach(althoughthisconclusionisbasedononecomparisonstudy).
Managedbrokerage 4 Helps 162,174,269,286
Pipelineforcing 4 Mixed
-Helps(2) 163,287
-Zero(1) 282
-Harmful(1) 217
Exportpromotion 4 Mixed
-Helps(3) 169,183,184
-Harmful(1) 165
Other 9 Mixed
-Helps(5) 167,177,182,284,285
-Zero(3) 170,195,276
-Harmful(1) 172
Comparisonbetweentypesofsupport
1 Intensivesupportmoreeffectivethanlighttouch
166
Evidence Review: Business Advice - May 2014 22
No. of studies
Findings (on balance of evidence)
Evaluation Reference Numbers
Overall
Sector specificitySectorspecific 2 Mixed
Thereisinsufficientevidencetoestablishtherelativeeffectivenessofsectorspecificprogrammescomparedwithmoregeneralprogrammes.
-Helps(1) 182
-Harms(1) 172
General 21 162,163,165,166,167,168,169,170,174,177,183,184,195,217,269,276,282,284,285,286,287
Evidence Review: Business Advice - May 2014 23
Detailed findings
Thissectionofthereportlooksatwhetherthereisanyevidenceofalinkbetweenspecificprogrammefeaturesandoutcomes.Forexample,weconsiderwhetherpublic-ledorprivate-leddeliverymodelsareassociatedwithbetteroutcomes.
Thisisnotstraightforwardbecausepossiblerelationshipscouldalsobeexplainedbyanumberofother‘confoundingfactors’thatmaybeinplay.Inaddition,thereissignificantvarietyinthetypesofsupportthatareexploredinthedifferentevaluationswehaveconsidered.Forexample,programmesmaybebroadlycategorisedbythosethatofferassistancetoindividualswhowishtostartbusinessesandthosethatassistalreadyestablishedfirms.Aswellasofferingaverydifferenttypeofsupport,themeasurablegoalsofsuchprogrammesarealsocontrasting,makingbroadcomparisonparticularlychallenging.
Inouremploymenttrainingreview,wewereabletoaddressthisproblembylookingatstudiesthatmadeexplicitcomparisonsbetweenprogrammes.Forbusinesssupport,suchcomparisonsare,unfortunately,notreadilyavailableandsowecanonlylookforanygeneralpatternorcorrelationbetweenthefeatureandtheoutcomewe’reinterestedin(forexample,deliverytypeandemployment).
Programme objectives
Most programmes target multiple or vague objectives. Only three evaluations found a clear link from programme objectives to improved firm outcomes for that objective.
Thefocusoftheprogrammeswelookedatvariessignificantly,asreflectedinanumberofmeasurableobjectives.Whileasmallnumberofprogrammeshaveexplicit,singularobjectivesthatareestablishedfromtheoutset,suchasincreasingtheexportsofsmallfirms,othersformpartofwiderpackagesofmarketinterventionswithmorecomplexsetsofgoals.
Thesecontrastingobjectivesreflectdifferentialpreferencesthatpolicymakershaveintermsofprogrammedesign,andalsotheultimatebeneficiariesofthesupport.Ashighlightedpreviously,anumberoftheprogrammescoveredinourreviewarefocusedongettingunemployedindividualsintoworkbysupportingthemtoestablishtheirownbusinessesandbecomeself-employed.Theseareoftenpartofwideractivelabourmarketpoliciesthatultimatelyaimtoincreaseemploymentand
08
Evidence Review: Business Advice - May 2014 24
durationoftimeinwork.Programmesthatareaimedtowardsexistingsmallbusinessestakethisastepfurther,sharingacommonunderlyinggoal,encouragingsmallbusinessesthatarealreadyestablishedtogrowfurther.Thismightbeachievedby,forexample,increasingexportsorimprovingproductivityinordertoboostsalesandprofits.Thepictureisfurthercomplicatedintheevaluationstage,withsomeprogrammesbeingdirectlyevaluatedagainststatedobjectives,whileinothercasesresearcherslookatabasketofwideroutcomes.
Weareinterestedinwhetherprogrammesmeettheirstatedobjectivesandwhethermorecomplexprogrammesaremoreorlesslikelytobesuccessful.Forthepurposesofthereview,weclassifyprogrammesintotwogroups:afirstgroupwhereobjectivesareclearlystatedandasecondgroupwhereobjectivesaremultipleornotclearlystated.Programmesinthefirstgroupallhavesingleobjectivesandaimtoeitherincreaseemploymentorexports.Programmesinthesecondgrouptendtohavemultipleobjectives–forexample,BusinessLink.
BusinessLinkprovedespeciallyhardtoclassify.25Eachevaluationemphasisesthemainobjectivesslightlydifferently,resultinginalackofclarityabouttheoverarchingcoreaims.SeveralreferenceHartandRoper(2003)who,insummary,statethatBusinessLinkwasdesignedtoimprovetheperformanceofthesmallbusinesssector,specificallyexistingbusinessesofbetween20and100employees.26However,giventhattheprogrammewasestablishedasanetworkoflocally-runoffices,theexactscopevariedasaresultoflocalpreferences.Indeed,HartandRoper(ibid)suggestthat,insomecases,start-upswereassistedaswellasexistingsmallfirms,somethingthatispickedupbytwootherevaluationsthatciteraisingself-employmentratesandraisingentrepreneurshipindeprivedareasasadditionalobjectives.27Inordertoreflectthesevariations,wehavecategorisedallBusinessLinkevaluationsunderthe‘hybrid’bracket.
Table 4: Programme Objectives
Objective No of studies Study reference numbers
Increasedemployment 2 217,287
Increasedexports 3 165,169,184
Hybrid(multipleobjectives) 18 162,163,166,167,170,172,174,177,182,183,195,269,276,282,
285,286
Onlytwoevaluationsconsiderprogrammesthatarefocusedsolelyonincreasingemployment(butnotethatoneofthesescored5ontheSMS).Bothfocusedspecificallyonself-employment,anddemonstratedmixedresults.Self-employmenttrainingandcoachingprovidedtorecipientsofBridgingSupport(Germany)wasgenerallylesssuccessful.28Incontrast,theWashingtonandMassachusettsSelf-EmploymentDemonstrators(US)wereshowntoincreasethelikelihoodofbeinginemploymentby14%,withtransitionoccurring5.9monthsmorequickly.Forself-employment,thelikelihoodincreasedby5%withtransition2.4monthsquicker.29
Thethreeevaluationsthatassessedtheeffectivenessofexport-onlyfocusedprogrammesallfoundatleastonesignificantpositiveeffectforfirmoutcomes.Oneofthesedidnotlookdirectlyatfirmexports
25 Studies166,167,177,284and285.26 Study167.27 Studies177and284.28 Study217.29 Study287.
Evidence Review: Business Advice - May 2014 25
butfoundpositiveeffectsforindirectmeasuresofsuccess.30Ofthetwothatdidlookatexports,whilstAftercare(UK)wasfoundtonoticeablyincreasethelevelandgrowthofexportintensity,31theOverseasMarketIntroductionService(UK)ledtoamarginallynegativeimpactonexportscomparedtofirmsinthecontrolgroup(usinganSMS3methodology). 32Thisisdespitepositiveeffectsonotherindicatorssuchasturnovergrowth(£611,000higher),employment(sevenadditionaljobsperfirmonaverage)andfirmsurvivalprobability.
Theremainingprogrammeshaveobjectivesthatareeithernotclearlydefinedorinvolveseveraldifferentoutcomemeasures.TheevaluationofBusinessLink,forexample,showsmixedresults:inoneexample,BusinessLinkwasfoundtogenerate4.4%employmentgrowth,butnosignificanteffectsonsales;33anotherfoundsimilarresults,witha2%upliftinemploymentgrowthbutnoimpactonsales;34whilstathirdfoundonlytentativeevidencethatBusinessLinkincreasedproductivitygrowth.35TheGATEprogramme(US),whichtargetedasuiteofentrepreneurshiptrainingtothosereceivingunemploymentbenefits,wasprincipallyaimedatincreasingself-employment,butwasalsodesignedtoraiseindividualearningsanddevelophigh-growthfirms.36However,increasesinthenumberofhoursspentinself-employmentwereoffsetbydecreasesinformalemploymenthours,thusresultinginaneutralneteffectonnumberofhoursinemployment.Householdandbusinessearningswerenotsignificantlydifferentfortreatmentandcontrolgroups.
Wehavenotfoundanyevaluationsthatexplicitlycompareprogrammeswithdifferentoutcomes.Overall,wefindnostrongdifferencesbetweenprogrammeswithmultipleobjectivesandprogrammeswithmorefocusedobjectives.Thelackofaclearlinkbetweenprogrammeobjectivesandspecificmeasuresofsuccessinthemajorityofcasesmakesitveryhardtoassesstheoveralleffectivenessofbusinessadviceprogrammes.Ourfindingshereechoexistingevidencereviewsonbusinessadviceandmentoring.37
National vs. local delivery
We found no evidence that would suggest one level of delivery is more effective than others.
The23evaluationsinvolveddeliverymodelsforbusinesssupportatseveraldifferentscales.Someprogrammesweredeliveredthroughnational-levelorganisations,bothgovernmentdepartments38andarms-lengthstateagencies,39whilstotherswerewhollydevolvedtotheregionalorlocallevel.40However,themajorityofprogrammescoveredinourreviewweredeliveredbyaformofpartnership,ofteninvolvingoverarchingnationalcontrolwithelementsofdevolvedpoweranddelivery.
30 Study184.31 Study169.32 Study165.33 Study28534 Study17735 Study16736 Studies163,282.37 ForexampleCurran(2000),Bennett(2008).38 UKTradeandIndustryprogrammesinstudies165,169and183.39 Studies166,167,177,284and285assessBusinessLink,whichwasanarms-lengthagencyadministeredand managedbyBISbeforelaterbeingdevolvedtotheregionallevel.40 Initslatteryears,controloverBusinessLinkwasdevolvedtoRegionalDevelopmentAgencies,with individual‘Links’incorporatedindividuallyasentrepreneurialorganisations(seestudy166formoredetails).The PLATOnetworkwasestablishedandrunbyanindependentlocalassociationofbusinesses,withfinancialsupport fromtheregionalgovernment.
Evidence Review: Business Advice - May 2014 26
Table 5: National or Local Delivery
Delivery model No of studies Study reference number
National 8 165,169,170,182,183,184,195,269
Local 6 162,168,172,174,276,286
Hybrid 9 163,166,167,177,217,282,284,285,287
Noneoftheevaluationslooksdirectlyatthequestionofwhetherprogrammesaremoresuccessfulwhentheyarelocally,regionallyorcentrallymanaged.Whenweclassifiedevaluationsaccordingtothelevelatwhichtheprogrammeisdelivered,wefoundnoevidencethatwouldsuggestonelevelofdeliveryisinherentlymoreeffectivethanothers.Thisreflectsthefindingsofotherevidencereviews.41Bennett(2008)notesthatdecentralisationofbusinessadviceislikelytohaveprosandcons:localknowledge,take-upandserviceperformancemayrise,butcostsandservicecomplexitymayalsoincrease.
Public vs. private-led delivery
We found no evidence that would suggest one delivery model (public, private, hybrid) is more effective than others.
Whilethereviewfocusesonevaluationsofpublicly-fundedprogrammes,themanagementanddeliveryofsuchprogrammesisoftendividedandsharedbetweenpublicandprivatesectororganisations.
Inseveralcases,publicsectorfundingisprovidedtoestablishedbusinessnetworkstorunsupportandnetworkingprogrammes,withtheoverallsteerisverymuchprovidedbyprivatefirms.42However,inthemajorityofcases,ahybridstructureisutilisedwhichinvolvesoverarchingprojectmanagementbypublicsectoragenciesandthesub-contractingofspecificadviceandothersub-programmestoamixtureofprivatesectorgroups.43Often,businessesorindividualsarescreenedbycustomer-facingbusinessadvisorsfrompublicsectororganisations,beforethenbeingreferredtoamorespecialistadviserfromtheprivatesector.44Somedirectlysubsidisedprivatesectorbusinessconsultancyorprofessionalservices,forexamplethroughtheprovisionof‘consultancyvouchers’.45
Table 6: Public / private delivery models
Delivery model No of studies Study reference number
Public-led 3 165,183,184
Private-led 4 168,170,182,286
Hybrid 14 162,163,166,167,172,177,217,269,276,282,284,285,287
Notstated 2 169,195
Ofthethreeevaluationsthatlookedatpublic-ledprogrammedelivery,theresultscangenerallybe
41 Curran(2000),Bennett(2008).42 Studies168and286providegoodexampleofthisapproach.43 Seestudies162or163forexamples.44 BusinessLinkutilisedthismodel.45 Forexample,Study170.
Evidence Review: Business Advice - May 2014 27
describedasmixed.Twostudiesfindpositiveeffectsonatleastonefirmoutcome.Forexample,firmsutilisingtheUKTI’sregionalsupportnetwork(UK)experiencedbetween8%and15%increasedturnovergrowth.46WhiletheUKTIPassportprogramme(UK)didgenerateturnovergrowthwhichwas20%fasterthanthecontrolgroup(albeitwithoutleadingtogreaterprofits).47Thefinalprogrammeismoreunusualshowingpositiveimpactsintermsofemployment,turnoverandproductivity,butnegativeimpactsonexports(theprogrammeobjective).
Inthecaseofthesimilarlysmallsampleoffourprivate-ledprogrammescoveredintheevaluations,theresultsarealsosomewhatmixed.Oneevaluation,whichlookedattheRegionalBusinessDevelopmentprogramme(Sweden),foundthatthedistributionofprivate-sectorconsultancyvouchershadnosignificantimpactonemploymentorproductivity.48Programmesthatutilisedamore‘handson’managementstyletendedtoshowmorepositiveresults(apointwereturntobelow).Twoweremanagedandrunbyprivatesectorbusinessnetworksandprovidedbothmentoringandnetworkingopportunitieswithbusinessexecutivesstillinactiveemploymentwithotherfirms.49Positiveimpactswereseenonsalesandassetvalue.Inthecaseofthelatterstudy,firms’netassetvalueincreasedbynearly5%.50
Delivery Technique
Programmes which used a hands-on, ‘managed brokerage’ approach may perform better than those using a light touch approach (although this conclusion is based on only one comparison study).Taken at face value, this suggests that a strong relationship and a high level of trust between advisor and client may be important to the delivery of positive programme outcomes. It is not clear, however, which of these two approaches is more cost-effective.
Theprogrammescoveredinourreviewuseawidevarietyofdifferenttechniquesintheirdelivery.IntheirreviewoftheBusinessLinkservice,Moleetaldevelopedaseriesofmodelswhichbroadlycategorisedhowdifferentaspectsoftheadviceserviceoperated.51Wehaveadaptedthiscategorisationtoallowustocomparedifferentprogrammetypesinourreview:
• Light Touch programmesarethosewithlittleornoengagementorfollowupbetweenadvisorsandtheirclients,forexample,thosewhichprovideholisticadvicewhichdirectsthemtowardsothersources;52
• Managed Brokerage programmeswereessentiallysimilar,butwiththeadvisorinmoreofarelationship-buildingposition;theyprovidemoretargetedindividualadvice;53
• Pipeline Forcing programmesinvolvemoreintensiveadvicetotheclientinordertogetthemtotheendofthepipeline.54Wehavecategorisedmostoftheprogrammesfocusedonself-employmentaspipelineforcing.
Wealsoseparatedoutthreeprogrammeswhichfocusedsolelyonexportpromotion(aswefeltthatthesedidnotfitappropriatelyintoanyofthesethreecategories).
46 Seestudy183.47 Study184.48 Study170.49 Seestudies168and286.50 Study286.51 Study177.52 Seestudy167.53 Study162isagoodexampleofthis,withconsultantsdiagnosingproblemsthatpreventgrowth,suggesting solutionsandassistingwithimplementation.54 Study287fitsneatlyintothiscategory,theassistancebeingshorttermandintense.
Evidence Review: Business Advice - May 2014 28
Table 7: Delivery Techniques
Delivery technique No of studies Study reference number
Lighttouch 1 168
Managedbrokerage 4 162,174,269,286
ComparisonofLighttouchvsmanaged
1 166
Pipelineforcing 4 163,217,282,287
Other 9 167,170,172,177,182,195,276,284,285
Exportpromotion 4 165,169,183,184
Oneevaluationlookedata‘lighttouch’programme,whichfoundthatbusinessadvisoryservicesresultedingreaterlevelsofsales,withsomeevidencetosuggestthattheprobabilityofpatentsisalsoincreased.Itshouldbenoted,however,thattheservicesreceived,though‘lighttouch’innature,weretailoredtotheneedsofbusinesses.55
Programmesutilisingmanagedbrokeragetechniquesaspartoftheirdelivery,whichwouldinvolvesomelevelof“fundingandmoreinteractionbetweentheclientsandadvisorinarelationship-buildingapproach”,tendedtobesuccessfulinimprovingatleastonefirmoutcome.Allfourevaluationsfoundsomepositiveresults,oneofwhichwastheresultofanSMS5randomisedexperiment.56Inthiscase,monthlyprofitswerereportedtobe120%higherversusthecontrolgroup.
Onthebasisofthesestudies,itishardtoreachanyconclusionaboutwhichapproachismoreeffective.Thereis,however,oneexampleofanevaluationthatdirectlycomparesintensiveandlighttouchsupportadministeredtobusinesses.57Itfoundthatmoreintensiveadvisoryservicesweremoreeffectivethan‘lighttouch’methods,headingto2.2%higherannualemploymentgrowth.Itisimportanttonotethatthisstudydidnotuserandomisedallocationtechniques;insteadsupportmethodswerebroadlycategorisedandusedtomakelatercomparisons.Takenatfacevalue,thisevaluationsuggeststhatastrongrelationshipandahighleveloftrustbetweenadvisorandclientmaybeimportanttothedeliveryofpositiveprogrammeoutcomes.However,itisimportanttonotethatsuchadvicemaybemoreexpensive,andsoitisunclearwhichapproachismorecost-effective.
Severalprogrammes(notablyBusinessLink)usedamixtureofthesetechniques,oranothertechnique.Onceagain,thecategorisationofevaluationsoftheBusinessLinkpolicywaschallengingbecauseeachevaluationprovidesslightlydifferentmaterialonthecontentoftheprogramme,andthescopeoftheservicesprovidedvariedovertimeandbetweenareas.Themajorityofevaluationssuggestamixedapproachoflighttouchandmanagedbrokerageandso,forconsistency,wehaveclassifiedalltheBusinessLinkevaluationsas‘mixed’.Interestingly,resultsarefarmoremixedforthesestudies.Ofthoseclassifiedas‘mixed’or‘other’approaches,fivefoundpositiveresultsforatleastonefirmoutcome.58Oftheremainder,threefoundnoevidenceofpositiveeffects,whileonefoundnegativeimpactsofbusinessadvice.59
Programmesthatfocusedsolelyonexportpromotionalsofoundpositiveeffectsforsomefirmoutcomes,thoughitisimportanttoacknowledgethatthethreeevaluationsincludedinourshortlistalllookedat
55 Study168.56 Studies162,174,296and286.Study162isanRCT.57 Study166.58 Studies166,177,182,284and285.59 Studies170,195and276foundinsignificanteffects.Study172foundnegativeeffects.
Evidence Review: Business Advice - May 2014 29
initiativesdeliveredbyUKTI,thuswecannotconcludewithcertaintythatallexportfocusedprogrammesaregenerallysuccessfulintheiroutcomes.Wediscussfindingsfromthesethreestudiesfurtherbelow.
Sector targeting
There is insufficient evidence to establish the relative effectiveness of sector specific programmes compared with more general programmes.
Themajorityoftheprogrammesinourreviewwerenotrestrictedbyfirmsector,butwedidfindtwoevaluationsofprogrammesthatweremoretargeted.Onewastailoredspecificallytoadvancedtechnologyfirms,whilstanotherwasonlyopentoBritish-basedmanufacturers.60
Table 8: Sector Targeting
Targeting No of studies Study reference number
Sector-specific 2 172,182
General 21 162,163,165,166,167,168,169,170,174,177,183,184,195,217,269,276,282,284,285,286,287
Oneofthetwosector-specificstudiesshowednostatisticallysignificanteffectsonemploymentandfirmsurvival61.Theothershowedsignificantpositiveeffectsonsalesandemploymentgrowthforsmallandmediumenterprises.62Overall,therearenotenoughstudiesthatfocusonsector-specificprogrammestosaywhethertheseprogrammestendtoleadtomore/lesspositiveoutcomesthanthegeneralprogrammes.Therearealsonodirectcomparisonsavailable.
Programme outcomes for firms
Business advice programmes show consistently better results for productivity and output than for employment. Results for sales, profits and exports are also mixed, although in the latter cases the count of studies is small.
Assetoutabove,mostprogrammesinourshortlisttargetmultipleorvagueobjectives,whichmakesitdifficulttoassessprogrammesuccessdirectlyagainstobjectives.Wewereonlyabletodothisdirectcomparisonforfiveofthe23shortlistedevaluations.
Forthatreason,wealsolookdirectlyatprogrammeoutcomes,regardlessofstatedobjectives.Inthisinstance,weareinterestedinwhetherbusinessadviceinterventionsworkbetterforsomefirmoutcomesthanothers.Inparticular,wewanttoknowwhetherprogrammestendtohavepositiveimpactsonfirmoutcomesthatrelatecloselytokeylocalgrowthoutcomes(particularlyproductivityoremployment)aswellasonrelevantfirm-leveloutcomes(suchassales)thatmightnotbetiedtolocaleconomicgrowthobjectives.
60 Study172istechnology-focused,182manufacturing-focused.Afurtherevaluation(study183)lookedspecifically atprogrammeimpactfromtheperspectiveofthemanufacturingandrealestateindustries,thoughforconsistency wehavedecidednottoincludeitinourdiscussionhereastheprogrammeitselfwasnotitselfsector-targeted.61 Study172.62 Study182.
Evidence Review: Business Advice - May 2014 30
Table 9: Programme outcomes
Outcome type Studies Positive Zero Negative Share positive
Productivity 165,166,167,169,284 4 1 4/5
GVA 170,174,183,269,286 4 1 4/5
Sales/turnover 162,165,167,168,177,182,184,195,269,282,284,285
7 5 7/12
Employment 163,165,166,167,169,170,172,174,177,182,195,217,
282,284,285,286,287
8 8 1 8/17
Employmentduration/survival
172,217 1 1 0/2
Assets/capital 172,184,286 2 1 2/3
Exports 165,169 1 1 1/2
Profits 162,184,195 1 2 1/3
Earnings/income 163,276,287 1 2 1/3
Ownership 282 1 0/1
Unemploymentbenefits 163 1 0/1
Patents 168 1 0/1
ResultsaregiveninTable9.Wefindthatthe10evaluationslookingatproductivityoroutput(GVA)showthemostconsistentlypositiveresults(8/10positive).Incontrast,ofthe17studiesthatlookatemploymentoutcomes,onlyeightreportpositiveprogrammeeffects.Asmanyevaluationsreportzeroeffectsonemployment,andoneevenfindsnegativeeffects.Forthetwoprogrammesthatlookatemploymentdurationorsmallbusinesssurvival,resultsaresubstantiallyworsethanotheremploymentcases.Resultsforsalesoutcomesareslightlybetterthanforemploymentbutworsethanforproductivity(withsevenoutof12programmesreportingpositiveresults).
Wealsofindevaluationsofarangeofotheroutcomes,someofwhichareonlyrelevantforspecificprogrammes(forexampleunemploymentbenefits,whichisonlyreportedforProjectGATE,aprogrammetargetedspecificallyatjoblesspeople).63Overall,foranygivenoutcomeresultsarequitemixedandlimitedtoasmallnumberofevaluations.
Inprinciplewemightworrythatsomeevaluationsareonlyreportingsuccessfulfindings,orhidingnegativeresults.Inpracticeweseeanumberofzeroornegativeeffectsinourshortlistedstudies.Forinstance,study165evaluatesanexportsprogrammeandreportsnegativeeffectsonexports,butpositivecoefficientsonanumberofotheroutcomes(productivity,employmentandturnover).Oursiftstrategy(seemethodologysectionabove)shouldalsoremovestudieswhichmakeinflatedorinaccurateclaims.Itmightalsobethatproductivitybenefitsoccurfirst,whileemploymentbenefitstakelongertomaterialise.Evaluationsthattakealongertermperspectivewouldhelpusunderstandwhetherthisisthecase.
63 Study163.
Evidence Review: Business Advice - May 2014 31
Specific UK ProgrammesAnumberofourevaluationslookatthesameprogramme,allowingustoattemptcollectivejudgementsabouttheeffectivenessofspecificpolicies.
FivestudieslookspecificallyattheBusinessLink(UK)policy,examiningprogrammeeffectsonarangeoffirmoutcomes.64Theweightofevidencesuggeststhatthepolicyhashadmixedresultsacrosstheseoutcomes.ThreeofthefivestudiesconcludethatBusinessLinkhadapositiveandsignificantimpactuponemployment,between2.2%and4.4%.65Intensiveassistanceappearstobeparticularlyeffective.66However,impactonsalesappearstobeparticularlyweakinallstudies,withonlytentative(andstatisticallyinsignificant)indicationsthatBusinessLinkleadstoupliftsinsalesandproductivity.67Thisisparticularlynoteworthygiventhatoneofthemainobjectivesofthepolicyistoimprovetheproductivityofsmallbusinesses.Anotherthingtonoteisthatallofthefiveevaluationsareranked3ontheSMSscale,thelowestqualitytypeofevaluationthatwehaveconsidered.
WealsofoundfourstudiesthatconsidertheimpactsofUKTI(UK)programmesthatassistsmallfirmstoenternewforeignmarketsandincreasetheirexports.68Threeofthefourevaluationsshowpositiveprogrammeeffectswhenlookedatinthecontextofseveraloutcomevariables.69Surprisingly,onlytwoofthefourstudiesofexport-focusedprogrammeslookspecificallyattheimpactonexports;ofthoseonefindsasignificantpositiveimpactonexportingitself,70whilstoneactuallysuggeststheopposite.71GiventhatacoreaimofUKTIistoallowfirmstoenterforeignmarkets,thisraisesquestionsabouttheoveralleffectivenessofsomepoliciesindeliveringtheirstatedobjectives,evenifsupportdoesappeartobroadlybeneficialtofirms.
64 Studies166,167,177,284and285.65 Studies177,284and285.66 Studies284and166comparetheeffectivenessoflighttouchandintensiveservices.67 Studies167and284.68 Studies165,169,183,and184.69 Studies165,169and18470 Study16971 Study165
Evidence Review: Business Advice - May 2014 32
Summary of findings
What the evidence shows1. Business support and advice had a positive impact on at least one business outcome
in 17 out of 23 evaluations. Fourevaluationsfoundthatbusinessadvicedidn’tworkandtwostudiesfoundthatbusinessadvicemightbeharmful.(Onestudyfoundnegativeeffectsagainstthestatedobjective,althoughotherpositiveeffectswerealsorecorded.)
2. Business advice programmes show consistently better results for productivity and output (GVA) than for employment. Evaluationslookingatproductivityoroutputeachshowpositiveeffectsineightoutof10cases.Bycontrast,evaluationsthatlookatemploymentreportpositiveeffectsinonlyeightoutof17cases(witheightcasesofzeroeffects,andzeroornegativeresultsfortwofurtherstudieslookingatself-employmentduration/survival).Resultsforsales,profitsandexportsarealsomixed,althoughinthesecasesthecountofstudiesissmall.
3. Programmes that used a hands-on, ‘managed brokerage’ approach may perform better than those using a light touch delivery model. Note,however,thatthisconclusionisbasedononlyonedirectcomparisonstudyanddoesnotconsidercost-effectiveness.
Where the evidence is inconclusive4. In most cases programmes had multiple or vague objectives. Ofthefiveevaluations
thatlookedatprogrammeswithaclearlyidentifiedobjective(i.e.increaseemployment/exports)three found positive effects.
5. We find no strong differences between programmes with multiple objectives and programmes with more focused objectives. Thelackofaclearlinkbetweenprogrammeobjectivesandspecificoutcomemeasuresinthemajorityofcasesmakesitveryhardtoassesstheoveralleffectivenessofprogrammesupport,andtounpickwhatfeaturesoftheprogrammesarelinkedtospecificoutcomesforfirms.Itisalsolikelytoraiseoperationalcomplexityandprogrammerisks(suchasthechanceofunintendedconsequences).Ourabilitytounderstandwhatworksforbusinessadvicewouldbeimprovedifpoliciesweredesignedwithclearobjectivesthatrelatedtomeasurable,relevantfirmoutcomes.
09
Evidence Review: Business Advice - May 2014 33
6. We found no evidence that would suggest one level of delivery is more effective than others. Resultsforbothnationally-ledandlocally-ledprogrammesismixed,asitisfor‘hybrid’programmesthecombinenationalandlocaldeliverystructures.
7. It is difficult to reach any strong conclusions on the effectiveness of private-led versus public-led delivery. Resultsappeartobemixedforbothpublic-ledandprivate-ledprogrammes.
8. Overall, it is difficult to reach any conclusions on the link from specific programme features to better firm outcomes.Toimproveourunderstandingofwhatworks,policymakersshoulddesignprogrammesthatallowforexplicitcomparisonsofprogrammefeatures–forexample‘lighttouch’versus‘handson’deliverymodels,ordifferentkindsof‘accelerator’approachesforstartups(seebelow).
Where there is a lack of evidence 9. Thereisinsufficientevidencetoestablishtherelativeeffectivenessofsector-specific
programmescomparedwithmoregeneralprogrammes.
10. Wefoundnohigh-qualityimpactevaluationsthatexplicitlylookathowbusinessadviceaffectsoutcomesforfemale-headedorBMEbusinesses.Moreresearchisneededhere.
11. Wefoundtwohigh-qualityevaluationsofprogrammesaimedatincubatingstart-ups(referredtoaboveas‘pipelineforcing’).Bothprogrammesweretargetedatunemployedpeopleandshowmixedresultsoverall.However,thereisalackofimpactevaluationforDragons’Den-typeacceleratorprogrammesthataimtolaunchhigh-growthbusinessesandinvolvecompetitiveentry.
Evidence Review: Business Advice - May 2014 34
How to use this review
Thisreviewconsidersaspecifictypeofevidence–impact evaluation.Thistypeofevidenceseekstoidentifyandunderstandthecausaleffectofpolicyinterventionsandtoestablishtheircost-effectiveness.Toputitanotherway,theyask,‘didthepolicywork’?
Thefocusonimpactreflectsthefactthatweoftendonotknowtheanswerstobasicquestionsthatmightreasonablybeaskedwhendesigninganewpolicy,suchas‘whatwillworkbestinthisscenario’and‘whatapproachoffersbestvalueformoney’.Beingcleareraboutwhatisknownwillenablepolicymakerstobetterdesignpoliciesandundertakefurtherevaluationstostartfillingthegapsinknowledge.
Supporting and complementing local knowledgeTheevidencereviewsetsoutanumberof‘BestBets’,whichoutlinetheapproachestobusinessadviceandmentoringhaveperformedmoststrongly,basedonthebestavailableimpactevaluations.
The‘BestBets’donotaddressthespecificsof‘whatworkswhere’or‘whatwillworkforaparticularindividual’.Insomecasesevaluationsdo breakoutresultsbyareatypeordifferentgroups.Butevenwhentheydo,detailedlocalknowledgeandcontextremaincrucial.
Reflectingthis,theoverallfindingsfromtheevaluationsshouldberegardedasacomplement,notasubstitute,forlocal,on-the-groundknowledge.
Businessadviceandmentoringinterventionswillneedtobetailoredandcarefullytargeted.Anaccuratediagnosisofthespecificlocalbusinesschallengesneedstobethefirststeptounderstandinghowtheevidenceappliesinanygivensituation.
‘BestBets’alsoraiseanoteofcautionforpolicymakersiftheydecidetointroduceaprogrammethathasnotworkedsowellelsewhere.
Specific recommendations
• Whendesigningaprogramme,localpolicymakersshouldidentify one or two clear programme objectives,andthenidentifyoutcomemeasuresthatarebothclearlyrelated
10
Evidence Review: Business Advice - May 2014 35
totheprogrammeobjectives,andfeasibletomeasure.Manyoftheprogrammeswelookedathadoverlycomplexorvagueobjectives,whichmakesassessingeffectivenessdifficult.
• Businessadviceprogrammesarebestplacedtohelpfirms’productivity, profits and sales–ratherthanraisingemployment.
• Programmes that used a hands-on, ‘managed brokerage’ approach may perform better than those using a ‘light touch’ delivery model. Note,however,thatthisconclusionisbasedononlyonedirectcomparisonstudyanddoesnotconsidercost-effectiveness.
Helping to fill the evidence gapsAsshouldbeclearfromthisreview,therearemanythingsthatwedonotknowabouttheeffectivenessofbusinesssupport.Muchofthepolicydebatefocusesonverybroadquestionsabouttheinstitutionalstructuresthatareputinplacetosupportbusinesses.Yetoveralltheevidenceprovidesnoclearsteeronwhetheroneparticulartypeofdeliverymodel(public/private;national/local)ismoreeffective.
Tohelpimprovebusinessadviceprogrammes,wewouldliketoseefarmorefocusonrobustlyevaluatingtheimpactofparticularaspectsofadviceprogrammesandcomparingtheircost-effectiveness.Forexample,thecostsoflighttouchversusmoreintensivesupportvarydramatically,yetwefoundonlyoneevaluationthatdirectlycomparedtheeffectivenessofthesetwotypesofsupport.Similarly,only5ofthe23shortlistedstudiesincludedcost-benefitanalysisthatassesscost-effectiveness,andnotalloftheseusedmeasuresthatarecomparableacrossstudies.Thereisaclearneedformore,consistentcost-benefitanalysisinbusinessadviceimpactevaluations.Webelievethatfurtherevaluationsofthiskind,involving,forexample,theprovisionofdifferenttypesofadvicetosimilarfirms,shouldbeapriorityforimprovingourunderstandingofwhatworksinbusinessadvice.Localflexibilitythatallowsforgreaterexperimentationprovidesanidealopportunitytoundertakesuchevaluations.
TheCentre’slonger-termobjectivesaretoensurethatrobustevidenceisembeddedinthedevelopmentofpolicy,thatthesepolicesareeffectivelyevaluated,andthatfeedbackisusedtoimprovethem.Toachievetheseobjectiveswewantto:
• Workwithlocaldecision-makerstoimproveevaluationstandardssothatwecanlearnmoreaboutwhatpolicieswork,where;and
• Establishupaseriesof‘demonstrationprojects’toshowhoweffectiveevaluationcanworkinpractice.
Interestedpolicymakerspleasegetintouch.
Evidence Review: Business Advice - May 2014 36
References
Acs,Z.J.,Audretsch,D.,Braunerhjelm,P.,&Carlsson,B.(2004).TheMissingLink:TheKnowledgeFilterandEntrepreneurshipinEndogenousGrowth.London:CEPR.
Aghion,P.,Blundell,R.,Griffith,R.,Howitt,P.,&Prantl,S.(2009).TheEffectsofEntryonIncumbentInnovationandProductivity.ReviewofEconomicsandStatistics,91(1),20-32.
Akerlof,G.(1970).Themarketfor‘lemons’:qualityuncertaintyandthemarketmechanism..QuarterlyJournalofEconomics,84(3),488-500.
Bartle,D.,andMorris,M.(2010)Evaluatingtheimpactsofgovernmentbusinessassistanceprogrammes:approachestotestingadditionality.ResearchEvaluation,19,275–280.
Batra,G.,&Mahmood,S.(2003).DirectSupporttoPrivateFirms:EvidenceonEffectiveness.Washington,DC:WorldBank.
Bennett,R.(2008).“SMEpolicysupportinBritainsincethe1990s:whathavewelearnt?”EnvironmentandPlanningC,26,375-397.
Benus,J.,Shen,T.,Zhang,S.,Chan,M.andHansen,B.(2010)GrowingAmericaThroughEntrepreneurship:FinalEvaluationofProjectGATE.WashingtonDC,USDepartmentofLaborEmploymentandTrainingAdministration.
Benus,J.M.,Wood,M.L.,Grover,N.,andAbtAssociates(1994)AComparativeAnalysisoftheWashingtonandMassachusettsUISelf-EmploymentDemonstrations.Cambridge,Mass,AbtAssociates.
Bravo-Biosca,A.,Marston,L.,&Westlake,S.(2011).VitalGrowth:Theimportanceofhigh-growthbusinessestotherecovery.London:NESTA.
Breinlich,H.,Mion,G.,Nolen,P.,andNovy,D.(2012)IntellectualProperty,OverseasSales,andtheImpactofUKTIAssistanceinEnteringNewOverseasMarkets(UKTI).London,UKTI.
Bruhn,M.,Karlan,D.S.,andSchoar,A.(2012)TheImpactofConsultingServicesonSmallandMediumEnterprises:EvidencefromaRandomizedTrialinMexico.YaleUniversity,DepartmentofEconomics.
Cumming,D.J.,andFischer,E.(2012)PubliclyFundedBusinessAdvisoryServicesandEntrepreneurialOutcomes.ResearchPolicy41,467–481.
Curran,J.(2000).WhatisSmallBusinessPolicyintheUKfor?EvaluationandAssessingSmall.InternationalSmallBusinessJournal,18(3),36-50.
Fairlie,R.W.,Karlan,D.,andZinman,J.(2012)BehindtheGATEExperiment:EvidenceonEffectsofandRationalesforSubsidizedEntrepreneurshipTraining.NBERWorkingPaper17804,Cambridge,Mass.NBER.
Gibbons,S.,Nathan,M.andOverman,H.(2014).EvaluatingSpatialPolicies.TownPlanningReview,forthcoming.
Girma,S.,Görg,H.,andPisu,M.(2005).Quantitativeanalysisandlinkedmicro-datastudyofUKTIservices-Finalreport.London,UKTI.
11
Evidence Review: Business Advice - May 2014 37
Godel,M.,andMantovani,I.(2012)EvaluationoftheimpactandcosteffectivenessofUKTI’sregionalnetworksupport.London,UKTI.
Grossman,S.J.,&Stiglitz,J.E.(1980).OntheImpossibilityofInformationallyEfficientMarkets.TheAmericanEconomicReview,70(3),393-408.
Haltiwanger,J.C.,Jarmin,R.S.,&Miranda,J.(2010).WhoCreatesJobs?Smallvs.Largevs.Young.Cambridge,MA:NBER.
Hart,M.,andRoper,S.(2003)ModellingtheEffectsofPublicSupporttoSmallFirmsintheUK-ParadiseGained?In:EuropeanRegionalScienceAssociation,EuropeanRegionalScienceAssociationConference.Jyväskylä,Finland,August2003.
Huggins,R.,&Williams,N.(2009).Enterpriseandpublicpolicy:areviewofLabourgovernmentinterventionintheUnitedKingdom.EnvironmentandPlanningC:GovernmentandPolicy,27(1),19-41.
Kosters,S.,andObschonka,M.(2011)PublicBusinessAdviceintheFoundingProcess:AnEmpiricalEvaluationofSubjectiveandEconomicEffects.EnvironmentandPlanningC,29,577–604.
Mansson,J.,andWiderstedt,B.(2012).TheSwedishBusinessDevelopmentProgram:Evaluationandsomemethodologicalandpracticalnotes.In:EuropeanRegionalScienceAssociation,EuropeanSocietyofRegionalAnalysis.Bratislava,Slovakia,2012.
Mole,K.,Roper,S.,Hart,M.,Storey,D.,&Saal,D.(2006).EconomicimpactstudyofBusinessLinklocalservice:finalreport.London:DepartmentforBusinessEnterpriseandRegulatoryReform.
Mole,K.,Hart,M.,Roper,S.,&Saal,D.(2008).DifferentialgainsfromBusinessLinksupportandadvice:atreatmenteffectsapproach.EnvironmentandPlanningC:GovernmentandPolicy,26(2),315-334.
Mole,K.F.,Hart,M.,Roper,S.,andSaal,D.S.(2009)AssessingtheEffectivenessofBusinessSupportServicesinEnglandEvidencefromaTheory-BasedEvaluation.InternationalSmallBusinessJournal,27,557–582.
Mole,K.F.,Hart,M.,Roper,S.,andSaal,D.S.(2011)Broaderordeeper?Exploringthemosteffectiveinterventionprofileforpublicsmallbusinesssupport.EnvironmentandPlanningA,43,87–105.
Oberschachtsiek,D.,Scioch,P.(2011)Theoutcomeofcoachingandtrainingforself-employment:astatisticalevaluationofnon-financialsupportschemesforunemployedbusinessfoundersinGermany.Nuremberg,InstituteforEmploymentResearchoftheFederalEmploymentAgency(IAB).
Ram,M.,&Jones,T.(2008).Ethnic-minoritybusinessesintheUK:areviewofresearchandpolicydevelopments.EnvironmentandPlanningC:GovernmentandPolicy,26(2),352-374.
Robson,P.J.A.,andBennett,R.J.(2000)SMEGrowth:TheRelationshipwithBusinessAdviceandExternalCollaboration.SmallBusinessEconomics,15,193–208.
Rogers,M.,andHelmers,C.(2008)IntellectualpropertyandUKTIPassportfirms-Stage2Report.London,UKTI.
Sanders,C.(2002)TheImpactofMicroenterpriseAssistancePrograms:AComparativeStudyofProgramParticipants,Nonparticipants,andOtherLow‐WageWorkers.SocialScienceReview,76,321-340.
Sherman,L.W.,Gottfredson,D.C.,MacKenzie,D.L.,Eck,J.,Reuter,P.,&Bushway,S.D.(1998).PreventingCrime:WhatWorks,WhatDoesn’t,What’sPromising.WashingtonDC:USDepartmentofJustice.
Schoonjans,B.,Cauwenberge,P.V.,andBauwhede,H.V.(2013)Knowledgenetworkingandgrowthinservicefirms.TheServiceIndustriesJournal,33(11),1051-1067.
Schumpeter,J.(1962).TheTheoryofEconomicDevelopment.Berlin:Springer.
Stiglitz,J.(1979).EquilibriuminProductMarketswithImperfectInformation.AmericanEconomicReview,69(2),339-345.
Stiglitz,J.(2002).InformationandtheChangeintheParadigminEconomics.TheAmericanEconomicReview,92(3),460-501.
Storey,D.(1994).UnderstandingtheSmallBusinessSector.London:Routledge.
VanCauwenberge,P.,VanderBauwhede,H.,andSchoonjans,B.(2013)Anevaluationofpublicspending:theeffectivenessofagovernment-supportednetworkingprograminFlanders.EnvironmentandPlanningC,31,24–38.
Wren,C.,andStorey,D.(2002)Evaluatingtheeffectofsoftbusinesssupportuponsmallfirmperformance.OxfordEconomicPapers,54,334–365.
Evidence Review: Business Advice - May 2014 38
Appendix A: Evidence Reviewed
Number Reference
162 Bruhn,M.,Karlan,D.S.,andSchoar,A.(2012)TheImpactofConsultingServicesonSmallandMediumEnterprises:EvidencefromaRandomizedTrialinMexico.YaleUniversity,DepartmentofEconomics.
163 Benus,J.,Shen,T.,Zhang,S.,Chan,M.andHansen,B.(2010)GrowingAmericaThroughEntrepreneurship:FinalEvaluationofProjectGATE.WashingtonDC,USDepartmentofLaborEmploymentandTrainingAdministration.
165 Breinlich,H.,Mion,G.,Nolen,P.,andNovy,D.(2012)IntellectualProperty,OverseasSales,andtheImpactofUKTIAssistanceinEnteringNewOverseasMarkets(UKTI).London,UKTI.
166 Mole,K.F.,Hart,M.,Roper,S.,andSaal,D.S.(2011)Broaderordeeper?Exploringthemosteffectiveinterventionprofileforpublicsmallbusinesssupport.EnvironmentandPlanningA,43,87–105.
167 Hart,M.,andRoper,S.(2003)ModellingtheEffectsofPublicSupporttoSmallFirmsintheUK-ParadiseGained?In:EuropeanRegionalScienceAssociation,EuropeanRegionalScienceAssociationConference.Jyväskylä,Finland,August2003.
168 Cumming,D.J.,andFischer,E.(2012)PubliclyFundedBusinessAdvisoryServicesandEntrepreneurialOutcomes.ResearchPolicy41,467–481.
169 Girma,S.,Görg,H.,andPisu,M.(2005).Quantitativeanalysisandlinkedmicro-datastudyofUKTIservices-Finalreport.London,UKTI.
170 Mansson,J.,andWiderstedt,B.(2012).TheSwedishBusinessDevelopmentProgram:Evaluationandsomemethodologicalandpracticalnotes.In:EuropeanRegionalScienceAssociation,EuropeanSocietyofRegionalAnalysis.Bratislava,Slovakia,2012.
172 Kosters,S.,andObschonka,M.(2011)PublicBusinessAdviceintheFoundingProcess:AnEmpiricalEvaluationofSubjectiveandEconomicEffects.EnvironmentandPlanningC,29,577–604.
174 VanCauwenberge,P.,VanderBauwhede,H.,andSchoonjans,B.(2013)Anevaluationofpublicspending:theeffectivenessofagovernment-supportednetworkingprograminFlanders.EnvironmentandPlanningC,31,24–38.
177 Mole,K.F.,Hart,M.,Roper,S.,andSaal,D.S.(2009)AssessingtheEffectivenessofBusinessSupportServicesinEnglandEvidencefromaTheory-BasedEvaluation.InternationalSmallBusinessJournal,27,557–582.
182 Wren,C.,andStorey,D.J.(2002)Evaluatingtheeffectofsoftbusinesssupportuponsmallfirmperformance.OxfordEconomicPapers,54,334–365.
183 Godel,M.,andMantovani,I.(2012)EvaluationoftheimpactandcosteffectivenessofUKTI’sregionalnetworksupport.London,UKTI.
184 Rogers,M.,andHelmers,C.(2008)IntellectualpropertyandUKTIPassportfirms-Stage2Report.London,UKTI.
195 Robson,P.J.A.,andBennett,R.J.(2000)SMEGrowth:TheRelationshipwithBusinessAdviceandExternalCollaboration.SmallBusinessEconomics,15,193–208.
217 Oberschachtsiek,D.,Scioch,P.(2011)Theoutcomeofcoachingandtrainingforself-employment:astatisticalevaluationofnon-financialsupportschemesforunemployedbusinessfoundersinGermany.Nuremberg,InstituteforEmploymentResearchoftheFederalEmploymentAgency(IAB).
269 Bartle,D.,andMorris,M.(2010)Evaluatingtheimpactsofgovernmentbusinessassistanceprogrammes:approachestotestingadditionality.ResearchEvaluation,19,275–280.
Evidence Review: Business Advice - May 2014 39
Number Reference
276 Sanders,C.(2002)TheImpactofMicroenterpriseAssistancePrograms:AComparativeStudyofProgramParticipants,Nonparticipants,andOtherLow‐WageWorkers.SocialScienceReview,76,321-340.
282 Fairlie,R.W.,Karlan,D.,andZinman,J.(2012)BehindtheGATEExperiment:EvidenceonEffectsofandRationalesforSubsidizedEntrepreneurshipTraining.Cambridge,Mass.,NationalBureauofEconomicResearch.
284 Moleetal(2007)EconomicImpactStudyofBusinessLinkLocalService.London,DeprtmentforBusinessEnterpriseandRegulatoryReform.
285 Moleetal(2008)DifferentialgainsfromBusinessLinksupportandadvice:atreatmenteffectsapproach.EnvironmentandPlanningC,26,315–334.
286 Schoonjans,B.,Cauwenberge,P.V.,andBauwhede,H.V.(2013)Knowledgenetworkingandgrowthinservicefirms.TheServiceIndustriesJournal,33(11),1051-1067.
287 Benus,J.M.,Wood,M.L.,Grover,N.,andAbtAssociates(1994)AComparativeAnalysisoftheWashingtonandMassachusettsUISelf-EmploymentDemonstrations.Cambridge,Mass,AbtAssociates.
Evidence Review: Business Advice - May 2014 40
Appendix B: Search Terms and Sources
Source Search Terms
EconLit “businesssupport”ANDjobs
EconLit “businesssupport”ANDemployment
EconLit “businesssupport”ANDturnover
EconLit “businesssupport”ANDproductivity
EconLit “businesssupport”ANDmeta
EconLit incubatorANDjobs
EconLit incubatorANDemployment
EconLit incubatorANDturnover
EconLit incubatorANDproductivity
EconLit incubatorANDmeta
EconLit business*ANDmentoringANDjobs
EconLit business*ANDmentoringANDemployment
EconLit business*ANDmentoringANDturnover
EconLit business*ANDmentoringANDproductivity
EconLit business*ANDmentoringANDmeta
EconLit business*ANDnetwork*ANDjobs
EconLit business*ANDnetwork*ANDemployment
EconLit business*ANDnetwork*ANDturnover
EconLit business*ANDnetwork*ANDproductivity
EconLit business*ANDnetwork*ANDmeta
WebofScience(SCCI)viaEndnoteweb “businesssupport”ANDjobs
WebofScience(SCCI)viaEndnoteweb “businesssupport”ANDemployment
WebofScience(SCCI)viaEndnoteweb “businesssupport”ANDturnover
WebofScience(SCCI)viaEndnoteweb “businesssupport”ANDproductivity
WebofScience(SCCI)viaEndnoteweb “businesssupport”ANDmeta
WebofScience(SCCI)viaEndnoteweb incubatorANDjobs
WebofScience(SCCI)viaEndnoteweb incubatorANDemployment
WebofScience(SCCI)viaEndnoteweb incubatorANDturnover
WebofScience(SCCI)viaEndnoteweb incubatorANDproductivity
WebofScience(SCCI)viaEndnoteweb incubatorANDmeta
Evidence Review: Business Advice - May 2014 41
Source Search Terms
WebofScience(SCCI)viaEndnoteweb business*ANDmentoringANDjobs
WebofScience(SCCI)viaEndnoteweb business*ANDmentoringANDemployment
WebofScience(SCCI)viaEndnoteweb business*ANDmentoringANDturnover
WebofScience(SCCI)viaEndnoteweb business*ANDmentoringANDproductivity
WebofScience(SCCI)viaEndnoteweb business*ANDmentoringANDmeta
WebofScience(SCCI)viaEndnoteweb business*ANDnetwork*ANDjobs
WebofScience(SCCI)viaEndnoteweb business*ANDnetwork*ANDemployment
WebofScience(SCCI)viaEndnoteweb business*ANDnetwork*ANDturnover
WebofScience(SCCI)viaEndnoteweb business*ANDnetwork*ANDproductivity
WebofScience(SCCI)viaEndnoteweb business*ANDnetwork*ANDmeta
WebofScience(SCCI)viaEndnoteweb mentor*ANDbusiness*ANDevaluat*
WebofScience(SCCI)viaEndnoteweb compan*ANDmentor*ANDevaluat*
WebofScience(SCCI)viaEndnoteweb mentor*andbusiness*ANDgrowth
GoogleScholar businessANDsupport
GoogleScholar “smallbusiness”ANDsupport
GoogleScholar businessANDmentoring
GoogleScholar entrepreneurANDsupport
GoogleScholar “localeconomicgrowth”AND“smallbusiness”
GoogleScholar SMEANDsupport
GoogleScholar micro-enterpriseANDsupport
GoogleScholar micro-firmANDsupport
GoogleScholar SMEANDmentoring
GoogleScholar businessadvi*
GoogleScholar “businesssupport”ANDmeta
www.gov.uk/publications(BIS,DCLG,HMTreasury,CabinetOffice,EconomicandSocialResearchCouncil,UKCommissionforEmploymentandSkills)
“smallbusiness”
www.gov.uk/publications “smallbusiness”ANDsupport
www.gov.uk/publications SME
www.gov.uk/publications SMME
www.gov.uk/publications micro-business
www.gov.uk/publications micro-enterprise
Evidence Review: Business Advice - May 2014 42
Source Search Terms
www.gov.uk/publications micro-firm
www.gov.uk/publications businessANDmentor
NationalAuditOffice-PublicationsSearch businesssupportjobs
NationalAuditOffice-PublicationsSearch businesssupportemployment
NationalAuditOffice-PublicationsSearch businesssupportturnover
NationalAuditOffice-PublicationsSearch businesssupportproductivity
NationalAuditOffice-PublicationsSearch incubator
NationalAuditOffice-PublicationsSearch businessmentoring
NationalAuditOffice-PublicationsSearch businessnetwork
IZAJournalofLaborEconomics Visualscanoffullpublicationslist
IZAJournalofLaborPolicy Visualscanoffullpublicationslist
IZAJournalofLaborandDevelopment Visualscanoffullpublicationslist
IZAJournalofEuropeanLaborStudies Visualscanoffullpublicationslist
IZA,ResearchinLaborEconomics Visualscanoffullpublicationslist
CEPR,DiscussionPapersonDevelopmentEconomics
Visualscanoffullpublicationslist
CEPR,DiscussionPapersonEconomicHistory Visualscanoffullpublicationslist
CEPR,DiscussionPapersonFinancialEconomics
Visualscanoffullpublicationslist
CentreforCities-Publications “businesssupport”
CentreforCities-Publications businessmentoring
CentreforCities-Publications network(incategory:businessandenterprise)
TheWorkFoundation Visualscanofreportssection
OECD businesssupport
OECD incubatorANDjobs
OECD incubatorANDemployment
OECD incubatorANDproductivity
OECD business+networkORnetworking+job
OECD business+networkORnetworking+employment
OECD business+networkORnetworking+productivity
OECD business+mentorORmentoring+jobs
OECD “businessmentoring”
REPECviaEconPapers “businesssupport”ANDjobs
Evidence Review: Business Advice - May 2014 43
Source Search Terms
REPECviaEconPapers “businesssupport”ANDemployment
REPECviaEconPapers “businesssupport”ANDturnover
REPECviaEconPapers “businesssupport”ANDproductivity
REPECviaEconPapers business*ANDmentor*ANDjobs
REPECviaEconPapers business*ANDadviceANDjobs
REPECviaEconPapers business*ANDadviceORadvis*ANDjobs
REPECviaEconPapers business*ANDadviceORadvis*ANDgrowth
REPECviaEconPapers SMEANDjobsANDsupport
REPECviaEconPapers SMEANDgrowthANDsupportORadvis*
REPECviaEconPapers SMEANDmentor*ANDemployment
REPECviaEconPapers SMEANDmentor*ANDjobs
REPECviaEconPapers SMEANDmentor*ANDgrowth
REPECviaEconPapers firm*ANDmentor*ANDemployment
REPECviaEconPapers firm*ANDmentor*ANDjobs
REPECviaEconPapers firm*ANDmentor*ANDgrowth
REPECviaEconPapers entrepreneur*ANDmentor*ANDproductivity
REPECviaEconPapers entrepreneur*ANDmentor*ANDjobs
REPECviaEconPapers entrepreneur*ANDmentor*ANDgrowth
REPECviaEconPapers business*ANDnetwork*ANDjobs
REPECviaEconPapers “smallbusiness”ANDnetwork*ANDjob*
REPECviaEconPapers business*ANDnetwork*ANDemploymentANDimpact*
REPECviaEconPapers business*ANDnetwork*AND“economicgrowth”ANDimpact*
REPECviaEconPapers business*ANDnetwork*ANDproductivityANDimpact*
REPECviaEconPapers SMEANDnetwork*ANDjob*ANDimpact*
REPECviaEconPapers SMEANDnetwork*ANDgrowth
REPECviaEconPapers SMEANDnetwork*ANDjob*
REPECviaEconPapers SMEANDnetwork*ANDproductivity
REPECviaEconPapers firm*ANDnetwork*ANDproductivityANDimpact
REPECviaEconPapers firm*ANDnetwork*ANDjobsANDevaluat*
REPECviaEconPapers firm*ANDnetwork*ANDjobsANDimpact*
REPECviaEconPapers firm*ANDnetwork*ANDgrowthANDimpact*
Evidence Review: Business Advice - May 2014 44
Source Search Terms
LEPwebsites(all) growthORjobs“businesssupport”
LEPwebsites(all) growthORjobsmentoring
IPPR “businesssupport”
IPPR mentoring
IPPR “businessadvice”
RDAarchives VisualscanofRDAwebsitesonNationalArchivewebsite
Eurofound “businesssupport”
Eurofound mentoring
Eurofound businessadvice
NESTA Visualscan,followedbysearchusingtermssuchas“businesssupport”&mentoring
UKCES Searchbytheme,keyterms(“businesssupport”)andvisualscanofevidencereports
CORDIS “businesssupport”
CORDIS “businesssupport”+impactjobsORgrowth
CORDIS mentoring+impactjobsORgrowth
CORDIS mentoring+businessjobsORgrowth
EuropeanCommission Visualscanwithsomesearches(“businesssupport”),butappearedtobeproblemwithsearchfunction
UNESCO Visualscan
TechnologyStrategyBoard(www.innovateuk.org)
smallANDbusiness
IPPR Visualscanoffullpublicationslist
Evidence Review: Business Advice - May 2014 45
Appendix C: Findings for RCT studiesAmongstthe23shortlistedevaluations,fourwereRandomisedControlTrialswhichscoredthemaximumfiveontheSMSscale.Theresultsofthetopscoringevaluationsaremixed.Twoofthetopfourscoringstudiesfoundpositiveresults,andtheothertwo(whichbothevaluatethesameprogramme)foundmixedresultswithminimalimpactonbusinessgrowthoutcomes.72
Study162looksataprogrammeinMexicotoprovidesubsidisedbusinessconsultingservicestomicro-,smallandmediumsizedbusinesses.Supportincludeddevelopingmissionstatements,visions,targetsaswellasadviceonaccounts,recordkeeping,pricing,teamworkandleadership.Volunteerswererandomlyselectedeithertoparticipateortoformpartofthecontrolgroupandfollow-upinterviewswereundertakentwoyearsafterprogrammeentry.Thestudyfoundverylargeproductivitygainsinfirmswhoreceivedtheadvicecomparedwiththecontrolgroup.Thosewhoparticipatedexperiencedincreasedprofits,andthereissomeevidenceofincreasedsales.Specifically,monthlysalesincreasedbyaround80%andprofitsincreasedby120%inthetreatmentgroupcomparedwiththecontrolgroup.Theaverageincreaseinprofitsisestimatedtoliebetween$7,600and$11,000permonth,comparedwithacostof$988permonthfortheconsultingservices.
Study287evaluatestwoUSprogrammesaimedatencouragingtheunemployedintoself-employment.TwoRandomisedControlTrialswerecarriedoutonfederally-fundeddemonstratorprojectsinWashingtonStateandMassachusetts.Thoseapplicantswhowererandomlyassignedtotheprogrammewereofferedentrepreneurialtraining,businesssupportandfinancialassistanceanalogoustounemploymentinsurancefortheearlyperiodofbusinessset-up.Thetwoprogrammesweredifferentintermsofdetail,butbothincludedelementsofclassroom-basedgrouplearningandinone-on-oneindividualadvicewithspecialistbusinesscounsellors.Followupsurveyswerecarriedoutaftertwoyears.Theevaluationfoundthatbothprogrammesincreasedthelikelihoodofentryintoself-employment21monthslater,andacceleratedmovesfromunemploymentintoself-employment(5.9monthsearlierfortheWashingtonprogramme,2.4monthsfortheMassachusettsprogramme).Bothprogrammesalsosignificantlyincreasedthechancesofbeingemployed/self-employed,whichwas14%higherforWashingtonparticipantsand5%higherforthoseintheMassachusettsinitiative.However,neitherprogrammesignificantlyincreasedparticipants’totalearnings,inpartbecausepeoplewereless likelytomoveintoaregularjob.Thedifferencesbetweenthetwoprogrammesseempartlyrelatedtowidereconomicconditionsinthetwostates;mostoftheUSwasinrecessionduringthedeliveryperiod,butWashington’seconomywaslargelyinsulatedfromthesemacrotrends.
TwoevaluationslookattheimpactsofProjectGATE(GrowingAmericaThroughEntrepreneurship),amicroenterprisesupportprogrammewhichaimedtosupportthedevelopmentoffledglingbusinesses,supportentrepreneurshipandmovepeoplefromunemploymenttoself-employmentaspartofawiderActiveLabourMarketProgramme.
Theprojectwassupportedbythefederalgovernmentanddeliveredlocallythrough14smallbusinessdevelopmentcentreswhichwerepubliclyownedanddeliveredanddeliveredasnon-profitorganisations.TheevaluationwascarriedoutbymeansofaRandomisedControlTrialatsevenofthe14locations(foururban,threerural)acrossthreeUSstates.Over4,000qualifyingapplicantswithabusinessideawereallocatedintoeithertreatmentorcontrolgroupsandtrackedbymeansoffollow-upsurveysforfiveyearsafterenteringtheprogramme.
Theofficialprojectevaluationfoundthatparticipantsweremorelikelytostartabusinessthanthe
72 Studies162,287
Evidence Review: Business Advice - May 2014 46
controlgroup,althoughthisimpactwasonlystatisticallysignificantduringthefirstfewquartersafterrandomassignment.73Businessesstartedbythetreatedgroupexperiencedsimilargrowthandprofitabilitytothosestartedbycontrolgroupmembers.Self-employmentandhoursworkedbyself-employedincreased(butthattheself-employmenteffectwasoffsetbyareductioninemploymentandemploymenthourswhichsuggeststhatindividualswere,onthewhole,transferringfromemploymenttoself-employmentratherthanmovingfromunemploymenttoself-employment.
Afollow-upevaluationbydifferentauthors,usingthesamedatasetaimstoprovideadditionalanalysisofProjectGATE’simpacts. 74Extendingtheoriginalstudy’sexplorationofself-employmentoutcomes,thisresearchconcludedthatmarginalbusinessescreatedasaresultofProjectGATEdonotsurviveinthelongrun.Self-employmentwas5.2%higherinthetreatmentgroup,butthisresultwasnolongerstatisticallysignificantafter18months.Theresearchersalsolookedatthesalesandemploymentoutcomesofnewbusinesses,findingnotreatmenteffect.Theprogrammealsohadnosignificanteffectonthelikelihoodofcreatinghigh-revenueorhigh-employment‘gazelle’firms.
Ingeneral,thesehigherqualityevaluationstendtofindmorepositiveprogrammeimpactsthanthelessrobustevaluations.TwooffouroftheRCTevaluationsfindpositiveresultscomparedto7of19SMS3studies.8oftheseSMS3studiesshowmixedresultscomparedtotwoRCTs(althoughtheseconsiderthesameprogramme).NoneoftheRCTsfindzeroorharmfuleffectsoverall,comparedwith4of19SMS3studies.Althoughtheseconclusionsarebasedonaverysmallnumberofstudies,theydosuggestthathighqualityevaluationmaybeimportantfordetectinganypositiveimpactofbusinessadviceprogrammes.
73 Study163.74 Study282.
Evidence Review: Business Advice - May 2014 47
Appendix D: Case Studies
Study 169: Matching; SMS level 3This2005studybySourafelGirmaandcolleaguesevaluatestwoprogrammesrunbytheUKTradeandInvestmentagency(UKTI).AftercareprovidessupportforinwardinvestorsalreadyintheUKandaimstoattractmoreinvestmentbytheseforeignfirmsbyhelpingthemintegratewithinthelocaleconomy.ThePassporttoExportservicesupportsdomesticfirmstoincreasetheirexportactivityandoffersarangeofservicessuchastraining,accesstoforeigncontracts,andexportdevelopmentservices.
ThefactthatUKTIchooseswhichoftheapplyingfirmsparticipatemakesithardertoassesstheprogramme’simpact.Therearetwosourcesofselectionbias:firstly,onlyfirmsthatexpecttobenefitwilllikelyapplyfortheprogram.Secondly,givenalimitedbudgetUKTImaybetemptedto‘pickwinners’byselectingfirmsthatlooklikelytosucceed.Overall,thismeansthatparticipatingfirmswilllikelybedifferentinperformancetootherfirms–andthismaybiastheestimatedeffectsoftheprogramme.Forexample,ifparticipantsweregenerallylargerfirms(observable)orhadmoreambitiousCEOs(unobservable)thendifferencesinperformancecomparedwithnon-participantscouldsimplybeaneffectofthesefactors.
Theauthorsusedadifference-in-differencematchingapproachtoestimatetheimpactofthepolicy.Acontrolgroupwasconstructedbycarefullymatchingprogramparticipantswithotherfirmsonthebasisofvariablessuchassize(employment),productivity(valueaddedperworker),assetsandexportingintensity.Thedifference-in-differenceestimateisthebefore-and-afterchangeinoutcomevariableforthetreatmentgroup(i.e.thosethatreceiveadvice)relativetothiscontrolgroup.Thiskindofmatchingbasedon‘observable’(orrecorded)characteristicscanneverbeperfect.Participantsandnon-participantswillalwaysdifferinsomeunobservedrespects.Becausetheevaluationisn’tabletodoanythingtoaddressthisconcern,wescorethismethoda‘3’ontheScientificMarylandScale.
Toimplementthisapproach,firmleveldatawastakenfromtheFAMEdatabasefrom1994to2003andlinkedwithUKTImanagementdata.Thereweretwootherexportpromotionprogrammesthatweredroppedfromtheanalysissincethelinkingbetweendatasetsforparticipantsoftheseprogrammeswasconsideredunsuccessfulbytheresearchers.Theirfinaltreatmentgroupsinclude299firmswhoreceivedPassportsupportatsomepointoverthistimeframeand509firmswhousedAftercare.
TheevaluationexaminestheeffectofAftercareandPassportonanumberofoutcomevariables,coveringnotonlycoreobjectives(exportingandFDI)butalsowideroutcomessuchasemployment,productivityandprofits.ForthisreviewwearemostinterestedinthePassportscheme.Specifically,participationinPassportisfoundtohavesuccessfullyincreasedthelevelofexportactivityforbothmanufacturingandservicesfirms–thecoreaimoftheprogramme.Passportisalsofoundtoincreasetotalfactorproductivity,comparedwiththecontrolgroup,butthereisnosignificanteffectonemployment.
Whatdotheseresultsmeanforpolicymakers?Onthefaceofit,theseresultsimplythattheUKTIprogrammeshavebeenverysuccessfulinreachingtheirgoalsandpromotingfirmperformanceingeneral.However,sincetheauthorsonlycontrolforobservablecharacteristics(SMS3)itmaybethattheseresultsaredrivenbythefirms’owngoals(whichareunobservable)aligningwiththepolicyobjectives.Forexample,firmsonthePassportprogrammearelikelytohavebeenconsideringincreasingtheirexportactivityinthefutureandmayhaveshownincreasescomparedwiththecontrolgroupeveniftheyhadnotbeenselected.Furtherresearchcontrollingforunobservables(throughrandomisation,instrumentsorquasi-experiments)wouldhelppindownprogrammeimpacts.
Evidence Review: Business Advice - May 2014 48
Study 170: Matching; SMS level 3InthispaperJonasMånssonandBarbroWiderstedtevaluatetheRegionalBusinessDevelopmentProgramme(RBDP)inSweden.TheRBDPprovidessupportforSMEsoperatinginruralareasofSweden–especiallyinthesparselypopulatednorth.Thepolicyaimstostimulatefirmperformancethroughacombinationofgrantsand‘consultancycheques’,whichfirmsspendonbusinessadviceandmentoring(thefocusoftheanalysis).Theprogrammeisvoluntary:firmsprepareabusinesscaseandapplyforfunding,withtheregionaldevelopmentagencymakingthedecision.In2009,theaverageawardwasabout€7,000(£5,700).
Thereareseveralchallengesinidentifyingtheeffectsofabusinesssupportprogrammelikethisone.First,firms’characteristicsratherthantheinterventionmaydriveperformance,andweneedsomewaytocontrolforthese.Second,becausetheprogrammeisvoluntaryfirmsmay‘select’intoit.Forinstance,ifthefirmswholeastneedsupportdominateapplications,thiswillbiasresultsupwards.Third,agencystaffmaynotmakeobjectivedecisions–reinforcingtheself-selectionproblem.Finally,eveniffirmsdon’treceiveRBDPfundingtheymaystillgetotherformsofbusinesssupport,whichmaycontaminatetheevaluationresults.
Todealwiththeseissues,theauthorsuseamatchingapproach,usingavailabledatatoconstructacontrolgroupthatlookssimilartothetreatmentgroup.Theyalsovarythematchingprocesstocontrolforcontaminationandself-selection–forexample,todealwithself-selectiontheycompareoutcomesfortreatmentfirmswithacontrolgroupofunsuccessfulRBDPapplicants.Theyavoidasimplecomparisonoftreatmentandcontrolgroupsafterthefirmsreceiveadvicebecausedifferencesinperformancemaybedrivenbyunobservablefirmcharacteristics.Instead,theylookatthechangeinfirms’performancebeforeandaftertreatment,andcheckifthedifferencebetweentreatmentandcontrolgroupsismeaningful.(Statisticianscallthisa‘differenceindifference’approach.)
Thisstrategyisstrongbutnotperfect.Matchingbasedon‘observable’characteristicsisneverperfect,asunobservablefactorsmaystillexplaindifferencesinfirmperformance.Theresearchersarealsounabletoseethegrant-makingdecision,sothatsourceofselectionbiasmaystillbepresent.Forthesereasons,wescorethismethoda‘3’ontheScientificMarylandScale.
Theresearchersuserichadministrativedataon1,010RBDPrecipientsand52,613controlfirmsbetween2004and2007,withsmallersamplesforcontaminationandselectiontests.Theyfindthatinthebasicanalysis,consultancycheques‘work’:valueaddedwas14.3%higherfortreatmentfirms,andemployment12.6%higher.Allowingforcontamination,effectsaresmallerbutstillstatisticallysignificant.However,theresultsbecameinsignificantoncetreatmentfirmsarecomparedwithunsuccessfulapplicants(ratherthanallotherfirmsintheregion).
Sodoestheprogrammehavezeroeffect?Theauthorssuggestthatit’sthetimefirmsspendthinkingaboutbusinessdevelopmentthatgeneratesimpacts–andthisispartoftheapplicationprocess,socanbeconsideredpartofthetreatment.Ineffect,theresultssuggestakindofplaceboeffectwheretheRBDPapplicationdoesthejobratherthanthesupportitself.Thatimpliesthatpolicymakersshouldlookforother,lesscostlywaysofdeliveringsupporttofirms–perhapsthroughoutreachratherthanstructuredadvice.
WeshouldbecarefulinusingtheseresultsinaUKcontext.Asnoted,therearesomeunresolvedissuesinidentifyingtrueeffects,andthefindingsmaynottranslatefromruralSwedentoruralBritain.TheevaluationcouldbereplicatedintheUK,forexampletestingcontrolgroupsagainstfirmsreceivingapplicationadviceandasecondgroupreceivingfullsupport.Ideally,thesegroupswouldberandomlyselected.
Evidence Review: Business Advice - May 2014 49
Study 162: Randomised Control Trial; SMS level 5ThisresearchbyMiriamBruhnandcolleaguesteststheimpactofsubsidisedconsultingservicesonoutcomesforsmallandmediumenterprises(SMEs)usingaRandomisedControlledTrialinPuebla,Mexico.TheprogrammeranfromMarch2008toFebruary2009andwasimplementedbytheStateGovernmentofPuebla.ItwasaimedatincreasingthesizeofSMEs,therebycreatingjobsandeconomicgrowthintheregion.Participatingbusinesseswerematchedwithanappropriateconsultingfirmthatwasaskedtodiagnoseproblemsthatpreventedgrowth,suggestsolutionsandassistwithimplementation.TheStateofPueblathencovered90%ofthecostofservicesformicroenterprises(definedhereaslessthan10employees),80%forsmallenterprises(11-30employees)and70%formediumenterprises(31-100employees).
Fromthe432SMEswhoappliedfortheprogramme(followingstate-wideadvertisements),150wererandomlyselectedintothetreatmentgroup.Theremaining282firmsmadeupthecontrolgroupanddidnotreceiveanysubsidisedservices.Thefactthattreatmentwasrandomisedmakesitmucheasiertoassesstheprogramme’simpact.Specifically,randomisingreceiptallowedtheresearchersto‘control’forbothobservablecharacteristics(e.g.ageoffirm)andunobservablecharacteristics(e.g.theCEO’sleadershipqualities)thatmightaffectoutcomes.Thisalsoavoids‘selectionintotreatment’issuesthatmightariseif(forexample)governmentofficialsdecidedwhogotsubsidisedconsulting,allowingustobeconfidentthatdifferencesinoutcomesforthetreatedgroupreallyarecausedbythefactthattheyreceivedbusinessgrowthadvice.
Inordertoexaminetheeffectsofthetreatment,theresearchersconductedabaselinesurveyofthefirmsinOctober-December2007andafollowupsurveyinMarch-May2009.Theresultsofthebaselinesurveydemonstratedthatthetreatmentandcontrolgroupswerelargelysimilaronobservablecharacteristics,suggestingthatrandomisationwassuccessful.Thebaselinecharacteristicsalsoshowedthatmoreproductive/profitablefirmsinthetreatmentgroup(i.e.thosereceivingadvice)weremorelikelytotakeuptheofferedservices(80ofthe150‘treated’firmsdecidedtotakeuptheservices).Solongasthesefirmsarekeptinthetreatmentgroupandsurveyed,thisdoesnotbiastheresultsbutsimplymeanstheevaluationcapturestheeffectsofbeingofferedratherthangettingsubsidisedconsulting.Therateofresponseforthefollowupsurveywasaround88%,withnodifferencesacrosstreatmentandcontrolgroups,norevidenceofcompositionalshiftsduetofirmsdroppingout(‘attrition’).Sincethetreatmentwassuccessfullyrandomisedandattritionwasnotabigproblem,wescorethisstudyasa5ontheMarylandScale.
Thisfieldexperimentfoundhugereturnstosubsidisedconsulting.Monthlysalesincreasedbyaround80%andprofitsincreasedby120%inthetreatmentgroupcomparedwiththecontrolgroup.Theaverageincreaseinprofitsisestimatedtoliebetween$7,600and$11,000permonth,comparedwithacostof$988permonthfortheconsultingservices.Inourmainreviewwefindthatbusinessadviceprogrammeshaveamixedrecordonimprovingemployment,andinthiscasetheresearchersdonotfindanyincreaseinthenumberofworkersemployedbytreatedfirms–althoughperhapsemploymentmaytakelongerthantwoyearstoadjust.Thecost-effectivenessoftheprogrammeposesthequestionofwhyfirmsdidn’tinvestintheseservicesthemselvesinthefirstplace.Theresearcherssuggestthemostlikelysourceofmarketfailureinthiscaseisfundingconstraints.Anotherpossibleexplanationisthattheysimplyneverconsideredtakingupconsultancyservicesordidnotknowsuchserviceswereavailable.
TowhatextentaretheresultsfromthistrialrelevanttoUKpolicymakers?Theresultshighlightthepotentialforworthwhileinterventionintheareaofbusinesssupport.However,suchapolicywouldbesuccessfulintheUKonlyifitwereaddressingasimilarmarketfailuresuchasalackoffunding/informationforSMEs.SincetheMexicaneconomyissignificantlydifferenttotheUKeconomyitmaybethatthepolicywouldnotbesosuccessful.TheonlywaytoknowforsureifitwouldbeeffectiveintheUKistoreplicatetheprogrammehereasarandomisedtrial.
Evidence Review: Business Advice - May 2014 50
Study 287: Randomised Control Trial; SMS level 5ThisresearchbyJohnBenusandcolleagues(1994)evaluatestwoUSprojectsdesignedtosupportunemployedpeopletostarttheirownbusinesses:theWashingtonStateSelf-EmploymentandEnterpriseDevelopmentProject(SEED),andtheMassachusettsUISelf-EmploymentDemonstration.Bothprogrammesranfrom1989to1991,andtestedwhetherpublicassistancecouldhelpparticipantsmoveintoemploymentorself-employment,raiseindividualincomesandincreasebusinessstart-uprates.Bothprogrammesprovidedamixofexperttrainingsessions,one-to-onesessionstodevelopbusinessideasandworkshopscoveringbusinessplanning,marketing,accountingandmanagement.Inaddition,theSEEDprogrammegaveparticipantscompletingthecoursealumpsumtohelpstarttheirnewfirm.
BothprogrammesweredesignedasRandomisedControlTrials(RCTs),withparticipantsrecruitedfrompeoplereceivingunemploymentinsurance.Arandomnumbergeneratorwasusedtoassignindividualstotreatmentorcontrolgroups,acrossanumberofsitesineachstate.(InthecaseofSEED,755peoplewereassignedtotreatment,withafurther752inthecontrolgroup;numberswasslightlylowerfortheMassachusettsprogramme.)
Randomisationmakesitmucheasiertoassesstheseprogrammes’impact.Specifically,itallowsresearcherstocontrolforparticipants’observablecharacteristics(suchasgenderorqualifications)andunobservablecharacteristics(suchasmotivation)thatmightaffectoutcomes.Italsoavoidsselectionissuesthatmightariseif(say)programmemanagersassignedonlythestrongestparticipantstotreatment.Thisallowsustobeconfidentthatahigherrateofself-employmentinthetreatedgroupisreallycausedbyprogrammeparticipation.OnepotentialproblemwithRCTsispeopledroppingoutofthestudy–theresearchersdealwiththisissueusingadministrativedata,whichallowstrackingevenifparticipantsdon’trespondtofollow-upsurveys.Wethereforescorethisstudy‘5’ontheScientificMarylandScale.
Theresearchfindsthatbothprogrammesincreasedthelikelihoodofentryintoself-employment21monthslater,andacceleratedmovesfromunemploymentintoself-employment(5.9monthsearlierforSEED,2.4monthsfortheMassachusettsprogramme).Bothprogrammesalsosignificantlyincreasedthechancesofbeingemployed/self-employed,whichwas14%higherforSEEDparticipantsand5%higherforthoseintheMassachusettsinitiative.However,neitherprogrammesignificantlyincreasedparticipants’totalearnings,inpartbecausepeoplewerelesslikelytomoveintoaregularjob.Thedifferencesbetweenthetwoprogrammesseempartlyrelatedtowidereconomicconditionsinthetwostates;mostoftheUSwasinrecessionduringthedeliveryperiod,butWashington’seconomywaslargelyinsulatedfromthesemacrotrends.
Whatdoesthismeanforpolicymakers?Thestudiesprovideveryhighqualityevidencethatbusinesssupportinterventionsareeffectiveinraisingemployment,butlesseffectiveiftheaimistoraisewagesandincomes.Whatislessclearisexactlywhichelementsofthetreatmentmixaremosteffective,andhowitcouldbefurtherfine-tuned.Thisisimportanttoknow,giventhestrongerperformanceofSEEDmayberelatedtoprogrammedesigndifferencesaswellasexternaleconomicconditions.Furthertrialswhichrandomisedtreatmentstrandswouldhelpanswerthesequestions.
The What Works Centre for Local Economic Growth is a collaboration between the London School of Economics and Political Science (LSE), Centre for Cities and Arup.
www.whatworksgrowth.org
This work is published by the What Works Centre for Local Economic Growth, which is funded by a grant from the Economic and Social Research Council, the Department for Business, Innovation and Skills and the Department of Communities and Local Government. The support of the Funders is acknowledged. The views expressed are those of the Centre and do not represent the views of the Funders.
May2014
WhatWorksCentreforLocalEconomicGrowth
[email protected]@whatworksgrowth
www.whatworksgrowth.org
©WhatWorksCentreforLocalEconomicGrowth2014