CTA Scotland ConferenceProve It! Effective Evaluation for the Community Transport Sector

Jonathan CoburnAssociate Director, EKOS

About the session

• An introduction to evaluation theory and practice• A wide-ranging overview

• Understanding what evaluation is and is not

• Giving an overview of the evaluation process

• Looking at some of the key concepts and terminology

• Exploring the main approaches and methods

• Some practical tips for evaluation

• ... all in the next 45 minutes!

Understanding Evaluation

What is evaluation?• Evaluation is a part of everyday life• It is about making judgements (criteria vs values)• It is comparative by nature• It is about using evidence• It is about cause and effect• It is about drawing on different perspectives• It is about critical reflection• It is about learning (and doing!)

Whatever it is, it’s becoming more prominent in all of our work ...

Evaluation in an era of accountability

•A period of financial austerity•Increasing competition for public funds•Tougher choices for funders/investors•A time of reckoning for all

= must demonstrate efficient and effective service delivery and prove your worth

Evaluation can help you to ...• Make your work more effective and efficient • Anticipate problems and deal with them early on• Identify gaps, improvements, and new directions• Celebrate progress and achievement• Motivate staff and volunteers• Develop your organisational memory• Tell the story more clearly or convincingly to others• Make the case for new/continuing resources

Why then is evaluation treated with such dread? Some myths

•Its a negative experience

•Its only for funders

•Its just dwelling on the past

•Its ‘damn lies and statistics’

•Its about making your case

•Its a wasteful distraction

•Its something done to you

The reality is, evaluation can either be ‘carrot or stick’

A ‘stick’ used to beat underperforming organisations with

A ‘carrot’ used to entice learning organisations committed to change

Two main roles for evaluation• Formative (or process) evaluation

• Takes place during the life of the project• Provides feedback on process and outputs• Examines progress towards your objectives

• Summative (or impact) evaluation• Takes place at the end of the project• Makes a judgement about the success of the work• Typically looks at outcomes and impacts

• .... but the distinction is not always as rigid as the above suggests

Evaluation is best viewed as part of a process

The Basics of the Evaluation Framework

Good evaluation depends on setting a clear direction

Mission (or overall aim). Why an organisation or project exists. The broad effect it wants to have. A summary of the difference it wants to make.

Specific aims. Statements about the differences that an organisation hopes to bring about, typically for its users. There will usually be several aims.

(SMART) Objectives. The areas of activity or practical steps that you will take to accomplish its aims.

Linking what you expect with what happens ... an evaluation framework

Mission

Specific aims This is why we do it

ObjectivesThis is what we do

Outcomes. The changes, benefits, learning or other effects that result from what the organisation makes, offers, or provides

Outputs. The things that result from an organisation’s activities that are intended to achieve change (outcomes).

Setting indicators ... going back to our basic evaluation framework ...

Mission

Specific aims This is why we do it

ObjectivesThis is what we do

Outcomes

Outputs

OutcomeIndicators

OutputIndicators

Using indicators• The different types of indicator are often confused

• Output indicators – assess the things that you do or accomplish and measure progress against objectives

• Outcome indicators – assess the changes, learning, or other effects and measure progress against aims

• 3 easy steps to developing your indicators• Ask yourself what output/outcome would look like if

you achieved it – give potential indicators

• Narrow these down to 2-3 key indicators

• Check appropriateness with others

Your output indicators

• Output indicators may be set for the following• Quantity – the things you deliver• Take up – the people/orgs that use them• Access – your ability to reach certain target groups• Quality – the extent to which standards are reached• Satisfaction – the ability to meet service user needs

• When selecting your output indicators• Be realistic about what you can measure

• Choose the most appropriate indicators

Your outcome indicators

• How are you actually bringing about change?• Outcomes are usually expressed as: the extent of ...; the

level of ...; the degree of ...; the ability to ...; etc.• The challenge is to put measurable indicators to this ...

• Positive or negative• Quantitative(countable) or qualitative (views)• Intended or unintended• Hard (tangible) or soft (pointers/proxies)

• When selecting your outcome indicators must prioritise only significant changes and make them simple

Everything is now about outcomes

• An increasing move in the public sector towards setting high level outcomes to drive performance

• These outcomes are now included in every Council’s strategy (e.g. SOAs) and the plans other public services

• Increasingly organisations are being asked how they fit and what they contribute to local area outcomes

• Funders and service commissioners are increasingly talking about outcome-based funding and contracts

• They are increasingly interested in the long-term outcome or impact of your work

A growing number of frameworks are focused on outcome/impact

• There are many new outcome/impact measurement frameworks that are now coming to the fore

• ... some are holistic e.g.• Social Accounting and Audit• Social Return on Investment

• ... some are specific e.g.• Volunteering Impact Assessment• LM3 – Local Multipliers 3• Prove It

SROI is the latest game in town

• Builds on established tools – CBA, Social Accounting, ROI• Outcome-based – all types of outcomes• About understanding and telling the story of change• A structured and systematic process – an Impact Map• Based on the perspective of stakeholders• Seeks to value the things that matter• Attempts to measure the unmeasurable - proxies• To be used with caution – not the only game in town

The Evaluation Process and Tools

The evaluation process

Manage the Evaluation

•Team•Timescale•Budget

1. Engage stakeholders

2. Establish focus 2. Collect

data 3. Analyse/ interpret 4. Use

findings

•Establish a Framework

•Determine use/users

•Define scope and purpose

•Select indicators

•Determine design

•Identify sources

•Select method(s)

•Determine sample

•Design instruments

•Pilot instruments

•Conduct fieldwork

•Process data•Analyse data•Interpret data•Develop main findings

•Explore implications

•Produce report•Share findings•Use in decision making

•Determine next steps

Methods of data collection• There are many ways of collecting required data,

typically based on social and market research methods• The most commonly used techniques are

• Monitoring records• Observation• Survey questionnaires• Interviews (face-to-face and by telephone)• Focus groups

• Other participatory methods often provide a creative, visual , and rewarding alternative

Monitoring data (records)•Existing sources of data or documentation routinely collected

•Examples include client database, timesheets, satisfaction forms, etc.

•Best if can provide a historical series of data otherwise unavailable

•Gives a first-level indication of progress and performance

•Often inflexible, incomplete, or inconsistently collected

•Usually a useful starting point

Observation•A holistic method providing a vivid picture of the project

•Complex process, with different forms of participant observation

•In this case the evaluator is the research instrument (reliability)

•Can be more or less structured depending on the purpose

•Good at gaining insights into group interactions and behaviours

•Often overlooked and therefore underused

Survey questionnaires •The most commonly used method in evaluation research

•Can be flexibly administered on-site, by post, by email, online, etc.

•Able to compile factual information across many respondents

•Can combine open and closed questions to good effect

•Less useful in exploring complex issues where views are important

•Questionnaires are not a straightforward option

Interviews (face-to-face or telephone)•Provides opportunity for personal, detailed interaction with interviewee

•Can be more or less structured depending on needs

•Used typically with complex issues and where in-depth views required

•Either telephone or face-to-face can be used depending on needs

•The success of interviews depends on the skills of the interviewer

•There will be differing resource implications

Focus groups•Collect data through group interaction on an agreed topic

•A tried and tested market research tool for gauging audience reaction

•Often used in combination with other methods

•Provides an opportunity to get a collective view and develop ideas

•The success of focus groups depends on good facilitation skills

•Choices in terms of focus, participation, and venue

Other creative and visual methods•A set of techniques often referred to as participatory appraisal

•Designed to work with groups of varying abilities and backgrounds

•Examples include graffiti walls, time lines, body maps, H-Form, photography, video booth, mapping networks, arts and drama, etc.

•Inclusive, participative and democratic approach

•Flexible and appropriate in many group settings

Choose the right tools ... and use them well

•The choice of tool(s) depends•On the questions to be answered•On the time available•On the resources available•On the technical skills of the team•On the research participants•On the intended audience

•Make sure that the mix of tools you use are appropriate

•Make sure that you have the skills, resources, and practice

Carrying Out and Using the Evaluation

Decide how much of the evaluation work to take on

yourself

Don’t underestimate the added workload

In-house vs external evaluation• Evaluation always works best when you build it into your

everyday activities (= self-evaluation) but there are choices

• In-house evaluation• Knowledge of the organisation• Organisational learning• Commitment to implement findings

• External evaluation• Specialist skills• Independence• Fresh perspective• Wider insights

• In practice, your use of external evaluators is likely to be determined by resources rather than preference

Make time for analysis•Summarise and organise your data in a logical, ordered way

•Analyse the info as you go along, don’t wait until the report is due

•Break the analysis down into manageable chunks

•Approach the analysis of quantitative, qualitative, and visual data in very different ways

•A systematic and rigorous process is required throughout

Interpretation ... focus on the questions that really matter

• Structure your thinking around the key questions• ... the things that you really should know (learning)

• What happened and why?• What has worked well and not so well?• Did we achieve what we set out to do?• What can we learn from this?

• ... the questions others will want answered (accountability)• Was the project implemented as planned?• Has the money been spent in the right way?• Did the project achieve its aims and objectives?• What difference and contribution has it made?• What proof is there of this?

• Your interpretation must be based on the facts

Make time to produce a compelling report ... some tips

Be well prepared • Collate the inputs, know the audience,

get a structure, leave time

Keep it simple• Focus on essential info only, tell a clear

story, and use headings, bullets, etc.

Make the meaning clear• Present findings in context, comment

on reliability, provide clear messages

Make it user friendly• Plain English, colour, graphics, charts,

case studies, quotes

Sharing the findings in other ways• There will be different audiences interested in the

evidence, not all of whom want to read a weighty report • Think creatively about ways of sharing the findings

• Summary report for circulation• Annual report• ‘Open day’ type events• Conferences• Feedback meetings/presentations• Newsletters• Websites• DVD• Radio interviews• Correspondence to key stakeholders

• Remember to adapt your approach and communication channel to each audience

Making use of the evidence•Share your findings to keep or get important decision-makers on board

•Begin dialogue with service users – ensure that their voice is important

•Use your findings to inform service design

•Don’t be afraid to make brave decisions where the evidence dictates

•Use the evidence as justification for funding applications or tenders

•Use the evaluation where you can to raise your profile

Embedding self-evaluation and maintaining momentum

• Makes sure that everyone understands and buys in• There must be commitment from the top• Someone must lead and take responsibility• It works best when built into activity, meetings, job roles• Ongoing communication to update and refresh • Use any early evidence to achieve some quick wins• Ensure that you build in the resources to do the job

Take the first steps today to improve your evaluation practice

Some resources to help you along the way

Evaluation Support Scotlandwww.evaluationsupportscotland.org.uk

Charities Evaluation Servicewww.ces-vol.org.uk

Prove and Improvewww.proveandimprove.org

Monitoring & Evaluation Resource Guidehttp://www.ces-vol.org.uk/downloads/meresourceguidecompletetext-574-582.pdf

Thank you!