Measuring what Matters Workbook

{name} Big Local, December 2014

Draft workbook

Early Years Evaluation Team

2014

Contact the team on 020 7520 2548 for more information or by email at biglocal2014@ncvo.org.uk

Introduction

Evaluation is where you use information to make judgements on how your programme or project is doing; it’s about assessing the value or worth of what you’re doing. It can help you in three main ways:

Evaluation involves four stages:

1. planning (what will success look like, what do you need to measure to understand how you’re doing?)2. collecting information (gathering the evidence you need)3. analysing it (figuring out what the evidence tells you, assessing how you’re doing)4. using it (to learn, to improve, to share and celebrate your successes)

This “Measuring what Matters” workbook focuses on the first stage of evaluation – that is, planning. In it we share our ten top tips for Big Local areas to plan how they could assess how they’re doing.

1

Plan and commitPlan ahead and see evaluation as an ongoing process to help track your journey as well as your destination. Commit to reviewing, reflecting and being honest about what works and what doesn’t so that you don’t just prove but also improve as you go along!

Plan ahead so that you can learn as you goEvaluation isn’t just about measuring success, it’s also about learning and helping you improve. Big Local is trying something different. Sometimes it’s about taking risks and trying new ways to tackle old problems. This means we’re all learning together about what works best, so it really helps to build into your plans time to review and reflect on how things are going and what you’re learning.

Remember you can do a lot yourselfSometimes people worry that evaluation is “best left to the experts”. This isn’t necessarily true. It’s important to value the skills and knowledge you already have – listening skills, observation skills, some good story-tellers, and a lot of local knowledge will get you far. Even better if your group has someone who likes to ask difficult questions - “but why?”, “but how?”, and “so what?” are the basis of some of the most interesting evaluation questions!

Planning ahead can save headaches laterPlanning ahead can save you headaches later because it’s really common when people aren’t sure what information to collect, to then collect either too little information, or too much, or simply not the right information you need to answer your important questions.

2

Define success clearlyClearly define success.  You can’t measure what you aren’t able to define.

Is there anything else we want to consider when we think about what success will mean for us?

What else matters to us? – eg, what people feel about what we do? making our million go

further?

Did we stay true to our values? eg, have we

been inclusive?

What difference have we made? to whom? was

it what we’d hoped?

Our activities – what did we do, was it what we intended, have

we met our targets?

Other things that matter to

us?

Success in how we’ve

worked

Success in making a

difference

Success in what we’ve

done

?

3

Define clear changes or resultsDescribe your intended changes clearly and simply. Try to get to the heart of them.  Break complicated changes down and use single, simple change statements to make things easier to measure.

What would you measure in this example?

To make things easier it helps to express the differences you want to make as single, simple statements; to break big changes down; and to try to get to the heart of the change so it’s clear what’s most important.

Activity Activity Change ChangeOut-of-work residents who use the club will feel more confident about seeking

support to find training/employment

Residents will report an increased confidence in accessing job

readiness and employment support services and a more positive outlook on improving their financial situation through work,

having been assisted through a new local high-street service that is friendly and community-led.

?

Members of the

community will help us run the job club, sharing their time and skills with others

We will set up an informal, friendly job club in our Big Local Hub to support long-

term unemployed residents

Out-of-work residents who use the club will feel more

positive about their chances of finding work.

4

Have a story that worksHave a story or map that explains the logic of your journey - a place where you’ve laid out clearly your assumptions about how what you do will contribute to the results or changes you want to achieve.

It’s important to make sure there are no steps missing – sometimes the steps are the things you want to measure.

Looking at your local story map, ask yourself:

(1) Does this tell our story? What would you add to tell a better story?

(2) Does the story ‘work’ – is there a logic to it - or are there steps (changes) missing?

We will offer youth nights at the community centre

Young people will be more employable

We will offer group work sessions during youth nights

at the community centre

Young people will improve their social and

communication skillsYoung people will be more

employable

?

5

Prioritise what matters mostBe realistic and prioritise what matters most.  You may not be able to collect information about everything you do as often as you’d like. Prioritising can help you collect less but collect smarter!

You can’t measure everythingBig Local is a long-term programme. A lot will be done and hopefully a lot will change. You can’t measure and report on all of it. It helps to prioritise and focus on what are the most important elements of your Big Local story. To decide on what matters most, it usually helps to focus on your goals.

Focus on what most helps you achieve your goalsOften when asked to measure their progress, people focus on how much they’ve done, what activities they’ve done. That information is really important, and you may find it’s what a lot of people are interested in – because telling others how many events you’ve held or how many people attended these gives a very tangible way of showing that you’re getting things done. However, you will also (and more importantly) want to know if your activities are moving you closer to your goals, to making the differences you want to make.

Be realistic and try to be proportionatePractical considerations also shape what you measure.

Some changes aren’t in your direct control or might happen after your work has ended. Focus first on the changes that are in your control.

Not all the changes you make may be equally important. Some things may simply be too difficult or time-consuming so you need to consider the next best thing that you can

realistically measure, remembering that what you do to measure your success shouldn’t get in the way of doing the work you want to do.

6

Ask ‘how will we know’ questionsWhen you’re thinking about assessing whether you’ve made a difference, it helps to think in advance about the signs of the change you could look for.  Asking yourself ‘how will we know?’ for each change really helps.

Asking “how will we know questions” can help you think about the signs you might see, things you could count, or things people could tell you that might help you evidence the difference you’re making. For instance … say one of the big, long-term changes we want to make is that we want to improve our local environment so that the whole community can benefit from it. We have a large park on our doorstep but it is under-used for a variety of reasons. We broke down our big change (improved local environment) into three specific changes that felt most important to us. We said, “We will know we’ve made progress with this change if ...

Our park is cleaner Our park is safer for families to use Our park is better used by the whole community

Then we asked, “how will we know these changes are happening? What will we count, measure or observe?” We ended up with the beginnings of a plan as below.

Big Change: Improved local environment for the whole community’s benefitSpecific changes How will we know?

What will we count, measure or observe? What might others tell us?Our park will be cleaner Level of littering

Level of graffiti Level of complaints Level of use of dog litter bins How clean people feel the space is

7

Be lawyers not scientists – building a caseThink of yourselves as lawyers (collecting evidence to make the strongest case you can), rather than scientists (collecting proof).  To make the strongest case it helps to try and measure things from more than one angle – using different types of evidence or perspectives to explore the same thing.

When we talked about signs of the change, we mentioned you might look for things you can see, things you can count or measure, or things others might tell you. All three could be valid and if you combine one or more you could make a stronger (more credible) case. Likewise you could think of assessing change from three different viewpoints to make a stronger case.

eg, monitoring, official records, surveys

eg, feedback forms, testimonials, interviews,

case studies, questionnaires

eg, observation, film, photos

your changewhat you can record/count

what others tell you

what you see or feel is

happening

8

Recycle and re-useWhen it comes to collecting information or evidence there are many toolkits, ideas, questionnaires and forms you could use, and there may also be others collecting information you need which would save you collecting it yourself.

Is there an easier way? Has someone else done this before?Always check if there’s an easier way to collect the information you need. This way you can avoid reinventing wheels, collecting information that’s already out there somewhere, or duplicating what someone else is doing. Check online or with other organisations working in your area about how they collect information. There are lots of great questionnaires, forms, ideas out there, many of which you can find by searching online. You can often adapt what someone else has tried and tested to collect information you need.

Has someone else got this information or could they collect it for you?Be a recycler! It’s also worth checking what information is already being collected about your community and seeing if you can use it. Your local council, NHS and voluntary organisations and schools may have really useful information you could use. Also, sometimes if you have a vital question you want to ask your community, one of your partners may be willing to add that to their own survey or forms and to share information with you.

Note of caution – do a test run first!If you are using or adapting someone else’s questionnaire or feedback form or using questions others have found helpful, always test them first with a few people from your target audience. What works in one setting or with one group of people, may not work in another.

9

Think about how you’ll use information before you collect itAlways think ahead to what you’ll do with information BEFORE you collect it, and try not to collect information unless you know you will use it.

Think in advance about how you’ll store information you collectDon’t collect personal information if you don’t need it. If you do, make sure you’re able to keep any personal information in line with data protection guidelines.

Think in advance about how you’ll analyse information you collectRemember that once you’ve collected information you have to be able to analyse it and use it. For instance, a survey might seem a great idea until you realise you don’t have time to enter 300 questionnaire responses into a computer, and you’re not sure how to make sense of all the questions people answered in words instead of numbers!

Think in advance about how you’ll use information you collectIt doesn’t make sense to collect information you won’t use “just in case it might come in handy one day”. Think carefully about how you want to use information in advance or you could end up collecting information you can’t use because you have too much of it and can’t make sense of it, because you collected it too late to be of use to you, or because it doesn’t really answer your questions.

10

Make the most of being part of a national programmeRemember you’re part of a national programme. Local Trust and its national partners can support you in measuring your progress in a number of ways.

Guidance, toolkits and case studies The Big Local Annual Review process includes tools to help you review how well you’re doing and to help you

assess yourself as a local partnership. Check Local Trust’s website or contact the team for a copy of the “Big Local: Checking Your Progress” guide or

the equally useful “Checking Your Progress” toolkit which contains lots of ideas and some useful sample documents

There are lots of Big Local case studies on the Local Trust website to inspire you and give you ideas for how you too could use stories to share what you’re doing and the progress you’re making.

On the horizon Local Trust plans to set up evaluation workshops to help areas explore more of the “how to” side of things.

Training on measuring impact may also be made available in the near future. The School of Public Health Research is currently researching the links between community control and health and

wellbeing. In time they hope to share useful data and ideas on how areas could look at the impact of Big Local on health in its broadest sense.

The Early Years Evaluation Team will be producing a database of changes and indicators (signs of change) that areas could visit to look at what others are doing and to get ideas for how to measure their changes over time.

Appendix 1. Reminder of our ten top tips for measuring what matters

11

Plan and commit to ongoing reflection and review

Define success clearly

Have clear changes and break them down where you can

Have a story that works – that links your activities and your results

Prioritise what matters most

Ask ‘how will we know’ questions

Be lawyers not scientists – build a strong case

Recycle and re-use

Think about how you’ll use information before you collect it

Make the most of being part of a national programmeAppendix 2. What a measurement plan might look like for a long-term goal

12