Multi-strand initiatives: using theory of change evaluations

Alan Dyson: D.A.Dyson@manchester.ac.ukKirstin Kerr: Kirstin.Kerr@manchester.ac.uk

Karen Laing: k.j.c.laing@newcastle.ac.ukLiz Todd: liz.todd@newcastle.ac.uk

Introduction

• Aims• The team – Alan Dyson, Karen Laing, Liz Todd• Plan:

Introduction to theory of change evaluationTheory of change in actionUsing evidenceFinal thoughts

Dyson & Kerr

Cummings, Dyson, Muijs, Papps, Pearson, Raffo, Tiplady & Todd

Cummings Dyson & Todd

Full service extended schools national evaluation

Colleen Cummings, Alan Dyson, Ivy Papps, Daniel Muijs, Diana Pearson, Carlo Ruffo, Lucy Tiplady, Liz Todd: Newcastle University, University of Manchester, Tecis Ltdhttp://www.ncl.ac.uk/cflat/ESnetwork.htm

Theory of Change

A systematic and cumulative study of the links between activities, outcomes and context of the initiative

Fullbright-Anderson, Kubisch and Connell, 1998: 16

5

Theories of Change

• Popularized in 1990s to capture complex

initiatives

• Outcomes-based

• Causal model

• Articulate underlying assumptions

What is involved in theory of change….

• Qualitative and quantitative data collected over time

• Identify early, intermediate and long-term outcomes

• Theorise retrospectively and prospectively

The starting situation

What it’s like now – and why

Strands of action

What we are going to do about it

Intended outputs

What we will need to do differently

Steps of change

How things will change (for beneficiaries)

Intended outcomes

How the starting situation will change for CYP

Steps of change

How a Theory of Change approach can help

• Working together• It is dialogical, involving conversation and

negotiation• Enables continuous feedback• Can feed into project planning• Provides guidance about data collection

methods and self evaluation

Asking the right questions

• What is the situation you face?• What needs to change?• How will these changes be made?• What actions will you take?• What effect will those actions have? On

whom?• How will you know if change is happening?• What will happen for person A, person B etc

Lack of jobs / unemployment

Inability to connect to local service jobs

Reluctance to commute (locally/into Newcastle)

Resentment – feeling that something has been taken away from the community (not always sure what)

Lost generation of grandparents / parents

Loss of area esteem – some fracturing of community (e.g. parts of the area ‘in work’ resented)

‘Unhealthy place’ physically & mentally

Presenting problems:

Youth unemployment Youth disorder Poor mental and physical health – unfit families Children starting KS3 below level 3 Poor early language development ‘Troubled’ families High benefit claimant rates – disability and JSA 

Underlying problems:

In some cases:

Parents have low expectations of their own children

Families don’t know how to be good parents or find it too hard

Families abdicate responsibility to services or refuse to engage with services

The hard to reach remain hard to reach

Children have aspirations but not the opportunities / capacities / support to act on these

Community not supportive of aspirations

Opportunities that are available are not recognised and acted on

 

A Children’s Community will seek to achieve the following outcomes for the area:

Reduction in presenting problems (no noticeable difference to other areas of N. Tyneside)

The area has the same opportunities and support structures as more advantaged areas. It provides the same ‘offer’ in terms of life chances (i.e. employment/ parental support/community support).

The community supports the realisation of aspirations. It establishes & sustains another way of behaving – i.e.:

-the community becomes self-empowered

-parents expect better for their children and act on this (services stop substituting for parents)

-the community is ‘fit for life’ mentally & physically

-there is a positive community identity linked to employment

-community members can connect to ‘wider world’ contexts and opportunities

There is upward mobility, both in and out of the area

The area attracts and sustains employment

 

 

Ship Building (Lost) 100% employment (Lost) Traditional gender roles (Maintained) Self-contained community (Maintained) Sufficient community fabric (Fractured)  

Past context:

Recent context:

Resulting in:

Pre

sen

tin

g a

s:

The Children’s Community

Suggested core strands of action for the Children’s Community:

1. Getting things right early – e.g. speech, language communication & parenting. Aim to set children on positive pathway rather than addressing deficits later

2. Being fit for life – e.g. making sure children are healthy (physically and mentally) and are able to engage with opportunities

3. Realising aspirations & behaving in new ways – e.g. empowering the community to support children to achieve their aspirations and to do things they have not done before – making the community positive for residents and attractive to others (inc. employers).

The perpetuating situation (c.2003)

Getting things right early actions:

Transition support

Support for parenting

High quality pre-school provision

Higher attainment in primary

schools

Higher attainment throughout schooling

Access better post-school

opportunities (including jobs)

Parents with higher aspirations for their children

Higher aspirations

Good schools capitalise on better

prior attainment

Better opportunities

available in area

Better parenting in the home

Children with capacity to take advantage of

schooling (confidence & resilience)

Children doing well through transition

Getting things right early

Mentors:

access children in primary schools identify children who may have

problems later but who do not have additional provision because they are coping in primary school (Jan-July)

provide nurturing relationships and establish friendship groups

liaise with parents offer summer school in secondary

school identify issues secondary school

needs to be alerted to

Children learn to trust mentors who become ‘the trusted adult’

Children become familiar with secondary school & become more confident about transition

Parents ‘buy into’ the programme & encourage their children to participate

Secondary school prepares appropriate action Children’s confidence grows

In secondary school (autumn term), mentors:

maintain contact with children (eg through breakfast clubs)

monitor children in school to identify emerging problems

equip children with coping skills (eg anti-bullying strategies)

pass information onto guidance team (& Relate counsellor in Churchill)

wean children off support so they can cope alone

Children develop coping strategies

Secondary school implement effective support strategies

Children cope with secondary school without support

Children cope with transition with support

Transition mentors

Guidance team implement strategies

Drawing conclusions

• Formative:– ‘Initiative X is triggering changes a.b.c….which are

likely/unlikely to lead to outcomes 1, 2, 3….’

• Summative:– ‘Initiative X triggered changes a.b.c….leading to outcomes 1,

2, 3….’ – ‘Initiative X also triggered changes l, m, n….which may yet

lead to outcomes 1, 2, 3….’

• But: • ‘Initiative X# may not trigger changes a.b.c….or lead to

outcomes 1, 2, 3….’

Drawing conclusions

• Building a theory:– Initiatives X, X#, X##...triggered similar changes

leading to similar outcomes’– Initiatives of type X are likely to lead to similar

outcomes• ToC and ‘what works’• Using a D & R approach