Outcomes Star™ Symposium · developed the Star - Outcomes Star for homelessness was published in...

1© Triangle Consulting Social Enterprise 2015

Outcomes Star™ Symposium

Sara Burns, Director

Triangle Consulting Social Enterprise

AMHOIC, 12 November 2015


The Outcomes Star™ Supporting and measuring change

Empowersservice users and supports

them in making changes

Promotes a consistent, holistic, assets based approach to key-work

Provides meaningful outcomes

information


Outcomes Star symposium – what I’ll cover

• Introduce myself, Triangle and origin of the Outcomes Star

• Overview of the Star as a tool

• Data and use in casework

• Development process and evidence base

• Current developments and key values


Triangle• 1980s: Statistics degree, research and evaluation work

• 1990s: Pioneered outcomes in the UK – listened, learnt, measured, published (but didn’t create the Star)

• 2002-3: A successful collaboration created the Star prototype:

• St. Mungo’s – bottom up process

• London Housing Foundation – sector perspective

• Me, Joy MacKeith and Kate Graham – head and heart

• 2003: Triangle formed – an outcome of the Outcomes Star


The Outcomes Star™ - early steps• Learnt alongside the homelessness sector and further

developed the Star - Outcomes Star for homelessness was published in 2006,

• The Mental Health Recovery Star was developed with 5 Mental Health services and MHPF and published in 2008

• Realised the value of sector-wide tools and had the vision of a suite of Stars

• Many embraced the Star, attitudes to outcomes changed, we responded and this vision is now a reality


Underpinned by a Journey of ChangeDifferent Stars are underpinned by different models of change, but the core journey is based on five steps:

Beleivingand trying things out

Learning

Accepting help

Stuck

Self-reliance


Recovery StarDescription of the Journey of Change


Recovery StarDetailed scales


Recovery StarCompleting the Star

• A collaborative process

• Discuss each point on the Star

• Transfer readings to the Star Chart and capture notes


The power of relationships

Worker Client

Worker and client worlds overlap in conversationWhere the magic happens – the power of relationships to transform

The Outcomes Star sits here and provides a framework for collaboration


What our staff are saying

• Outcomes Star

Tanzi


Data from the Outcomes Star™ Delivering at different levels

• Outcomes for an individual

• Outcomes across a worker's caseload or other group

• Outcomes for a project or service

• Outcomes across a large organisation

Star data is always a jumping off point for further enquiry


Using the Outcomes Star™As the basis for a support plan

The worker and service user can look at the shape together and ask:

What is going well? What are the problems?

What are the priorities?


An example: A family service and the Family StarThe data captured by the Star Online

Table: Change by Domain (All scores) (N=1137)

Domain ‘Starting’ Score Average

Last Score Average

Difference in Ave. Scores

Setting boundaries 4.3 6.4 2.1

Social networks 6.1 7.2 1.1

Supporting learning 4.6 7.1 2.5

Meeting emotional needs 6.0 7.0 1.0

Physical health 6.4 7.8 1.4

Home and money 5.3 6.6 1.3

Family routine 5.0 7.2 2.2

Keeping your child safe 5.7 7.2 1.5

Average change 5.4 7.1 1.7


Star data: macro level –service, cohort and investor data

Aggregated data for 50 peopleat the start and 6months into their journey with project A


Effectiveness:• Overall as a team or

organisation • Within individual

projects• With particular groups

(e.g. gender, ethnicity, age)

• Within particular areas of the Star

• What/where are our key strengths?

What can Star data be used to analyse?Areas for development:• Are there particular areas of

the Star that our clients aren't progressing in?

• Are there particular projects/members of staff that need extra support or training to move clients on?

• Do we need to develop more services to meet needs that aren't being met?

• What learning can we share between projects/individuals and with other organisations?

The nature of your service users:• On entering the service or

programme• On leaving the service• Are we taking on the right

service clients for the support level of each team or service?

• Do we need to review our referral pathways?


An example: St Mungo’s and the Homelessness StarLearnings and insights, from analysis of Star data

• 73% of clients benefit from the service (11% staying stable & 62% progressing)

• Best outcomes are among a minority group with specialist workers (86% made progress) – demonstrating that this approach works

• Women did not progress in the hostels – do we need to rethink provision for women or avoid hostels?

• Outcomes reduce after 12 months, especially without progression and activities in a hostel – highlighting the need for move-on accommodation


An example: A family service and the Family StarThe data captured by the Outcomes Star

Family Star Analysis (to date) sorted by change Service Initial Final Change No. of Stars Notes

Service A 4.5 6.8 2.3 13

Service B 6.1 7.9 1.8 12

Service C 5.0 6.7 1.7 14

Service D 6.4 7.9 1.5 84

Service E 5.2 6.7 1.5 15

Service F 6.7 7.9 1.3 79

TOTAL average for Organisation

5.65 7.3 1.6 217

• Organisational level - comparing services


An example: A family service and the Family StarThe data captured by the Outcomes Star

• Comparing services:


A growing family of StarsA unique set of sector-wide tools

Life Star (Learning Disability)

Work Star Recovery Star

(mental health)

Recovery Star Secure

(secure mental health services)

Community Star

Shooting Star(schools)

Older Person’s StarMy Star

(for children)

Empowerment Star(domestic violence)

Music Therapy Star

Youth StarSpectrum Star

(autism)Sexual Health

StarTeen Star Drug and Alcohol

Star

Well-being Star(long-term health

conditions)Family Star Family Star Plus Homelessness Star Young Person’s Star


Outcomes Star™ Translated across the world


When the Star is – and isn’t - the right tool

Use the Star for:

• On-going keywork

• For at least 6 weeks

• Holistic focus

Use a different tool for:

• Drop-in

• Very brief or crisis interventions

• Work on only 1-2 areas


The Outcomes Star™A distance travelled outcomes tool

• Hard outcomes are just the tip of the iceberg

• Long-term, sustainable change is built by taking steps towards fundamental changes

• In attitudes, skills, motivation and behaviours

• This is what the Star captures


How Outcomes Stars are developedA collaborative process

Scoping

Recruit working group (20 to 30

people) and small steering group

Workshop 1

1st draft of StarWorkshop 2

and second draft4-6 month pilot

Workshop 3 Final version of Star Publish PDFs and add to Star Online


The values of the Outcomes Star™Draws on and extends the principles of Participatory Action Research

Traditional approaches The Outcomes Star

Professional has expertise to effect a cure Focus on service user’s motivation, understanding, beliefs and skills as key factors in determining long-term change

Focus on problem severity Focus on relationship with the issue andstrengths as well as difficulties

Service users cast as passive recipients of help Servicer users cast as active co-producers of change

Empowerment



Either• Patient Reported Outcome MeasureOr• Professional assessment

Collaborative assessment based on• service user’s knowledge and understanding

of themselves and• workers’ observations and reflections on this

person’s behaviour.

The intention is that the process of measurement does not alter what is being measured

The intention is that the process may influence the perspective or both service user and worker

Does not lead to development of a shared perspective

Has the potential to give rise to a shared picture of the issues and next steps

Collaboration




Assessment and measurement is an additional task. The measurement aspect can be resented by workers who can see it as a distraction from the real work

Assessment and measurement is an integral part of key-work not a separate, additional process

Ideas underpinning the tools are not made explicit and shared with service user. Results of assessment may not be shared or shared in such a way as to be accessible to the service user

Ladders and Star visual are a shared, accessible resource supporting reflection and action planning

Integration



An evidence-based toolSummary of evidence and evaluation

• A growing body of evidence that the Star is both a positive keywork tool and a validated outcome measure

• Increasing research and evaluation using the Star is an indicator (eg Troubled Families work in Newham)

• For example:

Petersen et al (2014), Harris and Andrews (2013), York Consulting (2013) found the Star:

• Increases the effectiveness and consistency of keywork

• Promotes client change

“The Star helped develop service users’ resilience and helped workers reflect on their practice, particularly in terms of improving outcomes”

Dickens et al (2012), Killaspy et al (2012) reported good statistical validit, with new editions and quality assurance responding to findings around inter-raterreliability:

“Research into the psychometric properties of the Recovery Star has shown that it has good internal consistency, low item redundancy, high acceptability, good convergent validity and good test- retest reliability, but could be better on inter-rater reliability”


Benefits of the Outcomes Star™

For service users

For services and

keyworkers

Empowering – done with not done to

Visualises progress

Holistic and forward looking

Structures conversations

Encourages reflection

Consistent, person-centric keywork and

service culture

Integral tool – not

adding another process

Provides meaningful outcomes

data

Promotesmulti-agency

working

Ready-made

evidence-based tools


www.outcomesstar.org.ukInformation on research, good practice, access to preview versions and much more (pre-sale)

• http://www.staronline.org.uk/

http://www.outcomesstar.org.uk/

http://www.staronline.org.uk/


Supporting quality use of the Outcomes Star™ -the journey over the past few years

• The Outcomes Star is a copyright tool to support integrity – it can’t be changed

• All workers and their managers need training

• Star License gives access to all the materials – Licence 1 on the Star Online, License 2 the PDFs and license 3 in own software

• The Star is widely used in the UK by charities, housing associations, NHS and local authorities as well as internationally

• Learning with our clients more about what it takes to implement the Star – especially time and integration within systems and supervision


Now and going forward• New initiatives incluidng:

• Implementation tracker

• Data Assurance Tool

• Building an organisation in the UK and learning to support Star take-up internationally

• Increasing engagement with academic research and reporting and use of Star data

• Increasing appreciation of the social enterprise model for sustainability


The values triangle

HeartBelief in people

and the power of relationships

HeadClarity, integrity

and rigour

HandsBeing of service

in the world

BalanceCollaboration

Responsive Agile


Comments about the Outcomes Star™

“A useful tool for self-assessment and identification of needs; its use over time can demonstrate to commissioners that they are getting the right outcomes from the services they fund."

Department of Health, New Horizons: Towards a Shared Vision for Mental Health, July 2009

“I felt fully involved; it was clear and easy to understand and focused on me as a whole person, not just as a problem. It felt like a conversation; felt like the first time someone had really listened to me...”

Service User

“The services are more client-focussed because the Star makes us think about the client and what is changing for each individual.....it has helped staff to start thinking creatively about how to get greater improvements as well”

Keyworker


Research evidence of impact on service users Killaspy et al (2012) reported:

• 85% of service users felt the measure was useful/very useful in helping them and the staff understand how they were getting on and plan the support they needed

Harris and Andrews (2013) identified 4key benefits for clients including:

• “the opportunity to understand their journey in a holistic way beyond presenting need, and the relationship between different domains of their life”

York Consulting (2013) identified many benefits including:

• “Services users engaged with the Family Star, which gives them ownership of the change process. It helped them develop the skills and resilience to manage the issues they face.”


Research evidence of impact on workersBurns, MacKeith and Graham (2008)

• All organisations reported that key-work had improved becoming more focused on service user change, more systematic and consistent and covered a wider range of issues

Study carried out by the University of Newcastle and Hunter New England Mental Health, Australia on the Recovery Star

• Found that clinicians viewed all Recovery Star domains as important for care planning

Harris and Andrews (2013)

• Four key benefits identified including: “A focus on sustainable outcomes rather than limited hard outcomes… a strengths based approach to working with clients”

Killaspy et al (2012)

• 85% of staff who completed the tool with a service user felt it was useful/very useful for care planning


The Outcomes Star™ as an outcomes measurement toolExamples of findings from published studies

MHPF and St Andrews study (2012) - 203 service users:

• High internal consistency, low item redundancy, good responsiveness, 7/10 scales were normally distributed

• Some refinement needed on specific scales (now done in Edition 4)

UCL study (2012) with 172 services users and 120 staff:

• High acceptability, high Test: retest reliability, high convergent validity with a staff reported measure and mixed Inter-rater reliability

• Research informed Edition 4 but the method didn’t examine inter-raterreliability for collaborative readings and has been criticised in 4 peer-reviewed responses


Triangle’s work on reliability Journal of Housing Care and Support (2014)

• Responding to the lack of appropriate methodologies for establishing inter-rater reliability of a collaborative tool

• Case study approach using fictional service user profiles

• Initial results indicate that trained workers can achieve 80% reliability for the Journey of Change stages, when using the Family Star

• Implication is that if workers are trained and tested then Star data showing movement between Journey of Change stages is reliable

Outcomes Star™ Symposium · developed the Star - Outcomes Star for homelessness was published in...

Documents

Transcript of Outcomes Star™ Symposium · developed the Star - Outcomes Star for homelessness was published in...