Dr Sally Boa and Dr Joan Murphy Professor Pam Enderby Funded by NHS Education Scotland Conducted by...

Dr Sally Boa and Dr Joan Murphy

Professor Pam Enderby

Funded by NHS Education Scotland

Conducted by Talking Mats Limited

© Talking Mats Ltd 2014

Background

A Right to Speak (Scottish Government 2012)1. AAC services to demonstrate the effectiveness of AAC interventions.

2. National statistics on AAC to be gathered by relevant agencies to support future gathering of cost effectiveness data on AAC to ensure that AAC funding is sustained in the longer term.


Two stage project

1. Scoping of current methods of measuring outcomes

2. Testing the validity and inter-rater reliability of TOM AAC (Enderby et al 2006)


Main points arising from scoping project

• Agreed on what we mean by ‘Outcome Measure’

• Agreed on what is the best measure to use in AAC

• Identified the need for a measure to be workable across agencies


Definition of an outcome measure

Outcome Measures:Should measure changeCan be repeated over timeCan be used across clients, settings and disciplinesCan be used to give feedback to services, professionals, carers and clients (about what works as well as what doesn’t)Should be robust/standardised/well respectedShould inform dischargeShould be short and straightforward to useShould allow for comparison with different types of AAC/No AAC


Main contender for outcome measure in AAC

• TOMs AAC• (still in development – needed to be tested for

reliability and validity)


Stage 2

• Testing TOM AAC for validity and inter-rater reliability


Therapy Outcome Measure (TOM)• An Outcome Measure used by many rehabilitation

professionals to measure the impact of their interventions

• Based on the World Health Organisation’s International Classification of Functioning, Disability and Health (WHO ICF, 2001)

• Administered following assessment/intervention of an individual by a professional

• Individuals are rated using an 11 point ordinal scale with 6 defined points. People are rated in relation to four descriptors: Impairment, Activity, Participation and Well-being.


Normal limits0 0.5 1 1.5 2 2.5 3 3.5 4 4.5 5

MildProfound Severe Severe/ Moderate Moderate

Therapy Outcome Measure

Impairment/ Body

Disability/ Activity Participation Well-being

Carer Well-being Date

AdmissionIntermediateFinal

Reason for discharge


TOM AAC

• Adapted scale for use with an AAC population (children and adults)

• Domains of activity and participation can be rated with and without AAC


Recruitment of AAC professionals(n = 29)

Profession Numbers

SLT 16

Teacher 7

Rehab engineer/technologist 2

Social work 1

OT 1

Physio 1

Nurse 1


Validity testing

• Training provided• Each professional was asked to rate 10 AAC

users using TOM AAC• Feedback provided through:– Online survey– Focus groups


Findings from the survey

Coherence and relevance of the descriptors•Strong agreement that the descriptors in TOM AAC were appropriate and relevant for use with an AAC population•Respondents reported a good understanding of the difference between impairment, activity, participation and well-being

Usability of the measure in practice•Most agreed that the measure was quick and easy to use •Most agreed that the TOM AAC would help them to report on outcomes on their work with individuals who use AAC


Findings from Focus Groups

• TOM AAC was straight forward to use– “I think it’s really useful. I think it’s a really quick,

useful tool.” (Participant from focus group 3)

• Comments about wording of the impairment sections - a distinction between ‘expression’ and ‘comprehension’ rather than ‘speech’ and ‘language’


Strong face validity – changes made to descriptors of TOM AAC

• Wording changed to ensure consistency across the descriptors

• Within the impairment section, distinction was made between ‘expression’ and ‘comprehension’ rather than ‘speech’ and ‘language’.

• More detailed explanation was also provided for each descriptor to increase the clarity of meaning for each section


Inter-rater reliability testing

• AAC professionals (n = 17) provided with ten case histories to rate using TOM AAC

• Scores collated and analysed using Intraclass Correlation Coefficient (ICC)


Findings

• Agreement was generally acceptable–highest for physical impairment (0.887),

comprehension (0.831) and wellbeing (0.816)

– the lowest was for expression (0.486)


Inter-rater reliabilityTOM domain ICC

Physical Impairment 0.887

Cognitive Impairment 0.726

Sensory Impairment 0.605

Expression 0.486

Comprehension 0.831

Activity 0.622

Participation 0.703

Wellbeing 0.816

© Talking M

ats Ltd

Limitations

• We may not have provided sufficient information in case histories five and six, as ratings for these case histories gave the largest range between individuals.

• The domain of ‘expression’ appeared to produce the most variation between raters – need for clearer definition of descriptors

• Further investigation needed around ‘expression’


With adaptations


Greig


Example


Summary

• TOM AAC has been tested for face validity and inter-rater reliability

• TOM AAC has the potential to provide a quick, simple and reliable for all AAC professionals

• TOM next edition will include AAC


ReferencesCommunication Matters (2012) http://www.communicationmatters.org.uk/sites/default/files/downloads/standards/aac_outcome_measurement_sept_2012.pdf

Framework for Measuring Impact http://www.measuringimpact.org/

Scottish Government (2012) ‘A Right to Speak’http://www.scotland.gov.uk/Publications/2012/06/8416/0

Lacey, A. & Luff, D. (2001). Trent focus for research and development in primary healthcare: An introduction to qualitative analysis. London: Trent Focus.

Khangura et al 2012 Evidence summaries: the evolution of a rapid review approachhttp://www.systematicreviewsjournal.com/content/1/1/10

TOMS: Enderby P., John A., & Petheram B. (2006) Therapy Outcome Measures forRehabilitation Professionals: Speech and Language Therapy, Physiotherapy,Occupational Therapy, Rehabilitation Nursing & Hearing Therapists. Second Edition, JohnWiley & Sons Ltd.

CODES: http://keycomm.weebly.com/codes-framework.html

Shrout, PE, Fleiss, JL. Intraclass correlations: Uses in assessing rater reliability. Psychological Bulletin 1979; 86(2): 420-428


http://www.communicationmatters.org.uk/sites/default/files/downloads/standards/aac_outcome_measurement_sept_2012.pdf




http://www.measuringimpact.org/

Dr Sally Boa and Dr Joan Murphy Professor Pam Enderby Funded by NHS Education Scotland Conducted by...

Documents

Transcript of Dr Sally Boa and Dr Joan Murphy Professor Pam Enderby Funded by NHS Education Scotland Conducted by...