5/6/2011

1

Usability, Reliability, and Validity of Remote Autism Diagnostic Observation

Schedule (ADOS) Module 4 Administration

Jamie L. Schutte, MS, CRC, Michael McCue, PhD,

Bambang Parmanto, PhD, & Wayan I. Pulantara, MS

University of Pittsburgh

Outline

I. Background

– Adults with High-Functioning Autism

– Diagnosing ASD in Adults

II. Remote Assessment

III. Methods

I. ADOS

II. VISyTER

IV. Clinical Usability

IV. Procedural Usability Concerns

V. Maintaining Standard Administration Procedure

V. Reliability

VI. Validity

5/6/2011

2

Autism Spectrum Disorders

• ASDs are developmental disorders

• ASDs are usually diagnosed in early childhood

– Behavioral observation

– Developmental history

Statement of the Problem

• From 2002 to 2006, ASD prevalence increased an average of 57%

• Though ASD onset is prior to age 3 years, it is a lifelong disability

• Adults with ASD may be currently undiagnosed or misdiagnosed

3,326

8,450

15,670

1,407

3,825

10,140

19,587

-

5,000

10,000

15,000

20,000

25,000

2005 2010 2015 2020

Number of individuals diagnosed with autism in Pennsylvania ≥ 21 years of age

no new diagnosis assuming new diagnoses proportional to current rate

5/6/2011

3

Problem (cont.)

• The Autism Diagnostic Observation Schedule (ADOS) has recently become part of the gold standard in ASD diagnosis

• ADOS training opportunities are relatively limited and potentially difficult to access

• Proposed Solution: Remote ADOS module 4 administration system

Telehealth

Telemedicine

Telerehabilitaiton Telepsychiatry

Teleassessment

Telehealthcaree-Health/ Education

The remote administration of systematic procedures for observing and describing

behaviors with the aid of numerical scales or fixed categories, through use of interactive

videoconferencing between a client (and usually a technician) at a local site and a remotely located

assessment expert.

5/6/2011

4

Remote Assessment

Strengths

• Improved access

• Improved quality

• Reduced health-care costs/cost containment

Potential Limitations

• Policy

• Reliability and Validity

ADOS Module 4

• Adolescents/adults

• Verbally fluent

• 10-15 activities with focus on behaviors important in the diagnosis of ASD

• Cutoff scores for both a narrow diagnosis of Autism and a broader diagnosis of PDD

5/6/2011

5

Versatile and Integrated System for Telerehabilitation

(VISYTER)

Clinician-side

5/6/2011

6

Figure 5• Screenshot of the Autism Diagnostic Observation

Schedule module 4 remote administration system, clinician view.

Client-side

5/6/2011

7

5/6/2011

8

STUDY 1 – CLINICAL USABILITY

Clinical usability and participatory development

Develop a remote administration procedural manual

Usability Study

• Cognitive Walkthrough – Participants: 5 clinically reliable ADOS administrators

– Participants performed a remote ADOS module 4 assessment on a volunteer (mock client)

• Psychometric standardization:– “Were you forced to break standard face-to-face

administration procedure? (Yes or No.)

– If yes, how? (Narrative response)

– If yes, to what degree? (Likert scale, 1 to 7)

5/6/2011

9

Results:Procedural Usability Concerns

• Feedback:

– Typos on scoring form

– A glitch in the system that wipes out notes if the administrator forgets to his “save” regularly

– The problem of visual stimuli not staying on the screen simultaneously with the administrator taking notes

• PSSUQ:

Results:Maintaining Standard

Administration Procedure

• Of the 14 ADOS activities administered, only 6 were noted by at least one subject to force breaking standard, face-to-face administration

• Mean ranking was 2.4, indicating that generally breaks in standardization were minimal

5/6/2011

10

Conclusions

• The usability study identified major system deficits and elicited suggestions for making the system more usable

• A variable affecting the usability of the system was familiarity, proficiency, and comfort with computer technology

Conclusions, cont.

• When examining an alternative administration method for a previously standardized assessment, it is important to evaluate whether or not the new administration has equal reliability and validity to the way the evaluation was originally standardized

• In this study, the method of evaluating potential drift was a semi-structured interview with open- and close-ended question administered after each item

• This method could be considered a starting point for other researchers interested in evaluating whether or not remote administrations of standardized procedures are valid

5/6/2011

11

Remote ADOS

(n=14)

USABILITY, RELIABILITY, AND VALIDITY OF REMOTE

AUTISM DIAGNOSTIC OBSERVATION SCHEDULE (ADOS) MODULE 4 ADMINISTRATION

Study 1: Clinical Usability Study 2: Reliability Study 3: Validity

Given the number ofadults with autism in need

and the lack of trained clinicians, assessment services are

not available at thelevel required.

Adults with Autism Spectrum Disorders

(ASDs)

• Diagnostic criteria:

•Impairment in social interaction

•Restricted and repetitive interests and

behavior

•Impairment in communication

•“Urgent public health concern”

• 57% prevalence increase from 2002-2006

•Life-long disability, despite clinical and

research focus on children

Diagnosing ASD… in Adults• No genetic or medical test

• Adults may be undiagnosed or misdiagnosed

• Behavioral presentation – different in adults vs.

children

• Developmental history

• Onset prior to 3 years

• Difficult to access for adults

• Rule out – MR, psychosis, cognitive or

psychiatric disability

• Standardized assessments – adults receive little

attention

Autism Diagnostic Observation Schedule

(ADOS)

• Important component of “gold standard” ASD

diagnosis

• Semi-structured, standardized assessment of:

• Communication

• Social interaction

• Play or imaginative use of materials

• Must be administered by an experienced and

highly trained clinician

• # and location of trained administrators does

not meet need

Study sample

N=14

Face-to-face ADOS

(n=14)

Remote

ADOS

Face-to-face

ADOS

Was

hout

-3 m

onth

s

Remote ADOS

(n=14)

(Random assignment)

Teleassessment

Remote ADOS Module 4 Administration System

Within-subject crossover design:

Aim 1: Clinical usability and participatory development

•Phase 1 – an experienced ADOS administrator (Schutte) will participate, from the beginning, in the design and development of the ADOS teleassessment system

•Phase 2

•Methods

•Participatory study design

•Experienced clinicians who are not part of he development team will conduct a usability evaluation

•Participants – clinically or research reliable ADOS administrators with at least 1 year experience

Aim 2: Develop a remote administration procedural manual

Hypotheses

• ADOS scores will not differ significantly between remote and face-to-face assessment conditions

• Participants will indicate equivalent satisfaction between remote and face-to-face assessments

Participants

• Inclusion: Adults with PDD diagnosis

• Exclusion

• ADOS module 4 evaluation within 90 days

• Significant hearing or visual impairment

Power

• Recruiting 28 participants, with an assumed dropout rate of 20%, will yield a final sample size of n=22

• Yields 80% power to detect a significant result for ICC

Aim

• To estimate diagnostic accuracy of a remotely administered ADOS module 4

Test methods

• Reference standard – Diagnosis by 2 psychiatrists

• Readers of the remote ADOS administration and reference standard will be independent and blind to the results of other test

Participants

• Inclusion: Dx of Autism, ASD, NS, or NT; adolescent/adult, native English speaker

• Exclusion: ADOS module 4 evaluation with 90 days; significant hearing or visual impairment

Power

• For overall classification accuracy, a sample size of 50 yields a precision level of 0.1 using a 95% confidence interval

Statistical methods

• Sensitivity

• Specificity

• + and – predictive values

• + and – likelihood ratios

• Comparison to literature

DIAGNOSIS nAutism 20ASD 20Non-Spectrum 5No Dx 5TOTAL 50

THANK YOU!

Questions? Comment?