What Works Regarding Social Skills Interventions Using Single Subject

Design

Jeffrey Chenier, M.A., Aaron J. Fischer, Katherine Hunter, Emily Patty, Lisa Libster, M.A., Kristen O’Leary, Haley York, Natalie Robichaux

and Frank Gresham, Ph.D.

Introduction

• Scientifically Based Research – Section 9101(37) NCLB:

Research that involves the application of rigorous, systematic, and objective procedures to obtain reliable and valid knowledge relevant to education activities and programs

(No Child Left Behind Act, 2001)

Introduction

• What Works Clearinghouse (2006): Group Design– Meet Evidence Standards:

• “well designed and implemented randomized controlled trials”

– Meet Evidence Standards with reservations: • “quasi-experiments with equating and no severe

design or implementation problems or randomized clinical trials with severe design or implementation problems”

Introduction

• What Works Clearinghouse (2010): Single Case (SC)– Meet Evidence Standards

• IV must be systematically manipulated, with the researcher determining when and how the IV conditions change

• Each outcome variable must be measured systematically over time by more than one assessor, and the study needs to have IOA calculated 20% of the time in each condition, and IOA percentage must meet minimum thresholds

– 0.80 IOA or 0.60 Cohen’s Kappa• Study must include at least three attempts to demonstrate an

intervention effect at three different points in time or with three different phase repetitions

– Phase must have a minimum of three data points

– Effect size estimation follows if a study has either Strong Evidence or Moderate Evidence

Introduction

• Meta-Analysis– Strube & Hartmann (1982)

• Objective method for summarizing a body of empirical findings

– Emphasizes the direction and magnitude of effects across studies for a particular intervention

What Works Clearinghouse (2010)• No agreed upon method or gold

standard to calculate effect sizes from single-case design research – Problems

• How to quantify the effect?– How accurate is the effect?

• How comparable are the effects across other SC designs?

• How comparable are the effects compared to group design effect sizes?

Current Effect Size Estimators (WWC, 2010)• Nonparametric Methods

– Percentage of Nonoverlapping Data (PND), Percentage of All Nonoverlapping Data (PAND), Percent Exceeding the Median (PEM)

• “Distributional properties of these measures are unknown, so standard errors and statistical tests are not formally justified.”

– Additionally, trend is not addressed • Because of the lack of statistical justification, only use if an

approximate size of the effect is desired.• Wolery et al. (2010) compared four overlapping methods to

visual inspection of effect and each method had its own host of issues, so much that they called for their abandonment

– Visual analysis only agreed 121/160 on whether the treatment was effective or not

Current Effect Size Estimators (WWC, 2010)

• Parametric Methods– Regression Estimates

• Advantages– Familiarity– Ability to model trends– Ability to attain an Effect Size from a single case

• Disadvantages– Inability to deal with complex structures present in

single case design

Current Effect Size Estimators (WWC, 2010)

• Parametric Methods– Multilevel Modeling

• Advantages– Ability to account for complexity of design

• Disadvantages– Unfamiliarity– Technically challenging and time consuming– Different metric from group design Effect Sizes,

therefore the estimate is not comparable

Current Effect Size Estimators (WWC, 2010)• Quantitative Methods

– Differing methods to calculate a Standardized Mean Difference statistic (current study)

• Advantages– Encourages inclusion of SC designs in evaluating effects of

interventions– Potentially gives another method in which to rank order

interventions• Disadvantages

– Not completely comparable to group design research» Pooled within-group variance not comparable to pooled

within phase variance– Small n leads to imprecise estimates– Trend is not assessed

Summary of Effect Size Estimators for SC Design (WWC, 2010)

• Simply put, science is not there yet• Nonparametric estimators should be reported

with a parametric estimator (regression)– Multilevel methods are not ready

• Quantitative methods are not as statistically sound as they should be, but the base from which to build is present

Social Skills

• Learned behaviors that enable positive interactions and allow for escape/avoidance of negative interactions

• Academic Enablers (DiPerna & Elliott, 2002)– Better predictor of academic achievement in 8th

grade than 3rd grade academic achievement (Caprara et. al, 2000)

• Myriad of problems co-occurring with social skills deficits– Both externalizing and internalizing

• Does social skills training work?– Gresham, Cook, Crews, and Kern, 2004

Meta-Analysis n studies

ES g ES rBESD Control

BESD Treatment

Ang & Hughes (2001) 38 .62 .30 35% 65%

Beelman et al. (1994) 49 .47 .23 38% 62%

Losel & Beelmn (2003) 84 .38 .19 40% 60%

Schneider (1992) 79 .89 .40 30% 70%

Schneider & Byrne (1985) 51 .65 .31 34% 66%

Quinn et al. (1999) 35 .20 .10 45% 55%

Means M = .60 M = .29 M = 35% M = 65%

Introduction

Introduction

Introduction

• Godbold et. al, 2010– Contrast Analyses

Introduction

• Research question– Evidence is there for Primary Programs; is

there evidence for Secondary Programs?

Other Meta Analyses Since 2000Meta Analysis Disability Type n studies Intervention Type ES statistic ES Degree of Effect

Bellini & Akullian, 2007 Autism Spectrum 15 Video Modeling PND 81% Effective

Bellini & Akullian, 2008 Autism Spectrum 7 Video Self Monitoring PND 77% Questionable/Effective

Bellini et al., 2007 Autism 15 Child Specific PND 71% Questionable

Bellini et al., 2007 Autism 7 Collateral Skills PND 75% Questionable

Bellini et al., 2007 Autism 20 Comprehensive PND 72% Questionable

Bellini et al., 2007 Autism 10 Peer Mediated PND 62% Questionable

Bellini et al., 2007 Autism 55 Total PND 70% Questionable

Kokina & Kern, 2010 Autism Spectrum 18 Social Stories PND 60% Questionable

Wang & Spillane, 2009 Autism 2 CBT d .47-1.24, .24-.59 Medium - Large, Small - Medium

Wang & Spillane, 2009 Autism 1 CBT PND 100% Very Promising

Wang & Spillane, 2009 Autism 9 Others PND 80.77% Effective

Wang & Spillane, 2009 Autism 9 Peer Mediated PND 60.69% Questionable

Wang & Spillane, 2009 Autism 6 Social Stories PND 67.21% Questionable

Wang & Spillane, 2009 Autism 11 Video Modeling PND 84.25% Effective

Durlak & Weissberg, 2010 Typical 68

After School Programs (Targeting

Positive Social Behaviors)

g 0.19 Small (Significantly different from zero)

Schneider et al, 2008 Autism 19 Social Behavior phi 0.72 Large

Summary

• 7 total studies– 6 with Autism Spectrum, 1 with Typically

Developing

• Multiple interventions available• Effectiveness

– 3 very effective– 3 moderately effective– 1 not as effective (but still statistically significant)

Method

• Literature Search, 2000-2009– Keyword Search in PsycINFO

– 5940 Articles Total

Social Skills +

Competence

Intervention

Training

Method

• Coding 1 Primary Inclusionary Criteria– No Books, Reviews, Meta-Analyses, Group Designs,

Dissertations• Coding 1 Secondary Inclusionary Criteria

– “Is study a social skills intervention or does it target a social skill?” (YES)

– Does study focus on ages 3-21, or through high school? (YES)

– Does study target drugs, alcohol, or sexual offenders? (NO)

• 296 studies remained, 100% IOA in coding 1 (approximately 22% of articles)

Method• Coding 2 Primary Inclusionary Criteria

– Is the full article in English?– Does article include single subject graphs?

• No AB design

• Coding 2 Secondary Inclusionary Criteria– Does study fit our Social Skills definition?

• Gresham, Van, and Cook, 2006:– Facilitates initiating and maintain positive social relationships– Contributes to peer acceptance and friendship development– Results in satisfactory school adjustment– Allows individuals to cope with and adapt to the demands of the social

environment

– Is the study not part of a larger treatment package?• Coded 190 unique studies

– 64 studies on to Coding 3 (IOA = 92% for 38% of studies)

Method

• Coding 3 Primary Inclusion Criteria– If one participant, more than 1 replication

across setting or behavior– Presence of variability in baseline and

treatment conditions across at least 2 participants, settings, or behaviors

– Graphs in which UnGraph was able to score – 40 studies eligible for analysis (IOA 100% on

30% of studies)

Method

– Design Type and Subtype

– Research Question– Main Unit of

Comparison– Participant Info– Phase Info– Dataset Info

• Measurement Strategy• DV information• IOA• Treatment Efficacy

– Study Quality• Three replications

across or within?• Treatment Integrity?• IV Operationally

Defined?• DV Operationally

Defined?• IOA

• Coding 3

Method

• Data Extraction– UnGraph (Biosoft, 2004)

• Extracts numerical data from graphs and puts it into Microsoft Excel

– High reliability and validity in collecting data from single subject graphs (Shadish et al., 2009)

Method

• Effect Size Calculation (Shadish, 2007)– G = (Mt – Mb) / sp

• yields a standardized mean difference statistic

– Currently the best quantitative method available, but not absolutely accurate

Method1) Calculate Mean of Baseline and Tx Panels One, Two, and Three

2) Calculate BL and Tx Mean of Means across the three panels

3) Calculate Standard Deviation of BL and Tx from the Mean of Means

4) Calculate Effect Size for Positive Social Interactions

Results

Participants #  Disability Type  # Ethnicity # 

Total # 148 ADHD 24 % Reported of n 41.10%

Age 2-5 28 ODD 9 Caucasian 28

Age 6-8 8 Asperger's Disorder 9 African American 21

Age 9-12 50 Autism 37 Hispanic 6

Age 13-18 18 BD, EBD, ED 19 Asian 1

Unspecified 8 Language Delay 2 Bi-Racial 1

Male 123 Intellectual Disability 9 Vietnamese 1

Female 23 Typically Developing 44 Latino 1

Classrooms 2 PDD 1 Native American 1

    Williams 1    

JournalsJournal Name # of Articles

Education & Treatment of Children 5Behavioral Disorders 5Focus on Autism and Other Developmental Disabilities 5Journal of Positive Behavior Interventions 4Psychology in the Schools 3Behavior Modification 2Therapeutic Recreation Journal 2Journal of Autism and Developmental Disorders 2Education and Training 2Journal of Applied Behavior Analysis 2Behavior Therapy 1School Psychology Quarterly 1Remedial and Special Education 1Topics in Early Childhood Special Education 1Journal of Early Intervention 1Journal of the Association for Persons with Severe Handicaps 1International Journal of Play Therapy 1Education Psychology 1

Educational Psychology in Practice 1

Results

• Are social skills interventions evaluated by single subject methods effective?

• Yes• g=3.06

Results

• Independent Variable

IV n ESDirect Instruction 17 2.79Computer-Based 3 2.02*Peer Mediated 5 2.67

Self Management 6 3.53Social Stories 4 3.94

SODA 2 5.54*Reinforcement Based 3 2.17

*p<.05

Results

• Independent Variable– Largest effects occurred with interventions

that target self-awareness / self-monitoring of behavior as the primary independent variable

Results

• Dependent VariableDV n ES

AET/On Task Bx 5 1.57Total Destructive Behaviors (TBD)/ Problem

Behaviors / Negativistic Bx 7 1.30Prosocial Behaviors

Cooperative Play 2 5.54*Positive Statements, Terminations 4 2.12

Prosocial Behavior 13 2.80Sportsmanship 3 2.01

Social Play Skills - Recess 3 3.18SILA 2 1.21*

Seeking Help when lost 1 6.92*

*p<.05

Results

• Dependent Variable– Larger effects are seen when targeting

prosocial behavior over problem behavior

Results

Recipient Director

Recipient n ES

Student 37 3.15*

Classroom 1 2.68

Teacher 1 2.7

Peer 1 2.34

Director n ES

Experimenter 14 3.41

Teacher 10 3.02

Student 4 2.48

Computer 3 2.02*

Coach/Caregiver 3 2.04*

Interventionist 4 3.62*

Peer 2 3.81*

*p<.05 *p<.05

Results

• Recipient and Director– Largest effects are seen when the student

is the target of the intervention and when either an independent experimenter/interventionist or peer is the director of the intervention

Results

• Disability

Disability n ESAutism 13 4.04*

Asperger's 3 4.00*ADHD/ODD 4 2.47ED/BD/EBD 4 2.31

Language Impaired 1 2.68Intellectual Disability 2 1.82*Typically Developing 13 2.48

*p<.05

Results

• Disability*– Interventions are most effective with

children who have disabilities along the Autism spectrum

Results

• Setting

Setting n ES

School / Classroom 23 3.29*

School settings not classroom

6 2.70

Recess 5 2.43*

College Campus 2 2.60

Other 4 3.25*

Results

• Setting– The largest effects were seen when

interventions were implemented in schools

Results

• Number of Components

# Components n ES

2 4 4.02*

3 9 2.66

4 8 3.38*

5 3 2.31

6 10 3.02

7 4 3.34

8 1 1.84*

9 1 2.68

Results

• Components**– Increasing the number of components in

an intervention did not increase the magnitude of effect

• Size may not matter, quality matters

Results

• Treatment Integrity

Integrity n ES

Monitored 3 2.55

No 19 2.71

Yes 18 3.46*

Results

• Treatment Integrity*– Studies that report percentage of integrity

had largest effects

Results

• Reinforcement

Reinforcement Provided n ES

Yes 23 2.88

No 17 3.29

Results• Reinforcement**

– No difference in these studies in regards to reinforcement

• Could be a definitional issue. Coded articles that specified reinforcement given, not necessarily lack of reinforcement

– Reinforcement should be provided if necessary to acquire behavior change

• Quality of intervention may be more important for some

Results

• Study Quality

Quality n ES

3-3.5 5 3.49

4-4.5 22 2.30

5 13 4.16

Results

• Study Quality*– Highest effects were found in studies that

had highest quality ratings, although not statistically significant

– Studies/interventions should be implemented with highest quality possible

Summary of ResultsContrasting Significantly Higher Significantly Lower

IV Self Awareness(1) Computer-Based IVsDV Prosocial Behavior Problem Behavior

Recipient Student

Director Peer, Interventionist Computer, Coach/CaregiverDisability(2) Autism Spectrum Intellectual Disability

Setting School RecessComponents(3) Less More

Integrity Taken(2) Yes NoReinforcement(3) Neither NeitherStudy Quality Higher(4) Lower

1) Mixture of Ivs, all having self awareness qualities2) Matches effect of group design study3) Opposite of effect of group design study4) Not significantly higher, but higher, and similar to our group design finding

Limitations

• Effect size estimator not entirely accurate

• No correction for small sample size

• Stringent selection criteria / data analytic method responsible for abandonment of nearly 40% of single subject studies– Both parametric and nonparametric methods

would have had a larger n

Discussion and Future Directions• All interventions were effective

– If assessment leads to a social skills deficit, with almost any kind of student, there are interventions that work

• Single subject meta-analyses are not yet as informative/definitive as they could be, but current best practice is still to aggregate magnitude and direction of effects across studies– Calculate results using other available methods

and comparing effects (NASP in Philly 2012????)

Questions and Comments?

• For additional copies of this presentation, either check the NASP website or contact Jeffrey Chenier at jcheni1@tigers.lsu.edu

Selected Citations

• Kratochwill, T. R., Hitchcock, J., Horner, R. H., Levin, J. R., Odom, S. L., Rindskopf, D. M & Shadish, W. R. (2010). Single case designs technical documentation. Retrieved from What Works Clearinghouse website: http://ies.ed.gov/ncee/wwc/pdf/wwc_scd.pdf.

• Wolery, M., Busick, M., Reichow, B., & Barton, E.E. (2010). Comparison of overlap methods for quantitatively synthesizing single-subject data. The Journal of Special Education, 44, 18-28.