INS Investigators’ Workshop:

Methods for Single-Case Studies in NeuropsychologyJohn R. CrawfordSchool of Psychology

College of Life Sciences and Medicine

King’s College

University of Aberdeen

and

School of Psychology

Flinders University of South Australia

j.crawford@abdn.ac.ukwww.abdn.ac.uk/~psy086/dept/

Cognitive NeuroscienceResearch Group

Collaborators:

Prof Paul H Garthwaite The Open Universityalso

Prof David C Howell University of Vermont

Prof Keith R Laws University of Hertfordshire

Prof Annalena Venneri University of Hull

Dr Colin D Gray University of Aberdeen

Prof Addelchi Azzalini University of Padova

(Dr Sytse Knypstra University of Groningen)

Cognitive NeuroscienceResearch Group

The Importance of Dissociations

“Dissociation is the key word of neuropsychology.”

(Rossetti & Revonsuo, 2000, p. 2)

The Case for Single Cases

“Studies in groups of patients which aim at elucidating the neurological and

functional architecture of mental processes are useless and harmful,

since they provide misleading results. The only appropriate method is to

study individual patients”

(Vallar, 2000, p. 334)

The need for methodological rigour in single-case studies

“If advances in theory are to be sustainable they must be based on

unimpeachable methodological foundations.”

(Caramazza & McCloskey, 1988, p.619).

Evaluating Tests for Deficits in Single-Case Studies

Massive revival of interest in single-case studies

in neuropsychology and neurology

The arguments for single-case studies over group studies are viewed by many as compelling

However, it is clear that they present difficulties when it comes to statistical analysis

This aspect of single-case studies has been relatively neglected

Single-case research: The three basic approaches to drawing inferences concerning a patient’s

performance Patient is administered fully standardized

neuropsychological tests and performance is compared to large sample normative data

At other extreme, patient’s performance is not referenced to normative data or control performance; i.e., analysis is limited to intra-individual comparisons

Patient is compared to a (modestly sized) matched control sample

Limitations of the fully standardized approach

Can only be used in fairly circumscribed situations because:

New constructs are constantly emerging in neuropsychology

In contrast, collection of norms is a long and arduous process

Even where norms are available, may not be applicable to patient

Dangers of the intra-individual approach

Results can be very misleading as performance is not referenced to normal performance

Category specificity literature provides a good example of dangers

Reports of apparently striking dissociations between naming of living versus non-living things and even within these categories (Broccoli’s area?)

In vast majority of these studies inferences are drawn on the basis of chi-square tests comparing a patient’s living and non-living naming

Laws, Gale, Leeson & Crawford’s (2005) study on living / non-living naming

Laws et al. examined cases of AD who had or had not been classified as

exhibiting a dissociation using intra-individual approach (chi-square test)

It was found that the performance of some patients with “dissociations” was not unusual when referenced to control performance

Moreover, patients who had not been identified as exhibiting dissociations were identified as such when performance was referenced to controls

In one case a patient classified as exhibiting a dissociation in favour of non-living things was found to exhibit a dissociation in the opposite direction

Testing for a deficit in single-case studies: the “standard” method

*

X

x xz

s

Patient’s performance is converted to a standard score based on mean and SD of control sample and referred to table of areas under the normal curve

The statistics of the control sample are treated as population parameters

When sample size is large this is not too much of a problem as the statistics provide sufficiently accurate estimates of the parameters

However, large sample sizes are rare in the single-case literature

Testing for a deficit in single-case studies using Crawford & Howell’s (1998) proposed method:

*

1X

x xt

ns

n

Uses formula set out by Sokal and Rohlf (1995) Modified t-test: tests hypothesis that patient did not

come from the control population (under null hypothesis patient is an observation from this population)

Control sample statistics are treated as statistics Crawford & Garthwaite (2002) also developed method

of setting confidence limits on abnormality of score (using non-central t distributions)

Comparison of two methods for testing for a deficit: Type I errors (Crawford & Garthwaite, Neuropsychology,2005)

Monte Carlo simulation study

5 control sample sizes (N) were examined: 5, 10, 20, 50 and 100

For each value of N one million observations of N +1 were drawn from a normal distribution

The first N observations were taken as the control sample data and the N+1th observation as the control case

The alternative tests for deficits were applied to these data and the percentage of Type I errors compared to the specified rate of 5%

Perform statistical tests comparing control case andControl sample and record if significant, i.e. record if Type I error

Step (2) Get machine to repeat this one million times

Step (3) Meanwhile go and get yourself a…

Monte Carlo simulation: Sampling from the control population

Comparison of two methods for testing for a deficit: Type I errors (Crawford & Garthwaite, 2005)

10.37

7.57

6.255.53 5.285.01 5 5 5.03 4.98

0

2

4

6

8

10

12

5 10 20 50 100

Sample sixe (n )

Per

cent

age

Typ

e I

erro

rs

z

C&H

Departures from normality

Both z and modified t-test assume control data are drawn from normal distribution

However, in single-case studies there is often evidence of negative skew in scores of the control samples (ie control mean=50, SD=10, but max score =55)

We have run Monte Carlo simulations to examine control of Type I error rate when control data are non-normal

Same method as in previous study except that the N+1 observations were sampled from distributions that were skew and /or leptokurtic

Perform statistical tests and record ifsignificant, i.e. record if Type I error

Sampling from negatively skewed and / or leptokurtic distributions

Step (2) Repeat this one million times

Step (3) Meanwhile go and get yourself a…

Results of a Monte Carlo study: Robustness in face of moderate skew

11.48

8.59

7.236.5 6.26.06 6.04 5.97 6 5.97

0

2

4

6

8

10

12

14

5 10 20 50 100

Sample sixe (n)

Per

cen

tag

e T

ype

I er

rors

z

C&H

The Internet

Most of calculations involved with these methods are simple (exception being CLs)

However, still tedious and error prone

Therefore, we have written computer programs that implement these methods

Freely available on the web www.abdn.ac.uk/~psy086/dept/SingleCaseMethodsComputerPrograms.HTM

Calculations can be performed literally in seconds

In this example, the patient’s score is significantly below controls and so we conclude he/she has a deficit. Also, it is estimated that only 1.13% of the control population would exhibit this poor a score; the 95% CI on this estimate of abnormality is from 0.05% to 4.68%

Modifed T-Test Versus Modified ANOVA

Mitchell and colleagues (Mycroft et al, 200; Mitchell et al, 2004) have criticised the foregoing method

They argue that (a) a notional patient population will have markedly increased variance relative to the control population, and (b) our method will therefore produce inflated Type I errors

Mitchell and colleagues propose an ANOVA that employs more conservative critical values to overcome this perceived problem

Modifed T-Test Versus Modified ANOVA

We believe there are two major problems with Mitchell et al’s position:

The argument over Type I errors is untenable (Crawford et al, Cognitive Neuropsychology, 2004)

Statistical power to detect a deficit is very low for Mitchell et al’s method (Crawford & Garthwaite, Cognitive Neuropsychology, in press)

0 50 100 150 200 250

A graphic illustrating Mitchell et al’s. scenario: A notional patient population (gray line) has same mean as controls (dark line) but has greater variability

This scenario is not realistic: If the means

do not differ but patients are more

variable, then scores below control mean

must be exactly balanced by scores above control mean

0 50 100 150 200 2500 50 100 150 200 250

If the patient mean is lower than control mean (even marginally) then issue of Type I error does NOT arise: a deficit is present and the question is whether it can be detected (i.e., it is a power issue)

Power to detect a (2 SD) deficit: comparison of three methods (Crawford & Garthwaite, Cognitive Neuropsychology, in press)

0

10

20

30

40

50

60

70

5 10 20 50 100

Control sample size

Po

we

r (%

de

tec

ted

)

Crawford & Howell MM&K: Intermediate

MM&K: Extreme

Power to detect a (2 SD) deficit: comparison of three methods (Crawford & Garthwaite, Cognitive Neuropsychology, in press)

0

10

20

30

40

50

60

70

5 10 20 50 100

Control sample size

Po

we

r (%

de

tec

ted

)

Crawford & Howell MM&K: Intermediate

MM&K: Extreme

PART 2

Dissociations in Neuropsychology and Statistical Tests on Differences

DISSOCIATIONS

In neuropsychology, deficits are of limited

theoretical interest unless they are accompanied by preserved or less impaired performance on other tasks; i.e. the aim of many single-case studies is to demonstrate dissociations of function

Conventional Definition of a Classical Dissociation

“If patient X is impaired on task 1 but performs normally on task 2,

then we may claim to have a dissociation between tasks”

(Ellis and Young, 1996, p. 5)

A Classical Dissociation (based on Shallice, 1988)

Task X Task Y

Per

form

ance

The Importance of Dissociations

“Dissociations play an increasingly crucial role in the methodology of cognitive neuropsychology… they have provided critical support for

several influential, almost paradigmatic, models in the field.”

(Dunn & Kirsner, 2003, p. 2)

Criteria for Dissociations: Three Problems

What constitutes a “deficit” and being “within normal limits” is very poorly specified

One half of the typical definition essentially involves an attempt to prove the null hypothesis

A patient’s score on the “impaired” task could lie just below the critical value for defining impairment and the performance on the other test lie just above it (see Caramazza & Shelton, 1998 for similar point)

Problems with Conventional Criteria for a Classical Dissociation

Task X Task Y

Per

form

ance

Crawford, Garthwaite & Gray ( 2003):

Potential Solutions to the Three Problems

Crawford et al. (2003) provided fully explicit criteria for a

deficit (using Crawford & Howell’s test)

They also introduced a requirement that the patient’s performance on Task X should be significantly poorer than performance on Task Y

This criterion deals with the problem of trivial differences

It also provides us with a positive test for a dissociation (thereby lessening reliance on what boils down to an attempt to prove the null hypothesis of no deficit or impairment on Task Y)

Crawford et al’s. criteria for a classical dissociation

Task X Task Y

Per

form

ance

X significantly below controls on Crawford & Howell’s test (one-tailed)

Y not significantly different from controls on Crawford & Howell’s test (one-tailed)

X significantly different from Y

How should we test for significant difference between a patient’s score on Tasks X and Y ?

In most single-case studies the two tasks of interest will have different means and SDs

For example a patient’s performance on a ToM task with (mean=35, SD=12) is to be compared with performance on an executive task (mean=22 SD=6)

In order to meaningfully compare performance it is necessary to standardize scores on the two tasks

Testing for a difference between a patient’s performance on Tasks X and Y

A long established method is that of Payne & Jones (1957):

The Payne and Jones method

,2 2X Y

D

xy

Z Zz

r

Testing for a difference between a patient’s performance on Tasks X and Y: Crawford, Howell & Garthwaite (1998)

method

12 2

X YD

xy

Z Zt

nr

n

* *

2 2 2 22

2 2

( ) ( )

5 1 1 11 2(1 )2 2

1 2 1 2 1

X Y

X X Y Ys s

y r r y rn rr

n n n n

Revised Standardized Difference Test (Crawford & Garthwaite, 2005b; Garthwaite & Crawford, 2004):

Looks nasty but is essentially of familiar form…

difference between two quantities

standard error of the differencet

Monte Carlo Evaluation of tests for differences between tasks

5 control sample sizes (N) were examined: 5, 10, 20, 50

and 100

For each value of N and for each of 4 values of r (the correlation between tasks), one million pairs of observations of N +1 were drawn from a bivariate normal distribution

The first N pairs were taken as the control sample data and the N+1th pair was as the control case

The alternative tests for differences were applied to these data and the percentage of Type I errors compared to the specified rate of 5%

Perform statistical tests comparing control case with control sample and record if significant, i.e. record if Type I error

Simulation study of Type I errors for tests on differences X , YX , Y

X , YX , Y

X , YX , Y

X , YX , Y

X , YX , YX , Y

X , YX , Y

X , Y

X , Y

X , Y

Meanwhile, have a noodle about on the…

Monte Carlo simulation: Type I error rate for Revised Standardized Difference Test (rxy =0.5 in this example)

0

5

10

15

20

25

5 10 20 50 100

Sample size

Per

cen

tag

e T

ype

I err

ors

PJCHGRDT

Monte Carlo simulation: Type I error rate for Revised Standardized Difference Test (rxy =0.5 in this example)

0

5

10

15

20

25

5 10 20 50 100

Sample size

Per

cen

tag

e T

ype

I err

ors

PJCHGRDT

Evaluating criteria for classical dissociations

To recap: we have considered two sets of criteria for detecting dissociations – the conventional criteria and Crawford & Garthwaite’s (2005b) criteria

We now have suitable test for Crawford & Garthwaite’s third criterion (i.e. it requires a significant difference between a patient’s scores on X and Y)

We (Crawford & Garthwaite, 2005a, Neuropsychology) have examined performance of these two sets of criteria

Same approach as used for evaluating foregoing tests; i.e. sample from bivariate distributions but apply the sets of criteria rather than individual tests for components of these criteria

Type I error rate for Crawford & Garthwaite’s (2003; 2005b) criteria and conventional criteria for a classical dissociation (in

this example rxy = 0.5

1.67 1.55 1.52 1.47 1.47

14.6

11.01

9.34

8.2 7.85

0

2

4

6

8

10

12

14

16

5 10 20 50 100

Sample size

Per

cen

tag

e T

ype

I er

rors

C&G

Conventional

Type I error rate for Crawford & Garthwaite’s (2003; 2005b) criteria and conventional criteria for a classical dissociation (in

this example rxy = 0.5

1.67 1.55 1.52 1.47 1.47

14.6

11.01

9.34

8.2 7.85

0

2

4

6

8

10

12

14

16

5 10 20 50 100

Sample size

Per

cen

tag

e T

ype

I er

rors

C&G

Conventional

Evaluating criteria for a classical dissociation: Conclusions

Conventional criteria for a classical dissociation will

misclassify a worryingly high percentage of healthy controls as exhibiting a classical dissociation regardless of the size of the control sample (rate was as high as 18.6% in one of the scenarios)

In contrast, Crawford & Garthwaite’s (2003; 2005b) criteria are conservative; i.e. very low percentage of controls misclassified

Results underline importance of testing the difference between patient’s X and Y scores

Crawford & Garthwaite’s criteria were relatively robust in face of skewed control data

In this example, the patient is classified as exhibiting a dissociation by Crawford & Garthwaite’s (2005b) criteria. (Crucially) the RSDT shows that there is a significant difference between the patient’s scores on the two tasks. Task X is significantly below controls but Task Y is not: therefore it is a classical dissociation.

Further evaluation of criteria for a classical dissociation

Up to this point Type I errors have been defined as

incorrectly identifying a control as exhibiting a dissociation

However, there is another form of Type I error that should be considered

Alternative definition of Type I error: misclassifying a patient who has equivalent deficits on X and Y as exhibiting a classical dissociation

Perform statistical testsand record if criteria are met,i.e. record if Type I error

Lesion study: Type I errors now defined as misclassifying a patientX , YX , Y

X , YX , Y

X , YX , Y

X , Y

X , YX , YX , Y

X , YX , Y

X , Y

X , Y

X , Y

X , Y Lesion the control case, 2 SDs below premorbidScores on X and Y

X-2 , Y-2

X-2 , Y-2

Type I error rate for the competing criteria: Type I errors defined as identifying a patient as exhibiting a classical dissociation

32.6 32 31.6 31.3 31.1

6.3 5.94.6 4.2 4

0

5

10

15

20

25

30

35

5 10 20 50 100

Sample size

Per

cen

tag

e T

ype

I er

rors Conventional

C&G

Type I error rate for the competing criteria: Type I errors defined as identifying a patient as exhibiting a classical dissociation

32.6 32 31.6 31.3 31.1

6.3 5.94.6 4.2 4

0

5

10

15

20

25

30

35

5 10 20 50 100

Sample size

Per

cen

tag

e T

ype

I er

rors Conventional

C&G

Evaluating criteria for a classical dissociation: Further conclusions

Performance of conventional criteria for a classical

dissociation is extremely poor: very high percentages of patients will be incorrectly classified as exhibiting a classical dissociation (close to 50% in some scenarios)

In contrast, Crawford & Garthwaite’s (2005) criteria are much more conservative; i.e. percentage is below 5% in most scenarios

These results further underline importance of testing the difference between patient’s X and Y scores

Power to detect a classical dissociation

Up to this point concern has been with Type I errors, i.e. false positives

However, we should also be concerned with the statistical power of criteria for dissociations; i.e. the ability of these criteria to avoid false negatives

This issue has not previously been examined empirically

It does not make sense to examine power unless the Type I error rate is under reasonable control

Therefore, power only examined for Crawford & Garthwaite’s (2005b) criteria

Why power to detect a dissociation will be low

Crawford and colleagues have argued that power is almost inevitably low-to-moderate in single-case studies

An individual patient rather than a sample of patients is compared to a control sample

The control sample itself is usually modest in size

The existence of substantial individual differences in premorbid competencies is a further factor…

Why power to detect a dissociation will be low

Task X Task Y

Mean

+1 SD

-1 SD

Perform statistical tests and record if correctlyidentified as a dissociation

Lesion study: Power to detect a dissociation

X , YX , YX , Y

X , YX , Y

X , Y

X , Y

X , YX , YX , Y

X , YX , Y

X , Y

X , Y

X , Y

X , Y Lesion the control case, 2 SDs below premorbidScores on X ONLY

X-2 , Y

X-2 , Y

Simulation results: Power to detect a classical dissociation(for control sample N of 20)

0

10

20

30

40

50

60

Correlation between X and Y

Per

cen

tag

e co

rrec

tly

clas

sifi

ed

0 .3 .7 .8.5

Crawford & Garthwaite (2005a, Neuropsychology)

Power to detect a dissociation: Conclusions Results confirm that, if the Type I error rate is to be

controlled, power is low in single-case studies

Not surprisingly, power is higher with large control samples; therefore, control sample Ns should be larger than is typical currently

This is not unreasonable: if a researcher believes single-case studies are more useful than group studies then she/he should be willing to expend the effort

More encouragingly, power is higher when tasks are moderately to highly correlated

Single-Case Methods: Overall Summary

Massive revival of interest in single-case studies in neuropsychology and neurology

The arguments for single-case studies over group studies are viewed by many as compelling

However, it is clear that they present difficulties when it comes to statistical analysis

The methods and criteria commonly used in single-case studies are problematic

However, many of the problems can be overcome using the methods outlined

These latter methods are rigorous but easy to apply using the accompanying computer programs

Finally…

Regression equations can play a useful role in single-case research and clinical practice

For example, attempting to detect a deficit by comparing an obtained score with a predicted score based on demographic variables or a measure of premorbid ability

In neuropsychology, inferences concerning discrepancies between predicted scores and obtained scores are typically made using the standard error of estimate

This method produces inflated Type I error rates (Crawford & Garthwaite, Neuropsychology, in press)

Finally…

Building on earlier work by Crawford & Howell (1998b) we (Crawford & Garthwaite, in press) have recently proposed and evaluated an alternative method that controls Type I error rate

The method also produces confidence limits on the abnormality of the discrepancy

Methods implemented in accompanying computer program

Poster on this topic later today

THE ENDWeb page summarizing our work on single-case methods:

www.abdn.ac.uk/~psy086/dept/SingleCaseMethodology.htm

Web page containing the computer programs referred to in this presentation:www.abdn.ac.uk/~psy086/dept/SingleCaseMethodsComputerPrograms.HTM

A reference list for the methods is provided in your handout following the copy of this slide

end

ReferencesCrawford, J. R. (2004). Psychometric foundations of neuropsychological

assessment. In L. H. Goldstein & J. E. McNeil (Eds.), Clinical neuropsychology: A practical guide to assessment and management for clinicians (pp. 121-140). Chichester: Wiley.

Crawford, J. R., & Garthwaite, P. H. (2002). Investigation of the single case in neuropsychology: Confidence limits on the abnormality of test scores and test score differences. Neuropsychologia, 40, 1196-1208.

Crawford, J. R., & Garthwaite, P. H. (2004). Statistical methods for single-case research: Comparing the slope of a patient's regression line with those of a control sample. Cortex, 40, 533-548.

Crawford, J. R., & Garthwaite, P. H. (2005a). Evaluation of criteria for classical dissociations in single-case studies by Monte Carlo simulation. Neuropsychology, 19, 664-678.

Crawford, J. R., & Garthwaite, P. H. (2005b). Testing for suspected impairments and dissociations in single-case studies in neuropsychology: Evaluation of alternatives using Monte Carlo simulations and revised tests for dissociations. Neuropsychology, 19, 318-331.

References contdCrawford, J. R., & Garthwaite, P. H. (in press-a). Comparing an

individual's predicted test score from a regression equation with an obtained score: a significance test and point estimate of abnormality with accompanying confidence limits. Neuropsychology.

Crawford, J. R., & Garthwaite, P. H. (in press-b). Methods of testing for a deficit in single case studies: Evaluation of statistical power by Monte Carlo simulation. Cognitive Neuropsychology.

Crawford, J. R., Garthwaite, P. H., Azzalini, A., Howell, D. C., & Laws, K. R. (in press). Testing for a deficit in single case studies: Effects of departures from normality. Neuropsychologia.

Crawford, J. R., Garthwaite, P. H., & Gray, C. D. (2003). Wanted: Fully operational definitions of dissociations in single-case studies. Cortex, 39, 357-370.

Crawford, J. R., Garthwaite, P. H., Howell, D. C., & Gray, C. D. (2004). Inferential methods for comparing a single case with a control sample: Modified t- tests versus Mycroft et al's. (2002) modified ANOVA. Cognitive Neuropsychology, 21, 750-755.

References contdCrawford, J. R., Garthwaite, P. H., Howell, D. C., & Venneri, A. (2003).

Intra-individual measures of association in neuropsychology: Inferential methods for comparing a single case with a control or normative sample. Journal of the International Neuropsychological Society, 9, 989-1000.

Crawford, J. R., & Howell, D. C. (1998a). Comparing an individual’s test score against norms derived from small samples. The Clinical Neuropsychologist, 12, 482-486.

Crawford, J. R., & Howell, D. C. (1998b). Regression equations in clinical neuropsychology: An evaluation of statistical methods for comparing predicted and obtained scores. Journal of Clinical and Experimental Neuropsychology, 20, 755-762.

Crawford, J. R., Howell, D. C., & Garthwaite, P. H. (1998). Payne and Jones revisited: Estimating the abnormality of test score differences using a modified paired samples t-test. Journal of Clinical and Experimental Neuropsychology, 20, 898-905.

Garthwaite, P. H., & Crawford, J. R. (2004). The distribution of the difference between two t-variates. Biometrika, 91, 987-994.

Laws, K. R., Gale, T. M., Leeson, V. C., & Crawford, J. R. (2005). When is category specific in Alzheimer's disease? Cortex, 44, 452-463.