Intra-Individual Variability: OTBM by OTBM-SD

06/20/2012 Rohling - Meta-analysis & Base Rates 1

342

0

.575

.330

.328

3.410

Count

Num. Missing

R

R Squared

Adjusted R Squared

RMS Residual

Regression Summary

Intra-Individual SD vs. Overall Test Battery Mean (OTBM)

Inclusion criteria: Study Grp from RIM Response-Meyers Data.dat (imported).svd

1 1950.921 1950.921 167.731 <.0001

340 3954.618 11.631

341 5905.539

DF Sum of Squares Mean Square F-Value P-Value

Regression

Residual

Total

ANOVA Table



32.723 1.293 32.723 25.300 <.0001

-.197 .015 -.575 -12.951 <.0001

Coefficient Std. Error Std. Coeff. t-Value P-Value

Intercept

Overall Test Battery Mean (OTBM)

Regression Coefficients



0

5

10

15

20

25

30

35

Intr

a-I

ndiv

idual S

D

40 50 60 70 80 90 100 110 120

Overall Test Battery Mean (OTBM)

Y = 32.723 - .197 * X; R^2 = .33

Regression Plot


Intra-Individual Variability:

OTBM by OTBM-SD

Mercy Medical Center in Sioux City, IA

TBI Cases from John Meyers (MNB-RIM)


131

0

.56

.32

.31

3.17

Count

Num. Missing

R

R Squared

Adjusted R Squared

RMS Residual

Regression Summary

OTBM SD ss vs. OTBM ss

Row exclusion: Summary Scores.svd

1 605.27 605.27 60.17 <.0001

129 1297.58 10.06

130 1902.85


Regression

Residual

Total

ANOVA Table



35.084 2.554 35.084 13.739 <.0001

-.208 .027 -.564 -7.757 <.0001


Intercept

OTBM ss




0

5

10

15

20

25

30

OT

BM

SD

ss

40 50 60 70 80 90 100 110 120

OTBM ss

Y = 35.08 - .21 * X; R^2 = .32

Regression Plot



OTBM by OTBM-SD

RI Hospital Brown University

TBI Cases from Tremont


0

5

10

15

20

25

30

OT

BM

-SD

40 50 60 70 80 90 100 110 120

OTBM


OTBM by OTBM-SD

Sarasota, FL & Detroit, MI – TBI Cases from

Philip Leithen & Geoffrey Kantor (MNB-RIM)


0

5

10

15

20

25

30

OT

BM

SD

40 50 60 70 80 90 100 110 120

OTBM

Y = 39.83 - .24 * X; R^2 = .3

Regression Plot

Row exclusion: Green IISD.svd

891

0

.55

.30

.30

4.60

Count

Num. Missing

R

R Squared

Adjusted R Squared

RMS Residual

Regression Summary

OTBM SD vs. OTBM


1 8064.00 8064.00 380.89 <.0001

889 18821.43 21.17

890 26885.43


Regression

Residual

Total

ANOVA Table

OTBM SD vs. OTBM


39.83 1.17 39.83 34.09 <.0001

-.24 .01 -.55 -19.52 <.0001


Intercept

OTBM


OTBM SD vs. OTBM



OTBM by OTBM-SD

Edmonton, Alberta, Private Practice

Mixed Neurological Cases from Paul Green


0

5

10

15

20

25

30

Mean s

s S

D

40 50 60 70 80 90 100 110 120

Mean ss

Y = 32.02 - .18 * X; R^2 = .28

Regression Plot

114

0

.53

.28

.27

3.01

Count

Num. Missing

R

R Squared

Adjusted R Squared

RMS Residual

Regression Summary

Mean ss SD vs. Mean ss

1 388.88 388.88 43.01 <.0001

112 1012.63 9.04

113 1401.51


Regression

Residual

Total

ANOVA Table


32.024 2.482 32.024 12.901 <.0001

-.182 .028 -.527 -6.558 <.0001


Intercept

Mean ss




OTBM by OTBM-SD

Oklahoma Health Sciences University

Mixed Neurological Cases from Russell Adams


0

5

10

15

20

25

30

SD

40 50 60 70 80 90 100 110 120

Mean

Y = 39.649 - .257 * X; R^2 = .321

Regression Plot

339

0

.57

.32

.32

5.28

Count

Num. Missing

R

R Squared

Adjusted R Squared

RMS Residual

Regression Summary

SD vs. Mean

1 4440.49 4440.49 159.00 <.0001

337 9411.82 27.93

338 13852.30


Regression

Residual

Total

ANOVA Table

SD vs. Mean

39.65 2.04 39.65 19.46 <.0001

-.26 .02 -.57 -12.61 <.0001


Intercept

Mean


SD vs. Mean


OTBM by OTBM-SD

CNS Vital Signs

Xanax Drug Study from David Loring


0

5

10

15

20

25

30

OT

BM

sd

40 50 60 70 80 90 100 110 120

OTBM

Y = 34.71 - .2 * X; R^2 = .34

Regression Plot

192

1

.58

.34

.33

5.47

Count

Num. Missing

R

R Squared

Adjusted R Squared

RMS Residual

Regression Summary

OTBM sd vs. OTBM

1 2890.03 2890.03 96.45 <.0001

190 5693.22 29.96

191 8583.25


Regression

Residual

Total

ANOVA Table

OTBM sd vs. OTBM

34.711 1.797 34.711 19.319 <.0001

-.202 .021 -.580 -9.821 <.0001


Intercept

OTBM


OTBM sd vs. OTBM


OTBM by OTBM-SD

University of South Alabama – Analog Simulation

Study for CNS Vital from Martin Rohling


65

0

.43

.18

.17

3.54

Count

Num. Missing

R

R Squared

Adjusted R Squared

RMS Residual

Regression Summary

SD1 SS vs. OTBM1 SS

1 177.02 177.02 14.10 .0004

63 790.69 12.55

64 967.71


Regression

Residual

Total

ANOVA Table

SD1 SS vs. OTBM1 SS

28.411 2.109 28.411 13.473 <.0001

-.105 .028 -.428 -3.756 .0004


Intercept

OTBM1 SS


SD1 SS vs. OTBM1 SS

0

5

10

15

20

25

30

SD

1 S

S

40 50 60 70 80 90 100 110 120

OTBM1 SS

Y = 28.41 - .11 * X; R^2 = .18

Regression Plot


OTBM by OTBM-SD

Buffalo, NY – Dementia & MCI

Cases from Michael Santa Maria


99

0

.010

1.061E-4

•

3.662

Count

Num. Missing

|R|

R Squared

Adjusted R Squared

RMS Residual

Regression Summary

OTBMss-SD vs. OTBMss

1 .138 .138 .010 .9194

97 1300.985 13.412

98 1301.123


Regression

Residual

Total

ANOVA Table


14.847 4.942 14.847 3.004 .0034

.005 .051 .010 .101 .9194


Intercept

OTBMss



0

5

10

15

20

25

30

OT

BM

ss-S

D

40 50 60 70 80 90 100 110 120OTBMss

Y = 14.847 + .005 * X; R^2 = 1.061E-4


OTBM by OTBM-SD

University of Indianapolis – Normal College

Students from Brad Axelrod & Jacqueline Wall


Summary of descriptive

statistics of each sample.

Sample Source Sample Dx

Sample n

OTBM M

OTBM sd r p Intercept Slope R2

1. Mercy Medical Meyers’ MNB

TBI 342 84.3

(12.2) 16.1 (4.2)

.58 < .0001 32.7 -0.20 .33

2. RI Hosp Brown Geoff Tremont

TBI 148 93.3

(11.6) 15.5 (4.3)

.56 < .0001 35.1 -0.21 .32

3. MI/FL MNB-RIM Leithen/Kantor

TBI 136 91.0 (9.0)

17.8 (3.1)

.35 < .0001 29.0 -0.12 .12

4. Edmonton, CA Paul Green

Neuro Pseudo

891 92.6

(12.3) 17.2 (5.5)

.55 < .0001 39.8 -0.24 .30

5. Oklahoma HSU Russell Adams

Neuro Pseudo

114 88.9

(10.2) 15.9 (3.5)

.53 < .0001 32.0 -0.18 .28

6. CNS-Vital Signs David Loring

Drug Study

339 99.2

(14.1) 14.2 (6.4)

.57 < .0001 39.7 -0.26 .32

7. USA CNS-VS7 Logistic Regression

Simulator study

192 85.3

(19.3) 17.5 (6.7)

.58 < .0001 37.1 -0.20 .34

8. Buffalo, NY Michael Santa Maria

DAT MCI

65 73.5

(15.7) 20.7 (4.0)

.43 .0004 28.4 -0.11 .19

9. University of Indy B. Axelrod & J. Wall

Normal 99 97.3 (7.3)

15.3 (3.6)

.01 .9194 14.8 0.01 .0001

Summary Stats ALL 2091 91.8

(13.5) 16.6 (5.4) .54 < .0001 35.4 -0.20 .30


1 448.98 448.98 32.11 <.0001 32.11 1.00

3 166.95 55.65 3.98 .0086 11.94 .84

3 4.74 1.58 .11 .9524 .34 .07

234 3271.87 13.98

DF Sum of Squares Mean Square F-Value P-Value Lambda Pow er

Validity2

LOC Sev Grp.4

Validity2 * LOC Sev Grp.4

Residual

ANOVA Table for Intra-Individual SD

47 14.0 3.7 .5

51 14.9 4.0 .6

39 15.6 3.8 .6

41 17.0 3.6 .6

10 17.5 3.5 1.1

14 18.5 3.7 1.0

20 18.4 3.8 .8

20 19.9 3.3 .7

Count Mean Std. Dev. Std. Err.

Valid, LOC < 1 hour

Valid, LOC = 1-24 hours

Valid, LOC = 1-6 days

Valid, LOC => 7 days

Exag, LOC < 1 hour

Exag, LOC = 1-24 hours

Exag, LOC = 1-6 days

Exag, LOC => 7 days

Means Table for Intra-Individual SD

Effect: Validity2 * LOC Sev Grp.4

0

2.5

5

7.5

10

12.5

15

17.5

20

22.5

Cell

Mean

Valid Exag

Cell

LOC => 7 days

LOC = 1-6 days

LOC = 1-24 hours

LOC < 1 hour

Interaction Bar Plot for Intra-Individual SD

Effect: Validity2 * LOC Sev Grp.4

-3.4 1.1 <.0001 S

Mean Diff. Crit. Diff P-Value

Valid, Exag

Fisher's PLSD for Intra-Individual SD

Effect: Validity2

Significance Level: 5 %

-1.1 1.3 .1171

-1.9 1.4 .0055 S

-3.3 1.4 <.0001 S

-.9 1.3 .1918

-2.3 1.3 .0008 S

-1.4 1.3 .0437 S


LOC < 1 hour, LOC = 1-24 hours

LOC < 1 hour, LOC = 1-6 days

LOC < 1 hour, LOC => 7 days

LOC = 1-24 hours, LOC = 1-6 days

LOC = 1-24 hours, LOC => 7 days

LOC = 1-6 days, LOC => 7 days

Fisher's PLSD for Intra-Individual SD

Effect: LOC Sev Grp.4



OTBM by OTBM-SD

Mercy Medical Center in

Sioux City, IA – TBI Cases

from John Meyers

• Main effect for Effort or

Invalidity, as measured

by 9 embedded SVTs

• Main effect of TBI

Severity indexed by

LOC time

• No significant

interaction


1 356.22 356.22 16.75 <.0001 16.75 .99

2 116.49 58.25 2.74 .0655 5.48 .53

2 58.68 29.34 1.38 .2525 2.76 .29

539 11460.97 21.26


Fail WMT.2

PTA Groups

Fail WMT.2 * PTA Groups

Residual

ANOVA Table for Green's OTBM-SD

365 10.7 3.0 .2

59 10.6 2.0 .3

2 12.6 3.8 2.7

112 15.7 8.4 .8

6 15.8 5.8 2.4

1 27.0 • •


Gen WMT, PTA < 1 hr

Gen WMT, PTA = 1-24

Gen WMT, PTA = 1-6 days

Exag WMT, PTA < 1 hr

Exag WMT, PTA = 1-24

Exag WMT, PTA = 1-6 days

Means Table for Green's OTBM-SD

Effect: Fail WMT.2 * PTA Groups

0

5

10

15

20

25

30

Cell

Mean

Gen WMT Exag WMT

Cell

PTA = 1-6 days

PTA = 1-24

PTA < 1 hr

Interaction Bar Plot for Green's OTBM-SD

Effect: Fail WMT.2 * PTA Groups

-5.1 .9 <.0001 S


Gen WMT, Exag WMT

Fisher's PLSD for Green's OTBM-SD

Effect: Fail WMT.2


.7 1.2 .2193

-5.6 5.2 .0379 S

-6.3 5.3 .0209 S


PTA < 1 hr, PTA = 1-24

PTA < 1 hr, PTA = 1-6 days

PTA = 1-24, PTA = 1-6 days

Fisher's PLSD for Green's OTBM-SD

Effect: PTA Groups



OTBM by OTBM-SD



by the WMT

• Main effect of TBI

Severity indexed by

PTA time

• No significant

interaction

Edmonton, Alberta – TBI

Cases from Paul Green


1 1502.65 1502.65 44.03 <.0001 44.03 1.00

135 4606.82 34.12


Grp.6 SVTlog

Residual

ANOVA Table for OTBM sd

Inclusion criteria: Exclude Cases from CNSVS NAN Abstract.svd

67 21.1 5.4 .7

70 14.5 6.2 .7


Mal Log

Gen Log

Means Table for OTBM sd

Effect: Grp.6 SVTlog


0

2.5

5

7.5

10

12.5

15

17.5

20

22.5

Cell

Mean

Mal Log Gen Log

Cell

Interaction Bar Plot for OTBM sd



6.6 2.0 <.0001 S


Mal Log, Gen Log

Fisher's PLSD for OTBM sd





OTBM by OTBM-SD



by the WMT

USA – Analog Study for CNS

Vital from Martin Rohling


1. Low scores relatively common across all test batteries

2. Low scores depend on where you set your cutoff score

3. Low scores vary by number of tests administered

4. Low scores vary by the demographics of the examinee

5. Low scores vary by level of intelligence.

5 Psychometric Principles for

Abnormal Scores by Iverson

(2011) & Schretlen et al. (2008)


Mean ES distribution of actual mTBI patient that includes invalid cases

-2 -2 7 3

-2 -1 8 8

-1 -1 4 12

-1 0 15 12

0 1 14 8

1 1 2 3

1 2 0 1

Total 50 49

From (>=) To (<) Count Normal Count

-.50

.75

.11

50

-1.85

.93

-.38

-1.09

-.27

Mean

Std. Dev.

Std. Error

Count

Minimum

Maximum

Skewness

Kurtosis

Median

ES

1.362 48 14.210 <.0001

Mean Diff. DF t-Value P-Value

Genuine, Exaggerate

30 .044 .086 .294 .054

20 -1.317 .147 .383 .086

Count Mean Variance Std. Dev. Std. Err

Genuine

Exaggerate

0

3

6

9

12

15

18

Co

un

t

-2 -1 0 1 2ES

Mean ES

From

(>)

To

(<)

Freq

Dist

% of

Score

%

Norm

Ideal

Freq

-2.5 -2.0 7 14.0 6.7 3

-2.0 -1.5 8 16.0 8.7 4

-1.5 -1.0 4 8.0 16.0 8

-1.0 -0.5 15 30.0 18.6 10

-0.5 0.0 14 28.0 18.6 10

0.0 0.5 2 4.0 16.0 8

0.5 1.0 0 0.0 8.7 4

1.0 1.5 0 0.0 6.7 3

-- -- 50 100 100


0

5

10

15

20

25

30

35

Pe

rce

nt

0 10 20 30 40 50 60 70 80 90 100 110 120Skew ed Data

Histogram

43.92

12.64

.40

6.00

72.00

159.66

.29

66.00

-.72

.56

47.00

51.00

43.86

12.75

.40

8.00

82.00

162.69

.29

74.00

-.23

-.42

45.00

45.00

Mean

Std. Dev.

Std. Error

Minimum

Maximum

Variance

Coef. Var.

Range

Skewness

Kurtosis

Median

Mode

Patient 1 Normal 1

Descriptive Statistics

0

5

10

15

20

25

30

35

Pe

rce

nt

0 10 20 30 40 50 60 70 80 90 100 110 120Normal Data

Histogram

Low Functioning Patients

Impaired vs. Normal

Patient 1

Impaired

Normal 1

Unimpaired


0

5

10

15

20

25

30

35

Pe

rce

nt

0 10 20 30 40 50 60 70 80 90 100 110 120Skew ed Data 2

Histogram

63.92

12.64

.40

26.00

92.00

159.66

.20

66.00

-.72

.56

67.00

15.00

71.00

63.86

12.75

.40

28.00

102.00

162.69

.20

74.00

-.23

-.42

65.00

20.00

65.00

Mean

Std. Dev.

Std. Error

Minimum

Maximum

Variance

Coef. Var.

Range

Skewness

Kurtosis

Median

IQR

Mode

Patient 2 Normal 2

Descriptive Statistics

High Functioning Patients

Impaired vs. Normal

Patient 2

Impaired

Normal 2

Unimpaired

0

5

10

15

20

25

30

35

Pe

rce

nt

0 10 20 30 40 50 60 70 80 90 100 110 120Normal Data 2

Histogram


Low, Normal, and High (Genuine) Premorbid Function without Impairment

p = .11 p = .61 p = .68


Low, Normal, and High (Genuine) Premorbid Function with Impairment

p < .0001 p < .0001 p < .0001


Low, Normal, and High (Exaggerating) Premorbid Function - Dubious Impairment

p < .0001 p < .0001 p < .0001


1 38.43 38.43 9.34 .0025 9.34 .88

1 97.45 97.45 23.68 <.0001 23.68 1.00

1 .52 .52 .13 .7222 .13 .06

239 983.61 4.12

DF Sum of Squares Mean Square F-Value P-Value Lambda Power

Sev.2 Grp

SVT.2 Grp

Sev.2 Grp * SVT.2 Grp

Residual

44 9.3 1.9 .3

31 10.6 2.0 .4

82 10.1 2.0 .2

86 11.6 2.1 .2


Normal, Genuine

Normal, Exaggerate

Impaired, Genuine

Impaired, Exaggerate

0

2

4

6

8

10

12

Ce

ll M

ea

n

Normal Impaired

Cell

ExaggerateGenuine

Intra-Individual Variability (IIV)


Low scores are relatively

common across all test batteries

1. Average subjects can be expected to obtain a distribution of test scores in which approximately 16% will fall 1 SD below the mean.

2. The range of abnormal scores obtained by an average person might be as low as 8% or as high as 28%.

3. In fact, for the eight NAB normative subjects who Iverson (2010) identified as possibly suffering from impairment, on a battery of 25 test scores, the average number of abnormal scores per subject was 18%, with a minimum of 8% and a maximum of 28%.


Low scores depend on where

you set your cutoff score

4. For any particular battery administered to an average person, with abnormality defined as 1 SD below the normative mean, will result in 16% of the scores being labeled as abnormal, with a range from 8% to 28%.

5. Similarly, with abnormality defined as 1.5 SD below the normative mean will result in 7% of the scores being labeled as abnormal, w/ range from 3% to 14%.

6. Finally, with abnormality defined as 2.0 SD below the normative mean will result in 2% of the scores being labeled as abnormal, with a range from 1% to 5%.


Low scores vary by number of

tests administered

7. Multiple the number of test scores obtained by the percentage of scores expected to fall below the selected cutoff and you will obtain the number of expected abnormal scores.

8. Giving more tests results in more abnormal test

scores, but the percentage of abnormal test scores does not change by increasing the number of tests administered in the battery. More scores will increase the reliability and validity of the assessment.


Low scores vary by

demographics of the examinee

9. Higher levels of premorbid functioning will result in a lower frequency of obtained abnormal scores.

10. Lower levels of premorbid functioning will result in a higher frequency of obtained abnormal scores.

11. Demographic variables, such as age and education, influence levels of premorbid functioning.


Low scores vary by level of

premorbid intelligence

12. Higher levels of premorbid functioning will result in a lower frequency of obtained abnormal scores.

13. Lower levels of premorbid functioning will result in a higher frequency of obtained abnormal scores.

14. Demographic variables, such as age and education, influence levels of premorbid functioning.

Intra-Individual Variability: OTBM by OTBM-SD

Documents

Transcript of Intra-Individual Variability: OTBM by OTBM-SD