Diagnostics 1 Methods Comparison Studies for Quantitative Nucleic Acid Assays Jacqueline Law, Art...

44
Diagnostics 1 Methods Comparison Studies for Quantitative Nucleic Acid Assays Jacqueline Law, Art DeVault Roche Molecular Systems Sept 19, 2003

Transcript of Diagnostics 1 Methods Comparison Studies for Quantitative Nucleic Acid Assays Jacqueline Law, Art...

Page 1: Diagnostics 1 Methods Comparison Studies for Quantitative Nucleic Acid Assays Jacqueline Law, Art DeVault Roche Molecular Systems Sept 19, 2003.

Dia

gn

osti

cs

1

Methods Comparison Studies for Quantitative

Nucleic Acid Assays

Jacqueline Law, Art DeVaultRoche Molecular Systems

Sept 19, 2003

Page 2: Diagnostics 1 Methods Comparison Studies for Quantitative Nucleic Acid Assays Jacqueline Law, Art DeVault Roche Molecular Systems Sept 19, 2003.

Dia

gn

osti

cs

2

OutlineOutline

IntroductionPCR based quantitative nucleic acid

assaysLiterature referencesAcceptance criteriaExamplesReferences

IntroductionPCR based quantitative nucleic acid

assaysLiterature referencesAcceptance criteriaExamplesReferences

Page 3: Diagnostics 1 Methods Comparison Studies for Quantitative Nucleic Acid Assays Jacqueline Law, Art DeVault Roche Molecular Systems Sept 19, 2003.

Dia

gn

osti

cs

3

Methods Comparison Studies: to validate a new assayMethods Comparison Studies: to validate a new assay

Purposes:To show that the new assay has good

agreement with the reference assaysTo show that the assay performs similarly with

different types of specimen

Premises of methods comparison studies:A linear relationship between the two assays LOD, dynamic range have to be already

established Appropriate transformation to normalize the

data

Analysis: To detect constant bias and proportional bias

Purposes:To show that the new assay has good

agreement with the reference assaysTo show that the assay performs similarly with

different types of specimen

Premises of methods comparison studies:A linear relationship between the two assays LOD, dynamic range have to be already

established Appropriate transformation to normalize the

data

Analysis: To detect constant bias and proportional bias

Page 4: Diagnostics 1 Methods Comparison Studies for Quantitative Nucleic Acid Assays Jacqueline Law, Art DeVault Roche Molecular Systems Sept 19, 2003.

Dia

gn

osti

cs

4

Constant Bias: the difference between the two methods is constant across the data range

Constant Bias: the difference between the two methods is constant across the data range

Method X

Me

tho

d Y

2 3 4 5 6 7 8

34

56

78

Method Y vs. Method X

Average

Diff

ere

nce

3 4 5 6 7 8

-2-1

01

2

Difference vs. Average

Page 5: Diagnostics 1 Methods Comparison Studies for Quantitative Nucleic Acid Assays Jacqueline Law, Art DeVault Roche Molecular Systems Sept 19, 2003.

Dia

gn

osti

cs

5

Proportional Bias: the difference between the two methods is linear across the data range

Proportional Bias: the difference between the two methods is linear across the data range

Method X)

Me

tho

d Y

3 4 5 6 7

34

56

78

Method Y vs. Method X

Average

Diff

ere

nce

3 4 5 6 7 8

-0.5

0.0

0.5

1.0

Difference vs. Average

Page 6: Diagnostics 1 Methods Comparison Studies for Quantitative Nucleic Acid Assays Jacqueline Law, Art DeVault Roche Molecular Systems Sept 19, 2003.

Dia

gn

osti

cs

6

PCR based nucleic acid assaysPCR based nucleic acid assays

To quantify the viral load by PCR method

Characteristics:A wide dynamic range (e.g. 10cp/mL to

1E7 cp/mL)Skewed distribution (non-normal):

typically log10 transformation for the dataHeteroscedasticity: variance is higher at

higher titer levels log10 transformation may not achieve

homogeneity in variance (variance at lower end may increase)

Other transformation:

To quantify the viral load by PCR method

Characteristics:A wide dynamic range (e.g. 10cp/mL to

1E7 cp/mL)Skewed distribution (non-normal):

typically log10 transformation for the dataHeteroscedasticity: variance is higher at

higher titer levels log10 transformation may not achieve

homogeneity in variance (variance at lower end may increase)

Other transformation: 2log 2 xx

Page 7: Diagnostics 1 Methods Comparison Studies for Quantitative Nucleic Acid Assays Jacqueline Law, Art DeVault Roche Molecular Systems Sept 19, 2003.

Dia

gn

osti

cs

7

PCR based assays: a wide dynamic range - data are log10 transformed

PCR based assays: a wide dynamic range - data are log10 transformed

Nominal Titer (cp/mL)

Observ

ed T

iter

(cp/m

L)

0 2*10^7 6*10^7 10^8 1.4*10^8

05*1

0^7

10^8

1.5

*10^8

2*1

0^8

2.5

*10^8

3*1

0^8

Untransformed Data

Nominal Titer (Log cp/mL)

Observ

ed T

iter

(Log c

p/m

L)

2 4 6 8

-4-2

02

46

8

Log10 Titer

Page 8: Diagnostics 1 Methods Comparison Studies for Quantitative Nucleic Acid Assays Jacqueline Law, Art DeVault Roche Molecular Systems Sept 19, 2003.

Dia

gn

osti

cs

8

PCR based assays: log10 transformation may remove some skewness

PCR based assays: log10 transformation may remove some skewness

Untransformed

log10 transformed

0 2 4 6 8 10

0.0

0.1

0.2

0.3

0.4

Titer

7 cp/mL

20 40 60 80 100 120 140

0.0

0.01

00.

020

0.03

0

Titer

7.2E1 cp/mL

6*10^6 10^7 1.4*10^7

05*

10^-

81.

5*10

^-7

Titer

1.4E7 cp/mL

-1.5 -0.5 0.0 0.5 1.0

0.0

0.2

0.4

0.6

0.8

Titer

7 cp/mL

1.4 1.6 1.8 2.0 2.2

0.0

0.5

1.0

1.5

2.0

2.5

3.0

Titer

7.2E1 cp/mL

6.8 6.9 7.0 7.1 7.2

01

23

45

Titer

1.4E7 cp/mL

Page 9: Diagnostics 1 Methods Comparison Studies for Quantitative Nucleic Acid Assays Jacqueline Law, Art DeVault Roche Molecular Systems Sept 19, 2003.

Dia

gn

osti

cs

9

Literature references on Methods Comparison StudiesLiterature references on Methods Comparison Studies

Correlation coefficientOther coefficientsT-testBland-Altman plotOrdinary least squares regressionPassing-Bablok regressionDeming regression

Correlation coefficientOther coefficientsT-testBland-Altman plotOrdinary least squares regressionPassing-Bablok regressionDeming regression

Page 10: Diagnostics 1 Methods Comparison Studies for Quantitative Nucleic Acid Assays Jacqueline Law, Art DeVault Roche Molecular Systems Sept 19, 2003.

Dia

gn

osti

cs

10

Correlation coefficient R or R2Correlation coefficient R or R2

Measures the strength of linear relationship between two assays

Does not measure agreement: cannot detect constant or proportional bias

Correlation coefficient can be artificially high for assays that cover a wide range: how high is high? 0.95? 0.99? 0.995?

Measures the strength of linear relationship between two assays

Does not measure agreement: cannot detect constant or proportional bias

Correlation coefficient can be artificially high for assays that cover a wide range: how high is high? 0.95? 0.99? 0.995?

Page 11: Diagnostics 1 Methods Comparison Studies for Quantitative Nucleic Acid Assays Jacqueline Law, Art DeVault Roche Molecular Systems Sept 19, 2003.

Dia

gn

osti

cs

11

Other coefficientsOther coefficients

Concordance coefficient (Lin, 1989):Measures the strength of relationship

between two assays that fall on the 45o line through the origin

Gold-standard correlation coefficient (St.Laurent 1998):Measures the agreement between a new

assay and a gold standard

Concordance coefficient (Lin, 1989):Measures the strength of relationship

between two assays that fall on the 45o line through the origin

Gold-standard correlation coefficient (St.Laurent 1998):Measures the agreement between a new

assay and a gold standard

22122

21

212

C

GGDD

GGG SS

S

Page 12: Diagnostics 1 Methods Comparison Studies for Quantitative Nucleic Acid Assays Jacqueline Law, Art DeVault Roche Molecular Systems Sept 19, 2003.

Dia

gn

osti

cs

12

T-testT-test

Paired t-test on the difference in the measurements by two assays

Can only detect constant biasCannot detect proportional bias

Paired t-test on the difference in the measurements by two assays

Can only detect constant biasCannot detect proportional bias

Page 13: Diagnostics 1 Methods Comparison Studies for Quantitative Nucleic Acid Assays Jacqueline Law, Art DeVault Roche Molecular Systems Sept 19, 2003.

Dia

gn

osti

cs

13

Bland-Altman graphical analysis(Bland and Altman, 1986)Bland-Altman graphical analysis(Bland and Altman, 1986)

Methods:Plot the Difference of the two assays (D = X-Y)

vs. the Average of the two assays (A = (X+Y)/2)Visually inspect the plot and see if there are

any trends in the plot proportional biasSummarize the bias between the two assays by

the mean, SD, 95% CI constant biasModification: regress D with A, test if slope = 0

(Hawkins, 2002)

A useful visual tool: transformation, heteroscedasticity, outliers,

curvature

Methods:Plot the Difference of the two assays (D = X-Y)

vs. the Average of the two assays (A = (X+Y)/2)Visually inspect the plot and see if there are

any trends in the plot proportional biasSummarize the bias between the two assays by

the mean, SD, 95% CI constant biasModification: regress D with A, test if slope = 0

(Hawkins, 2002)

A useful visual tool: transformation, heteroscedasticity, outliers,

curvature

Page 14: Diagnostics 1 Methods Comparison Studies for Quantitative Nucleic Acid Assays Jacqueline Law, Art DeVault Roche Molecular Systems Sept 19, 2003.

Dia

gn

osti

cs

14

Bland Altman plot (continued)Bland Altman plot (continued)

Method X (log Titer)

Me

tho

d Y

(lo

g T

iter)

2 4 6 8

24

68

Method Y vs. Method X

Average (log Titer)

Diff

ere

nce

(lo

g T

iter)

3 4 5 6 7 8

-0.5

0.0

0.5

1.0

Difference vs. Average

Page 15: Diagnostics 1 Methods Comparison Studies for Quantitative Nucleic Acid Assays Jacqueline Law, Art DeVault Roche Molecular Systems Sept 19, 2003.

Dia

gn

osti

cs

15

Ordinary least-squares regressionOrdinary least-squares regression

Methods: Regress the observed data of the new assay

(Y) with those of the reference assay (X)Minimize the squared deviations from the

identity line in the vertical directionModifications: weighted least squares

Assumptions:The reference assay (X) is error free, or the

error is relatively small compared to the range of the measurements

e.g. in clinical chemistry studies, the measurement errors are minimal

Methods: Regress the observed data of the new assay

(Y) with those of the reference assay (X)Minimize the squared deviations from the

identity line in the vertical directionModifications: weighted least squares

Assumptions:The reference assay (X) is error free, or the

error is relatively small compared to the range of the measurements

e.g. in clinical chemistry studies, the measurement errors are minimal

Page 16: Diagnostics 1 Methods Comparison Studies for Quantitative Nucleic Acid Assays Jacqueline Law, Art DeVault Roche Molecular Systems Sept 19, 2003.

Dia

gn

osti

cs

16

Ordinary least-squares regression (continued)

Ordinary least-squares regression (continued)

If measurement errors exist in both assays, the estimates are biasedslope tends to be smallerintercept tends to be larger

If measurement errors exist in both assays, the estimates are biasedslope tends to be smallerintercept tends to be larger

Page 17: Diagnostics 1 Methods Comparison Studies for Quantitative Nucleic Acid Assays Jacqueline Law, Art DeVault Roche Molecular Systems Sept 19, 2003.

Dia

gn

osti

cs

17

Passing-Bablok regression (Passing and Bablok, 1983)Passing-Bablok regression (Passing and Bablok, 1983)

A nonparametric approach - robust to outliersMethods:

Estimate the slope by the shifted median of the slopes between all possible sets of two points (Theil estimate)

Confidence intervals by the rank techniques

Assumptions:The measurement errors in both assays follow the

same type of distribution (not necessarily normal)The ratio of the variance is a constant (variance not

necessarily constant across the range of data)The sampling distributions of the samples are

arbitrary

A nonparametric approach - robust to outliersMethods:

Estimate the slope by the shifted median of the slopes between all possible sets of two points (Theil estimate)

Confidence intervals by the rank techniques

Assumptions:The measurement errors in both assays follow the

same type of distribution (not necessarily normal)The ratio of the variance is a constant (variance not

necessarily constant across the range of data)The sampling distributions of the samples are

arbitrary

Page 18: Diagnostics 1 Methods Comparison Studies for Quantitative Nucleic Acid Assays Jacqueline Law, Art DeVault Roche Molecular Systems Sept 19, 2003.

Dia

gn

osti

cs

18

Deming regression(Linnet, 1990)Deming regression(Linnet, 1990)

Methods:Orthogonal least squares estimates: minimize the

squared deviation of the observed data from the regression line

Standard errors for the estimates obtained by Jackknife method

Weighted Deming regression when heteroscedastic

Assumptions:Measurement errors for both assays follow

independent normal distributions with mean 0Error variances are assumed to be proportional

(variance not necessarily constant across the range of data)

Methods:Orthogonal least squares estimates: minimize the

squared deviation of the observed data from the regression line

Standard errors for the estimates obtained by Jackknife method

Weighted Deming regression when heteroscedastic

Assumptions:Measurement errors for both assays follow

independent normal distributions with mean 0Error variances are assumed to be proportional

(variance not necessarily constant across the range of data)

Page 19: Diagnostics 1 Methods Comparison Studies for Quantitative Nucleic Acid Assays Jacqueline Law, Art DeVault Roche Molecular Systems Sept 19, 2003.

Dia

gn

osti

cs

19

Comparison of the 3 regression methods(Linnet, 1993)

Comparison of the 3 regression methods(Linnet, 1993)

Electrolyte study (homogeneous variance):OLS, Passing-Bablok: biased slope, large Type I

error, larger RMSE than DemingDeming: unbiased slope, correct Type I error

Metabolite study (heterogeneous variance):All have unbiased slope estimatesWeighted LS and weighted Deming are most

efficientType I error is large for OLS, weighted LS,

Deming and Passing-Bablok

Presence of outliers:Passing-Bablok is robust to outliersDeming regression requires detection of outliers

Electrolyte study (homogeneous variance):OLS, Passing-Bablok: biased slope, large Type I

error, larger RMSE than DemingDeming: unbiased slope, correct Type I error

Metabolite study (heterogeneous variance):All have unbiased slope estimatesWeighted LS and weighted Deming are most

efficientType I error is large for OLS, weighted LS,

Deming and Passing-Bablok

Presence of outliers:Passing-Bablok is robust to outliersDeming regression requires detection of outliers

Page 20: Diagnostics 1 Methods Comparison Studies for Quantitative Nucleic Acid Assays Jacqueline Law, Art DeVault Roche Molecular Systems Sept 19, 2003.

Dia

gn

osti

cs

20

SoftwareSoftware

Statistical packages: SAS, Splus

Other packages (for Bland-Altman plot, OLS regression, Passing-Bablok regression, Deming regression):Analyse-it (Excel add-on): does not

support weighted Deming regressionMethod Validator (a freeware)CBStat (Linnet K.)

Statistical packages: SAS, Splus

Other packages (for Bland-Altman plot, OLS regression, Passing-Bablok regression, Deming regression):Analyse-it (Excel add-on): does not

support weighted Deming regressionMethod Validator (a freeware)CBStat (Linnet K.)

Page 21: Diagnostics 1 Methods Comparison Studies for Quantitative Nucleic Acid Assays Jacqueline Law, Art DeVault Roche Molecular Systems Sept 19, 2003.

Dia

gn

osti

cs

21

Acceptance criteria for regression type analysisAcceptance criteria for regression type analysis

Independent acceptance criteria for slope and intercept estimates:e.g. slope estimate within (0.9, 1.1),

intercept estimate within (-0.2, 0.2)

Drawback: asymmetrical acceptance region across the data range

Independent acceptance criteria for slope and intercept estimates:e.g. slope estimate within (0.9, 1.1),

intercept estimate within (-0.2, 0.2)

Drawback: asymmetrical acceptance region across the data range

Page 22: Diagnostics 1 Methods Comparison Studies for Quantitative Nucleic Acid Assays Jacqueline Law, Art DeVault Roche Molecular Systems Sept 19, 2003.

Dia

gn

osti

cs

22

Asymmetrical acceptance regionAsymmetrical acceptance region

Method X (Log Titer)

Me

tho

d Y

(Lo

g T

iter)

2 4 6 8

24

68

Y = 0.2 + 1.1 * X

Slope=(0.9, 1.1), Int=(-0.2,0.2)

Y = -0.2 + 0.9 * X

Method X (Log Titer)

Bia

s =

Me

tho

d Y

- M

eth

od

X (

Lo

g T

iter)

2 4 6 8

-1.5

-1.0

-0.5

0.0

0.5

1.0

1.5

Asymmetrical Acceptance Region

Page 23: Diagnostics 1 Methods Comparison Studies for Quantitative Nucleic Acid Assays Jacqueline Law, Art DeVault Roche Molecular Systems Sept 19, 2003.

Dia

gn

osti

cs

23

Proposed acceptance criteriaProposed acceptance criteria

Goals: to show that the new assay is ‘equivalent’

to the reference assayto demonstrate that the bias between the

two assays is within some acceptable threshold across the clinical range

Acceptance Criteria:

Choice of tolerance level A: accuracy specification for the new assay

Goals: to show that the new assay is ‘equivalent’

to the reference assayto demonstrate that the bias between the

two assays is within some acceptable threshold across the clinical range

Acceptance Criteria:

Choice of tolerance level A: accuracy specification for the new assay

AXYEBiasE

Page 24: Diagnostics 1 Methods Comparison Studies for Quantitative Nucleic Acid Assays Jacqueline Law, Art DeVault Roche Molecular Systems Sept 19, 2003.

Dia

gn

osti

cs

24

Mathematical modelsMathematical models

Reference Assay:

New Assay:

where is the true concentration,

and are the independent random measurement errors

Bias:

i i i

i i i

i

i i

i i

X

Y

Y X

1

Acceptance Criteria: 1

i i i

i i iE Y X A

Page 25: Diagnostics 1 Methods Comparison Studies for Quantitative Nucleic Acid Assays Jacqueline Law, Art DeVault Roche Molecular Systems Sept 19, 2003.

Dia

gn

osti

cs

25

Comparison of the acceptance criteria:{Int (-0.2,0.2), Slope (0.9,1.1) } vs. { A= 0.5, L=2, U=7}

Comparison of the acceptance criteria:{Int (-0.2,0.2), Slope (0.9,1.1) } vs. { A= 0.5, L=2, U=7}

Method X (Log Titer)

Me

tho

d Y

(L

og

Tite

r)

2 3 4 5 6 7

23

45

67

Acceptance Region for the Data

Method X (Log Titer)

Bia

s: M

eth

od

Y -

Me

tho

d X

(L

og

Tite

r)

2 3 4 5 6 7

-1.5

-1.0

-0.5

0.0

0.5

1.0

1.5

Symmetrical Region

Page 26: Diagnostics 1 Methods Comparison Studies for Quantitative Nucleic Acid Assays Jacqueline Law, Art DeVault Roche Molecular Systems Sept 19, 2003.

Dia

gn

osti

cs

26

Acceptance region for the parameters:criteria for the intercept and slope are dependent

Acceptance region for the parameters:criteria for the intercept and slope are dependent

Intercept (Alpha)

Slo

pe

(B

eta

)

-0.5 0.0 0.5

0.8

0.9

1.0

1.1

1.2

Acceptance Region for the Parameters

Page 27: Diagnostics 1 Methods Comparison Studies for Quantitative Nucleic Acid Assays Jacqueline Law, Art DeVault Roche Molecular Systems Sept 19, 2003.

Dia

gn

osti

cs

27

Equivalence testEquivalence test

Methods:If the 90% two-sided confidence interval of

the Bias lies entirely within the acceptance region (- A, A), then the two assays are equivalent

Deming-Jackknife is used to do the estimation

Methods:If the 90% two-sided confidence interval of

the Bias lies entirely within the acceptance region (- A, A), then the two assays are equivalent

Deming-Jackknife is used to do the estimation

0 : vs. :aH Bias A H Bias A

where A is the accuracy specification of the new assay

Page 28: Diagnostics 1 Methods Comparison Studies for Quantitative Nucleic Acid Assays Jacqueline Law, Art DeVault Roche Molecular Systems Sept 19, 2003.

Dia

gn

osti

cs

28

Deming regression:(a.k.a. errors-in-variables regression, a structural or functional relationship model)

Deming regression:(a.k.a. errors-in-variables regression, a structural or functional relationship model)

Minimize the sum of squares:Minimize the sum of squares:

The solutions are given by:The solutions are given by:

2 2

1

n

i i i ii

S x y

2 21ˆ 42

ˆˆ

yy xx xx yy xyxy

S S S S SS

y x

where = Var()/Var() (assumed known or to be estimated)

Weighted Deming regression:Weighted Deming regression:

22 ˆˆ

11

iiii

YXSDw

Page 29: Diagnostics 1 Methods Comparison Studies for Quantitative Nucleic Acid Assays Jacqueline Law, Art DeVault Roche Molecular Systems Sept 19, 2003.

Dia

gn

osti

cs

29

Estimation of in Deming regressionEstimation of in Deming regression

Duplicate measurements:

>2 replicates: residual errors by ANOVA

Mis-specification of (Linnet 1998):biased slope estimatelarge Type I error

Duplicate measurements:

>2 replicates: residual errors by ANOVA

Mis-specification of (Linnet 1998):biased slope estimatelarge Type I error

2

2

221

2221

2

ˆ

2

1 ,

2

1

Y

X

iiYiiX

SD

SD

yyN

SDxxN

SD

Page 30: Diagnostics 1 Methods Comparison Studies for Quantitative Nucleic Acid Assays Jacqueline Law, Art DeVault Roche Molecular Systems Sept 19, 2003.

Dia

gn

osti

cs

30

Jackknife estimation: to obtain the final parameter estimates and the SEs

Jackknife estimation: to obtain the final parameter estimates and the SEs

Omit one pair of data at a time, obtain the Deming-regression estimates:

Omit one pair of data at a time, obtain the Deming-regression estimates:

The ith pseudo-values of the intercept and slope are:The ith pseudo-values of the intercept and slope are:

ˆ ˆ1

ˆ ˆ1

i i

i i

n n

n n

Final estimates and SEs for and are the mean and standard error of i and i

Final estimates and SEs for and are the mean and standard error of i and i

ii ˆ ,ˆ

Page 31: Diagnostics 1 Methods Comparison Studies for Quantitative Nucleic Acid Assays Jacqueline Law, Art DeVault Roche Molecular Systems Sept 19, 2003.

Dia

gn

osti

cs

31

Bias estimation by JackknifeBias estimation by Jackknife

( ) ( )1 1 1ii iBias n n

The bias estimate and the SE at each nominal level are the mean and SE of Biasi

The 90% CI of the bias at each nominal level are compared to the acceptance region (-A, A)

The two assays are concluded to be equivalent if all the CI lie entirely within (-A, A)

The bias estimate and the SE at each nominal level are the mean and SE of Biasi

The 90% CI of the bias at each nominal level are compared to the acceptance region (-A, A)

The two assays are concluded to be equivalent if all the CI lie entirely within (-A, A)

At each nominal level , the ith pseudo-value of the Bias is:

At each nominal level , the ith pseudo-value of the Bias is:

Page 32: Diagnostics 1 Methods Comparison Studies for Quantitative Nucleic Acid Assays Jacqueline Law, Art DeVault Roche Molecular Systems Sept 19, 2003.

Dia

gn

osti

cs

32

Example 1: methods comparison for two HIV-1 assays

Example 1: methods comparison for two HIV-1 assays

Reference Method (log Titer)

New

Met

hod

(log

Tite

r)

3 4 5 6

34

56

Methods Comparison Study

Page 33: Diagnostics 1 Methods Comparison Studies for Quantitative Nucleic Acid Assays Jacqueline Law, Art DeVault Roche Molecular Systems Sept 19, 2003.

Dia

gn

osti

cs

33

Bland-Altman plot: potential outliers in the dataBland-Altman plot: potential outliers in the data

Average of Reference and New (log Titer)

Diff

ere

nce

= N

ew

- R

efe

ren

ce (

log

Tite

r)

3 4 5 6

-0.5

0.0

0.5

1.0

Bland-Altman Plot: Full Data Set

Page 34: Diagnostics 1 Methods Comparison Studies for Quantitative Nucleic Acid Assays Jacqueline Law, Art DeVault Roche Molecular Systems Sept 19, 2003.

Dia

gn

osti

cs

34

Identify outliers: fitting a linear regression line to the Bland Altman plotIdentify outliers: fitting a linear regression line to the Bland Altman plot

Fitted Difference

Stu

dent

ized

Res

idua

l

0.0 0.01 0.02 0.03 0.04

-3-2

-10

12

3

Residual Plot for Diff vs. Avg

34860

34794

Samples

Leve

rage

0 10 20 30 40 50

0.02

0.04

0.06

0.08

0.10

34944

34851

Leverage

Page 35: Diagnostics 1 Methods Comparison Studies for Quantitative Nucleic Acid Assays Jacqueline Law, Art DeVault Roche Molecular Systems Sept 19, 2003.

Dia

gn

osti

cs

35

Remove outliers: Bland-Altman plot shows no trend in Difference vs. AverageRemove outliers: Bland-Altman plot shows no trend in Difference vs. Average

Average of Reference and New (log Titer)

Diff

eren

ce =

New

- R

efer

ence

(lo

g T

iter)

3 4 5 6

-0.6

-0.4

-0.2

0.0

0.2

0.4

Bland-Altman Plot

slope = 0.033 (p-value = 0.5)

mean difference = 0.02

(95% CI: -0.06, 0.10)

Page 36: Diagnostics 1 Methods Comparison Studies for Quantitative Nucleic Acid Assays Jacqueline Law, Art DeVault Roche Molecular Systems Sept 19, 2003.

Dia

gn

osti

cs

36

Regression analysis: results from the 3 methods are very similar

Regression analysis: results from the 3 methods are very similar

Reference Method (log Titer)

New

Met

hod

(log

Tite

r)

3 4 5 6

34

56

Regression Analysis

OLS: Y=0.084+0.986X Passing-Bablok: Y=-0.137+1.044XDeming-Jackknife: Y=-0.127+1.033X

Page 37: Diagnostics 1 Methods Comparison Studies for Quantitative Nucleic Acid Assays Jacqueline Law, Art DeVault Roche Molecular Systems Sept 19, 2003.

Dia

gn

osti

cs

37

Bias estimation: almost all 90% CI lie within the tolerance bounds (-0.2, +0.2)Bias estimation: almost all 90% CI lie within the tolerance bounds (-0.2, +0.2)

Reference Method (log Titer)

Diff

eren

ce =

New

- R

efer

ence

(lo

g T

iter)

3 4 5 6

-0.6

-0.4

-0.2

0.0

0.2

0.4

Estimated Bias (90% CI) by Deming-Jackknife Regression

Page 38: Diagnostics 1 Methods Comparison Studies for Quantitative Nucleic Acid Assays Jacqueline Law, Art DeVault Roche Molecular Systems Sept 19, 2003.

Dia

gn

osti

cs

38

Example 2: to show matrix equivalency between EDTA Plasma and SerumExample 2: to show matrix equivalency between EDTA Plasma and Serum

EDTA (log Titer)

Ser

um (

log

Tite

r)

2 3 4 5 6 7 8

23

45

67

8

Matrix Equivalency Study

Page 39: Diagnostics 1 Methods Comparison Studies for Quantitative Nucleic Acid Assays Jacqueline Law, Art DeVault Roche Molecular Systems Sept 19, 2003.

Dia

gn

osti

cs

39

Bland-Altman plot on average titer:most titers higher than 1E5 IU/mL, heteroscedasticity?

Bland-Altman plot on average titer:most titers higher than 1E5 IU/mL, heteroscedasticity?

Average of EDTA and Serum (log Titer)

Diff

ere

nce

= S

eru

m -

ED

TA (

log

Tite

r)

3 4 5 6 7

-1.0

-0.5

0.0

0.5

Bland-Altman Plot

slope = 0.03 (p-value = 0.6)

mean difference = -0.06 (95% CI: -0.16, 0.04)

Page 40: Diagnostics 1 Methods Comparison Studies for Quantitative Nucleic Acid Assays Jacqueline Law, Art DeVault Roche Molecular Systems Sept 19, 2003.

Dia

gn

osti

cs

40

Checking for heteroscedasticity:residual errors from random effects models

Checking for heteroscedasticity:residual errors from random effects models

Mean EDTA (log Titer)

SD

ED

TA

(lo

g T

ite

r)

3 4 5 6 7

0.0

20

.04

0.0

60

.08

0.1

00

.12

0.1

4

EDTA

Mean Serum (log Titer)

SD

Se

rum

(lo

g T

ite

r)

3 4 5 6 7

0.0

20

.04

0.0

60

.08

0.1

00

.12

0.1

4

Serum

Page 41: Diagnostics 1 Methods Comparison Studies for Quantitative Nucleic Acid Assays Jacqueline Law, Art DeVault Roche Molecular Systems Sept 19, 2003.

Dia

gn

osti

cs

41

1: Pooled within-sample SD for EDTA = 0.0706Pooled within-sample SD for Serum = 0.0715

1: Pooled within-sample SD for EDTA = 0.0706Pooled within-sample SD for Serum = 0.0715

Average of EDTA and Serum (log Titer)

Va

r(E

DTA

Err

ors

)/V

ar(

Se

rum

Err

ors

)

3 4 5 6 7

02

46

8 Median Lambda = 0.945

Lambda Estimation

Page 42: Diagnostics 1 Methods Comparison Studies for Quantitative Nucleic Acid Assays Jacqueline Law, Art DeVault Roche Molecular Systems Sept 19, 2003.

Dia

gn

osti

cs

42

Bias estimation: large variability at low titers due to sparse data - fail to demonstrate equivalency at low end

Bias estimation: large variability at low titers due to sparse data - fail to demonstrate equivalency at low end

EDTA (log Titer)

Diff

eren

ce =

Ser

um -

ED

TA (

log

Tite

r)

3 4 5 6 7

-1.0

-0.5

0.0

0.5

Estimated Bias (90% CI) by Deming-Jackknife Regression

Page 43: Diagnostics 1 Methods Comparison Studies for Quantitative Nucleic Acid Assays Jacqueline Law, Art DeVault Roche Molecular Systems Sept 19, 2003.

Dia

gn

osti

cs

43

ReferencesReferences

Bland M., Altman D. (1986). ‘Statistical methods for assessing agreement between two methods of clinical measurement’. Lancet 347: 307-310.

Hawkins D. (2002). ‘Diagnostics for conformity of paired quantitative measurements’. Stat in Med 21: 1913-1935.

Lin L.K. (1989). ‘A concordance correlation coefficient to evaluate reproducibility’. Biometrics 45: 255-268.

Linnet K. (1990). ‘Estimation of the linear relationship between the measurements of two methods with proportional bias’. Stat in Med 9: 1463-1473.

Linnet K. (1993). ‘Evaluation of regression procedures for methods comparison studies’. Clin Chem 39: 424-432.

Linnet K. (1998). ‘Performance of Deming regression analysis in case of misspecified analytical error ratio in method comparisons studies’. Clin Chem 44: 1024-1031.

Linnet K. (1999). ‘Necessary sample size for method comparison studies based on regression analysis’. Clin Chem 45: 882-894.

Bland M., Altman D. (1986). ‘Statistical methods for assessing agreement between two methods of clinical measurement’. Lancet 347: 307-310.

Hawkins D. (2002). ‘Diagnostics for conformity of paired quantitative measurements’. Stat in Med 21: 1913-1935.

Lin L.K. (1989). ‘A concordance correlation coefficient to evaluate reproducibility’. Biometrics 45: 255-268.

Linnet K. (1990). ‘Estimation of the linear relationship between the measurements of two methods with proportional bias’. Stat in Med 9: 1463-1473.

Linnet K. (1993). ‘Evaluation of regression procedures for methods comparison studies’. Clin Chem 39: 424-432.

Linnet K. (1998). ‘Performance of Deming regression analysis in case of misspecified analytical error ratio in method comparisons studies’. Clin Chem 44: 1024-1031.

Linnet K. (1999). ‘Necessary sample size for method comparison studies based on regression analysis’. Clin Chem 45: 882-894.

Page 44: Diagnostics 1 Methods Comparison Studies for Quantitative Nucleic Acid Assays Jacqueline Law, Art DeVault Roche Molecular Systems Sept 19, 2003.

Dia

gn

osti

cs

44

References (continued)References (continued)

Passing H., Bablok W. (1983). ‘A new biometrical procedure for testing the equality of measurements from two different analytical methods’. J Clin Chem Clin Biochem 21: 709-720.

Passing H., Bablok W. (1984). ‘Comparison of several regression procedures for method comparison studies and determination of sample sizes’. J Clin Chem Clin Biochem 22: 431-445.

St. Laurent R.T. (1998). ‘Evaluating Agreement with a Gold Standard in Method Comparison Studies’. Biometrics 54: 537-545.

Passing H., Bablok W. (1983). ‘A new biometrical procedure for testing the equality of measurements from two different analytical methods’. J Clin Chem Clin Biochem 21: 709-720.

Passing H., Bablok W. (1984). ‘Comparison of several regression procedures for method comparison studies and determination of sample sizes’. J Clin Chem Clin Biochem 22: 431-445.

St. Laurent R.T. (1998). ‘Evaluating Agreement with a Gold Standard in Method Comparison Studies’. Biometrics 54: 537-545.