Total Analytical Error Total Analytical Error Evaluation in Quantitative Evaluation in Quantitative

and Qualitative Assaysand Qualitative Assays May F. MoMay F. Mo

FDA/Industry Statistics WorkshopFDA/Industry Statistics Workshop

Washington, DC - September 29, 2006Washington, DC - September 29, 2006

BackgroundBackground

Total Analytical Error (TAE)Total Analytical Error (TAE) NOT a new conceptNOT a new concept Assay errors from Assay errors from all sourcesall sources of the of the

data collection experiment – including data collection experiment – including systematic and random errorsystematic and random error

CLSI Proposed Guideline January 2002CLSI Proposed Guideline January 2002 CLSI Approved Guideline April 2003CLSI Approved Guideline April 2003 FDA/Industry collaboration in FDA/Industry collaboration in

developing the guidelinedeveloping the guideline

Cause and Effect DiagramCause and Effect Diagramfor Total Analytical Errorfor Total Analytical Error

(from EP21-A)(from EP21-A)

Total Analytical Error

EP21

Random Error Systematic Error

Protocol Specific BiasEP10

ImprecisionEP5

Random Interferences

Protocol Independent

BiasEP9/EP6/EP7

DriftEP10

Sample CarryoverEP10

Constant BiasEP9

Linear BiasEP9

Non-linear BiasEP6

Reagent CarryoverInterferences

EP7

Importance of TAEImportance of TAE

For CliniciansFor Clinicians TAE provides a metric of assay quality TAE provides a metric of assay quality

that can directly equate to medical errorsthat can directly equate to medical errors For LaboratoriesFor Laboratories

TAE provides a simple, cost effective TAE provides a simple, cost effective method for comparing assay productsmethod for comparing assay products

For ManufacturersFor Manufacturers TAE guides manufacturers to set TAE guides manufacturers to set

appropriate design goals and evaluate appropriate design goals and evaluate the product directly to user needsthe product directly to user needs

TAE Evaluation MethodsTAE Evaluation Methods

TAE is evaluated by assessing if the TAE is evaluated by assessing if the combination of errors from all combination of errors from all sources is within some acceptance sources is within some acceptance limit. The common evaluation limit. The common evaluation methods are (Krouwer, 2002):methods are (Krouwer, 2002): Direct EstimationDirect Estimation Indirect EstimationIndirect Estimation Simulated EstimationSimulated Estimation

Note: Note: All methods are based on the All methods are based on the quantitative assays.quantitative assays.

Direct EstimationDirect Estimation

Direct EstimationDirect Estimation ( (Distribution of Distribution of Differences MethodDifferences Method)) EP21 RecommendationEP21 Recommendation Calculate differences of study assay vs. Calculate differences of study assay vs.

reference assay reference assay TAE limit estimated as the CLI of the TAE limit estimated as the CLI of the

differencesdifferences

- No need to estimate and combine individual - No need to estimate and combine individual source errorssource errors

Visual tools - Bland & Altman and “Mountain Visual tools - Bland & Altman and “Mountain Plot” Plot”

Direct Estimation Direct Estimation ((ContinuedContinued))

Is distribution of differences normal?Is distribution of differences normal?Parametric? Nonparametric? or Data Parametric? Nonparametric? or Data

Transformation?Transformation? Tolerance IntervalTolerance Interval: A range that contains a : A range that contains a

specified proportion of a sampled population specified proportion of a sampled population with a specified confidence. (EP21)with a specified confidence. (EP21)

Parametric or nonparametric methods to Parametric or nonparametric methods to report tolerance intervals are covered in report tolerance intervals are covered in Statistical IntervalsStatistical Intervals by Hahn and Meeker by Hahn and Meeker (1991). (1991).

Tolerance Interval (TI) Tolerance Interval (TI) CalculationCalculation

A two-sided 100(1-A two-sided 100(1-)% TI to contain at least 100)% TI to contain at least 100pp% % of the sampled differences from a sample of size n:of the sampled differences from a sample of size n:

Parametric:Parametric:

where k depends on where k depends on , , pp and and n, n, and can be and can be obtained from published tables, or derived using obtained from published tables, or derived using noncentral t-distribution (Odeh and Owen, 1980)noncentral t-distribution (Odeh and Owen, 1980)

NonparametricNonparametric

TI based on rank statistics is [TI based on rank statistics is [xx(l)(l), , xx(u)(u)].].The value of The value of ll and and uu depend on depend on , , pp and and n, n, and can and can be derived using published tables, or using be derived using published tables, or using equation provided in Hahn and Meeker (1991)equation provided in Hahn and Meeker (1991)

)*( skx

Indirect EstimationIndirect Estimation

Indirect EstimationIndirect Estimation ( (Simple Combination Simple Combination ModelModel)) TAE = Bias + k * ImprecisionTAE = Bias + k * Imprecision Simple to applySimple to apply Fits well with the Six-Sigma quality conceptsFits well with the Six-Sigma quality concepts Not recommended by EP21 - Underestimate TAENot recommended by EP21 - Underestimate TAE

Simulation Method (Simulation Method (Research UseResearch Use)) Cost-efficientCost-efficient Model the impact of an individual error component Model the impact of an individual error component

to TAEto TAE Can underestimate TAECan underestimate TAE

Industry PracticesIndustry PracticesQuantitative AssaysQuantitative Assays

Set Product RequirementSet Product Requirement Set allowable TAE (aTAE) limit based on Set allowable TAE (aTAE) limit based on

medical opinion and marketing needsmedical opinion and marketing needs Break aTAE down to precision and bias design Break aTAE down to precision and bias design

requirements using simple combination model. requirements using simple combination model.

Example: TAE = Bias + 2*SD (for 2 sigma Example: TAE = Bias + 2*SD (for 2 sigma performance)performance)

Design Verification and ValidationDesign Verification and Validation Verify design requirements individuallyVerify design requirements individually A direct assessment of total error should be A direct assessment of total error should be

conducted to ensure aTAE is met.conducted to ensure aTAE is met.

Industry PracticesIndustry PracticesQuantitative AssaysQuantitative Assays

TAE Claims in Package InsertsTAE Claims in Package Inserts Not Yet CommonNot Yet Common Concerns?Concerns?

Indirect method – Is it sufficient to establish a Indirect method – Is it sufficient to establish a TAE claim?TAE claim?

Direct method - When the reference assay is Direct method - When the reference assay is not a “gold standard”, TAE has measurement not a “gold standard”, TAE has measurement error from both assays … How to interpret?error from both assays … How to interpret?

User environment – Are our customers ready User environment – Are our customers ready to use TAE claims in the package inserts? to use TAE claims in the package inserts?

Generalize TAE toGeneralize TAE toQualitative AssaysQualitative Assays

MotivationMotivation TAE provides rationale to establish TAE provides rationale to establish

product requirementsproduct requirements TAE provides a final metric to TAE provides a final metric to

evaluate if a product meets medical evaluate if a product meets medical needsneeds

ChallengeChallenge TAE is a quantitative conceptTAE is a quantitative concept TAE does not apply to low negative TAE does not apply to low negative

and high positive samples and high positive samples

TAE in Qualitative AssaysTAE in Qualitative Assays

The performance of a qualitative The performance of a qualitative assay relies on its precision and bias assay relies on its precision and bias near the medical decision point near the medical decision point (cutoff) … (cutoff) … TAE evaluation applicable and desirable TAE evaluation applicable and desirable

for the cutoff zonefor the cutoff zone Direct estimation not realistic due to Direct estimation not realistic due to

limited patient samples around cutofflimited patient samples around cutoff Indirect modeled approach may be more Indirect modeled approach may be more

practicalpractical

aTAE in Qualitative aTAE in Qualitative AssaysAssays

How to define aTAE for the near cutoff How to define aTAE for the near cutoff zone?zone?

EP12-A Guidance:EP12-A Guidance: The 95% interval - the values above

and below the cutoff point at which repeated results are 95% positive or 95% negative, respectively

Grayzone - the zone of uncertainty between the 95% interval limits

aTAE in Qualitative aTAE in Qualitative AssaysAssays

Cutoff

5%5%

Upper 95%Interval

Lower 95%Interval

Grayzone

aTAE in Qualitative aTAE in Qualitative AssaysAssays

Set aTAE for the near cutoff zoneSet aTAE for the near cutoff zone Incorporating measurement errors from both

precision and bias, aTAE can be set as ½ of the grayzone. That is, the test result at 1.2 S/CO should read as positive (>1.0 S/CO) while test results at 0.8 S/CO should read as negative (<1.0 S/CO). Assuming that grayzone is 0.8-1.2 S/CO for an assay, aTAE will be 0.2 S/CO or 20% (assuming cutoff=1.0 S/CO)

When the above aTAE is met, there should be < (i.e., 0.05) probability that a test results will deviate >20% from the true value, and cause a change in the final interpretation.

aTAE in Qualitative aTAE in Qualitative AssaysAssays

Set aTAE for the near cutoff zoneSet aTAE for the near cutoff zone For an assay that reports an indeterminate

result, aTAE can be extended as the size the grayzone.

The test result at 1.2 S/CO should not read as negative (<0.8 S/CO) while test results at 0.8 S/CO should not read as positive (>1.2 S/CO). In this case aTAE can be set at 0.4 S/CO or 50% for results around 0.8 S/CO and 33% for results around 1.2 S/CO.

TAE Evaluation TAE Evaluation for Qualitative assayfor Qualitative assay

Direct Evaluation – Not realisticDirect Evaluation – Not realistic Indirect EvaluationIndirect Evaluation

TAE = Bias + TAE = Bias + * Precision * PrecisionChallenge:Challenge: How to define and measure How to define and measure Bias?Bias?

How to include ALL bias How to include ALL bias sources?sources?

Alternative: TAE = Alternative: TAE = * Reproducibility * ReproducibilityReproducibility includes variations contributed Reproducibility includes variations contributed from instrument, lot, calibration, operator, etc from instrument, lot, calibration, operator, etc … …

Underestimate TAE is still then main concern

TAE in Qualitative AssaysTAE in Qualitative Assays

How do different types of bias impact TAE?How do different types of bias impact TAE? Bias impacts both S and COBias impacts both S and CO

As a proportional biasAs a proportional bias

The will be no bias in S/COThe will be no bias in S/CO As a fixed biasAs a fixed bias

The bias in S/CO is unknown and depends on The bias in S/CO is unknown and depends on the value of S relative to CO (=1.0) and size the value of S relative to CO (=1.0) and size of the biasof the bias

COSCO

S

COCO

SSCOS /*

1

1

*

*/

unknownCO

SCOS

/

CO=1.0 Bias

S -0.2 -0.1 0.1 0.2

0.8 0.75 0.78 0.82 0.83

0.9 0.88 0.89 0.91 0.92

1.0 1.00 1.00 1.00 1.00

1.1 1.13 1.11 1.09 1.08

1.2 1.25 1.22 1.18 1.17

TAE in Qualitative AssaysTAE in Qualitative Assays

How does different type of biases impact How does different type of biases impact TAE?TAE? Fixed bias for both S and CO: The impact on Fixed bias for both S and CO: The impact on

S/CO is relatively small for the near cutoff S/CO is relatively small for the near cutoff range (0.8-1.2 S/CO)range (0.8-1.2 S/CO)

Biases that impact S and CO differently:Biases that impact S and CO differently: InterferencesInterferences Tube TypeTube Type StorageStorage Sample Carry OverSample Carry Over Crossreactivity …Crossreactivity …

ConclusionConclusion

TAE is an useful evaluation method TAE is an useful evaluation method for both quantitative and qualitative for both quantitative and qualitative assays. For qualitative assays, TAE assays. For qualitative assays, TAE should be evaluated for the near should be evaluated for the near cutoff zone using indirect estimation. cutoff zone using indirect estimation. Since indirect method Since indirect method underestimates TAE, it may not be underestimates TAE, it may not be sufficient to support a claim. sufficient to support a claim.

ReferencesReferences NCCLS. Estimation of Total Analytical Error for Clinical NCCLS. Estimation of Total Analytical Error for Clinical

Laboratory Methods; Approved Guideline. NCCLS Laboratory Methods; Approved Guideline. NCCLS Document EP21-A [ISBN 1-56238-502-X]. NCCLS, 940 West Document EP21-A [ISBN 1-56238-502-X]. NCCLS, 940 West Valley Road, Suite 1400, Wayne, Pennsylvania 19087-1898 Valley Road, Suite 1400, Wayne, Pennsylvania 19087-1898 USA, 2003.USA, 2003.

NCCLS. User Protocol for Evaluation of Qualitative Test NCCLS. User Protocol for Evaluation of Qualitative Test Performance; Approved Guideline. NCCLS Document EP12-Performance; Approved Guideline. NCCLS Document EP12-A [ISBN 1-56238-468-6]. NCCLS, 940 West Valley Road, A [ISBN 1-56238-468-6]. NCCLS, 940 West Valley Road, Suite 1400, Wayne, Pennsylvania 19087-1898 USA, 2002.Suite 1400, Wayne, Pennsylvania 19087-1898 USA, 2002.

Hahn GJ, Meeker WQ. Statistical Intervals: A Guide for Hahn GJ, Meeker WQ. Statistical Intervals: A Guide for Practitioners. New York: John Wiley and Sons, 1991.Practitioners. New York: John Wiley and Sons, 1991.

Krouwer JS, Monti KL. A simple graphical method to Krouwer JS, Monti KL. A simple graphical method to evaluate laboratory assays. Eur J Clin Chem Biochem 1995; evaluate laboratory assays. Eur J Clin Chem Biochem 1995; 33:525–7.33:525–7.

Odeh, RE, Owen, DB. Tables for Normal Tolerance Limits, Odeh, RE, Owen, DB. Tables for Normal Tolerance Limits, Sampling Plans, and Screening. New York Marcel Dekker, Sampling Plans, and Screening. New York Marcel Dekker, 1980.1980.