Statistical Considerations in Biomarker Method Development ... 2004 Devanarayan.pdf ·...
Transcript of Statistical Considerations in Biomarker Method Development ... 2004 Devanarayan.pdf ·...
Statistical Considerations in Biomarker Method Development & Validation
Viswanath Devanarayan, Ph.D.Eli Lilly & CompanyE-Mail: [email protected]
Presented at BASS-XI, November 1, 2004
V. Devanarayan, Biomarker MethodsBASS-XI, November 1, 2004
Copyright © 2004 Eli Lilly and Company 2
Acknowledgements
Biologists:• Len Boggs• Ron Bowsher• Karen Cox• Jean Lee• Peter O’Brien• Chad Ray• Sitta Sittampalam
Statisticians:• Bruno Boulanger• Ray Carroll• Walthere Dewe• Wendell Smith
V. Devanarayan, Biomarker MethodsBASS-XI, November 1, 2004
Copyright © 2004 Eli Lilly and Company 3
Outline
1. Assay/Method Background
2. Fundamental Validity, Similarity, Parallelism
3. Types of Biomarker Methods
4. Standard Curves, Weighting, Precision Profiles
5. Assay/Method Optimization
6. Pre-Study & In-Study Validation
7. Acceptance Criteria
8. Summary (Flow Scheme)
V. Devanarayan, Biomarker MethodsBASS-XI, November 1, 2004
Copyright © 2004 Eli Lilly and Company 4
1. Assay/Method Background
V. Devanarayan, Biomarker MethodsBASS-XI, November 1, 2004
Copyright © 2004 Eli Lilly and Company 5
Assay Purpose (Finney)
A (biological) assay is an experiment run to estimate the nature, constitution, or potency of a material, by means of the reaction that follows its application (to living matter).
V. Devanarayan, Biomarker MethodsBASS-XI, November 1, 2004
Copyright © 2004 Eli Lilly and Company
Stimulus
• The size of the stimulus (“dose”) is varied to obtain a dose response curve.
• “Potency” of a test sample is estimated by comparing its response to that of a standard preparation.
ExperimentalUnit Response
DoseApplication
SignalDetection
Assay Structure
V. Devanarayan, Biomarker MethodsBASS-XI, November 1, 2004
Copyright © 2004 Eli Lilly and Company 7
Assay Inference
Interest is primarily on “estimation” of some property of the material
Similar to methods of physical measurement, but with more complex sources of variation
Different from experimental studies that are designed to compare effects of known treatments
V. Devanarayan, Biomarker MethodsBASS-XI, November 1, 2004
Copyright © 2004 Eli Lilly and Company 8
Nature of a Standard Preparation“CRASS”
• Characterized and purified well
• Representative of samples to be assayed
• Available in large quantity
• Stable under well-defined conditions
• Accessible to participating laboratories
Note: Samples of the standard preparation must beincluded in each “run” of an assay!
V. Devanarayan, Biomarker MethodsBASS-XI, November 1, 2004
Copyright © 2004 Eli Lilly and Company 9
Biom arker C oncentration (R eference)10-1 100 101 102 103 104 105
O D
0.0
0.2
0.4
0.6
0.8
1.0
1.2
TY
Biomarker QuantificationStandard Curve
YT = Observed Assay signal of Test (T) sampleZT = Calibrated Biomarker level in test sample = F-1(YT, β)
Standard CurveY = F(Z, β) + ε
ZT
V. Devanarayan, Biomarker MethodsBASS-XI, November 1, 2004
Copyright © 2004 Eli Lilly and Company 10
2. Fundamental Validity, Similarity, Parallelism
V. Devanarayan, Biomarker MethodsBASS-XI, November 1, 2004
Copyright © 2004 Eli Lilly and Company 11
Similarity condition (Finney, 1978):1. Dose response functions for the test (T) and
standard (S) preparations must satisfyfor all doses z
2. and have the same functional form3. ρ is a constant (defined as relative potency)
Ft(z)=Fs(ρ•z)Ft(•) Fs(•)
What assumptions must be satisfied for a Biomarkerresult to be fundamentally valid when calibrated from a reference (standard) material?
Similarity Requirement
V. Devanarayan, Biomarker MethodsBASS-XI, November 1, 2004
Copyright © 2004 Eli Lilly and Company 12
100 101 102 103 104
OD
0.0
0.2
0.4
0.6
0.8
1.0
Standard Concentration
1/dilution factor
Standard Curve
)(FS ⋅
)zlog()zlog()log( ts −=ρ
ts z/z=ρTest Sample Curve
410− 210− 010110−310−
)(Ft ⋅
ParallelismIllustration
V. Devanarayan, Biomarker MethodsBASS-XI, November 1, 2004
Copyright © 2004 Eli Lilly and Company 13
1/dilution factor
10-5 10-4 10-3 10-2 10-1 100 101
Opt
ical
Den
sity
(OD
)
0.0
0.2
0.4
0.6
0.8
1.0
StandardRatio slopes (T/S) = 1.2Ratio slopes (T/S) = 1.1Ratio slopes (T/S) = 1.0Ratio slopes (T/S) = 0.9Ratio slopes (T/S) = 0.8
Effect of Non-ParallelismIllustration
V. Devanarayan, Biomarker MethodsBASS-XI, November 1, 2004
Copyright © 2004 Eli Lilly and Company 14
1/dilution factor
10-5 10-4 10-3 10-2 10-1 100 101
Rel
ativ
e Er
ror (
%)
-80
-60
-40
-20
0
20
40
60
80
100
120
Ratio = 1.2Ratio = 1.1Ratio = 1.0Ratio = 0.9Ratio = 0.8
Effect of Non-ParallelismIllustration (contd.)
100zRE%T
T ⋅
µµ−
=z
Tµ : nominal value: interpolated result
V. Devanarayan, Biomarker MethodsBASS-XI, November 1, 2004
Copyright © 2004 Eli Lilly and Company 15
ParallelismAssessment & Analysis
“Minor” differences in slopes can cause major effects in the “relative error” (bias).
Statistical significance:• The difference in slopes & other parameters can be tested
within the framework of nonlinear models.• Implementation: gnls()/nlme() function in Splus
Biological significance:• How much relative error (bias) is clinically acceptable?
Consider both Statistical & Biological Significance!
V. Devanarayan, Biomarker MethodsBASS-XI, November 1, 2004
Copyright © 2004 Eli Lilly and Company 16
3. Types of Biomarker Methods/Assays
V. Devanarayan, Biomarker MethodsBASS-XI, November 1, 2004
Copyright © 2004 Eli Lilly and Company 17
Types of Biomarker Methods
Biomarker Method Types
DefinitiveQuantitative
RelativeQuantitative
Quasi-Quantitative
Qualitative
V. Devanarayan, Biomarker MethodsBASS-XI, November 1, 2004
Copyright © 2004 Eli Lilly and Company 18
Reference Standard available• Well defined,
• Fully representative of the endogenous protein.
Analytical result is expressed in continuous units of the definitive reference standard.
Examples:• Human insulin
• Steroid Assays
Ideal situation
Types of Biomarker Methods Definitive Quantitative
V. Devanarayan, Biomarker MethodsBASS-XI, November 1, 2004
Copyright © 2004 Eli Lilly and Company 19
Reference Standard available.• Not well characterized,
• Not available in a purified form, or is
• Not fully representative of the endogenous protein
! Relative!
Analytical result is expressed in continuous units of the relative reference standard.
Example: Cytokine ELISAs
Types of Biomarker Methods Relative Quantitative
V. Devanarayan, Biomarker MethodsBASS-XI, November 1, 2004
Copyright © 2004 Eli Lilly and Company 20
Reference Standard not available or is not ‘valid’.
Continuous response
Analytical result: characteristic of the test sample• Assay Signal
Examples:• Enzymatic assays (Activity Units)
• Anti-drug antibody assays (titers)
• Flow cytometry assays
Quasi " “possesses certain attributes”
Types of Biomarker Methods Quasi Quantitative
V. Devanarayan, Biomarker MethodsBASS-XI, November 1, 2004
Copyright © 2004 Eli Lilly and Company 21
No Reference Standard
Discrete response
Analytical result: characteristic of the test sample• Assay Signal
Ordinal data:• Ordered & non-continuous responses
• Examples: +, ++, +++ or low, mid, hi
Nominal data:• Non-ordered & non-continuous responses
• Examples: reporting results as positive (+) or negative (-)
Types of Biomarker Methods Qualitative
V. Devanarayan, Biomarker MethodsBASS-XI, November 1, 2004
Copyright © 2004 Eli Lilly and Company 22
Types of Biomarker MethodsRecap!
Biomarker Method Types
DefinitiveQuantitative
RelativeQuantitative
Quasi-Quantitative
Qualitative
• Biomarker levels determined using a “reference standard”.
• Reference standard must be truly representative of test samples
– Parallelism– Standard Curve
Scope of this talk!
• Biomarker levels represented only by assay signal.
• Reference material not available or• Not representative of test samples.
– Non-Parallelism– Standard Curve not used
• Example: Mire-Sluis et al (2004, JIM)
V. Devanarayan, Biomarker MethodsBASS-XI, November 1, 2004
Copyright © 2004 Eli Lilly and Company 23
4. Standard Curves, Weighting & Precision Profiles
V. Devanarayan, Biomarker MethodsBASS-XI, November 1, 2004
Copyright © 2004 Eli Lilly and Company 24
Standard Curve
Standard Curve model includes
• Mean Function
• Response Error Function (Weighting)
Mean Functions:
• Polynomial (linear, quadratic, linear in log-scale, etc.)
• Nonlinear (example: Logistic models)
• Spline
Concentration (pg/ml)
Mea
n O
ptic
al D
ensi
ty
0.02 0.08 0.31 1.25 5.00
0.04
0.08
0.13
0.24
0.43
0.80
1.40
2.43
Log-Log Linear Calibration Curve
True Concentration
Per
cent
Rec
over
y
0.0195 0.0780 0.3130 1.2500 5.0000
4060
8010
014
0
Percent Recovery PlotFrom Log-Log Linear Fit
Weighted Four Parameter Logistic Calibration Curve
Concentration (pg/ml)
Mea
n O
ptic
al D
ensi
ty
0.02 0.08 0.31 1.25 5.00
0.0
0.5
1.0
1.5
2.0
2.5
True Concentration
Per
cent
Rec
over
y
0.0195 0.0780 0.3130 1.2500 5.0000
4060
8010
014
0
Percent Recovery PlotFrom Weighted FPL Fit
R2 = 0.99
Standard Curve: Mean Function
• Linear model is far worse than 4PL, even though R2 = 0.99
Percent Recovery = 100*(Estimated Conc. / True Conc.)
V. Devanarayan, Biomarker MethodsBASS-XI, November 1, 2004
Copyright © 2004 Eli Lilly and Company 26
Standard CurveResponse Error Function (Weighting)
Most laboratory software assume constant variance.
Heteroscedasticity is common with these data.
Ignoring this can affect method/assay performance.
Popular methods for weighting:• Model replicate SDs v.s. replicate means
• Pseudo-likelihood based methods (Carroll & Ruppert, 1988)
10 100 1000 10000
050
0010
000
1500
020
000
Standard Curve Without Weighting
4
1 22
32
( , )( )( , )
1 ( )
( )
Y f x
f x x
V a r
β
β εβ ββ β
β
ε σ
= +−
= ++
=
Variance is assumed constant
95% confidence band
Ass
ay S
igna
l
Biomarker Level (pg/ml)
10 100 1000 10000
050
0010
000
1500
020
000
Standard Curve With Weighting
Variance is modeled as a power of the mean response.
95% confidence band
[ ]
4
1 22
3
2
( , )( )( , )
1 ( )
( ) ( , )
Y f x
f x x
V a r f x
β
θ
β εβ ββ β
β
ε σ β
= +−
= ++
=
Ass
ay S
igna
l
Biomarker Level (pg/ml)
V. Devanarayan, Biomarker MethodsBASS-XI, November 1, 2004
Copyright © 2004 Eli Lilly and Company 29
Impact of WeightingIllustration: Precision Profiles
1 0 1 0 0 1 0 0 0 1 0 0 0 0
0.00
0.05
0.10
0.15
0.20
1 0 1 0 0 1 0 0 0 1 0 0 0 0
0.00
0.05
0.10
0.15
0.20
CV
Biomarker Level (pg/ml)
LQL UQLLQL UQL
Weighting
No Weighting
V. Devanarayan, Biomarker MethodsBASS-XI, November 1, 2004
Copyright © 2004 Eli Lilly and Company 30
Standard CurveMisconceptions About the “Linear Portion”
10 100 1000 10000
050
0010
000
1500
020
000
Ass
ay S
igna
l
Biomarker Level (pg/ml)
Linear Portion
Correct Working Range
Linear Portion π Working Range• Due to Heteroscedasticity!
V. Devanarayan, Biomarker MethodsBASS-XI, November 1, 2004
Copyright © 2004 Eli Lilly and Company 31
Note about Precision ProfilesPurpose
• Based only on standard curve, not validation samples! Doesn’t take into account of all sources of assay variability.
1. Educational tool– Illustrates the impact of weighting.
2. Method Development & Optimization– Objective endpoint for selecting reagents & protocol design/optimization.
3. Preliminary “Marker” of Method Performance.– Helps assess whether the assay is ready for Validation!
Points 2 and 3 will be evident in the next section.
V. Devanarayan, Biomarker MethodsBASS-XI, November 1, 2004
Copyright © 2004 Eli Lilly and Company 32
5. Assay/Method Optimization
V. Devanarayan, Biomarker MethodsBASS-XI, November 1, 2004
Copyright © 2004 Eli Lilly and Company 33
Method Optimization & Assessment
Commonly used Endpoint: • Assay’s Signal to Noise Ratio (Signal Window, Z-factor)• Appropriate for standard screening assays (binding, functional, etc.)• Inappropriate for Calibration applications.
Optimize/Assess Calibration Precision Profiles!
V. Devanarayan, Biomarker MethodsBASS-XI, November 1, 2004
Copyright © 2004 Eli Lilly and Company 34
Screening Experiments:
• Identify all potentially important assay factors/variables.
• Run 2-level fractional-factorial experiments to determine the factors that are statistically important.
Optimization Experiments:• Run 3-level experiments on these important factors using a Central-
Composite or Box-Behnken type design.
Generate standard curves for each trial of the experiment.
Estimate the optimum using response surface analysis.
Optimization Procedure
V. Devanarayan, Biomarker MethodsBASS-XI, November 1, 2004
Copyright © 2004 Eli Lilly and Company 35
Standard Curves and hence Precision Profiles can be generated for each assay condition.
Key endpoints determined from Precision Profile.• Lower Quantification Limit (LQL)
– Lowest concentration where the precision profile intersects 20% CV.
• Upper Quantification Limit (UQL)
• Working range (WR) = Log10(UQL / LQL)
• CVa = Average CV within the working range.
• Precision Area (PA) = WR x (20% - CVa).
Assay Optimization EndpointsDerived from Precision Profile
V. Devanarayan, Biomarker MethodsBASS-XI, November 1, 2004
Copyright © 2004 Eli Lilly and Company 36
Assay Optimization EndpointsDerived from Precision Profile
1 0 1 0 0 1 0 0 0 1 0 0 0 0
0.00
0.05
0.10
0.15
0.20
Working Range
LQL UQL
Precision AreaCVa
V. Devanarayan, Biomarker MethodsBASS-XI, November 1, 2004
Copyright © 2004 Eli Lilly and Company 37
Define important facets of the precision profile.
• LQL, WR, CVa
• PA & UQL are contained in the above
Now set specification limits on these facets.
• CLQL: largest acceptable value of LQL
• CWR: smallest acceptable value of WR
• 15%: largest acceptable value of CVa
Assay Optimization EndpointsDerived from Precision Profile
V. Devanarayan, Biomarker MethodsBASS-XI, November 1, 2004
Copyright © 2004 Eli Lilly and Company 38
The composite optimization endpoint is
%)15minmax
max
minmaxmin
minmaxmax
3
2
1
≤
−
−×
≥
−
−×
≤
−−
aI(CVw
)a(CV)a(CVaCV)a(CV
)WRCI(WRw
(WR)(WR)(WR)WR
)LQLCI(LQLw
(LQL)(LQL)LQL(LQL)
w1 + w2 + w3 = 1 are the relative weights chosen by biochemist.
Assay Optimization EndpointsComposite Endpoint from Precision Profile
LQL = 24.2UQL = 157
5 20 78 312 1250 5000Biomarker Level
01
23
OD
Optimum by Signal Window
5 50 500 5000Biomarker Level
0.0
0.1
0.2
0.3
0.4
CV
Precision Profile
5 20 78 312 1250 5000Biomarker Level
0.0
0.5
1.0
1.5
2.0
OD
Optimum by Precision Profile
5 50 500 5000Biomarker Level
0.0
0.1
0.2
0.3
0.4
CV
Precision Profile
LQL = 13.6UQL = 1662.3
Good Assay Signal does not imply good working range.Improper optimization # Unacceptable Assay!Focus must be on Calibration Range instead of Assay Signal.
Example: Endpoints matter!
V. Devanarayan, Biomarker MethodsBASS-XI, November 1, 2004
Copyright © 2004 Eli Lilly and Company 40
Biomarker MethodsAre you ready for Validation?
Predict the Sensitivity & Range (QLs) using precision profiles.
• Does not take into account of other assay problems– Cross-reactivity, Interference, Operational factors, etc.
• So the predicted QLs are optimistic!
If the predicted QLs are not within the target range:• Not ready for Validation!
– Re-optimize some assay conditions using precision profile.
If the predicted QLs are within the desired range:• Ready for Validation!
V. Devanarayan, Biomarker MethodsBASS-XI, November 1, 2004
Copyright © 2004 Eli Lilly and Company 41
6. Pre-Study & In-Study Method Validation, Acceptance Criteria
V. Devanarayan, Biomarker MethodsBASS-XI, November 1, 2004
Copyright © 2004 Eli Lilly and Company
To demonstrate that an analytical procedure is acceptable (suitable, reliable, ...) for its intended application.
Implicit assumption: Acceptance criteria are defined prior to the initiation of development.
Biomarker Analytical Validation Objective
V. Devanarayan, Biomarker MethodsBASS-XI, November 1, 2004
Copyright © 2004 Eli Lilly and Company 43
A1
A2
A3
A4
A5
A6
A7
A8
Which Assay is on Target?Dartboard Analogy!
V. Devanarayan, Biomarker MethodsBASS-XI, November 1, 2004
Copyright © 2004 Eli Lilly and Company 44
Which Assay is on Target?Bias + Precision
A3 A4 A6 A8
Ass
ays
Bia
sPr
ecis
ion
V. Devanarayan, Biomarker MethodsBASS-XI, November 1, 2004
Copyright © 2004 Eli Lilly and Company 45
Biomarker Analytical Validation Parameters
Primary parameters• Trueness (systematic error # bias)• Overall Precision (random errors # variance)
– Intra-Run, Inter-Run, Analyst, Equipment, Plate, …$ Total Error = |Bias| + Overall Precision
Derived parameters (from primary parameters)• Sensitivity (LQL)• Assay range (LQL, UQL)
Diagnostic parameters (provide insight into possible sources of systematic and random errors)
• Specificity• Dilution linearity• Parallelism• Stability
V. Devanarayan, Biomarker MethodsBASS-XI, November 1, 2004
Copyright © 2004 Eli Lilly and Company 46
Pre-Study ValidationExpectations for Biomarker Method Types
$Dilution Linearity
$Parallelism$$
LLOQ / ULOQAssayRange
$Standard & Reagent Stability
$$$Matrix Stability
$$$Specificity
$$$LLOQ
Sensitivity$$Precision
$Trueness (Bias)
QualitativeQuasi-
QuantitativeDefinitive & Relative
Quantitative
Biomarker Method Types
Reminder: Focus of this talk is on Definite & Relative Quantitative Methods
V. Devanarayan, Biomarker MethodsBASS-XI, November 1, 2004
Copyright © 2004 Eli Lilly and Company 47
Pre-Study Validation Experiment
Generate these data from each of 6 independent runs.• Standard Curve: 8-12 pt, triplicates
• Validation Samples: 6-8 concentrations, > 2 replicates
– Independent samples spiked with nominal amount of analyte.
– 2 conc near desired LQL, 2 near desired UQL, and 2-4 within the range.
Consider other sources of variation in the design.• Analyst, Equipment, Plate, Vendor, etc.
V. Devanarayan, Biomarker MethodsBASS-XI, November 1, 2004
Copyright © 2004 Eli Lilly and Company 48
Pre-Study Validation Data Analysis
Perform Variance Component Analysis• JMP, Proc Mixed in SAS, LME in S-plus, etc.
Estimate Bias, Overall Precision, Total Error.• Investigate each component of systematic and random errors.
Determine Sensitivity and Assay Range.• LQL, UQL
Confirm/Finalize the model for Standard Curve.• Compare popular models, weighting methods, etc.• Select/Confirm the optimal model based on the Total Error, Sensitivity, etc.
Assess Dilution Linearity• Determine Maximum Tolerable Dilution.
V. Devanarayan, Biomarker MethodsBASS-XI, November 1, 2004
Copyright © 2004 Eli Lilly and Company 49
Pre-Study Validation: IllustrationBias, Precision, Total Error, Sensitivity
Bias & Overall Precision
-60
-40
-20
0
20
40
60
1 10 100 1000
Biomarker Nominal Level (pg/ml)
%R
elat
ive
Bia
s +/
- Pre
cisi
on
Data from a pre-study validation experiment – Biomarker (IL-6) ELISA.Each data point represents the Mean %Bias across 6 runs.Error Bars represent the Overall Precision at each nominal level.
LQL UQL
V. Devanarayan, Biomarker MethodsBASS-XI, November 1, 2004
Copyright © 2004 Eli Lilly and Company 50
Pre-Study Validation: Illustration Bias, Precision, Total Error, Sensitivity
Total Error Profile
01020304050607080
1 10 100 1000Biomarker Nominal Level (pg/ml)
% T
otal
Err
or
5PL, Abs. Bias
5PL, Total Error
Criteria
Data from the same pre-study validation experiment (previous slide).Plotted differently to represent the Total Error, with |Bias| & Precision
LQL UQL
V. Devanarayan, Biomarker MethodsBASS-XI, November 1, 2004
Copyright © 2004 Eli Lilly and Company 51
Pre-Study Validation: Illustration Finalize/Confirm Model Selection
LL: Linear Model in log-scale (R2 = 98.5%)• Note that R2 is commonly reported by Lab software
4/5PL: Four/Five-Parameter Logistic Model
Calibration Curve
0.1
1
10
1 10 100 1000
Biomarker Level (pg/ml)
Ave
rage
Res
pons
e
Data 5PL LL
V. Devanarayan, Biomarker MethodsBASS-XI, November 1, 2004
Copyright © 2004 Eli Lilly and Company 52
Pre-Study Validation: Illustration Finalize/Confirm Model Selection
5PL vastly better than LL, and slightly better than 4PL.• With respect to Total Error, Bias and Precision
This confirms that for this particular assay, 5PL is the optimalchoice for the in-study (production) phase.
Total Error Profile: Model Comparison
01020304050607080
1 10 100 1000
Biomarker Nominal Level (pg/ml)
% T
otal
Err
or
4PL, Abs. Bias
4PL, Total Error
5PL, Abs. Bias
5PL, Total Error
LL, Abs. Bias
LL, Total Error
Criteria
V. Devanarayan, Biomarker MethodsBASS-XI, November 1, 2004
Copyright © 2004 Eli Lilly and Company 53
Pre-Study Validation: Illustration Finalize/Confirm Weighting
Data from a pre-study validation experiment. Assay characterization is greatly impacted by weighting.The optimal weighting factor estimated from the pre-study validation phase can be used to “fix” weights for the in-study (production) phase.
-90
-60
-30
0
30
60
90
10 100 1000
Nominal Biomarker Level (pg/ml)
% Re
lative
Erro
r WeightingNo Weighting
V. Devanarayan, Biomarker MethodsBASS-XI, November 1, 2004
Copyright © 2004 Eli Lilly and Company 54
In-Study Validation/QC“4-6-x Rule”
QC Samples in each run: • 3 levels (typically low, mid, high) in 2-3 replicates.
4-6-x Rule:• 2/3rd of all the samples must be within x% of the nominal.• Half the samples at each level must be within x% of the nominal.
The choice of “x” varies across applications and formats.• Typically, for biomarker immunoassays, x = 30%.
Parallelism of the test samples must be assessed at different points during the production phase.
V. Devanarayan, Biomarker MethodsBASS-XI, November 1, 2004
Copyright © 2004 Eli Lilly and Company 55
7. Acceptance Criteria
V. Devanarayan, Biomarker MethodsBASS-XI, November 1, 2004
Copyright © 2004 Eli Lilly and Company 56
“4-6-30” rule≤ 30%Total Error
-≤ 20 (25 at LQL)
Overall Precision (%CV)
-≤ ± 20 (± 25 at LQL)
Trueness (%Relative Bias)
In-studyValidation
Pre-studyValidationCharacteristic
Total Error = |%Relative Bias| + Overall Precision (%CV)
Biomarker Analytical Validation Acceptance Criteria (Immunoassays)
DeSilva, et al: Pharm Res 20(11): 1885-1900, 2003.
Lee, et al.: AAPS Biomarker Method White Paper, in preparation
V. Devanarayan, Biomarker MethodsBASS-XI, November 1, 2004
Copyright © 2004 Eli Lilly and Company 57
Comments on the Acceptance Criteria
This doesn’t hold as n is small and Total Error is just an estimate!
So for true consistency with the 4-6-x rule, we need to have Total Error < 30% - γ% as the pre-study criteria!% Choice of γ depends on level of uncertainty in the estimate of total error.
∞→−−→⇒
<+⇒
<−+−⇒
<−=
n as rule, 306430% within are results theof 2/3rd
%30 )(Precision Bias
%30)()(
30%Error Total
σ
µ
µ
XXX
X
i
i
Pre-Study & In-Study criteria “appear” to be consistent because
V. Devanarayan, Biomarker MethodsBASS-XI, November 1, 2004
Copyright © 2004 Eli Lilly and Company 58
8. Summary
V. Devanarayan, Biomarker MethodsBASS-XI, November 1, 2004
Copyright © 2004 Eli Lilly and Company 59
Summary Flow SchemeStatistical Thinking all the way!
Assay Development
Standard Curve? Model, Weighting, Software, …
Optimization/Selection of Assay Reagent Conditions• Use Precision Profiles
Is the Assay Ready for Validation?• Use Precision Profiles to decide
Pre-Study ValidationExperiment with spiked validation samples
• Bias, Precision, Total Error• Sensitivity, Assay Range, Linearity• Finalize Standard Curve model, weighting, …
Verify the reference standard valid?• Similarity/Parallelism using spiked donor samples
In-Study Validation/QC QC samples (4-6-X rule)Parallelism assessment
Intended use of the assay?• Define expectations/acceptance criteria
Def
initi
ve &
Rel
a ti v
e Q
u ant
it at i v
e M
etho
ds
V. Devanarayan, Biomarker MethodsBASS-XI, November 1, 2004
Copyright © 2004 Eli Lilly and Company 60
Statistical + Practical + Biological Thinking!
Use statistics to provide estimates of errors. Statistics do not directly tell you whether the method is acceptable.
- Westgard, 1998
Key issues in Biomarker Method Development & Validation are governed by a combination of
V. Devanarayan, Biomarker MethodsBASS-XI, November 1, 2004
Copyright © 2004 Eli Lilly and Company 61
Suggested Reading
• Belanger, B.A., et al.: D.M., Biometrics, 1995
• Callahan JD and Sajjadi NC: Bioprocessing J (Apr/May): 1-6, 2003.
• Carroll, RJ & Ruppert, D: Chapman & Hall, New York, 1988
• DeSilva, et al: Pharm Res 20(11): 1885-1900, 2003.
• Hartmann C, et al: Anal Chem 67: 4491-4499, 1995.
• Hubert H, et al: Analytica Chemica Acta 391: 135-148, 1999.
• Karnes HT and March C: J Pharm Biomed Anal 10-12: 911-918, 1991.
• Kringle R, et al: Drug Information Journal, Vol. 35, 1261-1270, 2000.
• Mire-Sluis et al.: J Immunological Methods, 2004.
• O'Connell, M.A., et al.: Chemo Intel Lab Systems, 20, 97-114, 1993
• Plikaytis BD, et al: J Clin Microbiol 32(10): 2441-2447, 1994.
• AAPS Biomarker Method White Paper, will be ready by late 2004