Setting Quality Standards - AACB
Transcript of Setting Quality Standards - AACB
John CallejaMelbourne Pathology ServicesAACB Quality SES, Adelaide: 30th Oct.2014
Setting QualityStandards
Discussion Content
John Calleja (M.P.S) Oct 2014 2
Laboratory Error Monitoring vs. Diagnosis Bias, Imprecision & Total Error 6 Sigma Considerations
Revisit Quality Goals – Setting Quality Standards What hierarchy we should use them in How much imprecision or Bias we can allow
Setting QC Targets & Limits• Referencing Quality Goals
Referencing Quality Goals when Assessing An assay shift Deciding which samples to re-run and which results to amend
following run failure
2
John Calleja (M.P.S) Oct 2014 3
1. Laboratory Error
John Calleja (M.P.S) Oct 2014 4
When patient samples arrive atour laboratories – they may havecome to us for one of two mainreasons ..
• For Diagnosis or Screening• Or for Monitoring..
John Calleja (M.P.S) Oct 2014c 5
Quality requirements Differ Slightlyfor each of these sample types..• For Diagnosis / Screening.
We consider Pat. Results vs. Reference Intervals &often against accepted C/O’s – to Rule In / Rule Out
disease
Bias is Important !
+/-2sd = 95.5%Reference Interval !
If we had aPositive Bias!
Patient classificationscould change!
C/O
John Calleja (M.P.S) Oct 2014c 6
• For Monitoring.
We consider Pat. Results relative to theirprevious results .. for: stability, responseto treatment or signs of deterioration .
GoodPrecision isImportant !
0
200
400
600
800
1000
1200
1400
1600
1800
Trop
onin
-T
Time / Date
Troponin-T Monitoring
Troponin-T
Patient hadInfarct ! Want to have confidence
that the decreases inTnT, signify a +ve
response to treatment
– NOT Imprecision !
John Calleja (M.P.S) Oct 2014 7
Whilst Bias & Imprecision areimportant categories forClassifying Error into ..
in the Laboratory
– We also need to considerthe Clinician !
John Calleja (M.P.S) Oct 2014 8
Total Error … Total Error.. considers error from the view-point of.
Clinicians .. who look at error in “Absolute Terms
It recognises that Analytical Error is made up of the ..two components..
* Systematic* Random
.. but views them as 1 parameter.
General Formula .. TEa = | x – u | + 2sd.
1.65sdFor 90% c.i. or 5% error at 1tail
John Calleja (M.P.S) Oct 2014 9
Total Error
_
True Value
u
Total Error: TEa = 1.65s + |x - u |
x - u
Systematic Error
Observed Value
x_
1.65sRandom Error This principle is further
developed by the Six-SigmaConcept
.. which views Bias &Imprecision as part of a Total
Error Budget
John Calleja (M.P.S) Oct 2014 10
Total Error
_
True Value
u
Total Error: TEa = 2s + |x - u |
x - u
Systematic Error
Observed Value
x_
2sRandom Error TEa = 2s + | x - u |
= (2x1) + | 137-140 |
= 5
Example:u = 140x = 137S = 1
John Calleja (M.P.S) Oct 2014 11
• We don’t really know which of these are for diagnosis& which are for monitoring – so minimising thesources of error for both of these sample categoriesis important to us.
• But Monitoring has the tighter requirements so ourQuality Standards should be set for monitoring.
Coming backto our PatientSamples ..
John Calleja (M.P.S) Oct 2014 12
2. Sources of QualityGoals
John Calleja (M.P.S) Oct 2014 13
Consider ..…
What guidelines or Quality Goals do we have .. to indicate
... the level of quality required
.. or the allowable error, we can permit
.. to ensure that our results are medically useful ?
(eg) If we have a CV of 3% for Na ...is this Clinically Acceptable ?
John Calleja (M.P.S) Oct 2014 14
2.1 Quality Goals- Addressed by ....
1. TONKS - 1958 - 25% of Normal Range
2. BARNETT - 1968 - limits based on the opinionsof Clinicians & Lab personnel .
3. COTLOVE/ - 1970 - Based on Intra-IndividualC.G.Fraser - 1990 Biological Variation.
CVa = 0.5 x CVi
4. Govt Bodies. - 1990 - C.L.I.A. -Clinical LaboratoriesInformation Act - 1988
5. QAP Providers - 1982 - RCPA-AACB QAP
6. Evidence Based - 1992 - HBA1c - DCCT Study
John Calleja (M.P.S) Oct 2014 15
2.2 Quality Goals– Hierarchy Provided by the Profession
◦ Evidence Based Studies DCCT Trial - HBA1c ( CV<2.5%)
◦ Biological Goals - Based on CVI CVa < 0.5 CVi, CVa < 0.25 CVi, CVa < 0.75 CVi
◦ Clinician Survey Barnett .. et al
◦ Profession Defined By group of experts
eg. RCPA-QAP Allowable Limits◦ Proficiency Testing Schemes State of the Art Method .or. +/- 2sd of all results submitted
◦ Publication by a Lab or Group
ISO Technical Committee212 Task Force, 1999
IFCCISOAACCRCPAAACB
John Calleja (M.P.S) Oct 2014 16
2.3 Quality Goals
Cotlove et al - 1970- Studied Intra & Inter individual variation of certain analytes- recommend that :
Allowable Limits of Performance, should be based on ...the Relevant Biological Variation,
Work later expandedon by Callum. G. Fraser
late 80’s / early 90’sOptimal Goal CVa < 0.25 x CVi
Minimal Goal CVa < 0.75 x CVi
- Apex of Hierarchy – Biological Variability
… for a result to be medically use-full.
CV < 0.5 x CV
analytic intra individual biological variation
John Calleja (M.P.S) Oct 2014 17
2.4 Quality Goals- Intra-Individual Biological Variability
Take several BloodSamples over 24hrs 02:00 04:00 06:00 08:00 10:00 12:00 14:00 16:00 18:00 20:00 22:00 24:00
Analyse Samples
CVT = (CVA2 + CVi
2) 1/2
Calculate CV(total CV) CVT
Use calculated CVT & known CVA.. to derive CVI
CVI Gluc = 6.5%
CVi = ( CVT2 - CVA
2 ) 1/2
John Calleja (M.P.S) Oct 2014 18
We can illustrate theimportance of Analytical CVon result interpretation !
.. Using inferences fromthe 1993 DCCT Trial ..
John Calleja (M.P.S) Oct 2014 19
Remember ...
When we analyse a patient sample, thereare 2 components of variation:◦ Analytical CVA &◦ Biological CVi
This is represented as
CVT = (CVA2 + CVi
2) 1/2
5
5.5
6
6.5
7
7.5
8
8.5
9
9.5
10
10.5
11
0 20 40 60 80 100 120 140
HBA1c - Effect of CV on result interpretation- CVa=0 %
low risk
High Risk
John Calleja (M.P.S) Oct 2014 20
5.6 Effect of CVA on Result Interpretation
In the DCCT Trial .. 2 Patient Cohorts◦ In Intensively Treated Cohort -> Mean HBA1c was 7%◦ In Conventional ly Treated Cohort -> Mean HBA1c was 9% . .
CVi (HBA1c) = 3.6% ...◦ We can use this study to illustrate the variation due to CVi alone, at these levels
Depiction of pureBiological SignalOnly !CVT = (CVA
2 + CVi2) 1/2
CVT = ( 02 + 3.62 ) 1/2
CVT = 3.6 %
John Calleja (M.P.S) Oct 2014 21
.. So what happens whenwe add in Analytical Variation
to these signals ?
John Calleja (M.P.S) Oct 2014 22
5
5.5
6
6.5
7
7.5
8
8.5
9
9.5
10
10.5
11
0 20 40 60 80 100 120 140
HBA1c - Effect of CVA on result interpretation -CVA=5.0%
Low Risk
High Risk
5.6 Effect of CVA on Result Interpretation
At the CVA = 5.0% (Typical of old HBA1c methodology) There is a significant overlap between cohorts & the
distinction between the groups is blurred !
Significant Overlap Blurringthe distinction between highrisk and low risk Grp for 2ryDiabetic complicationsCVT = (CVA
2 + CVi2) 1/2
CVT = (5.02 + 3.62) 1/2
CVT = 6.16%
John Calleja (M.P.S) Oct 2014 23
5
5.5
6
6.5
7
7.5
8
8.5
9
9.5
10
10.5
11
0 20 40 60 80 100 120 140
HBA1c - Effect of CVA on result interpretation -CVA=1.8 %
Series1
Series2
Effect of CVA on Result Interpretation
CVT = (CVA2 + CVi
2) 1/2
CVT = ( 1.82 + 3.62) 1/2
CVT = 4.02 %
Good Distinctionbetween Low & HighRisk Cohorts
At the Desirable HBA1c CVA Goal = 0.5CVi = 1.8% There is good distinction between cohorts !
John Calleja (M.P.S) Oct 2014 24
.. So .. minimising our analyticalvariation .. clearly assists in patientresult interpretation !
.. But how much is enough ?
.. Given to us by Callum Fraser
John Calleja (M.P.S) Oct 2014 25
.. Minimising our analytical variation ..assists in patient result interpretation !
.. But how much is enough ?
.. Given to us by Callum Fraser
John Calleja (M.P.S) Oct 2014 26
Effects of Imprecision on Test Result Variability
Ref. Biological Variation – Principles to Practice – Callum Fraser
0.25 CVi adds 3% variability.
0.50 CVi adds 12% variability.
0.75 CVi adds 25% variability.
Desirable Goal.
% increasein variability.
BiologicalCV Goal.
John Calleja (M.P.S) Oct 2014 27
What level of Biasis acceptable ?
John Calleja (M.P.S) Oct 2014 28
What level of Bias is Acceptable ?
Callum Fraser .. Tells us that .. the Reference Interval ismade up of;◦ With-in Subject & CVi◦ Between Subject Variation CVG
For all of us to use the same Reference Interval, .. theanalytical Bias should be less than ¼ of the groupedBiological Variation.
This is represented as:
BA < 0.25 (CVG2 + CVi
2) 1/2
Becomes aDefault Bias
Goal !
John Calleja (M.P.S) Oct 2014 29
Impact of a Shift or Bias
+/-2sd = 95.5%Reference Interval !
Results in False Positives,> 2.5% out of U.R.L.
Shift in Assaycausing +ve Bias
Results in<2.5% out of
LRL
J.Calleja - Melb. Path. - Oct 2014 30
2.1 Impact on Outcomes (eg) Chol
LDLC = Chol – HDLC - (Trig / 2.2 )
Ref: < 3.0
Chol / HDLC Ref: < 4.5
LDL / HDLC
Ref: < 3.5
CHOL Ref: 3.5 – 5.5
Chol.+ ve Bias
5.3
1.7
1.2
3.3
4.4
2.8
TRIG Ref: 0.5 – 2.0
HDLC Ref: > 1.0
5.8 *
1.7
1.2
3.8 *
4.8 *
3.2 *
5.6 *
1.7
1.2
3.6 *
4.7 *
3.0 *
+ 5% + 10%Baseline
Positive Bias
John Calleja (M.P.S) Oct 2014 31
How much of the Population is Displaced by Bias ?
Ref. Biological Variation – Principles to Practice – Callum Fraser
BA < 0.125 (CVI2 + CVG
2)1/2
adds 2%outside of Ref Interval.
BA < 0.250 (CVI2 + CVG
2)1/2
adds 16%
BA < 0.375 (CVI2 + CVG
2)1/2
adds 34%
% of ResultsOutside of URL
% of ResultsOutside of LRL
When Bias = 02.5% on either
side of Ref Limits
Allowable Bias
% Out ofeach
Ref Limit
Bias Goal.
John Calleja (M.P.S) Oct 2014 32
Combining Bias &Imprecision Goals !
Total Error Goal
<0.125(CVi2+CVg2)1/2 + 1.65(0.25CVi)
<0.25(CVi2+CVg2)1/2 + 1.65(0.5CVi)
<0.375(CVi2+CVg2)1/2 + 1.65(0.75CVi)
Bias Goal
<0.125(CVi2+CVg2)1/2
<0.25(CVi2+CVg2)1/2
<0.375(CVi2+CVg2)1/2
John Calleja (M.P.S) Oct 2014 33
CV Goal
Optimal CVa=0.25CVi
Desirable CVa=0.5CVi
Minimum CVa=0.75CVi
2.5 Combining Bias & Precision Goals for Total Error
• Generally we should aim for Desirable Goals.• If we are easily able to achieve them – then go for Optimal.• If the Biol Variabiity goals are very tight & we can’t achieve
them (eg. Na, Ca) – then go for Minimal.• If we can’t achieve Minimal – then should aim for State-of-
the-Art.
John Calleja (M.P.S) Oct 2014 34
Six-Sigma – is a means ofunderstanding & managingOur error .. in terms of an
‘Error Budget’
It’s one thing tohave a goal - but
you need a meansto achieve it
What is Six Sigma !
John Calleja (M.P.S) Oct 2014 35
Six sigma provides a means to monitor the ‘PerformanceCapability’ of a testing system
It was developed by Motorola in the 1980s, so that they couldvirtually eliminate defective products.
Motorola defined this as having .. ‘Six Sigmas (SDs) ofProcess Variation … fitting within the product tolerances.
The effect of ‘how many SDs you have spanning theproduct specifications’ on the defect rate and defects permillion is:
SD range Defect rate (%) Defects/Million± 2 SD 4.5 45,400± 3 SD < 0.27 ¬ 2,700± 4 SD 0.0063 63± 5 SD 0.0057 0.57± 6 SD 0.000002 0.002
Having a 6-SigmaProcesses -
virtuallyeliminates
defects
Six Sigma
John Calleja (M.P.S) Oct 2014 36
%TEA (%ATE) = Total Error Allowable %◦ Source = Biol .Var. Goals (Opt, Des, Min), RCPA,
CLIA, % Bias = Lab’s Bias vs. True Value (Target Value) % CV = Lab’s B/R precision Implication - Not all
6-Sigma estimatesare equivalent !
%TEA - % BiasSigma Metric = --------------------
% CV
Six Sigma- example.
John Calleja (M.P.S) Oct 2014 37
+ ATE = Allowable TotalError
- ATE
= (6-2)/1Sigma = 3
Ref: Total Analytical Error from Concept to Application – Westgard – Sept 13
Bias = 2SDSD = 1
John Calleja (M.P.S) Oct 2014 38
Various Sigma Metrics vs. Specification Limits
Ref. http://sixsigmatutorial.com/defect-based-six-sigma-metrics-dpo-dpmo-ppm-dpu-yield/276/
Consider theimpact of a
+/- 3sigma shift
LSL USL
Compare :a) 6∂ ..
LSL USL
b) 3∂ ..
-6 -5 -4 -3 -2 -1 0 1 2 3 4 5 6
-6 -5 -4 -3 -2 -1 0 1 2 3 4 5 6
Most Desirable
LessDesirable
4-Sigmaupwardsdesirable
John Calleja (M.P.S) Oct 2014 39
Method Decision Chart - Westgard
X-Axis max= (1/2 TEA)
Y-Axis max= TEA
Sigma forLab = 4
Draw Lines for each Sigma, From TEA on Y-Axis to:◦ TEA/2 on X-Axis for Sigma = 2 Line◦ TEA/3 on X-Axis for Sigma = 3 Line … etc
Plot Lab “Operating Point,” Bias & CV .. to give Sigma(=4)
Adv: Allowsyou to
Visualise toeffect of Bias &CV on Sigma
John Calleja (M.P.S) Oct 2014 40
Sigma SQC Selection Graph - Westgard
◦ ∆SEcrit = [(ATE – Bias)/SD] -1.65◦ [(ATE – Bias)/SD] = Sigma Metric◦ Sigma = ∆SEcrit + 1.65
Allows Graph tobe rescaled interms of Sigma
Draw vertical linecorresponding to
test Sigma
Read off which QC RulesIntersect w Sigma Line
@ 90 % Pr. of Rejection.
Power FunctionPlot
The size of themedicallyimportantsystematic
error
Various QC Rules
John Calleja (M.P.S) Oct 2014 41
Should we worry about Bias- If so when ?
So when we are considering6 Sigma -
John Calleja (M.P.S) Oct 2014 42
Should we Worry About Bias ? In the US – Yes
◦ Labs are assessed in the context of Regulatory Goals (CLIA) which are aboutTotal Error – of which Bias is a component
In Australia◦ We need to distinguish between Lab Bias & Method Bias
If Lab Bias◦ Definitely -Yes◦ Because – this generally indicates some sort of error in our
implementation of our method. If Method Bias
◦ The general position has been that … so long as we have areference interval that is appropriate to our method - then - No.
However – The pressure is increasingly - “Yes”◦ The existence of accepted Diagnostic, Action & Treatment cut-offs HBA1c, Glucose, Cholesterol, TSH
◦ Emergence of Common Health Records◦ Development of Common Reference Intervals
John Calleja (M.P.S) Oct 2014 43
Effects of usingdifferent TE goals on
Six-Sigma
“Six Sigma” is notan “absolute”Specification !
John Calleja (M.P.S) Oct 2014 44
Comparison of Sigma’s by different TEgoals
Various Goals.Labs Means &
SDs & CVs
AchievedSigmas
Analytes
John Calleja (M.P.S) Oct 2014 45
Comparison of Sigma’s by different TEgoals
Various Goals.Labs Means &
SDs & CVs
AchievedSigmas
For both qclevels
John Calleja (M.P.S) Oct 2014 46
Comparison of Sigma’s by different TE goals –Glucose
Example for Gluc. @ 2 QC concentrations CVs achieved are b/w optimal & desirable. Sigma’ s calculated for different TE Goals. Sigma achieved is different at different concentrations – for above, better at High
Conc. Sigma is better when you have no Bias
◦ Data Shown with & without a slight +ve Bias vs Targets of < 2%
Sigma is “Not an Absolute Parameter”-> If you are quoted a Sigma for a method – Keep all the above factors in
mind.
No Bias
With Bias
CV’s Des. -Opt
Ask: - Which TE goal used?
– Has Bias been included?
Sigma Varies, dependingon which TE Goal
selected
TE Goals -BV, RCPA,
CLIA
John Calleja (M.P.S) Oct 2014 47
Can View Visually – IgA - QC Unity Real Time
Nominate whichTE Goals to use
for givenAnalyte
SuperimposesCalculated Total
Error Range
TE
Plots LJ Plotagainst assigned
Target & SD limits.
SIGMA =(13.5-0.85) / 2.57
= 4.9
John Calleja (M.P.S) Oct 2014 48
3.0 What steps should wefollow when we set our QCTargets & Limits.
John Calleja (M.P.S) Oct 2014 49
3.1 Target Values.
John Calleja (M.P.S) Oct 2014 50
3.1 Various Sources ofTarget Values
QC Package Inserts Consensus Values from a centralised QC DataBase eg. BioRad’s QC-Net, Unity Real Time Set by Your Own Laboratory
.. Will not Discuss Target Values further- Focus will be on Allowable Limits
But – ( some Slides on Target Setting included )
John Calleja (M.P.S) Oct 2014 51
3.1.1 Manufacturer QC Kit Insert Values
The insert statesinstrument / method
specific values !
FT4: +/- 3sd QC Limits7.36 – 11.3 pmol/L22.3 – 35.9 pmol/L52.7 – 80.8 pmol/L
John Calleja (M.P.S) Oct 2014 52
3.1.2 Consensus Values from QC Database
World-Wide ReportMethod Specific Consensus
Values for QC Lot !
DisAdv: Lot must alreadybe in use by other
participants to be of use.
John Calleja (M.P.S) Oct 2014 53
3.1.3 Establish Your Own Target Values** Establish by
- Running the QC Material Several times
.. For all relevant tests
.. On all relevantInstruments / Modules / Channels.. Over a number of
days / runs / calibrations /operators .... Thereby exposing the process to as many possible
sources of variation, as is practical.
- Run the material passively, in parallel to your Existing QCs.
- When sufficient Data has been accumulated;
- Calculate Mean & SD- Remove any Outliers (Exclude values > 3sd)
- Re-Determine the Mean- Set this as the .... Target Value
John Calleja (M.P.S) Oct 2014 54
3.1.3.1 How Many Data Points ?
** Statistic: Standard Error of the Mean (SEM).
This statistic gives us the Amount of Error, associatedwith a given Target Value .. Given the
- SD of the data set &- the Number of QC observations.
The Accuracy of a Target Value .. Increases ...with an Increasing Number of Observations.
S where S = S.D.SEM = --- n = no. of control
n observations.
John Calleja (M.P.S) Oct 2014 55
0
10
20
30
40
50
60
0 1 2 3 4 5 6
Error of the Mean ( Arbitrary Units )
No. of Results
n = 30
n = 5
...No AppreciableDecrease in SEM
for n > 20-30
n = 20
3.1.3.1 How Many Data Points ?- Standard Error of the Mean (SEM)
John Calleja (M.P.S) Oct 2014 56
3.1.4 What if you haveMultiple Instrumentsmeasuring the same test ?
John Calleja (M.P.S) Oct 2014 57
Instrumentsof the same type ?
or ..On more than 1 measuringchannel, of an instrumentof the same type ?
Measure Cell 1 & 2
Instrument 1, 2 & 3
Between.. … DifferentInstrument types ?
Instrument 1, 2 & 3
YesNo
John Calleja (M.P.S) Oct 2014 58
3.1.4 Eg. MPS Glucose – Cobas c701- Are the recoveries across the 3 different c701s .. < 0.33CVi ?
Line 1
Line 2
Line 3
CV Goals, Opt, Des, Min = 1.63, 3.25, 4.9% / Bias Goal 0.33CVi = 2.15%
Max Bias L1: (4.72 – 4.75) = 0.03 | 0.03/4.735 = 0.63%
Max Bias L2: (15.48-15.4) =0.08 | 0.08/15.44 = 0.52%
Yes !Inter-Instrument
Bias is < 0.33CVi
John Calleja (M.P.S) Oct 2014 59
1.4.2 Recoveries differ for different instrumentswith different methodologies & calibrationtechniques
You generally, Can’t use the same targets,across different instrument types !
Roche e602
Siemens Vista
Vitros ECi
Abbott Architect
PSA
John Calleja (M.P.S) Oct 2014 60
3.2 What Steps should weFollow when we set our QCLimits !
John Calleja (M.P.S) Oct 2014 61
Step 2 - Put the labs achieved CV performance into perspective ..Compare to the Instrument / Reagent Manufacturer’s .. statedspecifications for Total CV - Refer to the manufacturers kit insert
Step 3 - Consider the Relevant Quality Goals ( CV Desired) , from the ISO TC212Hierarchy ; Evidence Based / Biol. Var’n / QAP Allowable Limits
Step 4 - Consider the assays Capability – or Six Sigma .
3.2 Setting Appropriate QC Ranges- Steps Involved
Step 1 - Review the SDs & CVs achieved from The Target Setting Studies- Compare this to the Labs Ongoing CVs at similar concentrations
Step 5 - Put all of this information together to determine what the allowable CV( or Quality Goal ) .. for your assay, should be .
Following Running-in Studies of the New QC.. Consider Quality Goals & Method Capability
– Don’t just settle on Mean +/- 2sd !
John Calleja (M.P.S) Oct 2014 62
Tgt = 1.00
SD = 0.051
CV = 5.1%
Range:
Min = 0.9
Max = 1.1
Tgt = 2.2
SD = 0.1
CV = 4.5%
Range:
Min = 2.0
Max = 2.4
Lab about to set targets & sd’s for 2qc’s for Trigs on a Beckman CX7 ..Based on achieved performance …
QC Level-1 QC Level-2
Setting QC Ranges- Acceptable Limits
Are theseranges
appropriate?
John Calleja (M.P.S) Oct 2014 63
? Appropriate QC Limits
3.2.1 Compare Evaluation QC CV toCV of Current QC Lot
John Calleja (M.P.S) Oct 2014 64
Trig - Expected Performance
CVs of current QCbetter (ie)
2.52% vs. 3.07%
Current QC
Evaluation QC
John Calleja (M.P.S) Oct 2014 65
? Appropriate QC Limits
3.2.2 Manufacturer Specifications
John Calleja (M.P.S) Oct 2014 66
Trig - Expected Performance
The Lab’s QCCVs are not as good
@2.0mmol/L
Note: ExpectedCVs
Tgt = 1.00
SD = 0.051
CV = 5.1%
Range:
Min = 0.9
Max = 1.1
Tgt = 2.2
SD = 0.1
CV = 4.5%
Range:
Min = 2.0
Max = 2.4
@ level
3.2
John Calleja (M.P.S) Oct 2014 67
? Appropriate QC Limits
3.2.3 Biologically Based CVa Quality Goals
John Calleja (M.P.S) Oct 2014 68
? Appropriate QC LimitsRefer to “Westgard Web-site”
www. westgard.com
CVi = 20.9%
John Calleja (M.P.S) Oct 2014 69
Lab’s CV Goals areWithin OptimalBiological Goals
Biological Variability
CVi (Triglyceride) = 20.9 %
1. Calculate theAnalytical CV Goals
CVa Minimal = 0.75 CVi= 15.75%
CVa Desirable = 0.5 CVi= 10.5%
CVa Optimal = 0.25 CVi= 5.25%
Tgt = 1.00
SD = 0.051
CV = 5.1%
Range:
Min = 0.9
Max = 1.1
Tgt = 2.2
SD = 0.1
CV = 4.5%
Range:
Min = 2.0
Max = 2.4
2. Compare Lab’s CVsw Biological Goals
Labs Trig QC Goals vs. Biological Limits
John Calleja (M.P.S) Oct 2014 70
? Appropriate QC Limits
3.3.4 QAP Allowable Limits
John Calleja (M.P.S) Oct 2014 71
? How do the Lab’sCV goals
Compare with QAPAllowable Limits
Labs Trig QC Goals .vs. QAP Limits
Tgt = 1.00
SD = 0.051
CV = 5.1%
Range:
Min = 0.9
Max = 1.1
Tgt = 2.2
SD = 0.1
CV = 4.5%
Range:
Min = 2.0
Max = 2.4
Labs CVs
John Calleja (M.P.S) Oct 2014 72
Labs Trig QC Goals .vs. QAP Limits -
What CV do we needto achieve 2,3,4,5,6
Sigma within the QAPAllowable Limits .. ?
Sigma 1sd CV .
2 Sigma (1.71–1.51)/2 = 0.1, (0.1/1.71)x100 = 5.8%
3 Sigma (1.71-1.51)/3 = 0.067 (0.067/1.71)x100 = 3.9%
4 Sigma (1.71-1.51)/4 = 0.05 (0.05/1.71)x100 = 2.9%
5 Sigma (1.71-1.51)/5 = 0.04 (0.04/1.71)x100 = 2.3%
John Calleja (M.P.S) Oct 2014 73
Labs Trig QC Goals .vs. QAP Limits - 2 Sigma
What CV do we needto achieve 95.5% ofall results .. withinthe QAP Allowable
Limits .. ?
• One way we can consider the QAP Target & Allowable Limitsis as .. mean +/- 2 sd (95% C.I.)
• So .. 1sd = (1.71 – 1.51) / 2 = 0.1
• So .. CV required is .. ( 0.1 / 1.71 ) x 100 = 5.8 %
John Calleja (M.P.S) Oct 2014 74
Labs Trig QC Goals .vs. QAP Limits – 3 Sigma
What CV do we needto achieve 99.7% ofall results .. withinthe QAP Allowable
Limits .. ?
• Can consider Allowable range as mean +/- 3 sd (99.7% C.I.)
• So .. 1sd = (1.71 – 1.51) / 3 = 0.067
• So .. CV required is .. ( 0.067 / 1.71 ) x 100 = 3.9 %
John Calleja (M.P.S) Oct 2014 75
Labs Trig QC Goals .vs. QAP Limits – 4 Sigma
What CV do we needto achieve results
within +/-4 sigma ..of the QAP Allowable
Limits .. ?
• Can consider Allowable range as mean +/- 4 sd (99.994% C.I.)
• So .. 1sd = (1.71 – 1.51) / 4 = 0.05
• So .. CV required is .. ( 0.05 / 1.71 ) x 100 = 2.9 %
John Calleja (M.P.S) Oct 2014 76
? Appropriate QC Limits
3.3.5 State of the Art Performances
John Calleja (M.P.S) Oct 2014 77
? Appropriate QC Limits – State-of-the-Art
Method Median CV = 3.5%
50th percentile CV = 3.2%20th percentile CV = 2.4%
Ranked Within Lab CVs
John Calleja (M.P.S) Oct 2014 78
3.4.6 Put it all Together
Transform Inputs .. into .. Appropriate QC Ranges
Give the Lab QC Limits w aSigma that as best as possible
insulate it against theundesirable impact of shifts &
increases in imprecision
John Calleja (M.P.S) Oct 2014 79
Put it All Together !
Tgt = 1.00
SD = 0.051
CV = 5.1%
Tgt = 2.2
SD = 0.1
CV = 4.5%
Lev 1: Tgt =1.01SD=0.033, CV= 3.3%
Lev 2: Tgt=2.2,1SD=0.065 CV=3.0% 79
Lab’s Evaluation CVs
CVa Minimal = 0.75 CVi= 15.75%
CVa Desirable = 0.5 CVi= 10.5%
CVa Optimal = 0.25 CVi= 5.25%
Manufacturers CV Specs.
RCPA QAP ALEs
QAP State-of-the-Art
Biological Goals
2 sigma QAP ALECV = 5.8%3 sigma QAP ALECV = 3.9%
Lab’s Ongoing CVs
Tgt = 0.95
SD = 0.031
CV = 3.3%
Tgt = 2.0
SD = 0.06
CV = 3.0%
50th %CV = 3.2%20th %CV = 2.4%
Method CV = 3.5%
Set Targets to:
4 sigma QAP ALECV = 2.9%
3.2
John Calleja (M.P.S) Oct 2014 80
? Appropriate QC Limits
3.4.7 Consider Six Sigma Capability
John Calleja (M.P.S) Oct 2014 81
Method Decision Chart – CalculateSigma
X-Axis max= (1/2 TEA)
Y-Axis max= TEA
Sigma forLab = 4.24
Draw Lines for each Sigma, From TEA on Y-Axis to:◦ TEA/2 on X-Axis for Sigma = 2 Line◦ TEA/3 on X-Axis for Sigma = 3 Line … etc
Plot Lab “Operating Point,” Bias & CV .. to give Sigma(=4)
T.E. Goal = Biol.Var. TE Goal
Desirable = 14%
John Calleja (M.P.S) Oct 2014 82
Consider Lab’s Sigma Capability
+ ATE =14%Allowable Total Error
- ATE = -14%
Sigma = 4CV=3.3 % x 3
=9.9%
Can tolerate a+/- shift of:14 - 9.9 = 4.1%
LSL USL
GoodCapability !
John Calleja (M.P.S) Oct 2014 83
3.4.8 A laboratory Toolto Assist the Process
John Calleja (M.P.S) Oct 2014 84
Laboratory Tool - Word Template / 2 Pgs./ Aanalyte
1. LJ Plots – All ActiveChannels
3 Levs - Current & Evaln. QC
7.Manufacturer
PrecisionSpecs
2. Summary Stats –Current & Evaln.
QC
8. Comments & FinalQC Target & 1sd
Settings
3. QAPPerformance Latest
Report
4. BiologicalGoals
Opt, Des, Min
6. State-of-the-Art20th,50th & Mthd
CV
5. QAP ALEs &2, 3 & 4 sigma
CVs
John Calleja (M.P.S) Oct 2014 85
Laboratory Tool - Word Template / 2 Pgs./ Aanalyte
1. LJ Plots – All ActiveChannels
3 Levs - Current & Evaln. QC
2. Summary Stats –Current & Evaln.
QC3. QAP
Performance LatestReport
4. BiologicalGoals
Opt, Des, Min
5. QAP ALEs &2, 3 & 4 sigma
CVs
6. State-of-the-Art20th,50th & Mthd
CV
7.Manufacturer
PrecisionSpecs8. Comments & Final
QC Target & 1sdSettings
1. LJ Plots – All ActiveChannels
3 Levs - Current & Evaln. QC
7.Manufacturer
PrecisionSpecs
2. Summary Stats –Current & Evaln.
QC
8. Comments & FinalQC Target & 1sd
Settings
3. QAPPerformance Latest
Report
4. BiologicalGoals
Opt, Des, Min
6. State-of-the-Art20th,50th & Mthd
CV
5. QAP ALEs &2, 3 & 4 sigma
CVs
John Calleja (M.P.S) Oct 2014 86
4.0 Using Quality Goalswhen Problems arise !
John Calleja (M.P.S) Oct 2014 87
4.1 How should we Assess /Action a Systematic shift ?
John Calleja (M.P.S) Oct 2014 88
Assessing / Actioning a ShiftSteps to Take:
Attribute a Cause Eg. Reagent or Cal Lot Change Reagent Reformulation
Assess Magnitude of Shift Percent deviation from QC Target or %Bias Pre & Post shift “patient comparison studies” Patient Data Extract – movement in averages & percentiles
Assess Clinical Relevance Clinical Consultation, 0.25 x (CVI
2 +CVG2)1/2, 0.33CVi
Consultation with Manufacturer about a corrective action. New Lot Number of Calibrator / Rgt Calibrator Set Point Reassignment
Change Pt. Ref. Intervals, to compensate for the shift
Apply a corrective Slope &/or Offset to results◦ Derived from Pre & Post shift .. patient comparison studies Examination of influence of shift on patient averages
Change QC Targets
TakeCorrective
Action
John Calleja (M.P.S) Oct 2014 89
4.1.1 Attribute the Causeof the Shift.
John Calleja (M.P.S) Oct 2014 90
Attribute the Cause ?
Start w LJ Plots Draw in dates of calibrator & reagent Lot changes etc .. Rule out Faulty Reagent/ Poor Calibration Attribute the Cause, of the shift -> Shift due to Reagent Lot
Change (Lot 602091)
eg. GGT
R: 23/8 602091
C: 27/8
R: 6/6 698158 R: 18/7 600696
C: 2013 169513
John Calleja (M.P.S) Oct 2014 91
Attribute the Cause ?
Start w LJ Plots Draw in dates of calibrator & reagent Lot changes etc .. Attribute the Cause, of the shift -> Shift due to Reagent Lot
Change (Lot 602091) Estimate Bias Magnitude Assess Significance % Chng < Optimal => Not Sig.
eg. GGT
Target Shift to Diff % Diff159 162 3 1.9 %
R: 23/8 602091
C: 27/8
R: 6/6 698158 R: 18/7 600696
C: 2013 169513
John Calleja (M.P.S) Oct 2014 92
4.1.2 Assess Magnitudeof Shift.
John Calleja (M.P.S) Oct 2014 93
eg. Perform Pt. Comparisons(eg) Assess Ca++ Bias .. Post vs. Pre New Lot of Calibrator
Y = 1.0965x – 0.1723
At URL: 2.6 mmol/LNew Result = 2.679
Bias = +0.0786%Bias = 3.02%
4.1.2.1 Estimate Magnitude of Shift
Results w Fomer Lot
ResultswNew Lot
John Calleja (M.P.S) Oct 2014 94
4.1.2.2 What if we initiallymissed the shift ..
&.. can’t perform
patient samplecomparisons ?
John Calleja (M.P.S) Oct 2014 95
4.1.2.2 Perform Patient Data Extracts
Extract Data from LIS for a period including before &after the shift
Calculate “moving medians”..per 30 .. or .. per 1000 sample results (if lots of data)
Plot .. Moving Median vs. Date/Time Examine for a significant shifts / Estimate Magnitude
Most useful.. when the detection of a shift hasinitially been missed.
How …
- Plot Patient Medians/Percentiles
John Calleja (M.P.S) Oct 2014 96
4.1.2.2 Patient Data Extracts
Write Formulas tocalculate the median forthe preceding 30 samples
=Median(F2:F31)
=Median (F3:F32)
=Median(F4:F33)
- example: raw data
Copy Formulas down spreadsheet .. toget rest of .. rolling 30 sample medians.
John Calleja (M.P.S) Oct 2014 97
4.1.2.2 Example: Plot the MovingMedian versus Date/Time …
Moving MedianSensitive to the ShiftBias ~ 0.15 mmol/L
John Calleja (M.P.S) Oct 2014 98
4.1.3 Assess Significanceof Shift.
Bias Goal
<0.125(CVi2+CVg2)1/2
<0.25(CVi2+CVg2)1/2
<0.375(CVi2+CVg2)1/2
John Calleja (M.P.S) Oct 2014 99
Optimal
Desirable
Minimum
4.1.3 Assessing Significance of Shift
• Assess Shift againstBiological VariabilityBias Goals !
John Calleja (M.P.S) Oct 2014 100
Callum Fraser ‘s .. Quality specifications ..for the allowable differences between twomethods .. used to analyse the same analytein the same laboratory;
4.1.3 To assess differences b/wtwo instruments.
Allowable difference < 0.33 CVi
John Calleja (M.P.S) Oct 2014 101
4.1.3 Estimate Significance of Shifteg. Pt. Comparisons for Ca++ Bias.. Post vs. Pre New Lot of Calibrator
Y = 1.0965x – 0.1723
At URL: 2.6 mmol/LNew Result = 2.679
Bias = +0.0786%Bias = 3.02%
1. Assess against, Inter-instrument Bias Goal:
0.33CVi (1.9) = 0.63%2. Assess against, Opt, Des & Min Biol. Bias Goals:
Desirable Bias: 0.250 (CVI2 + CVG
2)1/2 = 0.85%Minimal Bias Goal: 0.375 (CVI
2 + CVG2)1/2 = 1.27%
Our %Bias =3.02% -> Worse .=> Significant !
John Calleja (M.P.S) Oct 2014 102
4.1.3 Estimate Significance of Shifteg. Pt. Comparisons for Ca++ Bias.. Post vs. Pre New Lot of Calibrator
Y = 1.097x – 0.172
At URL: 2.6 mmol/LNew Result = 2.679
Bias = +0.0786%Bias = 3.02%
Assess against, Opt, Des & Min Biol. Bias Goals:Optimum Bias: 0.125 (CVI
2 + CVG2)1/2 = 0.43%
Desirable Bias: 0.250 (CVI2 + CVG
2)1/2 = 0.85%Minimal Bias Goal: 0.375 (CVI
2 + CVG2)1/2 = 1.27%
Our %Bias =3.02% -> Worse .=> Significant !
John Calleja (M.P.S) Oct 2014 103
4.1.4 What if we decide wehave a ClinicallySignificant Shift ..
John Calleja (M.P.S) Oct 2014 104104
4.1.4 Could Derive a Corrective Slope &Offset to realign the performances
Y = 1.097x – 0.172
Derive Corrective Slope & Offset(from Regression Eq):
Method Alignment Improved
Re-Perform Comparisons withSlope & Offset installed !
Corrective Slope = (1/1.097) = 0.91
Corrective Offset = (0.172/1.097) = +0.16
John Calleja (M.P.S) Oct 2014 105
4.1.5 What do we do withQC Targets ? ...
John Calleja (M.P.S) Oct 2014 106
4.1.5.1 What do we do about QC Targets ?
If Shift assessed as Clinically Tolerable
◦ Calculate the mean values for all relevant QC levels.. from the data after the Shift !
◦ Re-Assign the QC Target Values to these values
If Shift assessed as Clinically Significant◦ If Corrective Factors Implemented Should be no need to amend QC Targets
◦ If Other Corrective Action Taken(eg) Reference Intervals modified ..
due to assay Re-Standardisation Calculate the mean values for all relevant QC levels
.. from the data after the Shift ! Re-Assign the QC Target Values to these values
John Calleja (M.P.S) Oct 2014 107
4.1.5.2 What do we do about QC Limits ?
If we have already carefully calculated CV Goals ..-
Re-calculate our SDs .. to achieve equivalentCV goals, at the new target concentration.
◦ CV % = (SD/Mean ) x 100◦ If Former (Pre-Shift) Target = 100, SD = 5, CV Goal = 5%
◦ If Assay Shifted to Mean= 80 To maintain a CV Goal of 5% Re-arrange CV equation to solve for SD SD = (CV/100) x Mean SD = (5/100) x 80 SD = 4
John Calleja (M.P.S) Oct 2014 108
4.2 How should we Assess /Action .. a major QC Failure(eg) due to an Inst. Failure
John Calleja (M.P.S) Oct 2014 109
Considering the Scenario !
◦ Successful QC Event .. @ 10.00am Patient n= 1 Patient n= 2 Patient n= .. Patient n= .. Patient n= 50 Patient n= 51 Patient n= .. Patient n= . . Patient n=100 Patient n=101 Patient n= … Patient n= … Patient n= 150 Patient n= 151 Patient n= .. . Patient n= … Patient n=200
◦ Failed QC Event, > 3SD .. @ 2.00pm
200 Samplesrun in-betweenlast successful
QC & Failed QCEvent !
John Calleja (M.P.S) Oct 2014 110
4.2. What do we do with PatientResults that were Reported inthe Affected Run ?
◦ Consider that the failure .. may have occurred at any-time after the last successful QC event and the knownfailure.
◦ Therefore need to identify an accurate failure time-point!
◦ Procedure: Select representative samples from in-between the last
successful QC event right upto the failed QC Event Pick two to three samples in every 20 - 30 samples (depending
on size of failed batch) – but at shorter intervals towards failedQC Select the samples chronologically Re-run the selected samples Assess all samples for significant differences between the
repeats
John Calleja (M.P.S) Oct 2014 111
Pin-Pointing where run failed !
◦ Successful QCEvent Patient n= 1 Patient n= 2 Patient n= .. Patient n= .. Patient n= 50 Patient n= 51 Patient n= .. Patient n= . . Patient n=100 Patient n=101 Patient n= … Patient n= … Patient n= 150 Patient n= 151 Patient n= .. . Patient n= … Patient n=200
◦ Failed QC Event
◦ Selected Patients Patient n= 1 Patient n= 2 Patient n= 30 Patient n= 31. Patient n= 60 Patient n= 61 Patient n= 90 Patient n= 91 Patient n=120 Patient n=121 Patient n= 150 Patient n= 151 Patient n= 180 Patient n= 181 Patient n= 190 Patient n= 195 Patient n= 200
◦ Failed QC Event
Failure mayhave
occurredanywherewithin the
Batch !
SelectRepresentative
Patientsamples at
regular Time-Points fromBatch -> for
Re-run Checks
Check atsmallerintervalstowards
failed QC
Run Samples &Check where
SignificantDifferences
appear.
If FailurePin-pointed tobetween 150
& 180onwards ->
Rerun allpatients afterpatient 150
John Calleja (M.P.S) Oct 2014 112
4.2.1 Assessment of Differencesin Repeats –
Criteria Derived from :
For serial analysis of the same sample◦ Intra-individual Variation is N/A◦ Simplifies to:
Where .. Z= 1.96 (95% c.i.)
When the same sample is repeated .. asignificant difference is .. If a repeat resultdifference is
RCV (Reference Change Value) = √ 2 x Z x √ ( CVa2 + CVi
2 )
> 2.77 CVa or if the result classificationchanges
= √ 2 x Z x √ ( CVa2 ) = 2.77CVa
AnalyticalVariation
Intra-Individual BiologicalVariation
Two analysis
CoverageFactor
or z=1.65 (unidirectional)
> 2.33 CVa or if the result classification changes
John Calleja (M.P.S) Oct 2014 113
4.2.2 ∆SE or ∆RE critical
Critical Systematic or Random Error that wouldSHIFT or WIDDEN the result distribution enoughto exceed the allowable T.E. specification.
When the same sample is repeated .. asignificant difference is ..
∆SE critical = [ (TEA – Bias ) / CV ] –1.65
% Difference in Results > ∆SE Critical .. or∆RE Critical
Lab CV%Total Allowable Error % Lab Bias %
∆RE critical = [ (TEA – |Bias| ) / ( 1.65 xCV )
John Calleja (M.P.S) Oct 2014 114
Previously Reported ResultsExample: Trig. Problem - 2.77CVa method
Affected Samples ->Amend Results !
John Calleja (M.P.S) Oct 2014 115
Recovery from a Failed Run !
Final Step:
◦ If Repeated Result > 2.77CVa or considered clinicallysignificantly different (eg. Classification Change or byClinical Consultation – w Chemical Pathologist ) Amend Result Re-report with an Amended Report Comment if Critical Change in Result – Phone Dr.
John Calleja (M.P.S) Oct 2014 116
Patient samples come to us for Diagnosis & Monitoring Monitoring has the tighter quality requirement -> Quality
Standards should be based on Precision The Total Error concept & Six Sigma assists us in
considering both. When setting QC Targets & Limits - Consider Quality
Goals & in what Hierarchy to apply them. Construction of a Template incorporating all the relevant
Goals & Information may help the Process Quality Goals can also be used to assist with Assessing a Systematic Shift. Assessing which patients may need to be repeated
when there is a significant Run Failure.
8. Summary
J.Calleja - Melb. Path. - Oct 2014117
References
Chapter 19
John Calleja (M.P.S) Oct 2014 118
The End ...
Thankyou for your attention !