Analytical performance specifications two years after ... meeting 2016... · Analytical performance...

Analytical performance specifications two years after

Milan conference

Prof Mauro Panteghini

CIRME Scientific Coordinator

Definition

• Analytical performance specifications:

Criteria that specify (in numerical terms)

the quality required for analytical

performance in order to deliver laboratory

test information that would satisfy clinical

needs for improving health outcomes.

Profession

(e.g., JCTLM, IFCC, EFLM):

Define analytical objectives: reference

measurement systems (traceability chain)

and associated clinically acceptable

uncertainty (fitness for purpose)

Diagnostic manufacturers: Implement suitable analytical systems

(platform, reagents, calibrators, controls)

fulfilling the above established goals

End users (clinical laboratories): Survey assay and laboratory performance

through IQC and EQA redesigned to meet

metrological criteria

Adapted from Panteghini M, Clin Chem Lab Med 2010;48:7

Steps of the process and different responsibilities in implementing

traceability of patient results and defining their uncertainty

TH

E T

EM

PLE

OF

LAB

OR

AT

OR

Y S

TA

ND

AR

DIZ

AT

ION

REFERENCE METHODS

REFERENCE MATERIALS

ACCREDITED REFERENCE LABORATORIES

TRACEABLE REFERENCE INTERVALS

AND DECISION LIMITS

APPROPRIATELY ORGANIZED

ANALYTICAL QUALITY CONTROL

TARGETS FOR UNCERTAINTY AND ERROR OF

MEASUREMENT (FIT FOR PURPOSE)

Bra

ga

F &

Pa

nte

gh

ini M

, Clin

Ch

im A

cta 2

01

4;4

32

:55

•• Definition and approval of reference measurement systems, Definition and approval of reference measurement systems, possibly in their entirety;possibly in their entirety;

•• Implementation by IVD industry of traceability to such referenceImplementation by IVD industry of traceability to such referencesystems in a scientifically sound and transparent way;systems in a scientifically sound and transparent way;

•• Definition by the profession of the clinically acceptable Definition by the profession of the clinically acceptable measurement uncertainty for each of the analytes used in the measurement uncertainty for each of the analytes used in the clinical field;clinical field;

•• Adoption by EQAS providers of commutable materials and use of anAdoption by EQAS providers of commutable materials and use of anevaluation approach exclusively based on trueness;evaluation approach exclusively based on trueness;

•• Monitoring of the analytical performance of individual laboratorMonitoring of the analytical performance of individual laboratories ies by the participation in EQAS that meet metrological criteria andby the participation in EQAS that meet metrological criteria andapplication of clinically acceptable limits;application of clinically acceptable limits;

•• Abandonment by users (and consequently by industry) of Abandonment by users (and consequently by industry) of nonspecific methods and/or of assays with demonstrated nonspecific methods and/or of assays with demonstrated insufficient quality.insufficient quality.

The definition and use of the reference system

concept for standardization of measurements must be

closely associated with the setting of targets for

uncertainty and error of measurement in order to make

it clinically acceptable.

If these goals are not objectively defined and fulfilled,

there is a risk of letting error gain the upper hand,

thus obscuring the clinical information supplied by

the result and possibly nullifying the theoretical

advantages of metrological traceability and even

causing negative effects on patients' outcome.

Braga F & Panteghini M, Clin Chim Acta 2014;432:55

L Thienpont et al., Clin Chem Lab Med 2004;42:842

Barriers to practical achievement of

traceability scope

�Lack of definition of the clinically

allowable uncertainty for validation of

the metrological traceability chain of

each measurand

The Essential Question…

“What amount of

medical harm due to

analytical error is it OK

to let go undetected?”

The most innovative aspect of the new consensus is that it is recognized that some models are better suited for certain measurands than for others; the attention is therefore primarily directed towards the measurand and its biological and clinical characteristics.

Conference conclusionsDefining Analytical Performance Specifications

• Three models: outcome, biology, state of the art

• Important: w. high quality studies and updated data

• Measurands can have different performance

specifications depening on its use

• More work to be done to produce high quality data that can be used

for performance specifications

• More work to be done to judge how to apply the performance

specifications

An EFLM Task Force on Performance Specifications

in Laboratory Medicine (TF-PS)

has been created to coordinate the activities of the

Task & Finish Group (TFG) established as outcome of the

1st Strategic Conference

1. The measurand has a central role in diagnosis

and monitoring of a specific disease ⇒

outcome model

2. The measurand has a high homeostatic control

⇒ biological variability model

3. Neither central diagnostic role nor sufficient

homeostatic control ⇒ state-of-the-art model

Workflow for allocation of laboratory measurands to

different models for performance specifications

The application of the analytical

performance specifications can be

modulated depending on its use. For

example:

Reference material providers

Manufacturers producing calibrators

Individual laboratories who provide

patient results

EQAS organizations

100%

uu refref

(u(u22refref + u+ u22

calcal ))½½

(u(u22refref + u+ u22

calcal + u+ u22randomrandom ))½½

System imprecision

System calibration

uncertainty

Individual lab

performance

(IQC safety margin)

Measurement

uncertainty

budget

Uncertainty of

references

Measurand definition

Patient result

≤33%

≤50%

Recommended limits for combined uncertainty budget (expressed as

percentage of total budget goal) in traceability implementation

[Braga F, Infusino I, Panteghini M. Clin Chem Lab Med 2015;53:905]

uuCC >>2.5%>>2.5%

ERMERM--DA470DA470

U.S. National Reference U.S. National Reference

Preparation no. 12Preparation no. 12--0575C0575C

ERMERM--DA470k/IFCCDA470k/IFCC

ManufacturerManufacturer’’s working calibrators working calibrator

(master lot)(master lot)

ManufacturerManufacturer’’s product calibrators product calibrator

Routine sample resultRoutine sample result

value transfer protocolvalue transfer protocol

ManufacturerManufacturer’’s standing s standing

immunoassayimmunoassay

Commercial immunoassayCommercial immunoassay

value transfer value transfer

protocolprotocol

value transfer protocolvalue transfer protocol

uuCC 1.01%1.01%

uuCC 1.61%1.61%

uuCC 1.74%1.74%

Serum albumin: An exampleSerum albumin: An example

Combined Standard Combined Standard

Uncertainty (Uncertainty (uuCC))

Panteghini M, Clin Chem Lab Med 2012;50:1237Panteghini M, Clin Chem Lab Med 2012;50:1237

Infusino I & Panteghini M, Chim Clin Acta 2013;419:15Infusino I & Panteghini M, Chim Clin Acta 2013;419:15

Braga F & Panteghini M, Clin Chim Acta 2014;432:55Braga F & Panteghini M, Clin Chim Acta 2014;432:55

The uC associated with serum

albumin results on patient

specimens is greater than the

minimal goal for uncertainty

(≤2.4%), showing that the

uncertainty of albumin

measurement in serum is

probably too high to meet the

requirements for its clinical

application.

Specifications of reference measurement procedure defined by Specifications of reference measurement procedure defined by

intended useintended use……

……intended use is the certification of reference materialsintended use is the certification of reference materials……

……the specifications of certified reference materials are defined the specifications of certified reference materials are defined by by

the performance needs of the clinical assays.the performance needs of the clinical assays.FieldField

assaysassays

CRMsCRMs

RMPRMPu

nce

rtain

tyu

nce

rtain

ty

To assure that the expanded combined uncertainty associated withTo assure that the expanded combined uncertainty associated with patient patient

results fulfill the total budget goal, the higher order referencresults fulfill the total budget goal, the higher order references should es should

display uncertainty at most equal to 1/3 of the total budget goadisplay uncertainty at most equal to 1/3 of the total budget goal.l.

Turning the problem upside down Turning the problem upside down

Focus first on the field assaysFocus first on the field assays

System imprecision

System calibration

uncertainty

Individual lab

performance

(IQC safety margin)

Measurement

uncertainty

budget

Uncertainty of

references

According to the outcome-based study of misclassifi cation rates, the maximum allowable goal for 100% total uncertainty b udget of cTnIassays is 13% (minimum quality goal) for the clinic al result and which allows for <2% result misclassification.

Anticipate 50% of uncertainty budget (6.5%) at manu facturer’s calibration and value transfer level.

≤33% of uncertainty budget due to SRM uncertainty (~ 4.5%).

IFCC WGIFCC WG--TNI Technical DiscussionTNI Technical Discussion

Value assignment of NIST SRM 2922 and measurement uncertaintyValue assignment of NIST SRM 2922 and measurement uncertainty


Patient result




example:




patient results

EQAS organizations

Clin Chem Lab Med 2013; 51:973Clin Chem Lab Med 2013; 51:973

→→ Allowable limit for the expanded (combined) Allowable limit for the expanded (combined)

uncertainty of manufactureruncertainty of manufacturer’’s commercial s commercial

calibrators @ calibrators @ 50% of the goals 50% of the goals [note that these are goals for random variability, as [note that these are goals for random variability, as at the at the

calibrator level the systematic error (bias), in agreement with calibrator level the systematic error (bias), in agreement with

the metrological traceability theory, must be corrected if the metrological traceability theory, must be corrected if

present in a non negligible amount]present in a non negligible amount]

System imprecision

System calibration

(combined) uncertainty

Individual lab

performance

(IQC safety margin)

Measurement

uncertainty

budget

Need to define criteria for manufacturers that can be achieved

for their calibrators leaving enough uncertainty budget for the

laboratories to produce clinically acceptable results.


Patient result

The manufacturer must indicate the

combined uncertainty associated with

calibrators when used in conjunction

with other components of the analytical

system (platform and reagents).

Such uncertainty estimates provided by

the manufacturer must include the

uncertainty associated with higher

levels of the metrological traceability

chain.

Measuring system imprecision

Measuring system calibration

uncertainty

Individual lab

performance

(IQC safety margin)

Measurement

uncertainty

budget

Uncertainty of

references


Patient result

IVD ManufacturerIVD Manufacturer

IVD MANUFACTURER contribution to the

measurement uncertainty budget

Braga F & Panteghini M, Clin Chim Acta 2014;432:55




example:




patient results

EQAS organizations

[Braga F & Panteghini M, Clin Chem Lab Med 2013;51:1719]

HbA1c reference system and

associated combined standard uncertainty

uc

Clin Chem Lab Med 2013;51:1719Clin Chem Lab Med 2013;51:1719 ––2626

Further advances are needed to:

1. reduce uncertainty associated with

higher-order metrological references

(reference materials and procedures)

2. increase the precision of commercial

HbA1c assays

uc

Clin Chem Lab Med 2016; 54(3): e71–e73




example:




patient results

EQAS organizations

Analytical performance specification (APS) derivation

should be added to the Miller’s EQAS categorization

Category 1/2A → Milan model 1 or 2 as basis for APS

Category 1/2B → Other models

[Miller WG et al. Clin Chem 2011;57:1670][Miller WG et al. Clin Chem 2011;57:1670]

Infusino I et al. Clin Chem Lab Med 2016;in press.

99thth CIRME International Scientific MeetingCIRME International Scientific MeetingSTRUCTURING EQAS FOR MEETING METROLOGICAL CRITERIA:STRUCTURING EQAS FOR MEETING METROLOGICAL CRITERIA:

READY FOR PRIME TIMEREADY FOR PRIME TIMEMilano Milano –– 27 November 201527 November 2015

Basis for performance specifications

PROVIDER MODELS

RCPAQAP Australia Combination of BV and state of the art

SKML The Netherlands Combination of BV and state of the art

NOKLUS Norway Fixed percentage limits and based on a

combination of BV, state of the art and expert

opinion

SEQC Spain Combination of BV and statistical results

WEQAS UK Combination of BV and state of the art

SEHH Spain Statistical/state of the art/BV

CTCB France z-score/state of the art/limits given by

scientific societies or other/limits based on

clinical impact

TFG on Performance

Specifications for EQAS

• Apply Milan models to describe EQAS

performance specifications

• Develop common performance specifications

based on Milan models

• Focus on “type 1” EQAS (commutable materials,

reference measurement for target, repeated samples)

TFG on EQAS: Actions

• Develop terminology to describe EQAS

performance specifications

• Use terminology to describe current limits

• Support EQAS using descriptions to

communicate specifications (and meaning of

specifications) to users

• Consider best specifications to meet goals (may

be different for different schemes)

• A single result includes effects of both bias and

imprecision

• Bias and imprecision effects cannot be

separated

• Quality standards assess “total error”

• Applies to multiple samples, if they are

analysed separately

Single results in EQAS:

the interpretation

Clin Chem 1974;20:825

TE = bias + Z x SDA

Z=1.65

Conventional model is flawed:

TEA = 0,25 CVB + 1,65 (0,5 CVI)

Summing of mutual exclusive terms

Oosterhuis WP. Clin Chem 57 (2011):1334

Gowans et al. proposed an alternative

model in which the maximum allowable

bias and imprecision are interrelated and

described in a curve and the allowable

total error calculated from each point of

the graph.

Accuracy

Trueness

Precision

Performance characteristics

Type of errors

Systematic error

Measurement error

Random error

Bias

Uncertainty

Standard deviation

Quantitative expression of performance

characteristics

SIMPLIFY: What do we say

• ”Acceptable limits were established using

clinical criteria”

• ”The limits were based on biological variation”

• ”The limits were established using the state of

the art”

• Impact of analytical performance of test on clinical

classifications or decisions and thereby on probability of

outcomes (simulation or decision analysis).

• To model the clinical outcomes of misclassification requires

clinical evidence about the consequences for patients.

• Where clinical evidence about these consequences is not

available, the model estimates will be based on assumptions

drawn from what evidence there is about disease prognosis,

treatment benefits, harms, etc.

Defining analytical performance specifications using

indirect outcome data (Model 1b)

SimulationStudying the effects of varying analytical performance

Abbott Beckman Olympus Roche Siemens Ortho

1.3

1.2

1.1

1.0

0.9

0.8

0.7

0.6

Langlois MR et al. The EAS-EFLM Collaborative Project. Atherosclerosis 2014; 233:83

1.08

CDC RMPRotterdam

Dutch EQA survey (n=197 labs) of hypertriglyceridemic

serum (TG ~600 mg/dL)

Clinical impact of biased HDLc-risk multipliers,

simulated in men with initial SCORE of 4%

Method Labs (n)

HDL-C median (range) (mg/dL)

Error (mean bias)

SCORE >5%n (%)

Reference 1 42 [HDL multiplier, 1; SCORE = 4%] - -

Overall 197 35 (24-48) -15% 84 (43%)

Abbott 18 41 (38-42) -3% 0

Beckman 39 39 (31-45) -7% 2 (5%)

Roche 113 36 (26-48) -19% 71 (63%)

Siemens 14 31 (24-46) -22% 10 (71%)

Langlois MR et al. The EAS-EFLM Collaborative Project. Atherosclerosis 2014; 233:83

Effect of analytical performance of troponin measurement on

diagnostic misclassification

CV assuming unbiased results % misclassification

36.2% 7.7-15.2

24.6% 3.8-7.7

16.3% 1.8-3.8

13.0% 1.4-1.8

11.2% 1.2-1.4

9.4% 0.9-1.2

6.7% 0.5-0.9

[Sheehan P et al., Ann Clin Biochem 2002;39:213]

Performance specifications for troponin based on clinical needs

defined in terms of allowable misclassification rates

Panteghini M, AACB Troponin Monograph 2012

Healthy

Impaired fasting glucose (IFG)

Diabetes

110 mg/dL 125 mg/dL

A subject with a FPG of 117.5 mg/dL must be differentiate

from healthy condition (from one side)

and a frank diabetes diagnosis (from the other side).

Therefore, error of FPG measurement should be kept <7.5/117.5 = <6.38%,

so that a subject with an IFG cannot be misclassified

as diabetic (FPG >125 mg/dL) or healthy (FPG <110 mg/dL).

Defining allowable measurement error for plasma glucose

using indirect outcome data

117.5 -7.5 +7.5

Model 1b

Pasqualetti S et al., submitted

Impact of measurement error of plasma glucose on clinical classification: a simulation analysis

IFG represents a category at increased risk to develop DM. In this condition, the prevention of DM onset as well as of vascular hyperglycaemia-related complications is accomplished with interventions lowering PG over time. False negatives, i.e., IFG subjects misclassified as normoglycaemic, are therefore the most impacting results.In our outpatient population, measuring PG with an error of -6.38% would imply that 12.6% of individuals miss interventions necessary to stop the progression to DM and the worsening of related outcomes.

Pasqualetti S et al., submitted

• Considering the importance of BV data in laboratory

medicine, it is essential to experimentally derive them in

an accurate and reliable way

• Currently, the most commonly used information on the BV of

laboratory analytes is the SEQC compilation

(www.westgard.com/biodatabase1.htm)

The need to improve it by applying more stringent criteria in the selection and

review of available BV studies has been recognized…

but

This is the aim of the TFG created by EFLM

under the auspices of the Task Force on

Performance Specifications in Laboratory

Medicine

A checklist for critical appraisal of studies of biological variation

Bartlett WA et al., Clin Chem Lab Med 2015;53:879 Biological Variation Working Group

TFG on Biological Variation Database

�Refining and discussing the checklist

�Papers categorized as A, B, C and D depending on their methodological quality, with category A papers indicating high quality and D poor quality.

�The checklist contains 14 items and 22 items will be extracted from each paper and presented in the database.

�Established groups for different measurands

TFG on Biological Variation Database

0 2 4 6 8 10 12 14

02

46

810

12

Hb

A1

c (IFC

C d

efin

ed

me

asu

ran

d)

Type 2 diabetes mellitus

under good control


under acceptable control


under deficient control


all conditions

Healthy M

Healthy F

Healthy M&F

Healthy Roche Integra 800

Healthy Trinity Biotech,

Premier Hb9210

Healthy Tosoh G8

Healthy Trinity Biotech,

Ultra

CVCVII%%

““C C” ”

pa

pe

rsp

ap

ers

“ “B B” ”

pa

pe

rsp

ap

ers

We

stga

rdW

estg

ard

’ ’s da

tab

ase

s da

tab

ase

0

10

20

30

40

50

60

70

80

90C reactive protein

CV

CV

II%%

Ov

erw

eig

ht

wit

ho

ut

PC

OS

Pa

tie

nts

wit

h

sta

ble

an

gin

a

Ov

erw

eig

ht

wit

h P

CO

S

Sta

ble

po

st-M

I

pa

tie

nts

Healthy

WestgardWestgard’’s database (hss database (hs--CRP)CRP)

WestgardWestgard’’s database (CRP)s database (CRP)

Biological Variation Database Structure

The database will consist of:

1) an index page for all analytes [analyte, matrix and grading (A, B, C)].

2) a detailed table for each analyte and matrix:

-1st level: with estimates of CVs (CVI and CVG), analytical performance specifications

for imprecision and bias, grading (with the click you can go to the grading legend)

-2nd level: with all other information and a link to a cloud containing the compiled

papers included in the database.

3) a list of articles scored with D.

First group of analytes expected to be published by 2017.

• Evidence has been collected on the frequency and stratification of errors in laboratory medicine.

• The vulnerability of both the pre-analytical and post-analytical phase has been highlighted as well as the risk for quality and patient safety.

• Consensually defined criteria for setting extra-analytical quality indicators have been developed and data collected.

• This, in turn, should provide the way to define reliable performance specifications in the extra-analytical phases.

After the 1999 Stockholm Conference

Analytical Phase Pre/Post-Analytical Phase

Models for

performance

specifications

Defined Not defined

Possibly based on the State-of-the-Art

and on Outcome measures

Metrics Well defined Proposed

- Percentage

- Parts per million (ppm)

- Six sigma

Tools of measures Well defined

Internal Quality Control

External Quality Assessment

Recently defined

Quality Indicators

PERFORMANCE SPECIFICATIONS

Optimum

Desirable

Minimum

Unacceptable

Three quality levels for each indicator are proposed in order to allow laboratories to

evaluate how they are placed in comparison with other labs and if improvement

actions are needed.

<75th percentile

<50th percentile

<25th percentile

>75th percentile

In order to define

priorities in corrective

and preventive actions

Specimen not received 2.0 - 6.1 2.9

Insufficient specimen 0.07 - 0.80 0.15

Wrong container 0.02 - 0.20 0.11

3.0 Desirable

5.0 Minimum

2.5 Optimum

0.15 Desirable

0.60 Minimum

0.10 Optimum

0.11 Desirable

0.17 Minimum

0.05 Optimum

Range Median Specifications

Analytical performance specifications two years after ... meeting 2016... · Analytical performance...

Documents

Transcript of Analytical performance specifications two years after ... meeting 2016... · Analytical performance...