Measurement, accuracy,

precision, confusion

and validation

Ángel Alberich-Bayarri, PhD1 Biomedical Imaging Research Group GIBI230 La Fe Health Research Institute

2 QUIBIM – Quantitative Imaging Biomarkers in Medicine

angel@quibim.com

Disclosure

I am Founder & CEO of the spin-off

company QUIBIM which provides a

service of Imaging Biomarkers

• To learn metrology basics applied to radiology and

imaging biomarkers analysis

• To understand the differences of accuracy and

precision

• To know how to control potential measurement bias

and validation processes

Learning objectives

• Accuracy

• Precision

• Technical validation – Metrology

– Protocols

– Phantom calibration

• Clinical validation– Biases

– Reference technique calibration

– Relationship with Clinical Endpoints

Outline

Closeness of a measurement to the

true value

Accuracy

• JCGM 200:2008 International vocabulary of metrology — Basic and general

concepts and associated terms (VIM)

• BS ISO 5725-1: "Accuracy (trueness and precision) of measurement methods

and results - Part 1: General principles and definitions.", p.1 (1994)

Closeness of agreement among a

set of results

Precision

• JCGM 200:2008 International vocabulary of metrology — Basic and general

concepts and associated terms (VIM)

• BS ISO 5725-1: "Accuracy (trueness and precision) of measurement methods

and results - Part 1: General principles and definitions.", p.1 (1994)

Accuracy and precision

• JCGM 200:2008 International vocabulary of metrology — Basic and general

concepts and associated terms (VIM)

• BS ISO 5725-1: "Accuracy (trueness and precision) of measurement methods

and results - Part 1: General principles and definitions.", p.1 (1994)

High accuracy, Low precision High precision, Low accuracy

Validation

• Imaging Biomarkers must be both technically and

clinically valid

Validation

• Imaging Biomarkers are considered as a Class IIa

Medical Device in Europe, therefore, a CE mark must be

obtained in order to be considered for clinical use.

FDA 510kFDA 510k

TECHNICAL VALIDATION

OF IMAGING BIOMARKERS

Technical validation

• For the technical validation, we need to incorporate

new fields to our network…

METROLOGY

Technical validation:

Metrology

• Defined by the International Bureau of Weights and

Measures (BIPM) as "the science of measurement,

embracing both experimental and theoretical

determinations at any level of uncertainty in any field

of science and technology."

• 3 different fields:– Units of measurement

– Realization of units of measurement in practice

– Traceability to link measurements to reference standards

Sullivan DC, Obuchowski NA, Kessler LG, et al. Metrology

Standards for Quantitative Imaging Biomarkers.Radiology.

2015 Aug 12. Epub ahead of print. doi:

10.1148/radiol.2015142202.

Technical validation:

Metrology

• Technical validations should be addressed

in Facilities for Experimental Radiology

and Imaging Biomarkers

• The experiments allow to add insight into

the variability and utility of the candidates

to Imaging Biomarkers

• Technical validation against phantoms and

test cases with known conditions is

recommended.

Technical validation:

Metrology

• International initiatives like QIBA or EIBALL are specifying

protocols (from acquisition to postprocessing) resulting

from the committee’s periodically organized.

• Group of potential uncertainties:– Patient preparation details

– Pulse sequences (MR)

– Angulation

– Contrast administration

– Platform for Postprocessing

– Operative system1 of the software for postprocessing

Technical validation

Protocols

1Gronenschild et al. The Effects of FreeSurfer Version, Workstation Type, and Macintosh

Operating System Version on Anatomical Volume and Cortical Thickness Measurements.

• International initiatives like QIBA or EIBALL are specifying

full pipeline protocols (acquisition to processing) resulting

from the committee’s

Technical validation

Protocols

QIBA Profile: DCE-MRI Quantification v1.0 (Publicly Reviewed Version)

•(Citation reference)DCE-MRI Technical Committee. DCE-MRI Quantification Profile, Quantitative

Imaging Biomarkers Alliance. Version 1.0. Publicly Reviewed Version. QIBA, July 1, 2012.

QIBA Profile: CT Tumor Volume Change v2.2 (Publicly Reviewed Version)

•(Citation reference)CT Volumetry Technical Committee. CT Tumor Volume Change Profile,

Quantitative Imaging Biomarkers Alliance. Version 2.2. Publicly Reviewed Version. QIBA, August

8, 2012.

QIBA Profile: FDG-PET/CT as an Imaging Biomarker Measuring Response to Cancer Therapy

(Publicly Reviewed Version)

•(Citation reference)FDG-PET/CT Technical Committee. FDG-PET/CT as an Imaging Biomarker

Measuring Response to Cancer Therapy Profile, Quantitative Imaging Biomarkers Alliance.

Version 1.05. Publicly Reviewed Version. QIBA, December 11, 2013.

UPICT Protocol: FDG-PET/CT UPICT (Publicly Reviewed Version)

•(Citation reference) FDG-PET/CT Technical Committee. FDG-PET/CT UPICT, Quantitative

Imaging Biomarkers Alliance. Version 1.0. Publicly Reviewed Version. QIBA, July 08, 2014.

• International initiatives like QIBA or EIBALL are specifying

full pipeline protocols (acquisition to processing) resulting

from the committee’s

Technical validation

Protocols

• Example 1: Phantom for substances. Recommended to

have a ‘refill’ phantom

Technical validation

Phantoms

• Applications:– Diffusion

– Iron

– Fat

– T1 mapping

– T2 mapping

– Image Quality assurance

• Example 2: Phantom for liver components

Technical validation

Phantoms

• Application to callibration

– Fat

– Iron

– Collagen

– Water

Technical validation

• The imaging biomarker is measured with robustness and

quality but…

Covers an unmet clinical need?

CLINICAL VALIDATION

OF IMAGING BIOMARKERS

• The biases can be divided in 2: those related to patient

characteristics and those related to changes induced by

the disease.

• Patient-related biases: sex, age, race, laterality, …

• Pathology-specific biases: High iron overload biasing

accurate fat quantification.

• In the proof of principle, the most relevant characteristics

and potential bias of the population have to be taken into

account

Clinical validation

Biases

Clinical validation

Reference technique calibration

• The appropriate standard of reference must be

selected in order to validate the results of the imaging

biomarker.

• Example: liver iron concentration

[LIC]umol/g =6.21 + (0.15 x R2*)

Clinical validation

Reference technique calibration

• Nevertheless, sometimes the reference technique can

have a similar o even lower accuracy and precision

that the biomarker that we are proposing (25%

disagreement between pathologists1).

• Clinical endpoints, taking into account the evolution of

the patient is preferred as the reference standard, in

order to derive prognostic imaging biomarkers.

1Elmore JG, et al. Diagnostic concordance among pathologists

interpreting breast biopsy specimens. JAMA. 2015 Mar 17;313(11):1122-

32. doi: 10.1001/jama.2015.1405

Clinical validation

• Besides being technically validated, at this step, if there

is a relationship with disease that adds value to clinical

workflow, the biomarker is clinically validated an ready

for regulatory application.

Proposal for validation

Precision

•1. Image Acquisition: different centers, different vendors, protocol parameters, patient preparation.

•2. Methodology: algorithm, influence of human interaction (if any)

Accuracy

•1. Phantom: relative error to synthesized ground truth

•2. Pathology (biopsy): relative error to current gold standard

Clinical Endpoints

•1. Short term (diagnostic & therapeutic values)

•2. Long term (prognostic value)

Performance

IndicatorAssays Statistics

CoV (%)Coefficient of Var

iation

ε (%)Relative Error

CorrelationPearson, Spearman

Validated Imaging Biomarker✅

< 15% (20% LLOQ)

< 15% (20% LLOQ)

Acceptable*

*European Medicines Agency. Guideline on bioanalytical method validation. 21 July 2011

p<0.05

2/2

0/2

1/2

3/6

Minimum

Conclusions

• Imaging Biomarkers development workflow require both

Technical and Clinical validations to produce a paradigm shift

in the management of the disease.

• Technical validation is closely related to the Metrology

science.

• Clinical validation through Clinical Endpoints is preferred over

pathology.

• Integration in radiological workflow is crucial for promoting the

use of the biomarker, after technically and clinically validated.

Measurement, accuracy,

precision, confusion

and validation

Ángel Alberich-Bayarri, PhD1 Biomedical Imaging Research Group GIBI230 La Fe Health Research Institute

2 QUIBIM – Quantitative Imaging Biomarkers in Medicine

angel@quibim.com