Walloon Agricultural Research Centre 1 'Uncertainty in multivariate calibration: application to...

Wallo

on A

gri

cult

ura

l R

ese

arc

h C

entr

e

1

'Uncertainty in multivariate calibration:

application to embedded NIR data'

Juan Antonio Fernández Pierna

Scientific collaborator F.N.R.S. Brussels, Belgium-

Statistics and Informatics DepartmentUniv. of Agronomical Sciences of Gembloux (FUSAGx), Belgium

-Quality of Agricultural Products Department

Walloon Agricultural Research Centre (CRA-W), Gembloux, Belgium

IV Winter Symposium on Chemometrics, February 15-18, Moscow (Chernogolovka), Russia

Wallo

on A

gri

cult

ura

l R

ese

arc

h C

entr

e

2

PART I : Uncertainty study Embedded NIR

PART II : Imaging using a NIR camera

Wallo

on A

gri

cult

ura

l R

ese

arc

h C

entr

e

3

J.A. Fernández Pierna, L. Jin, F. Wahl, N. Faber, D.L. Massart Chemometrics and Intelligent Laboratory Systems 65 (2003) 281-291

‘Estimation of partial least squares regression prediction uncertainty when the reference values carry a sizeable measurement error’

PART I based on:

Wallo

on A

gri

cult

ura

l R

ese

arc

h C

entr

e

4

Summary

1- Introduction

3- How to determine the uncertainty

4- Examples

5- Conclusions

2- Uncertainty?

Wallo

on A

gri

cult

ura

l R

ese

arc

h C

entr

e

5

“A result is not complete without an associated measure of uncertainty.”

The uncertainty of a calculated value is statistically defined as the interval around that value such that

any repetition of the calculation will produce a new result that lies within this interval with a given

probability

Wallo

on A

gri

cult

ura

l R

ese

arc

h C

entr

e

6

“So, a result without reliability (uncertainty) statement cannot be published or communicated because it is not (yet) a result. I am appealing to

my colleagues of all analytical journals not to accept papers anymore which do not respect this

simple logic.”

P. De Bièvre, Editorial

“Measurement results without statements of reliability (uncertainty) should not be taken seriously”

Accreditation and Quality Assurance, 2 (1997) 269.

Source: N. Faber, BCS Workshop ‘Uncertainty estimation in multivariate calibration’ Antwerp, November 3, 2004

Wallo

on A

gri

cult

ura

l R

ese

arc

h C

entr

e

7

IUPAC guidelines for single component calibration

K. Danzer and L.A. Curry, “Guidelines for calibration in analytical chemistry. Part 1. Fundamentals and single component calibration”, Pure & Appl. Chem. 70 (1998) 993.

This document shows that the error analysis for univariate calibration is fairly simple.

IUPAC guidelines for multicomponent calibration

K. Danzer, M. Otto and L.A. Curry, “Guidelines for calibration in analytical chemistry. Part 2. Multispecies calibration”, Pure & Applied Chemistry, 76 (2004) 1215.

This document illustrates that the error analysis for multivariate calibration is relatively complex.


Wallo

on A

gri

cult

ura

l R

ese

arc

h C

entr

e

8

X true predictor matrix (NIR spectra)y true predictand vector (property of interest)

measured predictor matrix measured predictand vector

X~

y~

UNOBSERVABLE

OBSERVABLE

Notation

yyy~XXX

~

Unobservable measurement error

Wallo

on A

gri

cult

ura

l R

ese

arc

h C

entr

e

9

x~y~y 'unun

yyy~XXX

~

PLS regression

PLS prediction

UNCERTAINTY

Introduction

True values (unobservable)

Measured values (observable)

Unobservable measurement error

Xy

yXX~

y~ y~X~ˆ

Wallo

on A

gri

cult

ura

l R

ese

arc

h C

entr

e

10

Uncertainty

- PLS/PCR until now theory was scarce and not well-tested about how to estimate the quality of each individual prediction.

- PLS/PCR… RMSEP

Wallo

on A

gri

cult

ura

l R

ese

arc

h C

entr

e

11

Multivariate: empirical validation that implicitly accounts for all error sources

Root mean squared error of prediction (RMSEP) for test set of N samples:

2-1,ref

1

,ref

ˆRMSEP

ˆ prediction for sample

associated reference value

N

n nn

n

n

N y y

y n

y

Wallo

on A

gri

cult

ura

l R

ese

arc

h C

entr

e

12

Obvious problems with test set validation

The result (RMSEP) is a constant measure for prediction uncertainty that cannot lead to prediction intervals with correct coverage probabilities (say 95%).

A crucial assumption is that the reference values are sufficiently precise; this is certainly not always true (octane rating, classical Kjeldahl) - often the prediction is even better than the reference value.

High intrinsic variability of RMSEP estimate requires N to be large.


Wallo

on A

gri

cult

ura

l R

ese

arc

h C

entr

e

13

Some benefits of a sample-specific multivariate prediction uncertainty

Construction of prediction intervals, e.g. for monitoring the performance of an analysis using control samples (see ASTM standard E1655, “Standard practices for infrared, multivariate, quantitative analysis”).

Realistic estimation of limit of detection, since RMSEP - a constant value - poorly describes extreme samples.

Opportunities for sample design and variable selection.


Wallo

on A

gri

cult

ura

l R

ese

arc

h C

entr

e

14

How to determine the sample-specific multivariate prediction uncertainty?

1 - Repeating the experiment under relevant conditions

Not practical, cumbersome

3 - Equations in the literature

2 - Resampling methods (Monte-Carlo)

Noise addition: Boostrapping

MartensDe VriesFaber...

Wallo

on A

gri

cult

ura

l R

ese

arc

h C

entr

e

15

- Generation of data sets by introducing artificial perturbations that emulate the effect of the perturbation of the original data

Monte-Carlo simulation:Boostrapping

Wallo

on A

gri

cult

ura

l R

ese

arc

h C

entr

e

16

Noise addition method

PLSR (F factors)

Uncertainty estimation

N times

)(y )( X Ix1~K~

andIx

noiseyy*n

~

)xKI(~

tunX

ny sprediction ˆI t

s'y tore nˆS

Residuals e=(e1, e2…eI)

PLSR (F factors)

Wallo

on A

gri

cult

ura

l R

ese

arc

h C

entr

e

17

PLSR (F factors)

Uncertainty estimation

B times

)(y )( X Ix1~K~

andIx

*be~ yy*

b

)xKI(~

tunX

by sprediction ˆI t

s'y tore bˆS

Residuals e=(e1, e2…eI)

PLSR (F factors)

*besampleingBootstrapp

Boostrapping

e2*=(e1 e1 e2 e3)

e3*=(e1 e2 e3 e4)

e4*=(e2 e3 e3 e4)

...

Randomly sampling with replacement

e1*=(e2 e1 e2 e4)

Wallo

on A

gri

cult

ura

l R

ese

arc

h C

entr

e

18

The Martens-De Vries equation

- Expression used in the Unscrambler® software package (CAMO)

- S. De Vries, Cajo J.F. Ter Braak, Chemom. Intell. Lab. Syst. 30 (1995) 239-245

- M. Høy, K Steen, H Martens, Chemom. Intell. Lab. Syst. 44 (1998) 123-133

Wallo

on A

gri

cult

ura

l R

ese

arc

h C

entr

e

19

,,

1ˆ2

unxun y val un

xtot val

VU V h

V

Scores without error

Loadings without error

,,

ˆ 1 unxun y val un

xtot val

VFU V h

I V

PLS factors

y-residual variance in a validation set

Average x-residual variance in a validation set

x-residual variance in the prediction object un

number of objects in the calibration set

leverage

1un un cal cal unh t ' (T ' T ) t

The Martens-De Vries equation

cal,jTun,iun qty~y

2qt

2y~

2y cal,j

Tun,iun

ˆˆˆ

Wallo

on A

gri

cult

ura

l R

ese

arc

h C

entr

e

20

Variance of the measurement errors for the concentrations of the calibration set

Variance of the residuals for the calibration

Variance of the residuals for the new sample

Variance of the measurement errors in the spectra of the calibration set

Variance of the measurement errors in the spectra of the new sample

K Faber, B. R. Kowalski, Chemom. Intell. Lab. Syst. 34 (1996) 283-292

The Faber and Kowalski equation

leverage

2X

222X

22y

2un

1un unun

ˆˆˆˆˆhIU

Wallo

on A

gri

cult

ura

l R

ese

arc

h C

entr

e

21

Variations

2X

22y

2I

1i

2ii ˆˆˆy~y

1FI

1ECSM

22un

ˆˆ

0ˆ,ˆ 2X

2X un

R. Boqué, M. S. Larrechi, F. X. Rius, Chemom. Intell. Lab. Sys. 45 (1999) 397-408

2yun

12un

1un ˆhIˆ1hIU

2ˆ2ˆ y << 2un

1un ˆ1hIU

2ˆ y 2ˆ>> 2yun

1un ˆhIU

2yun

12yun

1un ˆhIˆECSM1hIU

X. H. Song, N. M. Faber, P. K. Hopke, D. T. Suess, K. A. Prather, J. J. Schauer, G. R. Cass, Anal. Chim. Acta 446 (2001) 329-343.

2yun

1un ˆECSM1hIU

Wallo

on A

gri

cult

ura

l R

ese

arc

h C

entr

e

22

Independent error estimates are not required: all ingredients are estimated directly from the data.

Valid if measurement errors can be neglected: assumed that all variables are observable

Can only be used if independent estimates for the measurement errors are available.

Valid for all situations

‘ Faber ’ ‘ De Vries ’

Comparison

True value=unobservable valueMeasured value=observable value

Xy

yyy~XXX

~

yXX~

y~

True value=observable value

Xy

0y,X

Wallo

on A

gri

cult

ura

l R

ese

arc

h C

entr

e

23

Embedded NIR

Wallo

on A

gri

cult

ura

l R

ese

arc

h C

entr

e

24

To have on the field a direct determination of the dry material of the forage.

For breeders having the dry material of the forage at the field is really important.

Wallo

on A

gri

cult

ura

l R

ese

arc

h C

entr

e

25

Diode ArrayZeiss instrument

Embedded NIR instrument

Wallo

on A

gri

cult

ura

l R

ese

arc

h C

entr

e

26

1

2 3 3

Data treatment

sabs

InGaAs detector128 diodes

GratingSource ZEISS CORONA 45

950-1700 nm

Embedded NIR instrument

Source: G. Sinnaeve, 2nd International Conference on ‘Embedded NIR spectroscopy’ Gembloux, November 18-19, 2004

Wallo

on A

gri

cult

ura

l R

ese

arc

h C

entr

e

27

Identification Weighting Fresh sample

Coarse grinding

Dry sample

NIR predictions

Oven70 °C , 48 h

Weighting

Hammer Mill

1st grindingCyclotec Mill

Fine grinding

Constructing the models

Wallo

on A

gri

cult

ura

l R

ese

arc

h C

entr

e

28

Dry Matter : PLS calibration

Calibration set Test set

PLS (8)

Wallo

on A

gri

cult

ura

l R

ese

arc

h C

entr

e

29

R2 = 0.893

6000

7000

8000

9000

10000

11000

12000

6000 7000 8000 9000 10000 11000 12000

Yield NIR

Yie

ld L

ab

.Yield (kg DM/ha)

-Comparison of the yield expressed in kg DM /ha using the oven and the embedded NIR methods

Wallo

on A

gri

cult

ura

l R

ese

arc

h C

entr

e

30

5000

6000

7000

8000

9000

10000

11000

0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23

Ray Grass cultivars

Yie

ld (

kg

DM

/ha

) .

5000

6000

7000

8000

9000

10000

11000

DM Embedded NIRDM Oven

-Comparison of the classification or the ranking of the cultivars according to their yield expressed in kg DM /ha

using the oven and the embedded NIR methods

Wallo

on A

gri

cult

ura

l R

ese

arc

h C

entr

e

31

Outlier in prediction?

Embedded data

95 x 366Xtest

600 x 366Xcal

Protein contentProperty (y)

Noise addition

Wallo

on A

gri

cult

ura

l R

ese

arc

h C

entr

e

32

Embedded data

2ˆ y 2ˆ<<

2un

1un ˆ1hIU

0.2321

0.0039

2ˆ

2yˆ

Faber and Kowalski equation

95 x 366Xtest

600 x 366Xcal

Protein contentProperty (y)

Wallo

on A

gri

cult

ura

l R

ese

arc

h C

entr

e

33

Gas oil data

0 5 10 15 20 25 30 35 400

0.01

0.02

0.03

0.04

0.05

0.06

Prediction sample

Sta

nda

rd e

rro

r o

f pre

dic

tion

: Formulao : Monte-Carlo

2ˆ y 2ˆ>>

Property (y) % Hydrogen

Split Duplex method

Xcal 199 x 2128

Xtest 40 x 2128

1.01e-4

0.0021

2ˆ

2yˆ

2yun

1un ˆhIU


Wallo

on A

gri

cult

ura

l R

ese

arc

h C

entr

e

34

Polyether Polyol data

0 5 10 15 20 251

1.5

2

2.5

Prediction sample

Sta

nda

rd e

rro

r o

f pre

dic

tion

: Formulao : Monte-Carlo

2ˆ y 2ˆ,

Property (y) -OH

Split Duplex method

Xcal 60 x 495

Xtest 24 x 495

1.59

0.49

2ˆ

2yˆ

2yun

1un ˆECSM1hIU


Wallo

on A

gri

cult

ura

l R

ese

arc

h C

entr

e

35

Formula versus resampling

Formula:

+ insight in dominant sources of error

+ evaluation is (usually) fast

- often difficult to obtain

- quite restrictive in their application, because of distributional assumptions

Wallo

on A

gri

cult

ura

l R

ese

arc

h C

entr

e

36

Formula versus resampling

Resampling:

+ easy to implement

+ not (very) restrictive in their application

- little insight (black box)

- evaluation is (relatively) slow

- not always clear how to resample

Wallo

on A

gri

cult

ura

l R

ese

arc

h C

entr

e

37

- Monte-Carlo methods allows working on residuals and directly operate on the noise

- Samples with large uncertainty possible prediction outliers in the prediction data set.

- Monte-Carlo methods are easy to implement

Conclusions PART I, 1

Wallo

on A

gri

cult

ura

l R

ese

arc

h C

entr

e

38

- The uncertainty obtained (Faber equation) is with respect to the true values and is sample-specific.

- The De Vries formula only works under the classical regression assumption that all variables are observable with negligible measurement noise.

- Leverage-based formulas have been recently proposed (successfully) for non-linear variations of PLSR, multiway PLSR and PLSR after OSC

Conclusions PART I, 2

- Good estimations of the uncertainty are obtained using the Faber equation working under all conditions of and .2ˆ 2

yˆ

- In the future all the techniques should be adapted for the estimation of the prediction uncertainty (ANN, SVM…)

Wallo

on A

gri

cult

ura

l R

ese

arc

h C

entr

e

39

PART II : Imaging using a NIR camera

Wallo

on A

gri

cult

ura

l R

ese

arc

h C

entr

e

40

NIR camera

- Camera InGaAs - 900 – 1700 nm / 10 nm - 240 x 320 pixels- Pixel size: 80 µm * 80 µm- Surface analysed : ±5 cm²- 76 800 spectra = 24 MB- 300 - 350 separated particles - Time of analysis : ± 5 minutes Spectral volume

WavelengthsPixels

Pix

els

50 100 150 200 250 300

50

100

150

200

Spectrum of a mineral particle

Spectrum of a bone particle

Wallo

on A

gri

cult

ura

l R

ese

arc

h C

entr

e

41

0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

1110 1210 1310 1410 1510 1610 1710 1810 1910 2010 2110 2210 2310 2410

Longueurs d'onde (nm)

Maize

Ab

so

rba

nc

e

0.1

0.2

0.3

0.4

0.5

0.6

0.7

1110 1210 1310 1410 1510 1610 1710 1810 1910 2010 2110 2210 2310 2410


Abso

rban

ce

Lin

0.2

0.3

0.4

0.5

0.6

0.7

0.8

1110 1210 1310 1410 1510 1610 1710 1810 1910 2010 2110 2210 2310 2410


Abso

rban

ce

Soya

Wallo

on A

gri

cult

ura

l R

ese

arc

h C

entr

e

42

title author range pixels

Analysis of meatDetection of meat and bone meal and fishmeal in compound feed Fernández et al. 900-1700nm 240x320Chicken heart disease characterization / chicken skin tumors Chao et al. 480-965nm 240x320Detection of the vegetal source of feed ingredientsScreening of compound feeds Fernández et al. 900-1700nm 240x320Analysis of fruits / vegetablesTomato sorting / ripeness of tomatoes Polder et al. 393-710nm 512x512Identification of major chemical components in fruits Martinsen et al. 650-1100nm 150x242Assessment of fruit quality Lu et al. 500-1000nm 320x240Analysis of fruit degradation Peirs et al. 900-1700nm 320x240Discrimination of different cereal components Robert et al.Determination of water content in leaves Tran et al. 1000-1700nm 7x7Visualization of sugar content in the flesh of a melon Sugiyama et al. 400-1100nm 768x512Single kernel analysisSingle kernel maize analysis Cogdill et al. 750-1090nm 512x512Automated sorting and single kernel analysis Stevermer et al. 700-1100nm 512x512Detection of pest insects and other contaminants in cereal grain Ridgway et al. 700-1100nm 256x256Food quality and safetySystem for food quality and safety Kim et al. 428-930nm 512x512Measuring the distribution of chemical components Taylor et al. 400-1100nm 240x320Visualising chemical composition and reaction kinetics Tran et al. 1000-1700nm 240x320Precision agriculture Yao et al.Characterisation of grassland canopy Buffet et al. 400-950nmDetermination of vegetation indices Wessman et al.Remote control and monitoring in agriculture Vane et al. 400-2500nmMineral exploration Stevens et al.Mapping habitat Earth Search Sciences Inc.Eco System Monitoring Abileah et al.Invasive vegetation Earth Search Sciences Inc.Hazardous waste remediation Swayze et al.Monitoring coastal environments Dunk et al. 435-2480nmDiscriminating and mapping soilsMapping variability in vineyards

Wallo

on A

gri

cult

ura

l R

ese

arc

h C

entr

e

43

PC 3i

iiiii

Analysis of raspberries by NIR imaging showing a grading in the maturity

(Berries i = low maturity; ii = medium maturity; iii = riped).

Source: Walloon Agricultural Research Centre, 2004 - 2005

Wallo

on A

gri

cult

ura

l R

ese

arc

h C

entr

e

44

1340 nm 1410 nm

NIR images at 1340 nm NIR images at 1410 nm

Analysis of white currants by NIR imaging.

Source: Walloon Agricultural Research Centre, Gembloux, Belgium (2004 – 2005)

Wallo

on A

gri

cult

ura

l R

ese

arc

h C

entr

e

45

1400 nm

PC 5

A)

B)

1

2

3

fifth PC image of the intact (1) and infested (2 & 3) coffee beans

Analysis of single kernels by NIR imaging to detect insect infested grains

image at 1400 nm of three infested wheat kernels


Wallo

on A

gri

cult

ura

l R

ese

arc

h C

entr

e

46

PC 61140 nm

NIR image at 1140 nm, spectra of the germ (dotted line) and albumen (continuous line), as well as sixth PC image bringing to the fore the germ of each kernel.

Analysis of wheat grains by NIR imaging


Wallo

on A

gri

cult

ura

l R

ese

arc

h C

entr

e

47

‘Estimation of partial least squares regression prediction uncertainty when the reference values carry a sizeable measurement error’ J.A. Fernández Pierna, L. Jin, F. Wahl, N. Faber, D.L. Massart Chemometrics and Intelligent Laboratory Systems 65 (2003) 281-291

Dr. N. Faber, http://www.chemometry.com/Dr. V. Baeten, CRA-WDr. G. Sinnaeve, CRA-WDr. P. Dardenne, CRA-WProf. J.J. Claustriaux, FUSAGxF.N.R.S. for financial support

References

Acknowledgements

‘Combination of Support Vector Machines (SVM) and Near Infrared (NIR) imaging spectroscopy for the detection of meat and bone meat (MBM) in compound feeds’ J.A. Fernández Pierna, V. Baeten, A. Michotte Renier, R.P. Cogdill and P. Dardenne. Journal of Chemometrics 18 (2005)

Imaging – NIR camera

Uncertainty

Walloon Agricultural Research Centre 1 'Uncertainty in multivariate calibration: application to...

Documents

Transcript of Walloon Agricultural Research Centre 1 'Uncertainty in multivariate calibration: application to...