The Best Method of Noise Filtering

Yuri Kalambet, Sergey Maltsev, Ampersand Ltd., Moscow, Russia;

Yuri Kozmin, Shemyakin Institute of Bioorganic Chemistry,

Moscow, Russia

kalambet@ampersand.ru

1

History: Adaptive peak approximation

2

Rough slope width estimate

• Evaluate baseline using default gap (minimum peak width Integration parameter)

• Evaluate peak height using default gap

• Count all points from peak apex to slope end with height bigger than half-height of the peak. Count obtained is an estimate of the slope width.

3

Properties of adaptive peak approximation

4

• Good noise suppression at each slope• Minimal peak shape disturbances• All peak parameters are resistant to oversampling • Baseline approximation may be poor – either noisy

(small gap) or disturbed (large gap).• No approximation outside of peaks• Does not improve formal signal/noise ratio• Baseline position is one of the most important

sources of error

Improvement 1: Non-central approximation

5

*x

2G

1G

Confidence intervals

6

Hei

ght

Concentration

0 1 2 3 4 5 6 7 8 9 100

2

4

6

8

123

456

789

Confidence interval estimate

7

where

n - number of data points used for polynomial approximation (gap of the filter);

p - power of the polynomial; X - matrix of x power values on independent axis (time);Y - vector of detector response values;

- Student’s coefficient for confidence probability (1-δ) and m degrees of freedom

x* - position at which smoothed (approximated) value is estimated.

*

)2/1( uStC pnY

pnS

)ˆ()ˆ(2 βXYβXY

*1

* xX)X(x *u

YXXXβ 1)(ˆ mt

},...,,1{ **pxx*x

Approximation using confidence intervals

8

*x

2G

*x

2G confidence interval

1 G

Algorithm of simple Confidence filter approximation

9

• Evaluate points and confidence intervals for new (shifted) window

Algorithm of simple Confidence filter approximation

10

• Evaluate points and confidence intervals for new (shifted) window

• Compare new confidence interval with that for previously evaluated point. If the new one is smaller than previous, replace approximated point and its confidence interval.

Algorithm of simple Confidence filter approximation

11

• Evaluate points and confidence intervals for new (shifted) window

• Compare new confidence interval with that for previously evaluated point. If the new one is smaller than previous, replace approximated point and its confidence interval.

• Computational complexity of Confidence filter is comparable to that of simple convolution, (e.g. Savitzky-Golay) and linearly depends on the product gap (degree of the polynomial)∙ .

Bonus #1: Correct handling of baseline steps and array boundaries

12

dotted – raw data; thick line – Confidence Filter; thin line – Savitzky-Golay filter

340 350 360 370 380 390 400 410 420 430 440 450 460 470 480 490 500 Nmeas

0

500

1000

1500

2000

mv

OriginalSGASG

Confidence filter algorithm improvement: Adaptive gap of the polynomial

13

• Repeat confidential filter algorithm for approximations with different windows (gaps)

• Computational complexity: degree gap (gap-1)/2∙ ∙

• Logarithmic step: next gap is k times smaller, than previous, e.g. gap2 = gap1/k, k>1; Computational complexity: degree gap k/(k-∙ ∙1)

Confidence interval estimate

14

where

n - number of data points used for polynomial approximation (gap of the filter);

p - power of the polynomial; X - matrix of x power values on independent axis (time);Y - vector of detector response values;

- Student’s coefficient for confidence probability (1-δ) and m degrees of freedom

x* - position at which smoothed (approximated) value is estimated.

*

)2/1( uStC pnY

pnS

)ˆ()ˆ(2 βXYβXY

*1

* xX)X(x *u

YXXXβ 1)(ˆ mt

},...,,1{ **pxx*x

t(df) for confidence probability 0.975

15

Confidence interval profiles for different slits (degree = 3)

16

-15 -14 -13 -12 -11 -10 -9 -8 -7 -6 -5 -4 -3 -2 -1 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 150.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1

Confidence Interval profiles, 31 points, 0…5 degrees

17

σ evaluation problems:

18

• Small gaps: accidental perfect fit• Large gaps: treating small peaks as a noise due to

large number of degrees of freedom• Is pump pulsation a noise or a signal? • Small gaps: confidence interval depends on

confidence level

σ evaluation solutions:• Evaluate in advance using the whole data array• Use the estimate for evaluation of confidence

intervals

Handling σ estimate

19

*)2/1( uStC pnY

*½ utC pnY

)(,

)(,22

22

bFormulaS

aFormulaSCY

),(222 pnSR

20

Noise Filtering: How it works 1

20 21 22 23 24 25 min

-20

0

20

Shift21

22

23 24 25

26

27

20 21 22 23 24 25 min0

10

20

Gap21

22

23

24

25

26 27

20 21 22 23 24 25 min

-0.005

-0.004

AU

280nm

21

22

23

2526 27

Smoo280

21

Noise Filtering: How it works 2

Automatic selection of degree and gap of

approximating polynomial

22

Raw

0…3

1…3

2...3

3

3…10

33 34 35 36 37 38 39 40 41 min

0.0784 mV

ch1

33 34 35 36 37 38 39 40 41 min

0.0733 mV

ch1

33 34 35 36 37 38 39 40 41 min

0.075 m V

ch1

33 34 35 36 37 38 39 40 41 min

0.0773 mV

ch1

33 34 35 36 37 38 39 40 41 min

0.0784 mV

ch1

33 34 35 36 37 38 39 40 41 m in

0.0784 m V

ch1

Is pump pulsation a noise or a signal?

23

Conclusions:

24

• Confidence filter introduces a measure of approximation quality

• Confidence filter helps to select the best set of functions that approximate the data set

• Confidence filter is metrologically the best noise filtering method and can be used in the fight with legal metrology

Patent pending

Thank you!

25