Slide 1

Machine Learning for Adaptive Bilateral Filtering I. Frosio 1, K. Egiazarian 1,2, and K. Pulli 1,3 1 NVIDIA, USA 2 Tampere University of Technology, Finland 3 Light, USA

Slide 2

Motivation (denoising) void denoise(float *img){ for (int y = 0; y < ys; y++){ for (int x = 0; x < xs; x++){ img(y*xs+x) = } }

Slide 3

Motivation: (Gaussian filter) t(x)d(x)

Slide 4

Bilateral filter Motivation (bilateral filter) C. Tomasi and R. Manduchi, Bilateral filtering for gray and color images, ICCV, 1998. t(x)d(x)

Slide 5

Motivation: (choice of the parameters) dd dd rr d(x)

Slide 6

Motivation (use intuition?) d scales with resolution r scaled with grey level dynamics possible automatic design of parameter values [BF, Tomasi and Manduchi, ICCV, 1998] image noise std n d = [1.5, 2.1], independently from n r = k n [BF, Zhang, Gunturk, TIP, 2008] local signal variance s 2 (x,y) d = [1.5, 2.1], independently from n r (x,y) = n 2 / s (x,y) [ABF, Qi et al., AMR, 2013] dd dd rr

Slide 7

Motivation (use machine learning!) dd rr

Slide 8

Framework (adaptive denoising) 3 features 6 unknowns d (x,y) r (x,y)

Slide 9

Framework (learning) Training images {t j } j=1..N Noise model (AWGN) Local image features, f x,y Image quality model (PSNR) Adaptive bilateral filter,

Slide 10

Entropy features FlatGradientTextureEdges 0.0 bit6.0 bits1.0 bit5.6 bits Shannons entropy i(x,y)

Slide 11

Entropy features FlatGradientTextureEdges eiei 0.0 bit6.0 bits1.0 bit5.6 bits egeg 0.0 bit 1.2 bit5.5 bits i(x,y) ||grad[i(x,y)]||

Slide 12

Entropy features eiei egeg

Slide 13

Framework (complete) Training images {t j } j=1..N Noise model (AWGN) Local image features f x,y Image quality model (PSNR) Adaptive bilateral filter, d (x,y) Logistic functions Local image features f x,y r (x,y) Noisy image EABF Filtered image

Slide 14

Results - PSNR BFBF [Zhang]ABF [Qi]EABF d (x,y) optimal1.8 our r (x,y) optimal 2n2n n 2 /(0.3 s ) our n = 5 36.1336.063636.27 n = 10 31.4531.431.4431.1 n = 20 27.1127.0927.3626.4 n = 30 25.072525.3224.68 n = 40 23.9423.6924.1123.76 n = 5 36.2936.0435.9236.4 n = 10 32.5332.1732.0332.81 n = 20 28.9628.4828.7529.51 n = 30 26.9626.3126.9827.62 n = 40 25.6324.7225.6826.17 n = 5 36.536.0735.9336.54 n = 10 32.6432.2332.1532.78 n = 20 29.3228.8129.1729.65 n = 30 27.7126.8427.6828.01 n = 40 26.6625.3326.5326.82 BFBF [Zhang]ABF [Qi]EABF d (x,y) optimal1.8 our r (x,y) optimal 2n2n n 2 /(0.3 s ) our n = 5 37.6937.537.237.81 n = 10 3433.7633.6634.37 n = 20 30.3129.6430.1731.11 n = 30 28.227.1228.229.24 n = 40 26.8625.4626.8327.78 n = 5 38.1737.8637.6438.45 n = 10 34.6434.0934.1835.17 n = 20 31.230.0231.0532.08 n = 30 29.3727.7829.2430.21 n = 40 28.225.9727.8328.77 n = 5 37.8137.7437.3737.86 n = 10 34.7534.3134.2234.98 n = 20 31.2730.431.2231.8 n = 30 29.1427.8529.329.75 n = 40 27.7925.9527.7128.3

Slide 15

Results - PSNR BF BF [Zhang] ABF [Qi] EABF d (x,y) optimal1.8 our r (x,y) optimal 2n2n n 2 /(0.3 s ) our n = 5 37.136.8836.6837.22 n = 10 33.3332.9932.9533.53 n = 20 29.6929.0729.6230.09 n = 30 27.7426.8227.7928.25 n = 40 26.5125.1926.4426.93 average n = 5 40 30.8830.1930.731.21 +1.01dB +0.51dB

Slide 16

Results Image quality Ground truth Noisy 20.11 dB BF [Zhang] 30.02 dB ABF [Qi] 31.05 dB EABF 32.08 dB

Slide 17

Machine learning vs. intuition: d (x,y), r (x,y) n = 20 BF [Zhang et al.]ABF [Qi et al.]EABF d (x,y) 1.8 r (x,y) 2s n [0.6, 2.6] [71, 85] [20, 110]

Slide 18

Machine learning vs. intuition: d = d ( n ) Flat areaEdge area

Slide 19

Machine learning vs. intuition: r = r ( n ) Flat areaEdge area

Slide 20

Conclusion Learning optimal parameter modulation strategies through Machine Learning is feasible. Modulation strategies are complicated But effective. PSNR

Slide 21

Conclusion Training images {t j } j=1..N Noise model (AWGN) Local image features, f x,y Image quality model (PSNR) Adaptive bilateral filter,

Slide 22

Conclusion Your training images Noise model (AWGN) Local image features, f x,y Image quality model (PSNR) Adaptive bilateral filter,

Slide 23

Conclusion Your training images Your noise model Local image features, f x,y Image quality model (PSNR) Adaptive bilateral filter,

Slide 24

Conclusion Your training images Your noise model Your local features, f x,y Image quality model (PSNR) Adaptive bilateral filter,

Slide 25

Conclusion Your training images Your noise model Your local features, f x,y Your image quality model, Q Adaptive bilateral filter,

Slide 26

Conclusion Your training images Your noise model Your local features, f x,y Your image quality model, Q A different adaptive filter,

Slide 27

Conclusion A general FRAMEWORK based on MACHINE LEARNING for the development of ADAPTIVE FILTERS