Profile based Fast Noise Estimation and High ISO Noise Reduction for Digital Cameras

Youngjin Yoo, HoCheon Wey, SeongDeok Lee and Chang-Yong Kim

Computing and Intelligence Laboratory, Samsung Advanced Institute of Technology (SAIT), San 14-1 Nongseo-Dong, Giheung-Gu, Yongin-si, Gyeonggi-do, Republic of Korea 449-712

ABSTRACT

In this paper, we present a new noise estimation and reduction scheme to restore images degraded by image sensor noise. Since the characteristic of noise deviates according to camera response function (CRF) and the sensitivity of image sensors, we build a noise profile by using test charts for accurate noise estimation. By using the noise profile, we develop simple and fast noise estimation scheme which can be appropriately used for digital cameras. Our noise removal method utilizes the result of the noise estimation and applies several adaptive nonlinear filters to give the best image quality against high ISO noise. Experimental results show that the proposed method yields significantly good performance for images corrupted by both synthetic sensor noise and real sensor noise.

Keywords: Nonlinear filtering, noise estimation, noise reduction, image sensor, high ISO

1. INTRODUCTION In digital photography, ISO measures the sensitivity of the image sensor. Higher ISO settings are generally used to get a faster shutter speed that enables us to take sports event when we want to freeze the action or indoor/outdoor shots in lower light for example. In low light situations, the sensor’s ISO can be increased by amplifying the image sensor’s signal (increasing its gain). However, unfortunately, the cost is that we will get more noisy shots. One of most important requirements for reducing high ISO noise effectively is finding accurate noise level. In general, image denoising algorithms assume white noise model. Shin et al. proposed white noise estimation algorithm1. Unfortunately, the image sensor noise characteristics and the nonlinear camera response function (CRF) cause camera noise model to become more complex. One of possible solution for estimating high ISO noise is piecewise white noise model approximation. However, over-estimation problem of noise level arises in this case as shown in figure 1.

0 50 100 150 200 2500

5

10

15

20

25

30

35

40Block filtering with uniform interval based noise estimation

Intensity

Noi

se L

evel

ground truthestimated noise level

Figure 1. Noise level over-estimation problem.

Digital Photography IV, edited by Jeffrey M. DiCarlo, Brian G. Rodricks,Proc. of SPIE-IS&T Electronic Imaging, SPIE Vol. 6817, 68170B

© 2008 SPIE-IS&T · 0277-786X/08/$18

SPIE-IS&T/ Vol. 6817 68170B-1

Recently, Liu et al. proposed a noise estimation method from a single image2. At first, they learn the space of the noise level function. Once the space is defined, Bayesian MAP estimation is used to infer the noise level with respect to image intensity from a single image. In this paper, we introduce fast noise estimation method for digital cameras. In digital cameras, processing time and accuracy of noise estimation are important issues. Our observation is that the noise level space is limited for typical cameras. Another observation is that noise level function mainly changes in lower intensity region for most typical cameras. These observations enable us to estimate noise level quickly and accurately by calculating the statistics of dark region noisy patch from noisy image.

A large number of methods have been proposed for image denoising and regularization. Wavelet-based methods have been very popular for recent years3, 4, 5, 6. Iteration based methods have been proposed, such as variations of total variation7 and anisotropic diffusion8. Recently image patch based denoising method attracts many researchers. Contributions from new image patch based techniques, such as principal component analysis (PCA) based method9, total least square image (TLS) approximation model10 and non-local denoising method11, helped spawn many new statistical models of image. However, we do not draw into comparison with the mentioned denoising methods since we are interested in practical solutions for digital cameras.

In this work, we mainly focus on spatial denoising filters such as Wiener filter or minimum mean square estimator (MMSE)12, Sigma filter13 and Bilateral filter14, because they are computationally less expensive for digital cameras. To our knowledge and based on our experiments, the most challenging issues for reducing high ISO noise using spatial denoising filters are edge region artifact and outliers as shown in figure 2. To overcome these problems, we develop simple local activity estimation method based on signal to noise ratio thresholding and morphological operations. Our noise removal method utilizes the result of local activity estimation and applies adaptive nonlinear filters to give good image quality against high ISO noise.

This paper is organized as follows. In Section 2, we present our observations on the noise level space for typical digital cameras and introduce our noise estimation algorithm. Section 3 describes the proposed high ISO noise reduction algorithm. In Section 4, we demonstrate the performance of our proposed method on images corrupted by both synthetic sensor noise and real sensor noise. Conclusion is given in Section 5.

Figure 2. Examples of artifacts of high ISO noise reduction: (a) edge region artifact; (b) outliers.

2. PROPOSED NOISE ESTIMATION ALGORITHM Typically, the standard deviation is used as noise level measurement. However, the standard deviation is expensive measure for digital cameras because it requires additional memory and square root operation. In this work, we use simpler noise level measure. Let Ωx be the set of pixel values in image patch centered at position x. For each pixel m∈Ωx our noise level measure can be represented by

SPIE-IS&T/ Vol. 6817 68170B-2

Intensity

(a)

-

—

IntenSty

0)

0

∑−

Ω∈+−

−=

1

111 N

IIN

xmmmxζ (1)

where N is the number of samples and I is the intensity value.

Typical output signal format of digital camera processor is YCbCr. Figure 3 depicts luminance noise level space of the High end level DSC with 10M CCD (a) and camera phone with 1.3M CMOS (b). The space is based on the noise level measure (1) and calculated by a variety of ISO, illuminations and test charts such as Macbeth color chart and gray scale chart.

Figure 3. Luminance noise level space: (a) High end level DSC with 10M CCD; (b) Mobile camera phone with 1.3M CMOS.

As shown in figure 3, the noise level spaces of the cameras are limited and the significant change of noise level function occurs in low intensity range. Thus, it is possible to find noise level function from input noisy image by only calculating statistics within lower intensity range.

We build noise profile from the calculated noise level space. Noise profile consists of several noise level functions. Then, intensity search range is defined based on the observation from the noise level space. It is simple observation but it enables us to reduce the estimation process time and complexity. The noise profile can be stored using look-up table (LUT). The operation so far is performed as an off-line process.

Once noisy image is given, textureless patch is found within the defined search range by using the noise level measure (1). The example of textureless patch is shown in figure 4.

SPIE-IS&T/ Vol. 6817 68170B-3

Figure 4. Example of textureless patch.

From the found patch, we obtain the statistics that are noise level ζ and mean intensity. Our assumption is that noise level function can be modeled as a linearly interpolated combination of noise profile components. Once the statistics are given, the closest two noise level function is found in the noise profile. Noise level function of noisy image is reconstructed by weighted interpolation between the obtained two noise level function. The example of weighted interpolation and search range is depicted in figure 5. As shown in figure 5, the point X in 2-D LUT of noise profile can be found from the statistics of the patch. The ratio for weighed interpolation between the closest two noise level function is calculated from the point X and then the noise level function of the noisy image is reconstructed. The dotted blue line is the reconstructed noise level function in figure 5.

0 50 100 150 200 2500

5

10

15

20

25

30

Intensity

Noi

se L

evel

(std

.)

Approximated 1.3M Micron CIS board Prior Noise Level Profile - Y

data 1data 2data 3data 4data 5data 6data 7

Defined search range Figure 5. Noise level reconstruction by using weighted interpolation between the closest two noise level function.

SPIE-IS&T/ Vol. 6817 68170B-4

3. PROPOSED NOISE REDUCTION ALGORITHM Our noise reduction method is based on the estimated noise level described in Section 2. The estimated noise level automatically determines parameters of our noise reduction method. As mentioned in Section 1, we attempt to reduce high ISO noise while minimizing the edge region artifact and outliers in this work. Our proposed method is depicted in figure 6.

Noise level adaptive nonlinear Gaussian filter

with Small Kernel

Median filter

Local Activity Estimation based

Blending

Impulse Detection based Blending

Noisy image

Restored image

Noise level adaptive nonlinear Gaussian filter

with Large Kernel

Figure 6. Our high ISO noise reduction scheme.

In general, noise level is high in images taken by high ISO setting. Large filter kernel is necessary to reduce high level noise because larger number of pixels of filter kernel gives more accurate estimate. However, large filter kernel creates artifacts around edges as shown in figure 2 (a). The bigger filter kernel size, the broader the distorted region. It was found that adaptive window size method is effective for this problem15. In this work, we apply noise level adaptive nonlinear Gaussian filter, which is a continuous version of weight of the Sigma filter, with both small and large filter kernel sizes to preserve details and naturalness. The filter formulation can be written as

∑∑

Ω∈

Ω∈

−−

⋅−−=

x

x

m xmx

mm xmxx )2/)(exp(

)2/)(exp(ˆ

22

22

σ

σ

II

IIII (2)

where σx is the estimated noise level. Based on the noise estimation result of Section 2, the amount of decay of the exponential term spatially varies. The exponential operation and division operation can be implemented in camera by using linear approximation and look-up table.

Local activity is estimated to blend the outputs of these two filters. First, based on noise level estimated as in Section 2, the ratio between local noise level and local signal activity is calculated by using the measure (1). We then threshold the ratio to obtain binary data. When calculating local signal activity, it is important to use enough pixels for avoiding estimation error due to noise. However, using large sample pixels causes over-estimation nearby sharp edges. Thus, over-estimation correction is performed on the binary data by utilizing morphological thinning operation. The corrected binary data is then blurred to reduce boundary artifacts. Example of local activity estimation is provided in figure 7. The local activity estimate is normalized to 0 to 1. The noise level adaptive nonlinear Gaussian filter is then estimated as

SLNG fLfLI ,,, )1(ˆxxxxx −+= (3)

where L represents the local activity estimate, and fL and fS represent nonlinear Gaussian filter with large kernel and small kernel respectively.

SPIE-IS&T/ Vol. 6817 68170B-5

(c) (d)

Order statistics have played an important role in statistical data analysis and especially in the robust analysis of data contaminated with outlying observations, called outliers16. In low light and high ISO condition, outliers are likely to appear. The median filter is the most popular example of filters based on order statistics due to its computational simplicity and robust estimation performance for outliers.

Figure 7. (a) Noisy Lena image; (b) local activity binary data; (c) over-estimation correction; (d) reduction of boundary artifact by blurring.

We reduce an outlier from nonlinear Gaussian filter output by blending with the median estimator. The blending amount is determined by fuzzy membership function of impulse detection. The fuzzy membership function indicates how much the probability of impulse noise is. We obtain the probability from minimum deviation of small filter kernel. The example of the fuzzy membership function is shown in figure 8.

Membership

function

Min. Deviation

1

min∆0

g (x)

Figure 8. Example of fuzzy membership function for the probability of impulse noise.

SPIE-IS&T/ Vol. 6817 68170B-6

The final noise filter estimate at the position x is

NGfinal IggmI ,, )1(ˆxxxxx −+=

(4)

where g is the calculated probability of impulse noise and m is the median estimate.

4. EXPERIMENTAL RESULTS In the first experiment, we have performed experiments of noise estimation on images corrupted by synthetic noise based on typical sensor noise model. Figure 9 shows noisy image used for noise estimation experiment. In figure 10, (a) is block filtering with uniform interval based noise estimation method introduced in Section 1, and (b) is our proposed noise estimation method. We have also computed the processing time. In our implementation, the processing time of our estimation method is 37% faster than that of the block filtering method for the test image with Intel Pentium 4 CPU 3.2G, 1GB RAM, showing that our algorithm is fast noise estimation method.

Figure 9. Test image for noise estimation experiment.

0 50 100 150 200 2500

5

10

15

20

25

30

35

40Ground truth

Intensity

Noi

se L

evel

0 50 100 150 200 2500

5

10

15

20

25

30

35

40Block filtering with uniform interval based noise estimation result

Intensity

Noi

se L

evel

Ground truth Block filtering with uniform interval based noise estimation result

Intensity Intensity

Noi

se le

vel

Noi

se le

vel

(a)

0 50 100 150 200 2500

5

10

15

20

25

30

35

40Proposed noise estimation result

Intensity

Noi

se L

evel

0 50 100 150 200 2500

5

10

15

20

25

30

35

40Ground truth

Intensity

Noi

se L

evel

Ground truth Proposed noise estimation result

Intensity Intensity

Noi

se le

vel

Noi

se le

vel

(b) Figure 10. Noise estimation results: (a) block filtering based noise estimation; (b) proposed noise estimation.

SPIE-IS&T/ Vol. 6817 68170B-7

Cbh I

JTTI

C h I

Intensity Intensity Intensity

(i)

Y dianne!

Y Channel Cb dianne! Cr Channel

______Intensity rntsi

(i)

Intensity

In the second experiment, images were corrupted by synthetic noise based on typical sensor noise model again. The figure 11 and 12 show the used test images and sensor noise model. Figure 13,14 and the MSE values in table Ⅰ show that the performance of the proposed algorithm compares very well against the existing spatial denoising filters. The used smoothing parameters of Wiener estimator is 18.0 for butterfly image and 15.0 for Lena image. The smoothing parameters of Sigma filter are 40.0 for butterfly image and 30.0 for Lena image. The geometric smoothing parameters of Bilateral filter are 2.5 for both butterfly and Lena images, and the photometric smoothing parameters are 35.0 for butterfly image and 25.0 for Lena image. The proposed algorithm outputs smoother surfaces in flat region while producing less artifacts in sharp edge region and preserving more details. In terms of computation, the processing time of our noise reduction method is 48% faster than that of Bilateral filter for the butterfly image with Intel Pentium 4 CPU 3.2G, 1GB RAM. The processing time of noise estimation was included in measured computation time.

Figure 11. Butterfly image: (a) original image; (b) noisy image; (c) synthetic noise model.

Figure 12. Lena image: (a) original image; (b) noisy image; (c) synthetic noise model.

SPIE-IS&T/ Vol. 6817 68170B-8

(e) (1)

Figure 13. Enlargements of butterfly image corrupted by synthetic sensor noise: (a) original image; (b) noisy image (c) wiener filter (d) sigma filter (e) bilateral filter (f) proposed method.

24.032838.267752.589354.6958180.1966Butterfly

23.635029.523037.305034.649599.6534Lena

ProposedBilateralSigmaWienerNoisy

24.032838.267752.589354.6958180.1966Butterfly

23.635029.523037.305034.649599.6534Lena

ProposedBilateralSigmaWienerNoisy

TABLE 1. RGB AVERAGE MSE FOR BUTTERFLY AND LENA IMAGES.

SPIE-IS&T/ Vol. 6817 68170B-9

(a) (b)

(C) (ii)

Fig. 14. Enlargements of Lena image corrupted by synthetic sensor noise: (a) original image; (b) noisy image (c) wiener filter (d) sigma filter (e) bilateral filter (f) proposed method.

SPIE-IS&T/ Vol. 6817 68170B-10

(a) (b)

(e) (1)

Fig. 15. Enlargements of image corrupted by real sensor noise: (a) original image; (b) noisy image (c) wiener filter (d) sigma filter (e) bilateral filter (f) proposed method.

In the third experiment, the proposed algorithm is tested on image corrupted by real sensor noise. The image in figure 15 is captured by camera phone with 1.3M CMOS with high ISO setting. The experimental result suggests that the proposed algorithm is robust to both edge-region artifact and outliers.

5. CONCLUSION In this paper, image sensor noise estimation algorithm and high ISO noise reduction algorithm were presented. In order to accurately and quickly estimate signal dependent sensor noise, noise level function was modeled as a linearly interpolated combination of noise profile components. Nonlinear filters were adaptively blended based on noise level adaptive local activity estimation and impulse detection to suppress artifacts of high ISO noise reduction. The simulation results for both proposed noise estimation and reduction algorithm showed promising performance, when compared to existing methods. While the proposed algorithms showed improved image quality, the measured computation time suggested that the processing time of the proposed algorithm was comparable to existing methods.

SPIE-IS&T/ Vol. 6817 68170B-11

REFERENCES

1. D. H. Shin, R. H. Park, S. Yang and J. Jung, “Block-based noise estimation using adaptive Gaussian filtering”, IEEE Transactions on Consumer Electronics, vol. 51, no. 1, Feb. 2005.

2. C. Liu, William T. Freeman, R. Szeliski, Sing Bing Kang, “Noise estimation from a single image”, proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition, 2006.

3. D. L. Donoho, I. M. Johnston, “Denoising by soft-thresholding”, IEEE Trans. on Information Theory, 41: 613-627, 1995.

4. J. L. Starck, E. Candes, D. L. Donoho, “The Curvelet transform for image denoising”, IEEE Trans. on Image Process., 11(6): pp 670-684, 2002.

5. J. Portilla, V. Strela, M. Wainwright, E. Simoncelli, “Image denoising using scale mixtures of Gaussians in the wavelet domain”, IEEE Trans. Image Process., 12(11): 1338-1351, 2003

6. S. G. Chang, B. Yu and M. Vetterli, “Spatially adaptive wavelet thresholding with context modeling for image denoising”, IEEE Trans. on Image Process., vol. 9, no. 9, 2000

7. L. I. Rudin, S. Osher and E. Fatemi, “Nonlinear total variation based noise removal algorithms”, Physica D 60 259-268, 1992

8. P. Perona and J. Malik, “Scale-space and edge detection using anisotropic diffusion”, IEEE Trans. on Pattern Analysis and Machine Intelligence, vol. 12, no. 7, 1990

9. D. Darian Muresan, Thomas W. Parks, “Adaptive principal components and image denosing”, proceedings of International Conference Image Processing, vol.1, Sep. 2003

10. Keigo Hirakawa, Thomas W. Parks, “Image denoising using total least squares”, IEEE Trans. Image Processing, vol. 15, no. 9, Sep 2006

11. Antoni Buades, Jean-Michel Morel, “A non-local algorithm for image denoising”, IEEE Computer Society Conference on Computer Vision and Pattern Recognition, vol. 2, p. 60-65, 2005

12. Lim, Jae S., “Two-dimensional signal and image processing”, Englewood Cliffs, NJ, Prentice Hall, 1990, p. 548 13. J. S. Lee, “Digital image smoothing and the sigma filter”, Comp. Vis. Graph. Image Process., 24: 255-269, 1983 14. C. Tomasi, R. Manduchi, “Bilateral Filtering for Gray and Color Images”, Proceedings of the 1998 IEEE

International Conference on Computer Vision, Bombay, India 15. Hilda Faraji and W. James MacLean, “CCD noise removal in digital images”, IEEE Trans. on Image Processing,

vol. 15, no. 9, 2006 16. Ioannis Pitas and Anastasios N. Venetsanopoulos, “Order Statistics in Digital Image Processing”, proceedings of

the IEEE, vol. 80, no. 12, 1992

SPIE-IS&T/ Vol. 6817 68170B-12