A LOW-COMPLEXITY, MOTION-ROBUST, SPATIO-TEMPORALLY ADAPTIVE VIDEO

DE-NOISER WITH IN-LOOP NOISE ESTIMATION

Chirag Jain, Sriram SethuramanIttiam Systems (Pvt.) Ltd., Bangalore, India

Image Processing, 2008. ICIP

Outline

• Introduction• Proposed Algorithm– LLMMSE based spatial filtering– Temporal filtering– Noise estimation

• Experimental Results• Complexity Comparison• Conclusions and Future Work

Introduction

• Many of the successful techniques determine the weights needed to combine the neighborhood of the pixel being de-noised.– e.g., bi-lateral[2], anisotropic diffusions

method[3],spatial non local means (NL-means) de-noiser

[4].– Drawback: high computational complexity

[2] C. Tomasi, and R. Manduchi.,” Bilateral filtering for gray and color images,” Proceedings of the Sixth International Conference on Computer Vision, pp. 839-846, 1998.[3] P. Bourdon, B. Augereau, C. Olivier, and C. Chatellier, “Noise removal on color image sequences using coupled anisotropic diffusions and noise-robust motion detection,” EUSIPCO'04 -Signal Processing XII, Wien (Austria), pp. 1194 – Sep. 2004.[4] A. Buades, B. Coll, and J. M. Morel, “Denoising image sequences does not require motion estimation,” IEEE Int. Conf. on Advanced Video and Signal based Surveillance, 2005.

Introduction

• We propose a de-noising algorithm using the following principles:– Block-based noise estimation and de-noising;– Use of block motion vectors by a video encoder;– Locally adaptive Linear Minimum Mean Square Error

(LLMMSE) methods.– Reliance on Infinite Impulse Response (IIR) temporal

filtering.– Spatially adaptive weight selection for the spatial-

temporal IIR (ST-IIR) filter.

Proposed AlgorithmCase 1: stationary blocks

Case 3: well Motion Compensated blocks

Case 2: under-compensated blocks

LLMMSE based spatial filtering

• An 3x3 Gaussian filter is applied to obtain a output L (low pass filtered ).

• The output H = X - L (X is the noisy image).• Spatially filtered output X’(i,j) is computed as:

– σf2: variance of noisy signal(X) for every 8x8 block.

– σn2: the estimated noise variance of the sequence.

Temporal filtering

• Let X(n) be the current frame, Y(n-1) be the previous denoised frame.

• σfr_diff2 : the estimated variance of the

difference data in stationary areas

between X(n-1) and Y(n-2)• σblk_diff

2 : the variance of the difference between cur_blk and its co-located block in Y(n-1)

(prev_blk)

Temporal filtering• Is block stationary? If (2) is satisfied

and at least one of (3) or (4) are true:

– Case 1: stationary blocks (CL)• Is MC successful? If (5) is satisfied:

– Case 3: well Motion Compensated blocks(MC)• All remaining blocks belong to case 2:

– Case 2: under-compensated blocks

Temporal filtering

• Case 2 : – Blocks only spatially filtered

• Case 1&3 : – The ST-IIR filtering is performed at a pixel level by using

the CL or MC pixel in Y(n-1):

– pixel_diff : the difference between the current pixel and its CL or MC pixel in Y(n-1).

– For pixels with pixel_diff > σfr_diff(n-1), α = e^(-1/k4)

σfr_diff(n-1) : the deviation of stationary (n-1)

Noise estimation

• If the current frame X(n) has noise variance σn2 and

Y(n-1) has noise variance σp2 , the variance σfr_diff

2(n) (stationary)will be:

• Ideally, σp2 should be zero, but we should introduce

a confidence c for robustness:

– c: the average value of α used for ST-IIR.

Noise estimation

• If the Y(n-1) is not filtered, c = 1, σn2 = σfr_diff

2 . • The convergence of standard deviation of the

noise against frame number for two sequences:

Experimental Results

• Additive white Gaussian noise of a specified variance was added to clean video sequences.

• Encoded using a H.264 baseline profile encoder.

• NL-means[4]: window size of 21x21 and a similarity square neighborhood of 7x7 with (a = 1 and h = n) [4].

Experimental Results

Experimental Results

Complexity Comparison

• The proposed algorithm perform the decisions at a block level and the filtering at a pixel level.

• The proposed algorithm requires only 38 operations(+,-,*,look-ups) per pixel

• NL-means has a complexity of 441x49 weighted Euclidean distance calculations and exponential function lookups per pixel.

Conclusions and Future Work

• The proposed algorithm comes close to the performance of more complex algorithms such as the NL-means method.

• It is ideally suited for real time embedded implementations.

• The spatial filter’s output can be used by the encoder to avoid spurious motion vectors.