Frequency domain methods for demosaicking of Bayer sampled color images Eric Dubois

Frequency domain methods for demosaicking of Bayer sampled color images Eric Dubois

Frequency-domain Bayer demosaicking

Problem Statement

Problem: Most digital color cameras capture only one color component at each spatial location. The remaining components must be reconstructed by interpolation from the captured samples. Cameras provide hardware or software to do this, but the quality may be inadequate.Objective: Develop new algorithms to interpolate each color plane (called demosaicking) with better quality reconstruction, and with minimal computational complexity.


Retinal Cone MosaicThe human visual system must solve a similar problem!


Construction of color image from color planes+


Lighthouseoriginal


Lighthousered original


Lighthousegreen original


Lighthouseblue original


Formation of Color planes


Lighthousered subsampled


Lighthousegreen subsampled


Lighthouseblue subsampled


LighthouseBayer CFA image


Color plane interpolationGAGBGLGRGIGreen channel: bilinear interpolation


Color plane interpolationRCRed channel: bilinear interpolationRNWRNERSWRSERS


Lighthousered interpolated


Lighthousegreen interpolated


Lighthouseblue interpolated


LighthouseInterpolated color image


Lighthouseoriginal


Can we do better?Color planes have severe aliasing. Better interpolation of the individual planes has little effect.


Lighthousered interpolatedwith bilinear interpolator


Lighthousered interpolatedwith bicubic interpolator


Can we do better?Color planes have severe aliasing. Better interpolation of the individual planes has little effect.We could optically prefilter the image (blur it) so that aliasing is less severe.


Lighthousered interpolatedwith bilinear interpolator


Lighthouseprefiltered red interpolatedwith bilinear interpolator


LighthouseInterpolated color image


Lighthouse Prefiltered & Interpolated color image


Lighthouse original


Can we do better?Color planes have severe aliasing. Better interpolation of the individual planes has little effect.We could optically prefilter the image (blur it) so that aliasing is less severe.We can process the three color planes together to gather details from all three components.


Can we do better?There have been numerous papers and patents describing different algorithms to interpolate the color planes they all work on the three planes together, exploiting the correlation between the three components.Gunturk et al. published an extensive survey in March 2005. The best methods were the projection on convex sets (POCS) algorithm (lowest MSE) and the adaptive homogeneity directed (AHD) algorithm (best subjective quality). We present here a novel frequency-domain algorithm.


Spatial multiplexing modelsubsamplingmultiplexing


Spatial multiplexing model


Frequency-domain multiplexing modelRe-arranging the spatial multiplexing expression


Frequency-domain multiplexing modelDavid Alleysson, EPFL


Luma and chrominance components


Luma and chrominance componentsLuma fLChroma_1 fC1Chroma_2 fC2


Lighthouse BilinearlyInterpolated color image


Frequency-domain demosaicking algorithmExtract modulated C1 using a band-pass filter at (0.5,0.5) and demodulate to basebandExtract modulated C2 using band-pass filters at (0.5,0.0) and (0.0, 0.5), demodulate to baseband, and combine in some suitable fashion (the key)Subtract modulated C1 and remodulated C2 from the CFA to get the estimated luma component L.Matrix the L, C1 and C2 components to get the RGB representation.


Spectrum of CFA signalab


Using C2a onlyUsing C2b only


OriginalFrom C2a onlyFrom C2b onlyDemosaicking using C2a only or C2b only -- details


Demosaicking Block Diagram


Spectrum of CFA signalab


Design IssuesHow to choose the filters h1, h2a and h2bFrequency domain design methodsLeast-squares design methodsSize of the filtersHow to combine the two estimates and Choice of features to guide weighting The two above issues may be inter-related.


Filter designGaussian filters (Alleysson)Window design or minimax designDefine ideal response, with pass, stop and transition bandsApproximate using the window design methodRefine using minimax or least pth optimizationCan design low-pass filters and modulate to the center frequency


Filter specification u v val0.000 0.00 1.00.110 0.00 1.00.110 0.02 1.00.000 0.10 1.00.030 0.10 1.00.070 0.06 1.00.338 0.00 0.00.338 0.05 0.00.050 0.36 0.00.000 0.36 0.00.184 .205 0.00.500 0.00 0.00.000 0.50 0.00.500 0.50 0.0


Ideal response perspective view


Ideal response contour plot


Window design perspective view


Window design contour plot


Least pth filter perspective view


Least pth filter contour plot


21 x 21 filters in SPL published algorithmuvh2ah2bh1


Adaptive weighting of C2a and C2bWe want to form the estimate of C2 by choosing the best between C2a and C2b, or perhaps by a weighted average. We have used

should be near 1 when C2a is the best choice, and near 0 when C2b is the best choice


Typical scenarios for local spectrumLC2aC2aC2bC2bC1C1C1C1B: C2b is better estimateLC2aC2aC2bC2bC1C1C1C1A: C2a is better estimateuvvu


Scenario AScenario B


Typical scenarios for local spectrumLC2aC2aC2bC2bC1C1C1C1B: C2b is better estimateLC2aC2aC2bC2bC1C1C1C1A: C2a is better estimateuvvu


Weight selection strategyScenario A: average local energy near (fm, 0) is smaller than near (0, fm ).Scenario B: average local energy near (0, fm ) is smaller than near (fm, 0).Let be a measure of the average local energy near (fm, 0), and be a measure of the average local energy near (0, fm ).


Gaussian filters for local energy measurementuvfm = 0.375


ResultsResults with this adaptive frequency-domain demosaicking method were published in IEEE Signal Processing Letters in Dec. 2005. All filters were of size 21 x 21. Filters h1, h2a and h2b were designed with the window method, with band parameters determined by trial and error. The method gave the lowest mean-square reconstruction error on the standard set of Kodak test images compared to other published methods.


Mean square error comparison


Frequency domain methods for demosaicking of Bayer sampled color images Eric Dubois

Documents

Transcript of Frequency domain methods for demosaicking of Bayer sampled color images Eric Dubois