Frequency domain methods for demosaicking of Bayer sampled color images Eric Dubois.

Frequency domain methods for demosaicking of Bayer sampled color

images Eric Dubois

Frequency-domain Bayer demosaicking 2

Problem Statement

• Problem: Most digital color cameras capture only one color component at each spatial location. The remaining components must be reconstructed by interpolation from the captured samples. Cameras provide hardware or software to do this, but the quality may be inadequate.

• Objective: Develop new algorithms to interpolate each color plane (called demosaicking) with better quality reconstruction, and with minimal computational complexity.


Retinal Cone Mosaic

The human visual system must solve a similar problem!


Construction of color image from color planes

+

Lighthouseoriginal

Lighthousered original

Lighthousegreen original

Lighthouseblue original


Formation of Color planes

Lighthousered subsampled

Lighthousegreen subsampled

Lighthouseblue subsampled

LighthouseBayer CFA image


Color plane interpolation

GA

GB

GL GR

)(4

1ABRLI GGGGG

GI

Green channel: bilinear interpolation


Color plane interpolation

)(4

1SESWNENWC RRRRR

RC

Red channel: bilinear interpolation

RNWRNE

RSWRSE RS

SESWS RRR 2

1

Lighthousered interpolated

Lighthousegreen interpolated

Lighthouseblue interpolated

LighthouseInterpolated color image

Lighthouseoriginal


Can we do better?

• Color planes have severe aliasing. Better interpolation of the individual planes has little effect.

Lighthousered interpolatedwith bilinear interpolator

Lighthousered interpolatedwith bicubic interpolator


Can we do better?


• We could optically prefilter the image (blur it) so that aliasing is less severe.

Lighthousered interpolatedwith bilinear interpolator

Lighthouseprefiltered red interpolatedwith bilinear interpolator

LighthouseInterpolated color image

Lighthouse Prefiltered & Interpolated color image

Lighthouse original


Can we do better?


• We could optically prefilter the image (blur it) so that aliasing is less severe.

• We can process the three color planes together to gather details from all three components.


Can we do better?• There have been numerous papers and patents

describing different algorithms to interpolate the color planes – they all work on the three planes together, exploiting the correlation between the three components.

• Gunturk et al. published an extensive survey in March 2005. The best methods were the projection on convex sets (POCS) algorithm (lowest MSE) and the adaptive homogeneity directed (AHD) algorithm (best subjective quality).

• We present here a novel frequency-domain algorithm.


Spatial multiplexing model

subsampling multiplexing


Spatial multiplexing model

))1(1)()1(1](,[4

1

))1(1)()1(1](,[4

1))1(1](,[

2

1],[

2121

2121

212121CFA

nn

B

nn

R

nn

G

nnf

nnfnnfnnf


Frequency-domain multiplexing model

212121

21212121

212121

2121

2121

212121CFA

)1()1(],[4

1],[

4

1

)1(],[4

1],[

2

1],[

4

1

],[4

1],[

2

1],[

4

1

))1(1)()1(1](,[4

1

))1(1)()1(1](,[4

1))1(1](,[

2

1],[

nn

BR

nn

BGR

BGR

nn

B

nn

R

nn

G

nnfnnf

nnfnnfnnf

nnfnnfnnf

nnf

nnfnnfnnf

Re-arranging the spatial multiplexing expression


Frequency-domain multiplexing model

)2/2exp()2/2exp(],[

)2/)(2exp(],[],[

)1()1(],[

)1](,[],[],[

21212

2121121

21212

212112121CFA

njnjnnf

nnjnnfnnf

nnf

nnfnnfnnf

C

CL

nn

C

nn

CL

)5.0,(),5.0()5.0,5.0(),(),( 221CFA vuFvuFvuFvuFvuF CCCL

David Alleysson, EPFL


Luma and chrominance components

2

1

41

41

41

21

41

41

21

41

2

1

211

011

211

0

C

C

L

B

G

R

B

G

R

C

C

L

f

f

f

f

f

f

f

f

f

f

f

f


Luma and chrominance components

Luma fL Chroma_1 fC1 Chroma_2 fC2

Lighthouse BilinearlyInterpolated color image


Frequency-domain demosaicking algorithm

1. Extract modulated C1 using a band-pass filter at (0.5,0.5) and demodulate to baseband

2. Extract modulated C2 using band-pass filters at (0.5,0.0) and (0.0, 0.5), demodulate to baseband, and combine in some suitable fashion (the key)

3. Subtract modulated C1 and remodulated C2 from the CFA to get the estimated luma component L.

4. Matrix the L, C1 and C2 components to get the RGB representation.


Spectrum of CFA signal

ab

Using C2a only Using C2b only


Original From C2a only From C2b only

Demosaicking using C2a only or C2b only -- details


Demosaicking Block Diagram

h2a

h2b

+ -

fCFA

(-1)n1+n2

-(-1)n2

(-1)n1

h1

combine

(-1)n1-(-1)n2

+

-

matrix

fR

fG

fB

fC2am

fC2bm

fC1m fC1

fC2a

fC2b

fC2

fL

fC1

fC2

21212

212112121CFA )1()1(],[)1](,[],[],[

nn

C

nn

CL nnfnnfnnfnnf


Spectrum of CFA signal

ab


Design Issues

• How to choose the filters h1, h2a and h2b

– Frequency domain design methods

– Least-squares design methods

– Size of the filters

• How to combine the two estimates and – Choice of features to guide weighting

• The two above issues may be inter-related.

aCf 2ˆ

bCf 2ˆ


Filter design

• Gaussian filters (Alleysson)• Window design or minimax design

– Define ideal response, with pass, stop and transition bands

– Approximate using the window design method

– Refine using minimax or least pth optimization

– Can design low-pass filters and modulate to the center frequency


Filter specification u v val0.000 0.00 1.00.110 0.00 1.00.110 0.02 1.00.000 0.10 1.00.030 0.10 1.00.070 0.06 1.00.338 0.00 0.00.338 0.05 0.00.050 0.36 0.00.000 0.36 0.00.184 .205 0.00.500 0.00 0.00.000 0.50 0.00.500 0.50 0.0

0 0.05 0.1 0.15 0.2 0.25 0.3 0.35 0.4 0.45 0.5

0

0.05

0.1

0.15

0.2

0.25

0.3

0.35

0.4

0.45

0.5

u

v

filter specification

1

0


-0.5-0.4

-0.3-0.2

-0.10

0.10.2

0.30.4

0.5

-0.5

0

0.50

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1

Horizontal frequency

Perspective plot of ideal filter response

Vertical frequency

Mag

nitu

de o

f fil

ter

resp

onse

Ideal response – perspective view


Ideal response – contour plot

Contour plot of ideal filter response

Horizontal frequency

Vert

ical fr

equency

-0.5 -0.4 -0.3 -0.2 -0.1 0 0.1 0.2 0.3 0.4-0.5

-0.4

-0.3

-0.2

-0.1

0

0.1

0.2

0.3

0.4


Window design – perspective view

-0.5-0.4

-0.3-0.2

-0.10

0.10.2

0.30.4

0.5

-0.5

0

0.5-0.2

0

0.2

0.4

0.6

0.8

1

1.2

Perspective plot of lowpass filter


Window design – contour plot

Contour plot of lowpass filter

-0.5 -0.4 -0.3 -0.2 -0.1 0 0.1 0.2 0.3 0.4

-0.5

-0.4

-0.3

-0.2

-0.1

0

0.1

0.2

0.3

0.4


Least pth filter – perspective view

-0.5-0.4

-0.3-0.2

-0.10

0.10.2

0.30.4

0.5

-0.5

0

0.5-0.2

0

0.2

0.4

0.6

0.8

1

1.2

Perspective plot of lowpass filter


Least pth filter – contour plot

0

0

0

0.4

Contour plot of lowpass filter

-0.5 -0.4 -0.3 -0.2 -0.1 0 0.1 0.2 0.3 0.4-0.5

-0.4

-0.3

-0.2

-0.1

0

0.1

0.2

0.3

0.4

21 x 21 filters in SPL published algorithm

-1-0.5

00.5

1

-1

-0.5

0

0.5

1

0

0.2

0.4

0.6

0.8

1

Horizontal frequency (c/pixel)Vertical frequency (c/pixel)

Mag

nitu

de

-1-0.5

00.5

1

-1

-0.5

0

0.5

1

0

0.2

0.4

0.6

0.8

1


Mag

nitu

de

-1-0.5

00.5

1

-1

-0.5

0

0.5

1

0

0.2

0.4

0.6

0.8

1


Mag

nitu

de

u

v

h2a h2b

h1


Adaptive weighting of C2a and C2b

• We want to form the estimate of C2 by choosing the best between C2a and C2b, or perhaps by a weighted average. We have used

• should be near 1 when C2a is the best choice, and near 0 when C2b is the best choice

],[ˆ]),[1(],[ˆ],[],[ˆ2122121221212 nnfnnwnnfnnwnnf bCaCC

],[ 21 nnw


Typical scenarios for local spectrum

L C2aC2a

C2b

C2b

C1C1

C1 C1

B: C2b is better estimate

L C2aC2a

C2b

C2b

C1C1

C1 C1

A: C2a is better estimate

u

v v

u

Scenario A

Scenario B


Typical scenarios for local spectrum

L C2aC2a

C2b

C2b

C1C1

C1 C1

B: C2b is better estimate

L C2aC2a

C2b

C2b

C1C1

C1 C1

A: C2a is better estimate

u

v v

u


Weight selection strategy

• Scenario A: average local energy near (fm, 0) is smaller than near (0, fm ).

• Scenario B: average local energy near (0, fm ) is smaller than near (fm, 0).

• Let be a measure of the average local energy near (fm, 0), and be a measure of the average local energy near (0, fm ).

],[ 21 nneX

],[ 21 nneY

],[],[

],[],[

2121

2121 nnenne

nnennw

YX

Y


Gaussian filters for local energy measurement

-1-0.5

00.5

1

-1

-0.5

0

0.5

1

0

0.2

0.4

0.6

0.8

1


Mag

nitu

de

-1-0.5

00.5

1

-1

-0.5

0

0.5

1

0

0.2

0.4

0.6

0.8

1

Horizontal frequency (c/pixel)Vertical frequency (c/pixel)M

agni

tude

u

vfm = 0.375


Results

• Results with this adaptive frequency-domain demosaicking method were published in IEEE Signal Processing Letters in Dec. 2005. All filters were of size 21 x 21. Filters h1, h2a and h2b were designed with the window method, with band parameters determined by trial and error. The method gave the lowest mean-square reconstruction error on the standard set of Kodak test images compared to other published methods.

http://www.site.uottawa.ca/~edubois/demosaicking/


Mean square error comparison

Frequency domain methods for demosaicking of Bayer sampled color images Eric Dubois.

Documents

Transcript of Frequency domain methods for demosaicking of Bayer sampled color images Eric Dubois.