Presented By: Anthony Karloff

RESEARCH CENTRE FOR INTEGRATED MICROSYSTEMS - UNIVERSITY OF WINDSOR

Presented By:Presented By:Anthony KarloffAnthony Karloff

Demosaicing with ImprovedDemosaicing with ImprovedEdge Direction DetectionEdge Direction Detection


OverviewOverview

• Demosaicing BackgroundDemosaicing Background• Basics and ChallengesBasics and Challenges• Advanced Methods (State of the Art)Advanced Methods (State of the Art)• Color Channel ReconstructionColor Channel Reconstruction• ConclusionConclusion


BackgroundBackground

BasicsBasics

Advanced Advanced MethodsMethods

• Incomplete color planes from CCD sensors.Incomplete color planes from CCD sensors.

Why Image Reconstruction?Why Image Reconstruction?

Demosaicing BackgroundDemosaicing Background

Color Filter Array (CFA) on image sensor.Color Filter Array (CFA) on image sensor.

Channel Channel ReconstructionReconstruction

ConclusionConclusion



BasicsBasics


Basics and ChallengesBasics and Challenges

• Must Interpolate color planes to re-create Must Interpolate color planes to re-create image.image.

Red Channel InterpolationRed Channel Interpolation

Color Plane InterpolationColor Plane Interpolation





BasicsBasics



• Pixel AveragingPixel Averaging- lose image resolution- lose image resolution

Color Plane Interpolation MethodsColor Plane Interpolation Methods

• Nearest NeighborNearest Neighbor- Poorest Quality- Poorest Quality

• Bilinear/SplineBilinear/Spline- Color artifacts at edges- Color artifacts at edges

G2 B1

R1 G1P1

G2 B1

R1 G1 P1

G4 B2

R1 G3

G5

R2

G1 B1 G2

P1





BasicsBasics



• Problem with most simple interpolation Problem with most simple interpolation algorithms is the presence of color artifacts.algorithms is the presence of color artifacts.

Color ArtifactsColor Artifacts

Original ImageOriginal Image Bilinear Interpolation of Bayer ImageBilinear Interpolation of Bayer Image





BasicsBasics



• Due to interpolation across edges.Due to interpolation across edges.

Color ArtifactsColor Artifacts

P9P6

P11

P7

P19P17

P8

P14

P16

P12 P13

P18

P15

P10

P1 P4P2 P3 P5

P20

P21 P24P22 P23 P25

P9P6

P11

P7

P19P17

P8

P14

P16

P12 P13

P18

P15

P10

P1 P4P2 P3 P5

P20

P21 P24P22 P23 P25

BilinearInterpolation

Artifacts

Dark to Light Edge over Bayer Pattern

Resulting Edge after Interpolation




Advanced MethodsAdvanced Methods

• Group pixels of similar objectsGroup pixels of similar objects

Advanced Techniques for Color Advanced Techniques for Color Plane InterpolationPlane Interpolation


BasicsBasics


• Interpolate along edges (not across)Interpolate along edges (not across)

• Interpolate green color plane firstInterpolate green color plane first

• Interpolate image more than one iteration Interpolate image more than one iteration (refinement)(refinement)

• Use color plane gradientsUse color plane gradientsChannel Channel ReconstructionReconstruction




• Better estimation of color plane behavior.Better estimation of color plane behavior.

Using Gradients for Image Using Gradients for Image ReconstructionReconstruction

Bayer Pattern for GreenBayer Pattern for GreenCentered PixelCentered Pixel

P1 P2P5

P3

P7 P9P4 P6

P8

2

64)5(

PPPDx

2

82)5(

PPPDy

22

73)5(

PPPDxd

22

91)5(

PPPDyd

• Notice that the differences are always from Notice that the differences are always from the same color plane.the same color plane.


BasicsBasics


1. GRADIENTS






BasicsBasics

Advanced Advanced MethodsMethods • Associates colors of the same object.Associates colors of the same object.

Kimmels ‘E’ Function for Pixel Kimmels ‘E’ Function for Pixel GroupingGrouping

P1 P2P5

P3

P7 P9P4 P6

P8

Ie. If P5 and Pi are part of the same Ie. If P5 and Pi are part of the same object, E will be close to unity.object, E will be close to unity.

22 )()5(1

1)5(

PiDiPDiPEi

• There are eight Ei values for each pixel. One There are eight Ei values for each pixel. One for each neighbor.for each neighbor.

Bayer Pattern for GreenBayer Pattern for GreenCentered PixelCentered Pixel

1. GRADIENTS

2. GROUPING






BasicsBasics


• Interpolation is best performed in the same Interpolation is best performed in the same direction as an edge.direction as an edge.

Using Edge Detection (Wide)Using Edge Detection (Wide)

Bayer Pattern for RedBayer Pattern for RedCentered PixelCentered Pixel

GGG PPPH 1412)13(

GGG PPPV 188)13(

P9P6

P11

P7

P19P17

P8

P14

P16

P12 P13

P18

P15

P10

P1 P4P2 P3 P5

P20

P21 P24P22 P23 P25RRRR PPPPH 1321511)13(

RRRR PPPPV 132233)13(

Edge detection of radius 3Edge detection of radius 3

1. GRADIENTS

2. GROUPING

3. EDGE DETECT 3 pxls






BasicsBasics


1. GRADIENTS

2. GROUPING


• Uses narrow edge detection to improve Uses narrow edge detection to improve edges by looking between color planes.edges by looking between color planes.

Narrow Edge DetectionNarrow Edge Detection


RGGGR PPPPH 1321412)13(

RGGGR PPPPV 13282)13(

P9P6

P11

P7

P19P17

P8

P14

P16

P12 P13

P18

P15

P10

P1 P4P2 P3 P5

P20

P21 P24P22 P23 P25

Edge detection of radius 2Edge detection of radius 2

BBBGBBGB PPPPPPPH 18219178297)13( 21

BBBGBBGB PPPPPPPV 142199122177)13( 21







BasicsBasics


1. GRADIENTS

2. GROUPING



• Compare average color differences in a 5x5 Compare average color differences in a 5x5 region to determine whether the Red or Blue region to determine whether the Red or Blue channel is more closely related to the green.channel is more closely related to the green.

Local Inter-Channel CorrelationLocal Inter-Channel Correlation

5555 xxGR RGC

5555 xxGB BGC

5. COLOR CORRELATION


P9P6

P11

P7

P19P17

P8

P14

P16

P12 P13

P18

P15

P10

P1 P4P2 P3 P5

P20

P21 P24P22 P23 P25






BasicsBasics


1. GRADIENTS

2. GROUPING




• Now we can complete the edge detectorNow we can complete the edge detector

GB

GRGR H

HHHH

otherwise

CCif GBGR

GB

GRGR V

VVVV

otherwise

CCif GBGR

Improved Edge DetectorImproved Edge Detector

34 5

6. IMPROVED EDGE DETECTION




Channel ReconstructionChannel Reconstruction


BasicsBasics



1. GRADIENTS

2. GROUPING





• Reconstruct the green channel using edge Reconstruct the green channel using edge detectors and the approximated red and blue.detectors and the approximated red and blue.

Channel Reconstruction OverviewChannel Reconstruction Overview

• Reconstruct the red and blue channels using Reconstruct the red and blue channels using the complete green channel.the complete green channel.

• For each pixel we now have:For each pixel we now have: )(PiEi H V


• Approximate the red and blue channels Approximate the red and blue channels using Bilinear Interpolation.using Bilinear Interpolation.




BasicsBasics



1. GRADIENTS

2. GROUPING





• For each Green pixel on a red center…For each Green pixel on a red center…

Green Channel ReconstructionGreen Channel Reconstruction


P9P7

P19P17

P8

P14P12 P13

P18

otherwiseE

PiPiE

VHifEE

PPEPPE

VHifEE

PPEPPE

PP

iPi

iRGPi

PP

RGPRGP

PP

RGPRGP

RG

18,14,12,8

18,14,12,8'

188

'18'8

1412

'14'12

)(

)1818()88(

)1414()1212(

1313

• A similar approach is taken to the finding the A similar approach is taken to the finding the green value at a blue centered pixelgreen value at a blue centered pixel





BasicsBasics



1. GRADIENTS

2. GROUPING





• Blue and red channels are then completed Blue and red channels are then completed using the full green channel.using the full green channel.

Blue and Red Channel ReconstructionBlue and Red Channel Reconstruction


P9P6

P11

P7

P19P17

P8

P14

P16

P12 P13

P18

P15

P10

P1 P4P2 P3 P5

P20

P21 P24P22 P23 P25

191797

1919171799771313

EEEE

KEKEKEKEPP GB

Where…Where… GB PiPiKi

Similar approach is taken for completingSimilar approach is taken for completingRed channel.Red channel.





BasicsBasics




• Highly computational and hence slow.Highly computational and hence slow.


• Not suitable for real-time applications.Not suitable for real-time applications.

• Drastically reduces color artifacts.Drastically reduces color artifacts.

• Improved Edge Quality.Improved Edge Quality.

Thank YouThank You


ReferencesReferences


BasicsBasics




[1] D. Darian Muresan, S. Luke, and T. W. Parks, “Reconstruction of Color Images From CCD Arrays,” Cornell University, Ithaca NY. 1485, [2] R. Kimmel, “Demosaicing: Image Reconstruction from Color CCD Samples” IEEE Transl. J. Image Processing, vol. 8, Sept. 1999.

[3] Xiaomeng Wang, Weisi Lin, Ping Xue, “Demosaicing with Improved Edge Direction Detection” IEEE Transl. J. Image Processing, 2005.

Presented By: Anthony Karloff

Documents

Transcript of Presented By: Anthony Karloff