CS6640 Computational Photography - Cornell University · CS6640 Computational Photography 7....

127
© 2012 Steve Marschner CS6640 Computational Photography 7. Digital camera processing pipeline 1

Transcript of CS6640 Computational Photography - Cornell University · CS6640 Computational Photography 7....

Page 1: CS6640 Computational Photography - Cornell University · CS6640 Computational Photography 7. Digital camera processing pipeline 1. Cornell CS6640 Fall 2012 Camera processing pipeline

© 2012 Steve Marschner

CS6640 Computational Photography

7. Digital camera processing pipeline

1

Page 2: CS6640 Computational Photography - Cornell University · CS6640 Computational Photography 7. Digital camera processing pipeline 1. Cornell CS6640 Fall 2012 Camera processing pipeline

Cornell CS6640 Fall 2012

Camera processing pipeline• read image out from sensor —see Sensors lecture• optional: HDR assembly —see Homework 2• color balance —see Color lecture• demosaic• noise processing• color matrix —see Color lecture• tone map

2

Page 3: CS6640 Computational Photography - Cornell University · CS6640 Computational Photography 7. Digital camera processing pipeline 1. Cornell CS6640 Fall 2012 Camera processing pipeline

Cornell CS6640 Fall 2012

Demosaicking• First question: how to spell it

I cite “picnicking”• Each photosite

senses only onecolor

• We need three measurements ateach pixel

3http://www.currentprotocols.com/WileyCDA/CPUnit/refId-ns0204.html

Page 4: CS6640 Computational Photography - Cornell University · CS6640 Computational Photography 7. Digital camera processing pipeline 1. Cornell CS6640 Fall 2012 Camera processing pipeline

Cornell CS6640 Fall 2012

Bayer array• Simple solution: make a half-resolution image

• Want sensor-resolution image? Make up 2/3 of the data!

4

? ? ?? ? ?? ? ?? ? ?? ? ?? ? ?? ? ?? ? ?? ? ?

??

??

??

??

??

??

?

?

?

?

?

?

??????

?

?

?

??????

?

?

?

??????

?

?

?

Page 5: CS6640 Computational Photography - Cornell University · CS6640 Computational Photography 7. Digital camera processing pipeline 1. Cornell CS6640 Fall 2012 Camera processing pipeline

Cornell CS6640 Fall 2012 5

Page 6: CS6640 Computational Photography - Cornell University · CS6640 Computational Photography 7. Digital camera processing pipeline 1. Cornell CS6640 Fall 2012 Camera processing pipeline

Cornell CS6640 Fall 2012 6

Page 7: CS6640 Computational Photography - Cornell University · CS6640 Computational Photography 7. Digital camera processing pipeline 1. Cornell CS6640 Fall 2012 Camera processing pipeline

Cornell CS6640 Fall 2012 6

bayer

Page 8: CS6640 Computational Photography - Cornell University · CS6640 Computational Photography 7. Digital camera processing pipeline 1. Cornell CS6640 Fall 2012 Camera processing pipeline

Cornell CS6640 Fall 2012 6

bayerbayer

Page 9: CS6640 Computational Photography - Cornell University · CS6640 Computational Photography 7. Digital camera processing pipeline 1. Cornell CS6640 Fall 2012 Camera processing pipeline

Cornell CS6640 Fall 2012 7

Page 10: CS6640 Computational Photography - Cornell University · CS6640 Computational Photography 7. Digital camera processing pipeline 1. Cornell CS6640 Fall 2012 Camera processing pipeline

Cornell CS6640 Fall 2012 8

Page 11: CS6640 Computational Photography - Cornell University · CS6640 Computational Photography 7. Digital camera processing pipeline 1. Cornell CS6640 Fall 2012 Camera processing pipeline

Cornell CS6640 Fall 2012 8

bayer

Page 12: CS6640 Computational Photography - Cornell University · CS6640 Computational Photography 7. Digital camera processing pipeline 1. Cornell CS6640 Fall 2012 Camera processing pipeline

Cornell CS6640 Fall 2012 8

bayerbayer

Page 13: CS6640 Computational Photography - Cornell University · CS6640 Computational Photography 7. Digital camera processing pipeline 1. Cornell CS6640 Fall 2012 Camera processing pipeline

Cornell CS6640 Fall 2012

Half-resolution demosaic• Idea 1: treat each block of four pixels as a pixel

Easy to code up in one line of Matlab. But what is wrong with this?

9

Page 14: CS6640 Computational Photography - Cornell University · CS6640 Computational Photography 7. Digital camera processing pipeline 1. Cornell CS6640 Fall 2012 Camera processing pipeline

Cornell CS6640 Fall 2012

Half-resolution demosaic• Idea 1: treat each block of four pixels as a pixel

Easy to code up in one line of Matlab. But what is wrong with this?1. throws away too much resolution

9

Page 15: CS6640 Computational Photography - Cornell University · CS6640 Computational Photography 7. Digital camera processing pipeline 1. Cornell CS6640 Fall 2012 Camera processing pipeline

Cornell CS6640 Fall 2012

Half-resolution demosaic• Idea 1: treat each block of four pixels as a pixel

Easy to code up in one line of Matlab. But what is wrong with this?1. throws away too much resolution2. produces subpixel shifts in color planes!

9

Page 16: CS6640 Computational Photography - Cornell University · CS6640 Computational Photography 7. Digital camera processing pipeline 1. Cornell CS6640 Fall 2012 Camera processing pipeline

Cornell CS6640 Fall 2012 10

Page 17: CS6640 Computational Photography - Cornell University · CS6640 Computational Photography 7. Digital camera processing pipeline 1. Cornell CS6640 Fall 2012 Camera processing pipeline

Cornell CS6640 Fall 2012 10

bayer

Page 18: CS6640 Computational Photography - Cornell University · CS6640 Computational Photography 7. Digital camera processing pipeline 1. Cornell CS6640 Fall 2012 Camera processing pipeline

Cornell CS6640 Fall 2012 10

bayerblock

Page 19: CS6640 Computational Photography - Cornell University · CS6640 Computational Photography 7. Digital camera processing pipeline 1. Cornell CS6640 Fall 2012 Camera processing pipeline

Cornell CS6640 Fall 2012 10

bayerblockbayer

Page 20: CS6640 Computational Photography - Cornell University · CS6640 Computational Photography 7. Digital camera processing pipeline 1. Cornell CS6640 Fall 2012 Camera processing pipeline

Cornell CS6640 Fall 2012 10

bayerblockbayerblock

Page 21: CS6640 Computational Photography - Cornell University · CS6640 Computational Photography 7. Digital camera processing pipeline 1. Cornell CS6640 Fall 2012 Camera processing pipeline

Cornell CS6640 Fall 2012 11

Page 22: CS6640 Computational Photography - Cornell University · CS6640 Computational Photography 7. Digital camera processing pipeline 1. Cornell CS6640 Fall 2012 Camera processing pipeline

Cornell CS6640 Fall 2012 11

bayer

Page 23: CS6640 Computational Photography - Cornell University · CS6640 Computational Photography 7. Digital camera processing pipeline 1. Cornell CS6640 Fall 2012 Camera processing pipeline

Cornell CS6640 Fall 2012 11

bayerblock

Page 24: CS6640 Computational Photography - Cornell University · CS6640 Computational Photography 7. Digital camera processing pipeline 1. Cornell CS6640 Fall 2012 Camera processing pipeline

Cornell CS6640 Fall 2012 11

bayerblockbayer

Page 25: CS6640 Computational Photography - Cornell University · CS6640 Computational Photography 7. Digital camera processing pipeline 1. Cornell CS6640 Fall 2012 Camera processing pipeline

Cornell CS6640 Fall 2012 11

bayerblockbayerblock

Page 26: CS6640 Computational Photography - Cornell University · CS6640 Computational Photography 7. Digital camera processing pipeline 1. Cornell CS6640 Fall 2012 Camera processing pipeline

Cornell CS6640 Fall 2012

Centered half-resolution• Average pixels in groups that all have the same

“center of gravity”avoids major color fringing

12

? ? ?? ? ?? ?? ? ?? ? ?? ? ?? ? ?? ? ?? ? ?

??

??

??

??

??

??

?

?

?

?

?

?

?

????

?

?

?

??????

?

?

?

??????

?

?

?

Page 27: CS6640 Computational Photography - Cornell University · CS6640 Computational Photography 7. Digital camera processing pipeline 1. Cornell CS6640 Fall 2012 Camera processing pipeline

Cornell CS6640 Fall 2012 13

Page 28: CS6640 Computational Photography - Cornell University · CS6640 Computational Photography 7. Digital camera processing pipeline 1. Cornell CS6640 Fall 2012 Camera processing pipeline

Cornell CS6640 Fall 2012 13

bayer

Page 29: CS6640 Computational Photography - Cornell University · CS6640 Computational Photography 7. Digital camera processing pipeline 1. Cornell CS6640 Fall 2012 Camera processing pipeline

Cornell CS6640 Fall 2012 13

bayerblock

Page 30: CS6640 Computational Photography - Cornell University · CS6640 Computational Photography 7. Digital camera processing pipeline 1. Cornell CS6640 Fall 2012 Camera processing pipeline

Cornell CS6640 Fall 2012 13

bayerblockcentered

Page 31: CS6640 Computational Photography - Cornell University · CS6640 Computational Photography 7. Digital camera processing pipeline 1. Cornell CS6640 Fall 2012 Camera processing pipeline

Cornell CS6640 Fall 2012 13

bayerblockcenteredbayer

Page 32: CS6640 Computational Photography - Cornell University · CS6640 Computational Photography 7. Digital camera processing pipeline 1. Cornell CS6640 Fall 2012 Camera processing pipeline

Cornell CS6640 Fall 2012 13

bayerblockcenteredbayerblock

Page 33: CS6640 Computational Photography - Cornell University · CS6640 Computational Photography 7. Digital camera processing pipeline 1. Cornell CS6640 Fall 2012 Camera processing pipeline

Cornell CS6640 Fall 2012 13

bayerblockcenteredbayerblockcentered

Page 34: CS6640 Computational Photography - Cornell University · CS6640 Computational Photography 7. Digital camera processing pipeline 1. Cornell CS6640 Fall 2012 Camera processing pipeline

Cornell CS6640 Fall 2012 14

naïve full-resdcrawnaïve full-resdcraw

Page 35: CS6640 Computational Photography - Cornell University · CS6640 Computational Photography 7. Digital camera processing pipeline 1. Cornell CS6640 Fall 2012 Camera processing pipeline

Cornell CS6640 Fall 2012 14

bayernaïve full-resdcrawnaïve full-resdcraw

Page 36: CS6640 Computational Photography - Cornell University · CS6640 Computational Photography 7. Digital camera processing pipeline 1. Cornell CS6640 Fall 2012 Camera processing pipeline

Cornell CS6640 Fall 2012 14

bayerblocknaïve full-resdcrawnaïve full-resdcraw

Page 37: CS6640 Computational Photography - Cornell University · CS6640 Computational Photography 7. Digital camera processing pipeline 1. Cornell CS6640 Fall 2012 Camera processing pipeline

Cornell CS6640 Fall 2012 14

bayerblockcenterednaïve full-resdcrawnaïve full-resdcraw

Page 38: CS6640 Computational Photography - Cornell University · CS6640 Computational Photography 7. Digital camera processing pipeline 1. Cornell CS6640 Fall 2012 Camera processing pipeline

Cornell CS6640 Fall 2012 14

bayerblockcenterednaïve full-resdcrawbayernaïve full-resdcraw

Page 39: CS6640 Computational Photography - Cornell University · CS6640 Computational Photography 7. Digital camera processing pipeline 1. Cornell CS6640 Fall 2012 Camera processing pipeline

Cornell CS6640 Fall 2012 14

bayerblockcenterednaïve full-resdcrawbayerblocknaïve full-resdcraw

Page 40: CS6640 Computational Photography - Cornell University · CS6640 Computational Photography 7. Digital camera processing pipeline 1. Cornell CS6640 Fall 2012 Camera processing pipeline

Cornell CS6640 Fall 2012 14

bayerblockcenterednaïve full-resdcrawbayerblockcenterednaïve full-resdcraw

Page 41: CS6640 Computational Photography - Cornell University · CS6640 Computational Photography 7. Digital camera processing pipeline 1. Cornell CS6640 Fall 2012 Camera processing pipeline

Cornell CS6640 Fall 2012

Naïve full-resolution interpolation• What if we don’t want to throw away so much sharpness?

15

? ? ?? ?? ? ?? ? ?? ? ?? ?? ? ?? ? ?? ? ?

??

?

??

??

??

??

?

?

?

?

?

?

??

???

?

?

?

??????

?

?

?

??????

?

?

?

Page 42: CS6640 Computational Photography - Cornell University · CS6640 Computational Photography 7. Digital camera processing pipeline 1. Cornell CS6640 Fall 2012 Camera processing pipeline

Cornell CS6640 Fall 2012 16

dcrawdcraw

Page 43: CS6640 Computational Photography - Cornell University · CS6640 Computational Photography 7. Digital camera processing pipeline 1. Cornell CS6640 Fall 2012 Camera processing pipeline

Cornell CS6640 Fall 2012 16

bayerdcrawdcraw

Page 44: CS6640 Computational Photography - Cornell University · CS6640 Computational Photography 7. Digital camera processing pipeline 1. Cornell CS6640 Fall 2012 Camera processing pipeline

Cornell CS6640 Fall 2012 16

bayerblockdcrawdcraw

Page 45: CS6640 Computational Photography - Cornell University · CS6640 Computational Photography 7. Digital camera processing pipeline 1. Cornell CS6640 Fall 2012 Camera processing pipeline

Cornell CS6640 Fall 2012 16

bayerblockcentereddcrawdcraw

Page 46: CS6640 Computational Photography - Cornell University · CS6640 Computational Photography 7. Digital camera processing pipeline 1. Cornell CS6640 Fall 2012 Camera processing pipeline

Cornell CS6640 Fall 2012 16

bayerblockcenterednaïve full-resdcrawdcraw

Page 47: CS6640 Computational Photography - Cornell University · CS6640 Computational Photography 7. Digital camera processing pipeline 1. Cornell CS6640 Fall 2012 Camera processing pipeline

Cornell CS6640 Fall 2012 16

bayerblockcenterednaïve full-resdcrawbayerdcraw

Page 48: CS6640 Computational Photography - Cornell University · CS6640 Computational Photography 7. Digital camera processing pipeline 1. Cornell CS6640 Fall 2012 Camera processing pipeline

Cornell CS6640 Fall 2012 16

bayerblockcenterednaïve full-resdcrawbayerblockdcraw

Page 49: CS6640 Computational Photography - Cornell University · CS6640 Computational Photography 7. Digital camera processing pipeline 1. Cornell CS6640 Fall 2012 Camera processing pipeline

Cornell CS6640 Fall 2012 16

bayerblockcenterednaïve full-resdcrawbayerblockcentereddcraw

Page 50: CS6640 Computational Photography - Cornell University · CS6640 Computational Photography 7. Digital camera processing pipeline 1. Cornell CS6640 Fall 2012 Camera processing pipeline

Cornell CS6640 Fall 2012 16

bayerblockcenterednaïve full-resdcrawbayerblockcenterednaïve full-resdcraw

Page 51: CS6640 Computational Photography - Cornell University · CS6640 Computational Photography 7. Digital camera processing pipeline 1. Cornell CS6640 Fall 2012 Camera processing pipeline

Cornell CS6640 Fall 2012 17

Page 52: CS6640 Computational Photography - Cornell University · CS6640 Computational Photography 7. Digital camera processing pipeline 1. Cornell CS6640 Fall 2012 Camera processing pipeline

Cornell CS6640 Fall 2012 17

bayer

Page 53: CS6640 Computational Photography - Cornell University · CS6640 Computational Photography 7. Digital camera processing pipeline 1. Cornell CS6640 Fall 2012 Camera processing pipeline

Cornell CS6640 Fall 2012 17

bayerblock

Page 54: CS6640 Computational Photography - Cornell University · CS6640 Computational Photography 7. Digital camera processing pipeline 1. Cornell CS6640 Fall 2012 Camera processing pipeline

Cornell CS6640 Fall 2012 17

bayerblockcentered

Page 55: CS6640 Computational Photography - Cornell University · CS6640 Computational Photography 7. Digital camera processing pipeline 1. Cornell CS6640 Fall 2012 Camera processing pipeline

Cornell CS6640 Fall 2012 17

bayerblockcenterednaïve full-res

Page 56: CS6640 Computational Photography - Cornell University · CS6640 Computational Photography 7. Digital camera processing pipeline 1. Cornell CS6640 Fall 2012 Camera processing pipeline

Cornell CS6640 Fall 2012 17

bayerblockcenterednaïve full-resbayer

Page 57: CS6640 Computational Photography - Cornell University · CS6640 Computational Photography 7. Digital camera processing pipeline 1. Cornell CS6640 Fall 2012 Camera processing pipeline

Cornell CS6640 Fall 2012 17

bayerblockcenterednaïve full-resbayerblock

Page 58: CS6640 Computational Photography - Cornell University · CS6640 Computational Photography 7. Digital camera processing pipeline 1. Cornell CS6640 Fall 2012 Camera processing pipeline

Cornell CS6640 Fall 2012 17

bayerblockcenterednaïve full-resbayerblockcentered

Page 59: CS6640 Computational Photography - Cornell University · CS6640 Computational Photography 7. Digital camera processing pipeline 1. Cornell CS6640 Fall 2012 Camera processing pipeline

Cornell CS6640 Fall 2012 17

bayerblockcenterednaïve full-resbayerblockcenterednaïve full-res

Page 60: CS6640 Computational Photography - Cornell University · CS6640 Computational Photography 7. Digital camera processing pipeline 1. Cornell CS6640 Fall 2012 Camera processing pipeline

slide by Frédo Durand, M

IT

Results of simple linear

Page 61: CS6640 Computational Photography - Cornell University · CS6640 Computational Photography 7. Digital camera processing pipeline 1. Cornell CS6640 Fall 2012 Camera processing pipeline

slide by Frédo Durand, M

IT

Results - not perfect

Page 62: CS6640 Computational Photography - Cornell University · CS6640 Computational Photography 7. Digital camera processing pipeline 1. Cornell CS6640 Fall 2012 Camera processing pipeline

slide by Frédo Durand, M

IT

Questions?

Page 63: CS6640 Computational Photography - Cornell University · CS6640 Computational Photography 7. Digital camera processing pipeline 1. Cornell CS6640 Fall 2012 Camera processing pipeline

slide by Frédo Durand, M

IT

The problem• Imagine a black-on-white corner• Let’s focus on the green channel for now

??

??

??

??

??

??

?

?

?

?

?

?

Page 64: CS6640 Computational Photography - Cornell University · CS6640 Computational Photography 7. Digital camera processing pipeline 1. Cornell CS6640 Fall 2012 Camera processing pipeline

slide by Frédo Durand, M

IT

The problem• Imagine a black-on-white corner

??

??

??

??

??

??

?

?

?

?

?

?

0 0 10 0 1

0 0 10 0 1

1 1 11 1 1

Page 65: CS6640 Computational Photography - Cornell University · CS6640 Computational Photography 7. Digital camera processing pipeline 1. Cornell CS6640 Fall 2012 Camera processing pipeline

slide by Frédo Durand, M

IT

The problem• Imagine a black-on-white corner

??

??

??

??

??

??

?

?

?

?

?

?

0 0 10 0 1

0 0 10 0 1

1 1 11 1 1

00

0

1

.25.75

.25

.751

0

.25

1

1

.1

1

0 0 10 0 1

0 0 10 0 1

1 1 11 1 1

.25 .75.75

Page 66: CS6640 Computational Photography - Cornell University · CS6640 Computational Photography 7. Digital camera processing pipeline 1. Cornell CS6640 Fall 2012 Camera processing pipeline

slide by Frédo Durand, M

IT

The problem• Imagine a black-on-white corner

??

??

??

??

??

??

?

?

?

?

?

?

0 0 10 0 1

0 0 10 0 1

1 1 11 1 1

00

0

1

.25.75

.25

.751

0

.25

1

1

.1

1

0 0 10 0 1

0 0 10 0 1

1 1 11 1 1

.25 .75.75

Page 67: CS6640 Computational Photography - Cornell University · CS6640 Computational Photography 7. Digital camera processing pipeline 1. Cornell CS6640 Fall 2012 Camera processing pipeline

slide by Frédo Durand, M

IT

Yep, that’s what we saw

Page 68: CS6640 Computational Photography - Cornell University · CS6640 Computational Photography 7. Digital camera processing pipeline 1. Cornell CS6640 Fall 2012 Camera processing pipeline

slide by Frédo Durand, M

IT

Green channel

Page 69: CS6640 Computational Photography - Cornell University · CS6640 Computational Photography 7. Digital camera processing pipeline 1. Cornell CS6640 Fall 2012 Camera processing pipeline

slide by Frédo Durand, M

IT

Edge-based Demosaicking

Page 70: CS6640 Computational Photography - Cornell University · CS6640 Computational Photography 7. Digital camera processing pipeline 1. Cornell CS6640 Fall 2012 Camera processing pipeline

slide by Frédo Durand, M

IT

Idea• Take into account structure in image

- Here, 1D edges• Interpolate along preferred direction

- In our case, only use 2 neighbors

0 0 10 0 1

0 0 10 0 1

1 1 11 1 1

Page 71: CS6640 Computational Photography - Cornell University · CS6640 Computational Photography 7. Digital camera processing pipeline 1. Cornell CS6640 Fall 2012 Camera processing pipeline

slide by Frédo Durand, M

IT

How do we decide• Look at the similarity of recorded neighbors

- Compare |up-down| and |right-le|- Be smart- See pset 4

• Called edge-based demosaicking

0 0 10 0 1

0 0 10 0 1

1 1 11 1 1

Page 72: CS6640 Computational Photography - Cornell University · CS6640 Computational Photography 7. Digital camera processing pipeline 1. Cornell CS6640 Fall 2012 Camera processing pipeline

slide by Frédo Durand, M

IT

Green channel -- naive

Page 73: CS6640 Computational Photography - Cornell University · CS6640 Computational Photography 7. Digital camera processing pipeline 1. Cornell CS6640 Fall 2012 Camera processing pipeline

slide by Frédo Durand, M

IT

Green channel -- edge-based

Page 74: CS6640 Computational Photography - Cornell University · CS6640 Computational Photography 7. Digital camera processing pipeline 1. Cornell CS6640 Fall 2012 Camera processing pipeline

slide by Frédo Durand, M

IT

Challenge with other channels

Page 75: CS6640 Computational Photography - Cornell University · CS6640 Computational Photography 7. Digital camera processing pipeline 1. Cornell CS6640 Fall 2012 Camera processing pipeline

slide by Frédo Durand, M

IT

Problem• What do we do with red and blue? • We could apply the edge-based principle• But we’re missing more information• But color transitions might be shifted

Page 76: CS6640 Computational Photography - Cornell University · CS6640 Computational Photography 7. Digital camera processing pipeline 1. Cornell CS6640 Fall 2012 Camera processing pipeline

slide by Frédo Durand, M

IT

Example• Black on white corner• Notion of edge-based unclear for pixels in

empty rows or columns

?

?

?

Page 77: CS6640 Computational Photography - Cornell University · CS6640 Computational Photography 7. Digital camera processing pipeline 1. Cornell CS6640 Fall 2012 Camera processing pipeline

slide by Frédo Durand, M

IT

Example• Black on white corner• Even if we imagine we can do some decent job

for each channel

Page 78: CS6640 Computational Photography - Cornell University · CS6640 Computational Photography 7. Digital camera processing pipeline 1. Cornell CS6640 Fall 2012 Camera processing pipeline

slide by Frédo Durand, M

IT

Example• Black on white corner• Even if we imagine we can do some decent job

for each channel• The channels don’t line up

- Because they are not recorded at the same location

Page 79: CS6640 Computational Photography - Cornell University · CS6640 Computational Photography 7. Digital camera processing pipeline 1. Cornell CS6640 Fall 2012 Camera processing pipeline

slide by Frédo Durand, M

IT

• Bad color fringes!

Example

Page 80: CS6640 Computational Photography - Cornell University · CS6640 Computational Photography 7. Digital camera processing pipeline 1. Cornell CS6640 Fall 2012 Camera processing pipeline

slide by Frédo Durand, M

IT

Recall color artifacts

Page 81: CS6640 Computational Photography - Cornell University · CS6640 Computational Photography 7. Digital camera processing pipeline 1. Cornell CS6640 Fall 2012 Camera processing pipeline

slide by Frédo Durand, M

IT

Green-based Demosaicking

Page 82: CS6640 Computational Photography - Cornell University · CS6640 Computational Photography 7. Digital camera processing pipeline 1. Cornell CS6640 Fall 2012 Camera processing pipeline

slide by Frédo Durand, M

IT

Green-based demosaicking• Green is a better color channel

- Twice as many pixels- Oen better SNR- We know how to do edge-based green interpolation

• Do the best job you can and get high resolution from green

• Then use green to guide red & blue interpolation

Page 83: CS6640 Computational Photography - Cornell University · CS6640 Computational Photography 7. Digital camera processing pipeline 1. Cornell CS6640 Fall 2012 Camera processing pipeline

slide by Frédo Durand, M

IT

Interpolate difference to green• Interpolate green

- using e.g. edge-based• For recorded red pixels

- compute R-G• At empty pixels

- Interpolate R-G naively- Add G

• Same for blue

Page 84: CS6640 Computational Photography - Cornell University · CS6640 Computational Photography 7. Digital camera processing pipeline 1. Cornell CS6640 Fall 2012 Camera processing pipeline

slide by Frédo Durand, M

IT

Black on white corner

Page 85: CS6640 Computational Photography - Cornell University · CS6640 Computational Photography 7. Digital camera processing pipeline 1. Cornell CS6640 Fall 2012 Camera processing pipeline

slide by Frédo Durand, M

IT

Measurements

Page 86: CS6640 Computational Photography - Cornell University · CS6640 Computational Photography 7. Digital camera processing pipeline 1. Cornell CS6640 Fall 2012 Camera processing pipeline

slide by Frédo Durand, M

IT

Edge-based green

Page 87: CS6640 Computational Photography - Cornell University · CS6640 Computational Photography 7. Digital camera processing pipeline 1. Cornell CS6640 Fall 2012 Camera processing pipeline

slide by Frédo Durand, M

IT

Red-Green difference• Zero everywhere!

Page 88: CS6640 Computational Photography - Cornell University · CS6640 Computational Photography 7. Digital camera processing pipeline 1. Cornell CS6640 Fall 2012 Camera processing pipeline

slide by Frédo Durand, M

IT

Red-Green interpolation• Easy!

Page 89: CS6640 Computational Photography - Cornell University · CS6640 Computational Photography 7. Digital camera processing pipeline 1. Cornell CS6640 Fall 2012 Camera processing pipeline

slide by Frédo Durand, M

IT

Add back green

Page 90: CS6640 Computational Photography - Cornell University · CS6640 Computational Photography 7. Digital camera processing pipeline 1. Cornell CS6640 Fall 2012 Camera processing pipeline

slide by Frédo Durand, M

IT

Same for blue

Page 91: CS6640 Computational Photography - Cornell University · CS6640 Computational Photography 7. Digital camera processing pipeline 1. Cornell CS6640 Fall 2012 Camera processing pipeline

slide by Frédo Durand, M

IT

Fully naive

Page 92: CS6640 Computational Photography - Cornell University · CS6640 Computational Photography 7. Digital camera processing pipeline 1. Cornell CS6640 Fall 2012 Camera processing pipeline

slide by Frédo Durand, M

IT

Edge-based green, naive red blue

Page 93: CS6640 Computational Photography - Cornell University · CS6640 Computational Photography 7. Digital camera processing pipeline 1. Cornell CS6640 Fall 2012 Camera processing pipeline

slide by Frédo Durand, M

IT

Green-based blue and red

Page 94: CS6640 Computational Photography - Cornell University · CS6640 Computational Photography 7. Digital camera processing pipeline 1. Cornell CS6640 Fall 2012 Camera processing pipeline

slide by Frédo Durand, M

IT

Still not 100% perfect• But will be good enough for pset 4

Page 95: CS6640 Computational Photography - Cornell University · CS6640 Computational Photography 7. Digital camera processing pipeline 1. Cornell CS6640 Fall 2012 Camera processing pipeline

Cornell CS6640 Fall 2012 53

Page 96: CS6640 Computational Photography - Cornell University · CS6640 Computational Photography 7. Digital camera processing pipeline 1. Cornell CS6640 Fall 2012 Camera processing pipeline

Cornell CS6640 Fall 2012 53

bayer

Page 97: CS6640 Computational Photography - Cornell University · CS6640 Computational Photography 7. Digital camera processing pipeline 1. Cornell CS6640 Fall 2012 Camera processing pipeline

Cornell CS6640 Fall 2012 53

bayerblock

Page 98: CS6640 Computational Photography - Cornell University · CS6640 Computational Photography 7. Digital camera processing pipeline 1. Cornell CS6640 Fall 2012 Camera processing pipeline

Cornell CS6640 Fall 2012 53

bayerblockcentered

Page 99: CS6640 Computational Photography - Cornell University · CS6640 Computational Photography 7. Digital camera processing pipeline 1. Cornell CS6640 Fall 2012 Camera processing pipeline

Cornell CS6640 Fall 2012 53

bayerblockcenterednaïve full-res

Page 100: CS6640 Computational Photography - Cornell University · CS6640 Computational Photography 7. Digital camera processing pipeline 1. Cornell CS6640 Fall 2012 Camera processing pipeline

Cornell CS6640 Fall 2012 53

bayerblockcenterednaïve full-resedge-based

Page 101: CS6640 Computational Photography - Cornell University · CS6640 Computational Photography 7. Digital camera processing pipeline 1. Cornell CS6640 Fall 2012 Camera processing pipeline

Cornell CS6640 Fall 2012 53

bayerblockcenterednaïve full-resedge-baseddcraw

Page 102: CS6640 Computational Photography - Cornell University · CS6640 Computational Photography 7. Digital camera processing pipeline 1. Cornell CS6640 Fall 2012 Camera processing pipeline

Cornell CS6640 Fall 2012 53

bayerblockcenterednaïve full-resedge-baseddcrawbayer

Page 103: CS6640 Computational Photography - Cornell University · CS6640 Computational Photography 7. Digital camera processing pipeline 1. Cornell CS6640 Fall 2012 Camera processing pipeline

Cornell CS6640 Fall 2012 53

bayerblockcenterednaïve full-resedge-baseddcrawbayerblock

Page 104: CS6640 Computational Photography - Cornell University · CS6640 Computational Photography 7. Digital camera processing pipeline 1. Cornell CS6640 Fall 2012 Camera processing pipeline

Cornell CS6640 Fall 2012 53

bayerblockcenterednaïve full-resedge-baseddcrawbayerblockcentered

Page 105: CS6640 Computational Photography - Cornell University · CS6640 Computational Photography 7. Digital camera processing pipeline 1. Cornell CS6640 Fall 2012 Camera processing pipeline

Cornell CS6640 Fall 2012 53

bayerblockcenterednaïve full-resedge-baseddcrawbayerblockcenterednaïve full-res

Page 106: CS6640 Computational Photography - Cornell University · CS6640 Computational Photography 7. Digital camera processing pipeline 1. Cornell CS6640 Fall 2012 Camera processing pipeline

Cornell CS6640 Fall 2012 53

bayerblockcenterednaïve full-resedge-baseddcrawbayerblockcenterednaïve full-resedge-based

Page 107: CS6640 Computational Photography - Cornell University · CS6640 Computational Photography 7. Digital camera processing pipeline 1. Cornell CS6640 Fall 2012 Camera processing pipeline

Cornell CS6640 Fall 2012 53

bayerblockcenterednaïve full-resedge-baseddcrawbayerblockcenterednaïve full-resedge-baseddcraw

Page 108: CS6640 Computational Photography - Cornell University · CS6640 Computational Photography 7. Digital camera processing pipeline 1. Cornell CS6640 Fall 2012 Camera processing pipeline

Cornell CS6640 Fall 2012 54

Page 109: CS6640 Computational Photography - Cornell University · CS6640 Computational Photography 7. Digital camera processing pipeline 1. Cornell CS6640 Fall 2012 Camera processing pipeline

Cornell CS6640 Fall 2012 54

bayer

Page 110: CS6640 Computational Photography - Cornell University · CS6640 Computational Photography 7. Digital camera processing pipeline 1. Cornell CS6640 Fall 2012 Camera processing pipeline

Cornell CS6640 Fall 2012 54

bayerblock

Page 111: CS6640 Computational Photography - Cornell University · CS6640 Computational Photography 7. Digital camera processing pipeline 1. Cornell CS6640 Fall 2012 Camera processing pipeline

Cornell CS6640 Fall 2012 54

bayerblockcentered

Page 112: CS6640 Computational Photography - Cornell University · CS6640 Computational Photography 7. Digital camera processing pipeline 1. Cornell CS6640 Fall 2012 Camera processing pipeline

Cornell CS6640 Fall 2012 54

bayerblockcenterednaïve full-res

Page 113: CS6640 Computational Photography - Cornell University · CS6640 Computational Photography 7. Digital camera processing pipeline 1. Cornell CS6640 Fall 2012 Camera processing pipeline

Cornell CS6640 Fall 2012 54

bayerblockcenterednaïve full-resedge-based

Page 114: CS6640 Computational Photography - Cornell University · CS6640 Computational Photography 7. Digital camera processing pipeline 1. Cornell CS6640 Fall 2012 Camera processing pipeline

Cornell CS6640 Fall 2012 54

bayerblockcenterednaïve full-resedge-baseddcraw

Page 115: CS6640 Computational Photography - Cornell University · CS6640 Computational Photography 7. Digital camera processing pipeline 1. Cornell CS6640 Fall 2012 Camera processing pipeline

Cornell CS6640 Fall 2012 54

bayerblockcenterednaïve full-resedge-baseddcrawbayer

Page 116: CS6640 Computational Photography - Cornell University · CS6640 Computational Photography 7. Digital camera processing pipeline 1. Cornell CS6640 Fall 2012 Camera processing pipeline

Cornell CS6640 Fall 2012 54

bayerblockcenterednaïve full-resedge-baseddcrawbayerblock

Page 117: CS6640 Computational Photography - Cornell University · CS6640 Computational Photography 7. Digital camera processing pipeline 1. Cornell CS6640 Fall 2012 Camera processing pipeline

Cornell CS6640 Fall 2012 54

bayerblockcenterednaïve full-resedge-baseddcrawbayerblockcentered

Page 118: CS6640 Computational Photography - Cornell University · CS6640 Computational Photography 7. Digital camera processing pipeline 1. Cornell CS6640 Fall 2012 Camera processing pipeline

Cornell CS6640 Fall 2012 54

bayerblockcenterednaïve full-resedge-baseddcrawbayerblockcenterednaïve full-res

Page 119: CS6640 Computational Photography - Cornell University · CS6640 Computational Photography 7. Digital camera processing pipeline 1. Cornell CS6640 Fall 2012 Camera processing pipeline

Cornell CS6640 Fall 2012 54

bayerblockcenterednaïve full-resedge-baseddcrawbayerblockcenterednaïve full-resedge-based

Page 120: CS6640 Computational Photography - Cornell University · CS6640 Computational Photography 7. Digital camera processing pipeline 1. Cornell CS6640 Fall 2012 Camera processing pipeline

Cornell CS6640 Fall 2012 54

bayerblockcenterednaïve full-resedge-baseddcrawbayerblockcenterednaïve full-resedge-baseddcraw

Page 121: CS6640 Computational Photography - Cornell University · CS6640 Computational Photography 7. Digital camera processing pipeline 1. Cornell CS6640 Fall 2012 Camera processing pipeline

Cornell CS6640 Fall 2012 55

Noise reduction• Users want noise out of their images

shot noise increases as sqrt(exposure)relative shot noise decreases as sqrt(exposure)relative readout noise increases prop. to exposurehigher ISO settings try to reduce relative readout noise

• Basic approach: average together measurements• Be clever, spatially, to avoid blurring image structure

• One approach: bilateral filtermore on this later

• Camera makes all have their proprietary methods

Page 122: CS6640 Computational Photography - Cornell University · CS6640 Computational Photography 7. Digital camera processing pipeline 1. Cornell CS6640 Fall 2012 Camera processing pipeline

Cornell CS6640 Fall 2012 56

worth a look:http://www.dpreview.com/reviews/nikon-d3200/12

dpreview.comcamera noise

comparison tool

Page 123: CS6640 Computational Photography - Cornell University · CS6640 Computational Photography 7. Digital camera processing pipeline 1. Cornell CS6640 Fall 2012 Camera processing pipeline

Cornell CS6640 Fall 2012 56

worth a look:http://www.dpreview.com/reviews/nikon-d3200/12

dpreview.comcamera noise

comparison tool

Page 125: CS6640 Computational Photography - Cornell University · CS6640 Computational Photography 7. Digital camera processing pipeline 1. Cornell CS6640 Fall 2012 Camera processing pipeline

Cornell CS6640 Fall 2012

Camera processing pipeline• read image out from sensor —see Sensors lecture• optional: HDR assembly —see Homework 2• color balance —see Color lecture• demosaic• noise processing• color matrix —see Color lecture• tone map —more on this in the project and papers

58

Page 126: CS6640 Computational Photography - Cornell University · CS6640 Computational Photography 7. Digital camera processing pipeline 1. Cornell CS6640 Fall 2012 Camera processing pipeline

slide by Frédo Durand, M

IT

Links from Frédo- http://www.csee.wvu.edu/~xinl/papers/demosaicing_survey.pdf

• http://www.unc.edu/~rjean/demosaicing/demosaicing.pdf• http://www.pages.drexel.edu/~par24/rawhistogram/

40D_Demosaicing/40D_DemosaicingArtifacts.html• http://www.guillermoluijk.com/tutorial/dcraw/index_en.htm• http://www.cambridgeincolour.com/tutorials/RAW-file-

format.htm• http://www.cambridgeincolour.com/tutorials/camera-

sensors.htm

Page 127: CS6640 Computational Photography - Cornell University · CS6640 Computational Photography 7. Digital camera processing pipeline 1. Cornell CS6640 Fall 2012 Camera processing pipeline

slide by Frédo Durand, M

IT

More• http://www.ists.dartmouth.edu/library/

edf0205.pdf• http://ieeexplore.ieee.org/

iel5/79/30519/01407714.pdf?isnumber=30519&prod=JNL&arnumber=1407714&arSt=+44&ared=+54&arAuthor=Gunturk%2C+B.K.%3B+Glotzbach%2C+J.%3B+Altunbasak%2C+Y.%3B+Schafer%2C+R.W.%3B+Mersereau%2C+R.M.