Computational Optical Imaging - Optique Numerique --...
-
Upload
truongtram -
Category
Documents
-
view
228 -
download
3
Transcript of Computational Optical Imaging - Optique Numerique --...
Ivo Ihrke / Winter 2013
Computational Optical Imaging -Optique Numerique
-- Noise, Dynamic Range and Color --
Winter 2013
Ivo Ihrke
Ivo Ihrke / Winter 2013
Organizational Issues
I received your email addresses
Course announcements will be send via email
Course webpage at http://giana.mmci.uni-saarland.de/
Teaching -> Computational Optical Imaging
Ivo Ihrke / Winter 2013
Noise
Ivo Ihrke / Winter 2013
Sources of noise
Photon shot noise
Dark current shot noise
Fixed pattern noise
Readout noise
…
[Janesick97]
Ivo Ihrke / Winter 2013
Noise Sources
[Reibel2003]
read noisephoton shot noise
dark current noise
Ivo Ihrke / Winter 2013
Photon shot noise
Variance in number of photons that are counted
they arrive in a Poisson random process
Standard deviation is square root of signal
relative noise decreases with signal
Fundamental limit on photodetector precision!
Can be reduced by averaging multiple exposures.
Ivo Ihrke / Winter 2013
Fixed pattern noise
Caused by variations in component values
Big problem for CMOS sensors
An amp at every pixel, and one for every column
Gain variation (proportional to signal PRNU)
Bias variation (independent of signal – dark current)
Can be partially canceled by correlated double sampling (CDS)
CCD’s transfer all charge to a single output amplifier
Ivo Ihrke / Winter 2013
Dark current
Things besides photons can knock electrons loose in the silicon. These are collected, too.
Highly temperature dependent
doubles every 5-8 degrees C
May be reduced by cooling the sensor.
Proportional to exposure time
Limits exposure durations—eventually, the dark current fills your well capacity.
Ivo Ihrke / Winter 2013
Dark Current Noise
Dark current has fixed pattern noise.
Dark current varies because of irregularities in the silicon.
Dark current has shot noise, too!
dominates in dark areas for long exposures
Mean dark current may be subtracted
but subtracting frames increases shot noise
subtract the average dark current
Dark current is why astronomers chill their image sensors.
Ivo Ihrke / Winter 2013
Peltier Cooling of CMOS chip
[Gary Honis]
Ivo Ihrke / Winter 2013
Thermal Noise
Generated by thermally induced motion of electrons in resistive regions (resistors, transistor channels in strong inversion…)
What does it mean?
Independent of the signal.
Zero mean, white (flat, wide bandwidth)
Another problem for CMOS, not CCD imagers
Dominates at low signal levels
Can limit dynamic range
Ivo Ihrke / Winter 2013
Dark Current Noise – Removal
cooling the chip
noise removal techniques to separate image data from noise
e.g. median filtering
25 s exposure time
uncooled cooled
Ivo Ihrke / Winter 2013
Noise, noise, noise…
Reset (kTC) noise
thermal noise when “resetting” the CMOS photodetector—a big deal, actually.
can be corrected with CDS
Amplifier noise
thermal
spatially non-uniform
1/f noise
non-linearities
Quantization noise
“truncate” analog value to N bits
Ivo Ihrke / Winter 2013
Analog/Digital Conversion
Ivo Ihrke / Winter 2013
Correlated Double Sampling
reduce noise by comparing against a reference charge
Ivo Ihrke / Winter 2013
Combined Noise Model [Reibel2003]
- fixed pattern noise
- readout noise
- thermal dark current shot noise
- photon shot noise
- photo response non-uniformity
- non-linear effects
NLPRNUPSNPSNDSNRFPNN CTOT
2222222
FPN2
R2
DSN2
PSN2
PRNU2
NLC
Ivo Ihrke / Winter 2013
Combined Noise Model [Reibel2003]
- fixed pattern noise (can be calibrated)
- readout noise (CDS)
- thermal dark current shot noise (cooling)
- photon shot noise (multiple exposures)
- photo response non-uniformity (per-pixel gain)
- non-linear effects (can also be calibrated for)
NLPRNUPSNPSNDSNRFPNN CTOT
2222222
FPN2
R2
DSN2
PSN2
PRNU2
NLC
Ivo Ihrke / Winter 2013
Noise Distribution
[Reibel2003]
ADU = Analog-Digital Unit, e.g. 1 ADU = 0.5 e-
Ivo Ihrke / Winter 2013
Digital Images
Images are now numbers (corrupted by noise)
Ivo Ihrke / Winter 2013
Digital
Sensor noise
Dynamic Range
Tone Curve
Recording “Medium”
Monochromatic
Optical
Distortions
Aberrations
Digital Images - Limitations
Ivo Ihrke / Winter 2013
Dynamic Range
Ivo Ihrke / Winter 2013
Dynamic Range
max output swing
noise in the dark
Saturation level – dark current
Dark shot noise + readout noise=
“noise in the dark” is random noise sources that
cannot be corrected with circuit tricks
Photon shot noise and read noise
dr =
Ivo Ihrke / Winter 2013
Dynamic Range of Standard Sensors
ww
w.d
xom
ark
.com
13.5 EVs or f-stops = contrast 11,000:1 = color negative
Ivo Ihrke / Winter 2013
Dependency of Dynamic Range on ISO
“Unity gain” is where 1 digital unit (ADU) equals 1 electron (e-)
This happens at different ISO settings for different sensors
Above that, the gain only increases the voltage before A/D conversion (possibly reducing the relative effect of some of the read noise)
“digital gain” multiplies the digital values
All gain settings beyond unity gain reducedynamic range
Ivo Ihrke / Winter 2013
What is High Dynamic Range (HDR)?
http://en.wikipedia.org/wiki/High-dynamic-range_imaging
Ivo Ihrke / Winter 2013
HDR Acquisition – Exposure Brackets
[Debevec & Malik 97]
Radiance Map Tonemapped HDR Image
Exposure Sequence
Ivo Ihrke / Winter 2013
Shutter Speed
F/stop (aperture)
Neutral Density (ND) Filters
Gain / ISO / Film Speed
(DOF)
(noise)
Ways to vary the exposure
Ivo Ihrke / Winter 2013
Combining the image
dttxlxI ),()(
• scene constant over exposure time (or ND-filter)
radiance
),()( xltxI
• assumes linear response (radiometric calibration!)
have several measurements with different t
Ivo Ihrke / Winter 2013
Combining the image
• introduce a weighting function for the pixels:
• centered at the sensor mean value,
e.g. Gaussian (image data in [0,1])
• compute final image as
2
2
2.0
)5.0)((
))((
xI
exIw
i
i
i
iii
finalxIw
txIxIw
xI))((
/)())((
)(
Ivo Ihrke / Winter 2013
• Important for many vision and graphics algorithms
EIg :1
• Use a color chart with precisely known reflectances.
• Use more camera exposures to fill up the curve.
• Method assumes constant lighting on all patches
• Works best when source is far away (example sunlight).
• Inverse exists (g is monotonic and smooth for all cameras)
Irradiance = const * Reflectance
Pix
el V
alu
es
3.1%9.0%19.8%36.2%59.1%90%
0
255
0 1
g
?
?1g
Radiometric Calibration
Ivo Ihrke / Winter 2013
Response Curve - Practice
Measurement: ColorCalibrationToolbox http://giana.mmci.uni-saarland.de/website-template/software.php
Example: 29 exposures of Gretag-Macbeth color checker
(uses EXIF info - exiftool)
Ivo Ihrke / Winter 2013
Color Calibration Toolbox
Zoom-in
Ivo Ihrke / Winter 2013
Color Calibration Toolbox
Mark the patch rectangle
Ivo Ihrke / Winter 2013
Color Calibration Toolbox
Make sure the patches are properly extracted
Ivo Ihrke / Winter 2013
Color Calibration Toolbox
Verify response curve – the example is for jpg on the Canon 5D mark II
Make sure the samples are fit well
Response curves
(R,G,B)
Inverse response curves
samples
Ivo Ihrke / Winter 2013
Color Calibration Toolbox
Check HDR image
Ivo Ihrke / Winter 2013
Color Calibration Toolbox
How is the curve estimated ?
Variant of Mitsunaga and Nayar, “Radiometric Self Calibration”, CVPR 1999
Polynomial fit to data samples
Variations:
─ enforce monotonicity (derivative > 0)
Prevents “wiggling”
─ enforce passing of curve through (0,0) and (1,1)
map range to range
─ Perform a weighted fit
accounts for sample non-uniformity
Ivo Ihrke / Winter 2013
Response Curve – Take Home Points
Usually linear for RAW images
Don’t rely on it – verify
Usually non-linear for jpg or other compressed/processed formats
Estimation from random images may be unstable
Use well defined target (color checker)
Prefer continuous-curve algorithms, especially for high bit depths
Ivo Ihrke / Winter 2013
Applications - HDR Display
• 47” TFT LCD, LED backlight
• aspect ratio 16:9
• resolution 1920 x 1080
• contrast >1,000,000:1
• brightness 4,000 cd/m2
Images courtesy Dolby
Ivo Ihrke / Winter 2013
Applications - Image Based Lighting
Slid
es b
y P
aul D
ebe
vec
Ivo Ihrke / Winter 2013
EDR and HDR Cameras
Grass Valley Viper
(10 bits log)
Panavision Genesis
(10 bits log)
Spheron
- Scanning
- 26 f-stops
Extended range High dynamic range
Ivo Ihrke / Winter 2013
Super CCD (Fuji)
octagonal grid
elements with differentsensitivity
extended DR
better in low light
Used in consumer products (Finepix)
http://www.henner.info/super_ccd.htm
Ivo Ihrke / Winter 2013
HDRC – Log Encoding
CMOS – pixel amplifier output is logarithmic
U - logarithmic
Ivo Ihrke / Winter 2013
Per-Pixel Exposure Time Controlwww.pixim.com
no pixim with pixim no pixim with pixim
Ivo Ihrke / Winter 2013
Adaptive Dynamic Range Imaging
[Nayar and Branzoi 03]
Ivo Ihrke / Winter 2013
Programmable Imaging
[Nayar
& B
ran
zo
i 0
6]
un
mo
du
late
dm
od
ula
ted
mo
du
latio
n s
ign
al
Ivo Ihrke / Winter 2013
Textbook
• HDR image / video encoding
• capture, display, tone reproduction
• visible difference predictors
• image based lighting, etc.
Ivo Ihrke / Winter 2013
Color
Ivo Ihrke / Winter 2013
Sensing color
Eye has 3 types of color receptors
Therefore we need 3 different spectral sensitivities
sourc
e: K
odak
KA
F-5
101ce
dat
a sh
eet
Ivo Ihrke / Winter 2013
Ways to sense color
Field-sequential color
simplest to implement
only still scenes
Proudkin-Gorskii, 1911
(Library of Congress exhibition)
Ivo Ihrke / Winter 2013
Ways to sense color
Field-sequential color
simplest to implement
only still scenes
Proudkin-Gorskii, 1911
(Library of Congress exhibition)
Ivo Ihrke / Winter 2013
Ways to sense color
Field-sequential color
simplest to implement
only still scenes
Proudkin-Gorskii, 1911
(Library of Congress exhibition)
Ivo Ihrke / Winter 2013
Ways to sense color
Field-sequential color
simplest to implement
only still scenes
Proudkin-Gorskii, 1911
(Library of Congress exhibition)
Ivo Ihrke / Winter 2013
Examples - Prokudin-Gorskij
Self-portrait 1915
Ivo Ihrke / Winter 2013
Photograph 1910, Emir of Bukhara , Prokudin-Gorskii
Examples - Prokudin-Gorskij
Ivo Ihrke / Winter 2013
Examples - Lew Tolstoy
1910 photograph, Sergey Prokudin-Gorskii1887 painting, Ilya Repin
Ivo Ihrke / Winter 2013
Color Wheel
one color channel is captured at one shot
3 times the acquisition time
static images only
Ivo Ihrke / Winter 2013
Liquid Crystal Tunable Filter (LCTF)
Computer controllable spectral filter
VariSpec LCTF
Ivo Ihrke / Winter 2013
Spectral Response of Lyot Stage
)(cos)( 2
max
t
T
T
Relative
Transmissivity:
Waveplate
parameters
(birefringence,
thickness)
Electrically Tunable Birefringence
is implemented by liquid crystal
in Lyot stage
Ivo Ihrke / Winter 2013
Spectral Response of 7 Differently tuned Lyot Stages
using several stages in sequence:
product = result
Ivo Ihrke / Winter 2013
VariSpec spectral curves
Ivo Ihrke / Winter 2013
Ways to sense color – 3-Chip Camera
dichroic mirrors divide light into wavelength bands
does not remove light: excellent quality but expensive
interacts with lens design
problem with polarization
image: Theuwissen
Ivo Ihrke / Winter 2013
Foveon Technology
3 layers capture RGB at the same location
takes advantage of silicon’s wavelength selectivity
light decays at different rates for different wavelengths
multilayer CMOS sensor gets3 different spectral sensitivities
don’t get to choose the curves
Ivo Ihrke / Winter 2013
Ways to sense color
Color filter array
paint each sensor with an individual filter
requires just one chip but loses some spatial resolution
“demosaicing” requires tricky image processing
G R
B G
C M
Y G
primary secondary
Ivo Ihrke / Winter 2013
Demosaicing
bilinear interpolation
sampling theory
edge-directed/pattern-based interpolation
correlation-based
Ivo Ihrke / Winter 2013
Demosaicing
Original image Bilinear interpolationRon Kimmel, http://www.cs.technion.ac.il/~ron/demosaic.html
Ivo Ihrke / Winter 2013
Demosaicing
Ron Kimmel, http://www.cs.technion.ac.il/~ron/demosaic.html
Bilinear interpolation Edge-weighted interpolation
Ivo Ihrke / Winter 2013
Bilinear Interpolation
perform interpolation for each color channel separately
G R
B G
= + +
Ivo Ihrke / Winter 2013
Bilinear Interpolation
G R
B G
= + +
4
3432141223
RRRRR
Ivo Ihrke / Winter 2013
Bilinear Interpolation
G R
B G
= + +
4
3432141223
RRRRR
2
343233
RRR
Ivo Ihrke / Winter 2013
Bilinear Interpolation
set all non-measured values to zero then convolve
G R
B G
= + +
4/
121
242
121
,
BRF 4/
010
141
010
GF
Ivo Ihrke / Winter 2013
Problem: Aliasing
[Alleysson & Suesstrunk05]
Ivo Ihrke / Winter 2013
Problem: Aliasing
[Alleysson & Suesstrunk05]
Ivo Ihrke / Winter 2013
Fourier Space
/4
Ivo Ihrke / Winter 2013
Excessive Blurring
Ivo Ihrke / Winter 2013
Grid Effect
Ivo Ihrke / Winter 2013
Bleak colors
Ivo Ihrke / Winter 2013
Color Aliasing
Ivo Ihrke / Winter 2013
optimize r1 and r2 to gain best separation
Low-pass filter luminance
High-pass filter chrominance (orthogonal filter)
Demultiplex chrominance
Interpolate opponent colors
Add luminance and interpolated colors
[Alleysson & Suesstrunk05]
Ivo Ihrke / Winter 2013
Gradient-based (dcraw)
1.Calculate gradients in 5x5 region
2.Select subset of gradients (below threshold)
3.Average color differences in the determined regions
[Chuang et al. 99]
Ivo Ihrke / Winter 2013
Gradients
“Gradient” S = |1323|3/|818| RRGG
2/|717|2/|919| BBBB
3/|1222|3/|1424| GGGG
Ivo Ihrke / Winter 2013
Regions
selection: “gradient” < 1.5*Min+0.5*(Max-Min)
e.g. {S,W,NE,SE}
S: R = (R13+R23)/2, G = G18, B = (B17+B19)/2
NE: R = (R13+R5)/2, G = (G4+G8+G10+G14)/4, B = B9
…
S
Ivo Ihrke / Winter 2013
Average
Rsum = (Rs + Rw + Rne + Rse)/4
Gsum = (Gs + Gw + Gne + Gse)/4
Bsum = (Bs + Bw + Bne + Bse)/4
average of color differences
G13 = R13 + (Gsum-Rsum); B13 = R13 + (Bsum-Rsum)
Ivo Ihrke / Winter 2013
Demosaicing – Take-home-points
2/3 of your image are just made up!
color resolution is less than image resolution
be careful with spiky BRDFs
combining multiple video frames might help
98% of all demosaicing algorithms are ad-hoc
─ smoothing based on constant hue assumption afterwards
Ivo Ihrke / Winter 2013
White Balance
daylight
flashflourescent
tungsten
capture the spectral
characteristics of
the light source to
assure correct
color reproduction
Ivo Ihrke / Winter 2013
White Balance
Human perception adapts to illumination condition
Practice: division of RGB values
Ivo Ihrke / Winter 2013
White Balance
Camera built-in function
derive scale from white point
sun
incandescent
tungsten
infraredred green blue
ultra violet
wavelength
Ivo Ihrke / Winter 2013
White Balance
Camera built-in function
derive scale from white point
infraredred green blue
ultra violet
wavelength
Ivo Ihrke / Winter 2013
White Balance
Camera built-in function
derive scale from white point
infraredred green blue
ultra violet
wavelength
Ivo Ihrke / Winter 2013
White Balance
Human perception adapts to illumination condition
Practice: division of RGB values
Theory: achieve a “neutral” spectrum
(only works for broad band sources and
broad band reflectance)
Conversion to RGB is an integral over the divided spectrum + linear transformation + gamma
Ivo Ihrke / Winter 2013
Spectrum to Image
do not have spectral display
not a huge problem: humans have only three types of cones (color vision) and one type of rod (night vision)
cones 6-7 million
rods ~120 million
rods more sensitive
Ivo Ihrke / Winter 2013
Color Vision
color vision by cones
significant overlap of the response functions
L = long
M = mid
S = short
Ivo Ihrke / Winter 2013
Color Vision
L ~63%, M ~31%, S ~6% of cones
eye least sensitive to blue, most sensitive to yellowish-green
spectral region outside of support of the response functions cannot be perceived
Ivo Ihrke / Winter 2013
Spectral response of human eye
reproducing color is tricky
color matching experiments
─ use light source with known spectral distribution
(i.e. assume uniform spectral distribution, can e.g. be achieved by normalization) filtered by a narrow band filter
─ additionally, use monochromatic sources
@ 700,546,435 nm
─ let human observers adjust apparent brightness of one of the sources to match the other
Color matching functions
Ivo Ihrke / Winter 2013
Color Spaces
─ RGB matching functions
negative !
Ivo Ihrke / Winter 2013
XYZ space
The CIE (1931) standard observer
Ivo Ihrke / Winter 2013
How to compute a tristimulus image from a spectral representation ?
We have to integrate with the spectrum with the appropriate color matching function
dxlfxIXX ),(ˆ)()(
dxlfxIYY ),(ˆ)()(
dxlfxIZZ ),(ˆ)()(
Ivo Ihrke / Winter 2013
Now to RGB
convert XYZ to RGB
Ivo Ihrke / Winter 2013
Horseshoe Diagram
Ivo Ihrke / Winter 2013
White Point for Different Color Temperatures
Planckian
Locus:
- convert black body
temperature to XYZ and
put intohorseshoe diagram
L_\lambda = spectral radiance [W/m^2/m]
lambda = wavelength [m]
h = Planck’s constant [J.s]
k = Boltzmann constant [J/K]
c = speed of light [m/s]
T = temperature of black body [K]
Ivo Ihrke / Winter 2013
Display Gamut
white point
Ivo Ihrke / Winter 2013
Bibliography
Holst, G. CCD Arrays, Cameras, and Displays. SPIE Optical
Engineering Press, Bellingham, Washington, 1998.
Theuwissen, A. Solid-State Imaging with Charge-Coupled Devices. Kluwer Academic Publishers, Boston, 1995.
Curless, CSE558 lecture notes (UW, Spring 01).
El Gamal et al., EE392b lecture notes (Spring 01).
Several Kodak Application Notes at http://www.kodak.com/global/en/digital/ccd/publications/a
pplicationNotes.jhtml
Reibel et al., CCD or CMOS camera noise characterization, Eur. Phys. J. AP 21, 2003
Ivo Ihrke / Winter 2013
Bibliography
D. Alleysson, S. Suesstrunk: Linear Demosaicinginspired by the Human Visual System, IEEE Trans. on Image Processing, 14(4), 2005.
B. K. Gunturk, Y. Altunbasak, R. M. Mersereau: Color Plane Interpolation Using Alternating Projections, IEEE Trans. on Image Processing, 11(9), 2002.
E. Chang, S. Cheung, D.Y. Pan: Color filter array recovery using a threshold-based variable number of gradients. Proc. SPIE, vol. 3650, pp. 36-43, 1999.
http://www-ise.stanford.edu/~tingchen/algodep/vargra.html
Ivo Ihrke / Winter 2013
Bibliography
Y. Takahashi, K. Hiraki, H. Kikuchi, S. Muaramtsu: Color Demosaicing Using Asymmetric Directional Interpolation and Hue Vector Smoothing, IEICE 20th Workshop on Circuits and Systems, 2008.
R. Kimmel, Demosaicing: Image Reconstruction from Color CCD Samples, IEEE Trans. on Image Processing. Vol. 8, No. 9, 1999.
Boris Ajdin, Matthias B. Hullin, Christian Fuchs, Hans-Peter Seidel, Hendrik P. A. Lensch: Demosaicing by Smoothing along 1D Features. Proc. of IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2008.
Ivo Ihrke / Winter 2013
(ICC –international color consortium)
color management system
capture the properties of all devices
─ camera and lighting
─ monitor settings
─ output properties
common interchange space
sRGB standard as a definition of RGB
ICC Profiles
input device
(e.g. camera)
input profile
profile
connection
space
output device
(e.g. printer)
output profile
display
device
(e.g.
monitor)
monitor
profile
Ivo Ihrke / Winter 2013
profile connection spaces
─ CIELAB (perceptual linear)
─ linear CIEXYZ color space
can be used to create an high dynamic range image in the profile connection space
allows for a color calibrated workflow
ICC Profiles and HDR Image Generation
input device
(e.g. camera)
input profile
profile
connection
space
output device
(e.g. printer)
output profile
...