Cameras and Image Processing
Transcript of Cameras and Image Processing
![Page 1: Cameras and Image Processing](https://reader036.fdocuments.us/reader036/viewer/2022072502/62dcfd3d1f61507f8e021915/html5/thumbnails/1.jpg)
Cameras and Image Processing
Computational Photography
CSE 291
Lecture 2
![Page 2: Cameras and Image Processing](https://reader036.fdocuments.us/reader036/viewer/2022072502/62dcfd3d1f61507f8e021915/html5/thumbnails/2.jpg)
Announcements
• Assignment 1 will be released today
– Due Apr 8, 11:59 PM
CSE 291, Spring 2020 2
![Page 3: Cameras and Image Processing](https://reader036.fdocuments.us/reader036/viewer/2022072502/62dcfd3d1f61507f8e021915/html5/thumbnails/3.jpg)
Traditional photography
• The acquisition of images by recording light (or other electromagnetic radiation) either electronically (using an image sensor) or chemically (using film)
• Typically, a lens is used to focus the light reflected or emitted from objects in the scene being imaged into an image on the light sensitive surface inside a camera during a timed exposure
CSE 291, Spring 2020 3
![Page 4: Cameras and Image Processing](https://reader036.fdocuments.us/reader036/viewer/2022072502/62dcfd3d1f61507f8e021915/html5/thumbnails/4.jpg)
Geometric image formation
• How do 3D world points project to 2D image points?
CSE 291, Spring 2020 4
![Page 5: Cameras and Image Processing](https://reader036.fdocuments.us/reader036/viewer/2022072502/62dcfd3d1f61507f8e021915/html5/thumbnails/5.jpg)
The projective camera
• Extrinsic Parameters: Since camera may not be at the origin, there is a rigid transformation between the world coordinates and the camera coordinates
• Intrinsic parameters: Since scene units (e.g., cm) differ image units (e.g., pixels) and coordinate system may not be centered in image, we capture that with a 3x3 transformation comprised of focal length, principal point, pixel aspect ratio, and skew
=
T
Z
Y
X
w
y
x
parameters extrinsic
by drepresente
tionTransforma Rigid
0100
0010
0001
parameters intrinsic
by drepresente
tionTransforma
3 x 3 4 x 4
CSE 291, Spring 2020 5
![Page 6: Cameras and Image Processing](https://reader036.fdocuments.us/reader036/viewer/2022072502/62dcfd3d1f61507f8e021915/html5/thumbnails/6.jpg)
The reason for lenses
CSE 291, Spring 2020 6
We need light, but big pinholes cause blur
![Page 7: Cameras and Image Processing](https://reader036.fdocuments.us/reader036/viewer/2022072502/62dcfd3d1f61507f8e021915/html5/thumbnails/7.jpg)
Thin lens, image of point
CSE 291, Spring 2020 7
O
All rays passing through lens and starting at Pconverge upon P’
So light gather capability of lens is given the area of the lens and all the rays focus on P’ instead of become blurred like a pinhole
F
P
P’
![Page 8: Cameras and Image Processing](https://reader036.fdocuments.us/reader036/viewer/2022072502/62dcfd3d1f61507f8e021915/html5/thumbnails/8.jpg)
Thin lens, image plane
CSE 291, Spring 2020 8
OF
P
P’
Image Plane
Q’
Q
A price: Whereas the image of P is in focus,the image of Q isn’t.
![Page 9: Cameras and Image Processing](https://reader036.fdocuments.us/reader036/viewer/2022072502/62dcfd3d1f61507f8e021915/html5/thumbnails/9.jpg)
Thin lens, aperture
CSE 291, Spring 2020 9
O
P
P’
Image Plane • Smaller Aperture-> Less Blur
• Pinhole -> No Blur
![Page 10: Cameras and Image Processing](https://reader036.fdocuments.us/reader036/viewer/2022072502/62dcfd3d1f61507f8e021915/html5/thumbnails/10.jpg)
Lens distortion
• Radial distortion
• Tangential distortion
CSE 291, Spring 2020 10
Images of straight lines should be
straight
![Page 11: Cameras and Image Processing](https://reader036.fdocuments.us/reader036/viewer/2022072502/62dcfd3d1f61507f8e021915/html5/thumbnails/11.jpg)
Photometric image formation
• What color is the projected point?
CSE 291, Spring 2020 11
Need bidirectional reflectance distribution function (BRDF) at
point on surface
![Page 12: Cameras and Image Processing](https://reader036.fdocuments.us/reader036/viewer/2022072502/62dcfd3d1f61507f8e021915/html5/thumbnails/12.jpg)
Lighting, reflectance, and shading
CSE 291, Spring 2020 12
BRDF
![Page 13: Cameras and Image Processing](https://reader036.fdocuments.us/reader036/viewer/2022072502/62dcfd3d1f61507f8e021915/html5/thumbnails/13.jpg)
Image acquisition
CSE 291, Spring 2020 13
![Page 14: Cameras and Image Processing](https://reader036.fdocuments.us/reader036/viewer/2022072502/62dcfd3d1f61507f8e021915/html5/thumbnails/14.jpg)
Digitization, one row of image
CSE 291, Spring 2020 14
![Page 15: Cameras and Image Processing](https://reader036.fdocuments.us/reader036/viewer/2022072502/62dcfd3d1f61507f8e021915/html5/thumbnails/15.jpg)
Digitization, whole image
CSE 291, Spring 2020 15
![Page 16: Cameras and Image Processing](https://reader036.fdocuments.us/reader036/viewer/2022072502/62dcfd3d1f61507f8e021915/html5/thumbnails/16.jpg)
Number of quantization levels
CSE 291, Spring 2020 16
256 128
64 32
16 8
4 2
![Page 17: Cameras and Image Processing](https://reader036.fdocuments.us/reader036/viewer/2022072502/62dcfd3d1f61507f8e021915/html5/thumbnails/17.jpg)
Image sensing pipeline
CSE 291, Spring 2020 17
Image processing
![Page 18: Cameras and Image Processing](https://reader036.fdocuments.us/reader036/viewer/2022072502/62dcfd3d1f61507f8e021915/html5/thumbnails/18.jpg)
Image processing
• A discipline in which both the input and output of a process are images
– There are usually other input parameters to the process
CSE 291, Spring 2020 18
![Page 19: Cameras and Image Processing](https://reader036.fdocuments.us/reader036/viewer/2022072502/62dcfd3d1f61507f8e021915/html5/thumbnails/19.jpg)
Demosaicing
CSE 291, Spring 2020 19
Image sensor
Color filter array (CFA)
Bayer pattern
CFA Interpolated (lower case)pixel values
![Page 20: Cameras and Image Processing](https://reader036.fdocuments.us/reader036/viewer/2022072502/62dcfd3d1f61507f8e021915/html5/thumbnails/20.jpg)
Image processing
• Color spaces
• Gamut mapping
• White balancing and color balancing
CSE 291, Spring 2020 20
![Page 21: Cameras and Image Processing](https://reader036.fdocuments.us/reader036/viewer/2022072502/62dcfd3d1f61507f8e021915/html5/thumbnails/21.jpg)
Image coding
• Common standards– Video
• Recommendation ITU-R BT.601 (standard-definition television (SDTV))
• SMPTE standard 240M (precursor to Rec. 709)• Recommendation ITU-R BT.709 (high-definition television
(HDTV))
– Image• sRGB• Adobe RGB• Wide gamut RGB (or Adobe Wide Gamut RGB)• ProPhoto RGB (or reference output medium metric (ROMM)
RGB)
CSE 291, Spring 2020 21
![Page 22: Cameras and Image Processing](https://reader036.fdocuments.us/reader036/viewer/2022072502/62dcfd3d1f61507f8e021915/html5/thumbnails/22.jpg)
Different codings, different pixel values
CSE 291, Spring 2020 22
![Page 23: Cameras and Image Processing](https://reader036.fdocuments.us/reader036/viewer/2022072502/62dcfd3d1f61507f8e021915/html5/thumbnails/23.jpg)
Color specification: chromaticity
• Chromaticity coordinates
– (x, y, z)
where x + y + z = 1
– Usually specified by (x, y)
where z = 1 – x – y
The CIE 1931 color space chromaticity diagram
CSE 291, Spring 2020 23
![Page 24: Cameras and Image Processing](https://reader036.fdocuments.us/reader036/viewer/2022072502/62dcfd3d1f61507f8e021915/html5/thumbnails/24.jpg)
Chromaticities
• Set of chromaticities
– Red
– Green
– Blue
– White (point)
CSE 291, Spring 2020 24
![Page 25: Cameras and Image Processing](https://reader036.fdocuments.us/reader036/viewer/2022072502/62dcfd3d1f61507f8e021915/html5/thumbnails/25.jpg)
Standard illuminantsHue of each white point, calculated with luminance Y = 0.54
CSE 291, Spring 2020 25
![Page 26: Cameras and Image Processing](https://reader036.fdocuments.us/reader036/viewer/2022072502/62dcfd3d1f61507f8e021915/html5/thumbnails/26.jpg)
Chromaticities of common video standards
CSE 291, Spring 2020 26
Color CIE x CIE y CIE z
Red 0.6400 0.3300 0.0300
Green 0.3000 0.6000 0.1000
Blue 0.1500 0.0600 0.7900
White 0.3127 0.3290 0.3583
Recommendation ITU-R BT.709
Color CIE x CIE y CIE z
Red 0.6400 0.3300 0.0300
Green 0.2900 0.6000 0.1100
Blue 0.1500 0.0600 0.7900
White 0.3127 0.3290 0.3583
Recommendation ITU-R BT.601625 lines
Color CIE x CIE y CIE z
Red 0.6300 0.3400 0.0300
Green 0.3100 0.5950 0.0950
Blue 0.1550 0.0700 0.7750
White 0.3127 0.3290 0.3583
Recommendation ITU-R BT.601525 lines
Color CIE x CIE y CIE z
Red 0.6300 0.3400 0.0300
Green 0.3100 0.5950 0.0950
Blue 0.1550 0.0700 0.7750
White 0.3127 0.3290 0.3583
SMPTE standard 240M
D65
D65 D65
D65
![Page 27: Cameras and Image Processing](https://reader036.fdocuments.us/reader036/viewer/2022072502/62dcfd3d1f61507f8e021915/html5/thumbnails/27.jpg)
Chromaticities of common image standards
CSE 291, Spring 2020 27
Color CIE x CIE y CIE z
Red 0.6400 0.3300 0.0300
Green 0.3000 0.6000 0.1000
Blue 0.1500 0.0600 0.7900
White 0.3127 0.3290 0.3583
sRGB
Color CIE x CIE y CIE z
Red 0.7347 0.2653 0.0000
Green 0.1152 0.8264 0.0584
Blue 0.1566 0.0177 0.8257
White 0.3457 0.3585 0.2958
Wide gamut RGB(or Adobe Wide Gamut RGB)
Color CIE x CIE y CIE z
Red 0.7347 0.2653 0.0000
Green 0.1596 0.8404 0.0000
Blue 0.0366 0.0001 0.9633
White 0.3457 0.3585 0.2958
ProPhoto RGB(or reference output media metric (ROMM RGB))
Color CIE x CIE y CIE z
Red 0.6400 0.3300 0.0300
Green 0.2100 0.7100 0.0800
Blue 0.1500 0.0600 0.7900
White 0.3127 0.3290 0.3583
Adobe RGB
D65
D50 D50
D65
![Page 28: Cameras and Image Processing](https://reader036.fdocuments.us/reader036/viewer/2022072502/62dcfd3d1f61507f8e021915/html5/thumbnails/28.jpg)
Chromaticity diagrams
Rec. 709 and sRGB35.9% of visible colors
Adobe RGB52.1% of visible colors
CSE 291, Spring 2020 28
![Page 29: Cameras and Image Processing](https://reader036.fdocuments.us/reader036/viewer/2022072502/62dcfd3d1f61507f8e021915/html5/thumbnails/29.jpg)
Chromaticity diagrams
Rec. 709 and sRGB35.9% of visible colors
Wide gamut RGB77.6% of visible colors
CSE 291, Spring 2020 29
![Page 30: Cameras and Image Processing](https://reader036.fdocuments.us/reader036/viewer/2022072502/62dcfd3d1f61507f8e021915/html5/thumbnails/30.jpg)
Chromaticity diagrams
Rec. 709 and sRGB35.9% of visible colors
ProPhoto RGB90% of visible colors
CSE 291, Spring 2020 30
![Page 31: Cameras and Image Processing](https://reader036.fdocuments.us/reader036/viewer/2022072502/62dcfd3d1f61507f8e021915/html5/thumbnails/31.jpg)
Academy Color Encoding Specification (ACES)
Color CIE x CIE y CIE z
Red 0.73470 0.26530 0.00000
Green 0.00000 1.00000 0.00000
Blue 0.00010 -0.07700 1.07690
White 0.32168 0.33767 0.34065
ACES
Approximately D60
100% of visible colors
CSE 291, Spring 2020 31
![Page 32: Cameras and Image Processing](https://reader036.fdocuments.us/reader036/viewer/2022072502/62dcfd3d1f61507f8e021915/html5/thumbnails/32.jpg)
Nonlinear encoding
• All of these standards use nonlinear encoding (gamma encoding)– Video
• Recommendation ITU-R BT.601 (standard-definition television (SDTV))
• SMPTE standard 240M (precursor to Rec. 709)
• Recommendation ITU-R BT.709 (high-definition television (HDTV))
– Image• sRGB
• Adobe RGB
• Wide gamut RGB (or Adobe Wide Gamut RGB)
• ProPhoto RGB (or reference output medium metric (ROMM) RGB)
CSE 291, Spring 2020 32
![Page 33: Cameras and Image Processing](https://reader036.fdocuments.us/reader036/viewer/2022072502/62dcfd3d1f61507f8e021915/html5/thumbnails/33.jpg)
Nonlinear encoding and conversion to linear
• Typical CRT monitors have a transfer function of gamma = 2.2
• Image and video standards were designed to be directly displayed on CRTs
– Pixel values are encoded to approximate gamma = 2.2
• Nonlinear to linear (floating-point) using a lookup table
• Linear to nonlinear calculation
CSE 291, Spring 2020 33
LinearNonlinear
![Page 34: Cameras and Image Processing](https://reader036.fdocuments.us/reader036/viewer/2022072502/62dcfd3d1f61507f8e021915/html5/thumbnails/34.jpg)
Nonlinear R’G’B’ color space and linear RGB color space
• Example: sRGB
CSE 291, Spring 2020 34
RGB linear
sRGB nonlinear
Slope of sRGB nonlinearin log-log space
![Page 35: Cameras and Image Processing](https://reader036.fdocuments.us/reader036/viewer/2022072502/62dcfd3d1f61507f8e021915/html5/thumbnails/35.jpg)
XYZ color space
• Encompasses all color sensations the average person can experience
• Standard reference– Many other color space definitions are based on XYZ
• Y is luminance• Z is quasi-equal to blue stimulation• X is a linear combination of cone response curves chosen to be
nonnegative• The plane parallel to the XZ plane and that Y lies on contains all
possible chromaticities at that luminance
CSE 291, Spring 2020 35
![Page 36: Cameras and Image Processing](https://reader036.fdocuments.us/reader036/viewer/2022072502/62dcfd3d1f61507f8e021915/html5/thumbnails/36.jpg)
RGB color space to XYZ color space
𝑋𝑌𝑍
=
𝑟𝑋 𝑔𝑋 𝑏𝑋𝑟𝑌 𝑔𝑌 𝑏𝑌𝑟𝑍 𝑔𝑍 𝑏𝑍
𝑅𝐺𝐵
𝑋𝑌𝑍
=
𝑟𝑥(𝑟𝑋 + 𝑟𝑌 + 𝑟𝑍) 𝑔𝑥(𝑔𝑋 + 𝑔𝑌 + 𝑔𝑍) 𝑏𝑥(𝑏𝑋 + 𝑏𝑌 + 𝑏𝑍)𝑟𝑦(𝑟𝑋 + 𝑟𝑌 + 𝑟𝑍) 𝑔𝑦(𝑔𝑋 + 𝑔𝑌 + 𝑔𝑍) 𝑏𝑦(𝑏𝑋 + 𝑏𝑌 + 𝑏𝑍)
𝑟𝑧(𝑟𝑋 + 𝑟𝑌 + 𝑟𝑍) 𝑔𝑧(𝑔𝑋 + 𝑔𝑌 + 𝑔𝑍) 𝑏𝑧(𝑏𝑋 + 𝑏𝑌 + 𝑏𝑍)
𝑅𝐺𝐵
𝑋𝑌𝑍
=
𝑟𝑥 𝑔𝑥 𝑏𝑥𝑟𝑦 𝑔𝑦 𝑏𝑦𝑟𝑧 𝑔𝑧 𝑏𝑧
𝑟𝑋 + 𝑟𝑌 + 𝑟𝑍 0 00 𝑔𝑋 + 𝑔𝑌 + 𝑔𝑍 00 0 𝑏𝑋 + 𝑏𝑌 + 𝑏𝑍
𝑅𝐺𝐵
CSE 291, Spring 2020 36
![Page 37: Cameras and Image Processing](https://reader036.fdocuments.us/reader036/viewer/2022072502/62dcfd3d1f61507f8e021915/html5/thumbnails/37.jpg)
RGB color space to XYZ color space
• Substitute in RGB white and XYZ white
CSE 291, Spring 2020 37
𝑊𝑅𝐺𝐵 =111
𝑊𝑋𝑌𝑍 =
𝑤𝑥𝑤𝑦1𝑤𝑧𝑤𝑦
𝑤𝑥𝑤𝑦1𝑤𝑧𝑤𝑦
=
𝑟𝑥 𝑔𝑥 𝑏𝑥𝑟𝑦 𝑔𝑦 𝑏𝑦𝑟𝑧 𝑔𝑧 𝑏𝑧
𝑟𝑋 + 𝑟𝑌 + 𝑟𝑍 0 00 𝑔𝑋 + 𝑔𝑌 + 𝑔𝑍 00 0 𝑏𝑋 + 𝑏𝑌 + 𝑏𝑍
111
𝑤𝑥
𝑤𝑦
1𝑤𝑧
𝑤𝑦
=
𝑟𝑥 𝑔𝑥 𝑏𝑥𝑟𝑦 𝑔𝑦 𝑏𝑦𝑟𝑧 𝑔𝑧 𝑏𝑧
𝑟𝑋 + 𝑟𝑌 + 𝑟𝑍𝑔𝑋 + 𝑔𝑌 + 𝑔𝑍𝑏𝑋 + 𝑏𝑌 + 𝑏𝑍
, solve for
𝑟𝑋 + 𝑟𝑌 + 𝑟𝑍𝑔𝑋 + 𝑔𝑌 + 𝑔𝑍𝑏𝑋 + 𝑏𝑌 + 𝑏𝑍
M_RGB_to_XYZ =
𝑟𝑥 𝑔𝑥 𝑏𝑥𝑟𝑦 𝑔𝑦 𝑏𝑦𝑟𝑧 𝑔𝑧 𝑏𝑧
𝑟𝑋 + 𝑟𝑌 + 𝑟𝑍 0 00 𝑔𝑋 + 𝑔𝑌 + 𝑔𝑍 00 0 𝑏𝑋 + 𝑏𝑌 + 𝑏𝑍
![Page 38: Cameras and Image Processing](https://reader036.fdocuments.us/reader036/viewer/2022072502/62dcfd3d1f61507f8e021915/html5/thumbnails/38.jpg)
Chromatic adaptation
• Estimating the appearance of a sample under a different illuminant– Convert between different
white points
• LMS color space– Response of the three types of
cones in the human eye• Long, medium, and short
wavelengths
• XYZ to LMS– Bradford transformation
matrix
• Chromatic adaptation– Adaptation matrix
CSE 291, Spring 2020 38
![Page 39: Cameras and Image Processing](https://reader036.fdocuments.us/reader036/viewer/2022072502/62dcfd3d1f61507f8e021915/html5/thumbnails/39.jpg)
Color mapping (using chromaticities)
• Image 1 to image 2– Calculate transformation matrix from image 1 RGB to XYZ
(M_RGB1_to_XYZ1) and transformation matrix from image 2 RGB to XYZ (M_RGB2_to_XYZ2)• White points in XYZ are the same as in RGB
– If white points are the same• M_RGB1_to_RGB2 = inv(M_RGB2_to_XYZ2) * M_RGB1_to_XYZ1
– Else (white points are different, include chromatic adaptation)• Bradford transformation matrix (M_XYZ_to_LMS)• Map white points to LMS and calculate adaptation matrix
(M_LMS1_to_LMS2)• Compose transformation matrices
– M_RGB1_to_RGB2 = inv(M_RGB2_to_XYZ2) * inv(M_XYZ_to_LMS) * M_LMS1_to_LMS2 * M_XYZ_to_LMS * M_RGB1_to_XYZ1
CSE 291, Spring 2020 39
![Page 40: Cameras and Image Processing](https://reader036.fdocuments.us/reader036/viewer/2022072502/62dcfd3d1f61507f8e021915/html5/thumbnails/40.jpg)
Note: luminance Y and luma Y’
• Luminance is calculated from linear RGB
– Y coordinate of XYZ
• Luma is calculated from nonlinear R’G’B’
• Example: sRGBY = 0.21263903 * R + 0.71516871 * G + 0.072192319 * B
Y’ = 0.21263903 * R’ + 0.71516871 * G’ + 0.072192319 * B’
(Coefficients are the middle row of M_RGB_to_XYZ)
CSE 291, Spring 2020 40
![Page 41: Cameras and Image Processing](https://reader036.fdocuments.us/reader036/viewer/2022072502/62dcfd3d1f61507f8e021915/html5/thumbnails/41.jpg)
Remember
• Common video and image standards use nonlinear encoding and may use different chromaticities
• Chromaticities define gamuts that contain a percent of the visible colors– Some contain more than others
• Mapping pixel data between these standards requires mapping to the XYZ color space and possibly the LMS color space
• Luminance is different than luma
CSE 291, Spring 2020 41